A Novel Software Trustworthiness Evaluation Strategy via Relationships between Criteria

Abstract

Software trustworthiness evaluation is regarded as a multi-criteria decision-making problem. However, most current software trustworthiness evaluation methods do not consider the relationships between criteria. In this paper, we present a software trustworthiness evaluation strategy via the relationships between criteria. Because the trustworthy attribute degree is evaluated by a criterion, a trustworthy attribute measurement method based on fuzzy theory is first proposed, and the relationships between criteria are described by cooperative and conflicting degrees between criteria. Then, a measure formula for the symmetric substitutivity between criteria is proposed, and the cooperative degree between criteria is taken as the approximation of the symmetric substitutivity between criteria. With the help of the symmetric substitutivity between criteria, the software trustworthiness measurement model obtained by axiomatic approaches is applied to aggregate the degree to which each optional software product meets each objective. Finally, the candidate software products are sorted according to the trustworthiness aggregation results, and the optimal product is obtained from the alternative software products on the basis of the sorting results. The theoretical rationality of the measurement model is validated by proving that it satisfies the desirable properties of software trustworthiness measures, and its effectiveness is demonstrated through a case study.

1. Introduction

Many concepts are related to software credibility, such as software dependability, software trustworthiness, and software quality. Software dependability refers to the ability of software to avoid the frequency and severity of service failure by exceeding the acceptable range [1], with a main focus on the acceptable service failure frequency and service failure severity of specific failure classes in a given use environment. Software quality indicates the ability of software to meet explicit or implicit requirements when used under specified conditions [2], with a main focus on system quality, taking into account the quality in use, usually only with respect to traditional quality attributes, such as correctness, reliability and safety, rarely considering the synthesis of different quality attributes. For example, the quality model defined in ISO/IEC 25010 only considers the measurement of each attribute rather than performing an overall evaluation of software quality according to these measures [2]. Although the McCall quality model [3] and Boehm quality model [4] consider the synthesis of each quality attribute, they only combine each quality attribute value as a measure of quality through a simple weighted average. The software quality models mentioned were introduced at different times and are the products of the evolution of such models. For example, ISO/IEC 25010 is the most recent standardized model, which was introduced in 2010, whereas the McCall model was proposed in 1977, and the Boehm model was proposed in 1978. This evolution continues at present in a slightly different and more specialized direction. Software trustworthiness refers to the extent which the dynamic behavior and results of software systems meet people’s expectations and provide continuous services when disturbed [5]. In addition to traditional quality attributes, the concept of software trustworthiness also attaches importance to new software attributes, such as survivability and controllability, as well as the synthesis of these attributes. Furthermore, software trustworthiness not only considers the objective quality of software but also the recognition of this objective quality in people’s minds. In this paper, we use the above definition of software trustworthiness.

Software trustworthiness is a new concept developed on the basis of many attributes, such as correctness, reliability, security, timeliness, integrity, availability, predictability, survivability, controllability, etc. [5]. It can be characterized by many attributes [6,7,8,9], which are referred to as trustworthy attributes. Because we deal only with software trustworthiness herein, the trustworthy attributes considered in this paper are only non-functional requirements related to software trustworthiness. Attribute-based measures of software trustworthiness typically translate the quantification problem of software trustworthiness into the selection of trustworthy attributes, measures of trustworthy attributes, and the aggregation of trustworthy attribute values [7]. There are three ways to choose trustworthy attributes. Given that different software has different trustworthy requirements, the first method dynamically constructs a software trustworthy attribute model. The second method is to establish the software trustworthy index system in advance. The third class integrates the first two methods. Some of the trustworthy attributes are provided up front, whereas others are provided by the user on demand. Most existing trustworthy attribute measurement methods first decompose high-level attributes into low-level sub-attributes that are easy to measure, with sub-attribute values then synthesized using mathematical tools, such as regression analysis, principal component analysis, and factor analysis [7,10]. The essence of this method is to predict the trustworthiness of attributes according to the internal attribute values. Typical models for the synthesis of trustworthy attribute values include machine learning [11,12,13,14,15], axiomatic approaches [16,17,18,19,20], uncertainty theory [21,22,23,24], system testing [25], the social-to-software framework [26], user feedback [27], heuristic-systematic processing [28], the development framework [29], crowd wisdom [30], etc.

The judgment degrees of trustworthy attributes are assessed by each criterion. Therefore, software trustworthiness evaluation can be regarded as a multi-criteria decision-making (MCDM) problem that consists of a series of criteria. The number of software trustworthy attributes is equal to the number of criteria used to evaluate software trustworthiness. Most existing approaches in MCDM are composed of two phases: (1) the aggregation of the judgements with respect to all criteria and per decision alternative and (2) the rank ordering of the decision alternatives according to the aggregated judgments. In practical decision-making problems, the criteria are often interrelated. For example, high performance and low power consumption are a pair of contradictory criteria. However, most approaches do not refer to the aspect of an explicit modeling of relationships between criteria, which makes the optimal solution obtained through MCDM discounted in use.

In this paper, a software trustworthiness evaluation strategy via the relationships between criteria is proposed. First, a trustworthy attribute measurement method based on fuzzy theory is presented. Then, we refer to the method described in [31,32], and the quantitative relationships between criteria are described by cooperative and conflicting degrees between criteria. Thirdly, the symmetric substitutivity between criteria is formulated, and the cooperative degree between criteria is used as the approximation of the symmetric substitutivity between criteria. By means of symmetric substitutivity between criteria, the software trustworthiness measurement model established by axiomatic approaches is used to aggregate the trustworthy attribute degree. Finally, the candidate software products are sorted on the basis of the results of trustworthiness aggregation, and the optimal product is obtained from the alternative software products according to the sorting results.

There are three main contributions in this work: (1) A novel trustworthy attribute measurement method based on fuzzy theory is given, which can more reasonably convert the fuzzy decision about trustworthy attribute to the determined real number. (2) The formal definitions of the quantitative relationships between criteria and the symmetric substitutivity between criteria are proposed, and a link between these two definitions is established. (3) A software trustworthiness evaluation model via the relationships between criteria is presented, which can not only be applied for the ranking of candidate software on trustworthiness, but also for the trustworthiness measurement of candidate software. Meanwhile, we theoretically validate this model by proving that it complies with the properties introduced in [19], and empirically verify it through a case study.

The rest of this paper is organized as follows. Section 2 describes the related work. A trustworthy attribute measurement method based on fuzzy set is given in Section 3. Section 4 proposes a software trustworthiness evaluation strategy via relationships between criteria. A case study is introduced to show the effectiveness of the presented methods in Section 5. Discussion and limitations are presented in Section 6. The conclusions and future work come in the last section.

2. Related Work

In this section some typical software trustworthiness models are selected for a detailed introduction.

Machine learning is widely used to solve complex problems in engineering applications and scientific fields, and it is also utilized to evaluate the software trustworthiness. Yuan et al. propose a partition consistency measurement method for applying software trustworthiness measurement in dynamic behavior feature datasets [11]. The dynamic behavior feature datasets are generated while the software is running. This method compares the datasets with the static attribute feature datasets generated during software testing. Medeiros et al. provide a comprehensive experiment to investigate how to effectively use software metrics to distinguish vulnerable code units from non-vulnerable code units [12]. To conduct this, they use Random Forest, Extreme Boosting, Decision Tree, SVM Linear, and SVM Radial to extract vulnerability-related knowledge from software metrics collected from the source code of Mozilla Firefox, Linux Kernel, Apache HTTPd, Xen, and Glibc. Lv et al., present an automatic online assessment of trustworthiness of cyber-physical system [13]. An evaluation framework based on machine learning knowledge is established, and an online ranking algorithm is designed to realize online real-time analysis and evaluation. Xu et al. build a QoS prediction model. In this model, they combine neural networks with matrix factorization to perform non-linear collaborative filtering on the potential feature vectors of users and services [14]. They also have conducted numerous experiments in a large-scale real QoS datasets, and the experimental results demonstrate the effectiveness of their method. Tian et al. build a software trustworthiness evaluation model based on a behavior trajectory matrix [15]. Checkpoints are set in the software behavior trajectory, and binary code is introduced to represent the software behavior trajectory tree. The scene information about the checkpoint is obtained and applied to establish the behavior trajectory matrix, which is used to represent the behavior trajectory. The behavioral trajectory matrix is transformed into gray scale images to train a deep residual network to classify the software behavior.

To more rigorously measure software trustworthiness and theoretically verify the model, Tao et al. evaluate software trustworthiness using axiomatic methods, present the expected properties of software trustworthiness measures, and build a series of models that satisfy these properties [16,17,18,19,20].

There are other classic software trustworthiness measurement models. Muhammad et al. classify software trustworthiness using values assigned to test case results. The rating strategy includes imposed test strategy, completeness of system test execution, test iterations, test case priority, and test case results from each iteration [25]. Yang et al. present a social-to-software framework of software trustworthiness measurement [26]. They first introduce a generalized index loss to unify the identity evidence, basic standard or norms evidence, and ability evidence of trustworthiness results. Then different methods are proposed for the three parts of software trustworthiness. Xu establishes a data-driven trust measurement model on the basis of the perceptual sources [33]. Direct trust calculation is implemented by sensing the relationship between nodes’ own data, and recommended trust calculation is realized by the relationship between neighboring nodes in the monitoring module. Deng et al. propose a software trustworthiness model based on evidence, which is called TDT [34]. The basic idea of building TDT is to distill the main characteristics into key components and continue distilling until the basic facts such as evidence are reached. TDT can be used as a communication means for different stakeholders to reach an agreement on the system attributes in the requirements analysis phase, and can be used for deductive reasoning to verify whether the system achieves credibility in the product verification phase. A real-time trust metric theory on the basis of the non-interference model is put forward by Zhang et al. [35]. In this theory, system calls are processed as atomic operations, and system call sequences are constructed as the actual behavior of the process. The theoretical expected behavior is calculated according to the mutual non-interference relationship between the corresponding security domains in the actual behavior. The trustworthiness of a process is evaluated by determining whether actual and theoretical expected behavior deviate. Wang et al. extract the evidence of software trustworthiness from the following aspects of process entity: behavior and product. They provide a software process trustworthiness model composed of 37 credibility principles, 182 process credibility evidence and 108 product credibility evidence, and they present a software process trustworthiness evaluation method based on this evidence [36,37].

3. Trustworthy Attribute Measurement Method Based on Fuzzy Set

Fuzzy set theory [38,39] provides a good idea for evaluation. Shi et al. apply the fuzzy set theory to evaluate the trustworthy attributes [40]. They first establish the mapping between the trustworthy attribute language variable and the triangular fuzzy number (TFN), then invite the evaluators to construct the fuzzy decision matrix, and finally measure the trustworthy attribute by defuzzification. This approach does not require measuring internal attributes and can model uncertainty or inherent imprecision of expert judgments. In this paper, a similar method is adopted to measure trustworthy attributes. Considering that the greater the software trustworthiness, the more difficult it is to improve the trustworthiness, and the higher the requirements for the trustworthy attribute value. The classification interval is unequal when constructing the mapping between the trustworthy attribute language variable and the triangular fuzzy number. The value interval increasing from the lowest level to the highest level is approximately reduced according to the gold ratio.

The trustworthy attribute is divided into five levels, namely, very low, low, middle, high and very high. The mapping relationship between trustworthy attribute level and TFN is shown in

Table 1. Mapping relationship between trustworthy attribute level and TFN.

Trustworthy Attribute	TFN
Very Low (VL)	(0.0, 0.0, 4.5)
Low (L)	(4.5, 6.0, 7.0)
Middle (M)	(7.0, 8.0, 8.5)
High (H)	(8.5, 9.0, 9.5)
Very High (VH)	(9.5, 10.0, 10.0)

.

Given the set of possible alternatives $A=\left\{a_1,a_2,\cdots a_m\right\}$, the set of criteria $C=\left\{c_1,c_2,\cdots c_n\right\}$ and $l$ evaluators, each evaluator rates the trustworthy attribute with the TFNs given in Table 1, and obtains the fuzzy decision matrix $D^k$ as shown in the following:

$$\begin{array}{l}\begin{array}{cccccc}\hspace{1em}\hspace{1em}& \hspace{1em}& \hspace{0.17em}\hspace{0.17em}{c}_1& \hspace{0.17em}{c}_2& \hspace{0.17em}\hspace{0.17em}\cdots & c_n\end{array}\\ D^k=\begin{array}{c}{a}_1\\ a_2\\ ⋮\\ a_m\end{array}\left[\begin{array}{cccc}{e}_{11}^k& e_{12}^k& \cdots & e_{1n}^k\\ e_{21}^k& e_{22}^k& \cdots & e_{2n}^k\\ ⋮& ⋮& \ddots & ⋮\\ e_{m1}^k& e_{m2}^k& \cdots & e_{mn}^k\end{array}\right]\end{array}$$

where $e_{ij}^k=(l_{ij}^k,m_{ij}^k,u_{ij}^k)$ is a TFN and expresses the fuzzy judgment rating of alternative $a_i$ concerning the criterion $c_j$ given by k-th evaluator. Suppose that each evaluator has the same importance, then the final fuzzy decision matrix $D$ can be determined, which is expressed as:

$$\begin{array}{l}\begin{array}{cccccc}\hspace{1em}\hspace{1em}& \hspace{0.17em}\hspace{0.17em}& \hspace{0.17em}\hspace{0.17em}{c}_1& \hspace{0.17em}{c}_2& \hspace{0.17em}\hspace{0.17em}\cdots & c_n\end{array}\\ D=\begin{array}{c}{a}_1\\ a_2\\ ⋮\\ a_m\end{array}\left[\begin{array}{cccc}{e}_{11}^{}& e_{12}^{}& \cdots & e_{1n}^{}\\ e_{21}^{}& e_{22}^{}& \cdots & e_{2n}^{}\\ ⋮& ⋮& \ddots & ⋮\\ e_{m1}^{}& e_{m2}^{}& \cdots & e_{mn}^{}\end{array}\right]\end{array}$$

where $e_{ij}^{}=(l_{ij}^{},m_{ij}^{},u_{ij}^{})=({\displaystyle \sum _{k=1}^l{l}_{ij}^k}/l,{\displaystyle \sum _{k=1}^l{m}_{ij}^k}/l,{\displaystyle \sum _{k=1}^l{u}_{ij}^k}/l)$.

In order to integrate the trustworthy attribute values with axiomatic approaches in the future, the fuzzy number is needed to convert to the determined real number through defuzzification technology. The graded mean integration representation method presented by Chen and Hsieh [41] is used in this paper. Denote the defuzzification value of $e_{ij}^{}=(l_{ij}^{},m_{ij}^{},u_{ij}^{})$ as ${\mu }_{c_j}(a_i)$, and ${\mu }_{c_j}(a_i)$ is obtained by Equation (1).

$${\mu }_{c_j}(a_i)=\frac{l_{ij}^{}+4m_{ij}^{}+u_{ij}^{}}{6}$$

(1)

${\mu }_{c_j}(a_i)$ is the degree to which the alternative $a_i$ meets criterion $c_j$. Since $0\le l_{ij}^{},m_{ij}^{},u_{ij}^{}\le 10$, then it follows that $0\le {\mu }_{c_j}(a_i)\le 10$.

4. Software Trustworthiness Evaluation Based on Relationships between Criteria

For a given alternative, intuitively speaking, the two criteria are contradictory if the increase (or decrease) in the satisfaction degree of one criterion will lead to the decrease (or increase) in the satisfaction degree of the other criterion. The two criteria promote each other if the increase (or decrease) in the satisfaction degree of one criterion will lead to the increase (or decrease) in the satisfaction degree of the other criterion. The two criteria are independent of each other if the change of the satisfaction degree of one criterion will not lead to the change of the satisfaction degree of the other criterion. On the basis of the above description, the formal definitions of conflicting, cooperative and irrelevant pairs are given, as shown in Definition 1.

Before providing this definition, the notations used in this paper are first presented as follows: Denote the set of possible alternatives as $A=\left\{a_1,a_2,\cdots a_m\right\}$ and the set of criteria as $C=\left\{c_1,c_2,\cdots c_n\right\}$. Suppose ${\mu }_{c_j}(a_i)$ is the degree to which the alternative $a_i$ meets criterion $c_j$, which satisfies $0\le {\mu }_{c_j}(a_i)\le 10$.

Definition 1

(Conflicting, cooperative and irrelevant alternative pairs [31,32]).

Suppose that$c$and$c^{\prime }$are the two criteria, and$A=\left\{a_1,a_2,\cdots a_n\right\}$is a set of alternatives,$\forall a_i,a_j\in A,i\ne j$. A set of conflicting alternative pairs about$c$and$c^{\prime }$is defined as,

$$CF(c,c^{\prime })=\{(a_i,a_j)|({\mu }_c(a_i)-{\mu }_c(a_j))\times ({\mu }_{c^{\prime }}(a_i)-{\mu }_{c^{\prime }}(a_j))<0\}$$

A set of cooperative alternative pairs about$c$and$c^{\prime }$is defined as,

$$CP(c,c\prime )=\{(a_i,a_j)|({\mu }_c(a_i)-{\mu }_c(a_j))\times ({\mu }_{c\prime }(a_i)-{\mu }_{c\prime }(a_j))>0\}$$

A set of irrelevant alternative pairs about$c$and$c^{\prime }$is defined as,

$$IR(c,c^{\prime })=\{(a_i,a_j)|({\mu }_c(a_i)-{\mu }_c(a_j))\times ({\mu }_{c\prime }(a_i)-{\mu }_{c\prime }(a_j))=0\}$$

Hence, for the two criteria $c$ and $c^{\prime }$, the set of pairs of alternatives $A_P(c,c^{\prime })=\{(a_i,a_j)|\forall a_i,a_j\in A\}$ can be divided into three categories: conflicting, cooperative and irrelevant, and it is easy to obtain that

$$A_P(c,c^{\prime })=CF(c,c^{\prime })\cup CP(c,c^{\prime })\cup IR(c,c^{\prime })$$

and

$$CF(c,c^{\prime })\cap CP(c,c^{\prime })\cap IR(c,c^{\prime })=\varnothing$$

Based on Definition 1, we present the conflicting and cooperative degrees between two criteria, as shown in Definition 2.

Definition 2 (Conflicting and cooperative degrees between two criteria).

 

Suppose that$c$and$c^{\prime }$are the two criteria, and$A_P(c,c^{\prime })=\{(a_i,a_j)|\forall a_i,a_j\in A\}$is the set of pairs of alternatives. Let$CF(c,c^{\prime })$denote the set of conflicting pairs about$c$and$c^{\prime }$, and$CP(c,c^{\prime })$represent the set of cooperative pairs about$c$and$c\prime$. The conflicting degree between the two criteria $c$ and $c^{\prime }$, is defined as:

$$cf(c,c^{\prime })=\frac{{\displaystyle \sum _{(a_i,a_j)\in CF(c,c^{\prime })}\left(\left|\frac{{\mu }_c(a_i)-{\mu }_c(a_j)}{{\mu }_c(a_i)}\right|+\left|\frac{{\mu }_{c^{\prime }}(a_i)-{\mu }_{c^{\prime }}(a_j)}{{\mu }_{c^{\prime }}(a_i)}\right|\right)}}{{\displaystyle \sum _{(a_i,a_j)\in A_p(c,c^{\prime })}\left(\left|\frac{{\mu }_c(a_i)-{\mu }_c(a_j)}{{\mu }_c(a_i)}\right|+\left|\frac{{\mu }_{c^{\prime }}(a_i)-{\mu }_{c^{\prime }}(a_j)}{{\mu }_{c^{\prime }}(a_i)}\right|\right)}}$$

The cooperative degree between the two criteria,$c$and$c^{\prime }$, is defined as:

$$cp(c,c^{\prime })=\frac{{\displaystyle \sum _{(a_i,a_j)\in CP(c,c^{\prime })}\left(\left|\frac{{\mu }_c(a_i)-{\mu }_c(a_j)}{{\mu }_c(a_i)}\right|+\left|\frac{{\mu }_{c^{\prime }}(a_i)-{\mu }_{c^{\prime }}(a_j)}{{\mu }_{c^{\prime }}(a_i)}\right|\right)}}{{\displaystyle \sum _{(a_i,a_j)\in A_p(c,c^{\prime })}\left(\left|\frac{{\mu }_c(a_i)-{\mu }_c(a_j)}{{\mu }_c(a_i)}\right|+\left|\frac{{\mu }_{c^{\prime }}(a_i)-{\mu }_{c^{\prime }}(a_j)}{{\mu }_{c^{\prime }}(a_i)}\right|\right)}}$$

In the following, the set of criteria $C=\left\{c_1,c_2,\cdots c_n\right\}$ is divided based on the cooperative degrees. Among all the criteria, the two criteria with the highest cooperative degree are selected as a group, and then the two criteria with the highest cooperative degree are selected from the remaining criteria as a group, and so on until the division is completed. Suppose $n$ is an even number, and the division result is $\left\{c_1,c_2\right\}$ as a group, $\left\{c_3,c_4\right\}$ as a group, …, $\left\{c_{n-1},c_n\right\}$ as a group. When $n$ is odd, the division results are similar, narrowly making $c_n$ itself a group.

The symmetric substitutivity between criteria is used to describe the quantitative change relationships between two criteria when the total criterion remains unchanged and only these two criteria values are changed, which is defined in Definition 3.

Definition 3 (Symmetric substitutivity between criteria).

 

For a given alternative$a$, assume that$c_i$and$c_j$are the two criteria, ${\mu }_{c_j}(a)$is the degree to which the alternative$a$meets criterion$c_j$and$T$is a differentiable function about${\mu }_{c_j}(a)$, the symmetric substitutivity between criteria$c_i$and$c_j$is defined as Equation (2).

$${\sigma }_{ij}=\frac{d\left({\mu }_{c_i}(a)/{\mu }_{c_j}(a)\right)}{d(h_{ij})}\times \frac{h_{ij}}{{\mu }_{c_i}(a)/{\mu }_{c_j}(a)}$$

(2)

where

$$h_{ij}=-\frac{\partial T/\partial {\mu }_{c_j}(a)}{\partial T/\partial {\mu }_{c_i}(a)}=\frac{d{\mu }_{c_i}(a)}{d{\mu }_{c_j}(a)}.$$

Axiomatic approaches formalize the empirical understanding of software attributes through defining ideal metric properties. They can offer precise terms for the software attributes’ quantification. They have been utilized to assess software trustworthiness [16,17,18,19,20,42]. In this paper, a simplified software trustworthiness metric model presented in [42] is applied to aggregate ${\mu }_{c_j}(a)$.

Definition 4

(Simplified software trustworthiness metric model given in [42]).

The simplified software trustworthiness metric model used in [41] is shown as Equation (3).

$$\begin{array}{c}T={\left({\left(\frac{{\alpha }_1}{{\alpha }_1+{\alpha }_2}{\left({\mu }_{c_1}(a)\right)}^{-{\rho }_{12}}+\frac{{\alpha }_2}{{\alpha }_1+{\alpha }_2}{\left({\mu }_{c_2}(a)\right)}^{-{\rho }_{12}}\right)}^{-\frac{1}{{\rho }_{12}}}\right)}^{{\alpha }_1+{\alpha }_2}\times {\left(\hspace{0.17em}{\left(\frac{{\alpha }_3}{{\alpha }_3+{\alpha }_4}{\left({\mu }_{c_3}(a)\right)}^{-{\rho }_{34}}+\frac{{\alpha }_4}{{\alpha }_3+{\alpha }_4}{\left({\mu }_{c_4}(a)\right)}^{-{\rho }_{34}}\right)}^{-\frac{1}{{\rho }_{34}}}\right)}^{{\alpha }_3+{\alpha }_4}\\ \hspace{1em}\\ \times \cdots \times {\left(\hspace{0.17em}{\left(\frac{{\alpha }_{n-1}}{{\alpha }_{n-1}+{\alpha }_n}{\left({\mu }_{c_{n-1}}(a)\right)}^{-{\rho }_{n-1,n}}+\frac{{\alpha }_{n-1}}{{\alpha }_{n-1}+{\alpha }_n}{\left({\mu }_{c_n}(a)\right)}^{-{\rho }_{n-1,n}}\right)}^{-\frac{1}{{\rho }_{n-1,n}}}\right)}^{{\alpha }_{n-1}+{\alpha }_n}\\ \hspace{1em}\hspace{0.17em}\hspace{1em}\end{array}$$

(3)

where

(1)

$a$is a possible alternative;

(2)

$n$is the number of criteria;

(3)

$c_j(1\le j\le n)$are the criteria;

(4)

${\mu }_{c_j}(a)$is the degree to which the alternative$a$satisfies criterion$c_j$, such that$0\le {\mu }_{c_j}(a)\le 10$;

(5)

${\alpha }_j$represents the weight of criterion$c_j$with$0\le {\alpha }_j\le 1$and${\displaystyle \sum _{j=1}^m{\alpha }_j}=1$;

(6)

${\rho }_{i,i+1}(1\le i\le n-1)$are parameters related to the symmetric substitutivity between the criteria$c_i$and$c_{i+1}$;

(7)

$T$is the software trustworthiness measure function of${\mu }_{c_1}(a),\dots ,{\mu }_{c_n}(a)$.

By Equation (2), the symmetric substitutivity between criteria $c_i$ and $c_j$ can be determined as follows:

$$\begin{array}{c}{\sigma }_{12}=\frac{1}{1+{\rho }_{12}}\\ {\sigma }_{34}=\frac{1}{1+{\rho }_{34}}\\ \dots \dots \\ {\sigma }_{n-1,n}=\frac{1}{1+{\rho }_{n-1,n}}\hspace{0.17em}.\end{array}$$

Using the cooperative degree between criteria $c_i$ and $c_j$ as an approximation of the symmetric substitutivity between criteria $c_i$ and $c_j$, we can determine

$$\begin{array}{c}{\sigma }_{12}=\frac{1}{1+{\rho }_{12}}=cp(c_1,c_2)\\ {\sigma }_{34}=\frac{1}{1+{\rho }_{34}}=cp(c_3,c_4)\\ \dots \dots \\ {\sigma }_{n-1,n}=\frac{1}{1+{\rho }_{n-1,n}}\hspace{0.17em}=cp(c_{n-1},c_n).\end{array}$$

It follows that

$$\begin{array}{c}{\rho }_{12}=\frac{1}{cp(c_1,c_2)}-1\\ {\rho }_{34}=\frac{1}{cp(c_3,c_4)}-1\\ \dots \dots \\ {\rho }_{n-1,n}=\frac{1}{cp(c_{n-1},c_n)}-1.\end{array}$$

Then the following software trustworthiness metric model based on relationships between criteria can be obtained.

Definition 5 (Software trustworthiness metric model based on relationships between criteria).

 

Software trustworthiness metric model based on relationships between criteria is defined as Equation (4).

$$\begin{array}{c}T={\left({\left(\frac{{\alpha }_1}{{\alpha }_1+{\alpha }_2}{\left({\mu }_{c_1}(a)\right)}^{-\left(\frac{1}{cp(c_1,c_2)}-1\right)}+\frac{{\alpha }_2}{{\alpha }_1+{\alpha }_2}{\left({\mu }_{c_2}(a)\right)}^{-\left(\frac{1}{cp(c_1,c_2)}-1\right)}\right)}^{-\frac{1}{\frac{1}{cp(c_1,c_2)}-1}}\right)}^{{\alpha }_1+{\alpha }_2}\times \\ \hspace{1em}\hspace{0.17em}{\left({\left(\frac{{\alpha }_3}{{\alpha }_3+{\alpha }_4}{\left({\mu }_{c_3}(a)\right)}^{-\left(\frac{1}{cp(c_3,c_4)}-1\right)}+\frac{{\alpha }_4}{{\alpha }_3+{\alpha }_4}{\left({\mu }_{c_4}(a)\right)}^{-\left(\frac{1}{cp(c_3,c_4)}-1\right)}\right)}^{-\frac{1}{\frac{1}{cp(c_3,c_4)}-1}}\right)}^{{\alpha }_3+{\alpha }_4}\times \\ \dots \dots \times \\ \hspace{1em}\hspace{0.17em}\hspace{1em}\hspace{0.17em}{\left({\left(\frac{{\alpha }_{n-1}}{{\alpha }_{n-1}+{\alpha }_n}{\left({\mu }_{c_{n-1}}(a)\right)}^{-\left(\frac{1}{cp(c_{n-1},c_n)}-1\right)}+\frac{{\alpha }_{n-1}}{{\alpha }_{n-1}+{\alpha }_n}{\left({\mu }_{c_n}(a)\right)}^{-\left(\frac{1}{cp(c_{n-1},c_n)}-1\right)}\right)}^{-\frac{1}{\frac{1}{cp(c_{n-1},c_n)}-1}}\right)}^{{\alpha }_{n-1}+{\alpha }_n}\end{array}$$

(4)

where

• $a$is a possible alternative;
• $n$is the number of criteria;
• $c_j(1\le j\le n)$are the criteria;
• ${\mu }_{c_j}(a)$is the degree to which the alternative$a$satisfies criterion$c_j$, such that$0\le {\mu }_{c_j}(a)\le 10$;
• ${\alpha }_j$represents the weight of criterion$c_j$with$0\le {\alpha }_j\le 1$and${\displaystyle \sum _{j=1}^m{\alpha }_j}=1$;
• $cp(c_i,c_{i+1})(1\le i\le n-1)$are cooperative degrees between criteria$c_i$and$c_{i+1}$;
• $T$is the software trustworthiness measure function of${\mu }_{c_1}(a),\dots ,{\mu }_{c_n}(a)$.

We once presented the expected properties of software trustworthiness measure, including monotonicity, acceleration, sensitivity and substitutivity [16]. Here we prove that the software trustworthiness metric model based on relationships between criteria conforms to these four properties. For convenience, in the following of this paper, let

$$b=\frac{{\alpha }_i}{{\alpha }_i+{\alpha }_{i+1}}{\left({\mu }_{c_i}(a)\right)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}+\frac{{\alpha }_{i+1}}{{\alpha }_i+{\alpha }_{i+1}}{\left({\mu }_{c_{i+1}}(a)\right)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}$$

Proposition 1.

T satisfies monotonicity, i.e.,$\frac{\partial T}{\partial {\mu }_{c_i}(a)}\ge 0$,$1\le i\le n$.

Proof of Proposition 1.

Since

$$\frac{\partial T}{\partial {\mu }_{c_i}(a)}={\alpha }_{i}T{b}^{-1}{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)-1};1\le i\le n$$

and

$$\{\begin{array}{ll}0\le cp(c_i,c_{i+1})\le 1,\hfill & 1\le i\le n-1\hfill \\ 0\le {\mu }_{c_i}(a)\le 10,\hfill & 1\le i\le n\hfill \\ 0\le {\alpha }_i\le 1,\hfill & 1\le i\le n\hfill \end{array}$$

then we can get

$$\frac{\partial T}{\partial {\mu }_{c_i}(a)}\ge 0,1\le i\le n$$

Therefore, T complies with monotonicity. □

Proposition 2.

Acceleration holds for T, i.e.,$\frac{{\partial }^{2}T}{\partial {\mu }_{c_i}{(a)}^2}\le 0$,$1\le i\le n$.

Proof of Proposition 2.

By calculating the second derivative of T with respect to each ${\mu }_{c_i}(a)\hspace{0.17em}(1\le i\le n)$, we can obtain

$$\begin{array}{l}\frac{{\partial }^{2}T}{\partial {\mu }_{c_i}{(a)}^2}={\alpha }_{i}T{b}^{-1}{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)-2}\times \\ \left({\alpha }_i{b}^{-1}{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}+\frac{{\alpha }_i}{{\alpha }_i+{\alpha }_{i+1}}\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)b^{-1}{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)-1}-\frac{1}{cp(c_i,c_{i+1})}\right)\end{array}$$

Due to

$$\begin{array}{l}{\alpha }_i{b}^{-1}{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}=\frac{{\alpha }_i{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}}{\frac{{\alpha }_i}{{\alpha }_i+{\alpha }_{i+1}}{\left({\mu }_{c_i}(a)\right)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}+\frac{{\alpha }_{i+1}}{{\alpha }_i+{\alpha }_{i+1}}{\left({\mu }_{c_{i+1}}(a)\right)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}=({\alpha }_i+{\alpha }_{i+1})\frac{{\alpha }_i{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}}{{\alpha }_i{\left({\mu }_{c_i}(a)\right)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}+{\alpha }_{i+1}{\left({\mu }_{c_{i+1}}(a)\right)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}}\le 1\end{array}$$

and

$$\begin{array}{l}\frac{{\alpha }_i}{{\alpha }_i+{\alpha }_{i+1}}\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)b^{-1}{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)-1}\\ =\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)\frac{\frac{{\alpha }_i}{{\alpha }_i+{\alpha }_{i+1}}{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)-1}}{\frac{{\alpha }_i}{{\alpha }_i+{\alpha }_{i+1}}{\left({\mu }_{c_i}(a)\right)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}+\frac{{\alpha }_{i+1}}{{\alpha }_i+{\alpha }_{i+1}}{\left({\mu }_{c_{i+1}}(a)\right)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}}\\ \le \left(\frac{1}{cp(c_i,c_{i+1})}-1\right)\end{array}$$

it follows that

$$\left({\alpha }_i{b}^{-1}{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}+\frac{{\alpha }_i}{{\alpha }_i+{\alpha }_{i+1}}\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)b^{-1}{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)-1}-\frac{1}{cp(c_i,c_{i+1})}\right)\le 0$$

Notice that

$$\{\begin{array}{ll}0\le cp(c_i,c_{i+1})\le 1,\hfill & 1\le i\le n-1\hfill \\ 0\le {\mu }_{c_i}(a)\le 10,\hfill & 1\le i\le n\hfill \\ 0\le {\alpha }_i\le 1,\hfill & 1\le i\le n\hfill \end{array}$$

Therefore

$${\alpha }_{i}T{b}^{-1}{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)-2}\ge 0$$

Then, for 1 ≤ i ≤ n, we have

$$\frac{{\partial }^{2}T}{\partial {\mu }_{c_i}{(a)}^2}\le 0$$

□

Proposition 3.

T meets sensitivity.

Proof of Proposition 3.

By computing, it is easy to obtain that for 1 ≤ i ≤ n,

$$\frac{\partial T}{\partial {\mu }_{c_i}(a)}\frac{{\mu }_{c_i}(a)}{T}={\alpha }_{i}T{b}^{-1}{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)-1}\frac{{\mu }_{c_i}(a)}{T}={\alpha }_i{b}^{-1}{\mu }_{c_i}{(a)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}$$

Because of

$$\{\begin{array}{ll}0\le cp(c_i,c_{i+1})\le 1,\hfill & 1\le i\le n-1\hfill \\ 0\le {\mu }_{c_i}(a)\le 10,\hfill & 1\le i\le n\hfill \\ 0\le {\alpha }_i\le 1,\hfill & 1\le i\le n\hfill \end{array}$$

it follows that

$$\frac{\partial T}{\partial {\mu }_{c_i}(a)}\frac{{\mu }_{c_i}(a)}{T}\ge 0$$

Then we obtain the conclusion that T is sensitive to ${\mu }_{c_i}(a)$. □

Proposition 4.

T complies with substitutivity.

Proof of Proposition 4.

Because this paper only focuses on the substitutivity of criteria within the same group, therefore only the substitutivity between criteria within the same group is calculated here.

Observe that for 1 ≤ i ≤ n − 1

$$h_{i,i+1}=\frac{{\alpha }_{i+1}{\left({\mu }_{c_{i+1}}(a)\right)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)-1}}{{\alpha }_i{\left({\mu }_{c_i}(a)\right)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)-1}}$$

Then the substitutivity between criteria $c_i$ and $c_{i+1}$ which belong to the i-th group can be derived as

$$\begin{array}{l}{\sigma }_{i,i+1}=\frac{d\left({\mu }_{c_i}(a)/{\mu }_{c_{i+1}}(a)\right)}{d(h_{i,i+1})}\times \frac{h_{i,i+1}}{{\mu }_{c_i}(a)/{\mu }_{c_{i+1}}(a)}\\ \hspace{1em}\hspace{1em}=\frac{{\left({\mu }_{c_{i+1}}(a)\right)}^{-1}d{\mu }_{c_i}(a)-{\mu }_{c_i}(a){\left({\mu }_{c_{i+1}}(a)\right)}^{-2}d{\mu }_{c_{i+1}}(a)}{\frac{{\alpha }_{i+1}}{{\alpha }_i}\frac{1}{cp(c_i,c_{i+1})}{\left({\mu }_{c_i}(a)\right)}^{\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}{\left({\mu }_{c_{i+1}}(a)\right)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)}\left({\left({\mu }_{c_{i+1}}(a)\right)}^{-1}d{\mu }_{c_i}(a)-{\mu }_{c_i}(a){\left({\mu }_{c_{i+1}}(a)\right)}^{-2}d{\mu }_{c_{i+1}}(a)\right)}\times \\ \hspace{1em}\hspace{1em}\hspace{0.17em}\frac{\frac{{\alpha }_{i+1}}{{\alpha }_i}{\left({\mu }_{c_i}(a)\right)}^{\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)+1}{\left({\mu }_{c_{i+1}}(a)\right)}^{-\left(\frac{1}{cp(c_i,c_{i+1})}-1\right)-1}}{{\mu }_{c_i}(a){\left({\mu }_{c_{i+1}}(a)\right)}^{-1}}=cp(c_i,c_{i+1}).\end{array}$$

The proposition follows immediately from what we have proved. □

For the given set of possible alternatives $A=\left\{a_1,a_2,\cdots ,a_m\right\}$, the set of criteria $C=\left\{c_1,c_2,\cdots c_n\right\}$, the set of the weights of the criteria $\left\{{\alpha }_1,{\alpha }_2,\cdots ,{\alpha }_n\right\}$, and the decision matrix about ${\mu }_{c_j}(a_i)\hspace{0.17em}(1\le i\le m,1\le j\le n)$, $CP(c_i,c_j)$ are first calculated according to Definition 1 and $cp(c_i,c_j)$ are computed based on $CP(c_i,c_j)$ and Definition 2. Then, the criteria are grouped through $cp(c_i,c_j)$, the two criteria with the highest cooperative degree are grouped together, and then the two criteria with the highest cooperative degree of the remaining criteria are grouped together, and so on until the division is completed. Assume that $c_i$ and $c_j$ are in the same group, then ${\rho }_{ij}=\frac{1}{cp(c_i,c_j)}-1$. Considering that when $cp(c_i,c_j)=1$, the corresponding ${\rho }_{ij}$ is 0, and the contribution of ${\mu }_{c_i}(a)$, ${\mu }_{c_j}(a)$ to the trustworthiness of the candidate alternative $a$ cannot be reflected, so simple preprocessing is performed. When $cp(c_i,c_j)=1$, the value of the corresponding ${\rho }_{ij}$ is set to 0.01. Finally, the metric model given in Definition 5 is used to evaluate each alternative in $A$, and the measurement results are arranged in descending order. The alternative corresponding to the first measurement result is the optimal alternative.

Without losing generality, assume that n is an even number. The algorithm for software trustworthiness evaluation based on relationships between criteria is given in Algorithm 1.

Algorithm 1 Algorithm for software trustworthiness evaluation based on relationships between criteria: for the given set of possible alternatives $A=\left\{a_1,a_2,\cdots ,a_m\right\}$, the set of criteria $C=\left\{c_1,c_2,\cdots c_n\right\}$, the set of the weights of the criteria $\left\{{\alpha }_1,{\alpha }_2,\cdots ,{\alpha }_n\right\}$, and the decision matrix about ${\mu }_{c_j}(a_i)\hspace{0.17em}(1\le i\le m,1\le j\le n)$, output the optimal alternative of $A=\left\{a_1,a_2,\cdots ,a_m\right\}$.

Input: $A=\left\{a_1,a_2,\cdots ,a_m\right\}$, $C=\left\{c_1,c_2,\cdots c_n\right\}$, $\left\{{\alpha }_1,{\alpha }_2,\cdots ,{\alpha }_n\right\}$, ${\mu }_{c_j}(a_i)\hspace{0.17em}(1\le i\le m,1\le j\le n)$ Output: the optimal alternative of $A=\left\{a_1,a_2,\cdots ,a_m\right\}$ $\mathrm{Let}M=\{(i,j)|1\le i<j\le n\}$; for  $(i,j)\in M$ do Calculate  $CP(c_i,c_j)$ according to Definition 1; end for for $(i,j)\in M$do Compute $cp(c_i,c_j)$$\mathrm{based}\mathrm{on}CP(c_i,c_j)$ and Definition 2; end for while  $M\ne \varnothing$ do $cp=0$; for  $(i,j)\in M$ do if  $cp(c_i,c_j)\ge cp$ then $cp=cp(c_i,c_j)$; end if end for $N=\varnothing$; for  $(i,j)\in M$ do if  $cp(c_i,c_j)=\hspace{0.17em}=cp$ then $N=N\cup \{(i,j)\}$; end if end for $\mathrm{Randomly}\mathrm{select}(i,j)\in N$$\mathrm{and}\mathrm{make}{c}_i$$\mathrm{and}{c}_j$ a group; if  $cp=1$ then ${\rho }_{ij}=0.01$; else ${\rho }_{ij}=\frac{1}{cp}-1$; end if $M=M-\{(i,j)\}$; end while According to the grouping results, substitute ${\rho }_{ij}$ into Equation (3) and obtain Equation (4); for  $i\in \{1,2,\dots ,m\}$ do for  $j\in \{1,2,\dots ,n\}$ do $\mathrm{Substitute}{\mu }_{c_j}(a_i)$$\mathrm{and}{w}_j$ into Equation (4); end for Obtain the software trustworthiness measurement result $T(a_i)$ of $a_i$; end for Sort $\{T(a_1),T(a_2),\cdots ,T(a_m)\}$ in descending order using the quick sorting algorithm; return the alternative corresponding to the first measurement result.

Theorem 1.

The time complexity of Algorithm 1 is $O(n^4+mn)$.

Proof of Theorem 1.

Steps 2–4 are a for loop, and they are applied to calculate the set of cooperative alternative pairs according to Definition 1. The number of loops is $\left|M\right|$. Since $\left|M\right|=n(n-1)/2$, the time complexity of Steps 2–4 is $O(n^2)$. Steps 5–7 are also a for loop, which are used to compute the set of cooperative degrees based on the results of Steps 2–4 and Definition 2. The number of loops is also $\left|M\right|$. Therefore, the time complexity of Steps 5–7 is also $O(n^2)$.

Steps 8–28 are a double nested loop, which are utilized to group the set of criteria $C=\left\{c_1,c_2,\cdots c_n\right\}$ and compute ${\rho }_{ij}$. The number of loops in first while loop is $\left|M\right|$. Steps 10–14 are the first for loop in the second loop, and are used to find the maximum cooperative degree in the set $\{cp(c_i,c_j)|(i,j)\in M\}$, and the number of loops is $\left|M\right|$. Steps 16–20 are the second for loop in the second loop, constructing the set composed of subscripts of maximum cooperative degree in the set $\{cp(c_i,c_j)|(i,j)\in M\}$, and the number of loops is also $\left|M\right|$. Step 21 randomly selects $(i,j)$ from $N$ and makes $c_i$ and $c_j$ a group. Steps 22–26 are an if-else conditional statement, computing ${\rho }_{ij}$, and Step 27 is used to update the loop variable of the while loop. It is easy to determine that the time complexity of Steps 21–27 is $O(1)$ and the time complexity of Steps 9–27 is $2\left|M\right|+O(1)$. Therefore, the time complexity of Steps 8–28 is $M\left(2\left|M\right|+O(1)\right)$. Notice that $\left|M\right|=n(n-1)/2$, and it follows that the time complexity of Steps 8–28 is $O(n^4)$.

Steps 30–35 are also a double nested loop, calculating the software trustworthiness measurement results $\{T(a_1),T(a_2),\cdots ,T(a_m)\}$. The number of loops in the first loop is $m$, and in the second loop is $n$. Consequently, we infer that the time complexity of Steps 30–35 is $O(mn)$. The software trustworthiness measurement results $\{T(a_1),T(a_2),\cdots ,T(a_m)\}$ are sorted in descending order using the quick sorting algorithm in Step 36, and it takes $O(n\log n)$.

In summary, we can obtain the time complexity of Algorithm 1 is $O(n^4+mn)$. □

5. Case Study

With the development of enterprise informatization, Product Lifecycle Management (PLM) software is becoming more and more important to improve the informatization level and core competitiveness of enterprises. Reference [40] presents a software trustworthiness evaluation approach based on combination weights and improved TOPSIS methods, and this approach is applied to evaluate the candidate PLM software trustworthiness for an aircraft equipment manufacturer. The candidate PLM software set consists of three PLM software, and the criterion set is composed of functionality, learnability, operability, co-existence, maintainability and portability. In this section, the method proposed in this paper is used to evaluate this case. Denote the three candidate PLM software as $a_1$, $a_2$,$a_3$ in turn, and functionality, learnability, operability, co-existence, maintainability, and portability as $c_1$, $c_2$, $c_3$, $c_4$, $c_5$, $c_6$. The combination weights of these six criteria ${\alpha }_1$, ${\alpha }_2$, ${\alpha }_3$, ${\alpha }_4$, ${\alpha }_5$, ${\alpha }_6$ obtained through the combination weighting method established in [40] are given in

Table 2. Combination weights of six criteria.

${\mathit{\alpha }}_1$	${\mathit{\alpha }}_2$	${\mathit{\alpha }}_3$	${\mathit{\alpha }}_4$	${\mathit{\alpha }}_5$	${\mathit{\alpha }}_6$
0.239	0.0799	0.1164	0.1997	0.2443	0.1206

. They are the weighted sum of the objective and subjective weights, where the objective weights are calculated by entropy weighting method and the subjective weights are determined by FAHP method.

Four exports $D_1$, $D_2$, $D_3$, $D_4$ are invited to give the fuzzy decision matrix about $a_1$, $a_2$,$a_3$ [36], as given in

Table 3. Fuzzy decision matrix.

	${\mathit{c}}_1$				${\mathit{c}}_2$				${\mathit{c}}_3$				${\mathit{c}}_4$				${\mathit{c}}_5$				${\mathit{c}}_6$
	D1	D2	D3	D4	D1	D2	D3	D4	D1	D2	D3	D4	D1	D2	D3	D4	D1	D2	D3	D4	D1	D2	D3	D4
a1	M	M	M	M	H	M	M	H	H	VH	VH	H	H	M	H	H	H	M	M	L	M	M	M	H
a2	H	H	H	H	M	M	L	H	M	H	H	M	M	M	M	M	VH	VH	H	H	H	H	H	VH
a3	VH	VH	VH	VH	VH	H	H	VH	M	H	H	L	M	L	M	H	H	H	M	M	M	L	L	H

.

Based on the mapping relationship between trustworthy attribute level and TFN defined in Table 1, the exports’ fuzzy decision can be transferred to the corresponding fuzzy numbers, and the integrated fuzzy decision matrix using TFNs can be obtained, as presented in

Table 4. Integrated fuzzy decision matrix.

	${\mathit{c}}_1$	${\mathit{c}}_2$	${\mathit{c}}_3$	${\mathit{c}}_4$	${\mathit{c}}_5$	${\mathit{c}}_6$
$a_1$	(7.000, 8.000, 8.5000)	(7.750, 8.500, 9.000)	(9.000, 9.500, 9.750)	(8.125, 8.750, 9.250)	(6.750, 7.750, 8.375)	(7.375, 8.250, 8.750)
$a_2$	(8.500, 9.000, 9.5000)	(6.750, 7.750, 8.375)	(7.750, 8.500, 9.000)	(9.000, 9.500, 9.750)	(9.000, 9.500, 9.750)	(8.750, 9.250, 9.625)
$a_3$	(9.500, 10.000, 10, 00)	(9.000, 9.500, 9.750)	(7.125, 8.000, 8.625)	(6.750, 7.750, 8.375)	(7.250, 8.500, 9.000)	(6.125, 7.250, 8.000)

.

Through defuzzying the integrated decision matrix with Equation (1), the decision matrix about ${\mu }_{c_j}(a_i)$ can be established as demonstrated in

Table 5. Decision matrix.

${\mathit{\mu }}_{{\mathit{c}}_{\mathit{j}}}({\mathit{a}}_{\mathit{i}})$	${\mathit{c}}_1$	${\mathit{c}}_2$	${\mathit{c}}_3$	${\mathit{c}}_4$	${\mathit{c}}_5$	${\mathit{c}}_6$
$a_1$	7.9167	8.4583	9.4583	8.7292	7.6875	8.1875
$a_2$	9.0000	7.6875	8.4583	9.4583	9.4583	9.2292
$a_3$	9.9167	9.4583	7.9583	7.6875	8.3750	7.1875

, where each element in the matrix is a real number between 0 and 10.

For any of the two criteria $c_i$ and $c_j$ $(1\le i<j\le 6)$, according to Definition 1 and Table 5, calculate the set of cooperative alternative pairs, and the calculation results are shown in Equation (5).

$$\begin{array}{l}CP(c_1,c_2)=\{(a_1,a_3),(a_2,a_3)\}\\ CP(c_1,c_3)=\varnothing \\ CP(c_1,c_4)=\{(a_1,a_2)\}\\ CP(c_1,c_5)=\{(a_1,a_2),(a_1,a_3)\}\\ CP(c_1,c_6)=\{(a_1,a_2)\}\\ CP(c_2,c_3)=\{(a_1,a_2)\}\\ CP(c_2,c_4)=\varnothing \hspace{1em}\hspace{1em}\hspace{1em}\\ CP(c_2,c_5)=\{(a_1,a_3)\}\\ CP(c_2,c_6)=\varnothing \\ CP(c_3,c_4)=\{(a_1,a_3),(a_2,a_3)\}\\ CP(c_3,c_5)=\{(a_2,a_3)\}\\ CP(c_3,c_6)=\{(a_1,a_3),(a_2,a_3)\}\\ CP(c_4,c_5)=\{(a_1,a_2),(a_2,a_3)\}\\ CP(c_4,c_6)=\{(a_1,a_2),(a_1,a_3),(a_2,a_3)\}\\ CP(c_5,c_6)=\{(a_1,a_2),(a_2,a_3)\}\end{array}$$

(5)

For any of the two criteria $c_i$ and $c_j$ $(1\le i<j\le 6)$, according to Definition 2 and the above calculation results, the cooperative degrees $cp(c_i,c_j)$$(1\le i<j\le 6)$ can be determined, as demonstrated in

Table 6. Cooperative degrees between criteria.

$\mathit{c}\mathit{p}({\mathit{c}}_{\mathit{i}},{\mathit{c}}_{\mathit{j}})$	${\mathit{c}}_2$	${\mathit{c}}_3$	${\mathit{c}}_4$	${\mathit{c}}_5$	${\mathit{c}}_6$
$c_1$	0.7551	0	0.2500	0.7662	0.2745
$c_2$		0.2580	0	0.2376	0
$c_3$			0.7348	0.2292	0.7066
$c_4$				0.7468	1
$c_5$					0.7662

.

It can be seen from Table 6 that the maximum value of $cp$ is $cp(c_4,c_6)$, whose value is 1. Therefore, $c_4$ and $c_6$ are selected as the first group. In the remaining criterion, the maximum value of $cp$ is $cp(c_1,c_5)=0.7662$, and $c_1$ and $c_5$ are chosen as the second group. Finally, $c_2$ and $c_3$ are taken as the third group. According to the grouping results, the weights in Table 2 and Definition 5, Equation (6) can be obtained. It should be noted that because $cp(c_4,c_6)=1$, ${\rho }_{46}$ is set to 0.01.

$$\begin{array}{l}T={\left({\left(\frac{0.2390}{0.2390+0.2443}{\left({\mu }_{c_1}(a)\right)}^{-\left(\frac{1}{0.7662}-1\right)}+\frac{0.2443}{0.2390+0.2443}{\left({\mu }_{c_5}(a)\right)}^{-\left(\frac{1}{0.7662}-1\right)}\right)}^{-\frac{1}{\frac{1}{0.7662}-1}}\right)}^{0.2390+0.2443}\times \\ \hspace{1em}\hspace{0.17em}\hspace{0.17em}{\left({\left(\frac{0.1997}{0.1997+0.1206}{\left({\mu }_{c_4}(a)\right)}^{-0.01}+\frac{0.1206}{0.1997+0.1206}{\left({\mu }_{c_6}(a)\right)}^{-0.01}\right)}^{-\frac{1}{0.01}}\right)}^{0.1997+0.1206}\times \hspace{1em}\\ \hspace{1em}\hspace{0.17em}\hspace{0.17em}{\left({\left(\frac{0.0799}{0.0799+0.1164}{\left({\mu }_{c_2}(a)\right)}^{-\left(\frac{1}{0.2580}-1\right)}+\frac{0.1164}{0.0799+0.1164}{\left({\mu }_{c_3}(a)\right)}^{-\left(\frac{1}{0.2580}-1\right)}\right)}^{-\frac{1}{\frac{1}{0.2580}-1}}\right)}^{0.0799+0.1164}\end{array}$$

(6)

Substituting the data in Table 5 into Equation (6), it follows that

$$\begin{array}{l}T(a_1)=8.2505\\ T(a_2)=9.0394\\ T(a_3)=8.4248\end{array}$$

Thus, among the three candidate PLM software, $a_2$ is the optimal software, and the result obtained is consistent with that in [40].

6. Discussion and Limitations

Software trustworthiness is a concept related to human cognition, and uncertainty theory can model human subjectivity well. Therefore, they are often used in software trustworthiness measurements. However, most of the existing methods use uncertainty theory to calculate the weight of trustworthy attributes. For example, references [21] and [22] both use fuzzy comprehensive evaluation methods to evaluate software trustworthiness, but they differ from the methods of selecting trustworthy attribute sets and computing the weights of trustworthy attributes. In reference [21], experts are invited to establish the trustworthy attribute set, and the weights of trustworthy attributes are computed based on the information entropy. In reference [22], the author’s trustworthy attribute set is determined in advance, and the weights of trustworthy attributes are obtained by using rough set theory and expert opinion. Gao et al. present a new weight distribution method by combining fuzzy analytic hierarchy process with standard importance index correlation, and establish a component-based software trustworthiness measurement model according to the four component composition structures on the basis of their weight distribution method [23]. Shi et al. also provide a calculation method of combination weight based on fuzzy analytic hierarchy process and entropy [40].

The fuzzy set theory is adopted by Shi et al. to measure the trustworthy attributes [40]. The mapping between the trustworthy attribute language variable and the triangular fuzzy number is first built, then the evaluators are invited to establish the fuzzy decision matrix, and finally the trustworthy attribute is measured by defuzzification. In this paper, a similar method is applied to measure trustworthy attributes; however, a more reasonable mapping between the trustworthy attribute language variable and the triangular fuzzy number is given. When building this new mapping, it is taken into account that the greater the software trustworthiness, the more difficult it is to improve the trustworthiness, and the higher the requirement for trustworthy attribute values. Shi et al. present a method to utilize an improved TOPSIS based on vertical projection distance [43] to evaluate the software trustworthiness. The advantage of the improved TOPSIS is that the alternative closest to the positive idea solution is farthest from the negative ideal solution. However, the evaluation method given by Shi et al. can only be applied for ranking the candidate software on trustworthiness. The method presented in this paper can not only be used for the ranking of candidate software on trustworthiness, but can also be used for the trustworthiness measurement of candidate software.

It should be noted that the method presented in this paper can be used for software trustworthiness evaluation, but only when multiple candidate software products must exist. Furthermore, the trustworthy attribute measurement method based on fuzzy theory can model the uncertainty or inherent imprecision of experts’ judgment. However, this method only gives the identification measurement of trustworthy attribute in the experts’ mind and does not give the measurement model of trustworthy attribute.

7. Conclusions and Future Work

To begin with, a trustworthy attribute measurement method based on fuzzy theory is presented in this paper, which is composed of the mapping relationship between trustworthy attribute level and TFN and defuzzification technology. This method can more reasonably transform the fuzzy decision of trustworthy attribute into a certain real number. Moreover, a software trustworthiness evaluation strategy based on the relationships between criteria is given, including the quantitative relationships between criteria described by the cooperative degrees between criteria, the symmetric substitutivity between criteria approximated by the cooperative degrees between criteria, and a software trustworthiness measurement model via the relationships between criteria. Lastly, we verify the theoretical rationality of the software trustworthiness measurement model by showing that it satisfies the expected properties of software trustworthiness measure. Meanwhile, the case study shows the effectiveness and practicality of the model. This strategy can be used not only to rank candidate software about trustworthiness, but also for the trustworthiness measure of software.

In the future, we will further improve the trustworthy attribute measurement method based on fuzzy theory and establish the measurement model of trustworthy attribute. We will also study the cooperative degrees between criteria for software in different fields, then estimate the symmetric substitutivity related parameters in Definition 3, and construct software trustworthiness measurement models for different fields. Further optimization of grouping methods based on cooperative degrees is also important work for the future.