All articles published by MDPI are made immediately available worldwide under an open access license. No special
permission is required to reuse all or part of the article published by MDPI, including figures and tables. For
articles published under an open access Creative Common CC BY license, any part of the article may be reused without
permission provided that the original article is clearly cited. For more information, please refer to
https://www.mdpi.com/openaccess.
Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature
Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for
future research directions and describes possible research applications.
Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive
positive feedback from the reviewers.
Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world.
Editors select a small number of articles recently published in the journal that they believe will be particularly
interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the
most exciting work published in the various research areas of the journal.
Software trustworthiness evaluation is regarded as a multi-criteria decision-making problem. However, most current software trustworthiness evaluation methods do not consider the relationships between criteria. In this paper, we present a software trustworthiness evaluation strategy via the relationships between criteria. Because the trustworthy attribute degree is evaluated by a criterion, a trustworthy attribute measurement method based on fuzzy theory is first proposed, and the relationships between criteria are described by cooperative and conflicting degrees between criteria. Then, a measure formula for the symmetric substitutivity between criteria is proposed, and the cooperative degree between criteria is taken as the approximation of the symmetric substitutivity between criteria. With the help of the symmetric substitutivity between criteria, the software trustworthiness measurement model obtained by axiomatic approaches is applied to aggregate the degree to which each optional software product meets each objective. Finally, the candidate software products are sorted according to the trustworthiness aggregation results, and the optimal product is obtained from the alternative software products on the basis of the sorting results. The theoretical rationality of the measurement model is validated by proving that it satisfies the desirable properties of software trustworthiness measures, and its effectiveness is demonstrated through a case study.
Many concepts are related to software credibility, such as software dependability, software trustworthiness, and software quality. Software dependability refers to the ability of software to avoid the frequency and severity of service failure by exceeding the acceptable range [1], with a main focus on the acceptable service failure frequency and service failure severity of specific failure classes in a given use environment. Software quality indicates the ability of software to meet explicit or implicit requirements when used under specified conditions [2], with a main focus on system quality, taking into account the quality in use, usually only with respect to traditional quality attributes, such as correctness, reliability and safety, rarely considering the synthesis of different quality attributes. For example, the quality model defined in ISO/IEC 25010 only considers the measurement of each attribute rather than performing an overall evaluation of software quality according to these measures [2]. Although the McCall quality model [3] and Boehm quality model [4] consider the synthesis of each quality attribute, they only combine each quality attribute value as a measure of quality through a simple weighted average. The software quality models mentioned were introduced at different times and are the products of the evolution of such models. For example, ISO/IEC 25010 is the most recent standardized model, which was introduced in 2010, whereas the McCall model was proposed in 1977, and the Boehm model was proposed in 1978. This evolution continues at present in a slightly different and more specialized direction. Software trustworthiness refers to the extent which the dynamic behavior and results of software systems meet people’s expectations and provide continuous services when disturbed [5]. In addition to traditional quality attributes, the concept of software trustworthiness also attaches importance to new software attributes, such as survivability and controllability, as well as the synthesis of these attributes. Furthermore, software trustworthiness not only considers the objective quality of software but also the recognition of this objective quality in people’s minds. In this paper, we use the above definition of software trustworthiness.
Software trustworthiness is a new concept developed on the basis of many attributes, such as correctness, reliability, security, timeliness, integrity, availability, predictability, survivability, controllability, etc. [5]. It can be characterized by many attributes [6,7,8,9], which are referred to as trustworthy attributes. Because we deal only with software trustworthiness herein, the trustworthy attributes considered in this paper are only non-functional requirements related to software trustworthiness. Attribute-based measures of software trustworthiness typically translate the quantification problem of software trustworthiness into the selection of trustworthy attributes, measures of trustworthy attributes, and the aggregation of trustworthy attribute values [7]. There are three ways to choose trustworthy attributes. Given that different software has different trustworthy requirements, the first method dynamically constructs a software trustworthy attribute model. The second method is to establish the software trustworthy index system in advance. The third class integrates the first two methods. Some of the trustworthy attributes are provided up front, whereas others are provided by the user on demand. Most existing trustworthy attribute measurement methods first decompose high-level attributes into low-level sub-attributes that are easy to measure, with sub-attribute values then synthesized using mathematical tools, such as regression analysis, principal component analysis, and factor analysis [7,10]. The essence of this method is to predict the trustworthiness of attributes according to the internal attribute values. Typical models for the synthesis of trustworthy attribute values include machine learning [11,12,13,14,15], axiomatic approaches [16,17,18,19,20], uncertainty theory [21,22,23,24], system testing [25], the social-to-software framework [26], user feedback [27], heuristic-systematic processing [28], the development framework [29], crowd wisdom [30], etc.
The judgment degrees of trustworthy attributes are assessed by each criterion. Therefore, software trustworthiness evaluation can be regarded as a multi-criteria decision-making (MCDM) problem that consists of a series of criteria. The number of software trustworthy attributes is equal to the number of criteria used to evaluate software trustworthiness. Most existing approaches in MCDM are composed of two phases: (1) the aggregation of the judgements with respect to all criteria and per decision alternative and (2) the rank ordering of the decision alternatives according to the aggregated judgments. In practical decision-making problems, the criteria are often interrelated. For example, high performance and low power consumption are a pair of contradictory criteria. However, most approaches do not refer to the aspect of an explicit modeling of relationships between criteria, which makes the optimal solution obtained through MCDM discounted in use.
In this paper, a software trustworthiness evaluation strategy via the relationships between criteria is proposed. First, a trustworthy attribute measurement method based on fuzzy theory is presented. Then, we refer to the method described in [31,32], and the quantitative relationships between criteria are described by cooperative and conflicting degrees between criteria. Thirdly, the symmetric substitutivity between criteria is formulated, and the cooperative degree between criteria is used as the approximation of the symmetric substitutivity between criteria. By means of symmetric substitutivity between criteria, the software trustworthiness measurement model established by axiomatic approaches is used to aggregate the trustworthy attribute degree. Finally, the candidate software products are sorted on the basis of the results of trustworthiness aggregation, and the optimal product is obtained from the alternative software products according to the sorting results.
There are three main contributions in this work: (1) A novel trustworthy attribute measurement method based on fuzzy theory is given, which can more reasonably convert the fuzzy decision about trustworthy attribute to the determined real number. (2) The formal definitions of the quantitative relationships between criteria and the symmetric substitutivity between criteria are proposed, and a link between these two definitions is established. (3) A software trustworthiness evaluation model via the relationships between criteria is presented, which can not only be applied for the ranking of candidate software on trustworthiness, but also for the trustworthiness measurement of candidate software. Meanwhile, we theoretically validate this model by proving that it complies with the properties introduced in [19], and empirically verify it through a case study.
The rest of this paper is organized as follows. Section 2 describes the related work. A trustworthy attribute measurement method based on fuzzy set is given in Section 3. Section 4 proposes a software trustworthiness evaluation strategy via relationships between criteria. A case study is introduced to show the effectiveness of the presented methods in Section 5. Discussion and limitations are presented in Section 6. The conclusions and future work come in the last section.
2. Related Work
In this section some typical software trustworthiness models are selected for a detailed introduction.
Machine learning is widely used to solve complex problems in engineering applications and scientific fields, and it is also utilized to evaluate the software trustworthiness. Yuan et al. propose a partition consistency measurement method for applying software trustworthiness measurement in dynamic behavior feature datasets [11]. The dynamic behavior feature datasets are generated while the software is running. This method compares the datasets with the static attribute feature datasets generated during software testing. Medeiros et al. provide a comprehensive experiment to investigate how to effectively use software metrics to distinguish vulnerable code units from non-vulnerable code units [12]. To conduct this, they use Random Forest, Extreme Boosting, Decision Tree, SVM Linear, and SVM Radial to extract vulnerability-related knowledge from software metrics collected from the source code of Mozilla Firefox, Linux Kernel, Apache HTTPd, Xen, and Glibc. Lv et al., present an automatic online assessment of trustworthiness of cyber-physical system [13]. An evaluation framework based on machine learning knowledge is established, and an online ranking algorithm is designed to realize online real-time analysis and evaluation. Xu et al. build a QoS prediction model. In this model, they combine neural networks with matrix factorization to perform non-linear collaborative filtering on the potential feature vectors of users and services [14]. They also have conducted numerous experiments in a large-scale real QoS datasets, and the experimental results demonstrate the effectiveness of their method. Tian et al. build a software trustworthiness evaluation model based on a behavior trajectory matrix [15]. Checkpoints are set in the software behavior trajectory, and binary code is introduced to represent the software behavior trajectory tree. The scene information about the checkpoint is obtained and applied to establish the behavior trajectory matrix, which is used to represent the behavior trajectory. The behavioral trajectory matrix is transformed into gray scale images to train a deep residual network to classify the software behavior.
To more rigorously measure software trustworthiness and theoretically verify the model, Tao et al. evaluate software trustworthiness using axiomatic methods, present the expected properties of software trustworthiness measures, and build a series of models that satisfy these properties [16,17,18,19,20].
There are other classic software trustworthiness measurement models. Muhammad et al. classify software trustworthiness using values assigned to test case results. The rating strategy includes imposed test strategy, completeness of system test execution, test iterations, test case priority, and test case results from each iteration [25]. Yang et al. present a social-to-software framework of software trustworthiness measurement [26]. They first introduce a generalized index loss to unify the identity evidence, basic standard or norms evidence, and ability evidence of trustworthiness results. Then different methods are proposed for the three parts of software trustworthiness. Xu establishes a data-driven trust measurement model on the basis of the perceptual sources [33]. Direct trust calculation is implemented by sensing the relationship between nodes’ own data, and recommended trust calculation is realized by the relationship between neighboring nodes in the monitoring module. Deng et al. propose a software trustworthiness model based on evidence, which is called TDT [34]. The basic idea of building TDT is to distill the main characteristics into key components and continue distilling until the basic facts such as evidence are reached. TDT can be used as a communication means for different stakeholders to reach an agreement on the system attributes in the requirements analysis phase, and can be used for deductive reasoning to verify whether the system achieves credibility in the product verification phase. A real-time trust metric theory on the basis of the non-interference model is put forward by Zhang et al. [35]. In this theory, system calls are processed as atomic operations, and system call sequences are constructed as the actual behavior of the process. The theoretical expected behavior is calculated according to the mutual non-interference relationship between the corresponding security domains in the actual behavior. The trustworthiness of a process is evaluated by determining whether actual and theoretical expected behavior deviate. Wang et al. extract the evidence of software trustworthiness from the following aspects of process entity: behavior and product. They provide a software process trustworthiness model composed of 37 credibility principles, 182 process credibility evidence and 108 product credibility evidence, and they present a software process trustworthiness evaluation method based on this evidence [36,37].
3. Trustworthy Attribute Measurement Method Based on Fuzzy Set
Fuzzy set theory [38,39] provides a good idea for evaluation. Shi et al. apply the fuzzy set theory to evaluate the trustworthy attributes [40]. They first establish the mapping between the trustworthy attribute language variable and the triangular fuzzy number (TFN), then invite the evaluators to construct the fuzzy decision matrix, and finally measure the trustworthy attribute by defuzzification. This approach does not require measuring internal attributes and can model uncertainty or inherent imprecision of expert judgments. In this paper, a similar method is adopted to measure trustworthy attributes. Considering that the greater the software trustworthiness, the more difficult it is to improve the trustworthiness, and the higher the requirements for the trustworthy attribute value. The classification interval is unequal when constructing the mapping between the trustworthy attribute language variable and the triangular fuzzy number. The value interval increasing from the lowest level to the highest level is approximately reduced according to the gold ratio.
The trustworthy attribute is divided into five levels, namely, very low, low, middle, high and very high. The mapping relationship between trustworthy attribute level and TFN is shown in Table 1.
Given the set of possible alternatives $A=\left\{{a}_{1},{a}_{2},\cdots {a}_{m}\right\}$, the set of criteria $C=\left\{{c}_{1},{c}_{2},\cdots {c}_{n}\right\}$ and $l$ evaluators, each evaluator rates the trustworthy attribute with the TFNs given in Table 1, and obtains the fuzzy decision matrix ${D}^{k}$ as shown in the following:
where ${e}_{ij}^{k}=({l}_{ij}^{k},{m}_{ij}^{k},{u}_{ij}^{k})$ is a TFN and expresses the fuzzy judgment rating of alternative ${a}_{i}$ concerning the criterion ${c}_{j}$ given by k-th evaluator. Suppose that each evaluator has the same importance, then the final fuzzy decision matrix $D$ can be determined, which is expressed as:
where ${e}_{ij}^{}=({l}_{ij}^{},{m}_{ij}^{},{u}_{ij}^{})=({\displaystyle \sum _{k=1}^{l}{l}_{ij}^{k}}/l,{\displaystyle \sum _{k=1}^{l}{m}_{ij}^{k}}/l,{\displaystyle \sum _{k=1}^{l}{u}_{ij}^{k}}/l)$.
In order to integrate the trustworthy attribute values with axiomatic approaches in the future, the fuzzy number is needed to convert to the determined real number through defuzzification technology. The graded mean integration representation method presented by Chen and Hsieh [41] is used in this paper. Denote the defuzzification value of ${e}_{ij}^{}=({l}_{ij}^{},{m}_{ij}^{},{u}_{ij}^{})$ as ${\mu}_{{c}_{j}}({a}_{i})$, and ${\mu}_{{c}_{j}}({a}_{i})$ is obtained by Equation (1).
${\mu}_{{c}_{j}}({a}_{i})$ is the degree to which the alternative ${a}_{i}$ meets criterion ${c}_{j}$. Since $0\le {l}_{ij}^{},{m}_{ij}^{},{u}_{ij}^{}\le 10$, then it follows that $0\le {\mu}_{{c}_{j}}({a}_{i})\le 10$.
4. Software Trustworthiness Evaluation Based on Relationships between Criteria
For a given alternative, intuitively speaking, the two criteria are contradictory if the increase (or decrease) in the satisfaction degree of one criterion will lead to the decrease (or increase) in the satisfaction degree of the other criterion. The two criteria promote each other if the increase (or decrease) in the satisfaction degree of one criterion will lead to the increase (or decrease) in the satisfaction degree of the other criterion. The two criteria are independent of each other if the change of the satisfaction degree of one criterion will not lead to the change of the satisfaction degree of the other criterion. On the basis of the above description, the formal definitions of conflicting, cooperative and irrelevant pairs are given, as shown in Definition 1.
Before providing this definition, the notations used in this paper are first presented as follows: Denote the set of possible alternatives as $A=\left\{{a}_{1},{a}_{2},\cdots {a}_{m}\right\}$ and the set of criteria as $C=\left\{{c}_{1},{c}_{2},\cdots {c}_{n}\right\}$. Suppose ${\mu}_{{c}_{j}}({a}_{i})$ is the degree to which the alternative ${a}_{i}$ meets criterion ${c}_{j}$, which satisfies $0\le {\mu}_{{c}_{j}}({a}_{i})\le 10$.
Definition1
(Conflicting, cooperative and irrelevant alternative pairs [31,32]).
Suppose that$c$and${c}^{\prime}$are the two criteria, and$A=\left\{{a}_{1},{a}_{2},\cdots {a}_{n}\right\}$is a set of alternatives,$\forall {a}_{i},{a}_{j}\in A,i\ne j$. A set of conflicting alternative pairs about$c$and${c}^{\prime}$is defined as,
Hence, for the two criteria $c$ and ${c}^{\prime}$, the set of pairs of alternatives ${A}_{P}(c,{c}^{\prime})=\{({a}_{i},{a}_{j})|\forall {a}_{i},{a}_{j}\in A\}$ can be divided into three categories: conflicting, cooperative and irrelevant, and it is easy to obtain that
Based on Definition 1, we present the conflicting and cooperative degrees between two criteria, as shown in Definition 2.
Definition2 (Conflicting and cooperative degrees between two criteria).
Suppose that$c$and${c}^{\prime}$are the two criteria, and${A}_{P}(c,{c}^{\prime})=\{({a}_{i},{a}_{j})|\forall {a}_{i},{a}_{j}\in A\}$is the set of pairs of alternatives. Let$CF(c,{c}^{\prime})$denote the set of conflicting pairs about$c$and${c}^{\prime}$, and$CP(c,{c}^{\prime})$represent the set of cooperative pairs about$c$and$c\prime $. The conflicting degree between the two criteria$c$and${c}^{\prime}$, is defined as:
In the following, the set of criteria $C=\left\{{c}_{1},{c}_{2},\cdots {c}_{n}\right\}$ is divided based on the cooperative degrees. Among all the criteria, the two criteria with the highest cooperative degree are selected as a group, and then the two criteria with the highest cooperative degree are selected from the remaining criteria as a group, and so on until the division is completed. Suppose $n$ is an even number, and the division result is $\left\{{c}_{1},{c}_{2}\right\}$ as a group, $\left\{{c}_{3},{c}_{4}\right\}$ as a group, …, $\left\{{c}_{n-1},{c}_{n}\right\}$ as a group. When $n$ is odd, the division results are similar, narrowly making ${c}_{n}$ itself a group.
The symmetric substitutivity between criteria is used to describe the quantitative change relationships between two criteria when the total criterion remains unchanged and only these two criteria values are changed, which is defined in Definition 3.
Definition3 (Symmetric substitutivity between criteria).
For a given alternative$a$, assume that${c}_{i}$and${c}_{j}$are the two criteria,${\mu}_{{c}_{j}}(a)$is the degree to which the alternative$a$meets criterion${c}_{j}$and$T$is a differentiable function about${\mu}_{{c}_{j}}(a)$, the symmetric substitutivity between criteria${c}_{i}$and${c}_{j}$is defined as Equation (2).
Axiomatic approaches formalize the empirical understanding of software attributes through defining ideal metric properties. They can offer precise terms for the software attributes’ quantification. They have been utilized to assess software trustworthiness [16,17,18,19,20,42]. In this paper, a simplified software trustworthiness metric model presented in [42] is applied to aggregate ${\mu}_{{c}_{j}}(a)$.
Definition4
(Simplified software trustworthiness metric model given in [42]).
The simplified software trustworthiness metric model used in [41] is shown as Equation (3).
Using the cooperative degree between criteria ${c}_{i}$ and ${c}_{j}$ as an approximation of the symmetric substitutivity between criteria ${c}_{i}$ and ${c}_{j}$, we can determine
${\mu}_{{c}_{j}}(a)$is the degree to which the alternative$a$satisfies criterion${c}_{j}$, such that$0\le {\mu}_{{c}_{j}}(a)\le 10$;
${\alpha}_{j}$represents the weight of criterion${c}_{j}$with$0\le {\alpha}_{j}\le 1$and$\sum _{j=1}^{m}{\alpha}_{j}}=1$;
$cp({c}_{i},{c}_{i+1})(1\le i\le n-1)$are cooperative degrees between criteria${c}_{i}$and${c}_{i+1}$;
$T$is the software trustworthiness measure function of${\mu}_{{c}_{1}}(a),\dots ,{\mu}_{{c}_{n}}(a)$.
We once presented the expected properties of software trustworthiness measure, including monotonicity, acceleration, sensitivity and substitutivity [16]. Here we prove that the software trustworthiness metric model based on relationships between criteria conforms to these four properties. For convenience, in the following of this paper, let
Then we obtain the conclusion that T is sensitive to ${\mu}_{{c}_{i}}(a)$. □
Proposition4.
T complies with substitutivity.
ProofofProposition4.
Because this paper only focuses on the substitutivity of criteria within the same group, therefore only the substitutivity between criteria within the same group is calculated here.
The proposition follows immediately from what we have proved. □
For the given set of possible alternatives $A=\left\{{a}_{1},{a}_{2},\cdots ,{a}_{m}\right\}$, the set of criteria $C=\left\{{c}_{1},{c}_{2},\cdots {c}_{n}\right\}$, the set of the weights of the criteria $\left\{{\alpha}_{1},{\alpha}_{2},\cdots ,{\alpha}_{n}\right\}$, and the decision matrix about ${\mu}_{{c}_{j}}({a}_{i})\hspace{0.17em}(1\le i\le m,1\le j\le n)$, $CP({c}_{i},{c}_{j})$ are first calculated according to Definition 1 and $cp({c}_{i},{c}_{j})$ are computed based on $CP({c}_{i},{c}_{j})$ and Definition 2. Then, the criteria are grouped through $cp({c}_{i},{c}_{j})$, the two criteria with the highest cooperative degree are grouped together, and then the two criteria with the highest cooperative degree of the remaining criteria are grouped together, and so on until the division is completed. Assume that ${c}_{i}$ and ${c}_{j}$ are in the same group, then ${\rho}_{ij}=\frac{1}{cp({c}_{i},{c}_{j})}-1$. Considering that when $cp({c}_{i},{c}_{j})=1$, the corresponding ${\rho}_{ij}$ is 0, and the contribution of ${\mu}_{{c}_{i}}(a)$, ${\mu}_{{c}_{j}}(a)$ to the trustworthiness of the candidate alternative $a$ cannot be reflected, so simple preprocessing is performed. When $cp({c}_{i},{c}_{j})=1$, the value of the corresponding ${\rho}_{ij}$ is set to 0.01. Finally, the metric model given in Definition 5 is used to evaluate each alternative in $A$, and the measurement results are arranged in descending order. The alternative corresponding to the first measurement result is the optimal alternative.
Without losing generality, assume that n is an even number. The algorithm for software trustworthiness evaluation based on relationships between criteria is given in Algorithm 1.
Algorithm 1 Algorithm for software trustworthiness evaluation based on relationships between criteria: for the given set of possible alternatives $A=\left\{{a}_{1},{a}_{2},\cdots ,{a}_{m}\right\}$, the set of criteria $C=\left\{{c}_{1},{c}_{2},\cdots {c}_{n}\right\}$, the set of the weights of the criteria $\left\{{\alpha}_{1},{\alpha}_{2},\cdots ,{\alpha}_{n}\right\}$, and the decision matrix about ${\mu}_{{c}_{j}}({a}_{i})\hspace{0.17em}(1\le i\le m,1\le j\le n)$, output the optimal alternative of $A=\left\{{a}_{1},{a}_{2},\cdots ,{a}_{m}\right\}$.
Input: $A=\left\{{a}_{1},{a}_{2},\cdots ,{a}_{m}\right\}$, $C=\left\{{c}_{1},{c}_{2},\cdots {c}_{n}\right\}$, $\left\{{\alpha}_{1},{\alpha}_{2},\cdots ,{\alpha}_{n}\right\}$, ${\mu}_{{c}_{j}}({a}_{i})\hspace{0.17em}(1\le i\le m,1\le j\le n)$ Output: the optimal alternative of $A=\left\{{a}_{1},{a}_{2},\cdots ,{a}_{m}\right\}$
$\mathrm{Let}\text{}M=\{(i,j)|1\le ij\le n\}$;
for$(i,j)\in M$do
Calculate$CP({c}_{i},{c}_{j})$ according to Definition 1;
end for
for$(i,j)\in M$do
Compute $cp({c}_{i},{c}_{j})$$\mathrm{based}\text{}\mathrm{on}\text{}CP({c}_{i},{c}_{j})$ and Definition 2;
end for
while$\text{}M\ne \varnothing $do
$cp=0$;
for$(i,j)\in M$do
if$cp({c}_{i},{c}_{j})\ge cp$then
$cp=cp({c}_{i},{c}_{j})$;
end if
end for
$N=\varnothing $;
for$(i,j)\in M$do
if$cp({c}_{i},{c}_{j})=\hspace{0.17em}=cp$then
$N=N\cup \{(i,j)\}$;
end if
end for
$\mathrm{Randomly}\text{}\mathrm{select}\text{}(i,j)\in N$$\text{}\mathrm{and}\text{}\mathrm{make}\text{}{c}_{i}$$\text{}\mathrm{and}\text{}{c}_{j}$ a group;
if$cp=1$then
${\rho}_{ij}=0.01$;
else
${\rho}_{ij}=\frac{1}{cp}-1$;
end if
$M=M-\{(i,j)\}$;
end while
According to the grouping results, substitute ${\rho}_{ij}$ into Equation (3) and obtain Equation (4);
for$i\in \{1,2,\dots ,m\}$do
for$j\in \{1,2,\dots ,n\}$do
$\mathrm{Substitute}\text{}{\mu}_{{c}_{j}}({a}_{i})$$\mathrm{and}\text{}{w}_{j}$ into Equation (4);
end for
Obtain the software trustworthiness measurement result $T({a}_{i})$ of
${a}_{i}$;
end for
Sort $\{T({a}_{1}),T({a}_{2}),\cdots ,T({a}_{m})\}$ in descending order using the quick sorting algorithm;
return the alternative corresponding to the first measurement result.
Theorem1.
The time complexity of Algorithm 1 is$O({n}^{4}+mn)$.
ProofofTheorem1.
Steps 2–4 are a for loop, and they are applied to calculate the set of cooperative alternative pairs according to Definition 1. The number of loops is $\left|M\right|$. Since $\left|M\right|=n(n-1)/2$, the time complexity of Steps 2–4 is $O({n}^{2})$. Steps 5–7 are also a for loop, which are used to compute the set of cooperative degrees based on the results of Steps 2–4 and Definition 2. The number of loops is also $\left|M\right|$. Therefore, the time complexity of Steps 5–7 is also $O({n}^{2})$.
Steps 8–28 are a double nested loop, which are utilized to group the set of criteria $C=\left\{{c}_{1},{c}_{2},\cdots {c}_{n}\right\}$ and compute ${\rho}_{ij}$. The number of loops in first while loop is $\left|M\right|$. Steps 10–14 are the first for loop in the second loop, and are used to find the maximum cooperative degree in the set $\{cp({c}_{i},{c}_{j})|(i,j)\in M\}$, and the number of loops is $\left|M\right|$. Steps 16–20 are the second for loop in the second loop, constructing the set composed of subscripts of maximum cooperative degree in the set $\{cp({c}_{i},{c}_{j})|(i,j)\in M\}$, and the number of loops is also $\left|M\right|$. Step 21 randomly selects $(i,j)$ from $N$ and makes ${c}_{i}$ and ${c}_{j}$ a group. Steps 22–26 are an if-else conditional statement, computing ${\rho}_{ij}$, and Step 27 is used to update the loop variable of the while loop. It is easy to determine that the time complexity of Steps 21–27 is $O(1)$ and the time complexity of Steps 9–27 is $2\left|M\right|+O(1)$. Therefore, the time complexity of Steps 8–28 is $M\left(2\left|M\right|+O(1)\right)$. Notice that $\left|M\right|=n(n-1)/2$, and it follows that the time complexity of Steps 8–28 is $O({n}^{4})$.
Steps 30–35 are also a double nested loop, calculating the software trustworthiness measurement results $\{T({a}_{1}),T({a}_{2}),\cdots ,T({a}_{m})\}$. The number of loops in the first loop is $m$, and in the second loop is $n$. Consequently, we infer that the time complexity of Steps 30–35 is $O(mn)$. The software trustworthiness measurement results $\{T({a}_{1}),T({a}_{2}),\cdots ,T({a}_{m})\}$ are sorted in descending order using the quick sorting algorithm in Step 36, and it takes $O(n\mathrm{log}n)$.
In summary, we can obtain the time complexity of Algorithm 1 is $O({n}^{4}+mn)$. □
5. Case Study
With the development of enterprise informatization, Product Lifecycle Management (PLM) software is becoming more and more important to improve the informatization level and core competitiveness of enterprises. Reference [40] presents a software trustworthiness evaluation approach based on combination weights and improved TOPSIS methods, and this approach is applied to evaluate the candidate PLM software trustworthiness for an aircraft equipment manufacturer. The candidate PLM software set consists of three PLM software, and the criterion set is composed of functionality, learnability, operability, co-existence, maintainability and portability. In this section, the method proposed in this paper is used to evaluate this case. Denote the three candidate PLM software as ${a}_{1}$, ${a}_{2}$,${a}_{3}$ in turn, and functionality, learnability, operability, co-existence, maintainability, and portability as ${c}_{1}$, ${c}_{2}$, ${c}_{3}$, ${c}_{4}$, ${c}_{5}$, ${c}_{6}$. The combination weights of these six criteria ${\alpha}_{1}$, ${\alpha}_{2}$, ${\alpha}_{3}$, ${\alpha}_{4}$, ${\alpha}_{5}$, ${\alpha}_{6}$ obtained through the combination weighting method established in [40] are given in Table 2. They are the weighted sum of the objective and subjective weights, where the objective weights are calculated by entropy weighting method and the subjective weights are determined by FAHP method.
Four exports ${D}_{1}$, ${D}_{2}$, ${D}_{3}$, ${D}_{4}$ are invited to give the fuzzy decision matrix about ${a}_{1}$, ${a}_{2}$,${a}_{3}$ [36], as given in Table 3.
Based on the mapping relationship between trustworthy attribute level and TFN defined in Table 1, the exports’ fuzzy decision can be transferred to the corresponding fuzzy numbers, and the integrated fuzzy decision matrix using TFNs can be obtained, as presented in Table 4.
Through defuzzying the integrated decision matrix with Equation (1), the decision matrix about ${\mu}_{{c}_{j}}({a}_{i})$ can be established as demonstrated in Table 5, where each element in the matrix is a real number between 0 and 10.
For any of the two criteria ${c}_{i}$ and ${c}_{j}$$(1\le i<j\le 6)$, according to Definition 1 and Table 5, calculate the set of cooperative alternative pairs, and the calculation results are shown in Equation (5).
For any of the two criteria ${c}_{i}$ and ${c}_{j}$$(1\le i<j\le 6)$, according to Definition 2 and the above calculation results, the cooperative degrees $cp({c}_{i},{c}_{j})$$(1\le i<j\le 6)$ can be determined, as demonstrated in Table 6.
It can be seen from Table 6 that the maximum value of $cp$ is $cp({c}_{4},{c}_{6})$, whose value is 1. Therefore, ${c}_{4}$ and ${c}_{6}$ are selected as the first group. In the remaining criterion, the maximum value of $cp$ is $cp({c}_{1},{c}_{5})=0.7662$, and ${c}_{1}$ and ${c}_{5}$ are chosen as the second group. Finally, ${c}_{2}$ and ${c}_{3}$ are taken as the third group. According to the grouping results, the weights in Table 2 and Definition 5, Equation (6) can be obtained. It should be noted that because $cp({c}_{4},{c}_{6})=1$, ${\rho}_{46}$ is set to 0.01.
Thus, among the three candidate PLM software, ${a}_{2}$ is the optimal software, and the result obtained is consistent with that in [40].
6. Discussion and Limitations
Software trustworthiness is a concept related to human cognition, and uncertainty theory can model human subjectivity well. Therefore, they are often used in software trustworthiness measurements. However, most of the existing methods use uncertainty theory to calculate the weight of trustworthy attributes. For example, references [21] and [22] both use fuzzy comprehensive evaluation methods to evaluate software trustworthiness, but they differ from the methods of selecting trustworthy attribute sets and computing the weights of trustworthy attributes. In reference [21], experts are invited to establish the trustworthy attribute set, and the weights of trustworthy attributes are computed based on the information entropy. In reference [22], the author’s trustworthy attribute set is determined in advance, and the weights of trustworthy attributes are obtained by using rough set theory and expert opinion. Gao et al. present a new weight distribution method by combining fuzzy analytic hierarchy process with standard importance index correlation, and establish a component-based software trustworthiness measurement model according to the four component composition structures on the basis of their weight distribution method [23]. Shi et al. also provide a calculation method of combination weight based on fuzzy analytic hierarchy process and entropy [40].
The fuzzy set theory is adopted by Shi et al. to measure the trustworthy attributes [40]. The mapping between the trustworthy attribute language variable and the triangular fuzzy number is first built, then the evaluators are invited to establish the fuzzy decision matrix, and finally the trustworthy attribute is measured by defuzzification. In this paper, a similar method is applied to measure trustworthy attributes; however, a more reasonable mapping between the trustworthy attribute language variable and the triangular fuzzy number is given. When building this new mapping, it is taken into account that the greater the software trustworthiness, the more difficult it is to improve the trustworthiness, and the higher the requirement for trustworthy attribute values. Shi et al. present a method to utilize an improved TOPSIS based on vertical projection distance [43] to evaluate the software trustworthiness. The advantage of the improved TOPSIS is that the alternative closest to the positive idea solution is farthest from the negative ideal solution. However, the evaluation method given by Shi et al. can only be applied for ranking the candidate software on trustworthiness. The method presented in this paper can not only be used for the ranking of candidate software on trustworthiness, but can also be used for the trustworthiness measurement of candidate software.
It should be noted that the method presented in this paper can be used for software trustworthiness evaluation, but only when multiple candidate software products must exist. Furthermore, the trustworthy attribute measurement method based on fuzzy theory can model the uncertainty or inherent imprecision of experts’ judgment. However, this method only gives the identification measurement of trustworthy attribute in the experts’ mind and does not give the measurement model of trustworthy attribute.
7. Conclusions and Future Work
To begin with, a trustworthy attribute measurement method based on fuzzy theory is presented in this paper, which is composed of the mapping relationship between trustworthy attribute level and TFN and defuzzification technology. This method can more reasonably transform the fuzzy decision of trustworthy attribute into a certain real number. Moreover, a software trustworthiness evaluation strategy based on the relationships between criteria is given, including the quantitative relationships between criteria described by the cooperative degrees between criteria, the symmetric substitutivity between criteria approximated by the cooperative degrees between criteria, and a software trustworthiness measurement model via the relationships between criteria. Lastly, we verify the theoretical rationality of the software trustworthiness measurement model by showing that it satisfies the expected properties of software trustworthiness measure. Meanwhile, the case study shows the effectiveness and practicality of the model. This strategy can be used not only to rank candidate software about trustworthiness, but also for the trustworthiness measure of software.
In the future, we will further improve the trustworthy attribute measurement method based on fuzzy theory and establish the measurement model of trustworthy attribute. We will also study the cooperative degrees between criteria for software in different fields, then estimate the symmetric substitutivity related parameters in Definition 3, and construct software trustworthiness measurement models for different fields. Further optimization of grouping methods based on cooperative degrees is also important work for the future.
Author Contributions
Conceptualization, H.T. and Q.C.; methodology, H.T. and Q.C.; algorithm, H.C., Y.X. and S.S.; writing—original draft preparation, H.T.; writing—review and editing, Q.C., H.C., Y.X., S.S. and X.N.; funding acquisition, H.T. All authors have read and agreed to the published version of the manuscript.
Funding
This work was financially supported by the National Natural Science Foundation of China (61906175), Doctoral Research Fund of Zhengzhou University of Light Industry (2016BSJJ037) and Science and Technology Project of Henan Province (212102210076, 202102210351).
Institutional Review Board Statement
Not applicable.
Informed Consent Statement
Not applicable.
Data Availability Statement
Not applicable.
Acknowledgments
A preliminary version of this work was presented at the 9th International Conference on Dependable Systems and Their Applications (DSA2022) and its title is “software trustworthiness evaluation method based on relationships between criteria”.
Conflicts of Interest
The authors declare no conflict of interest.
References
Avizienis, A.; Laprie, J.C.; Randell, B.; Landwehr, C. Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Trans. Dependable Secur. Comput.2004, 1, 11–33. [Google Scholar] [CrossRef][Green Version]
ISO/IEC 25010:2011; Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models. ISO: Geneva, Switzerland, 2011.
McCall, J.A.; Richards, P.K.; Walters, G.F. Factors in Software Quality: Concepts and Definitions of Software Quality; General Electric Co: Sunnyvale, CA, USA, 1977. [Google Scholar]
Boehm, B.W.; Brown, J.R.; Kaspar, H.; Lipow, M.; MacLeod, G.J.; Merritt, M.J. Characteristics of Software Quality; North-Holland Publishing Company: New York, NY, USA, 1978. [Google Scholar]
He, J.F.; Shan, Z.G.; Wang, J.; Pu, G.G.; Fang, Y.F.; Liu, K.; Zhao, R.Z.; Zhang, Z.T. Review of the Achievements of Major Research Plan of Trustworthy Software. Bull. Natl. Nat. Sci. Found. China2018, 32, 291–296. [Google Scholar] [CrossRef]
Tao, H.W.; Fu, L.Y.; Chen, Y.X.; Han, L.; Wang, X. Improved Allocation and Reallocation Approaches for Software Trustworthiness Based on Mathematical Programming. Symmetry2022, 14, 628. [Google Scholar] [CrossRef]
Tao, H.W.; Chen, Y.X.; Wu, H.Y.; Deng, R.M. A Survey of Software Trustworthiness Measurements. Int. J. Perform. Eng.2019, 15, 2364–2372. [Google Scholar] [CrossRef]
Tao, H.W.; Wu, H.Y.; Chen, Y.X. An Approach of Trustworthy Measurement Allocation Based on Sub-Attributes of Software. Mathematics2019, 7, 237. [Google Scholar] [CrossRef][Green Version]
Tao, H.W.; Chen, Y.X.; Wu, H.Y. A Reallocation Approach for Software Trustworthiness Based on Trustworthy Attributes. Mathematics2020, 8, 14. [Google Scholar] [CrossRef][Green Version]
Oman, P.; Hagemeister, J. Construction and Testing of Polynomials Predicting Software Maintainability. J. Syst. Softw.1994, 24, 251–266. [Google Scholar] [CrossRef]
Yuan, Y.; Han, H. A Software Behavior Trustworthiness Measurement Method Based on Data Mining. Int. J. Comput. Intell. Syst.2012, 4, 817–825. [Google Scholar] [CrossRef]
Medeiros, N.; Ivaki, N.; Costa, P.; Vieira, M. Vulnerable Code Detection Using Software Metrics and Machine Learning. IEEE Acc.2020, 8, 219174–219198. [Google Scholar] [CrossRef]
Lv, Z.H.; Han, Y.; Singh, K.A.; Manogaran, G.; Lv, H.B. Trustworthiness in Industrial IoT Systems Based on Artificial Intelligence. IEEE Trans. Industr. Inform.2021, 17, 1496–1504. [Google Scholar] [CrossRef]
Xu, J.L.; Xiao, L.J.; Li, Y.H.; Huang, M.W.; Zhuang, Z.C.; Weng, T.H.; Liang, W. NFMF: Neural Fusion Matrix Factorisation for QoS Prediction in Service Selection. Connect. Sci.2021, 33, 753–768. [Google Scholar] [CrossRef]
Tian, F.; Guo, Y.H. Software Trustworthiness Evaluation Model Based on Behavior Trajectory Matrix. Info. Softw. Technol.2020, 119, 106233. [Google Scholar] [CrossRef]
Tao, H.W.; Chen, Y.X. A New Metric Model for Trustworthiness of Softwares. Telecommun. Sys.2012, 51, 95–105. [Google Scholar] [CrossRef]
Tao, H.W.; Chen, Y.X.; Wu, H.Y. Decomposition of Attributes Oriented Software Trustworthiness Measure Based on Axiomatic Approaches. In Proceedings of the IEEE 20th International Conference on Software Quality, Reliability and Security Companion (QRS-C), Macau, China, 11–14 December 2020; pp. 308–315. [Google Scholar] [CrossRef]
Liu, H.; Tao, H.W.; Chen, Y.X. An Approach for Trustworthy Evidence of Source Code Oriented Aerospace Software Trustworthiness Measurement. AER Control Appl.2021, 47, 32–41. [Google Scholar] [CrossRef]
Tao, H.W.; Chen, Y.X.; Wu, H.Y. Theoretical and Empirical Validation of Software Trustworthiness Measure Based on Decomposition of Attributes. Connect. Sci.2022, 34, 1181–1200. [Google Scholar] [CrossRef]
Tao, H.W.; Chen, Y.X.; Liu, H.; Wu, H.Y.; Hu, Y.H. Attributes Oriented Software Trustworthiness Measure Based on Axiomatic Approaches. J. Int. Technol.2022, 23, 583–592. [Google Scholar] [CrossRef]
Shi, H.L.; Ma, J.; Zou, F.Y. A Fuzzy Comprehensive Evaluation Model for Software Dependability Based on Entropy Weight. In Proceedings of the 2008 International Conference on Computer Science and Software Engineering, Wuhan, China, 12–14 December 2008; pp. 683–685. [Google Scholar] [CrossRef]
Li, B.; Cao, Y. An Improved Comprehensive Evaluation Model of Software Dependability based on Rough Set Theory. J. Softw.2009, 4, 1152–1159. [Google Scholar] [CrossRef]
Gao, X.T.; Ma, Y.F.; Zhou, W. The Trustworthiness Measurement Model of Component-based Software Based on the Subjective and Objective Weight Allocation Method. In Proceedings of the IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C), Macau, China, 11–14 December 2020; pp. 478–486. [Google Scholar] [CrossRef]
Yang, M.; Gao, T.L.; Xie, W.Y.; Jia, L.; Zhang, T. The Assessment of Cloud Service Trustworthiness State Based on D-S Theory and Markov Chain. IEEE Acc.2022, 10, 68618–68632. [Google Scholar] [CrossRef]
Muhammad, D.M.S.; Fairul, R.F.; Loo, F.A.; Nur, F.A.; Norzamzarini, B. Rating of Software Trustworthiness Via Scoring of System Testing Results. Int. J. Dig. Enterp. Technol.2018, 1, 121–134. [Google Scholar] [CrossRef]
Yang, X.; Jabeen, G.; Luo, P.; Zhu, X.L.; Liu, M.H. A Unified Measurement Solution of Software Trustworthiness Based on Social-to-Software Framework. J. Comput. Sci. Technol.2018, 33, 603–620. [Google Scholar] [CrossRef]
Wang, B.H.; Chen, Y.X.; Zhang, S.; Wu, H.Y. Updating Model of Software Component Trustworthiness Based on Users Feedback. IEEE Acc.2019, 7, 60199–60205. [Google Scholar] [CrossRef]
Alarcon, G.M.; Ryan, T.J. Trustworthiness Perceptions of Computer Code: A Heuristic-Systematic Processing Model. In Proceedings of the 51st Hawaii International Conference on System Sciences, Waikoloa Village, HI, USA, 2–6 January 2018; pp. 5384–5393. [Google Scholar] [CrossRef][Green Version]
Maza, S.; Megouas, O. Framework for Trustworthiness in Software Development. Int. J. Perform. Eng.2021, 17, 241–252. [Google Scholar] [CrossRef]
Wang, H.M. Harnessing the Crowd Wisdom for Software Trustworthiness: Practices in China. ACM SIGSOFT Softw. Eng. Notes2018, 43, 6–11. [Google Scholar] [CrossRef]
Lee, J.; Kuo, J.Y. Fuzzy Decision Making Through Trade-off Analysis Between Criteria. Inf. Sci.1998, 107, 107–126. [Google Scholar] [CrossRef]
Lee, J.; Kuo, J.Y.; Huang, W.T. Fuzzy Decision Making Through Relationships Analysis Between Criteria. In Proceedings of the 1996 Asian Fuzzy Systems Symposium, Kenting, Taiwan, 11–14 December 1996; pp. 296–301. [Google Scholar] [CrossRef]
Xu, Z.S. Research on Software Trustworthiness Measurement Evaluation Model Based on Data Driven. In Proceedings of the 2nd International Conference on Computer Science Communication And Network Security (CSCNS2020), Sanya, China, 22–23 December 2020; pp. 1–4. [Google Scholar] [CrossRef]
Deng, Y.X.; Chen, Z.Z.; Du, W.J.; Mao, B.F.; Liang, Z.Z.; Lin, Q.S.; Li, J.H. Trustworthiness Derivation Tree: A Model of Evidence-Based Software Trustworthiness. In Proceedings of the IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C), Macau, China, 11–14 December 2020; pp. 487–493. [Google Scholar] [CrossRef]
Zhang, F.; Xu, M.D.; Chao, H.C.; Zhang, C.; Liu, X.L.; Hu, F.N. Real-time Trust Measurement of Software: Behavior Trust Analysis Approach Based on Noninterference. J. Softw.2019, 30, 2268–2286. [Google Scholar] [CrossRef]
Wang, D.X.; Wang, Q.; He, J. Evidence-Based Software Process Trustworthiness Model and Evaluation Method. J. Softw.2017, 28, 1713–1731. [Google Scholar] [CrossRef]
Wang, D.X.; Wang, Q. Trustworthiness Evidence Supporting Evaluation of Software Process Trustworthiness. J. Softw.2018, 29, 3412–3434. [Google Scholar] [CrossRef]
Tang, Y.; Pan, Z.; Pedrycz, W.; Ren, F.; Song, X. Viewpoint-based Kernel Fuzzy Clustering with Weight Information Granules. IEEE Trans. Emerg. Top. Comput. Intell.2022, in press. [Google Scholar] [CrossRef]
Tang, Y.; Zhang, L.; Bao, G.; Ren, F.J.; Pedrycz, W. Symmetric Implicational Algorithm Derived from Intuitionistic Fuzzy Entropy. Iran. J. Fuzzy Syst.2022, 19, 27–44. [Google Scholar] [CrossRef]
Li, S.; Yang, S.L.; Li, K. Developing an Evaluation Approach for Software Trustworthiness Using Combination Weights and TOPSIS. J. Softw.2012, 7, 532–543. [Google Scholar] [CrossRef][Green Version]
Chen, S.H.; Hsieh, C.H. Representation, Ranking, Distance, and Similarity of L-R Type Fuzzy Number and Application. Aust. J. Intell. Process. Syst.2000, 6, 217–229. [Google Scholar]
Tao, H.; Cao, Q.; Chen, H.; Xian, Y.; Shang, S.; Niu, X.
A Novel Software Trustworthiness Evaluation Strategy via Relationships between Criteria. Symmetry2022, 14, 2458.
https://doi.org/10.3390/sym14112458
AMA Style
Tao H, Cao Q, Chen H, Xian Y, Shang S, Niu X.
A Novel Software Trustworthiness Evaluation Strategy via Relationships between Criteria. Symmetry. 2022; 14(11):2458.
https://doi.org/10.3390/sym14112458
Chicago/Turabian Style
Tao, Hongwei, Qiaoling Cao, Haoran Chen, Yang Xian, Songtao Shang, and Xiaoxu Niu.
2022. "A Novel Software Trustworthiness Evaluation Strategy via Relationships between Criteria" Symmetry 14, no. 11: 2458.
https://doi.org/10.3390/sym14112458
Find Other Styles
Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.
Article Metrics
No
No
Article Access Statistics
For more information on the journal statistics, click here.
Multiple requests from the same IP address are counted as one view.
Tao, H.; Cao, Q.; Chen, H.; Xian, Y.; Shang, S.; Niu, X.
A Novel Software Trustworthiness Evaluation Strategy via Relationships between Criteria. Symmetry2022, 14, 2458.
https://doi.org/10.3390/sym14112458
AMA Style
Tao H, Cao Q, Chen H, Xian Y, Shang S, Niu X.
A Novel Software Trustworthiness Evaluation Strategy via Relationships between Criteria. Symmetry. 2022; 14(11):2458.
https://doi.org/10.3390/sym14112458
Chicago/Turabian Style
Tao, Hongwei, Qiaoling Cao, Haoran Chen, Yang Xian, Songtao Shang, and Xiaoxu Niu.
2022. "A Novel Software Trustworthiness Evaluation Strategy via Relationships between Criteria" Symmetry 14, no. 11: 2458.
https://doi.org/10.3390/sym14112458
Find Other Styles
Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.