Next Article in Journal / Special Issue
Framework and Capability of Industrial IoT Infrastructure for Smart Manufacturing
Previous Article in Journal
Employing Nanosafety Standards in a Nanomaterial Research Environment: Lessons Learned and Refinement Potential
Previous Article in Special Issue
Updating Mothers within an Hour of Newborn’s Admission to Neonatal ICU
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Combining Different Stakeholders’ Opinions in Multi-Criteria Decision Analyses Applying Partial Order Methodology

1
Awareness Center, Linkøpingvej 35, Trekroner, DK-4000 Roskilde, Denmark
2
Department of Ecohydrology, Leibniz-Institute of Freshwater Ecology and Inland Fisheries, Hutterbühlstrasse 20, D-91126 Schwabach, Germany
*
Author to whom correspondence should be addressed.
Standards 2022, 2(4), 503-521; https://doi.org/10.3390/standards2040035
Submission received: 18 May 2022 / Revised: 30 August 2022 / Accepted: 12 September 2022 / Published: 19 December 2022

Abstract

:
Multi-criteria decision analyses (MCDA) for prioritizations may be performed applying a variety of available software, e.g., methods such as Analytic Network Process (ANP) and Elimination Et Choice Translating Reality (ELECTRE III) as recently suggested by Kalifa et al. In addition to a data matrix, usually based on indicators and designed for describing the parts of the framework intended for the MCDA, these methods require input of a variety of other parameters that are not necessarily immediately obtainable. Often the indicators are simply combined by a weighted sum to obtain a ranking score, which is supposed to reflect the opinion of a multitude of stakeholders. A single ranking score facilitates the decision as a unique ordering is obtained; however, such a ranking score masks potential conflicts that are expressed by the values of the single indicators. Beyond hiding the inherent conflicts, the problem arises that the weights, needed for summing up the indicator values are difficult to obtain or are even controversially discussed. Here we show a procedure, which takes care of potential different weighting schemes but nevertheless does not mask any inherent conflicts. Two examples are given, one with a small (traffic) system and one with a pretty large data matrix (food sustainability). The results show how decisions can be facilitated even taking a multitude of stakeholder opinions into account although conflicts are not necessarily completely eliminated as demonstrated in the second case.

1. Introduction

1.1. Ranking

Ranking is a means to support decisions. The advantage of a ranking approach is that even if a best solution is found but not realizable, ranking provides other acceptable solutions. PROMETHEE [1,2,3], AHP [4], ANP [5], ELECTRE-family [6], and others are methods to obtain a ranking based on a data matrix and additional supporting information, such as weights, associated with indicators (ELECTRE, PROMETHEE). On one side, these methods are highly sophisticated; however, for a public understanding, they are rarely understandable. On the other side, partial order can explore the data matrix without complicated algebraic/arithmetic transformations but will not in all cases allow a unique decision, because the values of the single indicators lead to many conflicts which may be important for their own right, but clearly hampering a decision.
In the literature on application of partial ordering in decision making, many attempts can be found where—while keeping the framework of partial order—the degree of comparability (in a ranking the degree of comparability gets its maximum) is enhanced. Here, we present a procedure called General Linear Aggregation (GLA) [7] which not only enhances the degree of comparability, but also simultaneously covers the fact that different stakeholders may have different weightings in mind. The general idea with GLA is not to let the stakeholders try to harmonize their different opinions but to allow their own ones in the analysis. With GLA, the toolbox of partial order enrichments gets a further tool, which may be applied in selected cases. Some of these tools—including GLA—will be mentioned in subsequent sections of this paper.

1.2. Model Studies

The GLA is elucidated by two examples. (i) In a recent paper [8], two multi-criteria decision methods (MCDA), i.e., Analytic Network Process (ANP) and Elimination Et Choice Translating Reality (ELECTRE III), were applied to study and prioritize four paratransit transportation variants, BB: BodaBoda, Coa: Coaster, Kam: Kamunya, and TT: TukTuk, in Kampala, Uganda. The analyses were based on a total of 15 indicators covering B: benefits (8 indicators), C: costs (3 indicators), and R: risks (4 indicators) (cf. [8] Table 4). For the actual analyses, the indicators were aggregated into 3 main indicators (B, C, R) applying weight for the single indicators (cf. [8], Table 3). In both cases (ANP and ELECTRE) identical prioritizations were obtained, i.e., Coa > Kam > TT > BB, respectively. What is the influence of the weights, when simply weighted sums are applied for an aggregation and how to extend partial order methods to cope with cases such as here, whereby far more indicators than objects (here transportation variants) are applied? (ii) In the second example, a larger and thus more complex data matrix is investigated. Hence, the diet composition for 78 countries is to be compared with respect to food sustainability [9], which is not immediately quantifiable, but described by four indicators. In this study, four weighting schemes are adopted and GLA is applied to visualize what the combined effect of these four weighting schemes is.
It should be noted that the inclusion of the first example is important to demonstrate that the here-presented leads to an identical ranking.

1.3. Semantics

The advantages of the partial order-based approach may lose some value when the number of incomparabilities is increasing. To be clear: partial order may or even will disclose incomparabilities resulting from a multi-indicator system, which indicates the presence of conflicts within the set of indicators (details below).
It is appropriate to clarify the semantics of “importance of indicators” and “weighting schemes”. Within partial ordering, the concept “importance” has unfortunately often been associated with exchangeability of indicators expressing their low mutual importance. In connection with “weighting schemes” concerning the indicators importance is simply to stress the contextual importance of the single indicators.

2. Materials and Methods

2.1. Partial Ordering—Basics

The basics of partial ordering is the relation between objects (in our paper, either the four single transport systems or the 78 countries) to be ordered. The set of objects is called X. A priori, the data are analyzed without any pretreatments such as, e.g., aggregation of the single indicators. The only mathematical term applied in this context is “≤” (cf. e.g., [10,11]). By this, a relational instead of a numerical point of view is taken (cf. discussion). Two objects, x and y, are connected with each other if and only if the relation x ≤ y holds (see below, Equation (1)).

2.1.1. Main Equation, Compensation, Binary Relations

Object, x, characterized by the a set of indicators rs(x), s = 1, ..., m, can be compared with object y, characterized by the same set of indicators rs(y), if
rs(x) ≤ rs(y) for all s = 1, …, m
Application of Equation (1) needs a convention about the orientation of the single indicators, i.e., the larger the value of an indicator, the better a non-measurable quantity (a “latent construct”) by indicators being mutually co-monotone. Equation (1) is the basis for a comparison of objects. As the indicator values are not numerically combined, the problem of compensation (a “good” value of an indicator may compensate a “bad” one of another indicator) is avoided, which is one of the main advantages of partial ordering [12,13]. By several steps, the binary relations due to Equation (1) are prepared for a representation by a graph, the Hasse diagram [10,14]. Independent of a graphical representation, the result of the application of Equation (1) is a partially order set (poset) of the objects.

2.1.2. Hasse Diagram, Incomparabilities, Chains

In the Hasse diagram, comparable objects are connected by a sequence of lines [10,14]. If Equation (1) is not fulfilled for some objects x, y, then x is incomparable with y, denoted by x‖y. Generally, incomparability expresses that the data lead to conflicts between the objects. A subset X’ of X where for every x, y a ≤ - relation can be found is a chain, the number of objects constituting a chain is denoted its length. A subset X’’ of X, where for every x, y ∈ X’’ x‖y, is called an antichain. The number of incomparabilities, U, is an important characterizing quantity.
If for X a chain emerges, then U = 0 and the typical ranking (a linear order) is obtained.

2.1.3. Extension/Enrichment

An extension/enrichment of a partial order is transforming incomparabilities of the original poset into comparabilities (within the new poset) by maintaining the already given comparabilities.

2.2. Framework of Enrichments

As mentioned in Section 1, there are many attempts to enhance the degree of comparability without leaving the principles of partial order. As a detailed description is by far outside the scope of the paper, a table may be sufficient at this stage (Table 1). At the top is just the poset approach directly applied to data.
In the middle, there are more involved techniques to enrich the posets, and finally at the bottom, there are linear orders or construction of composite indicators, which condense the complexity inherent in data matrices to unidimensional representation. In that context, it is worth citing Arcagni et al. [15]: “Admittedly, however compressing the input posets into a simple linear order can be somewhat artificial and misleading…”.
Table 1. Attempts to enhance the degree of comparability.
Table 1. Attempts to enhance the degree of comparability.
MethodU (Number of Incomparisons)RemarkReferences
Application of Equation (1)May be very large.
Data matrix analyzed without any pretreatment by partial order leading to an “input poset”.
No external information needed beyond the data matrix [14,16]
Weights not as a sharp number but elements of certain intervalsU will be reducedStakeholders have to find intervals for the weights[17]
Different weighting systemsU will be reducedGLA (more details below)[18]
Matrix of mutual ranking probability (MRP)U will be reducedDominance structure of posets[15,19,20]
Bucket orderU will be reducedA systematic procedure to reduce U until the value 0[15,21,22,23]
POSAC (Partial Order Scalogram Analysis by coordinates)U will be reducedA bidimensional representation is searched keeping as much as possible the original comparabilities[24]
Ranking due to mean of different heightsU = 0Any partial order can be equivalently described by a set of linear orders.
The vertical position of an object within a linear order is called its height.
[25,26,27,28]: Concept of Composite, synthetic, indicator

2.3. Enrichment of the MIS—Generalized Linear Aggregation

Although the procedure is explained in detail in [7], we give a brief explanation for the sake of convenience of the reader.

2.3.1. Need of Normalization/Data Pretreatment

Whereas the original partial order method does not need a column wise normalization, the intended numerical combination of weights and indicator values (cf. Equation (4)) makes it favorable to ensure that both the indicator values and the weights are in the same order of amount. Hence, when indicators of different dimensionality are to be combined by weights, they both must be normalized to a common [0, 1] scale. Still worse is when indicators are on an ordinal scale, then the step of normalization is a critical and important step and there will be a trade-off: Either try to analyze all the incomparabilities, based on the original ordinal indicators, or perform a transformation which—in the last consequence—is a perhaps an acceptable data manipulation.

2.3.2. Orientation

In the first example, the 15 criteria values are oriented in both directions. In order to obtain equal orientation of the MIS (co-monotony of the single indicators) the six criteria c2, c9, c11, and c13–c15 (cf. Table 1 below) were all multiplied with −1 to make sure that for all criteria the higher values the better) A normalization of the indicators is performed as follows:
r ^ s = { ( r s ( x ) m i n ( r s ( ) ) ) / ( m a x ( r s ( ) ) m i n ( r s ( ) )   i f   r s ( x ) > 0 1 ( a b s ( r s ( x ) ) m i n ( a b s ( r s ( ) ) ) ) / ( m a x ( a b s ( r s ( ) ) ) m i n ( a b s ( r s ( ) ) )   i f   r s ( x ) < 0
One may consider the application of Equation (2) as an introduction of preference functions within the framework of partial order.
Therein, min or max of rs(…) and abs(rs(…)), respectively, is to be taken over the considered objects.

2.3.3. Aggregation Process

When the MIS (denoted as “MIS(old)” in order to emphasize the role of the aggregation process) is written in the form, where rr are the indicators and ei the studied elements
M I S ( o l d ) = e 1 e 2 e m r 1       r 2             r n [ ]
then the aggregation to a single scalar, Cf (composite indicator), which serves as a ranking index can be formulated by means of the weights for one single stakeholder, i.e., one single weight scheme, (Equation (4))
Cf⋯=⋯(g1⋯g2⋯…⋯gm) ⋯ MIS(old), i.e., Cf = ∑gi∙riold
where the selection of weights is responsible for the composite indicator Cf but is based on a system of indicators riold, where riold refers to the members of MIS(old).
Thus, performing the matrix-multiplication Equation (4), where a row of m entries is acting on each column of matrix MIS, the traditional weighted sum as expression for the aggregation process is obviously obtained. The difficulty in Equation (4) is not its mathematics, but the way how the weights can be found.
The weights bear important information concerning the roles played by the single indicators of an original MIS. There is no need for any equalizing of indicators as pointed out in [29]. The aggregation to a set of single scalar can be formulated as:
M I S ( n e w ) = [ g 11 g 21 g m 1 g 12 g 22 g m 2 ] · M I S ( o l d )
Equation (5) describes the calculation of a new MIS by a conventional matrix multiplication of a weight matrix, called G (having m columns, corresponding to the m indicators of the original MIS and as many rows as alternative weighting models that are/can be constructed) and the matrix MIS(old). Each row of matrix G is denoted as a weighting-scheme. Equation (5) can be more formally written as shown in in Equation (6), where the role of G as an operator Ĝ is stressed.
Ĝ*MIS(old) = MIS(new)
As mentioned above, in practical application of Equation (5), it is more convenient to accept any number for the weights, and to normalize them before Equation (5) is applied. It should be noted that preference functions are needed in other MCDA too and most MCDA further need weights, ignoring the inherent uncertainty in weight findings.
The advantages of the procedure, formulated by Equation (6) are that
(a)
the system of weight-regimes can be checked; for example, the matrix G can be analyzed by correlation measures, or even by posetic tools.
(b)
Equation (5) can also be written as a mapping, performed by an operator Ĝ. Ĝ can be applied to set of indicators of the MIS(old) leading to a set of new indicators, MIS(new) (Equation (7)).
Ĝ {r1, r2, …, rm} = {r′1, r′2, …, r′m}

2.3.4. The Number of Incomparabilities as a Controlling Quantity

This manner of consideration allows a partitioning: G can be thought of as consisting of submatrices, where each of the submatrices describes the opinions of a group of stakeholders. Correspondingly, Ĝ can be seen as obtained from sub-operators Ĝ(A), Ĝ(B), and Ĝ(C), when for instance three groups of stakeholders A, B, and C are considered. Then MIS(new) is just the combination of MIS(new,A), MIS(new,B), and MIS(new,C), respectively, and each of these groups of new indicators is obtained by application of Equation (5) with one of the submatrices. Each of these subsets of new indicators can be the basis for a visualization by a Hasse diagram and for each subset several incomparabilities U(A) (for example, when sub-operator Ĝ(A) is applied) can be obtained. Then, for U based on MIS(new), denoted as U(MIS(new)), the inequality.
U(MIS(old) ≥ U( MIS(new)) ≥ U(Y).
results, wherein Y stands for a subgroup of stakeholders, e.g., A, or B or C, respectively.
The ≥-sign indicates that the combination of subsets itself can generate new incomparabilities. The number of incomparabilities in a new MIS cannot be larger than that of the old MIS and will typically be lower.
Equation (8) allows to interpret the role of the sub-operators and hence of the different groups of stakeholders. When for instance selected Ĝ(Y) (Y is associated with any subgroups of stakeholders) in that manner that it is approximately just an m*m—unit matrix, then the corresponding MIS(new,Y) is just the MIS(old). Then, with respect to the incomparabilities, the role of other subsets no longer plays any role.
As mentioned above, one advantage of the procedure presented (Equations (5)–(8)) is that the multitude of stakeholders’ opinions can be mapped onto a matrix, that subsequently can be evaluated in a holistic manner. Here, the correlation measures characterizing G, i.e., the mutual correlations between the weight schemes, may be checked. Hence, if, e.g., the correlation between two regimes is very high, then one of the two rows (of matrix G) apparently is redundant and could be but must not be excluded. A perfect correlation (i.e., value 1 between two weight regimes) indicates that one of the two weight regimes could be ignored.

2.4. Software

All partial order analyses were carried out using the PyHasse software [30]. PyHasse is programmed using the interpreter language Python (version 2.6) [30]. Today, the software package contains more than 100 specialized modules and is available upon request from the developer, Dr. R. Bruggemann (brg_home@web.de).

3. Results

3.1. Evaluation Transportation Variants in Kampala

3.1.1. Normalization

The original MIS is adopted from Kalifa et al. [8] (Table 2) and the eventual normalized MIS is given in Table 3.
It is emphasized that for the criteria c2, c9, c11, c13, c14, and c15, the lower the values the better, whereas for the remaining 9 criteria, the higher the better is valid.

3.1.2. Application of Two Different G Matrices

In the present study, we applied two different G-matrices, comprising 4 and 6 weight-regimes, respectively. It should be noted that the weight-regimes applied here are partly artificially generated for illustrative purposed. In real life cases, the different weight-regimes are typically offered by the participants of the decision process. Thus, the number of weight-regimes is practically determined by stakeholders’ opinions.
The first G matrix includes in addition to the regime, where all weights have the value 1, the weights reported [8] (or) and subjectively chosen weights by the authors of the present paper (br and ca). The weight matrix G1 is shown in Table 4. Base on the Pearson correlations between the single regimes it is concluded that they are all mutually only to a minor extent correlated and as such that all should be taken into account.
The correlation (unsquared) is relatively large for “or” and “br”, thus the two weight schemes will not describe too different ideas as to how the coefficients in the weighted sum are to be selected (cf. Table 5). The other two correlations are positive too, but remarkably lower. Hence, one could expect that the weight scheme “ca” infers another conceptual idea. As mentioned above, the possible presence of strongly correlated weight-regimes as such do not pose a problem. Thus, in such cases it will be enough to include only one, i.e., reduced the G matrix, as including highly correlated weight-regimes will yield no new information.
Based on the G1 matrix (Table 4) and the normalized original MIS (Table 3) a new MIS is generated (Table 6), remembering that the criteria c2, c9, c11, c13, c14, and c15 are multiplied by −1 to secure identical orientation.
The same ranking is obtained as was shown in [8]. Although four different weight schemes are applied, with correlation coefficients varying from 0.1378 to 0.7589, the result is an invariant, namely a chain BB < TT < Kam < Coa. Its visualization as Hasse diagram is shown in Figure 1.
The second G matrix was generated based on 6 randomly generated weight-regimes, rd1–rd6. The random selection was [0, 25], respectively. The resulting G matrix is given in Table 7. A Pearson correlation showed that the 6 random weights display only very low correlation (in absolute terms) (Table 8).
As can be seen, in absolute terms the maximum correlation coefficient is found for the weight regimes rd1 and rd6 (−0.4426).
Based on the G2 matrix (Table 6) and the normalized MIS (Table 3) based on the originally adopted data from [27] (Table 2), a new MIS is generated (Table 9). The resulting Hasse diagram is identical to the one displayed in Figure 1. It seems as if a differentiation of the 15 indicators by weights is not important. Further, it is noted that the GLA method described here leads to the same ranking as previously reported by Kalifa et al. [8].

3.2. Evaluation of Food Sustainability within 78 Countries

The data for this part of the study were adopted from the Food Sustainability Index (FSI) 2021 [9] and comprises the fraction of the FSI that deals with diet composition ([9], sect. 8.1). This sub study includes 4 indicators (Table 10).
“All indicator scores are normalized to a 0 to 100 scale, where 100 indicates the highest sustainability and greatest progress towards meeting environmental, social and economic key performance indicators (KPI) and 0 represents the lowest” ([9], cf. Excel Workbook: Methodology). Hence, for all four indicators, the higher the indicator value, the better.
A fifth indicator (consumption of fruit and vegetables was left out due to the missing information concerning the weights of this indicators. The study comprises four possible weights, i.e., experts, uniform, outcome, and politics (vide infra). In total, 78 countries were included in the study [9]. In Table 11, the indicator values for the 78 countries are given.
Based on the above data (Table 11), a Hasse diagram was constructed (Figure 2).
Immediately, the diagram has a ‘broad’ structure and is highly complex and dominated by a high number of incomparabilities (1955) and a relative low number of comparabilities (1048). Although the number of incomparabilities is very high (around 66% percentage of all possible binary relations), there are nevertheless chains of length 8. By a special program of the software package PyHasse, details about chains can be found. Here, however, a more detailed discussion is out of the scope of the paper. The main point is that the Hasse diagram is complex; however, there are tools to unfold the jungle of lines.
In the analog study from 2017 [31], four different weighting schemes were discussed corresponding to the relative weighting of the four indicators by four different stakeholders (Expert, Political, Outcome, and Uniform). The four weight schemes are given in Table 12.
Applying the GLA methodology using the Table 11 and Table 12 gives rise to a new data matrix incorporation the original data as well as the four stakeholder opinions/weight schemes, the new indicators being denoted r1GLA, r2GLA, r3GLA, and r4GLA, respectively (Table 13).
The corresponding Hasse diagram is shown in Figure 3.
In Figure 3, we see a significantly enriched and much ‘slimmer’ Hasse diagram with only 285 incomparabilities and 2718 comparabilities.
Despite the obvious enrichment of the diagram, there are now chains of the length 30, we are not in this system achieving a strict linear order as was the above example studying various transportation forms. However, here, the opinions of all four stakeholders are simultaneously taken into account. Furthermore, the remaining incomparabilities, surviving the effect of the G*-operator motivate to further investigations, which is causing the conflicts. The relatively high number of comparabilities (compared to the original poset) indicates that the stakeholder opinions do not oppose each other but give gradually some more importance to the single indicators of the 4-indicator set. Hence, a possible subsequently decision process is remarkably facilitated.
It is here worth mentioning that simply the shape of Hasse diagrams can be a valuable tool in the analysis of complex data structures [32]. The input poset (i.e., the poset obtained by Equation (1)) of the food sustainability is broad and rather flat, whereas the Hasse diagram after GLA is rather slim and has a remarkable vertical range (cf. chain of lengths of 30).
A subsequent calculation of the average rank [26,33,34] makes much more sense than based on the diagram in Figure 2 although here the essential role of conflicts is no more visualized. In Table 14, the top 10 and bottom 10 countries based on an average ranking are shown.
Looking at the data shown in Table 14, it is immediately clear that USA, AUS, and ARG are virtually non-sustainable based on meat consumption (i2). Moreover, in the indicators i1 and i3, these three countries display non-sustainability, whereas in the case of salt consumption (i4), the three countries display values around 50.

4. Discussion

There are some points which should be stated in a clear manner:
(a)
Partial order takes a relational point of view, even if numerical algorithms, as indicated by Equation (5) are applied. Hence, the MIS(new) will once again analyzed in terms of a graph, indicating comparabilities and incomparabilities. Consequently, the data are only used to decide whether a ≤ -relation can be established. This is seen as some zooming out; however, clearly numerical details must be a posteriori analyzed.
(b)
In Equation (5), needs weights are combined with indicator values and then summed up. In a strict mathematical reasoning, this can only be done when the scaling level is metric. If this is not the case, or when the indicator values have very different ranges, which may depend on the used unit of measurements, then a normalization is needed. A normalization in turn requires metric values; when MIS(old) contains ordinal indicators, then a normalization is a crucial step which needs a carefully justification.
(c)
The characterization of the weight scheme (matrix G) can be performed in many ways, as, e.g., different correlation measures can be applied. Further, G itself can be investigated by partial order methods to disclose whether some weight regimes dominate some others.

5. Conclusions and Outlook

As mentioned above, there are constellations where partial ordering is insufficient and delivers only antichains or has an uncomfortable high degree of incomparabilities. Then, the generalized aggregation (Equation (5)) is still simple and remains within the theoretical framework of partial order methodology, since based on the normalized MIS (Equation (2)) a new MIS is generated, which is important on its own right, because stakeholder opinions and (measured) data are included. The new MIS still may have conflicts that require a deeper contextual discussion. In the most general case, Equation (5) cannot be applied without introducing preference functions, which in the framework of partial order are simple [0, 1] linear transformations (whereby it is not excluded that sometimes other preference models could be considered). Thus, in the results section, the effect whereby normalization the ranges of weight and of indicator values is made comparable is discussed. In the example of the Kampala transportation variants, a linear order is obtained, which is the most comfortable case for decision making. That a linear order is not necessarily the result when the matrix G has more than one row, shows the second example of food. Certainly, on the one side, a decision is not that easy (because both, the data of the original indicators (MIS(old)) and the weight regimes must be checked), but on the other side, the incomparabilities indicate that there are still conflicts, which must be discussed in a fair decision process.
However, accepting Equation (5) which frees the decision makers from the process of finding weights in a crisp manner (i.e., as a number with possibly some decimals) seems to be simple enough to be of help in public decision makings.

5.1. Limitations and Future Work

Obviously, there is a need of future work. Thus, the interpretation of ordinal data as metric ones is a big and crucial step that needs attention. However, not in every case such a step can be justified. In that case, GLA breaks down. Therefore, there is a need to find an enrichment procedure, where stakeholder opinions can be collected and can influence the poset. It seems as if the methodological way should start with the set of linear orders, representing the input poset. Hereto, a good basic material is given by Patil and Joshi [35]. The attempts presented therein aim at a final linear order, whereas our intention is an enrichment procedure, which keeps the most important conflicts. Another starting point could be the paper of Arcagni et al. [15], where especially the bucket poset approach seems to be attractive.

5.2. The Novelty of the Here-Presented Approach

Modern MCDA methods such as, e.g., PROMETHEE and members of the ELECTRE family, are typically based on an intricate combination of stakeholders opinions and arithmetic operations according to the methodology used. However, these close interactions make it very difficult to judge the actual effect of the stakeholders and effects that can be assigned to the pure arithmetic procedure.
The here-presented method separates completely the role of stakeholders from that of the procedure. Thus, the outcome is that statistical measures, made as Pearson correlations are available as a further tool to understand the stakeholders’ opinions as a whole. This task is facilitated by the—admitted—simplicity of the partial order methodology itself. The new here-described method is based on conventional matrix multiplication combined with the previously well-described partial order methodology cf., e.g., [7].

Author Contributions

Both authors have equally participated in all phases of the work leading to this paper. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Brans, J.P.; Vincke, P. A Preference Ranking Organisation Method (The PROMETHEE Method for Multiple Criteria Decision—Making). Manag. Sci. 1985, 31, 647–656. [Google Scholar] [CrossRef] [Green Version]
  2. Brans, J.P.; Vincke, P.; Mareschal, B. How to select and how to rank projects: The Promethee method. Eur. J. Oper. Res. 1986, 24, 228–238. [Google Scholar] [CrossRef]
  3. Brans, J.-P.; Mareschal, B. The PROMCALC & GAIA decision support system for multicriteria decision aid. Decis. Support Syst. 1994, 12, 297–310. [Google Scholar] [CrossRef]
  4. Saaty, T.L. How to Make a Decision: The Analytical Hierarchy Process. Interfaces 1994, 24, 19–43. Available online: https://www.jstor.org/stable/25061950?origin=JSTOR-pdf (accessed on 3 October 2022). [CrossRef] [Green Version]
  5. Saaty, T.L. Vargas, The analytical network process. In Decision Making with the Analytic Network Process; Saaty, T.L., Var-gas, L., Eds.; Springer: Berlin/Heidelberg, Germany, 2006; pp. 1–26. [Google Scholar] [CrossRef]
  6. Li, H.-F.; Wang, J.-J. An Improved Ranking Method for ELECTRE III. In Proceedings of the 2007 International Conference on Wireless Communications, Net-working and Mobile Computing, Shanghai, China, 21–25 September 2007; pp. 6659–6662. [Google Scholar] [CrossRef]
  7. Carlsen, L.; Bruggemann, R. Partial Order as Decision Support between Statistics and Multicriteria Decision Analyses. Standards 2022, 2, 306–328. [Google Scholar] [CrossRef]
  8. Kalifa, M.; Özdemir, A.; Özkan, A.; Banar, M. Application of Multi-Criteria Decision analysis including sustainable indicators for prioritization of public transport system. Integr. Environ. Assess. Manag. 2021, 18, 25–38. [Google Scholar] [CrossRef] [PubMed]
  9. Food Suistainability Index 2021. Available online: https://impact.economist.com/projects/foodsustainability/ (accessed on 3 October 2022).
  10. Bruggemann, R.; Patil, G.P. Ranking and Prioritization for Multi-Indicator Systems—Introduction to Partial Order Applications; Springer: New York, NY, USA, 2011; Available online: https//www.springer.com/gp/book/9781441984760 (accessed on 3 October 2022).
  11. Carlsen, L. Happiness as a sustainability factor. The world happiness index: A posetic-based data analysis. Sustain. Sci. 2018, 13, 549–571. [Google Scholar] [CrossRef]
  12. Munda, G. Social Multi-Criteria Evaluation for a Sustainable Economy; Springer: Berlin/Heidelberg, Germany; New York, NY, USA, 2008; p. 227. [Google Scholar] [CrossRef]
  13. Nardo, M.; Saisana, M.; Saltelli, A.; Tarantola, S.; Hoffman, A.; Giovannini, E. Handbook on Constructing Composite Indicators. Methodology and User Guide; OECD: Ispra, Italy, 2008; Available online: https://www.oecd.org/sdd/42495745.pdf (accessed on 3 October 2022).
  14. Brüggemann, R.; Carlsen, L. Introduction to partial order theory exemplified by the Evaluation of Sampling Sites. In Partial Order in Environmental Sciences and Chemistry; Bruggemann, R., Carlsen, L., Eds.; Springer: Berlin/Heidelberg, Germany, 2006; pp. 61–110. [Google Scholar] [CrossRef]
  15. Arcagni, A.; Avellone, A.; Fattore, M. Complexity reduction and approximation of multidomain systems of partially ordered data. Comput. Stat. Data Anal. 2022, 172, 107520. [Google Scholar] [CrossRef]
  16. Bruggemann, R.; Carlsen, L. (Eds.) Partial Order in Environmental Sciences and Chemistry; Springer: Berlin/Heidelberg, Germany, 2006; pp. 61–110. Available online: https://www.springer.com/gp/book/9783540339687 (accessed on 3 October 2022).
  17. Carlsen, L.; Bruggemann, R. Inequalities in the European Union—A Partial Order Analysis of the Main Indicators. Sustainability 2021, 13, 6278. [Google Scholar] [CrossRef]
  18. Bruggemann, R.; Carlsen, L. Uncertainty in Weights for Composite Indicators Generated by Weighted Sums. In Measuring and Understanding Complex Phenomena: Indicators and Their Analysis in Different Scientific Fields; Brüggemann, R., Carlsen, L., Beycan, T., Suter, C., Maggino, F., Eds.; Springer: Cham, Switzerland, 2021; pp. 45–62. [Google Scholar] [CrossRef]
  19. Brüggemann, R.; Carlsen, L.; Panahbehagh, B.; Pirintsos, S. A Study to Generate a Weak Order from a Partiallly Ordered Set, Taken Biomonitoring Measurements. In Measuring and Understanding Complex Phenomena; Indicators and Their Analysis in Different Scientific Fields; Brüggemann, R., Carlsen, L., Beycan, T., Suter, C., Maggino, F., Eds.; Springer: Cham, Switzerland, 2018; pp. 63–82. [Google Scholar] [CrossRef]
  20. de Loof, K.; de Baets, B.; de Meyer, H. Approximation of Average Ranks in Posets. MATCH—Commun. Math. Comput. Chem. 2011, 66, 219–229. Available online: https://match.pmf.kg.ac.rs/electronic_versions/Match66/n1/match66n1_219-229.pdf (accessed on 3 October 2022).
  21. Aledo, J.A.; Gámez, J.A.; Rosete, A. Utopia in the solution of the Bucket Order Problem. Decis. Support Syst. 2017, 97, 69–80. [Google Scholar] [CrossRef]
  22. Fernandez, P.L.; Heath, L.S.; Ramakrishnan, N.; Tan, M.; Vergara, J.P.C. Mining posets from linear orders. Discret. Math. Algorithms Appl. 2013, 5, 1350030. [Google Scholar] [CrossRef] [Green Version]
  23. Gionis, A.; Mannila, H.; Puolamäki, K.; Ukkonen, A. Algorithms for discovering bucket orders from data. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, 20–23 August 2006; pp. 561–566. [Google Scholar] [CrossRef] [Green Version]
  24. Shye, S. Partial Order scalogram analysis by coordinates (POSAC) as a facet theory measurement procedure: How to do POSAC in four simple steps. In Facet Theory and Scaling: In Search for Structure in Behavioral and Social Sciences Facet Theory; Cohen, A., Ed.; Association Press: New York, NY, USA, 2009; pp. 295–310. Available online: https://www.researchgate.net/publication/263932933_PARTIAL_ORDER_SCALOGRAM_ANALYSIS_BY_COORDINATES_POSAC_AS_A_FACET_THEORY_MEASUREMENT_PROCEDURE_HOW_TO_DO_POSAC_IN_FOUR_SIMPLE_STEPS (accessed on 3 October 2022).
  25. Bubley, R.; Dyer, M. Faster random generation of linear extensions. Discret. Math. 1999, 201, 81–88. [Google Scholar] [CrossRef] [Green Version]
  26. De Loof, K.; de Meyer, H.; de Baets, B. Exploiting the Lattice of Ideals Representation of a Poset. Fundam. Inform. 2006, 71, 309–321. [Google Scholar]
  27. Schröder, B.S.W. Ordered Sets—An Introduction; Birkhäuser: Boston, MA, USA, 2003. [Google Scholar] [CrossRef]
  28. di Bella, E.; Gandullia, L.; Leporatti, L.; Montefiori, M.; Orcamo, P. Ranking and Prioritization of Emergency Departments Based on Multi-indicator Systems. Soc. Indic. Res. 2018, 136, 1089–1107. [Google Scholar] [CrossRef]
  29. Mashunin, K.Y.; Mashunin, Y.K. Vector optimization with equivalent and priority criteria. J. Comput. Syst. Sci. Int. 2017, 56, 975–996. [Google Scholar] [CrossRef]
  30. Brüggemann, R.; Carlsen, L.; Voigt, K.; Wieland, R. PyHasse Software for Partial Order Analysis: Scientific Background and Description of Selected Modules. In Multi-Indicator Systems and Modelling in Partial Order; Springer: New York, NY, USA, 2014; pp. 389–423. [Google Scholar] [CrossRef]
  31. Food Sustainability Index 2017. Available online: http://foodsustainability.eiu.com/wp-content/uploads/sites/34/2016/09/FoodSustainabilityIndex2017GlobalExecutiveSummary.pdf (accessed on 3 October 2022).
  32. Comim, F. A Poset-Generalizability Method for Human Development Indicators. Soc. Indic. Res. 2021, 158, 1179–1198. [Google Scholar] [CrossRef]
  33. Brüggemann, R.; Carlsen, L. An Improved Estimation of Averaged Ranks of Partially Orders. MATCH Commun. Math. Comput. Chem. 2011, 65, 383–414. [Google Scholar]
  34. Brüggemann, R.; Sørensen, P.B.; Lerche, D.; Carlsen, L. Estimation of Averaged Ranks by a Local Partial Order Model. J. Chem. Inf. Comput. Sci. 2004, 44, 618–625. [Google Scholar] [CrossRef] [PubMed]
  35. Patil, G.P.; Joshi, S.W. Comparative Knowledge Discovery with Partial Orders and Composite indicators: Multi-indicator Sytemic Ranking, Advocacy and Reconciliation. In Multi-Indicator Systems and Modelling in Partial Order; Bruggemann, R., Carlsen, L., Wittmann, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2014; pp. 107–146. Available online: https://www.springerprofessional.de/comparative-knowledge-discovery-with-partial-orders-and-composit/1891480 (accessed on 3 October 2022).
Figure 1. Hasse diagram ranking the transportation modes by 4 aggregated indicators (cf. Table 4).
Figure 1. Hasse diagram ranking the transportation modes by 4 aggregated indicators (cf. Table 4).
Standards 02 00035 g001
Figure 2. “Input poset” visualized by a Hasse diagram (for details, see text).
Figure 2. “Input poset” visualized by a Hasse diagram (for details, see text).
Standards 02 00035 g002
Figure 3. Hasse diagram, based on the weighting schemes of Table 12 and the data obtained by GLA and shown in Table 13.
Figure 3. Hasse diagram, based on the weighting schemes of Table 12 and the data obtained by GLA and shown in Table 13.
Standards 02 00035 g003
Table 2. Original MIS adopted from Kalifa et al. [8].
Table 2. Original MIS adopted from Kalifa et al. [8].
Cluster Criteria Unit BodaBoda Coaster KamunyeTukTuk
Benefit c1 Score 1–9 9357
c2 Year 1.532.52
c3 Score 1–9 9357
c4 Score 1–9 3975
c5 Seats 129143
c6 Score 1–9 3975
c7 Score 1–9 3975
c8 Year 515155
Cost c9 kg/100 km·passenger 3.80.7611.63
c10 $/day 20403525
c11 $/passenger·year 16523.329.387.9
Risk c12 Score 1–9 3975
c13 Score 1–9 9135
c14 g CO2/pkm 69203147
c15 kJ/pkm 22004305501000
Table 3. Normalized Kalifa MIS with equal orientation.
Table 3. Normalized Kalifa MIS with equal orientation.
CriterianBBnCoanKamnTT
c1100.3330.667
c20−1−0.667−0.333
c3100.3330.667
c4010.6670.333
c5010.4640.071
c6010.6670.333
c7010.6670.333
c8011.0000.000
c9−10−0.079−0.286
c10010.7500.250
c11−10−0.042−0.456
c12010.6670.333
c13−10−0.250−0.500
c14−10−0.224−0.551
c15−10−0.068−0.322
Table 4. Weight-regimes of indifferent-weights (“cw1”), Kalifa et al. (“or”), Bruggemann (“br”) and Carlsen (“ca”).
Table 4. Weight-regimes of indifferent-weights (“cw1”), Kalifa et al. (“or”), Bruggemann (“br”) and Carlsen (“ca”).
Regimesc1c2c3c4c5c6c7c8c9c10c11c12c13c14c15
cw1111111111111111
or2.75.45.42.75.45.42.75.47.5157.5138.86.66.6
br3567781612151520552
ca21074107579101036210
Table 5. Pearson correlation coefficients between the four weight-regimes.
Table 5. Pearson correlation coefficients between the four weight-regimes.
cw1-orna
cw1-brna
cw1-cana
or-br0.7589
or-ca0.2291
br-ca0.1378
Table 6. The new MIS based on the generalized aggregation method (Equation (3)).
Table 6. The new MIS based on the generalized aggregation method (Equation (3)).
Transportation Modecw1orBrca
BB−0.200−0.289−0.256−0.274
Coa0.4000.4420.5040.353
Kam0.2810.2910.3430.239
TT0.0360.0000.0380.022
Table 7. Weight-regimes rd1 to rd6.
Table 7. Weight-regimes rd1 to rd6.
Regimec1c2c3c4c5c6c7c8c9c10c11c12c13c14c15
rd188201420181591152291615
rd2391860512201761921228
rd31219211159222021142143201
rd41518221111114410181169
rd525257118563857111050
rd6181421122391911841214255
Table 8. Pearson correlation coefficients between the 6 weight regimes.
Table 8. Pearson correlation coefficients between the 6 weight regimes.
rd1–rd20.2295
rd1–rd3−0.0201
rd1–rd40.3906
rd1–rd5−0.1483
rd1–rd6−0.4426
rd2–rd3−0.0804
rd2–rd4−0.0891
rd2–rd5−0.2134
rd2–rd60.447
rd3–rd4−0.2693
rd3–rd50.0107
rd3–rd6−0.1064
rd4–rd5−0.2319
rd4–rd6−0.2567
rd5–rd60.1221
Table 9. The new MIS based on the generalized aggregation method (Equation (3)).
Table 9. The new MIS based on the generalized aggregation method (Equation (3)).
Transportation Modesrd1rd2rd3rd4rd5rd6
BB−0.199−0.229−0.163−0.3640.015−0.164
Coa0.4030.3890.3250.5300.1760.452
Kam0.2810.2890.2560.3080.1610.326
TT0.028−0.0150.019−0.0410.0930.04
Table 10. Indicators of the food sustainability study.
Table 10. Indicators of the food sustainability study.
Indicator
r1Pct. of sugar in dietsPercent sugar in diet
r2Meat consumption levelsDifference in meat consumption (g/capita(day)) from daily recommended intake (90 g/capita/day)
r3Saturated fat consumptiong/capita/day
r4Salt consumptionAverage g/day sodium consumption
Table 11. Data matrix of the food sustainability. A total of 78 countries are characterized by the numerical values of four indicators.
Table 11. Data matrix of the food sustainability. A total of 78 countries are characterized by the numerical values of four indicators.
IDr1r2r3r4
AlgeriaDZA47.386.99124.9
AngolaAGO65.891.257.372.9
ArgentinaARG12.27.414.959.2
AustraliaAUS13.311.4648
AustriaAUT22.441.128.333.8
BangladeshBGD84.269.488.644.8
BelgiumBEL15.776.931.947.2
BrazilBRA21.726.826.229.5
BulgariaBGR40.970.869.442.6
Burkina FasoBFA78.279.480.862.5
CameroonCMR74.978.279.883.6
CanadaCAN19.635.348.840.2
ChinaCHN84.668.21510.2
ColombiaCOL27.569.430.530
Cote d’IvoireCIV77.276.14864.6
CroatiaHRV4.148.740.240.2
CyprusCYP46.852.258.130.8
Czech RepublicCZE33.944.343.733
Dem. Rep. of CongoCOG69.480.179.874.8
DenmarkDNK7.549.229.852
EgyptEGY43.899.190.841
EstoniaEST46.960.74833.8
EthiopiaETH77.572.594.378.8
FinlandFIN40.750.6036.5
FranceFRA25.449.68.338.6
GermanyDEU22.649.926.144.8
GhanaGHA7881.795.676.7
GreeceGRC45.956.171.238.6
HungaryHUN31.244.125.826.3
IndiaIND45.569.283.439.9
IndonesiaIDN56.878.573.149.6
IrelandIRE25.450.833.939.4
IsraelISL53.828.243.938.1
ItalyITA42.346.631.421.2
JapanJPN41.982.1778.6
JordanJOR15.899.864.829
KenyaKEN49.682.190.1100
LatviaLVA40.858.350.927.3
LebanonLBN7.398.388.655.8
LithuaniaLTU2844.333.930.6
LuxembourgLUX49.146.611.130.6
MadagascarMDG74.279.594.180.7
MalawiMWI68.678.895.595.2
MaliMLI77.28897.655.2
MaltaMLT16.950.963.629.8
MexicoMEX10.260.635.265.7
MoroccoMAR35.910088.124.1
MozambiqueMOZ60.275.460.579.6
NetherlandsNLD32.360.734.750.7
NigerNER10074.392.461.4
NigeriaNGA747369.464.1
PakistanPAK37.882.841.134.9
PhilippinesPHL43.796.848.224.7
PolandPOL18.7397.236.7
PortugalPRT52.33230.426
RomaniaROU50.465.563.829.2
RussiaRUS19.853.457.927.9
RwandaRWA70.473.380.496.8
Saudi ArabiaSAU42.487.322.353.9
SenegalSEN54.281.474.855.2
Sierra LeoneSLE84.974.45272.4
SlovakiaSVK29.173.965.126.3
SloveniaSVN45.360.264.126.3
South AfricaZAF30.466.45773.2
South KoreaKOR31.358.533.10
SpainESP3927.454.931.9
SudanSDN23.888.395.576.1
SwedenSWE27.856.24.141.8
TanzaniaTZA7076.682.166
TunisiaTUN3696.580.520.9
TurkeyTUR46.994.143.429.8
United Arab EmiratesARE29.361.868.141.3
UgandaUGA62.77869.283.1
United KingdomGBR40.951.631.742.9
United StatesUSA0041.543.2
VietnamVNM75.665.224.616.6
ZambiaZMB6685.787.178.8
ZimbabweZWE27.985.310056.6
Table 12. Four weighting schemes, as defined within the food sustainability study [31].
Table 12. Four weighting schemes, as defined within the food sustainability study [31].
Indicators:ExpertPoliticalOutcomeUniform
r1: Percentage of sugar in diets0.3750.1430.4000.250
r2: Meat consumption levels0.2500.2860.2000.250
r3: Saturated fat consumption0.1630.2860.2000.250
r4: Salt consumption0.2130.2860.2000.250
Table 13. The resulting new MIS following GLA.
Table 13. The resulting new MIS following GLA.
IDr1GLAr2GLAr3GLAr4GLA
AlgeriaDZA0.5950.6470.5950.625
AngolaAGO0.7230.7270.7060.718
ArgentinaARG0.2140.2500.2120.234
AustraliaAUS0.1900.2060.1840.197
AustriaAUT0.3050.3270.2960.314
BangladeshBGD0.7280.7000.7420.718
BelgiumBEL0.4030.4680.3750.429
BrazilBRA0.2540.2670.2520.261
BulgariaBGR0.5340.5810.5290.559
Burkina FasoBFA0.7560.7480.7580.752
CameroonCMR0.7840.7970.7830.791
CanadaCAN0.3270.3830.3270.360
ChinaCHN0.5330.3880.5250.445
ColombiaCOL0.3900.4100.3700.394
Cote d’IvoireCIV0.6950.6490.6860.665
CroatiaHRV0.2880.3750.2750.333
CyprusCYP0.4660.4700.4690.470
Czech RepublicCZE0.3790.3940.3780.387
Dem. Rep. of CongoCOG0.7490.7700.7470.760
DenmarkDNK0.3100.3850.2920.346
EgyptEGY0.6470.7220.6370.687
EstoniaEST0.4770.4740.4730.474
EthiopiaETH0.7930.8120.8010.808
FinlandFIN0.3570.3070.3370.320
FranceFRA0.3150.3120.2950.305
GermanyDEU0.3470.3770.3320.359
GhanaGHA0.8150.8370.8200.830
GreeceGRC0.5100.5400.5150.530
HungaryHUN0.3250.3190.3170.319
IndiaIND0.5640.6150.5670.595
IndonesiaIDN0.6330.6560.6300.645
IrelandIRE0.3610.3910.3500.374
IsraelISL0.4250.3920.4360.410
ItalyITA0.3710.3440.3680.354
JapanJPN0.5060.5390.5030.524
JordanJOR0.4760.5760.4500.524
KenyaKEN0.7500.8490.7430.805
LatviaLVA0.4390.4480.4360.443
LebanonLBN0.5360.7040.5150.625
LithuaniaLTU0.3360.3510.3300.342
LuxembourgLUX0.3840.3220.3730.344
MadagascarMDG0.8010.8330.8050.821
MalawiMWI0.8120.8680.8130.845
MaliMLI0.7850.7980.7900.795
MaltaMLT0.3570.4360.3560.403
MexicoMEX0.3870.4760.3640.429
MoroccoMAR0.5790.6580.5680.620
MozambiqueMOZ0.6820.7020.6720.689
NetherlandsNLD0.4370.4640.4210.446
NigerNER0.8410.7950.8560.820
NigeriaNGA0.7090.6960.7090.701
PakistanPAK0.4900.5080.4690.492
PhilippinesPHL0.5370.5470.5140.534
PolandPOL0.2570.2640.2410.254
PortugalPRT0.3810.3270.3860.352
RomaniaROU0.5180.5250.5190.522
RussiaRUS0.3610.4260.3580.398
RwandaRWA0.7840.8160.7830.802
Saudi ArabiaSAU0.5280.5280.4970.515
SenegalSEN0.6460.6810.6400.664
Sierra LeoneSLE0.7430.6890.7370.709
SlovakiaSVK0.4560.5140.4470.486
SloveniaSVN0.4800.4950.4820.490
South AfricaZAF0.5280.6050.5150.568
South KoreaKOR0.3170.3060.3080.307
SpainESP0.3720.3820.3840.383
SudanSDN0.6270.7770.6150.709
SwedenSWE0.3400.3310.3150.325
TanzaniaTZA0.7280.7420.7290.737
TunisiaTUN0.5510.6170.5400.585
TurkeyTUR0.5450.5450.5220.536
United Arab EmiratesARE0.4630.5310.4600.501
UgandaUGA0.7190.7480.7110.733
United KingdomGBR0.4250.4190.4160.418
United StatesUSA0.1600.2420.1690.212
VietnamVNM0.5210.4120.5150.455
ZambiaZMB0.7710.8130.7670.794
ZimbabweZWE0.6010.7310.5950.675
Table 14. Results of a linearization by LPOMext [33].
Table 14. Results of a linearization by LPOMext [33].
Objects.LPOMextRank
Top 10
MWI77.5561
GHA77.2752
MDG75.4673
NER73.754
ETH73.475
RWA73.3876
KEN727
MLI71.6198
ZMB70.7629
CMR69.70810
Bottom 10
HUN9.74869
HRV8.30570
AUT7.84471
FRA7.10972
KOR6.72873
BRA4.574.5
POL4.574.5
ARG376
AUS1.577.5
USA1.577.5
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Carlsen, L.; Bruggemann, R. Combining Different Stakeholders’ Opinions in Multi-Criteria Decision Analyses Applying Partial Order Methodology. Standards 2022, 2, 503-521. https://doi.org/10.3390/standards2040035

AMA Style

Carlsen L, Bruggemann R. Combining Different Stakeholders’ Opinions in Multi-Criteria Decision Analyses Applying Partial Order Methodology. Standards. 2022; 2(4):503-521. https://doi.org/10.3390/standards2040035

Chicago/Turabian Style

Carlsen, Lars, and Rainer Bruggemann. 2022. "Combining Different Stakeholders’ Opinions in Multi-Criteria Decision Analyses Applying Partial Order Methodology" Standards 2, no. 4: 503-521. https://doi.org/10.3390/standards2040035

Article Metrics

Back to TopTop