A Multi-Criteria Decision-Making Model for Selecting the Best Project Delivery Systems for Offsite Construction Projects

Abstract

Off-site construction (OSC) is an innovative construction method that transfers most of the site-based work to a more controlled environment. Construction waste minimization, speedy schedules, higher sustainability, and better quality are some of the perceived benefits of OSC. Therefore, significant research attention has been given to OSC. However, minimal research attention has been given to procurement management in OSC, which could impact its pace of adoption. Existing studies on the procurement methods of OSC projects have overlooked several criteria related to OSC that impact the selection of the appropriate procurement methods (i.e., design-build, construction management, etc.). In addition, the literature lacks decision-making tools to assist OSC practitioners in selecting the appropriate procurement method. In this regard, this study contributes to the body of knowledge by (1) identifying the criteria that impact the selection of OSC procurement methods; (2) developing a multi-criteria decision-making (MCDM) model to select the appropriate OSC procurement methods. The developed MCDM model uses a hybrid approach of analytic network process (ANP) and evidential reasoning (ER). The ANP, which considers the interdependencies among the collected OSC procurement criteria, is used to calculate the relative importance weights through questionnaire surveys. The ER method evaluates various OSC procurement methods in accordance with the criteria importance weights. The results indicate that project quality, cost control, and funding arrangement are the prominent selection factors. On the other hand, the model reveals that the integrated project delivery (IPD) and construction management (CM) methods have the highest utility scores. The MCDM model has been validated by comparing the results with similar studies. The present study could assist OSC practitioners in selecting the appropriate procurement method for OSC projects.

1. Introduction

The construction industry is characterized by intricacy, uncertainty, and fragmentation [1]. It is constantly subjected to requirements for greater speed and higher quality. These challenges have called for more efficient and integrated construction techniques. Therefore, vast attention has been given to solutions to transfer as many of the onsite construction works as possible to a more controlled environment, which is called the offsite construction (OSC) technique [2]. The OSC is a disruptive construction method that replaces the site-based processes with prefabricated sections that are developed in the factory and transported to the construction site to be installed and assembled [3,4]. The forms of OSC techniques are numerous, including prefabricated components, panelized elements, and fully prefabricated volumetric modules. Through the implementation of improved processes in the controlled environment (factory), OSC techniques leverage significant gains over traditional construction techniques, such as a speedy construction process, lesser construction waste, improved quality control, and higher sustainability [1,4,5].

Despite the excitement surrounding the merits of adopting the OSC techniques, they are characterized by a slow pace of adoption in most countries [2,6]. This raises the question of why or what is holding these innovative techniques back [2]. Research studies have identified several potential barriers that hinder the adoption of OSC techniques. These barriers are associated with various aspects of OSC implementation, including the complexity of the supply chain [4], cost uncertainty [2], and unclear procurement methods [7]. Procurement methods form a significant portion of the barriers associated with adopting OSC techniques [8]. Due to insufficient knowledge, the practitioners tend to adopt existing procurement methods, which are argued to be unsuitable and obstruct the advantages perceived by the OSC techniques due to the vast difference in processes [9].

Procurement methods can be defined as the determination of the contractual relationships among various project participants to build a facility [10]. It requires an understanding of the construction processes and the included stakeholders [8]. Regarding the OSC techniques, the appropriate procurement methods remain ambiguous and undefined. The unclarity of choosing the appropriate procurement approach in OSC techniques is related to various aspects. For instance, Salama et al. [7] stated that the procurement issues in OSC projects are tied to the significantly needed upfront costs, which are nearly equal to 60% of the project’s total cost. Stein [11] revealed that in modular construction projects, a type of OSC technique, manufacturers struggle to obtain financing through manufacturing processes due to the unwillingness of lenders to finance the projects unless the modules are delivered to the site (collateral). However, the regular duration for finishing the prefabricated modules is about six months, through which the manufacturers will have to pay for materials and labor, which would impact their cash flow and financial health [7]. Further, the developed modules are considered as the manufacturer’s personal ownership as long as they are not delivered to the site yet, where they transfer from the manufacturer’s ownership to the client’s private property. However, lenders generally refuse to release installments unless the modules are delivered to the site, raising legal issues and payment disputes [7,12]. These disputes over modules’ ownership complicate the financing arrangements and hence complicate the procurement methods.

The increasing attention to the procurement issues of the OSC techniques has motivated some researchers to investigate this topic. Ng and Ng [13] have studied several case studies of modular integrated construction (MiC) projects in Hong Kong to identify the challenges in the current procurement methods. Finnie et al. [14] have discussed the advantages of the procurement methods that enable an early involvement of the onsite contractor in the manufacture and design stages. Charlson and Dimka [9] have proposed a procurement model named design-manufacture-construct (DMC) for volumetric offsite manufacturing (VOSM) projects constructed in the UK. Their model asserts the need to have a single point of responsibility that promotes integration and enhances collaboration processes. Salama et al. [7] have investigated the financial issues in modular construction projects, asserting the need to modify the current procurement system to comply with the OSC aspects. Agapiou [8] has attempted to identify the factors that might affect the selection of procurement methods in offsite construction, concluding that the speed of construction and price competition are the most significant factors. Ramesh et al. [15] studied the factors causing the underutilization of OSC methods in construction projects. The study showed that the project delivery method is highly affected by the transaction cost needed for OSC methods. Furthermore, Assaf et al. [16] have developed a framework to facilitate the procurement of OSC methods and manage progress payments, emphasizing the need for a high level of integration and collaboration.

Despite the contributions of previous research on addressing procurement systems in offsite construction projects, they lack several aspects. For instance, hardly any study has gone beyond identifying the procurement issues in OSC techniques. Besides, the criteria identified in the majority of the studies to select the appropriate procurement approach were mainly related to conventional construction without incorporating numerous offsite aspects. Further, the literature lacks decision-making tools to assist OSC practitioners in selecting the appropriate procurement approach under different circumstances. In light of the mentioned limitations, this study aims to bridge these gaps by providing a multi-criteria decision-making approach for selecting the appropriate procurement method in OSC projects. The objectives of this study can be summarized as follows: (1) to review previous studies that addressed various procurement methods and the criteria used for their selection; (2) to identify the issues associated with procuring OSC projects and propose the criteria used to govern the procurement methods selection; (3) to prioritize the identified criteria by determining the weight of each criterion; (4) to develop a tool that assists practitioners in selecting the appropriate procurement approach for OSC projects.

2. Overview of the OSC Procurement Method

Several definitions can describe construction procurement methods. Jagtap et al. [17] defined procurement methods as the managerial process for the establishment and acquisition of a construction facility. El Asmar et al. [10] defined the procurement methods as the project delivery methods that establish the relationship among the construction project’s stakeholders and their engagement time to acquire a facility. Suresh and Arun Ram Nathan [18] acknowledged the procurement method as an activity to obtain facilities and goods for any organization. Procurement methods play a significant role in the construction industry. They govern the sourcing of material and labor, assign different responsibilities, and share risk among project participants to execute a project efficiently [19,20]. There are many forms of procurement methods. Masterman et al. [21] classified procurement methods into three main categories: separated, integrated, and management-based approaches. Figure 1 summarizes the arrangements of different procurement methods that were discussed in the literature [8,9,14,20]. Further details about the methods are discussed in the following paragraphs.

In the separated procurement approach, the client has separate contracts with each one of the project participants, i.e., onsite contractor, manufacturer, architect, supplier, etc. This is acknowledged as the design-bid-build procurement method and can also be referred to as the traditional procurement system. The integrated procurement methods can take various shapes, such as design-build (DB), integrated project delivery (IPD), and negotiated bid. In DB, the client has a contract with a single project party (main contractor) to carry out the design and construction [9,22]. In OSC projects, the main contractor will sublet the manufacturing processes (offsite processes) to a manufacturer to build the offsite components/sections. This approach is mainly used in constructing modular integrated construction (MIC) projects in Hong Kong [13]. Similarly, in the negotiated bid, the client will also contract with a single party of the project to perform the design, manufacturing, and onsite construction [23]. Instead of the onsite contractor, the manufacturer is hired initially by the client to select the appropriate onsite contractor and architect. Further, integrated project delivery (IPD), as a form of integrated procurement methods, is characterized by early engagement, collective processes, and effective collaboration among the project’s key stakeholders [24,25]. It adopts a single multi-party contract that is based on a risk/profit compensation model, which is known as the key element of the IPD approach [26]. However, the lack of proper establishment of compensation models hinders the adoption of the IPD approach in the construction industry.

In management-based procurement methods, the client has separate contracts with the architect and a constructor manager [9]. The constructor manager, which is also called the management contractor (MC), is appointed at a very early stage of the project and is responsible for the construction planning of a project and subletting all of the work to the subcontractors [22]. However, the MC is not in charge of performing any site work. Although this approach is argued to provide a high level of collaboration among the project’s stakeholders, its implementation is hindered by the high requirement of client involvement, calling for more sophisticated clients that are willing to be involved in the construction processes [27].

Selecting the appropriate procurement approach could significantly influence the project’s total duration, variations control, risk avoidance, and mitigation of adversarial relationships, and minimize possible issues during the project’s implementation [20,28,29]. Several internal and external criteria must be considered by the project’s client when selecting the procurement approach. It is argued that ignorance of these criteria would result in disputes, schedule delays, and cost overruns [30]. Initially, the selection of procurement methods was primarily cost-oriented [27]. Further studies have emphasized the integration of stakeholders in selecting procurement methods [31]. In addition, the rapid advances in technologies adopted in construction, such as building information modeling (BIM), have emphasized the criticality of considering communication in the selection criteria [27]. Further studies have asserted that quality and client satisfaction are significant indicators when selecting the appropriate procurement approach [32]. Other studies have argued that clients’ characteristics, including their involvement and willingness to share risks, can affect the selection of procurement methods [22].

3. Methodology

Figure 2 illustrates an overall framework of the adopted methodology to achieve the research objectives. The figure shows four main parts: (1) collection and validation of criteria considered in selecting the appropriate procurement approach in OSC projects; (2) calculation of the importance weights for each criterion and sub-criterion using the analytic network process (ANP); (3) development of the belief matrix for each criterion and sub-criterion through the ER technique; (4) calculation of the aggregated assessment of each procurement approach. The methods and procedures applied in each part are detailed in the following sections.

3.1. Parameters Identification

In this section, the publications that indicated the procurement selection criteria are collected, and the parameters are extracted using the following steps.

3.1.1. Databases Selection and Search Strategy

A preliminary search was carried out in Google Scholar using the terms procurement and construction to choose a representative set of keywords required for retrieving the related studies from search databases. The studies obtained from this process have helped select these keywords: procurement method, procurement system, procurement approach, procurement route, project delivery method, project delivery system, construction, criteria, and factors.

Several search databases can be used to collect related studies, including Scopus, Web of Science, and Google Scholar. As mentioned by Salihu et al [33] and Hong and W.M. Chan [34], Web of Science, and Scopus are widespread search databases that index engineering management publications. There is, however, a difference between the two databases in terms of the number of publications. In this study, the Scopus database was selected due to its broader coverage of publications, as well as its faster process of storing and updating recent publications [35,36,37,38]. The following formula was used in the Scopus search database:

(TITLE (“procurement method” OR “procurement approach” OR “procurement route” OR “procurement system” OR “project delivery system” OR “project delivery method”) AND TITLE-ABS-KEY (construction) AND TITLE-ABS-KEY (“criteria” OR “factors”)) Accordingly, 105 studies are obtained. These studies are then assessed in the next section according to the inclusion and exclusion criteria.

3.1.2. Evaluating the Collected Studies

In this section, the studies identified in the previous section are evaluated to exclude unrelated studies. First, the inclusion and exclusion criteria are discussed. The inclusion criteria in this paper would include (1) publications related to the engineering subject area; (2) publications that were written in English; (3) no limitations regarding countries, publication type, or publication year. The retrieved studies of this process are 78. Next, a full-text evaluation of the 78 publications has been conducted to ensure their contributions to the research topic. This process has resulted in excluding 45 publications. Hence, the retrieved number of publications to extract the PDS selection criteria is 33 studies (78 − 45 = 33). These publications are then studied to extract the criteria to select the appropriate procurement methods.

3.1.3. Determination of New Parameters in Selecting Procurement Methods in OSC Projects

In the collected studies, the discussed criteria for procurement method selection mainly contribute to the site-based projects. However, these criteria may differ when incorporating the activities that take place in the managed manufacturing environment [10,39]. One major concern of the OSC projects is the rising cost of transferring the components from the manufacturer to the site, including renting trailers and wrapping the prefabricated components [7]. Sutrisna et al. [2] have stated that the transportation cost can significantly impact the total cost of the project, forming around 3–4 percent of the project’s total cost. Another transportation-related aspect of implementing OSC projects is the flexibility of highway agency regulations, which could impact the project’s schedule and hinder its success [23]. Hence, selecting the procurement method should incorporate the accurate allocation of transportation costs and flexibility in the transportation processes.

Additionally, the cash flow systems used by participants in OSC projects should vary from those used in traditional ones. The cash flow issues for OSC project participants result from the significant upfront needed capital [1,40,41]. For instance, Sutrisna et al. [2] have investigated the financial feasibility of three case studies of OSC projects in Australia. The study indicated that the onsite contractors of all the cases had experienced a remarkably negative cash flow in the early stages of the OSC projects. The study also stated how different payment mechanisms could affect the cash flow of OSC project participants. Hence, when selecting the appropriate procurement approach for OSC projects, one should take into account the onsite contractor/manufacturer’s abilities to obtain early funding, as well as the client’s flexibility in providing advance payments.

Moreover, in most countries, financing OSC projects is challenging as the banks (lenders) are not yet familiar with the OSC arrangements. For instance, Mills [42] stated that, in New Zealand, lenders agree to offer significant funding to contractors at the early stage of conventional construction projects. However, in modular construction, lenders only offer funding when the prefabricated modules arrive onsite. This might risk the manufacturer’s financial stability, who will have to pay for materials and labor for around six months before the first module arrives on the site [7]. In modular construction, there is a lack of perception of the modules’ ownership. Through the manufacturing stage, the modules are considered the manufacturer’s property. However, this ownership transfers from the manufacturer to the client after the module’s arrival on the site [7]. This unclear ownership of the modules may complicate the financing structure of modular construction projects, as lenders seek collateral to agree on financing the projects [12]. Therefore, in selecting the appropriate procurement approach for OSC projects, the following parameters shall be considered: clear ownership of modules in the manufacturing stage and banks’ familiarity with the OSC projects.

One of the significant barriers to implementing OSC projects is the asynchronous nature of design, manufacturing, and onsite assembly [43]. Many scholars have asserted the need for early integration of onsite contractors in the design and manufacturing stages. For instance, Finnie et al. [14] suggested a two-stage integration model, emphasizing the value of contractors’ early involvement in OSC projects in New Zealand. The model includes a pre-construction contract for onsite contractors to enhance the procurement of OSC projects. Similarly, Charlson and Dimka [9] have proposed an integration model for OSC projects named design, build, and construct (DMC). The model asserts the value of a single point of responsibility that promotes integration among OSC processes. Therefore, decision-makers should consider the synchronization between offsite and onsite works and onsite contractors’ involvement in the design and manufacturing stages when selecting the appropriate procurement approach.

Another significant aspect of OSC projects is the just-in-time (JIT) method. JIT is the lean construction method that promotes the on-time delivery of material, or offsite components, to their correct location [44]. The JIT method promotes waste mitigation, smooth workflow, and quality management [45]. Much research attention has been given to the critical success factors of adopting the JIT method in offsite construction [46]. One identified critical success factor of adopting the JIT method is procurement management [47,48]. The researchers have emphasized the need to adjust the selection criteria to select partners who are willing to implement the JIT method instead of only concentrating on the lowest price [44,49]. Therefore, the ease of JIT method adoption should be considered when selecting the appropriate procurement approach for OSC projects.

The collected parameters through this section, along with the parameters identified in the past section, are gathered and presented in

Table 1. The project delivery system selection criteria.

Main Criteria	Sub-Criteria	ID	Parameters	Studies
Cost (A)	Cost estimating and control (A1)	A11	Lowest cost	[22,27,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65]
		A12	Precise estimation	[22,66]
		A13	Cost certainty	[54,56,57,58,67,68,69,70,71]
		A14	Control cost growth	[20,27,56,58,67,69,70,72,73,74]
		A15	Cash flow controlling systems *	[2,7]
		A16	Early consideration of transportation costs *	[2,7,9]
	Funding arrangements (A2)	A21	Ability to obtain early funding *	[1,2,7,40,41]
		A22	Funding methods/multiple funding resources	[20,50,52,53,55,57,60,61,64,75]
		A23	Banks’ familiarity with OSC projects *	[1,7,41]
Schedule (B)	Execution’s schedule control (B1)	B11	Shortest schedule	[20,22,27,50,51,52,53,54,56,57,58,61,62,63,64,65,66,67,68,69,70,72,75]
		B12	Control time growth	[27,52,58,60,62,66,68,69,70,73,74,75,76]
		B13	Flexibility in design and construction overlapping	[54,60,64,67,75]
		B14	Synchronization between offsite and onsite works *	[9,14,43]
	Material procurement plan (B2)	B21	Early procurement of materials to site/factory	[56,60,71,72,75]
Relationships and processes (C)	Risk/rewards Sharing (C1)	C11	Risk allocation among participants	[50,52,53,55,56,57,59,60,61,63,64,65,68,75,77]
		C12	Opportunity for partnership (risk sharing)	[60,68]
	Avoid claims and disputes (C2)	C21	Avoid conflict of interests	[52,60,68,75]
		C22	Minimize adversarial relationships	[52,55,57,58,60,61,65,67,69,73,74,78,79]
		C23	Minimize the number of contracted parties	[51,60,67]
		C24	Reduction of administrative parties	[52]
		C25	Clear ownership of modules in the manufacturing stage *	[7,12]
	Collaborative processes (C3)	C31	Effective communication	[27,51,52,55,57,58,59,60,61,74,77,79]
		C32	Collaboration of the project team	[27,52,57,59,60,64,69,74,76,77]
		C33	Distrust of new systems	[68,78]
		C34	Payment modality	[20]
	Integration (C4)	C41	Pre-construction arrangements	[51,57,61]
		C42	Involvement of the onsite contractors in the manufacturing/design stages *	[9,14,43]
Project characteristics (D)	Project quality (D1)	D11	Asset maintainability	[21,27,50,51,53,55,56,57,58,59,60,61,62,63,64,65,66,67,68,70,73,74,76]
		D12	Asset quality	[20,27,50,51,53,55,56,57,58,59,60,61,62,63,64,65,66,67,68,70,73,74,76]
		D13	Sustainable construction	[27,55,57,60,65,71,73,74]
		D14	Control construction impacts on facilities	[20,60,61,57]
	Prefabrication processes (D2)	D21	Complexity of manufacturing processes *	[8,9]
		D22	Flexibility in transporting the prefabricated components	[9,43]
		D23	Ease adoption of just-in-time delivery system *	[8,41,47,48,49]
	Project conditions (D3)	D31	Availability of appropriate contractors	[20,27,57,60,61,63,67,68,69,76]
		D32	Project type	[20,27,54,55,57,58,67,75,78,79]
		D33	Project scope	[20,50,51,52,53,55,57,58,61,63,64,65,67,68,69,71,72,74,75,78,79]
		D34	Clarity of the project scope	[22,52,57,58,62,64,66,69,71,72,71,78,79]
		D35	Project location and condition	[20,56,60,63,67,69,71,72,76]
		D36	Uniqueness of the project	[20,50,53,55,58,64,71,72,77,78,79]
	Project change (D4)	D41	Likelihood of the change	[50,52,53,56,57,60,61,64,68,72,73,76]
		D42	Flexibility upon change occurrence	[20,52,54,57,58,69,78,79]
	Project’s regularity (D5)	D51	Third parties’ agreements	[20,50,51,57,59,60,61,63,67,71,74,78,79]
		D52	Bidding competitions	[51,57,58,60,61,63,65,67,68,69,74]
		D53	Subjectivity of award process	[50,60]
Owner’s characteristics (F)	Owner’s capabilities (F1)	F11	Experience with particular procurement system	[20,50,51,52,54,57,60,61,62,64,65,69,74,75,79]
		F12	Owner’s availability of human resources	[53,54,55,63,64,65,67,74,78]
		F13	Owner’s management abilities	[51,57,60,62,63,78,79]
		F14	Complexity in decision making	[27,52,55,60,67,78]
		F15	Flexibility on advance Payments	[1,2,7,40,41]
	Owner’s preferences (F2)	F21	Owner’s involvement	[20,22,27,51,52,55,57,60,61,64,67,68,69,72,75,76,78,79]
		F22	Shifting roles and responsibilities	[20,22,51,55,57,60,62,63,65,67,69,74,75,78,79]
		F23	Owner preferences on particular stakeholders	[51,57,60]
		F24	Owner control over design	[22,52,55,57,62,73,74,78,79]

* indicates newly added parameters.

. The parameters are then classified into five main criteria, namely cost, schedule, relationships and processes, project characteristics, and client characteristics. Each of these contains a number of sub-criteria in which the parameters are presented. Further, Figure 3 shows a decision tree to clarify the collected criteria, subcriteria, and parameters. The following sections will discuss the interdependencies among criteria and sub-criteria and the weighting of each identified criterion and sub-criterion.

3.2. Determination of Importance Weights Using ANP

Multi-criteria decision-making (MCDM) problems can be solved through many techniques. Generally, MCDM problems comprise two major parts: prioritizing selected criteria and assessing various alternatives [80]. The analytic network process (ANP) and analytic hierarchy process (AHP) are some examples of the methods used in prioritizing the selected criteria [22,81,82]. Further, the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) and Evidential Reasoning (ER) are some of the alternatives assessments techniques [83,84]. Combinations of these methods have been widely used to solve MCDM problems in construction research [80,83,85,86].

Recently, researchers utilized more advanced MCDM techniques, such as the ordinal priority approach (OPA) technique. The OPA method was proposed by Ataei et al. [87] to solve intricate MCDM problems. The OPA method was extended afterwards to represent the uncertainties existing in the real world through the fuzzy and grey environments [88]. It was recently applied widely in construction research, including the performance measure of construction suppliers [89], the evaluation of construction sub-contractors [88], ranking the requirements needed for efficient implementation of blockchain technology in the construction supply chain [90], and prioritizing the risk factors in implementing blockchain technology in the construction industry [91].

The ANP method was established by Saaty [92] to solve intricate decision-making for real-world problems. The ANP method allows complex interdependencies among the decision elements. It develops the problems as a network that comprises goals, criteria, sub-criteria, parameters, and alternatives [93]. The applications of the ANP technique in construction research are numerous, including condition assessment of offshore pipelines [86], assessment of non-destructive technologies in bridge inspection [80], and safety assessment of construction systems [81]. The interactions among criteria, sub-criteria, and parameters profile the network of the method. The ANP method is used to solve the interdependencies by calculating the relative importance of criteria. It comprises several sequential steps that are discussed in the following sub-sections. SuperDecisions software tool and python codes were used to implement the ANP method.

• Step 1:Development of the network model

Selecting the appropriate procurement approach for OSC projects is a complex decision problem, as many criteria and sub-criteria are interrelated [9]. For instance, the prefabrication processes sub-criterion can directly be influenced by the criteria of cost, schedule, relationships and processes, and owner’s characteristics. Thus, the decision-maker might find that time (schedule) is more significant than other criteria when considering the prefabrication processes. Similarly, the material procurement plan sub-criterion could be directly impacted by cost and project characteristics criteria. These interrelationships among the criteria and subcriteria are efficiently represented by the ANP technique rather than the analytic hierarchy process (AHP), which only focuses on the hierarchical relationships of the criteria, as mentioned by Saaty [92], Mosleh et al. [86], and Abdelkhalek and Zayed [80]. Figure 4 shows the interrelations identified among various criteria and sub-criteria.

• Step 2:Construction of the pairwise comparisons

After establishing the network model of the decision problem, pairwise comparisons of the criteria and sub-criteria are performed to create the relative importance of the criteria and sub-criteria. Decision-makers in this step have been asked to answer a series of pairwise comparisons, through which two criteria or sub-criteria are compared simultaneously concerning a controlling upper level (goal, criterion, or sub-criterion) [82]. The decision-makers, through this step, have judged the pairwise comparisons following the AHP fundamental scale that was established by Saaty [94].

All of the relations in the developed ANP model should be evaluated using the same procedures to create the priorities of criteria and sub-criteria [82]. The number of pairwise comparisons in the developed network equals n × (n − 1)/2, where n is the number of the elements that are being compared. The pairwise comparison element a_ij is the output of the fundamental scale that reflects the ratio w_i/w_j, where w_i is the relative importance of the ith row criterion and w_j is the relative importance of the jth column criterion. For instance, if a_ij has a value of 1, that means equal importance of the ith row criterion and the jth column criterion. When a_ij has a value of 9, it means the ith row criterion is nine times more significant than the jth column criterion. The eigenvector of the pairwise comparison is calculated and used in developing the super-matrix, as discussed in the next section. As described by Ozkaya and Erdin [95], the general form of the pairwise comparison can be described as shown in Equation (1).

$$\begin{gathered} {\mathrm{C}}_1 {\mathrm{C}}_2\hspace{1em}\dots \hspace{1em}{\mathrm{C}}_{\mathrm{n}} \\ A=\begin{array}{c}{\mathrm{C}}_1\\ {\mathrm{C}}_2\\ ⋮\\ ⋮\\ {\mathrm{C}}_n\end{array}\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \cdots & ⋮\\ ⋮& ⋮& \cdots & ⋮\\ a_{n1}& a_{n2}& \cdots & a_{\mathrm{nn}}\end{array}\right] \end{gathered}$$

(1)

where A is the pairwise comparison matrix, a_ij = w_i/w_j = 1/ a_ji, a_ii = a_jj = 1, 1$\le i \le n$, and 1$\le j \le n$.

• Step 3:Checking the consistency

During the evaluation of the criteria and sub-criteria, there is a possibility that some of the evaluations are inconsistent. Therefore, a consistency check is developed to ensure the effectiveness and reliability of these evaluations. As mentioned by Genger et al. [96], the consistency can be checked by computing the consistency ratio (CR) and consistency index (CI) as shown in Equations (2) and (3).

$$\mathrm{CR}=\frac{\mathrm{CI}}{\mathrm{RI}}$$

(2)

$$\mathrm{CI}=\frac{{\lambda }_{\max }-n}{n-1}$$

(3)

where CI is the Consistency Index, RI is the random index, and λ_max is the maximum eigenvalue of matrix A. The RI value is chosen according to the number of compared elements in the pairwise matrix A. The values of RI with respect to the number of compared elements, as addressed by Saaty [94], are presented in

Table 2. Random index values, as addressed by Saaty [94].

No. of Compared Elements	Random Index Values
1	0
2	0
3	0.58
4	0.90
5	1.12
6	1.24
7	1.32
8	1.41
9	1.45
10	1.49

.

• Step 4:Construction of the super-matrix

The pairwise comparisons conducted between the criteria and sub-criteria are then converted into a two-dimensional super-matrix. The super-matrix indicates the priorities of the elements represented at the left of the super-matrix (rows) with regard to an element at the top of the super-matrix (column). Each column is not normalized in this matrix, and the summation of the priorities in each column does not equal one (unweighted matrix). A general representation of the super-matrix, as addressed by Magableh and Mistarihi [93], can be described as presented in Equation (4).

$$\begin{gathered} \begin{array}{}& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} {\mathrm{C}}_1\hspace{1em}\hspace{1em}& & & & \hspace{1em}{\mathrm{C}}_2\hspace{1em}\hspace{1em}& & & & \hspace{1em}{\mathrm{C}}_n& & \end{array} \\ \begin{array}{}\hspace{1em}\hspace{1em}{\mathrm{e}}_{11}{\mathrm{e}}_{12}\cdots {\mathrm{e}}_{1m1\hspace{1em}}& {\mathrm{e}}_{21}{\mathrm{e}}_{22}\cdots {\mathrm{e}}_{2m2\hspace{1em}}& {\mathrm{e}}_{n1}{\mathrm{e}}_{n2}\cdots {\mathrm{e}}_{nmn}\hspace{1em}\end{array} \\ w=\begin{array}{c}\\ {\mathrm{C}}_1\\ \\ \\ \\ {\mathrm{C}}_2\\ \\ \\ \\ {\mathrm{C}}_n\\ \\ \end{array}\begin{array}{c}{\mathrm{e}}_{11}\\ {\mathrm{e}}_{12}\\ ⋮\\ {\mathrm{e}}_{1m1}\\ {\mathrm{e}}_{21}\\ {\mathrm{e}}_{22}\\ ⋮\\ {\mathrm{e}}_{2m2}\\ {\mathrm{e}}_{n1}\\ {\mathrm{e}}_{n2}\\ ⋮\\ {\mathrm{e}}_{nmn}\end{array}\left[\begin{array}{ccccccccc}& & & & & & & & \\ & W_{11}& & & W_{12}& & & W_{1n}& \\ & & & & & & & & \\ & & & & & & & & \\ & & & & & & & & \\ & W_{21}& & & W_{21}& & & W_{2n}& \\ & & & & & & & & \\ & & & & & & & & \\ & & & & & & & & \\ & W_{n1}& & & W_{n2}& & & W_{nn}& \\ & & & & & & & & \\ & & & & & & & & \end{array}\right] \end{gathered}$$

(4)

where C_m represents the criteria m, e_nm represents the sub-criteria n included in the criteria m, and W_ij represents the priority rating of the jth under the control of the ith element. In the case of W_ij equals zero, the ith element has no influence on the jth element.

Next, the normalization of the direct relation matrix (unweighted super-matrix) is obtained so that the sum of each column equals one. As addressed by Uygun and Dede [85], the normalized direct relation matrix (weighted super-matrix) can be obtained as shown in Equations (5)–(7).

$$\overline{x}=\left[\begin{array}{cccc}{\overline{\mathrm{x}}}_{11}& {\overline{\mathrm{x}}}_{12}& \cdots & {\overline{\mathrm{x}}}_{1\mathrm{n}}\\ {\overline{\mathrm{x}}}_{21}& {\overline{\mathrm{x}}}_{22}& \ddots & {\overline{\mathrm{x}}}_{2\mathrm{n}}\\ ⋮& ⋮& \cdots & ⋮\\ {\overline{\mathrm{x}}}_{\mathrm{n}1}& {\overline{\mathrm{x}}}_{\mathrm{n}2}& \cdots & {\overline{\mathrm{x}}}_{\mathrm{nn}}\end{array}\right]$$

(5)

where

$${\overline{\mathrm{x}}}_{\mathrm{i}\mathrm{j}}=\frac{{\overline{\mathrm{w}}}_{\mathrm{i}\mathrm{j}}}{\mathrm{r}}$$

(6)

and

$$\mathrm{r}=\underset{1 \le \mathrm{i}\le \mathrm{n}}{\max } ({{\displaystyle \sum }}_{\mathrm{j}=1}^{\mathrm{n}}{\mathrm{w}}_{\mathrm{ij}})\hspace{1em}\hspace{1em}(\mathrm{i},\mathrm{j}=1,2,\dots \dots ,\mathrm{n})$$

(7)

• Step 5:Computation of the limited matrix

The final step in the ANP method is to obtain the limited super-matrix. This can be obtained by raising the power of the weighted super-matrix to a sufficiently high power k till the matrix obtains stabilization and convergence [93]. This step attempts to attain the global priorities of the criteria and sub-criteria. As mentioned by Hu et al. [82], Equation (8) is used to obtain the limited super-matrix:

$${\mathrm{w}}_{\mathrm{limit}}=\underset{\mathrm{k} \to \infty }{\lim }{{(\mathrm{w}}_{\mathrm{weighted}})}^{\mathrm{k}}$$

(8)

where k represents the used power, and w_weighted represents the weighted super-matrix.

3.3. Assessment of Different Alternatives Using Evidential Reasoning

The evidential reasoning (ER) approach is used in this study to evaluate the different procurement methods in OSC projects. The ER method is generally used to analyze multi-criteria decision-making (MCDM) problems that are associated with various types of uncertainties and incomplete information [97]. The implementation of the ER method can be done through several steps, as illustrated below.

• Step 1:Identification of assessments grades

Several evaluation grades (N) are defined in this step to assess the extent a specific alternative can satisfy a defined criterion. These grades can be described as shown in Equation (9).

$${\mathrm{H}=\{\mathrm{H}}_1, {\mathrm{H}}_2, \dots , {\mathrm{H}}_{\mathrm{n}}, \dots , {\mathrm{H}}_{\mathrm{N}}\}$$

(9)

where ${\mathrm{H}}_{\mathrm{n}}$ represents the nth evaluation grade. It is generally assumed that ${\mathrm{H}}_{\mathrm{n}+1}$ is a higher grade than ${\mathrm{H}}_{\mathrm{n}}$. In this study, the alternatives (procurement methods) are assessed through five main grades: Excellent, Very Good, Good, Fair, and Bad [88,98].

• Step 2:Construction of belief structures

The belief structures are a unified framework established to handle various types of uncertainties and incomplete information [83,99]. In belief structures, the assessment of any alternative (procurement approach) on a specific criterion can be expressed using belief degrees.

It is considered that there is a decision problem where there is a two-level hierarchy. Assume there are basic criteria (sub-criteria) g_j (j = 1, 2, …, L) inherent in main criterion G. Each of the given criteria and sub-criteria has a global normalized weight (calculated using the ANP method as illustrated in the previous section), given as ${\mathrm{w}}_1{, \mathrm{w}}_2{, \dots , \mathrm{w}}_{\mathrm{i}}{, \dots , \mathrm{w}}_{\mathrm{L}}$ (i = 1…..L), where $w_i$ is the weight of the main criterion (G_i) with $1\ge { \mathrm{w}}_{\mathrm{i}} \ge 0$ and ${\mathrm{w}}_{\mathrm{ij}}$ is the weight of the basic criterion (g_ij) $1\ge { \mathrm{w}}_{\mathrm{ij}} \ge 0$. For instance, the weight of the main criterion, Cost, is represented by $w_1$ and the weight of the second sub-criterion under cost (Funding Arrangement, g₃) is represented by ${\mathrm{w}}_{13}$. Therefore, a given assessment for g_j (j = 1, 2, …, L) under any alternative (procurement approach) can be mathematically represented as shown in Equation (10) [86,97,99].

$$\mathrm{S}\left({\mathrm{g}}_{\mathrm{i}}\right)=\{\left({\mathrm{H}}_{\mathrm{n}}{, \mathsf{\beta }}_{\mathrm{n},\mathrm{i}}\right), \mathrm{n}=1, \dots , \mathrm{N}\}$$

(10)

where $\mathrm{S}\left({\mathrm{g}}_{\mathrm{i}}\right)$ is the assessment of the criterion ${\mathrm{g}}_{\mathrm{i}}$, ${\mathrm{H}}_{\mathrm{n}}$ is the evaluation grade, and ${\mathsf{\beta }}_{\mathrm{n},\mathrm{i}}\ge 0$ and ${\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}{\mathsf{\beta }}_{\mathrm{n},\mathrm{i}} \le 1$, with ${\mathsf{\beta }}_{\mathrm{n},\mathrm{i}}$ indicating the degree of belief of a basic criterion $g_i$ to an evaluation grade $H_n$. An assessment $\mathrm{S}\left({\mathrm{g}}_{\mathrm{i}}\right)$ is considered complete when ${\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}{\mathsf{\beta }}_{\mathrm{n},\mathrm{i}}=1$ and incomplete when ${\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}{\mathsf{\beta }}_{\mathrm{n},\mathrm{i}}< 1$.

• Step 3:Construction of the ER algorithm

In this step, the ER algorithm is briefly discussed. Suppose $m_{n, i}$ is the basic probability mass that represents the extent to which a basic criterion $g_i$ supports the hypothesis that alternative $y$ is assessed to an nth assessment grade $H_n$. Thus, $m_{n, i}$ can be calculated as presented in Equation (11) [86,99].

$${\mathrm{m}}_{\mathrm{n}, \mathrm{i} }{=\mathrm{w}}_{\mathrm{ij}}{\mathsf{\beta }}_{\mathrm{n}, \mathrm{i}} \mathrm{n}=1,\dots \dots ,\mathrm{N}$$

(11)

where ${\mathrm{w}}_{\mathrm{ij}}$ is the global weight of the basic criterion ${\mathrm{g}}_{\mathrm{i}}$.

Further, in the case of not completed assessments (uncertainty), the remaining probability mass ($m_{H, i}$), which has not been assigned to any assessment grade after all assessment grades ($N$) have been considered to assess criterion $g_i$ with respect to alternative $y$, can be calculated as described by Mosleh et al. [86] in Equation (12).

$${\mathrm{m}}_{\mathrm{H}, i}=1-{\displaystyle \sum }_{\mathrm{n}=1}^{\mathrm{N}}{\mathrm{m}}_{\mathrm{n}, i}=1 -{ \mathrm{w}}_{\mathrm{ij}}{\displaystyle \sum }_{\mathrm{n}=1}^{\mathrm{N}}{\mathsf{\beta }}_{\mathrm{n}, i}$$

(12)

As defined before, ${\mathrm{G}}_{\mathrm{I}\left(i\right)}$ is the main criterion that comprises i subcriteria and can be represented, as shown in Equation (13).

$${\mathrm{G}}_{\mathrm{I}\left(\mathrm{i}\right)}{=\{\mathrm{g}}_1, { \mathrm{g}}_2 , \dots \dots ., { \mathrm{g}}_{\mathrm{i}}\}$$

(13)

Suppose ${\mathrm{m}}_{\mathrm{n}, \mathrm{I}\left(\mathrm{i}\right)}$ to be the probability mass that represents the extent to which all the basic criteria in ${\mathrm{G}}_{\mathrm{I}\left(\mathrm{i}\right)}$ support the hypothesis that $G$ is assessed to the grade $H_n$ with respect to alternative $y.$ Further, suppose ${\mathrm{m}}_{\mathrm{H}, \mathrm{I}\left(\mathrm{i}\right)}$ to represent the remaining that has not been assigned to any assessment grade after all assessment grades ($N$) have been considered to assess the criterion ${\mathrm{G}}_{\mathrm{I}\left(\mathrm{i}\right)}$ with respect to alternative $y$. The terms ${\mathrm{m}}_{\mathrm{n}, \mathrm{I}\left(\mathrm{i}\right)}$ and ${\mathrm{m}}_{\mathrm{H}, \mathrm{I}\left(\mathrm{i}\right)}$ can then be calculated by aggregating all the probability masses of the basic criteria included in the main criteria ${\mathrm{G}}_{\mathrm{I}\left(\mathrm{i}\right)}$. Equations (14)–(16) express the ER algorithm [83,99].

$${\mathrm{K}}_{\mathrm{I}(\mathrm{i}+1)}={\left[1-{\displaystyle \sum }_{\mathrm{t}=1}^{\mathrm{N}}{\displaystyle \sum }_{\begin{array}{c}\mathrm{j}=1\\ \mathrm{t}\ne \mathrm{j}\end{array}}{\mathrm{m}}_{\mathrm{t},\mathrm{I}\left(\mathrm{i}\right)}{\mathrm{m}}_{\mathrm{j},\mathrm{i}+1}\right]}^{-1}\mathrm{i}=1, 2,\dots \dots , \mathrm{L}-1$$

(14)

$${\mathrm{m}}_{\mathrm{n}, \mathrm{I}(\mathrm{i}+1)}{=\mathrm{K}}_{(\mathrm{i}+1)}\left({\mathrm{m}}_{\mathrm{n}, \mathrm{I}\left(\mathrm{i}\right)}{\mathrm{m}}_{\mathrm{n}, \mathrm{i}+1}{+\mathrm{m}}_{\mathrm{n}, \mathrm{I}\left(\mathrm{i}\right)}{\mathrm{m}}_{\mathrm{H}, \mathrm{i}+1}{+\mathrm{m}}_{\mathrm{H}, \mathrm{I}\left(\mathrm{i}\right)}{\mathrm{m}}_{\mathrm{n}, \mathrm{i}+1}\right)$$

(15)

$${\mathrm{m}}_{\mathrm{n}, \mathrm{I}(\mathrm{i}+1)}{=\mathrm{K}}_{(\mathrm{i}+1)}{\mathrm{m}}_{\mathrm{H}, \mathrm{I}\left(\mathrm{i}\right)}{\mathrm{m}}_{\mathrm{H}, \mathrm{I}(\mathrm{i}+1)}$$

(16)

where ${\mathrm{K}}_{\mathrm{I}(\mathrm{i}+1)}$ is a normalizing factor ensuring that ${\displaystyle \sum }_{\mathrm{n}=1}^{\mathrm{N}}{\mathrm{m}}_{\mathrm{n}, \mathrm{I}(\mathrm{i}+1)}{+\mathrm{m}}_{\mathrm{H}, \mathrm{I}(\mathrm{i}+1)}=1$.

Furthermore, the aggregated degree of belief (${\mathsf{\beta }}_{\mathrm{n}}$) can be determined as shown in Equations (17) and (18) [99].

$${\mathsf{\beta }}_{\mathrm{n}}{=\mathrm{m}}_{\mathrm{n}, \mathrm{I}\left(\mathrm{L}\right)} \mathrm{n}=1,\dots \dots \dots , \mathrm{N}$$

(17)

$${\mathsf{\beta }}_{\mathrm{H}}{=\mathrm{m}}_{\mathrm{H}, \mathrm{I}\left(\mathrm{L}\right)}=1 -{\displaystyle \sum }_{\mathrm{n}=1}^{\mathrm{N}}{\mathsf{\beta }}_{\mathrm{n}}$$

(18)

where ${\mathsf{\beta }}_{\mathrm{H}}$ represents the remaining degree of belief that has not been assigned to any individual assessment grade after all L basic criteria have been assessed, considering alternative y.

• Step 4:Utility intervals of the ER approach and ranking

In this study, the distributed assessments of the alternatives are not sufficient to show the difference between any two alternatives’ assessments (procurement methods), as well as the ranking of the studied alternatives. Therefore, in this step, the concept of the expected utility is introduced to generate the numerical values that are equivalent to the distributed assessments [97]. Let ${\mathrm{u}(\mathrm{H}}_{\mathrm{n}})$ represents the utility of the grade ${\mathrm{H}}_{\mathrm{n}}$, considering the following criteria shown in Equation (19)

$$\mathrm{u}\left({\mathrm{H}}_{\mathrm{n}+1}\right){ >\mathrm{u}(\mathrm{H}}_{\mathrm{n}})$$

(19)

where ${\mathrm{H}}_{\mathrm{n}+1}$ is a preferred assessment grade to $H_n$.

In cases where assessments are complete, ${\mathsf{\beta }}_{\mathrm{H}}$ will equal zero, and the expected utility of any alternative $\mathrm{y}$ can be calculated as shown in Equation (20).

$$\mathrm{u}\left(\mathrm{y}\right)={\displaystyle \sum }_{\mathrm{n}=1}^{\mathrm{N}}{\mathsf{\beta }}_{\mathrm{n}}\mathrm{u}\left({\mathrm{H}}_{\mathrm{n}}\right)$$

(20)

Considering there are two alternatives ${\mathrm{y}}_{\mathrm{a}}$ and ${\mathrm{y}}_{\mathrm{b}}$, it is believed that alternative $y_a$ is better than ${\mathrm{y}}_{\mathrm{b}}$ only if $\mathrm{u}\left({\mathrm{y}}_{\mathrm{a}}\right){ >\mathrm{u}(\mathrm{y}}_{\mathrm{b}})$. However, in uncertain cases, where the assessments of basic criteria are not complete, the value of ${\mathsf{\beta }}_{\mathrm{H}}$ will be positive. Therefore, three main measures can be adopted to express the assessment of alternative $y$: the minimum, maximum, and average expected utilities of the assessments [99]. Let ${\mathrm{H}}_1$ be the least preferred grade that has the lowest utility. On the other hand, suppose ${\mathrm{H}}_{\mathrm{N}}$ is the most prefered grade that has the greatest utility. Therefore, the three measures of utility for an alternative $y$ can be calculated as shown in Equations (21)–(23).

$${\mathrm{u}}_{\max }\left(\mathrm{y}\right)={\displaystyle \sum }_{\mathrm{n}=1}^{\mathrm{N}-1}{\mathsf{\beta }}_{\mathrm{n}}\mathrm{u}\left({\mathrm{H}}_{\mathrm{n}}\right)+\left({\mathsf{\beta }}_{\mathrm{n}}{+\mathsf{\beta }}_{\mathrm{H}}\right)\mathrm{u}\left({\mathrm{H}}_{\mathrm{N}}\right)$$

(21)

$${\mathrm{u}}_{\min }\left(\mathrm{y}\right)=\left({\mathsf{\beta }}_1{+\mathsf{\beta }}_{\mathrm{H}}\right)\mathrm{u}\left({\mathrm{H}}_1\right)+{\displaystyle \sum }_{\mathrm{n}=2}^{\mathrm{N}}{\mathsf{\beta }}_{\mathrm{n}}\mathrm{u}\left({\mathrm{H}}_{\mathrm{n}}\right)$$

(22)

$${\mathrm{u}}_{\mathrm{avg}}\left(\mathrm{y}\right)=\frac{{\mathrm{u}}_{\max }\left(\mathrm{y}\right){+\mathrm{u}}_{\min }\left(\mathrm{y}\right)}{2}$$

(23)

3.4. Data Collection

Structured questionnaires were used to collect the required data needed to establish the desired multi-criteria model. Three main stages were involved in this phase: the design of the questionnaire, a pilot study, and a full-scale study. The following subsections will illustrate each stage of data collection.

3.4.1. Questionnaire Design

It is essential to identify the required data to establish the multi-criteria model. As described in the above sections, the relative importance of criteria and sub-criteria are needed to perform the ANP technique. In addition, the assessment of each procurement system against each criterion and sub-criterion is required to perform the ER technique. Consequently, participants were asked to provide two different types of information. First, the participants were asked to provide their judgments of each criterion and sub-criterion using pairwise comparison matrices. This information was input into the ANP technique to compute the relative importance of each criterion and sub-criterion. Secondly, participants were also asked to provide their perceptions and assessments of different procurement methods’ performance in OSC projects with respect to each criterion and sub-criterion. This information was used to calculate the belief matrices and expected utilities using ER technique. In addition, the hierarchy of identified criteria and sub-criteria, as well as their identified interdependencies, were built into the Super Decisions software tool. All pairwise comparisons identified by the Super Decisions tool for criteria and sub-criteria were checked and included in the designed questionnaire. The questionnaire also included all required assessment grades for each procurement approach according to each criterion and sub-criterion. Other questions related to participants’ personal information and years of experience were also included in the questionnaire.

After identifying the required information for the MCDM model, the questionnaire was formulated and ordered. The questionnaire formulation was driven in accordance with previous MCDM studies [80,83,86] to ensure the appropriate design and clarification of the included parts. Four main parts were included in the designed questionnaire: (1) gathering personal information of the participants (i.e., years of experience and their most experienced type of OSC techniques); (2) illustrating the collected parameters and their categorization into sub-criteria and main criteria; (3) establishing the pairwise comparison matrices needed to compute the relative importance; (4) assessing different procurement methods regarding different criteria and sub-criteria.

Table 3. A sample of the pairwise comparison matrix.

With Respect to Cost Criteria
	The Relative Importance of the Compared Criteria
Factor X	(9) Absolute	(7) Very Strong	(5) Strong	(3) Moderate	(1) Equal	(3) Moderate	(5) Strong	(7) Very Strong	(9) Absolute	Factor Y
Cost estimating and control	-	-	-	X	-	-	-	-	-	Funding arrangements
With Respect to Time Criteria
	The Relative Importance of the Compared Criteria
Factor X	(9) Absolute	(7) Very Strong	(5) Strong	(3) Moderate	(1) Equal	(3) Moderate	(5) Strong	(7) Very Strong	(9) Absolute	Factor Y
Execution schedule control	-	-	-	-	-	-	X	-	-	Procurement plan

represents a sample of a pairwise comparison matrix conducted in part three of the questionnaire. This part was incorporated with instructions to guide participants in how to complete the pairwise comparison matrices, as shown in Figure 5. The pairwise comparison matrices were designed in a way that facilities the participant’s decision in indicating the degree of importance of criterion (X or Y) compared to the other with respect to a common goal. The scale of importance was stated in accordance with Saaty [100], from 1 to 9. Furthermore, in the last part of the questionnaire, participants were asked to evaluate different procurement methods with respect to each criterion and sub-criterion. Five main procurement methods were identified in this study to be assessed in this part. As shown in Figure 6, an assessment grade from 1 to 5 was adopted in the designed questionnaire that complies with the ER technique used in the data analysis [99]. It is worth mentioning that participants may choose not to assess a particular factor if they find it irrelevant. This part was also associated with instructions to participants to guide them into completing the assessment process, as shown in Figure 7. An assessment grade of 5 indicates that the understudy procurement approach strongly satisfies this sub-criterion. On the other hand, an assessment grade of 1 indicates that the understudy procurement approach poorly satisfies this sub-criterion.

3.4.2. Pilot Study

The included criteria, their interdependencies, and way of assessment should be tested prior to conducting a full-scale study [80]. Therefore, a pilot study was conducted at this stage to ensure the eligibility of the proposed questionnaire and gather suggestions from OSC experts. The pilot test also ensures the eligibility of the proposed questionnaire, such as the clarity of the questions, the appropriateness of their arrangement, and the presence of the necessary instructions. It can also help include more parameters that have not been considered prior to the full-scale study. A pilot study usually includes a small number of participants who provide their feedback on the questionnaire, which will be adjusted accordingly.

Therefore, four OSC experts were chosen to provide their comments on the proposed questionnaire. The participants in the pilot study include two academic members who specialized in OSC techniques, one OSC contractor, and one OSC consultant. The feedback did not report any mistakes, ambiguity, or disorder in the proposed questionnaire. However, there were concerns about the length of the questionnaire and the needed time to fill it. This can be justified by the included pairwise comparisons that measure the degree of importance for five main criteria and 15 subcriteria (comprised of 55 parameters). Further, the questionnaire also measures the evaluation of each identified procurement approach. All of this information included in the questionnaire calls for a lengthy questionnaire to gather the needed data for analysis.

3.4.3. Full-Scale Study

Having ensured the eligibility of the questionnaire, the full-scale study was distributed to OSC experts. These experts included contractors, manufacturers, consultants, and research institutions. A total of 58 experts were invited to participate in the proposed questionnaire over a period of six months. Only 21 responses were obtained, with a response rate of 36.2 %. For the eligibility of the questionnaire, the response rate should have specific limits. Baruch [100] stated that the response rate of a questionnaire survey should lie in the range of (55.6$\pm$ 19.7)%. Hence, the response rate in this study fulfills this criterion. Furthermore, the sample size of the questionnaire should comply with its purpose. According to Abdelkhalek and Zayed [80] and Waris et al. [101], the sample size of a survey that includes the AHP method, or similar, does not have to be large. In addition, several studies found in the literature conducted the AHP method with samples ranging from 10 to 25 responses [80,89,102].

As illustrated above, the first section of the questionnaire depicts the participants’ information. Figure 8 illustrates and summarizes this information. The majority of the participants exist in Hong Kong, where this study is conducted. Consequently, with the vast intention of modular construction projects lately in Hong Kong, most participants have indicated their areas of expertise as modular construction. Furthermore, 62% of the participants were related to the industry, including consultants, onsite contractors, and manufacturers. In addition, more than 50% of the respondents have more than five years of experience. Although this number might sound low, it can be justified by the fact that modular construction and many other OSC types are newly introduced to practitioners, such as in Hong Kong, where modular construction was introduced in 2017.

4. Results and Discussion

4.1. Importance Weights of Criteria and Subcriteria

The ANP method was implemented in this stage to determine the importance weights of criteria and subcriteria. As illustrated in Section 3.2, the ANP method is performed in four main stages: (1) development of unweighted supermatrix; (2) development of weighted supermatrix; (3) estimation of the limit matrix; and (4) assignment of local and global wrights of criteria and subcriteria. The formulation of equations illustrated in Section 3.2 was conducted on Excel sheets to determine the weighted and unweighted supermatrix. However, the limit matrix requires numerous iterations until the power that satisfies the convergence is defined. Therefore, a Python code associated with Excel sheets was developed to derive the limit matrix and aggregate the results of the respondents.

Table 4. The derived limit matrix of criteria and subcriteria.

	Goal	Cost	Owner’s Characteristics	Project’s Characteristics	Relationships and Processes	Time
Cost	0.168	0.168	0.168	0.168	0.168	0.168
Owner’s characteristics	0.151	0.151	0.151	0.151	0.151	0.151
Project’s characteristics	0.335	0.335	0.335	0.335	0.335	0.335
Relationships and processes	0.229	0.229	0.229	0.229	0.229	0.229
Time	0.118	0.118	0.118	0.118	0.118	0.118
Cost estimating and control	0.086	0.086	0.086	0.086	0.086	0.086
Funding arrangements	0.081	0.081	0.081	0.081	0.081	0.081
Execution’s schedule control	0.073	0.073	0.073	0.073	0.073	0.073
Procurement plan	0.045	0.045	0.045	0.045	0.045	0.045
Avoid claims and disputes	0.046	0.046	0.046	0.046	0.046	0.046
Collaborative processes	0.076	0.076	0.076	0.076	0.076	0.076
Integration	0.054	0.054	0.054	0.054	0.054	0.054
Risk/rewards sharing	0.052	0.052	0.052	0.052	0.052	0.052
Prefabrication processes	0.066	0.066	0.066	0.066	0.066	0.066
Project change	0.046	0.046	0.046	0.046	0.046	0.046
Project environment	0.042	0.042	0.042	0.042	0.042	0.042
Project quality	0.091	0.091	0.091	0.091	0.091	0.091
Project’s regularity	0.089	0.089	0.089	0.089	0.089	0.089
Owner’s capabilities	0.086	0.086	0.086	0.086	0.086	0.086
Owner’s preferences	0.065	0.065	0.065	0.065	0.065	0.065

shows the limit matrix developed after the assembly process. It is worth mentioning that this table does not show the entire limit matrix due to space restrictions. The highlighted parts represent the weights of the criteria and subcriteria within each cluster. The results of the limit matrix were verified using the super decision software tool. More about verification and validation is illustrated in Section 5.

As shown in Table 4, the final global weights of the main criteria indicate the significance of the project characteristics criterion, with a global weight of 0.335, in determining the suitable procurement approach for OSC projects. This criterion is followed by the relationships and processes, cost, owner’s characteristics, and schedule, with global weights of 0.229, 0.168, 0.151, and 0.118, respectively. The subcriteria included in each main criterion are influenced by the weight of the main criteria. Therefore, the final global weights were determined by proportioning the criteria and subcriteria of each cluster to themselves. This step ensures that the summation of all criteria and subcriteria equals 1.

Figure 9 shows the weights of the subcriteria distributed based on their significance in three main colors: red, orange, and yellow. The red color represents the significant factors with a global weight of more than 0.08. The orange color indicates a medium significance and includes subcriteria that fall into a global weight range of 0.05 to 0.08. Finally, the yellow color indicates the insignificant subcriteria with global weights less than 0.05. As can be seen from Figure 9, project quality is the most significant subcriteria with a global weight of 0.092, followed by the project’s regularity, cost estimating and control, owner’s capabilities, and funding arrangements with global weights of 0.089, 0.091, 0.086, and 0.085, respectively. Further, project conditions and procurement plan were the least significant subcriteria, with global weights of 0.034 and 0.032, respectively.

4.2. The Assessment of the Procurement Methods

Each of the identified procurement methods was evaluated in this step based on each criterion and subcriterion using the ER technique. As illustrated in Section 3.3, the ER technique includes the four main stages: (1) the development of belief structures; (2) the development of probability masses; (3) the aggregation of the assessments in each cluster and deriving the overall assessment of the cluster; and (4) the calculation of expected utility interval for each procurement system. The assessment grades of the procurement methods are distributed in five main grades: Excellent, Very Good, Good, Fair, and Poor. Furthermore, the assessment also includes an ignorance grade for missing or incomplete information. The prioritization of each procurement approach was developed based on the criteria and sub-criteria weights resulting from the ANP technique. A complete implementation of the ER to assess the DBB system is presented in the Supplementary Material.

All of the steps mentioned in the Supplementary Material are repeated for each procurement approach type. In addition, the expected utility intervals are calculated for each subcriterion and main criteria in assessing each of the procurement methods so that they can be compared. Figure 10 illustrates the average utility of the procurement methods according to the five main criteria. As can be seen, the IPD approach shows a high utility in most criteria, especially “relationships and processes.” Further, DB Approach indicates a high utility in the “cost” and “schedule” criteria, with medium support to the “relationships and processes” criterion. The NB approach indicates a similar assessment as the DB approach, except for lower support for the “schedule” criterion. It also highly supports the “owner’s characteristics” criterion. The CM approach highly supports the “project’s characteristics” criterion, with poor support for the “cost” criterion. In addition, the CM approach is the most supporting approach for the “owner’s characteristics” criterion. On the other hand, the DBB approach shows a low utility in almost every criterion, except for the “owner’s characteristics” criterion, where the DBB has an average utility.

Table 5. Utility intervals for individual procurement types in supporting the subcriteria.

Subcriteria	Utility Intervals	Procurement Methods
		DBB	DB	CM	IPD	NB
Cost estimating and control	$u_{max}$	3.15	3.90	3.42	3.68	3.11
	$u_{min}$	3.15	3.90	3.00	3.68	2.68
	$u_{avg}$	3.15	3.90	3.21	3.68	2.89
Funding arrangements	$u_{max}$	3.11	2.95	3.32	3.58	3.00
	$u_{min}$	3.11	2.95	2.89	3.58	2.79
	$u_{avg}$	3.11	2.95	3.11	3.58	2.89
Execution’s schedule control	$u_{max}$	2.37	3.58	3.95	3.89	3.32
	$u_{min}$	2.37	3.58	3.74	3.89	2.89
	$u_{avg}$	2.37	3.58	3.84	3.89	3.11
Procurement plan	$u_{max}$	3.16	3.53	3.53	3.68	3.06
	$u_{min}$	2.92	3.53	3.32	3.68	2.95
	$u_{avg}$	3.04	3.53	3.42	3.68	2.84
Avoid claims and disputes	$u_{max}$	2.16	3.47	3.32	4.00	2.94
	$u_{min}$	2.16	3.47	3.11	4.00	2.84
	$u_{avg}$	2.16	3.47	3.21	4.00	2.73
Collaboration processes	$u_{max}$	1.95	3.31	3.42	4.36	3.05
	$u_{min}$	1.95	3.31	3.32	4.36	2.84
	$u_{avg}$	1.95	3.31	3.21	4.36	2.95
Integration	$u_{max}$	2.27	3.21	3.00	4.16	3.32
	$u_{min}$	2.05	3.21	2.79	4.16	3.11
	$u_{avg}$	2.16	3.21	2.89	4.16	3.21
Risk/rewards sharing`	$u_{max}$	1.74	2.42	3.00	4.21	2.84
	$u_{min}$	1.74	2.42	2.79	4.21	2.63
	$u_{avg}$	1.74	2.42	2.89	4.21	2.74
Prefabrication processes	$u_{max}$	2.47	3.58	3.21	4.00	3.47
	$u_{min}$	2.47	3.58	3.00	4.00	3.26
	$u_{avg}$	2.47	3.58	3.11	4.00	3.37
Project change	$u_{max}$	2.32	3.11	3.26	3.69	2.95
	$u_{min}$	2.32	3.11	3.16	3.69	2.74
	$u_{avg}$	2.32	3.11	3.06	3.69	2.84
Project’s regularity consideration	$u_{max}$	2.68	3.11	3.58	3.42	3.21
	$u_{min}$	2.68	3.11	3.37	3.42	2.89
	$u_{avg}$	2.68	3.11	3.47	3.42	2.58
Project quality	$u_{max}$	3.00	3.32	3.84	3.89	3.37
	$u_{min}$	3.00	3.32	3.63	3.89	3.16
	$u_{avg}$	3.00	3.32	3.74	3.89	3.37
Project conditions	$u_{max}$	2.36	3.26	3.68	3.78	3.32
	$u_{min}$	2.16	2.84	2.84	3.16	2.47
	$u_{avg}$	2.26	3.05	3.26	3.47	2.89
Owner capabilities	$u_{max}$	3.31	3.37	3.68	3.89	3.59
	$u_{min}$	3.11	3.16	3.26	3.68	2.94
	$u_{avg}$	3.21	3.26	3.47	3.78	3.26
Owner preferences	$u_{max}$	3.37	3.16	3.53	3.53	3.31
	$u_{min}$	3.37	3.16	3.32	3.53	3.21
	$u_{avg}$	3.37	3.16	3.42	3.53	3.11

shows the utility intervals for each procurement type according to each identified subcriterion. As can be seen, each assessment is represented in three main values: maximum, minimum, and average. The reason behind the variety of assessments is the possible incompleteness of the information provided by the participants. The “project quality” subcriterion, i.e., the highest weighted factor by the ANP technique, can be highly supported by adopting the CM approach, followed by IPD and DB methods. In addition, the IPD approach has provided the best performance in assessing the “funding arrangements” subcriterion, which is the fifth-highest weighted factor based on the ANP technique. This subcriterion was also highly supported by the adoption of DB and DBB, with average utilities of 2.95 and 3.11, respectively. The DB approach also provided the best performance in assessing the “Cost Control” subcriterion, with an average utility of 3.90. In addition, IPD showed a dominant performance in assessing the subcriteria included in the ‘relationships and operations’ criterion, with a utility average of over 4 for all subcriteria.

To better understand the performance of the procurement methods, Figure 11 visualizes and compares the performance of each procurement approach according to each subcriterion. The DBB is represented in orange color, the DB approach in blue color, the CM approach in grey color, the NB approach in red color, and the IPD approach in green color. Decision-makers can easily refer to the performance of each sub-criterion according to their preferences. For instance, in situations that need a high level of integration and collaborative work, decision-makers may adopt the IPD approach in procuring their OSC projects. Similarly, when the high quality of the OSC project is the highest priority to the decision-makers (clients), they may select the CM approach.

5. Model Verification and Validation

5.1. Model Verification

To ensure the correctness of the developed model and assess whether it effectively reflects its targeted objectives, model verification is performed [103]. Various verification methods were adopted in the developed model, including collected data verification and analysis verification.

5.1.1. Collected Data Verification

The consistency ratios (CR) of the retrieved responses were calculated to exclude inconsistent ones. As advised by Saaty [92], if the CR value exceeds 0.1, then the response is unreliable, and the comparisons need to be repeated. Regarding this matter, the calculated consistency values fell between 0.011 and 0.092. Furthermore, to check the reliability of the collected data from a small sample, like in this case, Saaty [104] proposed a reasoning approach based on Chebyshev’s theorem statistical test. This theorem indicates that at least 75% of the data must fall within a $\pm 2$ standard deviation range to be reliable. Thus, Chebyshev’s theorem statistical test was adopted in this study to check the reliability of the obtained weights of criteria and subcriteria. The test results indicate that a range from 85.67% to 100% of the obtained results lies within a $\pm 2$ standard deviation range. Several studies have used this method to evaluate the reliability of the collected data from small sample sizes [80,105,106].

5.1.2. Analysis Verification

As illustrated in the previous sections, the ANP calculations were conducted using two approaches: Python codes associated with Excel sheets and the SuperDecisions software tool. Figure 12 illustrates the proposed approach. The Excel sheets were used initially to calculate the unweighted and weighed super matrices. Then, the calculated weighted super matrices were exported to the developed Python code. Following that, the Python code runs several iterations until the matrix is converged. The Limit matrix is then exported to the Excel sheets to conduct the ER technique. The Super Decisions software tool was used to calculate the unweight, weight, and limit matrics. This process is carried out for selected responses in this study to verify the results obtained by the Python code. The obtained limit matrics are then assembled to derive the local and global weights of the criteria and subcriteria.

5.2. Model Validation

In order to check the validity of the MCDM model and ensure that it provides accountable measures for OSC practitioners, a validation of the model is conducted. A validation model can be defined as a comparison of the model’s results with real-world practices to determine its suitability for its intended objectives [107]. Several validation techniques can be utilized for model validation. One of the validation techniques is to compare the developed model’s results with the results of similar studies. This method is adopted in this section to validate the study results. Although no MCDM models for assessing procurement methods in OSC projects have been found in the literature, some articles have studied some aspects of these procurement methods. Two main studies are introduced in this section to validate the MCDM model results. It is also worth mentioning that none of the found studies has considered the IPD procurement approach.

The first study used in the current study validation was conducted by Perera et al. [20] to assess the procurement methods in steel construction projects in Sri Lanka. Although it does not precisely match the topic of this study, their study included prefabrication-related factors, such as transportation and manufacturing processes of steel sections. They identified 26 parameters that may impact the selection of the procurement methods. Three main procurement methods were studied in their study: the traditional approach (DBB), DB, and CM. The study ranked the collected procurement parameters and assessed the considered procurement methods. Their results indicate the following: (1) the CM method provided the best performance in project quality, short construction schedule, government policies, and the client’s requirement; (2) the DBB approach had the lowest performance in almost all the identified parameters; (3) the DB approach overweighted other procurement methods in client’s financial capability parameter.

The above results generally validate the results obtained by the developed MCDM model in this study. The results of the current study indicate that CM and IPD methods have provided the best performance in the following criteria: project quality, project regularities’ consideration, execution schedule control, and owner’s preferences. In addition, the MCDM model indicates that the DBB approach has the lowest performance in OSC projects, which complies with the first validation study. Furthermore, the NB and IPD approaches have obtained the best performance in assessing the owner’s capabilities subcriterion. Although the DB was the best-performing approach in assessing the same subcriteria in the above study’s results, the above study did not consider the NB approach. The NB approach is similar to the DB approach, but the main party in charge is the manufacturer [9].

The second study used in the current study validation was conducted by Charlson and Dimka [9]. Their study targeted the procurement methods in OSC projects in the UK housing sector. Focus groups were formed, interviews were carried out in this study, and the ground theory was used to generate new themes of procurement methods. The results of the second study used in the validation indicated that the tender processes included in the DBB approach are quite lengthy and do not comply with the OSC projects. The study also asserted the need for partnerships that could foster the implementation of OSC projects. These findings by the mentioned study validate the findings obtained by the current research carried out by the MCDM model. Most participants in this study have indicated low-performance grades for the DBB approach. In addition, most participants have proposed IPD as a potential procurement method that can overcome the fragmentation in the OSC projects. The IPD was the best-performing approach, especially in the risk/reward sharing factor, which complies with the second validating study.

Furthermore, the mentioned study indicated that the default used procurement approach in the UK construction industry is the DB. Using the ground theory, the mentioned study has proposed a procurement model called design, manufacturer, and construct (DMC), which was similar to the DB system, but the manufacturer replaced the main contractor. This procurement method was included and evaluated in this study under the name of the negotiated bid (NB), as described by Schoenborn [23]. Charlson and Dimka [9] have tested the DMC model and concluded the following: (1) the DMC procurement approach promoted the integration between different sectors in OSC projects; (2) the DMC model supported the client’s requirements and early involvement of all parties; (3) the DMC model had weakly supported of the collaboration between the client’s advisors and the manufacturer’s in-house designer.

The above results of evaluating the DMC procurement approach validate the results of the developed MCDM model. The DMC approach, referred to as NB in this study, has provided the second-best performance in assessing the integration subcriterion. It was also the second-ranked approach, after the CM approach, in supporting the client’s preferences subcriterion. In addition, the result of the MCDM model also indicated a lower performance of the NB approach in the collaboration sub-criterion. This validates the findings of Charlson and Dimka [9], who recommended further collaboration between the client’s advisor and the manufacturer’s in-house designer.

6. Conclusions

This study is motivated by the lack of an investigation into OSC procurement methods. Therefore, this study sought to prioritize the selection criteria for OSC procurement methods and assess each procurement approach according to the identified criteria. An MCDM model is developed to evaluate the performance of procurement methods in OSC projects. The model incorporates 55 parameters, including newly identified parameters that reflect the distinguishing features of the OSC technique. These parameters are categorized into criteria and subcriteria. A questionnaire survey was developed to collect the data needed in the MCDM model. To analyze the collected data, two main techniques are used: ANP and ER techniques. The ANP is used to calculate the local and global weights of the criteria and subcriteria, considering their interdependencies. On the other hand, the ER technique is used to assess the procurement methods according to the identified individual criteria and subcriteria. In addition, the ER is used to calculate the utility intervals to prioritize the considered procurement methods.

The results obtained from the MCDM approach show that the project characteristics criterion was the most significant factor in selecting the appropriate procurement approach in OSC projects, with an importance weight of 0.334, followed by the relationships and processes criterion (0.229), and then the cost criterion (0.167). Furthermore, the model results indicate that the project quality, project’s regularity, cost control, and funding arrangements subcriteria achieve the highest importance weights among other subcriteria. As for assessing the procurement methods, CM and IPD are first ranked under assessing the schedule criterion, whereas CM is the first ranked under the owner’s characteristics criterion. For the rest of the criteria, the IPD approach achieves the highest performance grade. Overall, the IPD gets the highest aggregated assessment grade (3.83), followed by CM (3.32), and then NB (3.19). The results of the developed model are validated by comparing the model’s results with similar studies from the literature.

The current study extends the body of knowledge by the following contributions: (1) identifying the main criteria that govern the selection of the procurement methods in OSC projects; (2) designing a framework to evaluate the performance of various procurement methods in OSC projects, i.e., the MCDM model; (3) assessing five procurement methods and prioritizing them according to selection criteria, i.e., cost and schedule. Despite the contribution of this research, the study still has some limitations that should be addressed. Although the number of gathered responses is enough to obtain results, the reliability of these results could be enhanced by increasing the number of responses. Although most participants stated that the questionnaire survey covered the different aspects of the procurement methods, some participants criticized the length of the questionnaire survey. In addition, since OSC is a bit new construction method, it is challenging to find experts in this domain. Further, the MCDM model only considered one source of data, i.e., qualitative data using questionnaire surveys. Although this approach reflected OSC experts’ perception of procurement methods under various criteria, adopting more data sources, such as lessons learned from previous case studies, could drive more confidence in the findings of the developed model. Future research may target a larger sample size to increase the accuracy of the developed MCDM model used to assess procurement methods in OSC projects. In addition, future work may incorporate lessons learned on procurement methods from previous OSC case studies.