Fuzzy Comprehensive Evaluation of Mixed Reality Seismic Retrofitting Training System

Abstract

Due to the complexity of the construction environment and retrofitting methods, it is difficult to achieve the expected retrofitting effect. Therefore, effective seismic retrofitting training is a necessary way to ensure retrofitting workers acquire enough professional knowledge, skills and safe behaviors, which are critical to retrofitting. Mixed reality has huge potential in construction training. This paper conducts a fuzzy comprehensive evaluation of the MR seismic retrofitting training system to research the potential of MR in training. The purpose of this research was to provide scientific guidance and reference for the development, improvement and selection of MR training systems in the future. In this research, the evaluation indicators of the MR training system were firstly analyzed. Next, the weight of each evaluation indicator was calculated by the judgment matrix. Then, the evaluation model was established based on the qualitative–quantitative transformation principle of indicators. Finally, the results of the MR seismic retrofitting training system are obtained by combining the evaluation model with the membership matrix. The evaluation result of the training system in this paper is excellent.

1. Introduction

China is one of the countries most affected by earthquakes in the world. High frequency, high intensity, wide distribution and severe disasters are the characteristics of an earthquake. Therefore, earthquake prevention and disaster reduction have always been arduous tasks all over the world. The 2008 Wenchuan earthquake, with an epicenter intensity of XI, caused instant landslides, buildings collapsed, buildings were damaged and countless secondary disasters such as landslides and rockfalls, as well as damage to the ecological environment. The Wenchuan earthquake caused a huge loss of life and property, and nearly 462,000 people were sacrificial or missing. Countless buildings were damaged, and countless people lost their lives instantly [1]. In 2021 alone, there were 115 earthquakes of magnitude 6 or above in the world, including 19 earthquakes of magnitude 7 or above; the largest magnitude of an earthquake was 8.1 in the waters south of Alaska, the USA, on 29 July.

The destruction of buildings is one of the major consequences of earthquakes, ranging from cracks and deformations to severe collapse. Post-earthquake buildings have unique social and scientific value. Many scholars have been actively engaged in the research on seismic buildings reinforcement and retrofitting so as to realize the effectively protect and sustainable utilization of buildings. Due to the complexity of earthquakes and the irreversible consequences of seismic damage, many retrofitting methods of post-earthquake buildings always involve many new procedures and new tools, which may cause many inexperienced construction workers to be prone to errors and injuries.

Previous studies have shown that most construction accidents are caused by a lack of proactive and preventive measures, such as safety education. Even if the corresponding preventive measures are taken, because the workers lack professional knowledge and a low-efficiency knowledge grasp of the measures, it is difficult to guarantee the effectiveness of these preventive measures. In order to promote the knowledge and grasp of retrofitting methods and preventive measures, workers must accept efficient construction training in advance.

Situated learning is a learning theory proposed by American scholar Jean Lave [2]. This learning theory believes that the learning effect depends on the interaction between the learner, the environment and the machines in the learning situation. In other words, situational learning emphasizes the deep integration of learning and application by presenting knowledge in real situations so that learners can acquire knowledge in practice. The Mixed Reality (MR) training system can create a real learning environment for trainees, allowing trainees to experience and acquire knowledge in real scenes [3,4]. Thus, applying MR to retrofitting training can achieve situated learning. On the one hand, MR can solve the problems of poor visualization and poor interaction, which exist in face-to-face and paper training. On the other hand, with the raging of the COVID-19 pandemic, traditional offline training is no longer suitable for the epidemic era. Applying MR to retrofitting training can effectively isolate the virus among workers.

MR training system has unique characteristics, such as immersion, interaction and imagination (3I), which are difficult to achieve by other training methods [5]. In order to investigate whether MR training can indeed effectively promote the grasp of knowledge, skills and safe behaviors in retrofitting [6], this paper conducts a comprehensive evaluation of the MR seismic retrofitting training system, which is based on the theory of fuzzy mathematics and combines quantitative analysis-qualitative analysis. The comprehensive evaluation results can provide scientific guidance for the development, improvement, application and selection of the MR training system in the future.

2. Literature Review

2.1. The Extended Reality Technology in Construction Engineering

Construction engineering provides numerous jobs for millions of people around the world every year. However, fatal accidents in construction tend to be higher than in other industries [7]. The reasons for this phenomenon are that preventive measures (such as education and training) are often lacking in construction engineering, and the preventive measures are inefficient, which results in the lack of relevant knowledge, skills and safety behaviors of construction personnel [8]. In order to solve these problems, extended reality (virtual reality, augmented reality and mixed reality) has been introduced to advance the training development of construction engineering [9,10].

The core of virtual reality (VR) is to simulate immersive environments [11,12]. As the concept of VR was established in 1989, more and more research about VR appeared [13]. Sneha Bhoir et al. [14] conducted a literature survey on the adoption rate of virtual reality environments (VRE) for safety training. This survey aimed to explore the current adoption rate of VRE. Through the literature review and interviews with safety training organizations across the U.S., the authors found that the rate of VRE adoption was currently low, and construction workers preferred hands-on training. Augmented reality (AR) realizes the connection between the virtual and real worlds based on VR [15,16,17]. A. H. Behzadan et al. [18] developed a teaching system integrating remote videotaping, AR and ultra-wideband (UWB). This teaching system realized the interaction between the classroom and the construction site, which provided more intuitive information for students to learn.

MR contains the advantages of VR and AR. Moreover, MR can also realize collaborative work based on immersive and virtual–real interaction [19]. Therefore, MR was introduced to construction personnel training [20,21]. W. Wu et al. [22] researched the effectiveness of MR training systems in promoting students’ understanding of wooden structures. By comparing the performances of students using traditional paper construction drawings and using the MR system, the authors confirmed that the MR training system could effectively reduce the skills gap among construction workers. Justin Hartless et al. [23] compared the evaluation results of a house design scheme between beginners and experts, and the authors concluded that both beginners and experts had a high consistency in the understanding and application after training with the MR system. Thus, MR can be used to make up for the gap in the professional knowledge and skills of construction workers.

2.2. Research on Construction Retrofitting Training in the Construction Industry

The construction industry faces major challenges in safety issues, including buildings safety, construction safety and workers’ safety [24]. Carrying out construction retrofitting training and safety training in advance can improve the knowledge, skills and safety awareness of construction workers, which are useful to improve the safety of the construction industry. Construction training is considered an important way to reduce safety risks and improve workers’ ability to deal with emergencies [25]. The existing construction training methods mainly for retrofitting include lectures, conferences, video and paper training [26]. The most prominent shortcoming of these traditional training methods is that it is difficult for workers to participate fully in the learning environment, so their access to retrofitting information through memory may be hindered [27]. With the advent of Industry 4.0, construction retrofitting training also gradually has highly immersive and real-time interactive [28].

Shin et al. [29] conducted a wayfinding experiment using VR to narrow the knowledge gap among firefighters. The experiment results indicated that the firefighter using VR had a better wayfinding performance and obtained the route and survey spatial information more effectively. These results would help professionals design better training schemes for firefighters. Eiris et al. [30] researched the impact of VR on fall hazard recognition. Research results showed VR training, which had an immersive experience, took less time to acquire hazard knowledge. M. Wolf et al. [31] introduced AR to study the mechanism of data generating automatically. All the generated data can serve as personalized feedback for trained workers, which reduces the safety risks effectively in the next task.

As shown in Section 2.1, in addition to immersive and virtual–real interaction, MR can also perform multi-machine and multi-person collaborative work. Khaled El Ammari et al. [32] developed an MR system to support the interactive and collaborative management of construction equipment. The research realized the interactive visual collaboration of on-site workers and remote workers. The remote workers obtain the location of on-site workers through the MR system, which can formulate the inspection and maintenance task, and the remote workers can send the task to on-site workers through this MR system. Meanwhile, the inspection and maintenance results can be fed back to the remote workers in real time. Minze Chen et al. [33] built an MR space of virtual pedestrians in a real environment. Real-time interaction between virtual pedestrians and the real environment provided a new idea for high-risk evacuation research. Manas and Kawai et al. [34,35] also used a similar simulation approach to visualize scenarios during earthquakes and building fires, which aimed to improve the effectiveness of earthquake drills and fire evacuation drills.

2.3. The Fuzzy Comprehensive Evaluation in Construction Engineering

The fuzzy comprehensive evaluation method is a comprehensive evaluation method based on fuzzy mathematics theory; this evaluation method was proposed by Zadeh in 1965 to describe the uncertainty and ambiguity of things [36,37]. This evaluation theory was first applied to the economic field. With the development of fuzzy set theory, the fuzzy comprehensive evaluation method was also developed rapidly [38]. Thus far, the evaluation method has been applied to many fields, including construction engineering [38]. The fuzzy comprehensive evaluation firstly determines the levels of the analysis object through qualitative analysis and quantitative analysis, then makes an overall evaluation based on the influencing factors in each level [39,40].

Huang et al. [41] made a reasonable evaluation of green construction for construction enterprises. The authors evaluated the index system of 17 secondary indicators and five primary indicators (land saving, energy saving, material saving, water saving and environmental protection). The research by Huang showed that although the relationship between the indicators and green construction was fuzzy, the appliance of the membership function made it feasible to use the fuzzy comprehensive evaluation method to evaluate green construction. Zhang et al. [42] made a fuzzy comprehensive evaluation of a VR system used for mine safety training. The paper first determined the evaluation index system of the VR mine training system, and then the experts and students jointly determined the judgment matrix and weight indicators. Next, the comprehensive evaluation of the VR training system was moderate, and finally, the authors analyzed and discussed the reasons for this evaluation result. Meanwhile, this paper provided suggestions for improving the effectiveness of mining safety training systems in the future.

In summary, the fuzzy comprehensive evaluation method can not only determine the membership of each evaluation indicator but also comprehensively evaluate the evaluation object based on fuzzy mathematical theory. In actual construction projects, due to the limitations of technology, cost, time and other factors, the evaluation results are often not good, which means that the evaluation object still has space for improvement. Thus, according to the comprehensive evaluation results, reasonable optimization suggestions can be provided for construction engineering.

2.4. Research Gap

The complexity and diversity of the retrofitting environment make workers exposed to many potential risks [43]. In order to reduce construction risks, seismic retrofitting training in advance is very necessary. However, the current research on whether the training system can effectively improve the knowledge, skills and safe behavior of retrofitting workers is indeed limited. The existing studies on retrofitting training are mostly on seismic retrofitting training with VR and AR, which cannot work collaboratively. Even worse, these existing studies are largely focused on the training outcomes and participants’ subjective evaluations of training techniques, which lack evidence of the effectiveness of training methods. Additionally, these existing studies have also been limited by comparisons of a single function (such as immersion), which lacks a comprehensive evaluation of as many functions as possible.

In order to make up for the deficiencies of the above problems, this paper researched an MR seismic retrofitting training system that supports multi-person collaboration and obtained comprehensive evaluation results through the fuzzy comprehensive evaluation method. Firstly, the indicators of the MR training system were sorted out based on the Analytic Hierarchy Process (AHP), and then the evaluation model was established. Finally, the fuzzy comprehensive evaluation method was used to evaluate the MR training system. The contributions to this research include:

• Obtaining the comprehensive evaluation results of the MR seismic retrofitting training system. Multisource indicators affecting the comprehensive evaluation results were selected, and all the indicators were divided into two levels (decision layer and factor layer). The final evaluation result was obtained after the evaluation of the target layer;
• Establishing the comprehensive evaluation model of the MR seismic retrofitting training system. The qualitative analysis was used to determine the indicator evaluation system, and the quantitative analysis was used to eliminate the ambiguity among indicators. The fuzzy evaluation model was established with the transformation of qualitative analysis-quantitative analysis;
• Providing scientific guidance for the development and improvement of this MR seismic retrofitting training system. Meanwhile, this paper also provides a reference for the subsequent improvement of the effectiveness of seismic retrofitting training.

3. Methodology

This paper conducted a comprehensive evaluation of an MR seismic retrofitting training system based on fuzzy mathematical theory. The MR seismic retrofitting training system works by Hololens2, and the task of this MR training is to repair cracks. During this repair process, grouting nozzles and grout injectors are involved, so the virtual models of these tools are built first. Then, the authors mixed the virtual models and real retrofitting environment with Unity 3D. Finally, seven students as trainees completed the task of crack repairing, and the data involving evaluation were obtained after completing the task.

The comprehensive evaluation adopts a hybrid analysis method that combines qualitative analysis and quantitative analysis. The purpose of the hybrid analysis is to establish a comprehensive evaluation model of this MR training system and to quantify the evaluation results. The evaluation method includes three parts in total: data collection, evaluation model establishment and comprehensive evaluation. The framework of the comprehensive evaluation is shown in Figure 1.

Data collection: The raw data required for evaluation can be obtained through the MR training scenario. Therefore, it is necessary to build a seismic retrofitting training scenario. MR training system is a training system developed based on VR. Virtual models in fbx. format is required to create an MR seismic retrofitting scene. The virtual–real fusion was realized between the virtual model and the real environment through the transformation of coordinates. Finally, the scene after the fusion of virtual and real was used for seismic retrofitting training.

Evaluation model establishment: The establishment of the evaluation model includes three aspects: qualitative analysis, the transformation between qualitative and quantitative, and quantitative analysis. The indicator evaluation system of the MR seismic retrofitting training system should be determined firstly by qualitative analysis (Section 4.2). Then, all the indicators were compared in pairs, and the comparison results were quantified according to the qualitative–quantitative transformation principle (Section 4.3). In order to ensure the scientific of the indicator evaluation system and the transformation process, it is also necessary to conduct a consistency check. After passing the consistency check, the membership matrix is obtained according to the quantified comparison results. At this point, the evaluation model was established (Section 4.4).

Comprehensive evaluation: The indicator evaluation system established has three levels: target level, decision level and factor level. The comprehensive evaluation of the decision layer is before the target layer. The evaluation results of the decision layer consist of the membership matrix of the target layer. The evaluation result of the target layer is the comprehensive evaluation result of the MR seismic retrofitting training system.

4. Evaluation Model of Seismic Retrofitting Mixed Reality Training System

4.1. The Evaluation Process of the MR Training System

In this paper, the authors applied the MR to seismic retrofitting training. Every trainee wore Hololens2 to accept retrofitting training. The performances of different trainees in retrofitting operations were recorded. Finally, the recorded data and the trainee self-evaluation would be as parts of the original information for comprehensive evaluation. The evaluation data obtained from the MR seismic retrofitting training system are shown in Figure 2.

The training scheme should be first determined before the comprehensive evaluation of the MR seismic retrofitting training. In order to improve the generalizability of this evaluating method, it is necessary to select a group with the same practical experience and education experience in retrofitting. Therefore, seven students were recruited for this study. Each participant’s current research was not related to seismic retrofitting, and none of them had work experience related to retrofitting. Before the training, the task description and self-evaluation questionnaire were prepared for participants. The purpose of issuing the task description was to explain the requirements of the experiment and briefly introduce the retrofitting tasks to be undertaken. The purpose of issuing the questionnaire was to support the participant in completing the trainee self-evaluation. Parts of retrofitting training process are shown in Figure 3.

Marilyn Salzman et al. [44] believe that the indicators influencing the training effect mainly include the features of the trainees, the features of the training system, the interactive experience and the learning experience. Since the participants recruited have the same education and retrofitting experience, this paper evaluated the MR seismic retrofitting training system from three aspects: the features of the trainees, the features of the training system, and the interactive experience. According to the research of Lee et al. [45,46,47], the authors sorted out the indicators that affect the comprehensive evaluation results (as shown in Figure 4).

The research of Pooya Adami [48] shows that both qualitative analysis and quantitative analysis would be involved in the comprehensive evaluation of MR training. Therefore, this paper chose the quantitative-dominated hybrid method to analyze the evaluation indicators of the MR seismic retrofitting training.

For qualitative analysis, three aspects (the features of the trainees, the features of the training system, and the interactive experience) were considered in the comprehensive evaluation. For quantitative analysis, the weight of every evaluation indicator and the evaluation result were all expressed as a set of matrixes. The AHP [49] is an analysis method that considers qualitative and quantitative data, so AHP was chosen as the analysis method in this paper.

The fuzzy comprehensive evaluation was used to evaluate the MR training system. According to the membership degree theory of fuzzy mathematics, the evaluation method can transport qualitative evaluation into quantitative evaluation. The quantitative evaluation results are expressed in the form of a matrix, which solves various non-deterministic problems [50]. The specific comprehensive evaluation process is shown in Figure 5.

4.2. Analysis of Evaluation Indicators

The purpose of this research was to conduct a comprehensive evaluation of the MR seismic retrofitting training system so as to judge the performance and effect of the training system. Therefore, it is necessary to first determine the indicator system of the comprehensive evaluation.

By referring to the characteristics of extended reality equipment in the design and construction education summarized by Sepehr Alizadehsalehi et al. [51] and the experience summarized by this research team in the training process, the three indicators (the features of the trainees, the features of the training system, the interactive experience) were selected to consist of the decision level. According to the results of the indicators that affect the comprehensive evaluation result in Section 4.1, the indicator evaluation system is established using the AHP analysis method [52]. The AHP evaluation indicator system of the MR seismic retrofitting training system is shown in Figure 6.

4.3. The Construction of Judgment Matrix

The rationality of the AHP evaluation indicator system needs to be judged by the judgment matrix. In order to construct the judgment matrix of the evaluation indicator system, it is necessary to compare the importance of any two indicators at the factor level of the indicator evaluation system. The important comparison principles of the judgment matrix are shown in Figure 7.

The personnel involved in the construction of the judgment matrix includes graduate students, their tutors and professionals. The students and tutors all have a profound scientific theoretical foundation and practical experience in seismic retrofitting and MR training. As shown in Figure 6, the established AHP indicator evaluation system consists of three levels. The decision layer contains three indicators, namely the features of the trainees, the features of the training system and the interactive experience. The students and their tutors discuss and communicate with professionals about the importance of each two indicators, which is helpful in improving the rationality of the judgment matrix. See

Table 1. Judgment matrix of decision layer indicators of MR seismic retrofitting training system.

	i	The Features of the Training System	The Features of the Trainees	The Interactive Experience
j
the features of the training system		1	3	1/5
the features of the trainees		1/3	1	1/7
the interactive experience		5	7	1

for the judgment matrix from the discussed results of the decision layer. Then the secondary indicators subsets were analyzed, and all the judgment matrices were obtained. The judgment matrixes of the factor layer are shown in

Table 2. Judgment matrix of factor layer indicators of the features of the training system.

	i	Immersion	Perceived Ease of Use	Perceived Availability	Comfortable	Learnability
j
immersion		1	5	3	7	1/3
perceived ease of use		1/5	1	1/3	5	1/5
perceived availability		1/3	3	1	5	1/5
comfortable		1/7	1/5	1/5	1	1/9
learnability		3	5	5	9	1

,

Table 3. Judgment matrix of factor layer indicators of the features of the trainees.

	i	Emotional	Motivation	Self-Efficacy	Age	Pressure
j
emotional		1	1/2	1/3	5	4
motivation		2	1	1/2	5	5
self-efficacy		3	2	1	7	3
age		1/5	1/5	1/7	1	1/3
pressure		1/4	1/5	1/3	3	1

and

Table 4. Judgment matrix of factor layer indicators of the interactive experience.

	i	Reality	Collaborative	Accuracy	Ratio of Success
j
reality		1	1/3	1/5	1/7
collaborative		3	1	1/3	1/5
accuracy		5	3	1	1/3
ratio of success		7	5	3	1

.

4.4. Establish the Comprehensive Evaluation Model

The weight of each evaluation indicator can be determined after obtaining the judgment matrixes. In order to make the weight of each indicator more reliable and scientific, it is necessary to perform a consistency check.

The eigenvalue vector corresponding to the maximum eigenvalue ${\lambda }_{max}$ of the judgment matrix is expressed as $w$ after normalization (the sum of all the $w$ is equal to 1), which means the influence of the comparison indicators on a certain indicator of the upper level. In other words, the $w$ is the weight of the indicator at the same level. The next step is to perform a consistency check. The consistency check refers to the allowable range of inconsistency for the matrix. Wherein the unique non-zero eigenvalue of the n-order uniform matrix is $n$; the maximum eigenvalue of the n-order reciprocal matrix is $\lambda \ge n$, if and only if $\lambda =n$, $n$ is a uniform matrix. The more $\lambda$ is larger than $n$, the more serious the inconsistency of the matrix.

CI is used to measure the consistency of the matrix. The smaller the CI, the better the inconsistency. The greater the inconsistency of the matrix, the larger the judgment error caused. Therefore, the magnitude of the $\lambda \ge n$ value can be used to represent the degree of inconsistency of the matrix. The CI is defined as:

$$CI=\frac{\lambda -n}{n-1}$$

(1)

where $\lambda$ is the eigenvalue, and $n$ is the unique non-zero eigenvalue of the $n$-order uniform matrix.

When $CI=0$, the matrix is complete consistency; the closer to 0 the CI, the better the consistency; the closer to 1 the CI, the worse the consistency.

To measure the magnitude of the CI, a random consistency index RI is introduced:

$$RI=\frac{{{\displaystyle \sum }}_{i=1}^{n}C{I}_i}{n}=\frac{C{I}_1+C{I}_2+\cdots +C{I}_n}{n}$$

(2)

The random consistency index RI is related to the order of the judgment matrix. Generally, the larger the matrix order, the greater the possibility of a uniform random deviation. Considering that the deviation of consistency may be caused by random reasons, it is necessary to compare the CI with the random consistency index RI to judge whether the matrix has satisfactory consistency, and the comparison result can be defined as the coefficient CR. The formula of CR is as follows:

$$CR=\frac{CI}{RI}$$

(3)

In general, if CR < 0.1, the judgment matrix is considered to pass the consistency test. Otherwise, there is no satisfactory consistency, and the judgment matrix needs to be reconstructed. The excel template developed by Goepel [53] was used in this paper to show the weight value and CR value of each evaluation indicator. The weight values of all influencing indicators and CR values of each judgment matrix can be obtained by calculating the numbers in the excel template. The calculation results show that the CR value of each judgment matrix is less than 0.1, which indicates that the judgment matrix and analysis process are scientific and reliable. The detailed data on the weight values and CR values of the evaluation factors involved in this study are shown in

Table 5. Detailed data on the weight value and CR value of each evaluation indicator.

Target Layer	Decision Layer	Weight	CR	Factor Layer	Weight	CR	Comprehensive Weight
comprehensive evaluation of MR seismic retrofitting training system	the features of the training system	19.32%	0.06	immersion	27.46%	0.08	5.31%
				perceived ease of use	8.59%		1.66%
				perceived availability	13.38%		2.59%
				comfortable	3.07%		0.59%
				learnability	47.50%		9.18%
	the features of the trainees	8.33%		emotional	19.31%	0.06	1.61%
				motivation	27.92%		2.33%
				self-efficacy	39.39%		3.28%
				age	4.34%		0.36%
				pressure	9.04%		0.75%
	the interactive experience	72.35%		reality	5.69%	0.04	4.12%
				collaborative	12.19%		8.82%
				accuracy	26.33%		19.05%
				ratio of success	55.79%		40.36%

.

As shown in Table 5, the weights of the indicators in the decision level are as follows: the weight of the features of the trainees is 8.33%, the weight of the features of the training system is 19.32% and the weight of interactive experience is 72.35%. The weighted distribution indicates that the MR training system improvement should focus on the interactive experience, followed by the features of the training system. Through the discussion and analysis, the reasons for the weighted distribution of the decision layer indicators may be:

Interactive experience: the interactive experience occupies the most weighted distribution among the three indicators of the decision layer and is much more important than other evaluation indicators. MR includes VR and AR and realizes collaborative work based on immersive experience and virtual–real interaction. Collaborative work relies on the interaction between workers, environment and machines, which is reflected in the interactive experience of trainees.

As a unique feature of MR, collaborative work is also a very important function of seismic retrofitting. The seismic retrofitting task requires the cooperation of multiple workers, even the cooperation of workers and machines. When encountering difficult problems during retrofitting, it is also necessary to analyze and solve the problems with remote experts through the voice or video of the MR system. Collaborative work can maximize the advantages of each retrofitting worker through multi-person collaboration and can discover and even solve retrofitting mistakes in time through remote collaboration check of experts. Multiple people’s collaborative work not only ensures retrofitting safety of workers but also ensures retrofitting effect. Therefore, the weight of interactive experience is the largest.

The features of the training system: Among the three indicators of the decision layer, the weight of the features of the training system ranks second with 19.32%. The training features of the MR system are, of course, important, and a training system with strong learning ability and a better immersive naturally has great superiority in the training effect. However, from the perspective of the development of extended reality, VR has realized a great immersive experience with the virtual environment, and AR has realized the interaction between the virtual world and the real world. The training equipment for this research is MR equipment (Hololens2), which integrates the advantages of VR and AR. The related technologies about the subjective experience (such as immersive) have matured. Therefore, the features of the training system are not the most important and unique point.

In order to distinguish the differences between VR training and MR training and to better highlight the advantages of MR, the influence of the common features of VR, AR and MR on the comprehensive evaluation are deliberately weakened. Therefore, the weight of the features of the training system in the comprehensive evaluation ranks second with 19.32%.

The features of the trainees: Among the three indicators of the decision layer, the weight of the features of the trainees is the least, only 8.33%. The goal of this study was to evaluate the MR seismic retrofitting training system, which focuses on the comprehensive evaluation of the training experience and the effectiveness of the training system. Therefore, the features of the training system and interactive experience are more important than the features of the trainees. For the effectiveness of the training system, the trainees in this paper have the same ages, theoretical basis and retrofitting experience; there is little difference in trainees’ features. Thus, the weight of the features of trainees in the comprehensive evaluation is the least.

5. Fuzzy Comprehensive Evaluation of Mixed Reality Retrofitting Seismic Retrofitting Training System

5.1. The Fuzzy Comprehensive Evaluation of Decision Layer

After determining the indicator evaluation system and judgment matrix of the MR seismic retrofitting training system through AHP, the next step was to comprehensively evaluate the system. Since there was no clear correspondence between the indicators, this paper adopted the fuzzy logic theory combining qualitative analysis and quantitative analysis to comprehensively evaluate the MR training system.

The MR seismic retrofitting training system for a comprehensive evaluation in this study was developed by the authors based on Unity 3D. The purpose of this MR training was to repair seismic cracks, which is an important process of seismic retrofitting. The trainees were seven postgraduates majoring in civil engineering (without relevant retrofitting experience), and the performances of each trainee were recorded. The trainees needed to fill out a self-evaluation questionnaire after finishing crack repair. The previous evaluation of the MR training system always, respectively, comprised various indicators, lacking comprehensive evaluation. This paper makes up for this deficiency of previous studies. According to the requirement of the fuzzy comprehensive evaluation, it is necessary to determine the membership matrix before the fuzzy comprehensive evaluation of the MR seismic retrofitting training system. The data of the membership matrix come from the self-evaluation questionnaires.

Firstly, the membership matrix of each evaluation indicator can be divided into five levels, namely excellent, good, moderate, bad and very bad [51]. Next, each evaluation indicator was analyzed and scored by every trainee through self-evaluation questionnaires. Then, the data were normalized. Finally, the membership matrix of each indicator was calculated. According to the AHP theory, the fuzzy comprehensive evaluation should evaluate the indicators of decision level based on the corresponding membership matrix, and the evaluated results of the decision level are expressed as new matrixes. These new matrixes are considered the membership matrix of the indicator of the target layer. At this time, all the matrixes used for fuzzy comprehensive evaluation were obtained. The fuzzy evaluation membership matrix of the MR seismic retrofitting training system is shown in

Table 6. The fuzzy evaluation membership matrix of MR seismic retrofitting training system.

Indicators of Decision Layer	Indicators of Factor Layer	Membership
		Excellent	Good	Moderate	Bad	Very Bad
the features of the training system	immersion	0.58	0.14	0.14	0.14	0
	perceived ease of use	0.57	0.43	0	0	0
	perceived availability	1	0	0	0	0
	comfortable	0.72	0.14	0.14	0	0
	learnability	0.72	0.14	0.14	0	0
the features of the trainees	emotional	0.43	0.57	0	0	0
	motivation	0.29	0.57	0.14	0	0
	self-efficacy	0.86	0.14	0	0	0
	age	0.43	0.43	0.14	0	0
	pressure	0.71	0.29	0	0	0
the interactive experience	reality	0.42	0.29	0.29	0	0
	collaborative	0.57	0.29	0.14	0	0
	accuracy	1	0	0	0	0
	ratio of success	0.86	0.14	0	0	0

.

A comprehensive evaluation was conducted and combined the weight of each evaluation indicator with the membership matrix. Each evaluation indicator of the factor layer can determine a fuzzy matrix $R_i$. The fuzzy evaluation vector of each evaluation indicator can be calculated by using the mathematical calculation model $B_i=A_i\times R_i$.

The calculation process of the fuzzy evaluation vector $B_1$ of the features of the training system is as follows:

$$B_1 =A_1 \times R_1 ={\left[\begin{array}{c}0.275\\ 0.086\\ 0.134\\ 0.031\\ 0.475\end{array}\right]}^T·\left[\begin{array}{ccccc}0.58& 0.14& 0.14& 0.14& 0\\ 0.57& 0.43& 0& 0& 0\\ 1& 0& 0& 0& 0\\ 0.72& 0.14& 0.14& 0& 0\\ 0.72& 0.14& 0.14& 0& 0\end{array}\right]={\left[\begin{array}{c}0.701\\ 0.149\\ 0.112\\ 0.039\\ 0\end{array}\right]}^T$$

(4)

The calculation process of the fuzzy evaluation vector $B_2$ of the features of the trainees is as follows:

$$B_2=A_2 \times R_2={\left[\begin{array}{c}0.193\\ 0.279\\ 0.394\\ 0.043\\ 0.090\end{array}\right]}^T·\left[\begin{array}{ccccc}0.43& 0.57& 0& 0& 0\\ 0.29& 0.57& 0.14& 0& 0\\ 0.86& 0.14& 0& 0& 0\\ 0.43& 0.43& 0.14& 0& 0\\ 0.71& 0.29& 0& 0& 0\end{array}\right]={\left[\begin{array}{c}0.583\\ 0.370\\ 0.046\\ 0\\ 0\end{array}\right]}^T$$

(5)

The calculation process of the fuzzy evaluation vector $B_3$ of the interactive experience is as follows:

$$B_3=A_3 \times R_3={\left[\begin{array}{c}0.057\\ 0.122\\ 0.263\\ 0.558\end{array}\right]}^T·\left[\begin{array}{ccccc}0.42& 0.29& 0.29& 0& 0\\ 0.57& 0.29& 0.14& 0& 0\\ 1& 0& 0& 0& 0\\ 0.86& 0.14& 0& 0& 0\end{array}\right]={\left[\begin{array}{c}0.835\\ 0.131\\ 0.037\\ 0\\ 0\end{array}\right]}^T$$

(6)

After discussion and analysis, the reasons for this fuzzy evaluation result of decision layer are as follows:

• The features of the training system: This seismic retrofitting training system runs on MR equipment. The MR equipment is improved and optimized based on VR equipment and AR equipment. Thus, the virtual training environment is more real, and the training information is easier to be grasped;
• The features of the trainees: All the trainees in this study have similar educational backgrounds, retrofitting experience, and familiarity with MR equipment. All the trainees are also interested in the guidance and retrofitting of MR equipment. In general, although the trainees are full of confidence, the final training effects are affected due to the lack of retrofitting experience and professional knowledge;
• The interactive experience: The training system uses advanced Hololens2. The virtual model is built with Unity 3D and is equipped with the Microsoft 365 Guide engine. Therefore, the MR seismic retrofitting training system has good performances in task executability and multi-person collaboration.

5.2. The Fuzzy Comprehensive Evaluation of Target Layer

Through the analysis of the results of Section 5.1, it can be seen that the MR seismic retrofitting training system has an excellent performance in training. There is a more than 80% probability that the system performs well in system interaction; the training system also is thought to perform relatively well with an excellent ratio of 70%, but there is less than 60% probability that believes the system performs well from the term of trainee features, and nearly 40% probability think that it performs not so well. However, the different weighted distributions of the evaluation indicators would cause different comprehensive evaluation results. In order to obtain a more comprehensive and intuitive system evaluation result, a fuzzy comprehensive evaluation of the target layer of the training system is conducted.

Before the fuzzy comprehensive evaluation of the system, it is necessary to obtain the membership matrix of the target layer. The membership matrix of the target layer is composed of the fuzzy evaluation vectors of the decision layer, which can be obtained in Section 5.1. The calculation process is as in the following:

$$R=\left[\begin{array}{ccc}{B}_1& B_2& B_3\end{array}\right]=\left[\begin{array}{ccccc}0.701& 0.149& 0.112& 0.039& 0\\ 0.583& 0.370& 0.046& 0& 0\\ 0.835& 0.131& 0.037& 0& 0\end{array}\right]$$

(7)

$$B=A \times R={\left[\begin{array}{c}0.193\\ 0.083\\ 0.724\end{array}\right]}^T·\left[\begin{array}{ccccc}0.701& 0.149& 0.112& 0.039& 0\\ 0.583& 0.370& 0.046& 0& 0\\ 0.835& 0.131& 0.037& 0& 0\end{array}\right]={\left[\begin{array}{c}0.789\\ 0.154\\ 0.050\\ 0.008\\ 0\end{array}\right]}^T$$

(8)

5.3. Analysis of Evaluation Results

According to (the principle of maximum value [53]) the evaluation result, this training system belongs to the first level (excellent). Although the performance of trainee features is not as good as training system features and system interaction, the final comprehensive evaluation result of the system is still excellent due to the low weight of trainee features (evaluation result: excellent 78.9%, good 15.4%, moderate 5%, bad 0.8%, very bad 0%). This evaluation result is also consistent with previous research and the overall perception of the trainees.

Based on the fuzzy evaluation results of the system, some important references can be provided for the development, application, improvement and selection of MR retrofitting training system in the future. For example, with the progress of science and the development of technology, more and more efficient seismic retrofitting methods will be born. Although the new method can improve the retrofitting effect of seismic building structures, the knowledge, skills and safe behaviors contained in the new method are indeed urgent for retrofitting workers to master quickly. Therefore, the fuzzy evaluation results can be one of the important selection bases of different MR training systems.

6. Conclusions and Discussion

MR is very suitable for seismic retrofitting training because of the advantages of MR and the complication of retrofitting operations. At present, there are limited studies on seismic retrofitting training based on MR and fewer studies on MR system evaluation. In fact, the evaluation of the MR seismic retrofitting training system is an important basis for improving the efficiency of the training. With the development of MR and the proposal of efficient retrofitting methods, more and more MR training systems will be produced. Therefore, in the case of a large number of MR training systems in the future, it is easier to select a more suitable system based on the comprehensive evaluation results.

This paper aimed to comprehensively evaluate the MR seismic retrofitting training system. Firstly, the indicator evaluation system was determined by combining the AHP and fuzzy logic theory. Secondly, the weight of each indicator was calculated by constructing a judgment matrix. Finally, the evaluation results were obtained by the combination of the membership matrix and comprehensive evaluation model. From this research, the following conclusions can be drawn:

• The weight of each indicator in the decision layer was obtained by the hierarchical analysis. The weights are as follows: the features of the training system is 19.32%, the features of the trainees is 8.33% and the interactive experience is 72.35%.
• After a fuzzy comprehensive evaluation, the evaluation result of the MR seismic retrofitting training system is excellent. However, in the fuzzy evaluation of the decision layer, it can be found that the performance of the indicator of trainee features is not as good as other indicators. Therefore, the efficiency of training should not only pay attention to the development of technology but also need to improve the self-efficacy and emotional state of retrofitting workers.
• The research methods and ideas of this paper are universal, but due to the differences among individual experimenters, the specific indicators of the evaluation system may need to be modified. The indicator weighted in this paper are calculated based on the opinions of the trainees, so the results are subjective to a certain extent. This subjectivity is caused by the randomness of individuals. Similarly, the judgment matrix is indeed different according to the actual task. Generally speaking, the overall comprehensive evaluation ideas and methods remain unchanged and referential.

With the current aging of the labor force, the increase in the retirement rate, and the harsh construction environment, construction workers are fewer and fewer. Ineffective workforce training also leads to many problems, such as the construction workers in a dangerous construction environment, the construction efficiency being low and the construction quality being poor. Therefore, starting from the labor demand of civil engineering and according to the individual differences of construction workers, improving the effectiveness of construction training in the construction industry is the next main work of our team.