Development of Engine Maintenance Technology with Mixed Reality-Taking Pratt & Whitney JT9D as an Example

Abstract

The engine is the heart of the aircraft and deeply affects performance, flight safety, and airline management, which makes it a critical component requiring maintenance. With the advancement of technology, augmented reality (AR), virtual reality (VR), and mixed reality (MR) technologies have been introduced one after another. To increase maintenance skill ability, it is important to train for aviation maintenance. Therefore, the engine maintenance mixed reality (EMMR) system was developed through mixed reality technology in this study. There are three different scenarios have been developed in the EMMR system for making engine maintenance easy. In the first scenario, an exploded diagram of the disassembled engine with the main components has been made and an animation of the fuel flow has been demonstrated to introduce the basic structure of the engine. In the second scenario, interactive buttons and animations have been used to simulate the removal of the engine from the wing of an airplane so that the users can understand the process of disassembling the engine. In the third scenario, hybrid reality has been implemented to interact with the virtual objects, so that users could simulate the disassembly and replacement of the engine fan blades and get familiar with the whole process. Students have been introduced to and have been familiarized with the Microsoft HoloLens 2 and a survey has been conducted before and after using the developed technology to understand if the currently developed EMMR could help students to understand the process of each engine maintenance in the general classroom and make the learning more efficient at the same time.

1. Introduction

With the maturation of robotics and artificial intelligence, augmented reality (AR), virtual reality (VR) [1], and mixed reality (MR) technologies have been introduced which have helped in bringing in a huge leap in technological advancements through easing the complexities and increasing human–machine interaction. Many researchers are also experimenting with these technologies in gaming, medicine, arts, culture, and educational training [2,3].

Mixed reality is a new technology that has emerged in recent years, and the concept is similar to the combined augmented reality and virtual reality. This technology identifies the user’s surroundings through the visible camera and depth sensor on the device, then places the system’s virtual object in the corresponding spatial location, and displays the virtual object through a three-dimensional transparent lens, allowing the user to see the virtual object as if it is in the real environment and allowing them to interact with it without using other remote controls, thus achieving the characteristics of overlapping reality and increasing the information that the user can obtain in the real environment.

When an aircraft maintenance technician is faced with tens of millions of parts in an aircraft, it is impossible to completely remember the specifications, model number, and standard operation procedure (SOP) [4] for each component, and it is always necessary to look up the technical manual. By using the virtual and real features of mixed reality, it is possible to operate the actual components while looking at the digital virtual manual through glasses, saving the time and action of reading the paper technical manual.

2. Literature Review

Mixed reality (MR) is the combination of real and virtual worlds to create new environments and visualizations, where physical entities and digital objects coexist and can interact in real time, to be used as an analogy to real objects. In 1994, Paul and Fumio [5] proposed a definition of the reality–virtuality continuum, and the mixed reality is located at any point between the extreme values of the virtual continuum (VC). The two sides of the virtual continuum are the real environment and the virtual environment. The technology close to the real environment is augmented reality (AR), which uses the perceptual information created by the computer to increase the experience of real objects and additional information. On the other side of the spectrum is augmented virtuality (AV), which is a technology that adds physical elements to the experience of a virtual world. The unique feature of mixed reality is the fusion of the two at the same time. By superimposing reality and virtualization, users can interact with virtual objects in a real environment, shattering the basic concept of reality and imagination. As early as the 1930s, science fiction writers and inventors had already dreamed of creating an environment where people could escape from reality through art and machines. Many scientists have been evaluating the feasibility of virtual, augmented, and mixed realities for a long time, but the technology was not available at the time; it was only in the last decade that the technology became available.

In 2009, Microsoft announced a sensor called Project Natal [6], and in 2010 announced the official name of Project Natal as Kinect, which was applied to the peripheral devices of Xbox One and Xbox 360 consoles, and was considered a pioneering device to popularize the user’s physical control and changed the way of playing games at that time. In 2013, Google glass was launched [7], and since then, a large number of wearable products have emerged, including smart glasses, which have been left out of the market, and head-mounted display (HMD) products have become the stars of tomorrow. With the launch of smart glasses, information technology has taken a giant leap forward, and the biggest change is that the scene seen by the eyes can be calculated and recognized at any time, and the result will be presented to the user. The market for smart glasses is currently booming, and with many related applications coming out, smart glasses will probably become one of the most widely used products in the field of wearable devices in the next few years.

In 2016, Microsoft HoloLens was launched [8], which continued the Kinect’s body tracking technology and presented it as a head-mounted display, followed by Microsoft’s second-generation HoloLens 2 in 2019, highlighting three main improvements: immersion, ergonomics, and business friendliness. HoloLens 2 will be used in this study. Today’s mixed reality applications are mainly used with head-mounted displays (HMD), in which users can see virtual objects and appear to be in a real environment, similar to AR, but with the “immersion” and “virtual interaction” elements of VR. The combination of reality, augmented reality, and virtual reality technologies presents a mix of real and virtual worlds that is indistinguishable. In other words, MR not only retains the immersive and interactive nature of VR but also emphasizes the interactive elements of virtual objects in the real world more than AR. Wu and Vu [3,9] developed an aircraft maintenance virtual reality (AMVR) system for training students in the aviation education and evaluated the effectiveness of the system.

Guhl [10] presents a concept and architecture for interacting and programming robotic systems, especially industrial robots, using mobile devices such as HoloLens. The robots and mobile devices are connected through so-called agents and communicate using protocol buffers on top of TCP/IP connections. Various mobile devices running on different operating systems and hardware performance levels can be supported using sophisticated software components such as game engines. Sirilak et al. [11] found the use of HoloLens in critical care to be very beneficial in providing care and preventing or reducing morbidity and mortality, but HoloLens technology requires adequate prior training for users to implement, and users are not at all accustomed to the technology and are likely to continue to rely on the telephone for consultation, or to use applications such as Skype and LINE. They are likely to continue to rely on phone calls for consultation or use various applications such as Skype and LINE for video calls. On the other hand, using technology as an educational and training tool will go a long way toward overcoming the staffing shortage that is currently occurring. It can also be used to improve the efficiency of patient clinical teaching for physician teams and ICU nurses, increasing their experience and opportunities to learn alongside critical care medical professionals. Asgary et al. [12] have created an MR application called HoloVulcano that visualizes the various phenomena that can occur in a volcano for emergency management training, public education, and awareness. The application can provide a new way to visualize volcanic eruptions and volcanic emergencies, enhance learning and training, and improve preparedness and understanding of the risks. Gao et al. [13] have built a virtual museum with Vuforia, which not only scans and images the camera using mixed reality devices but also scans images to present models using augmented reality. By breaking the boundary between augmented reality and mixed reality, the two can play to their respective strengths and enhance user immersion and substitution. In cooperation with Taiwan Li-Wei Computer Machinery Company, Lai [14] designed the system for a five-axis machining machine called “V-40iT”, using the section “Troubleshooting blade jam” in its troubleshooting manual. After the completion of the system, we visited the company to conduct interviews and surveys. According to the results, the troubleshooting assistance system should be able to solve the shortcomings of traditional troubleshooting and increase the willingness of personnel to use the system again. Ghazali et al. [15] supplemented the learning process of desalination plant technology concepts with multimedia technologies such as HoloLens. The team built a working prototype of an MR game, as shown in the methodology section. The prototype includes a map scene that allows users to navigate the game, three tutorials that familiarize users with the basic gesture functions of HoloLens, and two mini-games that allow students to understand the desalination plant process. This project brought new learning experiences to the research team and analyzed developments in the field of education combined with mixed reality technologies. The team also expects that their research will create more directions for self-learning and educational purposes. Harper et al. [16] used the Microsoft HoloLens camera to scan QR codes and manipulate virtual scribe markers with gestures to perform a stone chipping simulation for training college-level stonemasons in stone grinding. By using virtual stones, a physical grinder can be used as a tangible interface without the associated safety risks. The main challenge of this investigation was the use of tangible objects as an interface to the virtual learning environment, which was achieved through low-latency, real-time tracking, allowing for trusted interaction with virtual content. The application was designed and developed in conjunction with several further education colleges in Scotland and has been evaluated by teachers and learners. Danie et al. [17] created an epidural analgesia simulator to increase the training space through the use of dummies, HoloLens, digital guidance, 3D audio feedback, and voice commands. The purpose of this study is to explore the potential of optical fluoroscopic augmented reality technology (OST-AR) for epidural anesthesia training. To do so, the system places relevant 3D anatomical teaching messages and visual guides on and around the dummy, supplemented by audio feedback to guide the teaching process.

3. Research Methodology

As mentioned in the previous research literature review, MR can operate virtual objects in real environments, provide information and environments that are not normally available in real environments, such as the prevention of explosions and aircraft fires, or the operation of expensive equipment replacement, and use MR for practice and training purposes. In addition, many researchers use the Microsoft HoloLens mixed reality device to conduct research. In this research, HoloLens 2 has been used to develop a mixed reality system called the engine maintenance mixed reality (EMMR) system [18]. There are three scenes to choose from engine introduction, engine removal from the pylon, and engine fan blade removal. These three scenes can be operated consecutively or directly into the desired scene by pressing the button directly from the menu to open the selected scene. The research follows the steps shown in Figure 1.

To generate scenarios and assign interaction functions, multiple software, programming languages, game engines, and 3D modeling tools are required (

Table 1. Commonly used software for MR systems.

Game Engine	Unity [21], Unreal Engine, JavaScript, …
Programming Language	C++, C#, Java/Java Script, …
3D Modeling Software	Blender, CATIA, SolidWorks, Autodesk Maya, Autodesk 3Ds Max, SketchUp, …

). Unity 3D version 2020, C#, and SolidWorks have been used in developing the EMMR. MR devices are indispensable for MR applications. To integrate into the virtual space in a real environment, users need MR glasses that look like a head-mounted display (HMD). The paper used Microsoft’s HoloLens 2 [19], and according to Microsoft’s own words [20], mixed reality is the next wave of technology after computers and smartphones, a hybrid of the physical and digital worlds in which humans, computers, and the real world can interact.

4. Results and Discussion

4.1. Engine Maintenance Mixed Reality System Design

Mixed Reality Toolkit [22] (MRTK) is a mixed reality development toolkit from Microsoft that allows developers to use the Unity game engine to develop applications for Microsoft’s own HoloLens and HoloLens 2 mixed reality devices, as well as WMR head-mounted displays and open VR platforms.

In the Mixed Reality Toolkit, basic scenarios for blended reality are provided, as well as ready-to-use UI elements buttons [23], sliders, labels, bulletin boards, etc., and a script for basic blended reality interaction with virtual objects. The script for basic blended reality interaction with virtual objects is also provided. The script is designed to make it easy for students to operate and understand the composition of the engine in aviation education. The initial screen of the engine maintenance mixed reality (EMMR) system is shown in Figure 2. The EMMR has three scenes to choose from, namely, introducing the engine, removing the engine from the pylon, and removing the engine fan blades [18]. You can see from the picture that the background is a normal classroom, and you can see the system screen and the outside physical environment at the same time when using the engine mixed reality maintenance system.

The first scene introduces the JT9D engine. This scene consists of a proximity menu, bulletin board tool, prompt label, animation system, and particle system, providing the introduction of JT9D engine parts, as shown in Figure 3a. The near menu has buttons from left to right, with the first button opening an exploded diagram animation of the engine, allowing students to get a closer look at the main engine components as well as models of some parts that are not normally visible inside the engine, as shown in Figure 3b. The second button opens the Tab Tool, which indicates the location of the main parts on the 3D model, and the tabs will face the user at any time with the direction of walking, so even if you walk to the back of the engine, you can still clearly know the location of the parts, as shown in Figure 3c. The third button opens a combined animation system and particle system to simulate the airflow when the engine is running, as shown in Figure 3d. Blue particles represent the cold air just entering the engine, while red particles simulate the hot air flow through the compression section and combustion chamber.

The second scene is the removal of the engine from the wing pylon, this scene consists of the near menu, buttons, and animation system, the operation can be done through the proximal menu to switch the process and camera position adjustment, and use the buttons to open each process of animation, as shown in Figure 4. In Figure 4a, the position of the view angle can be controlled by the near menu, which can be moved up and down, left and right, so that the user can easily see the view angle above the engine. The engine cowling button is designed to facilitate the inspection of the side of the engine, as shown in Figure 4b. To enrich the content of the scene, JT9D information is also provided in the scene. Students can find the screw position and document from the side panel for the removal of the pylon screws, as shown in Figure 4c. After the main bolt of the engine is removed, the engine can be removed and placed on the shipping stand, as shown in Figure 4d. Students can familiarize themselves with the disassembly steps of the engine bolt in this scene to reduce the danger in the process.

The third scene is the removal of the engine fan blades, this scene uses the bulletin board tool to list the process of fan blades removal, as shown in Figure 5a. Clicking on the step button opens the Bounds Control function that shows the virtual objects (white line) as in Figure 5b. Users can interact with the virtual objects with their hands and remove the virtual engine fan blades, and in the process, they can decide which objects can move through the script to create the disassembly process by taking advantage of the current position of the objects. For removing the engine fan blades, six bolts and washers have to be removed for removing the outer engine cone. The virtual engine cone is then removed and placed on the background table, as in Figure 5c. After removing the engine cone, twelve bolts and washers have to be removed for securing the rear segment support to the axial compressor rotor hub, then the rear compressor inlet cone is removed. The ring would be then used which is placed on the front to prevent the other blades from falling down and thus ensure safety during fan blades removal. Removing the fan blades are a bit more difficult than removing the pylon and the cone due to its small size but it has been designed in such a way that the student would not face many difficulties. The methods have been kept very user-friendly to make the participants much more comfortable. The scene is also shown the procedure for the removal of the fan blades from the engine manual when the student is unfamiliar with the part, as shown in Figure 5d.

4.2. Questionnaire Survey

This study has been done to understand how effective the developed mixed reality technology is effective for aircraft engine maintenance. The assumptions were made that (A) there were no other external factors in this study and students were not affected by emotional and physical stress; (B) the sub-items tested are not influenced by gender; (C) university students who had a preliminary concept of the engine were selected as the study population. Twenty-five students were invited to take the questionnaire and were asked to fill out the survey before and after training and used the EMMR system (taking the removal of the engine fan blades as an example). The questionnaire survey consisted of 17 questions [3,9] divided into five categories: problem organization, problem analysis ability, problem comment ability, problem-solving ability, and teamwork. The collected data were analyzed using SPSS. The mean and standard deviation were calculated to compute the efficiency of the training. Cronbach’s α [24] was calculated as shown in Equation (1). The results were used to evaluate the reliability of the questionnaire and consistency. The results are presented in

Table 2. Cronbach’s α of the question in the questionnaire survey.

	Number of Questions	Cronbach’s α
Problem Organization	3	0.819
Problem Analysis Ability	3	0.841
Problem Comment Ability	4	0.784
Problem-Solving Ability	4	0.763
Teamwork	3	0.836

.

$$\alpha =\frac{N·\overline{c}}{\overline{v}+\left(N-1\right)·\overline{c}}$$

(1)

where N is the number of items, $\overline{v}$ is the covariance between the item pairs, and $\overline{c}$ is the average of the variance.

Through Cronbach’s α results in the above table, we can see that Cronbach’s α values of the five question groups, among which the α values of problem commenting ability and the problem-solving ability range from 0.7 to 0.8, while the rest are higher than 0.8. Therefore, the results in the above table are acceptable and the questionnaire is consistent.

There were three questions in the problem organization section to assess students’ understanding of the problems in

Table 3. Results for problem organization.

Number	Question	Pre-Test	Post-Test
1	For the removal of the JT9D engine fan blades, students can find the defects by themselves.	3.20	3.88
2	When trained the remove the JT9D engine fan blades, students can find the required information in the manual.	3.48	3.76
3	When studying removing the JT9D engine fan blades, students can predict possible problems.	3.44	4.12
	Average	3.37	3.92

and Figure 6. In the first question, the results of using the mixed reality system showed that it helped students easily find the problem on their own, with the mean value changing from 3.2 to 3.88. In question 2, the data were similar, which shows that putting the manual scan file in mixed reality was comparable to using the traditional manual. In addition, the results for question 3 changed from 3.44 to 4.12, indicating that there was a significant change in the predicted problems that could be encountered when using blended reality in maintenance training.

As shown in

Table 4. The results for the problem analysis ability.

Number	Question	Pre-Test	Post-Test
4	Faced with a complicated aircraft maintenance manual, students can quickly complete the sorting and classification.	2.60	3.72
5	Students can infer information and learn from remove the JT9D engine fan blades data.	3.24	3.76
6	Student confidence in the logical thinking ability about the removal of the JT9D engine fan blades.	3.00	3.96
	Average	2.95	3.81

and Figure 7, the results of students’ classification and categorization ability increased from 2.60 to 3.72, while there was only a slight increase in question 5, presumably because the students were familiar with the process because they had attended a related practice course. This indicates that the mixed reality system helps students to think logically when learning.

The results for the question group on question commenting ability are shown in

Table 5. Results for the problem commenting ability.

Number	Question	Pre-Test	Post-Test
7	Students can judge the problems that others have not found.	2.76	3.68
8	Students can think backward to discuss problems.	3.40	3.84
9	Students can share their opinions on the current situation.	3.60	4.12
10	In the discussion of the remove the JT9D engine fan blades, they talk about the problems with each other.	3.68	4.08
	Average	3.37	3.92

and Figure 8. Between the questions, there was an increase in the results before and after using the mixed reality system. In particular, the mean value of question 7 increased by about 0.92 after using the mixed reality system, which is consistent with the fact that everyone identified and asked questions after using the system. The mean of questions 8 to 10 increased slightly by 0.4, indicating that each student’s commenting ability improved equally.

There were four questions in total in the problem-solving ability group to assess students’ problem-solving ability after removing the JT9D engine fan blades using the hybrid reality system, as shown in

Table 6. The results for problem-solving ability.

Number	Question	Pre-Test	Post-Test
11	When encountering a problem of removing the JT9D engine fan blades, students prefer to find information to find solutions.	3.68	4.08
12	In the process of solving the problem of removing the JT9D engine fan blades, students can formulate a solution strategy and review method.	3.36	3.80
13	In the process of solving the problem of removing the JT9D engine fan blades, students monitor their execution progress.	3.20	3.96
14	Students can draw a flowchart of the removal of the JT9D engine fan blades.	2.60	3.84
	Average	3.21	3.92

, and the results are shown in the graph in Figure 9.

Based on the results, it can be seen that the ability to draw a flowchart for removing the JT9D engine fan blades significantly improved in question 14, with the mean value increasing from 2.60 to 3.84. In question 13, the student’s ability to monitor their progress in solving the problem increased slightly from 3.20 to 3.96. The mean value of developing a solution strategy and reviewing the method in question 12 increased slightly from 3.66 to 3.80. Question 11 showed that after using the mixed reality system, more students actively searched for information to find out about the problem when they encountered it.

Finally, a total of three questions for the teamwork question set are presented in

Table 7. Results for teamwork.

Number	Question	Pre-Test	Post-Test
15	For the study of the remove the JT9D engine fan blades, students can understand what others mean.	3.36	4.04
16	For the study of the remove the JT9D engine fan blades, students can discuss with others to solve problems together.	3.40	4.24
17	During the study process, students coordinate work or cooperate with others in assignments.	3.52	4.00
	Average	3.42	4.09

and the results are shown in Figure 10. In the process of using the mixed reality system, the student’s ability to communicate with other students improved from 3.36 to 4.04, while the results obtained in the ability to communicate with each other to solve problems in question 16 ranged from 3.40 to 4.24. Additionally, the tested students found it easier to complete their assignments after discussing them with other students. Finally, the mean value of 3.42 was improved to 4.09, which shows that the engine maintenance mixed reality system is helpful for teamwork.

In

Table 8. Results of the questionnaire groups.

Questionnaire Groups	Pre-Test	Post-Test
Problem Organization	3.37	3.92
Problem Analysis Ability	2.95	3.81
Problem Comment Ability	3.37	3.92
Problem-Solving Ability	3.21	3.92
Teamwork	3.42	4.09
Average	3.69	4.14

, the mean scores for the problem organizing ability section increased from 3.37 to 3.92; the mean scores for the problem analysis ability section increased from 2.95 to 3.81; the mean scores for the problem commenting ability section increased from 3.37 to 3.92; the mean scores for the problem-solving ability section increased from 3.21 to 3.92; finally, the teamwork section improved from 3.42 to 4.09, as shown in Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10.

To find out the significance of the difference in knowledge of removing the JT9D engine fan blades before and after using the EMMR system, a paired-samples t-test [9,25,26] was conducted, as shown in

Table 9. Mean, standard deviation of the scores, and t-test result of the questionnaire survey using the EMMR system regarding the removal of the JT9D engine fan blades (n = 25).

Number	Pre-Test		Post-Test		t-Value	p-Value
	Mean	Stand-Devi	Mean	Stand-Devi
1	3.20	0.91	3.88	0.78	−3.778	0.001
2	3.48	0.65	3.76	0.97	−1.495	0.148
3	3.44	0.96	4.12	0.83	−3.778	0.001
4	2.60	0.76	3.72	0.94	−5.315	0.000
5	3.24	0.88	3.76	1.01	−2.397	0.025
6	3.00	1.00	3.96	0.98	−4.370	0.000
7	2.76	1.01	3.68	0.90	−5.662	0.000
8	3.40	0.91	3.84	0.75	−2.864	0.009
9	3.60	0.71	4.12	0.83	−3.161	0.004
10	3.68	0.69	4.08	0.95	−2.309	0.030
11	3.68	0.69	4.08	0.81	−2.828	0.009
12	3.36	0.76	3.80	0.82	−2.290	0.031
13	3.20	0.76	3.96	0.94	−3.919	0.001
14	2.60	1.00	3.84	1.18	−5.023	0.000
15	3.36	0.95	4.04	0.94	−3.302	0.003
16	3.40	0.91	4.24	0.83	−4.938	0.000
17	3.52	0.71	4.00	1.00	−2.493	0.020

. The scores of the questions showed significant differences before and after using the EMMR system with p-values less than 0.05, except for question 2, which did not show significant differences with p-values greater than 0.05. However, it seems that the students improved their knowledge to some extent with the help of the PBL method.

5. Conclusions

An engine maintenance mixed reality (EMMR) system has been developed for aviation educational and industrial training for students and professionals. Microsoft HoloLens 2, Unity 3D, C#, and SolidWorks have been used for developing mixed reality technology. The EMMR system application has achieved the predefined requirements. The system helped students to understand and master the process of unloading and removing the engine fan blades from the JT9D engine from the pylon. During the process, students also observed some possible problems in the operation of the system, thus improving their ability to recognize and solve problems. On the other hand, unloading an engine from an aircraft is difficult to physically access in an educational setting such as a normal school, because removing the engine from the wing requires a large enough area in addition to the entire aircraft. With the hybrid reality system, students can more clearly and easily understand the main process of engine disassembly from a pylon. In the engine introduction part, since the traditional teaching method is mostly paper-based, students rarely have the opportunity to see the internal structure of the actual engine even in the practical class, so the 3D virtual exploded view simulates the location of the main components with labels to let students understand at a glance. Finally, students can practice disassembling the JT9D engine fan blades in the virtual environment to avoid possible industrial safety risks and hazards.

The results of the questionnaire survey showed that the application of blended reality in engine maintenance courses can help students improve their learning abilities. After the survey, each student’s organizational, analytical, critical, problem-solving, and group cooperation skills improved after using the blended reality system, through which it can be seen that blended reality has great potential in engine maintenance training.

Since mixed reality is a new technology in recent years, most students are aware of or have been exposed to virtual reality (VR) and augmented reality (AR) technologies, but have not been exposed to mixed reality (MR). This makes most students find this technology novel and difficult to operate when they first encounter it, but after actually interacting with virtual 3D objects in a realistic environment, they find it very interesting and it improves their learning intentions.