Towards the Emergence of the Medical Metaverse: A Pilot Study on Shared Virtual Reality for Orthognathic–Surgical Planning

Abstract

Three-dimensional (3D) medical images are used for diagnosis and in surgical operation planning. Computer-assisted surgical simulations (CASS) are essential for complex surgical procedures that are often performed in an interdisciplinary manner. Traditionally, the participants study the designs on the same display. In 3D virtual reality (VR) environments, the planner is wearing a head-mounted display (HMD). The designs can be then examined in VR by other persons wearing HMDs, which is a practical use case for the medical metaverse. A multi-user VR environment was built for the planning of an orthognathic–surgical (correction of facial skeleton) operation. Four domain experts (oral and maxillofacial radiologists) experimented with the pilot system and found it useful. It enabled easier observation of the model and a better understanding of the structures. There was a voice connection and co-operation during the procedure was natural. The planning task is complex, leading to a certain level of complexity in the user interface.

1. Introduction

The metaverse [1,2,3] can be thought as an extension of earlier multi-user shared Virtual Reality (VR) systems. Support for collaboration is an essential feature in these systems in VR environments and has become an important topic in the field of human–computer interaction (HCI) [4,5]. Collaboration in VR may increase the quality of shared presence, communication, and interactions among multidisciplinary teams distributed in several locations [6,7,8,9,10]. Although issues of information security have been raised for the use of the metaverse [11], these are out of the scope of the present paper.

To enable truly effective collaboration in VR, one needs to find out the most important system features for the collaborative participants. Mütterlein et al. [12] studied the interplay between telepresence, interactivity, and immersion in collaborative VR and their results demonstrated the importance of interactivity and immersion; they claim that trust between users of the same team is essential for collaborative VR experiences. It was also shown that immersion is one of the main drivers. Liu et al. [13] demonstrated that the roles and actions of the participants in collaborative VR will affect participant experience. Drey et al. [14] studied collaborative learning systems where either both participants were using VR devices or one had a VR device and the other was using a tablet. They demonstrated that participants experienced greater presence and immersion, and lower cognitive load, when both had VR devices. Burova et al. [15] demonstrated that asymmetric remote collaboration, where some participants used VR and others attended by other means, was still useful.

Spatial perception is important when analyzing medical imagery for operation planning. Wu et al. [16] found that the use of collaboration in the VR environment improved cognition capability. Zaker and Coloma [17] showed that the use of VR made it possible to detect possible conflicts in spatial arrangements. Tea et al. [18] used VR to enable a real-time remote collaboration for the design review process and demonstrated that using an immersive VR application led to better results than using the traditional approach. Increased spatial understanding was also one of the benefits of using a collaborative VR system in a study by Heinonen et al. [19], who studied a system for technical document review and risk assessment.

In the medical field, collaboration in VR environments has been investigated in various contexts, for example in medical training where several participants working in the same environment and on the same data sets is natural. For example, Chheang et al. [20] studied how the training for a collaborative medical operation can be done in a VR environment and Schott et al. [21] used multi-user VR and mixed environments to learn liver anatomy. Kyaw et al. [22] collected a review about using VR for teaching in the health profession. In the learning situations, the teacher usually explains the phenomena or operation, and other participants will be observing a demonstration, but a true collaboration with active participants can also be arranged. Luxenburger et al. [23] demonstrated a system where the participants studied shared electronic patient records in VR and Liu et al. [24] had a collaborative VR system where several domain experts can study the same medical data. Butnaru and Girbacia [25] explored collaborative bone surgery planning in a tele-immersive environment with networked Cave Automatic Virtual Environment (CAVE) systems and other stereoscopic desktop displays.

To the best of our knowledge, this is the first study of collaborative VR for computer-assisted surgical simulation (CASS) in facial or craniomaxillofacial surgery, while VR has earlier been used for solo training [26] and planning [27]. Studying the use of VR in craniomaxillofacial (CMF) CASS is interesting because current surgical planning is still carried out on two-dimensional (2D) screens, but VR has been shown to have advantages over 2D in terms of perception and visualization [28,29,30].

Orthognathic surgery is a type of surgical procedure that is used to correct abnormalities or defects in the facial skeleton [31]. These abnormalities may include misaligned jaws, incorrect bite, and other problems that affect the appearance and function of the face. Having a precise pre-surgical plan is essential when performing surgical treatment of dentofacial deformities, as it helps to ensure that the procedure is successful and that the desired results are achieved [32,33]. In this experiment, to simulate a simplified mandibular bilateral sagittal split osteotomy (BSSO) operation, a scenario was created where a radiologist would make a plan together with a surgeon who is in charge of the final surgical operation. In this operation, the lower jaw is split sagittally in its posterior area [31].

An orthognathic surgery was chosen as a practical example of a collaborative VR procedure because the use of CASS is widely accepted in this field and has shown excellent results in previous studies, showing increased efficiency and more accurate osteotomies than with conventional planning methods [34]. CASS for Orthognathic Surgery is the three-dimensional (3D) integration of bone and soft tissue analysis into the execution of surgical movements to achieve a dental and skeletal harmony of tissues. The result is then transferred from the virtual environment to the surgical setting [35]. Ajoub and Pulijala [36] collated a review article about the use of VR in oral and maxillofacial surgery. Xia et al. [37] presented a use case back in 2001.

There are also several other types of craniomaxillofacial surgery where CASS is increasingly being used, such as cancer surgery and including reconstructions, major trauma reconstructions, and craniosynostosis surgery. The main goal in all of these is to improve surgical accuracy, functional and morphological outcomes, and to reduce operating time by using patient-specific cutting guides and implants. Finally, CASS can also reduce costs, and the use of VR could make the visualization of anatomical structures and the final surgical outcome more accurate than a 2D approach.

In earlier studies [16,17,18,19], collaboration in the VR system was shown to improve performance. The study concentrated on synchronous collaboration between two participants, similar to Pidel and Ackermann [38]. The two parties would make their contributions on the task simultaneously. As demonstrated by Mütterlein et al. [12], the interaction between the participants is important. In the experiment, participants were able to comment on the plan and suggest changes in a truly interactive way, and they were able to reach an agreement on the plan.

The objective of the experiment was to pilot test the functionality of the collaborative VR in the operation design and, in particular, to collect the subjective experience of collaboration between experienced participants. Our contribution through the study was to show that medical professionals appreciated the capabilities of the collaborative VR environment and would be interested in using a similar system in real operations.

2. Materials and Methods

A prototype version of a multi-user virtual reality system was built for the orthognathic–surgical planning operation to collect user comments of the feasibility of VR implementation. Four domain expert users (oral and maxillofacial radiologists) were asked to try out the prototype and experiment with the standard operations to get a feeling of the system, and then give comments and development ideas on how to improve the system. The comments were collected and analyzed after the experiments to continue the development.

The experimental task resembled the standard process of planning an orthognathic–surgical operation. The surgical plan is completed by a radiologist who manipulates the imaging data and 3D models, takes the necessary measurements, defines the provisional positions for the necessary osteotomy lines, defines the movements of the parts, etc. A surgeon who will perform the real-life operation is responsible for the final surgical plan and accepts the preliminary plans or modifies them according to the surgical requirements in this interdisciplinary process. For this reason, radiologists and surgeons work together in the final and decisive planning process, and a multi-user implementation of the system was necessary to enable this work.

2.1. Design of the Experimental System

Medical expert users were consulted during the development process and their suggestions were addressed in iterative development. The experimental system was designed to enable all the usual operations that are needed for orthognathic–surgical operation planning. This decision was made based on the preferences of the expert pilot users and the close resemblance of the tool to the ones they are using in their daily work. The system did not copy any existing design tools, as the objective was to concentrate on testing the VR in the design process, and the experience of collaboration between participants. Introducing more features would complicate the analysis by increasing the complexity [39].

The functions required for operation planning, which were implemented in the system, were as follows:

• The users may study the model by moving and observing it from different directions to obtain a better understanding of the model structures.
• The user can mark the necessary cephalometric points [40,41] on the model by simply placing small dots (Figure 1, left).
• Reference planes can be created on the model defined by three points set on the model. Relevant for the current operation planning were the Frankfort horizontal plane and the maxillary occlusal plane [42,43].
• Cutting planes can be created and their locations interactively edited on the model by moving the points (Figure 1, middle).
• The model can be cut into separate parts using the previously defined cutting plane (Figure 1, right).
• Separate parts of the model can be moved relative to each other and locked into the desired positions.
• The position and orientation of a model part can be accurately fine-tuned using virtual handles that restrict the moves (Figure 2, left and middle).
• The user can measure how much the model parts have been moved and the angle between the defined planes (Figure 2, right).
• For effective collaboration, users can discuss with each other and point to specific points in the design (Figure 3, left).

In the current experiment, the focus was on the functionality of collaboration between participants. It was important to observe the position of the other participant and to understand how the other participant sees the medical image. In the prototype system, only the positions of the HMD and controllers used by the other participant were shown, but it was enough to give us information on the participant’s position (Figure 3, middle). There was an audio connection between the participants built into the system. However, in these experiments, the participants were in the same large room and no audio connection was needed. The participants were able to move freely around the virtual object to observe the planning details, even overlapping their (virtual) viewpoints if necessary.

2.2. Experiment Design

The experiment was designed to collect participant experiences and participant comments. All the actions of the participants (controller and HMD moves, button taps, etc.) were recorded but they were not analyzed for execution times or similar objective measures as the system was a preliminary prototype, and the main focus was on the implementation and experiences of the collaboration functionality.

Subjective Measures

The participants were asked a set of subjective questions (

Table 1. The questions about the participant’s subjective impressions.

S1	How successful were you in accomplishing what you were trying to do?
S2	How confident were you in your ability to use the tools?
S3	How efficient was the system to use?
S4	How easy was the system to use?
S5	Could you imagine using the system for your daily work? (How likely would you be to use the system?)

), using a Likert scale with 7 levels from 1 (Not-at-all) to 4 (Somewhat) to 7 (Very).

3. Experiment

3.1. Apparatus and Participants

The VR system was built using the Meta Quest 2 HMD (“Oculus Quest 2”, Oculus, https://www.oculus.com/quest-2/, accessed on 7 June 2021) and using the Touch controllers (“Oculus Quest 2 Accessories”, Oculus, https://www.oculus.com/quest/accessories/, accessed on 7 June 2021) (Figure 3, right). The Oculus Quest 2 device uses an inside-out tracking technology, which means that no base stations are required for tracking. The VR system was built on the Unity 3D software development system, using version 2021.1. (“Unity Real-Time Development Platform”, Unity, https://unity.com/, accessed on 28 April 2021).

The skull model was segmented from the facial CBCT of an orthognathic surgery patient using Materialise ProPlan CMF 3.0.1. The CBCT was scanned with Promax Mid by Planmeca at the Department of Cranio- and Dentomaxillofacial Radiology, Tampere University Hospital, Finland. Voxel size was 0.3 mm. This study was based on a retrospective registry data set and did not involve human experimentation or the use of human tissue samples. No patients were imaged for this study. The study followed Finnish legislation, the Medical Research Act (488/1999), the Act on Secondary Use of Health and Social Data (552/2019), and the European General Data Protection Regulation 216/679. The use of data was approved by the research director of Tampere University Hospital, Finland, on 1 October 2019 (vote number R20558).

Four expert participants were recruited (two female and two male participants, oral and maxillofacial radiologists, who were readily available) who all had experience of orthognathic–surgical planning. Two participants performed them weekly and one performed them monthly. One was familiar with the planning procedure. Their experience in dentistry and radiology varied from 7 to 35 years and in CASS from 5 to 20 years. These radiologists were responsible for in-house CASS in their own hospital. (In other hospitals, the division of responsibilities may be different.) Participants ranged in age from 33 to 62, with an average age of 46. All the participants had some experience in using VR equipment, but none identified as an expert user.

3.2. Procedure

Each participant was introduced to the study procedure and to the equipment used. After the introduction, they were to read and to sign a consent form. After that, the participants filled in a background information form. The participants were asked to wear a Meta Quest 2 headset and hold the controllers and to practice using the system interface for as long as needed to feel confident that they could use the system. The practice sessions took between 10 and 20 min. The facilitator was available to explain and to help in trying out different functions. As the given task involved two participants, one was chosen to be the radiologist who made the orthognathic–surgical plan. The other was chosen to be the surgeon who observed the planning process and modified it according to the surgical needs.

The task started with both participants wearing the headset and the controllers. The facilitator started the system and the participants found themselves in a VR environment where there was a skull model. The participants saw each other’s avatars in the VR environment (Figure 3, left).

The radiologist was expected to first study the skull model. Then, they marked the cephalometric points and defined the Frankfort horizontal plane and the maxillary occlusal plane for reference. To cut the lower jaw in two places, the radiologist defined two cutting planes. The surgeon guided the process according to the surgical needs, and finally accepted the plan. The locations could be fine-tuned, after which the cutting process would be initiated. The radiologist would measure the length of the movement of the lower jaw relative to the original location. Then, the experiment would be over and the participants would be asked to remove the headsets. As a final task, the participants would fill in a subjective evaluation questionnaire (Table 1) and give free-form comments. The whole procedure took around an hour, including the practice session.

4. Results

The interaction system was subjectively evaluated (Table 1). The subjective evaluation results are visualized in Figure 4. The results show that, on average, the evaluations are at the upper end of the scale. Those participants who were selected to act as “radiologist” (who also happened to have the most planning experience) and one of the “surgeons” gave consistently high evaluations, while the other “surgeon” participant gave lower evaluations.

The participants gave a few general comments. One comment was about the clarity of the user interface and about the practice session, suggesting improvements. The user interface itself was fully functional, but it was found to be complicated to adopt during the relatively short experiment. The same issue was mentioned also in informal discussions with other participants after the experiments. In another comment it was proposed that the user should be able to scale the skull model. In the current implementation, the skull model was of fixed size and enabling scaling would make it possible to study details more easily.

Observations

During the experiment, the facilitator observed the participants. The participants knew each other before the experiment, so the start of the collaboration was easy. There was no need to become familiar with the working styles of each other. This would also apply to professional use if colleagues often work together.

The participants approved the possibility of share the VR environment and were chatting (as observed by the facilitator) about that at the beginning. Even as the participants were only able to observe the position and orientation of the other participant’s HMD and controllers in the environment, this alone created a sense of shared involvement in the task.

The radiologists were able to complete the intended tasks, but more slowly than they expected. Participants repeatedly reported problems with the user interface, mainly due to its complexity. Although the interface was shown to be functional, there were too many possible operations to be mastered in the limited amount of practice time available. They felt confident enough immediately after the practice but then found out during the experiment that they were not ready. For example, when asked to do certain operations, they could not find the necessary actions and had to ask the facilitator for help.

5. Discussion

Previous studies in VR collaboration [12] have shown that immersion and interactivity between collaborators are important factors in effective VR collaboration. From the participant behavior, one could observe that the system was sufficiently immersive and enabled effective collaboration between participants by providing comments, discussions, pointing out places, etc. The participants saw the benefits of everybody seeing the same 3D view of the medical image and being able to interactively agree on the details of the plan. Also, in the given task, the cognition of spatial arrangements was seen as being important for the planning task and it has been shown that VR collaboration is an effective tool for this [16,17,18].

Participants were generally satisfied with what the system offered, but there were small differences between the participants in different roles in the experiment. The “radiologists” gave slightly more positive evaluations than the “surgeons”, which might be related to the less active role of the “surgeons” as their experience was probably less immersive. Liu et al. [13] previously reported similar results.

Those two participants with the most extensive planning experience gave the highest subjective evaluation values. This may have been because they were more likely to see the benefits of using a 3D environment, such as the easier perception of 3D structures. With less experience of the 2D planning process, the benefits of the prototype system were probably less obvious and the problems related to the user interface were more prevalent for these users.

It was found that the interface was complicated, which affected the use of the interface and caused participants to ask for advice during the experiment. The main reason for this was the combination of the number of different functions needed in the complex procedure and the limited time to practice the interface. The complexity of the user interface was somewhat inevitable because the surgical planning task requires the user to perform many different functions. In professional use, the users are usually willing to spend a considerable amount of time training on user interfaces to master them. Still, it was recognized that the interaction design can be further improved in follow-up studies.

Most of the experiment participants approved the prototype virtual reality system and expressed interest in using a similar implementation in real orthognathic–surgical operation planning cases. They saw the benefits of everybody seeing the same 3D view of the medical image and being able to interactively agree on the details of the plan.

The main limitation of the study was that only four participants were used in the experiment, which inevitably introduces statistical uncertainty into the results. The reason for this choice was to collect relevant comments on the VR system, so the participants had to be highly qualified experts in the field, which inevitably limited their number. Nevertheless, it is believed that this small cohort is still representative of potential end users.

Also, as this was the first time the VR system was used in simulated surgical planning tasks, not all operations were as smooth as they would need to be in an operational system. This is inevitable in user-centered research and development, but will improve through later iterations of the systems. The “surgeon” who voted 1 for “easiness” did not elaborate on the reasons for their evaluation but considered the system too complicated. They were also less experienced with VR devices, which may have affected the given evaluation.

When planning future research activities, the functionality of the virtual reality system should be limited to only those functions that are strictly necessary for the intended tasks, in order to keep the complexity of the system under control. This makes the experimental setup less real, but helps focus on the functions studied in specific experiments. To ensure that the system is truly fit for purpose, development work needs be carried out in close collaboration with users who are the real experts in the field. Rapid utility for novice users should be balanced with the day-to-day needs of experienced users. Further work may also focus on conducting longitudinal studies of using the complete system to investigate learning effects.

There is an interesting possibility of not directly transferring the functionality of the current systems to the VR environment, as the expert users suggested, but of reinventing the ways the doctors use the system. This will presumably require changes in the working processes and a relearning of the ways the tasks are carried out, but this may finally lead to increased efficiency and improved user experience. However, such a further step is beyond the present work, and will require a long-term iterative process to produce successful results.

The system needs to be tested with more 3D radiological orthognathic surgery patient data, and the planning results need to be compared with traditional CASS for a meaningful assessment of VR usability. If VR CASS proves to be effective and reliable, it has the potential to be quickly adopted globally in clinical work. It would also be an effective tool for training medical specialists.

The more complex the surgical planning, the more specialists with different backgrounds will be involved, such as radiologists, oral surgeons, otolaryngologists (or Ear, Nose, and Throat (ENT) surgeons), or plastic surgeons. The shortage of doctors in both primary and specialist care is growing worldwide [44,45] and, therefore, there is a need for an efficient and trustworthy solution for CASS, even if the participants would be in separate hospitals, countries or even continents. In CASS, the loss of data in terms of image quality and the time lost when working separately are specific topics for future research. Cypersecurity is a specific issue for the implementation of the metaverse that will need to be addressed in future solutions when sensitive data are transferred between locations [11].

6. Conclusions

Four medical experts were recruited (oral and maxillofacial radiologists), who were familiar with orthognathic–surgical planning, to evaluate a prototype implementation of a multi-user VR planning system that enables collaboration between participants. The experts found the collaborative VR system beneficial for the planning task and expressed interest in using a similar system in real operation planning cases. The results support the expectation that a medical metaverse could provide a viable collaborative medium for medical professionals. The user interface was considered to require further development, but also more comprehensive training before deployment.