Next Article in Journal
Chatter Detection in Variable Cutting Depth Side Milling Using VMD and Vibration Characteristics Analysis
Next Article in Special Issue
GN-CNN: A Point Cloud Analysis Method for Metaverse Applications
Previous Article in Journal
Investigation of Charge Transport Properties in VTP: PC71BM Organic Schottky Diode
Previous Article in Special Issue
Metaverse for Cultural Heritages
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Electronics
Volume 11
Issue 22
10.3390/electronics11223778
electronics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Information and Communication Technologies Combined with Mixed Reality as Supporting Tools in Medical Education

by
Agnieszka Pregowska
1,*,
Magdalena Osial
1,
Damian Dolega-Dolegowski
2,
Radek Kolecki
3,4 and
Klaudia Proniewska
5,6
1
Institute of Fundamental Technological Research, Polish Academy of Sciences, Pawinskiego 5B, 02-106 Warsaw, Poland
2
Projekt LIDER XI NCBiR-HoloMed, Jagiellonian University Medical College, Kopernika 7E Str., 31-034 Krakow, Poland
3
University Hospital in Krakow, Jakubowskiego 2 Str., 30-688 Krakow, Poland
4
Jagiellonian University Medical College, Anny 12 Str., 31-008 Krakow, Poland
5
Center for Digital Medicine and Robotics, Jagiellonian University Medical College, Kopernika 7E Str., 31-034 Krakow, Poland
6
Department of Bioinformatics and Telemedicine, Jagiellonian University Medical College, Medyczna 7 Str., 30-688 Krakow, Poland
*
Author to whom correspondence should be addressed.
Electronics 2022, 11(22), 3778; https://doi.org/10.3390/electronics11223778
Submission received: 26 October 2022 / Revised: 8 November 2022 / Accepted: 15 November 2022 / Published: 17 November 2022
(This article belongs to the Special Issue Metaverse and Digital Twins)
Browse Figures
Versions Notes

Abstract

:
The dynamic COVID-19 pandemic has destabilized education and forced academic centers to explore non-traditional teaching modalities. A key challenge this creates is in reconciling the fact that hands-on time in lab settings has been shown to increase student understanding and peak their interests. Traditional visualization methods are already limited and topics such as 3D molecular structures remain difficult to understand. This is where advances in Information and Communication Technologies (ICT), including remote meetings, Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), and Extended Reality (XR, so-called Metaverse) offer vast potential to revolutionize the education landscape. Specifically, how MR merges real and virtual life in a uniquely promising way and offers opportunities for entirely new educational applications. In this paper, we briefly overview and report our initial experience using MR to teach medical and pharmacy students. We also explore the future usefulness of MR in pharmacy education. MR mimics real-world experiences both in distance education and traditional laboratory classes. We also propose ICT-based systems designed to run on the Microsoft HoloLens2 MR goggles and can be successfully applied in medical and pharmacy coursework. The models were developed and implemented in Autodesk Maya and exported to Unity. Our findings demonstrate that MR-based solutions can be an excellent alternative to traditional classes, notably in medicine, anatomy, organic chemistry, and biochemistry (especially 3D molecular structures), in both remote and traditional in-person teaching modalities. MR therefore has the potential to become an integral part of medical education in both remote learning and in-person study.

1. Introduction

Teaching resources have evolved from chalkboards to digital presentations and videos; however, certain aspects are changing more slowly, particularly in medical and pharmacy schools. Modern medical teaching aims to effectively train students in a controlled environment to ensure skill transferability to the clinical setting. The application of immersive technologies such as Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), Metaverse (i.e., Extended Reality XR), or a combination of imaging with biological and real-time diagnostic data is a promising solution [1,2,3,4,5]. Recently, we observed a strong uptick in the use of immersive technologies in medical education, particularly during the COVID-19 pandemic and subsequent lockdowns [6]. On the other hand, active learning is more effective than passive learning [7]. Among pharmacy students, active learning methods have been reported to stimulate the development of empathy [8]. Immersive Technologies also offer an interesting alternative for depicting structures on the molecular level in a more intuitive manner [9]. One of the greatest advantages of these technologies is the ability to simulate real-world situations with patients and reproduce them with the active involvement of the user. This lowers the cost associated with initially performing these tasks in clinical practice where there is a risk that mistakes may result in negative health outcomes or even death. Moreover, it has been shown that academic faculty noticed an improvement in the quality of education following the incorporation of VR and AR [10]. Immersive technology, i.e., the Metaverse, being a decentralized and stable online 3D virtual environment, creates an opportunity for users to interact as avatars [11]. An interesting solution was shown in [12]: the Metaverse platform VoRtex, which enables virtual collaborative learning and is based on Mannien’s matrix. The advantage of this compared to platforms such as Vircadia, Sansar is security.
Medicine, including emergency medicine and medical education, offers a key opportunity to apply immersive technologies [13,14]. Surgeons were the first to use VR, AR, and MR to carry out procedures, improve their skills, and practice future medical interventions [15]. For example, MR helps surgeons analyze and understand the complex morphologies of congenital heart disease [16]. Thus, the application of immersive technologies has also been developed in pharmacy, both at the educational and professional levels [17], e.g., Pharmacy Times uses AR applications to provide users with additional information about drugs [18]. Ref. [18] described the virtual simulation of a pharmacy that allows users to perform pre-laboratory activities, such as preparing and mixing ad hoc preparations, and producing labels, resulting in up to a 10-fold increase in workflow efficiency. Their proposal also highlighted increased interactions between students and their lecturers. Ref. [19] reported that a VR-based platform was successful in prescription processing, pharmaceutical calculations, and selecting, preparing, packaging, and dispensing medications. Participants believed the system provided a realistic, engaging, and interactive experience that allowed them to become more proficient in drug safety practices. In [20], MR was applied to the content of the brain anatomy. It turned out that HoloLens enabled a deeper understanding of content for students and at the same time no negative side effects such as nausea, disorientation or fatigue were noticed. Thus, the physiological extension possibilities of the application of VR headsets are described in [21].
Patients have been increasingly relying on virtual consultations with their doctors, which contributes to the willingness to apply this technology in pharmacy consultations [22]. AR-based applications can be used as patient monitoring tools to verify treatment compliance [23,24]. Immersive Technologies facilitate Phase IV monitoring post-market release to better identify rare adverse drug reactions [25,26]. Ref. [23] proposed a Medication Coach Intelligent Agent (MCIA) that interfaces with AR glasses and accesses the treatment plan, restrictions on drugs, interactions based on patient preferences, and sends notifications so that the patient does not miss a dose.
It is also important to remember that economics play an important role in education. However, even the significant equipment expense to incorporate immersive solutions is less than the cost associated with performing a cadaveric autopsy [27]. Moreover, immersive technologies contributed up to 50 percent in savings to healthcare worker protection measures during the COVID-19 pandemic [28].
In this paper, we discuss the utility of MR in medicine, including a discussion of how recent advances in immersive technologies may benefit medical education, particularly anatomical and 3D molecular structure visualizations. We propose an MR-based system dedicated to medical and pharmacy education that was designed to run on the Microsoft HoloLens2, a MR headset. The structural models are developed and implemented in Autodesk Maya and then exported to Unity software. It turned out that the proposed approach expands the effectiveness of education by increasing retention and comprehension of complex principles that enables a superior application of that knowledge in the future. While immersive technologies will never completely replace the critical interpersonal relationship between students and lecturers, they provide a highly valuable resource that will train even higher-quality professionals. The paper is organized as follows. In Section 2, we briefly overview the concept of immersive technologies in the field of education, in particular medical education, including chemistry and medical chemistry of training. In Section 3, details concerning the MR-based system are described. The results, namely 3D models, which can be displayed in HoloLens MR-glasses, are presented in Section 4. Section 5 contains a brief discussion and some concluding remarks, while Section 6 includes some direction for future development.

2. Immersive Technologies and Learning

Learning is a complex process that includes not only the acquisition of knowledge but also its comprehension and application. Traditional teaching methods include the use of lectures, pictures, and textbooks. Traditional lectures that last more than 20 min, however, have been shown to limit retention in short- and long-term memory [29]. Furthermore, students forget 90 percent of lectures after two weeks, while retaining only 30 percent of content immediately following the lecture itself [30]. Most students better absorb graphical information, rather than verbally or through text. The application of virtual simulators, in particular among future medical staff, provides a medium through which a greater degree of information may be retained. This is especially true in medicine where traditional methods of teaching anatomy have relied upon autopsies for visualization. However, these are still often limited due to ethical and financial constraints as well as how long a given cadaver may remain physically suitable for teaching [27,31]. Holograms may standardize the quality of education while developing spatial awareness through stereoscopic vision, which is critical to mastering anatomy [32].

2.1. Immersive Technologies in Chemical Classes

Two-dimensional models of complex chemical structures have existed for decades and translating them into an interactive 3D modality is a relatively new area of research where immersive technologies demonstrate particular value. In fact, the only molecular visualization program that has been translated into a virtual format is Chimera, while others have been adapted into AR and VR-based solutions [33,34], and the applications such as VRmol [35], our moleculARweb [36], and ProteinVR [37] are Web Browser-based. This opens the field for the development of various applications supporting chemistry education. Ref. [38] proposed AR tools to study the material structures and chemical equilibrium at the high school level, where students can use tools to conduct virtual chemical experiments. Here, a user proposes a chemical equation that includes reagents and coefficients on AR cards. The results are displayed as a submicroscopic image of the chemical reaction. When compared against groups taught traditionally, the proposed AR method resulted in better learning outcomes, particularly among students with lower baseline academic performance. Ref. [39] presented a mobile application, MolAR, that allows users to visualize molecules in AR directly from hand-drawn chemical structures. Another application is AR Chemistry Learning, which allows users to interact with virtual images that represent different molecules, where particular elements are marked in different colors [40]. This tool combines AR with Machine Learning to transform hand-painted structures into 3D images, with applications beyond education in visualizing protein structures and analyzing molecular properties by specialists. Thus, AR-based applications are a helpful teaching resource for understanding complex chemical principles [41].
Ref. [42] applied MR to microfluidics coursework. Their ALETHA platform was tailored to the Microsoft HoloLens and links experimental methods with descriptions of physical and chemical processes. This approach increased motivation among students. Ref. [43] provided a safe MR-based environment to perform a dangerous chemical experiment, i.e., diluting concentrated sulfuric acid. To make the experiments more realistic, it rendered a realistic image of the hand, beaker and virtual liquid.

2.2. Immersive Technologies in Medical Student Training

In our previous paper [44], we concentrated on possible applications of MR-based tools in medical education by conducting an anonymous electronic survey regarding the application of immersive technologies in the medical curriculum. A total of 70 percent of students believe that this teaching method was beneficial. While small in scope and limited to students of one university, this study nonetheless showed the beneficial and disruptive potential of MR technology. We have also presented the pilot study of patient experience-focused MR-teaching tools among the surgical oncology population, including cases of pancreatic, colorectal, hepatic, and gastric cancer along with adult polycystic kidney disease. This study directly led to the development of an MR-based laboratory for medical students, which enables students to become acquainted with modern medical imaging, and 3D image processing using high-fidelity 3D models. MR can also be successfully applied to entire anatomy courses [45]. The first commercial MR anatomy teaching tool, HoloAnatomy, was developed by CaseWestern Reserve University School of Medicine (Cleveland, OH, USA) in cooperation with the Cleveland Clinic (Cleveland, OH, USA) [46]. Immersive technologies can also be successfully applied in dentistry, particularly in pre-clinical education [47,48]. Ref. [49] used VR to teach root canal anatomy, while Ref. [50] used MR.
Immersive technologies are also valuable in clinical practice. Ref. [51] described how remote guidance from an experienced surgeon using Google Glass AR technology helped not only interns but also seasoned surgeons learn new surgical procedures such as arthroscopy or shoulder arthroplasty. This solution links imaging data with what the surgeon sees live, allowing them to proceed with the operation without taking their eyes off the patient. Ref. [52] described a VR surgical mentor, where expert knowledge was combined with virtual surgical tools and illustrative animations to lead to more confident decision making. We can also use MR technologies to optimize preprocedural planning and intraprocedural monitoring in advanced imaging in interventional cardiology [53]. In [54], the combination of ultrasound with Augmented Reality has been shown in the field of rabbit liver therapy.

2.3. Immersive Technologies in Medical Chemistry Training

Medical chemistry is traditionally taught using 2D representations of 3D molecules to understand chirality, electron distribution, steric bulk, and intra- and inter-molecular bonding. Many students struggle with this. Ref. [55] proposed an AR smartphone application and defined a protocol for creating and disseminating AR models. Ref. [56] proposed AmplifiedRx, an AR application used in pharmaceutical compounding laboratories that resulted in increased student comprehension and preparation for practical classes and exams. This weekly, hourlong Pharmaceutical Sciences Laboratory course (PHCY 752) lasts one semester and teaches dosing to gain proficiency in composing pharmacological preparations. Ref. [57] piloted an MR-based application supported by Artificial Intelligence, which included textual information and 3D structures related to pharmacogenomics.
Recently, Australia legalized vaccination for both children and adults that unexpectedly highlighted shortcomings in training to actually administer vaccinations [58,59]. Here, MR may also be helpful. Ref. [60] proposed an interesting solution using the Microsoft HoloLens, where students use the applications Holohuman and Holopatient to practice scenarios of post-vaccination anaphylaxis. They were able to thoroughly master human anatomy, particularly related to vaccine administration, and see what happened physiologically following the incorrect administration of vaccines.
An interesting study by [61] described the positive motivational influence AR technologies have on pharmacy students, highlighting the Pharma Compounds AR (PCAR) educational tool. This application allows users to view 3D models of particles using a smartphone. Similar utility was shown by [62] regarding inorganic stereochemistry. The motivation predictor was defined by a self-reported competency measure, where the application itself was determined to be both useful and engaging. Ref. [63] drew attention to the important issue that qualified pharmacy teaching staff are needed who can teach using these immersive technologies.

3. Materials and Methods

A strong emphasis is placed on chemistry during pharmacology studies. In medical studies, it is very important to learn the physiology and development of an organ to better understand its function. The same holds in pharmacology where an understanding of chemical structure and interaction with other compounds is necessary to be able to tailor treatments to patient needs. Historically, designing chemical compounds was carried out on a piece of paper. As computing technology has advanced, chemistry has received new tools to plan and visualize very complex structures in 3D. This is a massive leap that allows users to see how chemistry really looks. Other content is connected with human anatomy, in particular organs and their pathological, pathophysiological, or biochemical changes. Computer tomography (CT) or magnetic resonance imaging (MRI) serves as the basis for creating holograms of the organs or abnormalities, e.g., tumors. These are processed using semi-automatic algorithms in a process called segmentation, in which elementary volume elements (voxels or Hounsfield units) are initially labeled and subsequently post-processed. The most commonly applied algorithms for segmentation include thresholding, seeding, and filling slices. Thresholding involves specifying the minimum and maximum intensities based on the histogram of a region of interest in the slice view, i.e., where a high-intensity, white region such as bone is selected while omitting soft tissues. Seeding involves tagging two specific regions of interest in sagittal, coronal, and axial views of a CT scan: the object and the background, where the object corresponds to the desired organ and the background corresponds to all undesired material. After tagging these respective seeds, a customized GrowCut algorithm was applied to grow the region of seeded interest while excluding the region of undesired structures. This expands the binary “object” and “background” tags voxel by voxel along with a statistical weight as to the probability a given voxel still corresponds to the object. Filling slices is similar except it seeds only an “object” in one view, i.e., axial predominantly for vertical structures such as the inferior vena cava. Upon manually tagging the object in multiple slices of the CT scan, the Fill between Slices algorithm, based on ND morphological contour interpolation, is applied to form a 3D mask corresponding to the object. These algorithms are automated; however, their product requires varying degrees of manual correction, which is carried out before accepting the finalized 3D mask. This is accomplished either in the three 2D views, or the 3D preview generated, intending to define the boundary of the desired object as accurately as possible [64]. The finalized products are validated by experienced radiology and anatomy specialists and then exported into a stereo-lithography (STL) file format for holographic visualization [65,66,67]. Recently, there has been a push for AR and MR to open new possibilities for interacting with and understanding 3D space to display data in a new, attractive way.
We propose the use of advanced computer visualization methods in teaching and assessing skills. MR is an innovative method that can contribute to building an e-learning program which supports both knowledge acquisition and also skill mastery. Developing modern higher education systems need the use of immersive technologies. In this paper, we proposed a MR-based system dedicated to medical education, including medical chemistry, see Figure 1. It is tailored to run on Microsoft HoloLens2 headsets (Microsoft Corporation, Redmond, WA, USA), which supports eye-tracking, gesture control using individual fingers or both hands, and voice control. The last two are especially important in teaching. It is fully integrated with Microsoft Enterprise systems, which resembles other Windows operating systems. The key advantage of HoloLens2 advantage is the fact that it does not require a connection to a computer or other equipment, rather it is completely autonomous. The system utilizes a front-end MR application, HoloView, which has a simple and intuitive user interface. First, users decide where they want to place the generated model of an anatomical creation or chemical structure. This can be carried out using eye movement and gestures. The system can be effectively applied in classrooms, lab rooms, and even at home. An example medical chemistry class is shown in Figure 2.
Structural models are developed and implemented in Autodesk Maya (San Rafael, CA, USA), then exported to Unity software (Unity Technologies, San Francisco, CA, USA). The application uses Unreal Engine (Epic Games, Potomac, MD, USA) and Mixed Reality framework. It uses PDB files to generate 3D models that contain all information needed to place molecules and their connections to other structures in XYZ coordinates. Thanks to the Unreal Engine and simple sphere models (with different colors) we can visualize models in a simple, relatable manner. The Mixed Reality framework makes models interactive to better understand a given compound’s functionality.
All information gathered during the initial experiences with MR was used to create an extensive step-by-step guide to enable smaller teams to incorporate this technology. The goal of this handover was to have the team fully support the classroom, from HoloLens storage to creating and evaluating the 3D content at the level of detail necessary for students. Functions include highlighting dedicated objects with distinct colors, changing transparency or removing objects completely to better view structures of interest, creating 3D labels identifying different chemical structures and tagging anatomical structures representing human organs (digital twin). New pedagogical methods continue to evolve as students and faculty explore the MR teaching environment.

4. Results

Mixed Reality allows students to see and interact with everyone around them while viewing holographic information and models. We are currently exploring applications of the HoloLens to promote collaborative learning. The HoloLens can capture real-time MR video and audio to revolutionize the analysis of multimodal data and analyze chemical compounds in 3D to understand their structural properties. We used Autodesk Maya to create 3D models in a two-step process. Figure 3 shows a sample 3D model of chemical structures. Colors were used to distinguish different molecules, e.g., carbon was marked black, and oxygen was marked white, etc. The models were exported to Unity, an application dedicated to creating video games and AR software for different platforms. The input file is in PDB format, which contains specific information about the name of a molecule, its position in 3D space, and the name of the molecule to which it connects. After receiving the ball, the system reads how they are to be connected and draws the constraints. The system enables 3D visualization of more complex structures, such as the point cloud shown in Figure 4a. More complex structures such as spirals can be used to draw 3D models, Figure 4b.
The second step involved scripts, which enable anchoring the 3D object in a given area, animating model actions, and preparing management captions. The application was tested in the Unity emulator twitch export-ready project with Microsoft Visual Studio. This proposed approach allows images to be displayed as 3D objects, i.e., holograms. A user can touch and interact with them by rotating, changing view, moving, rescaling, or marking their components.
Figure 5 shows how a chemical model is displayed for users. The arrangement of particular elements in the molecule in space can be adjusted with sliders. Every chemical molecule can be grasped, moved, and connected with a different molecule by hand. This proposed application allows users to interact with chemical components in a very intuitive manner. Users see the graphical hands as their own. This tool supports cloud integration, meaning that 3D models can be uploaded and instantly disseminated among other HoloLens2 users using the platform. This tool has the potential to make learning pharmacology and chemistry easier and more intuitive.
Our system can be also applied to medical education, in particular anatomy. For example, a 3D model of a liver tumor was prepared from CT images. The liver tumor was removed using a surgical robot. Precise diagnostics are crucial to clearly define the surgical extent and feasibility of resection. To enable this, segmentation of the CT data was performed to highlight relevant structures. Then users display the resulting model as interactive holograms using the HoloLens 2 that can be rotated, move, and scaled with gesture or voice control. Preoperative imaging to produce holograms very precisely shows all the anatomical structures of the organs, see Figure 6. Thus, the 3D enables better visualization of the tumor’s location and improves the understanding of the spatial relation between anatomical structures. Thus, this solution is effective in enabling both students and junior surgeons to understand local anatomy in any given case. Moreover, this can be also used to simulate planned operations from different approaches and using various techniques. Consequently, the learning curve for surgeons may be expedited and the operating room workflow may be streamlined.
In Figure 7, we proposed an application to learn how to plan the removal of impacted teeth. The 3D model of the jaw based off an MRI image is displayed for the user. The points mark the place where the tooth is grasped (red) and the direction of pulling force (green). The application allows users to set coordinates for the grasp and pull points using sliders.
This approach can also be applied in maxillary surgery education, specifically treating mandible tumors (Figure 8). The subject was hospitalized due to a right-sided tumor at the angle of the jaw. Sections taken under general anesthesia were ultimately determined histopathologically to be: ameloblastoma, follicular type. The patient was in a good general condition without any obvious disease burden. Due to their young age, histopathology, and the lack of disease burden, the decision was made to perform a microsurgical resection of the mandible from the mandibular branch to tooth 42 and reconstruct the resulting defect using a right iliac crest allograft. 3D models of the jaw in Figure 8a show bone loss caused by the tumor, while Figure 8b shows a 3D model of the jaw with implants.
Resorption is a process involving the loss of hard tissues (dentin, root cementum, enamel) as a result of various factors. The model is made on the basis of the micro-CT of tooth 23, see Figure 9. Here, the bulbous widening of the tooth chamber is visible in the cross-sections. This model can be displayed in HoloLens 2. The point perforation of the mesial wall of tooth 23 can be observed after rotating the model.

5. Discussion and Conclusions

Initial attempts have been made to apply MR for distance learning for Science, Technology, Engineering, and Math (STEM) lectures and students quickly adapted to using these solutions [67,68]. Ref. [69] proposed a framework for creating instructional animations. Users can operate the system using gestures with the Leap Motion controller, while adaptive grasping of objects of various shapes and sizes was developed with the help of deep-learning neural networks. Ref. [70] present the application of AR to evaluate a patient’s posture and joints to match X-ray image information. Users, i.e., teachers, can easily visualize themselves performing chemical experiments and students can learn by following their movements. Ref. [71] described the virtual chemical laboratory using the Oculus Rift, in which students can repeat and change the course of chemical reactions.
Understanding molecular conformations, complex compound structures, and chemical transformations can be challenging for students, especially when it comes to the visualization of the molecules that are usually described as flat models in books. The same holds for teaching organ pathogenesis and morphology. The Metaverse is slowly gaining traction among the academic community, and particularly the medical community, because it provides an engrossing, collaborative telepresence [72,73]. In fact, forecasts predict that by the end of 2026, 25 percent of the population will spend at least an hour a day there [74]. Ref. [75] outline how the Metaverse will be applied in medicine. One part is to anchor the building as a static geometric model of the virtual world, including people, scenes, and events, while the other involves the dynamic fusion of the virtual and real world in a high precision virtual space. The latter is strongly dependent on the development of AI, 5G data, and brain–computer interfaces (BCI). An important consideration in the development of the Metaverse are ethical issues related to user privacy and personal information as well as challenges in collecting and linking data within the Metaverse. It has been shown that immersive technologies can simulate genuine emotion, including cognitive, social and behavioral reactions, while the presented content can be modified to provide effective communication and learning [76,77,78]. Ref. [79] showed that Virtual Reality does not produce additional workload but rather reduces it, as measured by the NASA Task Load Index: mental, physical, and temporal demand, performance, effort, and frustration. Ref. [80] compare traditional teaching and supplementary online lectures using digital handout notes based on AR in medical education. Superior outcomes were achieved by the group using immersive technologies.
Mixed Reality techniques bring enormous value to teaching new generations of doctors, physicists, and engineers. These types of systems allow for the analysis of real medical cases recorded with the help of three-dimensional medical imaging. The main difference between standard methods and those using holograms is that the student can see the object not as a flat visualization with added shadows simulating depth, but rather in a form similar to the actual patient lying on the operating table. An additional positive educational feature here is immersion, where medical records, imaging, and reference materials relevant to a patient’s case can be interacted dynamically and intuitively. Another important feature is the ability to analyze a specific data set by a larger group of users. This is advantageous from the educational point of view because it shares a physician’s commentary on a given case, the view from the perspective of the physician conducting the procedure and a visible hologram on the basis of which medical procedures are performed. All this leads to the modernization and improvement of the teaching process for medical students by leveraging the possibilities of modern technology in medicine. There are also tools for simulating a truly tactile experience using haptics, giving a new depth to the immersive experience. Such solutions are used in all kinds of simulators, because they are able to imitate the use of actual surgical instruments or laparoscopic cameras. This approach can also be used in conjunction with MR, which is applied in simulators where surgeons practice procedures and the system measures and compares parameters such as time, length of the movement path, smoothness of movement, tissue damage, and deviations from the planned course of the procedure against those of more experienced experts. MR can also be used in training complex specialties such as neurosurgery or interventional cardiology. Here, solutions are developed that allow the use of MR in conjunction with a surgical phantom where similar parameters are measured as described above. These systems are constantly being developed and provide more opportunities for training future medical staff. A multitude of solutions and their accessibility enable increasingly accurate and repeatable training modalities thanks to MR. More and more teams are working on implementing this type of training in various parts of medical and surgical education.
In this paper, we proposed the application of MR to simplify medical and chemistry education. The Microsoft HoloLens2 enables the visualization and gesture control of 3D models of chemical compounds, organs, and parts of the human body. The aim of this study was not to replace all traditional medicine and medical chemistry teaching paradigms, but rather to support them using modern solutions. This will provide students with a deeper understanding of chemistry and anatomy and save costs related to physical reagent use or autopsy by novice users, respectively. A significant limitation of our system is the number of users who can use it simultaneously. Another limitation may be throttled network traffic, which may lead to a less immersive experience. Moreover, using the virtual world for too long can also cause eye strain, headache, and nausea. These ailments can be reduced or eliminated as technology develops. Table 1 outlines a comparison of immersive technologies-based systems in education, particularly chemistry, medical chemistry and medicine, including proposed solutions.
This educational tool is based on advanced computer visualization methods and enables the development of the idea of collaborative learning in medicine, including pharmacy. 3D models of chemical compounds, organs, and bones are created from MRI and CT data using segmentation algorithms and visualized as holograms by the HoloLens 2.
The mobility of immersive technologies allows users to choose the time and place to learn. However, a stable internet connection remains essential [83]. The rapid development of immersive technologies makes it necessary to investigate how the information is actually absorbed. Ref. [84] showed that MR allows students to be more involved and feel more confident in their acquired knowledge of the spatial relationships between brain regions. Spatial ability in particular was found to be more important to the users learning with MR than to those learning traditionally. Moreover, an interesting consideration was shown by [76], who proposed the Contextaware Augmented Reality Model (ICAARM) to enable the analysis of human–computer interaction based on personalized gestures.
Ref. [85] described that some Microsoft HoloLens users experienced accessibility problems and a lack of flexibility. It remains difficult to categorically assess the impact of MR on education as studies are limited to small groups of students, have different curricula in different regions, and differ in the accessibility of immersive technologies [81].
Another important issue is related to the COVID-19 pandemic and lockdowns, which vastly increased interest in remote teaching methods [6]. Students in medical fields lacked laboratory and clinical experience along with patient interaction. There was a pressing need to provide tools that would make these possible. Most students are very comfortable adopting new technologies and thus a futuristic vision of science was met with great enthusiasm. For example, in some US institutions the HoloLens was loaned to medical students for the duration of the pandemic. As a result, they could not only take an anatomy course in three dimensions but also connect with a lecturer who discussed the content in detail. This provided the only possible student-lecturer contact with active interaction. MR can also be an effective tool to help students understand chemical compounds on the molecular level and grasp the complexity of the human body remotely and in person. Our preliminary results confirmed that MR can increase the effectiveness of delivering complex content via remote learning, which continues to play a significant role in education. The technology gives more control to students over their learning experience, enabling them to be more independent. However, more work is required to improve applications, adapt curricula for immersive technologies, and explore their impact on human psyche and perception.

6. Future Plans

Future plans include the formulation of new guidelines in education using Mixed Reality to prepare students for modern clinical conditions where advanced imaging systems use MR technology. We will also build new guidelines for adaptive MR medical training systems to incorporate a mechanism for tracking performance and recognize skill deficiencies to generate an optimal adaptive training schedule. It will be based, among others, on the NASA Task Load Index, while skill acquisition will be based on the skill retention theory. This will improve the user’s ability to interactively assess the level of skill learning and decay, optimize skill relearning across levels of experience, and positively impact skill maintenance. We also plan to develop and implement automatic anatomical segmentation tools through Machine and Deep Learning that will be validated by experts in anatomy and radiology. The following applications for Microsoft HoloLens 2 Mixed Reality goggles will be developed:
-
Software for displaying any 3D chemical models;
-
Software for displaying any computer-generated models of human anatomy elements;
-
Software for science popularization;
-
Courses/training consisting mainly of 3D models, animations and user interaction on the chemical and biochemical changes affecting disease, disease pathophysiology, how they can be visualized in 3D, and their diagnosis and treatment as a MR visualization.
In the next step, the medical Metaverse platform with scenarios similar to the real teaching environment and digital avatars for face-to-face communication will be developed and implemented. With technological advances, it will also become possible to completely mirror the traditional educational world.

Author Contributions

Conceptualization, A.P., K.P. and M.O.; methodology, D.D.-D. and A.P.; programming, D.D.-D.; processing and analyzing data, A.P. and M.O. writing—original draft preparation, A.P., K.P., R.K. and M.O.; writing—review and editing, A.P., K.P., R.K. and M.O. All authors have read and agreed to the published version of the manuscript.

Funding

This study was partially supported and financed by the National Centre for Research and Development under Grant Lider No. LIDER/17/0064/L-11/19/NCBR/2020. This study was also partially supported by grant Erasmus+ No. 2020-1-PL01-KA203-HE-082077.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Siriborvornratanakul, T. Enhancing User Experiences of Mobile-Based Augmented Reality via Spatial Augmented Reality: Designs and Architectures of Projector-Camera Devices. Adv. Multimed. 2018, 2018, 8194726. [Google Scholar] [CrossRef] [Green Version]
  2. Coyne, L.; Merritt, T.A.; Parmentier, B.L.; Sharpton, R.A.; Takemoto, J.K. The Past, Present, and Future of Virtual Reality in Pharmacy Education. Am. J. Pharm. Educ. 2019, 83, 7456. [Google Scholar] [CrossRef] [PubMed]
  3. Masood, T.; Egger, J. Augmented reality in support of Industry 4.0—Implementation challenges and success factors. Rob. Comput. Integr. Manuf. 2019, 58, 181–195. [Google Scholar] [CrossRef]
  4. Siriborvornratanakul, T. Through the Realities of Augmented Reality. In HCI International 2019—Late Breaking Papers; Stephanidis, C., Ed.; Springer: Cham, Switzerland, 2019; p. 11786. [Google Scholar]
  5. Kichloo, A.; Albosta, M.; Dettloff, K.; Wani, F.; El-Amir, Z.; Singh, J.; Aljadah, M.; Chakinala, R.C.; Kanugula, A.K.; Solanki, S.; et al. Telemedicine, the current COVID-19 pandemic and the future: A narrative review and perspectives moving forward in the USA. Fam. Med. Community Health 2020, 8, e000530. [Google Scholar] [CrossRef] [PubMed]
  6. Pregowska, A.; Masztalerz, K.; Garlińska, M.; Osial, M. A Worldwide Journey through Distance Education—From the Post Office to Virtual, Augmented and Mixed Realities, and Education during the COVID-19 Pandemic. Educ. Sci. 2021, 11, 118. [Google Scholar] [CrossRef]
  7. Zhuang, W.; Xiao, Q. Facilitate active learning: The role of perceived benefits of using technology. J. Educ. Bus. 2018, 93, 88–96. [Google Scholar] [CrossRef]
  8. Whitley, H.P. Active-learning diabetes simulation in an advanced pharmacy practice experience to develop patient empathy. Am. J. Pharm. Educ. 2012, 76, 203. [Google Scholar] [CrossRef] [Green Version]
  9. Habig, S. Who can benefit from augmented reality in chemistry? Sex differences in solving stereochemistry problems using augmented reality. Br. J. Educ. Technol. 2020, 51, 629–644. [Google Scholar] [CrossRef] [Green Version]
  10. Raja, M.; Lakshmi Priya, G.G. Using Virtual Reality and Augmented Reality with ICT Tools for Enhancing Quality in the Changing Academic Environment in COVID-19 Pandemic: An Empirical Study. In Technologies, Artificial Intelligence and the Future of Learning Post-COVID-19. Studies in Computational Intelligence; Hamdan, A., Hassanien, A.E., Mescon, T., Alareeni, B., Eds.; Springer: Cham, Switzerland, 2022; Volume 1019. [Google Scholar]
  11. Wu, T.C.; Ho, C.B. A scoping review of metaverse in emergency medicine. Australas. Emerg. Care 2022, 8, S2588-994X(22)00052-5. [Google Scholar]
  12. Jovanovic, M.; Guterman-Ram, G.; Marini, J.C. Osteogenesis Imperfecta: Mechanisms and Signaling Pathways Connecting Classical and Rare OI Types. Endocr. Rev. 2022, 43, 61–90. [Google Scholar] [CrossRef]
  13. Munzer, B.W.; Khan, M.M.; Shipman, B.; Mahajan, P. Augmented reality in emergency medicine: A scoping review. J. Med. Internet Res. 2019, 21, e12368. [Google Scholar] [CrossRef]
  14. Birlo, M.; Edwards, P.J.E.; Clarkson, M.; Stoyanov, D. Utility of optical see-through head mounted displays in augmented reality-assisted surgery: A systematic review. Med. Image Anal. 2022, 77, 102361. [Google Scholar] [CrossRef]
  15. Gerup, J.; Soerensen, C.B.; Dieckmann, P. Augmented reality and mixed reality for healthcare education beyond surgery: An integrative review. Int. J. Med. Educ. 2020, 11, 1–18. [Google Scholar] [CrossRef] [Green Version]
  16. Brun, H.; Bugge, R.A.B.; Suther, L.K.R.; Birkeland, S.; Kumar, R.; Pelanis, E.; Elle, O.J. Mixed reality holograms for heart surgery planning: First user experience in congenital heart disease. Eur. Heart J. Cardiovasc. Imaging 2019, 20, 883–888. [Google Scholar] [CrossRef] [Green Version]
  17. Korayem, G.B.; Alshaya, O.A.; Kurdi, S.M.; Alnajjar, L.I.; Badr, A.F.; Alfahed, A.; Cluntun, A. Simulation-Based Education Implementation in Pharmacy Curriculum: A Review of the Current Status. Adv. Med. Educ. Pract. 2022, 13, 649–660. [Google Scholar] [CrossRef]
  18. Nicolazzo, J.A.; Chuang, S.; Mak, V. Incorporation of MyDispense, a Virtual Pharmacy Simulation, into Extemporaneous Formulation Laboratories. Healthcare 2022, 10, 1489. [Google Scholar] [CrossRef]
  19. Tarng, W.; Tseng, Y.-C.; Ou, K.-L. Application of Augmented Reality for Learning Material Structures and Chemical Equilibrium in High School Chemistry. Systems 2022, 10, 141. [Google Scholar] [CrossRef]
  20. Moro, C.; Phelps, C.; Redmond, P.; Stromberga, Z. HoloLens and mobile augmented reality in medical and health science education: A randomised controlled trial. Br. J. Educ. Technol. 2021, 52, 680–694. [Google Scholar] [CrossRef]
  21. Siriborvornratanakul, T. A Study of Virtual Reality Headsets and Physiological Extension Possibilities. In Proceedings of the Computational Science and Its Applications—ICCSA 2016, Beijing, China, 4–7 July 2016; Springer: Cham, Switzerland, 2016; Volume 9787. [Google Scholar]
  22. Trenfield, S.J.; Awad, A.; McCoubrey, L.E.; Elbadawi, M.; Goyanes, A.; Gaisford, S.; Basit, A.W. Advancing pharmacy and healthcare with virtual digital technologies. Adv. Drug Deliv. Rev. 2022, 182, 114098. [Google Scholar] [CrossRef]
  23. Ingeson, M.; Blusi, M.; Nieves, J.C. Microsoft Hololens—A Mhealth Solution for Medication Adherence. In Proceedings of the International Workshop on Artificial Intelligence in Health, Stockholm, Sweden, 13–14 July 2018; Springer: Cham, Switzerland, 2018; pp. 99–115. [Google Scholar]
  24. Yang, S.; Pang, X.; He, X. A Novel Mobile Application for Medication Adherence Supervision Based on AR and OpenCV Designed for Elderly Patients. In Human Aspects of IT for the Aged Population. Supporting Everyday Life Activities, Proceedings of the 7th International Conference, ITAP 2021, Online, 24–29 July 2021; Springer: Cham, Switzerland, 2021; pp. 335–347. [Google Scholar]
  25. Karavasili, C.; Eleftheriadis, G.K.; Gioumouxouzis, C.; Andriotis, E.G.; Fatouros, D.G. Mucosal drug delivery and 3D printing technologies: A focus on special patient populations. Adv. Drug Deliv. Rev. 2021, 176, 113858. [Google Scholar] [CrossRef]
  26. Eleftheriadis, G.K.; Kantarelis, E.; Monou, P.K.; Andriotis, E.G.; Bouropoulos, N.; Tzimtzimis, E.K.; Tzetzis, D.; Rantanen, J.; Fatouros, D.G. Automated digital design for 3D-printed individualized therapies. Int. J. Pharm. 2021, 599, 120437. [Google Scholar] [CrossRef] [PubMed]
  27. Morimoto, T.; Kobayashi, T.; Hirata, H.; Otani, K.; Sugimoto, M.; Tsukamoto, M.; Yoshihara, T.; Ueno, M.; Mawatari, M. XR (Extended Reality: Virtual Reality, Augmented Reality, Mixed Reality) Technology in Spine Medicine: Status Quo and Quo Vadis. J. Clin. Med. 2022, 11, 470. [Google Scholar] [CrossRef] [PubMed]
  28. Levy, J.B.; Kong, E.; Johnson, N.; Khetarpal, A.; Tomlinson, J.; Martin, G.F.; Tanna, A. The mixed reality medical ward round with the MS HoloLens 2: Innovation in reducing COVID-19 transmission and PPE usage. Future Healthc. J. 2021, 8, e127–e130. [Google Scholar] [CrossRef] [PubMed]
  29. Hempel, D.; Sinnathurai, S.; Haunhorst, S.; Seibel, A.; Michels, G.; Heringer, F.; Recker, F.; Breitkreutz, R. Influence of case-based e-learning on students’ performance in point-of-care ultrasound courses: A randomized trial. Eur. J. Emerg. Med. 2016, 23, 298–304. [Google Scholar] [CrossRef]
  30. Ferrer-Torregrosa, J.; Jiménez-Rodríguez, M.Á.; Torralba-Estelles, J.; Garzón-Farinós, F.; Pérez-Bermejo, M.; Fernández-Ehrling, N. Distance learning ects and flipped classroom in the anatomy learning: Comparative study of the use of augmented reality, video and notes. BMC Med. Educ. 2016, 16, 230. [Google Scholar] [CrossRef] [Green Version]
  31. Duarte, M.L.; Santos, L.R.; Guimarães Júnior, J.B.; Peccin, M.S. Learning anatomy by virtual reality and augmented reality. A scope review. Morphologie 2020, 104, 254–266. [Google Scholar] [CrossRef]
  32. Ghaednia, H.; Fourman, M.S.; Lans, A.; Detels, K.; Dijkstra, H.; Lloyd, S.; Sweeney, A.; Oosterhoff, J.H.; Schwab, J.H. Augmented and virtual reality in spine surgery, current applications and future potentials. Spine J. 2021, 21, 1617–1625. [Google Scholar] [CrossRef]
  33. Pettersen, E.F.; Goddard, T.D.; Huang, C.C.; Couch, G.S.; Greenblatt, D.M.; Meng, E.C.; Ferrin, T.E. UCSF chimera—A visualization system for exploratory research and analysis. J. Comput. Chem. 2014, 25, 1605–1612. [Google Scholar] [CrossRef] [Green Version]
  34. Cortés Rodríguez, F.; Dal Peraro, M.; Abriata, L.A. Online tools to easily build virtual molecular models for display in augmented and virtual reality on the web. J. Mol. Graph. Model. 2022, 114, 108164. [Google Scholar] [CrossRef]
  35. Cortés Rodríguez, F.; Frattini, G.; Krapp, L.F.; Martinez-Hung, H.; Moreno, D.M.; Roldán, M.; Salomón, J.; Stemkoski, L.; Traeger, S.; Dal Peraro, M. MoleculARweb: A web site for chemistry and structural biology education through interactive augmented reality out of the box in commodity devices. J. Chem. Educ. 2021, 98, 2243–2255. [Google Scholar] [CrossRef]
  36. Cassidy, K.C.; Šefčík, J.; Raghav, Y.; Chang, A.; Durrant, J.D. ProteinVR: Web-based molecular visualization in virtual reality. PLoS Comput. Biol. 2020, 16, e1007747. [Google Scholar] [CrossRef]
  37. Xu, K.; Liu, N.; Xu, J.; Guo, C.; Zhao, L.; Wang, H.W.; Zhang, Q.C. VRmol: An integrative web-based virtual reality system to explore macromolecular structure. Bioinformatics 2021, 37, 1029–1031. [Google Scholar] [CrossRef]
  38. Tsang, W.Y.; Fan, P.; Raj, S.D.H.; Tan, Z.J.; Lee, I.Y.Y.; Boo, I.; Yap, K.Y.L. Development of a Three-Dimensional (3D) Virtual Reality Apprenticeship Program (VRx) for Training of Medication Safety Practices. Int. J. Digit. Health 2022, 2, 4. [Google Scholar] [CrossRef]
  39. Sakshuwong, S.; Weir, H.; Raucci, U.; Martínez, T.J. Bringing chemical structures to life with augmented reality, machine learning, and quantum chemistry. J. Chem. Phys. 2022, 156, 204801. [Google Scholar] [CrossRef]
  40. Macariua, C.; Iftenea, A.; Gîfua, D. Learn Chemistry with Augmented Reality. Procedia Comput. Sci. 2020, 176, 2133–2142. [Google Scholar] [CrossRef]
  41. Low, D.Y.S.; Poh, P.E.; Tang, S.Y. Assessing the impact of augmented reality application on students’ learning motivation in chemical engineering. Educ. Chem. Eng. 2022, 39, 31–43. [Google Scholar] [CrossRef]
  42. De Micheli, A.J.; Valentin, T.; Grillo, F.; Kapur, M.; Schuerle, S.J. Mixed Reality for an Enhanced Laboratory Course on Microfluidics. Chem. Educ. 2022, 99, 1272–1279. [Google Scholar] [CrossRef]
  43. Pan, Z.; Miao, J.; Cai, N.; Luo, T.; Zhang, M.; Pan, Z. Simulating the Dilution of Sulfuric Acid on Mixed Reality Platform. In Proceedings of the 2022 8th International Conference on Virtual Reality (ICVR), Nanjing, China, 26–28 May 2022; pp. 135–141. [Google Scholar]
  44. Kolecki, R.; Pregowska, A.; Dąbrowa, J.; Skuciński, J.; Pulanecki, T.; Walecki, P.; van Dam, P.M.; Dudek, D.; Richter, P.; Proniewska, K. Assessment of the utility of mixed reality in medical education. Transl. Res. Anat. 2022, 28, 100214. [Google Scholar] [CrossRef]
  45. Boomgaard, A.; Fritz, K.A.; Isafiade, O.E.; Kotze, R.C.M.; Ekpo, O.; Smith, M.; Gessler, T.; Filton, K.J.; Cupido, C.C.; Aden, B.; et al. A Novel Immersive Anatomy Education System (Anat Hub): Redefining Blended Learning for the Musculoskeletal System. Appl. Sci. 2022, 12, 5694. [Google Scholar] [CrossRef]
  46. Digital Trends. Available online: https://www.digitaltrends.com/virtual-reality/hololens-holoanatomy-award-jackson-hole-science-media-awards/ (accessed on 15 November 2022).
  47. Monterubbianesi, R.; Tosco, V.; Vitiello, F.; Orilisi, G.; Fraccastoro, F.; Putignano, A.; Orsini, G. Augmented, Virtual and Mixed Reality in Dentistry: A Narrative Review on the Existing Platforms and Future Challenges. Appl. Sci. 2022, 12, 877. [Google Scholar] [CrossRef]
  48. Blanchard, J.; Koshal, S.; Morley, S.; McGurk, M. The use of mixed reality in dentistry. Br. Dent. J. 2022, 233, 261–265. [Google Scholar] [CrossRef] [PubMed]
  49. Reymus, M.; Liebermann, A.; Diegritz, C. Virtual Reality: An Effective Tool for Teaching Root Canal Anatomy to Undergraduate Dental Students—A Preliminary Study. Int. Endod. J. 2020, 53, 1581–1587. [Google Scholar] [CrossRef] [PubMed]
  50. Dolega-Dolegowski, D.; Proniewska, K.; Dolega-Dolegowska, M.; Pregowska, A.; Hajto-Bryk, J.; Trojak, M.; Chmiel, J.; Walecki, P.; Fudalej, P.S. Application of holography and augmented reality based technology to visualize the internal structure of the dental root—A proof of concept. Head Face Med. 2022, 18, 12. [Google Scholar] [CrossRef] [PubMed]
  51. Kaylor, J.; Hooper, V.; Wilson, A.; Burkert, R.; Lyda, M.; Fletcher, K.; Bowers, E. Reliability Testing of Augmented Reality Glasses Technology: Establishing the Evidence Base for Telewound Care. J. Wound Ostomy Cont. Nurs. 2019, 46, 485–490. [Google Scholar] [CrossRef] [PubMed]
  52. Papalois, Z.A.; Aydin, A.; Khan, A.; Mazaris, E.; Rathnasamy Muthusamy, A.S.; Dor, E.J.M.F.; Dasguota, P.; Ahmed, K. HoloMentor: A Novel Mixed Reality Surgical Anatomy Curriculum for Robot-Assisted Radical Prostatectomy. Eur. Surg. Res. 2022, 63, 40–45. [Google Scholar] [CrossRef]
  53. Proniewska, K.; Khokhar, A.A.; Dudek, D. Advanced imaging in interventional cardiology: Mixed reality to optimize preprocedural planning and intraprocedural monitoring. Kardiol. Pol. 2021, 79, 331–335. [Google Scholar] [CrossRef]
  54. Na, F.; Wang, L.; Wu, C.; Ding, Y. Contrast-enhanced ultrasound combined with augmented reality medical technology in the treatment of rabbit liver cancer with high-energy focused knife. Comput. Intell. 2022, 38, 121–138. [Google Scholar] [CrossRef]
  55. Smith, C.; Friel, C.J. Development and use of augmented reality models to teach medicinal chemistry. Curr. Pharm. Teach. Learn. 2021, 13, 1010–1017. [Google Scholar] [CrossRef]
  56. Nounou, M.I.; Eassa, H.A.; Orzechowski, K.; Eassa, H.A.; Edouard, J.; Stepak, N.; Khdeer, M.; Kalam, M.; Huynh, D.; Kwarteng, E.; et al. Mobile-Based Augmented Reality Application in Pharmacy Schools Implemented in Pharmaceutical Compounding Laboratories: Students’ Benefits and Reception. Pharmacy 2022, 10, 72. [Google Scholar] [CrossRef]
  57. Roosan, D.; Chok, J.; Baskys, A.; Roosan, M.R. PGxKnow: A pharmacogenomics educational HoloLens application of augmented reality and artificial intelligence. Future Med. 2022, 23, 235–245. [Google Scholar] [CrossRef]
  58. Hope, D.; Haywood, A.; Bernaitis, N. Current research: Incorporating vaccine administration in pharmacy curriculum: Preparing students for emerging roles. Aust. J. Pharm. 2014, 95, 1129–1160. [Google Scholar]
  59. Bushell, M.; Morrissey, H.; Nuffer, W.; Ellis, S.L.; Ball, P.A. Development and design of injection skills and vaccination training program targeted for Australian undergraduate pharmacy students. Curr. Pharm. Teach. Learn. 2015, 7, 771–779. [Google Scholar] [CrossRef]
  60. Bushell, M.; Frost, J.; Deeks, L.; Kosari, S.; Hussain, Z.; Naunton, M. Evaluation of Vaccination Training in Pharmacy Curriculum: Preparing Students for Workforce Needs. Pharmacy 2020, 8, 151. [Google Scholar] [CrossRef]
  61. Essel, D.; Thompson, J.; Chapman, S. The effect of an augmented reality educational tool on the motivation towards learning in pharmacy students: An evaluative survey. Int. J. Pharm. Pract. 2022, 30, i30–i31. [Google Scholar] [CrossRef]
  62. Elford, A.; Gwee, C.; Veal, M.; Jani, R.; Sambell, R.; Kashef, S.; Love, P. Identification and Evaluation of Tools Utilised for Measuring Food Provision in Childcare Centres and Primary Schools: A Systematic Review. Int. J. Environ. Res. Public Health 2022, 19, 4096. [Google Scholar] [CrossRef]
  63. Silva, R.d.O.S.; de Araújo, D.C.S.A.; dos Santos Menezes, P.W.; Zimmer Neves, E.R.; de Lyra, S.P., Jr. Digital pharmacists: The new wave in pharmacy practice and education. Int. J. Clin. Pharm. 2022, 44, 775–780. [Google Scholar] [CrossRef]
  64. Slicer. Available online: https://www.slicer.org (accessed on 15 November 2022).
  65. Pinter, C.; Lasso, A.; Fichtinger, G. Polymorph segmentation representation for medical image computing. Comput. Methods Programs Biomed. 2019, 171, 19–26. [Google Scholar] [CrossRef]
  66. Zhu, L.; Kolesov, I.; Gao, Y.; Kikinis, R.; Tannenbaum, A. An Effective Interactive Medical Image Segmentation Method Using Fast GrowCut. In Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), Interactive Medical Image Computing Workshop, Boston, MA, USA, 4–18 September 2014. [Google Scholar]
  67. Zukić, D.; Vicory, J.; McCormick, M.; Wisse, L.; Gerig, G.; Yushkevich, P.; Aylward, S. ND morphological contour interpola-tion. Insight J. 2016, 17, 1–27. [Google Scholar]
  68. Birt, J.; Stromberga, Z.; Cowling, M.; Moro, C. Mobile Mixed Reality for Experiential Learning and Simulation in Medical and Health Sciences Education. Information 2018, 9, 31. [Google Scholar] [CrossRef] [Green Version]
  69. Gattullo, M.; Laviola, E.; Boccaccio, A.; Evangelista, A.; Fiorentino, M.; Manghisi, V.M.; Uva, A.E. Design of a Mixed Reality Application for STEM Distance Education Laboratories. Computers 2022, 11, 50. [Google Scholar] [CrossRef]
  70. Zhang, Z.; Wu, Y.; Pan, Z.; Li, W.; Su, Z. A novel animation authoring framework for the virtual teacher performing experiment in mixed reality. Comput. Appl. Eng. Educ. 2022, 30, 550–563. [Google Scholar] [CrossRef]
  71. Lee, T.H.; Munasinghe, V.; Li, Y.M.; Xu, J.; Lee, H.J.; Kim, S.J. GAN—Based Medical Image Registration for Augmented Reality Applications. In Proceedings of the 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS), Incheon, Republic of Korea, 13–15 June 2022; pp. 279–282. [Google Scholar]
  72. Duan, X.; Kang, S.J.; Choi, J.I.; Kim, S.K. Mixed Reality System for Virtual Chemistry Lab. KSII Trans. Internet Inf. Syst. 2020, 14, 1673–1688. [Google Scholar]
  73. Yang, D.; Zhou, J.; Chen, R.; Song, Y.; Song, Z.; Zhang, X.; Wang, Q.; Wang, K.; Zhou, C.; Sun, J.; et al. Expert consensus on the metaverse in medicine. Clin. eHealth 2022, 5, 1–9. [Google Scholar] [CrossRef]
  74. Pimentel, D.; Fauville, G.; Frazier, F.; McGivney, G.; Rosas, S.; Woolsey, E. An Introduction to Learning in the Metaverse. Meridian Treehouse. 2022. Available online: https://scholar.harvard.edu/files/mcgivney/files/introductionlearningmetaverse-april2022-meridiantreehouse.pdf (accessed on 15 November 2022).
  75. Gartner. Available online: https://www.gartner.com/en/newsroom/press-releases/2022-02-07-gartner-predicts-25-percent-of-people-will-spend-at-least-one-hour-per-day-in-the-metaverse-by-2026 (accessed on 15 November 2022).
  76. Sun, M.; Xie, L.; Liu, Y.; Li, K.; Jiang, B.; Lu, Y.; Yang, Y.; Yu, H.; Song, Y.; Bai, C.; et al. The metaverse in current digital medicine. Clin. eHealth 2022, 5, 52–57. [Google Scholar] [CrossRef]
  77. Llobera, J.; Spanlang, B.; Ruffini, G.; Slater, M. Proxemics with multiple dynamic characters in an immersive virtual environment. ACM Trans. Appl. Percept. 2010, 8, 1–12. [Google Scholar] [CrossRef] [Green Version]
  78. Plechatá, A.; Makransky, G.; Böhm, R. Can extended reality in the metaverse revolutionise health communication? NPJ Digit. Med. 2022, 5, 132. [Google Scholar] [CrossRef]
  79. Xi, N.; Chen, J.; Gama, F.; Riar, M.; Hamari, J. The challenges of entering the metaverse: An experiment on the effect of extended reality on workload. Inf. Syst. Front. 2022, 1–22. [Google Scholar] [CrossRef]
  80. Christopoulos, A.; Pellas, N.; Kurczaba, J.; Macredie, R. The effects of augmented reality-supported instruction in tertiary-level medical education. Br. J. Educ. Technol. 2022, 53, 307–325. [Google Scholar] [CrossRef]
  81. Pickering, J.D.; Panagiotis, A.; Ntakakis, G.; Athanassiou, A.; Babatsikos, E.; Bamidis, P.D. Assessing the difference in learning gain between a mixed reality application and drawing screencasts in neuroanatomy. Anat. Sci. Educ. 2022, 15, 628–635. [Google Scholar] [CrossRef]
  82. Hauze, S.W.; Hoyt, H.H.; Frazee, J.P.; Greiner, P.A.; Marshall, J.M. Enhancing Nursing Education through Affordable and Realistic Holographic Mixed Reality: The Virtual Standardized Patient for Clinical Simulation. In Biomedical Visualisation; Rea, P., Ed.; Advances in Experimental Medicine and Biology; Springer: Cham, Switzerland, 2019; Volume 1120. [Google Scholar]
  83. Abrecht, U.V.; Folta-Schoofs, K.; Behrends, M.; von Jan, U. Effects of mobile augmented reality learning compared to textbook learning on medical students: Randomized controlled pilot study. J. Med. Internet Res. 2013, 15, e182. [Google Scholar] [CrossRef]
  84. Ho, S.; Liu, P.; Palombo, D.J.; Handy, T.C.; Krebs, C. The role of spatial ability in mixed reality learning with the HoloLens. Anat. Sci. Educ. 2022, 15, 1074–1085. [Google Scholar] [CrossRef]
  85. John, B.; Kurian, J.C.; Fitzgerald, R.; Lian Goh, D. Students’ Learning Experience in a Mixed Reality Environment: Drivers and Barriers. Commun. Assoc. Inf. Syst. 2022, 50, 510–535. [Google Scholar] [CrossRef]
Electronics 11 03778 g001 550
Figure 1. System design, for use in remote and in-person medical education, particularly chemistry.
Figure 1. System design, for use in remote and in-person medical education, particularly chemistry.
Electronics 11 03778 g001
Electronics 11 03778 g002 550
Figure 2. Mixed Reality Laboratory with students using HoloLens 2 at Jagiellonian University Medical College in Krakow.
Figure 2. Mixed Reality Laboratory with students using HoloLens 2 at Jagiellonian University Medical College in Krakow.
Electronics 11 03778 g002
Electronics 11 03778 g003 550
Figure 3. Sample 3D models of molecules having a different chemical composition, in particular additional alkyl CH2 group in the right image.
Figure 3. Sample 3D models of molecules having a different chemical composition, in particular additional alkyl CH2 group in the right image.
Electronics 11 03778 g003
Electronics 11 03778 g004 550
Figure 4. The visualization of a large protein as (a) the 3D point cloud molecule, and (b) the spiral structure of the resulting protein.
Figure 4. The visualization of a large protein as (a) the 3D point cloud molecule, and (b) the spiral structure of the resulting protein.
Electronics 11 03778 g004
Electronics 11 03778 g005 550
Figure 5. MR in medical chemistry, displayed in front of the users wearing Microsoft HoloLens2.
Figure 5. MR in medical chemistry, displayed in front of the users wearing Microsoft HoloLens2.
Electronics 11 03778 g005
Electronics 11 03778 g006a 550Electronics 11 03778 g006b 550
Figure 6. Preoperative 3D imaging before tumor resection with the Microsoft HoloLens. The top row of color-coded anatomical structures includes the aorta (red), inferior vena cava (blue), portal vein with its intrahepatic branches (purple, solid), the liver (purple, partially transparent), gallbladder (green), spleen (yellow), and kidneys (dark green). The bottom right model displays bone (white), abdominal aorta with its branches (red), inferior vena cava along with its tributaries and ascending into the right side of the heart (blue), portal system and its tributaries (purple), gallbladder (green), liver (light brown, partially transparent), spleen (brown, solid), lungs (pink), and one kidney (brown, solid). The bottom left image shows the HoloLens2 system being worn along with its live-streamed view onto a local computer.
Figure 6. Preoperative 3D imaging before tumor resection with the Microsoft HoloLens. The top row of color-coded anatomical structures includes the aorta (red), inferior vena cava (blue), portal vein with its intrahepatic branches (purple, solid), the liver (purple, partially transparent), gallbladder (green), spleen (yellow), and kidneys (dark green). The bottom right model displays bone (white), abdominal aorta with its branches (red), inferior vena cava along with its tributaries and ascending into the right side of the heart (blue), portal system and its tributaries (purple), gallbladder (green), liver (light brown, partially transparent), spleen (brown, solid), lungs (pink), and one kidney (brown, solid). The bottom left image shows the HoloLens2 system being worn along with its live-streamed view onto a local computer.
Electronics 11 03778 g006aElectronics 11 03778 g006b
Electronics 11 03778 g007 550
Figure 7. The application of the MR in dental education. This view is displayed for users wearing the Microsoft HoloLens 2.
Figure 7. The application of the MR in dental education. This view is displayed for users wearing the Microsoft HoloLens 2.
Electronics 11 03778 g007
Electronics 11 03778 g008 550
Figure 8. Tumor of the jaw and preoperative 3D strategy planning. (a) 3D model of the jaw with bone loss due to the tumor (b) 3D model of the jaw with implants. The implants were marked in black and yellow. This view is displayed in front of the users wearing the Microsoft HoloLens 2.
Figure 8. Tumor of the jaw and preoperative 3D strategy planning. (a) 3D model of the jaw with bone loss due to the tumor (b) 3D model of the jaw with implants. The implants were marked in black and yellow. This view is displayed in front of the users wearing the Microsoft HoloLens 2.
Electronics 11 03778 g008
Electronics 11 03778 g009 550
Figure 9. Visualization of resorption in 3D. This view is displayed in front of the users wearing Microsoft HoloLens 2.
Figure 9. Visualization of resorption in 3D. This view is displayed in front of the users wearing Microsoft HoloLens 2.
Electronics 11 03778 g009
Table
Table 1. Comparison of immersive technologies-based systems in education, particularly chemistry, medical chemistry, and medicine.
Table 1. Comparison of immersive technologies-based systems in education, particularly chemistry, medical chemistry, and medicine.
ReferenceType of Immersive TechnologiesChemical ContentMedical ContentMedical Chemistry ContentApplication Fields
[79]Metaverse, ARNoNoNoWorkload evaluation
[12]MetaverseNoNoNoArchitecture lessons
[76]MetaverseNoYesNoDigital medicine
[73]MetaverseNoYesNoMedical education (a perspective)
[69]ARNoNoNoSTEM laboratories
[15]ARNoYesNoSurgery Education
[32]ARNoYesNoSurgery Education
[33]ARYesNoNoChemistry education
[35]ARYesNoNoChemistry and biology content
[38]ARYesNoNoMaterials structures and chemical equilibria
[55]ARNoNoYesMedical chemistry
[40]ARYesNoNoChemistry lessons
[56]ARNoYesNoPharmacy education
[36]VRYesNoNoChemistry education, protein
[18]VRNoYesNoPharmacy education
[49]VRNoYesNoDental Education
[48]VRNoYesNoDental Education
[81]MRNoYesNoAnatomy drawing screencasts
[82]MRNoYesNoNursing Education
[50]MRNoYesNoDental education
[16]MRNoYesNoSurgery Education
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Pregowska, A.; Osial, M.; Dolega-Dolegowski, D.; Kolecki, R.; Proniewska, K. Information and Communication Technologies Combined with Mixed Reality as Supporting Tools in Medical Education. Electronics 2022, 11, 3778. https://doi.org/10.3390/electronics11223778

AMA Style

Pregowska A, Osial M, Dolega-Dolegowski D, Kolecki R, Proniewska K. Information and Communication Technologies Combined with Mixed Reality as Supporting Tools in Medical Education. Electronics. 2022; 11(22):3778. https://doi.org/10.3390/electronics11223778

Chicago/Turabian Style

Pregowska, Agnieszka, Magdalena Osial, Damian Dolega-Dolegowski, Radek Kolecki, and Klaudia Proniewska. 2022. "Information and Communication Technologies Combined with Mixed Reality as Supporting Tools in Medical Education" Electronics 11, no. 22: 3778. https://doi.org/10.3390/electronics11223778

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop