Next Article in Journal
Robust Quadrotor Control through Reinforcement Learning with Disturbance Compensation
Next Article in Special Issue
High-Speed Dynamic Projection Mapping onto Human Arm with Realistic Skin Deformation
Previous Article in Journal
Sialic Acid as a Biomarker Studied in Breast Cancer Cell Lines In Vitro Using Fluorescent Molecularly Imprinted Polymers
Previous Article in Special Issue
Silhouettes from Real Objects Enable Realistic Interactions with a Virtual Human in Mobile Augmented Reality
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 11
Issue 7
10.3390/app11073253
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Augmented Reality, Virtual Reality and Artificial Intelligence in Orthopedic Surgery: A Systematic Review

by
Umile Giuseppe Longo
1,*,
Sergio De Salvatore
1,
Vincenzo Candela
1,
Giuliano Zollo
1,
Giovanni Calabrese
1,
Sara Fioravanti
2,
Lucia Giannone
2,
Anna Marchetti
2,
Maria Grazia De Marinis
2 and
Vincenzo Denaro
1
1
Department of Orthopedic and Trauma Surgery, Campus Bio-Medico University, via Alvaro del Portillo, 200, Trigoria, 00128 Rome, Italy
2
Research Unit Nursing Science, Campus Bio-Medico di Roma University, 00128 Rome, Italy
*
Author to whom correspondence should be addressed.
Appl. Sci. 2021, 11(7), 3253; https://doi.org/10.3390/app11073253
Submission received: 16 March 2021 / Revised: 24 March 2021 / Accepted: 31 March 2021 / Published: 5 April 2021
(This article belongs to the Collection Virtual and Augmented Reality Systems)
Browse Figure
Versions Notes

Abstract

:
Background: The application of virtual and augmented reality technologies to orthopaedic surgery training and practice aims to increase the safety and accuracy of procedures and reducing complications and costs. The purpose of this systematic review is to summarise the present literature on this topic while providing a detailed analysis of current flaws and benefits. Methods: A comprehensive search on the PubMed, Cochrane, CINAHL, and Embase database was conducted from inception to February 2021. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines were used to improve the reporting of the review. The Cochrane Risk of Bias Tool and the Methodological Index for Non-Randomized Studies (MINORS) was used to assess the quality and potential bias of the included randomized and non-randomized control trials, respectively. Results: Virtual reality has been proven revolutionary for both resident training and preoperative planning. Thanks to augmented reality, orthopaedic surgeons could carry out procedures faster and more accurately, improving overall safety. Artificial intelligence (AI) is a promising technology with limitless potential, but, nowadays, its use in orthopaedic surgery is limited to preoperative diagnosis. Conclusions: Extended reality technologies have the potential to reform orthopaedic training and practice, providing an opportunity for unidirectional growth towards a patient-centred approach.

1. Introduction

Nowadays, Virtual Reality (VR), Augmented Reality (AR), and Artificial Intelligence (AI) are commonly adopted as operative and training tools for surgery. However, the real advantages and fields of application of these technologies are still new and in constant evolution. VR and AR, also defined as “extended reality technologies” [1], are similar, but specific differences allow surgeons to use them in various contexts. With VR, the surgeon can interact with a virtual environment that is completely generated by a computer. In this setting, the surgeon can prepare preoperative planning or perform a surgical simulation [2,3,4]. However, the entire simulation is virtual and does not allow the surgeon to interact with the patient. Conversely, with AR, it is possible to visualize the real world [1]. A superimposed computer-generated image is directly projected on the surgeon’s field of view, overcoming VR limitations. In fact, the user cannot interact with the external environment, including the patient, the operating room and surgical tools, and cannot see the digital content and reality at the same time. Both VR and AR represent a step forward in surgical training since they simulate multiple surgical cases [5,6,7].
Orthopaedic surgery significantly benefits from these technologies, as they have been adopted for preoperative planning, intraoperative navigation and postoperative rehabilitation [8,9]. Extended reality technologies have already been used to improve the performance of young surgeons and residents in arthroscopic knee surgery [10] and hip arthroplasty [11]. Furthermore, AR is commonly used to assist surgeons during specific procedures, such as minimally invasive surgery, spine surgery, and robot-assisted surgery.
In some articles, authors also describe mixed reality (MR). MR may be defined as a hybrid between VR and AR. This technology is able to generate a virtual environment while also giving the user the possibility to interact with their surroundings. However, the functional and technical similarities in MR and AR applications make these two almost interchangeable [12].
AI is adopted in orthopaedic surgery as well, with particular applications in imaging analysis and surgical training. The mechanism behind AI employs algorithms that recognize patterns in complex medical data received by the computer [13,14,15]. This technology could also aid surgeons by improving diagnostic accuracy and preventing human error [16].
In conclusion, the following systematic review aims to assess the role of VR, AR and AI in orthopaedic surgery and their possible future value.

2. Materials and Methods

The present paper focused on studies concerning the use of VR, AR and AI in orthopedic surgery. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines were used to improve the reporting of the review (Figure 1).

2.1. Eligibility Criteria and Search Strategy

The research question was formulated using a PICOS approach—Patient (P); Intervention (I); Comparison (C); Outcome (O), and Study design (S). This study selects those articles that described orthopaedic surgical procedures (P) assisted by VR, AR and AI (I). The aim was to assess the role of these technologies in orthopedic surgery and their potential future applications (O). For this purpose, randomized studies (RCTs) and non-randomized controlled studies (NRCTs), as prospective, retrospective, cross-sectional, observational studies, case-series, pilot and case-control studies, were included (S).
A comprehensive search on the databases PubMed, Medline, Cochrane, CINAHL and Embase databases was conducted from inception to February 2021. The following keywords were used isolated and combined—Virtual Reality; Augmented Reality; Artificial Intelligence; Orthopedic Surgery; Reality; preoperative planning; training; intraoperative; and mixed reality. All the keywords were searched isolated and combined with their MeSH terms. More studies were searched among the reference lists of the selected papers. The exclusion criteria included reviews, books, case reports, technical notes, letters to editors, instructional courses, in vitro and cadaver studies.

2.2. Study Selection and Data Collection

This systematic review was carried out in February 2021. Only English and Italian publications were included. The initial search of the article was conducted by two authors (SF and LG) using the search protocol previously described. The following research order was adopted—titles were screened first, then abstracts and full papers. A paper was considered potentially relevant and its full text reviewed if, following a discussion between the two independent reviewers, it could not be excluded based on its title and abstract. The screening process was performed using CADIMA software [18]. The number of articles excluded or included was registered and reported in a PRISMA flowchart (Figure 1). For designing the PRISMA, the rules by Liberati et al. were followed [17].

2.3. Quality Assessment

The Cochrane Risk of Bias Tool was used to assess the quality and potential bias of the included Randomized Control Trials [19]. The Cochrane Risk of Bias Tool classifies evidence using seven different domains—sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting and other potential sources of bias. A quantitative score was assigned to each judgement as follows—Low risk of bias = 0; Uncertain risk of bias = 1; High risk of bias = 2. An overall quality score was calculated for each RCT study by summing the values of the different items using the following scale:
⚬
Overall score ≤1 High Quality;
⚬
Overall score ≤3 Moderate Quality; and
⚬
Overall score >3 Low Quality.
The Methodological Index for Non-Randomized Studies (MINORS) was used for quality assessment of non-randomized studies [20]. This score consists of 12 items—clearly stated aim; inclusion of consecutive patients; prospective data collection; endpoints appropriate to study aim; unbiased assessment of study endpoint; follow-up period appropriate to study aim; <5% lost to follow-up; prospective calculation of study size; adequate control group; contemporary groups; and baseline equivalence of groups and adequate statistical analyses. The reviewers individually evaluated all these items. The MINORS items were scored 0 if not reported, 1 when reported but inadequate, and 2 when reported and adequate. The ideal global score was 20 for NRCTs. The simplicity of MINORS comprising only 12 items makes this item readily usable by both readers and researchers. The reliability of this score has already been demonstrated [20].
Two reviewers independently evaluated (SF/LG) the potential risk of bias of the studies. The consensus was reached by the two reviewers (MGDM/VD) when there was a difference in opinion on an item. If no consensus was reached, the independent opinion of a third reviewer was decisive (SDS).

2.4. Data Synthesis and Analysis

Data were extracted and synthesized through Microsoft Excel. General study characteristics extracted were—author and year, type of study, levels of evidence, sample size, country, purpose (preoperative planning, intraoperative use and surgical training), hardware, surgical procedure and conclusions. Due to the heterogeneity of the study, only qualitative characteristics were described. Considering the heterogeneity of the included studies, it was not possible to perform a meta-analysis.

3. Results

According to the PRISMA protocol, a flow-chart diagram showing the selection process of the studies was reported (Figure 1). A total of 1452 studies were found (no additional studies were found in the grey literature, and no unpublished studies were retrieved). A total of 1211 studies after duplicate removal were maintained. Of that, 974 were excluded from the study through title and abstract screening because they were not in line with our objective (n = 572), non-orthopaedic topics (n = 488) or were reviews (n = 56). Then, 95 full-text articles were screened. Of these studies, 73 were excluded (not inherent to the study = 65; no full-text available = 7). After this process, 21 articles were eligible for this study.

3.1. Study Selection and Patient Characteristics

All of the studies included, excluding four articles, reported the patient sample size. Due to the quality of the data and their heterogeneity, a meta-analysis was not performed. The selected articles included three RCTs [5,21,22] and 18 NRCTs [11,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39] (five pilot, three prospective comparative, two prospective observational, two prospective cohort, two prospective case-control, two retrospective comparative, two retrospective cohort). Studies were published between 2013 [21] and 2021 [28]. VR was used for training procedures and preoperative planning [2,3,5,21,22,27,32,33,36]. AR was utilized for preoperative planning and intraoperative purposes [2,3,11,23,24,25,26,28,29,30,31,34,35,37,38,39]. AI was found reported only in two studies and was adopted to improve the preoperative diagnosis accuracy [27,33]. Limited studies reported extended reality technology usage in orthopaedic procedures compared to other surgeries [1,40].

3.2. Quality Assessment

The Cochrane Risk of Bias Tool was used to assess RCT quality. Concerning the RCTs, two studies with an overall “moderate” quality [5,21] and one with an overall “high” quality [22] were found. The MINORS tool was used to assess the risk of bias in NRCTs. Among these studies, fifteen studies (83.3%) [11,24,25,27,29,30,31,32,33,34,35,36,37,38,39] had a low risk of bias, and three studies (16.7%) [23,26,28] had a high risk of bias. The Cochrane Risk of Bias Tool and MINORS were reported in Table 1 and Table 2.

3.3. Results of Individual Studies

3.3.1. Outcome: Preoperative Planning

Three studies were included (one prospective comparative, one prospective case-control, one retrospective cohort) [27,32,33]. The authors of these studies reported that VR and AR were both excellent tools for preoperative planning. These two technologies allowed the surgeon to act in a fast, precise manner, providing an additional degree of safety when approaching the surgical phase [2,3]. Specifically, AR was useful in the visualization of preoperative planning images directly overlaying them on the patient. The most frequently utilized hardware were head-mounted displays (HMDs), including Magic Leap One, backed by Google and Microsoft HoloLens.
Two articles were found regarding AI use in preoperative planning. The former depicted its use in lumbar spinal stenosis detection [33]. The purpose of this study was to classify central lumbar spinal stenosis in four grades—none, mild, moderate and severe. This classification was compared with the independent readings of a spinal surgeon. The system can learn how to grade the severity of lumbar stenosis with outstanding results, providing many possible uses in epidemiological studies. The second article described the accuracy of AI in predicting, diagnosing, and classifying patients affected by cervical spondylotic myelopathy (CSM) [27]. The purpose of this study was mainly diagnostic, as it aimed to understand how neural networks can be used to predict CSM severity. Two different neural network models were employed to complete these tasks. Both models showed high accuracy percentages. According to MINORS, the overall quality of evidence in these studies was assessed in the range between “low” and “high”. All the results are reported in Table 3.

3.3.2. Outcome: Intraoperative Use

Fourteen studies were included (five pilot, one retrospective cohort, two retrospective comparative, two prospective comparative, one prospective case-control, two prospective cohort, one prospective observational) [11,23,24,25,26,28,29,30,31,34,35,37,38,39]. The authors of these studies reported that AR efficiently aided surgeons in performing surgical operations. Moreover, it supported the surgeon in a broad range of aspects, enabling them to understand the 3-D anatomy, providing extremely accurate guidance and reducing surgery time and radiation exposure at the same time. AR has been demonstrated to be successful in different surgical procedures, including K-wire placement, vertebroplasty and pedicle screw placement [2]. Only one study reported no differences in the results achieved by the AR group compared to the traditional group during total hip arthroplasty [34]. The hardware used in these procedures included head-mounted displays, such as HoloLens or Xvision (Augmedics). One of the articles mentioned MR [28] rather than AR. Different AR registration techniques were used in these studies. These include preoperative CT scans, also used for intraoperative capturing [3], or intraoperative computed tomography (iCT) imaging [25], which ensures high navigational accuracy. In other cases, an optically tracked reference plate was strapped to the patient’s limb [28] or a tracked tool was used to perform registration of preoperative CT data directly on the anatomical landmarks of the patient [30]. Lastly, CT scan data were integrated with ultrasound imaging [23], and markerless AR was used to perform real-time registration and to update imaging data during surgery [9].
According to MINORS, the overall quality of evidence in these studies was assessed in the range between “low” and “high”. All the results are reported in Table 3.

3.3.3. Outcome: Surgical Training

Four studies were included (three RCTs, one prospective observational) [5,21,22,36]. The authors of these studies reported that VR showed promising results, significantly improving trainee and resident skills. These studies showed that VR-trained residents performed surgery faster and with fewer errors compared to those who were traditionally trained. Moreover, VR-trained residents have proven more skillful in performing surgical tasks rather than other control groups, even with minimal guidance, resulting in overall better performance. All the mentioned studies showed high construct validity, as the levels of skill of trainees were successfully measured and improved. Moreover, one of the studies [5] showed evidence of criterion-based validity, because it compared a group of trainees using a simulator with one using conventional methods. The overall quality of evidence in the NRCT was assessed as “high”, according to MINORS. According to the Cochrane Risk of Bias Tool, one RCT had an overall score of zero, indicating high quality [22] and the other two had an overall score of two, indicating moderate quality [5,21]. All the results are reported in Table 3.

4. Discussion

Extended reality technologies are beginning to assume a central role in health care, assisting and improving the practice of both trainees and experienced surgeons. Nonetheless, these advancements come with inherent limitations. These technologies are active participants in reducing the learning curve for trainees. However, better and more cost-efficient ways of utilizing them still have to be theorized and employed.
VR reproduces the patient’s anatomy and surgical conditions in a virtual environment, in which the user is immersed through a wearable headset [42]. Furthermore, it can be enhanced by sounds and other inputs [1], providing a fully immersive and detailed experience for simulated surgical procedures. To reinforce its training capability, this technology also offers the possibility to implement features for recording and performance analysis. Simulators, devised to promote and upgrade learning prospects, are indeed being validated for both training and surgical practice. Contextually, it has been shown that VR-trained residents execute surgical tasks faster than those who were trained traditionally. Nonetheless, simulators are still not available in equal numbers in the different subfields of orthopaedics. As per Vaughan et al. [40], while numerous advanced simulators are available for being employed in hip trauma and drilling practices, other procedures seem neglected in comparison [40]. Observing how spinal and hip replacement surgery simulators are still at an embryological state, it is possible to assume that not enough research has been carried out regarding reality technology application in orthopaedic surgery during the last decades [40].
Nowadays, the possibility that orthopaedics will become one of the central focuses of reality technology-related research is feasible. This would, of course, be represented by a uniform development of VR, AR and AI in their applications to different subfields and procedures. Specifically, while VR is already widely accredited to create simulators for surgical training and preoperative planning, AR is more promising for intraoperative purposes [25]. This technology could aid the surgeon in understanding and visualizing the surgical site, including the topographical relationship between anatomical landmarks, orthopaedic implants and operating tools [43]. While offering a limited user–machine interaction potential, AR can create virtual images, project them onto real-life surroundings and superimpose them on data obtained by imaging, thereby providing a guided, minimally invasive approach [2,3,4]. However, in orthopaedic trauma surgery, AR use was not already validated [9]. As shown in numerous experimental studies [24,44,45,46,47,48,49,50], such as the one conducted by Cho et al. [51], AR’s potential practical applications appear countless. Some of these are still under development, while others have been proven successful in animals and still need to be redesigned for human application. Carl et al. [25] described several types of surgeries performed using AR. Indeed, it has been shown that combining iCT imaging with AR can greatly facilitate spine surgery by ensuring meticulous navigational accuracy and a deep understanding of the surrounding anatomy. In the study conducted by Ogawa et al. [34], forty-six patients underwent acetabular cup placement during total hip arthroplasty. In one group, AR navigation was used, while the other group received traditional surgery. No differences were found between the groups in the end, meaning that surgery was just as successful in the AR group as it was in the traditional one. Even though this result shows that AR does not yield benefits in all cases, it also shows that it does not have adverse effects.
Moreover, minimally invasive procedures represent an increasingly popular trend in the orthopedic field, delineating the value of augmented reality technologies even more clearly [52]. With its visualization potential, AR also has a lot to offer as a training tool, having been proven to reduce the learning curve for trainees.
Unanimity does not appear to have been reached regarding the drawbacks of these technologies due to most of them being improvable in the foreseeable future. Limitations of extended reality technologies include visual fatigue, inaccuracy of these devices, and high costs.
AI, a newly rising technology, achieved remarkable improvements in the medical field. Despite its almost flawless prediction accuracy in femur fracture classification cases [53] and general fracture detection [54], AI has been used the most for other purposes. Specifically, it has been used to grade the severity of lumbar spinal stenosis in epidemiological studies (31). These studies involved a considerable volume of highly consistent MRI spine data computed by the system thanks to its algorithm elaboration capabilities. Additionally, this technology has been used to predict the diagnosis and severity of cervical spondylotic myelopathy [27], further demonstrating its great accuracy. In the aforementioned studies, machine-learning, a subset of AI, was employed. This branch of computer science has the potential to revolutionize epidemiologic sciences [55]. The algorithms utilized by this technology can be used in the prediction, classification and clustering of several conditions. Hopkins et al. [27] utilized deep learning, which is defined as a subfield of machine-learning, which employs neural networks (algorithms inspired by the function and structure of the human brain). Neural networks have gained attention in the radiology field, as they can analyze various areas of the human body with great accuracy [56]. Even though AI has the incredible potential to revolutionize orthopaedics, not enough literature discusses its applications in orthopaedic surgery.
When applied to surgery, VR, AR and AI allow the surgeon to operate more consciously and safely. Physicians will work side-by-side with their machine counterparts and completely shift their focus towards more patient-centred health care, yielding unprecedented benefits. It appears clear that extended reality technologies are subject to continuous development, which will likely push them to become the object of a revolution in the medical field.
The limitations of this paper were the high heterogeneity between studies and the lack of data such as sample size in four studies or mean follow up. Moreover, due to the high heterogeneity of the data, it was impossible to perform a meta-analysis. Only English and Italian articles were included, constituting a limitation in the search string. Lastly, the quality of evidence of the studies included was low; therefore, it was impossible to obtain significant conclusions.

5. Conclusions

Extended reality technologies assist the surgeon in visualizing patient data and anatomical landmarks in a 3D setting. Moreover, they can significantly enhance preoperative planning and increase surgical accuracy. Additionally, these technologies allow trainees to experience surgical practice in a cost- and time-efficient manner. The high costs and availability issues, combined with technical limitations of these devices, are bound to be overcome by the growing interest in technology observable nowadays.
Conversely, further studies are needed on AI. Even considering the boundless computing potential and AI applications, not enough articles connecting it to the field of orthopaedic surgery were found in the literature.
With ongoing research and ulterior theoretical applications needing only to be put in practice, extended reality technologies may be considered as an exciting new milestone for orthopaedic surgical practice.

Author Contributions

Data curation, U.G.L. and S.D.S.; Investigation, V.C.; Methodology, G.Z. and S.F.; Resources, G.C. and A.M.; Software, L.G.; Supervision, U.G.L.; Visualization, M.G.D.M. and V.D.; Writing—original draft, S.D.S., G.Z. and G.C.; Writing—review and editing, S.D.S. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data presented in this study are available on request from the corresponding author.

Acknowledgments

This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Verhey, J.T.; Haglin, J.M.; Verhey, E.M.; Hartigan, D.E. Virtual, augmented, and mixed reality applications in orthopedic surgery. Int. J. Med. Robot. 2020, 16, e2067. [Google Scholar] [CrossRef] [PubMed]
  2. Sakai, D.; Joyce, K.; Sugimoto, M.; Horikita, N.; Hiyama, A.; Sato, M.; Devitt, A.; Watanabe, M. Augmented, virtual and mixed reality in spinal surgery: A real-world experience. J. Orthop. Surg. 2020, 28. [Google Scholar] [CrossRef] [PubMed]
  3. Yuk, F.J.; Maragkos, G.A.; Sato, K.; Steinberger, J. Current innovation in virtual and augmented reality in spine surgery. Ann. Transl. Med. 2021, 9, 94. [Google Scholar] [CrossRef] [PubMed]
  4. Brigham, T.J. Reality Check: Basics of Augmented, Virtual, and Mixed Reality. Med. Ref. Serv. Q 2017, 36, 171–178. [Google Scholar] [CrossRef] [PubMed]
  5. Logishetty, K.; Rudran, B.; Cobb, J.P. Virtual reality training improves trainee performance in total hip arthroplasty: A randomized controlled trial. Bone Joint J. 2019, 101-B, 1585–1592. [Google Scholar] [CrossRef] [PubMed]
  6. Madan, S.S.; Pai, D.R. Role of simulation in arthroscopy training. Simul. Healthc. 2014, 9, 127–135. [Google Scholar] [CrossRef] [PubMed]
  7. Blyth, P.; Stott, N.S.; Anderson, I.A. A simulation-based training system for hip fracture fixation for use within the hospital environment. Injury 2007, 38, 1197–1203. [Google Scholar] [CrossRef]
  8. Berton, A.; Longo, U.G.; Candela, V.; Fioravanti, S.; Giannone, L.; Arcangeli, V.; Alciati, V.; Berton, C.; Facchinetti, G.; Marchetti, A.; et al. Virtual Reality, Augmented Reality, Gamification, and Telerehabilitation: Psychological Impact on Orthopedic Patients’ Rehabilitation. J. Clin. Med. 2020, 9, 2567. [Google Scholar] [CrossRef]
  9. Negrillo-Cárdenas, J.; Jiménez-Pérez, J.R.; Feito, F.R. The role of virtual and augmented reality in orthopedic trauma surgery: From diagnosis to rehabilitation. Comput. Methods Programs Biomed. 2020, 191, 105407. [Google Scholar] [CrossRef]
  10. Cannon, W.D.; Eckhoff, D.G.; Garrett, W.E.; Hunter, R.E.; Sweeney, H.J. Report of a group developing a virtual reality simulator for arthroscopic surgery of the knee joint. Clin. Orthop. Relat. Res. 2006, 442, 21–29. [Google Scholar] [CrossRef]
  11. Ogawa, H.; Hasegawa, S.; Tsukada, S.; Matsubara, M. A Pilot Study of Augmented Reality Technology Applied to the Acetabular Cup Placement During Total Hip Arthroplasty. J. Arthroplast. 2018, 33, 1833–1837. [Google Scholar] [CrossRef]
  12. Speicher, M.; Hall, B.D.; Nebeling, M. What is Mixed Reality? In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK, 4–9 May 2019; pp. 1–15. [Google Scholar]
  13. Winkler-Schwartz, A.; Bissonnette, V.; Mirchi, N.; Ponnudurai, N.; Yilmaz, R.; Ledwos, N.; Siyar, S.; Azarnoush, H.; Karlik, B.; Del Maestro, R.F. Artificial Intelligence in Medical Education: Best Practices Using Machine Learning to Assess Surgical Expertise in Virtual Reality Simulation. J. Surg. Educ. 2019, 76, 1681–1690. [Google Scholar] [CrossRef]
  14. Mirchi, N.; Bissonnette, V.; Yilmaz, R.; Ledwos, N.; Winkler-Schwartz, A.; Del Maestro, R.F. The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine. PLoS ONE 2020, 15, e0229596. [Google Scholar] [CrossRef] [Green Version]
  15. Ramesh, A.N.; Kambhampati, C.; Monson, J.R.; Drew, P.J. Artificial intelligence in medicine. Ann. R Coll. Surg. Engl. 2004, 86, 334–338. [Google Scholar] [CrossRef] [Green Version]
  16. Panchmatia, J.R.; Visenio, M.R.; Panch, T. The role of artificial intelligence in orthopaedic surgery. Br. J. Hosp. Med. (Lond) 2018, 79, 676–681. [Google Scholar] [CrossRef]
  17. Liberati, A.; Altman, D.G.; Tetzlaff, J.; Mulrow, C.; Gotzsche, P.C.; Ioannidis, J.P.; Clarke, M.; Devereaux, P.J.; Kleijnen, J.; Moher, D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. PLoS Med. 2009, 6, e1000100. [Google Scholar] [CrossRef]
  18. Van der Mierden, S.; Tsaioun, K.; Bleich, A.; Leenaars, C.H.C. Software tools for literature screening in systematic reviews in biomedical research. ALTEX 2019, 36, 508–517. [Google Scholar] [CrossRef]
  19. Sterne, J.A.C.; Savović, J.; Page, M.J.; Elbers, R.G.; Blencowe, N.S.; Boutron, I.; Cates, C.J.; Cheng, H.Y.; Corbett, M.S.; Eldridge, S.M.; et al. RoB 2: A revised tool for assessing risk of bias in randomised trials. BMJ 2019, 366, l4898. [Google Scholar] [CrossRef] [Green Version]
  20. Slim, K.; Nini, E.; Forestier, D.; Kwiatkowski, F.; Panis, Y.; Chipponi, J. Methodological index for non-randomized studies (minors): Development and validation of a new instrument. ANZ J. Surg. 2003, 73, 712–716. [Google Scholar] [CrossRef]
  21. LeBlanc, J.; Hutchison, C.; Hu, Y.; Donnon, T. A comparison of orthopaedic resident performance on surgical fixation of an ulnar fracture using virtual reality and synthetic models. J. Bone Joint Surg. Am. 2013, 95, e60. [Google Scholar] [CrossRef]
  22. Hooper, J.; Tsiridis, E.; Feng, J.E.; Schwarzkopf, R.; Waren, D.; Long, W.J.; Poultsides, L.; Macaulay, W.; Consortium, N.V.R. Virtual Reality Simulation Facilitates Resident Training in Total Hip Arthroplasty: A Randomized Controlled Trial. J. Arthroplast. 2019, 34, 2278–2283. [Google Scholar] [CrossRef]
  23. Ma, L.; Zhao, Z.; Chen, F.; Zhang, B.; Fu, L.; Liao, H. Augmented reality surgical navigation with ultrasound-assisted registration for pedicle screw placement: A pilot study. Int. J. Comput. Assist. Radiol. Surg. 2017, 12, 2205–2215. [Google Scholar] [CrossRef]
  24. Tsukada, S.; Ogawa, H.; Nishino, M.; Kurosaka, K.; Hirasawa, N. Augmented reality-based navigation system applied to tibial bone resection in total knee arthroplasty. J. Exp. Orthop. 2019, 6, 44. [Google Scholar] [CrossRef]
  25. Carl, B.; Bopp, M.; Saß, B.; Voellger, B.; Nimsky, C. Implementation of augmented reality support in spine surgery. Eur. Spine J. 2019, 28, 1697–1711. [Google Scholar] [CrossRef]
  26. Fotouhi, J.; Unberath, M.; Song, T.; Hajek, J.; Lee, S.C.; Bier, B.; Maier, A.; Osgood, G.; Armand, M.; Navab, N. Co-localized augmented human and X-ray observers in collaborative surgical ecosystem. Int. J. Comput. Assist. Radiol. Surg. 2019, 14, 1553–1563. [Google Scholar] [CrossRef]
  27. Hopkins, B.S.; Weber, K.A.; Kesavabhotla, K.; Paliwal, M.; Cantrell, D.R.; Smith, Z.A. Machine Learning for the Prediction of Cervical Spondylotic Myelopathy: A Post Hoc Pilot Study of 28 Participants. World Neurosurg. 2019, 127, e436–e442. [Google Scholar] [CrossRef]
  28. Teatini, A.; Kumar, R.P.; Elle, O.J.; Wiig, O. Mixed reality as a novel tool for diagnostic and surgical navigation in orthopaedics. Int. J. Comput. Assist. Radiol. Surg. 2021, 16, 407–414. [Google Scholar] [CrossRef]
  29. Gu, Y.; Yao, Q.; Xu, Y.; Zhang, H.; Wei, P.; Wang, L. A Clinical Application Study of Mixed Reality Technology Assisted Lumbar Pedicle Screws Implantation. Med. Sci. Monit. 2020, 26, e924982. [Google Scholar] [CrossRef]
  30. Fotouhi, J.; Alexander, C.P.; Unberath, M.; Taylor, G.; Lee, S.C.; Fuerst, B.; Johnson, A.; Osgood, G.; Taylor, R.H.; Khanuja, H.; et al. Plan in 2-D, execute in 3-D: An augmented reality solution for cup placement in total hip arthroplasty. J. Med. Imaging 2018, 5, 021205. [Google Scholar] [CrossRef] [Green Version]
  31. Hu, M.H.; Chiang, C.C.; Wang, M.L.; Wu, N.Y.; Lee, P.Y. Clinical feasibility of the augmented reality computer-assisted spine surgery system for percutaneous vertebroplasty. Eur. Spine J. 2020, 29, 1590–1596. [Google Scholar] [CrossRef]
  32. Chen, Y.; Jia, X.; Qiang, M.; Zhang, K.; Chen, S. Computer-Assisted Virtual Surgical Technology Versus Three-Dimensional Printing Technology in Preoperative Planning for Displaced Three and Four-Part Fractures of the Proximal End of the Humerus. J. Bone Joint Surg. Am. 2018, 100, 1960–1968. [Google Scholar] [CrossRef] [PubMed]
  33. Ishimoto, Y.; Jamaludin, A.; Cooper, C.; Walker-Bone, K.; Yamada, H.; Hashizume, H.; Oka, H.; Tanaka, S.; Yoshimura, N.; Yoshida, M.; et al. Could automated machine-learned MRI grading aid epidemiological studies of lumbar spinal stenosis? Validation within the Wakayama spine study. BMC Musculoskelet. Disord. 2020, 21, 158. [Google Scholar] [CrossRef] [PubMed]
  34. Ogawa, H.; Kurosaka, K.; Sato, A.; Hirasawa, N.; Matsubara, M.; Tsukada, S. Does an Augmented Reality-based Portable Navigation System Improve the Accuracy of Acetabular Component Orientation During THA? A Randomized Controlled Trial. Clin. Orthop. Relat. Res. 2020, 478, 935–943. [Google Scholar] [CrossRef] [PubMed]
  35. Elmi-Terander, A.; Burström, G.; Nachabé, R.; Fagerlund, M.; Ståhl, F.; Charalampidis, A.; Edström, E.; Gerdhem, P. Augmented reality navigation with intraoperative 3D imaging vs fluoroscopy-assisted free-hand surgery for spine fixation surgery: A matched-control study comparing accuracy. Sci. Rep. 2020, 10, 707. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  36. Yari, S.S.; Jandhyala, C.K.; Sharareh, B.; Athiviraham, A.; Shybut, T.B. Efficacy of a Virtual Arthroscopic Simulator for Orthopaedic Surgery Residents by Year in Training. Orthop. J. Sports Med. 2018, 6. [Google Scholar] [CrossRef] [Green Version]
  37. Edström, E.; Burström, G.; Persson, O.; Charalampidis, A.; Nachabe, R.; Gerdhem, P.; Elmi-Terander, A. Does Augmented Reality Navigation Increase Pedicle Screw Density Compared to Free-Hand Technique in Deformity Surgery? Single Surgeon Case Series of 44 Patients. Spine (Phila Pa 1976) 2020, 45, E1085–E1090. [Google Scholar] [CrossRef] [PubMed]
  38. Zheng, C.; Li, J.; Zeng, G.; Ye, W.; Sun, J.; Hong, J.; Li, C. Development of a Virtual Reality Preoperative Planning System for Postlateral Endoscopic Lumbar Discectomy Surgery and Its Clinical Application. World Neurosurg. 2019, 123, e1–e8. [Google Scholar] [CrossRef]
  39. Ponce, B.A.; Jennings, J.K.; Clay, T.B.; May, M.B.; Huisingh, C.; Sheppard, E.D. Telementoring: Use of augmented reality in orthopaedic education: AAOS exhibit selection. J. Bone Joint Surg. Am. 2014, 96, e84. [Google Scholar] [CrossRef]
  40. Vaughan, N.; Dubey, V.N.; Wainwright, T.W.; Middleton, R.G. A review of virtual reality based training simulators for orthopaedic surgery. Med. Eng. Phys. 2016, 38, 59–71. [Google Scholar] [CrossRef] [Green Version]
  41. Ponce, B.A.; Menendez, M.E.; Oladeji, L.O.; Fryberger, C.T.; Dantuluri, P.K. Emerging technology in surgical education: Combining real-time augmented reality and wearable computing devices. Orthopedics 2014, 37, 751–757. [Google Scholar] [CrossRef] [Green Version]
  42. Brouwers, L.; Pull Ter Gunne, A.F.; de Jongh, M.A.; Maal, T.J.J.; Vreeken, R.; van der Heijden, F.H.W.M.; Leenen, L.P.H.; Spanjersberg, W.R.; van Helden, S.H.; Verbeek, D.O.; et al. What is the value of 3D virtual reality in understanding acetabular fractures? Eur. J. Orthop. Surg. Traumatol. 2020, 30, 109–116. [Google Scholar] [CrossRef]
  43. Pauly, O.; Diotte, B.; Fallavollita, P.; Weidert, S.; Euler, E.; Navab, N. Machine learning-based augmented reality for improved surgical scene understanding. Comput. Med. Imaging Graph. 2015, 41, 55–60. [Google Scholar] [CrossRef]
  44. Guo, M.L.; Yue, S.T.; Wang, J.Y.; Cui, H.X. Comparative study on the clinical application of mixed reality technology leading micro-invasive intervertebral foramen puncture location and blind puncture location. Pak. J. Med. Sci 2020, 36, 559–564. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  45. Kriechling, P.; Roner, S.; Liebmann, F.; Casari, F.; Fürnstahl, P.; Wieser, K. Augmented reality for base plate component placement in reverse total shoulder arthroplasty: A feasibility study. Arch. Orthop. Trauma Surg. 2020. [Google Scholar] [CrossRef]
  46. Creighton, F.X.; Unberath, M.; Song, T.; Zhao, Z.; Armand, M.; Carey, J. Early Feasibility Studies of Augmented Reality Navigation for Lateral Skull Base Surgery. Otol. Neurotol. 2020, 41, 883–888. [Google Scholar] [CrossRef] [PubMed]
  47. Liu, H.; Auvinet, E.; Giles, J.; Rodriguez, Y.; Baena, F. Augmented Reality Based Navigation for Computer Assisted Hip Resurfacing: A Proof of Concept Study. Ann. Biomed. Eng. 2018, 46, 1595–1605. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  48. Elmi-Terander, A.; Burström, G.; Nachabe, R.; Skulason, H.; Pedersen, K.; Fagerlund, M.; Ståhl, F.; Charalampidis, A.; Söderman, M.; Holmin, S.; et al. Pedicle Screw Placement Using Augmented Reality Surgical Navigation with Intraoperative 3D Imaging: A First In-Human Prospective Cohort Study. Spine (Phila Pa 1976) 2019, 44, 517–525. [Google Scholar] [CrossRef]
  49. Ma, L.; Zhao, Z.; Zhang, B.; Jiang, W.; Fu, L.; Zhang, X.; Liao, H. Three-dimensional augmented reality surgical navigation with hybrid optical and electromagnetic tracking for distal intramedullary nail interlocking. Int. J. Med. Robot. 2018, 14, e1909. [Google Scholar] [CrossRef]
  50. Zhou, Z.; Hu, S.; Zhao, Y.Z.; Zhu, Y.J.; Wang, C.F.; Gu, X.; Fan, G.X.; He, S.S. Feasibility of Virtual Reality Combined with Isocentric Navigation in Transforaminal Percutaneous Endoscopic Discectomy: A Cadaver Study. Orthop. Surg. 2019, 11, 493–499. [Google Scholar] [CrossRef]
  51. Cho, H.S.; Park, Y.K.; Gupta, S.; Yoon, C.; Han, I.; Kim, H.S.; Choi, H.; Hong, J. Augmented reality in bone tumour resection: An experimental study. Bone Joint Res. 2017, 6, 137–143. [Google Scholar] [CrossRef] [Green Version]
  52. Andress, S.; Johnson, A.; Unberath, M.; Winkler, A.F.; Yu, K.; Fotouhi, J.; Weidert, S.; Osgood, G.; Navab, N. On-the-fly augmented reality for orthopedic surgery using a multimodal fiducial. J. Med. Imaging 2018, 5, 021209. [Google Scholar] [CrossRef]
  53. Lee, C.; Jang, J.; Lee, S.; Kim, Y.S.; Jo, H.J.; Kim, Y. Classification of femur fracture in pelvic X-ray images using meta-learned deep neural network. Sci. Rep. 2020, 10, 13694. [Google Scholar] [CrossRef]
  54. Langerhuizen, D.W.G.; Janssen, S.J.; Mallee, W.H.; van den Bekerom, M.P.J.; Ring, D.; Kerkhoffs, G.M.M.J.; Jaarsma, R.L.; Doornberg, J.N. What Are the Applications and Limitations of Artificial Intelligence for Fracture Detection and Classification in Orthopaedic Trauma Imaging? A Systematic Review. Clin. Orthop. Relat. Res. 2019, 477, 2482–2491. [Google Scholar] [CrossRef]
  55. Bi, Q.; Goodman, K.E.; Kaminsky, J.; Lessler, J. What is Machine Learning? A Primer for the Epidemiologist. Am. J. Epidemiol. 2019, 188, 2222–2239. [Google Scholar] [CrossRef]
  56. Soffer, S.; Ben-Cohen, A.; Shimon, O.; Amitai, M.M.; Greenspan, H.; Klang, E. Convolutional Neural Networks for Radiologic Images: A Radiologist’s Guide. Radiology 2019, 290, 590–606. [Google Scholar] [CrossRef]
Applsci 11 03253 g001 550
Figure 1. Study selection process and screening according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) flow chart [17].
Figure 1. Study selection process and screening according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) flow chart [17].
Applsci 11 03253 g001
Table
Table 1. Cochrane Risk of Bias Tool for assessing the risk of bias in randomized trials.
Table 1. Cochrane Risk of Bias Tool for assessing the risk of bias in randomized trials.
AuthorSequence GenerationAllocation ConcealmentBlinding of Participants and PersonnelBlinding of Outcome AssessmentIncomplete Outcome DataSelective Outcome ReportingOther Sources of BiasOverall Score
Hooper, 201900000000
LeBlanc, 201300020002
Logishetty, 201900110002
Table
Table 2. MINORS tool for assessing the risk of bias in non-randomized trials.
Table 2. MINORS tool for assessing the risk of bias in non-randomized trials.
AuthorsClearly Stated AimInclusion of Consecutive PatientsProspective Data CollectionEndpoints Appropriate to Study AimUnbiased Assessment of Study EndpointFollow-Up Period Appropriate to Study Aim<5% Lost to Follow-UpProspective Calculation of Study SizeAdequate Control GroupContemporary GroupsBaseline Equivalence of GroupsAdequate Statistical AnalysesTotal Score (…/24)
Carl, 20182202000200008
Chen, 201822220222022220
Edstrom, 202022020002220012
Fotohui, 20182002000000026
Fotouhi, 20192NA02220NA220012
Gu, 202022021122022016
Hopkins, 201922020222000010
Hu, 202022020222222220
Ishimoto, 202022220002220012
Ma, 20172NA02000NA00026
Ogawa, 201822022222222020
Ogawa, 201922022222222222
Ponce, 201420222222020218
Shahram, 201822222002222220
Teatini, 20212NA02200NA00006
Terander, 202022020002222216
Tsukada, 201922020002222216
Zheng, 201822020222222218
Table
Table 3. Study characteristics.
Table 3. Study characteristics.
Author and YearCountryStudy Design, LOESamplePurposeHardwareProcedureConclusion
Carl, 2019 [25]GermanyPS, II10IntraoperativeARSpine surgeryAR greatly supports the surgeon in understanding the 3D anatomy, thereby facilitating surgery
Chen, 2018 [32]ChinaPComS, II131PreoperativeVRFracture reduction The clinical outcomes in both the virtual surgical and 3D printing groups were better than those in the conventional group. VR is more convenient and efficient
Edstrom, 2020 [37]SwedenRCohS, III44IntraoperativeARPedicle screw placementAR enables the surgeon to minimize the use of hooks in deformity surgery without prolonging the surgical time
Fotouhi, 2018 [30]UKRComS, III4 surgeonsIntraoperativeARTotal hip arthroplastyAR simplifies and allows accurate implantation of the acetabular cup
Fotouhi, 2019 [26]USARComS, IIINAIntraoperativeAR vs. X-RayPercutaneous fixationAR solution provides a shared augmented experience between the human and X-ray viewer
Gu 2020 [29]ChinaPComS, II50IntraoperativeAR (MR)Lumbar pedicle screws placementThe safety of spinal surgery and implantation accuracy of pedicle screw fixation system could be increased by AR
Hooper, 2019 [22]United StatesRCT, I14 residentsTraningVRTotal hip arthroplastyVR-simulation improves resident surgical skills but has no significant effect on medical knowledge
Hopkins, 2019 [27]USAPCCS, II28 PreoperativeAIPrediction of CSM diagnosis and severityAI provides a promising method for prediction, diagnosis, and even prognosis in patients affected by CSM.
Hu, 2020 [31]TaiwanPCCS, II18 IntraoperativeARPercutaneous vertebroplastyThe guidance of the AR system provided a more accurate bony entry point with reduced operative time and unnecessary radiation exposure.
Ishimoto, 2020 [33]UKRCS, III971 PreoperativeAILumbar spinal stenosisAn automated system can learn with an excellent performance against the reference standard
LeBlanc, 2013 [21]CanadaRCT, I22 residentsTrainingVRUlnar fixationThe procedural measures used to assess resident performance demonstrated good reliability and validity, and both the Sawbones and the virtual simulator showed evidence of construct validity.
Logishetty, 2019 [5]United KingdomRCT, I24 traineesTrainingVRTotal hip arthroplastyVR training advanced trainees further up the learning curve, enabling exact component orientation and more efficient surgery. VR could augment traditional surgical training to improve how surgeons learn complex open procedures
Ma, 2017 [23]ChinaPS, IINAIntraoperativeARIntramedullary nail fixation The AR-guided distal interlocking method is feasible and has many potential applications in clinic after further evaluation.
Ogawa, 2018 [11]JapanPS, II54 IntraoperativeARTotal hip arthroplastyAR system provided more accurate information than the conventional method
Ogawa, 2020 [34]JapanPComS, II46 IntraoperativeARTotal hip arthroplastyAR system did not show better results compared to the traditional group
Ponce, 2014 [41]USAPS, II15IntraoperativeVR—Virtual interactive presenceArthroscopic shoulder surgeryVIP technology was efficient, safe, and effective as a teaching tool
Yari, 2018 [36]USAPOS, II18 residentsTrainingVR—ArthroS simulatorKnee and shoulder arthroscopyResidents training on a virtual arthroscopic simulator made significant improvements in both knee and shoulder arthroscopic surgery skills
Teatini, 2021 [28]SwedenPCohS, II8 surgeonsIntraoperativeAR (MR)Visualization of joint and skeletal deformitiesAR improve diagnostic accuracy and allow for safer and more precise surgeries, as well as provide for better learning conditions for orthopaedic surgeons in training
Terander, 2020 [35]SwedenPOS, II20 IntraoperativeARSpine surgeryStatistically higher screw placement accuracy compared to the free-hand technique in a cohort of spinal deformity cases
Tsukada, 2019 [24]JapanPS, II10 IntraoperativeARTotal knee arthroplasty
AR system provides reliable accuracy total knee arthroplasty.
Zheng, 2018 [38]ChinaPCohS, II30 IntraoperativeVRDiscectomyStatistically higher accuracy was reported in the VR group
AR: augmented reality; CSM: Cervical spondylotic myelopathy; MR: mixed reality; NA: not applicable; PCCS: Prospective case-control study; PCohS: Prospective cohort study; PComS: Prospective comparative study; POS: Prospective observational study; PS: Pilot study; RCohS: Retrospective cohort study; RComS: Retrospective Comparative Study; RCT: Randomized Controlled Trial; VR: Virtual reality.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Longo, U.G.; De Salvatore, S.; Candela, V.; Zollo, G.; Calabrese, G.; Fioravanti, S.; Giannone, L.; Marchetti, A.; De Marinis, M.G.; Denaro, V. Augmented Reality, Virtual Reality and Artificial Intelligence in Orthopedic Surgery: A Systematic Review. Appl. Sci. 2021, 11, 3253. https://doi.org/10.3390/app11073253

AMA Style

Longo UG, De Salvatore S, Candela V, Zollo G, Calabrese G, Fioravanti S, Giannone L, Marchetti A, De Marinis MG, Denaro V. Augmented Reality, Virtual Reality and Artificial Intelligence in Orthopedic Surgery: A Systematic Review. Applied Sciences. 2021; 11(7):3253. https://doi.org/10.3390/app11073253

Chicago/Turabian Style

Longo, Umile Giuseppe, Sergio De Salvatore, Vincenzo Candela, Giuliano Zollo, Giovanni Calabrese, Sara Fioravanti, Lucia Giannone, Anna Marchetti, Maria Grazia De Marinis, and Vincenzo Denaro. 2021. "Augmented Reality, Virtual Reality and Artificial Intelligence in Orthopedic Surgery: A Systematic Review" Applied Sciences 11, no. 7: 3253. https://doi.org/10.3390/app11073253

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop