Single Evaluation of Use of a Mixed Reality Headset for Intra-Procedural Image-Guidance during a Mock Laparoscopic Myomectomy on an Ex-Vivo Fibroid Model

Abstract

Uterine fibroids represent the highest prevalence of benign tumors in women, with reports ranging from 4.5% to 68.6%, with a significant bias towards African American women. For uterine fibroids, a significant decision is determining whether fibroids can be successfully removed using minimally invasive (MI) techniques or their removal requires open surgery. Currently, the standard-of-care for intra-procedural visualization for myomectomies is ultrasound, which has low image quality and requires a specially trained assistant. Currently, the state-of-the-art is to obtain a pre-procedural MRI scan of the patient, which can be used for diagnosis and pre-procedural planning. Although proven incredibly useful pre-procedurally, MRI scans are not often used intra-procedurally due to the inconvenient visualization as 2D slices, which are seen on 2D monitors that do not intuitively convey the depth or orientation of the fibroids, as needed to effectively perform myomectomies. To address this limitation, herein, we present the use of a mixed reality headset (i.e., Microsoft HoloLens 2), as a tool for intra-procedural image-guidance during a mock myomectomy of an ex vivo animal uterus. In this work, we created a patient-specific holographic rendering by performing image segmentation of an MRI scan of a custom-made uterine fibroid animal model. A physician qualitatively assessed the usefulness of the renderings for fibroid localization, as compared to the same visualization on a 2D monitor. In conclusion, the use of mixed reality as an intra-procedural image guidance tool for myomectomies was perceived as a better visualization technique that could lead to improvements in MI approaches and make them accessible to patients from lower socioeconomic populations.

1. Introduction

Uterine fibroids represent the highest prevalence of benign tumors in women, with reports ranging anywhere from 4.5% to 68.6%, with a significant bias towards African American women [1]. It is estimated that the economic burden on the healthcare system from symptomatic women with uterine fibroids is nearly $34 billion [2]. For women with uterine fibroids, a major decision is determining whether fibroids can be successfully removed using minimally invasive (MI) techniques or their removal will require open surgery. Imaging variables that determine who is a candidate for MI surgery depends on the number of fibroids and locations of the myomas relative to the uterine wall. Interpreting these studies can be difficult as fibroids can lay on top of each other and present in any layer of the uterus. If a MI procedure is incorrectly selected, it can be significantly more difficult, if not impossible, thus increasing the risk of bleeding, and potentially aborting the MI procedure altogether, requiring a subsequent open surgery. It has been well documented that women in lower socio-economic status, particularly African American women, are less likely to be referred for minimally invasive procedures [3,4,5,6] even though there is universal insurance coverage for them [7,8].

The success of laparoscopic myomectomies over the last few decades has significantly reduced the mortality of patients. In addition, it is associated with lower intensity of postoperative pain, shorter hospitalization, less blood loss, less postoperative complications, and faster recovery. Yet, the recurrence rate is higher than a classical myomectomy, particularly in the case of small fibroids (e.g., <1 cm), and/or multiple myomas [9], which are more likely to be left in place after a laparoscopic myomectomy [10,11,12]. In addition, the 5-year recurrence rate after laparoscopic myomectomy reaches more than 50% [13]. Thus, tools to better remove all myomas to mitigate recurrence and to further improve procedural outcomes are desired. One of several options is to improve intra-procedural visualization, which currently the standard-of-care is the use of ultrasound. However, this imaging modality is often not used since it required a specialized technician and often provides difficult to interpret images.

Currently the state-of-the-art visualization technology for fibroids (also known as myomas) is a pre-procedural MRI, which provides a holistic 3D view of the entire women’s anatomy. This allows radiologists and gynecologists to more accurately determine the number and exact location of each myoma, relative to other anatomic structures (e.g., uterine wall, endometrium, cervix). However, the relatively poor spatial resolution MRI (on the order of millimeters) does still miss some small fibroids (<1 cm). This is not considered a significant disadvantage since the clinical burden of these small fibroids are not considered relevant and often would be left in to avoid other complications during the procedure.

Although proven significantly beneficial for determining the best treatment option, the use of MRI for intra-procedural guidance is very limited. Currently, to the best of our knowledge, there are no known clinical myomectomy studies on the use of MRI data intra-procedurally. However, it is known that currently gynecologists can leverage this pre-procedural MRI scan in two ways: (i) it can be directly viewed on a 2D monitor as 2D slices in a Digital Imaging Communication in Medicine (DICOM) Viewer, or (ii) it can be viewed from paper printouts that are hung next to physician during the procedure, shown in Figure 1. The goal for both methods is to convey the relative location of each fibroid to more easily allow the physician to find its location and determine where to make an incision to find it. The benefit of the first method is that it provides access to whole dataset where each slice can be toggled in real-time. However, this also requires someone to perform this manual toggling, which will take up procedural time to find each fibroid. The benefit of the second method is that each printout already contains the desired slice, however, there is a chance that not all the information is conveyed since there is a practical limit for how many printouts can be hung next to each other.

Visualization technology in the operating room has continued to improve both surgical workflow and training practices. It has provided an opportunity for multimodal integration, enabling surgeons to maximize all the relevant information during the procedure. These visualization modalities include digital stereomicroscopy, virtual reality (VR), augmented reality (AR), and mixed reality (MR). Unlike VR, which only offers an effective platform for pre-procedural planning, AR allows a surgeon to visualize the live surgical field with an enriched virtual data overlay, including different sets of data deriving from different imaging modalities, such as tractography, or angiography, or ultrasound imaging [5,14]. Yet, surgeons often cite the lack of tactile feedback and intuitive interaction as a limitation of AR technology. On the other hand, mixed reality holds the promise of providing digital enhancement to our vision while ensuring a fully immersed environment enabling the user to have an intuitive interaction with the overlaid digital contents [15,16]. Despite these advantages, there has been little adoption of mixed reality into clinical interventions due to limitations in hardware, lack of streamlined methods to generate 3D data to render in the mixed reality environment, and an inability to provide real-time updates to the model due to events occurring during the procedure.

Thus far, there are few publications on the use of AR systems in gynecologic surgery. Initially, Collins et al. [17] presented the first AR-based image guidance system for assisted myoma localization in uterine laparosurgery. This study leveraged the reference and the update registration methods to achieve a nonrigid registration of the uterus and myomas from the MRI to the laparoscopic video. Bourdel et al. [18] evaluated the potential benefits of an AR system for myomectomy and reported that an AR system could enhance the accuracy of the localization of fibroids by a factor of about twenty. In two follow-up publications, Bourdel et al. [19,20] developed an AR system for the individual cases and demonstrated an improved laparoscopic myomectomy and adenomyomectomy upon implementing AR during surgery.

In another study, François et al. [21] proposed object-class occluding contour detection (OC2D) framework to register preoperative 3D MRI model to intraoperative laparoscopy 2D images, to achieve augmented reality in laparoscopy. A user study was carried out to evaluate the impact of OC2D against manually marked occluding contours in augmented laparoscopy. The OC2D reduced 3 min and 53 s in surgeon time without sacrificing registration accuracy, indicating that fully automatic augmented laparoscopy is feasible. It is clear that such works have shed light upon the success of adapting extended reality systems (VR/AR/MR) in gynecologic practices and given us an improved understanding of its significance. However, despite these, there is a paucity of literature detailing the technical development and the refinement which could enable the engineers and developers to build and test the environment for collaborating physicians. To this end, herein, we present the use of a mixed reality headset, Microsoft HoloLens2, as an intra-procedural image-guidance tool during a mock myomectomy of an ex vivo animal model that was performed by a gynecologic surgeon at New York Presbyterian Hospital’s Skills Acquisition and Innovation Laboratory (SAIL). We discuss the manual segmentation of uterine fibroids and overview the file preparation from a MRI scan. Additionally, we tested the use of 3D rendering projected on a 2D monitor and qualitatively compared it with the mixed reality hologram.

2. Materials and Methods

2.1. Animal Model

The uterus model was custom-made by the company Gynesim with various bovine muscle layers to simulate the different textures of the uterus and myoma, shown in Figure 2A. We selected specific locations of the myomas to evaluate if they could be mapped in each layer of the uterus from endometrium to exophytic. The specific locations were as follows: 15 cm uterus with one 3 cm type zero myoma, one 4 cm 50% cavitary, three 4 cm intramural myomas (2 of them in same plane), one 5 cm fibroid (in separate plane), one 5 cm subserosal, and one 4 cm exophytic myoma.

2.2. MRI Scan and Post-Preparation

The ex vivo animal model was taken for MRI scans (isotropic (0.5 × 0.5 × 0.5 mm) T1, FATSAT T1 & T2 images) to allow for 3D visualization as would similarly be done in a leading clinical facility (Figure 2B,C). To process the anatomical structures of the uterus, we used a 3D image processing software (Materialize Mimics Research software 21.0) to import the MRI scan as a DICOM data file, depicted in Figure 2D. In Mimics, the specific thresholds are set to segment the fibroids and uterus wall, enabling a 3D representation of the fibroid in one mask, while maintaining all the relative positions. Next, the 3D segmentation was saved as a STL file. As can be seen from Figure 2E, to trim all the triangles, artifacts, and other elements that are not necessary for the model, we used Geomagic Wrap (3D Systems Geomagic Corporation, Morrisville, NC, USA).

2.3. Mixed Reality Device and Environment

The mixed reality scene is designed to provide a 3D rendering of the fibroids so the user will have an intuitive interpretation for the location of the fibroids within the anatomical structure, shown in Figure 2F. The 3D hologram was displayed via the 3D viewer app of a Microsoft HoloLens 2 headset during a mock laparoscopic myomectomy of the ex vivo animal model. To build our MR scene and display the hologram, initially we used SolidWorks assembly, which allowed us to set the color and texture of each part, as desired. This file, however, will not directly maintain its colors and texture properties in the Viewer app of HoloLens 2, so it was then uploaded to a free online-viewer (https://www.creators3d.com/online-viewer) accessed on 1 June 2021, and resaved as a GLB file, which was properly displayed in the Viewer app. This enabled us to have an intuitive interaction with the 3D assets and facilitated our workflow throughout development. The 3D Viewer app allows placement of the rendering anywhere in the room using a hand gesture to pinch and move the hologram from its base or scale it from its corners; the hologram can also be rotated along a single axis. There were no voice commands available for this experience. The mixed reality headset was used during the removal of the first three fibroids.

2.4. 3D Rendering on 2D Screen

After the segmentation in Mimics, postprocessing in Geomagic, and the assembly in SolidWorks, the 3D objects (fibroids, uterus wall) were exported as a 3D PDF file. Then, the PDF file illustrating 3D rendering was projected on the second monitor during myomectomy of the last three fibroids.

3. Results

As shown in Figure 3, the animal model was placed inside a laparoscopic trainer, which is equipped with an endoscopic camera, scissors, and grasper. Prior to the mock surgery, the gynecologist went through eye tracking user calibration in the HoloLens 2 for which the physician had to look at a set of holographic targets, shown in Figure 3A. This allowed the headset to adjust the scene for a more comfortable and higher quality 3D visualization experience and to ensure accurate eye tracking. After launching the hologram, the physician oriented and aligned the hologram according to the live laparoscopic video, depicted in Figure 3B, leveraging the mixed reality feature of real-world spatial positioning. During the mock procedure (see Video S1), a vertical incision was made through the myometrium using a monopolar laparoscopic scissor. Pearly white tissue was visualized, which indicated the target fibroid. Using a grasper the fibroid was extracted in its entirety. The hologram was then repositioned to match the animal model for the subsequent myomas. Furthermore, to qualitatively assess the usefulness of the mixed reality headset in comparison with 3D renderings on a 2D monitor, the gynecologist was asked to only rely on the second monitor for the last three fibroids, illustrated in Figure 3F.

4. Discussion

This system provided the benefit of a mixed reality display that was true to the animal model in terms of size and positioning, allowing for effective visualization and understanding of the location of fibroids. This capability has great potential benefit for reducing procedure time and minimizing the number of incisions made. However, for long surgeries, this particular headset (i.e., HoloLens 2) may be burdensome given its weight (566 g). This inconvenience may prevent surgeons from using the Hololens, in its current form, directly in procedures. However, as newer and lighter headsets are being developed (e.g., NuEyes, RealWear, Magic Leap 2), the same file processing workflow can be used to visualize the renderings on more comfortable headsets.

It should be noted that there is a learning curve to the use of the headset and that physicians will need ample training. Thus, creation of training tutorials or demos should be considered to facilitate this learning experience. The learning curve is associated with basic commands for the headset, but also with specific actions for the required task, such as orienting the hologram. These insights are provided by the co-author Tamatha Fenster, who has a subspeciality in Minimally Invasive Gynecologic Surgery. She currently serves as the Director of Biotechnology and Innovation at the Fibroid and Adenomyosis Center, where her role is to create innovative ways to improve women’s health care, with a specific emphasis on myomas. She currently holds two utility patents, and has two medical devices available to the public. She is also an Assistant Professor of Obstetrics & Gynecology at Weill Cornell Medicine and New York Presbyterian. She was the first MIGS fellowship trained surgeon at Weill Cornell Medical Center, and has had a flourishing surgical fibroid practice for over 10 years, during which she has performed over a thousand fibroid surgeries. She is currently, one of the highest volume MIGS surgeons at Weill Cornell Medicine.

The 3D rendering on a secondary monitor provides some immediate solutions for challenges of the headset, but the benefits of the real life feel of the hologram was lost, which may compromise on desired clinical outcomes such as shorter procedure times. The screen still depicted the number of myomas, but depth was not intuitively understood, requiring rotation of the rendering by another person, needing an additional screen, and extra space for the computer to sit.

Very recently, Condino et al. [22] proposed a hybrid simulation and planning platform harnessing Microsoft HoloLens for Cryosurgery. In this work, a qualitative study was carried out to preliminarily evaluate the proposed system for retrospective simulation of two surgical cases. All participants confirmed that using the desktop application and the hybrid simulator is easy and quick to learn, allowing a clear evaluation of the lesion margins and their distance from the adjacent anatomical structures, making the surgical planning easier than with the traditional method. Yet, they noted a prospective, randomized, single-center clinical trial is needed to compare cryotherapy procedures’ short- and long-term outcomes.

Our work serves as a single evaluation of use of a mixed reality headset for intra-procedural image-guidance during a mock laparoscopic myomectomy on an ex-vivo fibroid model. However, since this article is a communication, it has several limitations in its current stage, and we believe these limitations will be the seed for future developments for both our lab and others. These limitations include: (1) Lack of quantitative assessment. Systematically evaluating the improved timing and accuracy of fibroid detection and removal would better support the benefits of this system for myomectomies. (2) Lack of broad physician evaluation and feedback. A key factor in the iteration of the user interface and user experience is to get broad feedback from physicians, but this work was only limited to one since this work is primarily focused on the generation of the mixed reality environment from an MRI scan. (3) Lack of clinical evaluation. It is vital to confirm claims in actual clinical cases since our ex vivo animal model can’t recapitulate the complexity of an actual surgery. Therefore, the significance of MR-guidance in laparoscopic gynecology surgery should be evaluated in a clinical trial, comparing it to conventional guidance modalities.

Despite these limitations, we believe the workflow in this communication can be useful for other groups to develop other MR experiences to improve intra-procedural guidance of myomectomies. Furthermore, this workflow can be sued for any surgical procedure that a pre-operative MRI is acquired. We also believe this this workflow may directly assist in both patient consultations and pre-procedural planning, by providing a more intuitive visualization compared to traditional 2D DICOM viewers.

5. Conclusions

This work demonstrated that a mixed reality environment can potentially enable gynecologists to visualize and localize the relative positions of tumor fibroids for pre- and intra-procedural planning. This study used Microsoft HoloLens 2 as an intra-procedural image-guidance tool during a mock myomectomy of an ex vivo animal model. We discussed the manual segmentation of uterine fibroids and overviewed the file preparation from a MRI scan. Furthermore, we tested the use of the same 3D rendering displayed on a 2D monitor to qualitatively compare its effectiveness to guide fibroid localization compared to the mixed reality immersive experience. The development of this guidance tool will potentially lower the learning curve for these minimally invasive procedures such that these lower-risk procedures can be utilized for woman with high numbers of myomas, more complicated positions, and potentially to serve those from lower socioeconomic populations, where MIS are less available.