An Uncalibrated Image-Based Visual Servo Strategy for Robust Navigation in Autonomous Intravitreal Injection

Abstract

Autonomous intravitreal injection in ophthalmology is a challenging surgical task as accurate depth measurement is difficult due to the individual differences in the patient’s eye and the intricate light reflection or refraction of the eyeball, often requiring the surgeon to first preposition the end-effector accurately. Image-based visual servo (IBVS) control does not rely on depth information, exhibiting potential for addressing the issues mentioned above. Here we describe an enhanced IBVS strategy to achieve high performance and robust autonomous injection navigation. The radial basis function (RBF) kernel with strong learning capability and fast convergence is used to globally map the uncertain nonlinear strong coupling relationship in complex uncalibrated IBVS control. The Siamese neural network (SNN) is then used to compare and analyze the characteristic differences between the current and target poses, thus making an approximation of the mapping relationships between the image feature changes and the end-effector motion. Finally, a robust sliding mode controller (SMC) based on min–max robust optimization is designed to implement effective surgical navigation. Data from the simulation and the physical model experiments indicate that the maximum localization and attitude errors of the proposed method are 0.4 mm and 0.18°, exhibiting desirable accuracy with the actual surgery and robustness to disturbances. These results demonstrate that the enhanced strategy can provide a promising approach that can achieve a high level of autonomous intravitreal injection without a surgeon.

1. Introduction

Intravitreal injections are very common minor procedures in ophthalmology [1,2,3], and their automation benefits patients, physicians, and society. Intravitreal injections are usually used in the treatment of endophthalmitis [4], choroidal neovascularization [5], posterior uveitis [6], and so on. The commonly injected drugs include antibiotics [7], antivirals [8], antifungals [9], and anti-VEGF agents [10]. In practice, intravitreal auto-injection requires the injection needle to enter vertically into the inferior temporal quadrant of the human eye, 3.5 to 4 mm from the corneal limbus. With high precision required, inaccurate surgical procedures may lead to conjunctival hemorrhage, conjunctival scarring, intense pain, and even traumatic cataracts [11,12].

Advances in robotics, machine learning, and imaging endow the surgical approach with better clinical outcomes. Pre-clinical and clinical evidence suggests that automatic surgeries may standardize technical operations, increase efficiency, and reduce clinical complications [13]. Cehajic-Kapetanovic et al. used optical coherence tomography (OCT)–guided surgical robots for subretinal injections and achieved high precision. However, the 3D imaging techniques based on OCT, magnetic resonance imaging (MRI), and ultrasound are too expensive and complicated for minor procedures such as intravitreal injections [14,15,16,17,18]. Braun et al. performed simultaneous localization and mapping (SLAM) to conduct retinal vascular cannulation and laser photocoagulation based on the features of blood vessels in the retina [19]. S. Yang and B. C. Becker et al. also reported similar methods and results [20,21]. However, retinal vascular features are difficult to capture for intravitreal injections because the needle is located far outside the eyeball before the injection.

Visual servo uses machine vision to provide closed-loop position control for a robot end-effector [22]. The main advantage of this technique is that it imitates human vision and takes advantage of a large amount of information from the environment [23]. In recent years, visual servo has found many application scenarios in medical robot navigation, but its practical clinical implementation remains rare [24]. Image-based visual servo (IBVS) does not rely on accurate depth measurement [25,26], so it began to be used in procedures that lose depth perception such as minimally invasive surgery (MIS). C. Molnár et al. used the IBVS method to manipulate the da Vinci robot to reach the pre-placed marker [27]. P. Hynes et al. used the calibration-free IBVS technique to achieve surgical robotic manipulation of sutures with special markers [28]. These IBVS methods utilize the feature points or pre-placed special patterns; nevertheless, the eye has no easily extractable geometric features other than the pupil and iris. The eyelid, the blood vessels on the sclera, and the texture of the iris, on the other hand, are very different due to individual differences. The closest to intravitreal auto-injection is the ophthorobotics proposed by F. Ullrich et al. [29], which still requires the assistance of an ophthalmologist.

The main contribution of this work is the ability to achieve automatic and robust navigation of intravitreal injections without calibration and without depth information, using only limited information from the eye images. This proposed method can increase the level of automation of intravitreal injection and also helps to enhance the quality of the procedure as well as save valuable surgeon resources. We present a novel robust navigation method for autonomous intravitreal injection in a constrained environment using a visual servo-controlled robot. The developments presented in this work provide notably more autonomy compared with existing methods. To the best of our knowledge, this is the first intravitreal auto-injection system that is expected to achieve level of autonomy (LoA) 5 (a robotic surgeon that performs an entire process without the need for a human) [30,31]. This method combines a radial basis function (RBF) neural network [32], a Siamese neural network [33], and a robust SMC [34,35] to provide accurate and robust navigation for automatic intravitreal injection in the presence of inaccurate extracted features and noise. As a demonstration, we performed simulations on CoppeliaSim and compared the results with a generic multilayer perceptron (MLP) model to illustrate the advantages of this proposed method. The robustness of the method was also examined by introducing specific disturbances during the experiment. Finally, to verify the effectiveness of the algorithm, experiments were conducted using a simulated eyeball and a robotic injection device.

2. Materials and Methods

2.1. System Overview

The proposed method was based on an eye-in-hand system. In this system, two cameras fixed at the robot end-effector captured images. The controller guided the robot end-effector according to the captured image features, thus enabling the injector fixed at the robot end-effector to reach the injection pose, i.e., end-point closed-loop control [36]. The proposed method can also be used for a single camera, but two cameras can reduce errors by averaging effects.

The eyeball is a complex organ, and an accurate description of its geometry is difficult [37]. Based on the size data of the adult eyeball, it is an uncertain ellipsoid of 20.9–27.1 mm (transverse) × 20.5–26.4 mm (sagittal) × 19.9–27.0 mm (axial). There is an 11–12 mm diameter iris in the front of the eye [38]. The pupil size can vary from 10% to 80% of the iris [39]. Normally, saccades are the fastest and most amplified type of eyeball movement, with a peak speed of 450°/s [40]. However, the anesthetized eye movements are unlikely to reach the sweeping speed. We designed a simplified model: the eye was simplified as a 24 mm sphere, which could be rotated around the center of the sphere in both the X and Y directions perpendicular to the optical axis. Differences in the diameter of the patient’s iris caused differences in the target image features. The iris diameter was fixed at 12 mm in the experiment, while in practice, it needs to be pre-measured by the physician. It is extremely difficult to measure very small rotation angles in patients in the clinic, so we set a reference value of 5° in our experiments, but of course, the setting could be modified. In the simulation experiment, the eye could be directly manipulated to rotate, and in the physical experiment, the eye was pulled by a high-precision servo motor to achieve a rotation of ±4.4°. If the eye was rotated beyond ±5°, the entire procedure was forcibly aborted.

The injection needle path was discretized into N time intervals and represented as a set of needle tip poses at each time step $C_t=\left[\begin{array}{cc}{R}_t& p_t\\ 0& 1\end{array}\right] ϵ SE\left(3\right)$, and the image features were also in a corresponding state $s$. Following mainstream intravitreal injection requirements, the injection site was recommended to be in the inferotemporal quadrant 3.5 to 4 mm from the limbus, and the injection needle entered vertically at a depth of approximately 12 mm, which defined the final needle pose. There was an increased risk of retinal detachment if the injection site was too close to the iris and an increased risk of traumatic cataract formation if the injection site was too far from the iris [41]. In addition, throughout the process, the injection needle must not collide with the eye beyond the injection site. Since the iris diameter and the corneal diameter are close to each other, both around 12 mm, and the cornea is transparent, we used the iris as an approximate substitute for the cornea for navigation [42].

Many medical robots, such as the robot-assisted endonasal suturing implemented by J. Colan et al., use online path generation, mainly due to complex functional needs and the fact that longer paths can cause more trauma [43]. The intravitreal injection path had no such problems because it was simple to perform, and the needle was in the air rather than in the tissue before reaching the injection site. In addition, the uncertainty of our system was large. We suggest that designing a reference path manually, rather than generating it algorithmically, is more beneficial in ensuring the robustness of the system. We designed the path with the following considerations: (1) The closer the injection site was, the closer the needle tip was to the eye, and vice versa, to avoid accidental collisions. (2) The needle tip was gradually brought closer to the injection site from the inferior temporal direction, avoiding the bulging cornea. (3) After the path was discretized, the closer to the injection point, the denser the number of samples was to ensure accurate injection. (4) When the needle was moving on the reference path, the image features were located as close to the middle of the cameras’ field of view (FOV) as possible. (5) The length was as short as possible while ensuring the above premise. The reference path and the corresponding image features we designed are shown in Figure 1.

2.2. Visual Mapping Model

A visual mapping model that describes the changing relationship between the motion of the robot end-effector and visual features is a key link in the implementation of visual servo control. The classical interaction matrix approach requires the estimated parameters (depth) and the calibrated model of the camera. Moreover, the model of the robot can contain uncertainties engendered by the movement with sliding [44]. However, due to the reflective and refractive properties of the eyeball, the measuring depth cannot be accurate enough [25,26]. Although IBVS is quite tolerant to errors in measuring depth compared with position-based visual servo (PBVS), inaccurate depth estimates can still greatly affect the actual path of IBVS, which may increase the risk of the final stage of intravitreal injection [36]. In addition, it is also common to use state estimation algorithms such as Kalman filtering to estimate Jacobi online in simple visual servo tasks [45,46,47,48,49,50]. However, in practical applications, visual servo control is an uncertain nonlinear strong coupling system, and the accurate computation of the feature Jacobian matrix is very complicated [51]. Neural networks can be used to approximate such relations, hence avoiding computing the object’s feature inverse Jacobian, even at singular Jacobian postures [52]. Our proposed neural network-based approach does not require a priori knowledge of robot kinematics, hand–eye geometry, and camera models. To the best of our knowledge, this is the first visual mapping model to be used for the eye.

The output of the network proposed in other methods is mostly the relative pose of the camera or the absolute pose of the end-effector. Our network was trained to estimate the displacement ∆r required to move from the current pose to the target pose, according to the current image feature s and the target image $s^{\ast }$. $\Delta r={\left(\Delta t_f,\Delta {\eta }_f\right)}^T$ is the displacement of the pinpoint relative to the pinpoint reference system. The output of the network can be expressed in two forms: (1) $\mathsf{\Delta }r=f\left(\mathsf{\Delta }s,s\right)$ and (2) $\mathsf{\Delta }r=f\left(s^{\ast },s\right)$. We believe that the second approach was more favorable to calculate the relative displacement because the two inputs had the same order without the differential. A basic visual servo can be implemented with the following control laws:

$$\mathsf{\Delta }{r}_{k+1}=f\left(s_k+\lambda (s^{\ast }-s_k),s_k\right)$$

(1)

where $\mathsf{\Delta }{r}_{k+1}$ is the displacement of the next pinpoint and $\lambda$ is the gain, thus driving the end-effector of the robot ever closer to the target position. The inverse kinematic algorithm [53], which inverse solved the displacement of each joint of the robot from the robot end displacement, was provided by the official libfranka library.

As shown in Figure 2a, we used the following image features for robot guidance:

1. We used iris ellipse equation parameters. This was because the iris diameter varied very little from person to person compared with the pupil. We used least squares to fit the iris edge to obtain the general equation of the ellipse: $x^2+c_{1}xy+c_2{y}^2+c_{3}x+c_{4}y+c_5=0$, which has five independent parameters, $c_1,c_2,c_3,c_4, \mathrm{and} c_5$ [54]. The five parameters of the elliptic equation limited the five degrees of freedom (DOF) of the robot and ensured the exact distance from the final position to the iris edge.

2. We used the tilt angle of the line connecting the orientation indicator point to the center of the iris. The injection site needed to be on the inferior temporal side, so an orientation indicator point needed to be placed at an angle of 45° from the horizontal line. This feature was not accurate during eye rotations, but it was well tolerated in practical applications.

The image features captured by a single camera were a seven-dimensional vector, and the captures from the two cameras were combined into a 14-dimensional vector. The final robot pose is shown in Figure 2b.

The Lyapunov stability of neural network methods in visual servo is now proved [55,56,57]. To estimate the relative pose between the target image features and the current image features, we applied an SNN architecture. The SNN is one of the best choices for comparing two element vectors and outputting the similarity, where the two identical neural networks work in parallel and compare their outputs at the end [58]. The amplitude of the relative pose is positively correlated with the similarity between the current image features and the target image features, which is well suited for SNN. The network proposed by F. Tokuda et al. is very inspiring and also applies an SNN architecture, but the difference is that we used RBFNN instead of convolutional neural networks (CNN) to process the features because our input was a feature vector rather than the image itself [59]. As shown in Figure 3, our network contained two parts: the high-level feature extraction part and the regression part. The high-level feature extraction part contained two parallel embedding architectures with shared weights and parameters, consisting of an RBF layer and a fully connected layer. The main advantages of RBF were a short training phase and a reduced sensitivity to the order of presentation of training data. The regression part consisted of a subtraction process and a fully connected layer. The subtraction process constrained the extracted features to zero when the two input image features were the same. This constraint alleviated the high nonlinearity between the image feature space and the end-effector space, which made it easy for the network to learn feature embeddings. The last layer of the network was the fully connected (FC) layer, where the output was the next motion relative to the current pose. Unlike the output of F. Tokuda et al., our output was relative to the injection needle itself rather than the absolute coordinate system, which was experimentally found to be effective to improve the track performance in the case of eye rotation. We also use quaternions instead of Euler angles to represent the rotation to avoid the problem of gimbal lock [60]. All activation functions in the network were Gaussian error linear unit (GELU) functions, and Dan Hendrycks et al. showed that GELU has better performance in regression tasks compared with rectified linear unit (ReLU) and exponential linear unit (ELU) [61].

The data acquisition method was divided into two steps. In the first step, the space near the reference path was sampled. The reference path was interpolated to obtain the base point. The robot was then operated to move randomly around the base point, sampling a relatively small number of image features and pinpoint locations. In the second step, we selected two of the data collected in the first step and found the data used for training. We did not randomly select two data for calculation based on combinatorial theory as F. Tokuda et al. did. This was because the data set generated in this way was too large, and the large movements did not meet the requirements of practical applications. We required that the base points of the two sampled data must be adjacent to each other to limit the magnitude of the relative motion. In the first step, we captured the image feature $s_i$ and the absolute position of the pinpoint $r_i$. In the second step, the training data $\left(s_i,s_j,\mathsf{\Delta }{r}_{i, j}\right)$, $\left(s_j,s_i,\mathsf{\Delta }{r}_{j, i}\right)$, and $\left(s_i,s_i,0\right)$ were selected two at a time from the sample data, where $\mathsf{\Delta }{r}_{i, j}$ denotes the relative displacement of $r_j$ with respect to $r_i$.

The loss between the estimated vector and the ground truth vector was computed to regress the relative pose of the end-effector between the target and current images. As a reference, the loss function defined by F. Tokuda was

$$E=\alpha {\left|\left|\mathsf{\Delta }\widetilde{t_f}-\mathsf{\Delta }{t}_f\right|\right|}_2+\beta {\left|\left|\mathsf{\Delta }\widetilde{{\eta }_f}-\mathsf{\Delta }{\eta }_f\right|\right|}_2$$

(2)

where $\mathsf{\Delta }\widetilde{t_f}$, $\mathsf{\Delta }\widetilde{{\eta }_f}$, $\mathsf{\Delta }{t}_f$, and $\mathsf{\Delta }{\eta }_f$ are the ground truth of the relative translation, the ground truth of the relative orientation, the predicted relative translation, and the predicted relative orientation.

However, we defined the error of rotation as the angle between two quaternions instead of the mean square error of the Euler angles, so that the properties of the Lie group $\mathrm{SO}\left(3\right)$ space could be fully utilized [62]. $\alpha$ and $\beta$ are parameters used to adjust the training speed of the translation and rotation vector. In this paper, α = 1.0 and β = 1.0 were used. This loss function was more favorable for motion samples with larger amplitudes, whereas in the final stage of intravitreal injection, there were a large number of motions with small amplitudes. Therefore, we defined the loss function as the relative error as follows:

$$E=\alpha \frac{{\left|\left|\mathsf{\Delta }\widetilde{t_f}-\mathsf{\Delta }{t}_f\right|\right|}_2}{{\left|\left|\mathsf{\Delta }{t}_f\right|\right|}_2}+\beta \frac{{\left|\left|\mathsf{\Delta }\widetilde{{\eta }_f}-\mathsf{\Delta }{\eta }_f\right|\right|}_2}{{\left|\left|\mathsf{\Delta }{\eta }_f\right|\right|}_2}$$

(3)

The network was trained by AdaBelief [63] using PyTorch library [64] with learning rates of ${10}^{-3}$ and ${10}^{-4}$ for 700 and 300 rounds, respectively.

2.3. Visual Servo Controller

The vision mapping model enabled the simplest visual servo controller, but there was still room for optimization. The controlled system and controller are shown in Figure 4, where D(z) is the controller, V(z) is the visual mapping model, and I(z) is an integrating element. The proper design of D(z) can improve the performance of the whole visual servo system.

In the neighborhood of each node of the reference path, the system could be approximated as a linear time-invariant (LTI) system because the nodes were close to each other. With the input–output response of the neural network model, its approximate linear model $G\left(z\right)=\frac{b}{z-a}$ could be fitted using the least squares method, transformed into the differential equation as $y\left(k+1\right)=ay\left(k\right)+bu\left(k\right)$. Although the uncalibrated IBVS did not require consideration of the camera and tool calibration, model fitting errors, system identification errors, robot repetitive motion errors, and image recognition errors could make the system model inaccurate and varied with the pose of the robot end-effector. An SMC was then designed to overcome these problems, which was robust to model uncertainties and disturbances in the environment. To overcome the chattering problem associated with the SMC, we used quasi-SMC [65] instead of the classical SMC, i.e., we used the saturation function instead of the sign function. The design of the sliding mode surface was $s\left(k\right)=y\left(k\right)$, the convergence law was the constant rate reaching law $\Delta s\left(k\right)=s\left(k+1\right)-s\left(k\right)=-\eta sgn\left(s\left(k\right)\right)$, and the control law could be calculated as $u\left(k\right)=\frac{1}{b}\left[\left(1-a\right)y\left(k\right)-\eta sgn\left(s\left(k\right)\right)\right]$, defining the Lyapunov function of the system as

$$\begin{gathered} V\left(k\right)=\frac{1}{2}s{\left(k\right)}^2>0\left(s\left(k\right)\ne 0\right), \\ \mathsf{\Delta }V\left(k\right)=V\left(k+1\right)-V\left(k\right)=\frac{1}{2}\left[s{\left(k+1\right)}^2-s{\left(k\right)}^2\right] \\ \hspace{1em}\hspace{1em}\hspace{1em}=-\frac{\eta sgn\left(s\left(k\right)\right)}{2}\left[2s\left(k\right)-\eta sgn\left(s\left(k\right)\right)\right]<0 (\left|s\left(k\right)\right|>\frac{\eta }{2}) \end{gathered}$$

(4)

which meant that the system had Lyapunov stability. Then, the sign function was replaced with the saturated function

$$f\left(x\right)=\left\{\begin{array}{c}-1, x\le \Delta \\ \frac{x}{\mathsf{\Delta }}, -\Delta <x<\Delta \\ 1, x\ge \Delta \end{array}\right.$$

(5)

When $-\mathsf{\Delta }<x<\mathsf{\Delta }$, the controller was a linear controller, thus overcoming the chattering problem.

However, the optimal parameters tuned in the simulation may increase the risk of instability due to the presence of system uncertainty. Assuming that the system parameters had 10% uncertainty, we defined the problem with uncertain system parameters as a min–max robust optimization problem, then $a\in \left[0.9\widehat{a}, 1.1\widehat{a}\right]$ and $b\in \left[0.9\widehat{b}, 1.1\widehat{b}\right]$, and the problem transformed to

$$\begin{gathered} \underset{\left\{\eta , \mathsf{\Delta }\right\}}{\min }\underset{\left\{a,b\right\}}{\max }{\displaystyle \sum }_{k=0}^{N-1}qy{\left(k\right)}^2+ru{\left(k\right)}^2+py{\left(N\right)}^2 \\ \mathrm{s}.\mathrm{t}. a\in \left[0.9\widehat{a}, 1.1\widehat{a}\right] \\ b\in \left[0.9\widehat{b}, 1.1\widehat{b}\right] \\ \eta \in \left[0,1\right] \\ \Delta \in \left[0,1\right] \end{gathered}$$

(6)

After determining the solution window $N$ and the estimated values of the system parameters $\widehat{a}$ and $\widehat{b}$, the values of $\eta$ and $\Delta$ could be derived to obtain a robust sliding mode controller.

3. Experiments and Results

3.1. Simulation

To evaluate the performance of our proposed scheme, we first validated IBVS in a development simulation environment based on CoppeliaSim EDU V4.2 [66] and Python, in which we modeled visual features of the eye and simulated robot kinematics. There were 70,413 data sampled for model training. On the reference path, 20 values were inserted between every two neighboring nodes using the screw linear interpolation (ScLERP) [67] method, and then a uniformly distributed noise of different sizes was added.

To capture the response of the system in the face of deviation of image features, we selected two neighboring points from the sampled data as the initial and target points and ran 300 steps under the control of a proportional controller with a gain of 0.1 to obtain the input and output data of the system. For simplicity, we only analyzed the first image feature, and the other image features were similar. Based on the input and output data, the fitted discrete transfer function was $\frac{0.174452}{z-0.9310876}$, and the parameters of the robust sliding mode controller were $\eta =0.0005$ and $\Delta =0.01$ using the method described in the previous section. The general overshoot calculation formula was $(x_{max}-x_{\infty })/x_{\infty }$, and our calculation formula was $e_{min}/\left(f_{end}-f_{start}\right)=\left(f_{min}-f_{end}\right)/\left(f_{end}-f_{start}\right)$ because the feature 1 was reduced in this process, where $e_{min}$ is the minimum value of feature error 1, $f_{min}$ is the minimum value of feature 1, $f_{start}$ is the feature 1 at the reference start pose, and $f_{end}$ is the feature 1 at the reference end pose. The responses of the different methods to image feature deviations are shown in Figure 5.

We then used the proposed method to simulate the whole process. To verify the tracking performance of the algorithm, we also let the eye rotate at the 200th step (5°, 5°, 0°). The deviation of features from the target image is shown in Figure 6. All image features converged to near the ideal image feature but did not converge to 0. This was because the eye had rotated, and the target image was no longer accurate. The injection needle attitude also converged to near the ideal values, with a final deviation of 0.5 mm from the pinpoint position and an angle of 0.0438 rad for the injection needle attitude. The reference path, which was designed in Section 2.1, and the real path of the pinpoint are shown in Figure 7. Since the reference path was fixed relative to the eye, the reference path rotated accordingly after the eye rotated, which caused the jump in the reference path in Figure 7. We used the pytransform3d [68] library to calculate the reference path after rotation. Although the initial pose and the sudden eye rotation were located outside the task space, localization was accomplished through the generalization of the network. Proportional controller is commonly used in practice, and we used the proportional controller with gain $\lambda =0.12$ as the baseline [69,70]. As a comparison, we also performed simulations using the baseline method, and a comparison of the two is shown in

Table 1. Comparison of the baseline method and the proposed method during the simulation process.

Item	Baseline	Proposed
Rise steps	28	14
Overshoot	1.23%	1.62%
Standard deviation of balance position	1.09 × 10−3	9.48 × 10−4
Simulation test final translation error (mm)	5.1	0.5
Simulation test final rotation error (rad)	0.0607	0.0438
Distance between injection site and iris edge (mm)	1.31	3.68
Angle of injection point deviation from horizontal line (rad)	0.0937	0.2391

. The number of rising steps, overshoot, and standard deviation of the balance position in the table were derived using feature 1. The simulation results showed that the proposed method had a faster response, less oscillation at equilibrium, and smaller final error, which could achieve the required accuracy with the actual surgery.

3.2. Physical Model Experiments

The platform for the physical model experiments included a seven-degree-of-freedom robotic arm Franka Emika Panda, two symmetrically mounted cameras with USB video class (UVC) protocol, a coaxial ring light source, and an injector, as shown in Figure 8. The proposed methodology was implemented on a 2.9 GHz AMD computer running Linux (Ubuntu 20.04, Canonical) with real-time patches (RT-PREEMPT) [71]. During each cycle, the current robot pose was updated, and a target robot pose was computed and sent to the robot controller. Rigid body kinematics was implemented using the libfranka library that came with the robot system. Semantic segmentation of eye images is already a relatively mature technique and has been commonly used in, for example, the eye tracker and small incision lenticule extraction (SMILE) myopia surgery, so in the actual experiment, a simplified eye model (Figure 9b) was used instead of the complex actual surgical head model (Figure 9a) to execute the experiment. The simplified eye model replaced the real eye with a ball, the pupil with a solid blue circle in the center above the ball, and the iris with a red circle around it, and the rest was considered the sclera region, so that the pupil and iris regions could be accurately segmented using a simple method of color thresholding, allowing the study to focus on visual servo algorithms. The steering gear was connected to the eye model by an axis with an extension of the axis passing through the eye model sphere center. The steering gear controlled the rotation of the eye model around the axis to simulate the left–right rotation of the eye, with the rotation angle set from −4.4° to 4.4°. We conducted a robot-assisted intravitreal injection experiment to validate the performance of our system in a physically realistic environment. A total of 125,550 images were collected for model training from an initial position of the needle pinpoint 15 cm from the center of the eye to an end position where it finally touched the eyeball.

The experimental results are shown in Figure 10 and Figure 11, respectively. All image features converged to the vicinity of the desired image features, and the pose of the injection needle could reach the vicinity of the desired pose. In Figure 10, the simulation and physical model experiments used the same reference path design method, and their reference paths were not exactly the same due to differences in the positions of the eyeball model, robot, and camera. Figure 11 shows a partial zoomed-in view of the final approach to the target position to better represent the details. The reference path was specified manually during the design phase and later generated and executed by the machine in patients with the same iris diameter, using the method in Section 2.3. The translation error was 0.4 mm, and the rotation error was 0.0032 rad, which was increased compared with the simulation, and we think it was caused by inaccurate camera and image feature extraction, but it still met the needs of the surgery. As a comparison, the positional accuracy of Hynes et al. is similar to ours, although the systems and methods of both are very different [28]. The camera acquired images at convergence as shown in Figure 12. After repeated tests, the maximal distance between the pinpoint and the iris edge was approximately 3.6 mm, which could achieve the required accuracy with the actual surgery.

4. Discussion

4.1. Ablation Study

We enumerated the ablation studies in

Table 2. Ablation study.

	Proposed	w/o Robust Controller	w/o GELU	w/o Proposed Loss
Rise steps	14	25	5	6
Overshoot	1.62%	0.20%	2.04%	1.32%
Simulation test final translation error (mm)	0.5	0.7	1.6	0.8
Simulation test final rotation error (rad)	0.0438	0.0485	0.0407	0.0419
Distance between injection site and iris edge (mm), reference value: 3.5	3.68	3.55	4.03	3.45
Angle of injection point deviation from horizontal line (rad)	0.2391	0.241	0.195	0.240

to investigate the impact of the different components of the proposed approach. The ablation of the robust controller was studied by replacing it with a proportional controller with a gain of 0.1, the ablation of the GELU was studied by replacing it with a ReLU function, and the ablation of the loss function was studied by replacing it with a loss function of the original form proposed by F. Tokuda et al. The simulation results show that all components of the proposed method had significant positive effects.

4.2. Robustness to Noise

Due to the differences in equipment, environment, and lighting, there was noise pollution in the actual acquired images, which required the navigation system to have robustness to noise. To simulate this situation, we added different intensities of Gaussian noise to the image features. A comparison of the experimental results for different signal-to-noise ratio (SNR) cases with the noiseless case is shown in

Table 3. Robustness to noise.

SNR	∞	100 dB	80 dB	60 dB	50 dB	45 dB
Translation error (mm)	0.7	0.7	0.7	0.7	0.6	0.8
Rotation error (rad)	0.0485	0.0486	0.0486	0.0488	0.0489	0.0421
Distance between injection site and iris edge (mm)	3.55	3.55	3.55	3.53	3.54	3.98
Angle of injection point deviation from horizontal line (rad)	0.241	0.241	0.241	0.241	0.245	0.235

. For simplicity, robust controllers were not included in any of the methods compared. We also tried the simulation with an SNR = 40 dB, but the results were divergent. In the experimental results, the cases with SNRs of 100 dB, 80 dB, 60 dB, and 50 dB were almost indistinguishable from the case without adding noise, and the simulation results only deteriorated rapidly when SNR = 45 dB. The experiments illustrated that this algorithm had good robustness for environments containing noise.

4.3. Adaptability to Fewer Samples

The time cost of training data acquisition was high, so it made sense to achieve comparable accuracies on fewer samples for the practical application of the model. With all other experimental conditions unchanged, we randomly selected 10%, 25%, 50%, and 75% samples from the original training set to form a new training set for retraining the models. The comparison of the simulation results with the original data set is shown in

Table 4. Adaptability to fewer samples.

Sampling Rate	100%	75%	50%	25%	10%
Translation error (mm)	0.7	0.7	0.8	0.9	1.1
Rotation error (rad)	0.0485	0.0490	0.0481	0.0480	0.0437
Distance between injection site and iris edge (mm)	3.55	3.57	3.57	3.77	4.12
Angle of injection point deviation from horizontal line (rad)	0.241	0.228	0.247	0.243	0.188

. For simplicity, robust controllers were not included in any of the methods compared. In the experimental results, there was almost no difference between 100%, 75%, and 50% of the cases, and 25% of the cases were still within the acceptable range, while the remainder could not meet the requirements. The experiments illustrated that the algorithm was also adaptable to a few samples within a certain range.

5. Conclusions

In this work, a robust navigation system for autonomous intravitreal injection was proposed. Patient individual differences and the reflective and refractive properties of the eyeball make it impossible to use PBVS, which relies on accurate depth information. Moreover, the lack of point features in the eye that can be used for guidance also causes great difficulty for autonomous injections. To circumvent these difficulties, the system employed a combination of RBF-SNN and robust SMC for image-based visual servo. This enhanced autonomous strategy did not require a priori knowledge of robot kinematics, hand–eye geometry, and camera models. To verify this strategy, we compared the quality criteria of intravitreal injection—including the error of translation and rotation and the distance between the injection site and iris edge—of the developed system, mainstream intravitreal injection requirements, and similar visual servo tasks. Data from the simulation and the physical model experiments indicated that the developed system was capable of accurately guiding the end-effector of the robot to the intended injection location based on the limited information available to the eyeball. Although further technological advances are necessary for clinical applications, the principle and system may provide important information for the development of more advanced and automated robot systems for intravitreal injection in the future.