Grasping Complex-Shaped and Thin Objects Using a Generative Grasping Convolutional Neural Network ^†

Abstract

Vision-based pose detection and grasping complex-shaped and thin objects are challenging tasks. We propose an architecture that integrates the Generative Grasping Convolutional Neural Network (GG-CNN) with depth recognition to identify a suitable grasp pose. First, we construct a training dataset with data augmentation to train a GG-CNN with only RGB images. Then, we extract a segment of the tool using a color segmentation method and use it to calculate an average depth. Additionally, we apply and evaluate different encoder–decoder models with a GG-CNN structure using the Intersection Over Union (IOU). Finally, we validate the proposed architecture by performing real-world grasping and pick-and-place experiments. Our framework achieves a success rate of over 85.6% for picking and placing seen surgical tools and 90% for unseen surgical tools. We collected a dataset of surgical tools and validated their pick and place with different GG-CNN architectures. In the future, we aim to expand the dataset of surgical tools and improve the accuracy of the GG-CNN.

1. Introduction

Grasping has been a long and extensively studied area in Robotics. In an era of automation, robotic grasping allows for the quick, reliable, efficient, and reproducible handling of objects in many environments. While humans beings are able to grasp various kinds of objects under various environmental conditions, replicating the same skills in robots is not an easy task. The first step in a robotic grasping process is to detect an object’s pose, which is still challenging. Although active research by the computer vision and machine learning community has provided solutions to this problem, numerous challenges stem from the object’s properties of shape, appearance, color, occlusions, etc. An object such as a surgical tool is difficult to detect using traditional vision approaches. The tool has a complex shape, a reflective surface, and thinness, making it tough to infer its grasp pose using traditional vision-based methods confidently. Consequently, existing works have focused on machine learning and deep learning approaches to propose the grasp poses of such objects. In this regard [1,2] proposed a novel neural network architecture called the Generative Grasping Convolutional Neural Network to select grasp points and to predict the quality and pose of grasps. Although this method has been able to propose good grasp candidates for adversarial and household objects, it fails to address the problem of the thinness associated with surgical tools, and as a result, it performs poorly on such objects. We have addressed this issue by training a GG-CNN2 with only RGB images instead of only depth images as used by [2]. Our network’s output is still a grasp box in pixel coordinates, and its center can be taken as the grasp point. Moreover, the depth estimation of the grasp point is important for the grasping task. A color segmentation method with average depth calculation could provide the depth information between a camera and a tool in our experimental setup. We integrated the Generative Grasping Convolutional Neural Network with the segmentation method that could generate a reliable grasping point and experimented with it in an actual environment (Figure 1).

Concretely, the contributions of our work are the following:

• We created a new dataset of surgical tools composed of only complex-shaped and thin objects that are usually more difficult to grasp due to the challenging depth estimation resulting from their thinness;
• We proposed an architecture for grasping complex-shaped and thin objects, such as surgical tools, using the GG-CNN/GG-CNN2 with a segmentation method that provides the depth of the surgical tools images;
• We compared the performance of the GG-CNN model with the dataset by applying different encoder–decoder models with GG-CNN/GGCNN2 structures and evaluating the models with the IOU.
• We conducted preliminary experiment tests for validating the GG-CNN architecture for grasping the surgical tools of seen and unseen in noncluttered or cluttered environments.

This paper is organized as follows: Section 2 presents the related works. Section 3 describes the surgical tools dataset and the GG-CNN. Section 4 introduces the description of the experimental setup. Section 5 discusses the experiments, and finally, Section 6 concludes the paper.

2. Related Works

The grasp synthesis problem has been studied for decades and it can be divided into empirical and analytical methods [3,4]. Empirical methods [5,6,7] focus on using models and experience-based approaches. These techniques work with known items, associating good grasp points with an offline database of object models or shapes [8,9,10], or familiar items, based on object classes [11] or object parts [12], but they are unable to generalize to new objects. Analytic methods [13,14], instead, use mathematical and physical models of geometry, kinematics, and dynamics to calculate grasps that are stable [15,16]. However, they tend to not transfer well to the real world due to the difficulty in modeling physical interactions between a manipulator and an object [4,15,17].

Analytical metrics have been rigorously tested by comparing their predictions to the outcomes of real-world experiments [18,19], and the findings of these studies indicate that analytical metrics are not very good predictors of grasp success in the real world. This is why we decided to determine the optimal grasping area by evaluating the grasping quality based on the geometric position relationship. Another reason to avoid using analytical models is that they are costly in terms of computational resources [4].

In the past [10,11], most works used human-designed features to represent grasps in images or required the full 3D model of objects to generate grasps [20,21]. These methods were very popular, but they all faced the challenges of non-robust features or lacked the full 3D models when used in real-world applications. Moreover, these methodologies depended to a great extent on the the pre-built database, not generalizing well to novel objects.

Grasp Proposals Using Deep Learning Methods

Recently, deep learning methods [22] have proved effective in robotic grasp generation [23]. Neural networks are used to extract features from images and apply them to detect grasps to be executed by robots. Several of these techniques share a common pipeline: classifying grasp candidates sampled from an image or point cloud, and then ranking them individually using CNNs [2]. However, such approaches provide two key challenges. The first is the availability of diverse and high-quality data, which is widely recognized as an important prerequisite for training robust and generalizable models. The second one provides standardized, reproducible, and comparable evaluation methods and metrics [24]. In order to accelerate the detection speed, the recent GG-CNN proposed by Morrison et al. [1] directly evaluated the grasp quality and the pose of grasps at every pixel. This architecture has a high processing speed, overcoming the limitations of the current deep learning grasping techniques by avoiding the discrete sampling of grasp candidates. Moreover, the GG-CNN is orders of magnitude smaller while detecting stable grasps with an equivalent performance to the current state-of-the-art techniques [1]. The GG-CNN was used by Yu et al. [25] to build a regression network that is part of an efficient cascaded deep learning framework which has a real-time performance and guarantees grasp robustness for robotic manipulation in an unstructured environment. Wang et al. [26] modified the GG-CNN, developing an efficient neural network model to generate robotic grasps with high-resolution images, while Mahajan et al. [27] augmented the GG-CNN architecture with a decoder structure used in the vector-quantized variational autoencoder (VQ-VAE) model with the intuition that it should help to regress in the vector-quantized latent space.

The same authors improved the GG-CNN, creating a new network for real-time grasp prediction called GG-CNN2 (Figure 2). This architecture is a fully convolutional network based on the semantic segmentation architecture from Yu and Koltun [28], which uses dilated convolutional layers to provide an improved performance in semantic segmentation tasks. The GG-CNN2 uses the same input and outputs as the GG-CNN, contains 66,000 parameters, and has an average inference time of 3 ms, with the entire grasping pipeline taking on average 20 ms. In the GG-CNN2, the authors vary three parameters: the filter sizes, number of filters, and size of the dilated convolutions. The GG-CNN2 is used in [29] as a part of the ORientation AtteNtive Grasp synthEsis (ORANGE) framework that jointly solves a bin classification problem and a real-value regression. In [30], a comparison was made between the GG-CNN2 and the performance in grasping of a network presented by the authors. They proposed a novel Generative Residual Convolutional Neural Network (GR-ConvNet) model that can generate robust antipodal grasps from n-channel input at real-time speeds, and they stated that their network had a better performance in grasping than the GG-CNN2. In [31], the authors proposed a method for centering the object of interest in the field of view using visual serving based on RGB-D image data acquired from an eye-in-hand camera. Their method was designed to reduce propagation errors and eliminate the need for complex hand tracking algorithms, image segmentation, or 3D reconstruction. Regardless of the geometric complexity and physical properties of the object in question, the proposed approach can efficiently generate reliable multi-view object grasps. The system architecture proposed allows for simple and effective path generation as well as real-time tracking control. Furthermore, the system is modular, dependable, and precise in end-effector path generation and control. In another work [32], the authors took a different approach to grasp detection by treating it as image-space keypoint detection. Rather than a triplet or quartet of corner points, the deep network detects each grasp candidate as a pair of keypoints convertible to the grasp representation $g={\{x,y,w\}}^T$. This increases the performance by reducing the detection difficulty by grouping keypoints into pairs. A non-local module is incorporated into the network design to encourage the capture of the dependencies between the keypoints. A final filtering strategy based on discrete and continuous orientation prediction eliminates false correspondences, improving the grasp detection performance even further. Finally, in [33], starting with a 3D partial view of the object, the authors proposed an end-to-end learning solution for generating 6-DOF parallel jaw grasps. Their Learning to Grasp (L2G) method extracts information from an input point cloud using a novel procedure that combines a differentiable sampling strategy to identify visible contact points with a feature encoder that takes advantage of local and global cues. Overall, L2G is guided by a multi-task objective that optimizes contact point sampling, grasp regression, and grasp classification to generate a diverse set of grasps.

3. Methods

3.1. Grasping Generative Convolutional Neural Network

In [34,35,36], the grasp representation proposed by [37], and then simplified by [38], was used to generate antipodal robotic grasps using RGB-D images of objects. The grasp representation is defined by the following formula:

$$g=\{x,y,\varphi ,h,w\},$$

(1)

g represents a five-dimension rectangle perpendicular to x-y plane that includes the center of the rectangle (x,y), the orientation of the rectangle relative to the horizontal axis of the image $\varphi$, its width (w), and height (h). Morrison et al. [1] proposed a new representation of robotic grasps that changes g into this representation:

$$g=\{p,\varphi ,w,q\},$$

(2)

where p = (x, y, z) is the center position of the gripper, $\varphi$ is the rotation angle relative to the horizontal axis of the image plane, w is the gripper width, and q is the grasp quality, which represents the chances of grasp success. Concerning the computation of q, each grasping rectangle has set the corresponding area of q to a value of 1. All other pixels are 0.

Robotic grasps are detected in the depth image I $\in {\mathbb{R}}^{H\times W}$ with height H and width W. In the image space I, the grasp g is represented by

$$\tilde{g}=\{s,\tilde{\varphi },\tilde{w},q\},$$

(3)

where s = (u,v) represents the center point in pixels coordinates, $\tilde{\varphi }$ denotes the rotation relative to the camera frame, $\tilde{w}$ denotes the gripper width in pixels, and q the grasp quality. The grasp map G proposed by Morrison et al. [1] is

$$G=\{\mathsf{\Phi },\mathit{W},\mathit{Q}\}\in {\mathbb{R}}^{3\times H\times W}$$

(4)

$\mathsf{\Phi }$ is an image of dimension $H\times W$ which describes the angle of a grasp to be executed at each point, W an image which expresses the gripper width of a grasp to be executed at each point, q an image which defines the quality of a grasp executed at each point (u,v). $\mathsf{\Phi }$, $\mathit{W}$, $\mathit{Q}$ are each in ${\mathbb{R}}^{1\times H\times W}$ and each pixel contains the values $\tilde{\varphi }$, $\tilde{w}$, and q, respectively, at each pixel s. Following [1], the authors used this network to generate a grasp g for each pixel in depth image I, which denotes the pixel-wise representation:

$$M\left(I\right)=G,$$

(5)

where the map function M is a deep NN, and then the best grasp can be found by

$$g=\underset{Q}{max}G.$$

(6)

The Mean Squared Error loss is applied to predict grasp pose, grasp quality $\tilde{q}$, gripper width $\tilde{w}$, and rotation $\tilde{\varphi }$.

3.1.1. Contractive Networks

A deep contractive network imposes a layer-wise contractive penalty in a feed-forward neural network. The layer-wise penalty approximately minimizes the network outputs variance with respect to perturbations in the inputs, enabling the trained model to achieve “flatness” around the training data points [39]. The objective function of the model is the following:

$${\displaystyle J_{DCN}\left(\theta \right)=\sum _{i=1}^{m}L(t^{\left(i\right)},y^{\left(i\right)})+\sum _{j=1}^{H+1}{\lambda }_j{‖\frac{\partial h_j^{\left(i\right)}}{\partial h_{j-1}^{\left(i\right)}}‖}_2}$$

(7)

with outputs y$\in {\mathbb{R}}^{d_y}$, a target t$\in {\mathbb{R}}^{d_y}$, h is the hidden representation $\in {\mathbb{R}}^{d_y}$ with the number of hidden layers H, and $\lambda$ is a scaling factor that trades off reconstruction objective with contractive objective. ${‖\frac{\partial y^{\left(i\right)}}{\partial x^{\left(i\right)}}‖}_2$ is the Frobenius norm of the Jacobian matrix of $h_{\left(j\right)}$ with respect to $h_{(j-1)}$.

This layer-wise contractive penalty computes partial derivatives similarly to a contractive autoencoder and is easily incorporated into the backpropagation procedure. Moreover, it is a computationally efficient way to greedily propagate input invariance through a deep network.

3.1.2. Denoising Networks

The denoising encoder–decoder model introduces some noise to the input before the model starts training. This procedure achieves good representation because corrupted input data will help obtain a robust model useful for recovering the corresponding label data. From the state of the art, the most discussed types of noise in the literature are the additive white Gaussian noise (AWGN) [40], impulse noise [41], quantization noise [42], Poisson noise [43], and speckle noise [44]. For our work, random noise with noise factor are used to produce the noise input data.

3.1.3. Sparse Networks

Sparse neural networks are artificial neural information processing systems restricted in their structure and dynamics to conserve resources. The concept of sparseness refers to the network’s connectivity structure, so that each neuron receives inputs from only a limited number of other neurons, and to the network’s state which describes the level of activity of the entire neural population, so that only a few neurons are active at any one time [45].

3.1.4. Variational Autoencoder (VAE) Networks

VAE is an artificial neural network architecture introduced by Kingma and Welling, belonging to the families of probabilistic graphical models and variational Bayesian methods. VAE is meant to compress the input information into a constrained multivariate latent distribution (encoding) to reconstruct it as accurately as possible (decoding). In these networks, the input data are sampled by a parametrized distribution, and the encoder and decoder are trained jointly so that the output minimizes a reconstruction error in the sense of the Kullback–Leibler divergence between the parametric posterior and the true posterior [46].

3.2. Proposed Approach

In this paper, we propose an architecture for achieving precise grasping of surgical tools, using computer vision techniques and depth segmentation. The architecture integrates the GG-CNN/GG-CNN2 models with depth segmentation to achieve high-precision grasping of surgical tools (see Figure 1). The input to our system is an RGB image of a single surgical tool on a white flat surface. The image is cropped by the network to provide a better view of the tool, and then analyzed by the GG-CNN/GG-CNN2 models. These models output the x, y, and orientation of the best grasping rectangle for the tool in the camera frame. The GG-CNN/GG-CNN2 models have been shown to achieve state-of-the-art performance in robotic grasping tasks and are well suited to our application of surgical tool grasping. In addition to the GG-CNN/GG-CNN2 models, we incorporate depth segmentation to calculate the average z value of the tool in the camera frame. This segmentation output is a depth image of the tool, which is then transformed into the robot frame. The depth segmentation provides additional information about the tool’s position and orientation in 3D space, which is critical for achieving precise grasping. Once we have obtained the x, y, and z values of the tool in the robot frame, we issue a position command to the robot using open-loop position control. This command directs the robot to move to the specified position and orientation in order to grasp the tool with precision. The open-loop position control has the advantage of being simple and fast, which is important for real-time operation in surgical settings. Our proposed architecture has several advantages over existing approaches. First, it combines the strengths of the GG-CNN/GG-CNN2 models with depth segmentation to achieve highly accurate and efficient grasping of surgical tools. Second, it is based on open-loop position control, which is simple, fast, and reliable. We evaluated our proposed architecture on a dataset of surgical tools and achieved high accuracy and efficiency in grasping the tools. We evaluated our proposed architecture on a dataset of surgical tools and achieved high accuracy and efficiency in grasping the tools.

4. Experimental Setup

Our experimental setup uses a Universal Robot (UR5) equipped with a Robotiq gripper, and a Kinect v2 is used to acquire the images of the surgical tools (as shown in Figure 3a,b). The vision sensor is mounted at the top of a tripod, as shown in Figure 3c. We trained the surgical dataset on a PC running Ubuntu 16.04 LTS and using a GTX-1080ti GPU. The code was predominantly written in Python. For what regards the motion planning of the manipulator, we used the Moveit tool [47].

4.1. Datasets

Because our dataset size is very small compared to datasets such as the Cornell dataset and the Jacquard one, we augmented it using random rotations, zooming, jitter and white noises, and brightness variations. We chose to augment the dataset using random rotations, zooming, jitter and white noises, and brightness variations, because they are more “natural” ways to augment the dataset and because they represent possible variations in how the object can be found in reality. For example, the rotation augmentation is useful because the surgical tool can have different angular positions when it is placed on the table. Moreover, brightness is useful when the object is placed in different environments with several light conditions. Zooming augmentation is also important because the object can be grasped by the manipulator from different angles, and noises are also necessary to simulate different environment conditions.

In the end, we obtained a dataset of 9920 images (the original dataset consists of 320 RGB images). The dataset is composed of images showing a single surgical tool placed on a table-top scene with white background. We decided to have different categories in the training set and validation set to stress the network more and see if it can successfully grasp the new surgical tools even if it has not seen them before. Moreover, the figures are manually labeled with grasping rectangles. The augmented dataset is publicly available (the dataset was released on github: https://github.com/OliviaNocentini/SurgicalKit (accessed on 15 January 2023)).

Then, we used a 80/20% split for training/testing of the GG-CNN2. To determine the best network configuration, we trained the network by changing some parameters (

Table 1. Internal parameters of the GG-CNN2.

Batch size	8
Epochs	100
Batches per epochs	1000
Val-batches	250
Optimizer	Adam
Learning rate	0.001
Loss function	MSE

).

4.2. Experimental Method

During the experimental phase, we tested the grasping performance not only of the GG-CNN2 but also of the GG-CNN to compare the two networks. From the training of the GG-CCN and GG-CNN2, we obtained several models based on different combinations of augmentation data. We used the best models to predict the grasp poses of each surgical tool with the GG-CNN and GG-CNN2, with fixed depth or segmentation. For each configuration, we grasped each surgical tool 10 times and we used seen (used for training) and unseen objects (27 and 9, respectively) to evaluate quantitatively the grasping of the surgical tools using the GG-CNN2 with fixed-depth or with segmentation method (Figure 3a,b).

5. Experimental Results and Discussions

5.1. Quantitative Results

5.1.1. Network Evaluation

We evaluated the Intersection Over Union (IOU) with the GG-CNN/GG-CNN2 during the training. In this paper, the IOU refers to the Intersection Over Union between the ground-truth grasping rectangles, obtained by manually labeling all the trained images, and the predicted grasping rectangles. With regard to the ground-truth grasping rectangles, we annotated a txt file with 4 points for each grasping rectangle. These points represent the vertices of the grasping rectangle and were annotated following the reference paper [1]. The points are used to derive four pieces of information (position, sine and cosine angles, and width) that are used for calculating the loss.

The IOU is described with the following equation:

$$\mathrm{IOU}=\frac{|A\cap B|}{|A\cup B|}$$

(8)

where A and B are, respectively, the real and ground-truth rectangles. A success is counted if the grasp rectangle has a 25% IOU with a ground truth and is within 30 degrees [38]. In Figure 4, the ground-truth boxes (in blue) and the real grasping rectangle (in red) are shown. As can be seen, the real grasping rectangle is near the ground-truth rectangle where there is a high q value (yellow color in the center of the right grasping spot light).

Table 2. Comparison of the IOU of different networks.

Network	IOU (Max 1)
GG-CNN	0.79
GG-CNN2	0.88
Contractive-GG-CNN	0.78
Contractive-GG-CNN2	0.98
Denoising-GG-CNN	0.41
Denoising-GG-CNN2	0.90
Sparse-GG-CNN	0.73
Sparse-GG-CNN2	0.96
VAE-GG-CNN	0.18
VAE-GG-CNN2	0.26

describes the comparison of the IOU with different encoder–decoder models and GGCNN structures. Normally, the GG-CNN2 outperforms the GG-CNN with different models. Moreover, except for the performance of the VAE-GG-CNN/GG-CNN2 and Denoising-GGCNN, the average IOU of the models is higher than 80%. The original GG-CNN/GG-CNN2 already achieved a good evaluation over approximately 80% (with an MSE loss function), but other models conduct a greater performance. The reason is that contractive GG-CNN/GG-CNN2 models applied different loss functions with penalty terms. It is a kind of regularization technique that could extract useful features. In contrast, the performance of the VAE-GG-CNN/GG-CNN2 is dramatically low compared to the other models. Especially, the output from the encoder of the VAE is composed of latent distribution, which is $\mu$, and covariance as the input to the decoder. It helps to train the model and generate new data, but the latent distribution missed many of the features from the new dataset during the training. Moreover, we used three MSE loss functions with a GG-CNN structure that is not the same and the original VAE structure was not optimized.

5.1.2. Grasping Single Tool

An example of the outputs of the GG-CNN2 is shown in Figure 5a. In this figure, the RGB images of the surgical tool with the grasping rectangle, the depth image of the object, the q value, and the angle for four different surgical tools are shown. As we depicted in the images, the q value influences the grasping position, as the grasping rectangle location corresponds to a region with a higher value of q. In Figure 5b, the output of the segmentation of each object is shown.

With regard to the GG-CNN with a fixed depth, we achieved a success rate of 62% in grasping both the seen and unseen (73.64%) surgical tools. On the other hand, the GG-CNN2 obtained a success rate of 94.8% and 78.18%, respectively, in grasping the seen and unseen tools. Surprisingly, the performance was better for the unseen objects than for the seen objects. This can be attributed to the fact that the seen surgical tools have a more complex shape than the unseen ones. The GG-CNN2 with fixed depth exhibited a better performance both on the seen and unseen objects compared to the GG-CNN and this is due to its dilated convolutional layers, which provide an improved performance in semantic segmentation tasks. With respect to the GG-CNN and the segmentation method, we obtained a success rate of 74% for the seen objects and 66.36% for the unseen objects. For the GG-CNN2 with the segmentation method, the performance was 85.6% for the seen objects and 90% for the unseen objects. Therefore, we conclude that the segmentation method, when combined with grasping for unseen surgical tools, produces better results than a fixed depth (see

Table 3. To evaluate the performance of GGCNN and GGCNN2 for grasping in various environments, we use several datasets, including our Surgical tools (unseen only), Cornell, Daily supplies, Jacquard, and Restaurant object.

	Dataset	Surgical	Cornell	Daily	Jacquard	Restaurant
Models
GGCNN		66.36%	85.39% [48], 88% [49], 91% [1]	91.45% [50]		94.14% [51]
GGCNN2		90%	95.5% [52]		94% [2]

).

Our architecture’s reliability and effectiveness have been validated and discussed in Table 3. We compared the accuracy of the different grasping tasks using the GGCNN/GGCNN2 models. Based on the results, the models have proven to be reliable and effective for grasping tasks in various environments, with an accuracy of over 80% in actual environments. Given these results, our architecture could be applied to grasp surgical instruments in different environments. However, there are still limitations and improvements needed.

Limitations:

• Despite using a few variations in the training dataset, this model would still not adapt to novel objects due to limited variations in lighting conditions or other environmental factors.
• This model may struggle with objects that have complex geometries or are occluded, as it relies on a simple geometric grasping position.

Improvements:

• Improving the training dataset by incorporating more diverse object shapes, textures, and sizes.
• Exploring new learning techniques such as transfer learning or domain adaptation methods for handling new objects or environments [53,54].
• Using other types of sensors, such as tactile or additional information, provides the model with more information for grasping [55].

5.2. Qualitative Results

5.2.1. Grasping Multiple Tools

Regarding the grasping of multiple surgical tools concurrently, we decided to conduct the following experiments. We used 9 unseen surgical tools and attempted to grasp them using 5 configurations for each tool, resulting in 45 grasping attempts in total. This experiment is illustrated in Figure 6. We applied this protocol to the GG-CNN and GG-CNN2 with fixed depth and with segmentation. We only obtained good results for the GG-CNN with fixed depth, which achieved a success rate of 68.89%. The other combinations resulted in lower success rates for grasping the surgical tools.

The architecture is able to grasp the tools, but it has a problem of not completing the task. The issue is described in the following section.

5.2.2. Failure Grasping Examples

During the experimental part involving a single surgical tool, we encountered several common problems. The most frequent issue was the tool falling after being grasped. This was due to the large length of the objects and the lack of consideration of their center of mass. Moreover, the higher weight of the surgical tools and the material they are made of contributed to the problem. The shape of the surgical tools also played a critical role in the success or failure of grasping, as strange shapes are more difficult to grip and thicker objects are easier to hold than thinner ones. We also encountered two failure cases where the GG-CNN2 could not find the grasping box from the input data with differently colored tools (see Figure 7a) or where the segmentation was incorrect (see Figure 7b). Both of these issues arose from the lack of training examples. Additionally, we occasionally encountered an error related to the Moveit! software, which prevented us from grasping the surgical tool. This issue was resolved by running the grasping program a second time.

As for the issues we faced in grasping multiple surgical tools, we encountered the following problems. First, the grasping box was sometimes inaccurately located (see Figure 7c), preventing us from grasping any surgical tool. Second, due to the material or weight of the surgical tool, it would fall after being grasped (see Figure 7d). Finally, the computation of the surgical tool depth was sometimes incorrect because we used the average of the segmentation model’s output depth, leading to failures in grasping the object.

6. Conclusions

Grasping thin and complex-shaped objects is a challenging task in the field of manipulation. In this paper, we proposed an architecture that utilizes the GG-CNN with a segmentation method for grasping such objects. In the future, we plan to expand the dataset by using more surgical tools and modifying the GG-CNN architecture to achieve even higher success rates in grasping trials. Additionally, we aim to explore how different encoder–decoder network models with a GG-CNN architecture and segmentation can improve grasping multiple surgical tools based on the different colored height map [56] and with a compliance gripper [57]. Finally, we would like to study in depth the grasping of surgical tools with the Deformable Convolutional Net [58] or by adding different modules to the GG-CNN2.