Teleoperated Locomotion for Biobot between Japan and Bangladesh

Abstract

Biobot-based insects have been investigated so far for various applications such as search and rescue operations, environmental monitoring, and discovering the environment. These applications need a strong international collaboration to complete the tasks. However, during the COVID-19 pandemic, most people could not easily move from one country to another because of the travel ban. In addition, controlling biobots is challenging because only experts can operate the cockroach behavior with and without stimulated response. In order to solve this issue, we proposed a user-friendly teleoperation user interface (UI) to monitor and control the biobot between Japan and Bangladesh without onsite operation by experts. This study applied Madagascar hissing cockroaches (MHC) as a biobot hybrid robot. A multithreading algorithm was implemented to run multiple parallel computations concurrently on the UI. Virtual network computing (VNC) was implemented on the teleoperation UI as remote communication for streaming real-time video from Japan to Bangladesh and sending remote commands from Bangladesh to Japan. In the experiments, a remote operator successfully steered the biobot to follow a predetermined path through a developed teleoperation UI with a time delay of 275 ms. The proposed interactive and intuitive UI enables a promising and reliable system for teleoperated biobots between two remote countries.

1. Introduction

A teleoperation system is a technique that enables an operator to control the system from a remote location. Teleoperation is obtained from the Greek words Tele and Operation, which mean distance and task completion, respectively. The length could be physical, such as a human operator controlling a robot in a remote location, or a change in scale, such as a surgeon using teleoperation to perform surgery at the micro-scale level [1]. Nowadays, teleoperation technology is applied for various applications, including entertainment systems, industrial machinery, remotely operated vehicles (ROVs), remote surgery, unmanned aerial vehicles (UAV), etc. [2]. Mobile robots have been developed for search and rescue [3]. Mobile robots have difficulty running in an unknown environment, such as a disaster area. Moreover, mobile rescue robots cannot efficiently perform because they are expensive due to the initial cost of equipment, peripherals, installation cost, programming, and training [4]. This robot needs regular maintenance and a power source to drive the actuators. It cannot respond to emergencies and has limited sensors, vision systems, and real-time responses [5]. For these reasons, researchers are interested in biobots for search and rescue operations because this type of robot overcomes the limitation of mobile rescue robots [6]. Biobots reduce the equipment, installation, and training cost. The capabilities of sensors, vision systems, and real-time responses are higher than those of traditional mobile rescue robots due to their instinct. For the cyborg hybrid robot, one advantage is the power supply. It can be recharged or replaced with electrical power devices for a long time [7].

However, controlling biobots is a difficult task. Li et al. developed a brain–computer interface (BCI) to control the cockroach biobot [8]. They proposed a functional data transfer pathway from the human brain to the cockroach brain. Nguyen et al. developed sideways walking control of a cyborg beetle [9]. They developed a cyborg beetle control system that users can commend through infrared remote control. Cao and Sato developed remote radio-controlled insect–computer hybrid robots for search and rescue operations [10]. These cyborg controls are based on a wireless communication system for a limited distance range.

Teleoperation technology has been widely used to control various kinds of robotic applications. During pandemics, researchers have understood the necessity of a teleoperation system [11]. Tabaza et al. developed a robotics-assisted surgery robot using teleoperation technology [12]. Wang et al. developed a humanoid robot’s intuitive and versatile full-body teleoperation [13]. Diolaiti and Melchiorri researched the teleoperation of a mobile robot over haptic feedback [14]. Yuji et al. developed a teleoperation mobile robot that utilized the robot sensor network [15]. Ruan et al. created a motion control self-balance robot using vision-teleoperation [16]. By using a head-mounted display, Martine and Ventura built an immersive 3D teleoperation of a search and rescue robot [17]. Okabe et al. developed the teleoperation system for rescue robots by recording the target position of the end-effector [18]. Hong et al. proposed practical hardware design tactics and control methods for a rescue robot to save patients in disaster-stricken environments. The team created a powerful dual arm mechanism and a hybrid tracked-legged mobile platform. The motion was synthesized using dynamics-based optimization and a controlled hierarchical control scheme [19]. Teleoperation is one of the advanced technologies which is very popular for various applications in the modern world.

User interface (UI) is needed to be incorporated into the teleoperation system for efficient remote telecontrol of mobile robots [1,20,21]. The UI manages teleoperation processed, and displays measured data, a live-camera view to the operator. Communication channels such as the internet, radio frequency, Bluetooth, and infrared are commonly used for teleoperation. In previous research studies, Bluetooth and infrared are limited to the communication link range between the operator and remote robot [1,22]. Qian K et al. developed a small teleoperated robot with a control distance reaching 5000 m in an open area using radio wave communication [23]. Bluetooth, infrared, and radio wave are relatively low-cost. Internet communication allows a remote robot can be accessed and controlled from any part of the world [1,24]. Due to their size, various sensors, actuators, and cameras can be attached to these teleoperated robots

Previous biobot research studies are based on local communication and limited communication distance between computers and biobots [9,25,26,27]. Wireless technologies such as infrared [9,28,29], ZigBee [30,31,32], Bluetooth [8,33,34,35,36,37], WiFi [38,39], and radio frequency (RF) [25,27,40,41,42,43] were used to control the motion of biobots and wireless data transmission. These wireless devices cannot reach long-distance communication. The biobots cannot be controlled from a remote distance, such as city to city or country to country. Therefore, controlling the biobot using teleoperation technology from one country to another is quite a new and hot research topic. It will be useful during a pandemic when most people cannot travel from one country to another due to a travel ban. We propose a teleoperation system to monitor and control biobots from Bangladesh to Japan. This research will assist in the emergency or pandemic situation when biobot operators cannot physically come to the onsite area where the biobots are placed at a remote distance. These proposed methods will enable the operator to monitor and control the biobot from a remote location.

In this study, we proposed a biobot system that can be monitored and controlled from a remote distance through a developed teleoperation UI (remote and local UI). We employed two communication channels for the teleoperation UI. Virtual network computing (VNC) was selected and implemented for teleoperation communication between Japan and Bangladesh. The VNC was applied to provide live-stream video and send remote stimulation commands through remote UI. A radio frequency (RF) device was implemented for local communication between a computer and the biobot in Japan. The RF device sent remote commands from the computer to the backpack and received inertial measuring unit (IMU) data from an electronic backpack to the computer. The first camera was used as an online motion-tracking system on the local UI, while the second one was utilized for the remote live streaming video on the remote UI. The cockroach states were displayed on the local UI. A multithreading algorithm was applied to run multi-rate computational processes concurrently on local and remote UI. A remote operator in Bangladesh was tasked to steer the cockroach in Japan using the developed teleoperation UI.

2. Proposed Teleoperation System

This section presents a teleoperated cyborg system consisting of Madagascar hissing cockroaches, hardware, and software systems.

2.1. Hardware

Virtual network computing (VNC) was applied as a teleoperation connection system from Bangladesh to Japan. Due to security reasons, the secure shell protocol (SSH) cannot be implemented in the laboratory at Osaka University. Therefore, VNC was chosen to connect the User Interface (UI) on the Raspberry Pi 4, which a remote laptop can access in Bangladesh. A biobot was placed on the experimental testbed with a size of 88.2 cm × 88.5 cm. The proposed overall teleoperated system between Japan and Bangladesh is presented in Figure 1. The Hardware system in Japan comprised a webcam, biobot, wireless receiver transmitter, Raspberry Pi 4, and a high-performance desktop PC. A wireless cage comprising a micro servo motor and a wireless receiver was constructed using transparent plastic material and placed on an experimental testbed. A webcam (Logitech C922n) was placed on the experimental testbed at a distance of 122 cm. The camera was connected to a desktop PC to calculate the position of the biobot. The PC utilized 11th Gen Intel(R) Core (TM) i7-11700 @ 2.50GHz processor and 16 GB of RAM memory. Another web camera is the ELP 1080P Webcam purchased from Amazon Japan. It can acquire a picture with a size of 1280 × 720 pixels with a speed of 120 fps. It was connected to a Raspberry Pi 4 with a memory of 8 GB to display real-time streaming on an experimental testbed. A remote computer in Bangladesh can view it through VNC and a developed User Interface (UI).

By using a remote laptop In Bangladesh, a remote user can observe a live streaming video and provide commands to the biobot and wireless cage through the remote UI. A remote user can send the stimulation command to the cockroach and wireless cage by pressing predetermined keyboard keys. These commands were sent via VNC to the Raspberry Pi single-board computer (SBC) from Bangladesh to Japan. It sent commands to the electronic backpack attached to the cockroach through a wireless transmitter, as shown in Figure 2a. Seeeduino XIAO microcontroller from Seeedstudio was selected as the main controller for the wireless transmitter, wireless receiver, and electronic backpack. The dimension of the controller is 20 × 17.5 × 3.5 mm, making it suitable for a small-size electronic backpack. It utilizes ARM Cortex-M0+ CPU (SAMD21G18) running at up to 48 MHz. A 2.4 GHz wireless module (nRF24L01) was chosen as the two-way wireless communication module.

A 9-DOF inertial measuring unit (GY-955) was integrated into the electronic backpack, as depicted in Figure 2c, to capture the kinematics data of the biobot. A built-in Kalman filter processes the sensor output for estimated attitude angles of roll, pitch, and yaw. The electronic backpack receives stimulation and open/close wireless cage commands from a wireless transmitter connected to the Raspberry Pi. It is connected to the wireless transmitter through serial communication. The electronic backpack sends the estimated attitude angles and commands to the wireless transceiver (Figure 2b). A custom/local UI running on the desktop PC was developed under a Python environment to manage the data acquisition, computer vision, animation, and data recording. The PC reads the data received from the wireless transceiver through serial communication. The wireless transmitter sends open/close command to the wireless cage for opening or closing the gate through a wireless receiver on the cage.

2.2. Biobot

A Madagascar hissing cockroach (MHC) was chosen as the insect cyborg platform. MHC was selected because it can have a length of between 5 cm and 7 cm as an adult. The cockroach was treated weekly by providing new food and cleaning the container. This study utilized platinum wire (A-M systems) insulated with Teflon with a diameter of 0.127 mm as the electrode. We followed high ethical standards during this study. For the anesthetization method, cockroaches were placed in a container containing small chunks of ice. The cockroach was immersed in an ice cube for 30 min. The cockroach slept for 10 to 15 min. The electrodes were implanted on both antennae, both cerci, and the thorax. For easy attachment and detachment of the electronic backpack, a pin header of 2.54 mm (female 5-pin single-row strip) was utilized as the connector between the electronic backpack and implanted electrodes on the antennae, cerci, and thorax. The connector was placed and glued on the first segment of the thorax. The cockroach antenna was cut about 1 cm from the tip for electrode implantation on the antennae. The platinum electrode was inserted into the antenna to a depth of about 5 mm. Then, the antenna and electrode connection were coated with quick-drying glue. The antennae were attached to the pronotum and then glued to prevent cockroaches from releasing the electrodes on the antennae. A copper wire was connected to the platinum electrode for the electrode on the cerci. Another end of the copper wire was connected to the backpack connector. The cerci were cut to about 1 mm from the tip to provide a small opening. Therefore, the platinum electrode could be inserted into the cerci. The connection between the cerci and the platinum electrode was glued to prevent the cockroach from breaking the implanted electrode. The ground electrode was inserted near the midline of the thorax to a depth of about 3 mm. The electrode placements on antennae, cerci, and thorax are presented in Figure 3a. The cockroach was put back into the container to recover and rest for 24 h after surgical implantation. A lithium polymer (LiPo) battery with a 50 mAh capacity was chosen because it is relatively lightweight (1.4 g) and can supply power for more than 50 min. Figure 3b reveals an electronic backpack with a LiPo battery attached to a biobot. The weight of the electronic backpack is 5.2 g. The cockroach can easily carry this weight. Pulse-width modulation (PWM) signal with a 50% duty cycle and frequency of 50 Hz was selected as the electrical stimulation signal. This study used five cockroaches with platinum electrodes implanted.

2.3. Teleoperation User Interface

Two custom user interfaces (UI) were developed to assist the teleoperation system from Bangladesh to Japan. A multithreading algorithm was implemented for real-time computation in both user interfaces. Multithreading algorithm developed under Python programming run both user interfaces on Raspberry Pi and desktop PC in real-time. The overall system of the two user interfaces for the proposed teleoperation system of biobots is summarized in Figure 4. Bue dash–dotted arrow shows the internet connection using VNC, while the local wireless communication is applied using RF communication devices as shown with a green dash–dotted arrow.

2.3.1. Remote User Interface

A remote UI running on the Raspberry Pi 4 in Japan was aimed to remotely display a live streaming video of the experimental cockroach testbed and send stimulation and cage commands from Bangladesh, as presented in Figure 5. Multithreading algorithms were implemented to run three threads concurrently by a threading library in Python programming. The computations for multithreading on the Raspberry Pi are real-time video display, animation display, reading, and sending commands. VNC^® server was implemented to remotely display the UI accessed on a laptop in Bangladesh. A user in Bangladesh accessed the UI using VNC^® Viewer installed on his laptop. The UI was developed using the Tkinter library. A remote user from Bangladesh could provide commands to stimulate the biobot and open/close the wireless cage by pressing specific keys on the keyboard. Key binding in Tkinter was employed to read the user commands remotely. By using this strategy, a remote user from Bangladesh can direct the cockroach to follow a predetermined path and open or close the wireless cage by pressing the keyboard keys from Bangladesh through a remote laptop. This UI manages the internet and VNC for remote communication between Bangladesh and Japan.

2.3.2. Local User Interface

A local user interface on a desktop PC was built to manage teleoperation, computer vision, data acquisition, and plotting purposes. The UI consists of two tabs: teleoperated tab (Figure 6a) and the acquisition tab (Figure 6b). The first tab runs five threads concurrently using a threading library in Python. The threads are data acquisition, image processing, animation, data transmitter, and the data receiver. The first tab sets up the serial communication between PC and wireless transceiver. The electronic backpack sends commands and IMU data to the wireless transceiver connected to the PC. Euler angle estimation results from 9-DOF IMU (roll, pitch, yaw) and commands from Bangladesh (stimulation and open/close cage commands) are received via the wireless receiver and read by the UI via serial communication with a baudrate of 115.200. The animation thread consists of cockroach heading animation, stimulation animation, and wireless cage state (open/closed). The animation is augmented on the first tab to make it easier for a user in Japan to visualize what commands have been given from Bangladesh. The data acquisition process (start, record, stop) can be managed in this table. The UI’s second tab aimed to plot the recorded data acquisition. The recorded data were stimulation commands on the right antenna, left antenna, right cercus, left cercus, and the calculated Euler angles (roll ($\varphi$), pitch ($\theta$), and yaw ($\psi$)). The multithreading algorithm was not implemented in this second table. Recorded data were saved in .csv format because of its simplicity to access and retrieve. Matplotlib library was implemented for data visualization in the second table. The local UI regulates bidirectional local wireless communication between computers and biobots in japan.

2.3.3. Computer Vision

The image processing technique was implemented using OpenCV under Python programming to obtain the cockroach position. The acquired image was processed at 25 frames per second (fps). The image of the cockroach on the experimental testbed was obtained using the first web camera. It was resized to the image size of 760 pixels in width and 555 pixels in height. A Gaussian filter with a kernel size of 3 × 3 is implemented as a low-pass filter. The filtered image was converted to a grayscale image. The canny edge detection method was applied to obtain the cockroach edge on the experimental testbed. Image dilation with a kernel size of 6 × 6 and image erosion with a kernel size of 4 × 4 was applied to acquire a smoother edge image of cockroaches. The result of the edge image was then processed using the “findCountours” function in OpenCV to determine the contour area of the cockroach image. The processed contour area was fitted using the ellipse fitting method. Finally, the position of the center of mass (CoM) of the cockroach can be determined by calculating the center of the ellipse (x_c, y_c). This ellipse center was calculated in the image processing coordinate in pixel units. Equations (1) and (2) were used to calculate the position of the cockroach (x, y) from the image processing coordinate to the cartesian coordinate in the cm scale. The processed image of cockroach position and tracking was displayed on the first tab of local UI in real-time, as shown in Figure 6a. The general block diagram of the real-time image processing is summarized in Figure 7.

$$x (cm)=\frac{Calibrated length in x-axis (cm)}{Pixel width (px)}{x}_c (px)$$

(1)

$$y (cm)=Pixel height (px)-\frac{Calibrated length in y-axis (cm)}{Pixel height (px)}{y}_c (px)$$

(2)

3. Teleoperated Experiment

This study aimed to develop a biobot control method using the proposed teleoperation method between two remote countries. As shown in Figure 8, the experimental testbed was located in Morishima lab, Osaka University, Japan, while the operator is located in Chittagong, Bangladesh. Raspberry Pi 4 was connected to the wired LAN router for a faster internet connection. A semicircle path with an average width of 16.5 cm was drawn for the path following the mission, as shown in Figure 9a. For the finish area, a wireless cage with a dimension of 29 cm in length, 19 cm in width, and 7 cm in height was built using a micro servo motor augmented with the wireless receiver, as revealed in Figure 9b. The cage gate can be opened or closed using the teleoperated command.

Five biobots were used in this experiment for two schemes of teleoperated experiments: path with obstacle and path without obstacle. An obstacle was placed in the middle of the track with a diameter of 10 cm and a height of 5.5 cm. The operator was tasked to direct the cockroach remotely from Bangladesh to follow the path and avoid the obstacle to reach the finish area. Before the cockroach reached the finish area, the operator opened the gate and then directed the cockroach to enter the cage. After the cockroach entered the cage, it was closed by teleoperation command.

An operator from a remote place can send commands to the biobot and the cage by pressing predetermined keys on the keyboard. The keyboard keys for controlling the biobot from a remote laptop are summarized in

Table 1. Keyboard keys for controlling the biobot from a remote laptop in Bangladesh.

Keyboard Keys	Functions
Space bar	Open or close the gate on cage
a	Stimulation on right antenna
s	Stimulation on left antenna
d	Stimulation on right cercus
f	Stimulation on left cercus
up	Stimulation on right and left cerci
left	Stimulation on right antenna
right	Stimulation on left antenna

. Firstly, the biobot was placed at the testing area start point, as shown in Figure 9b. The operator monitored biobot movement on the web camera displayed on the remote UI, which was set in the experimental testbed in Japan. The operator controlled the biobot to follow the semicircle path and placed the cockroach in the wireless cage (finish area). After the biobot was in the wireless cage, the operator closed the gate by the teleoperation command. The experiment succeeded if the operator successfully operated the biobot to follow the semicircle path and placed the cockroach in the wireless cage/finish area. After that, IMU, commands, and computed position data were collected to analyze further.

Table 2. The sampling rate of the proposed teleoperated biobot using a multithreading algorithm.

Subsystem	Sampling Rate
Receive IMU and commands data wirelessly	20 Hz
Send stimulation signals wirelessly	10 Hz
Cockroach and cage animation	10 Hz
Data acquisition	20 Hz
Image processing and recording	25 fps

shows the sampling rate of local UI’s to run parallel computation simultaneously using a multithreading algorithm. Finally, the experiment was performed with and without obstacles. This study needs to investigate whether the operator can overcome the obstacle using the teleoperated command from Bangladesh.

The teleoperation UI applied two wireless communication channels to enable a teleoperation system between Japan and Bangladesh. Internet communication was applied for remote communication between Japan and Bangladesh. The RF device was implemented for local wireless communication between a desktop computer and the biobot in the local area (Japan). The transmission control protocol (TCP) and user datagram protocol (UDP) are widely used in internet communication protocols for the teleoperation of mobile robots [1] and industrial robots [44]. Due to security reasons, these methods could not be implemented in the experiments. Therefore, VNC and the developed teleoperation UI were applied to enable the teleoperation system between Bangladesh and Japan. VNC was selected because it uses the remote frame buffer (RFB) protocol. This protocol is suitable for the proposed teleoperation UI operation under the Python program. Remote UI with VNC provided a live-stream video from a cockroach experiment in Japan and sent remote commands from Bangladesh. Local UI received the IMU data from the biobot and remote commands from Raspberry Pi. The communication configuration implemented in the teleoperated system is presented in Figure 10.

4. Result and Discussion

4.1. Path following Tasks

In this section, biobots were steered remotely from Bangladesh to follow a predetermined path. Two schemes of teleoperated path following missions were tested: path following without and with the obstacle. Ten cockroaches were implanted with the electrode on both antennae and cerci in the initial preparation. After electrode implantation, the cockroaches were put back in the container for 24 h to recover and rest. The antennae with implanted electrodes were put back and glued to the cockroach pronotum to prevent the cockroach from breaking the electrode. Unfortunately, during recovering and resting time for 24 h, two cockroaches broke one of the antennae. Therefore, we could not use these two cockroaches for the experiments, although the implanted antennae were glued on the cockroach pronotum. Eight implanted cockroaches were tested using a developed electronic backpack with electrical stimulation. Only one cockroach could not respond to the given stimulus to one of the antennae of eight cockroaches. However, all cockroaches were successfully stimulated on the left cercus, right cercus, or cerci. From the ten implanted cockroaches, seven cockroaches were ready to use in the initial experiments.

The teleoperated experiments were divided into two sections, i.e., with an obstacle and without an obstacle placed in the middle of the predetermined path. We did not determine the sequence of the two sections/schemes of the teleoperation. Each section/scheme was repeated five times for each cockroach. Between the two sections/schemes, the cockroaches were put back on the container to rest after performing five trials on one of the experimental teleoperated schemes for about 15 min. Based on the teleoperated experiments, three cockroaches (first, second, and fourth) were successfully directed to follow the predetermined trajectory for a path with and without an obstacle. Two cockroaches broke their antenna during the initial teleoperated trial on a path with an obstacle. Therefore, we could not continue to use these two cockroaches for the experiments because one of the antennae from these two cockroaches was broken and could not be fixed. The third cockroach was controlled remotely from Bangladesh on a path with an obstacle, but the cockroach was able to break the antenna during the experiment on a path without an obstacle. The fifth cockroach could be controlled remotely on a path without an obstacle. However, the cockroach broke one of the antennae during the initial experiment on a path with an obstacle. Hence, we could not resume the teleoperation task using the fifth cockroach with an obstacle and the third cockroach without an obstacle.

The local UI wirelessly received the estimated Euler angles and stimulation commands from the electronic backpack attached to the cockroach. It was placed and glued on the first segment of the cockroach thorax. The IMU measures the cockroach thorax Euler angles and sends them to the desktop PC through a wireless transceiver. The IMU outputs zero value for the yaw angle measurement when the cockroach faces the north. All obtained data during one successful mission on a path with an obstacle are presented in Figure 11 and Figure 12. Figure 11 reveals the attitude angle responses concerning the stimulation inputs. Stimulation on the right antenna caused the cockroach to turn left, while stimulation on the left antennae caused the cockroach to turn left. Sometimes the cockroach did not move forward after stimulating the left or right antennae. Stimulation on the left or right cercus or both cerci triggered the cockroach to move forward with a slight turn to the right or left. By implementing this stimulation strategy, a remote operator from Bangladesh can direct the cockroach to follow a predetermined trajectory and avoid an obstacle in front of the cockroach. The position of the cockroach in pixels acquired from the image processing technique is shown in Figure 12.

The processed image of the cockroach trajectory was displayed on the local UI in real-time. The image was processed at a sampling rate of 25 fps. The displayed image of the cockroach trajectory commanded from Bangladesh is depicted in Figure 13. The cockroach is the fourth cockroach in the fifth trial. The photos show that a remote operator successfully directed the cockroach to follow a predetermined trajectory from starting point to the finish area. The cockroach could be controlled to avoid the obstacle in front of it. The overall teleoperated operation for the biobots is presented and discussed in Section 4.1.1 and Section 4.1.2.

We implemented five cockroaches to investigate the repeatability of our proposed teleoperated system. In this experiment, cockroaches were controlled remotely from Bangladesh for two teleoperation schemes. The first scheme was to follow the path with an obstacle and the second one was to direct the cockroach to follow the path and avoid an obstacle until the cockroaches entered the wireless cage. A remote operator opened the wireless cage when the cockroach was in the middle of the path and then closed the cage when the cockroaches entered the cage in the finish area. The teleoperated command was be considered successful if the cockroaches could be directed to follow the path and enter the cage.

4.1.1. Path without Obstacle

A remote operator was tasked to direct four cockroaches (first, second, fourth, and fifth) to follow a predetermined path without an obstacle. The operator monitored the cockroach’s current position on the remote UI. The remote stimulation and open/close the cage commands could be sent by pressing the predetermined keys on a remote laptop, as summarized in Table 1. The teleoperated trial was conducted five times for each cockroach. The teleoperated results for each cockroach are presented in Figure 14.

In the teleoperated experiment without an obstacle, first, second, and fifth cockroaches could be easily directed by a remote operator to follow a predetermined path. These cockroaches were successfully steered without touching both line paths on the right or left side, except for the third trial on the fifth cockroach, which slightly walked outside the left path. Two trials on the second cockroach (second and fifth trials) walked outside the left path. The cockroach ignored the stimulation command on the left antenna to turn right. After that, the cockroach was directed to follow the path and enter the cage. For all trials on four cockroaches, a remote operator opened the wireless cage when the cockroach reached the middle of the path and then closed the cage after the cockroach reached the finish area in the cage.

The second cockroach ignored a few stimulation commands on the right or left antenna. However, it still responded to the stimulation commands on the cerci. After carefully investigating the four cockroaches, we strongly believe that electrode implantation degradation caused the second cockroach to ignore some stimulation commands on the antennae. The cockroach utilized for the teleoperated mission was surgically implanted two weeks before. In two weeks, the cockroach tried to release or break the implanted electrode on the antenna, although certain antennae parts were glued on the pronotum. The second cockroach could be controlled to follow the predetermined trajectory, although few stimulation commands were ignored due to electrode implantation degradation. This degradation was not found on the first, fourth, and fifth cockroaches. We performed teleoperation experiments with these cockroaches one to two days after implanting electrodes. The cockroach almost responded to every stimulation to the left or right antenna by turning left or right. For the electrode implanted on the cerci, the cockroach has difficulty reaching the implanted electrode by using its legs.

Some results show that a remote operator effectively remotely controlled the cockroaches to follow the predetermined path without providing stimulation on the cerci. This motion could happen because the operator applied cockroach free walking motion to direct the cockroach on the path. However, for other trials, cockroaches tended not to move forward with antennae stimulation. Therefore, the operator stimulated the cerci to trigger the cockroach to move forward. After the cockroach moved forward with a slight turn, the operator directed the cockroach to turn left or right by providing stimulation to the antennae. Implementing this stimulation strategy allows the cockroaches to be steered remotely to follow the path and enter the cage.

4.1.2. Path with Obstacle

A cylindrical obstacle with a diameter of 10 cm and a height of 5.5 cm was put in the middle of the path. A remote operator was asked to steer the cockroach remotely from the starting position to the finish area in the wireless cage. The cockroach must avoid and pass the obstacle. After passing the obstacle, the operator opened the cage remotely and closed it after the cockroach had entered the finish area in the cage. This path with an obstacle is a relatively more complicated task than the previous one without an obstacle. Four cockroaches were utilized and repeated five times for each cockroach. The results for all the cockroaches are presented in Figure 15.

It revealed that the cockroaches could be steered to follow the path, avoid the obstacle, and enter the cage, although in some trials, the cockroach passed to the left or right line of the outside path. The first cockroach at the second trial could be directed to avoid the obstacle. However, after passing the obstacle, the cockroach ignored the stimulation on the right antenna to turn right. After passing the outside line of the right path, the cockroach did not ignore the stimulation on the right antenna to turn left and enter the cage. Due to the electrode implantation degradation on the antennae for the second cockroach, the cockroach could pass the outside left line path at the initial teleoperated command. The cockroach did not respond to the right stimulation commands, but the cockroach could be commanded to enter the path and avoid the obstacle. Then the cockroach could be directed to enter the cage. Overall, the cockroaches could be steered remotely to follow the path, avoid the obstacle, and enter the finish area in the cage.

We found that when the cockroach reached near the obstacle, the cockroach tended to circle the obstacle or did not move. In order to prevent this free motion, the operator stimulated the left or right cercus to trigger the cockroaches to move forward with a slight turn to the left or right. After the cockroach was triggered to a forward walking motion, the cockroach was steered to follow the path and enter the finish area in the cage. Based on the teleoperated experiment, the cockroaches did not respond to the stimulation signals on a few occasions, especially on the antennae. Previous research studies also reported that the cockroach did not always respond to the stimulation on the right or left antenna [27,38,45]. An action camera was put on the experimental testbed to record the cockroach’s response. The cockroach was steered to follow a path with an obstacle, as depicted in Figure 16. The figure demonstrates that the cockroach was successfully directed to follow the predetermined path, avoid the obstacle, and enter the cage. By using multithreading, we could run multiple computational processes simultaneously on the developed user interface as performed by previous research. Multithreading can run the embedded machine learning system in real-time [46]. The videos of the teleoperated locomotion for path following without (Video S1) and with obstacle (Video S2) can be seen in Supplementary Materials.

The time required to remotely steered the biobots from the starting position to the final position was measured for path following with and without obstacles. The average time required for each biobot is presented in Figure 17. Based on the results, the time required to steer the biobots is higher than that of biobots without obstacles (p-value = 0.08). This longer time happens because biobots tend to cycle the obstacle. The remote operator needs to control the biobots to avoid obstacles and follow the path until reaching the final position.

4.2. Accuracy for the Input Command

In the previous experiment for path following tasks, almost all cockroaches responded to stimulation given to the left and right antennae. The input commands were sent from Bangladesh, while the cockroach responses were recorded in Japan. On a few occasions, the second cockroach did not respond to the given stimuli on the antennae due to electrode implantation degradation, but all cockroaches responded to the stimuli provided to the cerci. Therefore, the second cockroach was excluded from the repeatability test command. The previous teleoperation commands were utilized to calculate the repeatability command. We recorded and collected command accuracy tests from the previous experiment session. The total number for this test is 120 samples. The test samples were collected from the recorded video of the experiment session for turning right, turning left, and moving forward commands. The stimulation commands were given randomly to the right antenna, left antenna, or cerci. Each command test result has 40 samples, as summarized in

Table 3. Repeatability commands for biobots.

Command	Number of Samples	Correct Movement	No Response	Accuracy ratio (%)
Right	40	35	5	87
Left	40	34	6	85
Forward	40	39	1	95

. Equation (3) is applied to calculate the command accuracy ratio.

$$Accuracy=\frac{Number of correct movement}{Number of total sample}100\%$$

(3)

The average accuracy for turning left, right, and moving forward commanded remotely from Bangladesh is 87%, 85%, and 97%, respectively. The biobots accepted all commands transmitted from Bangladesh to Japan. Delays occurred during data transmission from input commands in Bangladesh to the biobots in Japan. The delay was found in remote transmission using internet communication from Bangladesh to Japan and local transmission using RF devices from a computer to the insect. The computed delay in internet and RF communications is presented in Section 4.3. There is also an acceptable low delay in the live-video stream between Bangladesh and Japan. A remote operator in Bangladesh still successfully steered and monitored the insects to follow a predetermined path with and without obstacles.

4.3. End-To-End Time Delay

A proposed teleoperation system using internet communication can be accessed from Bangladesh for a long-range operation. However, internet communication is also susceptible to delay, data loss, jitter, and communication blackout, as presented by a previous study [47]. A remote operator controlled the biobots from Bangladesh to Japan in pure teleoperation mode. Time delay is one of the most significant factors affecting the performance of remote operations and manipulation in teleoperation systems [48,49]. Delays that occurred on internet and RF communications are presented in this section. The desired commands started and ended in Bangladesh. There are four kinds of delays for the end-to-end time delay: transmission delay, propagation delay, queuing delay, and processing delay. Transmission delay means the time needed to put the information/data bits to the transmission link. Mathematically, transmission delay is directly proportional to the length/size of data packets. Propagation delay is a time delay in transferring the information to the destination when the information is in the transmission link. In this study, queuing and processing delays were not measured for the end-to-end time delay for the teleoperation system. The measured propagation delay of Bangladesh to Japan and Japan to Bangladesh is shown in Figure 18. A command prompt was used to ping Japan to Bangladesh and Bangladesh to Japan with unique internet protocol (IP) addresses.

Transmission delay also occurred in the experimental field (local communication) in Japan during wireless transmission commands from the Raspberry Pi transmitter to the electronic backpack. The transmitter sent stimulation commands to the backpack with a sampling rate of 10 Hz. In order to measure this transmission delay, a step input signal, as shown in Figure 17, was implemented as the transmitted signal. The signal received by the electronic backpack was measured using a serial monitor on Arduino IDE. Based on the obtained transmitted and received signals, the delay from the Raspberry Pi transmitter to the electronic backpack was 0.19 s, as depicted in Figure 19.

The obtained average end-to-end time delay to control the biobot from Bangladesh to Japan is summarized in

Table 4. Average end-to-end time delay to control the biobot.

Destination	Time Delay (ms)
Bangladesh to Japan transmission delay	5
Bangladesh to Japan propagation delay	125
Japan transmission delay	19
Japan to Bangladesh propagation delay	125
Average end-to-end time delay	275

. The table shows that transmission and propagation delay Bangladesh to Japan and Japan to Bangladesh. The calculated average end-to-end time delay of this system was 275 ms. However, the time delay will be different in different circumstances depending on the internet connection speed. The obtained low time delay is still acceptable because the motion of the biobots is relatively slow. Therefore, a remote operator in Bangladesh could steer the cockroach on the path-following mission, as presented in Section 4.1.1 and Section 4.1.2.

4.4. Battery for Electronic Backpack

A Lithium Polymer (LiPO) battery with a capacity of 50 mAh was applied to power the electronic backpack. A digital multimeter was used to measure the battery’s voltage drop. In this test, the Raspberry Pi transmitter sent stimulation commands (50 Hz, 50% duty cycle of 3.3 V PWM signals) to the backpack for 30 s and then sent no stimulation signals for 30 s. The sequence of stimulation signals was repeated for more than 70 min. The backpack was powered with the LiPO battery and sent stimulation commands and IMU data to the wireless transceiver connected to the desktop PC. The developed local UI was utilized to observe the received stimulation commands and IMU data on the desktop PC. The backpack was not attached to the cockroach because too long, and many stimulation commands could damage the cockroach. The voltage drop of the battery was measured every two seconds. Three 50mAh LiPO batteries were used in this test. The results of the measured three batteries are revealed in Figure 20. The fully charged battery was 4.15 V. From the tested three batteries, the batteries could not supply the backpack if the voltage was below 2.50 V. Based on the test result, the batteries can provide power to the backpack from 52 min to 62 min.

5. Conclusions

In this research, we successfully developed teleoperation software and hardware for insect-based biobots that can be monitored and controlled from Bangladesh to Japan. The multithreading algorithm plays an important role in performing multiple parallel computations concurrently on the teleoperation UI. Stable remote communication between Bangladesh and Japan was performed using a developed remote UI equipped with VNC. Bidirectional RF devices provided local communication between electronic backpack and desktop PC. By using the proposed interactive and intuitive UI, a remote operator from Bangladesh successfully steered the biobots to follow a predetermined path from starting position to the finish area in the cage. In a few trials, the cockroach could walk on the outside path line, but the operator was able to direct the cockroach to walk on the path and enter the cockroach into the cage. The remote operator simply directed the cockroach by pressing the keyboard keys like playing a game. The developed electronic backpack could last for around 52 min. The average time for the end-to-end time delay on the proposed system between Bangladesh and Japan was 0.275 s. Based on the experiment results, the developed system is a potential and reliable system for the biobot teleoperation system for future search and rescue missions between two countries. In the future, a mini wireless camera will be developed and equipped for biobots to support teleoperated search and rescue missions.