EMG-Based Dynamic Hand Gesture Recognition Using Edge AI for Human–Robot Interaction

Abstract

Recently, human–robot interaction technology has been considered as a key solution for smart factories. Surface electromyography signals obtained from hand gestures are often used to enable users to control robots through hand gestures. In this paper, we propose a dynamic hand-gesture-based industrial robot control system using the edge AI platform. The proposed system can perform both robot operating-system-based control and edge AI control through an embedded board without requiring an external personal computer. Systems on a mobile edge AI platform must be lightweight, robust, and fast. In the context of a smart factory, classifying a given hand gesture is important for ensuring correct operation. In this study, we collected electromyography signal data from hand gestures and used them to train a convolutional recurrent neural network. The trained classifier model achieved 96% accuracy for 10 gestures in real time. We also verified the universality of the classifier by testing it on 11 different participants.

1. Introduction

Robot remote control has been studied in various fields, such as space exploration by robots [1,2,3], remote firefighting mobile robots for firefighting tasks at fire sites [4,5], military robots used in battlefields [6,7], and disaster rescue robots used at disaster sites [8,9,10]. Additionally, there are surgical robots [11], construction robots [12], and disability assistance robots that increase human work capacity [13]. Despite the development of robots with good mobility and work capabilities, it is difficult for robots to identify interactive scenarios and human intentions.

Therefore, research on human–robot interaction (HRI) technology that recognizes human information, judgments, and expressions has been actively pursued in recent years. HRI technology consists of recognition, judgment, and expression technologies for achieving communication between humans and robots. Various studies have used recognition technology to allow robots to detect human intentions based on face recognition, motion recognition, voice recognition, and hand gesture recognition. Additionally, studies have been conducted on specific HRI applications using facial image recognition [14,15], motion recognition, robot remote control systems based on skeletal data [16], autonomous driving robots based on voice recognition [17], and robotic hand control systems using hand gesture recognition [18].

Hand gesture recognition technologies are categorized into dynamic and static hand gestures. Static hand gestures are easier to recognize because they have little to no movement. However, they can be limited in expressiveness and are far from natural hand movements. Dynamic hand gestures can capture a wider range of natural hand gestures but can be more complex and difficult to classify due to the pattern changes during the gesture. In studies on controlling industrial robots, hand gesture recognition technology is mainly used as a recognition technology for communication based on the similarity between robot arms and human arms. Such a control scheme is intuitive and is used in various fields, including medicine, engineering, and rehabilitation. Hand gesture recognition technology has been applied in studies on controlling a six-degree-of-freedom robot arm [19], virtual reality interaction [20,21], and message notification for patient care systems [22].

Studies on hand gesture recognition have been conducted using electromyography (EMG) signals, which are a type of biosignal. EMG signals are microscopic electrical signals that can be obtained from muscular activity and are widely used in various fields, including robotics and rehabilitation medicine. However, hand gesture patterns must be learned because obtained EMG signals are not always the same, even for the same motion. Based on the recent development of AI technology, it performs excellently on various recognition tasks. Popular models include the convolutional neural network (CNN) and recurrent neural network (RNN). Various studies have been conducted on learning EMG-based hand gestures using CNNs [19,23], RNNs [24,25], and convolutional recurrent neural networks (CRNNs) [26,27]. A CNN is an artificial neural network inspired by the concepts of the visual cortex of the brain. CNNs are mainly used for image deep learning and have achieved excellent performance. An RNN is another type of artificial neural network that excels at processing time series data. Unlike a CNN, an RNN recursively refers to previous states when calculating future states. Long short-term memory (LSTM) has been proposed to solve long-term dependency issues associated with forgetting previous information when the time step grows. Additionally, the gated recurrent unit (GRU) is a high-speed model with performance similar to that of LSTM that simplifies the cells of the time steps constituting LSTM networks. A CRNN is a high-performance classification neural network that combines the feature extraction capabilities of a CNN with the classification capabilities of an RNN for time-series data. In the structure of a CRNN, input data pass through multiple convolutional layers, activation functions, and pooling layers. The resulting flattened features are inputted into an RNN layer, and the class with the highest probability is the output.

Although there have been many studies on hand gesture classification using deep learning, few studies have utilized a universal deep learning model to classify dynamic hand gestures and control robots. This paper proposes a system for dynamic hand gesture recognition on an embedded board using a deep learning model and the control of an industrial robot using a robot operating system (ROS). The proposed system uses a model that learns EMG signals using a CRNN structure. This model can recognize 10 hand gestures and works for users who have not participated in training. Figure 1 presents the configuration of the proposed system.

The main contribution of this study is to propose a novel classifier for dynamic hand gestures using CRNN based on EMG. As shown in [28], there are many studies on the classification of static hand gestures, but there is still a lack of research on the classification of dynamic hand gestures using EMG. Therefore, to classify the dynamic hand gestures, we develop a classifier that includes the CRNN structure trained by EMG data collected from a forearm. To verify the performance of the proposed classifier, we experiment with a gesture classification based on a group that did not participate in the collection of training data, and then the experiment results represent high accuracy compared to other papers. Furthermore, the proposed classifier is implemented on the embedded system in a ROS-based robot control system, resulting in the effective control of an industrial robot by dynamic hand gestures without requiring an external PC.

The remainder of this paper is organized as follows. Section 2 presents the configuration of the proposed system and the hardware components used in the system. Section 3 describes the proposed edge AI system, including defined hand gestures and control schemes, EMG data collection, and training and testing using CRNN structures. In Section 4, we present hand gesture classification and industrial robot control experiments to demonstrate the effectiveness of the proposed method. Finally, Section 5 concludes this paper.

2. System Configuration

2.1. Robot Arm and Gripper

The industrial robot considered in this study is the smallest UR3 robot in the Universal Robot CB3 series. The UR3 robot arm is a cooperative robot arm with a collision detection function. The UR3 robot consists of a six-axis joint robot arm with joints labeled as “base”, “shoulder”, “elbow”, “wrist 1”, “wrist 2”, and “wrist 3”, as shown in Figure 2. As shown in

Table 1. Specifications of the UR3 CB-series.

Technical Specifications of the UR3
Weight	11 kg/24.3 lbs
Payload	3 kg/6.6 lbs
Reach	500 mm/19.7 in
Repeatability	±0.1 mm/±0.0039 in
Degrees of freedom	6 rotating joints

, the maximum payload and maximum working radius are 3 kg and 500 mm, respectively. As shown in

Table 2. Working range and speed of the UR3 robot joints.

Robot Arm Joints	Working Range	Maximum Speed
Base	$\pm 360°$	$\pm 360°/$s
Shoulder	$\pm 360°$	$\pm 360°/$s
Elbow	$\pm 360°$	$\pm 360°/$s
Wrist 1	$\pm 360°$	$\pm 360°/$s
Wrist 2	$\pm 360°$	$\pm 360°/$s
Wrist 3	Infinite	$\pm 360°/$s

, the operating range of all joints is ±360°, excluding wrist 3, which can be rotated infinitely. The Universal Robot ROS Driver was installed to provide users with a stable interface between the UR robot and ROS.

As shown in Figure 3, the gripper consists of two fingers, each of which is 140 mm wide. The Robotiq ROS package was installed to control the gripper in the ROS environment.

2.2. Myo Armband Sensor

In this study, we used a commercial EMG sensor called the Myo gesture control armband, as shown in Figure 4, which presents the measurements of muscle integrity. The Myo armband is a lightweight and inexpensive product, so many studies on EMG have used this device. It also supports the desired ROS package, making it suitable for use in this study. The device has eight surface EMG electrodes, a nine-axis inertial measurement unit composed of a three-axis accelerometer, three-axis gyroscope, and three-axis magnetometer, and a Bluetooth module for transmitting EMG data. The Myo armband has a notch filter that is used to perform denoising at a frequency of 50 Hz. It is a gesture-based human interface device developed by Thalmic Labs. When wearing the Myo armband on the forearm, samples are obtained at a rate of 200 Hz. The ros_myo ROS package was installed to enable ROS communication of the data obtained from the Myo armband. The ros_myo ROS package uses a sampling frequency of 50 Hz.

2.3. Embedded Environment

The embedded board used in the proposed system is an NVIDIA Jetson Nano developer kit. The Jetson Nano is designed to support entry-level edge AI applications and devices. It also includes accelerated libraries for deep learning, computer vision, graphics, multimedia, etc. We installed image files, including the NVIDIA JetPack 4.6 SDK based on Ubuntu 18.04, to construct accelerated AI applications. For ROS communication, it was connected to the internet using a wireless LAN adapter Wi-Fi dongle. The CRNN model of our hand gesture classifier was trained using Python 3.6 in the TensorFlow GPU 2.4.0 environment.

3. Methods

3.1. Hand Gestures

The proposed system uses EMG signal data based on hand gestures as inputs. Ten gestures that can intuitively express the movements of industrial robots, grippers, and user intentions were defined. The defined gestures are newly defined gestures, separate from the five classes of gestures recognized by the Myo armband by default. Additionally, as shown in Figure 5, the gestures were defined by considering the reflection of human intentions, differences in hand gestures, and ease of hand gestures. The defined hand gestures are “Close”, “Open”, and “Fist” for gripper control, “Right”, “Left”, “Thumb up”, “Thumb down”, and “Supination”, “Pronation”, for industrial robot arm control, and “Rest” as the default state for classifying hand gestures. All gestures are performed with the right hand. Start at “Rest”, perform a specific action, and then return to “Rest” for 1 s. For “Rest”, hold “Rest” for 1 s.

“Rest” does not control anything on its own but serves as a base state. “Close”, which is the gesture of holding an object, causes the gripper to close. “Open”, which is a spreading gesture, causes the gripper to open. “Fist”, which is the gesture of clenching a fist, causes the gripper to close half way. “Right”, which is the gesture of pointing to the right with the index finger, causes the base of the industrial robot to turn clockwise. “Left”, which is the gesture pointing to the left with the index finger, causes the base of the industrial robot to turn counterclockwise. “Thumb up”, which is the motion of extending the thumb upward, causes the shoulder of the industrial robot to turn clockwise. “Thumb down”, which is the motion of extending the thumb downward, causes the shoulder of the industrial robot to turn counterclockwise. “Supination”, which is the supination of the forearm, causes the wrist 3 joint of the industrial robot to turn clockwise. “Pronation”, which is the pronation of the forearm, causes the wrist 3 joint of the industrial robot to turn counterclockwise.

3.2. Data Acquisition

EMG data are microscopic electrical signals that can be obtained from muscle activity. EMG data are collected when wearing the Myo armband, as shown in Figure 6. One hand gesture operation (one set) was collected in the form of 400 EMG data from 50 samples of 8 channels over 1 s. The 400 EMG data were normalized using min-max normalization and then reshaped into a two-dimensional form with dimensions of 50 × 8 for deep learning model training. Figure 7 illustrates how the EMG data collected for each gesture were normalized and plotted. EMG measurements of different magnitudes corresponding to different hand gestures are the main characteristics of the data.

Table 3. One set of normalized EMG data.

	Ch.1	Ch.2	Ch.3	Ch.4	Ch.5	Ch.6	Ch.7	Ch.8
1	0.00996	0.00996	0.07597	0.03238	0.01308	0.00685	0.0056	0.00623
2	0.01557	0.13574	0.06476	0.02428	0.01619	0.01308	0.04608	0.03674
3	0.01494	0.18991	0.06538	0.02304	0.01557	0.01059	0.04483	0.02491
4	0.01557	0.18493	0.06227	0.01806	0.0137	0.00996	0.04359	0.01681
∼
48	0.02242	0.16065	0.08219	0.02491	0.01059	0.00934	0.01121	0.01121
49	0.01743	0.16999	0.08468	0.02242	0.00996	0.00809	0.00934	0.00872
50	0.01681	0.17933	0.09278	0.0193	0.00934	0.00809	0.00747	0.00809

presents one set of EMG data. The data were collected from seven subjects (five males and two females) to create a universal classifier that can be used by both men and women. When collecting data for the ten hand gestures, each subject repeated each hand gesture 500 times. Of the 500 datasets collected, 400 and 100 were used as the training and validation data, respectively. Overall, 2800 training data were collected for each hand gesture, resulting in a total of 28,000 training data. We collected 700 validation data for each hand gesture, resulting in a total of 7000 data for validation.

3.3. Proposed CRNN

A CRNN is a high-performance classification neural network that combines the feature extraction capabilities of a CNN with the time series classification capabilities of an RNN. The proposed CRNN structure is presented in Figure 8. The input data are normalized EMG data. The convolutional layer consists of a 3 × 3 convolution filter. The pooling layer is a max-pooling layer with dimensions of 2 × 2. As the neural layers progress, the number of filters increases to 16, 32, and 64 to extract complex features. This structure is repeated three times to maintain the feature map size within a valid range. The extracted features are inputted into the fully connected layer. In the fully connected layer, a multidimensional array is flattened into a one-dimensional array and then inputted into the GRU layer. Finally, the data pass through a dense layer consisting of a rectified linear unit activation function and Softmax function, which output values for the 10 classes. Training was performed for 100 epochs, the learning parameters were set to 171 and 326, and the training data were shuffled. Figure 9 presents the training accuracy and loss according to the number of training epochs of the CRNN model. When the CRNN model is trained for 100 epochs, the accuracy increases and the loss decreases. Regarding the training results, the accuracy is 1.0 and the loss is 0.0247.

There are several methods available for checking the correctness of AI-based solutions. Such methods include formal verification, adversarial training, and abstract interpretation [29,30,31]. We evaluated the performance of our deep learning model using a confusion matrix. A confusion matrix is a measure that facilitates easy evaluation of how often an AI model confuses different classes. The values in a confusion matrix can be used to calculate performance metrics for deep learning model classification evaluation, including accuracy, precision, recall, and F1 score.

4. Experiments and Results

Our hand gesture classification and industrial robot control experiments were conducted in two ways. One experiment was conducted on the subjects from which training data were collected, and another experiment was conducted on other subjects for whom no training data were collected. Both experiments were conducted with the system configuration shown in Figure 1. Both experiments were conducted using the hand gesture classifier model trained on the EMG data from the first group of participants.

4.1. Subjects from Whom Data Were Collected

In the first experiment, the subjects from whom training data were collected were tested. Each subject performed each gesture 50 times, and the seven subjects performed a total of 3500 trials. The confusion matrix in Figure 10 represents the classification results for all the subjects in the first experiment. In this confusion matrix, each row and column represent predicted and actual values, respectively. The individual confusion matrices of the subjects can be found in Figure A1, Figure A2, Figure A3, Figure A4, Figure A5, Figure A6 and Figure A7 in Appendix A.

These figures reveal high classification rates for subjects 1 (97.0%), 2 (95.4%), 3 (96.6%), 4 (96.2%), 5 (97.0%), 6 (96.6%), and 7 (97.2%). The highest and lowest classification rates were obtained for subjects 7 (97.2%) and 2 (95.4%), respectively. The hand gesture classification and industrial robot control experiment yielded a high classification rate of 96.57% for all subjects participating in learning. The hand gesture with the highest classification rate was “Rest” (100%), while that with the lowest classification rate was “Thumb down” (94%). The hand gesture classifier in the proposed system extracted features accurately and achieved high classification performance for the EMG signals. It is also insensitive and robust to EMG signal variation.

4.2. Subjects from Whom No Data Were Collected

In the second experiment, the subjects from whom no training data were collected were tested. Each subject performed each gesture 50 times, and the four subjects performed a total of 2000 trials. The confusion matrix in Figure 11 presents the classification results for all the subjects in the second experiment. In the confusion matrix, each row and column represent predicted and true values, respectively. The individual confusion matrices of the subjects are presented in Figure A8, Figure A9, Figure A10 and Figure A11 in the Appendix A.

These figures reveal high classification rates for subjects 8 (95.8%), 9 (93.6%), 10 (95.8%), and 11 (95.2%). The highest classification rates were obtained for subjects 7 and 10 (95.8%), while the lowest classification rate was obtained for subject 9 (93.6%). The results of the hand gesture classification and industrial robot control experiment involving the subjects who did not participate in learning yielded a high classification rate of 95.10%. The hand gestures with the highest and lowest classification rates were “Rest” (100%) and “Pronation” (90%), respectively. The hand gesture classifier used in the proposed system exhibited high performance, even for subjects who did not participate in learning. This demonstrates that the hand gesture classifier is a transferable model that can learn the hand gesture data of multiple people, including men and women.

4.3. Comparisons to Previous Studies

Table 4. Comparisons to previous studies. (X: Not used, O: Used).

	Input	Method	Gesture	Subjects (Acquisition/Test)	Performance	Field	Edge AI
[32]	Image	DCNN	10	1/1	98%	Gesture recognition	X
[33]	Image	CNN	10	1/1	84.5%	Robotic arm control	X
[19]	EMG	CNN	7	-/18 (11 men and 7 women)	93.14%	Robotic arm control	X
[26]	EMG	CRNN	6	6/6	92.5%	Gesture recognition	O
[34]	EMG	K-NN	6	-/10	86.0%	Gesture recognition	O
[35]	EMG	FNN	6	120 (90 men and 30 women)/ 60 (45 men and 15 women)	96.87%	Gesture recognition	X
[36]	EMG	ANN	6	12/12	98.7%	Gesture recognition	O
[24]	EMG	RNN	17	-	86.7%	Gesture recognition	X
[41]	EMG	SKNN	6	4/4	95.83%	Gesture recognition	X
[37]	EMG & IMU	Adaptive LS-SVM	7	18 (12 men and 6 women)/22	92.9%	Gesture recognition	X
[38]	EMG	CRNN	5	7/12	84.2%	Robot arm contorl & Gesture recognition	X
[39]	EMG & IMU	MLP	10	20 (15 men and 5 women)/6	78.94%	Gesture recognition	X
[40]	EMG	ViT	23	6/7	97%	Gesture recognition	X
Proposed method	EMG	CRNN	10	7 (5 men and 2 women)/ 11 (8 men and 3 women)	96.04%	Robotic arm control	O

compares our results to those of previous studies in terms of inputs, methods, numbers of gestures, numbers of data collection and testing subjects, performance, field of application, and edge AI. Images were used as inputs in [32,33], while the other studies used EMG signals as inputs. Compared to [19,26,34,35,36], which used EMG signals as inputs, we achieved superior performance. Previous studies [37,38,39,40] used different deep learning models and EMG data to classify hand gestures. These studies tested the universality of their models by evaluating their accuracy on a group not involved in the data collection. The reported test accuracies in these studies were either lower or comparable to the accuracies achieved in this paper. In this study, an overall accuracy of 96.04% was obtained experimentally for seven participants who participated in learning and four participants who did not. Furthermore, compared to [19,24,26,32,33,34,35,36,40,41], our study contributes more to the development of edge AI systems.

5. Conclusions

We proposed a dynamic hand-gesture-based industrial robot control system using an edge AI platform and CRNN. The proposed system utilizes an edge AI system that can remotely control industrial robots using hand gestures, regardless of location. Embedded systems receive EMG signals collected by a Myo armband. A CRNN is used as a hand gesture classification model based on EMG data in the proposed system. The proposed system remotely controls industrial robots and grippers based on an ROS. The performance of the hand gesture classifier was evaluated through two experiments. The results of the first experiment revealed a high classification rate (96.57%) for subjects who participated in learning. The hand gesture classifier was insensitive to variations in EMG signals and exhibited robust classification performance. The results of the second experiment revealed a high classification rate (95.10%), even for subjects who did not participate in learning. Therefore, our hand gesture classifier is a transferable model that is applicable to both men and women. However, using only a single EMG sensor limits the number of muscles that can be monitored, and deep learning models have the disadvantage of requiring a large number of parameters, even for a basic structure. In the future, to classify more complex movements, we plan to conduct a study using two Myo armbands on the forearm and upper arm. Additionally, we will apply a network-in-network structure to the deep learning structure we developed to make it more lightweight.