Advanced Machine Learning for Gesture Learning and Recognition Based on Intelligent Big Data of Heterogeneous Sensors

Abstract

With intelligent big data, a variety of gesture-based recognition systems have been developed to enable intuitive interaction by utilizing machine learning algorithms. Realizing a high gesture recognition accuracy is crucial, and current systems learn extensive gestures in advance to augment their recognition accuracies. However, the process of accurately recognizing gestures relies on identifying and editing numerous gestures collected from the actual end users of the system. This final end-user learning component remains troublesome for most existing gesture recognition systems. This paper proposes a method that facilitates end-user gesture learning and recognition by improving the editing process applied on intelligent big data, which is collected through end-user gestures. The proposed method realizes the recognition of more complex and precise gestures by merging gestures collected from multiple sensors and processing them as a single gesture. To evaluate the proposed method, it was used in a shadow puppet performance that could interact with on-screen animations. An average gesture recognition rate of 90% was achieved in the experimental evaluation, demonstrating the efficacy and intuitiveness of the proposed method for editing visualized learning gestures.

1. Introduction

The development of technologies, which are based on intelligent big data such as virtual reality and invoked reality, have contributed to the increase in research interest in natural user interfaces (NUI) and natural user experience (NUX). Gesture-based systems maximize immersion by allowing end-users to intuitively manipulate contents. Numerous applications recognize end-user gestures such as the ability to remotely control a television using hand gestures or to activate stage effects triggered by body gestures. The successful implementation of applications based on gesture recognition requires a high gesture recognition accuracy, which requires defining and learning end-user gestures by immense collected gestures in advance. However, obtaining adequate gesture recognition accuracies is difficult because of the constraints imposed by different system components. First, each sensor has a limited sensory range in terms of capturing and measuring body parts in motion, reducing the accuracy possible for capturing complex gestures. Second, the end-users encounter a difficulty when they attempt to integrate their own gestures into the systems. For example, during the learning process applied to incorporate actual end-user gestures. Non-experts often face difficulty in gathering training gestures and learning gestures because this process requires a certain degree of programming knowledge.

In this study, a generic gesture recognition and learning framework is developed, which utilizes heterogeneous sensors, and enables end-users to modify their gestures based on intelligent big data. There are disadvantages associated with the use of heterogeneous sensors such as data synchronization and data fusion. However, it is difficult to accurately identify gestures without using heterogeneous sensors because of the limited range and recognition ability of each sensor. More complex gestures can be recognized using multiple sensors. Therefore, a suitable approach for utilizing heterogeneous sensors to increase the gesture recognition accuracy must be identified. In addition, for heterogeneous sensors, the most popular and low-cost sensors such as Kinect, Myo, and Leap Motion must be integrated for a better utilization of the proposed method. The approach must enable end-users to learn and recognize their specific gestures. To recognize gestures correctly, they must be learned by applying an appropriate learning algorithm using good training gestures. If the obtained results are not satisfactory, the end-user must be able to re-edit the learning data.

Therefore, a novel gesture recognition and learning method was developed and experimentally evaluated. The method comprises four end-user interfaces (UIs): Body selection, gesture learning, gesture editing, and gesture recording.

The remainder of this paper is organized as follows. In Section 2, the studies conducted on gesture learning and recognition are reviewed. Section 3 describes the structure of the proposed generic gesture recognition and learning framework. In Section 4, the implementation of the gesture learning and recognition process is described. Section 5 presents the experimental methods, results, and analysis of those results. Finally, Section 6 presents the conclusions of the study and the scope for future work.

2. Related Works

This section reviews the recent studies conducted in the field of gesture recognition. Moreover, active developments in gesture recognition techniques have led to the recent developments in virtual reality (VR) technology [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]. The gesture recognition accuracy, which is a key metric to evaluate the effectiveness of gesture recognition systems, has witnessed steady improvements using input data from multiple sensors [17,18,19,20,21,22]. Moreover, researchers have studied visual recognition and learning systems for more intuitive learning of gesture recognition. In particular, gesture recognition and learning using multiple sensors is being studied actively.

For example, Ibañez et al. [23] proposed a Kinect-based gesture recognition tool that uses only the skeleton generated by Kinect as input data, and it can visualize the results. However, this tool cannot support a variety of other sensors or perform partial recognition tasks. Further, the editing of the obtained results is not supported by the tool. Signer et al. [24] proposed a 2D-data-based gesture recognition tool with gesture registration and learning features; this tool also supports visualization and simple editing tasks. Zhang et al. [25] proposed a gesture recognition framework that uses electromyography and gyroscope data and can learn from these two datasets simultaneously. The proposed framework utilizes the hidden Markov model (HMM) algorithm as its learning algorithm. Truong et al. [26] proposed a 3D gesture recognition framework using Laban movement analysis and HMM models. Ma et al. [27] proposed an enhanced HMM model that could recognize handwritten characters in real time. In addition, the framework classified gestures into long-term/short-term gestures and dynamic/static gestures. Borghi et al. [28] also proposed a gesture recognition algorithm, which was based on the HMM algorithm, using 3D skeleton data. The proposed method divides the skeleton into upper, lower, left, and right parts and recognizes the corresponding gestures. Suma et al. [29] proposed a Kinect-based gesture recognition tool that included a network communication structure, and it could visualize end-user gestures. The learned gestures could be applied in game control using the communication system. However, this tool could not learn or recognize gestures. Gillian et al. [30] developed a gesture recognition library based on C++ that had high universality, and it supported learning and recognition and various types of input data. However, it did not offer editing features for visualizing and learning data. Gris et al. [31] used Kinect to develop a gesture recognition tool that could collect data by specifying the body parts required for the gesture. However, the tool only supported the recognition of static gestures. Maes et al. [32] proposed a recognition system based on learning gestures that enabled users to practice dancing. However, the proposed system only focused on learning and recognition, and it did not include a module to edit the data used to learn the gesture. Yavşan et al. [33] proposed an interface for gesture recognition and an NAO robot control using Kinect. In the proposed method, the learning and recognition experiments were conducted using K-nearest neighbor (K-NN) and feed-forward neural networks (FNNs). However, only static gestures were recognizable, and the system lacked editing capabilities for learning data.

In this study, we develop a method wherein various gestures are precisely learnt by collecting them from multiple sensors. This method offers GUI-based editing features in which even non-experts can easily edit the collected gestures. Moreover, the learning accuracy can be improved through the collection of static and dynamic gesture data and the editing of this collected data. After reviewing the visual learning results, the learning data can be re-edited to achieve better recognition. Further, the learning results can be applied to interactive systems such as smart TVs through a communication network.

3. Generic Gesture Learning and Recognition Framework

The proposed generic gesture recognition and learning framework is designed to obtain and learn various gestures based on sensing data by multi-sensor fusion. In this framework, end-users can define customized gestures by performing the corresponding gestures, which are learnt by the framework to recognize user-specific gestures. Moreover, the defined gestures can be edited after the end user analyzes the accuracy of the recognized gesture.

3.1. Overview of the Proposed Generic Gesture Recognition and Learning Framework

The proposed framework comprises five stages: Gesture registration, editing, learning, recognition, and transfer. Figure 1 illustrates the working of the proposed generic gesture recognition and learning framework.

First, in the gesture registration stage, the proposed method obtains gesture data from the multi-sensor fusion. Using two or more sensors, various types of gestures can be learnt and precise and accurate gesture recognition can be realized [17,18,19,20,21,22]. Moreover, the proposed method allows the end user to limit the range of gestures to be registered for recognizing specific body parts, thereby improving the recognition accuracy. Second, if the learnt gestures are not found to be satisfactory during the gesture editing stage, the end user can edit the collected gestures using the GUI editor. Thus, non-experts can easily define and learn various gestures. Third, during the gesture learning stage, the end user can also select the most appropriate learning algorithm. Subsequently, during the gesture recognition and gesture transfer stages, the learnt data can be applied to various interaction systems. In short, the proposed framework can realize accurate gesture recognition and learning through multi-sensor data fusion and gesture data.

3.2. Gesture Registration Stage

The proposed framework obtains gestures from Kinect, Myo, and Leap Motion. The gesture formats of Kinect, Myo, and Leap Motion are shown in

Table 1. Data format for each sensor.

Sensor	Data Format
Kinect	Skeleton orientation (x, y, z)
	Depth image (d)
	RGB image (r, g, b)
Myo	Electromyography ($e_0- e_7$)
	Gyroscope (= orientation) (x, y, z)
	Angular speed (s)
Leap Motion	Skeleton orientation (x, y, z)
	Depth image (d)

. The gesture registration stage consists of a module for selecting the relevant body part and a gesture recording module to resister a new gesture.

The body part selection module selects the body parts by taking the recognition accuracy into consideration. For example, when the end-user wants for the proposed method to learn gestures using the right arm, the gestures identified from body parts other than the right arm are excluded, and only the relevant gestures are learnt. Figure 2 shows six body parts recognized by Kinect, four body parts recognized by Myo, and two body parts recognized by Leap Motion.

In the proposed framework, human gestures, expressed using arms and legs, are recognized based on the movement of the twelve learning body parts: Head, trunk, left arm, right arm, left leg, right leg, lower left arm, lower right arm, upper left arm, upper right arm, left hand, and right hand. Mathematically, let $B$ be the set of all body parts, $B=\left\{b_1,b_2,\cdots , b_{12}\right\}$; let $B_t^{*}$ be the subset of $B$, i.e., body parts selected by the end-user at time $\mathrm{t}$, $B_t^{*}\in B$.

The gesture recording module obtains gestures from the sensor values of Kinect, Myo, and Leap Motion. Given that the sensor values in the proposed method consider only the orientation of the user, the values of each sensor values at time t are expressed by $s_{i,j,t}=\left\{x_{i,j,t},y_{i,j,t},z_{i,j,t}\right\}$, where $\mathrm{i}$ is the index of a body part in $B$, and j is the index of the sensors of the $i^{th}$ body part. Let $S$ be the set of all sensor values, and $S^{*}{}_t$ be the set of all sensor values for the selected body parts at time $\mathrm{t},B_t^{*}$. $s_{i,j,t}\in S$. $S^{*}{}_t=\left\{s_{1,1,\mathrm{t}},s_{1,2,\mathrm{t}},\cdots \right\}\cup \left\{s_{2,1,\mathrm{t}},s_{2,2,\mathrm{t}},\cdots \right\}\cup \cdots \}$. $S^{*}{}_t\subset S$.

Table 2. Available joints acquired by each sensor.

Device	Body part (Symbol)	Joints (Symbol)
Kinect	Trunk (b1)	Base spine (s1,1), Mid spine (s1,2), Shoulder spine (s1,3)
	Head (b2)	Neck (s2,1), Head (s2,2)
	Left arm (b3)	Left shoulder (s3,1), Left elbow (s3,2), Left wrist (s3,3), Left hand (s3,4), Left hand tip (s3,5), Left thumb (s3,6)
	Right arm (b4)	Right shoulder (s4,1), Right elbow (s4,2), Right wrist (s4,3), Right hand (s4,4), Right hand tip (s4,5), Right thumb (s4,6)
	Left leg (b5)	Left hip (s5,1), Left knee (s5,2), Left ankle (s5,3), Left foot (s5,4)
	Right leg (b6)	Right hip (s6,1), Right knee (s6,2), Right ankle (s6,3), Right foot (s6,4)
Myo	Lower left arm (b7)	Lower left arm (s7,1)
	Upper left arm (b8)	Upper left arm (s8,1)
	Lower right arm (b9)	Lower right arm (s9,1)
	Upper right arm (b10)	Upper right arm (s10,1)
Leap Motion	Left hand (b11)	Palm (s11,1), Wrist (s11,2), Thumb distal (s11,3), Thumb intermediate (s11,4), Thumb proximal (s11,5), Thumb metacarpal (s11,6) Index distal (s11,7), Index intermediate (s11,8), Index proximal (s11,9), Index metacarpal (s11,10) Middle distal (s11,11), Middle intermediate (s11,12), Middle proximal (s11,13), Middle metacarpal (s11,14) Ring distal (s11,15), Ring intermediate (s11,16), Ring proximal (s11,17), Ring metacarpal (s11,18) Pinky distal (s11,19), Pinky intermediate (s11,20), Pinky proximal (s11,21), Pinky metacarpal (s11,22)
	Right hand (b12)	Palm (s12,1), Wrist (s12,2), Thumb distal (s12,3), Thumb intermediate (s12,4), Thumb proximal (s12,5), Thumb metacarpal (s12,6) Index distal (s12,7), Index intermediate (s12,8), Index proximal (s12,9), Index metacarpal (s12,10) Middle distal (s12,11), Middle intermediate (s12,12), Middle proximal (s12,13), Middle metacarpal (s12,14) Ring distal (s12,15), Ring intermediate (s12,16), Ring proximal (s12,17), Ring metacarpal (s12,18) Pinky distal (s12,19), Pinky intermediate (s12,20), Pinky proximal (s12,21), Pinky metacarpal (s12,22)

represents available joints acquired by each sensor.

When the gesture recording module records and stores a gesture, the gesture at time t, g_t, includes $B_t^{*}$ and $S_t$ at time $\mathrm{t}$, $g_t=\left\{B_t^{*},S_t\right\}.$ Although the proposed method stores $S_t$, $S^{*}{}_t\subset S_t\subset$S, during the gesture registration stage, the learning model learns and recognizes only $S^{*}{}_t$ during the gesture learning and recognition stage in order to reduce the processing time required. Figure 3 illustrates the visual relationship between these notations and the gestures collected by Kinect, Myo, and Leap Motion sensors.

For multi-sensor data fusion, the framework obtains sensor values using different devices such as Kinect, Myo, and Leap motion. Therefore, the proposed method is susceptible to port collision when data is collected from various gesture recognition sensors through the communication model. To address this susceptibility, a port assignment module is designed that assigns a unique ID to each sensor such that the sensor can receive data from different ports, as shown in Figure 4.

As shown in Table 1, the sensor value type of each sensor is unique. However, in this framework, only the skeleton orientation information is used for training.

3.3. Gesture Editing Stage

The gesture editing stage comprises the visualization module and collected gesture editing module. Figure 5 illustrates the structure of the two modules in this stage. As the name suggests, the visualization module enables the visualization of the gestures when editing the collected gestures in order to realize a more intuitive editing experience for the end-user. It consists of a collected gesture viewer and learning information viewer. Each collected gesture is associated with a name, similarity, and trajectory, and the learning information includes the names, similarities, and counts of the gestures to be learnt.

The collected gesture editing module supports functions such as gesture re-recording, body part re-selection, and frame reselection for higher recognition accuracies. If the results obtained after learning and recognition are deemed unsatisfactory, the inadequate gesture can be edited, deleted, or re-recorded through editing to obtain the appropriate gesture. In Figure 5, the collected gestures are expressed as $g_k=\left\{B_k^{*},S_k\right\}$, where k is the index of the collected gestures in the database.

3.4. Gesture Learning Stage

As shown in Figure 6, the gesture learning stage comprises a learning algorithm selection module, body part filtering module, direction normalization module, and learning module. First, the end-user selects an algorithm to learn the collected gestures. Second, the body part filtering increases the learning accuracy by focusing on a selected body part when collecting gestures. Third, the direction normalization module normalizes each sensor’s orientation along eight directions in order to accelerate the learning accuracy. Finally, the learning module learns the normalized gestures using the selected algorithm.

In the proposed framework, the end-user can select from several learning algorithms including hidden Markov models (HMMs), dynamic recurrent neural networks (Dynamic RNNs), and dynamic time warping (DTW) algorithms. To enhance the gesture recognition accuracy, the appropriate learning algorithms can be selected through the learning algorithm selection module. In a previous study, it was shown that the recognition accuracies of HMMs or RNNs are higher than that of DTW [34,35,36]. Moreover, in our experiment, the highest accuracy was obtained when using HMMs. Therefore, the proposed framework uses HMM by default, and the user has the option of selecting another algorithm.

HMM is a statistical Markov model that comprises two elements: A hidden state and an observable state. HMM is suitable for tasks that involve recognizing patterns that change over time. Equation (1) represents hidden Markov model:

$${\lambda }_{B^{*}}=\left\{P_{B^{*}},G_{B^{*}},\pi ,A,B\right\},$$

(1)

where P represents the observable states of a posture, and G represents the hidden states of a gesture. π indicates a matrix of the initial probabilities of hidden states, while A indicates a matrix of the transition probabilities of hidden states; B indicates a matrix of the emission probabilities of hidden states. In this stage, gesture recognition is performed using B.

The HMM uses the following three algorithms. First, the observation probabilities are calculated using forward and backward algorithms. Second, the Viterbi algorithm is used to find the most appropriate state transition sequence for the observable result sequence. This algorithm selects the most probable transition state from the previous states and outputs a result by backtracking to the initial state. Third, the initialization, state transition, and observation probabilities—which are HMM parameters—are optimized to determine the maximum value of the observation probability. To optimize these parameters, the Baum–Welch algorithm is used to generate an optimized HMM for the observation sequence.

In the proposed method, gestures recognition using HMM is achieved by using one integer to represent the direction of one sensor. The directions, namely, up, down, left, right, front, and back, are represented by corresponding bits, i.e., 00100(2), 01000(2), 00010(2), 00001(2), and 10000(2), respectively. Therefore, one or more directions can be represented by one integer. For example, up/left/front is represented as 10110(2). Equation (2) represents the algorithm for calculating direction d of a sensor:

$$\begin{array}{c}\mathrm{if} \left|x_{i,j,t}^{}-x_{i,0,t}^{}\right|>\left|x_{i,j,t-1}^{}-x_{i,0,t-1}^{}\right|,\mathrm{then} d_{i,j,t}^{}\leftarrow d_{i,j,t}^{}+2^0\\ else d_{i,j,t}^{}\leftarrow d_{i,j,t}^{}+2^1\\ \mathrm{if} \left|y_{i,j,t}^{}-y_{i,0,t}^{}\right|>\left|y_{i,j,t-1}^{}-y_{i,0,t-1}^{}\right|,\mathrm{then} d_{i,jt}^{}\leftarrow d_{i,j,t}^{}+2^2\\ else d_{i,j,t}^{}\leftarrow d_{i,j,t}^{}+2^3\\ \mathrm{if} \left|z_{i,j,t}^{}-z_{i,0,t}^{}\right|>\left|z_{i,j,t-1}^{}-z_{i,0,t-1}^{}\right|,\mathrm{then} d_{i,j,t}^{}\leftarrow d_{i,j,t}^{}+2^4\\ \mathrm{else}{d}_{i,j,t}^{}\leftarrow d_{i,j,t}^{}+2^5.\end{array}$$

(2)

The direction is determined based on the differences in the 3D coordinates of the sensor values between previous and current frames. The direction sequence of the sensor values used in the gesture is calculated, and an optimized HMM for the gesture is created using the Baum–Welch algorithm. The gesture recognition result is obtained by calculating the observation probability of the HMM for each gesture using the Viterbi algorithm with the gesture direction sequence as the input value.

3.5. Gesture Recognition Stage and Gesture Transfer Stage

In the gesture recognition and transfer stage, the sensing module obtains gestures using Kinect, Myo, and Leap Motion. The body part filtering module filters the sensing values by considering the body parts selected by the end-user for the learnt gestures. Subsequently, the recognition module calculates the similarity of all learnt gestures, and then, the most similar learnt gesture is selected. The selected learnt gesture is deliveries provided as output to smart TVs and VR sets through the network module. Figure 7 illustrates the module structure chart of the gesture recognition and gesture transfer stage.

4. Generic Gesture Learning and Recognition Approach

The method developed in this study is based on the generic gesture learning and recognition framework. Figure 8 shows the UI Architecture of the proposed generic gesture learning and recognition framework.

4.1. Generic Gesture Learning and Recognition Overview

Figure 9 shows a snapshot of the UI designed for the gesture learning application. As shown in Figure 9, the end-user is displayed a list of learnt gestures. The end-user can select a gesture on the gesture list for editing. To add a new gesture to the gesture list, the end-user can define a new gesture in the gesture creation and option window, select body parts to learn, and start recording. The end-user can also re-learn or re-edit the collected gestures for learning.

4.2. Implementation of User Interface

In the body part selection UI, the end-user can select the body part(s) that will be used for recording the gestures. Figure 10 shows the body selection UI for Kinect. In the UI, the end-user can select up to six parts of the body—the head, trunk, left arm, right arm, left leg, and right leg—during the gesture learning stage. As shown in the right side Figure 10, the end-user can directly select the relevant body parts.

After choosing the body part, the end-user can record and save the gesture. The gesture recording UI is shown in Figure 11. In this UI, the end-user can visualize the original color video as well as the skeletal representation. The end-user can see if the recorded skeletal representation is not representative of the original color video. For example, if a frame displays an inappropriate motion trajectory of the target gesture, the inaccurate frame, in which a specific sensed value is measured incorrectly during the measurement or an empty frame is present between the recorded frames, it can be manually excluded. Moreover, learning using the obtained gestures that contain such frames leads to low recognition accuracies. Thus, the proposed method supports partial deletion and re-recording of such inappropriate frames, thereby improving the recognition accuracy.

As shown in Figure 12, by using the gesture editing UI, the end-user can edit the recorded gestures. The UI displays two previews—original and edited. Therefore, the end-user can compare the two previews and adjudge whether the editing is appropriate. Moreover, the end-user can use the slide bar to preview the recorded video, set the range of the video for deletion, and save the gesture to the gesture database.

After editing the recorded gestures, the end-user generates the HMM module using the gesture learning UI. The UI displays the set of gestures selected by the end-user for learning, as shown in Figure 13. In the gesture learning UI, the end-user can visualize the trajectory of the gestures and compare it against other gestures for the same type of data. Moreover, if differences arise in the two trajectories obtained from the data because of unstable sensors, the end-user can edit the data to reduce the noise. After the end-user finishes editing the gestures, the proposed method automatically learns all the gestures, and the end-user can visualize the similarity of each data point corresponding to each gesture. Here, similarity refers to the similarity score of the gesture data and the trained HMM model by the Viterbi algorithm. After training the HMM model, we use the dissimilarities to identify and correct noisy learning data, thereby enhancing the recognition accuracy of the HMM model. If the similarity is significantly lower than the average of other training data’s similarity, the corresponding data editing is required. In the proposed method, the GUI-based gesture learning UI allows the end-user to obtain improved recognition accuracy by the deleting and re-recording of low similarity data.

The gesture recognition UI shown in Figure 14 is used for the gesture recognition and gesture transfer stages. For the gesture recognition stage, the UI can be used to test whether the learnt gesture is accurate. The end-user can visualize the raw data obtained from sensors in the gesture data log panel. In the gesture succeed log, the end-user can identify the gesture recognized by the proposed method. If the recognition accuracy is lower than the threshold, the performed gesture is considered inadequate. For the gesture transfer stage, the end-user can transfer the recognized gesture result to other applications through the network module. Subsequently, after the HMM model is generated during the gesture learning stage, the Viterbi algorithm is utilized to calculate the similarity of end-user input data for each gesture. The gesture with the maximum similarity is chosen as the recognition result.

5. Experiments

In this study, the proposed method was implemented and evaluated. Kinect was used to identify and record gestures. In the experiments, the HMM algorithm was used as the learning algorithm.

5.1. Performance Show

The proposed method was experimentally evaluated using a shadow puppet performance show. As shown in Figure 15, the actors create certain animations on the screen and interact with it using hand gestures.

For the experiments, ten hand gestures were defined based on the storyline of the shadow puppet performance show. The defined gestures were hand gestures which were intended to activate stage effects during the performance show.

Table 3. The defined gesture list for the experiments.

Gesture 1. Left hand (top→bottom)	Gesture 2. Left hand (bottom→top)
Gesture 3. Left hand (bottom→middle)	Gesture 4. Right hand (top→bottom)
Gesture 5. Right hand (bottom→middle)	Gesture 6. Both hands (bottom→top)
Gesture 7. Both hands (middle→top)	Gesture 8. Both hands (bottom→middle)
Gesture 9. Both hands (bottom→top→bottom)	Gesture 10. Both hands (middle→bottom)

lists the gestures used in the performance show.

The proposed method recognizes the gesture performed by the actor, and the performance show contents are organized by animating a corresponding virtual character based on this gesture information.

5.2. Implementation of the Gesture Learning and Recognition Approach

The gesture learning and recognition approach was implemented using the Microsoft Foundation Class library (MFC), which is based on the C++ language, in a Windows 8.1 environment. We stored all gesture data in Hadoop Distributed File System (HDFS). The OpenCV library was used to visualize each gesture image.

5.3. Gesture Registration Stage Result

In the experiment, the recognition rates of gestures performed by three subjects were compared. Each participant provided 120 examples for each of the 10 types of gestures (a total of 1200 gestures), which served as learning data.

Table 4. Differences in the rendering of the selected body parts.

One Gesture During Testing	Collected Gesture Based on Each Selected Body Part
	Left arm	Right arm	Both arms

shows the gestures recorded by the proposed method for different body parts as selected by the end-user.

The user can visualize the specific body parts selected for the gesture data entered into the proposed system by using the learning gesture trajectory preview panel in the gesture recognition UI.

Table 5. Average sensing values of the right hand of 10 gestures per participant.

Gesture type	Average Sensing Values for Three Participants	Gesture Type	Average Sensing Values for Three Participants
Gesture 1		Gesture 6
Gesture 2		Gesture 7
Gesture 3		Gesture 8
Gesture 4		Gesture 9
Gesture 5		Gesture 10
horizontal axis: frame count, vertical axis: x, y, z value

presents right hand coordinates (x,y,z) of the ten gesture types. The graphs in the table represent the mean values of x, y, and z for the right hand data of three participants performing 100 repetitions of each gesture. The horizontal axes of all graphs indicates the frame count, and the vertical axes represents the x, y, and z values.

We confirmed that the coordinates of a particular body part are similar even when different subjects perform the same gesture. If the user selects specific body parts to be used for learning and recognition, gesture learning and recognition tasks are performed only for the corresponding body parts.

5.4. Gesture Editing Stage Result

To improve the accuracy of gesture recognition, the user can eliminate duplicated or invalid frames from the acquired gesture data.

Table 6. First key frame of the original skeleton and key frames of the transparent skeleton for different gestures.

Editing	Images of Frames
Before editing
Frame	1–9	10–19	20–29	30–39	40–49	50–59	60–69	70–79	80
After editing
Frame	1–9	10–19	20–29	30–39	40–49	50–59	60

shows the results of an end user selecting specific frames of each gesture. These include the specific frames of the end-user’s gestures.

As can be inferred from the above table, frames 1–9, 10–19, 70–79, and 80 are similar before the gesture is edited. The table also shows the results obtained after eliminating frames 10–19 and 70–79 by using gesture visualization and the editing UI.

5.5. Gesture Learning Stage Result

The criterion for determining the gesture recognition stage is the recognition accuracy of the learnt gestures. The gesture recognition accuracy is calculated as a percentage of successfully recognized gestures in the test data, as shown in Equation (3):

$$\mathrm{Accuracy}=\frac{\mathrm{Number}\mathrm{of}\mathrm{correctly}\mathrm{recognized}\mathrm{gestures}}{\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{samples}}\times 100.$$

(3)

In the experiment, the performance of each algorithm was compared. The number of participants was 21. For each participant, 100 examples were recorded for each gesture. Among these, 60 examples were used for learning, and the remaining 40 examples were used for testing. The threshold for the similarity of a gesture performed by a participant and the target gesture was set to 40%. If the similarity was below 40%, the gesture recognition task was considered unsuccessful.

Table 7. Comparison results of recognition accuracy among the hidden Markov model (HMM), dynamic recurrent neural networks (Dynamic RNN), and dynamic time warping (DTW).

Algorithm	HMM	Dynamic RNN	DTW
Gesture Type
Gesture 1	94.25	94.17	87.75
Gesture 2	91.92	92.75	82.58
Gesture 3	91.75	91.50	91.67
Gesture 4	92.33	92.08	88.58
Gesture 5	91.50	90.42	77.75
Gesture 6	90.75	90.17	82.75
Gesture 7	92.25	91.08	78.25
Gesture 8	91.58	90.75	78.67
Gesture 9	92.08	91.08	74.17
Gesture 10	91.58	92.92	74.92
Total average recognition accuracy	92.00%	91.69%	81.71%

lists the average gesture recognition accuracy for each gesture using HMM, Dynamic RNN and DTW. These were 92.00%, 91.98%, and 82.62% for HMM, Dynamic RNN, and DTW, respectively. The highest accuracy was obtained using HMM.

5.6. Gesture Recognition Stage Result

In addition to the above experiments, a scenario was designed using the 10 defined gestures to evaluate the recognition performance of the proposed method for each gesture.

Table 8. Testing scenarios to evaluate recognition performance.

1	2	3
4	5	6
7	8	9
10	11	12
13	14	15
16	17	18
19	20	21
22	23	24
25	26	27
28	29	30

shows the testing performance scenario. Between two gestures, a 1 s interval was added to distinguish between the start and end of a gesture.

Table 9. Gesture recognition results for scenarios.

Scene No.	1	2	3	4	5	6	7	8	9	10
Similarity rate	86.3%	97.8%	24.3%	65.4%	74.3%	85.4%	24.3%	74.3%	94.3%	54.3%
Recognition result	True	True	False	True	True	True	False	True	True	True
Scene no.	11	12	13	14	15	16	17	18	19	20
Similarity rate	84.3%	75.3%	43.2%	95.5%	43.4%	83.2%	64.3%	74.3%	62.1%	77.2%
Recognition result	True	True	True	True	True	True	True	True	True	True
Scene no.	21	22	23	24	25	26	27	28	29	30
Similarity rate	85.2%	4.3%	64.3%	86.5%	48.1%	84.3%	94.3%	84.3%	74.3%	79.9%
Recognition result	True	False	True	True	True	True	True	True	True	True
Avg. recognition accuracy					90.0%

lists the recognition rates for the testing scenario, which suggest that all gestures were recognized correctly even when multiple gestures were performed consecutively. Therefore, the recognition performance rate was 90% on average.

6. Conclusions

In this paper, a method for gesture recognition and learning was developed that can learn precise and detailed gestures using intelligent big data obtained from multiple sensors and an appropriate learning algorithm. This proposed method was integrated with interaction systems by using a communication network. In the proposed method, the collected gestures were visualized, and a GUI-based UI was developed to enable non-experts to input their gestures into the proposed method. The proposed method also allows end users to re-edit, delete, and re-record gestures to improve the gesture recognition rate. The experimental results showed that the end user can learn gestures more intuitively. Moreover, the proposed method achieved a recognition rate of 90% on average.

The proposed gesture recognition and learning method has applications in gesture recognition tasks in various interaction systems because of its integration with different types of sensors. Therefore, in the future, the proposed method can potentially interface with new types of sensors to learn and recognize more complex and detailed gestures.