Toward Improving the Reliability of Discrete Movement Recognition of sEMG Signals

Abstract

Currently, the classification accuracy of surface electromyography (sEMG) signals is high in literature, but the conventional recognition system may classify untrained movements or the trained movements of low reliability to one of its target classes by mistake. If such a system is used for prosthetic control, sometimes it may cause a disaster. A two-layer classifier that fuses the Gaussian mixture model (GMM) and k-nearest neighbor (kNN) in a sequential structure is proposed in this study. The proposed algorithm can reject the trained movements with low reliability and is efficient in rejecting the untrained movements, thus enhancing the reliability of the myoelectric control system. The results show that the proposed algorithm can produce 95.7% active accuracy in recognizing 12 trained movements and a 30.3% error rate for rejecting 12 untrained movements. When the movement number is six, the active accuracy for trained movements can reach 99.2%, and the error rate of untrained movement is only 17.4%, which is much better than previous studies. Therefore, the proposed classifier can accurately recognize the trained movements and reject untrained movement patterns effectively.

1. Introduction

Movement recognition with a surface electromyography (sEMG) signal has been investigated for many years. sEMG signal decoding with pattern recognition (PR) methods has attracted a lot of attention due to the rapid development of sensing technology and machine learning. Various feature extraction [1,2,3] and pattern recognition methods [4] have been developed to improve classification accuracy. An 89% classification accuracy can currently be obtained for as many as 52 hand movements [5]. When the number of movements is eight, the classification accuracy can reach 97% [6]. Though high classification accuracy has been achieved in the laboratory, the practical application of PR-based myoelectric control is limited due to the relatively low robustness of the PR-based control strategy [7,8].

One problem of conventional PR algorithms is that they are built on the assumption that the statistical feature of test samples is close to the samples used for training. However, the sEMG signals are affected by lots of factors, such as muscle fatigue, limb position [9], electrode shifting [10], and within/between day factor [11], leading to performance degradation of the PR system after prolonged use. It is hoped that the PR system can only make confident predictions. Otherwise, the PR system should avoid making predictions. This problem is known as ambiguity rejection in some studies [12,13], which is the first research question in this study.

Another problem of conventional PR algorithms is that they can only recognize a limited number of target movements. They always make a prediction for an input, which can be considered a drawback when the input is from an untrained movement class [14]. Suppose such a system is utilized for prostheses control. In this case, the system will classify the untrained movements as one of the target movements by mistake [15], resulting in an unexpected activation of prostheses, which will significantly reduce the usability and reliability of the myoelectric control system [16]. It may also frustrate the users and make them refuse to use the device again. Hence, the rejection of untrained movements (novelty rejection), which is the second research question in this study, is also essential.

As for prostheses control, it is believed that the error that causes accidental activations of the prostheses is more “costly” than those that cause a pause in motion [17]. A PR system with a rejection option to the sEMG signals with low similarities to training samples would reduce the potential risk due to uncertain decisions. Those sEMG signals can either be signals from trained movements or novel movements. The study aims to improve the reliability of the sEMG recognition system by increasing the accuracy of recognized samples and reducing the influence of untrained/unknown movement patterns [18] so that the system can be used for practical applications. An efficient two-layer classifier is proposed in this study. The proposed classifier predicts the test sample firstly by a time-efficient Gaussian mixture model (GMM) with a reject option (GMM-R) classifier. Then, the test sample will either be rejected or passed to the second-layer k-nearest neighbor (kNN) with a reject option (kNN-R) classifier based on the prediction result of the GMM-R classifier. Suppose the first layer classifier rejects the test sample. In that case, it is up to the second layer classifier to determine whether it gives a prediction or rejects the current decision. The advantage of the proposed classifier is that it takes advantage of GMM for its fast calculation and the kNN classifier for its high accuracy. So, it outperforms GMM and kNN in accuracy and is much more computationally efficient than kNN. With the proposed algorithm, objectives of this study can be achieved: (1) increasing the recognition accuracy of labeled samples while reducing their rejection rate and (2) enhancing the rejection rate of novel samples. Besides, the algorithm is computationally efficient for both model training and test sample recognition.

The paper is organized as follows. A literature review on classification with reject options is introduced in Section 2. Section 3 describes the dataset used in this study, signal processing and feature extraction method, and the model evaluation method. The description of the proposed model is introduced in Section 4. Then, the experimental results are presented in Section 5. A discussion on proposed algorithms and comparison with other studies are conducted in Section 6. Finally, the conclusion is drawn in Section 7.

2. Literature Review

Classification with a rejection option has been applied in many areas, such as optical character recognition [19,20], medical diagnosis [21], and engineering [22]. Despite the difference in their application background, the algorithms have something in common. Though ambiguity rejection and novelty rejection are two different rejection types, and they are both concerned in this study, the algorithms developed for one can be applied for another in most cases. So, we do not differentiate them on purpose when conducting the literature review but focus more on the algorithm itself.

A common algorithm used in related research is linear discriminant analysis (LDA). The authors of [17,23] proposed a multiple binary LDA classifier with “1-vs.-all” and “1-vs.-1” topology for the rejection of motions when the multiple binary classifiers cannot reach an agreement. Though it effectively limits the number of active movements, [23] could be problematic as the increase of motions and [17] suffer from too frequent rejections. The authors of [24] proposed a confidence-based rejection scheme based on the outputs of LDA. It generates a confidence score for each observation, and the observation with a confidence score lower than the predefined threshold will be rejected. A similar method is followed in [25], except that the class-specific threshold is automatically determined using receiver operating characteristic (ROC) curves.

The reject option can also be used with neural networks. The researchers of [26] developed a method for defining a reject option by estimating the classification reliability as measured by a reliability evaluator and applied the method to three neural network paradigms. A post-processing algorithm for a multiple layer perceptron (MLP) neural network was proposed in [27] to detect and remove possible misclassifications from LDA. Deep neural networks (DNNs) have also been used for novelty detection in [28,29]. Although these algorithms have demonstrated their effectiveness through experimental results, the training of neural networks is time consuming.

Previous publications have shown that combing classifiers will improve the accuracy of individual classifiers; refs [30,31,32] analyzed the error-reject trade-off in three linearly combined classifiers. The experiments showed that the linear combination of the classifiers might improve the error-reject trade-off of the individual classifiers; still, the improvement depends on the type of combination and level of rejection rate. An adaptive hybrid classifier consisting of one-class support vector descriptors (SVDDs) and a multi-class LDA was proposed in [33] to reduce the impact of interference on myoelectric pattern recognition. The LDA classifier will be adopted for classification if the test observation is not detected as an outlier by SVDD. A similar strategy has been adopted in [18]. In both studies, the classification accuracy of the identified active movements is mainly dependent on the selected classifier, and SVDD works as an outlier filter [12,34]. Besides, SVDD may result in a large run-time complexity as the increase of training set, which may introduce time delays to the control system.

Unlike studies above, a training strategy was proposed in [16] to categorize all the unwanted movements (UMs) into a new movement class (UMs-combined motion class) to reduce the impact of unwanted movements. Since the LDA classifier is trained by the sEMG signals from both target classes and the UMS-combined motion class, the usability of the proposed strategy is limited in practical applications because the unwanted movements are usually unknown in advance.

The studies mentioned above provide a good reference for developing classifiers with reject options. It can be noticed that many related studies developed algorithms based on an LDA classifier. Though an LDA classifier can be extended to have a rejection option conveniently and is computationally efficient, its classification accuracy is limited compared with other classifiers [35,36]. The other algorithms, such as neural networks, SVDD, and binary-type classifiers, may have significant problems in high computational costs. A classification scheme based on boosting and random forest classifiers is proposed in [37]. It was shown that the proposed classifier has good error distribution among the classes. A 92% accuracy can be obtained in recognizing trained movements and 20% for untrained movements. A comparison between the classifier proposed in [37] and the classifier proposed in this study has been conducted.

3. Methods

3.1. Dataset

The dataset applied in this study is from Ninapro database 1, Exercise A [35,36,38], which consists of 12 basic hand movements. The sEMG signals are collected using 10 active double-differential OttoBock MyoBock 13E200 electrodes. Eight electrodes are uniformly placed around the forearm. Additionally, two electrodes are placed on the flexor and extensor muscles of the forearm. The sampling rate of the sEMG signals is 100 Hz. The total number of subjects in the database is 27, which are all intact subjects. The movements used in this study are 12 finger movements, which are index flexion/extension, middle flexion/extension, ring flexion/extension, little finger flexion/extension, thumb adduction/abduction, thumb flexion/extension [36]. To evaluate the classifier’s performance on trained movements, the first five repetitions of the 12 movements are used for classifier training, and the left five repetitions are used for testing.

3.2. Signal Processing and Feature Extraction

The sEMG signals of Ninapro database 1 have already been amplified, band pass-filtered, and rectified. So, there is no signal preprocessing further applied to the sEMG signals in this study. The signals have been labeled so that the movement-related signals could be directly obtained. For feature extraction, a sliding window method is applied. The width of the window is 200 ms, and the increment is 20 ms. For each sliding window, two time-domain feature extraction methods, root mean square (RMS) and waveform length (WL), have been applied due to their computational simplicity and relatively better performance than the other features from my previous study [5]. The definitions of the feature extraction algorithms RMS and WL are described below.

• RMS

  $${\mathrm{RMS}}_j=\sqrt{\frac{1}{N}{\displaystyle \sum _{i=1}^N{X}_{{}_{ij}}^2}}$$

  (1)
• WL

  $$\mathrm{WL}={\displaystyle {\sum }_{i=1}^N|{\mathrm{x}}_i-{\mathrm{x}}_{(i+1)j}|}$$

  (2)

  where j is the column; N is the number of observations in the current window; and $X_{ij}$ represents the data point.

3.3. Evaluation

Accuracy is usually used to evaluate the performance of a classifier. For classifiers with rejection options, their performance depends not only on accuracy but also on rejection rate. To differentiate the accuracy defined in conventional classifiers and the one with a rejection option, total classification accuracy (tAcc), active classification accuracy (aAcc) [27], and rejection rate [16] are used to evaluate the performance of proposed algorithms, and they are defined as follows.

$$\mathrm{tAcc}=\frac{\mathrm{number} \mathrm{of} \mathrm{correct} \mathrm{classifications}}{\mathrm{total} \mathrm{number} \mathrm{of} \mathrm{classifications}}\times 100\%$$

(3)

$$\mathrm{aAcc}=\frac{\mathrm{number} \mathrm{of} \mathrm{correct} \mathrm{active} \mathrm{classification}}{\mathrm{total} \mathrm{number} \mathrm{of} \mathrm{active} \mathrm{classification}}\times 100\%$$

(4)

$$\mathrm{rejection} \mathrm{rate}=\frac{\mathrm{number} \mathrm{of} \mathrm{rejected} \mathrm{classifications}}{\mathrm{total} \mathrm{number} \mathrm{of} \mathrm{classifications}}\times 100\%$$

(5)

where active decisions are the decisions that would lead to prostheses moves when the algorithm is applied for prostheses control. If the classifier does not reject any observations, tAcc and aAcc will be the conventional classification accuracy, and the rejection rate will be zero. When a classifier tends to reject all of the observations, tAcc will be approaching 0%, aAcc will tend to 100%, and the rejection rate will be 100%. So tAcc, aAcc, and rejection rate should be used together [39] to evaluate the performance of a classifier with a reject option. A method for comparing classifiers with reject options by sketching an accuracy rejection curve (ARC) was proposed in [40]. The ARC plots the accuracy of a classifier against its rejection rate. Given the acceptable rejection rate, the best available classifier can be selected from the ARC curve with the highest classification accuracy. Therefore, the ARC curve is also sketched when the performance of the classifier is compared.

The method mentioned above is used to evaluate the performance of the classifier to discriminate between trained classes. To assess the ability of the classifier to reject untrained movements, leave-one-out error analysis (LEA) [37] is applied in this study. The error rate of the untrained movements is defined as follows.

$$\mathrm{error} \mathrm{rate}=\frac{\mathrm{number} \mathrm{of} \mathrm{active} \mathrm{decisions}}{\mathrm{number} \mathrm{of} \mathrm{total} \mathrm{decisions}}\times 100\%$$

(6)

For each untrained movement, if the classifier assigns a known class label to it, the decision is an active decision, which means it is an error. So, this index can be used to evaluate the capability of the classifier to reject unknown movement patterns. Each of the 12 movements is sequentially omitted during classifier training but sent to the classifier for prediction.

4. Proposed Methods

As mentioned above, ambiguity rejection and novelty rejection are two concerns of this study. Though complex models, such as DNNs [29], may produce highly confident predictions, they are more computationally expensive to train and require more data to learn the model [14]. Therefore, two simpler models are preferred in this study.

4.1. GMM Classifier with a Reject Option

GMM is a parameter-based classifier, and it makes an assumption about the samples. Our previous study shows that GMM has an excellent performance in sEMG signal-based movement recognition [5] for high accuracy and fast responses. Therefore, the GMM classifier is used as a baseline classifier in this study.

Theoretically, GMM can be seen as the weighted sum of several Gaussian components that best approximate the input. It can be modeled as follows.

$$p(x|\Theta )={\displaystyle \sum _{i=1}^{\mathrm{N}}{w}_{i}N(x|{\mathsf{\mu }}_i,{\Sigma }_i)}$$

(7)

$${\displaystyle \sum _{i=1}^N{w}_i=1}$$

(8)

where $w_i$ represents the prior probability of the i^th component; and $N(x|{\mathsf{\mu }}_i,{\Sigma }_i)$ is the Gaussian distribution of the i^th component defined by mean vector ${\mathsf{\mu }}_i$ and covariance matrix ${\Sigma }_i$. Then, given an observation x with dimension d, the conditional probability density of x obtained from the c^th category can be written as follows.

$$p(x|{\Theta }_c)={\displaystyle \sum _{i=1}^N{\pi }_i}\frac{1}{{(2\pi )}^{d/2}{|{\Sigma }_i|}^{\frac{1}{2}}}\exp \left(-\frac{1}{2}{(x-{\mathsf{\mu }}_i)}^T{\Sigma }_i{}^{-1}(x-{\mathsf{\mu }}_i)\right)$$

(9)

where ${\Theta }_c=\{\pi ,\mathsf{\mu },\Sigma \}$ represents the parameter set of the Gaussian mixture model of the c^th category, obtained from model training using the Expectation-Maximization (EM) algorithm. According to Bayes’ theory, the probability of x belonging to class c is proportional to the conditional probability density if each class’s prior probability is the same. Therefore, the class label of x can be determined by finding the category of the maximum conditional probability as long as the conditional probability densities of x from each class are obtained.

The test sample X of sEMG signals collected from contracting muscles for each discrete movement consists of many observations $x_1,\dots ,x_N$, where N is the number of observations. Therefore, the prediction on the discrete movement by using (10) is a series of class labels. To extend the GMM to have a rejection option (GMM-R), we introduce a majority voting (MV) scheme to the series of class labels obtained in this study. We tabulate these class labels into a 2-column table, where the first column contains the unique values of class labels, and the second column contains the percentage of each class label. The GMM-R makes its decision based on the maximum percentage value. Suppose the percentage of each class label is $G_1,\dots ,G_C$, and

$$\widehat{g}=\underset{c\in 1:C}{\arg \max }{ \mathrm{G}}_c$$

(10)

Then given a predefined threshold ${\delta }_g$, the decision of GMM-R can be expressed as follows.

$$g(X)=\{\begin{array}{l}\widehat{g}{, \mathrm{G}}_{\widehat{g}}>{\delta }_g\\ ⓤ, \mathrm{otherwise} \end{array}$$

(11)

where $ⓤ$ represents that the prediction is in doubt, and a further judgment should be made. In the GMM classifier, the number of mixture components plays a vital role in its performance. We have conducted a preliminary study on the influence of the number of mixture components on the performance of GMM [41], and it was found that when the number is three, a trade-off between classification accuracy and computational complexity can be obtained [5]. Therefore, the number of mixture components will be three in the following study.

4.2. GMM-kNN Fusion with a Reject Option

As a nonparametric method, kNN is one of the most widely adopted classification algorithms. Though it is simple to implement, it is shown that kNN can produce classifiers with performance close to the optimal Bayes classifier [42,43,44]. KNN does not make any assumptions about the data, and it can be used on many occasions. The limitation of kNN is that it compares the test sample with the training data each time a decision on the test sample is demanded, so it requires a lot of memory and is computationally expensive [45]. We have conducted a study on recognizing the 12 hand movements of 10 repetitions with sEMG signals collected from 27 subjects using GMM and kNN classifiers. When the first five repetitions are used as training samples, and the rest five are used as test samples, the result is shown in Figure 1. It can be observed from Figure 1a that the classification accuracy of kNN is superior to GMM in both its higher accuracy and lower standard deviation. However, the computational cost of kNN is much higher than GMM. The analysis shows that the average computational time for recognizing the sEMG signals for one movement by kNN is nearly 260 ms, almost 80 times more than GMM.

Considering the characteristics of kNN and GMM, we propose a two-layer classification scheme, which may take advantage of the two classifiers to improve the classification performance efficiently. The classifier is named GMM-kNN fusion with a reject option (GK-R). The proposed model consists of two layers, as shown in Figure 2. The first layer is a GMM-R classifier. The test sample will be sent to the GMM-R classifier for decision at first. If the maximum percentage is larger than the predefined threshold, the class label corresponding to the maximum percentage will be assigned to the test samples. Otherwise, the decision on the test sample is up to the second layer, which is a kNN with a reject option (kNN-R) classifier. The kNN-R classifier conducts a similar post-processing method on the class labels of each observation to what GMM does. Suppose the percentage of each class label obtained by kNN is ${\mathrm{K}}_1,\dots ,K_C$; the predicted class label can be determined as follows.

$$\widehat{k}=\underset{c\in 1:C}{\arg \max } K_c$$

(12)

Suppose ${\delta }_k$ is a predefined threshold used to evaluate the validity of the prediction by the second-layer classifier. Then the decision on the test sample X will be determined by two factors: the maximum percentage obtained by kNN and the predicted class label by the GMM-R and kNN-R classifier. When the two classifiers make the same prediction, or the maximum percentage obtained by kNN is higher than the predefined threshold, the predicted class label will be assigned to the test sample. Otherwise, the test sample will be labeled as “no motion”. The final decision can be expressed as follows.

$$f(X)=\{\begin{array}{l}\widehat{k},{ \mathrm{if} \mathrm{K}}_{\widehat{k}}\ge {\delta }_k or \widehat{g}=\widehat{k}\\ ⓡ, \mathrm{otherwise} \end{array}$$

(13)

where $ⓡ$ represents the prediction is rejected, and the movement will be labeled as “no motion”. The corresponding class label is 0.

The following classifiers are applied to prove the effectiveness of the proposed classification scheme.

• LDA-R(LDA with majority voting and a reject option). LDA-R is an extension to LDA. For each observation in test samples, the conventional LDA will assign a class label to it. Unlike the study in [24], a majority voting is performed to the series of class labels. The percentages of observations that fall into each class are obtained. Thresholding is performed on the maximum percentages. Additionally, for the one that is higher than the threshold, the corresponding class label will be assigned to it. Otherwise, the prediction will be rejected.
• GMM-R (GMM with a reject option). Since the proposed classifier is built on a GMM-R classifier, the effectiveness of the proposed algorithm will be compared with the GMM-R classifier.
• KNN-R (kNN with a reject option). Similar to LDA-R and GMM-R, kNN-R can reject a low-reliability decision. Although high computational requirements may prevent real-time application when the number of classes is large, its high classification accuracy is still attractive.
• GK-R (GMM and kNN fusion with a reject option). There are two parameters in the GK-R classifier; the parameters are tuned, and the performance of GK-R for different parameters will be presented.

5. Results

5.1. Classifiers with a Reject Option

The LDA, kNN, and GMM classifiers can be extended to classifiers with a reject option. The kNN and GMM classifiers with reject options have been presented before. The LDA classifier with a rejection option (LDAR) proposed in [24] gives a predicted probability for each observation. The probability was compared with a predefined threshold. If it is below the threshold, the prediction will be rejected. Otherwise, it will be accepted. The optimal threshold reported in [24] is 0.97, and the same value was used in [27]. Unlike the study in [24], the test sample of each movement applied in this study consists of multiple observations, so [24] cannot be used directly. To use LDA as a baseline classifier, we extend it by thresholding the percentage of the majority voting results. Additionally, the final prediction decision for the test sample is based on the thresholding result.

The classifier performance of LDA-R, kNN-R, and GMM-R on recognizing 12 trained hand movements with features extracted by using RMS and WL is shown in Figure 3. It can be observed in Figure 3a that the tAcc of kNN-R is the highest of all. The tAcc and aAcc of kNN-R and GMM-R are much higher than the LDA-R at the same rejection rate level. The performance of the classifiers with different features is different. GMM-R and LDA-R have a higher aAcc with WL feature than the RMS feature. However, for kNN-R, the RMS feature produces a higher accuracy than the WL feature at a low rejection rate, and then the aAccs of the two features tend to overlap at a high rejection rate. Therefore, for different classifiers, the more appropriate features are varied. For practical application, it is desired that the system has a low rejection rate and high accuracy. Considering the performance of kNN-R and GMM-R with the two features, the feature adopted by kNN is RMS, and the features used by GMM is WL in this study.

5.2. Performance of Proposed Classifier on Trained Movements

The accuracy–rejection curve proposed in [40] can be used to evaluate the performance of classifiers with rejection options. On the accuracy–rejection plot, accuracy is plotted against the rejection rate. The curve starts from a point (0, a%), where a% is the accuracy of the classifier when there is no rejection. A system with accuracy and rejection rates in the top left quadrant is the most desirable.

Figure 4 shows the classification results on the 12 hand movements with the classifiers mentioned above. The δ_g parameter of GK-R classifier varies from 35 to 75 with a step of 5, and δ_k = δ_g − 10:10:δ_g + 10. From Figure 4a,b, it can be seen that the tAcc and aAcc of GK-R are the highest, followed by kNN-R when the rejection rate is equal. Additionally, the accuracy of GMM-R is the lowest of all.

Figure 4b shows that although the aAcc of the GK-R classifier is lower than GMM-R with the same parameter δ_g, its rejection rate has been significantly reduced. It can be inferred from Figure 4a,b that the tAcc and aAcc of the GK-R classifier are much higher than GMM-R and kNN-R when their rejection rates are equal. Therefore, the second layer effectively enhances the accuracy in recognizing samples rejected by the first layer. For the same tAcc and aAcc, the rejection rate of GK-R is much lower than that of GMM-R, showing the advantage of GK-R in reducing the rejection rate and increasing recognition accuracy.

A grid search method has been applied to compare the performance of the three classifiers in Figure 4 at the same rejection rate level. When the rejection rate is 13.0% ± 0.5%, the tAcc and aAcc obtained by the classifiers are shown in Figure 5. It can be seen that the average tAcc and aAcc obtained from GK-R (δ_g = 65%, δ_k = 75%) are much higher than those results obtained by the LDA-R, GMM-R, and kNN-R classifiers. The improvements of GK-R in tAcc over LDA-R, GMM-R, and kNN-R are 9.0%, 2.9%, and 1.8%, respectively, and the improvements in aAcc are 11.2%, 3.7%, and 2.0%, respectively.

Figure 6 shows movement prediction using GMM, GK-R, and the first layer classifier (GMM-R) of GK-R for one subject performing 12 hand movements with five repetitions. It can be observed from Figure 6a that most of the test samples can be correctly recognized by the GMM classifier. Those samples that are wrongly predicted by the GMM classifier are all rejected by the first layer (GMM-R) classifier, although some samples that are correctly predicted by the GMM classifier but rejected by GMM-R. Figure 6b shows the prediction results with the proposed GK-R classifier. It can be found that the proposed model is effective in reducing the number of misclassifications in GMM with the help of the second layer classifier at the cost of increasing the number of rejections. However, misclassification is a much more severe problem than rejections for practical applications, especially when the rejection rate is limited.

5.3. Time Cost

As shown above, the accuracy of GK-R is much higher than GMM-R and kNN-R. Meanwhile, for the 1620 test samples collected from 12 hand movements of 27 subjects, the decisions on 52.7% of samples are made directly by the first layer GMM-R classifier, and the rest are then sent to the second layer kNN-R classifier. The percentage of decisions made by GMM-R determines the computational efficiency of the proposed classifier. It is said in [46] that a delay no larger than 300 ms will not be perceived by the user and can be used for prosthetic control. However, it is stated in [47] that delays greater than 200 ms are not acceptable, but the values were changed to 100–300 ms in the later paper [48]. A similar study in [49] showed that the maximum allowable time delays are 300–400 ms, so the user cannot notice the delay. Tests on different levels of controller delay ranging from 0 ms to 300 ms were conducted in [50] and it was found that the optimal controller delay lies between 100 ms and 175 ms. Though the conclusion on optimal controller delay varied in different studies, the less the delay, the better.

The comparison of average time cost for recognizing one movement by kNN-R, GMM-R, and GK-R is shown in Figure 7. In Figure 7, the values that deviate from the mean of the data by more than three standard deviations are considered outliers and are removed. The width of the bin in the histogram of Figure 7 is set to 75 ms. From Figure 7a, it can be found that the average time cost for recognizing one movement by GMM-R is much less than kNN-R, which is actually around five milliseconds. For kNN-R, about 12.5% of samples need less than 150 ms; 57.8% of samples need 150–300 ms; and the rest need more than 300 milliseconds to be recognized. GMM-R is much more time efficient compared with kNN-R. From Figure 7b, it can be found that the samples recognized within 150 ms are 57.5% by GK-R; it is only 12.5% by kNN-R, meaning more than half of the samples need less than 150 ms to be recognized by the GK-R classifier. Meanwhile, the percentage of samples recognized with more than 300 ms by kNN-R is 14.4% and is reduced to 8.7% by GK-R. So, the proposed GK-R classifier is much more time efficient than kNN-R.

5.4. Performance of Proposed Classifier on Untrained Movements

The leave-one-out error analysis (LEA) can be used to evaluate the reject ability of the classifier for untrained movements. If the classifier does not reject any untrained/unknown movements, it will result in an active decision, which is an error. So, LEA can be used to evaluate the ability of the classifier to decline untrained/unknown movement patterns. To assess the performance of the proposed classifier on unknown movements, each of the 12 hand movements is selected as an unknown movement sequentially, and the chosen unknown movement will not be used for training but used to test the trained classifiers. The classification result using the algorithms mentioned above is shown in Figure 8, obtained using the same parameters as Figure 5. From Figure 8, it can be observed that LDA-R and GMM-R classifiers have much higher error rates than GK-R in rejecting the untrained movements. The overall error rates of the LDA-R and GMM-R classifier are 78.3% ± 5.9% and 51.2% ± 6.7%, respectively, meaning they have unsatisfactory performances in rejecting the unknown movements. However, for GK-R, the error rate is reduced to 30.9% ± 7.6%, which shows a significant improvement.

6. Discussion

The proposed algorithm can enhance the classification accuracy by rejecting the movement with low reliability and unknown movement patterns. A classification scheme derived from the multi-class problem and through the linear programming boosting algorithm (MCLPBoost) was proposed in [37] to enhance the classifier’s robustness against untrained classes. We have compared the performance of the proposed algorithm with the one in [37]. Six most-similar hand movements were selected from Exercise B, Ninapro database 1 as t used in [37]. The six movements were wrist flexion, wrist extension, wrist pronation, wrist supination, fingers flexed together in a fist, and abduction of all fingers (corresponds to hand open) [35,36], where the total number of subjects was also 27. Though a different dataset was applied, it is shown that with the same classifier parameters used above, the average error rate of the six untrained movements obtained is 23.7%, which is comparable to the one obtained in [37]. The proposed GK-R classifier is trained with the first five repetitions of the six movements and tested with the left five for trained hand movements. It is shown that the total accuracy of the trained movements obtained is 95.6%, and the aAcc is 99.2%, which is much higher than the reported 80% accuracy, showing a significant improvement in rejecting unknown movements and high accuracy for trained movement patterns.

The GK-R classifier proposed in this study consists of two layers. The first layer classifier GMM gives a preliminary judgment on the test samples. If its threshold is too large, then more test samples will be passed to the second layer classifier kNN for a final decision, which may help to increase classification accuracy but at a higher computational cost. In contrast, if the threshold is small, the classification accuracy is dependent mainly on the GMM classifier. In the extreme case, when the threshold is small enough, the classifier’s performance is entirely determined by the GMM classifier, and no test samples will be rejected. So delicately choosing the value of δ_g is kind of important. The value of δ_g determines which index is more valued, computational cost or classification accuracy. For a fixed δ_g, the parameter δ_k of the second layer classifier determines the rejection ratio to some extent.

Figure 9 shows the performance of the proposed classifier on the six trained and untrained movements. The parameter δ_g varies from 55% to 65% with an increment of 5%, and δ_k increases from 65% to 95% with an increment of 5%. From Figure 9a, it can be found that the aAccs of different values of δ_g overlap with each other, and they are all 99.2%. Though the tAccs have different values, they show a similar tendency toward the increase of δ_k. For a smaller δ_g, more decisions are made by the first layer classifier, and fewer samples are rejected, which will result in a higher tAcc, as shown in Figure 9a. However, the side effect of a smaller δ_g is that the error rate of untrained movements will increase, as shown in Figure 9b. Therefore, when choosing the value of δ_g and δ_k, a compromise should be made between the accuracy of trained movements and the error rate of untrained movements. From Figure 9a,b, it can also be found that when the error rate of the untrained movements is 20.9%, the tAcc and aAcc of the trained movements are 95.6% and 99.2%, respectively. If we further increase the second parameter, the error rate of the untrained movements can reach 17.4%, while the aAcc will not be affected, and tAcc will decrease by 0.3%. The results have been much improved compared with [37].

7. Conclusions

A two-layer classifier has been proposed to enhance the reliability of the myoelectric control system in this study. Unlike the conventional classifier, the proposed classifier introduces a rejection option to decline the test samples that are not reliable enough. We have compared the proposed classifier with the LDA classifier with a reject option, the GMM classifier with a reject option, and the kNN classifier with a reject option on 12 hand movements. The proposed classifier produces much higher classification accuracy for test samples from trained movements than the other classifiers at the same rejection rate level. The improvement of proposed classifier over the other classifiers in terms of tAcc and aAcc are 9.0%, 2.9%, and 1.8% and 11.2%, 3.7%, and 2.0%, respectively. Though the improvement of the proposed classifier over the kNN classifier is limited compared with the other classifiers, the proposed classifier is much more computationally efficient than the kNN classifier, showing the effectiveness of the proposed model in ambiguity rejection. For test samples from untrained movements, the proposed classifier has a much lower error rate than the other classifiers, proving the effectiveness of the proposed classifier in novelty rejection. The proposed classifier has also been applied to recognize six hand movements. The results show that the tAcc and aAcc of the proposed classifier on trained movements can reach 95.6% and 99.2% when the error rate of untrained movement is 20.9%. The tAcc and aAcc of trained movements can be 95.3% and 99.2% by adjusting the threshold. The error rate of untrained movement is only 17.4%, showing a significant improvement in recognizing trained movements and rejecting the untrained movements compared with previous studies.