Current Trends, Challenges, and Future Research Directions of Hybrid and Deep Learning Techniques for Motor Imagery Brain–Computer Interface

Abstract

The field of brain–computer interface (BCI) enables us to establish a pathway between the human brain and computers, with applications in the medical and nonmedical field. Brain computer interfaces can have a significant impact on the way humans interact with machines. In recent years, the surge in computational power has enabled deep learning algorithms to act as a robust avenue for leveraging BCIs. This paper provides an up-to-date review of deep and hybrid deep learning techniques utilized in the field of BCI through motor imagery. It delves into the adoption of deep learning techniques, including convolutional neural networks (CNNs), autoencoders (AEs), and recurrent structures such as long short-term memory (LSTM) networks. Moreover, hybrid approaches, such as combining CNNs with LSTMs or AEs and other techniques, are reviewed for their potential to enhance classification performance. Finally, we address challenges within motor imagery BCIs and highlight further research directions in this emerging field.

1. Introduction

Brain–computer interfaces (BCIs) are an emerging field of technology that combines and allows the connection between the brain and a computer or other external devices. BCIs have the potential to revolutionize the way humans interact with machines, opening countless possibilities both in medical and nonmedical domains. In the medical field, it can help people suffering from locked-in syndrome to communicate [1]. Moreover, brain–computer interfaces (BCIs) are displaying potential in the realm of neuroprosthetics, offering the prospect for individuals with limb amputations or paralysis to command robotic limbs or exoskeletons through their brain signals [2]. In epilepsy management, BCIs are researched for real-time seizure detection and intervention, potentially mitigating the impacts of seizures [3]. In the nonmedical domain, an EEG BCI has applications in areas such as gaming where players can play a game using only their thoughts [4] and in fields such as entertainment where users can control drones or other robotic devices [5].

There are several methods and techniques used in BCI research (Figure 1). These methods can be categorized based on their degree of invasiveness as invasive and noninvasive. Invasive methods require physical access to the brain and include ECoG (electrocorticography) [6], which is the process of recording electrical activity in the brain by placing electrodes in direct contact with the cerebral cortex or surface of the brain. Noninvasive techniques do not require any type of surgical operation and can be further discriminated into (a) fMRI (functional magnetic resonance imaging), which measures the brain activity by detecting changes associated with the blood flow [7], (b) MEG (magnetoencephalography) [8], which is the measurement of the magnetic field generated by the electrical activity of neurons, (c) NIRS (near-infrared spectroscopy) [9], a brain imaging method that measures light absorbance to calculate oxyhemoglobin (oxy-HB) and deoxyhemoglobin (deoxy-HB), which provides an indirect measure of brain activity, particularly in the frontal cortex, and (d) electroencephalography (EEG), [6] which is this paper’s subject and refers to the measurement of the electrical activity produced by the brain by placing electrodes to the subject’s scalp.

Although there are disadvantages in EEG-based BCIs compared to other techniques [10], such as a low signal-to-noise ratio, a low spatial resolution, and variability in the signal quality due to artifacts, their portability, cost-effectiveness, noninvasiveness and high temporal resolution make them the most spread method for BCI applications.

There are a lot of approaches to further categorize EEG BCIs. Authors in [11,12] further categorize BCIs based on the user’s activity during the signal acquisition as passive/active. Other approaches are based on the stimuli needed for the activation of the BCI [13], which group noninvasive BCIs into evoked (also referred as exogenous) or spontaneous (also referred as endogenous). Another work [10] categorizes BCIs based on the ability of the system to respond on a specific time-window called synchronous or asynchronous.

In the active/passive approach (Figure 2), the term “Passive” refers to situations where the user is either at rest or involved in a task, and brain activity is being measured. The term “Active” refers to the situation where brain activity is measured while the user is carrying out specific tasks or actions. For example, in motor imagery, the user thinks about performing physical movement without actually performing the movement. On the contrary, in the word imagery paradigm also referred to as imagined speech, the user is required to think about certain words, while visually, auditory, and vibrotactile evoked-potential paradigms involve the measurement of brain activity in response to an external visual, auditory, and vibrotactile stimuli, respectively. Some applications of passive EEG BCIs are measuring the subject’s emotion [14], cognitive load [15], and attention [16], while active EEG BCIs have been used to control a cursor on a computer screen [17], moving a robotic arm [18], or even helping patient rehabilitation after a stroke [19].

A typical pipeline for an EEG motor imagery (MI) BCI application involves several stages (Figure 3). The first stage concerns the signal acquisition, whereas the subject is instructed to imagine a motor movement while their brain activity is measured using EEG. The next stage is signal preprocessing, where several different techniques are used to filter and enhance the quality of the signal. Afterwards, feature extraction is performed, where the processed signal is further analyzed to extract relevant features. These features are then subjected to feature selection, where the most relevant features for the application are identified. Finally, classification is performed using machine learning algorithms to classify the data into different categories, such as left- or right-hand movement, and turn these categories into machine instructions, such as moving a cursor in a 2D space, etc.

The success of this pipeline depends not only on the optimal selection of signal processing techniques, feature extraction methods, feature selection algorithms, and classification models, but also on the user’s ability to consistently generate distinguishable brain patterns [20].

Electroencephalogram (EEG) signals, captured noninvasively through electrodes on the scalp, are electrical patterns reflecting the synchronized firing of neurons in the brain. This technique has high temporal resolution, enabling researchers and clinicians to monitor real-time brain activity with millisecond precision. Although EEG signals are considered 1D-signal EEG recordings, they yield time series data, where the x-axis represents time, while the y-axis represents the electrical activity observed at each electrode. Time series data can undergo different transformations to be reshaped into 2D or 3D formats, with one common approach involving the creation of 2D images by incorporating time–frequency information.

Data classification typically involves one of the following three categories: machine learning, deep learning, and hybrid deep learning as shown in Figure 4. Machine learning provides strong statistical methods for smaller datasets, while deep learning uses neural networks for complex patterns in larger datasets. Hybrid deep learning merges the two previously mentioned methods, extracting benefits from each method to solve problems more effectively.

Traditional machine learning is a subfield of artificial intelligence that involves the development of algorithms which can learn from and make predictions or decisions based on data. Unlike deep learning, which requires large datasets and focuses on artificial neural networks, traditional machine learning techniques often work well with smaller datasets and include methods, as shown in Figure 5, such as decision trees, support vector machines, and linear discriminant analysis. These algorithms build models by analyzing input data features and finding patterns or statistical relationships, which can then be used to make predictions on new, unseen data.

Deep learning has recently become a key component in the development of numerous industries, from healthcare [21] to computer security [22,23,24,25,26]. Deep learning, a subfield of machine learning, excels at managing high-dimensional and complex datasets. Its capacity to represent complex patterns using neural networks with numerous layers has revolutionized a variety of applications, including image recognition [27], natural language processing [28], and the diagnosis of medical conditions [21]. Deep learning has the ability to learn representations directly from the input data, without relying heavily on feature engineering, which is often required in traditional machine learning. Through multiple layers of artificial neural networks, deep learning algorithms can automatically learn hierarchical representations, where initial layers often capture low-level features and deeper layers capture more complex and abstract features. This capability to automatically extract and learn features from raw data is a major reason why deep learning has been successful in numerous fields. Deep learning can be categorized into two main groups, deep learning and hybrid deep learning; deep learning includes models such as transfer learning, convolutional neural networks, and recurrent neural networks, as shown in Figure 6.

Recent advancements have led to the fusion of deep learning and machine learning techniques, the so-called hybrid deep learning, as shown in Figure 7. Deep learning techniques such as convolutional neural networks typically serve as a feature extractor from raw data, while machine learning methods techniques serves as a classifier. This hybrid approach leverages the strengths of both methods, creating more effective and interpretable models.

Several review papers have been published on the subject of EEG BCIs, but they are either outdated [13,29,30,31] or focus on specific methods [32,33,34,35,36,37,38]. For example, the authors in [33,34,35] provide a significant review but they do not explore the current trends such as deep learning methods. In [36,37], the authors focus only on deep learning methods and in [38], Habashi et al. examine only the proposed methods that involve generative adversarial networks (GANs). Finally, Rajwal et al. [32] focus on the state-of-the-art methods that deploy CNNs.

While some of the aforementioned review papers report state-of-the-art deep learning methods, they overlook the current research trend, i.e., hybrid deep learning techniques. This could be attributed to the fact that both the field itself and the application of deep learning within it are relatively new and rapidly evolving. In this review, our aim is to address this gap. We provide a comprehensive survey of the entire EEG BCI motor imagery system pipeline, focusing on deep learning methods and placing a particular emphasis on these emerging hybrid deep learning methodologies. Furthermore, we present an in-depth evaluation of the most commonly implemented algorithms in the field, assessing their accuracy and computational efficiency. By doing this, we offer an up-to-date and detailed perspective on this exciting and fast-growing area of research.

Scopus and Google Scholar were utilized as the electronic databases to retrieve the articles. This review includes articles related to the following keywords: (1) “EEG” or “Electroencephalogram” or “electroencephalography”, (2) “MI” or “Motor Imagery”, (3) “DL” or “Deep learning”. The Scopus database returned 201 results while Google Scholar returned 92 results adding up to a total of 293 results. From the returned results, we excluded duplicate papers and papers that were not relevant to the review, such as papers that included hybrid methods of signal acquisition, papers that did not utilize deep learning architectures, and papers with similar architectures, methods, and accuracies. After these filters were applied, the number of screened papers was 53 as shown in Figure 8.

The rest of this paper is organized as follows, in Section 2, “Datasets”, we present all the publicly available datasets that contains motor imagery tasks and relevant information about the datasets. Section 3, “Deep learning”, includes the deep learning architectures used in the literature for motor imagery brain–computer interfaces. In Section 4, “Hybrid deep learning”, we review the hybrid architectures that contain a combination of machine learning and deep learning architectures to tackle the problem. In Section 5, “Discussion”, we provide our conclusions and insights from the reviewed papers, and we provide directions for further research.

2. Datasets

Data play an indispensable role in the classification of EEG signals, as they provide the essential foundation for building and validating algorithms. Utilizing a rich set of EEG data, researchers and engineers can train classifiers to recognize patterns that correspond to different tasks. While private datasets may offer good and relevant data, their accessibility is often restricted to certain institutions or researchers. Public datasets, on the other hand, foster a more inclusive and collaborative approach, making them essential in advancing the field, and they serve as benchmarks in order to make the comparison among the proposed methods more efficient.

Table 1. Publicly available EEG motor imagery datasets.

Datasets	MI CLASSES *	Number of Electrodes	Sampling Rate, Hz	Number of Subjects	Number of Sessions	Number of Trials
BCI Competition II dataset 3b [39]	LH, RH	3	128	3	3	2800
BCI Competition III dataset 4a [40]	RH, RF	118	1000	4	1	1400
BCI Competition IV dataset 2a [41]	LH, RH, BL, T	22	250	9	2	5184
BCI Competition IV dataset 2b [41]	LH, RH	3	250	5	1	6480
EEGMMIDB [42]	BL, LF, RF, BF	64	160	12	1	9100
High Gamma Dataset [43]	LH, RH	128	160	14	13	1000
MIJoint [44]	RH, RE, RS	64	1000	25	7	7500
GigaDB [45]	LH, RH	62	1000	54	2	21,600
MISCP [46]	LH, RH, LL, RL, T	19	200	13	75	4000
Hohyun Cho et al. [47]	LH, RH	64	512	52	1	5500
Brodu N et al. [48]	LH, RH	11	512	1	3	560
Invividual imagery [49]	RH, FT	30	256	9	2	1400
Ofner et al. [50]	EF, EE, FS, FP, HO, HC	61	512	15	2	6300

* LH: left hand, RH: right hand, RL: right leg, LL: left leg, BL: both legs, T: tongue, RE: right elbow, RS: resting state, EF: elbow flection, EE: elbow extension, FS: forearm supination, FP: forearm pronation, HO: hands open, HC: hands closed, BF: both fists, LF: left fist, RF: right fist.

presents the characteristics of available MI public datasets, such as number and type of EEG motor imagery classes, number of electrodes, sampling rate, number of subjects per dataset, number of sessions, and total trials per dataset.

3. Deep Learning

In the following section, we delve into a comprehensive analysis of the most prevalent methods used in the deep learning domain for the interpretation of EEG (electroencephalography) data in the context of brain–computer interface (BCI) with a focus on motor imagery tasks. Our analysis explore a range of approaches from CNNs, which are extensively used due to their proficiency in handling spatial and temporal information, to transfer learning, deep neural networks, and other architectures.

3.1. Convolutional Neural Networks

CNNs mimic the operational principles of the human visual cortex and possess the ability to dynamically comprehend spatial hierarchies in EEG data, recognizing patterns associated with motor imagery tasks through multiple layer transformations [51]. A CNN architecture begins with an input layer that accepts raw or preprocessed EEG data as shown in Figure 9. These data can be represented in various formats, such as time–frequency images, allowing the network to effectively process and analyze the brain signals associated with motor imagery. These data are then convolved using multiple kernels or filters, enabling the network to learn local features. Subsequently, the network employs a pooling layer for dimensionality reduction, refining the comprehension of the information. As the model progresses through these layers, it acquires the capacity to understand increasingly complex features. The final component is a fully connected (dense) classification layer that maps the learned high-level features to the desired output classes, such as different types of motor imagery, effectively acting as a decision-making layer that converts abstract representations into definitive classifications.

Dose et al. proposed a CNN trained on 3 s of segments from EEG signals [53]. The proposed method achieved an accuracy of 80.10%, 69.72%, and 59.71% on two, three, and four MI classes, respectively, on the Physionet dataset.

Miao M et al. proposed a CNN with five layers to classify two motor imagery tasks, right hand and right foot, from the BCI Competition III-IV-a dataset, achieving a 90% accuracy [54].

Zhao et al. proposed a novel CNN with multiple spatial temporal convolution (STC) blocks and fully connected layers [55]. Contrastive learning was used to push the negative samples away and pull the positive samples together. This method achieved an accuracy of 74.10% on BCI III-2a, 73.62% on SMR-BCI, and 69.43% on OpenBMI datasets.

Liu et al. proposed an end-to-end compact multibranch one-dimensional CNN (CMO-CNN) network for decoding MI EEG signals, achieving 83.92% and 87.19% accuracies on the BCI Competition IV-2a and the BCI Competition IV-2b datasets, respectively [56].

Han et al. proposed a parallel CNN (PCNN) to classify motor imagery signals [57]. That method, which achieved an average accuracy of 83.0% on the BCI Competition IV-2b dataset, began by projecting raw EEG signals into a low-dimensional space using a regularized common spatial pattern (RCSP) to enhance class distinctions. Then, the short-time Fourier transform (STFT) collected the mu and beta bands as frequency features, combining them to form 2D images for the PCNN input. The efficacy of the PCNN structure was evaluated against other methods such as stacked autoencoder (SAE), CNN-SAE, and CNN.

Ma et al. proposed an end-to-end, shallow, and lightweight CNN framework, known as Channel-Mixing-ConvNet, aimed at improving the decoding accuracy of the EEG-Motor Raw datasets [58]. Unlike traditional methods, the first block of the network was designed to implicitly stack temporal–spatial convolution layers to learn temporal and spatial EEG features after EEG channels were mixed. This approach integrated the feature extraction capabilities of both layers and enhanced performance. This resulted in a 74.9% accuracy rate on the BCI IV-2a dataset and 95.0% accuracy rate on the High Gamma Dataset (HGD).

Ak et al. performed an EEG data analysis to control a robotic arm. In their work, spectrogram images derived from EEG data were used as input to the GoogLeNet. They tested the system on imagined directional movements—up, down, left, and right—to control the robotic arm [59]. The approach resulted in the robotic arm executing the desired movements with over 90% accuracy, while on their private dataset, they achieved 92.59% accuracy.

Musallam Y et al. proposed the TCNet-Fusion model, which used multiple techniques such as temporal convolutional networks (TCNs), separable convolution, depthwise convolution, and layer fusion [60]. This process created an imagelike representation, which was then fed into the primary TCN. During testing, the model achieved a classification accuracy of 83.73% on the four-class motor imagery of the BCI Competition IV-2a dataset and an accuracy of 94.41% on the High Gamma Dataset.

Zhang et al. proposed a CNN with a 1D convolution on each channel followed by a 2D convolution to extract spatial features based on all 20 channels [61]. Then, to deal with the high computational cost, the idea of pruning was used, which is a technique of reducing the size and complexity of the neural network by removing certain connections or neurons. In the proposed method, a fast recursive algorithm (FRA) was applied to prune redundant parameters in the fully connected layers to reduce computational costs. The proposed architecture achieved an accuracy of 62.7% in the OPENBCI dataset. A similar approach was proposed by Vishnupriya et al. [62] to reduce the complexity of their architecture. The magnitude-based weight pruning was performed on the network, which achieved an accuracy of 84.46% on two MI tasks (left hand, right hand) in Lee et al.’s dataset.

Shajil et al. proposed a CNN architecture to classify four MI tasks, using the common spatial pattern filter on the raw EEG signal, then using the spectrograms extracted from the filtered signals as input into the CNN [63]. The proposed method achieved an accuracy of 86.41% on their private dataset.

Korhan et al. proposed a CNN architecture with five layers [64]. The proposed architecture was compared using only the CNN without any filtering, then with five different filters, and finally, with common spatial patterns followed by the CNN with the last architecture, which achieved the highest accuracy of 93.75% in the BCI Competition III-3a dataset.

Alazrai et al. proposed a CNN network, with the raw signal transformed into the time–frequency domain with the quadratic time–frequency distribution (QTFD), followed by the CNN network to extract and classify the features [65]. The proposed method was tested on their two private datasets, with 11 MI tasks (rest, grasp-related tasks, wrist-related tasks, and finger-related tasks) and obtained accuracies of 73.7% for the able-bodied and 72.8% for the transradial-amputated subjects.

Table 2. Reviewed CNN architectures, datasets and their accuracies.

Authors	Accuracy	Dataset	MI Tasks
Dose et al. [53]	59.71%	EEGMMIDB	LH, RH, RS, BL
Miao M et al. [54]	90%	BCI III-4, private	RH, RF
Zhao R et al. [55]	74.10%, 73.62%, 69.43%	BCI III-2a, SMR-BCI, OpenBMI	RH, RF
Liu X et al. [56]	83.92%, 87.19%	BCI IV-2a, BCI IV-2b	LH, RH, BL, T
Han et al. [57]	83%	BCI IV-2b	LH, RH
Ma et al. [58]	74.9%, 95.0%	BCI IV-2a, HGD	LH, RH, BL, T
Ak et al. [59]	92.59%	Private	U, D, L, R
Musallam et al. [60]	83.73%, 94.41%	BCI IV-2a, HGD	LH, RH, BL, T
Zhang et al. [61]	62.7%	OpenBMI	LH, RH
Vishnupriya et al. [62]	84.46%	Lee et al.	LH, RH
Shajil et al. [63]	86.41%	Private	LH, RH, BH, BL
Korhan et al. [64]	93.75%	BCI III-3a	LH, RH, BL, T
Alazrai et al. [65]	73.7%, 72.8%	Private	RS, SDG, LG, ETG, RDW, EW, FI, FM, FR, FL, FT

LH: left hand, RH: right hand, RL: right leg, BL: both legs, T: tongue, RS: resting state, BF: Both fists, LF: left fist, RF: right fist, U: up, D: down, L: left, R: right, SDG: small-diameter grasp, LG: lateral grasp, ETG: extension-type grasp, RDW: ulnar and radial deviation of the wrist, EW: extension of the wrist, FI: flexion and extension of the index finger, FM: flexion and extension of the middle finger, FR: flexion and extension of the ring finger, FL: flexion and extension of the little finger.

summarizes the research articles that utilize CNNs along with the tasks, the datasets used, and their performance.

3.2. Transfer Learning

In the realm of machine learning, transfer learning offers a valuable approach to improve model performance and efficiency. It involves leveraging the knowledge learned from one task and applying it to a different but related task. The foundation of transfer learning lies in pretrained models, which have been trained on large-scale datasets. This approach is beneficial when considering the computational resources required for training deep learning models from scratch, i.e., time, complexity, and hardware. Transfer learning also offers a noble solution when available training data are not enough to train effectively novel deep learning models. In these situations, where collecting training data is hard or expensive, as in the case of EEG data, a need to develop robust models using available data from diverse domains arises. However, the effectiveness of transfer learning relies on the learned features from the initial task being general and applicable to the target task. By transferring this learned knowledge to a new model and fine-tuning it on a smaller, domain-specific dataset, we can effectively tackle new problems with limited labeled data.

Some popular deep learning models utilized for transfer learning are AlexNet [66], ResNet18 [67], ResNet50, InceptionV3 [68], and ShuffleNet [69]. These models are CNN networks trained on millions of images to classify different classes. For example, AlexNet consists of eight layers with weights, the first five are convolutional layers followed by max-pooling layers, and the last three layers are fully connected layers followed by a softmax layer to provide a probability distribution over the 1000 class labels. Figure 10 shows the architecture of the AlexNet architecture.

ResNet (residual network) is a deep learning model in which the weight layers learn residual functions with reference to the layer inputs. This method addresses the problem of vanishing gradients in deep neural networks by introducing skip connections, also known as residual blocks. The information can flow directly across multiple layers, making it easier for the network to learn complex features. There are various versions of ResNet utilized for EEG classification, e.g., ResNet34, ResNet50, etc. Figure 11 shows the architecture of ResNet.

While some research papers rely on pretrained models for transfer learning, others take a different approach. Let N be the total number of subjects in a dataset. Researchers train a custom CNN on N–1 subjects, and afterwards, they use the trained CNN as a base model for transfer learning. That is, they use the remaining N subject to train the aforementioned CNN, and afterwards, they finetune the whole model. Moreover, some research papers ([70,71]) opt for alternative architectures, such as the one proposed by Schirrmeister et al. [43], to facilitate transfer learning in their studies.

Zhang et al. utilized transfer learning to train a hybrid deep neural network (HDNN-TL) which consisted of a convolutional neural network and a long short-term memory model, to decode the spatial and temporal features of the MI signal simultaneously [72]. The classification performance on the BCI Competition IV-2a dataset by the proposed HDNN-TL in terms of kappa value was 0.8 (outperforming the rest of the examined methods).

Wei et al. [70] proposed a multibranch deep transfer network, the Separate-Common-Separate Network (SCSN) based on splitting the network’s feature extractors for individual subjects, and they also explored the possibility of applying maximum mean discrepancy (MMD) to the SCSN (SCSN-MMD). They tested their models on the BCI Competition IV-2a dataset and their own online recorded dataset, which consisted of five subjects (male) and four motor imageries (relaxing, left hand, right hand, and both feet). The results showed that the proposed SCSN achieved an accuracy of (81.8%, 53.2%) and SCSN-MMD achieved an accuracy of (81.8%, 54.8%).

Limpiti et al. used a continuous wavelet transform (CWT) to construct the scalograms from the raw signal, which served as input to five pretrained networks (AlexNet, ResNet18, ResNet50, InceptionV3, and ShuffleNet) [73]. The models were evaluated on the BCI Competition IV-2a dataset. On binary (left hand vs. right hand) and four-class (left hand, right hand, both feet, and tongue) classification, the ResNet18 network achieved the best accuracies at 95.03% and 91.86%.

Wei et al. [74] utilized a CWT to convert the one-dimensional EEG signal into a two-dimensional time–frequency amplitude representation as the input of a pre-trained AlexNet and fine-tuned it to classify two types of MI signals (left hand and right hand). The proposed method achieved a 93.43% accuracy on the BCI Competition II-3 dataset.

Arunabha proposed a multiscale feature-fused CNN (MSFFCNN) efficient transfer learning (TL) and four different variations of the model including subject-specific, subject-independent, and subject-adaptive classification models to exploit the full learning capacity of the classifier [75]. The proposed method achieved a 94.06% accuracy on for four different MI classes (i.e., left hand, right hand, feet, and tongue) on the BCI Competition IV-2a dataset.

Chen et al. proposed a subject-weighted adaptive transfer learning method in conjunction with MLP and CNN classifiers, achieving an accuracy of 96% on their own recorded private dataset [76].

Zhang et al. proposed five schemes for the adaptation of a CNN to two-class motor imagery (left hand, right hand), and after fine-tuning their architecture, they achieved an accuracy of 84.19% on the public GigaDB dataset [71].

Solorzano et al. proposed a method based on transfer learning in neural networks to classify the signals of multiple persons at a time [77]. The resulting neural network classifier achieved a classification accuracy of 73% on the evaluation sessions of four subjects at a time and 74% on three at a time on the BCI Competition IV-2a dataset.

Li et al. proposed a cross-channel specific–mutual feature transfer learning (CCSM-FT) network model with training tricks used to maximize the distinction between the two kinds of features [78]. The proposed method achieved an 80.26% accuracy on the BCI Competition IV-2a dataset.

A summary of the aforementioned methods can be found in

Table 3. Reviewed transfer learning architectures and their accuracies.

Authors	Accuracy	Dataset	MI Tasks
Zang et al. [72]	0.8 (kappa)	BCI IV-2a	LH, RH, BL, T
Wei et al. [70]	81.8%, 54.8%	BCI IV-2a, private	LH, RH
Limpiti et al. [73]	95.03%, 91.86%	BCI IV-2a	LH, RH, BL, T
Wei M et al. [74]	93.43%	BCI II-3	LH, RH
Arunabha M. [75]	94.06%	BCI IV-2a	LH, RH, BL, T
Chen et al. [76]	96%	Private	F, B, L, R, S
Zhang et al. [71]	84.19%	GigaDB	LH, RH
Solorzano et al. [77]	74%	BCI IV-2a	LH, RH, BL, T
Li D et al. [78]	80.26%	BCI IV-2a	LH, RH, BL, T

LH: left hand, RH: right hand, RL: right leg, BL: both legs, T: tongue, RS: resting state, BF: Both fists, LF: left fist, RF: right fist, F: forward, B: backward, L: left, R: right, R: rest, S: stop.

.

3.3. Deep Neural Networks

Deep neural networks, a subset of artificial neural networks, have the ability to tackle complex problems. Unlike shallow neural networks that consist of only a few layers, deep neural networks are characterized by their depth, featuring multiple hidden layers between the input and output layers. Each hidden layer progressively extracts higher-level features from the data, allowing the network to learn complex representations and patterns from vast quantities of data.

Suhaimi et al. [79] proposed a deep neural network with four layers each including 50, 30, 15, and 1 node, respectively, achieving a 49.5% classification accuracy in the BCI Competition IV-2b with two MI tasks selected (arm and foot movement).

Cheng et al. proposed a deep neural network which accepted as input multiple sub-bands of the raw signal extracted by a sliding window strategy [80]. Under these sub-bands, diverse spatial–spectral features were extracted and fed into a deep neural network for classification, achieving an accuracy of 71.5% on their private dataset.

Yohanandan et al. proposed a binary classifier (relaxed and right-handed MI tasks) using a deep neural network with the μ-rhythm (8–12 Hz frequency) data being fed into the network [81]. The authors used different sliding windows from 1 s to 9 s to determine the highest-accuracy window. An average accuracy of ~83% was achieved on their privately collected dataset from seven human volunteers.

Kumar et al. proposed a deep neural network for the classification of extracted features using a common spatial pattern in the BCI Competition III-4a dataset, achieving an accuracy of ~85% on two MI tasks (right hand and left foot) [82].

Table 4. Reviewed deep neural network architectures and their accuracies.

Authors	Accuracy	Dataset	MI Tasks
Suhaimi et al. [79]	49.5%	BCI Competition 2b	LH, RH
Cheng et al. [80]	71.5%	Private	LH, RH
Yohonanndan et al. [81]	83%	Private	RS, RH
Kumar et al. [82]	~85%	BCI Competition III-4a	RH, LF

LH: left hand, RH: right hand, LF: left foot, RS: resting state.

shows the performance of each one of the aforementioned architectures.

3.4. Others

Several alternative methods have been proposed for classifying motor imagery (MI) tasks, aiming at leveraging the potential of different deep learning techniques. Autoencoders [83], which are designed for data reconstruction, have been explored in the context of MI task classification. Autoencoders are neural network architectures that consist of two main phases: encoding and decoding. During the encoding phase, the input signal is passed through a neural network with a progressively reduced number of neurons in each layer until it reaches the bottleneck layer, which has a lower dimensionality compared to the input data. In the decoding phase, the network strives to reconstruct the original signal from this lower-dimensional representation, preserving essential information. This encoding stage in autoencoders enables them to effectively learn compressed representations of input data, such as EEG data, by reducing its dimensionality while retaining significant information. Figure 12 shows an autoencoder used to reconstruct an EEG signal.

Autthasan et al. proposed an end-to-end multitask autoencoder and tested it on three datasets, BCI Competition IV-2a, SMR-BCI, and OpenBMI, achieving accuracies of 70.09%, 72.95%, and 66.51%, respectively [85].

Similarly, capsule networks, which introduce a hierarchical structure to capture pose and viewpoint information, have shown promising results in MI task classification [86]. Capsules in capsule networks utilize vector-based representations. This property enables the network to capture hierarchical relationships and spatial dependencies among features. Each capsule comprises a group of neurons, with each neuron’s output representing a different property of the same feature, enabling the recognition of the whole entity by first identifying its parts. Ha et al. proposed a capsule network, using the images extracted with the short-time Fourier transform as input to the capsule network [87]. Their proposed method achieved a 77% accuracy on the BCI competition IV-2b dataset (left-hand and right-hand MI tasks).

Long short-term memory (LSTM) networks [88], a type of recurrent neural network, have been utilized to model temporal dependencies in MI data, enabling effective sequence learning for classification. Leon-Urbano et al. proposed an LSTM approach on an MNE python library dataset which consisted of two MI tasks (feet, hands), and after fine-tuning their model, they achieved a 90% accuracy [89]. Saputra et al. also deployed an LSTM network on the BCI Competition IV-2a dataset, achieving an accuracy of 49.65% [90]. Hwang et al. also performed a classification based on an LSTM on the BCI competition IV-2a dataset with a feature extraction based on overlapping band-based FBCSP (filter-bank common spatial pattern), with an accuracy of 97% [91].

Ma et al. proposed a parallel architecture including a temporal LSTM and a spatial bidirectional LSTM [92]. The proposed method was tested on the four MI tasks (moving both feet, both fists, left fist and right fist) from the EEGMMIDB dataset and achieved an accuracy of 68.20%.

Another proposed method is the restricted Boltzmann machine [93], a type of probabilistic graphical model, leveraging its ability to model joint probability distributions. Xu et al. utilized a restricted Boltzmann machine and a support vector machine (SVM) to classify and recognize deep multiview features [94]. The proposed method achieved an accuracy of 78.50% on the BCI competition IV-2a dataset.

Moreover, metalearning [95] empowers models to acquire the skill of learning on their own, with a limited quantity of data. This is achieved through training the model on a diverse range of tasks, allowing it to leverage the knowledge gained from these tasks when presented with new challenges. Among the various metalearning algorithms, one of the most prominent ones is MAML (model-agnostic metalearning) [95]. MAML trains the model to efficiently update its parameters, facilitating a rapid adaptation to new tasks with minimal updates. Li et al. proposed a metalearning method which learned from the output of other machine learning algorithms [96]. The proposed method achieved an 80% accuracy on the Physionet dataset (on left fist vs. right fist and both fists vs. both feet).

Contrastive learning [97] is a self-supervised learning technique that aims to create meaningful representations by contrasting positive and negative pairs of data. Han et al. proposed the so-called contrastive learning network. The proposed method was tested on the BCI competition IV-2a dataset achieving an accuracy of 79.54% when all the training labels were used [98].

A deep belief network (DBN) [99] is an unsupervised neural network known for its feature extraction from raw data. It uses a two-step training process: unsupervised pretraining with a restricted Boltzmann machine and supervised fine-tuning. Li et al. proposed a deep belief architecture where the time–frequency information from the raw EEG signal was fed into the DBN, which was used for the identification and classification [100]. The proposed method achieved an accuracy of 93.57% on the BCI competition II-3 dataset.

A synopsis of the above-mentioned proposals can be found in

Table 5. Other reviewed deep learning architectures and their accuracies.

Authors	Accuracy	Dataset	MI Tasks	Architecture
Autthasan et al. [85]	70.09%, 72.95% 66.51%	BCI IV-2a, SMR_BCI, Open BCI	LH, RH, BL, T	Autoencoder
Ha et al. [87]	77%	BCI IV-2b	LH, RH	Capsule network
Urbano et al. [89]	90%	MNE dataset	BF, BH	LSTM
Saputra et al. [90]	49.65%	BCI IV-2a	LH, RH, BL, T	LSTM
Hwang et al. [91]	97%	BCI IV-2a	LH, RH, BL, T	LSTM
Ma et al. [92]	68.20%	EEGMMIDB	LF, RF, BL, BF	LSTM, bi-LSTM
Xu et al. [94]	78.50%	BCI IV-2a	LH, RH, BL, T	Boltzmann Machine
Li et al. [96]	80%	EEGMMIDB	LF, RF, BL, BF	Meta-learning
Han et al. [98]	79.54%	BCI IV-2a	LH, RH, BL, T	Contrastive learning
Li et al. [100]	93.57%	BCI II-3	LH, RH	Deep belief network

LH: left hand, RH: right hand, RL: right leg, BL: both legs, T: tongue, RS: resting state, BF: both fists, LF: left fist, RF: right fist.

.

4. Hybrid Methods

Hybrid neural networks are a powerful fusion of different types of artificial neural networks, combining the strengths of various architectures to address complex problems effectively. By integrating components from different neural network types, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and feedforward neural networks (FNNs), hybrid networks can leverage their specialized capabilities to tackle complex problems.

4.1. CNN-Based

4.1.1. CNN and LSTM

Amin et al. proposed a novel deep learning-based lightweight model based on an attention–inception convolutional neural network and long short-term memory [101]. The model was tested on the BCI Competition IV-2a dataset and the High Gamma Dataset, achieving accuracies of 82.8% and 97.1%, respectively.

Khademi et al. proposed a CNN-LSTM hybrid network to extract spatial and temporal sequence features simultaneously and compared it with the same architecture but with pretrained CNN models, ResNet-50 and Inception-v3 [102]. A CWT was used to generate a 2D representation in the time–frequency domain as input for the CNN. The highest accuracy was 86% and achieved with the Inception-v3 hybrid network on the BCI Competition IV-2a dataset.

Echtioui et al. compared a CNN model and a CNN-LSTM model with similar architectures, showing that the addition of the LSTM layer in the model performed worse in terms of accuracy [103]. The CNN-LSTM model achieved a 55.55% accuracy on the BCI Competition IV-2a dataset as opposed to 62.45% for their CNN method.

Li et al. proposed a hybrid network combining a CNN, a DNN, and an LSTM, each as an independent classifier [104]. First, the HNN made a separate prediction for each of the three networks. These predictions were then normalized so that they added up to one. This was done to ensure that the predictions from the three networks could be compared with each other. The maximum prediction from each network was then selected. This gave three values, one for each network. The largest of these three values was then selected as the final prediction. The proposed method achieved an accuracy of 72.22% on the BCI competition IV-2a dataset.

Li et al. [105] proposed a hybrid neural network combining a deep-separation CNN (DSCNN), which is a convolution network divided into two or more convolutions to produce the same output, and a bidirectional LSTM (BLSTM). The proposed model achieved a 98.09% accuracy on the EEGMMIDB dataset, which contains five tasks (eyes closed, open/close left and right fists and open/close both fists, and open/close both feet).

Fadel et al. proposed a hybrid neural network consisting of a deep CNN followed by an LSTM, layer using as input to the model delta [0.5–4 Hz], mu [8–13 Hz], and beta [13–30 Hz] frequency bands [106]. The model was tested on the EEGMMIDB dataset, which includes five different classes (four motor imagery tasks and one resting task), achieved an accuracy of 70.64%.

Table 6. Reviewed hybrid CNN-LSTM architectures and their accuracies.

Authors	Accuracy	Dataset	MI Tasks
Amin et al. [101]	82.8%, 97.1%	BCI IV-2a, HGD	LH, RH, BL, T
Khademi et al. [102]	86%	BCI IV-2a	LH, RH, BL, T
Echtioui et al. [103]	55.55%	BCI IV-2a	LH, RH, BL, T
Li et al. [104]	72.22%	BCI IV-2a	LH, RH, BL, T
Li et al. [105]	98.09%	EEGMMIDB	RC, LF, RF, BF, BL
Fadel et al. [106]	70.64%	EEGMMIDB	RC, LF, RF, BF, BL

LH: left hand, RH: right hand, RL: right leg, RS: resting state, BL: Both legs.

shows in brief the hybrid CNN-LSTM architectures.

4.1.2. CNN and Autoencoders

Tabar et al. proposed a model with a CNN acting as a feature extractor and a stacked autoencoder with six hidden layers acting as a classifier [107]. The proposed architecture achieved an accuracy of 77.6% in the BCI Competition IV-2b dataset.

Dai et al. proposed a model with a CNN acting as a feature extractor but a variation autoencoder which is a directed model that uses learned approximate inference and can be trained purely with gradient-based methods, with seven layers acting as a classifier [108]. The proposed method was tested on the BCI competition IV dataset 2b achieving a kappa value of 0.564.

Hwaidi et al. proposed a network with a variational autoencoder to denoise the signal, followed by a CNN model to extract features and classify the processed signal [109]. The model was tested on the MI tasks of the Physionet dataset MI tasks, left fist, right fist, both fists, and both feet, achieving an accuracy of 98.20%.

A summary of these methods is given on

Table 7. Reviewed hybrid CNN + AE architectures and their accuracies.

Authors	Accuracy	Dataset	MI Tasks
Tabar et al. [107]	77.6%	BCI IV-2b	LH, RH, BL, T
Dai et al. [108]	0.564 (kappa)	BCI IV-2b	LH, RH
Hwaidi et al. [109]	98.20%	EEGMMIDB	RC, LF, RF, BF, BL

LH: left hand, RH: right hand, RL: right leg, RF: right fist, LF: left fist, RS: resting state, BL: both legs, T: tongue, BF: both fists.

.

4.1.3. Other CNN Architectures

Gomes et al. proposed a hybrid pretrained CNN (VGG16 and LeNet) and random forests with 100 trees to classify right- and left-hand MI tasks on the BCI Competition IV-2b dataset [110]. Pseudosinogram images were fed to the pretrained model to extract features and then to a random forest classifier. The VGG16 pretrained model slightly outperformed the LeNet with an accuracy of 89.13 and 89.10, respectively, after performing data augmentation.

Ma et al. proposed a CNN that extracted relevant spatial information, which was followed by a transformer layer which captured the long-range dependencies and temporal relationships between different EEG signal segments [111]. The proposed method achieved an accuracy of 83.91% on the BCI Competition IV-2a dataset.

Gao et al. [112] proposed a gated recurrent unit (GRU) and convolutional neural network (CNN), receiving inputs simultaneously, with the CNN extracting frequency and spatial features and the GRU extracting temporal features from the signal, combining their output as an input to the classifier. The proposed method was tested on the BCI competition IV-2a dataset and achieved an accuracy of 80.7%.

Ye et al. proposed a hybrid model with a CNN architecture with three layers with a “ReLU” activation, followed by a gated recurrent unit (GRU) with 64 units to extract the time-dependent relationship [113]. This method achieved an accuracy of 99.40% on the BCI competition IV-2a dataset.

In

Table 8. Other CNN-based architectures reviewed with their accuracies and architecture.

Authors	Accuracy	Dataset	MI Tasks	Architecture
Gomes et al. [110]	89.13%	BCI IV-2b	RH, LH	CNN + RF
Ma et al. [111]	90%	BCI IV-2a	RH, LH, BL, T	CNN + Transformer
Gao et al. [112]	80.7%	BCI IV-2a	RH, LH, BL, T	CNN + GRU
Ye et al. [113]	99.40%	BCI IV-2a	RH, LH, BL, T	CNN + GRU

LH: left hand, RH: right hand, RL: right leg, RS: resting state, BL: both legs, T: tongue.

, a summary of the discussed architectures is given.

4.2. Other Methods

Almagor et al. utilized an autoencoder to denoise the signal, comprising of three layers for the encoder and three layers for the decoder, followed by a feature extractor (filter-bank common spatial patterns) and a feature selector (mutual information-based best individual feature), to act as an input to the SVM classifier [114]. The proposed method achieved an accuracy of 59.6% on two MI tasks (right hand, relax) on their privately collected dataset.

Stephe et al. [115] proposed a method which first used an empirical mode decomposition (EMD) to decompose the EEG signals into intrinsic mode functions (IMFs), and then these signals were feed into a GAN which was trained to distinguish between EEG signals for two different motor imagery tasks (right hand, right foot). The proposed method achieved an accuracy of 95.29% on the BCI competition III-4a dataset.

Xu et al. proposed a method where time, frequency, time–frequency, and spatial features were extracted from the raw EEG signal, then fed into a multilayer restricted Boltzmann machine network followed by a support vector machine for the classification [94]. The proposed method achieved an accuracy of 78.50% on the BCI competition IV-2a dataset.

Jiang et al. proposed an autoencoder for dimensionality reduction, in combination with a transformer layer, which consisted of an encoder and a decoder, each with a multihead attention layer and a feedforward layer [116]. The proposed method was tested on the BCI Competition III-3 dataset and achieved an accuracy of 91.30% for left-hand and right-hand MI tasks.

A short summary of the above papers is presented in

Table 9. Other reviewed architectures and their accuracies and architecture.

Authors	Accuracy	Dataset	MI Tasks	Architecture
Almagor et al. [114]	59.6%	Private	RH, RS	AE + SVM
Stephe et al. [115]	95.29%	BCI III-4a	RH, RF	EMD + GAN
Xu et al. [94]	78.50%	BCI IV-2a	RH, LH, BL, T	RBM + SVM
Jiang et al. [116]	91.30%	BCI III-3	LH, RH	AE + transformer

LH: left hand, RH: right hand, RS: resting state, BL: Both legs, T: tongue, RF: right foot.

.

5. Discussion

This literature review provides a comprehensive overview of the flourishing research landscape within the realm of brain–computer interfaces (BCIs) utilizing electroencephalograms (EEGs), with a specific focus on the motor imagery paradigm. Notably, the integration of deep learning methodologies, particularly convolutional neural networks (CNNs), has emerged as a prominent and successful approach, yielding notable improvement in terms of accuracy rates when applied to relevant datasets. It is imperative to acknowledge that the efficacy of deep learning and hybrid deep learning, comes hand in hand with the substantial computational power it demands, thereby signifying a notable cost implication. The complex nature of CNN algorithms necessitates a substantial volume of training patterns, ranging from tens of thousands to even millions in certain instances, to facilitate an optimal performance and robust generalization. Complex algorithms benefit from extensive and robust datasets, with the BCI competition IV-2a dataset serving as a key reference point for researchers. However, the EEG-based BCI field urgently requires more comprehensive and larger public datasets to propel progress and innovation. While combining datasets may seem like a viable solution to enhance dataset size, the diverse sampling rates, electrode configurations, and motor imagery tasks across different datasets present integration challenges. To address this, some researchers turn to data augmentation techniques, such as utilizing GANs, to artificially expand dataset size. This not only helps prevent overfitting but also enhances model resilience, particularly in the presence of real-world environmental noise, resulting in more robust and practical EEG-based BCI models. Besides the requirement for large datasets, these complex algorithms also require a vast amount of computational power and time to train. For this reason, some authors such as Zhang et al. [61] have addressed this problem by using techniques such as pruning, to reduce the amount of time needed to train the network. As the field advances, it becomes imperative to not only address the computational demands of these intricate architectures but also to simultaneously enhance their efficiency and real-world applicability.

Within the realm of brain–computer interface (BCI) research, a consistent pattern emerges: the notable increases in accuracy can predominantly be attributed to the initial stages of preprocessing and the extraction of relevant features. This encompasses activities such as enhancing signals and exploring the time–frequency domain. This pivotal preparatory phase, which involves refining raw signals and extracting informative features, stands as a pivotal factor in achieving substantial progress in accuracy across various algorithmic approaches, which can be seen in Korhan et al.’s paper [64], which compares the same network architecture with different feature inputs.

The concept of “BCI illiteracy” [20] holds some significance. BCI illiteracy refers to the challenge that some individuals may face in effectively modulating their brain activity to produce distinguishable patterns during motor imagery tasks. This phenomenon can stem from various factors, such as a lack of familiarity with the task, insufficient cognitive engagement, or physiological variations that affect EEG signals. Addressing MI illiteracy is essential in designing inclusive and accessible BCIs, necessitating personalized training protocols, task adaptations, and innovative algorithms to accommodate users with varying levels of proficiency in generating discernible MI-related EEG patterns.

In the realm of motor imagery (MI), the number of commands that can be reliably extracted is inherently limited, as can be observed by the dataset’s distinct classes. This limitation arises due to the finite nature of distinct motor imagery tasks that users can effectively perform and differentiate in their mental simulations. The challenge lies in striking a balance between expanding the range of commands for diverse applications and maintaining a manageable set of tasks that users can consistently generate through mental imagery. This constraint underscores the importance of thoughtful task selection and user-centric design in the development of effective systems within the MI paradigm.

Finally,

Table 10. Advantages and disadvantages of the reviewed methods.

Architecture	Advantages	Disadvantages
Deep learning	High accuracy	High computational complexity, requires a vast quantity of data
CNN	Effective feature extraction	Computationally expensive
Auto encoder	Unsupervised feature learning	Potential for vanishing/exploding gradients
Neural network	High accuracy	Hyperparameter tuning may be needed
Hybrid deep learning	Combination of multiple architectures	Increased complexity, requires more computational resources, requires a vast quantity of data, requires more time to train the model
CNN + LSTM	Combined spatial and temporal information for improved accuracy	Computationally expensive
CNN + AE	Benefit from both spatial feature extraction and unsupervised pr-training	Computationally expensive, potential for vanishing/exploding gradients

offers a comparative analysis of the aforementioned algorithms, showcasing their respective strengths, weaknesses, and prospects for future research.

6. Future Research Directions

The consideration of computational time in real-time applications is crucial when evaluating machine learning and deep learning techniques. While deep learning indeed achieves elevated accuracy rates, the question of whether the benefits outweigh the computational burden becomes paramount. Although the inference of DL is quite fast and can be considered as real time, DL intricate architectures demand significant computational resources. That is, very powerful computers (in terms of processing power, memory, GPUs) are needed to train DL models, typically in a significant time. Balancing the pursuit of accuracy with the deployment of DL models that need less resources and less time to train remains a pivotal challenge.

The transition from a controlled lab environment with stable patients to a real-time application in a dynamic real-world setting introduces a notable shift in challenges. In the real environment, various types of noise can infiltrate the EEG signal, factors such as environmental interference, movement artifacts, and physiological variations. These noise sources can significantly degrade signal quality, making accurate interpretation and classification more difficult. Therefore, addressing the complexities of the real-world research field necessitates increased attention and consideration.

The dataset’s diversity and quality are crucial aspects, encompassing a range of characteristics specific to each dataset. Nevertheless, the absence of a standard benchmark dataset presents a notable constraint. This gap could potentially be mitigated through augmentation methods, including artificial augmentation. By layering additional variations onto existing data, these techniques aim to expand the dataset’s horizon, simulating diverse real-world scenarios. However, it is crucial to recognize that artificial augmentation might fall short in fully replicating the complexities of human experiences and interactions. Moreover, uncertainties persist regarding the exact conditions under which the experiments are conducted, adding an additional layer of complexity to the datasets.

7. Conclusions

In this paper, a thorough review of motor imagery (MI) EEG-based brain– computer interface (BCI) techniques was presented. This study delved into the realm of both deep learning and hybrid deep learning methodologies. Through an exploration of recent progress, this literature review provided a comprehensive and up-to-date insight into the advancements within this domain. Moreover, a discussion on the presented techniques was provided along with a comparison among them and future research directions.