Next Article in Journal
Dynamic Reconfiguration Method of Photovoltaic Array Based on Improved HPSO Combined with Coefficient of Variation
Previous Article in Journal
Chaos Moth Flame Algorithm for Multi-Objective Dynamic Economic Dispatch Integrating with Plug-In Electric Vehicles
Previous Article in Special Issue
Effects of Exercise Type and Gameplay Mode on Physical Activity in Exergame
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Electronics
Volume 12
Issue 12
10.3390/electronics12122743
electronics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Deep Comparisons of Neural Networks from the EEGNet Family

by
Csaba Márton Köllőd
1,2,3,*,†,
András Adolf
1,2,3,†,
Kristóf Iván
2,
Gergely Márton
2,3,‡ and
István Ulbert
2,3,‡
1
Roska Tamás Doctoral School of Sciences and Technology, 1083 Budapest, Hungary
2
Faculty of Information Technology and Bionics, Pázmány Péter Catholic University, 1083 Budapest, Hungary
3
Institute of Cognitive Neuroscience and Psychology, Research Centre for Natural Sciences, 1117 Budapest, Hungary
*
Author to whom correspondence should be addressed.
†
These authors contributed equally to this work.
‡
These authors contributed equally to this work.
Electronics 2023, 12(12), 2743; https://doi.org/10.3390/electronics12122743
Submission received: 16 May 2023 / Revised: 12 June 2023 / Accepted: 16 June 2023 / Published: 20 June 2023
(This article belongs to the Special Issue Advances in Augmenting Human-Machine Interface)
Browse Figures
Review Reports Versions Notes

Abstract

:
A preponderance of brain–computer interface (BCI) publications proposing artificial neural networks for motor imagery (MI) electroencephalography (EEG) signal classification utilize one of the BCI Competition datasets. However, these databases encompass MI EEG data from a limited number of subjects, typically less than or equal to 10. Furthermore, the algorithms usually include only bandpass filtering as a means of reducing noise and increasing signal quality. In this study, we conducted a comparative analysis of five renowned neural networks (Shallow ConvNet, Deep ConvNet, EEGNet, EEGNet Fusion, and MI-EEGNet) utilizing open-access databases with a larger subject pool in conjunction with the BCI Competition IV 2a dataset to obtain statistically significant results. We employed the FASTER algorithm to eliminate artifacts from the EEG as a signal processing step and explored the potential for transfer learning to enhance classification results on artifact-filtered data. Our objective was to rank the neural networks; hence, in addition to classification accuracy, we introduced two supplementary metrics: accuracy improvement from chance level and the effect of transfer learning. The former is applicable to databases with varying numbers of classes, while the latter can underscore neural networks with robust generalization capabilities. Our metrics indicated that researchers should not disregard Shallow ConvNet and Deep ConvNet as they can outperform later published members of the EEGNet family.
Keywords:
BCI; EEG; neural networks; EEGNet

1. Introduction

Artificial neural networks made a seminal contribution to the field of brain–computer interfaces (BCIs) when Schirrmeister et al. introduced Deep ConvNet and Shallow ConvNet in 2017 [1] for electroencephalographic (EEG) signal classification. Subsequently, neural networks have emerged as one of the most prominent topics in BCI literature.
BCIs are integrated systems comprising both software and hardware components. As delineated by Wolpaw et al. [2], these systems capture bioelectrical signals from the brain, extract useful information from the EEG–noise mixture, and translate them into computer commands. EEG is characterized as the fluctuation of postsynaptic membrane potential of neurons, recorded from the surface of the head. Figure 1 presents the components of a BCI system.
When a novel system is developed for motor imagery (MI) signal classification, it is frequently evaluated and contrasted with previously published systems utilizing one of the BCI Competition databases [3,4,5,6]. However, these datasets encompass records from a limited number of subjects, typically less than or equal to 10. Other open-access databases contain EEG records from more than 50 subjects but are predominantly avoided by researchers. One such database is the MI EEG dataset on PhysioNet [7] recorded using BCI2000 software [8], which comprises EEG records from 109 subjects. Another database was recorded using the OpenBMI toolbox [9] and contains data from 52 subjects, each of whom participated in two experimental days. Additionally, we have recorded our own dataset, which includes 25 experiments from 9 subjects [10]. Concerning the referenced literature, 39 instances employ one of the BCI Competition datasets, whereas a mere 6 instances utilize the MI EEG database available on PhysioNet. We presume that databases with more than 20 experimental days are sufficient for BCI system comparison.
In addition to offline comparisons, the Cybathlon competition [11] was established to assess the reliability of BCI systems operating in real-time, outside of laboratory conditions. Eleven teams successfully participated in the BCI discipline of Cybathlon 2016 [12], with two teams subsequently publishing their concepts, training protocols, and BCI systems [13,14]. As a continuation of this competition, the 2019 Cybathlon BCI Series and the 2020 Cybathlon Global Edition were organized, with multiple teams sharing their preparations and results [15,16,17,18,19,20].
Prior to the advent of neural networks, researchers endeavored to investigate and develop hand-crafted feature extraction methods in conjunction with simple classification algorithms. Blankertz et al. [21] successfully employed the common spatial patterns (CSP) algorithm with a linear discriminant analysis (LDA) classifier to control a cursor in one dimension. Barachant et al. [22] introduced Riemannian geometry for BCI with an LDA classifier, effectively classifying EEG covariance matrices. Lotte and Guan [23] proposed a unifying theoretical framework for regularizing the CSP and compared it with 10 other regularized versions of the CSP algorithm. Another feature extraction algorithm, based on the CSP, is the filter bank common spatial pattern (FBCSP) with a naïve Bayesian Parzen window classifier [24], which was compared with the ConvNets [1,25] on the BCI Competition IV 2a database. The winner of the BCI discipline of the Cybathlon competitions used the utilized power spectral density of the EEG signals as a feature [13,19] with a Gaussian classifier.
The introduction of Deep and Shallow ConvNets heralded a new trend in BCI development, shifting the focus from hand-crafted features to the creation of neural networks that not only classify signals but also incorporate the feature extraction step. Lawhern et al. [25] introduced EEGNet, drawing inspiration from previous neural networks designed for EEG signal processing, including MI-based BCIs [1,26,27,28]. It was demonstrated that EEGNet performs feature extraction similar to FBCSP. This neural network inspired numerous researchers, resulting in the development of many improved versions of EEGNet, culminating in the creation of an entire family (Table 1) of neural networks.
Several publications outside of the EEGNet family have underscored the importance of research on neural-network-based BCIs. Dokur and Olmez [43] presented a minimum distance network capable of learning at a faster rate than other deep neural networks. Fadel et al. explored the classification of image-like EEG data [44,45], while Han et al. focused on the development of parallel network architecture [46]. Jia et al. introduced a joint spatial–temporal architecture [47], which was further developed in [48] and successfully applied to cross-subject classification. Roy demonstrated [49] that classification accuracy can be further enhanced through the utilization of transfer learning.
Along with the development of neural networks, scientists started investigating the impact of transfer learning [50]. This methodology aims to transfer knowledge between two domains and increase classification accuracy. Khademi et al. [51] employed a CNN-LSTM hybrid model, which was pretrained on the ILSVRC subset of the ImageNet dataset to classify MI EEG signals. Their objective was to transfer knowledge from image classification and apply it to spatial EEG images generated using the continuous wavelet transformation method with a complex Morlet mother wavelet. Another approach is to utilize the entire EEG dataset and combine cross-subject and within-subject training, as demonstrated in [49,52,53]. In this case, knowledge is transferred from subjects whose data were not included in the neural network’s test set. The network is pre-trained on data from all but one subject, as in a cross-subject training procedure. However, the data of the test subject is also partitioned into training and test sets, as in within-subject training, and the training portion is used to fine-tune the pre-trained neural network. We opted for the latter version of transfer learning because it is architecture-independent and intended to apply it following artifact filtering.
In this article, all the experiments were conducted on data purified of artifacts because eye and muscle movement activity can distort the EEG signals [54]. This is attributable to the fact that the amplitude of electromyographic signals can be orders of magnitude greater than EEG signals. Furthermore, it has been demonstrated that artifacts can be successfully utilized for BCI purposes [55]; however, in our view, a genuine BCI should not rely on artifacts but solely on pure EEG signals. In addition, concerning a prominent international BCI competition, the Cybathlon “bionic Olympics” [11], participating BCI teams are required to implement an online artifact removal algorithm.
To reduce computational time for experiments, we arbitrarily selected Shallow and Deep ConvNet [1] as predecessors of EEGNet, the EEGNet itself [25], the EEGNet Fusion [30], and the MI-EEGNet [34] from the EEGNet family.

2. Materials and Methods

This section presents the databases and neural networks, along with the experimental setups and concepts. The code utilized in this study is accessible at: https://github.com/kolcs/bionic_apps (accessed on 12 June 2023).

2.1. Databases

We present the datasets employed for the EEGNet family comparisons. The databases were processed in an “independent days” configuration, meaning that if a subject participated in an experiment multiple times on different experimental days, the data were treated as if they had been recorded from distinct subjects. To our knowledge, EEG data can be significantly influenced by numerous factors, including recording setup, time of day, and the mental state of subjects, as also demonstrated in [56]. These could all lead to poorer performance if the data is merged concerning the subjects. It was also demonstrated in [57] that there is a great difference in cross-experimental day classification. With the independent days configuration, we aimed to overcome this problem and extend the number of subjects to strengthen the results of the statistical analyses, similar to in [58].

2.1.1. Physionet

The open-access PhysioNet database [7] is a valuable repository of numerous physiological datasets, including the EEG Motor Movement/Imagery Dataset, captured by Schalk et al. [8], using the BCI2000 paradigm control program. For convenience, we will refer to this specific dataset as the Physionet database. It encompasses four MI EEG signals obtained from 109 individuals: Left Hand, Right Hand, Both Hands, and Both Legs. The MI periods have a duration of 4 s and are interspersed with 4-second rest periods. The recordings were sampled at 160 Hz over 64 channels, without the use of hardware filters.
Four subjects out of the 109 were excluded from the database prior to the experiments. For subject 89, the labels were incorrect. In the case of subjects 88, 92, and 100, the timing was incorrect, with the execution of MI tasks and resting phases lasting 5.125 and 1.375 s, respectively. Moreover, the sampling frequency was altered from 160 Hz to 128 Hz. Other publications utilizing the Physionet database [30,52,59] also reported these problems.

2.1.2. Giga

Lee et al. [9] published an EEG dataset that included three paradigms: MI, event-related potential, and steady-state visually evoked potential. The experimental paradigms were conducted using the OpenMBI toolbox, custom written in MATLAB. We selected the files corresponding to the MI EEG paradigm from these three paradigms, which contains a 2-class classification problem, involving the imagination of Left Hand and Right Hand movement. The EEG signals were recorded using a 62-channeled BrainAmp amplifier system with a sampling rate of 1000 Hz. Fifty-four subjects participated in the experiments; each subject was present on two distinct experimental days. Therefore, in accordance with our independent days configuration, this dataset contains data from 108 subjects. To reduce the size of the raw EEG files, we resampled the data to a sampling frequency of 500 Hz.

2.1.3. BCI Competition IV 2a

Tangermann et al. [6] introduced the well-known and widely utilized BCI Competition IV database, which includes 5 sub-datasets with varying paradigms and challenges. This popular dataset is employed as a benchmark in the BCI literature to evaluate the developed methods and algorithms. In this study we utilize only the 2a sub-dataset, an MI dataset with Left Hand, Right Hand, Both Feet, and Tongue tasks. The EEG signals were recorded with a 250 Hz sampling frequency on 22 electrodes. The amplifier included a hardware bandpass filter between 0.5 and 100 Hz, and a notch filter at 50 Hz to remove the powerline noise.
This dataset was recorded with the assistance of 9 experimental subjects and each subject participated in two different experimental days. Therefore, concerning the independent days configuration, this dataset contains 18 subjects.

2.1.4. TTK

The TTK database [10], recorded at the Research Centre for Natural Sciences (TTK, as a Hungarian abbreviation), utilized a 64-channeled ActiChamp amplifier system (Brain Products GmbH, Gliching, Germany) to capture EEG signals at a sampling frequency of 500 Hz.
The EEG signals were recorded using a custom-built, MATLAB-based, paradigm leader code, General Offline Paradigm (GoPar), which was presented in the Supplementary Materials of [60] and is accessible at https://github.com/kolcs/GoPar (accessed on 12 June 2023). This code, inspired by the paradigm of the Physionet database, was designed to conduct multiple different MI paradigms with four tasks: Left Hand, Right Hand, Left Foot, and Right Foot. The paradigm began with an initial task consisting of a one-minute eye-open session followed by a one-minute eye-closed session, intended to capture the subjects’ full attention and prepare them for the core part of the experiment while serving as a baseline. Subsequently, two warmup sessions were conducted in which two of the four MI tasks were selected and practiced overtly and covertly to guide subjects on how to execute MI tasks. In total 25 experiments were conducted with 9 subjects. No hardware or software filters were applied during the EEG recording.

2.2. Signal Processing

Initially, EEG signals were filtered with a 5th-order Butterworth bandpass filter in the range of 1 to 45 Hz. Subsequently, a customized FASTER algorithm [54], as described in [60], was employed to eliminate artifacts associated with eye movements or muscle activity. The first stage involved the removal of EEG channels that exhibited consistent noise throughout the experiment, as determined by variance, correlation, and Hurst exponent measures. The second stage involved the exclusion of epochs containing motion artifacts (e.g., chewing, yawning) based on deviation from channel average, amplitude range, and variance parameters. In the third stage, eye-related artifacts were removed using independent component analysis. The fourth stage involved the individual filtering of EEG channels from epochs that were still considered noisy based on variance, median gradient, amplitude range, and channel deviation parameters. The fifth stage of the original FASTER algorithm, which involved the detection of artifacts across subjects, was omitted as our signal processing algorithm was designed to be subject-specific.
The resulting 4-s epochs were divided into 2-s windows with 0.1-s shifts to increase the sample size. The signals were then normalized using standard scaling, where the mean was set to zero and the standard deviation to one. These processed EEG windows were utilized for training and testing the classifiers of the BCI system.
For within-subject classification, 5-fold cross-validation was performed on a subject-wise basis, with the database split at the epoch level to ensure that windows originating from the same epoch were used exclusively in either the training or testing set. Approximately 10% of the training data was used as a validation set, with the split performed at the epoch level.

2.3. Neural Networks

This section describes the neural networks utilized in this study, as well as the methods and modifications employed in relation to the original networks.

2.3.1. Callbacks

During the training of the neural networks, a modified early stopping and model-saving strategy was implemented. The conventional early stopping approach [61] involves monitoring the validation loss and halting the learning process when it increases to prevent overfitting of the network. A patience parameter can also be specified to determine the number of training epochs that should elapse before the monitored value shows improvement. We extended this strategy by introducing an additional patience-like parameter termed “give up.” This strategy is intended to address training scenarios in which the validation loss increases above the initial training loss but subsequently decreases as the neural network begins to learn. The give up parameter specifies the number of training epochs that should elapse before the validation loss returns to its initial value. If the initial loss is reached within the give up limit, the original patience value is activated; otherwise, training is terminated.
Our model-saving strategy was designed to reflect our modified early-stopping approach. Until the initial validation loss was reached, model weights with the highest validation accuracy were saved. After reaching the initial validation loss, model weights were only saved if improvements were observed in both validation loss and validation accuracy. Prior to testing, the best model weights were restored.
Our experiments were conducted with a maximum of 500 training epochs, a give up value of 100, and a patience value of 20.

2.3.2. ConvNets

The Deep and Shallow ConvNets were implemented using the source code provided in [25] which employs several modified parameters relative to those originally published in [1]. No further modifications were made to the architecture of the networks.

2.3.3. EEGNets

The networks of the EEGNet family, including the EEGNet [25], the EEGNet Fusion [30], and the MI-EEGNet [34] were modified to enable automatic adaptation to databases with varying sampling frequencies, rather than requiring manual specification of input parameters. In the EEGNet publication [25], the authors explicitly stated that the filter size of the first convolutional block should be half of the sampling frequency. Accordingly, in our implementation, the kernel size was calculated based on the sampling frequency of the input signals, rather than being directly specified. This approach was also applied to EEGNet Fusion and MI-EEGNet.

2.4. Transfer Learning

In addition to subject-wise learning, we also investigated the effects of transfer learning. Test subjects were selected as distinct groups of 10, with the remaining subjects designated as pre-train subjects and used to establish the initial optimal weights for the neural networks. A validation set was separated from the pre-train data for use with our modified early stopping and model-saving strategy. Upon convergence of the pretraining phase, either through reaching the maximum number of training epochs or through early stopping, the best network weights were stored. For each test subject, 5-fold within-subject cross-validation was performed as described in the third paragraph of Section 2.2. Prior to each cross-validation step, the saved model weights were loaded and the selected training set for the test subject was used as fine-tuning data for the neural networks. During fine-tuning, validation sets were again employed in conjunction with our early stopping and model-saving strategies.

2.5. EEGNet Family Comparison

Extensive computational experiments were conducted on each database (Physionet, Giga, TTK, and BCI Competition IV 2a) to compare the performance of the neural networks from the EEGNet family (Shallow ConvNet, Deep ConvNet, EEGNet, EEGNet Fusion, MI-EEGNet). In cases where a subject participated in multiple experiments on different days, the data was treated as if it had been collected from multiple subjects, referred to as the independent days configuration. However, for the BCI Competition IV 2a dataset, we also conducted experiments in which data from a single subject was combined across recording dates to facilitate comparison with previous BCI studies. These experiments are denoted as “merged subject data”.
Both within-subject and transfer learning phases were conducted for each neural network and database. Cross-validation results were collected and normality tests were performed to determine the appropriate statistical test (t-test or Wilcoxon) for normally or non-normally distributed accuracy levels, respectively. The resulting p-values were adjusted using Bonferroni correction, with a preset significance level of 0.05.
In addition to comparing accuracy levels, we introduced two additional metrics to rank the performance of the neural networks. These metrics were evaluated on databases configured for independent days. The first metric measures the improvement in accuracy achieved by the EEGNet family relative to chance level, which can be applied to databases with varying numbers of classes. This metric was calculated and averaged for both within-subject and transfer learning. The second metric assesses the effect of transfer learning by comparing the results of within-subject classification with those of transfer learning classification. The difference between the two methods was calculated for each database configured for independent days.

2.6. Significance Investigation of Databases

To quantitatively evaluate our assumption that databases with more than 20 experimental days are sufficient for BCI system comparison, we investigated the number and quality of significant differences between databases. For each database configuration, two values were calculated: the sum of significance levels, as categorized in Table 2, and the count of significant differences. These values were then correlated with the number of subjects in each database.

3. Results

Upon obtaining five-fold cross-validated accuracy levels for all combinations of the four databases, five neural networks, and two learning methods (within-subject and transfer learning), normality tests indicated a non-normal distribution of the data. Consequently, the Wilcoxon statistical test with Bonferroni correction was employed for significance analysis. The results are presented in Figure 2 and Figure 3. In general, transfer learning was found to significantly improve performance across all databases except for BCI Competition IV 2a.
For the Physionet database (Figure 2A), within-subject classification using MI-EEGNet yielded the highest accuracy (0.4646) relative to other methods, while transfer learning using Deep ConvNet achieved the highest performance (0.5377).
For the Giga database (Figure 2B), MI-EEGNet achieved the highest accuracies of 0.725 and 0.7724 for within-subject and transfer learning, respectively. This network significantly outperformed other networks, with the exception of Shallow ConvNet in transfer learning mode.
Analysis of results from the TTK dataset (Figure 2C) revealed that EEGNet achieved the highest accuracies of 0.4437 and 0.4724 for within-subject and transfer learning, respectively. These results were significantly higher than those obtained using other networks, with the exception of MI-EEGNet.
For the BCI Competition IV 2a dataset, when treated as independent days (Figure 2D), Shallow ConvNet achieved accuracies of 0.719 and 0.733 for within-subject and transfer learning, respectively. In transfer learning mode, this network significantly outperformed other networks; however, its performance was comparable to that of EEGNet and MI-EEGNet in within-subject classification mode. When data from a single subject was merged across experimental days, Shallow ConvNet again achieved the highest accuracies of 0.749 and 0.7533 for within-subject and transfer learning, respectively; however, differences between networks were not significant.
To establish a hierarchy among the neural networks, we analyzed the improvement in accuracy achieved by the EEGNet family relative to chance level. Table 3 presents the ranking of these networks based on their training modes. Across all databases configured for independent days, MI-EEGNet exhibited the greatest average improvement in within-subject classification, while Shallow ConvNet outperformed other networks in transfer learning mode.
We also considered the extent to which neural network performance was enhanced by transfer learning, as presented in Table 4. Deep ConvNet exhibited the greatest improvement, achieving results that were on average 0.1 higher than those obtained using within-subject classification mode. In contrast, Shallow ConvNet, which ranked first in transfer learning performance, improved by only 0.05 relative to within-subject classification.
Finally, databases were ranked based on the number of significant differences observed between them. Table 5 presents the sum of significance ranges (corresponding to the number of stars in figures) and count of significant differences alongside the number of subjects in each database. The sum of significance ranges was found to be strongly correlated with the number of subjects in each database (r(3) = 0.7709), although this correlation was not statistically significant (p-value = 0.127014 > 0.05).

4. Discussion

Many articles presenting MI EEG signal classification using artificial neural networks from the EEGNet family report and compare their results on one of the BCI Competition databases. The aim of this study was to demonstrate the necessity of using datasets with large numbers of subjects for statistically significant comparisons. To this end, we compared the performance of five neural networks from the EEGNet family on four databases containing data from various subjects. With respect to the datasets, we introduced an independent day configuration in which data from a subject who participated in multiple experimental days were treated as if they had been collected from multiple subjects. This configuration was intended to increase the number of experiments and enhance the significance of comparisons. All four databases, namely BCI Competition IV 2a database [6], Physionet [7,8], Giga [9], and our TTK dataset [10], were used in this configuration. For the Physionet database, the authors reported that experiments were conducted with 109 volunteers, rendering the independent subject configuration irrelevant. For the BCI Competition IV 2a database, we also conducted an experiment in which data from a single subject was merged across experimental days (“merged subject data”) to facilitate comparison with other studies (Figure 3). These results were used to test our assumption regarding the correlation between the number of subjects in a database and the number of significant comparisons (Table 5). Although a strong correlation was observed between the number of subjects and our significance metric, it was not statistically significant. Nonetheless, Table 5 indicates that a database with only nine subjects is insufficient for significance testing. We therefore recommend using databases with large numbers of subjects, such as Physionet or Giga, for comparing BCI systems. Further investigation of our assumption will require additional open-access MI EEG databases.
We also wish to emphasize that our experiments used artifact-filtered EEG data, in contrast to previous studies on the investigated neural networks [1,25,30,34], which included only bandpass filtering and standardization prior to classification. In our signal processing step, we applied a fifth-order bandpass Butterworth filter with a range of 1 to 45 Hz, and utilized the FASTER algorithm [54] to detect and remove artifacts associated with eye movements and muscle activity. This is crucial to ensure that classification is performed on pure EEG signals rather than artifacts, because it has been demonstrated in [55] that electromyography can be successfully used for BCI purposes.
Many studies investigating the effects of transfer learning have utilized datasets without artifact filtering [49,51,52,53,62]. Our findings demonstrate that, even after artifact filtering, the implementation of transfer learning on databases with large numbers of subjects, such as Physionet and Giga, significantly enhances the accuracy of neural network classifications relative to within-subject classifications (Figure 2A,B). We also showed that Deep ConvNet exhibited the greatest improvement from transfer learning across all databases (Table 4). In contrast, Shallow ConvNet achieved the highest performance according to our “improvement from chance level” metric for all transfer-learning-trained neural networks (Table 3). Nevertheless, the differences between the ConvNets were insignificant concerning the Physionet and Giga databases (Figure 2A,B). In within-subject training mode, Deep ConvNet exhibited suboptimal performance, which may be attributed to an insufficient quantity of training data, a crucial factor for effective training of deep neural networks.
Our results highlight the importance of considering multiple factors when ranking the performance of neural networks. Relying solely on accuracy differences between networks and using unfiltered datasets with small numbers of subjects may lead to inconclusive results.
In addition to our findings, it is important to acknowledge the limitations of our research. Only a few neural networks were selected from the EEGNet family (Table 1) to shrink down the computational time. While it would be valuable to expand this comparison in future studies, the inclusion of additional networks may result in less significant findings due to the Bonferroni correction. Furthermore, several limitations were identified within the databases used. Only two databases, Physionet and Giga, were found to have more than 20 subjects. The TTK and BCI Competition IV 2a datasets were extended using our independent days configuration. The databases were recorded using different paradigms and contain varying amounts and types of motor imagery tasks. Additionally, they were recorded using different EEG amplifier systems with varying numbers of electrodes. As such, the consistency of the databases cannot be guaranteed. The aforementioned limitations may also have contributed to the observed variability in the classification results of the neural networks.
In future research, it would be worthwhile to explore the potential of transfer learning using data from multiple databases. However, this approach presents challenges due to variations in recording equipment and methodology across datasets, including differences in the position and number of electrodes, as well as sampling frequency. These issues must be addressed to facilitate effective transfer learning using data from multiple sources.

5. Conclusions

In this study, we conducted a critical comparison of neural networks from the EEGNet family, including Shallow ConvNet, Deep ConvNet, EEGNet, EEGNet Fusion, and MI-EEGNet, for the classification of MI EEG signals. Comparisons were performed using the BCI Competition IV 2a database as well as the Giga and Physionet databases, which comprise data from large numbers of subjects. Our TTK dataset was also utilized. Within-subject and transfer learning classifications were performed for each combination of database configuration and neural network, with all results subjected to five-fold cross-validation. Classification was performed on signals that had been cleaned of artifacts using the FASTER algorithm.
To our knowledge, this is the first study to compare neural networks from the EEGNet family on artifact-filtered databases comprising large numbers of subjects (>20) using cross-validated results. We demonstrated that transfer learning can improve classification performance even on artifact-filtered MI EEG data. To rank the performance of the neural networks, we introduced two metrics: one measuring improvement in accuracy relative to chance level and the other assessing improvement in classification performance achieved through transfer learning. These metrics indicated that Shallow ConvNet (0.2721, 0.0519) and Deep ConvNet (0.2598, 0.1075) outperformed more recently published networks from the EEGNet family. Finally, we showed that databases with small numbers of subjects (≤10) are insufficient for statistically significant comparison of BCI systems.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/electronics12122743/s1.

Author Contributions

Conceptualization, C.M.K.; methodology, C.M.K. and A.A.; software, C.M.K. and A.A.; formal analysis, C.M.K.; funding acquisition, K.I.; investigation, C.M.K. and A.A.; data curation, C.M.K.; writing—original draft preparation, C.M.K.; writing—review and editing, A.A., K.I., G.M. and I.U.; visualization, C.M.K.; supervision, G.M. and I.U.; project administration, I.U. All authors have read and agreed to the published version of the manuscript.

Funding

Prepared with the professional support of the Doctoral Student Scholarship Program of the Co-operative Doctoral Program of the Ministry of Innovation and Technology financed from the National Research, Development and Innovation Fund. Project no. FK132823 has been implemented with the support provided by the Ministry of Innovation and Technology of Hungary from the National Research, Development and Innovation Fund, financed under the FK 19 funding scheme. This research was also funded by the Hungarian Brain Research Program (2017 1.2.1-NKP-2017-00002) and the TUDFO/51757-1/2019-ITM grant from the Hungarian National Research, Development and Innovation Office.

Data Availability Statement

Databases and source codes are available under the following links, which were all accessed on 12 June 2023. Source codes: Signal processing and classification framework—https://github.com/kolcs/bionic_apps; Paradigm handler—https://github.com/kolcs/GoPar. Datasets: Physionet—https://physionet.org/content/eegmmidb/1.0.0/; Giga—http://gigadb.org/dataset/100542; BCI Competition IV—https://www.bbci.de/competition/iv/; TTK—https://hdl.handle.net/21.15109/CONCORDA/UOQQVK.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:
BCIBrain–Computer Interface
MIMotor Imagery
EEGElectroencephalography
CSPCommon Spatial Patterns
LDALinear Discriminant Analysis
FBCSPFilter Bank Common Spatial Pattern
TTKResearch Centre for Natural Sciences (HUN)
GoParGeneral Offline Paradigm

References

  1. Schirrmeister, R.T.; Springenberg, J.T.; Fiederer, L.D.J.; Glasstetter, M.; Eggensperger, K.; Tangermann, M.; Hutter, F.; Burgard, W.; Ball, T. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum. Brain Mapp. 2017, 38, 5391–5420. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  2. Wolpaw, J.R.; Birbaumer, N.; McFarland, D.J.; Pfurtscheller, G.; Vaughan, T.M. Brain–computer interfaces for communication and control. Clin. Neurophysiol. 2002, 113, 767–791. [Google Scholar] [CrossRef] [PubMed]
  3. Blankertz, B.; Muller, K.R.; Curio, G.; Vaughan, T.; Schalk, G.; Wolpaw, J.; Schlogl, A.; Neuper, C.; Pfurtscheller, G.; Hinterberger, T.; et al. The BCI competition 2003: Progress and perspectives in detection and discrimination of EEG single trials. IEEE Trans. Biomed. Eng. 2004, 51, 1044–1051. [Google Scholar] [CrossRef] [PubMed]
  4. Blankertz, B.; Muller, K.R.; Krusienski, D.; Schalk, G.; Wolpaw, J.; Schlogl, A.; Pfurtscheller, G.; Millan, J.; Schroder, M.; Birbaumer, N. The BCI competition III: Validating alternative approaches to actual BCI problems. IEEE Trans. Neural Syst. Rehabil. Eng. 2006, 14, 153–159. [Google Scholar] [CrossRef] [PubMed]
  5. Sajda, P.; Gerson, A.; Muller, K.R.; Blankertz, B.; Parra, L. A data analysis competition to evaluate machine learning algorithms for use in brain-computer interfaces. IEEE Trans. Neural Syst. Rehabil. Eng. 2003, 11, 184–185. [Google Scholar] [CrossRef]
  6. Tangermann, M.; Müller, K.R.; Aertsen, A.; Birbaumer, N.; Braun, C.; Brunner, C.; Leeb, R.; Mehring, C.; Miller, K.; Mueller-Putz, G.; et al. Review of the BCI Competition IV. Front. Neurosci. 2012, 6, 55. [Google Scholar] [CrossRef] [Green Version]
  7. Goldberger, A.L.; Amaral, L.A.; Glass, L.; Hausdorff, J.M.; Ivanov, P.C.; Mark, R.G.; Mietus, J.E.; Moody, G.B.; Peng, C.-K.; Stanley, H.E. PhysioBank, PhysioToolkit, and PhysioNet. Circulation 2000, 101, 215–220. [Google Scholar] [CrossRef] [Green Version]
  8. Schalk, G.; McFarland, D.J.; Hinterberger, T.; Birbaumer, N.; Wolpaw, J.R. BCI2000: A general-purpose brain-computer interface (BCI) system. IEEE Trans. Bio-Med Eng. 2004, 51, 1034–1043. [Google Scholar] [CrossRef]
  9. Lee, M.H.; Kwon, O.Y.; Kim, Y.J.; Kim, H.K.; Lee, Y.E.; Williamson, J.; Fazli, S.; Lee, S.W. EEG dataset and OpenBMI toolbox for three BCI paradigms: An investigation into BCI illiteracy. GigaScience 2019, 8, 16. [Google Scholar] [CrossRef] [Green Version]
  10. Köllőd, C.; Adolf, A.; Márton, G.; Wahdow, M.; Fadel, W.; Ulbert, I. TTK Dataset—4 Class Motor-Imagery EEG. 2022. Available online: https://hdl.handle.net/21.15109/CONCORDA/UOQQVK (accessed on 12 June 2023).
  11. Riener, R.; Seward, L.J. Cybathlon 2016. In Proceedings of the 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), San Diego, CA, USA, 5–8 October 2014; pp. 2792–2794. [Google Scholar] [CrossRef]
  12. Novak, D.; Sigrist, R.; Gerig, N.J.; Wyss, D.; Bauer, R.; Götz, U.; Riener, R. Benchmarking Brain-Computer Interfaces Outside the Laboratory: The Cybathlon 2016. Front. Neurosci. 2018, 11, 756. [Google Scholar] [CrossRef] [Green Version]
  13. Perdikis, S.; Tonin, L.; Saeedi, S.; Schneider, C.; Millán, J.d.R. The Cybathlon BCI race: Successful longitudinal mutual learning with two tetraplegic users. PLoS Biol. 2018, 16, e2003787. [Google Scholar] [CrossRef]
  14. Statthaler, K.; Schwarz, A.; Steyrl, D.; Kobler, R.; Höller, M.K.; Brandstetter, J.; Hehenberger, L.; Bigga, M.; Müller-Putz, G. Cybathlon experiences of the Graz BCI racing team Mirage91 in the brain-computer interface discipline. J. Neuroeng. Rehabil. 2017, 14, 129. [Google Scholar] [CrossRef] [Green Version]
  15. Benaroch, C.; Sadatnejad, K.; Roc, A.; Appriou, A.; Monseigne, T.; Pramij, S.; Mladenovic, J.; Pillette, L.; Jeunet, C.; Lotte, F. Long-Term BCI Training of a Tetraplegic User: Adaptive Riemannian Classifiers and User Training. Front. Hum. Neurosci. 2021, 15, 635653. [Google Scholar] [CrossRef]
  16. Hehenberger, L.; Kobler, R.J.; Lopes-Dias, C.; Srisrisawang, N.; Tumfart, P.; Uroko, J.B.; Torke, P.R.; Müller-Putz, G.R. Long-Term Mutual Training for the CYBATHLON BCI Race with a Tetraplegic Pilot: A Case Study on Inter-Session Transfer and Intra-Session Adaptation. Front. Hum. Neurosci. 2021, 15, 635777. [Google Scholar] [CrossRef]
  17. Korik, A.; McCreadie, K.; McShane, N.; Du Bois, N.; Khodadadzadeh, M.; Stow, J.; McElligott, J.; Carroll, A.; Coyle, D. Competing at the Cybathlon championship for people with disabilities: Long-term motor imagery brain–computer interface training of a cybathlete who has tetraplegia. J. Neuroeng. Rehabil. 2022, 19, 95. [Google Scholar] [CrossRef] [PubMed]
  18. Robinson, N.; Chouhan, T.; Mihelj, E.; Kratka, P.; Debraine, F.; Wenderoth, N.; Guan, C.; Lehner, R. Design Considerations for Long Term Non-invasive Brain Computer Interface Training with Tetraplegic CYBATHLON Pilot. Front. Hum. Neurosci. 2021, 15, 648275. [Google Scholar] [CrossRef] [PubMed]
  19. Tortora, S.; Beraldo, G.; Bettella, F.; Formaggio, E.; Rubega, M.; Del Felice, A.; Masiero, S.; Carli, R.; Petrone, N.; Menegatti, E.; et al. Neural correlates of user learning during long-term BCI training for the Cybathlon competition. J. NeuroEng. Rehabil. 2022, 19, 69. [Google Scholar] [CrossRef] [PubMed]
  20. Turi, F.; Clerc, M.; Papadopoulo, T. Long Multi-Stage Training for a Motor-Impaired User in a BCI Competition. Front. Hum. Neurosci. 2021, 15, 647908. [Google Scholar] [CrossRef] [PubMed]
  21. Blankertz, B.; Dornhege, G.; Krauledat, M.; Müller, K.R.; Curio, G. The non-invasive Berlin Brain–Computer Interface: Fast acquisition of effective performance in untrained subjects. NeuroImage 2007, 37, 539–550. [Google Scholar] [CrossRef] [PubMed]
  22. Barachant, A.; Bonnet, S.; Congedo, M.; Jutten, C. Riemannian Geometry Applied to BCI Classification. In Lecture Notes in Computer Science, Proceedings of the Latent Variable Analysis and Signal Separation, St. Malo, France, 27–30 September 2010; Vigneron, V., Zarzoso, V., Moreau, E., Gribonval, R., Vincent, E., Eds.; Springer: Berlin/Heidelberg, Germany, 2010; pp. 629–636. [Google Scholar] [CrossRef] [Green Version]
  23. Lotte, F.; Guan, C. Regularizing Common Spatial Patterns to Improve BCI Designs: Unified Theory and New Algorithms. IEEE Trans. Biomed. Eng. 2011, 58, 355–362. [Google Scholar] [CrossRef] [Green Version]
  24. Ang, K.K.; Chin, Z.Y.; Wang, C.; Guan, C.; Zhang, H. Filter Bank Common Spatial Pattern Algorithm on BCI Competition IV Datasets 2a and 2b. Front. Neurosci. 2012, 6, 39. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  25. Lawhern, V.J.; Solon, A.J.; Waytowich, N.R.; Gordon, S.M.; Hung, C.P.; Lance, B.J. EEGNet: A compact convolutional neural network for EEG-based brain–computer interfaces. J. Neural Eng. 2018, 15, 17. [Google Scholar] [CrossRef] [Green Version]
  26. Sakhavi, S.; Guan, C.; Yan, S. Parallel convolutional-linear neural network for motor imagery classification. In Proceedings of the 2015 23rd European Signal Processing Conference (EUSIPCO), Nice, France, 31 August–4 September 2015; pp. 2736–2740. [Google Scholar] [CrossRef]
  27. Sturm, I.; Lapuschkin, S.; Samek, W.; Müller, K.R. Interpretable deep neural networks for single-trial EEG classification. J. Neurosci. Methods 2016, 274, 141–145. [Google Scholar] [CrossRef] [Green Version]
  28. Tabar, Y.R.; Halici, U. A novel deep learning approach for classification of EEG motor imagery signals. J. Neural Eng. 2017, 14, 11. [Google Scholar] [CrossRef]
  29. Huang, W.; Xue, Y.; Hu, L.; Liuli, H. S-EEGNet: Electroencephalogram Signal Classification Based on a Separable Convolution Neural Network with Bilinear Interpolation. IEEE Access 2020, 8, 131636–131646. [Google Scholar] [CrossRef]
  30. Roots, K.; Muhammad, Y.; Muhammad, N. Fusion Convolutional Neural Network for Cross-Subject EEG Motor Imagery Classification. Computers 2020, 9, 72. [Google Scholar] [CrossRef]
  31. Musallam, Y.K.; AlFassam, N.I.; Muhammad, G.; Amin, S.U.; Alsulaiman, M.; Abdul, W.; Altaheri, H.; Bencherif, M.A.; Algabri, M. Electroencephalography-based motor imagery classification using temporal convolutional network fusion. Biomed. Signal Process. Control 2021, 69, 9. [Google Scholar] [CrossRef]
  32. Bria, A.; Marrocco, C.; Tortorella, F. Sinc-based convolutional neural networks for EEG-BCI-based motor imagery classification. arXiv 2021, arXiv:2101.10846. [Google Scholar] [CrossRef]
  33. Deng, X.; Zhang, B.; Yu, N.; Liu, K.; Sun, K. Advanced TSGL-EEGNet for Motor Imagery EEG-Based Brain-Computer Interfaces. IEEE Access 2021, 9, 25118–25130. [Google Scholar] [CrossRef]
  34. Riyad, M.; Khalil, M.; Adib, A. MI-EEGNET: A novel convolutional neural network for motor imagery classification. J. Neurosci. Methods 2021, 353, 109037. [Google Scholar] [CrossRef]
  35. Ma, W.; Gong, Y.; Zhou, G.; Liu, Y.; Zhang, L.; He, B. A channel-mixing convolutional neural network for motor imagery EEG decoding and feature visualization. Biomed. Signal Process. Control 2021, 70, 103021. [Google Scholar] [CrossRef]
  36. Riyad, M.; Khalil, M.; Adib, A. A novel multi-scale convolutional neural network for motor imagery classification. Biomed. Signal Process. Control 2021, 68, 102747. [Google Scholar] [CrossRef]
  37. Altaheri, H.; Muhammad, G.; Alsulaiman, M. Physics-inform attention temporal convolutional network for EEG-based motor imagery classification. IEEE Trans. Ind. Inform. 2022, 19, 2249–2258. [Google Scholar] [CrossRef]
  38. Li, H.; Ding, M.; Zhang, R.; Xiu, C. Motor imagery EEG classification algorithm based on CNN-LSTM feature fusion network. Biomed. Signal Process. Control 2022, 72, 103342. [Google Scholar] [CrossRef]
  39. Li, H.; Chen, H.; Jia, Z.; Zhang, R.; Yin, F. A parallel multi-scale time-frequency block convolutional neural network based on channel attention module for motor imagery classification. Biomed. Signal Process. Control 2022, 79, 104066. [Google Scholar] [CrossRef]
  40. Liu, X.; Shi, R.; Hui, Q.; Xu, S.; Wang, S.; Na, R.; Sun, Y.; Ding, W.; Zheng, D.; Chen, X. TCACNet: Temporal and channel attention convolutional network for motor imagery classification of EEG-based BCI. Inf. Process. Manag. 2022, 59, 103001. [Google Scholar] [CrossRef]
  41. Yao, H.; Liu, K.; Deng, X.; Tang, X.; Yu, H. FB-EEGNet: A fusion neural network across multi-stimulus for SSVEP target detection. J. Neurosci. Methods 2022, 379, 109674. [Google Scholar] [CrossRef]
  42. Gao, C.; Liu, W.; Yang, X. Convolutional neural network and riemannian geometry hybrid approach for motor imagery classification. Neurocomputing 2022, 507, 180–190. [Google Scholar] [CrossRef]
  43. Dokur, Z.; Olmez, T. Classification of motor imagery electroencephalogram signals by using a divergence based convolutional neural network. Appl. Soft Comput. 2021, 113, 107881. [Google Scholar] [CrossRef]
  44. Fadel, W.; Wahdow, M.; Kollod, C.; Marton, G.; Ulbert, I. Chessboard EEG Images Classification for BCI Systems Using Deep Neural Network. In Bio-inspired Information and Communication Technologies, Proceedings of the 12th EAI International Conference, BICT 2020, Shanghai, China, 7–8 July 2020; Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering; Chen, Y., Nakano, T., Lin, L., Mahfuz, M.U., Guo, W., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 97–104. [Google Scholar] [CrossRef]
  45. Fadel, W.; Kollod, C.; Wahdow, M.; Ibrahim, Y.; Ulbert, I. Multi-Class Classification of Motor Imagery EEG Signals Using Image-Based Deep Recurrent Convolutional Neural Network. In Proceedings of the 2020 8th International Winter Conference on Brain-Computer Interface (BCI), Gangwon, Republic of Korea, 26–28 February 2020; pp. 1–4. [Google Scholar] [CrossRef]
  46. Han, Y.; Wang, B.; Luo, J.; Li, L.; Li, X. A classification method for EEG motor imagery signals based on parallel convolutional neural network. Biomed. Signal Process. Control 2022, 71, 103190. [Google Scholar] [CrossRef]
  47. Jia, X.; Song, Y.; Yang, L.; Xie, L. Joint spatial and temporal features extraction for multi-classification of motor imagery EEG. Biomed. Signal Process. Control 2022, 71, 103247. [Google Scholar] [CrossRef]
  48. Jia, X.; Song, Y.; Xie, L. Excellent fine-tuning: From specific-subject classification to cross-task classification for motor imagery. Biomed. Signal Process. Control 2023, 79, 104051. [Google Scholar] [CrossRef]
  49. Roy, A.M. Adaptive transfer learning-based multiscale feature fused deep convolutional neural network for EEG MI multiclassification in brain–computer interface. Eng. Appl. Artif. Intell. 2022, 116, 105347. [Google Scholar] [CrossRef]
  50. Weiss, K.; Khoshgoftaar, T.M.; Wang, D. A survey of transfer learning. J. Big Data 2016, 3, 9. [Google Scholar] [CrossRef] [Green Version]
  51. Khademi, Z.; Ebrahimi, F.; Kordy, H.M. A transfer learning-based CNN and LSTM hybrid deep learning model to classify motor imagery EEG signals. Comput. Biol. Med. 2022, 143, 105288. [Google Scholar] [CrossRef] [PubMed]
  52. Mattioli, F.; Porcaro, C.; Baldassarre, G. A 1D CNN for high accuracy classification and transfer learning in motor imagery EEG-based brain-computer interface. J. Neural Eng. 2022, 18, 066053. [Google Scholar] [CrossRef]
  53. Zhang, R.; Zong, Q.; Dou, L.; Zhao, X.; Tang, Y.; Li, Z. Hybrid deep neural network using transfer learning for EEG motor imagery decoding. Biomed. Signal Process. Control 2021, 63, 102144. [Google Scholar] [CrossRef]
  54. Nolan, H.; Whelan, R.; Reilly, R.B. FASTER: Fully Automated Statistical Thresholding for EEG artifact Rejection. J. Neurosci. Methods 2010, 192, 152–162. [Google Scholar] [CrossRef]
  55. Noboa, E.; Rácz, M.; Szűcs, L.; Galambos, P.; Márton, G.; Eigner, G. Development of an EMG based SVM supported control solution for the PlatypOUs education mobile robot using MindRove headset. IFAC-PapersOnLine 2021, 54, 304–309. [Google Scholar] [CrossRef]
  56. Gibson, E.; Lobaugh, N.J.; Joordens, S.; McIntosh, A.R. EEG variability: Task-driven or subject-driven signal of interest? NeuroImage 2022, 252, 17. [Google Scholar] [CrossRef]
  57. Huang, G.; Hu, Z.; Chen, W.; Zhang, S.; Liang, Z.; Li, L.; Zhang, L.; Zhang, Z. M3CV: A multi-subject, multi-session, and multi-task database for EEG-based biometrics challenge. NeuroImage 2022, 264, 119666. [Google Scholar] [CrossRef] [PubMed]
  58. Castiblanco Jimenez, I.A.; Gomez Acevedo, J.S.; Olivetti, E.C.; Marcolin, F.; Ulrich, L.; Moos, S.; Vezzetti, E. User Engagement Comparison between Advergames and Traditional Advertising Using EEG: Does the User’s Engagement Influence Purchase Intention? Electronics 2022, 12, 122. [Google Scholar] [CrossRef]
  59. Fan, C.C.; Yang, H.; Hou, Z.G.; Ni, Z.L.; Chen, S.; Fang, Z. Bilinear neural network with 3-D attention for brain decoding of motor imagery movements from the human EEG. Cogn. Neurodyn. 2021, 15, 181–189. [Google Scholar] [CrossRef]
  60. Köllőd, C.; Adolf, A.; Márton, G.; Wahdow, M.; Fadel, W.; Ulbert, I. Closed loop BCI System for Cybathlon 2020. arXiv 2022, arXiv:2212.04172. [Google Scholar] [CrossRef]
  61. Prechelt, L. Early Stopping — But When? In Neural Networks: Tricks of the Trade, 2nd ed.; Montavon, G., Orr, G.B., Müller, K.R., Eds.; Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2012; pp. 53–67. [Google Scholar] [CrossRef]
  62. Kant, P.; Laskar, S.H.; Hazarika, J.; Mahamune, R. CWT Based Transfer Learning for Motor Imagery Classification for Brain computer Interfaces. J. Neurosci. Methods 2020, 345, 108886. [Google Scholar] [CrossRef] [PubMed]
Electronics 12 02743 g001 550
Figure 1. Components of a brain–computer interface system.
Figure 1. Components of a brain–computer interface system.
Electronics 12 02743 g001
Electronics 12 02743 g002 550
Figure 2. EEGNet family comparison on 4 databases handling the datasets in independent days configuration. The p-value annotation legend is the following: ns: 5 × 10 − 2 < p; *: 1 × 10 − 2 < p <= 5 × 10 − 2 ; **: 1 × 10 − 3 < p <= 1 × 10 − 2 ; ***: 1 × 10 − 4 < p <= 1 × 10 − 3 ; ****: p <= 1 × 10 − 4 . The mean of the data is presented with the ‘+’ symbol.
Figure 2. EEGNet family comparison on 4 databases handling the datasets in independent days configuration. The p-value annotation legend is the following: ns: 5 × 10 − 2 < p; *: 1 × 10 − 2 < p <= 5 × 10 − 2 ; **: 1 × 10 − 3 < p <= 1 × 10 − 2 ; ***: 1 × 10 − 4 < p <= 1 × 10 − 3 ; ****: p <= 1 × 10 − 4 . The mean of the data is presented with the ‘+’ symbol.
Electronics 12 02743 g002
Electronics 12 02743 g003 550
Figure 3. EEGNet family comparison on BCI Competition IV 2a. The p-value annotation legend is the following: ns: 5 × 10 − 2 < p; The mean of the data is presented with the ‘+’ symbol.
Figure 3. EEGNet family comparison on BCI Competition IV 2a. The p-value annotation legend is the following: ns: 5 × 10 − 2 < p; The mean of the data is presented with the ‘+’ symbol.
Electronics 12 02743 g003
Table
Table 1. EEGNet family.
Table 1. EEGNet family.
Nerual NetworkReference
Shallow ConvNet[1]
Deep ConvNet
EEGNet[25]
S-EEGNet[29]
EEGNet Fusion[30]
TCNet Fusion[31]
Sinc-EEGNet[32]
TSGL-EEGNet[33]
MI-EEGNet[34]
Channel-Mixing-ConvNet[35]
AMSI-EEGNet[36]
ATCNet[37]
FFCL[38]
MTFB-CNN[39]
TCACNet[40]
FB-EEGNet[41]
CRGNet[42]
Table
Table 2. Levels of significance tests.
Table 2. Levels of significance tests.
Levelp-Value Range
110 − 2 < p <= 5 × 10 − 2
210 − 3 < p <= 10 − 2
310 − 4 < p <= 10 − 3
4p <= 10 − 4
Table
Table 3. Ranking the performance of neural networks on all the databases concerning the independent days configuration.
Table 3. Ranking the performance of neural networks on all the databases concerning the independent days configuration.
ClassifierAvg. Acc. Improvement
from Chance Level		Rank
Within subjectShallow ConvNet0.20712
Deep ConvNet0.12495
EEGNet0.19973
EEGNet Fusion0.18714
MI-EEGNet0.23061
Transfer learningShallow ConvNet0.27211
Deep ConvNet0.25982
EEGNet0.25214
EEGNet Fusion0.23125
MI-EEGNet0.25373
Table
Table 4. Classification improvements by transfer learning on databases with independent day configuration.
Table 4. Classification improvements by transfer learning on databases with independent day configuration.
RankNeural NetworksPhysionetGigaTTKBCI Comp
IV 2a		Avg.
Impr.
1Deep ConvNet0.15570.14180.07080.06140.1075
2Shallow ConvNet0.09280.04970.05090.01410.0519
3EEGNet0.07160.04870.0288−0.00650.0357
4EEGNet Fusion0.03810.05860.03790.00070.0338
5MI-EEGNet−0.00580.04750.0564−0.00150.0241
Table
Table 5. Significance investigation.
Table 5. Significance investigation.
Significance Level
DatabaseSumCountSubjects
Physionet6318105
Giga4915108
TTK451625
BCI Comp IV 2a311518
BCI Comp IV 2a-
merged subject data		009
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Köllőd, C.M.; Adolf, A.; Iván, K.; Márton, G.; Ulbert, I. Deep Comparisons of Neural Networks from the EEGNet Family. Electronics 2023, 12, 2743. https://doi.org/10.3390/electronics12122743

AMA Style

Köllőd CM, Adolf A, Iván K, Márton G, Ulbert I. Deep Comparisons of Neural Networks from the EEGNet Family. Electronics. 2023; 12(12):2743. https://doi.org/10.3390/electronics12122743

Chicago/Turabian Style

Köllőd, Csaba Márton, András Adolf, Kristóf Iván, Gergely Márton, and István Ulbert. 2023. "Deep Comparisons of Neural Networks from the EEGNet Family" Electronics 12, no. 12: 2743. https://doi.org/10.3390/electronics12122743

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop