Multi-Channel EEG Emotion Recognition Based on Parallel Transformer and 3D-Convolutional Neural Network

Abstract

Due to its covert and real-time properties, electroencephalography (EEG) has long been the medium of choice for emotion identification research. Currently, EEG-based emotion recognition focuses on exploiting temporal, spatial, and spatiotemporal EEG data for emotion recognition. Due to the lack of consideration of both spatial and temporal aspects of EEG data, the accuracy of EEG emotion detection algorithms employing solely spatial or temporal variables is low. In addition, approaches that use spatiotemporal properties of EEG for emotion recognition take temporal and spatial characteristics of EEG into account; however, these methods extract temporal and spatial information directly from EEG data. Since there is no reconstruction of the EEG data format, the temporal and spatial properties of the EEG data cannot be extracted efficiently. To address the aforementioned issues, this research proposes a multi-channel EEG emotion identification model based on the parallel transformer and three-dimensional convolutional neural networks (3D-CNN). First, parallel channel EEG data and position reconstruction EEG sequence data are created separately. The temporal and spatial characteristics of EEG are then retrieved using transformer and 3D-CNN models. Finally, the features of the two parallel modules are combined to form the final features for emotion recognition. On the DEAP, Dreamer, and SEED databases, the technique achieved greater accuracy in emotion recognition than other methods. It demonstrates the efficiency of the strategy described in this paper.

1. Introduction

Emotions are the attitudes and related behavioral responses of individuals towards objective things. It is a complicated psychological and physiological state resulting from the interaction of emotions, thoughts, and actions. Changes in emotions frequently cause alterations in physiological and non-physiological signals, including intonation [1,2], body posture [2,3], facial expressions [4,5], and the EEG [4]. However, it is difficult to guarantee the reliability and validity of non-physiological signs such as facial expressions, gestures, and speech signals in practical applications [5]. Studies [6] in neuroscience and psychology indicate that EEG can intuitively represent emotional changes in individuals. Furthermore, because EEG signals are difficult to conceal and have outstanding real-time performance, the accuracy is superior to that of other methods, making it one of the most widely used for emotion recognition.

Deep learning is often used for EEG emotion identification since it provides automated feature extraction and classification from beginning to end. To collect EEG data with temporal and spatial characteristics, multiple electrode locations are used. EEG emotion recognition based on deep learning can be loosely divided into temporal recognition methods, spatial recognition methods, and spatiotemporal recognition approaches. Emotion recognition algorithms based on temporal characteristics primarily examine the contextual characteristics of EEG. Early studies on temporal features were based on manually extracted features [7,8,9,10,11], and deep networks were only used as classifiers. With the development of deep learning, the convolutional neural network (CNN) has been used for temporal feature extraction and has achieved excellent performance in emotion recognition [12,13,14,15,16,17]. However, CNN often depends on the size of the convolution kernel [18] and has certain defects in extracting global temporal features. The recurrent neural network (RNN) can solve the problem [19], and the long short-term memory model (LSTM) [20] and the gated recurrent network (GRU) [21] are used for the extraction of temporal features and gain satisfactory effects. These approaches are generally effective in extracting temporal features from EEG, but due to the limited amount of data, they are not very reliable at learning the electrode location connection. Methods for emotion recognition based on EEG spatial features account for the spatial interaction between electrodes and rebuild the EEG using electrode spatial information. Early, the spatial features extraction often used the CNN [22,23,24,25,26]. Later, the hybrid networks of CNN and other networks [27,28] perform well in extracting more features from reconstructed images. Such methods take the spatial information of each electrode into account, yet the reconstruction of EEG results in poor contextual feature extraction precision. Therefore, methods based on spatiotemporal characteristics have been proposed to address the abovementioned issues [29,30,31,32]. In these methods, temporal features are extracted first, and the features containing temporal information are inputted into the deep network to extract spatial features. However, such algorithms typically utilize only a single type of EEG data to extract features and cannot obtain spatial and temporal details.

To address the aforementioned issues, the study offers an EEG emotion identification model based on the parallel transformer and 3D-CNN spatiotemporal feature fusion. The parallel channel EEG map and position reconstructed EEG sequence are created first. Then, the model transformer [33] and 3D-CNN are employed to extract the respective temporal and spatial information. Finally, the simultaneously extracted spatial and temporal features are connected to a joint feature vector, and the Softmax layer receives it as input to predict emotion.

2. EEG Data Pre-Processing

This section prepares for emotion recognition using EEG. The utilized datasets are detailed in Section 2.1. In Section 2.2, a baseline preprocessing procedure is used to eliminate the impact of individuals’ pre-experiment emotions on the results. Section 2.3 creates the parallel channel EEG maps and position reconstructed EEG sequences for the temporal and spatial feature extraction portions.

2.1. EEG Dataset

The DEAP dataset was constructed by Sander Koelstra [34] as a multimodal dataset. The dataset gathered signals from the patients’ central and peripheral nerve systems by exposing them to numerous music videos. Utilizing a self-assessment form, the subject’s emotional information was gathered. The sample comprised of 32 healthy individuals between the ages of 19 and 37, including 16 males and 16 females. Each subject viewed forty one-minute music videos in accordance with the experimental parameters. According to their feelings, volunteers judged arousal, valence, liking, and dominance after each movie. The score range is 1 to 9. The experimental protocol is depicted in Figure 1. Valence and arousal are the two primary measures utilized to assess human emotions. As valence indicates that people’s emotional state goes from negative to positive [35], arousal indicates that people’s emotional intensity ranges from weak to robust, with a score over 5 indicating a high emotional state and a score below 5 suggesting a low emotional state.

The DEAP dataset contains 40 segments of 32-channel EEG signals from each of 32 participants, with each segment lasting 63 s. The signal has been reduced in frequency to 128 Hz. The initial three seconds of each channel’s EEG signal represent the preparatory period, whereas the latter sixty seconds represent the emotion-induced process. We treat the 40-segment EEG signals of a single individual as an independent dataset with the following format: where 40 represents the number of experiments, 32 represents the number of channels, and 8064 represents the number of sampling points.

In addition to the DEAP dataset, the Dreamer and SEED datasets are utilized to validate the model’s performance. The Dreamer dataset [36] collected the EEG and ECG signals of 23 subjects during emotion induction; arousal and valence continue to be used as labels for EEG data. If the value is greater than 3, it denotes a high emotional state; otherwise, it suggests a low emotional state. The SEED dataset [37] contains fifteen subjects. Positive, negative, and neutral labels correspond to the target emotion of emotion-evoked segments. The Dreamer and SEED datasets are trained and validated with the same procedure as DEAP.

2.2. Baseline Preprocessing

This article employs the baseline mean approach [38] to determine the respondents’ pre-test emotional state. First, remove the pre-test signals from the first three seconds of all channels, divide them into three 1-s segments, and take the mean of the three segments as the baseline mean, which represents the subjects’ basic emotions at that moment. Second, the last 60 milliseconds of EEG signal are separated into 60 pieces of equal size, and the baseline effect is subtracted using a Formula (1). Finally, as illustrated in the equation, all baselines shall be spliced (2).

$$Remove{d}_i=Rawdat{a}_i-Basemean$$

(1)

$$data=Concat(Remove{d}_1;Remove{d}_2;\dots ;Remove{d}_H)$$

(2)

where $Basemean\in R^{C\times L}$, $Rawdat{a}_i\in R^{C\times L}$, $Remove{d}_i\in R^{C\times L}$, $data\in R^{C\times (L\times H)}$; $C=32$ denotes the number of channels; $L=1 \mathrm{s}$, which represents the length of a segment; $H=60$ represents the number of segments; $Concat$ denotes the connected operations.

Figure 2a depicts the EEG signal of a random channel from the DEAP dataset. Figure 2b depicts the outcome of baseline preprocessing. With baseline interference removed, the EEG signal exhibits a smoother waveform and more pronounced wave features. Following baseline processing, the data format for each participant is where 40 represents the number of experiments, 32 represents the number of channels, and 7680 is the number of sample points in 60 s.

Five people are selected from the DEAP dataset to investigate the impact of baseline preprocessing on emotion recognition performance. Both baseline-preprocessed and unpreprocessed EEG data are used for emotion identification, and the results on arousal and valence are depicted in Figure 3.

Compared to the original EEG, the accuracy of arousal and valence has increased by 24.24 percent and 40.81 percent, respectively, when preprocessed by baseline. Therefore, the baseline is used to preprocess EEG data in this study.

2.3. Construction of Two EEG Representations

To simultaneously gather temporal and spatial features, we need EEG data in two formats that can account for both temporal and spatial aspects. Figure 4 illustrates the construction of parallel channel EEG maps and position-reconstructed EEG sequences.

Initially, the baseline-processed data is divided into EEG segments via a sliding window with all channels placed in a single image, which is referred to as parallel channel EEG. The size 9 × 9 matrix then stores the parallel channel EEG map based on the electrode distribution map [39]. The placement of the signal of each channel in the corresponding location of the matrix is referred to as the EEG reconstructed sequence. Where 0 indicates the absence of an electrode channel. The data format is then changed from the original matrix to a matrix sequence, which not only retains position information but also maintains temporal characteristics.

To segment EEG data, sliding windows with dimensions of 128, 256, 384, 512 and steps with sizes of 1/8, 2/8, 3/8, 4/8, 5/8, 6/8, 7/8, and 1 of the window widths were selected. Figure 5 depicts the obtained accuracy of arousal and valence. The maximum rate of accuracy is achieved when the window size is 128 and the step size is 1.

Each individual can obtain 60 data samples in two formats per experiment, for a total of 2400 EEG samples throughout 40 experiments. A parallel channel EEG map size of $32\times 128$ is utilized to derive temporal characteristics. The sequence size of a position-reconstructed EEG is $9\times 9\times 128$, which is used to extract spatial data.

3. Proposed Method

This section proposes TSFFN (Temporal–Spatial Feature Fusion Network), a parallel spatiotemporal feature fusion model based on transformer and 3D-CNN. It includes temporal feature extraction, spatial feature extraction, and temporal-spatial feature fusion modules. In Section 3.1, the attention model transformer in machine translation is utilized to extract the temporal characteristics in order to retrieve the contextual information of the parallel channel EEG maps. In Section 3.2, we employ 3D-CNN to determine the spatial properties of the position-reconstructed EEG episodes. Temporal and spatial features are used as the final emotion detection feature for classification in Section 3.3. The particular procedure is depicted in Figure 6.

3.1. TSFFN’s Temporal Characteristics Extraction Module

This module primarily extracts EEG temporal information, hence EEG parallel channel maps that have been preprocessed serve as input. Considering that the creation of emotions in the brain is a continuous and full process in time, getting the contextual characteristics of EEG is crucial for emotion detection. We used the Transformer to construct a model to extract the temporal characteristics of EEG because the attention mechanism can efficiently perceive the contextual relationships of the signal. The particular procedure is depicted in Figure 7.

Firstly, the position encoding module employs convolution with two channels to obtain the segment position characteristics [40]. The kernel size of convolution influences the accuracy of arousal and valence classification.

Table 1. The accuracy of arousal and valence classification based on kernel size.

Kernel Size	Arousal	Valence
(1,1)	96.34%	96.40%
(1,11)	97.16%	97.54%
(1,21)	96.58%	96.50%
(1,31)	97.33%	96.75%
(1,41)	97.29%	97.12%
(1,51)	98.53%	98.27%
(1,61)	97.45%	96.63%
(1,71)	96.83%	96.87%
(1,81)	96.15%	96.08%
(1,91)	97.50%	97.24%

displays the findings of the investigation into the effect of position kernel size on the accuracy of arousal and valence classification. The classification accuracy of arousal and valence is greatest when the convolution kernel size is set at $(1,51)$. Therefore, the $(1,51)$ is selected as the position encoding kernel parameter.

Secondly, the resulting feature maps are then fed into the dimension transformation module, which has a convolution kernel size of (32,16) and a channel count of 10. It transforms the feature maps into seven 1 × 10 vectors.

Thirdly, the one-dimensional vector sequences are input into the Transformer Encoder, which consists of 2 encoder blocks. After layer normalization, the input is fed into the multi-head attention (MHA) module in the encoder block. The input is turned into five vectors of size 7 × 2, which are then fed into five heads where the attention mechanism works in parallel. The outputs of all heads are then concatenated and converted to the original size 7 × 10 following Formulas (3) and (4). At the same time, there is a residual module here.

$$MultiHead(Q,K,V)=Concat(hea{d}_1,\dots ,hea{d}_h)W^o$$

(3)

$$Where hea{d}_i=Attention(Q{W}_i^Q,K{W}_i^K,V{W}_i^V)$$

(4)

where $W_i^Q\in R^{d_{model}\times d_k}$ $W_i^k\in R^{d_{model}\times d_k}$, $W_i^v\in R^{d_{model}\times d_v}$, $W_{}^o\in R^{h{d}_v\times d_{model}}$ denote the linear transformation used to obtain the query, key and value for each head; $hea{d}_i$ represents the outputs of the i-th head after attention mechanism; $Concat$ represents the operation used to concatenate all $hea{d}_i$; $W^0$ indicates a linear change to obtain the final result.

Then the normalization layer’s utput is as FF (feed-forward) module’s input. The first fully connected layer increases the number of features by four times 7 × 40. Another full connection layer is connected after the activation function GeLu and the normalization layer to reconstruct the characteristic number to the initial size 7 × 10.

Finally, a global averaging pooling layer is connected to average the one-dimensional vectors of all channel outputs as the extracted temporal features.

3.2. Spatial Characteristics EXTRACTION Module of TSFFN

The spatial feature extraction module primarily extracts electrode position features, hence the preprocessed position-reconstructed EEG sequence containing electrode position information serves as its input. Since the data comprises both position and temporal information for the electrodes, the 3D convolution [41] is utilized for feature extraction rather than the conventional 2D convolution. The particular procedure is depicted in Figure 8.

The model employs three successive 3D convolutional layers with kernel sizes of 1, 2, and 3, respectively. Convolution kernels in each layer are successively 32, 64, and 128. Because the pooling layer will result in data loss, it is not employed after the convolutional layer to store the electrode position data. Instead of the pooling layer, batch normalization (BN) is introduced to expedite model training. It is then followed by the activation function ReLu. Finally, we apply a complete connection layer to the extracted spatial characteristics to output them.

Table 2. The structural parameters of 3D-CNN.

Type of Layer		Filter			In Layer	Out Layer	Feature Size
		Width	Height	Length
conv	Conv1	4	4	2	1	32	32 × 6 × 6 × 127
	Conv2	4	4	2	32	64	64 × 3 × 3 × 126
	Conv3	3	3	2	64	128	128 × 1 × 1 × 125
fc	Fc-512	1	1	125	128	512	512

displays the model’s specific parameters.

3.3. Fusion of Temporal and Spatial Modules

Feature fusion and decision fusion are the two most prevalent fusion techniques. The research constructs a TSFFN based on spatiotemporal feature fusion and a TSDFN (Temporal–Spatial Decision Fusion Network) based on spatiotemporal decision fusion to evaluate the performance of the two fusion approaches.

TSFFN concatenates the temporal and spatial features extracted by the transformer and 3D-CNN to obtain a 1 × 522 vector, which is used as the final feature vector, and then uses a fully connected layer and SoftMax layer for emotion classification. The TSDFN model uses the weighted sum approach [14] to fuse the decision results of the transformer and 3D-CNN, as stated in Formula (5):

$$v_{final}={\displaystyle {\sum }_{i=1}^n{w}_i{v}_i}$$

(5)

Among them, n denotes the number of modules; $v_i=\{+1,-1\}$ is the prediction result of the ith model, where +1 represents a high emotional state, and −1 represents a low emotional state; $w_i$ is the probability of the ith model for emotional prediction; $v_{final}$ is the final decision of the model. If it is greater than or equal to 0, it is judged as a high emotional state, otherwise it is a low emotional state.

In this research, the experimental performance of TSFFN and TSDFN under the two fusion procedures is compared, and the findings are presented in

Table 3. The results of different fusion strategies.

Method	Accuracy (Arousal)	Accuracy (Valence)
TSDFN	96.14%	95.76%
TSFFN	98.53%	98.27%

. Table 3 demonstrates that the classification accuracy of the TSFFN based on feature fusion is higher for both arousal and valence than that of the TSDFN based on decision fusion. Consequently, feature fusion is picked as the model’s fusion strategy.

4. Experiments and Analysis of Results

This section verifies the performance of the model. Section 4.1 introduces the hardware and software environment and parameters for model implementation; Section 4.2 describes the model evaluation metrics; Section 4.3 shows the evaluation metrics scores, ablation experiments, and cross-dataset experiments to evaluate the performance of the model; Section 4.4 presents the comparison experiments with other existing EEG emotion recognition algorithms.

4.1. Environment of Software and Hardware for Model Implementation

The proposed model is implemented using Python 3.7 under the PyTorch 1.10.0 framework and trained on the GPU provided by NVIDIA Tesla k80. Python 3.7.0 was released by the python Software Foundation in Delaware, USA, and the python 1.10.0 framework was developed by Facebook in California, USA

4.2. Evaluation Metrics

The Accuracy, Precision, Recall, and F1-score metrics are used to evaluate the performance of the TSFFN. Among them, Accuracy indicates the correct proportion of the model’s prediction of all samples; Precision suggests the ratio of the actual positive samples to all the predicted positive samples; Recall indicates the proportion that the correctly predicted positive samples account for all the positive samples; F1-score is a comprehensive indicator of precision and recall, and the calculation formula is as follows:

$$Accurcacy=\frac{TP+TN}{TP+TN+FP+FN}\times 100\%$$

(6)

$$Precision=\frac{TP}{TP+FP}\times 100\%$$

(7)

$$Recall=\frac{TP}{TP+FN}\times 100\%$$

(8)

$$F1\text{-}score=2\times \frac{Precision\times Recall}{Precision+Recall}\times 100\%$$

(9)

where TP represents the prediction is positive and the actual is positive; FP represents that the prediction is positive and the actual is negative; FN represents that the prediction is negative and the actual is positive; TN represents that the prediction is negative and the actual is also negative.

4.3. TSFFN Model Performance Verification

The experiment is carried out on the DEAP dataset. After preprocessing the EEG data of 32 subjects, each subject contains 2400 samples in the form of a parallel channel map and a position reconstructed EEG sequence. The samples are divided into a training set and a test set by 8:2, in which the training set contains 1980 samples, and the test set contains 480 samples. The batch size is 56, epoch is 50, and learning rate is 1 × 10⁻³. It minimizes the cross-entropy loss function using the Adam optimizer for training. Moreover, in the case of subject independence, the average training time of the model TSFFN proposed in this paper lasts 714.4523 s.

The TSFFN model is utilized to conduct studies on 32 subjects with the arousal and valence labels, respectively. The results are presented in

Table 4. Recognition Accuracy for each subject on “Arousal”.

Subject	Accuracy	Subject	Accuracy	Subject	Accuracy	Subject	Accuracy
S01	99.74%	S09	98.77%	S17	97.27%	S25	99.31%
S02	96.74%	S10	99.89%	S18	98.31%	S26	96.2%
S03	99.91%	S11	96.95%	S19	98.37%	S27	98.47%
S04	96.31%	S12	98.66%	S20	99.72%	S28	96.43%
S05	98.22%	S13	99.06%	S21	99.39%	S29	99.33%
S06	98.29%	S14	97.39%	S22	98.29%	S30	98.95%
S07	98.68	S15	99.33%	S23	99.27%	S31	99.12%
S08	99.12	S16	98.85%	S24	99.47%	S32	98.85%

and

Table 5. Recognition Accuracy for each subject on “Valence”.

Subject	Accuracy	Subject	Accuracy	Subject	Accuracy	Subject	Accuracy
S01	99.95%	S09	97.45%	S17	97.62%	S25	97.79%
S02	95.25%	S10	98.87%	S18	98.93%	S26	98.37%
S03	98.97%	S11	95.29%	S19	97.58%	S27	98.97%
S04	97.12%	S12	98.22%	S20	99.25%	S28	97.68%
S05	98.43%	S13	97.47%	S21	98.64%	S29	99.12%
S06	98.89%	S14	97.41%	S22	97.87%	S30	99.14%
S07	99.00%	S15	98.85%	S23	99.62%	S31	98.83%
S08	98.04%	S16	99.29%	S24	98.43%	S32	98.35%

respectively.

The arousal and valence labeling accuracy of the TSFFN model is greater than 95 percent for all participants, as seen in the tables above. On arousal, the mean ± standard deviation of the accuracy is 98.53% ± 1.05% percent, and on valence, it is 98.27% ± 1.04%.

The assessment metrics are computed based on the outcomes of all predictions in the test set, as shown in

Table 6. The evaluation metrics scores of TSFFN.

Label	Accuracy	Precision	Recall	F1-Score
Arousal	98.53%	98.92%	98.91%	98.91%
Valence	98.27%	98.72%	98.90%	98.80%

. It can be seen that the TSFFN model’s arousal scores are higher than its valence ratings. It scored 98.91% on both the Recall and F1-Score tests and 98.92% on the accuracy test. Recall and F1-Score for label valence obtained 98.90% and 98.80% respectively. Precision increased by 98.72% as well.

Figure 9 depicts the confusion matrices for the arousal and valence labels. According to the data, the TSFFN has a greater recognition rate for intense emotional states. In the task of label arousal identification, the predictive accuracy of the model for high and low emotions is 99 and 98 percent, respectively. The model also achieves 99 percent accuracy in valence recognition for high emotional states and 96 percent accuracy for low emotional states.

The ablation experiments are carried out to verify the validity of each module of the TSFFN. Ablation variables include the temporal feature extraction module (temporal) and the spatial feature extraction module (spatial). Calculate the accuracy of each model’s arousal and valence label prediction according to Figure 10.

Figure 10 demonstrates that for the two labels of arousal and valence, the accuracy of TSFFN is superior to models based on temporal or spatial variables alone. Therefore, combining spatial and temporal variables can enhance the overall performance of a model.

4.4. TSFFN Performance Testing on Different Datasets

In addition to the DEAP data set, the study also verifies the performance of TSFFN in datasets Dreamer and SEED. On the basis of the results of all subjects, the Accuracy, Precision, Recall, and F1-score indicators are calculated as shown in

Table 7. The results of cross-dataset experiments.

Index	DEAP		Dreamer		SEED
	Arousal	Valence	Arousal	Valence
Accuracy	98.53%	98.27%	97.74%	96.80%	97.64%
Precision	98.92%	98.72%	98.66%	98.07%	98.86%
Recall	98.91%	98.90%	98.45%	97.48%	98.86%
F1-score	98.91%	98.80%	98.55%	97.77%	98.86%

, which reveals that on the Dreamer dataset, the accuracy of the model in the arousal label reaches 97.74% and the accuracy in the valence label reaches 96.80%. Additionally, TSFFN scores well in both Precision and Recall. The model also performs exceptionally well on the SEED dataset. The accuracy of the classification of emotions into three categories has reached 97.64%. Therefore, the TSFFFN has good performance not only in DEAP dataset, but also in Dreamer and SEED datasets.

4.5. Comparison Experiment with Existing Algorithms

In terms of arousal and valence accuracy, TSFFN compares favorably to PSD [42], DE_CNN [43], IJCNN [38], ACRNN [44], and FSA-3D-CNN [45]. PSD utilized time convolution to obtain spatial features from reconstructed EEG frames and then combines the extracted features with temporal properties for emotion recognition. DE_ CNN described a preprocessing method based on differential entropy (DE) characteristics for transforming raw EEGs to 2D frames, followed by classification of emotions using 3D-CNN. IJCNN captured spatiotemporal characteristics from raw EEG for emotion classification using a hybrid CNN and RNN network. ACRNN provided a hybrid convolutional recurrent network based on attention in which CNN is used to extract spatial features from EEG weighted by the attention mechanism and then an upgraded RNN combines temporal features for emotion recognition; FSA-3D-CNN proposed a four-dimensional representation of the EEG that incorporates spatiotemporal information and then employs a 3D-CNN to classify emotions.

Table 8. Performance comparison of different approaches.

Method	Accuracy (Arousal)	Accuracy (Valence)
PSD	85.49%	84.02%
DE_CNN	90.23%	89.78%
IJCNN	91.02%	90.79%
ACRNN	93.81%	92.56%
FSA-3D-CNN	95.87%	95.23%
TSFFN	98.53%	98.27%

provides the results. It can be seen from Table 8 that our proposed method has significantly better performance than methods that only use spatial information to reconstruct EEG (DE_CNN, FSA-3D-CNN) or only use time series (ACRNN) as input. In addition, with the method of fusion with spatial and temporal features (PSD, IJCNN), the accuracy of the proposed TSFFN is still far ahead.

5. Limitations

The research has serval limitations, although it achieved greater accuracy in emotion recognition. First, although parallel input can extract more detailed spatial and temporal features, it also requires more runtime. Second, the performance of this study is excellent on independent data sets, but for multi-person diverse data sets, the accuracy of the model will decrease due to individual differences.

In the future, we will try to simplify the network structure and reduce the running time of the model. The model’s generalization ability is improved by using multimodal data to minimize the impact of individual differences on the performance of the emotion recognition model.

6. Conclusions

This study introduces TSFFN, a parallel transformer and 3D-CNN-based spatiotemporal feature fusion model. Two forms of EEG are utilized to fully and successfully extract the temporal and spatial characteristics of EEG data. The transformer and 3D-CNN are selected to extract temporal and spatial features from the parallel channel EEG map and reconstructed position EEG sequence, which are then merged to generate the final emotion feature. Not only does the approach simultaneously consider the temporal and spatial aspects of EEG, but it also creates two data forms, which is useful for transformer and 3D-CNN to extract all information. The experimental findings on the DEAP, Dreamer, and SEED datasets demonstrate the adequacy of the suggested model.