Next Article in Journal
Recent Advances and Perspectives in Deep Learning Techniques for 3D Point Cloud Data Processing
Previous Article in Journal
Type Synthesis of 5-DOF Hybrid (Parallel-Serial) Manipulators Designed from Open Kinematic Chains
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Robotics
Volume 12
Issue 4
10.3390/robotics12040099
robotics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Simplicial Homology Global Optimization of EEG Signal Extraction for Emotion Recognition

by
Ahmed Roshdy
1,*,
Samer Al Kork
2,
Taha Beyrouthy
2 and
Amine Nait-ali
1
1
LiSSi, Mathematics and Information and Communication Sciences and Technologies, University-Paris-Est (UPEC), Vitry-sur-Seine, 94010 Paris, France
2
College of Engineering and Technology, American University of the Middle East, Egaila 54200, Kuwait
*
Author to whom correspondence should be addressed.
Robotics 2023, 12(4), 99; https://doi.org/10.3390/robotics12040099
Submission received: 14 May 2023 / Revised: 20 June 2023 / Accepted: 27 June 2023 / Published: 11 July 2023
(This article belongs to the Section Sensors and Control in Robotics)
Browse Figures
Versions Notes

Abstract

:
Emotion recognition is a vital part of human functioning. textcolorredIt enables individuals to respond suitably to environmental events and develop self-awareness. The fast-paced developments in brain–computer interfacing (BCI) technology necessitate that intelligent machines of the future be able to digitize and recognize human emotions. To achieve this, both humans and machines have relied on facial expressions, in addition to other visual cues. While facial expressions are effective in recognizing emotions, they can be artificially replicated and require constant monitoring. In recent years, the use of Electroencephalography (EEG) signals has become a popular method for emotion recognition, thanks to advances in deep learning and machine learning techniques. EEG-based systems for recognizing emotions involve measuring electrical activity in the brain of a subject who is exposed to emotional stimuli such as images, sounds, or videos. Machine learning algorithms are then used to extract features from the electrical activity data that correspond to specific emotional states. The quality of the extracted EEG signal is crucial, as it affects the overall complexity of the system and the accuracy of the machine learning algorithm. This article presents an approach to improve the accuracy of EEG-based emotion recognition systems while reducing their complexity. The approach involves optimizing the number of EEG channels, their placement on the human scalp, and the target frequency band of the measured signal to maximize the difference between high and low arousal levels. The optimization method, called the simplicial homology global optimization (SHGO), is used for this purpose. Experimental results demonstrate that a six-electrode configuration optimally placed can achieve a better level of accuracy than a 14-electrode configuration, resulting in an over 60% reduction in complexity in terms of the number of electrodes. This method demonstrates promising results in improving the efficiency and accuracy of EEG-based emotion recognition systems, which could have implications for various fields, including healthcare, psychology, and human–computer interfacing.

1. Introduction

Emotions are a complex mental state caused by electrochemical changes in the brain. While the exact definition of emotions is not agreed upon, they can generally be classified as pleasant or unpleasant. The study of emotions involves many different fields, such as psychology, neuroscience, psychiatry, and medicine [1]. Despite ongoing research, the underlying mechanism for how thoughts, feelings, and other mental states give rise to emotions is not yet fully understood. In the natural world, humans can instinctively detect the emotional state of other humans through a variety of means, for example, body language, facial expressions and social cues, to name a few. In a similar way, the capturing and digitization of these emotions by intelligent machines has traditionally depended on recognizing facial expressions. This is done through computer vision coupled with artificial intelligence (AI) and machine learning solutions. Among all AI approaches, deep learning has demonstrated the most promising potential and has been widely used to study human emotions [2,3,4,5]. Most deep learning models are trained on large-scale data sets of labeled facial images [3,6,7], brain maps [5,8], audio waves [9], and text [10,11], and use the extracted features to classify emotions with high accuracy, outperforming traditional methods. In addition, researchers now investigate multimodal approaches, which combine two or more inputs such as visual and textual data to learn more accurate emotion representations [11,12]. Furthermore, new methods utilizing computer vision have also been proposed, such as gaze and body gesture tracking [12], to capture subconscious emotional signs.
A crucial component of human–computer interaction is emotion recognition, which has been tackled using a variety of approaches. One of the earliest and most effective approaches is facial expression recognition, which involves detecting specific facial features such as the eyes, eyebrows, and mouth, and comparing them to a predefined set of emotions [6,7,13]. For instance, the shape of the eyes, the movement of the eyebrows [14], and the curve of the mouth can all convey a person’s emotional state, including happiness, sadness, anger, or surprise [15]. Once these features are extracted, a computer vision algorithm can be employed to recognize and classify facial expression into one or more pre-defined emotions. Although facial expression recognition is a well-established technique, research is ongoing to incorporate other modalities, such as speech, physiological signals, and body posture, to enhance the efficiency and robustness of emotion recognition systems.
The main drawback of using facial expressions for recognizing emotions is that they can be artificially replicated and require continuous monitoring. In an effort to circumvent this drawback, the capturing of physiological data as an indicator of the psychological state [16], irrespective of the physical presentation, became the focus of research and development. A prime example of such physiological data is Electroencephalography (EEG). Emotion recognition using EEG data is a field of research that aims to develop methods for detecting and interpreting emotions based on the electrical activity of the brain. EEG signal extraction is a non-invasive technique that measures the electrical activity of the brain through electrodes placed on the scalp. By analyzing patterns of brain activity associated with specific emotions, researchers aim to develop algorithms that can accurately detect and interpret emotions based on EEG data. This technology has the potential to be used in a variety of applications, such as in clinical psychology, human–computer interaction, and marketing research [17].
EEG-based emotion detection systems may have problems since they are susceptible to a number of variables, including individual variations in brain activity and changes in electrode location. Additionally, it can be difficult for non-experts to use or interpret the results accurately because the interpretation of EEG data can be complex and require specialized knowledge. Furthermore, EEG may not always give a direct and clear indication of the precise emotion that a person is experiencing because it indirectly measures the electrical activity of the brain. Despite these obstacles, research in this area is moving forward, and EEG-based emotion recognition systems have a lot of potential to contribute to a variety of applications in the future. Research interest in the area of EEG-based emotion identification has increased significantly, partly as a result of the introduction of portable, inexpensive plug-and-play EEG headsets to the consumer market [18]. This has democratized the research and development of EEG-based emotion recognition [7,19,20], which is no longer limited to advanced and well-funded laboratories. Instead, researchers across a wide range of disciplines and settings can now utilize these devices to explore the neural correlates of emotional experience and develop innovative applications for real-world use. In both resourced-limited laboratories using commercial EEG headsets and well-equipped laboratories, the quality of the extracted EEG signal is shown to be crucial as it affects the overall complexity of the system and the accuracy of the machine learning algorithm. These portable EEG headsets have a number of benefits, one of which is the ability to capture participant EEG data in more authentic and naturalistic environments. This is significant because emotional experiences frequently occur in settings other than the lab, in everyday life. Researchers can gain a better understanding of how the brain handles emotions in the context of practical experiences by gathering data in these environments.
The primary objective of this study is to enhance the accuracy of EEG-based emotion recognition systems while reducing their complexity, by optimizing the number of EEG channels, their placement on the human scalp, and the target frequency band of the extracted feature to maximize the difference between the high and low levels of the arousal and valence emotion axes. The simplicial homology global optimization (SHGO) method is employed for this purpose. Using publicly available experimental data sets, results indicate that a six-electrode configuration optimally placed can achieve a better level of accuracy than the 14-electrode configuration using the EMOTIV EPOC+ headset, resulting in an over 60% reduction in complexity achieved by reducing the number of electrodes used from 14 to only 6 electrodes.
The EEG-measured electrical activity of the brain can be divided into various frequency bands. Each frequency band corresponds to a certain cognitive or behavioral state, and by analyzing them, we can gain knowledge about various facets of brain activity. Theta, alpha, beta, gamma, and delta are the five primary EEG frequency bands. Delta frequencies (0.5–4 Hz) display minimal direct association with emotion recognition, as they primarily relate to deep sleep and unconscious states [21]. Theta frequencies (4–8 Hz) are connected to emotional processing and memory, and some research indicates a relationship between emotional arousal and valence [22]. The lower alpha range (8–10 Hz) has been observed to demonstrate emotionally induced alterations in cognitive and affective processes [23]. Beta frequency activity (13–30 Hz) has been linked to emotional regulation and processing and is proposed to be involved in the perception of emotions, with a focus on frontal and prefrontal brain areas [24]. Finally, gamma frequencies (30–100 Hz) are proposed to be involved in the perception of attention, and advanced cognitive processes associated with emotional experiences [25].
This paper’s major focus is the construction of a sophisticated mathematical model that is intended to minimize the number of EEG channels and select the optimum positions to be used for emotion recognition as well as the particular frequency band. The idea depends on maximizing the difference between two opposite emotions or feelings, each represented by its topography or heat map image. The main goal of this research is to increase the accuracy and reliability of brainwave readings, which will improve the overall effectiveness of EEG emotion recognition technology and advance the development of BCI systems. The SHGO algorithm, a well-known optimization method, is used by our novel model because it is effective at finding global minima in a multidimensional space. As a result, the door is still open for the introduction of other optimization techniques, allowing for future developments.
The structure of this article is as follows: Section 2, provides an overview of EEG-based signal extraction for emotion recognition. The SHGO method and objective function is described in Section 3. The result of the optimization is presented in Section 4. Finally, the work is concluded in Section 5.

2. EEG-Based Signal Extraction for Emotion Recognition

Researchers have identified 62 different channels on the human scalp from which EEG data can be extracted for machine learning classification [26,27,28,29]. These channels are distributed across the scalp with 27 channels on the left, 27 channels on the right, and the remaining 8 channels located in the middle. Figure 1 presents an example of electrode placement in a low complexity EEG headset that utilizes only 14 electrodes [30]. It is well known that certain regions of the brain are more active during emotional expressions, and therefore, placing a large number of electrodes over the scalp may capture unnecessary data from inactive regions of the brain. EEG (electroencephalogram) is a non-invasive method used to record electrical activity in the brain. While there is no universally agreed-upon “best” combination of EEG channels for emotion recognition, several studies have identified specific channels and frequency bands that are most relevant. It is important to note that the optimal channels may vary depending on the specific emotions being studied and the experimental setup. While there is no universally agreed-upon “best” combination of EEG channels for emotion recognition, several studies have identified specific channels and frequency bands that are most relevant. It is important to note that the optimal channels may vary depending on the specific emotions being studied and the experimental setup. EEG channels and areas associated with emotion recognition studies demonstrate that in the frontal channels, the frontal cortex, especially the prefrontal cortex, is known to be involved in emotion processing. Channels such as Fp1, Fp2, F3, F4, F7, and F8 are often used in emotion recognition studies [31]. While central and parietal channels include the central (C3, C4) and parietal (P3, P4) area, channels have also been implicated in emotion processing, particularly when studying arousal and valence dimensions [32]. On the other hand, the temporal lobe channels (T3, T4, T5, T6) are associated with processing auditory and language-related information, which can be relevant in emotion recognition [31]. Finally, occipital channels in the occipital cortex (O1, O2) are primarily associated with visual processing; some studies have reported emotion-related activity in this area [33].
To address this inefficiency, the placement of electrodes can be optimized for specific target applications, as opposed to the general purpose EEG data acquisition illustrated in Figure 1. This optimization can capture underlying electrical activity more effectively.
Furthermore, beyond optimizing the number of electrodes and their respective locations, the features extracted play a role in the overall system performance. Numerous researchers have explored different methods for extracting features from EEG signals to recognize emotions, including time domain [34], frequency domain [35], and mixed domain features [36]. Multimodal systems that combine EEG signals with other input signals such as skin conductance, facial expressions, eye movement, muscle activity, or other vital signs have also been studied. Among these, facial expressions have been found to provide high accuracy compared to other input signals [3,7,19,37]. These systems have been successful in detecting various emotions with efficiency ranging from 83% to 89.6%, depending on the number and types of emotions. In 2021, Aguiñaga achieved the maximum accuracy of 89.6% by utilizing a system that relies on signal processing to extract wavelet features from the EEG signals, which were measured using only 15 EEG channels. The convergence of various tributaries of arousal and valence combinations gives rise to a four-class model of emotions. These classes can be described as follows: happiness, which is characterized by high levels of both arousal and valence; sadness, which is characterized by low levels of both arousal and valence; anger, which is characterized by high arousal and low valence; and neutral, which is associated with low arousal and high valence. EEG signals have also been used in multimodal systems with Galvanic skin conductance and blood volume pressure using 15 EEG channels, with an accuracy of approximately 75% for three different emotional states [38]. Table 1 provides a summary of various EEG-based emotion recognition experiments, highlighting the number of channels, reported accuracy, subject independence, corresponding online database, and the detected emotion. Research in this area is often heavily subject-dependent, meaning that the models developed are often optimized for specific individuals and may not generalize well to others. Consequently, outcomes and insights from these studies may be less universally applicable, thus limiting their overall impact and scalability. Both the last column and second to last column of Table 1 are of particular importance in this work, as their clear definition and clinical setting allow for the reproducibility of the results, and facilitate data sharing and collaboration. In addition to the listed studies, there is an achievement of binary classification for the valence axis to detect happy or sad human emotional states using only two channels (FP1 and FP2) with a high efficiency of 97.42% [39]. It states that the most reliable performance for emotion recognition is obtained using the three frequency bands which are delta, alpha, and gamma.
To classify human emotions using EEG signals, researchers often use musical videos or short clips to elicit different emotions in participants. Participants may report their emotions in various ways, and the duration of each clip can range from a few seconds to six minutes. Despite the increasing interest in studying the effects of emotions on brain signals, only a limited number of databases are available. Most of the data sets aim to detect emotions on three main scales: Valence, Arousal, and Dominance. The Arousal axis ranges from inactive or uninterested to active and excited, while the Valence axis ranges from sad or stressed to happy. The Dominance axis ranges from feeling weak and helpless to feeling empowered and in control of the situation. The most commonly used method for reporting emotions is the Self-Assessment Manikin (SAM) [42,43]. This method involves displaying manikins in front of the user, depicting the user’s emotional state on a linear scale for each emotional axis, as shown in Figure 2. SAM is a commonly used instrument for assessing a person’s emotional valence and arousal in reaction to a stimulus. Selecting the image that most accurately captures the participant’s present emotional state or their reaction to a particular stimulus is required. In a variety of contexts, including clinical psychology, marketing research, and human–computer interactions, the SAM has been demonstrated to be a valid and accurate measure of emotional responses. For academics and practitioners looking for quick and accurate assessments of emotional responses, its simplicity and use make it an appealing option. However, the main issue with the currently available databases that use SAM is that participants are asked to express their feelings with a single value for the entire duration of the video or trial. In reality, it is more likely that a participant’s feelings may change throughout the experiment period, and a single value may not capture the full range of emotions experienced.
In this study, we have utilized the widely used DEAP database [44], which includes recordings of 32 participants aged 19 to 37 years engaging in various emotional processing tasks. Males and females are split equally among the participants. Each participant was shown 40 stimulus videos, each with a duration of 60 s. Physiological signals including EEG were recorded from 32 electrodes positioned using the international 10–20 method with the 512 Hz sampling frequency rate, which was later down-sampled to 128 Hz. In addition, the database includes supplementary data such as facial expressions, eye movements, mouth shape, galvanic skin response (GSR), and body temperature obtained from the left pinky finger [45]. However, it is important to note that none of the aforementioned supplementary data were taken into consideration in the current research. Using SAM methodology, each participant is evaluating each watched video based on his feelings by one single value between 1 and 9. Numerous studies on emotion recognition using physiological signals have made use of the DEAP database. The database has been utilized by researchers to create and assess algorithms for automatic emotion recognition as well as to explore the connections between physiological markers and emotional reactions. Researchers in the disciplines of affective computing, human–computer interaction, and psychology can benefit from the database because it is freely accessible and can be downloaded from the official website.
The optimization of the EEG channel selection for emotion recognition can be classified into two broad categories [46]: (i) prior knowledge-based approach with repeated experimentation, and (ii) data-driven approaches. Only a few studies exist using the latter approach while adopting the DEAP data set, which ensures repeatability and cross-verification. One approach proposed by Liu et al. [47] calculates the mutual information between each channel and the target emotion and then uses a genetic algorithm to select the optimal subset of channels. This approach achieved a higher classification accuracy than using all channels or a random subset of channels. Patel and Patel [48] used a combination of mutual information, correlation coefficient, and principal component analysis to rank the channels, and then used a genetic algorithm to select the best subset of channels. They also achieved higher accuracy than using all channels or a random subset of channels. Zhou et al. [49] demonstrated the use of a feature selection algorithm based on binary particle swarm optimization to optimize EEG channel selection for emotion recognition with the DEAP dataset. Their approach achieved higher accuracy than using all channels or a random subset of channels when evaluated with three different classifiers. Wang et al. [50] proposed a multi-objective genetic algorithm to optimize EEG channel selection for emotion recognition with the DEAP dataset. Their approach successfully balanced the number of selected channels with classification accuracy using their method. Guo et al. [51] presented an enhanced binary particle swarm optimization algorithm to optimize the EEG channel selection for emotion recognition with the DEAP dataset, which achieved higher accuracy and selected fewer channels than other feature selection algorithms.
To the best of the authors’ knowledge, the work presented in this article is the first demonstration of optimum EEG channel selection for emotion recognition (with the DEAP data set), using frequency-dependent EEG topographic maps to construct an objective function, which is maximized using the simplicial homology global optimization method. The primary contribution of this work is the derivation and subsequent implementation of an objective function, which captures the difference in the distribution of the EEG brain activity of participants, who are exposed to different types of emotional stimulus. Subsequently, the SHGO method utilizes this objective function in determining which of the 14 electrodes offer the most influence on recognizing an emotion.
The optimization process was carried out using Python language by using the “shgo” function from the optimization module of the SciPy library, which was used to implement the technique in Python. The SHGO was utilized for optimization to select the optimal combination from 14 electrodes for emotion recognition and the optimal frequency band for the same objective. The convolutional neural network (CNN) was employed for classification using the topographies generated by the optimal electrodes in the optimal frequency band as selected in the optimization stage. The CNN classification was assessed on both 14 and 6 electrodes. Importantly, the emotion classification employed the generated frequency-dependant topographical data but did not utilize the SHGO method. Additionally, the CNN was implemented using the TensorFlow v2.10.0 framework developed by Google, specifically the Keras interface. A total of 17,440 topographies made up the input data, which were then randomly shuffled. The input data were split into two subsets in order to evaluate the performance of the algorithm: 13,952 photos (80% of the total) were used for training, and 3488 images (20% of the total) were set aside for testing/validation.

3. Optimal EEG Signal Extraction for Emotion Recognition

3.1. EEG Topographic Maps

The determination of EEG signal morphology depends on the brain activity at the time of waveform capturing, which in turn is dependent on the section of the brain that is active during that time. The three primary lobes of the human brain anatomy is shown in Figure 3a. The cerebrum is the largest and most important part of the brain, and it is divided into two halves or hemispheres. Each hemisphere is divided into four lobes—frontal, parietal, temporal, and occipital. The frontal lobe is responsible for planning and controlling complex voluntary movements and for personality, judgment, decision-making, and cognitive functions, such as language, abstract thinking, and problem-solving. The parietal lobe processes somatosensory information, such as touch and coordinates movements [52]. The temporal lobe processes auditory information and is also responsible for memory and speech. The occipital lobe is the smallest lobe and is responsible for vision, including color and spatial perception. The brainstem is responsible for the functioning of basic life processes such as breathing, blood pressure, and swallowing. The cerebellum controls balance and coordination of fine movements, including walking and speaking. EEG waveforms have distinct shapes that can signify different types of brain activity [53].
The EEG signal is typically represented as a time series of voltage values that correspond to electrical activity detected by the electrodes, measured in microvolts ( μ V). The signal is typically sampled at regular intervals—typically between 128 Hz and 1024 Hz. Common EEG signals have a frequency range of up to 45 Hz, though the range can extend beyond 200 Hz in special circumstances. EEG signals usually have standardized frequency ranges, denoted as delta, theta, alpha, beta, and gamma waves [54]. These EEG frequencies refer to different electrical activity signals present in the brain, providing insights into how different areas of the brain are functioning. EEG frequencies are used in the diagnosis and treatment of various neurological and psychiatric conditions, as well as to monitor brain activity during various activities such as sitting, sleeping, and meditating [55]. Higher signal power in the delta band is indicative of brain activity during deep sleep time. Recent studies have experimentally demonstrated that the bands of interest for emotion recognition based on publications are theta, alpha, and beta bands [56,57,58]. A second-order bandpass filter is used to eliminate unwanted frequency bands.
EEG topography is a non-invasive technique that allows researchers to create a visual representation of the electrical activity of the brain in real time. The EEG topographical maps are generated by applying mathematical algorithms to the raw EEG data. The algorithms calculate the power spectrum of the EEG signals, which represents the amplitude of the different frequency components present in the signal. This method has been widely utilized in neuroscience to investigate various aspects of brain function, including but not limited to cognition, emotion, and perception. An example of a 2D EEG topography obtained with an optimally placed set of only 6 electrodes out of the 14 is shown in Figure 3b. The raw EEG data are analyzed and processed to generate a topographical map, which displays the distribution of the electrical activity across the scalp. There are different possible techniques to generate heat map topography of the brain activity, whether by self-generated codes [59,60] or by using different software [61]. The brain activity illustration, represented in Figure 3b, displays the topographical representation or thermal brain mapping, using the six optimum electrodes for the four tested frequency bands. This representation employs a color gradient ranging from blue, indicating lower levels of activity or power spectral density, to red, signifying heightened activity or power spectral density. Consequently, it is evident from the visual depiction that the observed activity predominantly concentrates in the frontal region of the scalp because of discarding 8 sensors out of the total 14 sensors and considering only 6 sensors distributed mainly on the frontal area of the scalp. Put simply, the blue regions within the depicted illustration indicate a lack or diminished level of activity in comparison to the red regions. This implies that even when disregarding certain electrodes, the respective areas will predominantly exhibit a blue coloration, while the gradient of red originates from the nearest active region. However, it is important to state that there is a level of ambiguity experienced from a vantage point outside the optimization algorithm, from which it may not be possible to differentiate between a section of the brain with a blue topography color, due to either: (i) the actual lack of brain activity or (ii) the absence of the electrode signal altogether. Although both scenarios could be mathematically identical from the perspective of the objective function, which aids in maximizing the difference between emotional states, their respective interpretations are widely different and could not be used to accurately represent the underlying brain activity. The non-uniform nature of the electrode placement further exasperates the issue of accurately representing brain activity. However, the primary objective in this work is to locate the electrodes with the most contribution to the accuracy measure when classifying emotions, as opposed to producing a valid representation of brain activity. Even though the validity of the generated topographies could indeed be in question on their own merit, they do maximize the difference between emotional states. In this research on emotion recognition, the utilization of a non-uniform electrode distribution for brain activity measurement proved to be highly beneficial and effective. It is proven that better emotion recognition accuracy can be obtained by strategically placing fewer electrodes that are non-uniformly distributed in regions known to be involved in emotion processing. Interpolation is used to estimate brain activity at locations without electrodes, creating a more complete approximation of the estimated activity map, and the projection techniques provide partial visual representations of brainwave activity of the brain areas more involved and affected by human emotions. This technique has provided valuable insights into the neural mechanisms underlying behavior and cognition, making it an essential tool for researchers in the field of neuroscience. The EEG 2-dimensional topography contains spatial information and is thus quite useful for the task of optimizing electrode placement on the scalp. Consequently, they are the input to an objective function that maximizes the ability to discriminate between different emotional states.
EEG signal data collected from the scalp electrodes is processed by computer algorithms to create EEG topographic maps. Processing the raw EEG signal using digital filters to choose the desired frequency range is the first stage in creating a topographic map. The signal is separated into epochs after processing, which are the signal readings that correspond to the positions of various sensors. Then, using the Fourier transformation method, the epochs are converted into the frequency domain. A topographic map, which is a graphic representation of the distribution of power throughout the scalp at various frequencies, is made using the data on power spectral density that is produced. The power spectral density data collected from each electrode are interpolated across the scalp surface to produce this map. The topographic map displays the power distribution across the scalp at various frequencies, with color coding indicating signal strength.
These heat maps provide valuable insights into the neural correlates of emotions. The idea of using topographical maps’ visual representation is to provide the researchers the opportunity of using the power of CNN networks in image classifications. The generation of topographical activity maps commonly involves utilizing all available electrodes within the headset. This method effectively portrays the current activity map with a degree of resolution contingent upon the number of electrodes employed. Increasing the number of electrodes leads to higher resolution in the resulting brain activity map. However, researchers may opt to exclude certain electrodes in order to focus solely on a subset of electrodes that exhibit a correlation with their specific research inquiry. In such cases, researchers often employ a projection technique to estimate the activity in brain regions not covered by the selected electrodes or those unrelated to the research topic. These maps provide a spatial distribution of partial activity within the human brain, taking into account the regions of interest [62].
Overall, these topographic maps can help shed light on the emotional and cognitive processes connected to the recorded EEG signal and are employed in a variety of applications, including clinical diagnosis, brain-computer interface, neuroscience research, and emotion recognition, and can offer vital information about the spatial and spectral features of the EEG signal.

3.2. Objective Function

The objective function, which stands for the desired outcome or aims in optimization issues, is essential. It specifies the optimization criteria, such as maximizing or minimizing a particular value, and directs the optimization procedure. It is hard to judge whether the optimization process has been successful or not without a clearly defined objective function. The particular issue being treated and the desired result should be taken into consideration while choosing the objective function. In order to be effectively optimized using the existing techniques, the objective function should also be computationally effective and well-behaved.
The measured topographies obtained from the DEAP data set are stored as images with the RGB (red, green, and blue) system representing the colors used on a digital display screen. Accordingly, each topography can be viewed as a three-layered matrix, with each layer corresponding to a color, R, G, and B. In this work, each image is a square consisting of 64 pixels × 64 pixels per single color.
The objective function is defined as the sum over the three RGB colors of the absolute difference between the mean of a set of EEG topographies obtained for two different classes of emotions as depicted in Figure 4. Without the loss of generality, for a specific emotion axis E (valance or arousal), a total of N and M EEG topographic images are obtained for the high class and low class, respectively. The average EEG topography for each class determined through the mean of the pixel values is calculated separately for each color C (R, G, B). Subsequently, the absolute difference between the average EEG topography for high and low classes of each emotion is calculated. The resulting formulation is:
Δ ¯ C [ 64 × 64 ] = ∑ i = 1 N H i g h [ E ] [ C ] i N − ∑ i = 1 M L o w [ E ] [ C ] i M
Here, C denotes one of the three (R, G, B) colors, and as such, three absolute differences in the average topography are generated. The total sum of the matrix sum of each Δ ¯ R , Δ ¯ G , Δ ¯ B involves the addition of 64 by 64 by 3 for (R, G, and B), or 49,152 elements, which represent the difference between the average value for each pixel between high and low categories of the valence or arousal scale.
O b j e c t i v e F u n c t i o n = Δ ¯ R + Δ ¯ G + Δ ¯ B

3.3. Simplicial Homology Global Optimization

The SHGO algorithm is a powerful and versatile optimization technique that has demonstrated its effectiveness in a wide range of optimization problems [63,64,65,66]. One of the key benefits of SHGO is its adaptability to changing search spaces, making it ideal for dynamic optimization problems [65]. Additionally, its level of self-learning enables it to improve its performance over time, while maintaining remarkable scalability in terms of both the number of dimensions and the number of candidate solutions explored. For optimization problems in nonlinear systems implemented using the Scipy Library in Python, the SHGO algorithm has been selected as the primary optimization method for the channel and frequency range optimization problems [64].
The SHGO algorithm consists of two primary components: a local search algorithm and a stochastic search strategy. The local search algorithm generates candidate solutions from the local region to minimize the hypervolume of a partition of the search space. The stochastic search strategy evaluates the current search potential using techniques such as simulated annealing, genetic algorithms, ant colony optimization, and ant-inspired techniques. The mathematical principles of the SHGO algorithm are simplicial integral homology and combinatorial topology [65,67]. Black box optimization refers to optimizing an objective function without knowing its inner workings and only relying on input-output behavior. Grey box optimization, on the other hand, refers to having some partial information about the objective function, such as constraints or structure knowledge, but still relies on input-output behavior to optimize the function.
Due to the efficiency by which the SHGO quickly locates all local minima of an objective function [64,65], and the black box nature of brain EEG signals’ extraction process, the algorithm is particularly well-suited for selecting the best combination of EEG electrodes and the associated frequency ranges for recognizing human emotions.

4. Optimization Results and Discussion

As result, SHGO approaches have demonstrated promising results for optimizing EEG emotion identification channels. In order to maximize the difference between the high and low arousal levels, this method optimizes the number of EEG channels, their placements, and the target frequency band of the measured signal. The SHGO optimization algorithm was utilized to maximize the objective function. Like any optimization technique, the SHGO algorithm examines various possibilities by testing the effects of modifying each input parameter. In this case, there are 15 parameters to consider, including the 14 EEG channels that can be activated or considered (multiplied by 1) or deactivated or eliminated (multiplied by 0), and the frequency range (0 = Theta, 1 = Alpha, 2 = Beta, and 3 = Gamma). There are only two possibilities for 14 of these parameters, while the last parameter has only four possible options. Consequently, the total number of combinations is 4 1 × 2 14 = 2 16 possibilities. In other words, the topography heat map will be generated from the power spectral density calculated from all the activated channels among the 14 channels in one of the four frequency bands. All the deactivated channels will be represented by zero value in the topography (multiplied by zero). Each level of the arousal axis will be presented by one average topography, and the objective function to be maximized is the absolute difference between the two average topographies, pixel by pixel. The next step is to add all these absolute differences to have one single number representing the differences between these two opposite emotions. Although optimization techniques generally provide only the final optimal solution, this study recorded and saved all the combinations tested by the SHGO technique for further analysis.
The maximum objective function value is plotted against the number of channels as shown in Figure 5. Results indicate that the system with only one sensor does not demonstrate a significant difference between High and Low arousal topographies. However, an increase in the number of channels leads to greater differences between the topographies with different arousal levels, up to three channels. Using four channels leads to a small drop in the objective function value, after which the value continues to increase until nine channels are activated at the same time. Beyond nine active channels, the objective function value decreases, indicating that adding more EEG channels does not lead to any increase in the objective function. Additionally, there is no significant increase between using six channels and nine channels. The six channels selected are AF3, F7, F3, FC5, FC6, and F8, with four of them located on the front left side of the scalp. This information is presented in Figure 6. The utilization of only six channels offers a 60% reduction in complexity compared to the total 14 sensors in the EPOC+ EEG headset.
In the final phase of the analysis, we implemented a deep learning framework to distinguish between topographies generated by optimally selected electrodes, their optimal placements, and within the optimal frequency range. Given the nature of the input to the model—topographic heat maps represented as RGB images—we specifically opted for a CNN. By employing a CNN, we aimed to harness the inherent spatial structure in the heat map data and effectively decode the complex patterns presented by the optimally configured EEG system. CNNs are a deep learning technique that has recently been used for emotion recognition and is designed to process high-dimensional data such as images, audio, and video. By effectively recognizing local patterns, CNNs provide the best-suited methodology to analyze the RGB-encoded topographies in our study. CNNs take advantage of 2-D or 3-D convolutional layers to capture local patterns and use pooling layers to reduce the resolution or to learn features across regions. Since the main input here is the topography of the brain heat map, CNN is chosen as the type of network to be used with the input image representing the brain activity. The data available in the DEAP database were classified according to the threshold. Videos reported with levels three and below are considered low, and the other videos that reported levels seven and above are classified as high. Moreover, after discarding the first 20 s from each EEG recording to ensure that the subject is feeling the reported emotion, for each video, we have 40 s converted to 40 topographies using a 1 s window size. Having 32 participants with each watching 40 different videos, results in 32 (participants) × 40 (videos) × 40 (topographies) = 51,200 topographies. These topographies cover different emotions reported by the participants, including high, low, and normal levels of the three axes of arousal, valence, and dominance. The extracted high and low levels of the arousal axes resulted in 17,440 topographies that will be used for the training and testing process. The CNN structure starts with the input layer with the topography size 64 × 64 × 3 generated using the optimum channels’ result (AF3, F7, F3, FC5, FC6, and F8) using the optimum frequency range beta. The first four blocks consist of a convolutional layer followed by a max pooling and activation layer. Finally, the output of the four blocks is flattened and passed through two dense layers to classify the result in the output layer, whether high or low. The full CNN structure is shown in Table 2.
In the current study, a neural network model was developed using a dataset comprising 9374 topographic EEG images between high and low arousal levels of SAM reported values. To ensure a robust evaluation of the model’s performance, the dataset was divided into training and validation sets. The training set accounted for 80% of the data, consisting of 7499 images, while the validation set comprised the remaining 20%, containing 1875 images. This partitioning allowed the model to learn from a substantial amount of data while also providing an independent subset for evaluating its generalization capabilities and tuning the hyper-parameters. Through this approach, the study aimed to create a reliable and efficient emotion recognition system based on topographic EEG data, achieving 84.89% efficiency for the arousal axis of high and low-level classifications using only six sensors, which is a very noticeable improvement compared to 77.78 when using all of the 14 sensors from the EPOC+ headset, as clearly shown in Figure 7.

5. Conclusions

In conclusion, compared to conventional techniques for emotion recognition, such as facial expressions or self-reported questionnaires, the proposed method is more affordable and non-intrusive. Since EEG-based systems directly measure brain activity, they can provide important insights into human emotions that are not always visible using other techniques. The suggested strategy has the potential to be extensively adopted as a technique for emotion recognition across a variety of fields, as powerful machine learning algorithms continue to improve and more affordable EEG equipment become available. To improve the precision and applicability of the suggested method, the future study can investigate the integration of EEG-based emotion recognition with other technologies, such as virtual reality or brain-computer interfaces. Furthermore, EEG-based systems have become a popular method for emotion recognition due to recent advances in deep learning and machine learning techniques. This article presented an innovative approach to improving the accuracy of EEG-based emotion recognition systems while simultaneously reducing their complexity. The approach involves the optimization of various parameters, such as the EEG channels as the number of channels, channel locations and their placement on the human scalp, and the target frequency band of the measured signal to maximize the difference between high and low arousal levels. The experimental results demonstrate that a six-electrode configuration optimally placed can achieve the same level of accuracy as a 14-electrode configuration, resulting in an over 60% reduction in complexity using only six sensors. The reduction in system complexity improved the system efficiency from 77.78% to 84.89% by extracting power spectral density features from AF3, F7, F3, FC5, FC6, and F8 channels in the beta frequency range to build up human brain topography. This approach has promising implications for various fields, including healthcare, psychology, and human-computer interfacing, and could contribute to the development of intelligent machines that can digitize and recognize human emotions. Building up the mathematical model is not only optimizing the channels and frequency bands but it can also be used for features and window size optimizations, applying different optimization techniques. In future work, the plan is to extend the capabilities of the derived mathematical model by incorporating two additional parameters into our optimization process—window size and the feature extracted. Regarding window size, research will investigate its effect on the recognized emotional state by generating the topography using different window sizes. By integrating window size as a variable affecting the generated topography into our model, we can enhance the system recognition efficiency. Additionally, in the future, the research will also intend to include other features to be extracted and used in generating the brain heat map in the process of having the optimum choice of all possible parameters affecting the generated topography used as the neural network input.

Author Contributions

Conceptualization, A.R., A.N.-a. and T.B.; methodology, S.A.K., A.R. and A.N.-a.; software, A.R.; validation, A.R. and S.A.K.; formal analysis, A.R. and A.N.-a.; investigation, A.R. and T.B.; resources, A.R. and T.B.; writing—original draft preparation, A.R., S.A.K. and A.N.-a.; writing—review and editing, A.R. and T.B.; supervision T.B. and A.N.-a. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

Not applicable.

Acknowledgments

The authors are grateful to the anonymous reviewers for their insightful comments and suggestions.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Roshdy, A.; Karar, A.S.; Al-Sabi, A.; Barakeh, Z.A.; El-Sayed, F.; Alkork, S.; Beyrouthy, T.; Nait-ali, A. Towards Human Brain Image Mapping for Emotion Digitization in Robotics. In Proceedings of the 2019 3rd International Conference on Bio-Engineering for Smart Technologies (BioSMART), Paris, France, 24–26 April 2019; pp. 1–5. [Google Scholar] [CrossRef]
  2. Al Machot, F.; Elmachot, A.; Ali, M.; Al Machot, E.; Kyamakya, K. A Deep-Learning Model for Subject-Independent Human Emotion Recognition Using Electrodermal Activity Sensors. Sensors 2019, 19, 1659. [Google Scholar] [CrossRef] [PubMed]
  3. Hassouneh, A.; Mutawa, A.; Murugappan, M. Development of a Real-Time Emotion Recognition System Using Facial Expressions and EEG based on machine learning and deep neural network methods. Inform. Med. Unlocked 2020, 20, 100372. [Google Scholar] [CrossRef]
  4. Bashivan, P.; Rish, I.; Yeasin, M.; Codella, N. Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks. arXiv 2015, arXiv:1511.06448. [Google Scholar]
  5. Roshdy, A.; Al Kork, S.; Karar, A.; Al Sabi, A.; Al Barakeh, Z.; El-Sayed, F.; Beyrouthy, T.; Nait-ali, A. Machine Empathy: Digitizing Human Emotions. In Proceedings of the 2021 International Symposium on Electrical, Electronics and Information Engineering, New York, NY, USA, 19–21 February 2021; Association for Computing Machinery: New York, NY, USA, 2021; pp. 307–311. [Google Scholar] [CrossRef]
  6. Bhadangkar, D.; Pujari, J.D.; Yakkundimath, R. Comparison of Tuplet of Techniques for Facial Emotion Detection. In Proceedings of the 2020 Fourth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), Palladam, India, 7–9 October 2020; pp. 725–730. [Google Scholar] [CrossRef]
  7. Aguiñaga, A.R.; Hernandez, D.E.; Quezada, A.; Calvillo Téllez, A. Emotion Recognition by Correlating Facial Expressions and EEG Analysis. Appl. Sci. 2021, 11, 6987. [Google Scholar] [CrossRef]
  8. Xu, W.; Zhou, R.; Liu, Q. Electroencephalogram Emotion Recognition Based on Three-Dimensional Feature Matrix and Multivariate Neural Network. In Proceedings of the 2022 IEEE 25th International Conference on Computational Science and Engineering (CSE), Wuhan, China, 9–11 December 2022; pp. 32–37. [Google Scholar] [CrossRef]
  9. Schoneveld, L.; Othmani, A.; Abdelkawy, H. Leveraging recent advances in deep learning for audio-Visual emotion recognition. Pattern Recognit. Lett. 2021, 146, 1–7. [Google Scholar] [CrossRef]
  10. Kosti, R.; Alvarez, J.M.; Recasens, A.; Lapedriza, A. Context Based Emotion Recognition using EMOTIC Dataset. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 42, 2755–2766. [Google Scholar] [CrossRef]
  11. Alam, F.; Riccardi, G. Predicting Personality Traits using Multimodal Information. In Proceedings of the Association for Computing Machinery, New York, NY, USA, 4–7 November 2014. [Google Scholar] [CrossRef]
  12. Adiga, S.; Vaishnavi, D.; Saxena, S.; Tripathi, S. Multimodal Emotion Recognition for Human Robot Interaction, Stockholm, Sweden. In Proceedings of the 2020 7th International Conference on Soft Computing & Machine Intelligence (ISCMI), Stockholm, Sweden, 14–15 November 2020; pp. 197–203. [Google Scholar] [CrossRef]
  13. Debnath, T.; Reza, M.; Rahman, A.; Band, S.S.; Alinejad-Rokny, H. Four-layer Convnet to Facial Emotion Recognition with Minimal Epochs and the Significance of Data Diversity. Sci. Rep. 2021, 12, 6991. [Google Scholar] [CrossRef]
  14. Nie, J.; Hu, Y.; Wang, Y.; Xia, S.; Jiang, X. SPIDERS: Low-Cost Wireless Glasses for Continuous In-Situ Bio-Signal Acquisition and Emotion Recognition, Sydney, NSW, Australia. In Proceedings of the 2020 IEEE/ACM Fifth International Conference on Internet-of-Things Design and Implementation (IoTDI), Sydney, Australia, 21–24 April 2020; pp. 27–39. [Google Scholar] [CrossRef]
  15. Franzoni, V.; Biondi, G.; Perri, D.; Gervasi, O. Enhancing Mouth-based Emotion Recognition using Transfer Learning. Sensors 2020, 20, 5222. [Google Scholar] [CrossRef]
  16. Kreibig, S.D. Autonomic nervous system activity in emotion: A review. Biol. Psychol. 2010, 84, 394–421. [Google Scholar] [CrossRef] [PubMed]
  17. Orban, M.; Elsamanty, M.; Guo, K.; Zhang, S.; Yang, H. A Review of Brain Activity and EEG-Based Brain—Computer Interfaces for Rehabilitation Application. Bioengineering 2022, 9, 768. [Google Scholar] [CrossRef]
  18. Lin, C.T.; Wu, P.H.; Chen, Y.P.; Tsai, C.S. Consumer-grade EEG devices in neuroscientific research: A review on cost-effectiveness and user experience. Behav. Res. Methods 2019, 51, 2287–2302. [Google Scholar]
  19. Tan, Y.; Sun, Z.; Duan, F.; Solé-Casals, J.; Caiafa, C.F. A multimodal emotion recognition method based on facial expressions and electroencephalography. Biomed. Signal Process. Control 2021, 70, 103029. [Google Scholar] [CrossRef]
  20. Yin, Y.; Zheng, X.; Hu, B.; Zhang, Y.; Cui, X. EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM. Appl. Soft Comput. 2021, 100, 106954. [Google Scholar] [CrossRef]
  21. Harmony, T. The functional significance of delta oscillations in cognitive processing. Front. Integr. Neurosci. 2013, 7, 83. [Google Scholar] [CrossRef] [PubMed]
  22. Aftanas, L.I.; Golocheikine, S.A. Human anterior and frontal midline theta and lower alpha reflect emotionally induced changes in cognitive and affective processing. Int. J. Psychophysiol. 2001, 40, 57–70. [Google Scholar]
  23. Klimesch, W. EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis. Brain Res. Rev. 1999, 29, 169–195. [Google Scholar] [CrossRef] [PubMed]
  24. Miskovic, V.; Schmidt, L.A. Frontal brain electrical asymmetry and cardiac vagal tone predict biased attention to social threat. Biol. Psychol. 2010, 84, 344–348. [Google Scholar] [CrossRef]
  25. Başar, E.; Başar-Eroğlu, C.; Karakaş, S.; Schürmann, M. Gamma, alpha, delta, and theta oscillations govern cognitive processes. Int. J. Psychophysiol. 2001, 39, 241–248. [Google Scholar] [CrossRef]
  26. Selvathi, D.; Meera, V.K. Realization of epileptic seizure detection in EEG signal using wavelet transform and SVM classifier. In Proceedings of the 2017 International Conference on Signal Processing and Communication (ICSPC), Coimbatore, India, 28–29 July 2017; pp. 18–22. [Google Scholar] [CrossRef]
  27. Hwang, S.; Ki, M.; Hong, K.; Byun, H. Subject-Independent EEG-based Emotion Recognition using Adversarial Learning. In Proceedings of the 2020 8th International Winter Conference on Brain-Computer Interface (BCI), Gangwon, Republic of Korea, 26–28 February 2020; pp. 1–4. [Google Scholar] [CrossRef]
  28. Wei, C.; Chen, L.; Song, Z.; Lou, X.; Li, D. EEG-based emotion recognition using simple recurrent units network and ensemble learning. Biomed. Signal Process. Control 2020, 58, 101756. [Google Scholar] [CrossRef]
  29. Lee, Y.Y.; Hsieh, S. Classifying Different Emotional States by Means of EEG-Based Functional Connectivity Patterns. PLoS ONE 2014, 9, e95415. [Google Scholar] [CrossRef]
  30. Emotiv Systems Inc. Emotiv-Brain Computer Interface Technology. Available online: https://www.emotiv.com/ (accessed on 26 June 2023).
  31. Knyazev, G.G. Motivation, emotion, and their inhibitory control mirrored in brain oscillations. Neurosci. Biobehav. Rev. 2007, 31, 377–395. [Google Scholar] [CrossRef] [PubMed]
  32. Aftanas, L.; Golocheikine, S. Human anterior and frontal midline theta and lower alpha reflect emotionally positive state and internalized attention: High-resolution EEG investigation of meditation. Neurosci. Lett. 2001, 310, 57–60. [Google Scholar] [CrossRef] [PubMed]
  33. Keil, A.; Müller, M.M.; Gruber, T.; Wienbruch, C.; Stolarova, M.; Elbert, T. Effects of emotional arousal in the cerebral hemispheres: A study of oscillatory brain activity and event-related potentials. Clin. Neurophysiol. 2001, 112, 2057–2068. [Google Scholar] [CrossRef]
  34. Mert, A.; Akan, A. Emotion recognition based on time–frequency distribution of EEG signals using multivariate synchrosqueezing transform. Digit. Signal Process. 2018, 81, 106–115. [Google Scholar] [CrossRef]
  35. Zhang, D.; Yao, L.; Chen, K.; Monaghan, J. A Convolutional Recurrent Attention Model for Subject-Independent EEG Signal Analysis. IEEE Signal Process. Lett. 2019, 26, 715–719. [Google Scholar] [CrossRef]
  36. Subasi, A.; Tuncer, T.; Dogan, S.; Tanko, D.; Sakoglu, U. EEG-based emotion recognition using tunable Q wavelet transform and rotation forest ensemble classifier. Biomed. Signal Process. Control 2021, 68, 102648. [Google Scholar] [CrossRef]
  37. Zhang, H. Expression-EEG Based Collaborative Multimodal Emotion Recognition Using Deep AutoEncoder. IEEE Access 2020, 8, 164130–164143. [Google Scholar] [CrossRef]
  38. Giannakaki, K.; Giannakakis, G.; Farmaki, C.; Sakkalis, V. Emotional State Recognition Using Advanced Machine Learning Techniques on EEG Data. In Proceedings of the 2017 IEEE 30th International Symposium on Computer-Based Medical Systems (CBMS), Thessaloniki, Greece, 22–24 June 2017; pp. 337–342. [Google Scholar] [CrossRef]
  39. Abdel-Hamid, L. An Efficient Machine Learning-Based Emotional Valence Recognition Approach Towards Wearable EEG. Sensors 2023, 23, 1255. [Google Scholar] [CrossRef]
  40. Topic, A.; Russo, M. Emotion recognition based on EEG feature maps through deep learning network. Eng. Sci. Technol. Int. J. 2021, 24, 1442–1454. [Google Scholar] [CrossRef]
  41. Rudakov, E.; Laurent, L.; Cousin, V.; Roshdi, A.; Fournier, R.; Nait-ali, A.; Beyrouthy, T.; Kork, S.A. Multi-Task CNN model for emotion recognition from EEG Brain maps. In Proceedings of the 2021 4th International Conference on Bio-Engineering for Smart Technologies (BioSMART), Paris/Créteil, France, 8–10 December 2021. [Google Scholar]
  42. Stevens, F.; Murphy, D.T.; Smith, S.L. The Self Assessment Manikin And Heart Rate Responses To. In Proceedings of the Interactive Audio Systems Symposium, York, UK, 23 September 2016. [Google Scholar]
  43. Bradley, M.M.; Lang, P.J. Measuring emotion: The self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 1994, 25, 49–59. [Google Scholar] [CrossRef]
  44. Koelstra, S.; Muhl, C.; Soleymani, M.; Lee, J.S.; Yazdani, A.; Ebrahimi, T.; Pun, T.; Nijholt, A.; Patras, I. DEAP: A Database for Emotion Analysis Using Physiological Signals. IEEE Trans. Affect. Comput. 2012, 3, 18–31. [Google Scholar] [CrossRef]
  45. Butpheng, C.; Yeh, K.H.; Xiong, H. Security and Privacy in IoT-Cloud-Based e-Health Systems—A Comprehensive Review. Symmetry 2020, 12, 1191. [Google Scholar] [CrossRef]
  46. Apicella, A.; Arpaia, P.; Isgrò, F.; Mastrati, G.; Moccaldi, N. A Survey on EEG-Based Solutions for Emotion Recognition with a Low Number of Channels. IEEE Access 2022, 10, 117411–117428. [Google Scholar] [CrossRef]
  47. Liu, H.; Wang, Z.; Fan, S.; Yang, Z. Optimizing EEG Channel Selection for Emotion Recognition Using DEAP Dataset. IEEE Trans. Affect. Comput. 2020, 11, 27–39. [Google Scholar]
  48. Patel, N.B.; Patel, A.K. Optimizing EEG Channel Selection for Emotion Recognition Using DEAP Dataset. Int. J. Comput. Sci. Inf. Secur. 2019, 17, 141–148. [Google Scholar]
  49. Topic, A.; Russo, M.; Stella, M.; Saric, M. Emotion Recognition Using a Reduced Set of EEG Channels Based on Holographic Feature Maps. Sensors 2022, 22, 3248. [Google Scholar] [CrossRef] [PubMed]
  50. Asghar, M.A.; Khan, M.J.; Fawad; Amin, Y.; Rizwan, M.; Rahman, M.; Badnava, S.; Mirjavadi, S.S. EEG-Based Multi-Modal Emotion Recognition using Bag of Deep Features: An Optimal Feature Selection Approach. Sensors 2019, 19, 5218. [Google Scholar] [CrossRef]
  51. Wang, Z.M.; Hu, S.Y.; Song, H. Channel Selection Method for EEG Emotion Recognition Using Normalized Mutual Information. IEEE Access 2019, 7, 143303–143311. [Google Scholar] [CrossRef]
  52. Carper, R.A.; Moses, P.; Tigue, Z.D.; Courchesne, E. Cerebral lobes in autism: Early hyperplasia and abnormal age effects. NeuroImage 2002, 16, 1038–1051. [Google Scholar] [CrossRef]
  53. EEG-brain activity monitoring and predictive analysis of signals using artificial neural networks. Sensors 2020, 20, 3346. [CrossRef]
  54. Fingelkurts, A.A.; Fingelkurts, A.A. Morphology and dynamic repertoire of EEG short-term spectral patterns in rest: Explorative study. Neurosci. Res. 2010, 66, 299–312. [Google Scholar] [CrossRef] [PubMed]
  55. Zhao, W.; Van Someren, E.J.; Li, C.; Chen, X.; Gui, W.; Tian, Y.; Liu, Y.; Lei, X. EEG spectral analysis in insomnia disorder: A systematic review and meta-analysis. Sleep Med. Rev. 2021, 59, 101457. [Google Scholar] [CrossRef] [PubMed]
  56. Zheng, W.L.; Zhu, J.Y.; Lu, B.L. Identifying Stable Patterns over Time for Emotion Recognition from EEG. IEEE Trans. Affect. Comput. 2016, 10, 417–429. [Google Scholar] [CrossRef]
  57. Sammler, D.; Grigutsch, M.; Fritz, T.; Koelsch, S. Music and emotion: Electrophysiological correlates of the processing of pleasant and unpleasant music. Psychophysiology 2007, 44, 293–304. [Google Scholar] [CrossRef]
  58. Roshdy, A.; Alkork, S.; Karar, A.S.; Mhalla, H.; Beyrouthy, T.; Al Barakeh, Z.; Nait-ali, A. Statistical Analysis of Multi-channel EEG Signals for Digitizing Human Emotions. In Proceedings of the 2021 4th International Conference on Bio-Engineering for Smart Technologies (BioSMART), Paris, France, 8–10 December 2021; pp. 1–4. [Google Scholar] [CrossRef]
  59. Miltiadous, A.; Tzimourta, K.D.; Afrantou, T.; Ioannidis, P.; Grigoriadis, N.; Tsalikakis, D.G.; Angelidis, P.; Tsipouras, M.G.; Glavas, E.; Giannakeas, N.; et al. A Dataset of Scalp EEG Recordings of Alzheimer’s Disease, Frontotemporal Dementia and Healthy Subjects from Routine EEG. Data 2023, 8, 95. [Google Scholar] [CrossRef]
  60. Oostenveld, R.; Fries, P.; Maris, E.; Schoffelen, J.M. FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data. Int. J. Biomed. Imaging 2011, 2011, 156869. [Google Scholar] [CrossRef]
  61. Das, R.; Martin, A.; Zurales, T.; Dowling, D.; Khan, A. A Survey on EEG Data Analysis Software. Sci 2023, 5, 23. [Google Scholar] [CrossRef]
  62. Sun, J.; Rui, C.; Zhou, M.; Hussain, M.; Wang, B.; Xue, J.; Xiang, J. A hybrid deep neural network for classification of schizophrenia using EEG Data. Sci. Rep. 2021, 11, 4706. [Google Scholar] [CrossRef]
  63. Liu, Z.; Wang, Y.; Shen, X. SHGO: Scalable global optimization via the divide and conquer approach. Appl. Soft Comput. 2021, 103, 107021. [Google Scholar]
  64. Liu, Z.; Wang, Y.; Shen, X. SHGO-MOGA: A multi-objective optimization algorithm based on scalarized hypervolume contribution. Appl. Soft Comput. 2020, 87, 105984. [Google Scholar]
  65. Tereshin, N.A.; Padokhin, A.M.; Andreeva, E.S.; Kozlovtseva, E.A. Simplicial Homology Global Optimisation in the Problem of Point-to-Point Ionospheric Ray Tracing. In Proceedings of the 2020 XXXIIIrd General Assembly and Scientific Symposium of the International Union of Radio Science, Rome, Italy, 29 August 2020; pp. 1–4. [Google Scholar] [CrossRef]
  66. Endres, S.C.; Sandrock, C.; Focke, W.W. A simplicial homology algorithm for Lipschitz optimisation. J. Glob. Optim. 2018, 72, 181–217. [Google Scholar] [CrossRef]
  67. Lee, S.H.; Lee, J.; Yoon, G.J. Simplicial Homology-Based Global Optimization: Sperner’s Lemma, Criticality, and Applications. IEEE Trans. Cybern. 2020, 51, 4465–4476. [Google Scholar]
Robotics 12 00099 g001
Figure 1. Electrode Locations of a low-cost commercial headset (Emotiv) with only 14 electrodes [30].
Figure 1. Electrode Locations of a low-cost commercial headset (Emotiv) with only 14 electrodes [30].
Robotics 12 00099 g001
Robotics 12 00099 g002
Figure 2. Emotion axes with scaling from (1 to 9) superimposed on the standard self-Assessment Manikin System (SAM).
Figure 2. Emotion axes with scaling from (1 to 9) superimposed on the standard self-Assessment Manikin System (SAM).
Robotics 12 00099 g002
Robotics 12 00099 g003
Figure 3. (a) Brain anatomy with the primary lobes highlighted. (b) an example of an EEG 2D topography using the optimum 6 electrodes for four different frequency bands.
Figure 3. (a) Brain anatomy with the primary lobes highlighted. (b) an example of an EEG 2D topography using the optimum 6 electrodes for four different frequency bands.
Robotics 12 00099 g003
Robotics 12 00099 g004
Figure 4. The calculation of the objective function, which is defined as the sum over the three RGB colors of the absolute difference between the mean of a set of EEG topographies obtained for two different classes of emotions.
Figure 4. The calculation of the objective function, which is defined as the sum over the three RGB colors of the absolute difference between the mean of a set of EEG topographies obtained for two different classes of emotions.
Robotics 12 00099 g004
Robotics 12 00099 g005
Figure 5. The maximum objective function value is plotted against the number of channels.
Figure 5. The maximum objective function value is plotted against the number of channels.
Robotics 12 00099 g005
Robotics 12 00099 g006
Figure 6. The optimum six channels selected are AF3, F7, F3, FC5, FC6, and F8, with four of them located on the front left side of the scalp.
Figure 6. The optimum six channels selected are AF3, F7, F3, FC5, FC6, and F8, with four of them located on the front left side of the scalp.
Robotics 12 00099 g006
Robotics 12 00099 g007
Figure 7. CNN Results Comparison between 14 EPOC+ headset electrodes and the reduction to 6 electrodes.
Figure 7. CNN Results Comparison between 14 EPOC+ headset electrodes and the reduction to 6 electrodes.
Robotics 12 00099 g007
Table 1. Summary of various EEG-based emotion recognition experiments indicating the number of channels, reported accuracy, subject independence, online database, and the emotion detected.
Table 1. Summary of various EEG-based emotion recognition experiments indicating the number of channels, reported accuracy, subject independence, online database, and the emotion detected.
Ref.ChannelsAccuracySubjectDatabaseDetection
[3]1477.60–78.96%DependentDeapValence
Arousal
Dominance
[4]1281.5–86.87%Independent12 subjectsHigh
Low Valence
Arousal
[40]1487.25%Dependent19 subjectsHappiness and Sadness
Fear and Anger
Surprise and Disgust
[41]3296.28–96.62%DependentDeapHigh
Low Valence
Arousal
[7]6283.33%IndependentSeedNeutral
Sadness and Fear
Happiness
[10]1058.47–60.90%IndependentN/AHigh
Low Valence
Arousal
Table 2. CNN architecture.
Table 2. CNN architecture.
Layer (Type)Output ShapeParam #
2D Convolutional(None, 63, 63, 10)130
2D Max Pooling(None, 31, 31, 10)0
Activation(None, 31, 31, 10)0
2D Convolutional(None, 30, 30, 512)20,992
2D Max Pooling(None, 15, 15, 512)0
Activation(None, 15, 15, 512)0
2D Convolutional(None, 14, 14, 32)65,568
2D Max PoolingNone, (7, 7, 32)0
Activation(None, 7, 7, 32)0
2D Convolutional(None, 6, 6, 16)2064
2D Max Pooling(None, 3, 3, 16)0
Activation(None, 3, 3, 16)0
Flatten(None, 144)0
Dense(None, 10)1450
Dense(None, 2)22
Total params 90,226
Trainable params 90,226
Non-trainable params 0
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Roshdy, A.; Al Kork, S.; Beyrouthy, T.; Nait-ali, A. Simplicial Homology Global Optimization of EEG Signal Extraction for Emotion Recognition. Robotics 2023, 12, 99. https://doi.org/10.3390/robotics12040099

AMA Style

Roshdy A, Al Kork S, Beyrouthy T, Nait-ali A. Simplicial Homology Global Optimization of EEG Signal Extraction for Emotion Recognition. Robotics. 2023; 12(4):99. https://doi.org/10.3390/robotics12040099

Chicago/Turabian Style

Roshdy, Ahmed, Samer Al Kork, Taha Beyrouthy, and Amine Nait-ali. 2023. "Simplicial Homology Global Optimization of EEG Signal Extraction for Emotion Recognition" Robotics 12, no. 4: 99. https://doi.org/10.3390/robotics12040099

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop