Research

14 pages, 1015 KiB

Open AccessCommunication

A New Network Structure for Speech Emotion Recognition Research

by Chunsheng Xu, Yunqing Liu, Wenjun Song, Zonglin Liang and Xing Chen

Sensors 2024, 24(5), 1429; https://doi.org/10.3390/s24051429 - 22 Feb 2024

Viewed by 604

Deep learning promotes the breakthrough of emotion recognition in many fields, especially speech emotion recognition (SER). As an important part of speech emotion recognition, the most relevant acoustic feature extraction has always attracted the attention of existing researchers. Aiming at the problem that [...] Read more.

Deep learning promotes the breakthrough of emotion recognition in many fields, especially speech emotion recognition (SER). As an important part of speech emotion recognition, the most relevant acoustic feature extraction has always attracted the attention of existing researchers. Aiming at the problem that the emotional information contained in the current speech signals is distributed dispersedly and cannot comprehensively integrate local and global information, this paper presents a network model based on a gated recurrent unit (GRU) and multi-head attention. We evaluate our proposed emotion model on the IEMOCAP and Emo-DB corpora. The experimental results show that the network model based on Bi-GRU and multi-head attention is significantly better than the traditional network model at detecting multiple evaluation indicators. At the same time, we also apply the model to a speech sentiment analysis task. On the CH-SIMS and MOSI datasets, the model shows excellent generalization performance. Full article

(This article belongs to the Special Issue Advanced-Sensors-Based Emotion Sensing and Recognition)

► Show Figures

Figure 1

19 pages, 1688 KiB

Open AccessArticle

Electromyographic Validation of Spontaneous Facial Mimicry Detection Using Automated Facial Action Coding

by Chun-Ting Hsu and Wataru Sato

Sensors 2023, 23(22), 9076; https://doi.org/10.3390/s23229076 - 09 Nov 2023

Cited by 1 | Viewed by 965

Abstract

Although electromyography (EMG) remains the standard, researchers have begun using automated facial action coding system (FACS) software to evaluate spontaneous facial mimicry despite the lack of evidence of its validity. Using the facial EMG of the zygomaticus major (ZM) as a standard, we [...] Read more.

Although electromyography (EMG) remains the standard, researchers have begun using automated facial action coding system (FACS) software to evaluate spontaneous facial mimicry despite the lack of evidence of its validity. Using the facial EMG of the zygomaticus major (ZM) as a standard, we confirmed the detection of spontaneous facial mimicry in action unit 12 (AU12, lip corner puller) via an automated FACS. Participants were alternately presented with real-time model performance and prerecorded videos of dynamic facial expressions, while simultaneous ZM signal and frontal facial videos were acquired. Facial videos were estimated for AU12 using FaceReader, Py-Feat, and OpenFace. The automated FACS is less sensitive and less accurate than facial EMG, but AU12 mimicking responses were significantly correlated with ZM responses. All three software programs detected enhanced facial mimicry by live performances. The AU12 time series showed a roughly 100 to 300 ms latency relative to the ZM. Our results suggested that while the automated FACS could not replace facial EMG in mimicry detection, it could serve a purpose for large effect sizes. Researchers should be cautious with the automated FACS outputs, especially when studying clinical populations. In addition, developers should consider the EMG validation of AU estimation as a benchmark. Full article

(This article belongs to the Special Issue Advanced-Sensors-Based Emotion Sensing and Recognition)

► Show Figures

Figure 1

21 pages, 8278 KiB

Open AccessArticle

ECG Multi-Emotion Recognition Based on Heart Rate Variability Signal Features Mining

by Ling Wang, Jiayu Hao and Tie Hua Zhou

Sensors 2023, 23(20), 8636; https://doi.org/10.3390/s23208636 - 22 Oct 2023

Viewed by 2072

Abstract

Heart rate variability (HRV) serves as a significant physiological measure that mirrors the regulatory capacity of the cardiac autonomic nervous system. It not only indicates the extent of the autonomic nervous system’s influence on heart function but also unveils the connection between emotions [...] Read more.

Heart rate variability (HRV) serves as a significant physiological measure that mirrors the regulatory capacity of the cardiac autonomic nervous system. It not only indicates the extent of the autonomic nervous system’s influence on heart function but also unveils the connection between emotions and psychological disorders. Currently, in the field of emotion recognition using HRV, most methods focus on feature extraction through the comprehensive analysis of signal characteristics; however, these methods lack in-depth analysis of the local features in the HRV signal and cannot fully utilize the information of the HRV signal. Therefore, we propose the HRV Emotion Recognition (HER) method, utilizing the amplitude level quantization (ALQ) technique for feature extraction. First, we employ the emotion quantification analysis (EQA) technique to impartially assess the semantic resemblance of emotions within the domain of emotional arousal. Then, we use the ALQ method to extract rich local information features by analyzing the local information in each frequency range of the HRV signal. Finally, the extracted features are classified using a logistic regression (LR) classification algorithm, which can achieve efficient and accurate emotion recognition. According to the experiment findings, the approach surpasses existing techniques in emotion recognition accuracy, achieving an average accuracy rate of 84.3%. Therefore, the HER method proposed in this paper can effectively utilize the local features in HRV signals to achieve efficient and accurate emotion recognition. This will provide strong support for emotion research in psychology, medicine, and other fields. Full article

(This article belongs to the Special Issue Advanced-Sensors-Based Emotion Sensing and Recognition)

► Show Figures

Figure 1

18 pages, 3182 KiB

Open AccessArticle

EEG and ECG-Based Multi-Sensor Fusion Computing for Real-Time Fatigue Driving Recognition Based on Feedback Mechanism

by Ling Wang, Fangjie Song, Tie Hua Zhou, Jiayu Hao and Keun Ho Ryu

Sensors 2023, 23(20), 8386; https://doi.org/10.3390/s23208386 - 11 Oct 2023

Cited by 1 | Viewed by 1336

Abstract

A variety of technologies that could enhance driving safety are being actively explored, with the aim of reducing traffic accidents by accurately recognizing the driver’s state. In this field, three mainstream detection methods have been widely applied, namely visual monitoring, physiological indicator monitoring [...] Read more.

A variety of technologies that could enhance driving safety are being actively explored, with the aim of reducing traffic accidents by accurately recognizing the driver’s state. In this field, three mainstream detection methods have been widely applied, namely visual monitoring, physiological indicator monitoring and vehicle behavior analysis. In order to achieve more accurate driver state recognition, we adopted a multi-sensor fusion approach. We monitored driver physiological signals, electroencephalogram (EEG) signals and electrocardiogram (ECG) signals to determine fatigue state, while an in-vehicle camera observed driver behavior and provided more information for driver state assessment. In addition, an outside camera was used to monitor vehicle position to determine whether there were any driving deviations due to distraction or fatigue. After a series of experimental validations, our research results showed that our multi-sensor approach exhibited good performance for driver state recognition. This study could provide a solid foundation and development direction for future in-depth driver state recognition research, which is expected to further improve road safety. Full article

(This article belongs to the Special Issue Advanced-Sensors-Based Emotion Sensing and Recognition)

► Show Figures

Figure 1

19 pages, 8077 KiB

Open AccessArticle

Harnessing Wearable Devices for Emotional Intelligence: Therapeutic Applications in Digital Health

by Herag Arabian, Tamer Abdulbaki Alshirbaji, Ramona Schmid, Verena Wagner-Hartl, J. Geoffrey Chase and Knut Moeller

Sensors 2023, 23(19), 8092; https://doi.org/10.3390/s23198092 - 26 Sep 2023

Cited by 1 | Viewed by 1147

Abstract

Emotional intelligence strives to bridge the gap between human and machine interactions. The application of such systems varies and is becoming more prominent as healthcare services seek to provide more efficient care by utilizing smart digital health apps. One application in digital health [...] Read more.

Emotional intelligence strives to bridge the gap between human and machine interactions. The application of such systems varies and is becoming more prominent as healthcare services seek to provide more efficient care by utilizing smart digital health apps. One application in digital health is the incorporation of emotion recognition systems as a tool for therapeutic interventions. To this end, a system is designed to collect and analyze physiological signal data, such as electrodermal activity (EDA) and electrocardiogram (ECG), from smart wearable devices. The data are collected from different subjects of varying ages taking part in a study on emotion induction methods. The obtained signals are processed to identify stimulus trigger instances and classify the different reaction stages, as well as arousal strength, using signal processing and machine learning techniques. The reaction stages are identified using a support vector machine algorithm, while the arousal strength is classified using the ResNet50 network architecture. The findings indicate that the EDA signal effectively identifies the emotional trigger, registering a root mean squared error (RMSE) of 0.9871. The features collected from the ECG signal show efficient emotion detection with 94.19% accuracy. However, arousal strength classification is only able to reach 60.37% accuracy on the given dataset. The proposed system effectively detects emotional reactions and can categorize their arousal strength in response to specific stimuli. Such a system could be integrated into therapeutic settings to monitor patients’ emotional responses during therapy sessions. This real-time feedback can guide therapists in adjusting their strategies or interventions. Full article

(This article belongs to the Special Issue Advanced-Sensors-Based Emotion Sensing and Recognition)

► Show Figures

Figure 1

16 pages, 431 KiB

Open AccessArticle

Diversifying Emotional Dialogue Generation via Selective Adversarial Training

by Bo Li, Huan Zhao and Zixing Zhang

Sensors 2023, 23(13), 5904; https://doi.org/10.3390/s23135904 - 25 Jun 2023

Viewed by 1009

Abstract

Emotional perception and expression are very important for building intelligent conversational systems that are human-like and attractive. Although deep neural approaches have made great progress in the field of conversation generation, there is still a lot of room for research on how to [...] Read more.

Emotional perception and expression are very important for building intelligent conversational systems that are human-like and attractive. Although deep neural approaches have made great progress in the field of conversation generation, there is still a lot of room for research on how to guide systems in generating responses with appropriate emotions. Meanwhile, the problem of systems’ tendency to generate high-frequency universal responses remains largely unsolved. To solve this problem, we propose a method to generate diverse emotional responses through selective perturbation. Our model includes a selective word perturbation module and a global emotion control module. The former is used to introduce disturbance factors into the generated responses and enhance their expression diversity. The latter maintains the coherence of the response by limiting the emotional distribution of the response and preventing excessive deviation of emotion and meaning. Experiments are designed on two datasets, and corresponding results show that our model outperforms existing baselines in terms of emotional expression and response diversity. Full article

(This article belongs to the Special Issue Advanced-Sensors-Based Emotion Sensing and Recognition)

► Show Figures

Figure 1

19 pages, 858 KiB

Open AccessArticle

Progressive Learning of a Multimodal Classifier Accounting for Different Modality Combinations

by Vijay John and Yasutomo Kawanishi

Sensors 2023, 23(10), 4666; https://doi.org/10.3390/s23104666 - 11 May 2023

Viewed by 1260

Abstract

In classification tasks, such as face recognition and emotion recognition, multimodal information is used for accurate classification. Once a multimodal classification model is trained with a set of modalities, it estimates the class label by using the entire modality set. A trained classifier [...] Read more.

In classification tasks, such as face recognition and emotion recognition, multimodal information is used for accurate classification. Once a multimodal classification model is trained with a set of modalities, it estimates the class label by using the entire modality set. A trained classifier is typically not formulated to perform classification for various subsets of modalities. Thus, the model would be useful and portable if it could be used for any subset of modalities. We refer to this problem as the multimodal portability problem. Moreover, in the multimodal model, classification accuracy is reduced when one or more modalities are missing. We term this problem the missing modality problem. This article proposes a novel deep learning model, termed KModNet, and a novel learning strategy, termed progressive learning, to simultaneously address missing modality and multimodal portability problems. KModNet, formulated with the transformer, contains multiple branches corresponding to different k-combinations of the modality set S. KModNet is trained using a multi-step progressive learning framework, where the k-th step uses a k-modal model to train different branches up to the k-th combination branch. To address the missing modality problem, the training multimodal data is randomly ablated. The proposed learning framework is formulated and validated using two multimodal classification problems: audio-video-thermal person classification and audio-video emotion classification. The two classification problems are validated using the Speaking Faces, RAVDESS, and SAVEE datasets. The results demonstrate that the progressive learning framework enhances the robustness of multimodal classification, even under the conditions of missing modalities, while being portable to different modality subsets. Full article

(This article belongs to the Special Issue Advanced-Sensors-Based Emotion Sensing and Recognition)

► Show Figures

Figure 1

15 pages, 1178 KiB

Open AccessArticle

Physiological Synchrony Predict Task Performance and Negative Emotional State during a Three-Member Collaborative Task

by Mohammed Algumaei, Imali Hettiarachchi, Rakesh Veerabhadrappa and Asim Bhatti

Sensors 2023, 23(4), 2268; https://doi.org/10.3390/s23042268 - 17 Feb 2023

Cited by 3 | Viewed by 1706

Abstract

Evaluation of team performance in naturalistic contexts has gained popularity during the last two decades. Among other human factors, physiological synchrony has been adopted to investigate team performance and emotional state when engaged in collaborative team tasks. A variety of methods have been [...] Read more.

Evaluation of team performance in naturalistic contexts has gained popularity during the last two decades. Among other human factors, physiological synchrony has been adopted to investigate team performance and emotional state when engaged in collaborative team tasks. A variety of methods have been reported to quantify physiological synchrony with a varying degree of correlation with the collaborative team task performance and emotional state, reflected in the inconclusive nature of findings. Little is known about the effect of the choice of synchrony calculation methods and the level of analysis on these findings. In this research work, we investigate the relationship between outcomes of different methods to quantify physiological synchrony, emotional state, and team performance of three-member teams performing a collaborative team task. The proposed research work employs dyadic-level linear (cross-correlation) and team-level non-linear (multidimensional recurrence quantification analysis) synchrony calculation measures to quantify task performance and the emotional state of the team. Our investigation indicates that the physiological synchrony estimated using multidimensional recurrence quantification analysis revealed a significant negative relationship between the subjectively reported frustration levels and overall task performance. However, no relationship was found between cross-correlation-based physiological synchrony and task performance. The proposed research highlights that the method of choice for physiological synchrony calculation has direct impact on the derived relationship of team task performance and emotional states. Full article

(This article belongs to the Special Issue Advanced-Sensors-Based Emotion Sensing and Recognition)

► Show Figures

Figure 1

12 pages, 1013 KiB

Open AccessArticle

Emotional Effects in Object Recognition by the Visually Impaired People in Grocery Shopping

by Michela Balconi, Carlotta Acconito and Laura Angioletti

Sensors 2022, 22(21), 8442; https://doi.org/10.3390/s22218442 - 03 Nov 2022

Viewed by 1788

Abstract

To date, neuroscientific literature on consumption patterns of specific categories of consumers, such as people with disability, is still scarce. This study explored the implicit emotional consumer experience of visually impaired (VI) consumers in-store. A group of VI and a control group explored [...] Read more.

To date, neuroscientific literature on consumption patterns of specific categories of consumers, such as people with disability, is still scarce. This study explored the implicit emotional consumer experience of visually impaired (VI) consumers in-store. A group of VI and a control group explored three different product shelves and manipulated target products during a real supermarket shopping experience. Autonomic (SCL, skin conductance level; SCR, skin conductance response; HR, heart rate; PVA, pulse volume amplitude; BVP, blood volume pulse), behavioural and self-report data were collected in relation to three phases of the in-store shopping experience: (i) identification of a product (recognition accuracy, ACC, and reaction times, RTs); (ii) style of product purchase (predominant sense used for shelf exploration, store spatial representation, and ability to orientate themselves); (iii) consumers experience itself, underlying their emotional experience. In the VI group, higher levels of disorientation, difficulty in finding products, and repeating the route independently were discovered. ACC and RTs also varied by product type. VI also showed significantly higher PVA values compared to the control. For some specific categories (pasta category), PVA correlates negatively with time to recognition and positively with simplicity in finding products in the entire sample. In conclusion, VI emotional and cognitive experience of grocery shopping as stressful and frustrating and has a greater cognitive investment, which is mirrored by the activation of a larger autonomic response compared to the control group. Nevertheless, VI ability to search and recognise a specific product is not so different from people without visual impairment. Full article

(This article belongs to the Special Issue Advanced-Sensors-Based Emotion Sensing and Recognition)

► Show Figures

Figure 1

15 pages, 5702 KiB

Open AccessArticle

Exploration of Emotion Dynamics Sensing Using Trapezius EMG and Fingertip Temperature

by Wataru Sato and Takanori Kochiyama

Sensors 2022, 22(17), 6553; https://doi.org/10.3390/s22176553 - 30 Aug 2022

Cited by 4 | Viewed by 1543

Abstract

Exploration of the physiological signals associated with subjective emotional dynamics has practical significance. Previous studies have reported that the dynamics of subjective emotional valence and arousal can be assessed using facial electromyography (EMG) and electrodermal activity (EDA), respectively. However, it remains unknown whether [...] Read more.

Exploration of the physiological signals associated with subjective emotional dynamics has practical significance. Previous studies have reported that the dynamics of subjective emotional valence and arousal can be assessed using facial electromyography (EMG) and electrodermal activity (EDA), respectively. However, it remains unknown whether other methods can assess emotion dynamics. To investigate this, EMG of the trapezius muscle and fingertip temperature were tested. These measures, as well as facial EMG of the corrugator supercilii and zygomatic major muscles, EDA (skin conductance level) of the palm, and continuous ratings of subjective emotional valence and arousal, were recorded while participants (n = 30) viewed emotional film clips. Intra-individual subjective–physiological associations were assessed using correlation analysis and linear and polynomial regression models. Valence ratings were linearly associated with corrugator and zygomatic EMG; however, trapezius EMG was not related, linearly or curvilinearly. Arousal ratings were linearly associated with EDA and fingertip temperature but were not linearly or curvilinearly related with trapezius EMG. These data suggest that fingertip temperature can be used to assess the dynamics of subjective emotional arousal. Full article

(This article belongs to the Special Issue Advanced-Sensors-Based Emotion Sensing and Recognition)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Advanced-Sensors-Based Emotion Sensing and Recognition

Share This Special Issue

Special Issue Editor

Special Issue Information

Published Papers (10 papers)

Research

Further Information

Guidelines

MDPI Initiatives

Follow MDPI