Submit to Sensors Review for Sensors Propose a Special Issue

Journal Menu

Journal Browser

► Journal Browser

Sensor Data Processing for Social Reasoning and Artificial Empathy in Human-Centred Systems

Print Special Issue Flyer
Special Issue Editors
Special Issue Information
Keywords
Published Papers

A special issue of Sensors (ISSN 1424-8220). This special issue belongs to the section "Sensors and Robotics".

Deadline for manuscript submissions: closed (25 March 2023) | Viewed by 17932

Share This Special Issue

Special Issue Editors

Dr. Erhan Ekmekcioglu

E-Mail Website
Guest Editor

Institute for Digital Technologies, Loughborough University London, London E15 2GZ, UK
Interests: video processing; video analytics; affective computing; biosignal processing; human-computer interaction
Special Issues, Collections and Topics in MDPI journals

Dr. Yücel ÇİMTAY

E-Mail Website
Guest Editor

Department of Computer Engineering, TED University, Ziya Gökalp Caddesi No:48 06420, Kolej Çankaya, Ankara, Turkey
Interests: signal and image processing; image and video restoration; hyperspectral image processing; remote sensing; computer vision; machine learning; deep learning

Special Issue Information

Dear Colleagues,

Recent technological breakthroughs in a myriad of distinct but interconnected domains such as wireless and wearable sensor systems, multimedia processing and analytics, virtual and augmented reality, machine learning, affective computing, Artificial Intelligence, robotics and autonomous environments have largely transformed the way people live and interact with their surroundings. The focus is now shifting towards making systems ever more human-centred, from design to implementation, addressing usability, user experience and interactivity. Areas of application for human-centred design vary in a wide spectrum, from education, training, healthcare, transportation to manufacturing, entertainment and gaming. One of the key prospects in truly human-centred systems, which embed human-to-robot or human-to-agent interaction, whether in physical, digital, or virtual reality environments, is that they will feature higher degrees of understanding of the human cognitive and emotional processes, as well as interaction patterns. This is so that they do empathetic and social reasoning and interact with humans accordingly. It is possible to create socially intelligent and reactive agents to interact and collaborate with humans in more natural and effective ways in various forms, such as software agents, virtual avatars, or physical robots. There are many challenges yet to address in this field from real-time personality analysis, emotional decoding, action recognition and anticipation to developing models of effective interaction or collaboration in physical and virtual spaces, designing reactive and empathetic systems leading to superior user experience. The goal of this Special Issue is to gather the results from recent research activities towards the creation of human-centred systems, which may integrate various forms of human-to-agent or human-to-robot interaction and novel interfaces (e.g., brain-computer interfaces). Emphasis will be on real-time processing of multimodal sensor data, human behaviour, cognition, and emotional decoding using advanced predictive models, artificial empathy, and human-centred collaboration design. It is expected to receive papers presenting novel algorithms, system architectures, and emerging applications, as well as comprehensive survey papers that review the novel technologies and new trends in this area.

Dr. Erhan Ekmekcioglu
Dr. Yücel ÇİMTAY
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Sensors is an international peer-reviewed open access semimonthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

affective computing
human-computer interaction
human-robot interaction
human-centred design
multimodal sensor data fusion
emotion recognition
intent recognition
activity recognition
deception detection
drowsiness detection
distraction detection

Published Papers (7 papers)

Download All Papers

Order results

Result details

Show export options Show export options

Select all

Export citation of selected articles as:

Research

17 pages, 3601 KiB

Open AccessArticle

EEG-Based Driver Fatigue Monitoring within a Human–Ship–Environment System: Implications for Ship Braking Safety

by Bin Ren, Wanli Guan, Qinyu Zhou and Zilin Wang

Sensors 2023, 23(10), 4644; https://doi.org/10.3390/s23104644 - 10 May 2023

Cited by 3 | Viewed by 1390

Abstract

To address the uncontrollable risks associated with the overreliance on ship operators’ driving in current ship safety braking methods, this study aims to reduce the impact of operator fatigue on navigation safety. Firstly, this study established a human–ship–environment monitoring system with functional and technical architecture, emphasizing the investigation of a ship braking model that integrates brain fatigue monitoring using electroencephalography (EEG) to reduce braking safety risks during navigation. Subsequently, the Stroop task experiment was employed to induce fatigue responses in drivers. By utilizing principal component analysis (PCA) to reduce dimensionality across multiple channels of the data acquisition device, this study extracted centroid frequency (CF) and power spectral entropy (PSE) features from channels 7 and 10. Additionally, a correlation analysis was conducted between these features and the Fatigue Severity Scale (FSS), a five-point scale for assessing fatigue severity in the subjects. This study established a model for scoring driver fatigue levels by selecting the three features with the highest correlation and utilizing ridge regression. The human–ship–environment monitoring system and fatigue prediction model proposed in this study, combined with the ship braking model, achieve a safer and more controllable ship braking process. By real-time monitoring and prediction of driver fatigue, appropriate measures can be taken in a timely manner to ensure navigation safety and driver health. Full article

(This article belongs to the Special Issue Sensor Data Processing for Social Reasoning and Artificial Empathy in Human-Centred Systems)

► Show Figures

Figure 1

22 pages, 950 KiB

Open AccessArticle

Exploring Prosodic Features Modelling for Secondary Emotions Needed for Empathetic Speech Synthesis

by Jesin James, Balamurali B.T., Catherine Watson and Hansjörg Mixdorff

Sensors 2023, 23(6), 2999; https://doi.org/10.3390/s23062999 - 10 Mar 2023

Viewed by 1470

Abstract

A low-resource emotional speech synthesis system for empathetic speech synthesis based on modelling prosody features is presented here. Secondary emotions, identified to be needed for empathetic speech, are modelled and synthesised in this investigation. As secondary emotions are subtle in nature, they are difficult to model compared to primary emotions. This study is one of the few to model secondary emotions in speech as they have not been extensively studied so far. Current speech synthesis research uses large databases and deep learning techniques to develop emotion models. There are many secondary emotions, and hence, developing large databases for each of the secondary emotions is expensive. Hence, this research presents a proof of concept using handcrafted feature extraction and modelling of these features using a low-resource-intensive machine learning approach, thus creating synthetic speech with secondary emotions. Here, a quantitative-model-based transformation is used to shape the emotional speech’s fundamental frequency contour. Speech rate and mean intensity are modelled via rule-based approaches. Using these models, an emotional text-to-speech synthesis system to synthesise five secondary emotions-anxious, apologetic, confident, enthusiastic and worried-is developed. A perception test to evaluate the synthesised emotional speech is also conducted. The participants could identify the correct emotion in a forced response test with a hit rate greater than 65%. Full article

(This article belongs to the Special Issue Sensor Data Processing for Social Reasoning and Artificial Empathy in Human-Centred Systems)

► Show Figures

Figure 1

13 pages, 1192 KiB

Open AccessArticle

Synergy Masks of Domain Attribute Model DaBERT: Emotional Tracking on Time-Varying Virtual Space Communication

by Ye Wang, Zhenghan Chen and Changzeng Fu

Sensors 2022, 22(21), 8450; https://doi.org/10.3390/s22218450 - 3 Nov 2022

Cited by 1 | Viewed by 1115

Abstract

Emotional tracking on time-varying virtual space communication aims to identify sentiments and opinions expressed in a piece of user-generated content. However, the existing research mainly focuses on the user’s single post, despite the fact that social network data are sequential. In this article, we propose a sentiment analysis model based on time series prediction in order to understand and master the chronological evolution of the user’s point of view. Specifically, with the help of a domain-knowledge-enhanced pre-trained encoder, the model embeds tokens for each moment in the text sequence. We then propose an attention-based temporal prediction model to extract rich timing information from historical posting records, which improves the prediction of the user’s current state and personalizes the analysis of user’s sentiment changes in social networks. The experiments show that the proposed model improves on four kinds of sentiment tasks and significantly outperforms the strong baseline. Full article

(This article belongs to the Special Issue Sensor Data Processing for Social Reasoning and Artificial Empathy in Human-Centred Systems)

► Show Figures

Figure 1

17 pages, 2804 KiB

Open AccessArticle

Shift Pose: A Lightweight Transformer-like Neural Network for Human Pose Estimation

by Haijian Chen, Xinyun Jiang and Yonghui Dai

Sensors 2022, 22(19), 7264; https://doi.org/10.3390/s22197264 - 25 Sep 2022

Cited by 6 | Viewed by 2642

Abstract

High-performing, real-time pose detection and tracking in real-time will enable computers to develop a finer-grained and more natural understanding of human behavior. However, the implementation of real-time human pose estimation remains a challenge. On the one hand, the performance of semantic keypoint tracking in live video footage requires high computational resources and large parameters, which limiting the accuracy of pose estimation. On the other hand, some transformer-based models were proposed recently with outstanding performance and much fewer parameters and FLOPs. However, the self-attention module in the transformer is not computationally friendly, which makes it difficult to apply these excellent models to real-time jobs. To overcome the above problems, we propose a transformer-like model, named ShiftPose, which is regression-based approach. The ShiftPose does not contain any self-attention module. Instead, we replace the self-attention module with a non-parameter operation called the shift operator. Meanwhile, we adapt the bridge–branch connection, instead of a fully-branched connection, such as HRNet, as our multi-resolution integration scheme. Specifically, the bottom half of our model adds the previous output, as well as the output from the top half of our model, corresponding to its resolution. Finally, the simple, yet promising, disentangled representation (SimDR) was used in our study to make the training process more stable. The experimental results on the MPII datasets were 86.4 PCKH, 29.1PCKH@0.1. On the COCO dataset, the results were 72.2 mAP and 91.5 AP50, 255 fps on GPU, with 10.2M parameters, and 1.6 GFLOPs. In addition, we tested our model for single-stage 3D human pose estimation and draw several useful and exploratory conclusions. The above results show good performance, and this paper provides a new method for high-performance, real-time attitude detection and tracking. Full article

(This article belongs to the Special Issue Sensor Data Processing for Social Reasoning and Artificial Empathy in Human-Centred Systems)

► Show Figures

Figure 1

11 pages, 324 KiB

Open AccessCommunication

Sentiment Analysis and Emotion Recognition from Speech Using Universal Speech Representations

by Bagus Tris Atmaja and Akira Sasou

Sensors 2022, 22(17), 6369; https://doi.org/10.3390/s22176369 - 24 Aug 2022

Cited by 16 | Viewed by 5600

Abstract

The study of understanding sentiment and emotion in speech is a challenging task in human multimodal language. However, in certain cases, such as telephone calls, only audio data can be obtained. In this study, we independently evaluated sentiment analysis and emotion recognition from speech using recent self-supervised learning models—specifically, universal speech representations with speaker-aware pre-training models. Three different sizes of universal models were evaluated for three sentiment tasks and an emotion task. The evaluation revealed that the best results were obtained with two classes of sentiment analysis, based on both weighted and unweighted accuracy scores (81% and 73%). This binary classification with unimodal acoustic analysis also performed competitively compared to previous methods which used multimodal fusion. The models failed to make accurate predictionsin an emotion recognition task and in sentiment analysis tasks with higher numbers of classes. The unbalanced property of the datasets may also have contributed to the performance degradations observed in the six-class emotion, three-class sentiment, and seven-class sentiment tasks. Full article

(This article belongs to the Special Issue Sensor Data Processing for Social Reasoning and Artificial Empathy in Human-Centred Systems)

► Show Figures

Figure 1

14 pages, 734 KiB

Open AccessArticle

Effects of Data Augmentations on Speech Emotion Recognition

by Bagus Tris Atmaja and Akira Sasou

Sensors 2022, 22(16), 5941; https://doi.org/10.3390/s22165941 - 9 Aug 2022

Cited by 7 | Viewed by 2211

Abstract

Data augmentation techniques have recently gained more adoption in speech processing, including speech emotion recognition. Although more data tend to be more effective, there may be a trade-off in which more data will not provide a better model. This paper reports experiments on investigating the effects of data augmentation in speech emotion recognition. The investigation aims at finding the most useful type of data augmentation and the number of data augmentations for speech emotion recognition in various conditions. The experiments are conducted on the Japanese Twitter-based emotional speech and IEMOCAP datasets. The results show that for speaker-independent data, two data augmentations with glottal source extraction and silence removal exhibited the best performance among others, even with more data augmentation techniques. For the text-independent data (including speaker and text-independent), more data augmentations tend to improve speech emotion recognition performances. The results highlight the trade-off between the number of data augmentations and the performance of speech emotion recognition showing the necessity to choose a proper data augmentation technique for a specific condition. Full article

(This article belongs to the Special Issue Sensor Data Processing for Social Reasoning and Artificial Empathy in Human-Centred Systems)

► Show Figures

Figure 1

25 pages, 4505 KiB

Open AccessArticle

A Sentiment Analysis Method Based on a Blockchain-Supported Long Short-Term Memory Deep Network

by Arif Furkan Mendi

Sensors 2022, 22(12), 4419; https://doi.org/10.3390/s22124419 - 11 Jun 2022

Cited by 8 | Viewed by 2303

Abstract

Traditional sentiment analysis methods are based on text-, visual- or audio-processing using different machine learning and/or deep learning architecture, depending on the data type. This situation comes with technical processing diversity and cultural temperament effect on analysis of the results, which means the results can change according to the cultural diversities. This study integrates a blockchain layer with an LSTM architecture. This approach can be regarded as a machine learning application that enables the transfer of the metadata of the ledger to the learning database by establishing a cryptographic connection, which is created by adding the next sentiment with the same value to the ledger as a smart contract. Thus, a “Proof of Learning” consensus blockchain layer integrity framework, which constitutes the confirmation mechanism of the machine learning process and handles data management, is provided. The proposed method is applied to a Twitter dataset with the emotions of negative, neutral and positive. Previous sentiment analysis methods on the same data achieved accuracy rates of 14% in a specific culture and 63% in a the culture that has appealed to a wider audience in the past. This study puts forth a very promising improvement by increasing the accuracy to 92.85%. Full article

(This article belongs to the Special Issue Sensor Data Processing for Social Reasoning and Artificial Empathy in Human-Centred Systems)

► Show Figures

Journal Menu

Journal Browser

Sensor Data Processing for Social Reasoning and Artificial Empathy in Human-Centred Systems

Share This Special Issue

Special Issue Editors

Special Issue Information

Keywords

Published Papers (7 papers)

Research

Further Information

Guidelines

MDPI Initiatives

Follow MDPI