Recognition of Human Mental Stress Using Machine Learning: A Case Study on Refugees

Abstract

This paper introduces a study on stress recognition utilizing mobile EEG and GSR sensors. The research involved collecting samples from a group of 55 refugees who participated in Virtual Reality stress-reduction sessions. The timing of the study coincided with an influx of refugees, prompting the development of software specifically designed to alleviate acute stress among them. The paper focuses on presenting an EEG/GSR signals pipeline for classifying stress levels, emphasizing selecting the most informative features. The classification process employed popular machine learning methods, yielding results of 86.7% for two-stress-level classification and 82.3% and 67.7% for the three- and five-level classifications, respectively. Most importantly, the positive impact of the system has been proven by subjective assessment in alignment with objective features analysis. Such a system has not yet reached the level of autonomy, but it can be a valuable support tool for mental health professionals.

1. Introduction

The term stress, used as a descriptor of a human condition, first appeared in the scientific literature in the 1930s [1]. However, it came into everyday use in the 1970s [2]. Today it is often used to describe various unpleasant feelings, illustrating frustration, anger, dilemmas, and being overwhelmed or exhausted. Although the term is widely used, “stress” is a rather vague and challenging concept to define precisely [3].

According to the WHO, stress is defined as any type of change that causes physical, emotional, or psychological strain [4]. Stress is the human body’s response to a disturbance that requires attention or action [5]. People experience positive and negative emotions that affect their interactions with others, perception of reality, and general well-being. How one responds to a stressful situation defines their life’s comfort and health [6].

Stress causes changes in vegetative-somatic indicators. When in danger, the sympathetic system is activated, adrenaline secretion by the adrenal glands increases, blood pressure rises, the heart rate accelerates, blood glucose levels rise, and digestion processes are inhibited [7]. This is because stress affects the nervous system (especially the hypothalamus and pituitary gland) and mobilizes the body to a “fight or flight” response [8]. Short-term stress mobilizes the body for effort and action, while long-term stress may cause harm. The symptoms of an acute stress reaction usually subside quickly, depending on a person’s psychological ability to cope and recover. Untreated stress leads to more severe conditions, strains the nervous system, and causes somatic ailments [9]. That is why treatment and stress therapy is so important.

So-called stressors, i.e., events that disrupt the body’s equilibrium and strain or exceed our ability to cope effectively are invariably linked to the definition of stress. Their source can be physical (e.g., overcrowding, disease, natural disasters) or social (e.g., lack of employment, family problems, loss of a loved one, or crime) [10]. Stressors can also vary in terms of their strength. The weakest of them can be described as ordinary, everyday nuisances. However, despite their low strength, they can be problematic due to their commonness (e.g., tardiness, conflicts). The next level is severe stressors, which are associated with so-called life changes. These are ground-breaking circumstances, such as getting married, changing jobs, or the birth of a new family member [11]. The last level is catastrophic stressors, which dramatically affect entire groups of people (such as wars, natural disasters, and terrorist attacks), striking at the most basic human needs and values, such as life or safety. The stress accompanying them is usually extreme, sometimes traumatic, and can leave a long-lasting psychological effect [12]. Undoubtedly, refugees are in such a situation.

According to various estimates, more than 3 million refugees from Ukraine appeared in Poland in a short period [13]. Women and children found themselves in an environment that was utterly unfamiliar, often without essential personal belongings, money, or a guaranteed place to stay. Separated from their loved ones, who remained in the bombed-out cities, these individuals were naturally highly stressed [14].

Having previous experience in using VR technology for stress reduction [15,16], a Ukrainian-language VR-therapeutic application involving various relaxation techniques such as breathing methods, visualization, and bilateral stimulation (BLS) [17] was developed. Furthermore, relatively minimally invasive biomedical sensors (dry frontal electroencephalography—EEG [18], Galvanic Skin Response—GSR [19], and induced eye movement tracers) enhanced the application to continuously monitor users’ activities. As a result, user interaction with the application was completely natural, relying on gestures and bio-responses. Before and after the session, participants filled out questionnaires about their current mental health, stress levels, and general mood. The primary goal of this study is to validate the effectiveness of the proposed system using data recorded by portable EEG and GSR devices during VR-based training. This work’s contribution may be summed up as follows:

• An EEG/GSR-head mounted display (HMD) system is used to build a relaxation training system and assess stress levels;
• It has been demonstrated that a portable integrated HMD can collect high-quality EEG/GRS data during relaxation training in virtual reality, allowing for objective and quantitative monitoring of the stress level;
• The main contributions of this research include the creation of a novel EEG/GRS corpus for stress recognition, along with comparative experimental results that evaluate different approaches to managing this issue.

The rest of the paper is organized as follows. Section 2 presents the experiment specification and explains the selection of features used in the study. Section 3 presents the results of the experiment. I discuss the findings in Section 4 and provide some final remarks.

2. Materials and Methods

In this section, an EEG/GRS-based system for stress level recognition is presented. As depicted in Figure 1, to achieve this goal, the system has four sequential components: signal acquisition, preprocessing, pattern recognition, and device output (stress level).

2.1. VR-Based Stress Reduction Training

For the purpose of this study, a VR application was developed, specifically designed to simulate the process of stress reduction therapy. Within the virtual environment, a quiet mountain apartment is depicted, carefully crafted to create a soothing atmosphere. The initial scene serves as a setup and tutorial, offering customization options for the virtual camera and the patient’s location.

During the therapy session, each participant is encouraged to relax in a comfortable chair (see Figure 2). The visual scene presented to the participant is automatically calibrated based on their head position, ensuring that the central element—a window showcasing a stunning mountain view—is perfectly aligned vertically and horizontally for the individual participant. The VR device incorporates built-in sensors to accurately determine the participant’s position.

To ensure the participant’s safety and minimize the potential occurrence of negative effects during the simulation, such as cybersickness or VR sickness, a 2 min adaptive session is initially employed. This session helps acclimate the participant to the virtual environment before proceeding to the actual relaxation simulation. Throughout the experience, a native Ukrainian narrator guides the user, providing instructions on various relaxation techniques, including breathing techniques, visualization, and bilateral stimulation.

2.2. Course of the Experiment

To recruit participants, an information campaign was conducted in cooperation with the City of Łódź. This campaign began with a press conference, widely covered in local media, such as television, newspapers, and social media, dealing mainly with Ukrainian affairs.

Before each session, users were informed about the course of the VR experience and the exclusionary conditions, and provided informed consent. The entire course of the session was monitored by the supervisory staff. It was assumed that each participant would be able to benefit from five therapeutic sessions (one session per week). Unfortunately, due to personal reasons and relocation, not everyone has completed the session sequence.

Finally, we recruited n = 55 participants, 52 female, 3 male, aged M = 35, SD = 10.93. Detailed participants characteristics are presented in

Table 1. Participant characteristics.

Partic.

#1

#2

#3

#4

#5

#6

#7

#8

#9

#10

#11

#12

#13

#14

#15

#16

#17

#18

#19

#20

Sex

F

F

F

F

F

F

F

F

F

F

F

F

F

F

F

F

F

F

F

M

Age

31

24

29

46

26

33

28

40

41

24

31

22

32

45

52

54

31

31

71

35

Partic.

#21

#22

#23

#24

#25

#26

#27

#28

#29

#30

#31

#32

#33

#34

#35

#36

#37

#38

#39

#40

Sex

F

F

F

F

F

F

F

F

F

F

F

M

F

F

F

F

M

F

F

M

Age

47

35

35

31

22

44

45

35

58

45

36

19

39

33

39

44

38

31

51

20

Partic.

#41

#42

#43

#44

#45

#46

#47

#48

#49

#50

#51

#52

#53

#54

#55

Sex

F

F

F

F

F

F

F

F

F

F

F

F

F

F

F

Age

19

24

31

25

46

32

23

28

31

24

35

39

39

57

36

. Most participants came to the sessions repeatedly (up to 5 times/once a week). The study was conducted at Łódź University of Technology, in a period when the university was closed (there were no students), in a quiet, air-conditioned room.

The participant’s primary objective is to track a moving sphere with their eyes as it traverses an invisible horizontal line outside the window. Sensors integrated into the display glasses continuously monitor the participant’s gaze-tracking accuracy throughout the session. The study comprises eight 90 s sessions of BLS (see Figure 3), wherein the participant focuses on following the sphere located at the center of their visual field. This division into BLS sessions and breaks ensured that the study participant had adequate time to relax and unwind, thus avoiding fatigue and keeping the BLS session as effective as possible.

2.3. EEG

A non-invasive EEG system, Looxid Link, was used for the study to maintain comfort during the session. Looxid Link is a VR headset-compatible hardware configuration that includes a frontal six-channel EEG system (F3, F4, F7, F8, Fp1, and Fp2) conforming to International System 10-10 [20] (see Figure 4). Unlike conventional wet-EEG with wires and voluminous amplifiers, wireless dry-electrode mobile EEG allows for effortless attachment to the VR headset and it is virtually unnoticeable by the user [21].

Table 2. Data generated at the output of Looxid Link.

Data Type	Description
TimeStamp	The moment in time when specific data was recorded during the session; on average, 90 entries per second;
leftActivity	A marker indicating the activity of the left hemisphere of the brain, calculated from sensors located on the left side of the prefrontal cortex (an algorithm built in Looxid Link);
rightActivity	A marker indicating the activity of the right hemisphere of the brain, calculated from sensors located on the right side of the prefrontal cortex (an algorithm built in Looxid Link);
attention	A marker that determines the subject’s overall focus in the range of 0–1 (an algorithm built in Looxid Link);
relaxation	A marker determining the relaxation of the subject in the range of 0–1 (an algorithm built in Looxid Link);
asymmetry	A marker determining the ratio of left to right brain hemisphere activity;
frequency bands	Delta, theta, alpha, beta, and gamma for each of the 6 sensors.

shows the data generated at the output of the device.

Delta, theta, and alpha brain waves are categorized as “slow” brain waves, indicating a more relaxed or tranquil state of the brain. For instance, delta brain waves are associated with deep sleep and help facilitate falling asleep. Alpha waves are activated when someone is engaged in activities like meditation or enjoying a leisurely day. On the other hand, beta, high beta, and gamma brain waves are classified as “fast”. When these brainwaves dominate, it signifies an active and engaged state of the brain. Beta waves are associated with significant cognitive activities, while gamma waves are involved in deep concentration [22].

During times of stress, beta brain waves become hyperactive, triggering the amygdala, which is responsible for emotional processing. This leads to the production of high beta brain waves, making them fast and active. It activates the nervous and adrenal systems, preparing the body for a physical response to the stressor. Ideally, once the stressor is no longer present, the intensity of high beta waves diminishes, allowing the individual to return to a relaxed state [23].

This research utilized the Grove GSR sensor in conjunction with a Raspberry Pi 3. GSR sensors measure and record changes in the electrical resistance of the skin. The skin’s resistance is influenced by different levels of hydration due to sweat gland activity controlled by the sympathetic nervous system. A typical GSR reaction involves a relationship between skin resistance and sweating—as the sweat glands fill up, the skin becomes more conductive. GSR is believed to be influenced by the psychological state of the individual, where increased activity in the cerebral cortex is reflected in sweating and higher GSR readings. Conversely, a decrease in stimulation leads to higher skin resistance and lower GSR [24]. Typically, the measured value is the skin resistance or its inverse conductance (R = 1/G). The process involves placing two dry electrodes on two fingers (refer to Figure 4). It is important to maintain an appropriate room temperature during the experiment to avoid inducing additional sweating that could affect the accuracy of the collected measurements. Furthermore, participants were instructed not to move their hands during the session, to prevent any artifacts in the signal. An example of a GSR waveform recorded for a random participant during the relaxation session is presented in Figure 5.

3. Results

After discarding files unsuitable for further analysis (unsteady connection of EEG electrodes or GRS sensors during testing), a total of 155 samples were obtained (simultaneous EEG and GSR). However, dividing the recordings into “before” and “after” samples, it was found that 5 of them had stopped registering roughly in the middle. Therefore, for classification, ultimately, 305 samples were used, of which 155 were collected before the VR sessions and 150 at the end of the VR sessions. To recognize stress levels, two approaches were used. In the first one, only the psychological condition into two states—before and after the session—was classified. The second was based on participants’ self-assessment before and after the session. The following sections will describe the results of both approaches.

3.1. Two-Stress-Level Analysis

Since the majority (98%) of study participants indicated a significant reduction in stress levels during the proposed VR session, the first step was to verify the possibility of recognizing the initial (higher stress levels) and final (reduced stress levels) states. Thus, two fragments of the signal waveforms were juxtaposed: the last 60 s from the Intro session and the first 60 s from the Finish session (just after the last BLS). From these fragments, a high-dimensional feature set, which initially consisted of 312 descriptors, was created. All the features were based on the outputs of both EEG and GSR sensors. More specifically, statistical characteristics such as mean, median, standard deviation, first and third quartiles, kurtosis, and skewness were determined from all outputs. Since the final set contained 1.2% missing data, an argumentation with the average/most frequent value was performed. Then, several different selection methods were juxtaposed to minimize the feature set: one wrapper method (Sequential Forward Selection, SFS), one filter method (Mutual Information, MI), and one embedded method (LASSO). The methods were implemented with preliminary feature correlation analysis, which rejected features with a degree of correlation higher than 0.8. This operation reduced the initial set size from 312 to 108. The selected features, listed in the ranking order, are presented in Figure 6. Since none of the classifiers achieved maximum accuracy performance with a set of more than 20 features, the table is limited to this number.

As can be noticed in the case of the SFS and MI methods, 80% of the features overlap. In contrast, for the LASSO method, the similarity of the sets is much lower, amounting to 35% and 45% with the SFS and MI methods, respectively. It should also be noted that the first two methods rank quite high for GSR-related features not present in the LASSO set. The distribution of EEG-based features derived from specific electrodes is shown in Figure 7. The most frequently chosen features come from the left hemisphere from electrodes Fp1, F7, and F3 for each selection method.

Next, five different classifiers (k-NN [25], AdaBoost [26], Random Forest RF [27], multilayer perceptron MLP [28]: 200 neurons in hidden layers, activation function ReLu, and CN2 induction algorithm [29]) with previously mentioned feature selection methods were employed. Each model’s parameters were chosen experimentally to maximize the results. Tests were performed using 10-fold cross-validation. The results are presented in

Table 3. Classification results for two stress levels (before and after relaxation session). The best result is bolded.

	k-NN	AdaBoost	RF	MLP	CN2
SFS	74.5	77.4	83	86.7	82.8
MI	74.5	80	84.3	86.1	85
LASSO	76.5	67.5	76.1	78.4	76.4

.

MLP yielded the most favorable outcomes when distinguishing between two states: before and after relaxation. The impact of feature selection methods on the quality of classification was minimal for SFS and MI. For LASSO, the classification drops by more than 8%. When considering SFS and MI, the results were comparable, reaching an accuracy level of 86%. Figure 8 illustrates the correlation between the number of features and the classification outcomes achieved through the SFS selection method for two classes. Furthermore, Figure 9 depicts the confusion matrix for MLP and SFS selection.

3.2. Stress Level Analysis Based on Participants Labels

At this stage, subjective ratings given before and after the survey by the participants were used to identify the stress level. Since the questionnaire answer range was set from 0 to 10, to allow successful classification the adjacent answers were grouped in classes. This solution has been tested for two cases, three and five classes, as illustrated in Figure 10.

Thus, in the first case, if a participant identified their overall stress in the range of 0 to 3, it was marked as low, 4 to 7 as medium, and 8 to 10 as high. In the second case, a division was made into five ranges, and so if a participant defined their overall stress in the range of 0 to 1, it was marked as very low, 2 to 3 as low, 4 to 6 as medium, 7 to 8 as high, and 9 to 10 as very high. The distribution of labels obtained from participants before and after the session is presented in Figure 11.

Similar to the previous case, different feature selection methods were used, and the results of the various classification methods were compared. The results obtained are shown in

Table 4. Classification results for three stress levels (low, medium, and high).

	k-NN	AdaBoost	RF	MLP	CN2
SFS	73	76.8	78.4	82.3	82.3
MI	72.4	76.1	76.7	80.4	80.4
LASSO	68.4	73.4	75.9	78.8	79

and

Table 5. Classification results for five stress levels (very low, medium, high, and very high).

	k-NN	AdaBoost	RF	MLP	CN2
SFS	58.4	57.8	63	67.7	65.7
MI	58	56.2	61.1	66.9	65.1
LASSO	55.1	54.9	59.9	61.1	63.4

for three and five classes, respectively.

As for two-class recognition, the highest classification quality was in the 16–20 descriptor range. Therefore, Figure 12 refers only to the top 20 features (listed in ranking order). The results present descriptors selected by the SFS method. The distribution of EEG-based features derived from specific electrodes is shown in Figure 13.

For three classes in Figure 14, the best performance was achieved by the MLP and CN2 algorithms, reaching an accuracy of 82.3%. Similarly to the previous case, the chosen feature selection method exhibited minimal influence on the outcomes, resulting in a mere 2% variation. Upon partitioning the data into four distinct classes, the algorithms exhibited comparable behavior, albeit with a noticeable decline in recognition accuracy. The highest achievement was recorded by MLP with SFS, yielding 67.7% accuracy, which signifies a high deterioration in performance compared to the three-class classification. Figure 15 depicts the confusion matrix for MLP and SFS selection for both cases.

4. Discussion and Conclusions

This paper explores the feasibility of classifying stress levels among refugees using an EEG/GRS dataset. The signals were extensively analyzed, emphasizing feature selection to predict stress across three scales. The findings suggest that it is possible to achieve a 67.7% accuracy in recognizing five stress classes. Notably, reducing the number of levels to three leads to a significant improvement, increasing the classification results by 14.6%. Furthermore, the study demonstrates the ability to discriminate between the psychical condition before and after the session with an accuracy of 86.7%.

The accuracy results obtained for both five and three classes fall short of meeting the requirements for implementing the proposed system as a reliable decision-support tool in real-world clinical applications. However, it is important to note that this limitation can be attributed, at least partially, to the limited size of the training dataset, particularly due to the uneven distribution of subjective labels. Participants’ subjective evaluation of their stress levels can introduce ambiguity in the dataset. Utilizing a 0 to 10 scale may prove challenging for humans to determine accurately, and this can lead to difficulties in effectively training the algorithm. In this case, a scale ranging from 1 to 5 would have been preferable. Additionally, there were issues with the equipment used in the study, as it sometimes disconnected during the sessions. Consequently, the number of complete samples available for analysis was significantly reduced compared to the number of participants involved and the number of sessions conducted. This equipment-related problem posed an additional challenge in achieving reliable and robust results.

However, the system demonstrates a high level of reliability in distinguishing between states before and after the session, indicating the positive influence of the system on participants. This assertion is further supported by the participants’ subjective opinions, which align with the system outcomes. Additionally, the recordings were captured in an authentic environment amidst a challenging group of individuals and with background noise. These factors make this particular database highly valuable for research and analysis purposes.

The successful integration of machine learning methods for identifying and classifying stress levels holds great promise in quantifying therapy response. To further enhance the accurate discrimination of stress levels, incorporating additional information alongside EEG/GRS features shows potential. This augmentation can improve classification performance, enabling monitoring systems to predict treatment response. As a result, the entire framework can be deployed as a decision support system for psychologists, offering valuable insights and aiding in treatment planning.