EEG-Based Empathic Safe Cobot

Abstract

An empathic collaborative robot (cobot) was realized through the transmission of fear from a human agent to a robot agent. Such empathy was induced through an electroencephalographic (EEG) sensor worn by the human agent, thus realizing an empathic safe brain-computer interface (BCI). The empathic safe cobot reacts to the fear and in turn transmits it to the human agent, forming a social circle of empathy and safety. A first randomized, controlled experiment involved two groups of 50 healthy subjects (100 total subjects) to measure the EEG signal in the presence or absence of a frightening event. The second randomized, controlled experiment on two groups of 50 different healthy subjects (100 total subjects) exposed the subjects to comfortable and uncomfortable movements of a collaborative robot (cobot) while the subjects’ EEG signal was acquired. The result was that a spike in the subject’s EEG signal was observed in the presence of uncomfortable movement. The questionnaires were distributed to the subjects, and confirmed the results of the EEG signal measurement. In a controlled laboratory setting, all experiments were found to be statistically significant. In the first experiment, the peak EEG signal measured just after the activating event was greater than the resting EEG signal (p < 10⁻³). In the second experiment, the peak EEG signal measured just after the uncomfortable movement of the cobot was greater than the EEG signal measured under conditions of comfortable movement of the cobot (p < 10⁻³). In conclusion, within the isolated and constrained experimental environment, the results were satisfactory.

1. Introduction

Collaborative robots (cobot) [1] are special robots that can collaborate with humans. They were originally intended for industrial applications [2,3,4,5,6,7,8,9], but they have also found use in biomedical [10,11,12,13,14,15,16,17,18], domestic [19,20,21,22], and military [23,24,25,26] fields.

Because of the physical proximity of humans and robots, many scientific works in the field of mechatronics [27,28,29] focus on the relationship’s safety [30] and comfort [31]. Several authors have evaluated the use of natural interface systems, such as vision [32,33] and face recognition and communication [34,35,36,37], gestures [38,39,40], and spoken natural language [41,42,43,44], to improve the relationship. This process has led to research into incorporating the concept of empathy [45] into the human–robot relationship by giving the robot the ability to decode the emotional state of the human subject [46]. The results in the literature are almost entirely based on the analysis of human facial expressions [47,48,49,50], sometimes supplemented by gestures and body language [51,52,53] or by voice modulation [54,55,56,57].

On the basis of psychological studies, the literature differentiates at least two groups of emotions, namely primary and secondary emotions [58]. The category of primary emotions is thought to be intrinsic [58]. Literature says that the main emotions developed during phylogeny to help people act quickly and instinctively when danger is near. Primary emotions are also thought to be prototype emotion kinds that can be assigned to one-year-old children [59]. Secondary emotions such as “relief” or “hope” are thought to emerge from higher cognitive processes based on the ability to evaluate choices over outcomes and expectations. Then, secondary emotions are acquired through social learning processes. The adjective “secondary” is used in the literature to allude to “adult” emotions. Furthermore, secondary emotions, like main emotions, influence bodily (and facial) manifestations of primary emotions. Different authors listed different primary and secondary emotions, i.e., according to Becker et al. [60], the nine primary emotions are: angry, annoyed, bored, concentrated, depressed, fearful, happy, sad, surprised. Always according to Becker et al. [60], the three secondary emotions are: hope, fears-confirmed, and relief.

Studies on functional neuroanatomy of emotion of brain show that in the majority of investigations, across individual emotions and induction methods, no unique brain region was consistently active, implying that no single brain region is routinely activated by all emotional tasks [61]. The same meta-analysis article [61] classifies five primary emotions, happiness, fear, anger, sadness, and disgust and enumerated the studies in the literature that reported the activation of twenty brain regions. Respectively, happiness, fear, anger, sadness, and disgust activated 18, 17, 16, 19, and 13 brain regions. So, it might be a good idea to use a mathematical relationship based on ElectroEncephaloGraphic (EEG) signals and the activation or sequence of activation of multiple brain regions to figure out what emotions a person is feeling.

At first, applications for EEG were developed to help people with disabilities [62] in a variety of areas, including substitution of natural verbal and non-verbal communication [63,64,65], the connection with ICT devices [66], and the use of assistive orthoses [67], such as mobile systems [68] and robotic manipulators [69], exoskeletons [70], prosthetic limbs [71], or grippers [72]. Assisting patients suffering from paralysis [73], amputations [74], and dysfunctions of the central nervous system [75] are some of the other uses for EEG-based prostheses.

EEG-based brain–computer interface applications have been used also in non-medical domain, i.e., the entertainment industry [76]. There have been studies conducted in non-medical fields, such as the creation of devices to evaluate the level of attentiveness [77]. Another topic has been investigated is ways to ensure the overall safety in professional and daily living activities [78,79,80]. Applications based on EEG might be used to control the operation of intelligent houses [81]. EEG has the potential to be used for a variety of non-medical applications, including the empowerment of non-pathological subjects’ performances [82]. The use of drones [83] and robot control [84] might also fall under the category of applications, along with video games [85] designed to increase a person’s ability to concentrate or pay attention.

Some authors have used EEG technology [86,87,88] to identify emotions [89,90,91,92,93], but this approach does not appear to have been applied to robotics, particularly collaborative robotics, to the best of the authors’ knowledge. In general, the EEG signal is used in robotics to identify stressful conditions or decreased concentration [94,95] or to send voluntary commands to the robot [96].

The work proposed in [97] analyzed the EEG signal to discriminate between: interest, stress, relaxation, excitement, engagement, long-term excitement, and focus. This approach may be of interest, for example, in assessing operator stress and ability to concentrate in order to reduce one of the risk factors for an accident and recommend a break or slowdown in operations, as described in [98]. The work proposed in [99] describes a brain–computer–interface based on an EEG sensor to command the motion of a virtual robot. All these research exhibited an accuracy limited between the range 80% and 90%.

Thus, in this work, we introduce a first novel element in comparison to the literature: we use EEG to transfer information about the emotional state of the human subject from the human subject to the cobot, thereby promoting the formation of an empathic relationship between human and cobot. We want to focus on the aspect of safety in particular: when a human subject perceives danger, he or she feels fear, which causes hyperactivation of various brain regions [100] and then produces a rapid reaction to move away from the dangerous situation. As a result, monitoring the EEG signal could quickly transfer the fear information to the cobot. To be able to use this approach, an initial hypothesis will be introduced in this paper, namely that the subject and the robot proceed in their activity without interactions with other subjects, so that the human subject’s brain hyperactivation can actually be associated with a fear emotion. Fear is the most important and intense emotion because it allows the subject to survive [101]; thus, the rationale of this article is in favor of the subject’s safety; that is, even if other strong emotions distract the subject from interacting with the robot, the robot will interpret them as an emotion of fear in order to maintain itself in a safe condition. Fear is transmitted from one subject to another in human society through the decoding of facial expressions and nonverbal signals, with the receiving subject unaware of the reason for the fear [102]. This evolutionary phenomenon serves to shorten the reaction time to danger and protect the entire society and all of its members. As a result, the approach proposed in this paper tends to facilitate the transition from a collaborative to a social dimension, even for current collaborative robots that become sobots [103]. Following the perception of the fear signal, the robot implements a strategy to contain the danger by moving to a safe position; this behavior should be recognized by human subjects as a nonverbal message of danger, allowing for a rapid transmission of fear from the robot to the human subjects.

2. Materials and Methods

Materials consists of two distinct but interconnected systems: an ElectroEncephaloGraphic (EEG) system and a collaborative robot. There are two experimental protocols in place: one in which decision thresholds are identified and one in which decision thresholds are validated. In the first protocol, we only use the EEG system, whereas in the second protocol, we use both systems, EEG and cobot, which are made to communicate with one another.

2.1. EEG Sensor System

The used EEG system (Figure 1) [104] and consists of a band to which some electrodes are attached. The electrodes are connected to an acquisition board, and the band is worn around the head of a human subject. The acquisition board receives and partially processes the EEG signal before wirelessly transmitting it to a USB dongle connected to a PC.

The electrical scheme of the acquisition board is described in Appendix A.

In the first experimental protocol, the data are saved offline and later reprocessed using the following Matlab script, where the first lines on input file configuration are omitted.

... %data input filename = strcat(path, slash, file); fileID = fopen(filename); data=textscan(fileID,’%f%f%f%f%f%f%f%f%f%f%f%f%f% f%f%f%f%f%f%f%f%f%f%f%q’, ‘Delimiter’, ‘,’,’headerlines’, 5); fclose(fileID); sample_index = data{1}; k=1; for i = 2:9 eeg_data(:,k) = data{i}; %electrodes measures [microV] k = k+1; end reference = data{10}; %example parameters fs= 250; %[Hz] - sampling rate Channel = 8; %number of channels n = length(sample_index); time = data{25}; start_time = time(1); epoch_start = datetime(start_time); end_time = time(n); epoch_end = datetime(end_time); elapsed_time = epoch_end - epoch_start; elapsed_time = seconds(elapsed_time); %registration duration [s] t = linspace(0, elapsed_time, n); %bandpass filter Wp = [3 15]/(fs/2); %pass band Ws = [2 20]/(fs/2); %attenuation band Rp = 1; %[db] maximum bandwidth loss value Rs = 60; %[db] attenuation value [N, Wp] = ellipord(Wp,Ws,Rp,Rs); [B, A] = ellip(N,Rp,Rs,Wp); X = filtfilt(B,A,double(eeg_data)); %calculation of values at rest and during fright std_val = mean(m(fs*4:fs*10)); %resting average [V] between 4–10 s [fright_val, i]= max(m(fs*10:length(m))); %maximum after 10 s fright_time = i/fs + 10; threshold = 100; %fright threshold example if fright_val - std_val > threshold disp(‘frightened’) else disp(‘unfrightened’) end

Instead, in the second experimental protocol, data are processed inline by a ROS (Robot Operating System) application to develop decision strategies, and a command signal is sent to a cobot that interacts with the subject wearing the EEG system.

2.2. Cobot

Rethink Robotics’ Sawyer is the adopted cobot (Figure 2). It consists of a 7-degree-of-freedom robotic arm, a gripping system (tool), a screen (head), a vision system with two cameras (one on the end effector and one on the head), and a control system. The arm is outfitted with seven servomotors that allow movement in all seven degrees of freedom; it has a maximum extension of 1260 mm and a maximum payload of 4 kg. The entire system weighs 19 kg.

According to the Denavit and Hartenber notation, the robot can be represented by the parameters listed in

Table 1. Saywer robot structure described with DH parameters a, d, α, and θ sequentially from the base joint to the gripper joint.

α	θ	a	d
−π/2	θ1	8.1 × 10−2	0
π/2	θ2	0	1.91 × 10−1
−π/2	θ3	0	3.99 × 10−1
π/2	θ4	0	−1.683 × 10−1
−π/2	θ5	0	3.965 × 10−1
π/2	θ6	0	1.360 × 10−1
0	θ7	0	1.785 × 10−1

.

The robot communicates with a ROS node via a proper driver. The cobot transfers information to the ROS node via the driver from encoders that measure the angular positions at the joints, torque sensors at the joints, and cameras mounted on the cobot itself. The ROS node also sends commands to the cobot via the driver based on the required function, as well as by processing data from the cobot’s sensors or other sensors (EEG) connected to the PC. The ROS application is described in Appendix B.

2.3. Identification of the Decision Threshold

2.3.1. Participants

A total of 100 subjects were recruited ranging in age from 19 to 30 years. The subjects were divided into two groups, experimental group and control group, for an equal number of 50 subjects in each group. Subjects completed an informed consent form, as well as information and evidence of consent to the processing of personal data. Following that, subjects underwent a preliminary screening based on the following inclusion criteria:

-

No cardiovascular disease, which could pose a risk factor during the experiment;

-

The absence of neurological disorders that could change the intensity, shape, and latency time of the response signal;

-

Absence of abnormal eating habits, excessive sports activity, not having over-hydrated or exercised shortly before the experiment; particularly to limit changes in skin hydration;

-

No creams or other cosmetic or medicinal products applied to the skin in the area where the electrodes will be applied, as well as no long hair, to limit changes in the contact impedance between the skin and the electrodes;

-

No substance abuse that alters the psychophysical state or general hydration level (alcohol, drugs, systemic medications).

2.3.2. Experimental Protocol

The experimental observations were carried out in a robotics laboratory while all other electrical and electromechanical equipment were turned off. The season was winter and the indoor temperature was set at 21 °C with relative humidity in the 40–60% range. All cables and measuring equipment were shielded. The computer connected to the measurement system via Bluetooth was placed at the maximum possible distance so as to limit possible electrical interference.

EEG data were collected using active electrodes placed on the scalp at the location(s) of interest (

Table 2. Placement of electrodes on the subject according to Figure 3 and respective connections to the data acquisition board.

Electrode Code	Electrode Type	Board Pin	GUI Channel
FP1	Flat	N1P	1
FP2	Flat	N2P	2
FPZ	Flat	N3P	3
TP7	Spikey	N4P	4
TP8	Spikey	N5P	5
P7	Spikey	N6P	6
P8	Spikey	N7P	7
OZ	Spikey	N8P	8
A1	Ear clip	SRB	-
A2	Ear clip	BIAS	-

and Figure 3). The reference was the average ear value. The impedances of all electrodes were kept under 6 kΩ. The sampling rate of the EEG channels was 500 Hz, while the accelerometer sampling rate was 25 Hz.

Using the definitions in Table 2 and the positioning standard in Figure 3 according to the 10-10 system using Modified Combinatorial Nomenclature (MCN), the following order was used to come up with the steps for putting the electrodes on the patient.

• The Velcro strip was put on the person, and a mark was made on the Velcro strip to show the standard place to put the spikey electrodes;
• The band was removed from the subject, and then all the spikey electrodes were mounted in the correct position indicated in the previous step;
• The first lobe clip electrode was connected to the correct pin on the board (BIAS);
• The second lobe clip electrode was connected into the correct pin of the board (SRB);
• The three flat electrodes were connected to the three respective pins of the board (N1P, N2P, and N3P).
• The five spikey electrodes were connected into the five respective pins of the board (N4P, N5P, N6P, N7P and N8P);
• The Velcro band with the eight electrodes (three flat and five spikey) was placed on the subject;
• Finally, it was verified that the position of the electrodes was correct after the assembly had taken place.

Electrooculogram activity was not recorded because its effect on the EEG signal is very small compared to the hyperactivation associated with the fear phenomenon.

Face-muscular activity was not recorded because the subject was required to maintain a relaxed condition, and until the relaxed condition was achieved, data acquisition did not begin. To ensure this condition, the subject was asked to raise his or her hand slightly when he or she felt ready, and an operator who was able to observe his or her face would, subsequently, give a start signal with his or her hand when he or she observed the subject in the relaxed state. The operator observing the subject never checked the macro-movements of the subject’s face. Had macro-movements of the face been observed, the measurement would have been discarded and never repeated on the same subject.

The data of the subjects were then anonymized using a three-digit numerical code. To avoid bias in the analysis stages, the research team member who performed the screening and anonymization did not participate in the later stages of the study and was also kept anonymous to the other members of the research team, aside from the group leader.

The subject is made to sit in the data collection station, which includes a chair and a table to support the device. In order to obtain reliable results, the subject is placed away from any nearby electronic devices and facing away from the research group. A member of the research team ensures that the electrode band is properly placed on the patient to ensure that the sensors and the subject’s head are in constant contact. The test subject is instructed to remain calm and relaxed. After the sensors have been tested to make sure they work, the Open BCI software records the signal.

The EEG signal is recorded for approximately 20 s for each subject; after 10 s, a member of the research team induces a fear reaction in the subject through the percussion of a pair of metal objects in order to detect any changes in the EEG signal.

Following the experiment, a member of the research team asks the subject if he or she was scared in order to confirm the presence of the fear emotion.

2.4. Randomized-Controlled Trial (RCT) of Emphatic Collaboration

2.4.1. Participants

The sample size for the RCT described in this section could be estimated using the research protocol described in Section 2.3; but, as will be noted in results Section 3.1, the statistics do not require a large sample. The sample size is chosen to be 100 subjects conventionally to account for the difference between the two experiments; in particular, the subject in the protocol described here is exposed to the motion of a cobot while at rest, so his or her level of EEG activation may differ from that identified in Section 2.3. A total of 100 subjects, all of whom are different from those who participated in the previous experimental protocol and range in age from 19 to 30 years, are recruited. The subjects were divided into two groups, experimental A group and control group B, for an equal number of 50 subjects in each group. The subjects completed an informed consent form, as well as information and expressions of consent to the processing of personal data. Following that, subjects were subjected to the same screening described in Section 2.3.

The data of subjects are then anonymized using a three-digit numeric code. To avoid bias in the analysis stages, the member of the research team who performed the screening and anonymization does not participate in the subsequent stages of the study and is also kept anonymous to the other members of the research team, apart from the group leader. Subjects are put into one of two groups, A or B, based on a random draw. Group A is the intervention group, and Group B is the control group.

2.4.2. Experimental Protocol

The subject is made to sit in the data collection station, which includes a chair and table to support the device, as well as a collaborative robot in front of the subject. To ensure reliable results, the subject is placed away from any nearby electronic devices to avoid interference. A member of the research team ensures that the electrode band is correctly placed on the patient to ensure that sensors and the subject’s head are in constant contact. For safety reasons, a member of the research team stays on the subject’s right side by holding an emergency button and ensuring that the subject is always behind the edge of the table with his or her hands under the table. The subject being tested is instructed to remain calm and relaxed. After the correct sensor activity has been checked with the Open BCI software, the signal is then recorded.

If the subject is assigned to group A, the robot moves at a distance of 800 mm for the first 10 s, then moves rapidly toward the subject to a distance of 150 mm and a height of 400 mm from the table, and finally moves back slowly, closing on itself. The EEG signal is measured for a total of 10 s. This rapid movement is sufficient to generate a feeling of fear and still provide a sufficient level of safety for the subject. In fact, it is known in the literature that an intense acoustic stimulus coming from a source near a subject produces a stunning effect. The stunning effect is followed by autonomic reflexes such as increased heart rate and danger avoidance behaviors. It has been observed in these subjects that the autonomic reflex is closely related to the emotion of fear [105]. Therefore, it can be said that autonomic reflex and fear occur together in the subject.

If the subject is in group B, the robot moves for 20 s while staying 800 mm away from the subject. During the robot’s work session, the EEG signal is measured according to the procedure described in Section 2.3.

Following the experiment, a member of the research team asks the subject in both groups A and B whether he or she was scared to check the manifestation of the emotion of fear. This questionnaire serves to confirm the hypothesis in the literature.

3. Results

This section is divided into two parts: the first is for the experiment described in Section 2.3, in which the subject is subjected to rapid high-intensity sound stimulation; the second is for the experiment described in Section 2.4, in which the subject is subjected to rapid, close-range movement of a collaborative robot.

3.1. Results—Identification of the Decision Threshold

In accordance with the protocol outlined in Section 2.3, 100 subjects were screened for enrolment. As shown in Figure 4, five subjects were discarded: one had engaged in sports activity just before the test session, one had abused alcohol the night before, two had hair that was too long, and two had applied an oily based moisturizer to their face. Two more subjects were discarded after the experimental phase because the measurement was incomplete, most likely due to the detachment of one or more wires during the experiment. It was better not to repeat the experiment with the same subject so that the experiment would not be biased because he or she would already know what to do.

Table 3. Comparison between rest, peak, and peak to rest ratio distributions of the EEG signal; where m is the minimal observed value, M is the maximum observed value, µ is the mean value, σ is the standard deviation, p is the statistical significance for the comparison with the rest distribution.

Signal	m	M	µ	σ	p 1
rest	0.0043	7.9100	1.1796	1.6861	-
peak	53.58	5004.4	1386.95	1362.19	<0.000
peak/rest	69.06	127,785.2	11,142.21	25,189.99	<0.000

¹ p measures the difference with rest distribution and is computed with Mann–Whitney U test with independent samples. m, M, µ, and σ are expressed in mV for the rest and peak distributions; whereas they are non-dimensional for the peak/rest distribution.

summarizes the experiment’s results, emphasizing the significant statistical difference between the data composing the rest distribution and the data composing the peak distribution. From a statistical standpoint, there is a significant difference between the data that comprise the rest distribution and the data that comprise the peak/rest distribution.

The probability that the mean value of the rest distribution is equal to 53.58 (the minimal value of the peak distribution) is computed with the Mann–Whitney U test and produced a p value < 10⁻³. According to this result and to results in Table 1, the rest distribution is always under the minimum value measured for the peak distribution.

Figure 5 and Figure 6 show comparisons of the probability distributions in Table 3, where it can be seen qualitatively, among other things, that the probability distributions are not normal, so the choice of the statistical criterion for comparison, the Mann–Whitney U test, is easily justified.

Similarly, it is possible to measure how much the ratio of rest to peak exceeds a threshold value that is less than the minimum observed value obtaining the same result. This second analysis, although more accurate, requires a measurement of the rest value over a period of time before one can assess whether a peak is present.

The raw data (Figure 7) acquired from the eight EEG channels listed in Table 2 and depicted in Figure 3 illustrate an example of acquisition for a single individual. After band-pass filtering that preserves the fundamental signal content, the same obtained data are depicted in Figure 8 in the frequency domain and in Figure 9 in the time domain. In addition, Figure 10 shows the mean of the data received from the 8 EEG channels already mentioned, and this mean is utilized as the primary variable to identify the peak circumstances already discussed in this section and according to the technique given in Section 2.1.

3.2. Results—Randomized-Controlled Trial (RCT) of Emphatic Collaboration

A total of 100 subjects were identified in accordance with the protocol outlined in Section 2.4. As shown in Figure 11, two subjects were discarded: one had abused drugs the night before, and the other had applied moisturizer to his face. The remaining 98 subjects were divided into two groups of 49 each: experimental group A and control group B.

Table 4. Comparison between rest, peak, peak to rest ratio distributions in the experimental group A and the rest distribution in the control group B of the EEG signal; where m is the minimal observed value, M is the maximum observed value, µ is the mean value, σ is the standard deviation, p is the statistical significance for the comparison with the rest distribution.

Signal	m	M	µ	σ	p 1
rest—group A	0.1976	8.853	2.7174	2.3624	-
peak group A	49.03	3492.3	1578.7	961.44	<10−3
peak/rest A	33.54	7641.8	1253.0	1526.7	<10−3
rest—group B	0.5775	8.7525	4.732	2.459	0.287

¹ p measures the difference with rest distribution of the experimental group A and is computed with Mann–Whitney U test with independent samples. m, M, µ, and σ are expressed in mV for the rest and peak distributions; whereas they are non-dimensional for the peak/rest distribution.

summarizes the experiment’s results, highlighting a statistically significant difference between the data composing the rest distribution and the data composing the peak distribution in experimental group A (p < 10⁻³). In experimental group A, there is a statistically significant difference between the data composing the rest distribution and the data composing the peak/rest distribution (p < 10⁻³). Finally, there is no statistically significant difference (p = 0.287) between the data that comprise group A’s rest distribution and the data that comprise control group B’s rest distribution.

Figure 12, Figure 13 and Figure 14 show comparisons of the probability distributions in Table 4, where it can be seen qualitatively, among other things, that the probability distributions are not normal, so the choice of the statistical criterion for comparison, the Mann–Whitney U test, is easily justified.

To determine whether the subjects felt fear, they were given a questionnaire in which they were asked to assign a value from zero to ten to the amount of fear they felt during the experiment, with zero indicating no fear and ten indicating maximum fear.

Table 5. Comparison between the measured fear of the subjects in the experimental group A and in the control group B.

Signal	m	M	µ	σ	p 1
group A	6	10	8.5	1.4	-
group B	0	1	0.6	0.5	<10−3

¹ p measures the difference with rest distribution and is computed with Mann–Whitney U test with independent samples.

summarizes the findings, demonstrating that the experimental group’s level of fear is significantly higher (p < 10⁻³) than the control group’s level of fear.

Figure 15 shows a comparison of the probability distributions in Table 5.

4. Discussion

There are multiple articles in the literature on the recognition of emotional state by means of EEG instruments [106]. In particular, the emotion manifested in the observed subject can be elicited in several ways, the most popular being: visual-based elicitation using images, prepared task, audio-visual elicitation using short film video clips, audio-based elicitation using music, multiple techniques, imagination techniques/memory recall, social interactions. This paper presents two experiments. In the first experiment, emotion is elicited through an auditory stimulus. In the second experiment, there is a task that is performed to distract the subject, there is an interaction with a robot that is the source of the emotion elicitation. It should be noted that the interaction with the robot takes place in the visual field domain, although the phenomenon cannot be limited to a merely visual dimension, as the robot enters the subject’s peripersonal field. Therefore, compared to the literature, the second experiment has a rather hybrid elicitation technique, but certainly tending toward interactivity.

In the literature [107] there are several methods of feature classification and extraction, the main ones are: in the frequency domain, in the time domain, in the wavelet domain, based on statistical features, using support vector machine, adopting K-nearest neighbor, with linear discriminant analysis, using artificial neural network. The algorithm proposed in this paper, although it uses filtered data in the frequency domain, is based on analysis in the time domain. Probably the class that classification methods that comes closest to those proposed in the literature is the one based on statistical features, in that the discriminant is the maximum value of the mean of the EEG signals reached in the time frame of observation and synchronized with the source of artificial elicitation of emotion.

At present, there are several types of applications in the literature that can be attributed to a medical and a non-medical macro class. Medical classifications often follow the pathology of the subject or the stage of health treatment. Non-medical classifications are often divided into two large families: entertainment and safety. Certainly, the application proposed in this paper is non-medical and aimed at safety.

From the point of view of the hardware used, we distinguish the type of electrodes that influences measurement; although the type of electrode is a very random indication, because various factors influence the measurement, surely it is more relevant to state the contact impedance and in our case it is kept under 6 k. The number of channels ranges from 5 channels up to 256 channels and in our case it is 8 channels.

Some scientific works are based on standard emotion databases. In our case it was not possible to use these standard data because we are interested in a source of emotion elicitation based on interaction with the robot that is not represented in standard databases.

The number of emotions that are recognized ranges from 2 to 11. In ours, under an example hypothesis that will be expounded later, our model is based on two emotions: fear and pleasure. To expose this hypothesis clearly, we introduce Russell’s 2D emotion model, denoted by the expression:

$$arousa{l}^2+valenc{e}^2=1$$

In the two-emotion model, both can reach all values of arousal, that is, between −1 and +1, while from the perspective of valence, one emotion is considered positive (pleasure) and the other is considered negative (fear). Therefore, both fear and pleasure take values in the form between 0 and 1, but fear is only the name of valence when this is negative and pleasure is only the name of valence when this is negative.

Now come the working example hypotheses. The first hypothesis is that the maximum of fear is always greater than the maximum of pleasure. Since valence measures the intensity of emotion, this assumption is reasonable in the industrial work environment, where fear of losing one’s life in the face of a dangerous event (valence in modulus of 1) may manifest; but it is difficult, if not impossible, for such intense pleasure (valence in modulus of 1) to manifest. This also represents a limitation on the type of work, as there are rare cases of work that produces intense emotions: think of a performer on stage. The second limitation is related to the intense positive or negative emotions that may be though produced in a work environment due to non-work causes that may occur unexpectedly. This second case, although not contemplated still remains to be considered a condition of subsequent danger as it distracts the subject’s attention from his activity, so it is right for the cobot to react by interrupting the work activity and producing a safe situation.

There is another previous work [108], which presents the same hypothesis as ours, using pleasantness emotion in the case of odor-induced EEG signal measurement.

Finally, regarding the accuracy of emotion recognition [109], apart from the case where valence is measured indiscriminately by the sign taken, the literature presents various results ranging from 50% to 95%, although most works present accuracies between 75% and 85%. There are no papers that introduce classification accuracy since they implicitly introduce a strong limitation considering the measurement database as universal i.e., exactly statistically representative of the actual sample from which it is extracted. In this, our work introduces a methodological innovation by using probability distributions reconstructed from the data and thus being able to assess accuracy as well as precision. Moreover, the accuracy results obtained are greater than 99.99%.

As stated in Section 3.1, when a subject is relaxed, an abrupt sound signal of high intensity can be used to produce the occurrence of an artificially induced fear emotion in the subject. This can be accomplished simply by using EEG sensors and placing electrodes in areas of the encephalon that are not particularly relevant or precise. This property is thought to be due to the fact that primary emotions activate various brain regions [61] and that, in particular, fear is associated with high EEG signal intensities [110,111], making this phenomenon easily measurable, as was later confirmed experimentally.

Eye movement, as just mentioned in Section 2, produces a negligible artifact in relation to the observed phenomenon [112]. There is an accelerometer signal in the measurement system that is used by the board’s internal system for appropriate compensation (Figure A9). As previously stated in Section 2, the autonomic response to a sudden, intense sound from a source close to the subject is inextricably linked to the emotion of fear [105]. As a result, whether we measure one or the other, the important thing for us is that the cobot receives information that there is a danger, and this result was obtained. During the experimental phase, we realized that the phenomenon observed is so evident that excessive signal processing is not required to highlight the presence of the subject’s perception of danger. Artifacts are present in the signal, but their significance is meaningless. This observation enables us to accelerate signal processing and quickly send a reaction decision to the robot. To avoid danger, it is critical that the reaction be quick.

The results of the second experiment, conducted with a collaborative robot and described in Section 3.2, produced the same result. This second study was performed in an especially rigorous manner using an RCT (randomized controlled trial) scheme, preserving an extremely high level of statistical reliability. It can be seen, for example, that identifying a discrimination threshold between the relaxation and fear conditions using an EEG sensor is always statistically possible. It is important to note, however, a numerical detail that is unusual in comparison to the literature: the intensity of the peak signal associated with the occurrence of fear is extremely high, much higher than would be expected. In fact, some authors report that the presence of strong emotions, such as fear, is associated with EEG signal intensities greater than 100 µV [110], while others indicate EEG signal intensities greater than 200 µV [111]; in particular, the experiment by Kometer et al. [111] identifies an EEG signal intensity of 500 µV in subjects watching scary movies. This phenomenon is most likely associated with the presence of significant artifacts as a result of macroscopic and microscopic movements in response to the emotion of fear. As a result, the spike appears to be associated with the artifacts rather than the EEG signal itself. In this particular case, artifacts should not be discarded or deleted because they always occur concurrently with the emotion of fear, serving as an indicator of the discrimination threshold between normal and fearful conditions. It is possible for the artifacts to appear a few milliseconds after the fear emotion has occurred in the encephalon, but reaction times to a fearful event are extremely limited, so even while maintaining a high EEG threshold, the decision can be identified with a delay of a few tens of milliseconds from the occurrence of the emotion in the encephalon. As a result, a fear communication channel was established between the human subject and the robot.

After detecting the fear condition, the robot can react and move to a position that reduces the chances of colliding with the subject in the work cell. In the experiment described in Section 2.4 and the results of which are listed in Section 3.2, the collaborative robot closes in on itself, moving away from the area where the human subject is present (which is separated from the robot by a work table), reducing its overall moment of inertia and stopping, bringing its kinetic energy to zero. An alternative reaction would be to switch to a control that only compensates for static actions, so that any dynamic actions from the subject would allow the robot to move away quickly and easily, while any static actions (including the weight of the robot itself and any objects carried by the gripper) would be compensated for. If the human subject is aware of and recognizes the robot’s reaction to a dangerous situation, the human subject is also capable of receiving a signal equivalent to its fear emotion, this time emanating from the robot toward the human subject. In the future, it would be interesting to test this phenomenon experimentally, but for now, we can assume it as a reasonable working hypothesis.

As previously stated in Section 1, the phenomenon of fear transmission through the members of a subject society, even without knowing the exact cause of the danger but only by decoding the attitudes of the frightened subjects, is a property of societies that promotes the preservation of those societies and their members. Because it can send and receive fear, the EEG sensor-based empathic cobot becomes a part of society and helps to keep it and its members alive.

One might object that the subject interacting with the cobot is not always in a relaxed state and may experience emotions other than fear. There are different possible responses to these observations. In the case where the subject has standard conditions other than rest, as long as there are no sudden spikes in the EEG signal, i.e., no strong emotions, a decision threshold that is relative to the subject and determined by the ratio of standard signal to spike signal using only data from that subject can be used; that is, the cobot must be aware of the peculiarities of the subject interacting with it and, possibly, know how to identify him/her. In the case where the subject exhibits sudden spikes in the EEG signal that cannot be associated with the phenomenon of fear, it can be observed that the phenomenon of fear is due to a danger. Thus, identifying the EEG spikes as associated with a potential danger allows the effect of these potential dangers to be limited as much as possible. This approach may reduce the efficiency of the interaction, especially if there are too many interruptions that are not related to fear. In the latter case, it would be beneficial to introduce additional information via other sensors to differentiate between different emotions. The robot’s response time should be carefully monitored; it should not be increased excessively, as this would reduce the safety level of the interaction.

5. Limitations and Future Developments

5.1. General Considerations

This study proposes a new communication model between human and cobot that aims to safeguard the safety of the human.

A human and a cobot arm are connected in the previously illustrated manner, which can be summarized as follows: the human has a band of electrodes on his head that record electroencephalographic signals of his brain activity; the electrodes are connected via cable to a sensor; the sensor is connected via Bluetooth to a computer, which in turn is connected via cable to the cobot arm. By means of a simple AI programme in the computer, the cobotic arm is able to distinguish, among various electroencephalographic signals, a certain brain hyperactivity of the human, decode it as a fear signal, and stop and close in on itself upon its detection.

The reasoning that guided the study was as follows: where the EEG signal indicates a certain brain hyperactivity, this must be interpreted by the cobot as fear; if the human feels fear, then it means there is danger; if there is danger, the cobot must stop.

The study is valuable because it maintains the basic direction of cobotics research which, since its inception in the 1990s [1] to the present day, has always focused on the relationship between human safety and machine performance. In the first decade of the 2000s, a group of scholars from the University of Southern Denmark, coordinated by E. Østergaard [113], focused on the issue of safety and in 2008 realized the UR5 cobot, equipped with virtual barriers that are as effective as they are sustainable.

As valuable as it is, however, this study has its limitations, which should be worked on in order to develop its strengths and potential. Our analysis first turns to the three logical assumptions underlying the study:

• Fear in humans is an emotion capable of more intensely activating brain activity. This ‘cerebral hyperactivity’ enables fear to be distinguished from other emotions;
• Precisely because it is recognizable-distinguishable, fear is precisely detectable by the EEG;
• If the human feels fear, then there is a real danger that requires the cobot to stop.

Consider points 1 and 2, whereby fear causes a unique brain hyperactivity in the human, different from any other emotion, which makes it easily detectable by the EEG.

Some recent neuroscientific studies, coordinated by Alexander Shackman (University of Maryland) in collaboration with other universities [114], have shown that this is not the case at all, leading the scientific community to revise the neurobiological model by which the etiology of anxiety and that of fear were considered different and separate. Shackman showed that psychological-behavioral reactions to threats, whether concrete, possible, or imaginary, are actually controlled by common neural circuits. According to previous scientific literature, fear was controlled by the amygdala, which in the event of perceived (or supposed) danger causes the hormones associated with attack and flight to be released, and the circulatory, muscular, and intestinal systems to be activated. Anxiety, on the other hand, was found to depend on the nucleus of the terminal stria (BNST).

Studies coordinated by Shackman challenged this difference. One hundred volunteers were subjected to a painful shock associated with an unpleasant image and sound, with or without warning, and through functional magnetic resonance imaging, brain responses were observed to determine which areas were activated. Well, the experiment found that when subjects perceived a threat, both the neural circuits afferent to the amygdala and the nucleus of the stria terminalis were activated in them. The two structures gave statistically indistinguishable responses. The study was therefore able to conclude that there is a common neural basis controlling both anxiety and fear.

It is now clear that if the studies coordinated by Shackman are right, assumptions 1 and 2 lose their value: one cannot distinguish between fear and anxiety, and who knows whether this also applies to other primary emotions. The cobotic arm is therefore exposed to the misinterpretation of the electroencephalographic signal. A human subject who is particularly anxious, or even just prone to anxiety, would continually generate ambiguous signals, because his anxiety-driven brain activity would be neurobiologically identical to fear and the cobot would interpret it as fear, hence danger, and would continually stop. By not distinguishing anxiety from fear, the cobotic arm would be blocked continuously by false alarms. Result: ensuring the safety of the human would kill the performance of the machine.

Such a machine would not only increase production performance, it would actually damage it. It could in fact lead to the psychological alteration of the human, who would be irritated by the constant interruption of the machine for no reason. In the case of an anxious human, the blocking of the machine would feed the anxiety, and the constant occurrence of anxiety in the human—interpreted by the cobot as fear—would lead to further and continuous stoppage of the cobot. In an endless spiral, anxiety and blockage would feed each other in a directly proportional way. Human–cobot interaction would spiral into an endless loop.

In addition, it cannot be ruled out that other brain activities could also be interpreted by the machine as fear, since here the cobot does not use any special magnetic resonance imaging, but only a simple EEG. In that case, any hyperactivity could lead to an unmotivated blocking of the machine, resulting in performance inhibition and psychological alteration of the human.

First outcome of the analysis: the impossibility of distinguishing fear nullifies the fear/danger equation on which this work is based.

But beyond the hermeneutic difficulties, is the fear-danger equation really valid in an absolute sense? Even if the cobot’s interpretation were correct and the human was afraid, would that fear signify such danger that the cobot’s arrest was necessary? Here is assumption 3.

5.2. If the Human Feels Fear, It Is Because There Is Some Situation of Real Danger

Assuming that the machine has correctly registered the feeling of fear: the human subject is not anxious, not euphoric, but genuinely frightened. Fear could indicate real danger and therefore the stopping of the cobot would be correct. The conditional mode has been used because the reasoning here is based on the validity of another equation, taking it for granted whereas it is not: granted that the human is really afraid, does this fear of his certainly relate to a danger relative to the human–cobot interaction, which therefore requires the cobot to be stopped?

The study does not ask this question. Yet without the relation between danger and the cobot’s work, there is no need for the cobot to stop: if the danger does not concern what the cobot is doing with the human, but an external and unrelated situation, why should the cobot stop?

This second problem is undoubtedly inferior to the first, but it is not absent: the fear of the human is real and well read by the machine, but it is related to factors that are completely external and unrelated to the interaction with the cobot. The outbreak of a thunderstorm or the memory of an event or any other event, extrinsic to the human-machine collaborative relationship, might frighten the human: however, that fear, actually experienced and correctly registered by the machine, would not affect the human–cobot interactive activity at all and would not require the cobot to stop. But the cobot stops. It cannot do anything else, because it cannot distinguish between fear that involves danger and fear that does not involve danger. It is not the hermeneutics of fear that does the damage here, but the fear-danger equation, as it is understood in an uncritical, undifferentiated, and indeterminate way. Hence the risk of the cobot getting stuck unnecessarily, damaging the human’s performance and psychological state.

5.3. How to Solve the Problems?

One possible solution to the problems that emerged could be to enhance the cobot’s AI.

Owing to the increased AI, the cobot would be able to analyze its surroundings and obtain useful data to recognize the different nature of events. More importantly, it would be able to observe the human’s behavior and obtain valuable additional information on his/her nature. It could retrieve that information through natural language as well as through biometric facial, sound, and tactile recognition, as well as through temperature and heart rate detection and pupil observation. With this data, it could analyze human behavior, recognize its reactions, and distinguish its emotions.

Owing to AI, the cobot would self-learn from the relationship, could make autonomous decisions, and react to stimuli in a relevant and effective manner, also owing to facial and sound expressions. Its interaction with the human would be much more empathetic.

The human would then have to learn to decipher the cobot’s language correctly in order to interact with it. The interaction between human and cobot would be similar in many respects to that between human and social robot. The only difference would be the context: personal that of social robots, work that of cobots. Even at work, the interaction could be personalized: the cobot could learn the specific anthropological characteristics of the subject it works with and conform completely to them in manner and timing. The cobot would then be an ideal collaborator for the human, because in addition to safety, it would also guarantee the serenity of the working environment.

6. Conclusions

An EEG signal was used to connect a human agent and a cobot agent. This link allowed the human agent to transmit a fearful emotion to the cobot agent. Furthermore, the cobot’s reaction allowed a message of fear to be transferred from the cobot agent to the human agent. The social relationship between the two agents was strengthened by the transmission of fear. The experiments revealed an extremely high level of statistical reliability. The experiments were carried out in a controlled environment. In a more complicated setting, there may be strong limitations to how this technique can be used, necessitating the development of more complex strategies than those proposed in this paper.