Evaluation of Fatigue in Older Drivers Using a Multimodal Medical Sensor and Driving Simulator

Abstract

In recent years, the spread of wearable medical sensors has made it possible to easily measure biological signals such as pulse rate and body acceleration (BA), and from these biological signals, it is possible to evaluate the degree of biological stress and autonomic nervous activity in daily life. Accumulated fatigue due to all-day work and lack of sleep is thought to be a contributing factor to distracted driving, and technology to estimate fatigue from biological signals during driving is desired. In this study, we investigated fatigue evaluation during a driving simulator (DS) using biological information on seven older subjects. A DS experiment was performed in the morning and afternoon, and no significant differences were observed in the change over time of heart rate variability and skin temperature. On the other hand, in the afternoon DS, changes in arousal and body movements were observed based on BA calculated from the three-axis acceleration sensor and fingertip reaction time in a psychomotor vigilance test. It is suggested that by combining biological information, it may be possible to evaluate the degree of fatigue from the presence or absence of arousal and changes in body movements while driving.

1. Introduction

Older drivers may be at increased risk for traffic accidents, and one of the main factors is fatigue. Older adults generally sleep less well and are more prone to daytime drowsiness and fatigue. Driving under these conditions can lead to reduced driving ability and reaction time, posing a serious risk to road safety [1,2].

The objective of this study is to develop an effective method for detecting fatigue in older drivers. Specifically, we will examine which indicators have prognostic power by using a combination of conventional biometric autonomic estimation, the analysis of bio-accelerations, sitting pressure, subjective evaluation, and arousal level testing.

Previous studies have proposed many methods for detecting driver fatigue. Some studies have attempted to identify fatigue states by measuring physiological parameters (heart rate (HR), nystagmus, electromyography, etc.). Others use sensors and cameras in a vehicle to collect information about the driver’s behavior patterns and vehicle controls to help identify fatigue. However, the application of physiological parameters has significant noise processing problems, and facial image analysis of cameras in the vehicle cabin has human privacy issues and lighting environment limitations [3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18].

This study explores a simple index for detecting fatigue in the older through multimodal signal processing, including wearable sensors. This will contribute to human-centered design in the age of automated driving. Our aim is to evaluate the degree of fatigue during driving by the multimodal technology that combines multiple types of biological information using inexpensive and highly versatile medical sensors that can easily measure biological information. In this paper, in addition to HR and pulse wave measurement, we investigated fatigue evaluation for seven older subjects using a three-axis acceleration sensor, skin temperature (ST), seat pressure, reaction time of fingertips, and so on, during a driving simulator (DS).

This paper is organized as follows. Section 2 (Methods) describes the experimental methodology and data collection techniques. Section 3 (Results) reports the experimental and analytical results. Finally, we provide a discussion and conclusions.

2. Methods

2.1. Participants

In order to avoid bias in the subject group, the selection criteria for the subjects were that in addition to the absence of medication and chronic diseases, the subjects must be healthy older adults with driving experience and no traffic accidents or serious violations in the past year. In addition, the following 6 exclusion criteria were used to select subjects:

  i.

Cataracts that require treatment (excluding those after treatment).

 ii.

Color blindness.

iii.

Persistent atrial fibrillation and frequent premature contractions.

 iv.

Hospitalization history within 3 months.

  v.

Diabetes requiring insulin treatment.

 vi.

Debilitating diseases and acute diseases.

An announcement was made to recruit subjects, 7 older subjects applied, and all older subjects were accepted. A screening process was conducted for eligible subjects. This included medical evaluations, driving history checks, and schedule adjustments. 7 healthy older subjects (4 males) participated in this study. The mean age of the participants was 70 ± 3 y.o., and their mean driving history was 35 years. All subjects received an explanation from the experimenter in advance, and after being fully satisfied with the content, agreed to participate in the experiment.

2.2. Measurement Techniques

2.2.1. Measurement of Physiological Parameters

To assess the fatigue status of the participants, electrocardiogram (ECG), ST, and physical activity (body movement) were measured. ECG was measured by an ECG sensor attached to the chest, and ST activity was obtained from a skin conductivity sensor attached to the wrist during driving. Figure 1 shows the sensors used in the experiment and the situation during the DS.

ECG was measured using the Holter ECG with a built-in 3-axis accelerometer (Cardy303 Pico, Suzuken, Japan, Figure 1a). It has dimensions of 28 mm (W) × 42 mm (D) × 9 mm (H), a weight of 13 g, and a recording time of 24 h and uses a button-type lithium battery. The sampling frequency is 125 Hz, the sampling frequency of the acceleration is 31.25 Hz, and the 3-axis acceleration of −4.5 to 4.5 G can be measured. From the ECG signal, all R waves (sharp deflections corresponding to the ventricular electrical excitation) were detected, and R-R interval time series were obtained. If there was an arrhythmia, the R-R interval was deleted and resampled at 2 Hz using the step function. Time domain indices and frequency domain indices of heart rate variability (HRV) were calculated from the R-R interval time series. The mean HR [bpm] was calculated as an index of the time domain. As indices in the frequency domain, LFP (power of low-frequency component; 0.04~0.15 Hz, [ln, ms²]), HFP (power of high-frequency component; 0.15~0.45 Hz, [ln, ms²]), LFP/HFP (LFP to HFP ratio), and HFF (peak frequency of the HF component, Hz) were calculated. Indices in the frequency domain were calculated by FFT (fast Fourier transform). LFP reflects both sympathetic and parasympathetic activity and baroreceptor reflex sensitivity, and HFP reflects para-sympathetic activity. LFP/HFP reflects sympathetic nerve activity, and HFF reflects respiratory frequency [19,20].

Body acceleration (BA) was calculated from the 3-axis acceleration. The x-axis is recorded in the right-hand direction, the y-axis in the vertical upward direction, and the z-axis in the forward direction. The 3-axis acceleration was resampled to 10 Hz, and the square root of the sum of the squares of each axis was calculated to obtain the resultant acceleration. Furthermore, a high-pass filter was used to extract fluctuating components, and the sum of absolute value for the fluctuating components was defined as BA [G].

ST was measured with a wristwatch-type wearable sensor (Silmme22, TDK, Tokyo, Japan, Figure 1b,c) [21]. The sensor size is approximately 52 mm × 24.5 mm × 13.5 mm, and the weight is approximately 26 g. The following four types of sensors are inside the device: a pulse sensor, an acceleration sensor, a UV sensor, and a temperature sensor. In this experiment, it was used for the purpose of measuring ST. The temperature sensor uses a digital semiconductor temperature sensor to measure the ST of the wrist in the range of −10 °C to 45 °C. Measured values are output every minute.

2.2.2. DS Software

The DS software used was CarSim (Mechanical Simulation, Version 2022.0, Ann Arbor, MI, USA). The cockpit consisted of a car seat, steering wheel controller games for games, and an accelerator pedal. The course was designed as a public road, assuming a commuting route. The backgrounds were set in the morning and in the evening, assuming commuting time and return time. It was designed to follow the vehicle in front at a speed of 50 km/h or less. The driving time was 60 min. Figure 1d shows the situation during the experiment.

2.2.3. Measurement of Seating Pressure

The pressure and positioning of the sheet during the experiment were measured using a portable contact pressure measuring device Palm Q (CAPE CO., LTD., Kanagawa, Japan, Figure 1e) [22]. The body size of Palm Q is 65 mm (W) × 175 mm (L) × 35 mm (H), its weight is 160 g, and its power source is a 9 V square alkaline battery. The sensor pad size is 130 mm (W) × 130 mm (L), the pressure tube is about 75 cm long and weighs 50 g, and the materials are urethane, ABS resin, and polyacetal resin. The pressure measurement range is 0 mmHg–200 mmHg, and the measurement accuracy is ±3 mmHg. The sensor pad is placed at four points on the circumference at equal angles around one point, allowing you to check the position of the highest pressure.

2.2.4. Measurement of Fatigue Stress

The subjects’ fatigue stress before and after the experiment was evaluated using an Autonomic fatigue measurement device (MF100, Murata, Kyoto, Japan) [23]. The body size of MF100 is 100 mm (W) × 152 mm (L) × 67 mm (H), its weight is 110 g, and its power source is 2 AA alkaline batteries. MF100 uses optical and electrical measurement methods to measure the thumb pulse rate for approximately 90–120 s. Low-frequency components (0.04–0.15 Hz) and high-frequency components (0.15–0.40 Hz) are calculated from the pulse wave interval time series by frequency spectrum analysis. The area of the LF component is evaluated as sympathetic nerve activity, the area of the HF component as parasympathetic nerve activity, and the sum of the LF area and HF area as the overall autonomic nerve activity. Autonomic balance (ANB), deviation value (DV), total pulse rate (TPR, [beats]), and arrhythmia rate (AR, [beats]) is calculated from autonomic nerve activity. The higher the ANB value, the more dominant sympathetic nerve activity. The DV value indicates the degree of fatigue, and the standard value is 42. If the value is lower than 42, the degree of fatigue increases. The vital data measured by the MF100 is analyzed on a cloud server connected to the Internet, and the results are displayed on the iOS/Android app on a mobile device.

The psychomotor vigilance test (PVT) is widely used to measure sleepiness and arousal based on finger reactivity [24]. The PVT was conducted to examine arousal before and after driving, fingertip reaction time (RT, [ms]), and the number of times RT was 500 ms or more (ML: minor lap frequency). The subject clicked the mouse at the moment the elapsed time was suddenly displayed on the screen, and the elapsed time became the reaction time of the fingertip. The PVT used a 15.6-inch laptop (Panasonic, Windows11, Let’s note SV8). The distance from the subject’s eyes to the screen was approximately 70 cm, and the PVT font was 72 pt.

After the DS, as a subjective evaluation using a questionnaire, the participants answered the sickness questionnaire (SSQ) to estimate their degree of sickness. SSQ is the most commonly used subjective evaluation of motion sickness. This is a result of a factor analysis of 1119 pairs of Motion Sickness Questionnaire (MSQ) data measured before and after the simulator experience, from which 16 subjective items were extracted (e.g., eyestrain, blurred vision, dizziness, nausea, burping) that are more effective in evaluating simulator sickness [25]. These 16 items can be quantified into the following 4 items: nausea, oculomotor, disorientations, and total scores.

2.3. Experimental Protocol

The participants participated in an experiment in a DS where they drove a specific route. The experiment took place in the morning and in the afternoon, with each session lasting one hour. No specific task was given to the participants during the driving session, and they drove for one hour on a plain, flat road.

The experiment was performed at Tohoku University between 2 February 2023 and 12 February 2023. The subjects were prohibited from drinking alcohol starting the evening before the experiment. DS driving was performed for 1 h each in the morning and afternoon for each subject.

Table 1. Experimental protocol.

Events	MF100	PVT	DS	SSQ	MF100	PVT
Time [min]	2	5	60	2	2	5

The same experiment was performed in the morning and afternoon.

shows the experimental protocol. The morning experiment started between 7:30 and 8:00. First, the subjects performed the PVT and measurement of fatigue stress (MF100) before DS driving and then wore the measurement equipment. ECG, BA, ST, and seating pressure were measured during DS driving. Seating pressure (Palm Q, Cape Inc., Yokosuka, Japan) was measured once before and after starting DS driving. After DS driving, SSQ, PVT, and MF100 were measured, and the morning experiment ended. Afterward, the subjects were allowed some free time before performing the experiment in the afternoon. The afternoon experiment started between 16:30 and 17:00. The experimental protocol for the afternoon was similar to the morning. The morning DS background was the morning background, and the afternoon DS background was the evening background. The mean illuminance in the laboratory during DS driving was 601 lux in the morning and 21 lux in the afternoon.

2.4. Statistical Analysis

Statistical analysis was performed to determine if significant changes in biological information were observed between the morning and afternoon DS driving. The mean values of HRV indices during DS driving were compared between the morning and afternoon using a paired t-test. In addition, to examine the significance of changes in the HRV indices, ST, and BA over time, the mean values for each 10-minute period were analyzed using one-way ANOVA. Post hoc tests used Bonferroni multiple comparisons; PVT, Palm Q, and MF100 were compared before and after the DS driving operation and between the morning and afternoon using paired t-tests; and SSQ was compared between the morning and afternoon using paired t-tests. These statistical analyzes were performed for the older subjects (7 subjects). The statistical software used was IBM SPSS Statistics (version 28.0.1.0, Armonk, NY, USA). The significance level was set at 5%, p < 0.05 for significance, and p < 0.1 for trend.

3. Results

The SSQ, PVT, and standard fatigue detection test for the assessment of autonomic function were used to evaluate driver fatigue detection, and the results showed that drivers had higher finger reactivity and arousal in the afternoon than in the morning (

Table 2. Comparison of Palm Q, MF100, PVT, and SSQ.

Indices		Morning		Afternoon		p-Value
		Before	After	Before	After	AM (Before vs. After)	PM (Before vs. After)	Before (AM vs. PM)	After (AM vs. PM)
Palm Q	Seat pressure [mmHg]	79.3 ± 6.4	57.4 ± 8.2	92.7 ± 16.4	73.4 ± 10.1	0.067	0.444	0.390	0.147
MF100	ANB	0.99 ± 0.27	0.75 ± 0.15	0.90 ± 0.24	0.99 ± 0.16	0.399	0.760	0.794	0.294
	DV	46.7 ± 8.5	49.6 ± 15.9	43.6 ± 7.2	36.9 ± 3.0	0.738	0.230	0.498	0.364
	TPR [beats]	125 ± 7	117 ± 4	124 ± 6	119 ± 5	0.136	0.562	0.630	0.729
	AR [beats]	8 ± 7	5 ± 3	6 ± 6	2 ± 2	0.408	0.411	0.105	0.243
PVT	RT [ms]	283 ± 8	286 ± 5	274 ± 10	280 ± 4	0.804	0.539	0.052	0.241
	ML [frequency]	2.1 ± 0.6	0.9 ± 0.3	0.9 ± 0.5	0.6 ± 0.3	0.122	0.654	0.049	0.604
SSQ	Nausea	—	19.1 ± 8.8	—	17.7 ± 10.5	—			0.846
	Oculomotor		40.1 ± 11.4		31.4 ± 12.8				0.311
	Disorientation		31.8 ± 16.8		31.8 ± 19.4				1
	Total scores		26.2 ± 9.5		23.0 ± 11.4				0.482

). In addition, physiological indices such as ECG and bio-acceleration were analyzed in this experiment, and the results are shown in the table below (Figure 2, Figure 3 and Figure 4).

The evaluation of the effectiveness of the detection model showed that changes in BA could be detected during afternoon DS driving when arousal is high. Additionally, changes in seat pressure were observed before and after morning DS driving. The position of seat pressure varied depending on the subject, and some subjects’ body movements transitioned while others did not.

Figure 2 shows the mean and standard error of the HRV indices during morning and afternoon DS driving. There was no significant difference in HRV indices. Figure 3 shows the change in HRV indices over time during morning and afternoon DS driving. Figure 4 shows the change in ST and BA over time during morning and afternoon DS driving. No significant changes were observed in SK for either morning or afternoon DS driving. BA from 50 to 60 min was significantly increased compared with 30 to 40 min during afternoon DS driving (p = 0.036) based on Bonferroni multiple comparisons. Table 2 shows the mean values and standard errors of Palm Q, MF100, PVT, and SSQ. There was a tendency for seat pressure to decrease after DS driving in the morning compared with before DS driving in the morning (p = 0.067). In the PVT, RT tended to decrease (p = 0.052) and ML decreased significantly (p = 0.049) before the start of DS driving in the afternoon compared with before the start of DS driving in the morning. There were no significant differences in the MF100 and SSQ indicators before and after DS driving or between the morning and afternoon.

Table 3. Change in position of the highest seat pressure before and after DS driving.

Subjects	Sex	Morning		Afternoon
		Before	After	Before	After
1	male	right	front	right	right
2	male	front	left	front	left
3	male	back	back	left	left
4	male	left	left	front	front
5	female	right	right	front	left
6	female	front	left	front	front
7	female	left	left	left	left

shows the change in the position of the highest seat pressure before and after DS driving. Regardless of whether it was the morning or afternoon, there were subjects whose positions changed (five cases) and subjects whose positions did not change (nine cases). Among the subjects whose posture changed, four cases showed a tendency to sit on the left side after DS driving.

4. Discussion

This paper presents a novel approach to fatigue detection in older drivers based on a subjective evaluation and multimodal analysis of biological signals. The novelty of this research lies in the fact that it focuses on characteristics unique to older drivers that have been overlooked by conventional fatigue detection systems. Because older people are generally prone to arrhythmias, conventional physiological assessment of autonomic nervous system analysis using biometric signal analysis had limitations [26,27,28]. In this study, we demonstrated the possibility of detecting fatigue continuously from transitions in seat pressure and BA by taking into account the onset of fatigue during long drives, which is unique to older people. This has the potential to significantly reduce the risk of traffic accidents for older drivers. Such fatigue technology could be incorporated into vehicle systems and smartphones. In particular, the possibility of connecting to smartwatches, smart bands, and other wearable devices was considered. Not only would this facilitate more convenient monitoring of fatigue conditions for drivers, but it would also provide a means of continuous monitoring without compromising data quality.

The monitoring of seat pressure is non-invasive, and the measurement of BA also places a low measurement burden on the subject. Further exploration of the integration of these research findings with everyday technology could benefit from more integrated real-life applications. Revisiting the possibilities of connecting to smart devices could increase the applicability of our solution to a wider audience and facilitate its integration into everyday life.

However, this study has a number of limitations. Focusing only on older participants and the relatively small sample size are notable limitations. Future studies should seek to address these limitations by conducting larger studies that include a broader range of participants and test the applicability of the study results across different age groups and categories of drivers.

In this experiment, which evaluated fatigue during DS driving in the morning and afternoon, no significant difference was observed in HRV analyzed from ECG between the morning and afternoon, and no significant difference was observed in the changes over time during DS driving. Furthermore, no significant difference was observed in ST over time, but ST was clearly higher in the afternoon than in the morning. This may be due to the regulation of body temperature by human circadian rhythms [29]; the interrelationship between HRV and circadian rhythms is not clear due to individual differences, and HRV increases or decreases depending on lifestyle and activity status [30]. Furthermore, HRV decreases with age [31], so HRV in an older adult is unlikely to change under the burden of DS driving. In this experiment, SSQ was investigated to determine if simulator sickness occurred in addition to fatigue caused by DS driving. The results showed no significant difference between the morning and afternoon. However, interpretation of the score values is difficult [32]. Regarding fatigue indices, there were no significant differences before and after DS in the morning and afternoon. The fatigue index measured by MF100 uses pulse wave variability to observe the balance of autonomic nerve activity and the rate of arrhythmia. Pulse wave variability is affected by HRV and aging, so it is possible that no significant changes were observed before and after DS. However, just because there are no significant changes in HRV or pulse wave variability in older people does not mean that they are not fatigued. It is also thought that a significant difference is difficult to find due to the decrease in HRV associated with aging. Furthermore, the incidence of arrhythmia changes depending on the time of day, so it cannot be determined whether this is because of the effects of DS [33]. Comparing RT and ML with the PVT before DS driving, reactivity and arousal improved more in the afternoon than in the morning. Thus, it is suggested that in the older group, the experiment was conducted in the afternoon when finger reactivity and arousal were higher.

BA (body activity) during DS driving changed significantly in the afternoon; DS driving time was 60 min, and a gradual increase in body activity was observed, especially in the second half of the driving time. This phenomenon is thought to be due to unconscious body movement to prevent loss of concentration after 30-60 min of driving, and the increase in body movement during the latter half of the DS in the older group is thought to have increased their arousal level.

Regarding seat pressure, in the morning, it was lower after DS driving than before DS driving. In the afternoon, no significant difference was observed before and after DS driving. In the morning and afternoon, considering the position of the measured seat pressure, there were five cases in which the seat position changed during DS driving, and four of these cases tilted to the left after DS driving. Among the nine cases in which the seat pressure position did not change, there were four cases in which the seat pressure tilted to the left after DS driving, and a total of eight cases, or approximately 57% of all cases, tilted to the left after DS driving. Due to experimental constraints, these seat pressure measurements were taken once before the start of DS driving and once after the end of DS driving, so changes in seat pressure during DS driving are unknown. However, based on these results, seat pressure is thought to be related to individual differences in sitting style and daily driving habits, as well as the effects of fatigue and arousal, so a more detailed study is required.

Studies on seating pressure during driving have investigated factors such as driver comfort, seating pressure, seat shape, and cushioning material during long-distance driving. Pascaline L et al. (2021) evaluated the effects of driver seating behavior during long-distance driving by measuring seat pressure in three different seats and found that an increase in general discomfort with driving time was similarly observed in the three seats [34]; Mathieu L et al. (2020) used surface electromyography to assess neuromuscular fatigue and discomfort during long-distance driving [35]. Soft and hard seats have been shown to exhibit different neuromuscular fatigue profiles, with soft seats inducing greater activation of lumbar back muscles and hard seats increasing lumbar back support. Sooho C et al. (2021) reported that self-shaping cushions improve sitting comfort. They reported that by adjusting the shape of the cushion to the driver’s body shape, the pressure distribution becomes more uniform, reducing pressure concentrated on the sitting bones and improving sitting comfort and safety [36].

Based on the above, older subjects in this DS driving experiment had higher arousal in the afternoon than in the morning, and their BA increased in the latter half of DS driving. No significant differences were observed overall in HRV or ST over time during DS driving, and no significant differences were observed in fatigue indicators based on pulse wave fluctuations. It is difficult to estimate driver fatigue with these indices. The seat pressure measured in this study was not in a time series, so it was not possible to confirm changes in seat pressure or seat pressure position over time. Previous research has revealed the possibility of fatigue evaluation using seat pressure. It has been suggested that by combining BA and PVT in addition to a detailed analysis of sitting pressure, a detailed fatigue evaluation based on the balance of arousal may be possible.

No significant differences were found in HRV or ST changes calculated from ECG over time, but the results suggest that sitting pressure and body movement have some power to detect fatigue even in such cases. Body motion and BA, in particular, have already been recognized for their usefulness in biometric measurements in various fields. Seat pressure sensors and body motion sensors play an important role in health management and lifestyle monitoring, as well as in the field of care for the older and disabled.

Sitting pressure and body movement sensors monitor sitting and movement patterns in daily life, which can be useful for health management and lifestyle improvement. This makes it possible to identify health risks, such as a lack of exercise and prolonged sitting habits, and to take preventive measures. We look forward to the upcoming possibilities in conventional medical diagnosis and treatment support as well, as it will be useful in diagnosing specific diseases and disorders and monitoring the effects of treatment.

5. Conclusions

In conclusion, this study investigated the feasibility and effectiveness of a system to detect fatigue in older drivers through multimodal analysis. By focusing on characteristics specific to this population, such as BA and seat pressure, as well as driving behavior, we were able to identify indicators that can detect to some extent the state of driving fatigue in older drivers, which has been difficult in the past, such as in subjects with frequent irregular heartbeats. Our findings underscore the importance of considering age-specific factors. As the population ages and the number of older drivers increases, addressing the specific needs of this population will be essential to ensure road safety.