Round 1

Reviewer 1 Report

Review on

Detection of drivers’ anxiety invoked by driving 2 situations using multimodal bio-signals

Authors have established an experiment on investigating driver’s stress through biosignals (EEG, PPG, EDA, PS) analysis.

Abstract

please modify bio-signals->biosignals  throughout the whole manuscript

while their bio-signals were measured -> while their biosignals were recorded

Introduction

In general, related work should be added in the introduction describing the state of the art  for the problem under investigation.

Some references should be added to support that biosignals are affective in detecting stress states,

I provide you some recent reviews on the topic

Alberdi, A. Aztiria, and A. Basarab, "Towards an automatic early stress recognition system for office environments based on multimodal measurements: A review," Journal of biomedical informatics, vol. 59, pp. 49-75, 2016.

Giannakakis, D. Grigoriadis, K. Giannakaki, O. Simantiraki, A. Roniotis, and M. Tsiknakis., "Review on psychological stress detection using biosignals.," IEEE Transactions on Affective Computing, 2019.

Also another interesting publication regarding estimation of a driver’s stress is

J.A. Healey, R.W. Picard, Detecting stress during real-world driving tasks using physiological sensors, IEEE Transactions on Intelligent Transportation Systems, 6  156-166 2005.

Also some references regarding detection of stress from EEG features, PPG features, EDA features should be added.

2. Materials and Methods

(15 Female -> 15 females, 16 males

This study was carried out in accordance with the recommendations of Institutional Review Board of the Ulsan National Institute of Science and Technology (UNISTIRB-18-45-C)

Please provide reference

Hazard Perception Test provided by England Driver and Vehicle Standard Agency

Please provide reference

We collected thirty 30-s driver perspective video clips from Youtube, which contained one of the three anxiety-invoking events above. Each video clip included one anxiety-invoking event (video of anxiety: VA).

Please provide the videos used how the anxiety invoking event is ensured

2.5.1

with a 1-s window and 50% overlapping

Please consider increase the window length to 2sec in order to have more reliable estimates.

using Least Absolute Shirinkage and Selection 128 Operator (LASSO) regression analysis. Please provide reference

PPG please consider to increase the window length at least 30 sec. You can use overlapping windows to maintain your analysis. You can see details about minimum window lengths in this article

Hejjel, E. Roth, What is the adequate sampling interval of the ECG signal for heart rate variability analysis in the time domain?, PHYSIOLOGICAL MEASUREMENT, 2004

Please consider using also EEG features related to stress such as frontal asymmetry, β/α ratio, BLI except extracting only EEG rhythms

Giannakakis, D. Grigoriadis, and M. Tsiknakis. Detection of stress/anxiety state from EEG features during films watching. In Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology, Milano, Italy, 25-29 August 2015.    

2.6

a cumulative feature count (CFC), please provide reference and explain method.

It is not clear how many features are retained after the CFC application, please clarify.

3.2.

Please provide also a standard 10-fold cross validation analysis with selected features (and not one feature for each signal*) in order to compare the performance measures with the proposed methodology.

*Please clarify this as said in section 2.6.

Please provide also 10-fold cross-validation results instead of leave one out which is a very optimistic approach.

The achieved accuracy is comparable with other relevant studies results in the literature. Please, refer other accuracies and discuss.   

General comments – Overview

The paper address an interesting research topic. I would recommend authors to address all the comments referred.

Author Response

We would appreciate your valuable comments. Below are responses to each point.

Abstract

please modify bio-signals->biosignals throughout the whole manuscript

while their bio-signals were measured -> while their biosignals were recorded

According to the comment, we modified bio-signals to biosignals throughout the manuscript.

Introduction

In general, related work should be added in the introduction describing the state of the art for the problem under investigation. Some references should be added to support that biosignals are affective in detecting stress states, I provide you some recent reviews on the topic

Alberdi, A. Aztiria, and A. Basarab, "Towards an automatic early stress recognition system for office environments based on multimodal measurements: A review," Journal of biomedical informatics, vol. 59, pp. 49-75, 2016.

Giannakakis, D. Grigoriadis, K. Giannakaki, O. Simantiraki, A. Roniotis, and M. Tsiknakis., "Review on psychological stress detection using biosignals.," IEEE Transactions on Affective Computing, 2019.

Also another interesting publication regarding estimation of a driver’s stress is

J.A. Healey, R.W. Picard, Detecting stress during real-world driving tasks using physiological sensors, IEEE Transactions on Intelligent Transportation Systems, 6  156-166 2005.

Also some references regarding detection of stress from EEG features, PPG features, EDA features should be added.

As suggested, we added new references about stress detection using biosignals to the revised manuscript (Line 36 and Line 32-34.).

2. Materials and Methods

(15 Female -> 15 females, 16 males

As suggested, we added the number of male participants to the revised manuscript (Line 66).

This study was carried out in accordance with the recommendations of Institutional Review Board of the Ulsan National Institute of Science and Technology (UNISTIRB-18-45-C)

Please provide reference

Hazard Perception Test provided by England Driver and Vehicle Standard Agency

Please provide reference

using Least Absolute Shirinkage and Selection Operator (LASSO) regression analysis.

Please provide reference

The requested references for the methods were added in Line 76 (about hazard perception test) and Line 140-141 (about LASSO) of the revised manuscript.

However, we could not find other ways of citing a reference for IRB except the document registration number, which was already provided in the manuscript (UNISTIRB-18-45-C).

We collected thirty 30-s driver perspective video clips from Youtube, which contained one of the three anxiety-invoking events above. Each video clip included one anxiety-invoking event (video of anxiety: VA).

Please provide the videos used how the anxiety invoking event is ensured

We would like to add videos along with the revised manuscript, but we could not make it because of the MDPI system. We asked to the Journal editorial office and received the reply that the size of video files is too big for their e-mail system so that they could not pass them on the reviewers. The Journal editorial office recommended us to attach the available download link to the response to the reviewer’s comment. Here is a link:

https://drive.google.com/file/d/1AZGDvsV0h15HvmWKiLM6vt-_A8BnLQWk/view?usp=drive_web

2.5.1

with a 1-s window and 50% overlapping

Please consider increase the window length to 2sec in order to have more reliable estimates.

As suggested, we increased the window size for EEG features from 1-s to 2-s window with 0.5-s non-overlapping and applied it to the decoding analysis (Line 141-142). However, increasing the time window length did not improve estimation accuracy (Table B2, B3, Line 229-235).

PPG please consider to increase the window length at least 30 sec. You can use overlapping windows to maintain your analysis. You can see details about minimum window lengths in this article

Hejjel, E. Roth, What is the adequate sampling interval of the ECG signal for heart rate variability analysis in the time domain?, PHYSIOLOGICAL MEASUREMENT, 2004

We appreciate this valuable comment about PPG signal processing. Unfortunately, we could not apply it to our study. In the study of Hejjel and Roth, the 30-s window of PPG signals was necessary because the main feature focused on heart rate variability (HRV). On the other hand, we focused on the change of PPG before and after the anxiety onset during 30-s driving videos. For this reason, we did not consider HRV features but rather statistical features such as mean, minimum and maximum amplitudes of PPG signals. These statistical features have been also used in a similar study (Alberdi et al., 2016).

Please consider using also EEG features related to stress such as frontal asymmetry, β/α ratio, BLI except extracting only EEG rhythms

Giannakakis, D. Grigoriadis, and M. Tsiknakis. Detection of stress/anxiety state from EEG features during films watching. In Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology, Milano, Italy, 25-29 August 2015.   

As suggested, we added new EEG features related to stress and applied to the further analysis (Line 135-137).

2.6

a cumulative feature count (CFC), please provide reference and explain method.

CFC was developed in this study (Line 196), so there is no reference for it. Instead, we explain this method in more details in the revised manuscript (Line 204-205).

It is not clear how many features are retained after the CFC application, please clarify.

CFC was developed not for selecting features but for ranking weights assigned to each feature in logistic regression after the features had been selected. Therefore, The number of features was not change after CFC application The high ranked features from the CFC analysis represented that those features were highly involved in classification. We modified the corresponding text to elaborate CFC more (Line 204-205).

3.2.

Please provide also a standard 10-fold cross validation analysis with selected features (and not one feature for each signal*) in order to compare the performance measures with the proposed methodology.

*Please clarify this as said in section 2.6.

As suggested, we added a 10-fold cross validation analysis and clarified in Line 188-190 in section 2.6.

Please provide also 10-fold cross-validation results instead of leave one out which is a very optimistic approach.

We appreciate your comment. In the revised manuscript, we decide to show both results of LOTO and 10-fold CV, instead of replacing LOTO with 10-fold CV, because LOTO showed higher accuracy than 10-fold CV (Table B2-B3, Line 226-235).

The achieved accuracy is comparable with other relevant studies results in the literature. Please, refer other accuracies and discuss.

According to this valuable comment, we compared accuracy of this study with other relevant studies in Line 317-321. Overall, estimation accuracy achieved in this study is lower than those in other studies: 77% vs {82%, 82.03%, 89.70%, 100%, 77.95%}. However, other studies demonstrated the estimation of drivers’ states different from anxiety, such as stress or specific emotions (happy and angry), where they discriminated these emotional states from a normal state. In contrast, our study estimated changes in anxiety derived by sudden events in driving situation. This comparison is newly added to Discussion section (Line 321-325)

Reviewer 2 Report

My main concern is  the selection of features of EEG described in section 3.3. It looks like the set of selected features is specific for each individual. Thus, the decoding model (LR) would be specific for each individual. This makes questionable the significance of the result. The authors should consider a different approach.

From data in table B1, it looks like for specific individuals the multimodal approach does not improve the accuracy over the presented by EEG only. This has not been discussed in the paper.

Errors in presentation:

Figure 7: Why  some subjects (2,4,5,6,8...) are missing from this bar plot? Please either provide an explication, or present all the subjects in the plot.

Table 3: The caption of table 3 does not correspond with the data presented. It looks like the caption is the same of table 2.

Author Response

My main concern is the selection of features of EEG described in section 3.3. It LOTOks like the set of selected features is specific for each individual. Thus, the decoding model (LR) would be specific for each individual. This makes questionable the significance of the result. The authors should consider a different approach.

We appreciate this valuable comment on the issue of individualized feature sets. In this study, we observed that an optimal combination of multiple biosignals varied across individuals. As such, it was difficult to draw a set of common features for anxiety detection across individuals. Nonetheless, we attempted to extract a common feature set from all the participants and examine decoding performance using it. But, decoding performance was only close to a chance level. This might be because the features varied across individuals as expected. We added this additional examination of using a common feature in Discussion (Line 315-317).

From data in table B1, it LOTOks like for specific individuals the multimodal approach does not improve the accuracy over the presented by EEG only. This has not been discussed in the paper.

As you mentioned, some participants (7 out of 23) did not show improved accuracy for the multimodal approach compared to single modal one (i.e. EEG). We added the need for future work of individual difference for improved accuracy using multimodal signals in Line 318-319.

Errors in presentation:

Figure 7: Why some subjects (2,4,5,6,8...) are missing from this bar plot? Please either provide an explication, or present all the subjects in the plot.

As suggested, we added the reason for missing participants in Line 265-267 of the manuscript.

Table 3: The caption of table 3 does not correspond with the data presented. It LOTOks like the caption is the same of table 2.

We appreciate this comment and change the caption of table 3 as ‘The performance comparison between feature sets’.

Reviewer 3 Report

The research explain some experiments to detect the anxiety situation in drivers. The paper explian very well the data collection and the obtained results. I suggest the authors some point to try to increase the quality of their contribution:
- I suggest to introduce a sumary of the whole article at the end of the introduction.
- I think that it is better to clarify the 30-seconds video in line 67 (as appear in line 69).
- The authors talk about the logistic regression that they applied to the bio-signals; but I think they could try to use any soft-computing algorithm like Artificial Neural Networks, only to check if with these technique, the results are improved.
- I miss a future work section, I think that this research could be continue with different type of drivers (older, with more driving time...).
- I don't know if the authors try to use some clustering technique before the logistic regression was applied. In some previous research I found that this type of hybrid technique (with clustering), increase the performance of the prediction.

Author Response

The research explained some experiments to detect the anxiety situation in drivers. The paper explained very well the data collection and the obtained results. I suggest the authors some point to try to increase the quality of their contribution:

- I suggest to introduce a summary of the whole article at the end of the introduction.

We appreciate this valuable comment and add a summary in the last paragraph of the introduction accordingly (Line 57-62).

- I think that it is better to clarify the 30-seconds video in line 67 (as appear in line 69).

As we understood from your comment, a more concrete explanation is needed for videos. Thus, we would like to add videos along with the revised manuscript, but we could not make it because of the MDPI system. We asked to the Journal editorial office and received the reply that the size of video files is too big for their e-mail system so that they could not pass them on the reviewers. The Journal editorial office recommended us to attach the available download link to the response to the reviewer’s comment. Here is a link:

https://drive.google.com/file/d/1AZGDvsV0h15HvmWKiLM6vt-_A8BnLQWk/view?usp=drive_web

- The authors talk about the logistic regression that they applied to the bio-signals; but I think they could try to use any soft-computing algorithm like Artificial Neural Networks, only to check if with these techniques, the results are improved.

According to the comment, we additionally applied an ANN to the data to see if it could improve decoding accuracy. However, unfortunately, ANN did not help improve. Nonetheless, this new analysis with ANNs was added to Tables B2, B3 and Line 226-235 in the revised manuscript.

- I miss a future work section, I think that this research could be continue with different type of drivers (older, with more driving time...).

We appreciate this valuable comment and add a future work part at the end of the discussion (Line 329-331).

- I don't know if the authors try to use some clustering technique before the logistic regression was applied. In some previous research I found that this type of hybrid technique (with clustering), increase the performance of the prediction.

We appreciate this very interesting suggestion. However, our original aim was to find which combination of multimodal biosignals would be effective to detect anxiety. So, if we applying a hybrid technique (e.g. clustering, PCA, etc.) to biosignals before logistic regression, it may make it difficult to tell which combinations are more effective. Thus, in this study, we decided not to apply a hybrid technique to multimodal biosignals. However, this idea is still promising so we add this approach as future work in Discussion (Line 331-334).

Round 2

Reviewer 1 Report

Authors addressed most issues accordingly. 

There are some minor issues to be fixed  

The requested references for the methods were added in Line 76 (about hazard perception test)

I think the link for the test is https://hazardperceptiontest.net and not the https://hazardperceptiontest.net/hazard-perception-tips/ provided

and Line 140-141 (about LASSO) of the revised manuscript.

Please provide a reference for LASSO methodology and not just the MATLAB function.

Author Response

We appreciate these valuable comments. As suggested, we revised the link for the test (Line 76):

The old link,

https://hazardperceptiontest.net/hazard-perception-tips/

Is changed to a new link,

https://hazardperceptiontest.net.

We also added a reference for LASSO (Line 140) as:

Tibshirani, R., Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society. Series B (Methodological) 1996, 58 (1), 267-288

Reviewer 2 Report

Previous comments has been addressed satisfactorily.

Author Response

We think the previous comments truly help improve the manuscript and appreciate all those valuable comments again.

Reviewer 3 Report

After reviewed the new version of the paper, I think that the content improves its quality.

Author Response

We think the previous comments truly help improve the manuscript and appreciate all those valuable comments again.