Vehicle-to-Cyclist Collision Prediction Models by Applying Machine Learning Techniques to Virtual Reality Bicycle Simulator Data

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This study uses virtual reality bicycle simulator data to develop vehicle-to-cyclist collision prediction models based on machine learning techniques. Overall, the paper is well-written. Pls see my comments as follows.

1. While the authors have summarized the relevant studies, their limitations are unknown. This leas to that the contributions of this study are not clear.

2. The interaction between the vehicle and the bicycle is an interesting topic. How to ensure the accuracy of VR simulation? Especially for the vehicle.

3. The subtitles for sections 3.5 and 3.6 are the same. Pls check.

4. The used supervised Machine learning algorithms should be introduced in details in the methodology section. Further, a separate section regarding the used methods are necessary.

5. Practical applications of the proposed vehicle-cycle collision prediction models are suggested to be discussed.

Author Response

Dear,

Thank you for your insightful comments, which we have carefully considered. We have incorporated the following changes in response to your suggestions, presented in the order you provided. All changes have been conducted using the MS Word change control tool:

1. The authors have highlighted in the text the contributions of the research conducted in the manuscript, providing papers related to the use of Machine Learning and Deep Learning techniques in road safety and implementation in vehicle technology. In particular, the models described in this paper are based on supervised classifiers with low computational cost due to the processing of kinematic variables, unlike those used in vehicle implementations addressed in the scientific literature (based on convolutional networks and feature extractors). Also, most of the supervised models proposed in the paper have been developed in macroscopic studies of collision risk assessment and injury severity. In our case, more emphasis is placed on the vulnerable user, pointing out the critical actions in the decision-making process that can lead to collision in different types of collision scenarios. For this purpose, a cutting-edge technology (Virtual Reality) has been used, with a calibrated Virtual Reality simulator that guarantees a high level of immersion and realism.

2. The authors have included an analysis of the level of immersion of the participants in the VR simulator, based on the results obtained in the System Usability Scale (SUS) questionnaire and the Presence Questionnaire (PQ). 80% of users rated the application as acceptable (maximum values, 71-100), and 20% as marginal (51-70). Likewise, the PQ questionnaire shows a high mean score for all participants, indicating that there is a high level of immersion while interacting with the 3D environment
3. The corresponding subtitles have been corrected.
4. The authors have included a Materials and Methods subsection (following the Methodology workflow) dedicated to explaining the supervised Machine Learning classification models addressed in this article, as well as a description of their development phases (explained in detail in the next section).
5. The authors have highlighted in the State of the Art section and in the Conclusion the main applications at the vehicle level that the proposed models would have. In particular, it would follow the line of previous work on the development of commercial AEB systems (https://doi.org/10.3390/app122211364, https://doi.org/10.3390/vehicles5040084). The objective is for this model to regulate the emergency braking pressure in the AEB system, applying full pressure when a collision is predicted and partial braking if the cyclist's action would avoid the accident. Promising results in our previous scientific output include a higher avoidance rate, a reduced probability of serious head injury, and increased braking distance and time gap for following vehicles equipped with original AEB.

We sincerely appreciate your valuable feedback and believe that these modifications have enhanced the clarity and rigor of our paper.

Sincerely,

The Authors

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The article investigates cyclist behavior in urban environments using a VR simulator to develop better driver assistance systems that can help prevent accidents. The topic of the manuscript is current because the development and implementation of assistance systems for cyclists could significantly reduce the number of accidents and injuries to cyclists and contribute to safer and more sustainable transport. Graphically, the article is well prepared. I recommend describing the breakdown of the article in the introduction. The authors' motivation for the article is clear, and I recommend better describing the novelty and contribution of the article. In the conclusion, I recommend writing a continuation of the research and recommendations for further research.

Author Response

Dear,

Thank you for your insightful comment, which we have carefully considered. We have incorporated the following changes in response to your suggestions, presented in the order you provided. All changes have been conducted using the MS Word change control tool.

The authors have included in the second section a more complete description of the novelty and contribution of a predictive collision model based on cyclist behaviors in VR for an AEB system. We have included references to our previous works, where we developed a similar predictive model for pedestrians, and whose integration in a commercial AEB system together with an AES system allowed us to increase both the avoidance effectiveness of the original system and to increase the minimum distance and time gap to reduce the possibility of rear-ending collisions.

We have also included in the conclusions some of the future lines of our research: a multi-user application to evaluate conflicts between VRUs; an application that generates bespoke scenarios to address a more comprehensive assessment of VRU behavioral patterns, and the development of an e-scooter user VR simulator, based on calibration with an inertial full-body capture measurement system.

We sincerely appreciate your valuable feedback and believe that these contributions have enhanced the clarity and rigor of our paper.

Sincerely,

The Authors

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

Please view the attachment.

Comments for author File: Comments.pdf

Author Response

Dear,

Thank you for your insightful comments, which we have carefully considered. We have incorporated the following changes in response to your suggestions, presented in the order you provided. All changes have been conducted using the MS Word change control tool:

1. The application has been designed primarily to evaluate the cyclist's actions in front of the vehicle. The kinematic parameters of the vehicle have been adjusted to the average speed of traffic in the city of Madrid, the typical deceleration to that of a vehicle with an AEB system tested on track in our facilities, and the driver's reaction time to the value of 1 s (accepted value in the traffic safety literature). The prior values have been modified accordingly for the SC5 scenario, due to the type of vehicle-VRU encounter (vehicle reversing out from a parking lot). This description has been included in the manuscript, in the subsection describing the VR scenarios.
2. Section 3.2 includes now a description of the stereophonic sounds integrated into the application. These represent the environmental background sound (traffic, people's voices), engine, rolling and the sound of vehicle braking. All these audible stimuli are received by the user through the headphones of the HTC Vive VR headset.
3. The authors have developed an ANN model, although the accuracy results and performance metrics were similar to those of supervised classifiers (Accuracy=0.82, Precision=0.84, Recall=0.82 and f1-score=0.83). Since the explanation of the model (introduction, network layers, activation functions, etc.) would extend the length of the manuscript, and the performance is similar to that of the other classifiers, its inclusion in the first version of the manuscript was discarded. If you believe it is convenient, we can integrate it as well.
4. By integrating ML predictive model into the AEB system, it is plausible to regulate the braking pressure during emergency braking: if the model predicts that a collision will occur, the vehicle brakes with maximum pressure; but if it predicts that the VRU action avoids the potential collision, then it will apply partial braking. This reasoning was used in our previous works, with certainly promising results: a similar predictive model for pedestrians was deployed, and its integration in a commercial AEB system together with an AES system allowed to increase the avoidance effectiveness of the original system and to increase the minimum distance and time gap to reduce the possibility of rear-ending collision. This dissertation has been conveniently included in the State of the Art (with the corresponding references) and in the Conclusions sections.
5. Currently, the application cannot generate bespoke scenarios. Nevertheless, one of our research lines in which we are immersed is based on the development of this application, where customized urban scenarios are generated through a series of input parameters, such as: number of lanes, direction of traffic flow, infrastructural elements of urban furniture, different types of vehicle-VRU interaction or the existence of traffic light regulation. This future line has been included in the Conclusions section.
6. The current application is focused only on vehicle-cyclist interactions. Previously, our research team developed a homologous application to evaluate the behavioral patterns of pedestrians in road conflicts with vehicles. However, one of the research lines we are working on has been included in the Conclusions section: the development of a multi-user application (several VR headsets connected at the same time) to evaluate a greater variety of conflicts, including interactions between VRUs.
7. An image of the first-person VR cyclist view has now been included (new Figure 9).
8. The authors have reduced the extent of the subscripts in the equations of the numerical variables. The expression Eye Tracker has been abbreviated, and it has been specified which subject in the simulation corresponds to each timestamp and position, according to the new nomenclature.

We sincerely appreciate your valuable feedback and believe that these contributions have enhanced the clarity and rigor of our paper.

Sincerely,

The Authors

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

The paper deals with vehicle-to-cyclist collision prediction models by applying machine learning techniques. This topic is hihly actual and is relevant for journal. The paper is well structured. The concept, approach and methodology are clear and the language quality is appropriate for a journal. However, there are several drawbacks to be mentioned:

1) I think the Introduction could be completed with a brief paragraph describing a content of upcoming sections. It will help the readers with navigation.
2) Several aspects could be addressed in more detail in "State of the art" section, e.g. "Machine learning".
3) I would reconsider the section titles, some of them are quite general ("State of the art", "materials and methods"). E.g. in Section 3 is nothing related to materials.
4) Figure 7a) contains lot of empty space. Perhaps in could be zoomed to in better way.
5) Similarly, I would put the horizontal labels and caption to the bottom of the figure and unify that in the whole paper.
6) In the whole paper, I would simplify notation as "XCILISTA_EYE_TRACKER_vehicle_start" or "𝑑𝐸𝑌𝐸𝑇𝑅𝐴𝐶𝐾𝐸𝑅𝑣𝑒ℎ𝑖𝑐𝑙𝑒"
7) Moreover, looking at line 297, it is not clear how index "i" is implemented in the sum
8) Please check the quality (resolution) of all Figures
9) Conslusions should be completed via ideas for future work in order to track the research line
10) The list of references should be extended as this is a journal paper
11) I would appreciate a bit more details on how the approache could be adapted to electric bikes and other electric personal transportation devices, which are getting more and more popular in urban areas.
12) Also a bit more references to authors previous works could be added to track the research baseline
13) There are lots of abbreviations in the paper, please doublecheck if all are defined before being used (see e.g. "SSQ").
14) There is a big block of "Pedestrian behaviour modeling" in Fig. 1. However, it is not addressed in the text in sufficient detail.
15) Also machine learning classifiers and related topic could be addressed in more detail.
16) CATS project is refered quite often. I think clear link to project webpage will help the readers.
17) I would improve the "Results" section and clarify more precisely the figures and charts presented, e.g. what are the units on vertical axes on Fig. 19?


Clearly, there are lots of positive aspect like: Overal concept and and methodology nicely summarized in Fig. 1., approach tested on various diverse scenarios; clear positive societal impact, namely related with human safety.

Consequently, despite mentioned drawbacks, I think the paper could be revised again after update respecting comments from all reviewers;

Comments on the Quality of English Language

The language quality is adequate for a journal only minor final check is required.

Author Response

Dear,

Thank you for your insightful comments, which we have carefully considered. We have incorporated the following changes in response to your suggestions, presented in the order you provided. All changes have been conducted using the MS Word change control tool:

1. The authors have now included a paragraph at the end of the Introduction section, describing the contents of the following sections.
2. The authors have included further explanation about the Machine Learning literature. In particular, it has been indicated that the supervised classifiers addressed in our work have been mostly implemented in the scientific literature through the macroscopic study of road safety with cyclists, generating collision risk and injury severity models. While at the vehicle level most optimization focuses on the development of convolutional neural networks for user detection and accident prediction, techniques based on Deep Learning also focus mainly on the analysis of data and macroscopic variables of traffic in a specific area.The article proposes the use of supervised classifiers (distance-based, tree-based) in a cyclist AEB system based on behavioral patterns measured in individual interactions and with kinematic variables, a topic not addressed until the moment in literature. This work is a continuation of our research line on the application of Machine Learning techniques to improve semi-autonomous driving in commercial vehicles in terms of pedestrian protection (https://www.mdpi.com/2076-3417/12/22/11364; https://doi.org/10.3390/vehicles5040084).

3. The authors have modified the subsections of the manuscript to make the content more compact and clearer. The Materials and Methods section has been divided into: Methodology, Virtual reality scenarios, Materials and equipment (new, separate), Adaptation of the VR simulator braking system, Calibration of the steering system, User interface and simulation options, Sample definition and Supervised Machine Learning classifiers.
4. Figure 7a) has been conveniently adjusted to eliminate the empty space.
5. The authors have modified the figures so that horizontal labels and captions appear at the bottom in all cases.
6. The authors have reduced the extent of the subscripts in the equations of the numerical variables. The expression Eye Tracker has been abbreviated, and it has been specified which subject in the simulation corresponds to each timestamp and position, according to the new nomenclature.
7. The authors have included the explanation of the subscript i in the above expression: “… d_{EYE_TRACKER_vehicle} is the distance traveled by the cyclist in each observation period (i) prior to the accident or theoretical point of collision. Each observation period i starts with an ET_vehicle_start and ends with an ET_vehicle_end”
8. The authors have reviewed the quality of the images, improving the resolution of those of low quality. We have included a new ZIP with all the images, for subsequent editing of the manuscript, if required.
9. We have included in the conclusions some of the research future lines: a multi-user application to evaluate conflicts between VRUs, an application that generates bespoke scenarios to address a more comprehensive assessment of VRU behavioral patterns, and the development of an e-scooter user VR simulator, based on calibration with an inertial full-body capture measurement system.
10. As stated before, the authors have included more references, related to the use of Machine Learning and Deep Learning techniques (implementation at the level of collision risk prediction in macroscopic traffic studies with cyclists and implementation in vehicular systems), relevant literature relating to the main simulator settings, as well as questionnaires to evaluate the level of usability and immersion of the VR application. The number of bibliographic citations has increased from 20 to 39.
11. The authors have included in the Conclusions section one of our future research lines on the design of a VR simulator for user Personal Mobility Vehicles (PMV), such as e-scooters. For this, besides the corresponding calibration of the braking and steering system, we use a sensorized full-motion capture system, to determine the forces and accelerations on the rider's limbs and thus design the simulator anchorages and fine-tune the throttle sensitivity. Currently, we are in the post-processing phase of the data obtained in track tests for the simulator dimensioning.
12. As indicated in point 2, references to previous papers related to the subject matter of this manuscript and published by the authors have been included. In particular, a similar collision predictive model for pedestrians based on VR techniques was deployed, and its integration in a commercial AEB system together with an AES system allowed to increase the avoidance effectiveness of the original system and to increase the minimum distance and time gap to reduce the possibility of rear-ending collision.
13. The authors have now checked all abbreviations and their definitions.
14. The authors regret the error in the outline of the methodology. We have corrected the figure, replacing the text with "ML COLLISION PREDICTIVE MODELS BASED ON CYCLIST BEHAVIOR", whose workflow is described in the last subsection of the Materials and Methods section and fully developed in the Results section.
15. In addition to adding more relevant Machine Learning literature in the State of the Art section, we have included a subsection in Materials and Method (Supervised Machine Learning classifiers) where the Machine Learning concept, its usefulness, the classifiers used (both distance-based and tree-based), and the workflow sequence are explained. Likewise, some details have been explained in the Model fitting section, such as the purpose of dividing the sample into the training set and test set, and the concept of k-fold cross-validation in the optimization process for the search of hyperparameters. Lastly, in the Discussion section, a broader explanation of the performance metrics was included, and how their evaluation is related to the potential implementation in an AEB system.
16. The authors have added a reference to the CATS project web page in CORDIS (https://cordis.europa.eu/project/id/234341)
17. The authors have corrected the corresponding image, this time generating a matrix of bar charts to perform the individual comparison of each metric performance. In this way, in addition to the most extensive explanation of the performance metrics and how to evaluate them to analyze the suitability of the models in a decision algorithm of the AEB system, it is plausible to better follow the content of the Discussion section.

We sincerely appreciate your valuable feedback and believe that these contributions have enhanced the clarity and rigor of our paper.

Sincerely,

The Authors

Author Response File: Author Response.pdf

Reviewer 5 Report

Comments and Suggestions for Authors

The authors propose a relevant work for the state of the art, however, some comments should be addressed to enrich and facilitate the reading.

1.- In the introduction section, the contributions of the work should be highlighted.

2.- In the line 484 there’s two times Figure17.

3.- The writing of $t_2$ should be checked as it does not respect the subindices (line 275). The same with $d_1$ and $d_3.

4.- In KNN why is selected k equal to 3? How would be the performance with other value?

Comments on the Quality of English Language

The authors propose a relevant work for the state of the art, however, some comments should be addressed to enrich and facilitate the reading.

1.- In the introduction section, the contributions of the work should be highlighted.

2.- In the line 484 there’s two times Figure17.

3.- The writing of $t_2$ should be checked as it does not respect the subindices (line 275). The same with $d_1$ and $d_3.

4.- In KNN why is selected k equal to 3? How would be the performance with other value?

Author Response

Dear,

Thank you for your insightful comments, which we have carefully considered. We have incorporated the following changes in response to your suggestions, presented in the order you provided. All changes have been conducted using the MS Word change control tool:

1. There is a double scientific contribution of this research, now mentioned in the text and explained in more detail: i) the proposed models are based on supervised Machine Learning classifiers, with low computational cost by processing kinematic data and not requiring image processing through feature extractors ii) it is a microscopic study of vehicle-cyclist interactions, based on cutting-edge techniques such as Virtual Reality, through a calibrated simulator that guarantees a high level of immersion and an exhaustive assessment of the cyclist's critical actions. Therefore, the novelty lies in the more realistic assessment of the cyclists’ behavior in safety-relevant situations, by means of a VR simulator that allows the user to perform actions similar to real ones. This element is crucial to program the response logic of Advanced Driver-Assistance (ADAS) systems to avoid accidents, and to adapt them to the typical cyclist’s behavior patterns in these perilous situations. These aspects have been now included in the reviewed manuscript.

2. The authors have now deleted the repeated expression
3. The authors have reduced the extent of the subscripts in the equations of the numerical variables. The expression Eye Tracker has been abbreviated, and it has been specified which subject in the simulation corresponds to each timestamp and position, according to the new nomenclature. The meaning of the index i in the subscript of the equations has also been explained in more detail.
4. The value of K=3 has been obtained through the GridSearchCV optimization function, as well as the rest of the hyperparameters of the Machine Learning classifiers. This value, as well as the parameter p, allows us to obtain the best performance of the KNN. A more detailed description of the procedure to find the hyperparameters that optimize the performance of the models, as well as the cross-validation procedure it consists of, has been included in the corresponding subsection.

We sincerely appreciate your valuable feedback and believe that these contributions have enhanced the clarity and rigor of our paper.

Sincerely,

The Authors

Author Response File: Author Response.pdf

Reviewer 6 Report

Comments and Suggestions for Authors

Here are some concerns about this paper:

1. What exactly is the innovative aspect of this paper? The author is hoped to clarify this in the relevant sections of the article.

2. What is the logic of the literature review section? What are the problems with existing AEB-cyclist systems? Does the bicycle simulator for a Virtual Reality application mentioned in this paper solve the aforementioned problems?

3. The equations throughout the article are not numbered, and some table captions have periods while others do not. The author is advised to carefully check the formatting of the entire document again.

4. The results section of the paper spends a considerable amount of space discussing the selection of different model parameters and evaluation metrics. Is this meaningful for reaching the conclusion that KNN is the most compatible and suitable option to be integrated into the decision algorithm of an AEB system? It is recommended to adjust the expression in the results section.

5. The results in Figure 19 are not very intuitive. It is recommended to produce a separate comparative effect graph for each different evaluation metric.

6. The literature review section should be expanded to include recent studies in this field. Some relevant studies in this journal using machine learning approaches for prediction should be reviewed. For example, see: Vehicle acceleration prediction based on machine learning models and driving behavior analysis. applied sciences, 12(10), 5259. A Deep Reinforcement Learning Approach for Efficient, Safe and Comfortable Driving. Applied Sciences, 13(9), 5272.

Comments on the Quality of English Language

The paper writing is fine.

Author Response

Dear,

Thank you for your insightful comments, which we have carefully considered. We have incorporated the following changes in response to your suggestions, presented in the order you provided. All changes have been conducted using the MS Word change control tool:

1. There is a double scientific contribution of this research, now mentioned in the text and explained in more detail: i) the proposed models are based on supervised Machine Learning classifiers, with low computational cost by processing kinematic data and not requiring image processing through feature extractors ii) it is a microscopic study of vehicle-cyclist interactions, based on cutting-edge techniques such as Virtual Reality, through a calibrated simulator that guarantees a high level of immersion and an exhaustive assessment of the cyclist's critical actions.Therefore, the novelty lies in the more realistic assessment of the cyclists’ behavior in safety-relevant situations, by means of a VR simulator that allows the user to perform actions similar to real ones. This element is crucial to program the response logic of Advanced Driver-Assistance (ADAS) systems to avoid accidents, and to adapt them to the typical cyclist’s behavior patterns in these perilous situations.These aspects have been now included in the reviewed manuscripts.

2. The first part of the literature review focuses on the development of the bicycle simulator, providing a line of technological progression and advancement from the first implementations to the most current developments. Next, the authors have now highlighted in the text the contributions of the research conducted in the manuscript, now providing more papers related to the use of Machine Learning and Deep Learning techniques in road safety and implementation in vehicle technology. In particular, the models described in this paper are based on supervised classifiers with low computational cost due to the processing of kinematic variables, unlike those used in vehicle implementations addressed in the scientific literature (based on convolutional networks and feature extractors). Also, most of the supervised models proposed in the paper have been developed in macroscopic studies of collision risk assessment and injury severity. In this case, more emphasis is placed on the vulnerable user, pointing out the critical actions in the decision-making process that can lead to collision in different types of collision scenarios. For this purpose, a cutting-edge technology (Virtual Reality) has been used, with a calibrated Virtual Reality simulator that guarantees a high level of immersion and realism. To date, no collision predictive models have been developed for AEB of cyclists based on a comprehensive characterization of the user's behavior and actions on the road.

3. The equations have been numbered and periods have been included in all table captions. The authors have carefully revised the formatting of the manuscript again.
4. The results section has included the phases corresponding to the Machine Learning workflow, as well as an exhaustive study of the cyclist's actions in potential accident situations through exploratory analysis, the decision logic of the tree-based models and the importance of the variables. We have also included a section corresponding to the evaluation of the usability and level of immersion of the VR application.The evaluation of the performance metrics in the Discussion section and in the conclusions is necessary to analyze the capacity of the models to detect cases of Collision and Avoidance, critical in the suitability of their implementation in an AEB system (an explanation has been included more detailed of them, to favor subsequent analysis). In particular, these metrics, along with the ROC curve and the AUC, are analyzed for each model, comparing all of them to indicate the most suitable one for the purpose of the present investigation

5. The authors have conveniently modified Figure 19 (now Figure 21) to establish separate bar charts for each performance metric. Likewise, in addition to the most extensive explanation of the performance metrics and how to evaluate them to analyze the suitability of the models in a decision algorithm of the AEB system, it is plausible to better follow the content of the Discussion section.
6. As mentioned in point 2, a larger number of recent studies in the field have been included, with a special focus on the use of Machine Learning techniques. Likewise, the two papers proposed in your comment have been incorporated, relating them to the section on technological innovation and implementation of Machine Learning in in-vehicle predictive systems.

We sincerely appreciate your valuable feedback and believe that these contributions have enhanced the clarity and rigor of our paper.

Sincerely,

The Authors

Author Response File: Author Response.pdf

Round 2

Reviewer 4 Report

Comments and Suggestions for Authors

The second revision of the paper has a significantly higher quality. The authors adressed point by point all critical comments. I think the paper can be accepted after final gramar and formatting check, if all reviewers agree.

Comments on the Quality of English Language

I think laguage is fine, I will just make a final check and edits.

Reviewer 5 Report

Comments and Suggestions for Authors

After review of the version submitted by the authors. It is clear that they have attended to the reviewers' comments, and it is a valuable article to be accepted for publication in such a prestigious journal. 

Comments on the Quality of English Language

After review of the version submitted by the authors. It is clear that they have attended to the reviewers' comments, and it is a valuable article to be accepted for publication in such a prestigious journal. 

Reviewer 6 Report

Comments and Suggestions for Authors

The comments are well addressed.

Comments on the Quality of English Language

The paper writing is fine.