Software Development for Processing and Analysis of Data Generated by Human Eye Movements
Round 1
Reviewer 1 Report
The paper presents software for processing and analysis of data generated by human eye movements.
This is an interesting area and the work done shows promise, it builds on earlier work that has been untaken by the authors in 2018.
The Introduction section starts very generally discussing the general (scientific) need to collect and analyse experimental data. This introduction is too general and does not motivate why this research/software development in particular was required. The existing work reviewed is general in nature and not specifically related to eye tracking or analysis software for eye tracking data. The introduction would be greatly improved if the focus was narrowed and specifically focused on eye tracking software/data analysis.
There is information on pg 3-4 about the authors previous work on the software and the level of detail provided seems too great. I think this could be improved with a diagram showing the architecture of the exisiting software and an Entity Relationship diagram to show the existing data model. An additional diagram could then be added to show the additional work done in this research.
The information in line 127-130 would be better communicated through a link that describes the file format or a tabular description of the data fields. As currently described it would not be possible to understand the data format if presented with a file in this structure.
The information presented on pg 4-5 about the existing web service and the methods available would be better presented in a table or as a API description.
The materials and methods section focuses mainly on the conversion of raw .bin files and microphone recordings into a format suitable for storage in the database. This is well described, however it is not clear why MicMin and MicMax are the values that are of value from the microphone recording rather than the actual utterances/words of the participant at that point in the experiement.
The results are presented in section 3 and show images generated by the software and also summary data (Table 2 & 3). What is not clear is how the data here were generated. There is no description of the tasks that users under took, the number of users and the details of the experimental setting.
The statistics presented are descriptive statistics (percentage correct/incorrect) with no more robust statistical analysis performed.
Total session times are shown as reaction time, it would seem that looking at aggregate times (even if for one user) would hide variations in performance for individual tasks and therefore is not a good representation of performance. It is better to show average time and ranges or some other normalized result.
Figures 12 a & b are summed rather than averages with error bars which means they do not show the impact of participants who struggled with different movement types from each other.
The conclusions state that "SimulationCenter application has all the 603 necessary functionalities that have been experimentally tested and verified" however no experimentation or verification of the software (as contrasted with the experiemental data) is presented. There are no user studies that look at the effectiveness of the software, it's ability to do things "better" than other approaches or other available software.
Overall, whilst the software produced is interesting, more focused reporting on the new aspects of this software, details around the experimental setup and stronger statistical analysis along with user studies of the application of the system would greatly improve this paper.
Author Response
Point 1: The Introduction section starts very generally discussing the general (scientific) need to collect and analyse experimental data. This introduction is too general and does not motivate why this research/software development in particular was required. The existing work reviewed is general in nature and not specifically related to eye tracking or analysis software for eye tracking data. The introduction would be greatly improved if the focus was narrowed and specifically focused on eye tracking software/data analysis.
Response 1: The "Introduction" section has been updated, with the presented information now focusing mainly on the research itself and some of its aspects related to the processing and extraction of the data generated during the experimental sessions.
Point 2: There is information on pg 3-4 about the authors previous work on the software and the level of detail provided seems too great. I think this could be improved with a diagram showing the architecture of the exisiting software and an Entity Relationship diagram to show the existing data model. An additional diagram could then be added to show the additional work done in this research.
Response 2: The information on pages 3-4 has been reduced. The description of the data model is presented more briefly in a table.
Point 3: The information in line 127-130 would be better communicated through a link that describes the file format or a tabular description of the data fields. As currently described it would not be possible to understand the data format if presented with a file in this structure.
Response 3: The information presented between lines 127-130 (from the original version of the manuscript) is essential in the development of the software.
Point 4: The information presented on pg 4-5 about the existing web service and the methods available would be better presented in a table or as a API description.
Response 4: Information related to the mvsemws web service, and its web methods is presented in a table.
Point 5: The materials and methods section focuses mainly on the conversion of raw .bin files and microphone recordings into a format suitable for storage in the database. This is well described, however it is not clear why MicMin and MicMax are the values that are of value from the microphone recording rather than the actual utterances/words of the participant at that point in the experiement.
Response 5: The MicMin and MicMax values are needed to synchronize the signals from the saccade sensor (which operates at a frequency of 1000 Hz) and the signal from the microphone (which is recorded at a frequency of 8000 Hz). MicMin and MicMax contain respectively the lowest and highest value of the audio signal for each segment of 8 values. The participants in the experiment did not speak during the sessions. The audio signal is important because of the background noise being analyzed.
Point 6: The results are presented in section 3 and show images generated by the software and also summary data (Table 2 & 3). What is not clear is how the data here were generated. There is no description of the tasks that users under took, the number of users and the details of the experimental setting.
Response 6: The main objective of the present work is to present the software by which all experimental sessions can be recreated. We have presented detailed information about the data in publications [4, 8, and 9].
Point 7: The statistics presented are descriptive statistics (percentage correct/incorrect) with no more robust statistical analysis performed.
Response 7: A more detailed statistical analysis of the experimental data from our research is presented in publications [13 and 31].
Point 8: Total session times are shown as reaction time, it would seem that looking at aggregate times (even if for one user) would hide variations in performance for individual tasks and therefore is not a good representation of performance. It is better to show average time and ranges or some other normalized result.
Response 8: From the results of our research published so far, presented in more detail in sources [9, 13, and 31], only summary data that we have presented in the present study has not been published.
Point 9: Figures 12 a & b are summed rather than averages with error bars which means they do not show the impact of participants who struggled with different movement types from each other.
Response 9: We have published a detailed analysis related to exactly this particularity of the experimental study in [31].
Point 10: The conclusions state that "SimulationCenter application has all the 603 necessary functionalities that have been experimentally tested and verified" however no experimentation or verification of the software (as contrasted with the experiemental data) is presented. There are no user studies that look at the effectiveness of the software, it's ability to do things "better" than other approaches or other available software.
Response 10: All images presented are generated by the SimulationCenter application. This is a prerequisite (and proof) of its applicability. User opinions on the usefulness of the software will be considered in our next work.
Point 11: Overall, whilst the software produced is interesting, more focused reporting on the new aspects of this software, details around the experimental setup and stronger statistical analysis along with user studies of the application of the system would greatly improve this paper.
Response 11: As the scope of the research itself is large, at the next stage we will conduct a user opinion survey on the usefulness of the software and publish the results.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The authors proposed “Software Development for Processing and Analysis of Data Generated by Human Eye Movements.” The manuscript is well written. However, some issues need to be addressed to improve the quality of the manuscript:
1. The introduction is incoherent and not smooth; the flow is juggling up and not clearly written. Please improve it.
2. Clarify the novelty and contribution of your work.
3. All results need more explanations; for example, describe the mechanism and how the proposed methods improve the outcomes.
4. Elaborate on the future work of the proposed approach. For example, in which area or parameters can be improved?
5. There are several grammatical errors; these need to be resolved by proofreading multiple times.
6. All the references used are not in the same format. Please check it.
Author Response
Point 1: The introduction is incoherent and not smooth; the flow is juggling up and not clearly written. Please improve it.
Response 1: The information on pages 3-4 has been reduced. The description of the data model is presented more briefly in a table.
Point 2: Clarify the novelty and contribution of your work.
Response 2: The "Introduction" section has been updated, with the presented information now focusing mainly on the research itself and some of its aspects related to the processing and extraction of the data generated during the experimental sessions.
Point 3: All results need more explanations; for example, describe the mechanism and how the proposed methods improve the outcomes.
Response 3: The "Results" section has been updated.
Point 4: Elaborate on the future work of the proposed approach. For example, in which area or parameters can be improved?
Response 4: As the scope of the research itself is large, at the next stage we will conduct a user opinion survey on the usefulness of the software and publish the results.
Point 5: There are several grammatical errors; these need to be resolved by proofreading multiple times.
Response 5: An additional spelling and grammar check has been made and any inaccuracies and errors noticed have been removed.
Point 6: All the references used are not in the same format. Please check it.
Response 6: All cited references have been redacted in accordance with the journal citation format, using the journal abbreviations instead of their full names.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
The authors have addressed the comments made in the previous review. In many cases the information included in the comments would be useful to include in the text and these places are outlined below.
Also is SimulationCentre available for download? It would be good to add that detail to the Data availability statement if it is.
RK5 - include why audio is recorded in the main text.
RK6, 7, 8 9 include in the text the references to these studies with the statement that the data is a summary of data presented in more detail elsewhere.
Author Response
Point 1: The authors have addressed the comments made in the previous review. In many cases the information included in the comments would be useful to include in the text and these places are outlined below.
Response 1: Thank you very much!
Point 2: RK5 - include why audio is recorded in the main text.
Response 2: The text "The audio signal is used for background noise analysis." has been added to the manuscript.
Point 3: RK6, 7, 8 9 include in the text the references to these studies with the statement that the data is a summary of data presented in more detail elsewhere.
Response 3: We added information about the publications in which a more in-depth statistical analysis of the experimental results was published.
Point 4: Also is SimulationCentre available for download? It would be good to add that detail to the Data availability statement if it is.
Response 4: We added information about the SimulationCenter application.
Author Response File: Author Response.pdf
Reviewer 2 Report
I don't have any more suggestions.
Author Response
Thank you very much for your support!
