Risk Determination versus Risk Perception: A New Model of Reality for Human–Machine Autonomy

Round 1

Reviewer 1 Report

This paper presents a case study from a recent drone strike in Afghanistan by the U.S. Military that involved an erroneous risk assessment, resulting in the death of 10 Afghan civilians. The author explores this case at length and demonstrates how the determination of risk facing U.S. forces at the airport in Kabul was attributable to a process breakdown, which failed to break confirmation bias and resulted in a flawed perception of risk. He discusses how this potential for process failure becomes even more problematic as the potential deployment of lethal autonomous systems is explored by Nations. He then presents a physics model that predicts human-machine teams/systems will perform best under situations where risk perceptions are debated openly and thoroughly, as compared to a minority-rule or consensus-seeking situations where one perception dominates and all other perceptions are suppressed. This finding is also discussed with respect to innovation, government rule, and suppression of ideas/perceptions, and how minority rule wastes energy and reduces MEP.   

This paper has an unconventional style with lengthy quotations, however the majority are thought provoking and contribute to the paper. It would be useful to trim some and paraphrase others, because the lengthy quotes can begin to detract from the overall flow and point of the section, leaving the reader somewhat lost. For example, I don’t think lines 263- 273 and 569-579 add much to the paper, and can be removed or paraphrased. Furthermore, the long quotes within the conclusion, while interesting, detract from the overall point of the paper and I would suggest either moving theses to another section or drastically reducing each of them and including a summarizing, closing paragraph at the end.

• In the “1.1. Situation” section I suggest broadening this description to AI or machine learning, rather than solely robotics, in order to help with the flow to section 1.2 which opens describing hate speech online, and not robotics.
• Section 1.2.1 hangs without a “so what” at the end since it ends on a quote. Also, the facebook example is not mentioned again in the paper, which makes me question its value/inclusion in the paper
• Suggest ending 1.2.2 noting that this example will be discussed further in section 1.3.3
• Section 1.3. - suggest first succinctly stating what the “human-machine drone problem” is, before jumping into the fix.
• Rename 1.3.3. header to Discussion of…[Military Case Study]?
• Line 203 – close quotes or indent and remove the first opening quote
• In section 3 you state “for the rest of this article, we review the predictions from Equation 1, which are counter-intuitive” – I suggest removing this, because I was anticipating that the rest of the paper would be focused on the equation, but only a small portion was.
• Define c in equation 1
• Line 311 – change citation of Lawless, 2019 to [26]
• Combine 3 paragraphs into one, between lines 341 – 355
• I don’t think section 3.1 added much to the paper, potentially remove or rewrite to make relevance more obvious.
• Paragraph 450 – 459, while interesting seems superfluous and could be removed

Author Response

Dear Reviewer,

Please find the responses to review report below.

Best，

Author Response File: Author Response.pdf

Reviewer 2 Report

The manuscript reviews critical issues about the risk assessment of autonomous human-machine systems. Generally, the manuscript is well organized. However, some changes should be implemented to reach the publication level:

1. Please provide a table of acronyms.
2. The body text needs proofreading. In some sections, the text is messy and needs to be revised.
3. Generally, the author is recommended to provide graphical classifications and/or table presentations. The current version includes text descriptions only. 
4. I believe that the conclusion section should be revised. In this section, some brief information is expected. In the present version, the study is discussed. So, some parts may be transferred to the discussion section. A more brief and informative conclusion is highly appreciated.  

Author Response

Dear Reviewer,

Please find the responses to review report below.

Best，

Author Response File: Author Response.pdf