Murder on the VR Express: Studying the Impact of Thought Experiments at a Distance in Virtual Reality

Round 1

Reviewer 1 Report

Please see comments attached.

Comments for author File: Comments.pdf

Author Response

Thank you so much to all four referees for their very helpful feedback. We have made numerous changes to the manuscript to accommodate as many of the comments as we could, though in some places comments pulled in different directions and we had to make choices.

 

The largest change to the manuscript has been to clarify our research questions in the introduction. The discussion section has been reframed to address those research questions. The analysis in the results section has been rerun and presentation has been streamlined to focus on the research questions as well. The study is largely exploratory in nature, meant to provide an initial foray into the viability of running this sort of study at a distance in virtual reality, while also providing a valuable tool for researchers, educators, and an interested public. To this end, we have also included more details about the development of the experience itself and provided further recommendations for future studies. Finally, we have added numerous citations to the bibliography, both on the recommendations of the referees and in the course of addressing feedback. We hope that you find the changes satisfactory and thank you for your consideration.

Reviewer 2 Report

See pdf

Comments for author File: Comments.pdf

Author Response

Thank you so much to all four referees for their very helpful feedback. We have made numerous changes to the manuscript to accommodate as many of the comments as we could, though in some places comments pulled in different directions and we had to make choices.

 

The largest change to the manuscript has been to clarify our research questions in the introduction. The discussion section has been reframed to address those research questions. The analysis in the results section has been rerun and presentation has been streamlined to focus on the research questions as well. The study is largely exploratory in nature, meant to provide an initial foray into the viability of running this sort of study at a distance in virtual reality, while also providing a valuable tool for researchers, educators, and an interested public. To this end, we have also included more details about the development of the experience itself and provided further recommendations for future studies. Finally, we have added numerous citations to the bibliography, both on the recommendations of the referees and in the course of addressing feedback. We hope that you find the changes satisfactory and thank you for your consideration.

Reviewer 3 Report

This is an exploratory study with VR and moral decision-making. It’s contribution to the field is primarily methodological. Author(s) use a new way of collecting data, which involves people outside of a laboratory setting, promises to scale up. I think this is indeed promising and probably the best way to frame and justify this research. The aims of replication should be made more explicit, with statement of effects and appropriate discussion of past research.

Each section has significant problems, which together make this paper not publishable in present form.

Introduction:

The purpose of thought experiments in philosophical arguments in not typically to make complex philosophical ideas accessible to a wider audience. This is a misrepresentation of philosophical methods in general and the purpose of the trolley dilemma example in particular. The Matrix and other things that are cited are not a part of professional philosophy in any sense.  This is misleading and irrelevant to what comes after.

Things are not better when researchers report on the use of the trolley dilemma in empirical research. First, the trolley dilemma is not a tool for gathering information about what people would do, even though I presume it has been purported to do that in some empirical work that used it as a scenario and collected responses. To my understanding, it is a scenario (one of many) that can be used to us something about the nature of moral judgments in general.  Empirical work that uses it, such as Greene’s, or theoretical work, such as Mikhail’s, does not obviously aim to tell us something about “what people would actually do in real life.” I am not sure what value there would be in finding such a thing out, anyway.

I believe author(s) make an excellent point about using text-based questionnaires, including those that use the trolley dilemma, to carry about empirical research about the way people actually make moral judgments. Namely, it is not ecologically valid, and probably not a great way to investigate the mechanisms that underlie human moral decision-making, from an empirical perspective.  That is a point worth developing further. It is worth substantiating that claim with evidence or a better literature review, and perhaps also substituting all the references to popular media with how it bears directly on the choice to use VR.

There is previous research that uses trolley dilemmas in a VR environment or just in a lab.  It may be good to review that literature and place this study within it.

What would have been even better is a clear statement of the hypotheses being tested, albeit since this appears to be primarily aimed at replication in a new setting, some of these could be borrowed from other studies.

Methods:

The experimental set-up is entirely unclear. At this point this is a between-subject design, with one group per condition, and 6 or 7 independent variables that are being measured, in addition to a post-experiment questionnaire (it would be nice to have a better description of the construct that this tool is measuring). 33 participants is very small for an experiment of this complexity. 

It is also not clear why participants were assigned to each condition *before* completing the moral control check. This has to be better justified. 

It is also not clear why the experimenters chose to maintain an imbalance in their study sample, especially with such a small group.

What is most puzzling is that this assignment was apparently random, but the subsequent analysis looked at individual differences in groups.  Why were these criteria not used to create groups to begin with? 

There is no justification given for the size of the sample; no power analysis for the expected effect size is provided.

It is not at all clear why all this demographic information was collected about participants, which is worrysome, given the sensitive nature of research into moral decision-making and the fact that this data is used in analysis just to tell us about differences in decision-making. One possibility is that this data was used in an exploratory way, to find correlations after collecting responses. If that is the case, then author(s) should say so.

What is the moral dilemma check that three participants failed?  How does one fail it?  Is it the same as the moral control scenario?

Results:

The plots appear to be a pixelated screenshots taken from Jupyter notebooks or R and should be better quality.

Figure 5 is very difficult to interpret, since the y axis is in percents.  If I understand it correctly, then 100% of people that prefer not to specify their gender chose to kill 1 and spare 5.  That could have been 1 person, but also 5.  Not clear.  Also not clear why this is significant and why it is tested for.  What is the research question that is being answered here?

Figure 6 is easier to interpret, but again not at all clear what its connected to the stated research questions would be. Are we trying to find out whether Christians are more likely to not pull the switch?  More likely to push someone on the tracks?  Both?  Is this, then, an experiment meant to tell us something about the differences between Christians and non-Christians?  If so, then the experiment needed to take that into account in the beginning and follow proper methodology to set it up accordingly, including group size.

Discussion:

No substantive or easy-to-read take away point that would state the significance of results, perhaps because no hypotheses are stated.  No real discussion of limitations.  Some discussion of future directions, albeit mostly connected to the use of the same technology and method of collecting data, rather than anything that has to do with what was being measured or tested.  Also, not sure why Cushman’s dual-process theory is the standard by why we should interpret this sort of work. 

 

 

 

Author Response

Thank you so much to all four referees for their very helpful feedback. We have made numerous changes to the manuscript to accommodate as many of the comments as we could, though in some places comments pulled in different directions and we had to make choices.

 

The largest change to the manuscript has been to clarify our research questions in the introduction. The discussion section has been reframed to address those research questions. The analysis in the results section has been rerun and presentation has been streamlined to focus on the research questions as well. The study is largely exploratory in nature, meant to provide an initial foray into the viability of running this sort of study at a distance in virtual reality, while also providing a valuable tool for researchers, educators, and an interested public. To this end, we have also included more details about the development of the experience itself and provided further recommendations for future studies. Finally, we have added numerous citations to the bibliography, both on the recommendations of the referees and in the course of addressing feedback. We hope that you find the changes satisfactory and thank you for your consideration.

Reviewer 4 Report

GENERAL COMMENTS

This paper presents a VR study of the two main versions (switch and push) of the trolley dilemma, whose main contribution is that it was conducted outside traditional laboratories, with participants at home and using commercial VR headsets. It replicates similar results to those achieved in previous VR research.

The article is well written, adequately motivates the research problem, and has a good study design. In particular, I find attractive the authors’ idea of defending the validity of conducting moral dilemma studies outside conventional settings such as universities or research centres, using the participants’ domestic VR equipment. This is an approach that could be pursued by future research, and I think it is a relatively original contribution of this article (which also shows some strategies to avoid concrete problems). Furthermore, the article fits in well with the special issue to which it has been submitted.

Among the shortcomings of the article, I would highlight that it only presents a single study with a relatively small sample. The authors collect valid data from only 33 participants, of which they removed three who failed the Moral control scenario, so the final sample is only 30. Authors say in the discussion that their sample is consistent with other similar investigations, citing Navarrete et al. (2012), Patil et al. (2014), and Francis et al. (2016). However, this is only the case for Patil et al. (with a sample of 40). In Navarrete et al. the sample is considerably higher (+300) and, in Francis et al., there were various studies with slightly higher samples. All in all, the authors of this paper concentrate their results on the statistically significant results they achieved. But the reduced sample is a limitation that editors should consider.

 

SPECIFIC COMMENTS

Lines 334 and 342: a space is missing before the words in italics. Also in line 363.

In the Push Dilemma, authors get a very similar result to the studies of Francis et al. (approximately 60% of approval of the act of pushing). Still, they only cite Cushman, who can offer the theoretical explanation but not the result. I recommend citing the Francis et al. studies when discussing replication.

One of the virtues of this type of in-home research would be to precisely reach audiences that are less common in virtual reality research. Some leaders in VR research have noted that their studies commonly suffer from the limitation of being conducted with university students and young people, sometimes from affluent backgrounds [Bailenson, J. (2018). Experience on demand: What virtual reality is, how it works, and what it can do. WW Norton & Company]. In this study, interestingly, the participants fit the same profile as in traditional settings: the majority are young, university-educated people. It would be good if they could briefly discuss this limitation a bit more (they already mention it in the discussion) and comment on possible strategies for these studies to reach wider audiences.

 

Bibliographical recommendations:

-       Cite Pan & Slater’s seminal paper on using VR to study sacrificial dilemmas such as the trolley problem: Pan, X., & Slater, M. (2011, July). Confronting a moral dilemma in virtual reality: a pilot study. In Proceedings of HCI 2011 The 25th BCS Conference on Human Computer Interaction 25 (pp. 46-51).

-       On the ecological validity of VR to study moral dilemmas I also recommend the article by Parsons, T. D. (2015). Virtual reality for enhanced ecological validity and experimental control in the clinical, affective and social neurosciences. Frontiers in human neuroscience, 9, 660.

 

-       There is a recent debate about the ethical aspects of conducting this particular type of VR research into the trolley dilemma. Without any obligation, perhaps it might be in the authors' interest to briefly mention this debate in the discussion. See, Ramirez, E. J., & LaBarge, S. (2020). Ethical issues with simulating the Bridge problem in VR. Science and Engineering Ethics, 26(6), 3313-3331. And Rueda, J. (2022). Hit by the Virtual Trolley: When is Experimental Ethics Unethical?. Teorema: Revista internacional de filosofía, 41(1), 7-27.

Author Response

Thank you so much to all four referees for their very helpful feedback. We have made numerous changes to the manuscript to accommodate as many of the comments as we could, though in some places comments pulled in different directions and we had to make choices.

 

The largest change to the manuscript has been to clarify our research questions in the introduction. The discussion section has been reframed to address those research questions. The analysis in the results section has been rerun and presentation has been streamlined to focus on the research questions as well. The study is largely exploratory in nature, meant to provide an initial foray into the viability of running this sort of study at a distance in virtual reality, while also providing a valuable tool for researchers, educators, and an interested public. To this end, we have also included more details about the development of the experience itself and provided further recommendations for future studies. Finally, we have added numerous citations to the bibliography, both on the recommendations of the referees and in the course of addressing feedback. We hope that you find the changes satisfactory and thank you for your consideration.

Round 2

Reviewer 1 Report

I would like to thank the authors for responding to all of the reviewers' comments so thoroughly and in detail. The paper is much improved and I think this makes an important contribution to a sub-field in moral psychology adopting VR (and associated) paradigms in empirical research.

Author Response

Thank you very much for the feedback, the paper has improved greatly as a result.

Reviewer 3 Report

Fine improvements, especially on the change of focus, and analysis.  Good take-away points, clear methodological contribution, and well-motivated in the context of other studies of this sort.

I am still puzzled by all of the references to popular media in the introduction.  The explanation of the role of thought experiments in philosophical methodology is improved, but still lacking.  This could be improved with a straightforward literature search.  Thought experiments and their use has been recently the subject of monographs and papers.  Those may be good to cite here.  That said, this is not a major issue, since this discussion serves only to introduce the trolley dilemma and the motivation for the study.

Secondly, the labels used for coding participant responses post-experiment (deontological, consequentialist, etc.) need better explanation with more references to contemporary definitions of these terms in first-order ethics theorizing.  I am sure this is easy to do, so, again, this is a minor revision.

Author Response

Thank you for your continued consideration of our manuscript. We have added additional citations and discussion of some of the various uses of thought experiments in the introduction. These changes can be seen in lines 18- 47. We have decided to retain references to popular culture as evidence of an increased public interest in engaging with thought experiments. We think this is important for at least two reasons. 1) It gives some reason to think that non-philosophers would be interested in participating as subjects in research using thought experiments, 2) It suggests potential societal impacts of broader access to thought experiments. We take this second point to be particularly relevant, given the journal's goals.  

We have also clarified the terms "consequentialism" and "deontologist" with reference to contemporary and historical examples. These changes can be seen in lines 266-285.

Once again, thank you for the feedback.