Usability of WebXR Visualizations in Urban Planning
Round 1
Reviewer 1 Report
General comments
The paper focuses on evaluating web-based eXtended Reality (XR) in the scope of spatial planning. Authors designed and implemented six WebXR examples, which usability were subsequently verified. In general, this is an actual and interesting topic. From my point of view, the main ideas are clearly explained. I also appreciate that the examples created and evaluated are available online, including source codes.
Introduction
- From the description of user testing, I understood that it was exploratory research, i.e., research questions and hypotheses were not defined in advance. I would provide information that the study was exploratory in the description of the aims of the paper.
- Page 3, rows 120-123: I think that there are other approaches to distinguishing between Augmented Reality (AR) and Mixed Reality (MR).
Materials and Methods
- From my point of view, this section is clearly written.
- The only possible problem with the experiment's design I see is that the order of the tasks was probably fixed. For this reason, the results could be affected by the so-called learning effect. This issue should be further discussed in the Discussion or Study limitations sections. If participants solved the tasks in random order, this information should be stated.
Results
- Page 8, rows 315-317: I would move this information to the methodological section. It would also be appropriate to add whether '1' means "best" or "worst".
- Table 5: I would also recommend adding the standard deviations.
Discussion
- Page 11, row 425: Citation "Kitchin et al." does not have a numeric link assigned.
- An interesting topic for discussion is also the level of detail of the displayed 3D models. In general, it can be assumed that web technologies, including WebXR, require less detailed (more generalized) 3D models. On the other hand, users can expect or require more complex 3D models. From my point of view, the example called VR4 is perhaps too simplified to be seen from a first-person perspective.
- The authors used the Oculus Quest headset; the question is how other devices would display the presented examples, such as Google Cardboard. Alternatively, it is possible to use standard devices such as desktop PCs, tablets, or smartphones to display examples after certain modifications of its control.
Study Limitations
- As I indicated above, I would also discuss the limitations based on the chosen design of user testing.
I recommend adding a "Conclusions" section, which will briefly summarize the results achieved and indicate possible directions for future work (based on the topics discussed in the previous two sections). The paper will also not end in a slightly negative spirit, which represents the limitations of the conducted research.
Other minor comments
- Authors should correct the formatting of some paragraphs (e.g., 1st paragraph in the introduction) and remove some spaces in the text (e.g., before Table 4).
- Page 2, row 51: I recommend following the exact term "Virtual Geographic Environments "(including word order).
- Table 2: I do not know what the letter K in the column "Gender "means. The terms male-female (M – F) or men-women (M – W) are usually used.
- Page 11, row 421: There is probably a typo at the beginning of this row.
Author Response
Dear Reviewer,
Thank you for taking time to read and review our paper with detailed comments. We have corrected and updated the paper according to your suggestions and we think that the resulting paper is much better in the result. Below you will find our detailed answers for each point that you have made.
Best Regards and thank you!
Comments and responses below:
Introduction
From the description of user testing, I understood that it was exploratory research, i.e., research questions and hypotheses were not defined in advance. I would provide information that the study was exploratory in the description of the aims of the paper.
A: Yes that is true. Only research questions were formulated as stated in the introduction. To highlight the exploratory status of the study we’ve added necessary information as requested in the paragraph where main goals are stated.
Page 3, rows 120-123: I think that there are other approaches to distinguishing between Augmented Reality (AR) and Mixed Reality (MR).
A: That is absolutely corrected. We’ve reformulated our description to include more traditional definitions that distinguish AR and MR using interactivity (eg. Holz et al. 2012) and added the appropriate references.
Materials and Methods
The only possible problem with the experiment's design I see is that the order of the tasks was probably fixed. For this reason, the results could be affected by the so-called learning effect. This issue should be further discussed in the Discussion or Study limitations sections. If participants solved the tasks in random order, this information should be stated.
A: It is quite true that this could very much take place in the experiment. However, since the comparison of the environments was not our main goal we opted for fixed order. We’ve hoped that this will be easier for the users and give them a chance to focus more on exploring rather than learning. But it is also true that this need to be revealed to the user and we have added the necessary information in the Study Limitations section.
Results
Page 8, rows 315-317: I would move this information to the methodological section. It would also be appropriate to add whether '1' means "best" or "worst".
A: The information has been moved as requested and we’ve added the additional description indicating the meaning of the numbers in the methodological section as well as beside the Table 4.
Table 5: I would also recommend adding the standard deviations.
A: Of course. We should have done that. Corrected and SD added.
Discussion
Page 11, row 425: Citation "Kitchin et al." does not have a numeric link assigned.
A: Corrected with numeric link.
An interesting topic for discussion is also the level of detail of the displayed 3D models. In general, it can be assumed that web technologies, including WebXR, require less detailed (more generalized) 3D models. On the other hand, users can expect or require more complex 3D models. From my point of view, the example called VR4 is perhaps too simplified to be seen from a first-person perspective.
A: This is indeed very important and not very documented. Therefore we have added mention of this topic into future research perspective that we are describing in the newly added Conclusions section.
The authors used the Oculus Quest headset; the question is how other devices would display the presented examples, such as Google Cardboard. Alternatively, it is possible to use standard devices such as desktop PCs, tablets, or smartphones to display examples after certain modifications of its control.
A: That is an interesting point. We’ve mentioned briefly that it could be important to test the results using desktop environments. We’ve also added additional explanation to the Study Limitations section.
Study Limitations
As I indicated above, I would also discuss the limitations based on the chosen design of user testing.
A: We’ve added the explanation and brief discussion on the choice of equipment that we have used in the Study Limitations. In our case it may be assumed that Oculus Quest 2 represent the best case scenario for practical reasons. Still, it may be especially important to test the assumptions that desktop environments can also be used to access WebXR. We also point out this in Conclusion section that was added to the paper (See also answer below).
I recommend adding a "Conclusions" section, which will briefly summarize the results achieved and indicate possible directions for future work (based on the topics discussed in the previous two sections). The paper will also not end in a slightly negative spirit, which represents the limitations of the conducted research.
A: Good point. As requested we have added the brief Conclusions that describe the results and provide future avenues of research
Other minor comments
Authors should correct the formatting of some paragraphs (e.g., 1st paragraph in the introduction) and remove some spaces in the text (e.g., before Table 4).
A: Corrected – with the added note that this is the result of the incosistences in the way MDPI template is being read by various editors. We hope that this will be corrected in full in the publishing process.
Page 2, row 51: I recommend following the exact term "Virtual Geographic Environments "(including word order).
A: Of course! It was an editing mistake. Corrected.
Table 2: I do not know what the letter K in the column "Gender "means. The terms male-female (M – F) or men-women (M – W) are usually used.
A: Corrected to Male/Female.
Page 11, row 421: There is probably a typo at the beginning of this row.
A: That is true. Corrected.
Reviewer 2 Report
This is a really nice paper that gives insight in the use of XR in spatial planning.
I would advise the authors to translate in English the text that appears in the
application as the use of polish is a barrier to international readers.
Specific remarks:
Please explain "try to load the model in the places indicated by the tester' in Table 3 line 1
Line 421: Delete "A"
Line 425: change reference as expected
Good luck in publishing your paper
Author Response
Dear Reviewer,
Thank you for taking time to read and review our paper We have corrected the paper and updated the application according to your suggestions Below you will find our detailed answers for each point that you have made.
Best Regards and thank you!
Comments and responses below:
I would advise the authors to translate in English the text that appears in the
application as the use of polish is a barrier to international readers.
A: That is true. The text withing the full application has been translated into English. Additional figure has also been added to the text to further present the environments in use.
Please explain "try to load the model in the places indicated by the tester' in Table 3 line 1
A: The translation of the instruction has been corrected.
Line 421: Delete "A"
A: Corrected.
Line 425: change reference as expected
A: Corrected with numeric link.
Reviewer 3 Report
Interesting introduction and analysis of the relative literature, Nevertheless I would like to see more clearly the opinion of the authors of what is AR, VR XR, and MX in spatial planning by the example.
Nice idea in general, some parts are really original.
More images/ pictures would help a lot.
In the end, I have a "why" question... What are the advantages of the proposed method in comparison to the 3D in a GIS of region after and before a specific spatial intervention? or even that is VR?....
Author Response
Dear Reviewer,
Thank you for taking time to read and review our paper We have corrected the paper according to your suggestions Below you will find our detailed answers for each point that you have made.
Best Regards and thank you!
Comments and responses below:
(...)I would like to see more clearly the opinion of the authors of what is AR, VR XR, and MX in spatial planning by the example.
A: We have added more clear definitions of the components if the spectrum and provided brief explanations of what they may be used for withing urban planning.
More images/ pictures would help a lot.
A: We agree. We have added two figures that show VR environments in more detail.
In the end, I have a "why" question... What are the advantages of the proposed method in comparison to the 3D in a GIS of region after and before a specific spatial intervention? or even that is VR?.…
A: This is a very good point. To elaborate on this we have added the following explanation to the section 1.2 of the paper: “XR environments in comparison to non-immersive 3D visualisation often used in urban planning offer some advantages. The most important one is that through immersion they can provide the correct circumstances to induce the feeling of spatial presence – being in another location. This can in turn emotional engage the user, which is crucial in participatory planning and help with understanding of the changes in the urban space. For example it can be much easier for users to perceive scale in XR.”
Round 2
Reviewer 1 Report
All my suggestions have been responded to in the text of the paper and explained in the provided cover letter. The clarity of the paper has increased. I only have the following formal comments for the provided second version of the manuscript:
- Page 3, row 122: There is probably a typo („Another“).
- Page 7: Blank lines between figures.
