Factors Influencing the Effectiveness of Serious Gaming in the Field of Vocational Orientation

Round 1

Reviewer 1 Report

A very well constructed and conducted study, the results are discussed and interpreted appropriately. 

Going into great detail on its design and statistical analysis often makes it difficult for readers to perceive.

Author Response

Dear Reviewer 1

Thank you for your efforts and kind comments in the review.

We have tried to describe the research design in as much detail as possible so that readers can understand how we adapted the scales. In this respect, we hope to offer other researchers the opportunity to use our findings as well as measurement tools (i.e., our scale) for similar research projects.

Author Response File: Author Response.pdf

Reviewer 2 Report

 

This is a neatly written and clear paper that reports an interesting experimentation with the like2be serious game. The paper has merit, but I have 3 concerns:

1. The authors seems to be very expert in research statistics, but much less in serious game design. They tend to assume that serious games are only computer-based (line 73) and that they are equivalent to gamified learning methods (176). I suggest revising the concepts in the theoretical framework to match the current discourse about serious games considering for example the educational escape room strand (domain where typically digital and physical blend).

2. The study is quasi experimental, but readers are provided very little information about the control group. As far as I can read, the control group simply "did not play like2be". Were they offered similar or equivalent activities? With what content or approach? Describing non-treatment groups in education is paramount, otherwise we will not be able to interpret the results. For example, the test consisted in questions about all the job profile in the like2be game; did the control group have an opportunity to learn about all these profiles as well?

3. In the same way, very little information is provided about the game and its playing setting. We learn it is a point-and-click game with no narrative and that was not particularly enjoyable (!!) - but we do not know if there were scores or a roster, or how long sessions lasted. Also, a screenshot might help understand the flavour of the game. Moreover, did students chose to play or was it mandatory? Where teachers observing, providing feedback or debriefing? These elements definitely impact the learning outcomes.

4. A major part of the paper is about how the researchers adjusted the model to find a suitable fit, which they only partially found, using already existing scales. However, the manipulations they did were not marginal, and the results are not that strong, even if they can be meaningfully interpreted. While this does not subtract merit to the paper, I read another story here, that I think should be addressed by the authors: the fact that games are not equal. The authors took scales from other studies designed for other games, and find that the scales do not fit. Might this be because the games (so, the very object of the study!!) were different? For example, they decided to drop the "control of the game" subscale, because they found it irrelevant in a point-and-click game - but why did they consider it in the first place with that kind of game? In other words, while reading I had the impression that the effort was put in making the statistics work, and not on actually understanding the research object (the game and the students). This reflects the scarce information about the game itself (point 3). A lot of research on GBL (and in education in general) work this way, as if the nature of the game (or of the instruction) was irrelevant in selecting research and evaluation instruments. I think a critical reflection on this would be important.

Another general critical remark: a game that is not fun can still teach, but is it a good game? Or just a way to propose a colourful drill exercise?

Finally, I would strengthen the game design conclusions and make them more concrete. What does this study say about how we can improve like2be or design new serious games for vocational orientation?

Minor issues (by line):

Abstract: for readers not familiar wiith the Swiss education system, please indicate this study takes place in lower secondary education (not just secondary).

115: "Adventure games" is a more common expression

153-166 these lines seem to repeat what was already said before.

434-436 this sentence repeats the previous one

Figure 1: increase font size, it would be more legible

509 You mention here attention capacity, short-term and visual memory... why would you have measured them? I understand that the paper referenced here did, but this study has a totally different focus, so I find this passage weird.

540 double space before "Furthermore"

571 the expression "additional instructional materials such as reflection, modelling, ..." is sloppy. Instructional materials can support reflection, but cannot be reflection. I read this as another confirmation of the scarce attention paid to the actuality of the game and of its embedment into a concrete learning situation. 

Author Response

Dear Reviewer 2

Thank you for your constructive criticism. We are happy to address your comments and suggest our adjustments (please see the attachment). 

Author Response File: Author Response.pdf