Next Article in Journal
Exploring an Affective and Responsive Virtual Environment to Improve Remote Learning
Previous Article in Journal
Cybersickness in Virtual Reality Questionnaire (CSQ-VR): A Validation and Comparison against SSQ and VRSQ
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Virtual Worlds
Volume 2
Issue 1
10.3390/virtualworlds2010003
virtualworlds-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Comparison Study on the Learning Effectiveness of Construction Training Scenarios in a Virtual Reality Environment

by
Yee Sye Lee
1,
Ali Rashidi
2,*,
Amin Talei
1,
Huai Jian Beh
1 and
Sina Rashidi
3
1
Civil Engineering Discipline, School of Engineering, Monash University Malaysia, Selangor 47500, Malaysia
2
Future Building Initiative, Monash University, Melbourne, VIC 3145, Australia
3
Department of Architecture, Faculty of Engineering, Azad University-Pardis Branch, Tehran 1658174583, Iran
*
Author to whom correspondence should be addressed.
Virtual Worlds 2023, 2(1), 36-52; https://doi.org/10.3390/virtualworlds2010003
Submission received: 10 December 2022 / Revised: 13 January 2023 / Accepted: 28 January 2023 / Published: 2 February 2023
Browse Figures
Versions Notes

Abstract

:
While VR-based training has been proven to improve learning effectiveness over conventional methods, there is a lack of study on its learning effectiveness due to the implementation of training modes. This study aims to investigate the learning effectiveness of engineering students under different training modes in VR-based construction design training. Three VR scenarios with varying degrees of immersiveness were developed based on Dale’s cone of learning experience, including (1) Audio-visual based training, (2) Interactive-based training, and (3) Contrived hands-on experience training. Sixteen students with varying backgrounds participated in this study. The results posit a positive correlation between learning effectiveness and the degree of immersiveness, with a mean score of 77.33%, 81.33%, and 82.67% in each training scenario, respectively. Participants with lower academic performance tend to perform significantly better in audio-visual and interactive-based training. Meanwhile, participants with experience in gaming tend to outperform the latter group. Results also showed that participants with less experience in gaming benefited the most from hands-on VR training. The findings suggest that the general audience retained the most information via hands-on VR training; however, training scenarios should be contextualized toward the targeted group to maximize learning effectiveness.

1. Introduction

Construction hazards leading to fatalities have been a major concern in the construction industry for decades, as it remains one of the highest death rates among the occupations in the workforce [1,2]. Several factors are associated with these incidences occurring on the construction site, primarily led by human factors, such as unsafe behaviors, insufficient education requirements, and incompetency among construction personnel [3,4]. Prior studies showed that realistic training scenarios successfully reduced the frequency of construction accidents, delivering safety-related information effectively [5,6]. Nevertheless, exposure to on-site risks and high deliverance costs has hindered the deployment of on-site training methods [7,8]. Furthermore, advanced visualization technologies are bound to revolutionize construction practitioners’ methods of information and knowledge exchange and provide an opportunity for trainees to get first-hand experience of a construction site to prepare and equip them for future jobs [9,10]. In this regard, with the rapid advancement of VR technology, the adoption of VR technology in construction training gained popularity, as it similarly presents information to on-site training while eliminating the aforementioned risks [11]. While VR-based construction training proved to be a more effective method in memory retention and recall over conventional training methods, there is a lack of study on the impact of the deliverance method within the virtual environment on the learning effectiveness of users. Different types of VR-based construction training reported substantial variance in the improvement of learning effectiveness, which suggests an underlying factor affecting learning effectiveness due to the development process of VR training [12,13,14,15]. Hence, it is crucial to investigate the drawbacks of VR training scenarios under different constraints regarding memory and knowledge retention. It is essential to address some of the uncertainties present in the knowledge gap of VR training development to maximize its benefits for learners.
This study aims to (1) improve the learning effectiveness of VR-based construction training through passive and active approaches; (2) compare the performance of engineering students of varying demographics and academic performance under different VR-based construction training scenarios; and (3) propose development directions to maximize the benefits of VR-based construction training. To achieve the aim of this study, a research question was formulated as follows: What are the implications of virtual construction training scenarios under varying immersiveness conditions on engineering students of different demographics?
In this study, three different VR training scenarios were developed based on Edgar Dale’s cone of experience relating to real-world applications of learning methods. It is hypothesized that as the training approach moves from passive to active methods, the learning effectiveness should improve overall. The first VR scenario is developed to deliver information through an audio-visual-based training approach, which is considered a passive method. Engineering students are taught based on a virtual instructor showing students building components while delivering a speech in the virtual environment. The second VR scenario is conducted through a semi-passive approach, which is through an interactive-based training approach. In this case, students are given opportunities to interact with the building components during the learning process, and animations are simulated in the virtual environment to aid in the visualization process. The third VR scenario utilizes an active approach, which is known as a contrived hands-on experience training. In addition to providing interaction with the building components, students are required to perform hands-on tasks to complete the training. At the end of each scenario, students are required to undergo a quiz to test their knowledge retention and recall. The results are then analyzed and critically discussed in further sections.

2. Background

2.1. Virtual Reality Technology in Construction Engineering Education

The use of VR technology has steadily gained popularity in construction engineering education since decades ago, long before semi or fully immersive virtual techniques were introduced [16,17,18]. The advancement of VR technology has proven its effectiveness in delivering knowledge to learners in the construction industry, primarily due to the realistic reproduction of the construction site in a virtual environment [19,20,21]. VR’s ability to simulate the construction site is highly advantageous over conventional training methods as it removes the on-site construction hazards and risks, which would otherwise be exposed to the learners [22,23]. Aside from that, VR-based construction training can mitigate the high costs of real-world training, further incentivizing the deployment of VR-based training [14,24,25].
There is a vast area of VR-based applications in the construction industry, especially in education and training. Building information modeling (BIM) technology is commonly integrated with VR to facilitate information flow between the educational game and building components [21,26]. Various VR-based educational training has been developed over the years, covering different aspects of the construction industry, such as construction safety [27,28,29], operation of construction equipment [30], construction process [31], construction management [32], and so on. The wide application of VR technology in construction education is mainly attributed to the highly interactive learning environment, providing engagement and motivation for learners to focus between training sessions [33,34]. In addition, VR technology also serves as an effective collaboration platform due to its excellence in representing the spatiotemporal dimensions [35,36]. Therefore, there is strong evidence of the benefits of VR in construction education applications [37]. Learners who were trained using the VR platform also generally performed better over conventional methods, as VR simulates hands-on experience [38]. However, some research found that there are no significant differences between VR and conventional methods, despite the strong beliefs of the participants [39]. Besides that, VR provides an improved spatial perception of the environment over conventional and non-immersive VR (e.g., desktop VR) methods [40,41]. Nevertheless, most research was conducted independently on the virtual environment or on a comparison study against conventional training methods [42,43]. Despite research reporting variance in the learning outcomes, there is insufficient knowledge of the factors affecting these results. Hence, this study posits that comparing different developmental approaches toward VR-based construction educational training allows further insights to verify these occurrences and understand their rationale.

2.2. Critical Factors Affecting the Learning Effectiveness of VR-Based Educational Training

Language is considered one of the most basic tools in delivering information in both real and virtual environments, whether through text or speech [44]. It acts as a medium for action, which is crucial in the learning process [45]. Therefore, an effective speech system is necessary to achieve high learning effectiveness in a training environment. In the context of VR-based training, this can be achieved by implementing audio speech or in combination with subtitles. Audio and visual stimuli in an immersive environment also act as a cue for an action or reaction, providing engagement with the users [46,47]. Therefore, visual and sound FX is considered an essential aspect of game development as it maintains the game’s momentum [48]. Gestures are also deeply integrated with cognitive capabilities, representing emotions in a dialogue [49]. Hence, the implementation of gestures is indispensable in creating an interactive environment, which is achieved by animations in a VR game. In the educational theory on the learning process, it is argued that learning through actions is the basic building block of humanity [50]. It is further expanded that the conduct of acts is a highly effective tool for learning new information [51]. Thus, through the simulation of a hands-on experience in the virtual environment, VR contributes to a natural and effective learning method for the users.

3. Methodology

The experimental design of this study is presented in this section. The details of the participants, experimental design, and comparative evaluation is discussed in each subsection.

3.1. Participants

A total of 32 engineering students from the civil, mechanical, and electrical departments of Monash University Malaysia were selected to participate in this study. All participants were in their final year of study (the age of participants are roughly equal), and no monetary or non-monetary incentives were given out during the study. The participants were split equally into two groups in a random order, whereby 16 students were each separated into the control and test group. The sample size consists of 22 civil, 6 mechanical, and 4 electrical engineering students, whereby the gender distribution of the sample size consists of 24 males and 8 females. A study conducted by Wang and Dunston showed that a sample size of 16 was sufficient to obtain a reliable result [52]. Another VR education system conducted in another study also proved reliable and consistent results using a sample size of 25 [53]. A comparison study related to VR applications in the assembly process also showed reliable results with a sample size of 15 [54]. The control group was used to ensure similar difficulties between three different VR scenarios and was verified through a paper-based study to verify and minimize persisting biases across all the VR scenarios. Meanwhile, the test group underwent all three developed VR scenarios, and the collected data were used to analyze students’ learning effectiveness under different VR training approaches.

3.2. Experimental Design

In this study, a quantitative approach is designed and applied to evaluate and analyze the results, as suggested by Creswell [55]. All participants from the test group are subject to three VR scenarios consecutively to minimize internal and external factors affecting the results. The VR scenarios are also designed to have the same scoring system to eliminate biases. Besides that, the type, size, and color of the user interface and graphical objects used in the testing process persist across all VR scenarios. The content of each VR scenario also shares a very similar topic and is further adjusted to ensure similar difficulties through the control group. The control group is subject to a paper-based method, whereby the learning process is conducted through a physical-based lecture on the same content as the developed VR scenarios. Then, it is immediately subjected to a multiple-choice quiz containing the same questions as the VR scenarios. Meanwhile, the demographics, assessment scores, and completion time of the participants from the test group are recorded and exported into a Microsoft Excel spreadsheet for further analysis.

3.3. Comparative Evaluation

The evaluation measures are designed based on the Charette Test Method, using a scoring and time measurement system [56]. At the end of each VR scenario, a quiz consisting of five questions, each carrying the same weightage, is provided in the virtual environment. Every question comprises three to four multiple choices and does not contain multiple correct answers. During the questioning phase, the timer is paused while the virtual instructor reads out the question. This ensures that the time measurement system is not affected by the length of the questions. Before the start of training, participants are required to fill in a pre-test demographics survey, identifying the participants’ engineering disciplines, academic performance, experience in gaming, and gender. An external validation process is adopted in this study based on the charette test method and similar past research, which consists of three criteria: effectiveness, speed, and usability [56,57]. These three criteria are used to establish a relation between the three VR scenarios developed based on Dale’s cone of experience [58], as shown below in Figure 1.
The cone of learning as the visual metaphor for types of learning content from the concrete learning experience at the bottom of the cone engaged more senses of the learner such as hearing, seeing, touching, etc. than the abstract experience such as seeing the words or hearing the instructions. The report by Fadel et al., (2008) supported Edgar Dale’s Cone of Experience/Learning which represents the tangible reality of learning experience first-hand [59]. It is demonstrated that the least effective methods to learn new tasks are by means of reading a manual or instructions and listening to a presentation. In contrast, the most effective method of learning experiences involves doing a physical task and simulating the real task, which is called learning by doing. In this regard, the construction training scenarios in a VR environment are designed in three different stages according to Edgar Dale’s cone of learning.

4. Design Framework and System Description

4.1. Design Framework

A design framework was developed for this study to systematically design and verify the VR scenario’s integrity, shown below in Figure 2.
The topic selected in this study is the building components of a typical two-story residential townhouse with a Light Steel Framing system. Through VR-based training, learners are expected to gain insights into the key components of the building and its associated functionality from four different aspects, including the structural, mechanical, electrical, and plumbing point of view. After each training scenario, the external validation test was conducted immediately to assess the learners’ memory retention and recall time capabilities. The learning topic slightly differs for every scenario to eliminate memory dependency from any previously learned topic.

4.2. Game Development

The VR training is developed on Unity® 2019.3.7f1 version to support the various types of VR devices such as Personal Computer-based VR and stand-alone VR headsets. All BIM-based building components were imported from Autodesk® Revit® as FBX files, while the material and texture files were converted using Autodesk® 3DS Max®. The building components consist of light steel framing (LSF) system, one of the commonly used industrialized construction systems. The developed VR training scenarios are supported on the SteamVR platform only. The game is tested on a few different head-mounted displays (HMDs), including HTC Vive, Valve Index, and Oculus Quest 2 through Oculus Link. Nevertheless, the actual test is conducted on HTC Vive. Oculus Quest 2 was not used as there is some network latency, which could cause nausea, leading to an external factor affecting the results [40]. Meanwhile, Valve Index was not used during the actual tests as it weighs roughly 50% more than HTC Vive, which may lead to user discomfort. The game supports up to 144Hz to minimize flickering while maximizing user comfort. The computer specifications used in the development and testing stages are Nvidia GeForce RTX™ 3090, AMD Ryzen™ Threadripper™ 3990x, 128GB RAM, and 2TB SSD. The game utilizes a teleport-based movement system, and interaction is performed by grabbing, hovering, or colliding. A scene manager system is developed through visual scripting, handling different events, including audio clips, dialogues, subtitles, animation, visual effects, triggers, and game flow. A time and score manager system is also integrated into the game, which mainly interacts with the scene manager system to record data within the game. The game development times of the audio-visual, interactive, and contrived hands-on training scenarios were five, three, and three weeks, respectively. The relatively longer development time of the audio-visual training scenario was due to the initial development of movement, interaction, and manager system, which would subsequently be reused in the other training scenarios.

4.3. Audio-Visual-Based VR Training

The focus of this training is to deliver information through an audio-visual-based approach. In this scenario, users started outside of the residential house. A virtual instructor begins by introducing the LSF system used in the building, as shown in Figure 3a, including some structural components and their associated function. Afterward, the user is required to enter the building. The virtual instructor follows the user and then begins explaining the wall system used in the building, such as material, thermal and acoustic properties. Throughout the scenario, the user and environment interaction is minimal. The virtual lecturer is fully rigged with animation, and the speech is delivered through both audio recordings and subtitles. Some visual effects are provided to facilitate the explanation process, as shown in Figure 3b. The user is then required to participate in a quiz in the virtual environment. The quiz consists of 5 questions in total, whereby 3 questions are on the topic of the LSF system, and 2 questions are on the topic of a wall system.

4.4. Interactive-Based VR Training

The philosophy of this training method is to approach users with a highly interactive environment. The training begins by introducing some key components and the functionality of the electrical system inside the house. Then, the virtual instructor asks the user to proceed to the second floor of the building. The virtual instructor briefly discusses some implementations of smart technologies in a residential house, such as smart light bulbs. Afterward, the user is required to enter a bedroom. The virtual instructor explains the application of the heating, ventilation, and air conditioning (HVAC) systems and the factors that need to be considered during the design stage. An example of an HVAC system used in this training is the chilled water system. Lastly, the virtual instructor presents the piping system to the user, including the main water supply, greywater, and blackwater system. Throughout the scenario, the virtual environment is highly interactive in addition to the basic functionality introduced in the audio-visual-based training. For instance, users are required to open and investigate the electrical system, as shown in Figure 4a. An automatic user detection system activates or deactivates the light bulbs whenever the user enters or leaves the room. The users are also able to visualize the air ventilation in the building, as shown in Figure 4b. Aside from that, users can visualize the supply of clean water and its purpose, as shown in Figure 4c. Besides that, all HVAC and piping systems are fully animated to provide a highly immersive environment, as shown in Figure 4d. After the training, the user is required to answer five questions related to the learned topic, one on the electrical system, two on the HVAC system, and the remaining two on the water piping system.

4.5. Contrived Hands-On Experience VR Training

This training is designed to deliver knowledge through performing hands-on tasks. In addition to the functionality of the interactive-based training, hands-on tasks are also introduced in this scenario. The training begins with an introduction to the roofing system used in this building, which is a warm deck-pitched roof. Then, the user is allowed the freedom to obtain further information by interacting with the components freely. Afterward, the user is briefed on the safety measures before performing maintenance tasks. After the user completes the safety tasks, the virtual instructor teaches the user about piping maintenance methods. The user is required to perform hands-on tasks, such as unscrewing bolts, replacing gaskets, and so on. The virtual environment is highly interactive throughout the scenario, and a contrived hands-on experience is introduced whenever possible. For instance, Figure 5a shows the user removing the pipe from the connections, one of the many hands-on tasks required to complete the training. Meanwhile, Figure 5b shows the user performing a contrived hands-on task from a real-world perspective. After completing all hands-on tasks, the user is required to undergo a quiz. The quiz consists of five questions, where three are related to the roofing system, and two to the safety and maintenance works.

5. Results and Discussion

This section highlights the study’s findings and provides an in-depth analysis of the results.

5.1. Overall Performance Comparison between All VR Training Scenarios

The comparison of the participants’ overall performance between three developed VR scenarios is evaluated based on two criteria: the quiz performance score obtained in each scenario and the time required to complete it. Figure 6 shows the mean score of the participants in all VR training scenarios.
From Figure 6, it is seen that participants scored 4% higher in the interactive-based training compared to the audio-visual-based training, which is equivalent to a 5.2% improvement over the audio-visual-based training. Therefore, an inference that a highly interactive virtual environment positively reinforces learning effectiveness can be established from the results. The improvement is likely due to a stronger engagement between the users and the training through a higher degree of immersiveness. Aside from that, the incorporation of contrived hands-on tasks in the virtual environment further led to an improvement of 1.3% in performance scores over interactive-based training. The performance gain from implementing a contrived hands-on experience in the VR training is less significant when compared to the performance improvement upon implementing a highly interactive environment. However, a one-way analysis of variance (ANOVA) test with a 95% confidence level showed that the effect of the training scenarios on the participants’ performance score is insignificant, with F = 0.59 (DoF = 2) and p-value = 0.55. Although the statistical test showed that there are no significant differences between the mean performance under different training scenarios, it should be noted that the difficulty of each training scenario was scaled to be roughly equal. Hence, it is expected that the mean performance score should be roughly equal as well. Nevertheless, the implementation of a contrived hands-on experience benefits through memory recall time improvements, as shown in Table 1.
From Table 1, it is seen that there is a decreasing trend in the mean completion time as the virtual environment becomes increasingly immersive. This further strengthened the methodology of the developmental approach, strongly relating the phenomenon to the real-world application of learning approaches. The mean completion time decreased by approximately 8% when comparing the interactive-based training against the audio-visual-based training, showing an improvement in memory recall rate. There is also a further improvement of 3% when contrived hands-on experience is exposed to the participants. An observation is made that the standard deviation and the maximum value of the completion time on contrived hands-on experience training are exceptionally high. This is likely due to the complexity of the training scenario, whereby users are required to perform hands-on tasks. Therefore, users who are unfamiliar with the controls may be overwhelmed by the situation, leading to confusion or incapability to retain information properly. A one-way ANOVA test with a 95% confidence level conducted on the completion time between three scenarios showed that F = 3.18 (DoF = 2) and p-value = 0.05.

5.2. Demographic Factors Affecting the Performance Score of the VR Training

A few key demographic factors showed varying learning behaviors between the groups. Figure 7 shows the mean performance score of different weighted average mark (WAM) groups in all VR training scenarios, whereby WAM is a scoring system used at Monash University to calculate students’ academic performance. The WAM system is calculated based on the following formula:
W A M = ∑ C P y = 1 × P y = 1 × 0.5 + ∑ C P y = 2 , 3 , 4 × P y = 2 , 3 , 4 ∑ C P y = 1 × 0.5 + ∑ C P y = 2 , 3 , 4
Whereby C P y is the credit points of the units undertaken during their respective years, and P y is the marks obtained for the units undertaken during their respective years. The WAM score of each student ranges from 0–100, whereby the WAM group with a score of 60–69, 70–79, and 80+ falls under the category second class honors division B, second-class honors division A, and first-class honors, respectively. In the sample size of this study, there are no students who held a WAM score of 90 and above.
As can be seen in Figure 7, there is a performance score improvement across all WAM groups when comparing the interactive-based training against the audio-visual-based training. This showed that a highly interactive virtual environment is always preferred to maximize the learning effectiveness of the users. However, there is a mixed response from different WAM groups when subjected to contrived hands-on experience training. Participants from the lowest academic performance group tend to perform worse when a hands-on training approach is utilized during the learning process. A one-way multivariate analysis of variance (MANOVA) test using Pillai’s Trace showed that p-value = 0.76, F (12,36) = 0.67, and Pillai’s V = 0.55, indicating that there are no significant differences between the performance of participants and each academic group. Nevertheless, participants stated that it was generally more difficult to focus on the auditory (audio recordings) and visual (subtitles) feedback as they were distracted by the given hands-on tasks, which may be affecting the performance score. Meanwhile, participants from the middle academic performance group retained knowledge more efficiently through contrived hands-on experience training. Participants felt that practicing this method is highly effective in maintaining attention, which contributes to higher memory retention capabilities. Meanwhile, a t-test conducted between male and female participants showed a p value of 0.90, indicating that there are no significant differences between the mean performance between the gender groups. Aside from that, users’ experience in gaming notably affects the learning effectiveness of VR training in general as well, as shown in Table 2.
The obtained results were further categorized using the demographics survey. Participants with a gaming frequency of less than once a week, regardless of the gaming platform, were categorized as “rarely/sometimes”. Meanwhile, the latter group with a gaming frequency of more than once a week is considered as “often” regarding the gaming experience. Participants with more experience in gaming completed the training at a faster rate (50 s faster) and scored better (21% higher) at the same time. Participants who are familiar with gaming expressed comfort and ease of navigation in the virtual environment. Among the participants with less experience in gaming, it is generally agreed that clarity and ease of use are very significant in navigating the virtual environment.

5.3. System Usability of the VR Training Scenarios

The system usability of the developed VR training scenarios was also evaluated based on the performance score and is tabulated in Table 3.
The system usability score is based on a maximum score of 5, which is derived from the evaluation of the quiz performance. From Table 3, it is seen that the system usability score across all VR training scenarios is above 3. Aside from that, the minimum score obtained in all training scenarios is above 3 as well. This indicates that all participants successfully completed each training scenario, which suggested that the developed VR training scenarios were highly usable. Nevertheless, there is a mixed reaction toward different stimuli across participants with different demographics. Although there is a noticeable difference in the usability of the training scenarios with increasing immersion, it should be noted that there are numerous factors that were not considered in this study, such as the impact of spatial perception characteristics on the usability of the training scenarios.

5.4. Suggestions for VR Training Development Guidelines

As discussed in the previous subsections, it is evident that the learning effectiveness of VR-based training improves as the user receives a higher degree of immersiveness. Therefore, a VR learning effectiveness pyramid is suggested, as shown in Figure 8.
From Figure 8, it is suggested that adding the functionalities in VR-based training results in a higher degree of immersiveness, leading to higher learning effectiveness. Nevertheless, this developmental approach is generalized and should only be viewed as a theory. The findings discussed in the previous subsections also showed that some functionalities adversely affect certain demographic groups. Therefore, it is essential to identify the target audience before developing the VR training. After identifying the background of the targeted group, the VR training should be contextualized and personalized to maximize its learning effectiveness. Nevertheless, the results can only be generalized after repeating the study with a larger sample size.

6. Conclusions

There is currently insufficient knowledge on how the development process of VR-based training affects the learning effectiveness of users, despite research reporting significant variance in memory retention and recall capabilities in developed VR-based training. Most research tends to focus on comparing paper-based and VR-based methods, which leaves a research gap in the comparison between different VR-based methods. Therefore, this study developed three different VR-based training scenarios, varying from passive to active approaches, to further identify the critical success factors affecting the learning effectiveness of VR training. The training scenarios are developed based on Dale’s cone of learning as a guideline, whereby each subsequent scenario contained an increasing degree of immersiveness. The developed training scenarios were comparatively evaluated through performance scores and completion time, whereby participants were required to partake in a virtual quiz at the end of each scenario. The findings showed that there is a positive correlation between the degree of immersiveness and learning effectiveness. However, the demographics of the participants play a significant role in dictating the outcome of the results. For instance, participants with lower academic performance reacted adversely to contrived hands-on experience training. A learning effectiveness pyramid is also proposed in this study as a general guideline in the development of VR-based training. Nevertheless, VR-based training should be contextualized based on the background of the targeted group to maximize its benefits. A limitation of this comparison study includes the limited sample size to draw a definitive conclusion. Besides that, the user cognitive load is not measured, despite the difference in the complexity of workload between the three VR scenarios. In addition, factors affecting user performance such as user spatial perception were not considered in this study. Further studies should be made by incorporating different implementation combinations of learning methods to better understand how developmental approaches affect learning outcomes. Aside from that, future work will study the effects of other factors on user learning performance, particularly on the cognitive load of users when subjected to different stimuli using an electroencephalogram (EEG).

Author Contributions

Conceptualization, A.R.; methodology, A.R.; software, Y.S.L.; validation, H.J.B. and Y.S.L.; formal analysis, Y.S.L.; investigation, H.J.B.; resources, A.R. and A.T.; data curation, Y.S.L. and H.J.B.; writing—original draft preparation, Y.S.L.; writing—review and editing, A.R. and A.T. and S.R.; visualization, Y.S.L.; supervision, A.R. and A.T.; project administration, A.R.; funding acquisition, A.R. and A.T. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki, and approved by the Ethics Committee of Monash University (Review Reference 2020-23844-48677).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to protecting participants privacy under Monash University Human Ethics with project reference number 23844.

Acknowledgments

The authors would like to acknowledge the support of the School of Engineering at Monash University Malaysia and last but not least special thanks to the Departments of Architecture and Civil Engineering at Monash University Australia.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

References

  1. Work-related fatal injuries in Great Britain, Health and Safety Executive. 2022. Available online: https://www.hse.gov.uk/statistics/pdf/fatalinjuries.pdf (accessed on 7 December 2022).
  2. Crane-Related Work Deaths Trended Down from 1992 to 2017, U.S. Bureau of Labor Statistics. 2019. Available online: https://www.bls.gov/opub/ted/2019/crane-related-work-deaths-trended-down-from-1992-to-2017.htm (accessed on 7 December 2022).
  3. Premji, S.; Smith, P.M. Education-to-job mismatch and the risk of work injury. Inj. Prev. 2012, 19, 106–111. [Google Scholar] [CrossRef] [PubMed]
  4. Hide, S.; Atkinson, S.; Pavitt, T.C.; Haslam, R.; Gibb, A.; Gyi, D. Causal Factors in Construction Accidents. 2003. Available online: https://repository.lboro.ac.uk/articles/online_resource/Causal_factors_in_construction_accidents/9353051 (accessed on 7 December 2022).
  5. Cuqlock-Knopp, V.G.; Wilkins, C.A.; Torgerson, W.S. Multiple cue Probability Learning and the Design of Information Displays for Multiple Tasks. In Multiple Task Performance; CRC Press: Boca Raton, FL, USA, 2020; pp. 139–152. [Google Scholar] [CrossRef]
  6. Lapinski, M.K.; Rimal, R.N. An Explication of Social Norms. Commun. Theory 2005, 15, 127–147. [Google Scholar] [CrossRef]
  7. Fang, Y.; Teizer, J.; Marks, E. A Framework for Developing an As-built Virtual Environment to Advance Training of Crane Operators. In Proceedings of the Construction Research Congress 2014, Atlanta, GA, USA, 19–21 May 2014. [Google Scholar] [CrossRef]
  8. Getuli, V.; Capone, P.; Bruttini, A.; Isaac, S. BIM-based immersive Virtual Reality for construction workspace planning: A safety-oriented approach. Autom. Constr. 2020, 114, 103160. [Google Scholar] [CrossRef]
  9. Ibrahim, R.; Rashidi, A.; Said, N.S.; Othman, M.S. Engaging Capability Training in Serious Game Technology for Delivering Industrialized Construction. In Proceedings of the 2014 International Conference on Computing in Civil and Building Engineering, Orlando, FL, USA, 23–25 June 2014. [Google Scholar] [CrossRef]
  10. Rashidi, A.; Rahinah, I. Computer-Aided Theoretical Model for Low Tech Laborers Capability Training in Industrialized Construction Industry. In Proceedings of the The 14th International Conference on Computing in Civil and Building Engineering, Moscow, Russia, 27–29 June 2012. [Google Scholar]
  11. Lee, Y.S.; Rashidi, A.; Talei, A.; Arashpour, M.; Pour Rahimian, F. Integration of deep learning and extended reality technologies in construction engineering and management: A mixed review method. Constr. Innov. 2022, 22, 671–701. [Google Scholar] [CrossRef]
  12. Beh, H.J.; Rashidi, A.; Talei, A.; Lee, Y.S. Developing engineering students’ capabilities through game-based virtual reality technology for building utility inspection. Eng. Constr. Archit. Manag. 2022, 29, 2854–2877. [Google Scholar] [CrossRef]
  13. Goulding, J.; Nadim, W.; Petridis, P.; Alshawi, M. Construction industry offsite production: A virtual reality interactive training environment prototype. Adv. Eng. Informatics 2012, 26, 103–116. [Google Scholar] [CrossRef]
  14. Sacks, R.; Perlman, A.; Barak, R. Construction safety training using immersive virtual reality. Constr. Manag. Econ. 2013, 31, 1005–1017. [Google Scholar] [CrossRef]
  15. Le, Q.T.; Pedro, A.; Park, C.S. A Social Virtual Reality Based Construction Safety Education System for Experiential Learning. J. Intell. Robot. Syst. 2014, 79, 487–506. [Google Scholar] [CrossRef]
  16. Shelbourn, M.; Aouad, G.; Hoxley, M. Multimedia in construction education: New dimensions. Autom. Constr. 2001, 10, 265–274. [Google Scholar] [CrossRef]
  17. Radford, A. Games and learning about form in architecture. Autom. Constr. 2000, 9, 379–385. [Google Scholar] [CrossRef]
  18. Toyoda, R.; Russo-Abegão, F.; Glassey, J. VR-based health and safety training in various high-risk engineering industries: A literature review. Int. J. Educ. Technol. High. Educ. 2022, 19, 42. [Google Scholar] [CrossRef]
  19. Vahdatikhaki, F.; El Ammari, K.; Langroodi, A.K.; Miller, S.; Hammad, A.; Doree, A. Beyond data visualization: A context-realistic construction equipment training simulators. Autom. Constr. 2019, 106, 102853. [Google Scholar] [CrossRef]
  20. Fumarola, M.; Poelman, R. Generating virtual environments of real world facilities: Discussing four different approaches. Autom. Constr. 2011, 20, 263–269. [Google Scholar] [CrossRef]
  21. Yan, W.; Culp, C.; Graf, R. Integrating BIM and gaming for real-time interactive architectural visualization. Autom. Constr. 2011, 20, 446–458. [Google Scholar] [CrossRef]
  22. Li, X.; Yi, W.; Chi, H.-L.; Wang, X.; Chan, A.P. A critical review of virtual and augmented reality (VR/AR) applications in construction safety. Autom. Constr. 2018, 86, 150–162. [Google Scholar] [CrossRef]
  23. Li, H.; Chan, G.; Skitmore, M. Visualizing safety assessment by integrating the use of game technology. Autom. Constr. 2012, 22, 498–505. [Google Scholar] [CrossRef]
  24. Zhang, Y.; Liu, H.; Kang, S.-C.; Al-Hussein, M. Virtual reality applications for the built environment: Research trends and opportunities. Autom. Constr. 2020, 118, 103311. [Google Scholar] [CrossRef]
  25. Song, H.; Kim, T.; Kim, J.; Ahn, D.; Kang, Y. Effectiveness of VR crane training with head-mounted display: Double mediation of presence and perceived usefulness. Autom. Constr. 2020, 122, 103506. [Google Scholar] [CrossRef]
  26. Goulding, J.S.; Rahimian, F.P.; Wang, X. Virtual Reality-based Cloud BIM Platform for Integrated AEC Projects. J. Inf. Technol. Constr. 2014, 19, 308–325. [Google Scholar]
  27. Chellappa, V.; Mésároš, P.; Špak, M.; Spišáková, M.; Kaleja, P. VR-based Safety Training Research in Construction. In Proceedings of the OP Conference Series: Materials Science and Engineering, Civil Engineering Conference (CEC 2022), Košice, Slovakia, 6–8 June 2022; Volume 1252, p. 12058. [Google Scholar] [CrossRef]
  28. Rokooei, S.; Shojaei, A.; Alvanchi, A.; Azad, R.; Didehvar, N. Virtual reality application for construction safety training. Saf. Sci. 2023, 157, 105925. [Google Scholar] [CrossRef]
  29. Hoang, T.; Greuter, S.; Taylor, S.; Aranda, G.; Mulvany, G.T. An Evaluation of Virtual Reality for Fear Arousal Safety Training in the Construction Industry. In Proceedings of the 021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Bari, Italy, 04–08 October 2021; pp. 177–182. [Google Scholar] [CrossRef]
  30. Juang, J.; Hung, W.; Kang, S. SimCrane 3D+: A crane simulator with kinesthetic and stereoscopic vision. Adv. Eng. Informatics 2013, 27, 506–518. [Google Scholar] [CrossRef]
  31. Sampaio, A.Z.; Martins, O.P. The application of virtual reality technology in the construction of bridge: The cantilever and incremental launching methods. Autom. Constr. 2014, 37, 58–67. [Google Scholar] [CrossRef]
  32. de Vries, B.; Verhagen, S.; Jessurun, A. Building Management Simulation Centre. Autom. Constr. 2004, 13, 679–687. [Google Scholar] [CrossRef]
  33. Yang, F.; Goh, Y.M. VR and MR technology for safety management education: An authentic learning approach. Saf. Sci. 2022, 148, 105645. [Google Scholar] [CrossRef]
  34. Wang, R.; Lowe, R.; Newton, S.; Kocaturk, T. Task complexity and learning styles in situated virtual learning environments for construction higher education. Autom. Constr. 2020, 113, 103148. [Google Scholar] [CrossRef]
  35. Azarby, S.; Rice, A. Scale Estimation for Design Decisions in Virtual Environments: Understanding the Impact of User Characteristics on Spatial Perception in Immersive Virtual Reality Systems. Buildings 2022, 12, 1461. [Google Scholar] [CrossRef]
  36. Bjørn, P.; Wulff, M.; Petræus, M.S.; Møller, N.H. Immersive Cooperative Work Environments (CWE): Designing Human-Building Interaction in Virtual Reality. Comput. Support. Coop. Work (CSCW) 2021, 30, 351–391. [Google Scholar] [CrossRef]
  37. Soliman, M.; Pesyridis, A.; Dalaymani-Zad, D.; Gronfula, M.; Kourmpetis, M. The Application of Virtual Reality in Engineering Education. Appl. Sci. 2021, 11, 2879. [Google Scholar] [CrossRef]
  38. Abidi, M.H.; Al-Ahmari, A.; Ahmad, A.; Ameen, W.; Alkhalefah, H. Assessment of virtual reality-based manufacturing assembly training system. Int. J. Adv. Manuf. Technol. 2019, 105, 3743–3759. [Google Scholar] [CrossRef]
  39. Poshdar, M.; Zhu, Y.; Ghodrati, N.; Alqudah, H.; Tookey, J.; Gonzáles, V.A.; Talebi, S. The Efficacy of Virtual-Reality Based Safety Training in Construction: Perceptions Versus Observation. In ECPPM 2021—eWork and eBusiness in Architecture, Engineering and Construction; CRC Press: Boca Raton, FL, USA, 2021. [Google Scholar]
  40. Paes, D.; Arantes, E.; Irizarry, J. Immersive environment for improving the understanding of architectural 3D models: Comparing user spatial perception between immersive and traditional virtual reality systems. Autom. Constr. 2017, 84, 292–303. [Google Scholar] [CrossRef]
  41. Gómez-Tone, H.; Escapa, J.B.; Escapa, P.B.; Martin-Gutierrez, J. The Drawing and Perception of Architectural Spaces through Immersive Virtual Reality. Sustainability 2021, 13, 6223. [Google Scholar] [CrossRef]
  42. Sampaio, A.Z.; Ferreira, M.M.; Rosário, D.P.; Martins, O.P. 3D and VR models in Civil Engineering education: Construction, rehabilitation and maintenance. Autom. Constr. 2010, 19, 819–828. [Google Scholar] [CrossRef]
  43. Han, Y.; Yang, J.; Diao, Y.; Jin, R.; Guo, B.; Adamu, Z. Process and Outcome-based Evaluation between Virtual Reality-driven and Traditional Construction Safety Training. Adv. Eng. Inform. 2022, 52, 101634. [Google Scholar] [CrossRef]
  44. Kilby, L.; Lennon, H. When words are not enough: Combined textual and visual multimodal analysis as a Critical Discursive Psychology undertaking. Methods Psychol. 2021, 5, 100071. [Google Scholar] [CrossRef]
  45. Molder, H.T. Discursive psychology. Handbook of Language and Social Interaction; Lawrence Erlbaum Associates, Inc.: Mahwah, NJ, USA, 2005; pp. 1–11. [Google Scholar]
  46. Haus, M.; Rooney, C.; Barnett, J.; Westley, D.; Wong, W. Evaluating the Effect of Startling and Surprising Events in Immersive Training Systems for Emergency Response. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2012, 56, 2467–2471. [Google Scholar] [CrossRef]
  47. Al-Ahmari, A.; Abidi, M.H.; Ahmad, A.; Darmoul, S. Development of a virtual manufacturing assembly simulation system. Adv. Mech. Eng. 2016, 8, 1687814016639824. [Google Scholar] [CrossRef] [Green Version]
  48. Suwanaposee, P.; Gutwin, C.; Cockburn, A. The influence of audio effects and attention on the perceived duration of interaction. Int. J. Human-Computer Stud. 2021, 159, 102756. [Google Scholar] [CrossRef]
  49. Roth, W. From gesture to scientific language. J. Pragmat. 2000, 32, 1683–1714. [Google Scholar] [CrossRef]
  50. Bruner, J. The Process of Education; Harvard University Press: London, UK, 1976. [Google Scholar]
  51. Bruner, J. Toward a Theory of Instruction; Harvard University Press: London, UK, 1974. [Google Scholar]
  52. Wang, X.; Dunston, P.S. Compatibility issues in Augmented Reality systems for AEC: An experimental prototype study. Autom. Constr. 2006, 15, 314–326. [Google Scholar] [CrossRef]
  53. Pedro, A.; Le, Q.T.; Park, C.S. Framework for Integrating Safety into Construction Methods Education through Interactive Virtual Reality. J. Prof. Issues Eng. Educ. Pr. 2016, 142, 4015011. [Google Scholar] [CrossRef]
  54. Abidi, M.H.; Ahmad, A.; El-Tamimi, A.M.; Al-Ahmari, A.M. Development and Evaluation of a Virtual Assembly Trainer. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2012, 56, 2560–2564. [Google Scholar] [CrossRef]
  55. Creswell, J.W. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 3rd ed.; SAGE Publications: Los Angeles, CA, USA, 2009. [Google Scholar]
  56. Clayton, M.; Kunz, J.; Fischer, M. The Charette Test Method; Stanford University, Centre For Integrated Facility Engineering: Stanford, CA, USA, 1998. [Google Scholar]
  57. Feinstein, A.H.; Cannon, H.M. Constructs of Simulation Evaluation. Simul. Gaming 2002, 33, 425–440. [Google Scholar] [CrossRef]
  58. Dale, E. Audiovisual Methods in Teaching, 3rd ed.; Education Resources Information Center: Washington, DC, USA, 1969.
  59. Fadel, C.; Lemke, C. Multimodal Learning Through Media: What the Research Says. 2008. Available online: https://www.cisco.com/c/dam/en_us/solutions/industries/docs/education/Multimodal-Learning-Through-Media.pdf (accessed on 7 December 2022).
Virtualworlds 02 00003 g001 550
Figure 1. Developmental approach of VR construction training scenarios based on Edgar Dale’s cone of experience.
Figure 1. Developmental approach of VR construction training scenarios based on Edgar Dale’s cone of experience.
Virtualworlds 02 00003 g001
Virtualworlds 02 00003 g002 550
Figure 2. Research design framework to evaluate the learning effectiveness of different VR scenarios.
Figure 2. Research design framework to evaluate the learning effectiveness of different VR scenarios.
Virtualworlds 02 00003 g002
Virtualworlds 02 00003 g003 550
Figure 3. Audio-visual-based training scenario. (a) Virtual instructor and LSF system; (b) Examples of visual effects used.
Figure 3. Audio-visual-based training scenario. (a) Virtual instructor and LSF system; (b) Examples of visual effects used.
Virtualworlds 02 00003 g003
Virtualworlds 02 00003 g004 550
Figure 4. Interactive-based training scenario. (a) User interacting with the electrical system; (b) Visualization of the air ventilation in the building; (c) User interacting with the main water supply system; (d) Fully animated HVAC and piping systems showing air and water flow directions.
Figure 4. Interactive-based training scenario. (a) User interacting with the electrical system; (b) Visualization of the air ventilation in the building; (c) User interacting with the main water supply system; (d) Fully animated HVAC and piping systems showing air and water flow directions.
Virtualworlds 02 00003 g004
Virtualworlds 02 00003 g005 550
Figure 5. Contrived hands-on experience training scenario. (a) User replacing the piping system; (b) Contrived hands-on task from the real-world perspective.
Figure 5. Contrived hands-on experience training scenario. (a) User replacing the piping system; (b) Contrived hands-on task from the real-world perspective.
Virtualworlds 02 00003 g005
Virtualworlds 02 00003 g006 550
Figure 6. Mean performance score based on different learning methods in the VR environment.
Figure 6. Mean performance score based on different learning methods in the VR environment.
Virtualworlds 02 00003 g006
Virtualworlds 02 00003 g007 550
Figure 7. Mean performance score of different WAM groups of participants when subjected to different VR training scenarios.
Figure 7. Mean performance score of different WAM groups of participants when subjected to different VR training scenarios.
Virtualworlds 02 00003 g007
Virtualworlds 02 00003 g008 550
Figure 8. Learning effectiveness pyramid in a virtual reality training. Modified from [59].
Figure 8. Learning effectiveness pyramid in a virtual reality training. Modified from [59].
Virtualworlds 02 00003 g008
Table
Table 1. Comparison of the quiz completion time between the developed VR training scenarios.
Table 1. Comparison of the quiz completion time between the developed VR training scenarios.
VR Training ScenarioSample Size, NMean Completion Time(s), (S.D.)% Completion, (S.D.)Max. Completion Time (s)
Audio-visual based training16519 (48)100 (0)611
Interactive-based training16479 (31)100 (0)534
Contrived hands-on experience training16464 (90)100 (0)646
Table
Table 2. Comparison of the quiz completion time between the developed VR training scenarios.
Table 2. Comparison of the quiz completion time between the developed VR training scenarios.
Experience in GamingMean Completion Time (s)Mean Performance Score (%)
Rarely/sometimes 49979
Often44990
Table
Table 3. Comparison of the quiz completion time between the developed VR training scenarios.
Table 3. Comparison of the quiz completion time between the developed VR training scenarios.
Participants CategoryMean System Usability Score of VR Training Scenario
Audio-Visual Based TrainingInteractive-Based TrainingContrived Hands-on Experience Training
Overall3.874.074.13
Low academic performance4.334.674.00
Middle academic performance3.403.604.20
High academic performance4.004.144.14
Low frequency in gaming3.774.004.08
High frequency in gaming4.504.504.50
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Lee, Y.S.; Rashidi, A.; Talei, A.; Beh, H.J.; Rashidi, S. A Comparison Study on the Learning Effectiveness of Construction Training Scenarios in a Virtual Reality Environment. Virtual Worlds 2023, 2, 36-52. https://doi.org/10.3390/virtualworlds2010003

AMA Style

Lee YS, Rashidi A, Talei A, Beh HJ, Rashidi S. A Comparison Study on the Learning Effectiveness of Construction Training Scenarios in a Virtual Reality Environment. Virtual Worlds. 2023; 2(1):36-52. https://doi.org/10.3390/virtualworlds2010003

Chicago/Turabian Style

Lee, Yee Sye, Ali Rashidi, Amin Talei, Huai Jian Beh, and Sina Rashidi. 2023. "A Comparison Study on the Learning Effectiveness of Construction Training Scenarios in a Virtual Reality Environment" Virtual Worlds 2, no. 1: 36-52. https://doi.org/10.3390/virtualworlds2010003

Article Metrics

Back to TopTop