A Virtual Reality Laboratory for Blended Learning Education: Design, Implementation and Evaluation

Abstract

Launched during the pandemic, the EU-funded JANUS project aimed to ensure the continuity of student workshops at universities using a virtual reality (VR) robotics laboratory. With the return to normality, the project has been redesigned to capitalise on the positive outcomes of the experience. The VR lab provides safe and unrestricted access to the labs and experiments with the machines, reducing the consequences of student mistakes and improving the user experience by allowing the experiment to be repeated from different angles, some of which are impossible to access in the real lab. In addition, integration with an interactive learning platform called “ViLLE” allows for continuous assessment of the learning experience. Self-evaluation of the material taught and learned can be integrated with the execution of the exercises that pave the way for Kaizen. Two VR workshops for the blended learning of robotics were developed during the JANUS project. Their evaluation reported favourable responses from the students whose learning performance was indirectly measured.

1. Introduction

Blended learning (BL) didactics—a combination of traditional face-to-face and online learning—received unprecedented attention in higher education after the onset of the pandemic [1]. The shift to widespread online education, necessitated by movement restrictions during the peak of the outbreak, highlighted some limitations in the context of STEM (Science, Technology, Engineering and Mathematics): the difficulty of delivering workshops [2] and the lack of effectiveness of online assessments [3]. In addition, Karalis and Raikou [4] show that distance learning is modern, appropriate, and convenient, but cannot replace the social interaction with colleagues and teachers in the classroom and laboratory, as well as the effectiveness of online assessments.

The ERASMUS+ Janus project [5] was proposed during the first COVID-19 lockdown. The aim of the project was to overcome the main shortcoming in online teaching subjects such as the STEM disciplines (i.e., the impracticality of access to laboratory workshops). Based on a blended learning scenario, the project implemented a digital twin of a robotics laboratory to be used either as remote access or as an extension of the actual experience in the real world. The digital twin is the real-time digital counterpart of a physical entity [6].

To overcome another limitation of online education, the difficulty of assessment, the project includes an interactive assessment system that allows students to self-assess. Several universities experimented with various forms of virtual laboratory during the pandemic, for example, see Kapilan [7]. The use of self-assessment learning techniques combined with access to a virtual laboratories is missing in all the studies considered and is the main innovative element of the project. The design of the experience is organised as a low-level Poka-Yoke—a Lean Thinking (LT) solution, which immediately indicates the mistakes made by the students during the reproduction of laboratory experiments [8]. This solution forces the students to systematically correct all the mistakes if they want to continue with the workshop. To evaluate the potential of the proposed approach, we adopted the guidelines provided by the four-dimensional framework for VR learning analytics, which details how data can be used collectively to target instructional objectives.

Specifically, the four-dimensional framework for VR learning analytics includes four dimensions: data collection, data analysis, feedback generation, and feedback presentation. In this project, the interactive assessment system collects data on the students’ self-assessments and virtual laboratory activities. The collected data are then analysed to identify patterns and trends that can inform instructional objectives. Based on the analysis, feedback is generated and presented to the students in a way that is timely, relevant, and actionable.

The use of self-assessment learning techniques combined with access to a virtual laboratory is particularly beneficial for students studying STEM subjects, as it allows them to practice and apply their knowledge in a safe and controlled environment. The interactive assessment system also provides immediate feedback, which is crucial for helping students correct mistakes and improve their understanding of the subject matter.

Overall, the design of the experience as a low-level Poka-Yoke solution ensures that students are held accountable for their learning and are forced to systematically correct mistakes. This approach can help students develop a growth mindset and a willingness to learn from their mistakes, which are essential qualities for success in any field. Using the terminology of ‘quality management’, it can be said that there is a shift from assessing the quality of teaching to ensuring the quality of learning.

The difficulty of assessing student knowledge is recognised as the main limitation of distance learning, whereas in the current approach, similar to a video game, to pass a level the student must complete all the tasks assigned to the previous level.

By adopting the four-dimensional framework for VR learning analytics, the project ensures that data are used effectively to inform instructional objectives and improve the learning experience for students.

During the project, two different applications of VR are developed to support the robotic laboratory of two different Universities. The reason for the double experiment is the opportunity/necessity of applying two different BL models to the workshop experience. In the application developed for the Polytechnic of Turin, the difficulty of physically visiting the laboratory for a large number of students has to be overcome. The BL adopted is the rotational model. In the application developed for the Politechnika Rzeszowska, a group of students has to learn how to program a robot by taking turns on the physical machine, after having learned all the safety procedures in a virtual environment. The BL chosen is the face-to-face driver model.

The project provides secure access to robotics workshops for both local and remote students, without time constraints, using a VR model that runs in a standard internet browser. The disadvantage of desktop VR is that it is difficult to interact with the virtual lab using standard input devices: mouse and keyboard. The virtual experience is, therefore, limited to those workshops characterized by a low intensity of interaction with the robots.

Computer science and mechatronics are two disciplines that are used in academic robotics courses on a multidisciplinary basis [9,10]. The study of robotics also makes it possible to master topics from fields such as mechanical engineering and electronics effectively [11].

Some examples of the use of VR in robotics are described in [12]. This study tests the impact and potential benefits of virtual reality technology by creating virtual simulations of industrial robot operations. The work aims to develop games using Unity and Oculus Rift headsets for virtual visualisation, enhance student interaction with robots in virtual environments, and introduce pedagogical innovations. Another example of the use of VR in robotics is the work of [13], where the authors propose a blended form of learning.

The paper presents an experiential education based on blended learning methods, using a virtual and remote robotics laboratory called RobUALab as an experimental tool. Students practise manual exercises using RobUALab, first in face-to-face classes where they experiment with real plants in the field, and then access the experimental environment to complete their practical exercises remotely outside the laboratory. The results of the evaluation of the proposed teaching methods demonstrate their effectiveness in terms of student learning and performance. The above work is an example of the growing interest in the use of VR for robotics education.

Furthermore, the combination with BL is a promising pedagogical approach that has multiple benefits. It enhances students’ academic outcomes, competence development, personal wellbeing, and social interactions. It also transforms the role of the teacher into a facilitator of meaningful learning.

The research questions that the project tries to answer are therefore:

RQ1—How to provide a virtual lab experience that is comparable to the real lab?

RQ2—How can the limitations of interaction with other users of the virtual laboratory be used to set up a learning system based on self-assessment?

To get feedback on the effectiveness of introducing the VR laboratory in the curriculum, questionnaires and Voice of the Customer (VOC) were employed. Questionnaires allowed us to collect quantitative data from a large sample of students and teachers and to measure their satisfaction, learning outcomes, and preferences. VOC allowed qualitative data to be collected from a smaller sample of students and to gather useful suggestions.

The paper combines two elements of innovation in teaching robotics with the support of the laboratory: VR access to the laboratory and self-assessment of the learning outcomes.

2. Materials and Methods

The project developed two Blended Learning scenarios in which the teaching team was present on the laboratory premises whereas the students participated remotely. The first workshop scenario in Politecnico di Torino consisted of industrial assembly tasks performed by dual-arm collaborative robots (Figure 1a). The second scenario in Politechnika Rzeszowska presented the machining process of grinding the vehicle rim (Figure 1b).

The digital twins, made up of robot, workbench, and workpiece, were created in a desktop view using Unity 3D. The pairing with the real workstation is obtained by data exchange through the Message Queuing Telemetry Transport (MQTT) protocol, a standard messaging protocol for Internet of Things (IoT). Further details are found in [14].

Figure 2 shows a VR (Virtual Reality) replica of a dual-arm assembly station, which is complemented with additional information. This means that the VR model not only represents the physical appearance of the assembly station, but it also includes additional information that provides more context or insight into the system. This additional information could be in the form of data overlays, annotations, or visualizations of relevant metrics such as temperature, pressure, or machine status.

The scenario for the Rzeszow University of Technology involved a face-to-face BL model. After attending regular lectures in class, students had access to the VR platform, where they prepared for workshops to be held in the physical laboratory. The VR platform allowed harmlessness learning the safety procedures to apply in the robot laboratory. The students also used a robot simulator software, RobotStudio, to practice the programming algorithms for controlling the movement of the robotic arm. RobotStudio also allows students to obtain detailed data the signals from the sensors located in the robot for further analysis within their own work. RobotStudio is software provided to the Rzeszów University of Technology as part of the cooperation with the manufacturer of the ABB robot, which is in the laboratory. In this context, the students were able to gain knowledge during the actual laboratories, additional learning on the VR platform developed as part of the project, and the RobotStudio software integrated into the laboratory.

The educational project is presented in Figure 3. The two figures show the physical machining part and its tool (a), and the VR replica of the same part (b). The VR replica is referred to as a digital twin because it is a continuously updated virtual replica of the physical machine. The digital twin allows for virtual testing and simulation of the machining process before performing it on the physical part.

In Figure 3b, the path of machining is indicated, which means that the virtual model shows the exact path that the tool will take when machining the part. This information is important because it allows for the optimisation of the machining process and helps to identify any potential issues or errors in the process before it is performed on the physical part. Overall, the use of VR and digital twin technology can improve the efficiency and accuracy of industrial processes.

The paper concerns the continuous improvement of the VR workshop. Continuous improvement can be implemented in two ways. The first way is improvement in small steps (kaizen), which takes place when increasing the knowledge of students using the methodology proposed in this work, with a set of interactions and self-answering questions as illustrated in Figure 4 and Figure 5. The use of kaizen in education is proposed for example by Wiid [15].

The second way of improvement is the implementation of a breakthrough change, which takes place in the case of the implementation of the proposed VR to the education process. DMAIC methodology was previously successfully used for education process improvement by Navas et al. [16]. DMAIC is a data-driven process-improvement methodology that focuses on properly identifying the problem, utilising data to determine the root cause, developing and implementing the solution, and verifying that the solution continues to be effective over time. DMAIC is an acronym that stands for Define, Measure, Analyse, Improve, and Control. The sections of the paper broadly refer to DMAIC, with the definition of the workshop, the collection of ‘voice of the student’, the analysis of the survey, and the discussion of possible overcoming to the emerged weak characteristics of the blended learning VR workshop.

2.1. Apply Blended Learning to a Robotic Workshop

There is a strong focus on VR to engage learners to create active learner positions where the student is involved in the learning content [17,18], shifting from teaching to learning. Speaking about how laboratory workshop can be organised in blended learning, there are a variety of learning methods that can be developed to involve students in the learning process, promote higher levels of thinking, creativity, initiative, dissemination of effective ideas, and full integration of teamwork [19].

Learning with other students can be developed in synchronous/asynchronous collaboration with the teacher. Sepasgozar [20] states that some of the most important knowledge we gain comes from collegiality and debate. Group-based learning methods consist of open-ended problem solving, practical projects, and team-oriented communication. Laboratory tasks are a powerful pedagogical strategy for developing competences, allowing students to understand the theoretical topics explained in the classroom in a practical way [21]. Virtual laboratories are able to simulate typical operating environments, as well as extreme situations in the operation of different equipment. In general, VR technology supports the idea of providing simulated environments for hands-on experiments as additional learning material. In the context of the blended learning approach and the experiential learning theory, the VR Robotics Lab thus provides an effective and efficient format for digital teaching [22].

As for the actual implementation of the workshop, it can be carried out in the following modes: physical visit to the laboratory; VR visit to the laboratory; extended reality access to the laboratory, where students can interact with virtual objects in the digital model; digital twin of the laboratory, where the VR model is a faithful dynamic replica of the real laboratory; video presentation.

In the project, some modes were combined in two macro-scenarios:

• All students attend the lab and simultaneously access its digital twin (synchronous), which is only possible with a small number of students.
• Only the teacher is present in the lab and the students experience the workshop through a VR replica with interaction (asynchronous). There is no limit to the number of students.

2.2. Collect the ‘Voice of the Customer’

The primary aim of this study was to investigate the potential of VR on students’ comprehension of intricate processes performed by automation robots. To attain this goal, a custom VR experience was developed and coupled with a Learning Management System (LMS) named ‘ViLLE’ [23]. The LMS supported the collection and storage of diverse data modalities and further provided Learning Analytics (LA) services which enabled educators and learners to monitor progress and identify areas in need of improvement [24,25].

The experiment was meticulously structured as a three-stage intervention process. In the initial stage, participants recorded within the LMS their background information (demographics) and English proficiency level as all the activities were conducted in English which was not their native language. Concurrently, a custom knowledge quiz (pre-test) became available assessing participants’ theoretical understanding of the subjects that were to be covered during the course of the intervention, thereby establishing a baseline for subsequent comparisons. The final part of the pre-intervention stage included a preliminary survey which evaluated participants’ perspectives and expectations regarding the implementation of educational VR. The aforementioned information enabled us to generate the student profiles and further to connect them with the activities that were to be performed in the VR platform. Following this stage, students engaged with the VR platform which supported self-guided practice and self-assessment via the asynchronous modality of the digital twin. For the self-assessment, a predetermined set of Multiple-Choice Questions (MCQs) was strategically integrated for each topic aligning with the actions that the robots were executing or preparing to execute. Upon completion of the VR-based educational activities, participants returned to the LMS for the post-test assessment, which aimed at evaluating their newly acquired theoretical knowledge. The post-test utilised the same questions as the pre-test but in randomised order to prevent answer memorisation. Following completion of the post-test, a comprehensive survey became available gathering feedback on participants’ present learning experience and willingness to embrace educational VR in future academic pursuits.

Concerning the data collection instruments, the LMS examined several subject-related theoretical topics including (a) theoretical concepts of collaborative robotics and robots, (b) set-up and programming of Universal Robots, (c) types of robot movements, and (d) robot’s set-up. On the other hand, the self-assessment quizzes in the VR tool were streamlined to the examination of participants’ conceptual understanding across the following themes: (a) Pick and Place, (b) Palletising, and (c) Welding. As for the psychometric instruments, to assess learners’ prior experience with educational VR we adopted and adapted the instrument provided by Küçük et al. [26] (constructs: Satisfaction, Aversion, Willingness; Likert Scale: 1-Strongly Disagree, 5-Strongly Agree) whereas, for the evaluation of participants’ current learning experience and willingness to adopt educational VR in the future, we adopted and modified the instrument provided by Fokides et al. [27] (constructs: Immersion, Enjoyment, Perceived Usefulness, Perceived Realism, Appropriateness of learning material, Perceived Ease of Use, and Motivation; Likert Scale: 1-Strongly Disagree, 5-Strongly Agree).

2.3. Design of the Self Learning Procedure

Self-learning can be conducted through a variety of methods, including reading books, watching instructional videos, taking online courses, and practicing skills through trial and error. This latter was the method adopted by the VR laboratory workshop. There are many benefits to self-learning. Self-learning allows the students to explore new topics and ideas at their own pace, leading to personal growth and development. Self-learning allows students to learn on their own schedule, making it a flexible option for those with busy lives or living at a distance.

Thus, staying motivated while self-learning can be challenging [28], but there are several strategies that were followed in our project. The workshop was organised into stand-alone learning modules, each one with a proper goal on which the student can focus. There was a rewarding mechanism, borrowed by computer games: to advance in the VR experience (pass level), it is necessary to answer correctly to the questions. It is possible to retry after a wrong answer.

3. Results

The results presented here refer to the scenario from Politecnico di Torino, due to differences in the approach to blended learning and the implementation of the simulation scenario. As PRZ also used a robot simulator, the responses of PRZ students do not allow them to separate the VR experience from the experience with the robot simulator. In order to have unbiased feedback on VR only, it was decided to present only the surveys answered by the students of the Politecnico di Torino. A detailed elaboration of the results from Rzeszow University of Technology will be the subject of further work. The paper includes a description of both teaching cases and scenario realisations, as both approaches to BL for robotics can be valuable for the reader interested in this topic.

3.1. Participants

In total, 119 students were invited to participate in the study but only 107 provided their background information at the start. The same students (n = 107) also provided responses to the preliminary survey (perceptions and attitudes toward educational VR) but only 102 undertook the preliminary knowledge quiz (pre-test). As for the knowledge-related assessments that took place during the conduct of the intervention (i.e., within the VR tool), the following submissions were made to the individual topics: (a) Pick and Place (25 students), (b) Palletizing (16 students), and (c) Welding (13 students). Finally, the conclusive survey (perceived usefulness of the Blended Learning VR experience) was filled in by 84 students, whereas the conclusive knowledge quiz was completed by 82 students.

In the proposed evaluation plan, we had intended to systematically measure students’ academic performance and attitude toward the use of educational VR across the intervention stages. However, the volunteering nature of the study prohibited us from achieving this objective as seen from the decline in receiving complete submissions across the predefined milestones. Therefore, in the analyses that follow, the performance metrics were not considered as they could lead to incorrect and inaccurate conclusions. Instead, we explored and analysed the data that emerged from the psychometric surveys which investigated learners’ preconceptions and attitudes toward educational VR as well as their perceptions toward the Blended Learning VR experience. Further studies will be necessary to address the issue of student engagement with the predefined tasks especially when it comes to pilot studies.

3.2. Descriptive Statistics

The final sample for the current investigation consists of 70 postgraduate students (40 males, 30 females) from the Polytechnic University of Turin aged between 18 and 24 years old. With regard to their English proficiency, the majority of the participants (n = 33) did not provide any information about their IELTS scores. One possible explanation for this omission might be that some participants had taken alternative English language proficiency tests, such as TOEFL, instead of the IELTS exam. As a result, these students might not have had an IELTS score to report. Notwithstanding, in accordance with the university’s regulations, students are required to demonstrate a certain level of competency in English to be considered for admission into the program which, in the present case, acts as a safeguard to the data analysis. Among those who did disclose their IELTS scores, only a small number (n = 4) reported having ‘Basic’ English competence, while a significant portion indicated ‘Intermediate’ English competence (n = 25). In contrast, just a few participants reported ‘Advanced’ (n = 7) or ‘Proficiency’ (n = 1) levels of English.

Table 1. Descriptive statistics of the prior experience ratings and workshop evaluations.

Variable	Cronbach’s α	Min	Max	M	SD	Skewness	Kurtosis
Prior experience
Satisfaction	0.79	2.00	4.71	3.45	0.60	−0.19	−0.29
Aversion	0.83	1.00	4.17	2.07	0.69	0.79	0.16
Willingness	0.63	1.00	5.00	3.30	0.85	−0.07	−0.06
Workshop Evaluation
Immersion	0.80	1.00	4.67	3.04	0.66	−0.27	1.10
Enjoyment	0.85	1.67	5.00	3.50	0.75	−0.15	−0.52
Usefulness	0.87	1.33	5.00	3.30	0.72	−0.16	0.88
Realism	0.85	1.00	5.00	3.12	0.83	−0.25	0.81
Difficulty	0.73	1.30	3.70	2.94	0.47	−0.67	1.11
Motivation	0.89	1.00	5.00	3.50	0.88	−0.51	0.34

shows the descriptive statistics of the variables that were used in this study to measure the students’ prior experience and how they evaluated the Blended Learning VR workshop. All scales showed sufficient to good internal consistencies, as indicated by Cronbach’s alpha values that ranged from 0.63 for willingness to 0.89 for motivation. Further, the scales showed approximately normal distributions with little skewness and kurtosis. It can also be seen that, whereas some variables showed the full range of potential values (1–5), none of the students had an average below two on Satisfaction and for Difficulty, the range was relatively small, with all students indicating at least some difficulty, yet none giving a maximum score.

3.3. How Does the Blended Learning VR Workshop Impact Knowledge Acquisition?

As mentioned in Section 3.1, given the limited number of responses received in the VR-based assessments, it was deemed wise not to process this data to avoid drawing conclusions based on a statistically insignificant sample size. This decision was made to maintain the integrity and validity of our study and to prevent misinterpretation of the findings. However, we processed the pre-post knowledge assessments from the LMS-based quizzes which concerned the theoretical knowledge advancement (

Table 2. Descriptive statistics for the pre-post-test knowledge assessment scores.

Variable	Min	Max	M	SD	Skewness	Kurtosis
Pretest
Theoretical concepts of collaborative robotics and robots	0	1	0.38	0.48	0.485	1.242
Set-up and programming of Universal Robots	0	1	0.47	0.5	0.116	1.089
Type of movements	0	1	0.4	0.49	0.394	1.164
Configuration of Universal Robots	0	1	0.19	0.39	1.573	3.54
Posttest
Theoretical concepts of collaborative robotics and robots	0	1	0.38	0.48	0.465	1.222
Set-up and programming of Universal Robots	0	1	0.41	0.49	0.355	1.205
Type of movements	0	1	0.42	0.49	0.304	1.10
Configuration of Universal Robots	0	1	0.12	0.33	2.243	6.142

Note: For presentation purposes the test scores have been scaled (min = 0, max = 1).

). These assessments provided a more substantial sample size, allowing for a more comprehensive analysis of participants’ learning outcomes. To analyse this data, we employed paired t-tests to compare participants’ scores before and after completing the training program.

The results for the first subject, “Theoretical concepts of collaborative robotics and robots”, revealed no statistically significant difference between the pre-test (M = 4.59, SD = 1.89, and SEM = 0.23) and post-test (M = 4.57, SD = 1.72, SEM = 0.21) scores, t(69) = 0.0427, p = 0.96. This finding is somewhat surprising, given that VR technology has been increasingly touted as a transformative tool for education. Next, in the domain of “Set-up and programming of Universal Robots”, the difference between pre-test (M = 0.47, SD = 0.50, and SEM = 0.06) and post-test (M = 0.40, SD = 0.49, and SEM = 0.06) scores was also not statistically significant, t(69) = 0.8434, p = 0.40. This finding suggests that the VR tool did not significantly enhance the students’ understanding of UR robot set-up and programming. It is possible that the complexity of this topic may have made it difficult for the VR intervention to effectively convey the necessary information to the students. For the theme “Types of robotic movements”, there was no significant difference between the pre-test (M = 3.23, SD = 1.24, SEM = 0.15) and post-test (M = 3.36, SD = 1.61, SEM = 0.19) scores, t(69) = 0.5653, p = 0.5737. This result raises questions about whether the VR tool was sufficiently engaging or well-designed to effectively teach this particular aspect of robotics. Lastly, in the topic of “Configuration of Universal Robots”, although there was a slight decrease in the mean score from the pre-test (M = 0.39, SD = 0.62, SEM = 0.07) to post-test (M = 0.24, SD = 0.52, SEM = 0.06), the difference was not statistically significant, t(69) = 1.4871, p = 0.1415. This may suggest that the VR tool might not have effectively addressed the challenges that students face when learning about setting up robotics.

The observed lack of significant knowledge increase across all topics could be attributed to two main factors: the duration of the VR training and the complexity of the subject matter. First, the insufficient time allocated for the VR training could have limited the students’ ability to fully absorb and process the information provided. In the context of learning complex subjects, such as robotics, it is essential for students to have ample time to practice and consolidate new concepts. Research has consistently demonstrated that effective learning occurs when students are given the opportunity to engage with the material, apply it to real-world situations, and reinforce their understanding over time. It is possible that the duration of the VR intervention was not adequate for students to achieve these learning goals, thereby contributing to the absence of significant knowledge gains. Second, the complexity of the subject matters could have played a role in the observed results. Robotics encompasses a wide range of topics many of which can be challenging for students to grasp. The VR tool might not have been able to effectively break down and simplify the content in a way that was easily accessible for the students. In some cases, traditional teaching methods, such as lectures, demonstrations, or hands-on workshops, could be more effective for conveying complex information. This is particularly relevant when considering that the learning process often requires multiple approaches, the scaffolding of knowledge, and opportunities for students to ask questions and receive clarification from their instructors.

Considering these factors, future research should investigate the optimal duration for VR-based educational interventions in robotics to ensure that students have sufficient time to engage with and assimilate the material. Additionally, studies should explore strategies to effectively present complex information in VR environments, potentially through the integration of traditional teaching methods, to facilitate a more comprehensive and accessible learning experience for students. By addressing these issues, educational practitioners can better harness the potential of VR technology to enhance learning outcomes in robotics and other complex subjects.

3.4. How Is the Blended Learning VR Workshop Evaluated?

One sample t-tests were also used to test if the student evaluations were negative, neutral, or positive by using the neutral scale midpoint of three for comparison. It was found that, on average, students gave neutral evaluations with respect to immersion, t(69) = 0.54, p = 0.590. This means that the students did not particularly experience more immersion than normal. The average enjoyment they experienced, however, was found to be significantly higher than three, thus more positive than neutral, t(69) = 5.55, p < 0.001. Similarly, the students were also positive about the usefulness, t(69) = 3.45, p < 0.001. The students were neutral about the realism of the materials, t(69) = 1.19, p = 0.239. The average scores also indicated a neutral evaluation with respect to difficulty, t(69) = −0.98, p = 0.329, which means that the level of the material was about right. Finally, students’ experiences of motivation were significantly higher than the scale midpoint, t(69) = 4.81, p < 0.001. It can, therefore, be concluded that on average, the VR workshop was evaluated positively.

3.5. What Explains How Different Students Evaluate the VR Workshop?

Males and females exhibited highly similar average scores in the VR Workshop, leading us to perform a MANOVA to further explore these similarities. The results confirmed that there was no significant gender effect on the evaluations, with an F-value of 6.63 and a p-value of 0.821, indicating that gender did not play a role in the assessment of the VR workshop. This finding seems to diverge from previous studies that suggest potential gender differences in VR learning environments [29,30,31]. However, it is important to note that research on gender and age effects in VR environments is still limited and urgently needed [32].

To better understand the relationship between the different variables,

Table 3. Correlations between prior experiences and current workshop evaluations.

Constructs	IELTS	1	2	3	4	5	6	7	8
1. Satisfaction	−0.19
2. Aversion	0.02	−0.53 **
3. Willingness	−0.12	0.38 **	−0.31 **
4. Immersion	0.17	0.11	−0.04	0.09
5. Enjoyment	−0.17	0.44 **	−0.27 *	−0.05	−0.11
6. Usefulness	−0.11	0.38 **	−0.14	0.14	0.03	0.72 **
7. Realism	−0.14	0.28 *	−0.07	0.10	0.15	0.51 **	0.67 **
8. Difficulty	0.08	0.01	0.10	−0.08	0.73 **	−0.18	−0.15	0.07
9. Motivation	0.05	0.16	−0.25 *	−0.07	−0.19	0.64 **	0.49 **	0.39 **	−0.25 *

* p < 0.1, ** p < 0.5.

presents the correlations between the IELTS scores, prior responses to VR, and evaluations of the VR Workshop. Interestingly, the IELTS scores did not exhibit any significant relationship with the other variables, suggesting that language proficiency did not influence participants’ evaluations.

The evaluation of VR-based educational interventions is often influenced by the so-called “novelty effect” [33]. Likewise, individuals’ prior experiences with a particular tool (in this case VR) can shape their willingness and preconceptions before diving into a new experience. These combined factors can either positively or negatively affect students’ satisfaction levels at the end of an intervention. However, this satisfaction may prove short-lived as enthusiasm wanes, particularly if the intervention fails to deliver lasting, meaningful learning experiences. In view of these, it was deemed wise to cross-examine whether any interplay exists between students’ prior VR experiences (i.e., willingness to use VR) and satisfaction levels in the present intervention.

Upon closer inspection of the data, we observed that Satisfaction exhibited positive correlations with Enjoyment, Usefulness, and Realism evaluations. In other words, participants who were more satisfied with the VR Workshop tended to rate it higher in terms of enjoyment, usefulness, and realism. Conversely, Aversion was negatively correlated with Enjoyment and Motivation, indicating that those who experienced aversion towards the VR Workshop were less likely to enjoy it or feel motivated by it. It is important to note that prior willingness did not demonstrate any relationship with the evaluations, suggesting that participants’ initial attitudes towards the workshop did not affect their evaluations.

Moreover, the evaluations displayed various intercorrelations among themselves. For instance, Immersion and Difficulty exhibited a strong, positive correlation, indicating that participants who found the VR Workshop more immersive also perceived it as more challenging. Additionally, Difficulty demonstrated a moderate, negative association with Motivation, suggesting that as the perceived difficulty increased, motivation decreased. Furthermore, Enjoyment, Usefulness, Realism, and Motivation all displayed positive associations with each other, implying that these factors are interconnected and potentially contribute to an overall positive evaluation of the VR workshop.

4. Discussion

Blended learning has become increasingly popular in higher education, especially in the context of the pandemic. However, implementing blended learning in STEM fields has presented some challenges, such as the lack of access to laboratory workshops and the difficulty of conducting effective online assessments [2,34,35]. The present solution was designed to address these challenges through the use of a digital twin of a robotics laboratory and an interactive self-assessment system.

The digital twin allowed students to remotely access and interact with a real-time digital counterpart of a physical laboratory (answer to RQ1). This approach overcame the limitations of online learning and provided safe access to robotic workshops without time limitations. The interactive self-assessment system, organised as a low-level Poka-Yoke, provided immediate feedback to students, forcing them to correct their mistakes systematically before continuing with the workshop. This approach helped shift the focus from assessing the quality of teaching to ensuring the quality of learning, addressing one of the main limitations of distance learning (answer to RQ2).

Τhe study shows that students maintained a positive attitude toward the VR workshop before and after the experience. This was particularly true for students who already had felt satisfaction with respect to the use of VR prior to the workshop and who were not aversive with respect to VR uses. Interestingly, the overall willingness to use VR for other subjects as well was unrelated to the current experiences. This may indicate that students may have specific ideas about topics for which VR is suitable or find it difficult to correctly estimate its potential value without actual real experience. In line with this idea, it was found that prior satisfaction, that was formed by reflections on previous lessons that had used VR applications, was positively associated with motivation.

In the current study, but also confirmed by literature, the level of difficulty was negatively associated with motivation [36]. This can be an advantage for VR that can be combined with Artificial Intelligence to create learner-adapted experiences that are customised to the knowledge and skills of individual students [31].

This could be a next step in the further adjustment of the workshop. In the current study, a Poka-Yoke approach was used, that allowed students to process how they were performing. Concurrent feedback is seen as one of the factors that foster immersion [32]. Yet, the combination of concentration and enjoyment that makes students forget about their environment or what is also referred to as ‘flow’ results from an optimal balance between skill and challenge [37].

Further improvement could include adding simple tasks to VR to ensure a high level of student motivation and easy assimilation of new knowledge. The control process should involve ongoing monitoring of student performance to identify tasks that lower the level of motivation and improve them.

Another way to improve the workshop could be to incorporate collaborative activities within the VR environment. This could involve group problem-solving activities or team-based challenges that require students to work together to achieve a common goal. Collaboration has been shown to increase engagement and motivation as well as improve learning outcomes [38].

Additionally, providing opportunities for students to customise their VR experience could also enhance their motivation and engagement. This could involve allowing students to personalise their avatars or choose the virtual environment they want to work in. Customisation provides a sense of ownership and control, which can increase motivation and engagement.

Finally, incorporating real-world scenarios and challenges into the VR environment could also improve student motivation and engagement. By simulating real-world situations, students can see the practical applications of the knowledge they are learning and understand how it can be applied in their future careers. This can increase their motivation to learn and improve their learning outcomes.

Overall, there are many ways to improve the VR workshop to increase student motivation and engagement. Incorporating collaborative activities, customisation, real-world scenarios, and ongoing monitoring of student performance can all contribute to a more engaging and effective learning experience.

5. Conclusions and Future Directions

BL combines online and face-to-face learning activities to enhance students’ engagement and outcomes. However, its effectiveness and suitability for different contexts and learners need to be carefully examined. STEM students may benefit from experiencing the relevance and applicability of what they learn, which can foster their motivation.

The COVID-19 pandemic has accelerated the use of online methods and increased the demand for more adaptive and personalised digital learning environments. VR is one of the technologies that has been shown to have potential for STEM education [39].

VR laboratories have several implications for education, research, and society:

-

They can enhance students’ engagement, motivation, and learning outcomes by providing interactive and immersive experiences that mimic real labs.

-

They can increase the accessibility and inclusivity of science education by allowing students from different locations, backgrounds, and abilities to participate in lab activities.

-

They can foster collaboration and communication among students and instructors by enabling online feedback, peer review, and data sharing.

The current study evaluates a VR workshop developed in the JANUS project and finds positive feedback from the participants.

However, the study also has some limitations that should be acknowledged. First, it was not possible to compare the VR workshop with a conventional workshop using the same group of students. Therefore, it is unclear whether the Blended Learning approach would be preferred over a traditional one. Second, the learning performance was measured only indirectly (time spent online, number of wrong answers to online questions), which does not provide conclusive evidence of the learning outcomes. Third, the VR workshop was accessed individually by each student, which prevented collaborative learning.

Based on the findings of the present study, some future research directions can be recommended. First, it is important to investigate whether students’ positive attitudes towards the use of VR in the workshop translate into improved learning outcomes and long-term retention. Future studies could, therefore, explore the impact of VR on student learning outcomes by comparing the performance of students who have used VR with those who have not. In addition, the potential of combining VR with other emerging technologies, such as AI, to create personalised learning experiences should be explored. The present study suggests that AI could be used to adjust the level of difficulty based on individual students’ knowledge and skills. Another area of research could be the effectiveness of the interactive self-assessment system in improving learning outcomes. Future studies could explore the impact of the system on student performance in different STEM fields and assess its adaptability to other domains. Furthermore, the use of digital twins could be extended to other STEM fields beyond robotics. The present study focused on the use of digital twins in a robotics laboratory, but the approach could be applied to other fields, such as chemistry or biology, where access to physical laboratories may be limited. Finally, comparative studies evaluating the effectiveness of different BL approaches should be undertaken. The present study evaluated the effectiveness of the digital twin and interactive self-assessment system approach, but it would be interesting to compare it with other approaches such as flipped classrooms [40] or hybrid learning. A control group attending alternative learning strategies could be included to address these issues.

From the point of view of the robotics laboratory, it is important in future work to extend the developed scenarios to include other activities that remain in the curriculum. The BL approach can be considered here as a dual-mode activity, i.e., participation in the physical activity plus additional preparation before the activity, e.g., including a focus on safety. The developed VR system forms the basis for the design of other activities that use such robots.