The Influence of Competency-Based VR Learning Materials on Students’ Problem-Solving Behavioral Intentions—Taking Environmental Issues in Junior High Schools as an Example

Abstract

With the increasing price–performance ratio of virtual reality equipment, there has been an increase in research on the use of VR in education. The learning content of biology courses for junior high schools often introduces the interaction between organisms and their surroundings. Through the interactive design of VR learning materials, students can enhance their understandings of the impact of human behaviors on the environment. They can become immersed in the virtual world using a virtual camera to explore environmental pollution and record pollution sources, which exist in their surrounding environment. In this study, one-group pre- and post-test designs were adopted to collect and analyze data on students’ reactions before and after implementation of the VR learning course for environmental issues in a junior high school. The questionnaires were analyzed to evaluate the impact on course identity and behavioral intentions of students regarding problem-solving. The results reveal the VR interactive learning course has positive effects on both the course identity and the students’ behavioral intentions for problem-solving.

1. Introduction

The rapid development of information technology leads educators to apply advanced technologies, such as AR/VR, into competency-based learning [1,2,3,4,5,6]. In the 21st century, students need to acquire the foundation of learning processes, such as problem-solving, critical thinking, creativity, innovation, metacognition, communication, etc. to survive in the modern world [7]. The new K-12 education curriculum guidelines implemented in the 2019 academic year in Taiwan take the key competencies as the backbone of the coherence and integration of basic education courses and thereby guide the development of curriculum and teaching in various fields. Learning materials have a great influence on self-learning for students; therefore, how to implement teaching content that conforms to the idea of competency-based instructions is one of the most important issues.

Problem-solving is the foundation of learning processes and is a cognitive process that requires the ability to define and identify a problem, find a solution, and implement the solution in relatively new situations [8]. Problem solving is often challenging for students because they do not understand the problem-solving process. In [8], Yu, K.C. et al. reveal context-based learning may effectively enable students to establish a complete problem-solving process. Many studies reveal virtual reality technology has demonstrated positive educational outcomes can engage and motivate users and potentially assist cognitive processing and knowledge transfer in some subjects [2,3,4,5,6]. Within the paper [6], authors conclude for high school students, the use of VR for subjects associated with chemistry was positive in terms of motivation and confidence since knowledge is required to handle these technologies. Currently, problem-solving has been identified as an important educational skill for primary school students [9]. VR technology enables the possibility of making abstract concepts concrete while increasing students’ engagement and motivation in the learning process [10,11]. In the paper [3] proposed by Ronaghi, M.H., it shows VR-based training courses, as well as holding traditional training courses, has been effective on sustainable behavior and has been effective in learning sustainable behavior.

Sustainability is one of the global challenges, and awareness of environmental protection is even more urgent in education. Environmental sustainability covers a wide range of issues from regional locations to the globe. Global issues include concerns about greenhouse gas reduction, climate change, and renewable energy, while regional issues are soil erosion, water management, soil quality, and air and water pollution. Thus, understanding the interaction between human and environmental ecology is one of the important topics on environmental education. In the study, a competency-based VR learning course on environmental education is implemented. One-group pre- and post-test designs were adopted to evaluate the effects of the VR learning course on students’ behavioral intentions for problem solving.

1.1. Interactive Design in VR Content

Interactive design refers to a mechanism by which designers interact with products or services. Human-computer interaction design based on user experience should consider the user’s background, user experience, and feelings during operations to design a product that gives a satisfactory user experience.

Using human intuition and experience can effectively reduce users’ thinking and ego depletion. A user can understand and use a design immediately without consciously thinking about how to do it. Thus, intuitive design can help users to immerse themselves in the experience of content without the burden for actions in the virtual environment [12]. An overview of some key terms relevant to interaction design in virtual reality were popularized in the book [13]. According to Nielsen [14], ten usability heuristics for digital interface design are proposed to create better experiences:

(1)

Visibility of system status;

(2)

Match between the system and the real world;

(3)

User control and freedom;

(4)

Consistency and standards;

(5)

Error prevention;

(6)

Recognition rather than recall;

(7)

Flexibility and efficiency of use;

(8)

Aesthetic and minimalist design;

(9)

Help users recognize, diagnose, and recover from errors;

(10)

Help and documentation.

We follow these guidelines to design VR interactive content for better user experiences. In this work, we make efforts to follow these guidelines. In particular, the two guidelines of (1). visibility of system status and (2). match between the system and the real world have implications for both interaction design and scenarios.

1.2. Design of Competency-Based Learning Materials

The new K-12 education curriculum guidelines in Taiwan take the key competencies as the backbone of the coherence and integration of basic education courses and thereby guide the development of curriculum and teaching in various fields. Different from many traditional learning methods using summative testing, competency-based learning focuses on the idea that students may only identify and demonstrate the mastery of individual learning outcomes [15,16]. Learning materials have a great influence on self-learning for students; thus, how to implement teaching content that conforms to the idea of competency-based instruction is one of the most important issues.

A framework to design the experience for sustainable behaviors on exploring the use of virtual reality was proposed by Giulia et al. [4]. It summarized the VR experiences could be explorations, games, simulations, and so on. Additionally, the objectives of VR experiences are related to the three behavioral dimensions, including emotional, rational, and practical. The use of VR experiences to support design teaching–learning activities can be related to the practical dimension. These designed solutions, presented and tested in VR, can explore strategies involving the rational and emotional ones.

As described in the paper [5], different teaching methods affect competency development and effectiveness in different ways. Novel VR tools require appropriate teaching methods to foster students’ abilities to develop competencies. Additionally, in [17] it also proposes approaches of modeling and implementations to make the design of competency-based educational content easier and serve as the recommendation of a set of resources concerning a skill to be acquired.

2. Aim

In the study, it proposes a competency-based VR learning material on environmental education, which emphasizes students’ exploration and recording about surrounding objects and provides formative assessments in the final exploration phase. Further, a VR learning course using the competency-based learning material is implemented and then evaluates whether the effect on students’ behavioral intentions of problem solving has been improved in the junior high school.

3. Materials and Methods

3.1. Hardware Components

The user application is based on Oculus Quest 2, which has a better price–performance ratio, easy maintenance, and classroom preparation. Users can query their learning records from the web server by web browser with the user ID and the school ID. These learning records can also be used as materials for students to explain and elaborate their understanding of concepts in further teaching activities. The details are described later.

3.2. Software Components

Good and flexible software development tools are helpful for the efficiency of system implementation and integration. In the development of the VR interactive learning materials, related development tools include the Unity 3D engine Ver. 2019.1 LTS [18], Autodesk Maya [19], MRTK v2.7 [20], Oculus Integration SDK Package [21], etc. Additionally, the XAMPP [22], an easy to install Apache distribution, extends web information server with integrated services, including PHP development environment, MySQL Database services, etc.

3.3. VR Content Design

The ADDIE model [23], including analysis, design, development, implementation, and evaluation steps, has been used as a framework for designing and developing educational and training programs for many years. The clearly defined steps help educators create effective teaching–learning materials.

In the analysis phase of the work, the target group is the students in the junior high school. There are petrochemical factories, orchards, pig farms, and other fields near their living environment. The purpose of the learning materials is to increase local students’ interests in understanding the classification of pollution sources and understanding the interaction between humans and the living environment.

In the content design phase, the types of environmental pollution sources are classified into five categories: eutrophication, heavy metal pollution, dioxin pollution, solid waste, and volatile organic compounds, VOCs, in the VR interactive material, as shown in Figure 1. Students must judge and classify the pollution sources collected by themselves into these classification kinds to complete the task, and the interactive learning system will feedback appropriate information in time. Additionally, it also supports students to query their learning records by web browsers and further discuss and share their learning results in the extended teaching–learning activities. It presents an example of the student’s learning records in Figure 2.

The system architecture of the VR content is illustrated, as shown Figure 3. The Unity Player application is developed for Oculus Quest 2, which is an all-in-one VR system and gains a better price–performance ratio in the current period. The database server is mainly used to maintain users’ learning records, including users’ test scores, users’ taking pictures in each scene, and recording times. Students using the Unity Player applications can upload their learning results to the cloud service.

3.4. Functional Architecture

In Figure 4 derived from Figure 1, it explains the VR learning objects in each different scene will be constrained to its preset pollution types. For example, in the house scene, it focuses on water pollution, air pollution, and solid waste. In the interactive design of VR learning material, we use the familiar camera element as a collection tool for users to explore in the virtual world. Using familiar visual elements can reduce burdens of human cognition for functionalities, but it is undeniable there are still many hardware interface design limitations that need to be improved, such as VR handle’s operations and so on. For unfamiliar user interfaces, users need a longer adaptation period.

4. Experimental Design and Data Description

4.1. Background and Experimental Design

One-group pre-test–post-test design is adopted in this study, and the participants’ questionnaire responses are analyzed. In the questionnaire, the main purpose is to evaluate students’ perceptions of course identification, analyze differences of the student ability scale, identify students’ intentions for problem-solving, and assess the students’ impacts on the VR teaching-learning activities. The process of the experimental test for VR experience is shown in Figure 5.

In this study, the experiment was conducted in September 2022, and the subjects were 29 first-year students at a junior high school in Kaohsiung City, Taiwan. There were 10 girls and 19 boys in this experimental class. The experimental activity of VR experience was held in school club classes. Before the VR experience activity, the teacher gave a brief activity description. Each student had approximately 25 min of VR experience time. Before and after the experience, participants filled out the pre-test and post-test questionnaires within 10 min each.

4.2. Experimental Method

The VR learning content consists of five scenes, including petrochemical factory, orchard, house, pig farm, and the laboratory. The laboratory scene is created for assessment after the learning experiences of the students. Other built scenes are related to the local students’ living environments. The scripts of these scenarios lead students to encounter the related environmental pollution problems in their lives.

As we know, the impact of pollution sources on the environment is complex and not classified into a single type of pollution. Therefore, we set different pollution weights for specified objects as references for the game exploration scoring. For example, as shown in Figure 6, each of the pollution objects in different scenes are assigned with different weighted values for five classified pollution types, including eutrophication, heavy metal pollution, dioxin, acid rain, and solid waste.

A total of 29 first-year junior high school students participated in the VR learning experience. Students were divided into two groups. Due to equipment limitations, the two groups were tested at different time periods. First, the teacher explained the content of the VR experiment and conducted a pre-test of the survey. Next, students had turns with the VR experiences. Finally, a post-test questionnaire was conducted. For each pre- and post-test survey, students needed to complete the questionnaire in 10 min.

4.3. Results Based on Questionnaires

In the experimental test, a total of 29 first-year junior high school students participated in the pre- and post-tests. For each pre- and post-test survey, students completed the questionnaire surveys in 10 min. The test results are described as follows.

4.3.1. Pre-Test and Post-Test Data Overview

VR Course Identity

One hundred percent of students agreed the “VR Learning Course” is helpful for personal learning in the pre- and post-tests, as shown in Figure 7. On a scale of 1 to 4, the average scores for the pre-test and post-test in the class were 3.38 and 3.84, respectively. The details are presented in

Table 1. Student’s pre- and post-test responses for VR course identity (n = 29).

Items	Pretest		Posttest
	Means. (Max. 4)	Std. Deviation(σ)	Means. (Max. 4)	Std. Deviation (σ)
Q1	3.39	0.5	3.9	0.3
Q2	3.43	0.66	3.86	0.36
Q3	3.26	0.75	3.86	0.36
Q4	3.26	0.54	3.9	0.3
Q5	3.26	0.54	3.67	0.58
Q6	3.13	0.69	3.76	0.54
Q7	3.35	0.65	3.9	0.3
Q8	3.48	0.51	3.81	0.4
Q9	3.43	0.66	3.86	0.36
Q10	3.39	0.58	3.81	0.51
Q11	3.48	0.51	3.81	0.4
Q12	3.65	0.49	3.9	0.3
Q13	3.39	0.72	3.86	0.36
Average	3.38		3.84
Cronbach’s Alpha	0.98		0.96
Scale: 1–1.74 (disagree), 1.75–2.49 (a little bit disagree), 2.5–3.24 (a little bit agree), 3.25–4 (agree). Q1. I think the “VR learning course” makes my learning more effective. For example, it helps me to better understand the content of the class and apply the knowledge I have learned flexibly in my life. Q2. I think the “VR learning course” helps me study more effectively. For example, I finish studying in less time. Q3. I think after class, I want to continue to use the software and hardware of the “VR learning course” to help myself learn, such as querying materials, cooperative learning, and after-class exercises. Q4. I hope future courses can also be conducted using the “VR learning course” method. Q5. I think the “VR learning course” helps me better understand how to communicate and interact with my classmates. Q6. I think the “VR learning course” helps me better understand how to work with my classmates as a team. Q7. I think the “VR learning course” makes me more capable of creative thinking. For example, thinking about things can be more imaginative, flexible, unique. Q8. I think the “VR learning course” enables me to have more critical thinking skills, e.g., being able to judge the truth or falsehood of information and be able to reason and prove without being influenced by others. Q9. I think the “VR learning course” makes me more capable of problem-solving. For example, when I encounter problems, I can come up with more solutions, and I can think about the effectiveness and feasibility of various solutions. Q10. I think the “VR learning course” can help me improve my ability to use software, e.g., can use writing software as a notebook, presentation software as PowerPoint, other software, and video editing software. Q11. I think the “VR learning course” can help me improve my ability to operate hardware devices. Q12. I think the “VR learning course” can help me improve my information ethics literacy, e.g., you cannot arbitrarily copy other people’s text or pictures as your own, respect others’ speeches on the Internet, be responsible for your own speeches and be able to correctly judge Internet rumors. Q13. I think my family will support the teaching and learning activities of the “VR learning course”.

.

Intentions for Problem-Solving

It was shown 100% of students had good intentions for problem-solving in the pre-test and post-test, as shown in Figure 8. On a scale of 1 to 5, the average scores for the pre-test and post-test in the class were 4.12 and 4.59, respectively. The detail are presented in

Table 2. Pre- and post-tests for students’ behavioral intentions regarding problem solving (n = 29).

Items	Pretest		Posttest
	Means (Max. 5)	Std. Deviation(σ)	Means (Max. 5)	Std. Deviation (σ)
Q1	4.18	0.85	4.81	0.51
Q2	4.27	0.83	4.67	0.48
Q3	4.14	0.89	4.52	0.75
Q4	4.27	0.88	4.71	0.56
Q5	4.05	0.84	4.57	0.6
Q6	4.05	0.84	4.48	0.68
Q7	4.41	0.8	4.67	0.58
Q8	4.45	0.67	4.62	0.59
Q9	4.27	0.77	4.62	0.67
Q10	4.18	0.73	4.57	0.75
Q11	3.95	0.79	4.67	0.58
Q12	4	0.69	4.52	0.75
Q13	3.77	0.97	4.57	0.68
Q14	3.95	0.84	4.48	0.6
Q15	3.77	0.87	4.57	0.68
Q16	3.91	0.87	4.57	0.6
Q17	4.09	0.81	4.62	0.67
Q18	4.27	0.7	4.38	0.8
Q19	4.23	0.81	4.76	0.44
Q20	3.91	0.87	4.52	0.68
Q21	4.36	0.73	4.62	0.67
Q22	4.05	0.9	4.71	0.56
Q23	4.23	0.81	4.48	0.75
Q24	4.18	0.66	4.52	0.6
Q25	4.05	0.79	4.62	0.67
Q26	4.14	0.71	4.43	0.75
Q27	3.77	0.97	4.62	0.59
Q28	4.5	0.67	4.62	0.67
Average	4.12		4.59
Cronbach’s Alpha	0.99		0.98
Scale: 1–1.99 (very bad), 2.0–2.99 (not good), 3.0–3.99 (good), 4.0–5 (very good). Q1: When I encounter a problem, I believe I can solve it. Q2: When I had a problem before, I solved it. Q3: With my own efforts, I believe I can solve the problem I encounter. Q4: I am willing to face the problem and find a solution. Q5: When I have a problem, I do not run away. Q6: I can often ask questions about things around. Q7: Before solving a problem, I would first think about what the problem is. Q8: I think it is important to know where the problem is before solving it. Q9: I know what exactly the teacher is asking. Q10: In addition to the problem, I also understand the reasons associated with the problem. Q11: In the process of solving problems, I often collect relevant information. Q12: When encountering a problem that needs to be solved, I will first think about the methods and steps to solve the problem. Q13: I will work with others to solve problems together. Q14: When solving problems, I can distribute everyone’s work. Q15: I want to come up with fun and creative ways to solve problems. Q16: I can think of many ways to solve the problem. Q17: In the process of solving problems, I can be honest and do not cheat. Q18: When solving a problem, I think it is necessary to compare the possible consequences of each solution. Q19: I think a standard comparison is needed to determine whether the used method is appropriate. Q20: I will design some experiments to try and see if I can solve the problem. Q21: I can ask questions or suggest solutions that others propose. Q22: I can judge which solution is better for everyone’s comments. Q23: After I come up with a solution to the problem, I will seriously implement it. Q24: During the problem-solving process, I was very patient until the problem was solved. Q25: When I fail to solve the problem, I will try another method. Q26: After the problem is solved, I will compare the difference between the expected and the actual result. Q27: Although the problem is solved, I will still think about whether there is any other better way. Q28: I will apply the methods I learned to solve the problems encountered in my life.

.

4.3.2. Data Description

Total post-test results of the VR course identity and problem-solving intentions were more highly significant (p < 0.05) than the pre-test results in Table 1 and Table 2. The means and standard deviations were calculated for further discussions.

Part 1: VR Course Identity

Item values in the questionnaire represent the degrees of VR course identification. Item values < 1.74 are considered as disagree; values between 1.75 and 2.49 as a little bit disagree; values between 2.5 and 3.24 as a little bit agree and scores more than 3.25 as agree.

Based on the analysis results, as shown in Table 1, the reliability of the survey for both student’s pre- and post-test responses for VR course identity ≥ 0.9 (number of questions = 13); thus, it can be concluded the survey results developed for VR course identity has excellent internal consistency and high reliability. Additionally, 100% of the students in the class agree the “VR learning course” is helpful for self-learning. The average scores of the pre- and post-tests in the class were 3.38 and 3.84 on a scale of 1 to 4, respectively.

Part 2: Students’ Behavioral Intentions Regarding Problem Solving

Item values in the questionnaire represent students’ intentions regarding their problem-solving. Item values < 1.99 are considered as “very bad”; values between 2.0 and 2.99 as “not good”; values between 3.0 and 3.99 as “good”; and scores more than 4 as “very good”.

Based on the analysis results, as shown in Table 2, the reliability of the survey for both pre- and post-test for students’ behavioral intentions regarding problem solving ≥ 0.9 (number of questions = 28); thus, it can be concluded the survey results developed for students’ behavioral intentions regarding problem solving has excellent internal consistency and high reliability. Additionally, the result shows 100% of the class has good or very good problem-solving intentions. On a scale of 1 to 5, the average values of pre- and post-tests in the class were 4.12 and 4.59, respectively.

4.4. Impact on Learning Outcomes

In

Table 3. Paired t-tests analysis results comparing pre-test and post-test mean scores for VR course identity.

	Variable 1	Variable 2
Mean	3.376923077	3.838461538
Variance	0.01705641	0.004497436
Observations	13	13
Pearson Correlation	0.884276178
Hypothesized Mean Difference	0
Degrees of Freedom, df	12
T Stat	−21.36959658
P(T ≤ t) one-tail	0.003213992
t-Critical one-tail	1.782287556
P(T ≤ t) two-tail	0.006427984
T Critical two-tail	2.17881283

, the p-value is much less than 0.05, which proves it has a significant effect on the course identification after the implementation of the VR learning course.

In

Table 4. Paired t-tests analysis results comparing pre-test and post-test mean scores for students’ behavioral intentions for problem solving.

	Variable 1	Variable 2
Mean	4.121428571	4.59
Variance	0.039597884	0.0096
Observations	28	28
Pearson Correlation	0.972790695
Hypothesized Mean Difference	0
Degrees of Freedom, df	27
T Stat	−23.36134756
P(T ≤ t) one-tail	0.009589838
t-Critical one-tail	1.703288446
P(T ≤ t) two-tail	0.001917968
T Critical two-tail	2.051830516

, the p-value is much less than 0.05, which proves it has a significant effect on students’ behavioral intentions for problem solving after the implementation of the VR learning course.

5. Discussions

Many studies [1,2,3,4,5,6] reveal virtual reality technology has demonstrated positive educational outcomes. In VR learning courses, students’ use of VR teaching materials has a positive impact on their learning outcomes. In this case report, the aim is not only to develop competency-based VR learning materials but also to evaluate students’ VR course identity and problem-solving behavioral intentions. First, in the interactive design of competency-based VR learning materials, we followed human intuitions and experiences to reduce users’ thinking and ego depletion for a better user experience. In the implementation of interactive VR learning material, we use the familiar camera element as a collection tool for users to explore in the virtual world. Using familiar visual elements can reduce burdens of human cognition for functionalities, but it is undeniable that there are still many hardware interface design limitations that need to be improved, such as VR handle’s operations and so on. For unfamiliar user interfaces, most users need a longer adaptation period. Secondly, using the learning records, teachers can encourage students to explain and elaborate their understanding of concepts in new teaching–learning activities.

Additionally, we assessed the impact of the VR learning course on students’ VR course identity and behavioral intentions for problem-solving using the proposed competency-based VR learning materials. An increase in learning intent will have a positive impact on learning outcomes. The results of the questionnaire reveal 100% of the first-year students at a junior high school all agree the “VR learning course” is helpful for personal learning. Additionally, they have a high course identity with VR learning courses, and their behavioral intentions for problem-solving have been improved significantly.

6. Conclusions

Different teaching methods affect competency development in different ways, and different types of learning materials have a great influence on students’ self-learning [5]. In the VR competency-based learning experiment, we attempted to improve the VR course identity and problem-solving intentions and stimulate the learning of problem-solving abilities through VR experience.

In this case report, the interactive competency-based VR learning materials are implemented following human intuitions and experiences to reduce users’ thinking and ego depletion. It also analyzes and discusses the learning impact results before and after the VR experience in junior high school. In the VR experimental results, it was found the first-year students at a junior high school have a high course identity with VR learning courses, and their behavioral intentions for problem-solving improved significantly. It is expected through the VR content, students could learn to observe and pay much attention to their surroundings to understand the interaction between humans and the environment. Next, we will conduct a quasi-experimental study using the proposed competency-based VR materials to further analyze participants’ responses with VR course identity and learning intentions of problem-solving abilities.

The VR immersive experience can effectively enhance motivation and engagement of students’ learning. Different from traditional teaching ways, the VR courses require the development of their own teaching methods and learning material designs. However, the evaluation of learning effectiveness for competency-based VR course plans have always been one of the important research topics for educations. Critical thinking and problem solving are some of the most important development skills for 21st century learning outcomes. Educators should think about how to cultivate students to develop these skills in the learning processes. Thus, it is becoming increasingly important to develop competency-based lesson plans.