VR-Enhanced Cognitive Learning: Method, Framework, and Application

Abstract

Both constructivist learning and situation-cognitive learning believe that learning outcomes are significantly affected by the context or learning environments. However, since 2019, the world has been ravaged by COVID-19. Under the threat of the virus, many offline activities, such as some practical or engineering courses, have been subjected to certain restrictions. Virtual Reality (VR) is an emerging, promising, and rapidly developing technology that enables users to obtain a near-real immersion experience by combining technologies such as computer science, communication, vision, etc. In the context of COVID-19, the advantages of VR immersive experiences are highlighted. By constructing a virtual learning environment, VR technology can greatly compensate for the shortage of traditional teaching conditions and help learners to carry out cognitive learning better. However, currently, VR-enhanced cognitive learning is still in its infancy, along with numerous problems and limitations. Therefore, this paper first conducted an in-depth study of some related concepts, such as constructivist learning and situated cognition learning. Then it proposes a general VR-enhanced cognitive learning framework and designs the general steps for constructing learning situations with VR technology. Based on the proposed model and framework, it developed a campus knowledge-learning APP using VR tools. Through a case study, it verified the validity and performance of the model and strategies. Questionnaire survey and experimental results show that the new model achieves a good learning effect and improves the efficiency of learning by at least 20% compared to the traditional learning methods.

1. Introduction

Constructivism is a branch of cognitive psychology that studies the way individuals perceive and understands the world. Constructivists believe that learning is no longer part of a simple transfer of knowledge from teachers to students but a process of knowledge self-construction process based on the existing knowledge base and the specific environment. In the view of constructivists, learners are the leaders of learning, not teachers. The connotation of learning is that learners actively explore knowledge and construct meaning with the help of facilitators (teachers or learning partners) in specific situations (cognitive environment, social context, cultural context, and so on). Obviously, in the process of cognitive learning, the creation of a situation is the premise, the reconstruction of meaning is the goal, and conversation and negotiation are the means to achieve the goal. Therefore, context is the key factor in implementing efficient cognitive learning.

However, in the traditional teaching mode, most of the knowledge is explained by teachers orally, and the concepts and principles are presented through text, pictures, or videos, which requires students to give full play of their imagination to understand. Naturally, the teaching effect is not very ideal, especially for teaching engineering practice-related courses. For these courses, if students can be in the field site and directly participate in the practice process, the learning effect will greatly improve. However, for the sake of safety or cost, not all of the engineering courses are suitable or possible for students to experience directly. Thus, the role of virtual reality technology appears. It can create near-reality but repeatable scenes, thereby reducing costs and improving the participation ratio of students. Virtual reality technology is good at creating cognitive scenes which are needed for education and teaching, including virtual teaching environments, virtual laboratories, etc., to make up for the deficiency of real conditions.

Since 2019, COVID-19 has broken out all over the world. The spread of the virus is extremely strong, causing many infections and deaths worldwide. The group gathering behaviors, such as centralized classroom teaching, field experience activities, and especially cross-regional activities, have been strongly restricted, which becomes a serious obstacle to the construction of the learning environment [1]. The advantages of VR technology in creating virtual reality scenes and classrooms are gradually showing. Theoretically, virtual reality can virtually display any person, anything, or any object according to learning requirements and design dynamic hierarchical scenes according to task requirements. With the aid of VR technology, educators and learners can feel the unique situation of teaching practice without leaving home and complete the learning process through the interaction between learners and the environment. Immersion and interactive features created by VR teaching mode are one of the effective ways to realize constructive learning.

Recently, using VR technology to promote constructive learning has attracted the attention of some researchers. For example, Niţu et al. [2] developed a VR-based teaching application for the computer architecture course. Meyer et al. [3] designed a VR game for simultaneous multiple people engineering training. Seo et al. [4] developed a VR learning system for anatomy education. However, the research on VR-enhanced cognitive learning is still in the primary stage; still, many problems need to be addressed: (1) the relationship between VR and cognitive learning has not been dissected, (2) a lack of a general VR reinforcement learning model, (3) the existing VR cognitive learning application platforms being strong scene limitations, and (4) the limit of the learning effect, etc.

This paper studies the theory of VR-enhanced cognitive learning and demonstrates the relationship between them. Based on it, it designs a general VR learning model to improve the learning effect in the context of COVID-19 and makes a systematic development and empirical analysis.

The innovations of this paper are as follows.

• It makes an in-depth investigation and analysis of the existing VR educational applications and demonstrates the relationship between VR, constructivism learning, and situated cognition theory;
• It constructs a novel VR-enhanced cognitive learning model and designs the general method and steps for VR learning situation creation in restricted conditions;
• A VR learning APP is developed under the framework of the proposed learning model, and its effectiveness is verified by experiments.

The following sections are organized as follows: Section 2 introduces the theoretical background, Section 3 describes the research status of the related technologies and general VR-enhanced education, Section 4 proposes a teaching model, Section 5 is the case study, and the last section, the conclusion and future work.

2. Theoretical Background

2.1. Constructivist Learning Theory

Constructivist learning is a novel cognitive theory. Contemporary constructivism theory can be traced back to the theory of children’s cognitive development founded by the famous Swiss psychologist Jean Piaget. Constructivism maintains that the world exists objectively; however, the understanding of things is determined by each individual [5]. Different people have different understandings of the same thing because of their different experiences. Therefore, learning should guide students to construct their characteristic knowledge based on their original experience.

The viewpoint of constructivist learning theory includes knowledge view, learning view, and teaching view. The knowledge view of constructivism questions the objectivity and certainty of knowledge and holds that knowledge is subjective and dynamic, which is constantly changing and deepening with people’s in-depth understanding of things. The learning view emphasizes that learning is a process in which students actively receive information and construct information according to their personal experience and form their own opinions. The teaching view holds that teachers should attach importance to students’ autonomy in the teaching process and guide students to construct new knowledge systems based on their prior knowledge.

There are four major elements in the constructivist learning theory: Situation, Collaboration, Conversation, and Meaning construction [6]. In addition, the characteristics of learning include (1) Initiative, learning should be a process in which learners actively construct new knowledge based on the existing cognitive structure; (2) Diversity and heterogeneity, the world is diverse, and the learning background and environment are different. Therefore, the knowledge structure constructed by different individuals is different; (3) Collaborative learning should be full of cooperation. Knowledge structure is constantly revised and improved. Constructivism advocates that learning processes are dominated by learners and supplemented by teachers and collaborators. Teachers are required to guide students in constructing knowledge rather than blindly instilling knowledge. Students are required to construct knowledge spontaneously rather than passively accept information. Students choose and process information according to their own experience and finally form their unique understanding.

2.2. Situated Cognition Theory

Situated cognition theory is another famous theory that appears at the same time as constructivist learning after the stimulus-response learning theory of behaviorism and the “information processing” learning theory of cognitive psychology. Situated cognition theory attempts to correct the mistakes of cognitive symbolic operation methods, especially the cognition that completely depends on rules and information description, which only focuses on the cognition of conscious reasoning and thinking, ignoring the cognition of cultural and physical background.

Situated cognition theory holds that practice is not independent of learning, and meaning is not separated from practice and situation context. Meaning is produced by the interaction between practice and situation. Knowledge ceases to be a collection of facts and rules but a dynamic construction and organization. It is only an interactive state constructed in the interaction process between the individual and the environment. Situated cognition theory also believes that the learning content comes from the working or task simulation situation; thus, the construction of knowledge is a function of the situation to a certain extent. Students form their unique knowledge systems under the influence of individuals and special situations.

Viewpoints of Constructivist learning that learners can obtain personalized cognitive structure under the influence of prior knowledge and specific context coincide with the emphasis on the impact of the situation (practice) on cognitive structure in situated cognition theory. Exploration of constructivist learning can be combined with situating cognition learning.

3. Literature Review

3.1. Virtual Reality

Virtual Reality (VR) is a kind of technology that makes use of computer technology to build simulated real scenes, providing users with immersion in real-world experience in the process of interaction with the virtual world. Because the object, landscape, or scene created by VR technology is simulated rather than actual existence, it is known as virtual reality. VR is the combination product of computer graphics, digital image processing, computer simulation, multimedia, artificial intelligence, sensors, and the Internet. In the research field of VR technology, panoramic virtual reality is one of the hot spots. Panoramic virtual reality technology comprehensively restores the three-dimensional spatial relationship of the scene by analyzing and reconstructing the geometric relationship of the scene image or picture to realize the omnidirectional display of the target scene [7].

In recent years, panoramic technology has developed rapidly, including 360-degree omnidirectional live images, videos, and Augmented Reality (AR) technology further derived from VR, which is widely used in many fields such as medicine, art, design, real estate, military, archaeology, entertainment and so on, bringing huge economic benefits and social benefits to the society.

Based on a variety of traditional technologies, the widely hyped Metaverse can be regarded as an upgraded version of virtual reality. The Metaverse is a virtual living space with a new social system constructed by human beings using digital technology, which is mapped or transcended by the real world and can interact with the real world. The head companies related to Metaverse include Roblox, Unity, Nvidia, Meta (Facebook), Microsoft, etc. At present, the study of Metaverse is still in its infancy, showing the characteristics of diversification and crossover [8].

3.2. Application of Virtual Reality in Education

In the field of teaching, the use of VR technology can make up for the shortcomings of real conditions, establishing a more systematic and vivid virtual practice environment to stimulate learners’ interest in learning and break through the key and difficult problems in teaching [9]. VR has become an effective way to improve teaching methods.

As the founder and leader of VR technology, the United States and European countries have made a lot of attempts in the field of military, sport, and medical education and teaching [10]. Chinese universities have also created a large number of VR teaching platforms in physics, aviation, architecture, chemistry, and other disciplines.

3.2.1. Effect

The promoting effect of VR technology on teaching is mainly reflected in the following three aspects.

• Creating an immersion experience
  VR technology allows students to have an immersion experience. Students accept the influence of simulated real scenes through various sensory stimuli, to strengthen the memory of knowledge, increase practical experience, and improve learning interest.
• Realizing the interactive teaching
  VR helps students interact with the scene through their senses. For example, students can control characters’ walking and interact with virtual characters in VR scenes. This increases interest and helps students discover information, explore and expand their knowledge. In the teaching process, teachers use VR technology to present complex data in an accessible, interesting, and simple way, improving the teaching effect [11].
• Efficient learning
  With the help of VR technology, teachers can make students connect theory and practice to learn curriculum knowledge through the design of VR scenes and the support of a large number of interactive technologies. This helps students make up for the confusion of theory caused by not being in the background era, enabling them to seriously think and analyze problems from multiple angles and to obtain a satisfactory learning experience.

3.2.2. Research Progress

Applications of VR in education are mainly in the following aspects.

• Virtual learning environment
  Second Life is an immersion and cooperative online game-learning platform developed by Delucia et al. [12]. It is used to teach many subjects such as educational technology, computer, history, media, and writing. The Second Life platform has a large market in Western developed countries, including 80% of the universities in the UK, and more than 150 universities in the United States for establishing teaching and scientific research platforms [13].
  Sloodle is a three-dimensional learning environment developed based on the second life, which supports various forms of interactive teaching activities such as role-playing, collaborative learning, and activity construction [14].
  i3DVLE is a three-dimensional virtual learning platform developed by Cai and Yao [15] for primary and secondary school teaching. After that, the research team successively developed a series of teaching assistant software. There are also many other VR-assisted teaching systems, such as the probability theory learning tool developed by Li et al. [16], and the AR English learning system developed by Li et al. [17]
• Virtual lab
  As is known, further understanding of knowledge needs the combination of theory and practice. However, many experiments require certain environments or specific types of equipment, which are often accompanied by certain risks. For this reason, many students can not personally participate in such experiments. The emergence of VR laboratories allows students to experience some experiments with high requirements for environments and equipment. This not only reduces the experimental expenses but also allows students to have an immersion experimental experience and strengthen their understanding.
  Cai et al. [18,19,20] combined AR and somatosensory technology to realize the visualization of physical experiments. Kerawalla et al. [21] developed an astronomy experiment teaching platform based on VR technology. Kaufmann et al. [22] built a virtual reality mechanics laboratory. Chiang and Yang [23] developed a virtual experiment system for science teaching in middle schools. The geographical and atmospheric learning experimental platforms were designed by Xiao and Chang [24].
  Buono et al. [25]. designed a set of virtual experimental teaching environments for fire escape training, and the effect is remarkable. With the help of VR technology, Eugenia et al. [26] built a laparoscopy surgery training environment for training basic surgical skills. Chu et al. [27] designed the learning systems or tools for the teaching of natural science courses.
• Virtual Library
  In a VR library, students can read electronic books and search for relevant book information. The virtual library facilitates the storage and reading of information while protecting the content of books and preventing the loss of books. The earliest e-book realized by VR technology is the magic book made by Billinghurst and Kato [28]. It turns books into 3D scenes and animation for a vivid presentation. Cai et al. [29] developed a virtual book to show a large number of laws in physics, which is called the book of the future.

At present, the main limitations of VR-based learning applications are as follows.

• It can not well reflect the characteristics of autonomous learning. Although learners can independently choose the learning content and progress of the platform, they can not actively participate in the production or planning of learning objectives and content.
• The interleaving of teaching and learning and the interactive design of teaching and learning are still not perfect, which can not well combine the two tasks of virtual learning and teaching management.
• The sense of experience and immersion in virtual reality need to be further improved, and there are still some differences between the virtual learning environment and the real scene.

4. Methodology

4.1. Research Design

This paper designs a VR-enhanced cognitive learning model, as shown in Figure 1. The model is divided into three layers: the perception layer, the interaction layer, and the application layer. The perception layer is the physical realization of the VR learning environment. On the computer (machine) side, various elements of the physical world are captured by sensors, audio, and video equipment, simulated through physical devices, and a human-computer interaction interface is provided to realize the interaction with learners. Learners fully use their senses, limbs, and brain to integrate with the VR learning environment. The interaction layer realizes a higher level of the virtual world interacting with people from multiple perspectives than physical immersion, including sensory, behavioral, scene, and social simulation. In addition, the learners’ activities include perception, mental process, stress reaction, and cognitive model. The application layer realizes deep immersion and interaction oriented to specific functions and specific application scenes and constructs VR scenes, VR education, VR training, and the VR laboratory, to help learners explore solutions to specific subjective problems in the simulated world.

Depending on the goals and requirements of each layer in the cognitive learning model, the VR-enhanced cognitive learning framework is proposed in Figure 2. The framework contains four phases: the design phase, modeling phase, implementation phase, and evaluation phase. The core issues in the design phase are: analyzing cognitive goals, setting VR goals according to the learning goal, analyzing the specific learning scene and the content faced by learners, designing the VR scene, and finally constructing the virtual elements of the VR world according to the various learning elements involved in the learning process. The main task of the modeling phase is to accurately model learners’ senses, emotions, learning behaviors, and abilities. In the implementation phase, computer software tools are utilized to realize the VR learning environment and platform, and the main functional modules in the implementation phase are planned according to the level of the immersion experience. In the evaluation shape, the effect of the learning model is comprehensively evaluated, a judicious evaluation index system is designed, and the model is optimized and improved.

4.2. VR Situation Construction Method

There are two ways to construct the VR situation: (1) to analyze the elements and then associate the elements; (2) to generalize and describe the situation and then conclude the specific relationship. The creation of the situation forces us to transform the traditional teaching design into the design of the situation learning environment and integrate the content into the situation learning experience according to different learning objectives. Therefore, students can obtain better learning methods and enhance learning efficiency through vivid experience, which is one of the crucial goals of teaching design. The use of VR technology in the construction of the situation can better reflect the interactivity, immersion, and imagination of the situation. VR learning situation includes four basic dimensions: user situation, technology situation, resource situation, and service situation [30]. When using VR technology to design teaching situations, the principal elements to be considered include dramatic story statements, performance, reinforcement, compression, single action, termination, scope, interaction, and suspension of doubt [31]. Further design of these elements can better stimulate learners’ interest and make learners feel that they are involved in situation learning and have a better interactive experience so that learners can have higher learning efficiency.

Figure 3 shows the methods and steps of creating VR situations designed in this paper. Firstly, we utilize VR modeling tools to build a virtual learning scene according to the need of a specific learning objective (experimental environment). Secondly, it creates problems or conflicts artificially in the scene to realize learners’ experience of possible problems. The third is to generate problem-solving method space in the database to guide learners to learn how to resolve the problems. The fourth is to construct derivative problems to stimulate learners’ innovative thinking. Finally, it guides learners to summarize knowledge points and make in-depth exploration.

5. Case Study

Under the guidance of the above methods, this paper uses VR technology to build a campus knowledge learning APP to help freshmen quickly get familiar with campus geography, learn about school history and culture, the latest activities, and campus encyclopedia knowledge.

5.1. System Design

The APP mainly includes four parts: a VR campus, an encyclopedia of school history, an automatic navigation system, and games, as shown in Figure 4.

• VR campus module. This is the main interface module of the system. Through the disc, the player can control the virtual characters to move around and browse the campus environment from the perspective of the characters. The player can also switch the weather through the weather control button to experience the campus scenery under diverse weather conditions. The system can also automatically identify the day and night to change the scene.
• School history encyclopedia module. This module realizes many functions, such as VR school history museum, campus culture introduction, school activities, campus address book, Knowledge Q&A, etc. When the player controls the character to walk near a special landmark building, the introduction of the corresponding place will appear to help him get familiar with the campus environment and learn relevant campus knowledge.
• Navigation system module. The player can choose any place he wants to go, and the navigation system will automatically realize route planning. The player can also click the scene-switching button to select the place he wants to go to trigger the direct place jump.
• Game module. The system designs several fun games, including the role-playing games. By filling out the tasks in the game, players can obtain game coins, which can be consumed in a VR mall.

5.2. Main Functions and Features of the VR App

5.2.1. Support Simultaneous Control by Mobile Phone and Computer

The VR app supports simultaneous control by mobile phone and computer, and realized 3d realistic experience, as is shown in Figure 5. The player can choose a VR role, and with the 3D view of the campus as the background, he can walk freely around the campus. When he is close to the landmark buildings, the introduction of campus activities appears, which makes him better understand the campus environment and get familiar with campus life.

5.2.2. VR Scene Modeling and Design

Scene modeling and design of the app can restore 90% of the real campus. As shown in Figure 6, we have carried out a real-scene shooting and 3D modeling restoration of landscapes and buildings. At the same time, the app can automatically identify day and night and freely switch between different weather conditions, such as sunny, rainy, heavy rain, cloudy, etc.

5.2.3. VR Character-Free Action

In the campus browsing interface, the player can control the characters to move around and visit the campus from the perspective of the characters, as shown in Figure 7. The app supports free walking and direct destination jumping. If the player clicks the navigation button, the navigation starts working automatically, and without controlling the disc manually, it will plan the route and help the character to walk to the corresponding place.

5.2.4. VR School History Museum

In the VR School History Museum, the player can personally browse the relevant information on school history and culture for a better understanding. As shown in Figure 8, the VR school history museum is presented in the form of panoramic VR, and the interface will rotate automatically. The player can also manually drag the interface to browse, click the corresponding block and jump to the corresponding campus knowledge introduction.

Figure 7. VR character action.

Figure 7. VR character action.

5.2.5. Immersive Role-Playing Games

Figure 9 shows the role-playing games designed in the App. The games are configured with multiple independent scenes. Players can interact with the scenes and props and can switch freely in different scenes. Most of the scenes comprise knowledge quiz games, where players can upgrade their level or gain a certain number of game coins by the appropriate interact behaviors or correct answers. The questions are randomly selected from the database.

6. Results and Discussions

In this paper, two groups of experiments are designed to evaluate the effect of the mentioned VR app on cognitive learning. The first experimental group belongs to the subjective evaluation, while the second belongs to the objective evaluation.

6.1. App Satisfaction Questionnaire

This paper conducted a user satisfaction survey on the VR campus knowledge learning app. The questionnaire focuses on three contents: (1) the testers’ experiences when using the VR App (Question1 8), (2) users’ evaluation of the promotion effect of the APP on their cognitive learning structure (Question 9), and (3) users’ view on VR promoting learning (Question 10). The questionnaire was distributed to 120 learners who learned campus knowledge through the APP. The subjects were volunteers who agreed to use the app and accepted the questionnaire survey, whose gender, age, occupation, and educational level varied, including 58 males and 62 females, aged between 18 and 30 years. All the subjects signed the ICF (Informed Consent Form). The design of app satisfaction questionnaires is according to the methodology provided in [31].

Figure 10 shows the design of the questionnaire, and Figure 11 shows the results. From the results, we can see that most users are satisfied with the functionality, interactivity, and VR realism of the app, and are willing to continue to use and will recommend the app to other users. Secondly, the vast majority of learners believe that VR is conducive to cognitive learning, and are optimistic about the application of VR technology in the field of education and teaching. The results also demonstrate that some users are not satisfied with the running speed of the app, which shows that building a virtual learning environment with VR is greatly affected by the software and hardware conditions and the Internet speed. To improve VR and enhance cognitive learning, we are required to improve the corresponding teaching support conditions at the same time.

6.2. Effect of VR on Enhancing Cognitive Learning

This paper also designed a set of experiments to test the auxiliary effect of VR-enhanced cognitive learning. In this experiment, we also invited 120 subjects. Different from the previous app experience experiment, this time the participants are all university freshmen, with similar ages and education levels, with little campus-related knowledge. The participants were divided into four groups: (1) VR learning group, (2) flat learning group (obtaining information by printed resources or online resources), (3) field group (obtaining information by visiting the place), and (4) non-learning group. A test paper with a full score of 100 regarding campus-related knowledge was designed for them. We explicitly told the testers the learning targets, gave the first three groups half a day to study, and then required them to finish the test. The discipline of the test is also based on the methodology proposed in [31].

Figure 12 shows the scores of the learners in the four groups. The horizontal axis represents the 30 learners in each group, and the vertical axis represents their scores. Figure 13 shows the comparison of the standard deviation and the average score. We compared the standard deviation (blue histogram) and average score (orange histogram) of each group.

It is obvious that the VR learning group scored the highest, followed by the field investigation group. Field visitors can obtain the most intuitive impression of the knowledge, which is obviously helpful to improve the learning effect. However, there are certain shortcomings in the learning of pure knowledge content. VR-assisted learning can effectively combine offline (through virtual reality scenes) and online (knowledge unit) to achieve the best learning effect. This test also proves that the objective learning effect is reinforced.

7. Conclusions and Future Work

Cognitive learning emphasizes the learners’ subjective initiative. It is proposed to create a reasonable situation to stimulate learners’ interest in learning to achieve good learning results. VR is one of the most efficient tools for learning situation construction, especially in the context of COVID-19. The learning environment created by VR can not only make up for the shortcomings of traditional teaching conditions but also greatly stimulate learners’ initiative, exploration, and innovation ability through a multi-dimensional and multilevel immersion experience. At present, although VR has been more and more commonly used in the field of education, its theoretical research, development, and application are still in the primary stage. This paper conducts an in-depth study on constructivist learning, situation cognitive learning, and VR technology, and explains the relationship between the related technologies. Based on it, it puts forward a novel VR-enhanced cognitive learning model and designs the general methods and steps of VR to create a learning situation. The implementation process and learning effect of the model and method are analyzed through a case study. The VR app based on the proposed model improved the testers’ understanding of the corresponding knowledge by more than 20%.

Although the design and development of the VR app are based on the proposed theoretical model and methodology, its functions and implementation technology seem a bit simple. Therefore, in future work, we will continue the in-depth research of the theoretical model, and we hope to carry out more complex applications to test the performance and effect of VR-improved cognitive learning.

Creating a cognitive learning situation with VR can essentially promote the meaningful integration of VR and cognitive education, and help to realize learners’ knowledge construction in a visual, intelligent, and personalized way, so as to achieve the best learning effect.