Exploring the Perception of the Effect of Three-Dimensional Interaction Feedback Types on Immersive Virtual Reality Education

Abstract

Immersive virtual reality (VR) systems are becoming widely used for education with three-dimensional (3D) information. Specifically, three-dimensional spaces to create virtual environments can help increase students’ learning interest and ability with spatial interaction. Also, with the use of multimodal interaction, VR systems can provide highly effective ways to solve problems through natural experiences. Additionally, immersive environments can bring together people in remote locations, which has been increasingly applied in education applications with the use of technology to simulate real situations. However, effective interaction methods that improve the learning ability of people participating in educational activities in 3D immersive environments are yet to be well defined. In this study, we investigated the effect of the interaction feedback types on the perception of students participating in VR environments. We conducted the experiment on three types of interaction responses, and our study was designed as a virtual chemistry class. Our experimental study showed that the interaction feedback type of a deformable object had a greater educational effect than other types of visual or audio feedback, and our results are expected to provide guidelines on how to create effective immersive education content and interaction methods.

1. Introduction

Recently, there have been many extensive developments in immersive interactive content; in particular, the influence of virtual reality (VR) systems is expected to grow rapidly as a futuristic technology for education by adding effective information and realistic interaction [1,2]. VR systems can create various situations in immersive 3D environments and provide interaction to improve a user’s experience [3]. Also, 3D virtual spaces can increase educational effectiveness by allowing learning to take place anytime and anywhere [4]. More recently, with continued advances in sensing technologies, multimodal interaction with high accuracy has become much more feasible, and it is now possible to express interactive feedback realistically such as haptic feedback in VR systems [5]. Thus, students in VR environments can better understand the subject with interactive 3D materials. Also, remote learners can improve their comprehension by experiencing and delivering personalized feedback in collaborative environments [6]. Moreover, some researchers suggested that an interaction such as visual and tactile feedback may enable students to construct knowledge for science learning, and perception in immersive environments plays an important role in the learners’ cognitive development [7]. Despite such provisions, there is a lack of research on how to interact with virtual objects for education, and interaction responses for users in VR systems still have limited designs and limited effectiveness [8]. For example, for an immersive VR setup, we need to know which effective interaction is more suitable or natural in the case of the specific 3D situation (e.g., visual or audio). Furthermore, one recent study presented the relationship between learning and haptic feedback [9], and another research work suggested a method to enhance the VR experience in situations in which the physical haptic configuration and the virtual objects do not match [10]. Therefore, there has been a continued requirement to study interaction technologies that can replace haptic approaches in an immersive education environment.

Along these lines, we investigate the perception of the effect of interaction feedback in the case of the manipulation of 3D virtual objects for immersive environments, and specifically, we focus on finding appropriate interaction responses to improve learning ability for educational content. Thus, using chemistry class scenarios as one example, we performed comparative experiments on two types of visual interaction feedback and an audio response. During our experimental study with subjects participating in VR environments, we measured the interest, understanding, preference, and remembering in terms of the main evaluation elements of the learning effects.

Table 1. Candidates of test conditions in 3D environments with respect to the interaction feedback types.

Test Conditions	Feedback	Comparative Examples
Visual	Color change	Original color vs. changed color
	Object halo	No change vs. halo effect
	Scale transformation	No change vs. size change
	Deformable body object	No change vs. shape change
Non-visual	Audio	No audio feedback vs. audio output
	Haptics	No haptics vs. a sense of touch

shows the classification of the interaction feedback techniques [11,12]. We decided to use three types of interaction feedback that focused on measuring a scale factor and a deformable body parameter related to changing the shape representation of the virtual object for visual feedback and the audio element for non-visual feedback. Note that haptic interaction was excluded from this study due to the need for additional devices, and the study on haptic feedback is left for future research as mentioned earlier.

Figure 1 depicts a conceptual representation of the three types of interaction response scenarios for chemistry education, studying molecular bonding models. The participants in the immersive VR systems interacted with the virtual object (e.g., molecular 3D model) by wearing a head-mounted display (HMD) to study the chemistry concepts. Then, they used virtual fingers to select an individual element (or atom) and received feedback on the interaction results. In this study, we evaluated three types of interaction feedback (see Table 1), and the proposed comparative conditions were set to two types of visual feedback and one type of audio feedback. First, for the scale transformation, when the participant placed his/her finger on the molecular element, an animation growing and shrinking in the form of breathing was implemented to provide visual feedback. Second, to provide audio, the participant could continuously hear the element’s name. As the third and final feedback, also visual, when pressing the element with the virtual finger, as if a physical force was applied, the virtual element was made to react in a distorted form with a deformable body shape. Then, a participant in our VR system performed comparative evaluations of their interest, understanding, preference, and remembering associated with the quality of learning. Detailed information is provided in the section describing the implementation and evaluation.

The remainder of the paper is structured as follows: Section 2 addresses some research results directly related to this paper. Section 3 provides a general description of our approach and experimental configurations, and Section 4 provides detailed descriptions of the implementation. Section 5 reports the main results and provides a discussion about the analysis of the approach used in this study. Finally, the overall conclusions and future work are presented in Section 6.

2. Related Work

2.1. VR Systems for Education

Virtual reality technology is a real-time 3D computer graphics technique that experiences immersive virtual worlds [13]. Recently, in addition to the experience of a certain educational condition in a real space, students can increase their effectiveness with a highly immersive experience in a mixed-reality (MR) area by providing five types of sensory information rather than learning through 2D textbooks [14]. Educational studies in immersive environments have been conducted on whether the experience of participants in a mixed-reality setting is equivalent to a real-world setting [15]. For example, researchers developed AR education content to visualize virtual examples registered in textbook images, and the AR content was helpful to intuitively learn educational information by using more interactive situations [16]. As a result of another study, Kaminska et al. presented an insight into positive education outcomes when using AR technology and various evaluation methods to test its performance. In this paper, they provide detailed approaches to evaluate students’ performance and educational tools [17]. Also, such virtual 3D educational environments can include head-mounted display (HMD) devices. Pathan et al. suggested that VR learning environments have been mostly based on perception [18]. Additionally, Oliveira et al. investigated whether vibrotactile feedback was helpful to improve participants’ intercommunication in collaborative virtual environments (CVEs) [19], and Sarmiento et al. measured the collaboration degree by interaction elements and evaluated interaction quality in CVEs using five characteristics including predictability and awareness [20]. Here, their results suggested that interactive feedback would be helpful for immersive remote learning environments. Gradually, the interactive applications of immersive education are gradually increasing [21,22], and it is crucial to investigate how the interactive factors can be influenced to make the education effective [23]. However, in recent works, there have been few studies that have suggested the influence of responses on participants’ actions toward virtual objects. In our study, some of these concepts of previous works were borrowed and revised to handle interaction feedback and response in VR education environments.

2.2. Interaction Methods in Immersive Environments

In the field of virtual reality (VR) research, many studies have been centered on interactive designs for 3D object selection and manipulation [24]. Interaction is an input technology of VR systems to communicate with computing systems using natural interfaces [25]. For example, haptic interaction to improve the experience in VR environments is being used as a major approach increasing the participant’s sense of the presence of physical objects. More recently, many researchers have been concerned about appropriate interaction methods depending on the VR content scenario being experienced [26]. In one notable research work, an interactive technique called redirected walking was proposed for immersive environments to allow participants to walk in a virtual space larger than the real walking one [27]. Another example is raycasting to select virtual objects used in immersive environments, and the method, which is similar to a real-life laser pointer, is implemented in the form of emitting a ray of light to the target for 3D object selection [28]. With the recent and ongoing advances in sensing technologies, the virtual hand generated from interaction devices (e.g., glove-type, joystick, and depth-sensing-based bare-hand tracking) can recognize the participant’s hand motions [29]. However, no comprehensive research in terms of developing interaction feedback in a more effective way for immersive education systems has yet been conducted. It remains to be determined how the interactions should be configured to result in a high level of learning in the virtual environment. Our work was designed to explore the effect of 3D interaction feedback on the student’s perception and experience.

2.3. Comparative Studies for Interaction Elements

Recent research in virtual reality fields has focused on developing the selection and manipulation of 3D objects in the immersive environment [29]. Also, some results were recently introduced showing that sufficient sensory stimulation related to the feeling of the participants in virtual environments helped to improve the sense of presence through comparative studies [26]. Moreover, many researchers have considered users’ perception in using multimodal interaction technologies in order to evaluate the sense of presence when interacting with virtual objects in immersive environments [10]. Determination of the optimally provided interaction elements to overcome the limitations of conflicting information between the real object and the virtual one is a developing issue [10]. We propose an effective interaction strategy to support a high-level experience that improves the learning ability of educational activities in 3D immersive environments.

3. System Overview

Figure 2 shows the system configuration of the proposed virtual chemistry classroom: the participant wearing the immersive HMD can interact with a molecular model of the virtual object (e.g., an atom) using a joystick (or controller). Some elements of the educational process and evaluation methods were borrowed from recent research works on the chemistry VR classroom [30,31]. For the test conditions, 3D virtual molecular models were constructed using Unity3D, and the participating students interacted with 3D models in a given VR classroom environment. In the chemistry VR class, although learning content in VR without exposure to dangerous situations is more appropriate, but because we focused on interactive feedback, we designed it to be learned with a molecular model. Thus, we compared interaction response types (e.g., two types of visual feedback and one type of audio feedback) in terms of improving learning ability for educational content. For instance, in chemistry education, it is very important to have a geometrical understanding of complex chemical formulas and molecular structures. Using immersive virtual reality systems, students in the VR class will be able to understand 3D structures with suitable interactions that cannot be presented in 2D textbooks [32]. Thus, we assumed that our immersive VR environment would have a positive impact on students’ learning by providing 3D visualization, interaction, and an intuitive learning experience.

In our VR environment, before evaluating the three types of interaction feedback, we compared the effectiveness of learning between the typical 2D and immersive VR classroom environments. Then, we investigated the three types of interaction methods with experimental configurations concerning the differences in students’ perception of the learning effects (e.g., interest, understanding, preference, and remembering) (see

Table 2. Our configured experiment conditions: we conducted evaluations in two stages, and we first compared a typical 2D-based class and an immersive 3D-VR one in relation to the participants’ perception of the learning effects such as interest, understanding, preference, and remembering in phase 1. In phase 2, we investigated the three types of interaction methods.

Experiment Conditions by Interactions			Learning Effects
Phase 1	Typical 2D class vs. Immersive VR class		Interest Understanding Preference Remembering
Phase 2	Visual	Scale transformation (Breathing animation)
		Deformable body object (Physical shape interaction)
	Audio (Non-visual)	Audio feedback

).

Figure 3 illustrates the flowchart related to the evaluation procedure for our approach. The left side of Figure 3 shows how we compared the VR and non-VR systems to determine the difference that spatial information makes in an educational environment. The flowchart on the right side of Figure 3 shows the comparison process of the three types of interactions in the VR environment. Here, the measurement items for impacts on learning were set the same. Our interaction handling process between the participant and the virtual object was based on the virtual object’s response according to the participant’s input [33,34]. For example, the participant’s specific interaction information (e.g., selection of the virtual object) was extracted from the controller, and the system recognized whether the participant performed an interaction motion at a specific point. Then, the virtual object was allowed to change. In addition, to further increase interest in the VR class, the participating student was captured in real time, and the reconstructed human was included in the virtual environment (see Figure 2, left-hand image).

4. Implementation

Figure 4 depicts the results of the three types of interactions. To evaluate the interaction feedback in our study, we used two visual responses and one non-visual (or audio) response. This study conducted an analysis of the impact of the interactions within an immersive chemistry education environment to determine the effectiveness of interaction on interest, understanding, preference, and remembering related to the quality of learning. In the first condition (see Figure 4, top image), when the participant in the VR environment touched a virtual chemical object (or atom) with the virtual hand, it repeatedly grew and shrank in a certain cycle like a breathing effect to provide visual feedback. Then, the frequency and size of the virtual object could be controlled to enhance visual information. In the second condition (see Figure 4, middle image), when the user touched the virtual object with his or her virtual finger, the name of the virtual element was played continuously through the speaker. To enhance auditory stimulation, the participant could choose between male and female voices using artificial intelligence text-to-speech (TTS). As the final experimental condition (see Figure 4, bottom image), when the participant touched the element, a distortion effect was implemented with deformable body objects. When the user pressed and released the element with his/her virtual finger, it was restored to its original form. To enhance visual stimulation, the intensity was adjusted by the depth of the finger pressure.

In the evaluation conditions for immersive 3D environments, the participant wore a head-mounted display (Meta Quest [35]) to view and interact with the virtual molecular element. Also, we installed blue screen technology to make the participant visible as a virtual human in the virtual space (see Figure 2, left-hand image) [36]. The virtual classroom, virtual datasets of the chemical element, and the synthesized person were configured with resemblance to an actual chemistry classroom using the Unity3D authoring toolkit [37].

5. Experiment and Discussion

We now describe our experiment that validated how participants responded to the three different types of interaction responses. In recent previous articles, most research works have relied on questionnaires on the user experience [38], and we also used them to evaluate the effectiveness of interaction feedback for performance verification. The experiment was designed as a within-subject measurement. The subjects of our experiment were third- and fourth-year university students participating in chemistry classes. A total of 58 paid subjects (40 men and 18 women) with a mean age of 23.5 years participated in our experiment. After collecting their basic background information, we explained the purpose of the experiment and instructions for the interaction task, and our virtual chemistry class was controlled by an administrator. After experiencing the virtual class, participants filled out the questionnaire to evaluate the effects on their interest, understanding, preference, and remembering on a seven-level Likert scale compared to a typical textbook-based class. The first phase examined the effectiveness of education in immersive environments.

Table 3. Evaluation results from the immersive VR classroom. In the table, the ratios of effectiveness according to interest, understanding, preference, and remembering are presented.

	Interest	Understanding	Preference	Remembering
Strongly Agree	3.4%	6.9%	1.7%	5.2%
Agree	31.0%	29.3%	24.1%	29.3%
Somewhat agree	58.6%	48.3%	69.0%	60.3%
Neutral	6.9%	15.5%	5.2%	5.2%
Somewhat disagree	0.0%	0.0%	0.0%	0.0%
Disagree	0.0%	0.0%	0.0%	0.0%
Strongly disagree	0.0%	0.0%	0.0%	0.0%

shows the level of the participants’ perception of the results with interactive content. In the results, all participants responded that the VR class had a better than neutral effect; in particular, the immersive and interactive system was able to achieve very high results in terms of memory for learning. As shown in previous studies, we also found that participants tended to highlight the interactive content’s long-term effectiveness.

Figure 5 and

Table 4. Statistical results with respect to scale transformation feedback: we compared the nominal situation without interaction feedback (e.g., class is indicated as off in the table below) and the given interaction response.

Group	Class	Mean	t	Sig.
Interest	Off	5.31	−2.564	0.012
	On	5.76
Understanding	Off	5.28	−1.985	0.050
	On	5.66
Preference	Off	5.22	−2.233	0.027
	On	5.64
Remembering	Off	5.34	−1.603	0.112
	On	5.64

show the results regarding the scale transformation feedback as a factor in our conducted experiments. As a result, people tended to have a high overall learning effectiveness compared to not providing feedback. Statistically significant differences in understanding and remembering questions were not found (p > 0.05). However, in the case of interest and preference, as a result of the t-test, it was found that there was a significant difference in confidence level between the two conditions (feedback off vs. on) (p < 0.05). The results of the audio feedback showed that the experimental conditions did have major effects on participants’ interest, understanding, and remembering. In the other visual feedback, all evaluation results related to the learning effects were revealed to be statistically significant in our experiment (p < 0.05) (see Table 4). To summarize our experimental results, audio feedback is more suitable for remembering, from which we assume it can be an advantage to simultaneously provide two or more multimodal types of feedback such as visual and auditory. Also, participants found the deformable objects more stimulating, which resulted in a greater learning effect in terms of interaction. Moreover, we used the results to analyze whether the effectiveness could be applied to other general classes. As a previous result of meaningful research, Campos et al. presented understanding as the highest in a pyramidal structure to study specific learning topics and the most objective [7]. In our study, understanding was more effective than other measurement factors, e.g., the agree and strongly agree values were 36.2%. Typical chemistry experiments are limited by many issues such as the speed of reaction, cost, and safety [30], and our VR-based teaching environments allow students to have experiences of virtual experimental operations. Also, in similar research works related to interactive feedback, although they mainly focused on virtual object recognition for selection and manipulation [31], we studied changes in virtual objects without using haptic equipment. Thus, we believe that our results will help develop a general-purpose VR system that can overcome the limitations of haptic equipment, and our findings have implications for the design of more effective VR classroom systems in terms of higher learning effectiveness. However, it was found that there was difficulty in interacting with touch using a virtual hand due to accuracy issues. We will leave it to future research to overcome this difficulty. Additionally, it is necessary to find out whether the VR system has directly positive results on students’ learning capabilities. In the future, we plan to update our experiments on perception according to learning theories and quality with respect to educational approach such as cognitive load, and problem solving [18]. This research direction is expected to be a way to learn more about the relationship between user perception and learning effects. Then, we plan to perform additional measurements based on the technology acceptance model (TAM), and we will expand the survey in terms of self-efficacy, ease of use, and usefulness [38].

6. Conclusions and Future Work

Recent advances in immersive virtual reality technologies will provide more helpful information to assist in real-world educational situations. In our study, we investigated the effects of a few interaction responses on the participants’ perceptions of interest, understanding, preference, and remembering. We focused on the perception of learning effects, and it was found that the interaction feedback of a deformable object improved the participants’ perception of educational effect. Also, participants reported that the audio response was most effective in the case of remembering. Our results are expected to provide guidelines on how to focus on constructing educational VR content.

However, there are several selection and manipulation methods for interaction in a virtual environment. There are still many aspects in need of improvement for the practical applicability of a VR education system. In future work, we will continue to explore the effects of other interaction responses to improve learning effects. We will also conduct research to analyze the characteristics of various classes such as mathematics, music, and art, and we intend to investigate the optimal interaction strategy that can be used with the proposed approach in our study. Finally, it will also be necessary to use measurement tools in a more accurate way, and we also plan to conduct research related to disorientation regarding psychological and physiological effects.