Influence of Avatar Facial Appearance on Users’ Perceived Embodiment and Presence in Immersive Virtual Reality

Abstract

Immersive virtual reality (VR) based on head-mounted displays has been identified as one of the key interaction technologies of the future metaverse, which comprises diverse interconnected virtual worlds and users who traverse between those worlds and interact with each other. Interaction in immersive VR entails the use of avatars that represent users. Previous research has shown that avatar appearance (e.g., body type, body visibility, and realism) affects the senses of embodiment and presence, which are among the key indicators of successful immersive VR. However, research on how the similarity between an avatar’s face and the user’s face affects embodiment and presence is lacking. We conducted a mixed-method experiment with 23 young adults (10 males, 13 females, mean age: 25.22) involving a VR scene with rich embodiment, a virtual mirror, and interaction with a virtual character. The participants were assigned to two groups: Group 1 had avatars based on their own faces, and Group 2 had avatars based on a stranger’s face. The results indicated that Group 1 experienced higher embodiment with no significant differences in presence scores. Additionally, we identified moderate and significant correlations between presence and embodiment, including their subscales. We conclude that the realism and similarity in an avatar’s appearance is important for embodiment, and that both embodiment and presence are intertwined factors contributing to immersive VR user experience.

1. Introduction

One of the biggest technology buzzwords for 2023 is “metaverse,” which refers to the next generation of the Internet, built upon interconnected and interoperable virtual worlds that can host an unlimited number of users who can seamlessly interact with each other and traverse between these virtual worlds at ease. Although the metaverse has not yet been implemented, several proto-metaverses, such as RoBlox (Available online: https://www.roblox.com/ (accessed on 27 December 2022)), Zepeto (Available online: https://web.zepeto.me/en (accessed on 27 December 2022)), Meta Horizon Worlds (Available online: https://www.oculus.com/horizon-worlds/ (accessed on 27 December 2022)), Spatial (Available online: https://www.spatial.io/ (accessed on 27 December 2022)), and Engage (Available online: https://engagevr.io/ (accessed on 27 December 2022)), have emerged as candidate components of the metaverse to come [1]. Some of the common features of these proto-metaverse applications include support for diverse social interactions, customizable avatars, user-created content, and a virtual economy [2]. Moreover, they can be accessed by various technologies, such as virtual reality (VR), augmented reality, and computer-based three-dimensional (3D) virtual world clients. Out of these methods, VR is particularly promising as it can provide highly immersive and engaging experiences.

In this study, we focus on immersive VR that uses head-mounted displays (HMD) to enable users’ high immersion and engagement in the activities of a virtual world. Although VR has existed for a long time [3], it has recently become a mainstream interaction technology due to the availability of affordable HMDs, along with rich content development tools. Today, multi-user VR environments such as Meta Horizon Worlds, Spatial, and Engage provide users with means to develop their virtual avatars for meeting and interacting with their family, friends, colleagues, and even strangers in immersive virtual worlds. Although these tools and their underlying VR technology are developed at a fast pace, significant challenges remain in the development of VR, such as presence and embodiment [4]. Presence was defined by the International Society for Presence Research [5] as a “psychological state or subjective perception in which even though part or all of an individual’s current experience is generated by or filtered through human-made technology, part of all of the individual’s perception fails to accurately acknowledge the role of the technology in the experience.” In other words, to facilitate the sense of presence, the VR experience must be so compelling that the user forgets about the role of technology in it. To achieve presence, several conditions must be satisfied, such as realism of the experience [6] and a strong sense of control by the user [6,7].

Embodiment is another important success measure of immersive VR experiences. It refers to the combination of sensations that the user feels when being inside, having, or controlling a body [8]. In VR, the body is typically that of a virtual avatar which synchronizes with the user’s body through accurate sensor-based tracking (e.g., motion trackers, eye tracker, and facial tracker). Consequently, the user’s bodily movements and interactions are replicated within the body of the avatar, thus creating a sensation of being in control of the avatar’s body.

Understanding the influence of the user’s avatar’s facial appearance realism on embodiment and presence is essential for achieving strong senses of embodiment and presence in future metaverse experiences based on social interactions. Although a substantial body of literature has addressed both embodiment and presence, scholars have paid less attention to their relationships with an avatar’s facial appearance. Several studies have explored the effects of the avatar’s body type (e.g., half-body, full-body, realistic, or cartoon) and body synchronization on presence [9,10]; other studies have investigated the relationships of embodiment and presence [10,11], agency/control [11,12], and camera perspective [12]. However, only Waldemate et al. [11] considered the realism of facial appearance as an experiment variable.

In this study, we aim to explore how an avatar with a facial appearance based on the user’s facial photograph affects the senses of presence and embodiment in an immersive VR environment with rich embodiment implemented using motion trackers, an eye tracker, and a facial tracker. A mixed-method strategy based on a questionnaire and interviews was harnessed to gather the perceptions and opinions of the participants who were divided into two groups; the experimental group used avatars based on their own facial image and the control group used avatars based on a same-gender stranger’s face. Through conducting this experiment, we provide the following contributions to the body of immersive VR research: exploration on the effects of an avatar’s facial appearance on presence and embodiment and investigation of the relationship between embodiment and presence.

2. Background

2.1. Embodiment and Avatar Appearance

When a user enters a virtual environment in a first-person perspective using immersive VR equipment, they control the virtual body of a human or a non-human avatar. Such an immersive experience can lead to embodiment, which is defined by Kilteni et al. [8] as the combination of sensations that arise with being inside, having, or controlling a body. In immersive VR, the body is virtual; it typically belongs to the user’s avatar and is synchronous with the user’s limbs, head, and torso. In another study, Gonzalez-Franco and Peck [13] propose that embodiment comprises several constructs: the feeling of owning the virtual body of an avatar, the sense of agency that enables one’s independent control of the virtual body, the location of the virtual body in relation to the user’s body, and the virtual body’s external appearance. Body ownership is based on the theory of the rubber hand illusion [14]: when a rubber hand is placed next to a real hand and is stimulated by stroking, the participant’s mind falsely recognizes the rubber hand as their own hand. As such, the rubber hand illusion caused by visual–tactile synchronization enables a sense of embodiment, but only for the hand. In immersive VR, accurate motion tracking technology can extend this illusion to the whole body, which can be further improved by installing a virtual mirror in front of the user’s avatar [15]. Through this body ownership illusion, the user feels the virtual body as if it were their own, which can increase the immersion in the virtual world. Gao et al. [16] also studied the effects of a virtual mirror and various levels of avatar visibility (only controllers, only hands, upper body, or full body) on embodiment, presence, and bow-shooting performance in an immersive VR environment. They found that the full-body avatar was the most preferred for embodiment and presence, but the unsynchronized legs are easy to ignore, making it equal to the upper-body avatar. Moreover, it was found that most of the participants preferred the controllers-only case rather than the hands-only case. In immersive VR, when the user looks at their avatar’s feet from a first-person point of view, the user can feel a sense of embodiment when finding the avatar’s feet in place of their own. When the virtual body is moved away from the first-person view to a third-person view, the user experiences a fluid escape, which reduces the sense of embodiment [17]. Moreover, Krekhov et al. [18] found that body ownership through embodiment in a virtual world can also be applied to animal bodies; in addition, embodiment can affect the user’s emotional experience in a virtual environment. Gall et al. [19] conducted a study measuring the participants’ emotional responses according to the presence or absence of embodiment in immersive VR. They discovered that increased embodiment translates to increased emotional responses in users.

It is known that the sense of embodiment can be affected by the avatar’s external appearance [13]. As the focus of this study is to explore the effects of the avatar’s facial similarity to the user’s own face, it is important to summarize studies that have explored the avatar’s appearance in the context of embodiment. Studies related to the Proteus effect [20,21], a phenomenon in which the user’s avatar’s appearance affects their attitude and behavior, have been actively conducted, and it has been found that the avatar’s age [22,23,24], race [25], gender [17], and body type [20,22] affect such attitude and behavior. Yee and Bailenson [22] confirmed the difference in negative bias toward non-persons according to the age of the users’ avatars. As a result, experiments showed that experiencing embodiment using an avatar older than the user reduces negative prejudice against the elderly. In addition, researchers have mentioned that using VR for perspective-taking in psychology can overcome the limits of the user’s imagination [22]. Studies have also been conducted to lower negative prejudices regarding skin color. Peck et al. [25] demonstrated that prejudice against people with dark skin was lowered by using a dark-skinned avatar. In an experiment, light-skinned participants used dark-skinned avatars to confirm the avatar’s appearance on a mirror in VR and experienced embodiment through full-body visual-motion synchronization. As a result, an Implicit Association Test confirmed that prejudice against dark-skinned races was significantly reduced compared to before experiencing VR. In another study, Yee and Bailenson [22] found that an avatar with an attractive face made users come closer when introducing themselves in VR, and that these tended to respond in a friendly way, thus giving more information about themselves. Yee and Bailenson also found that users with tall avatars tended to act confidently when negotiating.

Waltemate et al. [11] explored the impact of avatar personalization on embodiment and the effects of embodiment on body ownership, presence, and emotional responses. They created synthetic avatars, non-personalized body-scanned avatars (including face), and personalized body-scanned avatars for an experiment involving 32 participants. They also employed two virtual mirror setups: a wall projection mirror and a virtual mirror via an HMD. The results indicated that personalized body-scanned avatars led to the highest body ownership and presence among the avatar conditions. Moreover, the study found that the HMD mirror increased body ownership, agency, and presence more than the projection mirror.

2.2. Presence and Avatar Appearance

VR has become a mainstream immersive technology and a key component of the metaverse to come. Despite having different target groups, content, interaction methods, hardware configurations, and level of context-awareness, most VR systems share the goal of eliciting presence in users where they feel that the virtual environment is similar to being present in the real environment [2,26]; this feeling of presence is the result of psychological, perceptual, and cognitive immersion [27]. However, it is misleading to talk about presence in a singular form because researchers have identified diverse types. Lee [2] argues that presence can be evaluated by identifying which reality is affected between the “reality” in which one lives and the “virtual reality” one experiences through a HMD. He divides presence into three types: physical presence, self-presence, and social presence. Many studies related to presence have focused on physical presence, which Lee [2] defines as the feeling of presence arising from the elements that constitute the virtual space or the relationship between the virtual space and the user. In contrast, self-presence refers to the psychological state in which the virtual self is experienced as the real self, and social presence is the sense of presence arising when the presence of another person is not perceived as an artificial construct in VR [2]. Bulu [26] also explores physical presence and social presence as components of presence; in addition, she introduces co-presence as another component that describes the sense of being together with others in a virtual environment.

Studies to measure presence have been conducted since the 1990s, and several questionnaires have been developed as tools to measure it, of which Witmer and Singer’s [27] Presence Questionnaire (PQ) and the Slater-Usoh-Steed Questionnaire [28] are representatives. The version of PQ adapted for this study consists of four presence factors—control, realism, involvement, and adaptation—and the presence score is calculated by adding the item scores on a seven-point Likert scale. These factors are defined by Witmer and Singer [27] as follows. Control is the extent and responsiveness of the user’s control over the tasks and interactions in a virtual environment. Realism refers to the level of authenticity in the appearance of the virtual environment and its objects, as well as the consistency of the presented information and responses to interactions in relation to what might be expected in the real world. Involvement is defined in terms of focusing one’s energy and attention on meaningful stimuli, activities, and events in a virtual environment, where meaningfulness is subjective. It is closely related to immersion, which is defined as “perceiving oneself to be enveloped by, included in, and interacting with an environment that provides a continuous stream of stimuli and experiences” [27]. Aligned with this definition, immersion and involvement are particularly strong in HMD-based VR systems that isolate the user from the surrounding physical environment. Finally, adaptation refers to the user’s ability to adapt to the virtual environment [29].

Pan and Hamilton [4] identify presence as one of the contemporary research challenges in VR. They note that although current VR systems may elicit a sense of presence in the user, VR experiences still differ too much from the real world to make the user confused between the two. Building a realistic VR environment that supports natural interactions is a substantial technical challenge, where things can go wrong in terms of enabling the sense of presence. Therefore, successful VR experience development requires the consideration of many factors that contribute to presence, including but not limited to the quality of information [6,30]; user control [6,7]; realism [6]; cognitive, emotional, and behavioral factors [7,31]; and overall technical quality [30]. In addition, the development process should consider the aspects of VR content, hardware, software, or the physical surroundings that may disturb the sense of presence. Some examples of these are cybersickness [7,32], awareness of the VR equipment [7], and obstacles in the physical environment.

Previous studies have shown that a user’s avatar and its appearance can influence the sense of presence; for example, the avatar’s body type (e.g., half-body, full-body, realistic, or non-realistic) and the degree to body synchronization has been studied from the perspective of presence [9,10,11,33]. When Waltemate et al. [11] studied the impacts of different levels of embodiment and avatar appearance on presence, they found that personalized avatars based on 3D models created by scanning users increase the sense of presence; conversely, avatars based on non-personalized scanned models and synthetic 3D models showed less effect on presence. In another study, Heidicker et al. [10] investigated how an avatar’s body appearance and movement mapping affect presence and co-presence in the context of social VR. They prepared three types of avatars: full-body avatar with predefined animations; full-body avatar with synchronized motion; and head-and-hands avatar with synchronized motion. The results indicated that full-body avatars with synchronized motion had the highest effect on presence. Moreover, synchronized avatars and head-and-hands avatars affected co-presence the most. The relationship between an avatar’s appearance and co-presence in social VR was also explored in a study by Freiwald et al. [33] involving a realistic humanoid and abstract avatar (snowman designs), where participants were only able to see the avatars of other users. As opposed to the findings of Heidicker et al. [10], Freiwald et al. [33] found no significant effects of realistic humanoid avatars on co-presence. In a related study, Yoon et al. [9] measured how an avatar’s body presentation (head-and-hands, upper body, or full body) and style (realistic or cartoon) affect social presence in a remote collaboration scenario based on VR and augmented reality technologies. They found no significant difference between realistic and cartoon avatars on social presence but identified the full-body avatar to be the best for social presence, followed by the upper-body avatar.

2.3. Summary

While the previous studies covered above have explored the effects of an avatar’s appearance on presence and embodiment in different conditions, our study improves upon their shortcomings. Firstly, our study uses only realistic humanoid avatars and focuses on similarity to the user’s face. Secondly, we focused on user self-image rather than on avatar appearances in a social VR scenario, which was the case in many studies related to presence (e.g., [9,10,33]). Thirdly, we used a rich avatar embodiment system which attaches multiple motion trackers and other body sensors to the user to enable realistic body synchronization. Thirdly, we provided virtual mirrors for our participants to better enable them to see and feel their connection with an avatar.

3. Materials and Methods

3.1. Research Objective and Questions

This study aims to investigate the effects of the user’s avatar’s facial appearance on perceived embodiment and presence in an immersive VR environment. Moreover, we sought to identify any meaningful relationships between presence and embodiment in this context. The corresponding research questions are as follows:

• How does the similarity between an avatar’s face and the user’s face affect perceived embodiment?
• How does the similarity between an avatar’s face and the user’s face affect perceived presence?
• What is the relationship between embodiment and presence?

3.2. Participants

The experiment was conducted with 23 Korean adults (10 males, 13 females) between 22 and 32 years of age (mean: 25.22) who were students and staff members of Ajou University. The participants voluntarily applied for and participated in the experiment. After considering their previous VR experience, participants were randomly assigned to one of two groups with similar numbers of experienced and inexperienced participants. Group 1 comprised participants who experienced the VR content with a same-gender avatar that had a face based on the participant’s own facial image. In contrast, the participants in Group 2 experienced the VR content using a same-gender avatar that had a face based on a non-participating person’s facial image.

3.3. VR Stimulus

To facilitate the senses of embodiment and presence, we used an interactive VR content running on HMDs. Rich embodiment was implemented utilizing sensors such as motion trackers, an eye tracker, and a facial tracker. The VR content was situated in a virtual environment that resembled a typical high school classroom in the Republic of Korea, and the user’s avatar was a high school student wearing a school uniform. The user could observe themselves through a virtual mirror, and their actions (e.g., body movement, eye gestures, and facial gestures) were synchronized with the avatar’s body. The VR content also presented a dialogue with another virtual character with whom the user could interact. We created avatar faces based on participant as well as non-participant photographs. More details of the VR stimulus are presented in Section 4.

3.4. Data Collection Instruments

To collect the experimental data on presence and embodiment, we adapted Witmer and Singer’s [27] PQ and Gonzalez-Franco and Peck’s [13] Embodiment Questionnaire (EQ), respectively. Both questionnaires used a 7-point scale to give each question a score from −3 to 3 to measure embodiment and presence. An amount of 13 items measured the sense of embodiment, and 14 items measured the sense of presence, for a total of 27 items. The questions based on PQ and EQ are presented in Appendix A. In the following sections, we describe the questionnaires in detail and provide an overview of the interview design.

3.4.1. Presence Questionnaire

We used the following subscales of PQ to measure presence in our experiment: (i) Control, (ii) Realism, (iii) Involvement, and (iv) Adaptation. These factors were selected by comparing the subscales of PQ with the subscales used in previous presence studies [27,29]. The items in the Control subscale evaluate the user’s reaction, control, and degree of control in VR. The Realism subscale evaluates the similarity with reality, such as the virtual environment, interaction, and naturalness of movement in VR. The Involvement subscale comprises items that measure the extent to which the user focuses their mental energy and attention to the VR stimulus; it also includes items for measuring immersion, such as how isolated the user feels from the real world. Finally, Adaptation measures how quickly the user adapts to VR, including adaptability in moving and interacting as the avatar in VR. The scoring method for presence is explained in

Table 1. Presence scoring method (prs: presence item).

Subscale	Scoring Method
Control	(prs1 + prs2 + prs3)/3
Realism	(prs4 + prs5 + prs6 +prs7)/4
Involvement	(prs8 − prs9 + prs10 + prs11 + prs12)/5
Adaptation	(prs13 + prs14)/2
Total	(Control + Realism + Involvement + Adaptation)/4

.

3.4.2. Embodiment Questionnaire

We selected the following subscales from EQ [13] to measure the sense of embodiment: (i) Body Ownership—Face, (ii) Body Ownership—Body, (iii) Agency and Motor Control, and (iv) External Appearance. The Body Ownership—Face subscale measures the sense of ownership of the avatar’s face. Body Ownership—Body refers to the sense of ownership of the avatar’s body. Agency and Motor Control evaluates the extent and accuracy with which the avatar’s body can be controlled within the virtual environment. Lastly, External Appearance measures the effect of the avatar’s external appearance on the sense of embodiment. We excluded the subscales of Tactile Sensations, Location of the Body, and Response to External Stimuli in EQ because they are not applicable to the VR stimulus used. The scoring method of the sense of embodiment based on the questionnaire is summarized in

Table 2. Embodiment scoring method (em: embodiment item).

Subscale	Scoring Method
Body Ownership—Face	(em1 − em2)/2
Body Ownership—Body	(−em3 + em4 − em5)/3
Body Control	(em6 + em7 + em8 − em9)/4
External Appearance	(em10 + em11 + em12 + em13)/4
Total	(Face×2 + Body×2 + Body Control×2 + Ext. Appearance)/8

.

3.4.3. Semi-Structured Interview

We conducted semi-structured interviews with the participants after they completed the questionnaire, to receive their in-depth opinions and thoughts about the VR content experienced. The interviews were recorded with consent for later analysis. The interview questions, as shown in Appendix B, probed what the participants felt while experiencing the content and why, as well as their thoughts on the content. Additionally, the interview contained specific questions depending on the group. Group 1 participants, who used an avatar based on their own facial image, were asked why they felt or did not feel that the avatar’s facial appearance affected their sense of embodiment or presence. They were also asked whether their answers to the embodiment and presence questions would have been different if they had experienced an avatar with an unfamiliar facial appearance. On the other hand, Group 2 participants, who experienced the VR content using an avatar based on another person’s facial image, were asked whether there would be a difference in the sense of embodiment and presence if they had the same experience with an avatar based on their own facial image.

3.5. Experiment Preparation

Before proceeding with the experiment, frontal photographs of the participants’ faces were taken after obtaining consent from all of them, regardless of their group. Researchers took pictures of the participants’ faces using a Canon G7 X Mark II digital camera in the same space, and it was announced that the photographs were for the purpose of making avatar faces. The facial pictures were used to create avatars for the participants of Group 1 using Character Creator 3 (Available online: https://www.reallusion.com/character-creator/ (accessed on 27 December 2022)), a 3D character creation tool, and the Headshot plugin, an AI-based 3D head creation tool (Available online: https://www.reallusion.com/character-creator/headshot/ (accessed on 27 December 2022)). The avatars for the participants in Group 2 were created in a similar manner based on facial photographs of people who did not participate in this experiment. All avatars had a same-gender body type and wore a school uniform. The created avatars were imported to the VR stimulus.

3.6. Experimental Procedure

For the study, we divided the experiment into two stages (Figure 1), where the first stage was for photographing the participants and the second stage for the VR experience and data collection. It lasted approximately 30 min. At the beginning of Stage 1, the experimental steps and required time were explained to the participants. After checking the participants’ condition, the researchers explained the possible side effects of the experiment, such as cybersickness and headache. They also explained that each participant’s behavior and screen would be recorded and that the recordings and other collected data would not be used for purposes other than anonymously informing this study. Finally, the participants signed an informed consent form.

At the beginning of Stage 2, we explained the VR equipment (Oculus VIVE Pro Eye HMD, two VIVE controllers, three VIVE motion trackers, and one VIVE Facial Tracker) to the participants. One motion tracker was worn below the chest, and one on each elbow, and the facial tracker was attached to the lower part of the HMD and worn together with the HMD. After the participant held the controller in both hands, as shown in Figure 2, the VR content was executed. The participant remained seated during the entire experiment.

After experiencing the content, participants answered a questionnaire to measure perceived presence and embodiment. They were able to see the screen recording of their experience while answering the questionnaire to review the details of their avatar. Then, a researcher conducted an interview to gather the participants’ thoughts about the experienced VR content.

3.7. Data Analysis

Our study adopted three methods to analyze the quantitative questionnaire data on embodiment and presence with SPSS. The Mann–Whitney U Test brought out the difference between independent groups. The Kruskal–Wallis H Test (significance level: 0.05), which compares independent samples from three or more groups, analyzed differences in embodiment and presence according to gender and previous VR experience in the two groups. The Spearman’s Rank Correlation examined the correlation between the embodiment and presence results.

4. VR Content Production

4.1. VR Equipment and Avatar Embodiment

The VR equipment used in the experiment was an HTC VIVE Pro Eye HMD with two controllers, three HTC VIVE Trackers (3.0), and one HTC VIVE Facial Tracker. The HMD’s eye tracker was used to synchronize the user’s eye gestures to the avatar’s eyes. In addition, the facial tracker was used to track the user’s lips and jaws, which we used to control the shape of the avatar’s mouth. The controllers and motion trackers were used to track the movements of the user’s hands, arms, and torso. SteamVR (Available online: https://store.steampowered.com/steamvr (accessed on 27 December 2022)) was used to track the VIVE controller and trackers so that the avatar could imitate the user’s movements. Moreover, the avatar’s natural movement was implemented with the Final IK (Available online: https://assetstore.unity.com/packages/tools/animation/final-ik-14290 (accessed on 27 December 2022)) tool using the inverse kinematics technique. In addition, we used SRanipal SDK (Available online: https://developer-express.vive.com/resources/vive-sense/eye-and-facial-tracking-sdk/ (accessed on 27 December 2022)) to capture eye tracking and facial tracker data. The VR content for this experiment was implemented with Unity 2020.3.9f1 and the XR Interaction Toolkit. The specifications of the computer running the VR contents during the experiment are presented in

Table 3. Specifications of the computer used in the experiment.

Operating System	Windows 10 Pro (64 bit)
Processor	Intel® Core™ i7-7700 processor (4 cores, 8 MB cache, Turbo Boost 2.0, 4.2 GHz)
GPU	NVIDIA® GeForce® GTX 1070 (8 GB GDDR5)
RAM	16 GB
Storage	1 TB
Software	Unity 2020.3.9f1, XR Interaction Toolkit, SteamVR, SRanipal SDK

, followed by images of the hardware and experiment station in Figure 3. The implementation details of the avatar synchronization are presented in Moon et al. [34].

4.2. VR Content Creation

The content used in this experiment is the XR Lab support project funded by the Korea Radio Promotion Association (RAPA) in the virtual convergence economy era in 2021, which links content development with start-up and commercialization to secure master- and doctoral-level professional labor force. The theme of this project is to develop an immersive content platform for the diagnosis of empathy types and educational contents for each type. The experiment utilized scenarios of the empathy type diagnosis content, including scenes of checking the appearance of the avatar in a mirror (Figure 4 top) and interacting with a virtual character (Figure 4 bottom). These scenes were chosen because they contain elements and an environment that could induce a sense of embodiment and presence. The mirror scene has a virtual mirror that allows the user to explore the avatar synchronization and resulting embodiment. The interaction scene was chosen to make the participant feel that they are not alone and they are participating in an event (conversation) in the virtual environment; this was aimed to promote presence. The two scenes were modified according to the experiment’s requirements. For example, additional instructions were added to the mirror scene for the participants to use their body, as this was aimed to enhance embodiment.

The VR content was set in a high school classroom and consisted of three scenes: (i) first mirror, (ii) dialogue, (iii) second mirror. In the first mirror scene, the user sat in a chair in a virtual high school classroom with a mirror on the desk in front of them. A voice instruction asked the user to pick up the mirror and place it on a panel in front of them. Then, a large virtual mirror appeared, where the user could observe their motion-synchronized avatar. The user heard voice instructions for actions such as “wave your right hand to say hello” and “blink your eyes and move your mouth” to induce the user’s experience of embodiment with a synchronized avatar. After performing all the instructions, the user clicked a button to proceed to the dialogue scene where a virtual school student character interacted with the user by offering a student ID. The user then grabbed it and the character proceeded to tell the user a story about making a confession to a friend of another gender. At the end of the scene, the second mirror scene played in the same manner as the first.

To create users’ avatars, we used Reallusion’s Character Creator 3, a 3D character creation tool, and an image-based character was created with the Headshot plugin. After taking facial photos of the participants in the experiment, avatars were created using the facial photos of the participants in Group 1. In addition, the users’ clothes were made as school uniforms using Marvelous Design and applied to the characters. Two body types were used for females and males, respectively. In the case of the Group 1 avatars, the shape and color of the hair and facial hair were applied according to the participants’ photographs. Examples of female and male avatars created based on photographs are presented in Figure 5.

5. Experiment Results

5.1. Demographics and Previous Experience

As mentioned above, the participants were divided into Group 1 and Group 2, based on whether their avatar was based on their facial appearance or not, respectively. As shown in

Table 4. Experiment participants and previous VR experience.

	Group 1	Group 2	Total
Males	7	3	10
Females	5	8	13
Previous VR experience	5	6	11
No previous VR experience	7	5	12
Total participants	12	11	23

, Group 1 consisted of 12 people (seven males and five females), and Group 2 comprised 11 people (three males and eight females). Eleven people had previous VR experience—five in Group 1 and six in Group 2. Moreover, twelve people had no experience in VR—seven people in Group 1 and five people in Group 2. Therefore, the ratios between those who had experienced VR and those who had not were similar between the two groups.

5.2. Embodiment and Presence by Group

To find whether any significant differences existed between Group 1 and Group 2 in terms of embodiment and presence, we conducted the Mann–Whitney U test. The results presented in

Table 5. Results of the Mann–Whitney U test on presence and embodiment between Group 1 and Group 2.

	Embodiment		Presence
Group	1	2	1	2
N	12	11	12	11
Mean Rank	17	6.55	13.33	10.55
Sum of Ranks	204	72	160	116
Mann–Whitney U	6		50
Wilcoxon W	72		116
Z	−3.693		−0.985
Asymp. Sig. (two-tailed)	<0.001		0.325
Exact Sig. [2*(one-tailed Sig.)]	<0.001		0.347

show that Group 1 had higher mean ranks for both embodiment and presence. In the case of embodiment, the mean ranks of Group 1 and Group 2 were 17.00 and 6.55, respectively. The low Mann–Whitney U value (6.0) indicates that the two groups were different, and the test result is significant (<0.001). In the case of presence, the mean rank of Group 1 (13.33) was also greater than that of Group 2 (10.55), but the difference between the groups was not significant (0.325). These results suggest that when the avatar’s facial appearance is based on the user’s facial image, the user’s sense of embodiment increases; however, in the case of presence, the effect of the avatar’s facial appearance could not be determined. In the following sections, we present the detailed results of the embodiment and presence questionnaire.

5.2.1. Embodiment Scores

Table 6. The mean scores of embodiment, calculated according to Gonzalez-Franco and Peck [13] based on the embodiment questionnaire (scale from −3.0 to 3.0).

	Group 1	Group 2
Body Ownership—Face	1.667	−1.133
Body Ownership—Body	0.750	0.222
Agency and Motor Control	2.000	1.817
External Appearance	0.896	−0.217
Overall Embodiment	1.389	0.228

presents detailed results of the embodiment questionnaire, namely the mean scores for each subscale and the total mean as calculated according to Gonzalez-Franco and Peck [13]. The total mean values of embodiment for Group 1 and Group 2 were 1.389 and 0.228, respectively, thus confirming that Group 1 experienced a higher sense of embodiment due to their avatars’ faces resembling their own. Among the subscale means, body control received the highest mean scores from Group 1 (2.000) and Group 2 (1.817), thus showing the effect of using motion trackers to replicate the user’s movement in the avatar. Other mean scores in Group 2 were significantly lower, with negative mean scores for face ownership (−1.133) and appearance (−0.217). In contrast, the mean scores for face ownership (1.667) and appearance (0.896) were significantly higher in Group 1. These results further confirm the positive effect on embodiment when using the user’s face as a model for the avatar’s face.

5.2.2. Presence Scores

The results of analyzing the differences between the groups in measured presence are presented in

Table 7. Mean scores based on the presence questionnaire (scale from -3.0 to 3.0).

	Group 1	Group 2
Control	1.555	1.044
Realism	1.292	0.783
Involvement	0.317	0.333
Adaptation	2.208	2.366
Overall Presence	1.383	1.132

. The mean values of overall presence are 1.389 for Group 1 and 1.132 for Group 2. This result indicates that the difference of the sense of presence between the groups is small, although Group 1 members felt it slightly more. Among the individual presence factor scores, the largest differences between the groups were found in Control and Realism, where Group 1 showed higher presence. This result indicates that when the avatar’s face is based on the user’s face, it can enhance the feelings of control and realism, which, in turn, affect the overall presence.

5.3. Embodiment and Presence by Gender and Previous VR Experience

We analyzed the questionnaire data through a Kruskal–Wallis H test to determine whether the participants’ gender had an impact on embodiment and presence. The results showed no significant difference in the senses of embodiment and presence between females and males. Similarly, we utilized a Kruskal–Wallis H test to investigate whether any significant differences existed between the participants with previous VR experience and those without previous VR experience with regard to experienced embodiment and presence and found none.

5.4. Correlation Analysis

We conducted a correlation analysis between the participants’ sense of embodiment and presence to identify any meaningful relationships that could help explain how the two concepts are related; for this purpose, we utilized Spearman’s Rank Correlation Analysis. The identified moderate and significant correlations between embodiment and presence and their subscales are illustrated in Figure 6. The results indicate a moderate correlation (0.462) between embodiment and presence. The highest level of correlation (0.766) was found between the External Appearance subscale of embodiment and the Realism subscale of presence, thus indicating the importance of avatars’ external appearance in promoting a realistic VR experience. In addition, External Appearance had high (0.502) and moderate (0.377) correlations with the Control and Immersion subscales of presence, respectively. The correlations between the Body Ownership—Face subscale of embodiment and the Control and Realism subscales of presence were 0.451 and 0.350, respectively. Moreover, the Agency and Motor Control subscale of embodiment showed moderate correlations with the Control (0.380) and Adaptation (0.317) subscales of presence. In sum, we observed significant correlations between the subscales of embodiment and presence, with External Appearance being a major factor in the embodiment–presence relationship.

5.5. Interview Results

The analysis of the questionnaire data revealed that avatars’ facial appearance affected the sense of embodiment, but not that of presence. To identify the reasons for this, we conducted a post-interview to gather the participants’ detailed thoughts and opinions. In the following summary of the interview results, we use M to denote males and F to denote females, followed by their participants’ number and age.

We asked participants about the reason why they felt embodiment in virtual reality and a majority explained that embodiment occurred due to projected movements. This is demonstrated through the following excerpts:

When I waved or shook my hand, it moved the same way, so I could feel a sense of embodiment.

(M3, 25)

When I moved, the avatar moved the same way, so I felt a sense of embodiment.

(F2, 27)

I felt a sense of embodiment because the body moved in the same way as it was shown visually [in the mirror].

(F5, 23)

The questionnaire results indicated that face ownership and external appearance were the most significant factors for the participants in Group 1 feeling higher embodiment than the participants in Group 2. This was also discussed in the interview; when Group 1 participants were asked whether the avatar’s facial appearance was identical to their own, the majority confirmed the likeness and the effect of the familiar face on embodiment, as these comments indicate:

Because the avatar’s face is my face, I felt like I was really in this body.

(M3, 25)

Because it was an avatar with my own face, I was more immersed and seemed to be myself.

(F4, 26)

When we asked the participants in Group 2 if the avatar’s face in the mirror would have affected the sense of embodiment had it been based on their own, most of them gave an affirmative response. On the other hand, when we asked all participants about whether the avatar’s face had any effect on their sense of presence, some of them pointed out that the space in VR had a greater effect on presence than the avatar’s face, as confirmed in the following interview excerpts:

I have the feeling of being in virtual reality more when I look at the environment around me than when I see my face.

(F23, 22)

Since you cannot see your face when you look around, it does not affect the feeling of being in this space.

(M3, 25)

These results confirm that when the avatar’s facial appearance in VR is based on the user’s own face, it significantly affects their feeling of embodiment. However, in the case of presence, the feeling that the user is in the virtual environment is affected more by the virtual space and movement they are experiencing than by the avatar’s appearance. Therefore, the findings of the interview analysis confirm the results of the questionnaire analysis. It can be interpreted that since the time to observe the avatar’s face through the mirror in the VR experience was short, the influence of the avatar’s facial appearance on the sense of presence was not significant. However, another study is needed to evaluate whether a longer observation of one’s virtual face would have a significant effect on presence.

6. Discussion

Implementation of rich social interactions in the metaverse and social VR applications entail the use of avatars to represent human users and autonomous virtual characters. As previous research has shown, an avatar’s appearance (e.g., body type, body part visibility, and realism) can have various impacts on the user’s experience in a virtual environment, such as on their emotional engagement [19], attitude and behavior [20,21], and social interactions [22]. Moreover, an avatar’s appearance can influence embodiment [11] and presence [9,10,11,33], which are among the key components of immersive VR environments and have been identified as research issues to be solved [4]. Motivated by these previous studies, we conducted a mixed-method experiment to explore how an avatar’s facial similarity to the user affects the senses of embodiment and presence in an immersive VR experience that utilizes state-of-the-art sensors for rich embodiment. Two conditions of an avatar’s facial appearance were evaluated by 23 participants who were assigned to two groups: in Group 1, the avatars’ faces were modeled based on the participants’ facial photographs; in Group 2, the avatars’ faces were modeled based on non-participants’ facial photographs. The results of our analysis of the collected questionnaire and interview data indicated that an avatar’s facial similarity to the participant’s face had a positive effect on embodiment; however, the analysis did not reveal any significant influence on presence. Moreover, we could not find any significant effects of gender and previous experience on embodiment and presence. However, we identified a moderate correlation between presence and embodiment (0.462), with significant positive correlations between the subscales of External Appearance and Control (0.502) and External Appearance and Realism (0.766). These findings suggest that when the avatar’s face has similarity to that of the user’s, the sense of embodiment is increased, along with the control and realism components of presence. Future immersive VR system developers and metaverse researchers can promote embodiment through body ownership—and, to a certain extent, presence—by allowing the user to construct an avatar based on their own face.

During the experiment, the participants sat on a chair in front of a table. The table’s height was synchronized with that of the virtual table, thus allowing the participants to place their hands naturally on the virtual table. Due to the seated posture, the results presented in this study may not be applicable to room-scale VR experiences with full-body embodiment, where the player can traverse the virtual space by walking around; therefore, the effects of an avatar’s face based on the user’s face requires more investigation in these specific VR experiences. Moreover, since the participants sat down during the experiment, the avatar’s legs were partially hidden under the virtual table. Consequently, the participants did not wear motion trackers on their legs, and we only implemented a half-body embodiment that involved the participant’s torso, arms, hands, and head. Since we focused on facial appearance and upper-body embodiment in the study, the absence of legs’ synchronization was deemed irrelevant. Previous research has shown that the body realism and visibility of virtual body parts may influence the presence and embodiment experienced by the user [9,10,11,16,33]; however, half-body synchronization has been shown to be equal to full-body synchronization in terms of embodiment [16]. This result is promising because providing full-body embodiment is not only costly but also more prone to tracking errors than half-body embodiment.

As discussed in the Background section, most previous studies exploring the effects of an avatar’s appearance on presence and embodiment have focused on the avatar’s body rather than its face. Waltemate et al.’s [11] study bears some resemblance to ours as one of their experiment conditions contained a full-body avatar based on photogrammetry with a 3D scanned model of the user. Their results showed that such a realistic avatar resulted in increased body ownership and, therefore, embodiment [11]. However, they also found that realistic avatars based on users’ appearance increased the sense of presence; this contradicts our results as we did not identify such an effect. A possible explanation of this difference could be derived from our study’s experimental setup, where the participants sat down and experienced half-body embodiment with generic male and female bodies dressed as high school students; their inability to traverse the virtual environment by walking, along with the generic body appearance, may have resulted in decreased presence. Consequently, the avatar’s facial similarity to the user’s face has little or no effect on presence. However, this hypothesis has not yet been evaluated.

Personalizing an avatar’s facial appearance for achieving realistic quality can be a complicated procedure. Waltemate et al. [11] note that creating a realistic model of the user based on photogrammetry is both time-consuming and costly. Our avatar creation pipeline was different, but it also involved various steps and software. The process of creating an avatar’s head based on a model’s photograph with our pipeline lasts between one hour and one day, depending on the quality of the input photograph and the desired level of realism. The complexity of the avatar creation process is a key issue if we are to provide masses of future metaverse residents with personalized and realistic avatars in the days to come. Some commercial social VR platforms such as Spatial (Available online: https://www.spatial.io/ (accessed on 27 December 2022)) allow the quick generation of an avatar based on the user’s photograph taken with a webcam, but such generated virtual faces do not look very realistic, possibly for performance reasons. However, this aspect is currently being improved by researchers. For example, in a recent article, Cao et al. [35] proposed a method based on deep learning for creating a photorealistic avatar face based on a captured video of the user utilizing a camera with a depth sensor (e.g., some iPhone models). Once an avatar is created and fine-tuned by the deep learning model in six hours, it can be rendered in VR in real time, thus making it a potential solution for implementing realistic avatars in social VR applications. Cao et al. also demonstrate that their model can create realistic original faces based on combinations of recorded data by replicating recorded facial expressions. This research trend suggests that we could soon see easy-to-use, effective, and fast tools for creating realistic avatar faces for immersive VR as well as metaverse applications, which will help achieve deeper embodiment.

The application of the study findings has limitations. First, we explored only the effects of the avatar’s facial similarity on presence and embodiment, while leaving out other parameters related to its appearance (e.g., body type, body visibility, human versus non-human, gender, age, and so forth). Consequently, the results cannot be generalized to other types of avatars without further investigation. Second, the sample comprised university students of a similar age group. It is possible that younger or older users might feel differently about presence and embodiment in similar conditions, and further examination is required to study this aspect. Third, our study did not investigate social presence or co-presence, which have been shown to be affected by the avatar’s body appearance and embodiment in social VR settings [9,10]. Therefore, a future study that measures co-presence and social presence is required to understand to what extent an avatar with a similar face can impact the user.

7. Conclusions

The objective of this study was to explore how the facial appearance of an avatar influences a user’s presence and embodiment experiences in the context of immersive VR. Specifically, we sought to identify how presence and embodiment are perceived when the avatar’s face resembles that of the user’s and when it does not. To determine this, we conducted a mixed-method experiment comprising an immersive VR stimulus, questionnaire, and interview that measured the senses of presence and embodiment, along with subjective perceptions related to the experience. The 23 participants were assigned to two groups based on how their avatars’ faces were constructed: based on their own image (Group 1) or on another person’s image (Group 2). In the following, we summarize the answers to the research questions based on our analysis of the experiment data.

The first research question—“How does the similarity between an avatar’s face and the user’s face affect perceived embodiment?”—was answered by analyzing the data collected by Gonzalez-Franco and Peck’s [13] Embodiment Questionnaire. The results indicate a significant difference between Group 1 and Group 2 in terms of embodiment, with the participants in the former group scoring higher. This result indicates that embodiment can be enhanced by allowing the user to use an avatar based on their own appearance. In contrast, we could not find any significant differences between the two groups in response to the second research question—“How does the similarity between an avatar’s face and the user’s face affect perceived presence?” The participants’ presence scores were measured by Witmer and Singer’s [27] PQ, which suggests that the similarity in the faces of the avatar and the user appears to have little or no effect on perceived presence. To answer the third research question: “What is the relationship between embodiment and presence?,” we analyzed the correlations between the presence and embodiment scores, including their subscales. Identifying several moderate and significant correlations confirms a relationship between these success indicators of immersive VR.

For future investigations based on this study, first, we seek to conduct a similar study in social VR to measure effects including social presence and co-presence in a social scenario. Second, we plan to conduct another experiment involving different age groups to see whether people of different ages have different perceptions of presence and embodiment in the context of facial similarity of an avatar. Third, as the participants of our study were seated and experienced only half-body embodiment, we would like to conduct a similar experiment in a room-scale immersive VR setup with full-body embodiment. Finally, it would be interesting to involve other evaluation parameters, such as an avatar having the face of a familiar person who is not the user, or an avatar with a face based on the user’s childhood photograph.