A Modular Haptic Agent System with Encountered-Type Active Interaction

Abstract

Virtual agents are artificial intelligence systems that can interact with users in virtual reality (VR), providing users with companionship and entertainment. Virtual pets have become the most popular virtual agents due to their many benefits. However, haptic interaction with virtual pets involves two challenges: the rapid construction of various haptic proxies, and the design of agent-initiated active interaction. In this paper, we propose a modular haptic agent (MHA) prototype system, enabling the tactile simulation and encountered-type haptic interaction of common virtual pet agents through a modular design method and a haptic mapping method. Meanwhile, the MHA system with haptic interaction is actively initiated by the agents according to the user’s intention, which makes the virtual agents appear more autonomous and provides a better experience of human–agent interaction. Finally, we conduct three user studies to demonstrate that the MHA system has more advantages in terms of realism, interactivity, attraction, and raising user emotions. Overall, MHA is a system that can build multiple companion agents, provide active interaction and has the potential to quickly build diverse haptic agents for an intelligent and comfortable virtual world.

1. Introduction

The increasing engagement of people in virtual reality (VR) has led to some psychological and cognitive problems in some relatively extreme situations. For example, long-term living or working in VR may increase users’ loneliness and anxiety [1]. This results from the big difference between the virtual environment and the real life of humans. To tackle this problem, a very intuitive approach is adding more participants (i.e., virtual agents) into the virtual environments because virtual agents can provide users with companionship in a virtual environment [2]. The companion virtual agents mainly fall into three categories: virtual avatars of real humans [3], virtual characters [4], and virtual pets [5]. However, virtual avatars of real humans cannot provide companionship all the time due to the limitation of the humans behind the avatars. Additionally, virtual human characters are not widely used because of the complexity of appearance and interaction design, especially the lack of high-level intelligence [6]. Different from the above two, virtual pets can keep an online state all day long and can be built much more easily compared to virtual humans [7]. As a result, virtual pets have emerged as a prominent choice for companion agents [8]. A good companion agent needs to have two important properties; it not only should have variable appearance to provide different companionship but must also sense the user’s interaction and give feedback [9]. Existing work mainly focuses on the appearance and behavior of virtual agents in the visual channel [5], ignoring the haptic channel of virtual agents. However, haptic perception has an important role in the interaction between users and agents [10]. It can significantly improve the user’s sense of immersion and help users to understand the interactive behaviors of virtual agents [11]. So, the design of haptics is necessary for a good interaction experience. Among several kinds of haptics, the encountered-type haptics can provide force feedback based on the user’s behavior, which can improve the interaction experience and provide a better sense of companionship [12].

Besides the effects on visual and tactile perception, the paradigm of virtual agent-initiated interaction can further enhance the sense of companionship [13]. This type of agent-initiated interaction demonstrates the intelligence of the agent, leading to a more immersive and engaging user experience [14]. However, due to the complexity and variety of the haptic proxies’ design process, the haptic behaviors initiated by agents have not been considered in previous research [15,16]. Currently, the haptic proxies of virtual agents are still difficult to design and face major challenges in widespread applications.

In this paper, we present a modular haptic agent (MHA) prototype system, which enables the rapid construction of diverse virtual agents’ haptic proxies and interaction modes as shown in Figure 1. The MHA system focuses on addressing two key challenges, i.e., the diverse design of haptic proxies and the active interaction of agents. For the first problem, MHA adopts a modular design structure, and different pet agents with physical haptic modules are implemented. Distinct tactile elements of various agents are compiled to create diverse haptic proxies and physical props. By changing different patches and props on the module, MHA can simulate different tactile elements of various pets and map different parts of pets through a movable mechanical structure. For the second problem, we demonstrate different interactions between users and virtual pets and divide the interactive modes between users and virtual agents into passive interaction and active interaction. Passive interaction is easy to perform, while active interaction requires some additional effort. To support the active interaction, we achieved pet-initiated interaction by monitoring the user’s head movement, hand position, and the position of the interactive object in real time. The design methods of MHA enable designers to build diverse virtual agent haptic proxies in VR and improve the user’s interactive experience to obtain better accompaniment.

Our main contributions are three-fold:

1.

We propose the design methods of the haptic module of virtual agents by deconstructing the tactile elements of haptic proxies. We also provide a haptic mapping method for haptic simulation to fit the appearance and animation of different virtual agents.

2.

We develop a prototyping system called MHA to build haptic proxies of virtual agents in VR. The MHA system allows for predicting some of the user’s interaction intentions and enables the agent to initiate a haptic interaction with the user.

3.

Three user studies have been conducted to verify the advantages of building modular haptic agents based on the modular design method and the encountered-type haptic interactions. Experimental results have proved that MHA has the potential to quickly build diversified companion agents with haptic interactions in VR.

The remainder of this paper is structured as follows. Section 2 reviews the related work about encountered-type haptics and companion agents. Section 3 and Section 4 introduce the design methods and the system implementation, respectively. Section 5 conducts experiments about the system performance through user studies. Section 6 discusses some further results, and Section 7 gives the conclusion.

2. Related works

2.1. Virtual Pets

Pets have long been regarded as partners of human beings. The interactions between humans and animals can influence health and mood on both psychological and physiological levels [17]. There have been many studies on the value of human–pet interactions. Pet companions can reduce anxiety, loneliness, and depression, thereby delaying the onset of stress-related illness [17,18]. Pets can also increase sensory stimulation while lowering blood pressure and heart rate, making people of all ages feel comfortable [19]. In addition, studies have shown that virtual pets have a very good adjunctive therapeutic effect on children with autism [20] or asthma [21].

Ways to interact with pets or virtual pets have been the focus of researchers. One of the ways is interacting without physical haptics through a display screen, and another is to interact with physical virtual pet toys or pet haptic proxies to engage multiple senses of the users. Chahyana et al. proposed a virtual pet system based on the Android platform, in which users could interact with each other through applications and understand the daily behavior of pets [22]. Their virtual pets could only provide visual feedback, and they expressed their concern that the application of non-haptic virtual pets cannot replace the real ones. Pretty Pelvis added a force feedback device to the interaction system to obtain the user’s sitting posture, thus guiding the user to adjust their sitting posture promptly by interacting with virtual pets [23]. On top of this, MobiliMB presented a 5-degree-of-freedom mechanical device to be installed at the bottom of the phone, giving the user limited force feedback of a virtual pet [24]. Metazoa Ludens allowed users to play with their pets together remotely in a mixed reality environment [25].

Another approach is simulating physical pets through pet toys or proxies to provide a human–pet interaction experience. One of the most typical examples is the robot dog “AIBO” developed by Sony. AIBO had an image sensor that allowed it to walk to its target, kick a ball, or bump its head. Users could also actually touch AIBO’s head sensor to obtain force feedback [26]. AIBO had a wide range of applications not only in the field of entertainment but also in medical treatment and education [27]. It is also worth mentioning that iCat had the function of signal input and output at the same time, which could obtain the user’s intention and provide feedback with expressions and actions [28]. In addition, Lee et al. proposed a remote interaction device based on mixed reality, which could simulate the actions of remote pets through mechanical entities to achieve both-way interactions [16].

2.2. Encountered-Type Haptics in VR

The haptic interface has become essential for enhancing user experiences in VR systems [29]. In early research, McNeely proposed the concept of robotic graphics [30] and pointed out the need for various types of haptic feedback in VR. Encountered-type haptics refers to tactile behavior, in which one or both users or interactive objects move towards each other [31,32]. Among them, haptic solutions in VR fall on a spectrum from passive props to active props [33].

To achieve encountered-type haptics with passive props, physical props need to be matched with virtual environments or virtual objects to provide haptic feedback. Many representative works have been put forward, such as using integrated modules to achieve different haptics [34,35], manipulating props [36,37], self-assembled cell modules [38], and building synchronized scenes [39]. Snake Charmer is a typical example [35], which uses the rotation of the polyhedron to dynamically change the surface and make contact with the users to simulate multi-angle texture, temperature, and force feedback information. However, passive props have certain limitations. The physical props must be meticulously aligned with their virtual counterparts in terms of shape, position, and tactile feedback [40,41]. Furthermore, passive props do not offer force feedback during user interaction, which could diminish the user’s sense of presence [31].

To provide more general haptic feedback in VR, researchers proposed active props in encountered-type haptics. For example, robots [42] or humans [43,44] carried the devices to the target position to match the users’ feelings in the virtual world [45]. When users contacted virtual objects, haptic devices could dynamically transfer physical props or provided force feedback to users, which could significantly improve the users’ sense of realism. Active props also provided new possibilities for human–robot interactions.

Bouzbib divided the current encountered-type haptic devices into two dimensions, which were physicality and actuation [11]. Physicality refers to whether the haptic system uses real haptic props to simulate haptics. Actuation refers to whether motor-based hardware is deployed to simulate the autonomous movement of haptics. For a virtual pet agent, we suggest that it is necessary to simulate both the tactile sensation with real props for the physical dimension and the kinetic feedback with mechanical devices for the actuating dimension. Although several works have investigated encountered-type haptic devices that provide haptic feedback by predicting the user’s intention [37,46,47,48], little attention has been paid to achieving both tactile sensation and kinetic feedback when interacting with virtual agents [32], especially for virtual pets. One way to solve this problem is to build different interaction objects, using changeable interaction devices [49,50]. We would like to further work on the changeable interaction devices concept to achieve a variety of haptic interactions with virtual intelligent pets through the modular design method and the haptic mapping method.

3. Design Methods

Designing haptic proxies and human–agent interaction in VR involves answering two main research questions.

• RQ1: What is an efficient method for constructing a large number of haptic proxies for different agents using minimal props?
• RQ2: How can agent-initiated interactions be designed to enhance the sense of companionship?

To design haptic interaction between users and virtual pets in VR with quality and efficiency, we propose two key schemes to solve the two problems above, respectively. For the diverse haptic proxies construction, we adopt the modular design method and the visual–haptic mapping method. For agent-initiated interaction, we classify human–agent interaction and design an active interaction system. We use a system prototype of various virtual pets to illustrate our modular haptic proxy design and interaction design.

3.1. Haptic Proxies Construction with Minimal Props

3.1.1. Classification of Tactile Elements

Tactile elements are the sets of haptic props that share the same haptic rendering properties, reflecting a similar class of attributes (e.g., size, texture, softness, temperature, etc.) [51,52,53]. Tactile elements of the same type can be used to approximate the tactile sensation of a class of objects [54,55]. Therefore, haptic elements of the same class can be represented by the same haptic props.

We surveyed more than 20 common household pets through questionnaires and expert interviews to investigate pets’ tactile elements and interactive behavior. Though different pets have different shapes and haptic rendering properties, through our survey, we found that most pets have very similar tactile elements. Their shapes and tactile elements can be roughly divided into five categories, i.e., flat surface, cambered surface, hard sharp corner, triangular-shaped or stripe-shaped ears and tails, and other special categories. As a result, the tactile sensation of different parts of a virtual pet can be simulated by constructing different categories of tactile props and mapping them to the position of the specified virtual avatar. We demonstrate different categories of tactile elements, some examples of haptic props, and the different parts of the virtual pets they correspond to as shown in Figure 2. Each of the five categories of tactile elements is described below.

• Flat surface: A large flat soft surface, often covered with long or short fur (e.g., the heads and bodies of dogs, pet pigs, and pet calves). When the users come in contact with the flat surfaces, their hands are often extended, and users touch along the plane or gently pat along the normal direction of the plane.
• Cambered surface: A soft surface with a large or small radius of curvature, often covered with long or short fur (e.g., the heads of kittens, ducks, and rabbits). When the users come in contact with the cambered surfaces, their hands are often half open, and users touch them with their palms or fingertips, sliding in a circle with their hands.
• Sharp corner: A hard, sharp pyramid or cylinder. For small sharp corners (e.g., the mouths of ducks), users often touch them with the tips of their fingers or palms. For large sharp corners (e.g., the horns of pet calves or lambs), users often touch or hold them with multiple fingers and palms.
• Ears and tails: A soft triangular or elliptical piece or column, often covered with long or short fur (e.g., the triangular ears and elliptical tails of dogs, pet pigs, and kittens). When the users come in with contact the ears, their hands are either fully extended or bent to pinch. When the users come in with contact the tails, their hands often hold the tails or rub up and down.
• Others: Some special pets have special tactile elements that are very unique (e.g., the long ears of rabbits, the mane of pet ponies, and the nose and tail of pet pigs). When interacting with these pets, users will make specific haptic interactions according to specific situations (e.g., patting the ears of rabbits, fondling the mane of ponies, and pinching the nose of pigs).

3.1.2. Modular Design Method

Based on the composition of virtual pets, we can use the five tactile elements to represent the physical proxies of virtual pets. Therefore, tactile proxies can be built in the form of a module to achieve a realistic pet tactile sensation in the virtual environment. The main concept behind the modular design method is to create virtual pet proxies using a modular structure. By assembling modules containing tactile elements, designers can quickly construct haptic proxies.

We design and produce the prototype MHA module. The MHA module consists of a module core, detachable patches, and props on the patches (see Figure 1a). The module core is located in the center of the module and is connected to the patches and mechanical devices. The detachable patches can be secured on the module core as holders for props. Props are closely connected to the patches to form tactile elements of virtual pets. In addition, a variety of sensors and drivers can be installed on the patches to detect the users’ touch behavior and provide feedback. By combining the module core, the detachable patches, and the props, all the tactile elements of a pet can be represented.

3.1.3. Haptic Mapping Method

When all the tactile elements in the haptic proxies are formed through the modular method, they need to be allocated in different specific spatial positions to achieve the tactile sensation of the pets. Since we use the same module core, a vision–haptic mapping method ought to be programmed to map different sizes of virtual agents when constructing the haptic proxies. We show examples of different assemblies and the mapping of three different virtual pets. For the same module core, we use different patches and props to construct the tactile elements required by three virtual pets.

For the small pet duck, the actual module size basically corresponds to the size of the virtual duck’s head. By moving and fixing the actual module to the position of the virtual image, as shown in Figure 3a, the simulation of the duck can be achieved.

For the medium-sized pet dog, the actual module size is smaller than the entire size of the virtual dog. Due to the matching characteristics of users’ vision and touch, users often look at the position they are about to touch [56]. We monitor the user’s viewing position according to the collision between the user’s vision and parts of the virtual image to further predetermine the position that the user intends to touch. By using a mechanical device that continuously drives the module to move to positions where the user looks, and aligning the haptic props to the tactile elements on the virtual images, designers can use one small module to match the whole virtual dog’s tactile elements as shown in Figure 3b.

For a large pet calf, we can use the same mapping method to achieve the tactile stimulation of different parts of the pet calf as shown in Figure 3c. Furthermore, for other pets, designers can reasonably divide different areas of tactile elements according to the size of the virtual image and achieve different mapping methods.

3.2. Agent-Initiated Active Interaction

A large part of human–agent communication is achieved through haptics. Interactive behavior can be initiated by either users or virtual agents. Suzuki proposed the concept of explicit interaction and implicit interaction in his work [57]. In explicit interaction, the users directly touch the interaction props with their hands or bodies, while implicit interaction takes the user’s implicit motion as input, such as the user’s position or proximity to the robot [58]. In haptic interaction, we divide these two forms of interaction as passive interaction and active interaction depending on the haptic initiating agent.

From the perspective of virtual agents, we define passive interaction as user-initiated haptic interaction, while active interaction is interaction initiated by an agent when it notices some of the user’s intention to interact. In passive interaction, users approach and touch the pets, and then the pets give feedback according to the touch behavior of the users. In active interaction, pets sense the user’s intention and spontaneously approach users to exert force on users.

Based on the survey mentioned in Section 3.1, five common haptic interactions of common household pets are summarized, i.e., head touch, body touch, discipline, feeding, and rubbing. Head touch and body touch fall under the category of passive interaction; the user touches different parts of the pets, and pets respond to the user’s haptic behavior. Pets react differently depending on where the user touches them. While discipline, feeding, and rubbing are active interactions, the pets sense the user’s intent and interact spontaneously with the user. In particular, the term “discipline” in our study refers to the haptic interaction initiated by the virtual pet with the user after the user has issued a non-haptic command to the pet, such as beckoning the pet to come closer.

The judgment of user intention in active interaction includes head direction [59,60,61], proximity [58,62], gesture [63,64], voice [65,66], and other interaction modalities [57,67]. Among them, head direction and proximity interaction are the two most important aspects [68,69]. Head movements can provide a convenient and natural addition to human–agent interaction [70,71], and proximity interaction effectively reflects the interpersonal relationship between the human and the agent [72]. We use two assumptions for active interaction decisions in our system: i) if the user’s head direction is oriented to the virtual agent for a period of time, the agent will initiate an interaction, and ii) if the user’s hand or interactive object approaches the virtual agent, the agent will initiate the interaction.

Not every agent has all these types of interactions. For instance, some birds do not allow their bodies to be touched in passive interaction, and some pets do not understand the user’s intent and may not rub themselves against the user’s hand. Though the above situations exist, these interaction types basically cover most of the interaction design of all kinds of virtual pets. Compared with previous works [7,15], the active interaction design expands the possibility of interaction between users and virtual agents. Combining agent-initiated interaction with user-initiated interaction is more rational, and more likely to be preferred by users and provide better companionship.

4. System Implementation

4.1. Overview

According to the modular design method and the mapping method, we proposed the modular haptic agent, a prototype system for the rapid construction of virtual agents’ haptic proxies and interactions. We attached the MHA module with all tactile elements of a virtual pet to a mechanical arm that drove it to achieve tactile simulation, encountered-type passive and active interaction as shown in Figure 4. Our prototype system included (i) a haptic module with tactile elements, (ii) a mechanical device to provide haptic mapping and force feedback, (iii) a microcomputer for signal exchange and circuits, and (iv) a suit of virtual reality devices for display and tracking. Additional information can be found in the supplementary material on Video S1.

4.2. Module Design

4.2.1. Module Core

The module core is an empty cube shell, one side of which is fixed with the mechanical device, and the other five faces are perforated to connect different detachable patches to simulate tactile elements. To ensure the authenticity of interactions, the size of the module core has certain requirements. The module should not be too small, for the user’s whole palm should be able to touch the tactile elements. However, the module should not be too large due to the limitation of the mobile range of the mechanical manipulator. We investigated the sizes of common pets and the palm sizes of people of different ages and genders in advance and found that the plane range of 12 cm × 12 cm was accepted by users. We made a cube with a side length of 12 cm using the 3D printer, served as the module core.

4.2.2. Detachable Patches and Props

Detachable patches were holders for sensors and props. Various props could be used to simulate the tactile elements of different pets. Each patch was perforated in a standardized way at the same position on the module core surface. Any of the patches was able to connect to any of the surfaces of the module core.

After fixing the patches, we installed the sensors and servos on the surfaces of the patches for haptic input and haptic output, as shown in Figure 5. Hand tracking is not always accurate due to the loss of depth information when the user’s hands are trapped in the plush. So, we used photo sensor pasting on the patch and punched a tiny hole on the surface of the props for haptic input. When the user touches the patch, their hand will cover the photo sensor, and thus the sensor will capture the touch signal. The reason we did not use touch sensors is because their large metallic surface might have affected the users’ sense of touch. We were able to install a servo motor on the patch with slots and place the props on the top to form movements of the duck’s mouth for haptic output. Since the servo motor can move quickly, the tiny movements of the duck’s mouth were able to be simulated.

We used an Arduino Uno microcomputer to process the user’s different touch signals and the servo motor driving signals. The real-time signal exchange was deployed between the microcomputer and processor. The microcomputer transmitted the touch signal to the processor, and the processor transmitted the servo motor moving signal to the microcomputer.

4.3. Tracking System

We captured some of the user’s interaction intentions in real time by the line of sight, the hand position, and the position of interactive objects.

(1) Line of sight. The center of the head-mounted display emitted a virtual ray to determine the user’s visual location, detecting the head direction of the user. We detected whether this ray crosses the mesh of the virtual agent in real time.

(2) User’s hand position. We used Leap Motion to track the user’s hands. The Leap Motion Controller provided a virtual hand image, guiding the user to touch the virtual pet, and monitoring the distance between the user’s hand and the module.

(3) Position of interactive objects. When the user interacted with the virtual agent using an interactive object, the tracker on the object was used to determine the spatial position for active interaction.

4.4. Encountered-Type Haptic Interaction

To better illustrate the MHA encountered-type haptic design process, we used an example of interaction with a pet dog to illustrate our design methods. It is important to note that the encountered-type haptic interaction of different agents could be designed similarly.

We assembled a module with five patches. We divided the virtual dog into a head part and a body part, each of which approximates the size of the module. The props on the patches were composed of a flat plane prop (top of the head), two large curved props (can be either the cheeks of the head or the back of the body), a small curved prop (front face of the head), and a tail prop (tail of the body). Each patch had a sensor to register the user’s touch behavior. We pre-edited the positions, moving speeds, and strength of the mechanical arm to provide the shape simulation and encountered-type haptic interaction.

The virtual dog was about the size of two modules. We divided the virtual dog into a head area and a body area. When a user’s visual ray hit an area, the processor would send the moving signal to the mechanical arm, driving the same module to match different areas. In the encountered-type haptic interactions, we edited the force of the mechanical arm to match the feedback animation of the virtual pets.

Passive interaction. The users came close to the virtual pets and touched them. The virtual pets gave the user movement feedback and force feedback after the user’s touch behavior. For the pet dog, different animation and force feedback can be given when different parts of the dog are touched. As shown in Figure 6a, when the users touched the dog’s face, the virtual dog would nod, and the mechanical arm drove the module to move in an arc to provide force feedback.

Active interaction. When the user’s hands or the tracked food container were close to the dog, it would understand some of the user’s intentions and come up to the user. When the distance between the users’ palm or interactive props and the MHA module was less than the threshold value of 10 cm, the animation system played the animation of the pets moving towards the users, and the mechanical arm synchronously drove the module to the user’s hand position to provide the active interaction behavior of the virtual pet. Different animation feedback and force feedback could be triggered when the user placed their hands or interactive objects in different positions. For instance, as shown in Figure 6b, when the users’ single hand or both hands were placed in front of the virtual dog for a period of time, the dog would take the initiative to rub against the user’s palm, and the mechanical arm drove the module to move back and forth to provide force feedback.

4.5. Software

We used Unity3D to render virtual pets on the processor, and Unity3D was responsible for signal processing and logical judgment of all kinds of interactions. The processor received the user’s sight position, hand position, tracker position, and touch signal in real time, judging the user’s interaction intention and behavior. Based on the different signal inputs, the processor could determine the current animation to play, as well as the signal to be sent to the servo motor and the mechanical arm. The pseudo-code of our designed interaction logic is shown in Algorithm 1.

Algorithm 1 MHA system control algorithm.

1: Input: line of sight L, haptic input $H_i$, hand-module distance $D_h$, tracker-module distance $D_t$, head orientation time threshold t = 5 s, interaction distance threshold d = 10 cm. 2: Output: mechanical device terminal position P, mechanical device force feedback M, module haptic output $H_o$, virtual animation feedback A. 3: for each time of interaction do 4:    Move P to align the position of L; (Haptic Mapping) 5:    if $H_i!=0$ then 6:      Passive interaction(user-initiated) is detected; 7:      Activate M feedback, activate $H_o$ feedback, activate A feedback; 8:    else 9:      Remain stationary and be ready to move; 10:      if $D_h$ < d or $D_t$ < d or (the time of L on P) > t then 11:         Active interaction(user’s intent of interaction) is detected; 12:         Activate M feedback, activate $H_o$ feedback, activate A feedback; 13:      end if 14:    end if 15: end for

5. System Evaluation and User Study

To verify that the MHA system had the potential to address both RQ1 and RQ2, we conducted three studies to evaluate the MHA system. For RQ1, Study 1 and Study 2 were designed to evaluate the modular design method and the haptic mapping method, respectively. For RQ2, Study 3 was designed to evaluate the enhancement of the virtual agent active interaction for the user experience. Study 1 was to verify whether the MHA system could simulate the tactile sensation with a modular unit that was smaller than a virtual agent. Meanwhile, the MHA system could provide the same user experience as a larger haptic prop that matched the size and shape of the virtual agent. Study 2 discussed the system generalization performance when users touched virtual pet agents of different sizes and shapes. Study 3 tested the effect of passive and active interaction on users’ interactive experience in the encountered-type interaction.

5.1. Participants

In total, 30 participants (10 females and 20 males) aged 21–32 ($M=24.2$) were recruited to participate in these studies. All participants reported no fear or resistance towards the virtual pets we designed. We taught each participant for 5–10 min to ensure that they were familiar with the images of the virtual pets and could interact with the MHA system safely.

A total of 30 participants were enrolled in the system evaluation, and each individual participated in all three studies (Studies 1, 2, and 3). The order of the three studies was randomized for each participant.

5.2. Apparatus and Materials

The virtual environments were displayed with an HTC VIVE Pro. A Leap Motion Controller was attached to the head-mounted display for users to observe their hands, and the HTC VIVE Tracker was used for tracking interactive props. The experiment was conducted on a computer with the Intel Core i7 processor and the NVIDIA RTX 3080 graphics card, and the software was implemented in Unity 2019.4.10f1. All virtual pets’ 3D models were obtained from the Unity Asset Store. The module containing all tactile elements for each study was assembled in advance and fixed on a KUKA lbr iiwa R820 mechanical arm, whose manipulator end can move in a 1 m cubic area in 2 s, and provide forces of less than 30 N in any direction. Arduino Uno was selected for signal processing and exchange, and an SG-90 servo motor was used to provide feedback on the module. The haptic proxies available for haptics on the module are equipped with three-pin photo sensors to facilitate the perception of touch signals. We planned the movement path of the mechanical arm in advance according to different pets and programmed the mechanical arm. Although we limited the movement speed and force of the robotic arm, we were concerned that the participants might hit the module or the robot arm if they moved their arms blindly, which could be dangerous. Therefore, we informed the participants in advance of the limited area of the virtual pet that they could touch, and we provided instructions to participants with and without the VR headset.

5.3. Questionnaire

There were four main aspects to consider when designing a virtual agent with a sense of companionship, which were realism, interactivity, attraction, and user emotion [15,73]. We designed our questionnaire based on standardized human–computer interaction questionnaires [74,75,76]. The questionnaire was a 7-point Likert scale, with scores ranging from zero for “strongly disagree” to six for “strongly agree”. The participants were asked to fill out this questionnaire after finishing a group of tests. A total of $4\left(questions\right)\times 4\left(indexes\right)=16\left(questions\right)$ were asked on each scale, which were related to realism, interactivity, attraction, and user emotion, respectively. The three studies shared the same questionnaire to maintain the same evaluating standard. The questionnaire can be found in the appendix (see Appendix A).

5.4. Study 1: Module Design Method and Mapping Method

5.4.1. Experiment Design and Procedure

We adopted a comparative experiment to verify our modular design method and haptic mapping method. Each participant interacted with a virtual dog in a random group order. For each set of groups, the image and animation of the virtual dog were the same, and the only difference was the haptic perception. Users interacted with the virtual dog for three to five minutes in each group.

• NHD group (none haptic dog): Participants could see the pet dog but could not touch it (see Figure 7a). When their virtual hands collided with the virtual dog, the virtual dog provided animation feedback.
• HapticDog group (MHA haptic dog): We designed the module in advance and planned the path of the mechanical arm (see Figure 7b). When a user looked at different positions of the dog, the mechanical arm drove the module to move to the corresponding positions of the virtual dog. When their virtual hands touched the virtual dog, the dog provided animation feedback.
• ST group (stuffed toy): We prepared a plush toy dog similar in shape and size to the virtual image (see Figure 7c), which could be regarded as a 1:1 haptic proxy of the virtual dog based on previous research [77]. The size and shape of the stuffed toy were identical to those of the avatar. Therefore, it was only necessary to position the stuffed toy in the corresponding location.

5.4.2. Results

We collected 30 results from each group (see Figure 8). Except for the three indexes of realism, interactivity, and attraction in the ST group, the rest of the data were all in line with normal distribution. A Wilcoxon signed-rank test is used to analyze the results.

Significant differences were found between the NHD group and the HapticDog group in realism (Z = −4.60, p < 0.001), interactivity (Z = −4.75, p < 0.001), attraction (Z = −4.46, p < 0.001) and user emotion (Z = −4.58, p < 0.001). Additionally, significant differences were found between the NHD and ST groups in realism (Z = −4.67, p < 0.001), interactivity (Z = −4.66, p < 0.001), attraction (Z = −4.53, p < 0.001) and user emotion (Z = −3.88, p < 0.001). No significant difference was found between the HapticDog group and the ST group in realism (Z = −1.60, p = 0.11), interactivity (Z = −1.12, p = 0.24), attraction (Z = −0.30, p = 0.77) and user emotion (Z = −0.32, p = 0.76).

5.4.3. Results Discussion

We found that participants with VR experience rated the ST group lower, while participants without VR experience rated the ST group higher, which might be the reason why the three groups of data for stuffed toys did not conform to a normal distribution. Since some participants were more familiar with haptic interaction in VR environments, their expectations may have affected their evaluation status.

In addition, we found improvements in four indexes between the NHD group, the HapticDog group, the NHD group and the ST group. The experimental results showed an enhancement of the haptic interaction system compared to the visual system. There was no significant difference between the HapticDog group and the ST group, indicating that, with a small module, it is possible to achieve a realistic haptic simulation of large haptic props by mapping methods.

5.5. Study 2: Simulation of Different Pets

5.5.1. Experiment Design and Procedure

To verify that our system could simulate pets of different sizes and shapes, we designed a pet calf. Each participant interacted with the calf in a random group order.

• NHC group (none haptic calf): Participants could see the pet cow but could not touch it (see Figure 7d). When their virtual hands collided with the virtual calf, the calf provided animation feedback.
• HapticCalf group (MHA haptic calf): We reassembled the module and planned the path of the mechanical arm (see Figure 7e). When a user looked at different positions of the calf, the mechanical arm drove the module to move to the corresponding positions of the virtual calf. When their virtual hands touched the virtual calf, the calf provided animation feedback.

The interaction process with the virtual calf was the same as that of the virtual dog, and the animation feedback of the virtual calf was the same as that of the virtual dog in Study 1. Users interacted with the virtual calf for three to five minutes in each group. We still used four indexes in each group to describe the realism, interactivity, attraction, and users’ emotion. We compared the results of Study 2 with Study 1 to verify whether there were differences in the four indexes regarding the user’s score of various agents.

5.5.2. Results

The results of the NHD, HapticDog, NHC and HapticCalf groups were compared together (see Figure 9). The four indexes of all groups were under the normal distribution. We paired the four indexes of the four groups and conducted several paired t-tests to analyze the differences.

From the comparison of different pet images, no significant difference was found in any of the factors, including realism (t = −1.07, p = 0.29), interactivity (t = −0.50, p = 0.62), attraction (t = −0.92, p = 0.37) and user emotion (t = −1.00, p = 0.33) between the NHD and NHC groups. Additionally, there was no significant difference in the realism (t = −1.38, p = 0.18), attraction (t = −0.30, p = 0.77) and user emotion (t = −1.19, p = 0.24) between the HapticDog and HapticCalf groups. A significant difference was found in the interactivity index (t = −2.80, p = 0.01) between the HapticDog and HapticCalf groups.

From comparison on haptic feedback, significant differences were found between the NHD and HapticDog groups in realism (t = −8.15, p < 0.001), interactivity (t = −9.32, p < 0.001), attraction (t = −6.86, p < 0.001) and user emotion (t = −7.53, p < 0.001). Meanwhile, significant differences were found between the NHC and HapticCalf groups in realism (t = −8.36, p < 0.001), interactivity (t = −11.00, p < 0.001), attraction (t = −8.27, p < 0.001) and user emotion (t = −6.27, p < 0.001).

5.5.3. Results Discussion

Through comparisons of Study 1 and Study 2, there were no significant differences in the visual aspects of the virtual pet images experienced by the participants, thus providing a baseline for haptic interaction with different pets. Among them, the participants felt certain differences in interactivity between the HapticDog and HapticCalf groups. We thought the possible reason might be that the virtual calf was larger than the virtual dog, and the participants were more likely to observe the calf’s feedback and, therefore, higher ratings were given to the interactivity index. Through comparisons on Study 2, we found that the method of module design and mapping could effectively improve all indexes.

According to the results, we verified the potential of the MHA system to simulate virtual pets of different sizes and shapes. Our proposed modular design method and mapping method has the potential to be expanded in the design of different pet agent haptic proxies. MHA could significantly improve users’ virtual experience compared with the ones with no haptic feedback.

5.6. Study 3: Passive Interaction and Active Interaction

5.6.1. Experiment Design and Procedure

To explore the influence of force feedback on passive interaction and active interaction, we adopted another set of comparative studies. To eliminate the influence of the previous two experiments, we adopted another virtual avatar of a duck and reassembled the module. Each participant interacted with the duck in a random group order:

• NF group (no feedback): When a participant touched different parts of the virtual duck, it responded to the user’s touch with only animation feedback.
• PF group (passive feedback): When a participant touched different parts of the virtual duck, the mechanical arm drove the module to generate different animation feedback and force feedback for the participant.
• AF group (active feedback): When the hands of a participant approached different parts around the virtual duck or the participant’s head oriented towards the virtual duck for 5 s, the mechanical arm drove the module to generate different animation feedback and force feedback for the participant.

The participants stood in front of a virtual duck in the virtual environment. We placed two photo sensors on different module patches. In the NF group, when a photo sensor received the touch signal, the duck provided animation feedback of touching the part of the duck. In group PF, when a photo sensor signal received a touch signal, the mechanical arm drove the module to provide force feedback, and the virtual duck played the equivalent animation feedback at the same time. In the AF group, in addition to the PF group, we tracked the user’s head direction and hand position. When the center ray of the head-mounted display hit the bounding box of the virtual duck for 5 s, or one of the palm centers of the user approached the virtual duck to a range of a given threshold (e.g., 10 cm), the mechanical arm drove the module to the user’s palm, and the virtual duck played the equivalent animation feedback at the same time (see Figure 10). For each set of experiments, the virtual duck had the same image and animation. Again, users interacted with the virtual duck for three to five minutes in each group and rated 16 questions on all 4 indexes.

5.6.2. Results

We collected 30 results on the NF, PF, and AF groups (see Figure 11). The four indexes of the three groups of data were all under the normal distribution. The results of the paired t-test showed that significant differences were found between the NF and PF groups in realism (t = −6.98, p < 0.001), interactivity (t = −6.63, p < 0.001), attraction (t = −5.32, p < 0.001) and user emotion (t = −8.08, p < 0.001). Furthermore, there were further improvements in realism (t = −5.53, p < 0.001), interactivity (t = −4.51, p < 0.001), attraction (t = −2.97, p < 0.001) and user emotion (t = −5.89, p < 0.001) when we compared the AF and PF groups. Additionally, significant differences were found between the AF and NF groups in realism (t = −8.93, p < 0.001), interactivity (t = −7.29, p < 0.001), attraction (t = −5.49, p < 0.001) and user emotion (t = −9.01, p < 0.001).

5.6.3. Results Discussion

The comparison between the NF and PF groups showed that encountered-type haptic passive interaction had better comments on realism, interactivity, attraction, and users’ emotion. The comparison between the PF and AF groups showed that the design of the encountered-type haptic active interaction could further promote user experiences when interacting with virtual pets.

It is worth mentioning that in the AF group, the average score of the user’s emotion index was 5.29 out of 6. In terms of the prompt “it makes me feel happy”, 24 out of 30 people gave “strongly agree”, indicating that active interactions had a great effect on improving the psychological feeling of the users and providing better companionship.

6. Discussion

6.1. Further Analysis

Three user experiments evaluated our designed MHA system from different perspectives, focusing on both the design of different haptic proxies and the active interaction of agents.

Aiming at the problem of how to quickly construct multiple haptic agents with limited haptic props, Study 1 and Study 2 together illustrate the modular design method and haptic mapping method for haptic proxies, allowing the construction of multiple haptic agents with a small and limited number of haptic props. By modularizing the tactile elements and utilizing a mechanical device to map different parts of the agent, designers can assemble different haptic proxies to simulate the shapes and tactile elements of different agents, and thus the number of haptic props can be reduced to some extent. At the same time, the design methods of MHA allow designers to build unlimited types of haptic proxies with limited haptic props, which has a design space that can be further extended.

In addition, we carried out a haptic interaction design actively initiated by the haptic agents. Haptic agents can understand certain interaction intentions of the users through their head movements, hand positions, and interactive object positions, and take the initiative to operate a haptic interaction. The design of active interaction makes up for the lack of agent-initiated haptic interaction in VR and receives better comments from users. The MHA system can improve realism, interactivity, the attraction of virtual agents, and user emotion in VR, and provide better companionship for users. The design methods of the MHA system can help stakeholders quickly build active–interactive virtual pet agents and have the potential to be applied to the haptic design of various intelligent agents in VR.

6.2. Limitations and Future Work

Based on our design and application demonstrations of the MHA system, we identified several issues and potential future works to discuss the prototyping of haptic proxies.

Our entire system currently uses a 3D-printed module, some handmade patches and props for pets, and a lightweight robotic arm. The system is expensive, as it is still in the prototype stage. Since virtual companion agents may be used in home scenarios in the future, further research may investigate users’ needs for haptic agent design, including simpler methods for patch design, and smaller robotic arms or unmanned vehicles to provide encounter-type haptics.

We present three examples of different pets in this paper to illustrate our module design method and haptic mapping method for MHA, showing that our system can be adapted to different agents. Although we were able to achieve haptic interaction at different locations for some pets with one module, the challenge of continuous haptics remains, and advanced settings are required during module assembling and mapping. Our current system uses a cube module with a side length of 12 cm, which is sufficient to simulate the tactile sensation of a certain part of the virtual pet. However, when touching a large agent from head to tail, the user’s hand inevitably crosses a 90-degree angle, which may break the immersion. Moreover, the accidental touching of haptic props that should not be touched may confuse the user. Therefore, future research could explore the use of multiple, smaller self-assembling modules or robots to achieve continuous haptics. Smaller robots could enable automatic assembly and spatial topology reorganization during user interaction, resulting in a continuous haptic approach.

In the MHA system, the line of sight position, hand position, and tracked object position are used together to predict some user’s interaction intention and drive the agent to initiate interaction with the user. However, due to the diverse interaction scenarios and different virtual agents, the user’s interaction intention is difficult to predict. In the future, we plan to use a larger algorithmic model for the companion agent to process the multimodal user input, jointly determine the user’s multiple interaction intentions, and provide feedback in a proper manner.

In this work, we use the Leap Motion Controller to capture and present the users’ hand position. The Leap Motion Controller has the advantage of allowing haptic interactions with bare hands, users do not need extra devices on their hands. However, the Leap Motion Controller, as an infrared detector, may be affected by the shape of the module. For instance, when a user touches a soft part of the patches, the user’s fingers may become trapped inside the soft patches. Losing the depth information of the fingers makes it difficult to recognize the users’ fingers. Future work may investigate a way to obtain the hand position more accurately, which can bring a better experience for users.

7. Conclusions

In this paper, we propose an MHA prototype system based on the modular design method, the haptic mapping method, and the encountered-type haptic device. We demonstrate our system with examples of human–agent interactions. The MHA system has two notable advantages. First, we put forward the modular design method and haptic mapping method to construct multiple haptic proxies with limited props. Second, the haptic agents can initiate interaction with the user through the real-time tracking of the user’s eyes, hands, and interactive objects. Study 1 and Study 2 prove that the MHA system can achieve the tactile simulation of different sizes and shapes through a single module. Study 3 shows that virtual agents with active interactions have better reviews than passive interactions and non-haptic virtual agents on realism, interactivity, attraction, and users’ emotions, and thus provide better companionship.

We believe that the modular design method and the use of the haptic mapping method to achieve haptics at different locations are effective in the design of haptic proxies and have the potential to be extended to other types of haptic proxies’ designs. In addition, the encountered-type active interaction design will expand the design space of haptic feedback, which can provide a better companion in virtual environments. We expect that our system and demonstrations will benefit future researchers to provide better and more diverse experiences in VR.