Multimodal Mixed Reality Impact on a Hand Guiding Task with a Holographic Cobot

Abstract

In the context of industrial production, a worker that wants to program a robot using the hand-guidance technique needs that the robot is available to be programmed and not in operation. This means that production with that robot is stopped during that time. A way around this constraint is to perform the same manual guidance steps on a holographic representation of the digital twin of the robot, using augmented reality technologies. However, this presents the limitation of a lack of tangibility of the visual holograms that the user tries to grab. We present an interface in which some of the tangibility is provided through ultrasound-based mid-air haptics actuation. We report a user study that evaluates the impact that the presence of such haptic feedback may have on a pick-and-place task of the wrist of a holographic robot arm which we found to be beneficial.

1. Introduction

Industry 4.0 [1] is the term used for an ongoing fourth industrial revolution, according to which factories of the future will comprise machinery, warehousing and production facilities in the shape of cyber-physical systems.

Collaborative robots, also known as ‘cobots’, are one such type of technological assets that interact and cooperate with workers within the same shared workspace.

In order to tag a robot as collaborative, it needs to meet some key requirements [2]. The first requirement is trust, which is essential for collaboration and ensures the safety of the human worker during the task. A robot with this feature also needs to match or improve the efficiency of the task performance (e.g., the time spent doing the work). Related to trust, it is also necessary for the user to feel a sense of control. In case of feeling uncomfortable, the human worker must be able to intuitively stop the robot. Finally, the robot has to be predictable in order to avoid unexpected movements that could endanger the human worker.

The ability of cobots to be programmed easily by technically naïve workers is an important requirement for their easy adoption by manufacturing industries [3]. ‘Manual guidance’ (the technique to program a robot by freely moving it by hand within the workspace [4]) is a very useful and widely used technique to program robots in an easy and technically non-demanding way. This is also known as programming by demonstration (PbD) [5]. Moving a robotic arm by hand is intuitive for workers, who are able to transfer their job expertise to the robot without the need of technical knowledge of robot kinematics or programming languages.

Making use of hand guidance for programming imposes, however, the requirement is that the robot has to be available (and not engaged in any other automation-related task) for a worker to have exclusive access to the robot while programming it for a new task. This non-productive time for robot re-training has an evident negative impact on productivity.

As a way around this limitation, we propose performing manual guidance on the holographic representation of the digital twin [6] of the cobot that needs to be programmed, by using current off-the-shelf mixed-reality (MR) equipment. Creating a hologram of the robot that is only visual, however, has the limitation that is lacking the tangibility of grabbing and moving a physical robot. In order to replicate the experience of physical hand-guidance more closely, in this paper we propose that some aspect of the tangibility of the robot might be reproduced by providing tactile feedback to the hand that holds the robot. To do so in a seamless way (i.e., without attaching any equipment to the workers hand), we explore the possibilities offered by a mid-air haptics technology that makes use of ultrasound-based actuation. The goal is that the spatial and temporal combination of visual and tactile augmentations may convey a more realistic representation of the robot, supporting manipulation actions that resemble interacting with the physical robot.

Relevant prior work aiming towards manipulating a hologram does exist in the literature, as it is reviewed in the next section. Such examples from the state of the art made use of different kinds of augmented reality and virtual reality devices in order to program a robotic arm movement. One of the outcomes from such work was the lack of naturalness of the manipulation, which further supports our approach of creating a visuo-tactile multimodal interface.

In preliminary earlier work [7], we prototyped this concept by creating an application with which the user could perform a pick-and-place task of the wrist of a holographic robot. This task represents a part of the hand guidance programming process. We also reported an initial pilot evaluation with a small group of users, with the aim to obtain early indication of the impact that using haptic feedback in performing the task might have on task execution and user experience. Five inexperienced users collaborated in the preliminary study work where we obtained a first impression related to the user experience and distance error and total time spent in the task values. While results suggested that the interaction might be promising, that pilot study was non-conclusive, given the small user group and the limited amount of data collected. In this paper, we report a full study with 16 naïve users performing the same task and procedure, but collecting a larger set of results with the employment of additional research methodologies. With the goal to obtain more solid insight into performance and user experience aspects, we employed a broader range of both quantitative and qualitative methods and metrics.

We monitored and analysed additional quantitative metrics, and we collected subjective experience data with the use of additional user research methods: extended versions of the validated SEQ questionnaire and the NASA-TLX index. The same four-question ad hoc questionnaire as in the previous paper was also still used for comparison (all questionnaires used in the studies are included in Appendix A) and we monitored some data related to the user performance to compare performance with visual only feedback and the multimodal experience.

With the present study, we wanted to shed light on two aspects of the interaction proposed. First, we wanted to know how feasible it was for a user to perform hand guidance of a virtual robot using off-the-shelf mixed reality devices. The devices that we focused on were (i) a Hololens HMD device to create visual holograms and (ii) an Ulrahaptics Stratos Explore device to create the tactile feedback on the user’s hand. In addition, and more specifically, we wanted to learn about the role that adding ultrasound-based tactile stimulation might play in the user’s performance and in the UX obtained. Thus, we formulated the following two research questions:

• Is it possible to program a robotic arm through hand guidance using current mixed reality devices?
• What is the impact of the Ultrahaptics Stratos device in manual guidance through the application?

To adapt the experimental task to naïve users (with no prior experience collaborating with robots), we simplified a programming-by-demonstration task to a task of pick-and-place of the wrist of the robot.

In this way, the user could grab the holographic robot end effector, displacing it into a new position.

The simplification of the task comes from not taking into account the kinematics constraints of the physical robot, which is an important part in a PbD solution because it defines the rotation and movement limits of the robot. However, we put the research focus on the impact that the tangibility of the hologram has during the execution of the task.

The rest of the paper is structured as follows. Section 2 reviews the prior work. Section 3 describes the design of the experimental user study reported in this paper. In the Section 4, we describe the experimental user study procedure and the results are reported in Section 5. These results are analysed in Section 6. Finally, in Section 7, we present the conclusions and future work.

2. State of the Art

As we know it today, augmented reality almost exclusively addresses the visual channel, with some examples of auditory augmentation as well. However, as far back as 1962, the concept of multiple sensory modalities was proposed by Morton Heilig’s Sensorama [8]. This device is one of the first machines related to the multimodal virtual reality (VR), with stereoscopic 3D view, stereo sound, wind and smell effects and mobile seat.

In 2002, visual virtual reality was introduced in medicine [9] in order to train and, thus, improve surgeon’s skills. During the following years, a number of studies continued to focus on this area [10,11,12].

Head-mounted display (HMD) technology has advanced in these recent years, allowing the representation of complex 3D scenes rendered in real time and with a good image quality and frame ratio per second.

Oculus Rift [13], PlayStation VR or HTC Vive are examples of HMDs which allow a realistic immersive simulation of a virtual world.

Regarding augmented reality HMDs, Google presented in 2012 the advanced prototype “Google Glasses” in 2012, a device which had the aim of coexisting in the society. In 2015 Google decided not to sell their device commercially to the public and, in 2017, they presented a new release aimed for business and industrial environment.

Human–robot interaction (HRI) has made profilic use of augmented reality, as seen in recent review publications, such as in [14], which identifies a roadmap for the field. In the same way, in [15], a review of advanced robot programming approaches is presented where the authors explain the current augmented reality-based programming approaches.

Augmented reality and virtual reality fit perfectly in the manufacturing industry as support for operators. A good example is found in Makris et al. [16] for the programming of welding robots, or in [17] for industrial robots supported by virtual reality. Using different methodologies, algorithms and tools like in [18], it is possible to improve training, programming, maintenance and process monitoring. Defining fiducial markers in the 3D world [19] that are directly linked to the real robot close spots working with augmented reality or defining the path that the robot has to follow [20] could also simplify the difficulty of the task. Related to the safety requirements for human–robot collaboration in industrial settings, in [21], authors present a depth-sensor based model for workspace monitoring and an interactive AR user interface for safe human–robot collaboration.

Through these kinds of augmented reality devices, the virtual representation of a robot makes the safety factor easy to comply with. Moreover, due to the possibility of overlapping additional information over the real world, it is possible to improve time to complete the task using marks and labels [22]. In 2017 [23], Ni et al. developed an augmented reality system that mixes haptic and visual experiences for the programming of welding robots using a haptic PHANToM device. In this case, authors used a screen for visual augmented reality.

Puljiz et al. [24] presented a work on programming a collaborative robot using Microsoft Hololens mixed reality HMD with a KUKA KR-5 collaborative robot and ROS as an operating system. Luebbers et al. [25] presented a novel augmented reality interface for the visualization and directed control of robot skill learning from demonstration (LfD). Rosen et al. [26] proposed a mixed-reality HMD visualisation of the intended robot motion, allowing users also to adjust the goal pose of the end effector through hand gestures.

In addiction, [27,28,29] are examples of systems in which systems that use mixed reality HMD were used for the interactive programming of industrial robots.

A limitation common to all the prior work is the lack of haptic feedback that can increase the realism of the pick-and-place task. An exception is [23], where the autors did use a PHANToM haptic device, but the visual experience was limited to a screen. If the application does not contain predefined target points to perform the task, a naïve user would have difficulties to move the robot and have spatial perception. Moreover, we decided to use a mid-air haptic device as an alternative.

With the currently available mixed reality (MR) head-mounted displays, the capability of representing a hologram of a robot and its workspace allows creating a better context to perform a task [30]. With hand tracking technologies, the worker could perform the pick-and-place task in a similar way as with a physical robot. In the examples from the literature mentioned above, the movement is done in the air without grabbing any physical object, and consequently with a lack of tangibility, which could be addressed by using a haptic device, making the execution of the task potentially more intuitive and natural [31]. In the work presented here, the haptic device selected is an Ultrahaptics Stratos device, which is able to present tactile sensation on the skin in mid-air, without the intrusion of a physical interface.

The main goal of the present study was to combine the use of a haptic device with a MR virtual environment to measure the feasibility and the improvement introduced by the Ultrahaptics Stratos device.

3. Design of the Experimental User Study

The goal was to investigate the research questions mentioned above, regarding the programming of the robots through manual guidance of the holographic representation of its digital twin, in the sub-task of performing a pick-and-place operation in the context of a virtual scene and robot.

For this implementation, we used a Microsoft Hololens mixed reality device to provide visual augmented reality. This device consists of 2 transparent lenses in 16:9 aspect ratio, sensors including an inertial measurement unit, depth camera, environment understanding cameras and microphones among others, and input/output connections (Bluetooth, Wi-Fi, etc.) which allow communication between this head-mounted display with the haptic device. As mentioned, tactile feedback was generated using an Ultrahaptics Stratos Explore ultrasound actuator [32]. This option of mid-air rendering of tactile sensations was deemed more appropriate than alternatives such as haptic gloves, or an ‘AIREAL’ [33] air vortex emitter device. The Ultrahaptics Stratos device consists of 256 ultrasound transducers that are combined to create tangible sensations on specific points in space that the hand occupied at each moment in time. It presents the advantage that it does not require wearing any physical device on the hand, although it has the limitation that the interaction space is small (up to less than 70 cm above the actuating panel). This device integrates a Leap Motion hand tracker that permits the localization of the user’s hand within the interaction space. Through the position of the hand and the capacity to create local air disturbances, it is possible to create a range of sensations on the user’s skin. When the user places his/her hand in the interaction region, a holographic hand is superimposed with the real hand, which corresponds to the reconstruction of the user’s hand as it is tracked by the Leap Motion device.

Using these devices and the Unity cross-platform game engine, we created a 3D environment which simulated a collaborative robot that moved within the interaction space of the Ultrahaptics device.

The setup to run the application is shown in Figure 1. A PC managed the creation of the 3D environment and, with a script, it also collected all the data during the task execution. The Ultrahaptics Stratos Explore device was connected through a USB port to the same PC. The Microsoft Hololens HMD device was connected to the application through WiFi connection in order to render the 3D visual representation of the scenario.

Figure 2 shows an overview of the interactive scene. Through the visual holograms, the haptic device can be distinguished as a rectangular panel placed on a desk. Above the haptic device, the hologram of a 6 degrees of freedom robot is shown. A sphere is displayed as the wrist of the robot, which the used could grab and move by hand for the execution of the pick-and-place task. As to the end effector, the cobot had a two-finger gripper. For this experimental study, when the program was launched, the grasping tool was always pointing towards the target, which was the centre of the handle (marked in red) of a holographic tool box located on the left side of the Ultrahaptics Stratos device. In Figure 2, we can also see a linear slider (composed by a white bar and a small blue box which indicates the rotation angle), with which the user could rotate the gripper through the Z axis. This slider was not used during the study. The user could view this scene from any angle, obtaining any perspective that could aid with the task.

When the user’s hand entered the scene, a holographic version of his/her real hand appeared, superimposed with it. This was indication that the system was tracking the hand successfully. If the hand stopped being detected by the Leap Motion device, the holographic hand disappeared from the scene. Once in the scene, the hand could grab the spherical wrist of the robot. If grabbed successfully, the user felt a continuous tactile sensation on his/her hand, and the sphere (plus the whole kinematic chain of the robot) would follow the user’s hand as it displaced. If for any reason the sphere dropped from the hand, the tactile sensation disappeared, and the robot would not displace following the hand anymore. The tactile sensation served, thus, as a non-visual confirmation that the robot was grabbed by the hand. To voluntarily release the sphere, the user extended the fingers of the hand, opening it completely.

The interaction with the scene also includes voice commands in order to stop (stop robot) or resume (move robot) the mobility of the robot. When the user activated the ‘stop robot’ voice command, the user could not move it even if he/she tried to grab the end effector. In order to allow the robot movement again, the user had to activate the ‘move robot’ voice command. This was the mechanism used to carry out different repetitions of the pick-and-place task during the study. Each time the movement was activated, the target toolbox appeared in a different place in the scene, traversing three possible position in a triangle. Activating and deactivating the robot in this way would be used, during the study, for a participant to initiate and finish each execution of the experimental task. The decision to use voice commands was made because working with interactive menus with a narrow field of view would be more damaging than beneficial. Menus take up a lot of vision space that the current Hololens device version, with a narrow field of view, cannot fit in easily without occluding the scene and diverting the user’s attention from the main objective. With a ‘show help’ voice command, a list of all the available voice commands was displayed in text next to the robotic arm.

4. User Study

A group of 16 inexperienced participants volunteered to take part in the study. Most of the participants (12) were in the 18–30 age range, two participants in 31–50 age range and another two participants in the 51–65 age range. Before starting the study, all participants were notified that their performance data would be collected and that they should complete a questionnaire. They were informed that they could drop out at any point if they decided to do so, and that any data collected up to that point would be discarded. No instance of a drop out took place.

As we stated before, the objective of this study was to answer the research questions proposed:

• Is it possible to program a robotic arm through hand guidance using current mixed reality devices?
• What is the impact of the Ultrahaptics Stratos device in manual guidance through the application?

To investigate these questions, we designed a single factor, repeated measures user study in which the factor was the presence of tactile feedback from the Ultrahaptics Stratos device (i.e., in one condition the user would feel tactile feedback while grabbing the robot, but not in the other condition). Participants executed the pick-and-place task three times in each condition (more details about the task are provided below). The order of presentation of the conditions (interacting with tactile feedback in the first or in the second place) was fully counterbalance between the participants.

The research methodology employed included the measurement and assessment of a substantially larges set of variables that were used in the prior pilot study [7]. During the study, data were collected by two means: (i) validated and ad hoc questionnaires filled in by each participant at different points during the session, and (ii) observed (i.e., objective) data about the interaction logged by the computer system managing the session. The second category (logged objective observations) included time to complete the task, the number of times the user’s hand lost grasp of the sphere, the total amount of time spent moving the cobot, the distance error between the gripper and the target in the final positioning, the ratio between the time moving the cobot and the total amount of time spent on the task. Participants thus generated two kinds of results. The quantitative variables which had been collected during the users executions, and qualitative variables collected through the questionnaires.

Taking into account that the users performed the task in two conditions (with and without tactile feedback), and in order to answer the research questions, any differences between conditions found in the data from this controlled study would be attributed to the effect of the condition.

Statistically, we analysed quantitative data using t-tests [34] as well as the Wilcoxon test [35], depending on the normality of the data distribution (for which we used the Shapiro–Wilk test). All data analysis and production of figures were done using the R language [36]. The p-value used was 0.05.

The quantitative metrics mentioned could not capture differences in UX and other subjective aspects. For this reason, we also make use of research methods that participants would assess or respond to subjectively. These included ‘Raw NASA-TLX’ [37] and ‘SEQ’ [38] questionnaires. The Raw NASA-TLX [39] was extended with the categories “sense of control” and “physicality” (i.e., the in what degree it was perceived that the hologram grabbed with the hand was physical). We added these extensions because we wanted to analyse these two aspects explicitly. They were, of course, not added in the calculation of the TLX Task Load Index. Similarly, the SEQ questionnaire was also extended with an additional category, “satisfaction with the result obtained” on the same scale format. The preliminary pilot study [7] did not make use of any of these methodologies, but it did include a brief ad hoc questionnaire. For the sake of comparison of results between the preliminary pilot study and this full study, we did also include the same ad hoc questionnaire in this case.

Each participant started with filling in a demographic questionnaire that included prior experience with augmented/mixed/virtual reality. Whether or not they had experience with those technologies and related devices, all the participants underwent a 15 min training session where they learned how to interact with the devices and the 3D scene in order to perform the experimental task. Training included sampling different examples of haptic feedback from the Ultrahaptics device, as well as different kinds of interaction with Leap Motion and the elements in the scene. One of these interactions was moving a sphere. This task was the same one they completed later with the collaborative robot’s end effector but, in this case, the interaction was out of any context.

In the experimental part of the study, participants performed the hand guiding task with the holographic cobot. The goal of the task was to grab by the hand the sphere on the wrist of the robot and to bring it to the holding point on the handle of the tool box (marked in red), thus indicating to the robot where it should place its end effector in order to lift up the tool box (see in Figure 3). The lifting of the toolbox did not follow as part of the study. Participants were asked to trade-off between speed of execution and accuracy: they were instructed to carry out the task in as short an amount of time as possible, providing a positioning of the end effector that was as accurate as possible. However, there was no time limit for each execution. To execute this task, the participant said the voice command ‘move robot’, which started the task execution timer. The participant then moved his/her hand to the ball, grabbed it (curving down the five fingers on the hand, as if surrounding the sphere with them) and dragged it (keeping the hand in that same semi-closed position), until the end effector had been brought to the target position next to the tool box. The user could release the ball (by opening the hand) and grab it again repeatedly as desired, e.g., to achieve a best positioning. In the moment that the user thought that the tool was pinching correctly the tool case mark (i.e., when the participant was satisfied with the execution of the task), the user stopped dragging the end effector, released the sphere (by opening the hand) and indicated that the execution had ended with the “stop robot” voice command, which stopped the timer and logged the performance values corresponding to that execution.

Each participant made three executions of the task with tactile feedback from the Ultrahaptics Stratos haptic device and another three executions without any tactile feedback. As stated before, the order of the conditions was fully counterbalanced between participants. Half of the participants started with the haptic feedback enabled, while the other half started without any haptic feedback. After every execution, the participants completed the extended version of the single ease question (SEQ) that asked about the task difficulty in each execution and about satisfaction obtained with each execution. Both categories made use of the same 7-point Likert scale, so that providing first-impression responses (which was the aim of this post-task questionnaire) as easy as possible. After participants performed the three executions of a condition, they completed the post-condition raw NASA-TLX questionnaire [39] that rated, for that condition, the perceived workload in order to assess the task. In the same way as with the SEQ test, we extended this NASA-TLX with two additional categories: perceived control and physicality. These two values are not standard in the NASA-TLX, and do not contribute to the calculation of the index of this test. However, we included them because they might capture a difference between the conditions.

With the purpose of knowing more precisely the participants’ opinions, we also administered the same post-study ad hoc questionnaire as in the preliminary pilot study, with the same four questions on a 7-point Likert scale (see details about the questionnaire in the Appendix A of this paper).

5. Results

The results shown below contain the early work variables as well as the new measured variables. First we show the quantitative variables, which are distance error, time spent on the task, time moving the cobot, hologram movement time ratio and number of times the user lost the ball. Then, the results related to the questionnaires are shown (extended SEQ and extended RAW NASA TLX).

5.1. Quantitative Results

These key variables can measure objectively if there exists a real difference in the task performance. These variables are related to the distance, the time and the times that the user paused his/her activity.

5.1.1. Distance Error

The distance error refers to the distance measured between the target point, and the gripper tool center point (TCP) is shown in Figure 4. Both data sets appeared to be similarly distributed, with apparently small differences between distributions ((M_V+H = 2.706 mm, SD_V+H = 2.351 mm); (M_V = 2.259 mm, SD_V = 1.998 mm)).

The Wilcoxon test gave a p-value greater than 0.05 (0.4183), which means that there is no significant difference between the mean distance error in both cases.

5.1.2. Time Spent on the Task

We analysed 3 values related to the time needed to complete the task. The first one measures the full time spent on an execution since the user first grabbed the robotic hologram of the end effector of the robot, until the user released it.

When we checked the results through the box plots, we found an outlier value from User 11. This value corresponds to the first execution with tactile feedback and the first try of this user during the whole of his/her participation. The time value that we considered as an outlier is 307.2 s, the gap with the next higher value (69.5) is 237.7. Consequently, we also discarded the task execution value of that same participant value without tactile feedback. With 47 executions in total for each condition (instead of 48), we obtained the data obtained in Figure 5, left, showing similar distributions in both conditions ((M_V+H = 24.027, SD_V+H = 21.483); (M_V = 26.614, SD_V = 27.473)).

The Wilcoxon test returned a p-value (p = 0.962), which showed no statistically significant difference between conditions.

5.1.3. Time Moving the Cobot

The second time-related variable analysed is the total amount of time the user spent moving the hologram with the hand (i.e., discarding the time from the execution during which the hologram was not being moved). Like with the total amount of time spent on the task, we omitted the first executions of User 11 in each condition. In Figure 5, right, shows in a box plot the distribution of the data obtained. Also, in this case, the distributions from each condition were very similar ((M_V+H = 16.338, SD_V+H = 12.638); (M_V = 16.242, SD_V = 11.747)) and far from significant according to a Wilcoxon test (p = 0.987).

5.1.4. Hologram Movement Time Ratio

The last variable related to time that we analysed is the proportion between the time that the user was actually moving the robot and the total amount of time spent on the task. With this information we wanted to know if the percentage of time with the holographic cobot grew using the Ultrahaptics Stratos Explore device. The inspection of the data represented in Figure 6 suggested that the distributions were very similar ((M_V+H = 0.748, SD_V+H = 0.165); (M_V = 0.715, SD_V = 0.197)).

In this case, we used the t-test because of the normal distribution of both data sets. The obtained p-value is 0.578, suggesting there was no difference in this case either.

5.1.5. Lost Ball

We analysed the number of times the user’s hand lost grip of the ball that was fetched to drag the hologram (leaving the hologram behind immobile until it was fetched again). We got the results plotted in Figure 7.

Like with the previous parameters, the distributions of the data obtained were very similar in both conditions ((M_V+H = 6.681, SD_V+H = 7.553); (M_V = 7.319, SD_V = 8.577)), and no statistically significant difference was obtained from a Wilcoxon test (p = 0.698), suggesting that the presence of tactile feedback might not influence the number of times the hologram dropped off from the user’s hand.

Summarising all the above analysis, none of the observed quantified variables showed significant differences between conditions.

The following two sub-sections present subjective data captured and quantified with two standard questionnaire-based methods, extended SEQ questionnaires and the 210 extended NASA-TLX test.

5.2. Qualitative Results

Now, we are going to analyse the users’ opinions extracted from the extended SEQ test, the extended RAW NASA-TLX test and the customized questionnaire.

5.2.1. SEQ—Task Difficulty

The Single Ease Question is a test that consists of only one question about the difficulty of the task that the user has performed in the test. After each task execution, users register on a 7-point Likert scale how easy they found it to execute that task (three times for each test condition) where a low value means high difficulty and high value means a lower difficulty. Figure 8 shows the distribution of the answers provided. The Wilcoxon test (p = 0.147) found no perceived difference in difficulty that was statistically significant. The distribution of the obtained data shows the similarity ((M_V+H = 6.291, SD_V+H = 1.031); (M_V = 6.104, SD_V = 0.973)).

5.2.2. SEQ—Satisfaction with the Result

We extended the SEQ questionnaire with a second question that participants responded to in the same way (same Likert scale) and immediately after the first one. In this scale, participants rated how satisfied they were with the result they had obtained in that task execution, meaning the higher the value, the better the performance. Results are plotted in Figure 9. In this case, the data distribution obtained is [(M_V+H = 6.354, SD_V+H = 6.0); (M_V = 0.887, SD_V = 1.167)], and the Wilcoxon test returned a value (p = 0.033), corresponding to a statistically significant difference between series.

5.2.3. NASA-TLX

The index obtained from the “Raw NASA-TLX (Task Load Index)” questionnaire is a measure of the work load experienced by participants when executing the experimental task. The questionnaire consists of six dimensions, all of which are rated on a 21-point (0 to 20) Likert-like scale: mental demand, physical demand, temporal demand, performance level achieved, effort expended and frustration experienced. Keeping the same format, we added two dimensions to the questionnaire: sense of control and perceived physicality. The sense of control dimension has been used as part of an extended TLX questionnaire (in e.g., [40]), and is highly relevant in the task chosen in this study. We added also the perceived physicality [41] dimension to assess the subjective degree of realism that tactile feedback might be adding to the manipulation of the hologram. To calculate the index, we used the raw version of NASA-TLX [39], which does not make use of weighted pairwise comparisons.

Figure 10 shows the distribution of the Raw NASA-TLX values obtained from the 16 users, suggesting a lower TLX with tactile feedback ((M_V+H = 5.646, SD_V+H = 3.999); (M_V = 6.271, SD_V = 4.484)). However, after performing the t-test, we obtained a p-value greater than 0.05 (0.096), which means that no significant difference between the mean of both conditions was found from our data. Still, we went on to examine each dimension of TLX individually, to see which dimension(s) the difference in mean TLX value originated from.

Figure 11 plots the data distribution of the six constituent dimensions of TLX.

As a result, we found a highly significant statistical difference in the temporal demand (p-value < 0.01). Participants reported a lower temporal demand with tactile feedback from the Ultrahaptics Stratos explore device.

In addition, regarding the perception of control while carrying out the task, data from the experiment revealed ((M_V+H = 14.625, SD_V+H = 4.470); (M_V = 12.25, SD_V = 4.524)) a difference that was also statistically significant in their mean difference (p-value = 0.04), with perception of control being superior when the user performed the task with tactile feedback from the Ultrahaptics Stratos Explore device (Figure 12).

Similarly, perceived physicality increased when the users performed the task with haptic feedback as opposed to without it. This difference (p-value = 4.751 × 10⁴) can be seen in Figure 13 ((M_V+H = 15.437, SD_V+H = 6.375); (M_V = 3.521, SD_V = 4.843)).

5.2.4. Ad Hoc Questionnaire

Finally, we administered the same ad hoc questionnaire as in the preliminary study [7]. This questionnaire consisted of four statements that the users had to answer to on a Likert 7-point scale, indicating whether they agreed with them (3) or not (−3):

• Q1—With tactile feedback, I have perceived a certain advantage to carry out the task.
• Q2—I haven’t done complete the task faster when I felt the ball.
• Q3—I have achieved better accuracy with tactile feedback.
• Q4—The perception that I was handling the robot was the same with and without tactile feedback.

The distribution of the answers is shown in Figure 14.

Some consensus was shown for Q1 and Q4, with opinions regarding Q3 and Q4 distributed around the neutral region of the scale (with a rather even split of opinions). Thus, for Q1, 13 out of the 16 users perceived quite a clear advantage in carrying out the task with tactile feedback. As for Q4, participants largely supported the opinion that handling the hologram with or without tactile feedback felt different.

6. Discussion of the Results

With the results obtained, we can evaluate the impact of the presence of tactile feedback from an Ultrahaptics Stratos device on the task performance and on the user experience.

Regarding the objective quantitative metrics measured from the participants’ task executions (distance error, time spent on the task, net time grabbing the ball, ratio between both time values, number of times the participant lost the ball), we observed that the effect of introducing tactile feedback was small and, in fact, we found no statistically significant differences between conditions. An immediate conclusion that can be drawn from this is that the presence of the tactile feedback from an Ultrahaptics did not affect performance on the task of the experiment, either positively or negatively.

In contrast, with the analysis of quantified subjective data obtained through the various questionnaires administered, some results emerged that showed positive effects from the presence of tactile feedback, with no negative effects detected. Starting with the extended SEQ questions (ease of task execution and satisfaction with result obtained), mean values of the distributions suggested positive effects from the presence of feedback, although those differences were not found to be statistically significant. A similar trend was found for the TLX index and for its six constituent dimensions, where, in all cases, mean values of response distributions were numerically lower with tactile feedback, suggesting positive effects from its presence. Statistical comparison of distributions showed that the difference was significant in the case of temporal demand, where it was reported to be lower (with a high level of statistical significance) when tactile feedback was present. This same trend showing a positive effect was also present in the categories that extended the NASA-TLX questionnaire (not included in the calculation of the TLX index): sense of control and perceived physicality. In both cases, not only were distribution means higher in the condition with tactile feedback, but the differences were statistically significant in both cases, with a particularly high level of significance in the case of the perceived physicality. Further confirmation of the positive impact from the presence of tactile feedback was found in the data collected from the ad hoc questionnaire (the four statements that participants could agree or disagree with). Regarding the two questions gathering most consensus in the responses, Q4 showed that a clear difference was perceived when handing the hologram with or without tactile feedback and, according to the broad consensus around Q1, the presence of tactile feedback offered an advantage for the execution of the experimental task.

We hypothesize that the positive impact of having tactile feedback (reflected in the data from several of the metrics) might have been due to the effectiveness of feedback as a mechanism to confirm to the participant that he/she was grasping the hologram, as long as the feedback was felt. If such grasp got lost and the hologram was left behind in the process of dragging it, participants could notice immediately the change in sensation on the palm of their hand and react quickly to fetch the hologram again and resume the task. This could account for the significant differences found in lowering temporal demand, improving the sense of control, and providing an enhanced perception of physicality of the hologram. This interpretation of the results was reinforced by discussions with the participants about their experience during the study, who provided comments stating that it was helpful to receive tactile feedback, mostly because they could know better if they were holding the robot at each moment or not. Based on the data presented above, our interpretation is that the presence of tactile feedback was fulfilling the expectation of participants to be feeling in their hands the (virtual) object that they were holding. Tactile feedback increased the naturalness of that experience, and the sensation felt was reassuringly familiar, paving the way for an interaction that did not need to be learned.

7. Conclusions and Future Work

In response to the first research question in the introduction, this paper presents the implementation of a functional demonstrator based on a Hololens head mounted display, with which a user can manipulate (move) a holographic robot, for the execution of a pick and place task, achieving positioning precision that remains under 3 mm. This functionality is a building block of interactions in more complex programming-by-demonstration scenario. The second research question (impact of tactile feedback from mid-air haptics actuators on a pick and place task) motivated the main contribution of this paper. The paper reports a user study in which one such task was performed by participants in two conditions, with and without tactile feedback. As discussed above, the results obtained suggest that feedback does not affect observed performance (no significant numerical differences were recorded, although the trend was for better scores obtained with tactile feedback).

As for the qualitative results, subjective scores from participants on a set of questionnaire-based methods employed supported also the trend that tactile feedback was noticeable and with an impact that was positive. The strongest evidence of that was found on the reduction of temporal demand, improved sense of control, enhanced perceived physicality of the hologram, plus the consensus among participants that the noticeable effect introduced by tactile feedback gave an advantage in the successful execution of a pick-and-place task of a holographic robot arm.

Building on the results obtained in this paper, our next steps are focused on implementing a programming-by-demonstration method and scenario, based on the handing of a holographic robot arm. We will reproduce a realistic context in which a real robot can reproduce the procedure demonstrated to the hologram. Alongside this process, we will investigate further haptic actuation techniques and multimodal interaction that can lead to improved performance and user experience, as well as individual the impact of each device used in the interaction task.