Design of Control Elements in Virtual Reality—Investigation of Factors Influencing Operating Efficiency, User Experience, Presence, and Workload

Abstract

Virtual reality (VR) enables prototypes of devices to be evaluated in a simulation of a future usage environment. A disadvantage is the insufficient feedback design during the interaction with prototypes. In this study, we investigated how virtual control elements in VR must be designed to compensate for the lack of feedback. Therefore, 35 participants performed VR tasks using a virtual joystick and virtual rotary control. According to the design of experiments method, 12 factors, such as haptic feedback, sensitivity, size, and shape, were systematically varied. The control accuracy, task time, user experience, presence, and mental workload were recorded. The effect of a factor on the recorded parameters was examined using multifactorial ANOVA. Linear regression was used to calculate the mathematical models between the factors and parameters. These models were used to calculate the optimal design of the control elements in the VR. For rotary control, eight factors had a significant influence on the recorded parameters. There were seven factors for the joystick. With mathematical models, optimized control element designs for VR were calculated for the first time. These findings can help to better adapt prototypes and human–machine interfaces to different modalities in VR.

1. Introduction

Touch screens, rotary controls, and joysticks are part of modern human–machine interfaces (HMIs) in industrial contexts. They are often integrated into complex HMIs and used to control cranes, drilling rigs, agricultural machines, or medical devices [1,2,3,4,5,6]. The development of complex HMIs is ideally conducted according to a user-centered design process [7]. The focus of this process is on the user requirements. Based on these requirements, prototypes are developed, tested by the users, optimized, and retested [8]. Several prototypes must be developed for this iterative process. However, the production of prototypes is expensive and time-consuming [9]. Companies are increasingly using virtual prototypes to reduce the number of prototypes that need to be manufactured [10,11].

Virtual reality (VR) technology offers the possibility of visualizing and experiencing virtual prototypes in high detail. Users can test and evaluate prototypes by simulating a later usage environment [12]. The first studies showed that the evaluation of prototypes in VR is comparable to the evaluation of real prototypes [13,14,15,16,17,18,19]. In the industrial context, however, completely virtual prototypes are hardly used for user tests because the lack of haptic feedback limits the evaluation, in particular [19,20]. Therefore, virtual prototypes are often evaluated in mixed-reality environments [21,22,23,24]. In this case, the virtual environment is mixed with real control elements.

However, completely virtual prototypes are useful, particularly in the early stages of development. Different HMI concepts can be evaluated without the need to manufacture real control elements. In addition, test participants can evaluate new prototypes independent of their location. Many studies and textbooks exist on the design of real control elements to achieve high control accuracy and user satisfaction [25,26,27,28,29,30,31,32,33,34,35,36,37,38,39].

Table 1. Recommendations for the design of rotary controls.

Parameter	Recommended Value Anguelov [36]	Recommended Value Schmidtke [35]
Diameter	40–50 mm	≥30 mm
Number of detent pos.	25–35	8–24
Torque	30–40 mNm	200–700 mNm

shows, for example, the recommended value ranges for the optimal haptic value impression of rotary controls according to Anguelov [40], as well as recommendations for safe operation according to Schmidtke and Jastrzebska-Fraczek [30].

The table shows that an important parameter in the design of rotary controls is the torque that the user must apply to switch from one position to another. This torque cannot be simulated using VR. Consequently, the user lacks an important feedback component. This may lead to reductions in positioning accuracy and speed, which can decrease performance and user acceptance. Force feedback is also an important feedback source for joysticks. The user feels resistance when moving the joystick and can therefore better control the deflection [31,41,42,43].

To adapt HMIs in VR to different modalities, customized interaction types are often used, such as laser pointers, additional virtual menus, or input devices such as tablets [44,45,46,47,48,49,50]. Many studies have focused on the optimal human–computer interaction for specific applications in VR, for example, games [51], desktop navigation [52], or learning tasks [53]. However, to evaluate new prototypes in VR, the control elements and the way of interaction should match the later reality as closely as possible. Hardware solutions, such as data gloves and haptic devices, that are intended to enable natural interaction in VR are still very expensive and require extensive setup time [54]. To be able to interact with VR content as naturally as possible, studies have already been conducted on realistic grasping in VR [55,56]. However, there is a lack of studies investigating how control elements (joysticks, rotary controls, etc.) can be adapted to VR so that they can be operated well by users. In addition, there is a lack of studies investigating the influence of different control element parameters such as size, shape or type of feedback on subjective parameters such as user experience or workload. Therefore, in this study, we investigated how rotary controls and joysticks must be designed in VR to perform positioning tasks as quickly and precisely as possible. In addition, we will investigate parameters that influence user experience, presence, and mental workload, and how the control elements can be optimized with regard to these criteria.

2. Materials and Methods

Figure 1 shows the methodological approach in this study. The individual steps are explained in the following sections.

2.1. Experimental Setup and Design

2.1.1. Identification of Relevant Control Element Parameters

To investigate how the control elements can be adapted to VR, a workshop with drillers and experts in usability and VR identified factors that could influence positioning accuracy, time required for positioning tasks, user experience, and workload. Drillers control the drilling process on offshore rigs and are experts in operating joysticks and rotary controls.

Table 2. Identified factors that could influence the operation and subjective perception of users.

	Factor	−	+
Rotary control	Angular resolution	5°/Value	45°/Value
	Diameter	20 mm	110 mm
	Inclination	0°	90°
	Shape	Knurling/Toggle/Cylinder
Joystick	Max. angle of deflection	10°	45°
	Max. positioning speed	2.5%/s	12%/s
	Size (vertical/horizontal)	15/7 cm	32/17 cm
	Angular resolution	3-Stage	linear
	Visual feedback	Yes	No
	Shape	Vertical	Horizontal
Rotary control and joystick	Haptic feedback	Yes	No
	Acoustic feedback	Yes	No
	Hand visualization	Yes	No
	Position of the participant	Sitting	Standing
	VR environment	Table	Crane
	Input device used	HTC Vive	Valve Index

lists the identified factors that were examined in this study. In addition, the experts defined value ranges in which the factors vary.

The angular resolution factor describes the sensitivity of the rotary control. With an angular resolution of 5°/value (−), the rotary control must be turned by 5° such that the value displayed on the screen changes by 1. The inclination of the control element describes the angle between the control element and table (refer to Figure 2: inclination 0°). Rotary controls with knurling, without knurling (cylinder), and with toggles were tested. In addition, the diameter of the control elements was varied.

The maximum deflection angle was varied to examine the joystick. For example, the joystick could be deflected forward or backward by a maximum of 45° (+) from the zero position (perpendicular to the table). The factor max. positioning speed describes the maximum possible rate of change in the maximum deflection of the joystick. The factor angular resolution describes the ratio of the deflection angle to the change in the displayed value of the bar graph. With a linear angular resolution (+), the change in the positioning value is proportional to the deflection angle. With a three-stage angular resolution (−), three angular ranges are defined, in which the change in the positioning value is the same.

A joystick with a vertical handle (cf. Figure 2. left) and a joystick with a horizontal handle, such as that used as an aircraft thrust lever, were tested. The size of the joystick was tested on three levels. In addition to the extreme values shown in Table 2, a medium size was tested (vertical: 23.5 cm; horizontal: 12 cm). In addition, the visual feedback was varied. Either the test person saw only a bar graph with the current positioning value (refer to Figure 2, top right) or a bar graph with the current rate of change in the positioning value (refer to Figure 2, top left).

For both control elements, we investigated whether haptic or acoustic feedback as well as the visualization of the hand influenced the operation of the control elements. For example, if the participant activated the joystick, the hand grasping the joystick was visualized in VR. If there was no visualization, the participant saw only the joystick and its deflection. If the value changed, the participant heard a click and/or felt the vibration of the controller.

It was investigated whether the design of the VR environment influenced the operation and especially the subjective evaluation of the control elements. In addition, the influence of the design of the VR controller on the operation was investigated. Valve Index controllers (+) have integrated sensors to detect hand and finger positions. This enables the control elements to be “grasped” similarly to reality. For HTC Vive (−) controllers, interaction is performed by moving the controller close to the control element and pressing the “trigger button” on its back. Images of the controllers used can be found in the Supplementary Material (refer to Figure S1).

The various control elements and the virtual environments were created in the Unity 2020.2.3f1 development environment (Unity Technologies, San Francisco, CA, USA) and coded in C# (Microsoft Corporation, Redmond, WA, USA). Visualization was performed using the Valve Index HMD (Valve Corporation, Bellevue, WA, USA) and a PC with an i7 processor and a GeForce GTX 1070 graphics card (NVIDIA Inc., Santa Clara, CA, USA). Figure 2 shows examples of the different factor characteristics listed in Table 2.

2.1.2. Design of Experiments

To investigate whether the factors listed in Table 2 influenced the positioning accuracy, time required for positioning tasks, user experience, presence, and workload, the different levels of the factors (e.g., 40 mm or 110 mm diameter) were varied and tested. Owing to the high number of factors, a classical experimental design where only one factor is varied (e.g., the diameter) and the other factors are kept constant would have led to an enormous experimental effort. Therefore, we used the design of experiments (DOE) method in this study. The DOE is a standardized methodology for the planning and statistical evaluation of experiments [57]. This methodology aims to learn as much as possible about the relationships between influencing factors and results with as little experimental effort as possible. The information obtained is statistically confirmed, and the effects of the input variables on the output are quantifiable [58]. In the DOE, several factors are varied simultaneously. The factors are varied according to a specific system (test design) such that the influence of a single factor can be statistically calculated [59].

In this study, a fractional factorial I-optimal response surface test design with 35 trials (joystick) and 32 trials (rotary control) was created using Design Expert 13 software (Stat-Ease, Minneapolis, MN, USA). In I-optimal designs, the algorithm selects points that minimize the integral of the prediction variance in the entire design space [60,61].

Table 3. The first three runs of the test design for the rotary control.

Run	A: Angular Resolution (°/s)	B: Diameter (mm)	C: Inclination (°)	D: Haptic Feedback	…
1	10	40	45	No	…
2	45	110	0	Yes	…
3	45	40	45	No	…
…	…	…	…	…	…

lists an excerpt from the test design for the rotary control as an example. The levels of the factors listed in Table 2 were varied systematically. The complete test designs can be found in the Supplementary Material (cf. Tables S1 and S2).

2.2. Experimental Procedure

To examine the influence of the factors, test participants (n = 35; f: 14; m: 21; age: 24.7 ± 3.6 years) evaluated the different joysticks and rotary controls according to the test design in VR. Each test participant tested only one configuration of a control element. This means that the first test participant tested the configuration of run one and the second test participant tested the configuration of run two and so on (cf. Table 3). Each participant tested both a rotary control and a joystick in randomized order.

Of the 35 participants, 10 had no VR experience, 19 had little experience (less than 10 h), and 6 participants had more than 10 h of VR experience. The test participants had no previous work experience in the operation of control elements. All participants were enrolled in a bachelor’s or master’s degree program with a technical focus at the time of the study and stated that they used computers on a daily basis.

In VR, the participants viewed two screens. One screen showed a vertical bar graph on a scale from 0% to 100% when the joystick was examined. If the participant pulled the joystick towards themselves, the displayed value increased. When examining the rotary control, a screen showed a numerical display with a value between 0% and 100%. Clockwise rotation increased this value. Figure 3 shows the experimental setup used in VR.

The test participants sat on a height-adjustable chair with castors and were requested to adjust the seat height at the beginning of the test. This meant that the test participants did not have to adopt a predefined posture and could position themselves freely in relation to the virtual control elements.

At the beginning of each experiment, the test leader introduced the test participants to the respective VR controllers in a standardized manner. The test participants were shown how to grip and operate the control elements. Subsequently, the test participants were requested to set the maximum value of the scale with the control element.

After the introduction, the participants performed four tasks per control element (

Table 4. Tasks of the user tests.

Task	Rotary Control	Joystick
1	Set the value from 0 to 12	Set the value from 0 to 42
2	Set the value from 12 to 11	Set the value from 42 to 41
3	Set the value from 11 to 31	Set the value from 41 to 8
4	Set the value from 31 to 10, then to 15, then to 8	Set the value from 8 to 98, then to 40, then to 50

). The tasks were displayed on a separate screen in VR and performed sequentially by the test participants. Each participant performed all the tasks three times.

2.3. Evaluation Parameters

2.3.1. Time on Task

The time required to complete the task was measured. The measurement began when the test participant gripped the control element in the VR and ended when the required value was set.

2.3.2. Positioning Accuracy

The positioning accuracy was determined by counting the positioning errors and adding them. For example, if the test participant turned the rotary control in Task 1 and reached a value of 14 instead of 12, then returned to 11, and then to the required value of 12, three errors were noted.

2.3.3. Presence

Presence describes the feeling of being in a virtual world [62,63]. After completing the tasks, the test participants evaluated the control element used and the user test with regard to the perceived presence using the Slater–Usoh-Steed–Questionnaire (SUSQ) in a German-adapted version [64,65].

2.3.4. User Experience

User experience was measured using the User Experience Questionnaire (UEQ) [66]. The UEQ consists of 26 dual-pole items divided into the following six dimensions:

• Attractiveness: describes the general impression of the product.
• Perspicuity: describes the user’s feeling that interaction with a product is easy, predictable, and controllable.
• Efficiency: describes how quickly and efficiently the user can use the product.
• Dependability: describes the feeling of being in control of the system.
• Stimulation: describes the user’s interest and enthusiasm for the product.
• Novelty: describes whether product design is perceived as innovative or creative.

Participants rated the items using a seven-point Likert scale. Each box on the Likert scale was assigned a score between −3 and +3, where +3 corresponded to an adjective with positive connotation. The UEQ score of a dimension is the mean of its respective scores [67].

2.3.5. Perceived Workload

The NASA RAW-TLX by Hart and Staveland [68] consists of six items representing the dimensions of mental, physical, and temporal demands, as well as performance, effort, and frustration, on a 20-point scale. A German translation was used in this study [69].

2.4. Data Analysis

2.4.1. Identification of Significant Factors Influencing the Evaluation Parameters

The time required to complete the tasks and number of positioning errors per task were averaged over the three trials of the test participants. By means of multifactorial ANOVA (α = 0.05), it was analyzed for each factor (e.g., diameter of rotary control) whether the change in the factor (e.g., from 40 mm to 110 mm diameter) had a significant influence on an evaluation parameter (e.g., time on task). If a factor had a significant influence on an evaluation parameter, the mathematical relationship between the factor and evaluation parameter was calculated via linear regression. These equations were represented in a coded form. This means that the equations did not use numerical factor values, but −1 for the lower level and +1 for the upper level. In this representation, the relative effect of a factor can be determined by comparing factor coefficients. Statistical analyses and calculation of the mathematical models were conducted using Design Expert 13 software. The Cook’s distance, which is a measure of how much the entire regression function changes when the values of a test run are not considered, was calculated to verify the accuracy of the mathematical models [70]. Large differences between Cook’s distances indicate outliers. The Cook’s distances above a value of one are considered critical and can falsify the model.

2.4.2. Optimization of the Control Elements for VR Using Mathematical Models

Numerical optimization calculations were performed for each control element, whose factor values could be used to achieve a high level of control accuracy, user experience, and presence with a low level of time expenditure and mental workload. These calculations were performed using ramp functions. This transformation makes the evaluation parameters dimensionless, and outliers cannot distort the overall result [59]. Each ramp can only take values between 0 and 1. A value of 0 corresponds to a poor result, and a value of 1 corresponds to a very good result. The minimum and maximum observed values of the respective evaluation parameters were used as the corner points of the x-axis.

The ramp function results from the case distinction. Let y be the value of an evaluation parameter, and y₁ and y₂ be the respective corner points. The value q of the ramp function is calculated as follows:

$$q=\left\{\begin{array}{c} 0 \forall y\le y_1 \\ \frac{y-y_1}{y_2-y_1} \forall y_1<y<y_2 \\ 1 \forall y\ge y_2 \end{array}\right.$$

(1)

Subsequently, the desirability function D(X) was calculated [71]. Therefore, the function values (q) of all ramps were multiplied as follows:

$$D={(q_1{\times q}_2\times \dots {\times q}_n)}^{\frac{1}{n}}={\left(\prod _{i=1}^{n_q}{q}_i\right)}^{\frac{1}{n_q}}$$

(2)

In this case, n denotes the number of evaluation parameters. If an evaluation parameter or factor is outside the desired range, then the entire function becomes zero. Subsequently, the desirability function was optimized using simplex algorithms [72], and it was calculated with the factor values with which the highest possible control accuracy and user experience could be achieved [59].

2.4.3. Validation of the Optimized Control Elements

To investigate whether the mathematically determined control element parameters led to high control accuracy, user experience, and presence, as well as low time consumption and mental workload, confirmation runs were conducted with the optimized control elements. Therefore, four additional test participants (m: 4; age: 26.8 ± 4.5 years) performed the procedure described in Section 2.2 with the joystick and rotary control optimized for VR. Subsequently, the evaluation parameters were determined. These parameters were then compared with the calculated parameters. For example, the mathematical models calculate that a time on task of 20 ± 10 s should be possible with the optimized design of the rotary control. This mathematical model could be confirmed if the value determined in the confirmation runs is within the 95% prediction interval (95% PI).

3. Results

3.1. Rotary Control

3.1.1. Identification of Significant Factors Influencing the Evaluation Parameters

At least one factor had a significant effect on the evaluation parameters time on task (ToT), positioning accuracy (Acc.), presence (SUSQ), workload (NASA TLX), perspicuity (UEQ_P), efficiency (UEQ_E), and dependability (UEQ_P) of the User Experience Questionnaire (p < 0.05).

Table 5. Factors that had a significant effect (p < 0.05) on the evaluation parameters when operating the rotary control. If a factor had a significant influence on an evaluation parameter, the significance value was entered. If the factor has no significant influence, “---” was entered.

Factor	ToT	Acc	SUSQ	NASA TLX	UEQP	UEQE	UEQD
A	<0.001	0.003	---	<0.001	---	<0.001	0.012
B	---	---	---	0.038	---	<0.001	---
C	<0.001	---	---	---	---	---	---
D	0.020	---	---	---	---	---	---
E	---	---	---	---	---	---	---
F	---	---	---	---	---	0.004	---
G	---	---	---	0.004	---	---	---
H	---	---	---	---	---	---	---
J	---	0.017	0.006	---	---	---	---
K	---	---	---	---	0.002	---	0.019
AC	<0.001	---	---	---	---	---	---
AG	---	---	---	---	---	---	0.040
BE	---	---	0.006	0.002	<0.001	---	---
EF	---	---	---	---	---	0.006	---
EH	---	---	---	.003	---	---	---
HK	---	---	---	---	---	<0.001	---

A: angular resolution; B: diameter; C: inclination; D: haptic feedback, E: acoustic feedback; F: visualized hand; G: shape; H: position participant; J: input device; K: VR environment.

shows a comparison of the factors and evaluation parameters as well as the significance values calculated using variance analysis. If no significance value was entered, this factor had no significant effect (p ≥ 0.05) on the respective evaluation parameter. The factors AC, AG, etc., are interaction effects.

Figure 4 shows the effect of the factor angular resolution on the evaluation parameters time on task (ToT) and positioning accuracy (Acc.). The figure shows that the participants with an angular resolution of 5° needed on average 54.4 s less to complete the tasks than the participants with a resolution of 45°. The angular resolution had the opposite effect on positioning accuracy. The test participants with an angular resolution of 5° made more errors.

Table 6. Mathematical models of the rotary control with coded factors. The italicized factors alone had no significant influence on the respective evaluation parameters. They were included in the model because the interaction with this factor had a significant influence on the evaluation parameter.

Mathematical Models	R2Adj	SNR
ToT = 45.0 + 27.2A + 12.8C + 4.8D + 15.0AC	0.84	19.70
Acc = 3.0 − 1.7A + 0.9J	0.30	6.83
SUSQ = 4.4 − 0.3B − 0.1E − 0.7J − 1.1BE	0.28	6.96
NASA = 4.09 + 2.5A + 0.9B − 0.1E + 1.3G1 + 0.2G2 − 0.4H − 1.5BE − 1.2EH	0.75	12.85
UEQP = 2.5 − 0.07B − 0.01E − 0.3K + 0.5BE	0.38	7.98
UEQE = 1.5 − 0.8A − 0.7B − 0.04E + 0.3F + 0.08H − 0.1K − 0.3EF − 0.5HK	0.71	12.20
UEQD = 1.7 − 0.5A − 0.4G1 + 0.04G2 − 0.4K − 0.7AG1 + 0.3AG2	0.34	9.90

A: Angular resolution; B: Diameter; C: Inclination; D: Haptic feedback, E: Acoustic feedback; F: Visualization hand; G: Shape; H: Position participant; J: Input device; K: VR environment.

presents the mathematical models used to calculate the evaluation parameters. In addition, the adjusted coefficient of determination (R²_Adj) and signal-to-noise ratio (SNR) are presented. The critical Cook’s distance was not exceeded in any of the tests.

By doubling a factor coefficient, the effect of the level change (e.g., 5° angular resolution to 45°) on the respective evaluation parameter could be determined in the respective unit. Figure 4 shows that by changing the angular resolution (A) from the low level (5°) to the high level (45°), the test participants needed on average 54.4 s longer to complete the tasks. This difference can also be seen in the equation “ToT” in Table 6. By doubling the factor coefficient 27.2, the difference of 54.4 s could be determined. This equation thus also shows that the test participants without haptic feedback (D) took on average 9.6 s longer to complete the tasks.

In addition to the angular resolution factor, the factor input device also significantly influenced the positioning accuracy. Test participants who used the Valve Index controller made about two more errors than test participants who used the HTC Vive controller. The input device factor also has a significant influence on the perceived presence. Users of the HTC Vive controllers rated presence 1.4 points higher than users of the Valve Index controllers.

The factor angular resolution had the strongest influence on mental workload. Averaged over all test runs, the mental workload was rated significantly lower at an angular resolution of 5° (NASA = 2.0) than at an angular resolution of 45° (NASA = 6.9). The factor angular resolution also significantly influenced the dimensions efficiency and perspicuity of the UEQ. Test participants rated the efficiency dimension significantly better (UEQ_E = 2.3) when they used rotary control with an angular resolution of 5°. Rotary control with an angular resolution of 45° achieved a mean UEQ_E score of 0.7. The test participants also rated the user experience better when the rotary control was tested in the detailed VR environment (crane). The visualization of the hand in the VR also had a significant influence on the user experience. The test participants rated the dimension perspicuity 0.6 points higher when no hand was shown in the VR.

3.1.2. Optimization of the Control Elements for VR Using Mathematical Models

The final numerical optimization calculation resulted in the following configuration for the rotary control:

A 

Angular resolution: 10–15°/value;

B 

Diameter: 33–45 mm;

C 

Inclination: 0°;

D 

Haptic feedback: yes;

E 

Acoustic feedback: yes;

F 

Visualized hand: no;

G 

Shape: knurling;

H 

Position participant: sitting;

J 

Input device: HTC-controller;

K 

VR environment: crane.

With these factor values, based on the experiments, it was computationally possible to achieve low time on task, minimum positioning errors, high user experience and presence, and low workload. Figure 5 shows an image of the optimized rotary control system for VR.

3.1.3. Validation of the Optimized Control Elements

The optimized rotary control shown in Figure 5 was then tested by four additional test participants in confirmation runs.

Table 7. Results of confirmation runs for the rotary control. Predicted parameter results vs. measured results of the validation runs.

Parameter	Pred. Mean	Std Dev	95% PI Low	Data Mean	95% PI High
Time on Task	20 s	10 s	7 s	18 s	34 s
Pos. Accuracy	3.13	1.99	0.80	1.28	5.47
SUSQ	4.90	1.21	3.39	4.50	6.42
NASA	0.00	1.50	−3.08	2.63	2.12
UEQP	2.96	0.37	2.49	2.06	3.43
UEQE	2.43	0.46	1.72	1.88	3.15
UEQD	2.42	0.72	1.46	2.13	3.38

show the results of the confirmation runs for the rotary control. The “Pred. Mean” column lists the values that should be mathematically possible with the rotary control optimized for VR. If the measured values of a parameter (data mean) deviate significantly from the predicted value and thus lie outside the 95% prediction interval, the value is marked red.

Table 7 shows that the measured values of the confirmation runs for the parameter NASA (mental workload, NASA-RAW-TLX) and the dimension perspicuity of the UEQ were outside the 95% PI interval. Consequently, the mathematical models for these parameters could not be confirmed. All other mathematical models were confirmed.

3.2. Joystick

3.2.1. Identification of Significant Factors Influencing the Evaluation Parameters

Table 8. Factors that had a significant effect (p < 0.05) on the evaluation parameters when operating the joystick. If a factor had a significant influence on an evaluation parameter, the significance value was entered. If the factor has no significant influence, “---” was entered.

F	ToT	Acc	SUSQ	UEQP	UEQE	UEQS	UEQN
A	<0.001	<0.001	---	---	<0.001	0.016	---
B	---	---	---	---	---	---	---
C	<0.001	---	---	---	---	---	---
D	0.014	0.010	---	---	---	0.028	---
E	---	---	---	0.014	---	---	---
F	---	---	---	---	---	---	---
G	---	---	---	---	---	---	0.031
H	---	---	---	---	---	---	---
J	---	---	---	---	---	---	---
K	---	---	0.008	---	0.039	---	---
L	---	---	---	---	---	---	---
M	---	---	---	---	0.007	0.004	<0.001
EK	---	0.003	---	---	---	---	---
AG	---	---	<0.001	---	---	---	---
FL	---	---	---	---	---	---	0.002
BC	---	---	---	---	0.009	---	---

A: positioning speed; B: deflection angle; C: size; D: position participant; E: shape; F: VR environment; G: angular resolution; H: input device; J: haptic feedback; K: acoustic feedback; L: visual feedback; M: visualized hand.

shows a comparison of the factors and evaluation parameters, including the significance values calculated by ANOVA for the joystick analyses. At least one factor had a significant influence (p < 0.05) on the evaluation parameters time on task (ToT) and positioning accuracy (Acc.), presence (SUSQ), perspicuity (UEQ_P), efficiency (UEQ_E), stimulation (UEQ_S), and novelty (UEQ_N) of the User Experience Questionnaire.

Figure 6 shows the effect of the factor positioning speed on the evaluation parameters time on task (ToT) and positioning accuracy (Acc.). The figure shows that the test participants with a high rate of change needed, on average, 64 s less to complete the tasks than the test participants with a low rate of change. A higher change rate had the opposite effect on positioning accuracy. The higher the positioning speed, the more errors the test participants made.

Table 9. Mathematical models of the joystick with coded factors. The italicized factors alone had no significant influence on the respective evaluation parameters. They were included in the model because the interaction with this factor had a significant influence on the evaluation parameter.

Mathematical Models	R2Adj	SNR
ToT = 82.3 − 31.9A + 3.3C1 − 15.7C2 + 6.8D	0.76	15.20
Acc = 25.3 + 13.4A + 5.1D + 1.8E − 0.2K − 6.1EK	0.52	15.20
SUSQ = 4.1 + 0.02A + 0.35G1 − 0.35G2 − 0.6K + 1.5AG1 −1.3AG2	0.34	6.61
UEQE = 1.2 + 0.7A + 0.1B − 0.02C1 + 0.1C2 − 0.2K − 0.3M − 0.1BC1 − 0.8BC2	0.49	9.76
UEQS = 1.2 + 0.5A − 0.3D − 0.4M	0.33	9.10
UEQN = 1.2 + 0.2F + 0.4G1 − 0.5G2 + 0.2L − 0.5M + 0.5FL	0.38	7.40

A: positioning speed; B: deflection angle; C: size; D: position participant; E: shape; F: VR environment; G: angular resolution; K: acoustic feedback; L: visual feedback; M: visualized hand.

lists the mathematical models for the calculation of the evaluation parameters, including the adjusted coefficient of determination (R²_Adj) and SNR for the examination of the joystick. The critical Cook’s distance was not exceeded in any of the tests.

With the medium joystick size, the test participants needed the least amount of time to complete the tasks (ToT_medium = 59.8 s). With the small and large versions, the test participants needed more time (ToT_small = 78.8 s; ToT_large = 87.6 s). Test participants who operated the joystick while standing needed on average 13.6 s more time and made about 10 more errors than test participants who were seated. With acoustic feedback, the test participants perceived a higher level of presence (SUSQ_{no acc feedback} = 3.9) than without it (SUSQ_{acc feedback} = 5.0). The interaction of the factors positioning speed (A) and angular resolution (G) also had a strong influence on the presence. Test participants who tested the joystick with a high positioning speed and a linear angular resolution perceived a higher level of presence (SUSQ = 6.5) than participants who tested high positioning speed with a three-step angular resolution (SUSQ = 3.0).

The equations show that the factor positioning speed (A) had the strongest influence on the user experience. Test participants rated the dimensions efficiency and stimulation of the UEQ significantly better when the joystick had a high positioning speed (12%/s).

3.2.2. Optimization of the Control Elements for VR Using Mathematical Models

The final numerical optimization calculation resulted in the following values for the significant factors of the joystick:

A

Positioning speed: 11%/s;

B

Deflection angle: ±10–17°;

C

Size (length/diameter): 240/43 mm;

D

Position participant: sitting;

E

Shape: vertical handle;

F

VR environment: crane;

G

Angular resolution: linear;

K

Acoustic feedback: yes;

L

Visual feedback: no;

M

Visualized hand: yes.

Factors H (input device) and J (haptic feedback) had no significant effect on the evaluated parameters, either individually or in interaction with other factors (p > 0.05). Figure 7 shows an image of the optimized joystick.

3.2.3. Validation of the Optimized Control Elements

Table 10. Results of confirmation runs for the joystick. Predicted parameter results vs. measured results of the validation runs.

Parameter	Pred. Mean	Std Dev	95% PI Low	Data Mean	95% PI High
Time on Task	34.65	14,.3	15.46	23.10	53.81
Pos. Accuracy	23	10	9	10	37
UEQP	1.97	1.20	0.60	2.25	3.35
UEQE	2.91	0.65	1.94	2.06	3.88
UEQS	2.29	0.76	1.32	1.13	3.27
UEQN	3.00	0.71	1.75	1.31	4.29

show the results of the confirmation runs for the joystick. If the measured values of a parameter (Data Mean) deviated significantly from the predicted value (Pred. Mean) and thus lie outside the 95% prediction interval, the value is marked red.

The validation of the mathematical models of the joystick shows that the models for the dimension stimulation and novelty of the UEQ cannot be confirmed. All other mathematical models were confirmed.

4. Discussion

4.1. Factors Influencing Operation of Control Elements in VR

In this study, we identified, for the first time, the factors that influence the positioning accuracy, time on task, presence, user experience, and mental workload when operating joysticks and rotary controls in VR. Mathematical models were used to calculate how these factors must be designed to enable users to operate the control elements efficiently in VR and to experience a high presence and user experience and a low mental workload. Finally, the VR-adapted control elements were validated by user tests.

Figure 4 and Figure 6 show the influence of the angular resolution and the positioning speed of the control elements on the evaluation parameter time on task and positioning accuracy. The higher the angular resolution or the positioning speed, the less time the test participants required to complete the tasks. However, the test participants with sensitive control elements also made more errors. The 8%/s calculated by the optimization calculation is therefore a compromise between time on task and positioning accuracy.

The diameter of the rotary control had a significant influence on the efficiency dimension of the UEQ and on mental workload. Test participants rated the mental workload higher and the efficiency dimension lower when the rotary control had a large diameter of 110 mm. However, the diameter of the rotary control had no influence on the positioning time or positioning accuracy. Thus, the test participants only had the feeling that the operation of the big rotary control was not as efficient. Objects are often displayed larger in VR for better readability or representation. The results of this study show that, at least for control elements, an enlarged display may not be useful despite good operability, because test participants perceived the operation of the control element as less efficient.

The factor input device had a significant effect on the positioning accuracy. The test participants with the Valve Index controller primarily had problems releasing the control element at the end of the positioning process. The test participants had to remove their fingers from the controller simultaneously to release the control element in VR. If this was unsuccessful and the test participants continued to move their hands, the control element continued to rotate. With the HTC Vive controller, only the trigger button must be released. Problems with the release did not occur with this controller.

Test participants who tested rotary controls with haptic feedback were able to complete the tasks significantly faster than test participants who tested rotary controls without haptic feedback. We expected that haptic feedback would also influence the positioning accuracy. However, the test participants apparently acted more cautiously and thus slower with the rotary controls without haptic feedback in order to make as few errors as possible. The test participants were not informed that the time for the positioning tasks or the number of errors were measured.

The shape of the rotary control had no significant influence on the time on task and positioning accuracy, but it influenced the mental workload. The test participants with the knurled rotary control experienced the lowest mental workload. The highest mental workload was reported with the cylindrical rotary control. Because of the knurling, the rotation of the rotary control was better recognizable than with the other two variants. We assumed that this additional visual feedback reduced the mental workload of the test participants.

Test participants were able to operate the joystick significantly better while sitting than while standing. Fewer errors were made in the sitting position, and the test participants needed less time for the tasks. In real-life standing workplaces where control elements are operated, forearm supports exist or the operators can rest their arms on a table [73]. The test participants in this study stood freely in the room. Movements in the upper body consequently had a direct effect on the operation of the control element. In reality, small movements in the upper body or arm have no or only a minimal effect because the fixed mounting and the restoring force of the joystick prevent accidental or unintentional operation. In VR, this force is nonexistent. Further studies should investigate how joysticks can be adapted to operation while standing. We suggest that the sensitivity of the joystick should be further reduced. The results show that the position of the test participant should be considered in usability tests of new prototypes and can influence the usability of the prototype in VR.

The factors of acoustic feedback and visualization of the hand influence presence and user experience. Overall, the test participants rated the user experience of the joystick better when it had acoustic feedback and when the hand enclosing the joystick was visualized. Conversely, rotary control was rated better when no hand was visualized. Other studies have described clear benefits of visualizing hands in VR [74,75]. In the study by Lougiakis et al. [75], the effects of virtual hand representation on interaction and embodiment in VR environments were investigated. The test participants in this study had to move an object through a type of obstacle course. Although more errors were made with the hand representation than with the representation of a controller, it was preferred by most users. We assumed that, in our study, the hand covered the rotary control too much and that the participants were better able to detect the rotation of the rotary control without a visualized hand. Another reason for the lower user experience could be the virtual hand posture. In our study, the virtual hand gripped the cylindrical rotary control completely with all fingers (cf. Figure 3). Another type of grip for rotary controls would be a two-finger grip with thumb and index finger or thumb and middle finger. Whether this type of grip is more suitable for VR and feels natural to the user must be investigated in further studies.

Table 8 shows that the size of the joystick in VR had a significant effect on the time on task. The test participants who evaluated the joystick with a total height of 32 cm took significantly longer to complete the tasks. We hypothesized that a large joystick suggests a high actuating force and large range of motion, causing the test participants to pull up on the VR joystick more quickly. Participants who evaluated the smallest VR joystick took less time to complete the task than participants who evaluated the large joystick but took more time than participants who evaluated the medium joystick. We suspected that the small joystick, in contrast, suggested a small range of motion and fine motor work, which did not correspond to the actual work task.

The interaction between the factors positioning speed and angular resolution had a significant influence on the perceived presence of the test participants. The angular resolution was varied because we suspected that a linear angular resolution, i.e., the proportional increase in the positioning speed depending on the deflection of the joystick, could lead to a poor positioning accuracy due to the lack of force feedback in the VR. However, the angular resolution had no influence on the positioning accuracy. With the three-stage angular resolution, the angular range of the joystick was divided into three equal sections in which the same positioning speed prevails. If the test participant moved the joystick within the defined section, the positioning speed did not change. It only changed when the joystick was moved from one section to the next section. The three-stage angular resolution in combination with a high positioning speed led to a strong increase in the change in the positioning speed between the sections. We suspected that this behavior felt very unnatural to the test participants, causing presence to be rated very low for this combination. Overall, it could be concluded that a linear angular resolution is suitable in VR, despite the lack of force feedback, because both the three-stage and the five-stage subdivisions did not lead to a reduction in position errors. A linear angular resolution is more in accordance with the user’s expectations and, therefore, also makes sense in VR.

In addition to the control-element-specific factors, we also varied the VR environment. The test participants performed the trials either in a detailed crane environment or in an empty virtual room. In the literature, the advantage of VR is stated to be the ability to create realistic virtual environments that simultaneously provide a high degree of control over experimental conditions and good ecological validity [76]. In our study, the test participants rated the user experience of the control elements higher when the control elements were tested in the detailed crane environment (refer to Figure 2). This could either have been due to the realistic virtual environment or the detailed crane environment increased the novelty effect, in which test participants rate products better due to the novel technology [77]. A comparative study between VR and reality should further investigate whether the same control elements are similarly evaluated in a real crane environment. If not, this could be an indication of a bias caused by the novelty effect.

Overall, the factor positioning speed had a significant influence on most of the evaluation parameters. Objective parameters such as time on task or positioning accuracy as well as subjective parameters such as presence and user experience depend on the positioning speed of the joystick. Therefore, when evaluating prototypes in VR and simulating operating processes, it should be especially considered that the positioning speed is adapted to the changed modalities in VR. The recommendations for the design of a joystick in VR developed in this work offer a first orientation.

4.2. Optimized Control Elements for Operation in VR

A comparison between the calculated optimal rotary control in VR and the recommendations of Anguelov [40] and Schmidtke and Jastrzebska-Fraczek [30] shows that the calculated optimal diameter of 33–45 mm corresponds to the recommendations for real rotary controls (refer to Table 1). The optimization calculation computed a sensitivity of 24–36 detents per revolution. This is close to the recommended range of Anguelov [40] with 25–35 detents per revolution. We assume that haptic feedback in the form of vibration pulses per detent can compensate for the missing detent torque.

The handle length of a real joystick as recommended Schmidtke and Jastrzebska-Fraczek [30] is 75–125 mm. This recommendation only refers to the grip handle and not to the overall length of the joystick. In our study, we varied the overall length including the grip handle. The joystick optimized for VR had an overall length of 240 mm. In the optimized vertical shape, the length of the VR handle was 95 mm and thus corresponds to the recommendations of Schmidtke and Jastrzebska-Fraczek [30].

The recommended diameter for real joysticks is between 20 and 40 mm, and the actuating angle should be less than 45°. The calculated optimal handle diameter in VR was 43 mm, which is slightly higher than the recommendations. The calculated optimal angle range for joysticks in VR was 10–17°, which is within the recommendations for real joysticks. Overall, the design recommendations for real rotary controls can be transferred to VR if haptic and acoustic feedback types in the form of vibration pulses and clicking sounds are implemented. The recommendations for the design of real joysticks are also transferable to VR; however, the feedback should only be acoustic.

4.3. Limitations

In this study, we used the DOE method and examined the control elements with an fractional factorial I-optimal response surface test design. The test design used had the resolution IV, i.e., the effects of the main factors (A, B, C, etc.) were neither mixed with each other nor with effects of the two-factor interactions (AB, AC, etc.). However, effects of the main factors were mixed with three-factor interactions (ABC, ACD, etc.), as well as effects of the two-factor interactions among themselves. This type of design is used to identify factors that have a significant effect on the evaluation parameters. The mathematical models created and optimization calculations performed in this study provide initial guidance. In a follow-up study, detailed investigations with the significant factors will be conducted with a full factorial design. The important factors will be evaluated on more levels and with a significantly increased number of experiments in order to further increase the accuracy of the mathematical models.

The main field of application of the DOE is chemical and process engineering [59]. However, the DOE is also being used more and more in the field of human factors [78]. Compared with chemical and process engineering, signal-to-noise ratios (SNRs) for human test participants must be expected to be higher owing to human variabilities such as age, affinity for technology, and prior VR experience. The SNR presented in Table 6 and Table 9 indicates that the tests had high scatter in some cases. In our estimation, these are mainly due to the different prior experiences of the test participants with the VR systems. In addition, the standard deviation of questionnaires in the field of usability and user experience is high [79].

This also made it difficult to perform confirmatory runs. Table 7 and Table 10 show the results of the confirmation runs, in which four additional test participants evaluated the optimized control elements. The theoretical parameter values calculated using the mathematical models for these control elements deviated for both the joystick and rotary control for the dimensions of the User Experience Questionnaire. However, the calculated 95% prediction intervals were partially outside the scale of the questionnaire. The upper limits of the 95% prediction interval were above a value of three for all UEQ dimensions, and thus above the scale. In addition, it could be assumed that some test participants tended towards the middle [80,81]. Consequently, extreme values of the questionnaire were selected less frequently. However, during the optimization calculations, the design of the control elements was calculated to precisely achieve these extreme values. Therefore, the mathematical model for the mental workload for operating the rotary control could not be confirmed. According to the optimization calculation, the optimized rotary control should achieve a score of zero. However, this score is unlikely in practice. The optimized rotary control achieved a score of 2.63 in the confirmation runs. This corresponded to a low mental workload.

Overall, the experiments showed that although the response behavior of the test participants could not be precisely modeled mathematically, the models were sufficient to calculate suitable control elements that achieved good results on the questionnaires. Comparatively high values were achieved for the UEQ and SUSQ. The positioning times (time on task) and positioning accuracy were even better than those predicted by the mathematical models. Generally, these human performance data, such as positioning accuracy or the time on task, can be better represented mathematically than factors for perception, such as mental workload.

4.4. Interaction with the Virtual World

The test participants interacted with the virtual control elements using VR controllers (Valve Index/HTC Vive). Studies have also shown that interactions without a controller can be beneficial. In this case, the user’s hands were captured and displayed in VR [82,83,84,85]. Recent innovations, such as the Apple Vision Pro, use a combination of eye and hand tracking to optimize interaction with VR, eliminating the need for controllers [86]. However, for the user evaluation of prototypes, e.g., a new car, in VR, it is important that the interaction modality be transferrable to reality as much as possible. Human-machine interfaces based on eye tracking are not yet established in reality. Accurate hand tracking, such as that provided by the Apple Vision Pro, could enable realistic interacting with virtual prototypes. However, hand tracking alone does not provide haptic feedback. Our study shows that haptic feedback has a significant impact on the operation of control elements in VR. Precise hand tracking, haptic feedback, and force feedback enable data gloves that theoretically allow for the most realistic interaction. However, the implementation of data gloves is currently still very complex [54]. How the control elements have to be designed for operation with optical hand tracking systems or with data gloves has to be investigated in further studies. To this end, our study provides factors that have a significant influence on the usability and subjective perception of the user when operating control elements in VR.

5. Conclusions

Overall, this study shows that it is generally possible to adapt control elements to different modalities in VR and optimize operation. Factors that have a significant impact on user experience, presence, and perceived workload when operating control elements in VR were identified. The results showed that rotary controls in VR should have a resolution of 10–15°/value, a diameter of 33–45 mm, and knurling. Haptic feedback in the form of vibration pulses and acoustic feedback in the form of clicks should be implemented. Joysticks in VR should have a maximum rate of change of 11 values/s, a maximum deflection of ±10–17°, a length of 240 mm, and a diameter of 43 mm. The feedback when operating joysticks in VR should be acoustic.

With VR-adapted control elements, prototypes of machines and devices can be better tested by users in VR by minimizing distortion due to the altered modalities in VR. Possible applications include cranes, drilling rigs, agricultural machines, or medical devices. In these tests, the usability of the new product as well as initial user evaluations can be examined without the need to manufacture real prototypes or control elements. It can be examined whether the selection of control elements and their layout are suitable for the work tasks. These completely virtual tests should be followed by tests in mixed-reality environments in which the selected control elements can be operated as realistically as possible. Overall, this approach could improve the development process, reduce manufacturing costs, and optimize the HMIs of products.

How well control elements can be operated in VR depends strongly on the type of interaction. In this study, VR controllers were used to interact with the virtual joysticks and rotary controls. Currently, more and more interaction types based on hand and eye tracking are entering the market. Data gloves, some of which provide force feedback, are also being developed further. This study shows that haptic feedback in the form of vibration pulses significantly improves the operation of rotary controllers in VR. Therefore, VR controllers or data gloves should be used for user testing of prototypes in VR. Especially, the possibility of force feedback could make the interaction with VR prototypes even more realistic. The influence of the use of data gloves and force feedback on the factors investigated in this study, e.g., user experience, needs to be investigated in further studies.