2.1. Startles, Fear, and Premonition
Startles are widely used in in psychology as a physiological means of measuring psychological characteristics [
9,
10,
11,
12,
13]. The review by Blumenthal et al. [
14] provides an excellent overview of the established techniques for eliciting a startle response, as well methods used for measuring the response. Eye twitches are one of the oldest metrics used in psychology research. EMG measurements of eye muscle activation are typically used to measure eye muscle activity as an indicator of a response to stimuli [
14], although eyeblinks can be measured by cameras or mechanical devices [
15], yielding similar results. In this research, we used EMG measurements of muscle activity near the eye, back, and neck, because the different startles were expected to activate muscles in those areas based upon the participant’s whole-body physical response (e.g., jumping, looking up, and blinking) due to the startles. Our results suggest, however, that the classical eye squint response measurements may not be valid for measuring response to strong environmental displays because they naturally induce squinting.
In psychology research, startle responses are elicited using loud audio, electrical stimulation, magnetic stimulation, light flashes, or mechanical stimulation [
14]. One can imagine that some of these techniques may be more desirable for VR. Audio stimulation, the most common technique, consists of a short burst of white noise (~50 ms) played at high sound levels (~100 dBA) to ensure that a startle response is elicited, although research indicates that audio levels as low as 50 dBA can elicit a startle response [
16]. Audio is a natural part of a VR experience, but bursts of white noise are not. Hence, researchers must decide if their goal is to elicit a clinically accepted startle response, or if the goal is to evaluate the effect of different startle audio. Clinically accepted white noise audio bursts have been used in VR research to evaluate the effects of conditioning [
17], although some do use natural sounds from the environment, such as the sounds of explosions and sirens in VR training simulators [
18]. Our results highlight that enabling environmental display appropriately can actually magnify or diminish the startle effect of loud audio.
To drown out background noise (e.g., machinery), noise-cancelling headphones have been used to mask the sounds of apparatus [
19]. The role of artificial background noise is debated because background noises in the 65–75 dBA range have been shown to reduce startle responses (i.e., pre-pulse inhibition) while masking the sound of apparatus [
20]. In contrast, louder background levels, such as 75 dBA or 85 dBA, have been shown to increase startle responses [
21]. In this study, we used noise-cancelling headphones to reduce sounds from the apparatus and then used background sounds that would naturally occur in the virtual world; startle sounds were also natural, corresponding to the event that the user experienced.
Visual stimulation, such as bright lights with increasing intensity [
22] or rapidly approaching threatening objects [
23], are also used by psychologists to elicit startle reflexes. Psychologists sometimes use still images to evoke fear [
24], but videos have been shown to be more effective for generating fear than static images [
25]. Similarly, light and darkness have been shown to elicit stress in participants akin to real life as they drive cars through tunnels in VR [
26]. VR simulations for emergency responders use graphical representations of a car exploding coupled with the sound of an explosion, instead of bursts of noise [
18]. Startles consisting of loud white noise were used in [
27] as a user travelled through different parts of a virtual world. Scary games are used with surprising events, such as falling furniture or ghouls that appear suddenly [
28], which are also combined with startle audio characteristic of those phenomena. Our virtual world was most similar to the former in terms of visual startle variety, but our phenomena were typical of real life; we employed a small bird that suddenly fluttered across the display, a beam that fell from the ceiling of a barn, and a large flash of light due to lightning, all of which included audio that was appropriate to each startle.
A variety of other startle stimulation techniques have been used. Electrical stimulation is achieved by the application of an electrical potential above the threshold of detection and below the threshold of pain [
29]. Magnetic stimulation can also be used to elicit eyeblink responses [
30]. Air stimulation can elicit startle responses using brief puffs of low-pressure air [
31,
32], typically applied to the forehead or temporal regions of the head [
33]. Mechanical stimulation, such as tapping or ballistic impacts, has also been used [
34]. The study presented here used wind as an environmental display, but this is quite different compared with the air pulses used to generate startles in the abovementioned studies. When enabled, wind was used as a steady haptic display until the thunder startle, whereupon wind was increased as a premonition of an impending storm. Others have used haptic displays, such as vibrating interfaces, to elevate the heartrate in VR, but this does not result in a startle response [
35].
The usage of startles in VR research is often focused on psychology. Studies have used startle responses in VR to study phobia reactions [
17], the motivational mechanisms of cravings [
36], and to improve the effectiveness of post-traumatic stress disorder treatments [
37]. These types of studies measure startle responses to evaluate the efficacy of their therapies. Studies using VR to study startle responses have found that users who were startled while involved in a complex task exhibited smaller startle responses [
18], and that social anxiety is associated with a greater startle response [
38]. Other studies have also used startle responses in VR to study the effects of extinction learning [
27], effectiveness of certain trauma treatment methods [
28], and conditioning for phobias [
39,
40]. In contrast, startles [
41] and audio warnings [
42] have also been examined to alert a user to an impending impact so that a tensed reaction can be used to better protect themselves; although these were not focused on VR, the latter used a treadmill interface similar to the one used in this research in order to deliver warnings at precise instances during a running gait. One of the long-term goals of this research is to augment that work to evaluate the effectiveness of different audio and visual warnings in controlled realistic VR environments as users wear protective gear such as smart helmets [
8]. Finally, it is worth noting that not all VR environments promote startles or rely on fears; work by Noronhona used a natural outdoor VR world, as we do, but they were focused on promoting calmness via peaceful graphical presentations and soothing sounds as a means of therapy [
43]. Our VR world focused on pleasing natural environments, such as a walk along a river, through the forest, and in the mountains; however, we did not focus on promoting peacefulness, which could be a focus of future research given the advanced haptic displays presented here.
Premonitions created by VR were also a feature of this study. Specifically, the thunder startle was designed to feature graphics and environmental displays which cued the participant that a startle was about to occur. This is typical of extinction learning [
44], where repeated exposure to a stimulus is used to reduce the effect of the stimulus. In this case, we relied upon associations of the stimuli with the thunder startle to diminish its impact. Previous VR studies have used imagery of spiders as the unconditioned stimulus [
39] and colored light as the conditioned stimulus reducing the fear response. Images of fierce dogs and falling walls have also been used as unconditioned stimuli [
24]. The thunder stimulus is fundamental to child development to the point that it is known to have become “extinct through awareness” rather than requiring any specific conditioning [
45], which is why it was selected for this study. We have yet to find any papers that deal with environmental display or graphical display in VR as cues (e.g., the conditioned stimuli) for thunder startles. Hence, we believe that this paper highlights advancements regarding the ability of graphical and environmental displays in virtual worlds to leverage these natural premonitions engrained in humans from childhood.
It is important to note that premonition is different from exposure therapy, where a user is repeatedly exposed to a stimulus to reduce its effect, such as the treatment of acrophobia [
40]. In fact, this study only presents each stimulus once. Likewise, psychological research focuses on pre-pulse inhibition for a startle, meaning that things that happen moments before the startle can diminish the magnitude of the startle. Pre-pulse tones, for example, can diminish the intensity of startle response to loud white-noise pulses [
1], but as indicated by [
46], tones that occur 15 to 400 milliseconds before the startle inhibit reactions, whereas longer pre-pulse periods (e.g., 2 s) become ineffective. This is in contrast to a fear-potentiated startle, such as the dark haunted house used in [
28] to potentiate startles, which is typical of fear-potentiated VR research. In this study, however, we created a startling situation, a thunderstorm, and examined the effect of an environmental display (wind, mist, heat, and odor) and graphical display (darkening skies) several seconds before the startle to attenuate the startle response instead of potentiating it.
2.2. Simulator Technology
Development of computer graphics technology in the 1960s and 1970s led to the early development of VR simulators for training pilots [
47]. As indicated by [
48], contemporary simulators range from a computer screen and joystick to high-fidelity mockups with 6 DOF motion platforms and graphical displays to create realistic experiences [
49]. Similar difficulties operating the simulations have been reported across the range of simulators; however, neurological activity is noted to be significantly increased with VR-based simulators. VR simulators provide an improved sense of presence, and many use haptic interactions with remote systems and simulations, leading to extended reality (XR), which is used in applications such as piloting remotely operated vehicles [
50,
51] and surgical robotics [
52]. Haptic feedback is often an important part of this process so that user can better perceive the conditions and interactions of a remote system or simulation. We have not found any simulators that evaluate the effect of environmental display on their training results, although it is common for simulators to make users respond to the effect of adverse environmental conditions, suggesting that environmental displays could add a new dimension to these training scenarios.
To enhance realism in VR, many studies have used sensory feedback systems such as olfactory, heat, and wind. Few, however, combine as many as MS.TPAWT. Examples of olfactory feedback systems include the Lotus system, which uses a directional mist system, and the VE VIREPSE, which uses a fan-based system [
53,
54]. Both systems differ from ours in that they focus solely on the effects of olfactory feedback.
Warmth has also been used across many types of VR systems to induce a greater feeling of presence. Examples of this being implemented include using several heating/cooling elements to enable a seated user to feel changes in temperature as they explore a space with their hand [
55], or through heatpads worn with a mobile headset [
56]. Our system allows ambient heat to be felt by the user in a way realistic to sunlight without limiting locomotion of the user.
An example of a wind system is the WindCube [
57], which can provide wind from several directions, but impedes heavily on the virtual environment, and thus, the total immersion [
58]. Simpler, less intrusive wind systems include the VR Scooter [
59] and Sensorama [
60], which both use fixed fans hidden from the user to simulate speed. Ambiotherm combines a thermal display with a wind display as an accessory for head-mounted displays [
56]. These setups minimize intrusion on the VR experience, but do not allow for multidirectional wind or user movement. Our system created multidirectional wind from a device hidden from the user and allowed for user movement.
Head-mounted displays have become popular commodities for VR research and home applications due to their affordability and portability; however, CAVE display systems are still a common occurrence for research-related VR systems [
4,
61,
62] and commercial locomotion systems [
61]. Without locomotive input, they have comparable effectiveness in some scenarios to higher-end mobile displays such as the Samsung Gear VR [
62]. CAVE displays combined with locomotion systems are very popular for locomotion studies and gait therapy, however [
4,
63]. The user can interact with their physical environment, for example, using the railings on a treadmill or via advanced haptic displays that render terrain features [
4], slopes [
64,
65,
66,
67], or inertial forces [
6,
68], while also experiencing their virtual world unencumbered. Physical therapists can more easily interact with the users, helping to guide their therapy. Bodyweight support systems [
5] allow for safety in applications such as gait therapy for spinal cord rehabilitation, or in simulations of reduced gravity. Tether-based systems also allow for perturbations to be applied during gait [
43,
69]. Such systems often provide sufficient space perturbations and for motion capture systems that can be used for characterizing gait properties and controlling interactions with the user. CAVE displays also remove the added weight of a head-mounted display, which would perturb the user’s kinematics, allowing the user to move more freely and naturally. Although MS.TPAWT provides all of the above features, only a subset were applied in this research. Future research would expand on the results from this paper to evaluate the effects of other haptic interactions and user experiences.
Most of the related studies described above use up to two haptic feedback devices. MS.TPAWT used four haptic feedback devices coupled with a locomotive input and visual/audio display. These effects are designed to be non-intrusive, allowing the user to freely interact with their environment. The combination of these systems enables the effective study of startling experiences which would be difficult or dangerous were they to be performed without VR.