Enhancing Operational Police Training in High Stress Situations with Virtual Reality: Experiences, Tools and Guidelines

Abstract

Virtual Reality (VR) provides great opportunities for police officers to train decision-making and acting (DMA) in cognitively demanding and stressful situations. This paper presents a summary of findings from a three-year project, including requirements collected from experienced police trainers and industry experts, and quantitative and qualitative results of human factor studies and field trials. Findings include advantages of VR training such as the possibility to safely train high-risk situations in controllable and reproducible training environments, include a variety of avatars that would be difficult to use in real-life training (e.g., vulnerable populations or animals) and handle dangerous equipment (e.g., explosives) but also highlight challenges such as tracking, locomotion and intelligent virtual agents. The importance of strong alignment between training didactics and technical possibilities is highlighted and potential solutions presented. Furthermore training outcomes are transferable to real-world police duties and may apply to other domains that would benefit from simulation-based training.

1. Introduction

As first responders, police officers often find themselves in highly stressful operational situations in the field [1]. When confronted with such stressful and sometimes threatening situations, police officers have to be able to make split-second decisions and act upon them under ambiguous and complex circumstances. Decision-making and acting (DMA) of police officers is therefore decisive of whether a situation is resolved adequately or puts bystanders, suspects, or the officers themselves at risk of harm. We define DMA as the interplay between cognitive and sensorimotor processes [2]. According to this definition, cognition and movement are inseparable, intertwined processes that determine DMA behavior. For instance, when a police officer is confronted with a use of force situation, the distance between the suspect and officer may inform the officer about which type of force is appropriate to use in this particular situation (e.g., pepper spray, service weapon, baton, taser, physical force). In addition, the officer needs to consider the legal framework in which he or she can safely operate, further informing the decision on subsequent actions. Hence, the officers’ movements and behaviors (e.g., creating distance to a suspect) inform his or her decision and as the situation evolves (e.g., having a legal basis to use lethal force), his or her decisions inform the movements and behaviors. The process between action and decision-making is intertwined and emergent rather than separate and sequential [3]. To train and improve police officers’ DMA, it is necessary that officers have the opportunity to perform DMA behavior in training.

In general, police training aims to prepare police officers to confront and solve complex and stressful on-duty situations. To this end, simulation-based training can be an efficient way to train police officers, as it allows officers to actively practice the application of what they have learned while being exposed to the perceptual, motor, and cognitive deficits that are associated with high levels of stress [4,5,6,7]. However, the delivery of training in accordance with a learner-centered didactical approach is pertinent to enhance learning and transfer to real-life situations [8]. To enhance learning, transfer, and consequently on-duty performance, training methods should align with didactical approaches to police training.

Virtual Reality (VR) is a great addition to current training practices. It should not be seen as digitalization or virtualization of existing training methods, but rather as an additional option that can be used to train situations that are difficult to practice in real-world training. For example, effective virtual environments (VEs) may be developed for training tactical skills and personal safety procedures. Contrary, for firearms training (as shooting precision is often not sensitive enough in VR [9,10]), or close combat skills (as it is currently not possible to recreate realistic physical contact with a Non-Player Character (NPC)), VR might not be the best medium for training. Not all training objectives can be met using VR-based training. Some training areas important for Law Enforcement Agencies (LEAs) are perfectly trainable in VR and will fall short of meeting the needs of others. Hence, it is crucial to define training goals and frameworks before considering to implement a VR training system.

VR training offers a number of advantages over traditional methods. In several industries it has proven itself as cost-effective, engaging and provides opportunities to include objective performance measurement techniques as well as a high degree of personalization [11,12,13]. It is especially considered a useful tool for training in professions where errors in real situations can seriously endanger people and even cost lives [14,15,16]. Specifically for the domain of police training, VR-based training offers possibilities to safely train dangerous situations that would not be allowed in real-life training because of the high risk of physical injuries to trainees or bystanders [15]. It allows for training scenarios that include vulnerable populations (e.g., children), animals (e.g., dogs) or dangerous equipment (e.g., explosives) [15,17].

This article builds upon learnings from previous research and reports findings from the three-year European Horizon 2020 project SHOTPROS (https://shotpros.eu/ accessed on 15 December 2022). The project had the aim to investigate the influence of psychological and contextual human factors (HFs) on the behaviour of decision-making and acting of police officers under stress and in high-risk operational situations (DMA-SR). Throughout the different development stages of the VR training system, several requirement and development workshops, human factor studies, and field trials have been conducted, as will be discussed in the following chapter.

Previous publications from the SHOTPROS consortium provide an outline of requirements collected from police officers, police trainers, and field experts in workshops at the beginning of the project which discuss strengths, weaknesses, opportunities, and threats of the adoption of VR training within LEAs [15,18]. A concept of using stress cues as audio-visual representation of stressors in VR has been proposed [19]. Murtinger and colleagues [17] present how an assistance system can support police trainers in VR with real-time recording and visualization to immediately react on trainee’s performance. Uhl and colleagues [20] demonstrate how multi-sensory stimuli affect threat perception in virtual police training and lead to more realistic DMA in stressful situations.

Other related research provides insight into increased stress level and threat perception of full-body avatars compared to only head and hands visible [21] and evidence that VR preparation for a police surveillance task leads to increased performance and decreased stress, workload and recuperation [22]. Furthermore, it shows that police officers with VR preparation make less direction changes when finding target locations, had lower heart rate variability (HRV) during the surveillance, indicating lower stress levels, hence higher stress resilience. Marler and colleagues [23] present a framework for implementing low-cost, game-based, VR technology to improve decision-making under stress in a pilot study, identifying stressful scenarios for police officers including skills required and a plan to implement the proposed gaming technology in police department training curricula. A study investigating psychophysiological responses associated with controlled breathing during a firearms training proposes a potential framework for bio-feedback based adaptations of the virtual environment (VE) [10]. Research from related domains such as other first responders [24,25,26,27,28,29], military [30,31,32], aviation [33,34,35] and CBRNE [36,37] has started to emerge.

In the following chapters we will provide an overview on the materials and methods used to gather insights on VR training for DMA-SR in the context of police training (Section 2), report results from a variety of studies and field trials (Section 3) and discuss the foundations of DMA-SR training and its delivery including scenario design (Section 3.1) as well as technical considerations such as realism and immersion (Section 3.2), interaction modalities and virtual agents (Section 3.3). Furthermore, we will highlight physiological stress measurement and visualization approaches (Section 3.4.1), performance management and ethical aspects (Section 3.4.4) that should be considered in the context of VR training. In section Future work (Section 4) we propose four topics for further investigations before we conclude with a short summary of this paper.

2. Materials and Methods

2.1. Requirement Workshops

At the start of the SHOTPROS project, six one and a half day End-User Workshops were organized with 60 experienced police officers and police trainers of six LEAs in Europe (Brussels, Amsterdam, Bucharest, Stockholm, Selm and Berlin) to collect requirements specific to police training in VR. The workshops consisted of a variety of sessions on topics important for the development of VR training for police officers, to gain knowledge about (a) the human factors and stressors that influence police officers’ on-duty performance, (b) their perceptions on current police training practices and future training needs, (c) their functional and non-functional needs and requirements for a future VR-system for DMA training, and (d) the type of scenarios they would want to train in such a VR training system. The data collected during these workshops served as the starting input for the development process of a VR scenario-based DMA training for police officers. An agile development approach was used to develop virtual test scenarios and advance the VR system and its physical props, by means of weekly scrums and the continuous systematic collection and further implementation of end-user feedback.

2.2. Human Factor Studies

To gain further insights into specific aspects of VR training for police officers, several human factor studies were conducted and included a variety of different study designs. The results of these studies have been published in separate articles and research deliverables as mentioned below. The effect of adding a pain stimulus to virtual training simulators such as VR was investigated with Swiss police officers [38]. An empirical study for assessing the stress inducing capabilities of stressors highlighted in the requirement workshops was conducted [39]. A qualitative analysis of the organization and delivery of European police training identifies strengths and challenges of current training practices amongst European law enforcement agencies. The results report how law enforcement agencies organize their training practices focusing on training curricula and components, skill and knowledge assessment, and resource availability. In addition, the results show how current training practices are delivered, including the role of the instructor, didactical approaches and concepts, and creation of effective training environments [8].

2.3. End-User Feedback Studies

To receive initial feedback on user experience two end-user feedback studies including VR system try-outs were conducted with 96 police officers and trainers one and a half year into the project. The VR system was developed and provided by RE-liON (https://www.re-lion.com accessed on 15 December 2022) and the studies conducted in empty gym halls on the premises of two participating LEAs. The training set up (Figure 1) included a dressing area including storage for police equipment (#1), a Battery Charging Station (#2), a 30 × 30 training area, surrounded by eight radio beacons connected to the Execution Control (EXCON) station (#3), four to six full body Smartvests (#6) for trainees and role players and a trainer station (#4) for de-briefing. The system works with a client-server architecture, with the Smartvests being the clients, sending input (sensory system) and receiving output (Head Mounted Display (HMD), sound and haptics). The server runs in the EXCON Station and processes the ‘business’ logics (scenario execution, ballistics, etc.). The VR engine and software used was custom-built and programmed based on similar VR training environment for military training (https://www.re-lion.com/blacksuit.html accessed on 15 December 2022). A video summarizing the features of the system can be found on the projects YouTube channel: SHOTPROS results video (https://youtu.be/xQ9AnHqtOOQ accessed on 15 December 2022).

Both studies consisted of five testing days that were split into 2 slots with three hour testing. Participants were grouped into teams of four based on the feedback received from LEAs that this is the group size they usually operate in. After participants signed the informed consent, they were dressed up in the VR suites and headsets and went through a introductory tutorial to familiarize themselves with the virtual environment. Afterwards they went through three different scenarios with increasing levels of difficulty, followed by a debriefing with the trainer. After being relieved of the VR suites and equipment participants were brought into a quite room next to the training hall to fill out several questionnaires investigating their experience. User experiences were assessed based on Technology Acceptance (TAS) [40], Quality of Experience (QoE) [41], Quality of Learning (QoLE) [42], Sense of Presence Inventory (SOPI) [43], Visual Analogue Scales (VAS) for stress [44,45] and mental effort [46], and open qualitative feedback.

The results of the studies were grouped in themes (hardware, graphic/visualisation/ animation, scenario scripts, scenario editor, introduction/set up/calibration, non-player characters, physiological stress measurement, stressors, tactical belt, participants interaction and communication, tactile feedback, performance monitoring, trainer dashboard and debriefing) and transferred into a backlog, which the agile development team prioritized in collaboration with LEAs into development tasks to further improve the VR system.

2.4. Field Trials

Towards the end of the project six field trials were organized, each resembling a real police training session, to test the final version of the VR training system. The same principle VR system was used as in the end-user feedback studies. However, with improved features based on the analyzed data and feedback provided throughout the project [47]. The most important considerations and improvements are discussed in the results section (see Section 3). To give each LEA the opportunity to test the VR system for their individual training needs, LEAs were able to select and adapt VR training scenarios to fit their training curriculum. The procedure of the studies was adapted from the end-user feedback studies as described above. Each field trial consisted of three to four testing days and included a total of 234 participants (active police officers and trainers from the participating LEAs). To receive further in-depth feedback, seven focus groups were conducted during the field trials with a total of 22 participants who completed the VR training. The focus groups aimed to provide further insights into the experience of participants regarding their perception of cognitive and sensory information in VR, their stress mitigation strategies in VR, and their ability to practice with task-relevant and task-irrelevant information in VR. In addition, systematic observations of the VR training were conducted on the basis of established criteria for high quality training of police officers [48]. During the systematic observations, examples were gathered of how didactical criteria such as high quality instruction, constructive feedback, and variation and differentiation are used in VR training [49].

3. Results and Discussion

3.1. Foundation of Technical Guidelines

To effectively implement a training technology such as VR into police training, various foundational aspects should be considered. These considerations will inform the design, implementation, and application of VR in police practice. First, a theoretical basis to support the training of certain behaviors (e.g., DMA behavior) in VR should guide the design decisions of hardware and software (e.g., simultaneous full-body motion tracking of multiple team members). Second, the possibilities that VR offers need to align with training areas and objectives that benefit from the advantages of VR or that currently have limitations in real-life practice. Lastly, the VR system used for training police officers needs to be able to enhance learning and elicit stress since police officers operate under stressful conditions and should be prepared for those experiences in training. In the following, we discuss the three considerations (i.e., theoretical basis, training objectives, and training delivery) in detail.

3.1.1. Theoretical Basis for DMA VR Training

DMA is an integral part of operational police performance. Particularly under stressful circumstances, police officers’ ability to decide and act is challenged due to the influence of stress [7]. Exposing police officers to stressful situations in training and allowing them to familiarize themselves with the influence of stress on their behavior may improve their performance under these circumstances [5]. In order to use and develop a VR training system that enhances DMA under stress, a human factors model of police officer’s DMA under stress and in high-risk situations (referred to as “HF model”) was conceptualized [50,51]. The HF model is based on the integrated model of anxiety and perceptual-motor performance [6] and was adjusted to the context of VR police training within the SHOTPROS project.

Following the HF model, various human factors including personal, contextual, organizational, and societal factors influence whether a stress response occurs. Generally, when a police officer perceives the demands of a situation to outweigh their capabilities to resolve the situation, the situation is perceived as stressful to their well-being [52]. The stress response may then influence attentional processes that can lead an officer to focus on task-irrelevant, threat-related information (such as bystanders, sounds, etc.) instead of task-relevant, goal-directed information (such as the possibilities to restrain a suspect). Consequently, these attentional processes influence DMA-SR and are thus decisive of the outcome of a stressful situation. In order to train DMA-SR in VR, the HF model provides important foundational considerations:

• DMA is an integrated process between cognition and movement. Hence, VR systems need to provide police officers with the opportunity to move about freely and realistically.
• DMA behavior depends on the available action possibilities. Hence, the VR system should allow police officers to perform a variety of actions (e.g., multiple types of force).
• DMA behavior is influenced by human factors such as personal factors (e.g., personal stressors, physical strain) and contextual factors (e.g., implementing the ‘unexpected’, bystanders, sensory elements) [28]. Hence, the VR system should include and display a variety of relevant human factors in the training environment.
• DMA behavior is influenced by a shift in attentional processes, mostly between task-relevant, goal-directed and task-irrelevant, threat-related information. Hence, in VR, the training environment should include information that helps a police officer solve a scenario as well as information that may be distracting from the goal (e.g., loud music, clutter, multiple bystanders).

According to the model underlying the VR solution [50,51] one of the aims of training for stressful circumstances is to be able to maintain or restore goal-directed attention. In order to train these attentional processes, distractors need to be present in the VE. Ideally, these distractors are similar to the ones that are present in real life (e.g., bystanders, mess in a house during a search, noises in the surroundings). The use of VR in itself may evoke distracting information that is not related to real life circumstances (e.g., initial familiarization with wearing VR equipment), however, this may still provide practice opportunities to remain task-focused (i.e., goal relevant attentional processes).

External sounds (i.e., sounds coming from outside the VE) are an example of distraction that adds unnecessary and therefore undesirable extraneous cognitive load. In the VR solution that resulted from the project, trainees are equipped with a noise canceling headset that eliminates most external sound, except from sounds within the VE. Proof of this noise canceling capacity were multiple moments when instructors were trying to communicate with trainees outside of the system and failed to make themselves heard.

As with all types of learning, and in line with cognitive load theory [53,54], training instructors and scenario designers need to ensure that the extraneous cognitive load (i.e., learning- irrelevant demands placed on the trainee) in VR training is minimized (e.g., by effective instructional design for VR training [54] and technological developments such as latency minimization [55].

3.1.2. Training Objectives in VR

To conduct VR training effectively, the training objective of the lesson VR may be used for should align with the technical advantages that VR provides. For instance, in VR, a variety of VEs can be designed, displayed, and adjusted on the fly, making VR an excellent training tool for DMA-SR. DMA-SR skills should ideally be trained in various settings. In their training curricula, police agencies include a variety of training components and areas implemented to prepare police officers for their job [8]. Based on a review of training-related documents from six European law enforcement agencies, we selected ten training areas commonly implemented across all six agencies and evaluated the usefulness of VR for the training of these areas based on systematic observations during the field trials and expert ratings of police trainers [49,56]. The results are displayed in

Table 1. Police training areas derived from usefulness ratings by police trainers. The usefulness scale specifies a range from 1 to 5 stars where 1 star indicates a minimal usefulness, and 5 stars indicates a maximal usefulness.

Training Area	Usefulness	Observation and Recommendation
Tactical training	*****	The possibility of quickly varying location and scenario context in VR creates the groundwork for the training of tactical strategies in many different situations (e.g., tactical movement, car procedures, extreme violence situations).
Perception and action training	*****	In real-life, the training design needs to be adapted to the training location’s infrastructure. VR does not have this limitation, making it extremely useful for perception and action training. VR offers training environments that a trainer can manipulate on the fly (e.g., adding extra bystanders).
Law and regulation training	****	The After-Action Review (AAR) (visual debriefing of the training session) is an excellent VR-specific feature to provide feedback on information regarding law and regulation that cannot be monitored and reviewed easily in real-life training.
Communication training	***	VR is helpful for communication training as it allows quick customisation of the avatar’s appearance (gender, skin, cultural aspects) and how trainees respond to and communicate with the avatar. Communication relies on the appropriate interpretation of emotions. In VR, there are limitations regarding the display of accurate and interactive facial expressions. We recommend to work with a professional actor as a role-player to optimise the display of adequate emotions.
Shooting and weapon handling training	**	The AAR provides relevant information on shooting performance (e.g., hit rates, shooting lines, cross-fire) that cannot be evaluated readily in real life, making VR useful for training positioning in reference to suspects and colleagues. Currently, VR has limitations in weapon handling and aiming (e.g., precision of shot accuracy, realistic reloading). As these skills are extremely relevant for on-duty performance, they should be executed and trained in real-life with the service weapon.
Physical training (combat, fitness training)	*	All actions involving physical contact (e.g., handcuffing, controlling and restraining of suspect, use of weapon) are not suitable for VR training due to safety for trainees, fragility of materials, and necessary real-life haptic experience.

.

Training areas, particularly in the context of DMA-SR, for which VR appears beneficial include tactical training, training for perception and action, and law and regulation training (see Table 1). Conversely, if a training objective includes areas in which VR has technical limitations (e.g., physical training of combat skills, interactive communication training, or service weapon handling training), alternative modes of training should be selected (see Table 1).

3.1.3. Training Delivery in VR

Training delivery is an integral part of how the training is perceived by the trainees and which learning opportunities trainees are provided with within the training. Training delivery is largely the task of police instructors, however, for the delivery to be as effective as possible, the training modality should provide certain features aligned with the training objective. In the case of DMA-SR, police officers need to experience stress during the training. Therefore, the VR system needs to elicit a certain level of stress. In accordance with the seminal work of Yerkes and Dodson [57], learning takes places at an optimal level of stress following an inverted U-shaped continuum. Thus, to train DMA-SR, VR needs to be able to elicit moderate levels of stress in police officers to enhance learning. Because there are individual differences in the perception and experience of stress, VR scenarios should be adjustable in such a way that police trainers can manipulate the stress levels of trainees in real-time; for instance, through the introduction of additional stress cues in the scenario.

Training delivery is heavily dependent on the didactical approach a police agency and their police instructors take to training [8]. Historically, police training relied on teacher-centered approaches (e.g., the teacher demonstrating a skill in front of the class) rather than a learner-centered approach (e.g., the learner performing and practicing the skill) [58]. More recently, police agencies seem to shift to a training approach that puts the learner at the center [8]. VR training may be particularly supportive of this pedagogical approach since it offers trainees the opportunity to be actively involved in their learning process. To this end, police agencies can take advantage of the implementation of VR as a new training tool that follows a didactical approach in line with the learner at the center of training.

Hutter and colleagues [48] have developed didactical criteria for high-quality training of police officers. The criteria were initially developed for real-life practice but can be translated to and implemented in VR training. To do so, the didactical criteria described in

Table 2. Didactical criteria and technical guidelines.

Didactical Criterion	Technical Guideline
Clear assignment	Training objective alignment
High quality instruction	VR instructional tutorial
Well-designed practice situation	Scenario design
Model learning	After-Action Review
Variation and differentiation	Scenario editor
Self-regulation of the learning process	Selection of tools
Constructive, motivating feedback	After-Action Review

were used to conduct systematic observations of VR trainings. As a result, clear links between a didactical criterion and the resulting technical guideline could be established [49]. The technical developments that follow the didactical guidelines are described in detail in the sections to follow. In general, the didactical criteria described in Table 2 provide the foundation and guidance for the development of technical features to create an effective and efficient VR training tool for police practice.

When structuring and designing a training session with VR, various considerations should be taken into account such as training duration, scenario length, and repetitions of scenarios. First, based on the training sessions conducted within the field trials, a VR training session should be scheduled for at least 90 min to allow for sufficient scenario execution time in VR. Ideally, a VR session should be scheduled for 120 min to allow ample time for the AAR and account for potential delays in set-up and calibration. Depending on the state of the trainees (not too drained, overloaded, or cyber sick) instructors should vary the length of the scenarios, the actual time spent in VR, and the length of breaks between scenarios (e.g., time spent on the AAR or preparing for the next scenario). Particularly for first time users, gradual, incremental exposure to active execution time in VR with sufficient breaks between intervals provides effective familiarization with and adaption to the VE [59].

VR offers the benefit of having multiple scenario repetitions in quick sequences. Due to the newness of VR as a training tool in police, and the high likelihood of police officers to experience VR for the first time, measures should be taken to reduce the occurrence of cybersickness as much as possible. Therefore, the VR training scenarios should be designed in such a way that the scenario length coincides with a gradually increasing exposure time to the VE [59]. Hence, the scenario repetitions should be structured in such a way that trainees start with short scenario sequence, have sufficient recovery time before re-exposure and then begin the next, slightly longer scenario sequence. Within the SHOTPROS project, we have structured a VR training session to contain an initial VR familiarization tutorial followed by three operational training scenarios that built on each other in increasing length of VR exposure, content of training tasks and task complexity.

3.1.4. Scenario Design and Live Editor

The possibility to create a wide range of VEs and scenarios is one of the most essential advantages of VR training. The choices for scenarios are essentially limitless, ranging from simulating a domestic violence call to handling an active shooter situation in a school or office building or dealing with agitated and confused mentally ill people in realistic environments.

The scenario design should begin with defining the training objective of the training session. Once the training objective and area have been defined, the expertise of the police trainers should be utilized to design relevant on-duty situations. In accordance with the didactical criterion of a well-designed practice situation, the scenario should present a realistic problem, invite realistic stress and provide trainees with the opportunity to find realistic solutions [48]. Once these initial steps have been considered, the technical development of the scenario can begin.

Setting up a new scenario requires several steps and can be divided into four layers (Figure 2). In the SHOTPROS system the creation of the overall virtual environment is split into two tools. The Terrain Editor is used to model the terrain, streets, buildings, open fields or replication of real-life locations. This part requires a 3D modelling expert and is expected to be supplied by the VR platform developer or an external agency. With the Live Editor, the details in the buildings and rooms are enriched, NPCs are added and their behavior is specified including trigger zones. In the SHOTPROS VR solution the Live Scenario Editor is also used by the training operator to engage with trainees during the training. NPCs can be steered in real-time and stress cues can be added or removed. This feature was highly appreciated during the field trials as it increased the realism of a situation and provided the opportunity for realistic, responsive behavior of NPCs. Operators ideally have a wide range of avatars (children, elderly people, physiologically disabled people) and environments (greenery, construction sites, lively city areas, weather conditions) at their disposal to offer scenario variation and resemble the real world as close as possible. The importance of realistic, immersive experiences with correct kinematic movement and ballistics, e.g., ability to shoot through windows, has also been highlighted throughout the project.

3.2. Realism and Immersion

3.2.1. Movement

DMA-SR training is very physical and thus realistic movement plays an important role. Hence, locomotion should be as limitless as possible. Locomotion refers to the technology that enables movement from one place to another within VEs and is an extremely important factor when it comes to the level of embodiment and immersion in VR training [60]. It can be distinguished between physical (actual walking) and artificial locomotion techniques.

For DMA-SR training physical locomotion is the preferred option, ideally in a 1:1 simulation of the scenario to the real-world (1 m walked represents 1 m in the VE). Most VR solutions have some sort of restriction when it comes to the size and space of the VE. Therefore, VR scenarios should be designed to make this restriction seem as natural as possible by, for example, adjusting the size of a virtual building or room to the actual training space available. However, natural locomotion in VR is only possible in full-body VR solutions and comes with both technological challenges and space requirements of training facilities. Artificial alternatives, such as controller-based “floating” or teleportation, have been developed and improved over the past decades [61,62] but should not be considered for tactical police training, where natural movement is such an important part of the training. Within the SHOTPROS project both artificial locomotion options have been tested in a compact version of the system. For most LEAs it was not a feasible option for training, because body contact within teams is of high importance and needs to be replicated in VR. A considerable amount of test subjects have also experienced motion sickness and reported very low level of immersion in both artificial locomotion options, floating and teleportation.

3.2.2. Spatial Sound

Spatial sound in VR is incredibly important and should not be neglected when developing VEs or evaluating different VR systems. Sound is an important factor for overall immersion but also to localize an event source or stress factor. Especially for police officers it is one of the senses used to identify potential threats.

3.2.3. Physical Props

The SHOTPROS tactical belt is an adaptation of a physical tactical belt and is a significant innovation in the latest generation of virtual reality training equipment for law enforcement. Physical props that realistically represent (in size, weight, haptic) gear police officers wear in real life are crucial in scenario-based training. Furthermore, in order to train DMA, it is important that trainees can choose from all possible tools that are available to them in real-life situations and are not restricted to solely shoot/no shoot options. The SHOTPROS belt (Figure 3) is equipped with the following components:

• Handgun: realistic model (size, form and weight as well as realistic behavior regarding trigger pulling and changing of magazine).
• Pepper spray: including realistic functionality (infinite spraying capacity or need to re-fill after a certain amount of usage) and effect (if used inside a room, the trainee will have limited visibility for a short amount of time and NPCs need to react accordingly e.g., hands in front of their eyes).
• Electroshock gun: this tangible device fires a single or a double shot (trainers can select mode for each trainee or on group level).
• Baton: length of a retracted baton that can be virtually expanded.
• Handcuffs: handcuffing an NPC is still a challenge because trainees don’t have the physical arms to place them. Currently, they need to be held towards the opponent’s hands until a “success” message is displayed.
• Flashlight: including an on/off switch to which the VE and its lighting reacts accordingly.

Introducing virtual equipment into a VR training system requires the development of a tutorial to explain and give trainees the opportunity to practice using the equipment before entering actual training scenarios.

3.2.4. Multisensory Experience

The majority of our daily life experiences are multisensory in nature, consisting of what we see, hear, feel, taste and smell. There is a lot of opportunity for making VR training experiences more realistic and threats more stressful by incorporating multi-sensory components such as heat, wind, pain, and scent.

A variety of experiences have been tested in the project and published [20]. Pain stimuli are recommended for increased behavioral and psychological realism because they intensify dangerous situations in VR and cause trainees to behave more realistically. It was discovered, for example, that trainees kept their distance from a perpetrator much better when they knew a light shock could be administered if the perpetrator shot or stabbed them. Scent administered by a specialized olfactory device for VR goggles (https://ovrtechnology.com/ accessed on 15 December 2022) provides context when analyzing the VE and therefore increases the perceived realism of a situation. For example, a puddle of liquid was enhanced with the smell of gasoline, a fire with the smell of smoke, and a perpetrator with the smell of sweat and alcohol.

3.3. Responsive Worlds

3.3.1. Multi-User and Interaction Modalities

All participating LEAs reported conducting scenario-based DMA-SR training in teams of two to four trainees. Therefore a VR solution should support multi-user training with realistic communication options (including communication with external actors like dispatchers) and sensory modalities that go beyond the visual and auditory. Interacting with the virtual environment or other trainees in the scenario still poses some challenges with current technology. To be able to use nonverbal forms of communication commonly used by police officers during operational situations, such as touching team members on the shoulder, tracking precision must be enhanced. For example, during the field trials, trainees sometimes grasped into space when anticipating their teammate, leading to uncertainty and reduced immersion. This issue exists with various tracking systems and must be rectified for a more realistic haptic and social training experience.

Tangible or physical interaction in VE has enormous potential because of the opportunity to provide rich haptic cues that are frequently neglected in consumer VR experiences. In VR, users frequently manipulate virtual objects using generic controllers, which reduces immersion. Using tangible interfaces makes interaction with the virtual world feel more natural, intuitive, and realistic. Furthermore, when the tangible interface is an exact replica of a virtual object, training exercises accurately reflect day-to-day working practices and trainees can virtually repeat tasks with physical representations of their real tools, developing muscle memory that aids in learning retention and that is transferable to real life.

3.3.2. Non-Player Characters

Computer-generated Non-Player Characters (NPCs) play a crucial role in VR training for first responders. All aspects of interpersonal communication and engagement, including voice (e.g., volume, tonality), facial emotions (e.g., friendly, furious), posture and movement, gesturing with the arms and hands, and maintaining or not maintaining interpersonal distance, are deciding elements in how realistic NPCs appear to the trainees. In real-life scenario training, role-playing is often time consuming, costly, and cannot cover the diversity in characters (e.g., age, gender, appearance, disabilities, ethnicity, reactions, behaviour) relevant for successful and realistic training.

The ideal NPC would have realistic facial expression, speech recognition, language processing and ability to react to trainees’ movements. However, off-the-shelf solutions are not that advanced yet [63]. An alternative work-around, that has been tested in the SHOTPROS project, includes a predefined set of reactions that is applied to the NPCs ad-hoc by the operator or trainer during the training (e.g., run towards the policeman, attack, raise hands, get on the knees, lay on the ground).

To go one step further, the addition of scent to increase realism has been tested and evaluated in several field trials. An offender portraying a homeless person might, for instance, smell like alcohol and urine. End-user feedback clearly indicates that adding scent can significantly improve the experience and help officers in the process of identifying the level and kind of threat an NPC represents.

3.3.3. Role-Player Characters and Live Interactions

Role-player characters can be used as an addition or alternative to NPCs. A role player-character is a virtual character whose actions, behaviour and communication are controlled by a real person. This feature offers the opportunity to combine the advantages of a virtual character (visualisations of a variety of different person characteristics) and realistic behaviour and communication (executed by the role player). They can be used to steer scenarios, escalate situations and act as additional stressor for trainees. Feedback given by trainees who trained scenarios with and without role player-characters emphasized the strong influence they have on the realism of the training scenario. Trainers can also participate in the VE by either temporarily taking over virtual avatars (thus acting as role-players), wearing a trainer vest (as they would in real life) or in a ghost mode (invisible to trainees) so they can follow the action live without interfering in the training.

3.4. Beyond Novelty—Human Factors and Performance

3.4.1. Stress Factors for Police Officers

Standardized stress tests, which have been used by researchers to manipulate stress levels accurately and consistently, provide insight into how to produce stress in a laboratory environment [64,65]. However, research investigating the manipulation of stress levels in VR training has highlighted the importance of using contextual stressors that are relevant to real-life situations [30,66,67]. LEAs participating in this project further highlighted the importance that potential stressors used in the VE are relevant to their work and resemble stressors that occur in the day-to-day operational environment. To identify the most important stressors for police officers, in their line of work, a list of contextual stressors was collected through end-user workshops [19] ranked and later developed into the following audio-visual stress cues in VR: (1) a stranger suddenly walking into a room, (2) a person holding a handgun, (3) rocks falling from a building very close to participants, (4) finding an injured person at a crime scene, (5) a child crying at a potential crime scene, (6) an unidentifiable scream at a crime investigation, (7) a door suddenly closing with a loud noise, (8) smoke coming from an apartment, (9) the light suddenly switching off at a potential crime scene, (10) a loud scream during an apartment building search, (11) an aggressively barking dog, (12) being filmed during an investigation in a day scenario, (13) being filmed during an investigation in a night scenario. The ecological validity of these stress cues was analyzed in a study with 22 police officers (8 female, 14 male) ages 22 to 51 (Mean = 33, SD = 12) and analyzed based on subjective ratings (Visual Analogue Scales (VAS) stress and anxiety). A VR prototype was developed that exposed participants to the above mentioned stress cues while searching an apartment building (scenario 1) and investigating an outdoor crime scene (scenario 2). All participants where exposed to the stress cues in the same sequence. This can be considered as a limitation of the study. In future studies we will create a prototype that allows for randomization of the appearance of each stressor. The descriptive results are presented in

Table 3. VAS (stress and anxiety) for individual stress cues.

	Subjective Measurements
		Stress			Anxiety
Stressor	n	Mean	SD	Rank	Mean	SD	Rank
Stranger	22	41.45	26.24	1	28.41	23.94	1
Weapon	22	38.18	19.27	2	26.23	16.71	2
Falling Rocks	19	32.32	22.06	3	18.05	15.11	3
Injured	22	27.91	18.72	4	17.95	11.25	4
Crying Child	22	27.41	23.88	5	13.00	20.09	7
Scream (Night)	22	25.41	17.39	6	13.00	11.54	6
Door Closing	22	24.05	22.58	7	17.14	21.17	5
Smoke	22	23.09	18.52	8	11.82	13.04	10
Dark Room	22	19.45	20.69	9	12.59	14.55	8
Scream Indoor	22	17.68	14.97	10	9.82	9.64	11
Dog	22	17.18	12.73	11	11.86	9.71	9
Photo (Day)	22	16.59	15.94	12	5.68	5.40	13
Photo (Night)	22	14.73	15.56	13	5.95	5.64	12

. Furthermore, physiological responses to each stressor have been tested based on heart rate variability (HRV) and heart rate (HR) (see Figure 4). The results show initial indications that the stressors can be realistically implemented in VR and have a high probability of eliciting the desired stress response in the trainees.

These stress cues, based on known real-world stressors for police officers, can be implemented to a stress cue repository and used for VR training for DMA-SR as described by Nguyen and colleagues [19].

3.4.2. Stress Dashboard

As stress plays such an important role in DMA-SR training, trainers would benefit from the possibility to monitor trainees’ stress levels during the training and have them recorded to be included in behavior analysis in the debriefing. A solution called Stress Dashboard has been developed for the SHOTPROS system and tested in the field trials. It includes (a) Stress Level Assessment panel, (b) Stress Cue Control panel, and (c) stress level indicators in all live VR view options.

In the literature HRV has been identified as a valid indicator of real-time stress levels [68,69] and has the advantage over other bio-signals (e.g., EDA, EMG and EEG [70]) that it can be measured via a consumer-friendly ECG chest belt. In the SHOTPROS project the Zephyr™ BioHarness™ (chest strap) (https://www.zephyranywhere.com/ accessed on 15 December 2022) has been used and tested for this purpose [71]. Throughout the field trials the device has proven itself as a reliable sensor that is relatively easy to put on, comfortable to wear during the training and relatively resistant to movement artifacts. For the stress indicator to work, an individual baseline (2 min) needs to be recorded before the training [72]. To assess each trainee’s stress level in real-time throughout the training in the SHOTPROS system, a 2 min baseline measurement exercise at the beginning of the training has been built in. During the training, 30 s moving averages intervals of HR and HRV (based on the RMSSD method [68]) are compared to the trainee’s individual baseline and weighted according to our stress score based on a combination of HRV and HR. The resulting value is classified into one of four categories: (1) normal, (2) increased, (3) high and (4) very high. If no values are available due to technical issues, this is indicated by a grey colour and a red cross). Each trainee’s stress level is visualized in the Stress Level Assessment panel and all available live VR viewing options as icons above each trainee (see Figure 5).

To give trainers the opportunity to not only monitor but also manipulate stress levels during the live training, a Stress Cue control panel has been developed (see Figure 5). Pre-defined stress cues can be added to the training either ad-hoc with instant playback or time-controlled via a time axis (e.g., a dog starts barking or a phone ringing). End-user requirements indicated that this feature needed an easy-to-use interface design that required little mental effort from the trainers because of the multitude of tasks they have to fulfill during a live training with multiple trainees. For VR DMA-SR training, it is important to be able to increase the level of complexity of a scenario by adding augmentations and stressors. A content pool with a variety of relevant assets and animations should be available to choose from.

3.4.3. Performance Monitoring and Management

Performance monitoring is an innovative element of VR Training. Real training sessions with multiple trainees make live performance evaluations difficult. In VR every movement and performance outcome is recorded and can be reviewed in the After-Action Review (AAR), where movements, tactics and behaviours can be tracked and evaluated. Sections in the training session can be directly selected, paused or repeated, the perspective can be changed and evidence-based feedback, supported by stress measurement and Key Performance Indicators (KPIs), can be provided.

Key Performance Indicators. Several LEA workshops have highlighted displaying and measuring a selection of KPIs as a priority [17]. Experienced police trainers identified the most important KPIs to gain a better understanding of the trainee’s behavior and analyze training performance [39]. This feature was implemented, in close collaboration with LEAs, into a Real-time In-action Monitoring panel (Figure 6) and After-Action Review (Figure 7).

Real-time In-Action Monitoring. Effective DMA-SR training not only provides feedback at the end of a training session but also integrates feedback throughout the training session. In-action monitoring is the ability, for the trainer to live-view the training scenario with a real-time update on pre-defined KPIs and the trainer’s didactical view (Figure 6). The current SHOTPROS solution includes the following KPIs:

• Physical positions, motion and pose of trainees and NPCs
• Trainee walking paths
• Line of sight
• Field of view
• Firing events
• Firing lines
• Impacts
• Radio chatter and other sound

After-Action Review. Events are automatically bookmarked by the system or can be added manually to efficiently navigate through the debriefing process and focus on important decision-making points. Each behaviour can be examined in relation to the environment and the trainee’s stress level, allowing to pinpoint the cause of a behavior more precisely and in context (e.g., signs of avoidance behavior, freezing, and hesitation). In addition, AAR provides an excellent opportunity to save critical training data in order to track progress and performance over time. However, for a successful deployment, legislators and LEA representatives must evaluate data anonymity, short-term vs. long-term storage, accessibility, and visibility of individual data to others. As VR offers additional data and review options of performance, there may be an extra responsibility for the instructor and the participants to ensure psychological safety in the training and review environment and to use physiological data in a responsible and ethical way.

3.4.4. Ethics

There are four domains that seem of particular ethical relevance when designing a VR program for police training, although these domains are not exhaustive: (1) users’ and psychological well-being, (2) ethics in VR (and scenario) design, (3) user data protection, and (4) prevention of possible misuse, abuse or dual use.

Users’ psychological well-being. As VR training fully immerses trainees into scenarios, with the goal to elicit similar cognitive, behavioural and affective responses as would occur in real-life situations, it might also induce real averse emotions in trainees [73]. Several studies have shown that even though people know that the experience in VR is not physically real, it can feel psychologically real [74,75], especially since VR technology is increasingly moving towards ’superrealism’ [74]. Similar to the possible short- and long-term psychological effects police officers can endure after experiencing stressful and high-risk real-life situations (e.g., anxiety, trauma, PTSD), real emotional pain can thus also occur after virtually experiencing such situations [73,74,76,77]. As the explicit goal in DMA-SR police training is exactly to train police officers in how to perform optimally in such stressful and threatening operational situations, they will be exposed to potentially traumatising situations that can cause emotional backlash. Therefore, it is important to use a didactical framework to design effective learning scenarios and to not just deliberately expose trainees to an overload of hardship, just because it is possible. Furthermore, police trainers should pay attention to physiological reactions during the training that might indicate high emotional discomfort. After an intensive training, trainers should also always make sure how the trainees are feeling and to provide psychological support should this be necessary.

Ethics in VR design. Human behavior can be (even unconsciously) influenced by external factors, including by technology [76]. Some research suggest that, for example, VR can be used as a beneficent application to reduce implicit ethnic bias, by experiencing a situation in the skin of a person with a different ethnicity [78]. However, this means that the opposite might also occur [74]. That is, that continuous negative interactions with a person from a certain ethnicity (e.g., always depicting them as offenders), may increase the implicit bias towards the entire ethnic group [74,76] and thus alter trainees’ views of actual individuals or groups [79]. Therefore, developers should make sure that trainers have a large and diverse repository of skins/avatars available to be used in scenarios. But equally important, trainers should continuously reflect on the choices they make in selecting avatars for their scenarios and to diversify as much as possible and avoid stereotyping the types of avatars they use for different types of characters (e.g., offender, bystander, victim) [80,81]. Furthermore, attention should also be given to offering a diverse set of skins for the trainees themselves, differing in gender, age, race, and appearance. An important part of police training involves the improvement of team performance. Police officers rarely do interventions individually, but mostly in teams of two to four (and sometimes even more). It is important that they get to know their ’partners’ and learn how to work together as a team. Thus, for the learning objectives, it is reasonable to assume that trainees should be able to ’recognize’ and get acquainted with their colleagues. For that reason, it might be advisable to choose avatars that relate as closely as possible to the real trainee, and for trainees to systematically use the same avatar in their trainings, although no research has specifically focused on this topic yet. Brey [81] does note that the avatar chosen can function as a manifestation of the user itself (in appearance, behavior and responses), or as a character that has no direct relation to the user and merely plays a role. To be able to maximally transpose virtual learning situations to real-life situations, it might be better to stay as close as possible to the real identity of the trainee.

User data protection. During training sessions in VR, a tremendous amount of data is collected from each trainee, such as personal data (e.g., body measurements, gender, trainee name), physiological data (e.g., HR, HRV), haptic and tracking data (e.g., routes taken, distance to others), audio-visual data (e.g., audio recordings) [82,83]. Many data are necessary to collect, either for the VR system to be able to function properly (e.g., height of trainee and tracking of movement to correctly depict the trainee’s avatar) or to meet the learning goals of the training (e.g., HR and HRV to calculate stress level for DMA-SR training). Still, the principle of ’data minimization’ should be followed: only relevant and necessary data should be collected [84]. Pertinent questions about the protection of the data collected from trainees will highly depend on the choices LEAs make regarding their desired functionalities of the system. For example, if a LEA chooses to delete all the data after each training session, there will be limited issues regarding data storage, protection and access. However, a LEA could also opt to make full use of the VR training system and allow trainers or trainees to keep (individual) performance records, to re-examine training sessions, assess learning curves or set personalised training goals. In that case, it is advised that there is a thorough and transparent discussion between technology provider and end-user about (a) where the data will be stored (e.g., at the premise of the end-user or on servers of the VR-provider), (b) how the data will be secured (e.g., encryption, password protection), (c) if there is software or hardware used that also sends certain data to third parties, (d) who is authorized to access the data (e.g., trainers, individual trainees, police management, ICT-personnel), and (e) what the data may be used for (e.g., solely for training purposes or also for research, performance reviews, lawsuits against police officers) [85]. All these elements can be written down in a Data Processing Agreement between the technology provider and the LEA. It is also recommended that LEAs should get informed consent from their trainees regarding the data that will be collected from them, how they will be protected, and what they can or will be used for.

Prevention of possible misuse, abuse or dual use. VR is considered a persuasive technology. It has the capacity to influence or modify behavior, values or attitudes of people [73]. In a training context, we want it to be persuasive, as we aim to change and improve DMA behavior. However, persuasion can also be used for malicious purposes [74]. If not based on a proper didactical framework, police training in VR might unintentionally result in teaching police officers wrong procedures or behaviors (misuse), which is of course also possible in real-life training situations. However, if such a training system falls into the wrong hands, there is also a risk that the possibilities of the system would be used for unethical or illegal purposes. For example, a person might use the system to prepare for a crime they want to commit (abuse). Similarly, it could also be used by military organisations or governments for purposes that the system was not intended for, for example, to train torture techniques or to prepare for situations to repress a particular group of civilians (dual use). Necessary safeguards should thus be in place to maximally prevent such misuse and abuse. Not only should the data ’inside’ the system be protected, but the system itself should also be secured so it cannot be hacked or used by unauthorized people.

Developers thus have a clear responsibility to consider ethical aspects in their design process and to think and communicate about possible effects to potential users in advance [81]. However, also police trainers who use the system for training (future) police officers, have an equally important ethical responsibility for the well-being of their trainees and for achieving correct and safe learning experiences.

4. Future Work

Although the solution developed within the SHOTPROS consortium is currently one of the most advanced VR solutions for training DMA-SR, four topics have been identified and frequently discussed as opportunities to further advance training solutions of that kind.

4.1. Inclusive Performance Management System and Training Personalization

Moving away from the “one size fits all” approach, VR training is also shifting its focus to consider individual needs, interests and capabilities [86]. By adapting the learning experience to a trainee’s skills, fitness levels, previous experience and current role, individuals can train scenarios that challenge them on the right level. Their track record should be recorded in and connected to the overall performance management system and not in isolation. Benefits include:

• Higher engagement through challenging yet accomplishable scenarios
• Better retention through enough repetition and training of the right difficulty level
• Targeted training to rank, location and experience level
• Motivation to pursue learning
• Trainee satisfaction

For VR training within LEAs this would require a centralized data storage system that is connected to the VR training system and ideally also includes personal information on training history (not limited to VR), experience level, skills and gaps. Aspects of data storage and privacy regulations appropriate for the LEAs individual country and standards might differ from one to another and need to be carefully considered prior to adoption.

4.2. Artificial Intelligence and Machine Learning Enhanced VR Training

The role of artificial intelligence (AI) and machine learning (ML) has been explored within the SHOTPROS project in terms of feasibility and potential for application. Three areas were identified as helpful: (a) performance monitoring and insights, (b) real-time adaptive VE and (c) scenario design. Trainers could be supported with insights on behavior and meaningful correlations between variables captured during training (e.g., stress level, field of view, attention) and visualized during the training or in the AAR. With artificial smart scenario control, VE could be automatically adapted to trainees’ real-time performance and physiological state. For example, trainees in low stress levels could be exposed to additional stress cues but also be protected from over-exposure, which may compromise the self-esteem of the trainee and result in potentially life-threatening mistakes while on duty. AI could be used to create completely new and personalized training scenarios based on the training requirements selected by the trainer or trainee themselves. However, to engage AI and ML in VR training a trustworthy relationship and collaboration between the technology and the trainer needs to be established. Currently that is not the case and European police trainers are very critical about its usage [87], which has only been investigated on a theoretical level so far. In future work we are planning to implement such features and test them in field studies to get a more concrete understanding of its potential.

4.3. High-End Content Streaming

The majority of current full-body VR solutions require users to wear a backpack that holds a computer necessary to power the VR headset. With the development of high-end streaming directly into the headset, trainees would be able to move more freely and stand back-to-back, without having the backpacks in-between them, a limitation that has been discussed frequently with the current SHOTPROS solution. The current SHOTPROS solution works via a local WiFi network but in the future such training solutions would benefit from a connection to a cloud-based library, that would also provide the opportunity to share best practise scenarios amongst LEAs.

4.4. Multi-Sensory VR Experiences

Multi-sensory stimuli enable the detection of environmental threats and context to situation awareness, that are otherwise hard or impossible to portray through audio-visual stimuli alone. Scents of dangerous gases, liquids or smoke are important factors when assessing the threat of a given situation, which is made possible with multi-sensory VR. Although some “wizard of oz” solution have been tested and its impact evaluated within the SHOTPROS project [20], further research needs to be conducted on (a) technical framework of implementation, operation and potential automatization, (b) their ecological validity (c) impact on immersion (d) impact on DMA-SR and (e) trainees’ safety. In future work we are planning to investigate the use of a robotic platform that could be operated on three levels of user involvement and control (a) manual control, (b) pre-defined triggering of multi-sensory cues, similar to current operation of NPC behavior or (c) integrated into the VR solution with fully automated triggering based on trainees’ location or vicinity to objects and virtual agents.

5. Conclusions

This article summarizes findings from a three-year project investigating the influence of psychological and contextual human factors (HFs) on the behaviour of decision-making and acting (DMA) of police officers under stress and in high-risk operational situations in VR. We highlight the most important aspects of VR training solutions such as scenario design, realism and immersion, interaction modalities of responsive Virtual Environments, performance management and ethical considerations. Furthermore we cover the importance of considering and integrating real-world training practices such as clarity of training objectives, relevant human factors and the theoretical foundations of DMA. Training DMA in VR has several advantages over real-world training including the possibility to create a variety of scenarios and interact with trainees through avatars that can take on a variety of appearance and personalities. Another characteristic of VR training that surfaced as a major advantage is objective measurement of performance (e.g., line of sight, firing events) and psycho-physiological factors such as stress. The AAR including these measurements was considered a major advantage over current training practices. However, VR training is not the best solution for all training areas and it is important to select appropriate training objectives, follow didactical principles and ensure trainee’s have the right equipment (e.g., tactical belt). The training areas considered most beneficial from VR training are “tactical training” and “perception and acting training” because they require varying locations, scenario context and trainer’s can manipulate the VE on the fly as needed. Overall studies of this project have highlighted the great opportunities the inclusion of VR can bring to the training industry.

VR training is not yet widely used by LEAs, and organisations are still struggling with the question of where to start on this topic. This article provides guidelines on the different steps necessary, such as requirements definition, training objectives, technical consideration and evaluations methods. Although a very specific application area (DMA-SR of police) was described in this article, considerations and study methods will apply to a variety of other application areas and contribute to the development and testing of the use of VR training in related domains. Furthermore, the human factor model and its application described may be relevant to other types of simulation training. Learnings from this project can be transferred to a variety of domains exposing users to stressful situations. As VR technology continues to advance and becomes more widely adopted, it is likely that we will see an increase in the use of VR in training across a wide range of industries.