Ranking Crossing Scenario Complexity for eHMIs Testing: A Virtual Reality Study

Abstract

External human–machine interfaces (eHMIs) have the potential to benefit AV–pedestrian interactions. The majority of studies investigating eHMIs have used relatively simple traffic environments, i.e., a single pedestrian crossing in front of a single eHMI on a one-lane straight road. While this approach has proved to be efficient in providing an initial understanding of how pedestrians respond to eHMIs, it over-simplifies interactions which will be substantially more complex in real-life circumstances. A process is illustrated in a small-scale study (N = 10) to rank different crossing scenarios by level of complexity. Traffic scenarios were first developed for varying traffic density, visual complexity of the road scene, road geometry, weather and visibility conditions, and presence of distractions. These factors have been previously shown to increase difficulty and riskiness of the crossing task. The scenarios were then tested in a motion-based, virtual reality environment. Pedestrians’ perceived workload and objective crossing behaviour were measured as indirect indicators of the level of complexity of the crossing scenario. Sense of presence and simulator sickness were also recorded as a measure of the ecological validity of the virtual environment. The results indicated that some crossing scenarios were more taxing for pedestrians than others, such as those with road geometries where traffic approached from multiple directions. Further, the presence scores showed that the virtual environments experienced were found to be realistic. This paper concludes by proposing a “complex” environment to test eHMIs under more challenging crossing circumstances.

1. Introduction

Connected and automated vehicles (CAVs) have the potential to drastically reduce the number of road fatalities and improve traffic flow by avoiding errors and delays caused by human behaviour [1]. In fact, as opposed to human drivers, CAVs will be capable of accomplishing the driving task with high precision and will not be exposed to common risks such as fatigue or distraction [2,3]. This would particularly benefit vulnerable road users, especially pedestrians, whose deaths represent more than one fifth of road fatalities each year [4]. However, the successful deployment of CAVs within traffic networks depends greatly on users’ acceptance towards this new technology as well as on the prediction and mitigation of potential unsafe behaviours among road users. Recently, there has been an emergence of literature around pedestrian–AV interaction, particularly studies which have looked at external human–machine interfaces (eHMIs) to ease communication between pedestrians and automated vehicles in the absence of a human driver. Research has shown that eHMIs applied on an automated vehicle can have positive effects, e.g., increases in pedestrians’ trust [5], acceptance [6], and clarification concerning the vehicle’s intentions and behaviour [7,8,9]. Additionally, eHMIs have been reported to influence pedestrian crossing behaviour, e.g., leading to shorter crossing decision times [10,11]. At present, most of the existing research investigates the efficacy of eHMIs in relatively simple traffic settings, i.e., a single pedestrian crossing in front of a single eHMI. A few studies only have looked at eHMIs in more complex scenarios. For example, Dey et al. (2021) [12] evaluated different eHMIs designed with two pedestrians to address the question of scalability to multiple road users. The study showed that traditional eHMIs might not be suitable to communicate the vehicle’s intention to the intended user. Another study [13] investigated the effect of pedestrian group behaviour as well as vehicle autonomy levels and AV–pedestrian interfaces on pedestrians’ crossing decisions. The results showed that considering more complex traffic conditions can provide useful insights into suitable interface design and pedestrian crossing strategies. Hoggenmüller et al. (2022) [14] evaluated an eHMI for automated vehicle–pedestrian interactions in complex scenarios where multiple road users shared the same space, using 360° recordings. Their findings emphasise the significance of evaluating eHMIs more comprehensively in complex urban situations to design information that is relevant, timely and properly targeted to the intended user. Lee et al. (2021) [15] investigated the effect of eHMIs on pedestrians’ behaviour on a two-way two-lane road and found that pedestrians exhibited less cautious behaviour when an eHMI was displayed. A study by Bazilinskyy et al. (2021) [16] examining the effect of eHMIs on pedestrians’ perceived safety looked, among other elements, at whether the presence of a billboard might divert pedestrians’ attention from the eHMI. No significant differences were found between the distraction condition and the condition without a distraction. However, the level of visual distraction investigated was kept to a minimum, and the authors recommended the need for further research in more distracting environments.

These studies provide some valuable progress towards a more holistic evaluation of AV–pedestrian interactions. However, at present, the range of potential pedestrian–AV real-life interaction scenarios investigated remains limited. It is acknowledged that urban traffic interactions are a complex and dynamic process [17], where different aspects are at play. There is evidence that elements which make up a traffic environment such as the road structure and the presence and behaviour of other road users can have an impact on a road user’s expectations [18], which in turn affect their evaluation of the situation [19] and their decision-making in traffic [20]. A link also seems to exist between traffic setting complexity and pedestrian crashes. Research suggests, in fact, that crashes involving pedestrians tend to happen at locations with a higher degree of road configuration complexity, such as junctions and intersections where the traffic approaches from different directions altogether and with more than one lane and wide roads [21]. The visual degree of complexity (visual cluttering) of the environment, created for instance by the presence of advertising, can lead to reduced visual attention [22,23,24], as can the presence of distractions such as a phone call. Studies carried out in a controlled virtual environment demonstrated that distracted pedestrians who are engaged in a phone call or who are texting display a reduced awareness of the situation and attention to the oncoming traffic, with higher chances of being hit by a vehicle [25,26,27,28,29]. In accordance with Thomson et al.’s (1996) [30] description of the stages of a successful road crossing, a pedestrian must be able to focus their attention on relevant details in the road environment and switch their attention efficiently between them while timely blocking out distractions. If the complexity of the surrounding traffic environment increases, a signal sent by an AV may be missed or misinterpreted, thus losing its efficacy. Eisma et al. (2020) [31] suggested that pedestrians may, for instance, rely more on their peripheral vision, without continued visual attention towards eHMIs in a complex traffic environment. Additionally, the presence of multiple AVs with eHMIs may “increase the complexity of traffic constellations” [32] (p. 80) even further.

Although there is a general acknowledgement of the variety and intricacy of factors which impact traffic interactions, it is not an easy task to fully capture real-world complexity in AV–pedestrian research. A major challenge comes from the human component of the interaction, i.e., the pedestrian. If, on one hand, the AV is governed by the basic principle of collision avoidance, pedestrians’ behaviour can be erratic and unpredictable.

Pedestrians have their own abilities, limitations, characteristics, perceptions of the surrounding environment, walking attitudes and related preferences, which all contribute to shaping their crossing decision-making. For example, age differences are known to affect crossing decisions and behaviour, with older and very young pedestrians showing a tendency to adopt unsafe crossing styles dictated by perceptual, physical and cognitive limitations [33,34]. Gender differences are also a well-documented determinant of pedestrian behaviour. A considerable amount of literature has found that male pedestrians engage in riskier behaviours than females [35,36,37]. Individual characteristics such as different perceptions of risk and risk-taking propensity, shaped among other factors by the built environment [38], are recognised to impact crossing behaviour [39]. The greater unpredictability of pedestrian behaviour can also be attributed to lower compliance with traffic regulations compared to drivers. Pedestrians tend to compromise between convenience, time pressure, comfort and safety when making decisions [40,41], which can result in disobeying the rules in favour of more advantageous crossing strategies, e.g., crossing diagonally and making dangerous actions to save time [42]. Moreover, pedestrians often make subconscious decisions that are difficult to explain or measure [43].

When designing automated vehicle interactions with pedestrians, a holistic approach should be followed that accounts for pedestrian behaviours and perceptions within the broader context of the interaction. In other words, looking at the pedestrian-perspective side of the interaction and investigating their behaviour in different and more or less (from the pedestrian viewpoint) lifelike complex crossing scenarios. By considering contextual factors early in eHMI development stages, safety challenges can be identified and tackled appropriately [44], and AV behaviour can be designed to adapt and respond efficiently to traffic interactions.

1.1. Simulating Complex Traffic Environments

Investigating AV–pedestrian interactions in complex traffic environments, especially when the full crossing task is accomplished, can be challenging. Safety is an issue, as there is a risk for actual collisions to happen when humans interact with real vehicles experimentally, thus endangering the participants’ wellbeing. Additionally, there is the problem of controlling and isolating single variables in the environment, e.g., traffic density and presence of other road users. Simulated environments offer a tool to overcome some of these obstacles. Virtual reality (VR) can be used to re-create immersive, interactive and risk-free experiences where different elements of pedestrian behaviour can be measured systematically. Additionally, virtual reality is highly flexible, thus making it possible to alter different elements of the scene relatively quickly and at a low cost. An increasing number of studies employ virtual reality to assess different aspects of pedestrian crossing decisions and behaviour, as well as elements which pertain to the user experience, in the field of AV–pedestrian interactions [45]. For example, ref. [46] investigated how pedestrian crossing intentions were affected by the appearance of an AV coupled with an eHMI in a VR experiment. In this study, previously recorded 360° videos were shown to participants through a head-mounted display, and at the end of each trial, participants were asked whether they would cross the road. Mahadevan et al. (2019) [13] developed a desktop-based VR simulator to study AV–pedestrian interactions in mixed traffic conditions, where the parameters of the traffic scene, i.e., vehicle factors, elements of the street and traffic attributes, pedestrians’ characteristics and AV–pedestrian interfaces, can be easily manipulated and adjusted. A more interactive approach to explore the effects of different eHMIs on pedestrians’ crossing decisions was followed by Bindschadel et al. 2021 [47]. They used a set-up which made it possible for pedestrians to physically cross the road in front of a vehicle in a virtual environment rather than just asking the participants to communicate their crossing intention. Similarly, Lee et al. (2022) [48], using a CAVE-based virtual reality simulator that allowed participants to walk to the other side of the road, measured the pedestrians’ willingness to cross and the crossing initiation time for novel and more conventional eHMIs. A set-up such as the ones cited above allows behavioural measures to be recorded in addition to self-reported ones, allowing for a more comprehensive and objective approach to investigating pedestrians’ crossing behaviour but with the advantage of a controlled set-up.

The immersive properties of a virtual environment are believed to be a necessary condition for behavioural performance. To address the question of the ecological validity of behaviours observed in virtual reality set-ups, the sense of presence (i.e., “of being there” in the virtual world [49]) is measured as an indication of the extent to which a participant is immersed in the virtual scenario and perceives the virtual environment as real. The perceived sense of presence, as well as the quality of sensory stimuli experienced in a virtual environment, can affect participants’ risk perception, for instance [50,51], consequently altering their behaviour. Studies have shown that VR can be highly immersive [52,53], and when this is the case, pedestrian behaviour in a VR simulation can reflect real-world attitudes [54]. With specific regard to the domain of AV–pedestrian interactions, it has been shown that the level of immersion of the simulated environment can impact the pedestrians’ perceived realism and spatial awareness of the scene, which in turn is how eHMIs are assessed [55]. Therefore, it is important to run a validation process to ensure the ecological validity of the results gathered in virtually simulated environments.

1.2. Aims and Research Contribution

The aim of this study was twofold. First, this research aims to show a process to identify a range of indicators that contribute to the crossing scenario complexity and then rank these by increasing complexity. A set of crossing scenarios was developed and tested for this purpose in a simulated environment. According to previous findings, it was assumed that different elements of the traffic environment can affect pedestrians’ perception of complexity and crossing behaviour. Further, the study aimed to measure the realism of the different scenarios proposed. Previous research has stressed the link between the sense of presence in the virtual environment and the ecological validity of the results, and therefore, this was deemed a necessary step to re-create a simulated environment which elicits realistic pedestrian crossing behaviour.

To the best of the authors’ knowledge, this is the first study that empirically attempts to reproduce and rank the complexity of different pedestrian crossing task scenarios. The insights from this research can be used as a starting point to assist the categorisation and ranking of different scenarios, which can then serve to test eHMIs for pedestrian interactions with AVs.

The next section provides an overview of the approach followed.

2. Materials and Methods

2.1. Study Scenarios Development

The following section describes the experimental conditions developed to investigate crossing scenario complexity. All scenarios were designed with embedded audio, congruent with the level of traffic and clutter of the environment. A table summarising all the scenarios is provided at the end of this section (

Table 1. Summary of study crossing scenarios.

Scenario	Description
Baseline 1	No time pressure, bus arriving after some time
Baseline 2	Time pressure (bus arriving any minute)
Traffic flow (2–5 s)	High traffic flow = 2–5 s gaps
Traffic flow (5.5–7 s)	Medium traffic flow = 5.5–7 s gaps
Traffic flow (7.5–10)	High traffic flow = 7.5–10 s gaps
One lane road	One lane, straight road
Two-lane road	Two-lane road with two-way traffic
T-Junction	T-junction with traffic coming from multiple directions
Daylight, good weather	Daytime with good weather and visibility
Daylight, poor weather	Daytime, rainy weather and poor visibility
Dusk, poor weather	Dusk, rainy weather and poor visibility
Visual load < 100	Low visual load, No. of elements in the scene < 100
Visual load = 100–199	Medium visual load, No. of elements in the scene = 100–199
Visual load = 200–300	Medium visual load, No. of elements in the scene = 200–300
Distraction (phone call)	Participants engaged in a simulated phone call

).

2.1.1. Baseline Conditions

Two baseline conditions were included, one without an ‘urgency task’ and one where participants were required to reach the other side of the road to make it in time for the approaching bus (visible on the road). Although the latter was the one against which the other conditions were compared, both were included to evaluate whether the more lifelike crossing goal would make a difference to the perceived realism of the crossing task and crossing behaviour. In both conditions, participants were reminded to only cross when they felt safe to do so as much as possible, as they would in real life. However, in the latter, they were also told that a bus would be arriving any minute and that they would need to make it to the bus stop on the other side of the road in time. A similar task design was implemented by Morrongiello et al. (2015) [56] in a VR study on pedestrians’ behaviour.

2.1.2. Traffic Density

In order to select a gap that allows for safe crossing, pedestrians must correctly judge the spatial and temporal size of the gap with respect to their own ability to cross the gap in time. A minimum 2 s gap has been found to be the smallest acceptable gap [57]. Rasouli and Tsotsos [58] found that critical gap acceptance ranges from 3–7 s. A gap of 5 s is considered a safe crossing gap, 7 s a safe and comfortable gap, and most pedestrians will accept gaps between 5.3 and 9.4 s [59].

Three different traffic density conditions were designed as follows:

• For 2–5 s: Starting at smaller gaps increases traffic density. A 5 s gap was included in this condition to give participants an opportunity to cross when they wouldn’t cross with the smaller gap times. This level of traffic density, i.e., gaps between 2 and 5 s, was repeated across all the other conditions, with the exception of the baseline scenarios. The reason for choosing smaller gaps was that these would allow for more efficient traffic flow, which is a desired outcome of AV deployment. If smaller gaps are acceptable across different crossing scenarios, it becomes possible to design gaps which are comfortable enough for pedestrians to cross but whilst also being efficient.
• For 5.5–7 s: A lower density of traffic with more comfortable traffic gaps but which still establishes a condition of traffic conflict requiring individuals to carefully evaluate traffic in order to make safe crossing decisions.
• For 7.5–10 s: Low traffic density where most pedestrians would accept a gap.

Time gaps were designed with 0.5 s increments across all conditions.

2.1.3. Road Geometry

The process of selecting a gap that allows for safe crossing is further complicated when there is more than one lane of traffic [60]. When the flow of traffic increases in both directions, available safe crossing gaps become smaller and less frequent. Additionally, if visual obstructions are present, such as bends in the road, and traffic is approaching from multiple directions, as what happens at junctions, making safer crossing decisions becomes even more challenging. Scenarios were developed which encompassed different road layouts and, accordingly, lower or higher levels of traffic. These were (a) a one-lane road comprising one line of traffic only, (b) a two-lane road with bi-directional traffic, which required left and right traffic checks in order to cross safely, and (c) a T-junction, where traffic came from different ways and was not visible at all times due to the road layout blocking the subjects’ range of view, requiring them to continuously monitor the road for traffic.

2.1.4. Weather and Visibility

As an active transport mode, the act of walking is susceptible to changes in weather and visibility conditions. Evidence suggests that adverse weather conditions, such as the presence of rain, are associated with a higher likelihood and severity of pedestrian injury in crashes, as jaywalking might be more prevalent [61] and in general pedestrians might exhibit riskier and less compliant crossing behaviour [62]. The presence of rain has also been found to impact pedestrians’ walking speed [63,64]. There is some indication that pedestrians’ speed can be higher at dusk as a “walk-quickly” strategy might be adopted [65]. Additionally, it has been reported that pedestrians’ estimation of vehicles’ speed can vary under different weather conditions [66]. Three different weather and visibility conditions were included in this study: daytime good weather, daytime rainy weather and dusk with rainy weather. Figure 1 below shows example scenarios for these different conditions.

2.1.5. Distractions

When the cognitive demand required for crossing is loaded by additional distractions, the pedestrian’s awareness is reduced, resulting in unsafe and risky crossing behaviour [67,68,69]. As previously discussed, different types of distractions can impact a pedestrian’s crossing behaviour. It was decided that both static and dynamic distractions would be investigated for a more comprehensive representation of real-life crossing circumstances.

These included three different levels of visual clutter in the crossing scene: (a) <100, (b) =100–199, (c) 200–300 elements, as investigated by Tapiro et al. (2020) [24]. Elements embedded in the scene included road furniture, vegetation, billboards and other pedestrians. Additionally, the type of built environment was also altered across the conditions, with progressively taller and ‘bulkier’ buildings as the level of clutter increased [70]. Figure 2 shows these scenarios.

A distraction task coming from engaging in a secondary activity while crossing, e.g., engaging in a phone call, was also simulated. Talking on the phone has been found to have the greatest effect on pedestrian behaviour compared to other secondary activities, e.g., texting/viewing content on mobile phone or listening to music [71]. Conversations in the phone call condition were conducted between the researcher and the participant. The researcher asked a number of open-ended questions, and when the participant showed interest in a topic, follow-up related questions were asked to keep the conversation going. The questions used were formulated by Neider et al. (2010) [25] and contained topics which can be of interest to a wide range of people. Participants were asked to hold their mobile device to their ear in this condition to make the task more realistic; therefore, of the two VR headset headphones, the one close to the ear where the phone was being held next was kept lifted.

2.2. Measures

A combination of subjective and objective measures was collected as indirect indicators of the scenarios level of complexity.

2.2.1. Simulation Task Load Index

There is currently no measure which has been specifically designed to quantify the complexity of the crossing task. As mental workload has been previously used as a measure of difficulty for operating a system or accomplishing a task (e.g., [72,73]) participants’ workload was recorded. This was measured with a VR-simulator-adapted version of the NASA-TLX questionnaire, the simulation task load index (SIM-TLX), proposed by Harris et al. [74]. This index covers some of the subscales present in the original scale, such as mental demands, physical demands, temporal demands, frustration and additional sources of load, i.e., task complexity, situational stress, distraction, perceptual strain and task control. These additional dimensions were deemed relevant to the research at hand, in that they can give a more comprehensive overview of how different environments would be perceived.

2.2.2. SUS Questionnaire

The Slater-Usoh-Steet questionnaire developed by Slater et al. (2000) [75] was used to measure presence in simulated environments. The questionnaire revolves around three themes on which the questions are based, which are the sense of being in the virtual environment, the degree to which the virtual environment becomes the prevailing reality and how well the virtual environment is recalled as a real place.

2.2.3. Crossing Behaviour Measures

A frame-by-frame analysis of HD video recordings captured with the GoPro cameras synchronized with desktop screen recordings—which captured the subjects’ view in the simulated virtual environment and their position with respect to approaching vehicles in the environment—was used to assess pedestrian crossing behaviour.

The following measures were computed: (1) crossing initiation time, i.e., the time between the start of the simulation and the participant initiating crossing, and (2) crossing speed, which was recorded as the walking speed average speed between the pedestrian-initiated crossing and when they completed crossing. These measures are commonly used to study pedestrian behaviour (e.g., [76,77]). Whether pedestrians did or did not cross the road was additionally recorded. Floor markings were used to designate the road and crosswalk, which matched the crossing facility in the simulated environment (Figure 3).

2.2.4. Gaze Behaviour

The gaze behaviour recorded was based on head movements due to technical issues encountered with the eye-tracking equipment. Screen recordings of the subjects’ view in the virtual environments were processed, and the amount of time that participants looked at traffic versus the time where participants looked at other areas of the environment was calculated for each trial. This was used as a measure of attention to traffic. Due to the lack of eye-tracking data, it was not possible to determine accurate visual targets. However, it was still possible to establish areas of interest [78], specifically whether the road and traffic were in the participants’ visual area or not [79] (Figure 4).

2.2.5. Additional Data Collected

The Misery Scale (MISC) [80] was used to assess the participants’ wellbeing throughout the session.

Semi-structured interviews closed the session. Questions were asked to gain a better understanding of the participants’ experiences in the simulated environments, specifically with regards to workload crossing difficulty and realism of the scenes.

2.3. Participants

Ten participants (six females) took part in the study. Their age ranged from 19 to 48 years (M = 29.9, SD = 8.75). Demographic characteristics are reported in

Table 2. Demographic characteristics of participants.

Category	Sub-Category	Frequency (N)
Sex	Male Female	4 6
Age	18–24 25–34 35+	3 4 3
Driving license	Yes No	8 2

. All participants were residents in the UK, had normal or corrected to normal vision and normal walking abilities. Participants were screened for photosensitive epilepsy and panic attacks to ensure their safety during the experiment. Two subjects stated that they had prior experience with virtual reality, one being an experienced game player, whereas the rest had not previously experienced any virtual reality.

2.4. Apparatus

The laboratory setup was carefully designed to make sure that the data was collected not only effectively but also safely. This is described in Section 3. The study was carried out in the Human Factors Lab (School of Design and Creative Arts, Loughborough University, Loughborough, UK). A set of immersive virtual environments were created using the modelling software Blender (Blender version 3.1, Blender Foundation, Amsterdam, The Netherlands) and run with Unity VR (Unity 2021 1.7f1, Unity Technologies, Copenhagen, Denmark) and the HTC Vive Eye Pro Headset device (HTC Corporation, Taoyuan, Taiwan). The virtual reality system consists of a head-mounted display with an HTC Vive wireless adapter and two external Vive bases that track the position and orientation of the Vive headset in the real world and translate its position and orientation in the virtual space. The base trackers were positioned diagonally and opposite each other in the lab space at a distance of about 8 m to delimit a usable area of about 4 m × 7 m, which allowed participants to physically cross the road, which was 3–5.5 m wide, during the experiment (see Figure 5).

The equipment was supported by a desktop computer. Four cameras, Cameras model GoPro 7, (GoPro Inc., San Mateo, CA, USA) were used to record the participants’ crossing behaviour during the experiment, positioned at the front, back and side of the walking area. Additionally, a webcam was used to synchronise the recorded video data with a view of what the study subjects were seeing in the virtual environment. This was displayed on a secondary screen and recorded with OBS Studio Software (OBS Studio version 28.1.1, OBS Project, San Francisco, CA, USA).

The study featured nineteen virtual environments: the “lobby”, three familiarisation environments and fifteen test environments (previously described in Section 2.1). The test environment consisted of an urban scene, whose level of detail and design changed depending on the scenario and was focused on a freeway traffic scenario. Traffic was present in the environments, which were also complemented by noise and sounds. Participants started on the sidewalk and walked all the way through to the other side of the road to complete the crossing. Once the participant crossed the street, the simulation was stopped, and participants walked back to the starting position in the lobby environment.

3. Experimental Procedure

Participants were first given an overview of the study and the equipment. After they gave consent to take part in the study, they were shown the questionnaires. Later, they were asked to familiarise themselves with the questionnaire that they would complete after each trial and set of trials. Participants were given an opportunity to familiarise themselves with the VR and later undertook the trials and questionnaires. At the end of the experiment, participants were asked some questions about their experience with the VR and the different scenarios they encountered throughout the study.

The full procedure is discussed in more depth in the following section, and a schematic overview is shown in Figure 6.

3.1. Introduction to VR, Informed Consent and Familiarisation with Questionnaires

Participants arrived at a previously scheduled time and were explained the purpose of the study, as well as given an overview of the data collected and the equipment used, including the VR hardware and safety measures in place. Consent was obtained before proceeding further with the session. All participants were shown the questionnaires that would later be administered, and were given an opportunity to ask any questions they might have. They were then asked to indicate how they felt using the Misery Scale (MISC) before experiencing the virtual reality to collect a baseline measurement.

3.2. VR Familiarisation

After collecting the MISC measurement, participants were asked to wear the headset and explore a familiarisation environment that consisted of a simple one-lane straight road without any traffic, and were asked to walk from one side of the road to the other to become comfortable with walking and moving in the virtual environment. Participants who felt more hesitant were given more time to move through the environment until they gained enough confidence. All participants were shown the “safety grid” delimiting the playground area, and it was explained to them that they could safely move within the available free space. Additionally, another researcher was present during the session to supervise the participants’ safety when they were wearing the headset to prevent collisions from happening. Participants were then exposed to two other familiarisation environments: a one-lane road and a two-lane road, which were this time populated with traffic. Participants were instructed to cross the road when they felt safe to do so, similar to how they would do in real life. After experiencing each of the test environments, a MISC measurement was collected. As participants were now familiar with the Misery Scale, there was no need for them to remove the headset to complete it, and it was enough for the researcher to check with them regarding how they were feeling with respect to the different points listed in the questionnaire.

3.3. Data Analysis

Data analysis consisted of two stages. The descriptive statistics distribution of scalar variables was first assessed for normality using Shapiro–Wilk tests. One data point was excluded from the Workload analysis due to the participant completing the questionnaire erroneously. A series of repeated measures ANOVAs was conducted in order to assess the main effects of scenario type on workload score, presence score, time gazed at traffic and crossing initiation time. The data was checked for sphericity using Mauchly’s test, and where violated, Greenhouse–Geisser was applied. Where significant results were found, Holm–Bonferroni adjusted post-hoc tests were performed to determine which scenarios were significantly different from each other. The significance level was set at α = 0.05.

Effect sizes were classified using eta squared (η²) for repeated measures ANOVA (small effect: η² = 0.01; medium effect: η² = 0.06; large effect: η² = 0.14) [81], and Kendal’s W following the non-parametric Friedman test (small effect: W = 0.1; medium effect: W = 0.3; large effect W = 0.5) [82].

Attention was paid to conditions which were significantly different from baseline scenario 2. This baseline condition was chosen as the base for comparison, as it differentiated from baseline 1, having the time pressure built in as the rest of the conditions.

4. Results

4.1. Subjective Measures

4.1.1. NASA SIM-TLX

A repeated measures ANOVA was performed to compare the effect of scenario type on the subjects’ perceived workload ratings as a surrogate measure for scenario complexity. The results showed a statistically significant effect of scenario type on workload ratings F (5.124, 40.994) = 13.085, p < 0.001, η² = 0.621. These results suggested that some scenarios were perceived as involving a higher workload than others. Figure 7 shows the workload ratings for each scenario. Post hoc tests with Holm–Bonferroni correction indicated that the baseline condition scored significantly lower workload ratings for the dusk poor weather condition (M = 38.243, SD =24.731, p < 0.043), as well as the visual load = 200–300 (M = 42.283, SD =20.143, p = 0.003), the distraction task (M = 42.499, SD = 22.266, p = 0.003), the two-lane road (M = 48.611, SD = 16.766, p < 0.001), and the junction (M = 57.901, SD = 19.303, p < 0.001) conditions, which had significantly higher workload ratings compared to the baseline condition (M = 19.647, SD = 13.895). These conditions did not significantly differ from each other, with the exception of the junction condition being significantly higher than the dusk poor weather condition, p = 0.023.

The junction scenario also had significantly higher workload scores then all of the remaining scenarios, whereas the two-lane road scenario ratings were significantly higher than the different traffic flow conditions, which are arranged in order: 7.5–10 s (M = 16.651, SD = 13.969, p < 0.001), 5.5–7 s (M = 21.851, SD =10.410, p < 0.001), 2–5 s (M = 15.138, SD = 6.648), and the daylight poor weather condition (M = 28.364, SD =14.493, p = 0.016). Workload ratings for the distraction task scenario were significantly higher than all the traffic conditions (in order p < 0.001, p = 0.012, p < 0.001) as well as the daylight good-weather condition, p = 0.005.

Table 3. SIM-TLX mean ratings by scenario (arranged from lowest to highest mean score).

Scenario	Mean	SD	Min	Max
Baseline 1	11.17	8.63	0.00	28.33
Traffic flow (7.5–10 s)	16.65	13.97	2.08	41.39
Baseline 2	19.65	13.90	0.00	43.47
Traffic flow (5.5–7 s)	21.85	10.41	7.50	36.25
Daylight and good weather	20.59	14.51	5.14	50.69
Traffic flow (2–5 s)	15.14	6.65	9.03	27.36
One lane road	30.83	14.85	8.61	47.78
Daylight, poor weather	28.36	14.49	9.58	53.61
Visual load =100–199	34.30	13.17	23.89	64.58
Visual load < 100	37.61	17.32	11.81	68.06
Dusk, poor weather *	38.24	24.73	11.81	86.11
Visual load =200–300 **	42.28	20.14	10.97	79.17
Distraction task (phone call) **	42.50	22.27	15.42	89.86
Two-lane road ***	48.61	16.77	24.86	72.22
Junction ***	57.90	19.30	38.19	73.61

* p < 0.05; ** p < 0.01; *** p < 0.001.

summarises the workload scores across scenarios.

Figure 8 illustrates the raw scores for the scenarios which scored a significantly higher workload than the baseline condition. Some factors, such as temporal demands (which scored progressively higher ratings across these scenarios), task complexity, distractions, situational stress and temporal demands, contributed more heavily on the workloads, whereas others, e.g., physical demands, scored lower ratings.

4.1.2. SUS

Scores from the Slater-Usoh-Steed (SUS) questionnaire are reported in the table below (

Table 4. SUS count and mean scores.

Scenario	Count Mean Scores (SD)	Cumulative Mean Scores (SD)
Baseline 1	2.50 (1.65)	5.00 (0.94)
Baseline 2	3.40 (1.58)	5.31 (0.88)
Traffic flow (7.5–10 s)	2.70 (2.11)	5.40 (0.76)
Traffic flow (5.5–7 s)	3.30 (2.11)	5.20 (1.69)
Traffic flow (2–5 s)	2.30 (2.26)	5.35 (0.88)
One lane road	2.50 (1.65)	5.00 (0.94)
Two-lane road	3.40 (1.58)	5.31 (0.88)
Junction	2.70 (2.11)	5.40 (0.76)
Daylight, good weather	3.30 (2.11)	5.20 (1.69)
Daylight, poor weather	2.30 (2.26)	5.35 (0.88)
Dusk, poor weather	2.50 (1.65)	5.00 (0.94)
Visual load < 100	3.40 (1.58)	5.31 (0.88)
Visual load = 100–199	2.70 (2.11)	5.40 (0.76)
Visual load = 200–300	3.30 (2.11)	5.20 (1.69)
Distraction task (Phone call)	2.30 (2.26)	5.35 (0.88)

). One column (left) reports the SUS count mean scores and the other the SUS cumulative mean scores, i.e., the mean of the SUS count of ‘6’ and ‘7’ scores amongst the 6 questions only, and the mean scores across the 6 questions, respectively.

The baseline-1 scenario scored the lowest presence rating—2.50 count mean and 5.0 mean. All the other conditions received higher presence ratings, with the junction scenarios having the highest score—5.5 (SD = 0.7) and 6.1 (SD = 0.8).

4.1.3. Misery Scale

The misery scale scores collected throughout the experiment showed that none of the participants experienced any symptoms of discomfort, with all the scores collected being equal to zero (i.e., no symptoms of sickness), including during the practice trials. At no point did participants request to take a break from the experiment or to cease it due to simulator sickness.

4.2. Objective Measures

4.2.1. Crossing Initiation Time

A Friedman ANOVA was carried out because an inspection showed that data for crossing initiation time was not normal. This showed a significant difference in crossing initiation time across the scenarios with a large effect size χ²(14) = 59.347, p < 0.001, W = 0.606. Conover’s post hoc pairwise comparisons showed that the crossing initiation time for the baseline condition was significantly lower than the two-lane road condition (p < 0.001), the junction condition (p = 0.038) and the visual load = 100–199 condition (p = 0.004). The difference between the two-lane road and the junction scenario was non-significant.

The baseline condition also had lower crossing Initiation times than the traffic flow for 5.5–7 s and 2–5 s, the one-lane road, the good and poor weather and visibility conditions, levels of visual load < 100 and = 200–300 and the distraction task condition, although these were not significant.

It should be noted that almost all participants crossed the road throughout the trials. However, for the two-lane road and the junction scenarios, two and three subjects, respectively, did not cross the road. One subject stepped onto the road but then made their way back onto the pavement. The participants who did not cross reported not being able to find a comfortable enough gap between vehicles.

Table 5. Summary of crossing behaviour data for each condition.

Scenario	Crossed the Road (N)	Median and IQR	Mean SD
		CIT (s)	Average Crossing Speed (m/s)
Baseline 1	10	8.2 (3.8–10.8)	1.17 (0.17)
Baseline 2	10	3.2 (2.6–3.7)	1.35 (0.12)
Traffic flow (7.5–10 s)	10	4.6 (3.2–8.0)	1.47 (0.16)
Traffic flow (5.5–7 s)	10	5.7 (3.4–9.5)	1.56 (0.22)
Traffic flow (2–5 s)	10	6.4 (6.0–9.8)	1.77 (0.52)
One lane road	10	10.7 (6.9–17.7)	1.33 (0.31)
Two-lane road	8	32.5 (23.4–36.7)	1.25 (0.08)
Junction	7	10.7 (8.5–18.8)	1.37 (0.51)
Daylight, good weather	10	5.7 (5.5–10.2)	1.54 (0.29)
Daylight, poor weather	10	9.3 (5.9–17.8)	1.50 (0.27)
Dusk, poor weather	10	7.5 (5.2–9.7)	1.37 (0.18)
Visual load < 100	10	8.2 (7.3–15.7)	1.46 (0.21)
Visual load = 100–199	10	12.3 (8.5–31.2)	1.31 (0.20)
Visual load = 200–300	10	8.3 (7.6–8.7)	1.65 (0.55)
Distraction task (Phone call)	10	9.8 (8.1–10.1)	1.45 (0.22)

summarises the median crossing initiation times, the number of participants who crossed the road by scenario, as well as the average crossing speeds, which are discussed in Section 4.1.2.

4.2.2. Crossing Speed

The repeated measures ANOVA for speed showed a significant result: F(14, 70) = 1.960 p = 0.034, η² = 282.

Significant results were not found, however, between the baseline-2 condition (urgency built-in task) and the other scenarios, but rather when comparing some scenarios to the baseline-1 condition.

Although not significant, mean crossings speed varied across the different scenarios, with the lowest walking speed levels being for a two-lane road (M = 1.25, SD = 0.08) and the highest speeds for the high traffic flow condition (M = 1.77, SD = 0.52), followed by the level of visual clutter = 200–300 condition (M = 1.65, SD = 0.55).

4.2.3. Gaze Behaviour

The Friedman ANOVA showed a significant difference in the percentage of total time gazing at traffic across scenarios χ²(14) = 47.283, p < 0.001, W = 0.563.

Post hoc tests were used to look at differences between scenarios.

The percentage of time spent gazing at traffic was significantly higher than the baseline condition for the one-lane road (p = 0.001), the two-lane road (p < 0.001), visual clutter < 100 (p = 0.003) and visual clutter = 100–199 (p = 0.001). Median percentages of time gazing at traffic across the other scenarios were higher than the baseline, although these were not significant. Gaze behaviour data is reported in Figure 9.

4.3. Qualitative Analysis

4.3.1. Distracting Tasks and Environments

When discussing crossing task difficulty, themes emerged which pointed out different elements. Some individuals also expressed how a particular scenario made them feel with regard to the road-crossing task.

A major theme could be identified with the perceived level of distraction and how this impacted participants’ self-reported behaviour, for instance by paying less attention to the traffic or missing crossing gaps. The difficulty presented by “distractions” came mainly from the level of detail in the environment, which is higher levels of visual load (visual load =100–199 and = 200–300 conditions). The following participants’ comments illustrate this theme:

‘more difficult was the distraction, when those screens for example were playing something, or people talking, or people passing, that became more distracting’, ‘several distractions, made me not pay much attention to traffic’, ‘I think when there are more distractions, you probably won’t be looking for a gap as often’. So there were some gaps where I was probably the same. But if I was distracted, I would wait a second, maybe to see it and then I’d be too late. So yeah, I think distractions was a key thing. Things like sound, or like a lot of bright things moving around like, in your peripheral vision’, ‘The environment felt extremely realistic and distracting’, ‘With a lot of people that was a bit more difficult ‘cause you get distracted’, ‘(it) was very loud, there were a lot of loud noises, that also made it very distracting for me’.

Engaging in a phone call was also perceived as distracting:

‘Was very distracting and although the environment was less complicated because there was less detail overall, it raised the task complexity much higher’, ‘The distracting phone call made me focus left on the surroundings’.

Higher levels of detail also made some subjects feel more pressured or rushed to cross:

‘When there are more elements in the environment the task becomes more realistic and engaging and there is also more pressure to cross’, ‘seeing other people at the bus stop made me feel rushed to cross the road’. This was also the case for the lower levels of visual load condition (<100): ‘I felt very rushed off and frustrated’, ‘this experience felt more real to me and I did feel the pressure of getting across safely and watching out for cars’.

Although distracting and potentially pressuring, the presence of other pedestrians and the level of detail made some individuals feel more at ease:

‘The more the noise and the detail in the environment and also the presence of other pedestrians, the more engaged and comfortable I felt’, ‘Very lively environment (... it felt) less stressful and more comfortable because there were people’.

4.3.2. Pressure to Cross and Sense of Danger

It emerged from the post-session interviews that the presence of rain when coupled with dusk put some pressure on subjects and increased the task difficulty:

‘The rain and the darkness did feel real and in some way urged me to cross as soon as I had a chance. It felt more urgent’, ‘(with) things like rain, like darkness it became a little more difficult for me to perform the task’. One individual reported the condition feeling dangerous ‘The cars and the headlights didn’t look very realistic but the scenario overall laid down on be a bit of fear, so it felt like, scary and dangerous to cross. So it’s a bit contradictory, it didn’t feel as real as the others but at the same time if felt more dangerous to cross’.

Pressure to cross was also reported when recalling the junction condition, which was found to be difficult due to the road layout causing reduced visibility of traffic:

‘Yes, on the one with the junction, I felt really pressured. I felt I was never gonna be able to see so I had to take a chance. Because of the two corners I was never gonna be able to see when it was clear’, ‘Felt dangerous. One of the cars surprised me, had to look a few times before crossing’, ‘the corner and high traffic are challenging, visibility is an issue, that made me go for a run even without being one hundred percent sure it was safe’.

The level of traffic and the two-way traffic in the two-lane road condition made some pedestrians feel rushed and aware of the danger of traffic:

‘With traffic coming from both sides, I felt rushed to cross’, ‘Was scared to be hit by a car’, ‘I felt really pressured, I think it was the one with the two-lane road’ and in general reported that this road layout required more concentration on their side to find a suitable gap to cross the road.

Conversely, lower traffic density and no-traffic scenarios were found easier to deal with:

‘open fields with light traffic, that was easy’, ‘when there were less elements in the scene, when there was just the road’.

4.3.3. Sense of Presence and Perceived Realism of the Virtual Environment

The qualitative analysis showed that most study participants perceived the virtual environments generally as quite realistic, and they felt engaged in the experience. They reported having to pay attention to the road in a similar way as they would in real life:

‘it was difficult as it was still crossing the road, you had to look after distractions, treat it like a real-life experience’, ‘I felt like it was the real world, and I was making decisions as I would make them in the real world’ ‘In completing the task I did feel the rush to cross and sensed the danger of cars. To me that is very much an experience of being there’, ‘Cars made me act as if I was on an actual road. While crossing I didn’t think of the experiment but felt instinctive instead’.

Different aspects of the virtual experience contributed to presence or detracted from it in a few cases. These are discussed in the following paragraphs.

Level of Detail and Visual and Auditory Accuracy of the Scene

The qualitative analysis showed that the more the level of detail in the scene, the higher the perceived realism of the simulation. This included buildings, traffic, street furniture, and pedestrians.

‘When there are more elements in the environment, the task becomes more realistic and engaging and there is more pressure to cross as well’, There were many elements—buildings/pedestrians—that made it feel real’, ‘The higher number of cars didn’t leave time to think whether it was real or not’, ‘Very detailed landscape, made it more realistic’, ‘Very lively environment, looks like a city. Seeing people around helps feel like you are actually present’ ‘People moving about helped realism’, ‘Pedestrians and more street furniture made it much more realistic’.

On the other hand, scenarios with lower levels of detail diminished the sense of presence. This was often the case for baseline scenario 1 and the conditions with less dense traffic:

‘Flatness of the scenery pulled me out’, ‘Open fields with light traffic. It was easy so I could think about being in a simulation’ ‘Very plain settings, less realistic’ ‘Plainer surroundings, made it less realistic’.

Road noises and sounds also played a key role in the sense of presence and the perceived danger of traffic:

‘The sensorial experience was well orchestrated—the road felt real, the sounds (birds chirping and road noise) gave the idea of a real place’, ‘Seeing the virtual environment along with the audio helps feel like you are really there’, ‘Noises of traffic made it feel more real’ ‘The car noise when it came closer made it more realistic. I crossed too soon and I felt that it (the approaching vehicle) was too close’, ‘(With vehicle’s noises embedded in the environment), you feel like the cars are quite close to you. And I think that really made you ‘Oooh’ more aware of your surroundings ‘cause you know it’s coming, it can hit you’.

Engaging in Lifelike Tasks

The level of engagement and distraction resulting from performing a lifelike task, such as in the case of taking a phone call and having to catch a bus, resulted in higher perceived immersion, even when the visual scenery was bare.

‘Needing to pay attention to the “phone call” made me almost forget I wasn’t on an actual road’ ‘The phone call was actually a very realistic task and it made me feel like I was there’, ‘Real life activity e.g. being on the phone’ contributed to enhanced sense of presence, ‘The urgency of wanting to complete a task when there were cars made the VR experience believable and like I was really there’, ‘The bus approaching made it feel more real’ the ‘timing element of waiting for a bus’, made the experience more realistic.

What also made crossing more lifelike was being able to physically move in the environment and walk to the other side of the road:

‘Walking and moving provided a sense of being on the road’, ‘It was better than I expected. I was kind of expecting the use of controllers but the fact that you got the whole room set up, it was much more immersive and real’.

5. Discussion

This research aimed to show a process to explore the nature of complexity of different crossing conditions and identify scenarios for testing eHMIs in conditions which more realistically reflect real-life crossing circumstances. In the first step, the criteria were established for the scenarios’ selections based on previous pedestrian research and existing frameworks.

The second research step consisted of measuring complexity to address the question of whether different traffic scenarios affected a pedestrian’s perception of complexity and crossing behaviour. The results showed that the pedestrian’s perceptions of complexity differed across scenarios. Some scenarios significantly increased the workload scores of the crossing task compared to the baseline condition, while “simpler” environments scored lower workload ratings; these were higher for more complex and detailed scenarios. The analysis of the elements of the pedestrians’ crossing behaviour also revealed some differences across scenarios in terms of crossing initiation time and time gazing at traffic. Some of the objective measures were found to be consistent with the self-reported workload as well as with the participants’ comments and feedback.

For other scenarios, the subjective and objective measures differed.

Table 6. Subjective and objective data by scenarios from the lowest to the highest value, as indicated by the arrows.

Workload (SIM-TLX) (Lower to Higher ↓)	Crossing Initiation Time (Shorter to Longer ↓)	% Tot Time Gazed at Traffic (Shorter to Longer ↓)
Baseline 1	Baseline 2	Baseline 2
Traffic flow (7.5–10 s)	Traffic flow (7.5–10 s)	Traffic flow (7.5–10 s)
Baseline 2	Traffic flow (5.5–7 s)	Dusk, poor weather
Traffic flow (5.5–7 s)	Daylight, good weather	Baseline 1
Daylight and good weather	Traffic flow (2–5 s)	Traffic flow (5.5–7 s)
Traffic flow (2–5 s)	Dusk, poor weather	Junction
One lane road	Baseline 1	Daylight and good weather
Daylight, poor weather	Visual load < 100	Visual load =200–300
Visual load =100–199	Visual load = 200–300	Distraction task (Phone call)
Visual load < 100	Daylight, poor weather	Traffic flow (2–5 s)
Dusk, poor weather	Distraction task (Phone call)	Daylight, poor weather
Visual load =200–300	One lane road	Visual load < 100
Distraction task (phone call)	Junction	One lane road
Two-lane road	Visual load = 100–199	Visual load =100–199
Junction	Two-lane road	Two-lane road

summarises the subjective and objective data by scenarios from the lowest to the highest value.

In the following sections, the results are discussed in detail, and reflections are made with regards to eHMIs evaluation.

5.1. Self-Reported Workload

Perceived complexity for the crossing task across different scenarios was assessed by using self-reported workload, an indirect measure. Significant differences in perceived workload for more complex road layouts were found when compared to the simple, one-lane road scenario. The two-lane road and junction scenarios were in the conditions that scored the highest workload ratings in general. It has been acknowledged by previous research that handling traffic approaching from multiple directions is a considerably more challenging task than crossing a single lane. A pair of gaps must be selected, a near-lane gap and a far-lane gap, taking into account the spatiotemporal relationships between gaps to make a safe crossing decision [60]. Also, crossing a two-way traffic road requires alternating one’s attention left and right, requiring a higher number of gazes prior to crossing [83].

Participants reported a higher perceived workload for the distraction task scenario. Previous literature has shown that pedestrians who engage in a phone call conversation make slower crossing decisions, are less aware of the situation and pay lower attention to traffic [25,26,27]. It is reasonable to assume that performing a dual task, i.e., taking on the phone and crossing in parallel, required higher cognitive effort [84,85]. Significant differences in perceived workload with respect to the baseline scenario were also found for high levels of visual clutter. Higher workload scores, although not significant, were also reported for the low and medium visual load conditions, indicating that these were perceived as more cognitively demanding by the participants. This is in line with previous research which shows that busier road environments can be more taxing on a pedestrian’s attention, resulting in missed safe crossing opportunities [24] and leading to riskier crossing behaviour, especially for younger pedestrians [86]. Visual distraction has also been found to increase a driver’s workload [87] and interfere with driving performance [88]. From the qualitative analysis, it emerged that it was both visual elements which contributed to perceived distraction, such as the presence of other pedestrians, screens and billboards as well as auditory elements, e.g., other people talking and loud noises in general. As crossing is a task which requires many attentional resources that use the same (limited) visual channel [89], the presence of a high number of objects which detract attention from the road requires pedestrians to work harder to keep a focus on traffic and make safe crossing decisions [90]. Additionally, ambient noise from background sources can make it harder to determine a vehicle’s travel path [91,92,93] and has been reported to affect pedestrians’ perception of the surrounding environment [94]. Finally, accomplishing the crossing task with poor weather and visibility conditions was found to be more demanding than in the baseline scenario. It has been suggested that pedestrians might be more vigilant when crossing at dusk [65], being aware that road crossing would be more dangerous under such circumstances. Higher workload thresholds for this scenario reflect a tendency to exert more caution in a situation perceived as more demanding [95].

Participants’ level of workload indicates that pedestrians’ perception of the crossing task can differ depending on traffic condition. A diverse set of scenarios that accounts for different levels and natures of difficulty of the crossing task can be used for comprehensive safety and efficiency eHMI testing, as these could elicit unique design and deployment challenges. Moreover, the addition of eHMI to an already-complex traffic scene could further increase the workload. Therefore, an evaluation of the interplay between traffic scenario and eHMI could help design an optimal information capacity that is appropriate for different interaction circumstances.

When looking at SIM-TLX raw scores, it is interesting that some factors were found to have a greater impact than others. In particular, temporal demands were perceived as higher than the other elements. It is likely that the level of urgency embedded in the crossing task also played a part in the perceived workload to some level. However, for the same task, the temporal demands still differed across scenarios, indicating that there were other elements at play. It might be interesting to investigate how the workload varies for the same scenarios without time pressure.

5.2. Crossing Behaviour

In general, crossing initiation times were higher for all conditions compared to the baseline (time pressure built in task) scenario, as was the percentage of time spent by participants gazing at traffic. The presence of traffic, even if at lower densities, made pedestrians more cautious and required more time for them to evaluate the traffic. The time before pedestrians stepped on the road was particularly longer when crossing the two-lane road and the junction, as more traffic gazes were needed in order to find a suitable gap [80]. The longer waiting time and the higher amount of time spent gazing at traffic might suggest cautious behaviour, as pedestrians made sure that a safe gap was found before attempting to cross. Some pedestrians did not cross the road when multi-directional traffic was present, showing that despite the pressure of getting across, the perceived risk made them behave more cautiously. The average speed for the two-lane crossing scenario was lower, although not significantly, compared to the other conditions. The lower speed might be due to the pedestrians being more cautious as well as to them shifting to a two-stage crossing or a rolling gap strategy, thus adjusting their crossing speed to the (heavier) vehicular flow [96]. In the junction condition, faster crossing average speeds were recorded, and although the crossing initiation time was significantly higher than the baseline scenario, there was a considerable variation of crossing initiation times across participants, and likewise for the time subjects allotted to gazing at traffic throughout the trial. Variation across participants in this circumstance suggests that some participants compensated for the higher perceived risk of the situation, i.e., achieved safety by trying to get across the road quickly [76]. Participants did indeed report being aware of the riskier nature of this type of road configuration due to traffic coming from multiple directions and the reduced visibility resulting from the road layout. In general, as the level of traffic flow increased, so too did the time spent gazing at traffic, the time before the crossing was initiated and the workload ratings. This is consistent with previous findings indicating that the level of traffic flow can impact crossing difficulty [97]. Whether eHMIs could affect how pedestrians respond to riskier and traffic-heavy circumstances is yet to be determined. Testing scenarios where the level of vehicular flow and density is varied can help address this question and establish whether and when it is safe to communicate a message.

Participants took longer to step on the road for medium levels of visual clutter, although interestingly, the same cannot be said for the high levels of clutter. It is possible that the considerable level of detail in the environment was found distracting, leading to a longer time to find a suitable crossing gap, but the presence of a greater number of pedestrians in the latter condition contrasted the distracting element, affecting the subjects’ crossing decision. Previous research has shown that social influence can impact crossing decision-making, e.g., the presence of other individuals might increase a pedestrian’s propensity to cross even in risky situations [98]. This would also explain the lower time spent gazing at traffic for this condition with respect to the lower visual load scenarios. The higher average crossing speed recorded for this condition would also point in this direction. It has been reported that the higher presence of people on the street, as well as higher perceived noise, can result in higher walking speed [99]. Participants indeed reported feeling pressured to cross the road due to the presence of the many elements in the environment and of other pedestrians waiting at the bus stop. Social influence on pedestrians’ crossing behaviour and the presence and proximity of other road users should be accounted for in eHMIs evaluation, as also suggested by previous research (e.g., [13,14]), to avoid designing information which has the potential to further pressure pedestrians or which does not target the intended user. It would also be sensible to investigate how distracting scenes could impact eHMI signal receptivity and responsiveness. Finally, it is likely that the type of road scene induced the participants’ expectations about the behaviour of other road users [100,101,102], i.e., because of the busy road environment and the higher presence of people, there was an expectation that drivers would be more tolerant and cautious. This aspect is particularly relevant to the design of eHMIs as well as of AV kinematics, which have been shown to play a key role in a pedestrian’s crossing decisions (e.g., [6,103,104]). If pedestrians formulate expectations based on the surrounding environment and interaction context, they might expect an AVs’ behaviour to reflect these, e.g., by driving at lower speeds, decelerating and stopping for them more often.

Other scenarios presented longer initiation times, although not significantly, which are consistent with previous research. For example, start delay was longer for the cell phone distraction scenario compared to the baseline condition, although this difference was not significant. Distracted pedestrians have been reported to take longer to step on the road when crossing, missing more safe gaps and causing later detection of approaching vehicles [25,105].

The time pressure built into the crossing task might have impacted the crossing initiation times and crossing speed across scenarios. In baseline 1, where no time pressure was given, average crossing initiation times were longer, although not significantly, compared with the urgency baseline. Previous research has shown that time pressure when crossing can result in shorter periods to appraise the traffic before crossing [56] and riskier behaviours in general. It should be noted, however, that the experience of time pressure in day-to-day life is not uncommon, and pedestrians must often choose between safety, convenience and saving time in their crossing decision [40]. It could be argued therefore that presenting pedestrians with a time pressure task is representative of crossing circumstances encountered in everyday life and would provide a realistic task to investigate pedestrians’ spontaneous reactions to eHMIs and AVs.

5.3. Presence and Sickness

A strong sense of presence is thought to promote simulator validity [106,107]. Cumulative presence scores showed that overall, the scenarios performed well, as all average scores were equal to or greater than five (the maximum possible score being 7).

The lack of any symptom of discomfort throughout the study sessions indicates a good sense of presence in the virtual environment. It has been suggested that cybersickness in virtual reality and presence are negatively related [108,109,110,111]. It is likely that the ability of participants to physically walk in the environment, coupled with the visual quality of the virtual scenarios, contributed to lower sickness and higher sensorial vividness, due to the matching of physical and virtual displacement in the environment [112,113]. The exposure time to the simulated environment being quite short for each trial possibly also contributed to the lack of cybersickness symptoms across scenarios. The mean SUS scores showed more variation in sense of presence across scenarios. The highest score was achieved for the junction scenario that was reported to resemble a more realistic road appearance and shape with drivers approaching from different directions. This was also the scenario that scored the highest workload ratings, suggesting that the increased perceived difficulty of the scenario required participants to concentrate more intensely on the crossing task, which in turn led to a stronger immersion [114]. Scenarios with a measure of visual clutter elicited higher presence scores. This is in line with previous research showing that the more realistic and elaborate the visual graphics are, the higher the perceived realism [51,115]. In general, the presence of other people and street furniture helped portray a visual scene that resembled a more lifelike road where all these elements would often be found. This was also reported by the study participants.

Lower scores than the scenarios mentioned above, although still higher than the baseline scenario, were reported for the distraction task condition. Participants explained that engaging in a real-life task made them feel more immersed in the experience, despite the environment being relatively bare in this condition. Previous research has reported that the higher the involvement in a VR experience, the more the simulated experience can elicit genuine negative and positive emotions such as fear and anxiety, and in turn the higher the emotional engagement in the simulated experience [116]. One element that was reported to favour a stronger sense of presence was the presence of road and car noise. It is very likely that the combination of sensorial stimuli more closely mimicked real-life conditions, enhancing the participants’ sense of presence [117]. Additionally, it is known that pedestrians combine data coming from different perceptual modalities when detecting traffic, which include auditory cues, particularly in urban areas [118]. Pedestrians’ crossing behaviour, e.g., speed and perception of the surrounding environment can be affected by the level and quality of surrounding noise [95]. Ambient noise also has the potential of masking the sound of approaching vehicles, leading pedestrians to hear vehicle noises with insufficient safety margins [93]. The validation and standardization of eHMIs and different cue modalities would greatly benefit from visual and auditory stimuli closer to a real traffic environment for increased realism as well as to test the efficacy of different designs and multimodal cues where elements of the surrounding scene could interfere with successful AV–pedestrian communication.

5.4. Complex “Worst Case” Scenario

Based on the findings of this study, a “complex” environment was developed (Figure 10). The following elements which contributed, in one way or another, to adding difficulty to the crossing task were combined: traffic approaching from two lanes and joining the main road from side streets, adverse weather and poor visibility conditions, high levels of clutter in the scene as well as the presence of the phone call distraction. The junction allowed for the investigation of a task where pedestrians have to negotiate a two-way street crossing, putting them in a more cognitively demanding situation, while also dealing with problematic weather and visual and external distractions. In other words, this could be viewed as a “worst case scenario” and could help uncover issues associated with eHMI signals which might not emerge in simpler interaction situations.

6. Conclusions, Limitations and Future Work

This study was aimed at showing how the complexity of different crossing scenarios can be ranked by means of a simulated environment. In brief, the analysis clearly showed that some factors can be perceived as more taxing than others by pedestrians, leading to increased complexity, e.g., higher levels of traffic approaching from multiple directions. The overall goal of the study was to validate more complex simulated scenarios for the realistic evaluation of pedestrians’ interactions with AVs via eHMIs. This was achieved through a human factors approach by looking at objective and subjective data to measure the complexity and realism of the simulated environments and will provide a starting point for future work to test different eHMIs.

Future research will be carried out using a more complex scenario, such as the one described in Section 5.4, as a test environment to evaluate different eHMIs designs. Although higher complexity might not necessarily translate into altered crossing behaviour, it is still possible that this might be the case in some circumstances, e.g., when eHMIs are present. Therefore, testing eHMIs in such a scenario can bring to light potential safety and efficiency issues which might not arise in a simpler crossing situation, leading to the development of effective countermeasures. The authors suggest that auditory stimuli should be embedded in test scenarios in future eHMI research to match as authentically as possible real-life crossing conditions, as confirmed by this study.

Although the research employed in this study was aimed at eHMIs for AV–pedestrian communication in particular, the approach followed could benefit other application domains such as road traffic control, pedestrian facilities and infrastructure optimisation by addressing aspects of the traffic environment that can pose a challenge to pedestrians and affect traffic flow and safety. Scenarios could be scaled up to involve multiple pedestrians to assess group behaviour and interactions in this context. Additionally, this research could be adjusted to other road-vulnerable road users, e.g., cyclists, to investigate traffic negotiation strategies and interactions with AVs in more complex scenarios.

There are limitations to this research that the authors would like to acknowledge. One limitation, which is intrinsic to the use of VR, is that participants crossed the road in a risk-free environment, thus making it possible for them to exert less caution than they would in real life. However, research has shown that VR can be successfully used to investigate pedestrian crossing behaviour, with crossing decisions being similar in VR to in the real world [119,120]. Furthermore, VR provided the flexibility needed to manipulate the crossing environment as required, which would not have been possible otherwise, thus providing a useful and cost-effective research tool.

Another limitation is the rather small sample size involved in the study, which might have resulted in some significant differences not being detected amongst the variance in the data. Although there were differences in the subject responses between scenarios, only a few of these were significant, and many scenarios overlapped for the different indicators. Therefore, although the results provide an overview of how the scenarios performed across measures, they were mainly informative and should be interpreted cautiously. Nevertheless, the study still provides valuable insights into how crossing scenario complexity can be investigated. Future work might replicate this study with a larger participant sample for a higher statistical power. Additionally, only non-disabled pedestrians took part in this study, limiting the reach of the findings. However, users with different needs should be involved in future research, and inclusive simulation methods and tools should be considered to this extent, such as the ones proposed by Schieber et al. (2022) [121] and Liu et al. (2021) [122] for visually impaired people. Another limitation was the limited number of scenarios tested in this study, dictated by practical reasons. Future research could expand the number of crossing case studies and explore the interaction effects between individual factors. Finally, as the field of view was used as a measure of gaze behaviour, it was not possible to record the precise gaze patterns of the participants. In future research, eye-tracking data could be collected for a more in-depth understanding of how complexity impacts pedestrians’ visual search strategies when crossing, especially when eHMIs are present.