Human Factor in Navigation: Overview of Cognitive Load Measurement during Simulated Navigational Tasks

Abstract

This paper is intended to give an overview of the experiments to evaluate the cognitive load of the officer on watch (OOW) during a collision avoidance maneuver in a full-mission simulator. The main goal is to investigate the possibilities of recording the biometric parameters of an OOW during a simulated collision avoidance maneuver. Potentially dangerous navigation errors known as human erroneous action (HEA) are induced by excessive cognitive load. Despite modern navigational aids on the ship’s bridge, investigators of maritime incidents typically link the reason for incidents at sea with human factors, including high cognitive load. During the experimental tasks on the bridge, the biometric parameters of the OOW are recorded. Statistical tools are used to visualize the data and evaluate the cognitive load of the OOW. Biometric peaks of the OOW typically occur either during the collision avoidance maneuver or when the OOW has been exposed to disturbing factors that increase reaction time and cause potentially dangerous navigation. Assessing the cognitive load of OOWs in the simulator is challenging for several reasons: e.g., the environmental conditions of the simulator, the type of task to be simulated, and even the type of sensor used. After careful study of the available literature, an original experimental design using non-invasive biometric sensors is proposed.

1. Introduction

In recent years, the trend of the officer on watch’s (OOW’s) cognitive research has risen. As a result of a literature study in the maritime field, the experiments conducted during actual ships’ operation are challenging due to uncontrolled environment parameters, like strong sunlight [1] or unexpected traffic situation [2], which is likely to affect the results. During the research, it has become apparent that the training and cognitive load research are rarely conducted experimentally by exposing participants to marine simulators’ safe environment ([3], Tables 1–3). Despite the aim to contribute maritime safety, the simulator and technical equipment have limitations. Thus the generalization of the OOW’s cognitive load recorded in the simulator, with the cognitive load during actual ships’ maneuvering, is not always an easy task.

Nevertheless, the simulation is the safest way to expose the stakeholders to simulated danger without risking the collision, losing property, and marine pollution. The observed high cognitive load during experiments’ is induced by simulated traffic or weather conditions or additional tasks of the OOW (e.g., n-back test) [4]. According to the analysis of maritime accidents in the merchant navy [5], 75% of the incidents are caused by HEA (human erroneous action), mainly due to the high cognitive load of the OOW. Despite the most modern electronic navigation equipment integrated with a modern ship’s bridge, the OOW remains responsible for positioning and decision making during navigation and berthing [6,7]. In the available studies dealing with the cognitive load research on the ship’s bridge, the OOWs’ cognitive load is typically correlated with working (practical) experience. Studies, where the comparison between experienced officers vs. trainees [8] or students vs. experienced officers [9] reports, experienced participants perform specified tasks better. The findings are consistent with the theory of information processing in working memory [10]. As work has an impact on both human safety and health [11], the recommendation regarding workload is included in the International Maritime Organization (IMO) conventions [12,13], which proposed recommendations for the working environment and working hours to avoid high cognitive load generally, and thus HEA [14].

The goal of this paper is to investigate the impact of the distraction on the observed biometrical parameters. The cognitive load increases when certain situations occur; distractions such as fire alarms, dangerous overtaking of a ship, or modern distractions like checking social media websites, occupy the working memory, which cannot process all available navigational information at once, resulting in a longer response time. In a collision avoidance maneuver, even under normal circumstances, the OOW has but a limited amount of time available, so any increase in reaction time due to high cognitive load will result in a potentially dangerous HEA, resulting in minor failures, misunderstandings, major errors and even direct rule violations [14]. Cognitive load studies using the nautical simulator as a test environment are summarized in Section 2—Materials and Methods. The direct assessment reflects available maritime studies that have observed stress hormone levels or analyzed brain wave intensities to assess workload directly. Indirect approaches typically use eye tracker data, heart rate, or response time values to analyze an OOW’s workload. For method comparison, studies from a road traffic area are also considered [7,15]. A preliminary testing proposal based on the literature study is described in Section 3, where OOW’s biometric parameter measurement was conducted. By measuring pupil diameter, heart rate, blood pulse, electrodermal activity, wrist acceleration, the biometric equipment’s connectivity is tested, together with the visualization of the complex biometric results to assess the influence of the disturbance factor. In Section 4, the conclusion is delineated, highlighting future research directions of the full experimental design for quantifying the cognitive load of OOWs during their navigation tasks.

2. Materials and Methods

2.1. Related Work

In this section, a decision on the most suitable methods of cognitive load measurement will be described as well as the typical experimental design. According to the available literature, two approaches for collecting an OOW’s biometry are typically conducted: direct, and indirect (

Table 1. Biometrical data collecting.

Direct Approach	Indirect Approach
Brain waves intensity (EEG)	Heart rate
Stress hormone rate (Cortisol)	Electro-Dermal Activity
-	Pupil diameter Electrocardiogram (ECG)
-	Body acceleration

).

The advantage of a direct approach is that it yields a more in-depth view of the cognitive process. Brain wave intensity measurements with an electroencephalograph (EEG) observe brain activity and extract cognitive load data in real-time. The challenge with using this method is the difficulty of interpretation of the recorded EEG brain wave signals due to the noise in the raw data. Nevertheless, this method provides insight into the mind process, from which the cognitive load [4] and psycho-physical state (emotions) [16] of the participants can be understood. Noise in the raw data is induced by invasive multi-wire sensors; and procedures in which sampling of the participant’s stress hormone cortisol [8] was found disturbing and stressful, affecting the clarity of the final results. Comparison of the direct approach cognitive load studies involving a nautical simulator is challenging due to different research aims and different sensors used in the experiments (

Table 2. Direct workload assessment.

PAPER	Main 2018	Liu 2017	Miklody 2017
Environment	Simulator	Simulator	Simulator
Task	Maneuvering	Overtaking	Maneuvering
Aim	Stress	Emotions	Workload
Sensor	Cortisol	EEG	EEG
Questionnaire	Nasa TLX	Profile	Profile
Statistical tool	ANOVA	-	-
Processing raw data	-	Support vector ML	Spectral decompositions ML
Reason for cognitive load	Weather	Traffic	Weather, n-back test

). Considering this fact and with a careful study of the available literature, a less invasive approach was proposed in our experiment. On the other hand, processing raw data with the machine learning (ML) support vector method [15] and the spectral decompositions method [2,4], was found useful in the post-processing phase, including data mining.

Sensors in the indirect approach are less invasive. Thus, noise in raw data due to wearable sensors is typically lower compared with expected with EEG or with the stress hormone level method. According to the available studies (

Table 3. Indirect approach of cognitive load assessment.

PAPER	Kim 2005	DiNocera 2015	Hareide 2019	Fredman 2018
Environment	Simulator	Simulator	Navy Ship	On-Road
Task	Awareness	Navigation	High speed	Car driving
Aim	Decision-time	Workload	Visual attention	Workload
Sensor	Timer	Eye-tracker	Eye-tracker	Camera
Questionnaire	Profile	Nasa-TLX	Profile	Profile
Statistical tool	t-test	ANOVA	Average	Classifier
Processing raw data	Custom SW	Toby-Studio	TobyPro2	ConvNet
Reason for cognitive load	Traffic	Traffic	High speed	n-back test
PAPER	Saus 2012	Orlandi 2018	Kim 2007	Arenius 2010
Environment	Simulator	Simulator	Simulator	Simulator
Task	Navigation	Berthing	Navigation	Navigation
Aim	Awareness	Workload	Performance	Human error
Sensor	ECG	EEG, ECG	ECG	Eye-tracker
Questionnaire	Personality test	NASA TLX	NASA-TLX	Interview
Statistical tool	Regression	ANOVA	-	-
Processing raw data	rMSSD *, Interplt. *	MatLab	t-test	-
Reason for Cog. load	Traffic	Familiarity	Alcohol	Emotions

* root mean of the squared successive differences, interpolation.

), many different body markers support the indirect approach—e.g., reaction time [9], heart rate (HR), electro-dermal activity (EDA), electro-cardio graph (ECG), [17,18,19], pupil diameter, [1,20,21], and acceleration of the body’s extremities. Even when measured indirectly, they can achieve a high degree of accuracy of up to 90%, indicating a level of cognitive load validated in several experiments tailored for key operators of different modes of transport [15]. As shown in Table 3, cognitive research in the maritime field in recent years has different aims, although a common denominator is based on cognitive research. In these studies, typically, eye-tracking glasses are used to reduce the disturbance factor caused by classical-type wired biometrical sensors. Eye-tracking manufacturers also provide high-tech software based on ML classifiers, which helps us gain additional information on cognitive load by data post-processing.

After the presented papers were carefully studied, it was decided to collect biometrical data with pupil diameter, heart rate, EDA, and wrist acceleration.

2.2. The Current Experiment

For the current experiment (Figure 1), five captains volunteered to participate. They were, on average, 48 years old and had 20.5 years of navigation experience. The primary source for data collecting are observables from biometrical sensors. The simulated ship’s bridge log-files and data from interviews (questionnaire) are secondary sources of data. By comparing participants’ experimental and control phases, the preliminary analyses and visualization of the biometrical data are performed (Figure 2). Like a computer processor, more tasks in the OOWs’ working memory (during navigation) cause higher cognitive load and consequently prolong reaction times, which may eventually lead to a potentially dangerous situation [10].

The experiment is carried out in the full mission marine simulator, Wärtsilä’s TechSim5000, where the navigation bridge’s design is comparable to that of the latest merchant ships. Four video projectors provide a 170° forward viewing angle during the voyage simulation, ensuring a high degree of realism. Side and aft views are afforded by LCD monitors on port and starboard bridge wings with a visual angle and zoom option. Environment conditions within the bridge simulator are recorded by a temperature sensor, humidity sensor, noise sensor, an illuminance sensor to measure light conditions and provide the same testing conditions for all the participants. There is only one OOW on the bridge during the simulation, to control the cognitive load-sharing effect, which typically appears in multi-participant’s experiments [16]. Navigational tasks, devised by experienced captains, are designed for the northern Adriatic traffic separation zone [22,23]. The task is divided into two groups, easy and hard; each consists of 15 min of navigation. The easy task aims to collect reference biometrical observables (control phase). Hard navigation (experimental phase) consists of collision avoidance in the traffic separation zone, where several ships’ routes cross when they head to the ports of Koper, Trieste, and Monfalcone. During the hard task, the OOW has experienced the disturbance factor—the fire alarm’s sound—simulating a fire in the engine room. The aim is to determine if and how the alarm’s sound affects the volunteering OOW’s biometrical markers (and cognitive process) during the navigational task. More details on the participants are further explained in Section 3.

2.3. Raw Data Sampling

For biometrical data recordings, participants wear two sets of sensors: on the head and the wrist. Head-mounted “Pupil-Core” eye-tracking glasses measure average pupil-diameter, and the Empatica E4 wristband measures multiple parameters as presented on

Table 4. Sensor’s sampling rate.

Type of Parameter	Manufacturer	Sampling Rate
PUPIL DIAMETER (mm)	PUPIL CORE	200 Hz
EDA (μS)	EMPATICA	4 Hz
BVP	EMPATICA	64 Hz
SHIPS’ PARAMETERS	WäRTSILä	10 Hz
HEART RATE (BPM)	EMPATICA	1 Hz
ACCELERATION (m/s2)	EMPATICA	32 Hz
TEMPERATURE (°C)	EMPATICA	4 Hz

. Ship’s parameters, e.g., ship’s speed, the telegraph position, rudder position, ship’s heading, and engine RPM, are recorded in the simulator’s log file and at the end of the simulation extracted as a CSV-file. As per Table 4, the sampling rate of each sensor is different due to the Nyquist rate, wherein the sampling frequency is twice the maximum frequency of the signal being sampled, which prevents loss of the information.

3. Results

3.1. Experimental Procedure

The quantification of the human factor has been the subject of research in all transport sectors for many years (Table 2 and Table 3). This particular study was launched last year when EMSA issued a report which found that 75% of marine accidents are caused by human factors [5]. Human factor deficiencies include lack of competence, inadequate supervision, inattention, and certain states of working memory of the OOW. The working memory in our brain consists of two parts from an engineer’s point of view. The first part performs logical and cognitive operations. The second part is a limited temporary memory in which data from our body sensors is stored before the cognitive part processes it. Thus, cognitive load is a term that describes the mental effort involved in processing information. Mental effort causes typical biometric changes that can be measured [10].

The following study is an attempt to measure the rate of biometrical changes caused by mental effort during a simulated navigational task, the connectivity of the biometrical equipment, the impact of the chosen disturbance factor, and the visualization of the complex biometrical results. The aim is to isolate the weak points of the proposed experimental design before engaging in a full-scale experiment.

Each of the participants was given the same instruction before the experiment. The first simulation is collision avoidance under normal (typical) conditions, and the second simulation is affected by the disturbance factor (sound of fire alarm). In the particular experiment, the large amount of data was not obtained by a large number of participants, but by a wide scale of biometrical markers acquired on the small (N = 5) homogeneous sample of experienced OOW. For the determination of the change in the cognitive process in the working memory, we assume that a statistically significant change in biometric markers (e.g., heart rate and EDA) indicates the state of cognitive load of the OOW [10]. As seen in Figure 2, during the collision avoidance simulation, multiple data are processed in the brain (e.g., target distance, CPA, TCPA, speed, and course, etc.). After the disturbance factor is added, the cognitive process in working memory becomes high. Thus, the OOWs’ reaction becomes significantly different than under the standard (control) conditions. The described negative effect reduces the time for collision avoidance, which is limited even in normal conditions. When the OOW is facing a lack of collision avoidance time, and with the appearance of the disturbance factor, typical measurable biometrical changes occur (Figure 3). Increased average values of the observables in the time when the disturbance factor is involved increase the pressures of the cognitive load. According to the literature study, invasive methods typically cause noise in the results. Consequently, we pay special attention to the selection of the least invasive wireless sensor devices. The environmental conditions (e.g., temperature, humidity, noise level, and light conditions) have a great influence on the biometric readings; therefore, they are controlled during the task. This means that the environmental parameters of the simulator are the same for all participants. The room temperature is set to 23 °C, the humidity to 38%, the noise level is 45 dB with a peak value of 75 dB during the alarm.

3.2. Results

The typical data from the biometrical sensors (Figure 3) shows the moment where disturbance factor occur between T1 and T2. Again, the disturbance factor is the sound of the fire alarm with its red rotating light. Biometrical values in the demanding phase become higher compared with the control phase. We experienced issues if the right-handed participants wear the accelerometer on the left wrist. More accurate readings from the sensor are recorded when the accelerometer was on the working wrist. The average HR rate of participants in the test phase is significantly higher compared to navigation in the control phase. Blood volume pulse (BVP) shows higher values during the disturbance factor interval. The more the blood-vessels expand (vasodilation), the higher is the amplitude of the signal, which implies the brain’s high cognitive process. The heart rate (HR) is denoted by the distance between the peaks (the Inter-Beat Interval (IBI)). However, the analysis of the individual biometric reaction shows that the participants vary in the degrees of reaction recorded by the biometric data. The differences seem to correspond to the level of experience of the individual participants in their profession. From the literature study, we assume that the biometric response is related to the personality of the participant (openness, conscientiousness, extraversion, comfort, and neuroticism), including a potentially post-traumatic stress response that occurred when the participant confronted with a situation remembers the dangerous experience from reality [10].

In the current experiment, we faced an unexpected challenge with pupil diameter measurement. Participants with correction glasses could not wear an eye-tracking sensor at the same time. Thus this part of the experiment was excluded from this research.

3.3. Post Processing and Data Visualization

In the post-processing phase, the log files from sensors are resampled, synchronized, and processed with Excel, E4-Connect firmware, and Python. This phase aims to visualize the complex biometrical results that determine the influence of the disturbance factors on the cognitive load. After the simulation starts, the navigational task begins, and the participant takes control over the ship’s bridge (Figure 4). The moment of processing the available navigation information in the memory of the OOW from simulated bridge instruments causes the jump of biometric signals EDA, HR, and BVP, which indicate a high cognitive load. The acceleration of the wrist is high, which is indicated by the detection of the moving hand on the radar screen and on the electronic chart ECDIS. The biometrics during the disturbance factor (Figure 5) shows higher values during all disturbance factor sequences. An interesting observation is the gap of the BVP pulse shortly after the initiation of the disturbance factor. The gap indicates an expectation due to real-life experience that the alarm is not essential [24] and will be silenced shortly, or that the participant’s working memory dismissing this sound as likely a false alarm [25]. That is not the same as ignoring an alarm but is yet another variable that likely, at times, affects the behavior of people in perilous situations. Responding to impending unexpected disasters, there is also a tendency toward disbelief that affects certain human personality types.

The EDA observation confirms the assumptions from the literature that participants have a measurable response to stress. In our case, collision avoidance and noise of the disturbance factor are reasons for stress and cause changes in skin conductance, so that the EDA varies (Figure 5). The data on the acceleration diagram shows the movement of the participant’s wrist. The higher intensity and amplitude are observed at the moment of actual collision avoidance when the participant’s wrist jumps quickly from one navigational instrument to another to avoid the dangerous obstacle safely.

4. Discussion

In typical cognitive load studies, including those conducted in a nautical simulator (Table 2 and Table 3), the experimental design is based on the OOW’s biometrical markers’ readings. The authors face challenges regarding the experimental design and type of sensors used. The major shortcoming of the studies where the N is relatively small, in our case N = 5, the generalization of the results is not applicable. However, the particular experiment was not intended to include large numbers of participants. Instead, we assessed a small homogeneous sample of OOW with a wide scale of biometrical markers to indicate a broad spectrum of biological and behavioral responses to the task estimate tendencies that are likely to occur during the workload; studies with a larger number of selected markers are awaiting further research. The advantage of this proposed experimental method is applying modern wirelessly monitored sensors to achieve a relatively non-invasive approach, where pupil diameter BVP, EDA, and HR ratio as biometrical indicators for assessing the cognitive load are recorded. Non-invasive methods have the advantage of eliminating the biometric reaction caused by wearing a sensor, influencing the results. Unfortunately, at the beginning of the experiment, we found that most volunteers wore glasses. Participants with correction glasses could not, at the same time, wear an eye-tracking sensor. After we equipped the OOW with an eye-tracking sensor only and without the participant’s correction glasses, the participants could not clearly see the navigation monitors. The conclusion was that even before starting the experiment, the OOW’s biometry showed stress and indicate a high cognitive load. Consequently, the pupil-diameter measurement was excluded from the current experiment.

The current test shows a conspicuous difference in the average values of the biometrical markers between timelines in Figure 4, where disturbance factors occur. Readings of the parameters EDA and HR show higher average values when disturbance factors occur, which is promising for further research. The participants’ self-evaluation reveals that the fire alarm’s sound as a disturbance factor evoked a stress response, which affected the cognitive load during the collision avoidance maneuver and, consequently, the HEA (Figure 2). The assumption is that the mentioned fact is more evident in participants which in his career facing dangerous situations onboard which indicate to post-traumatic response, as explained to the participants during the self-evaluation interview.

Statistically, the distance between the upper and lower quartiles represents the interquartile range (IQR). The box plot visualization (Figure 6) shows a five-digit summary from bottom to top: the minimum line, the 1st (25%) quartile, the sample median, the 3rd (75%) quartile, and the maximum line. The dots below the minimum and above the maximum represent the samples that exceed the 1.5 of the IQR range either below the 1st quartile or above the 3rd quartile.

Observation of the HR rate (Figure 5) reveals a comparable participant’s HR pattern during the collision avoidance (CA) task, with a median 79.05 BPM, which we assume is a typical condition. With disturbance factor simulation (DF), the HR median rises to 84.65 BPM compare to the CA task. What is interesting is the appearance of the outliers. There are two explanations for them. First, we consider whether there is a sensor issue or some similar hardware problem. If the pattern repeats, we consider the second explanation, which is the participants’ characteristics. In the interview, participants confirmed that dangerous situations experienced on board in the past explained HR extremes (outliers) seen on the DF-P1 and DF-P5 (Figure 6). At zero visibility task (ZVT), the safety of navigation depends only on navigation instruments. We assume that the brain’s cognitive process will increase due to the saturation of the working memory. The median, in this case, rises to 92 BPM, which indicates a higher cognitive load due to the lack of visibility.

The aim of the experiment was to determine the appearance of stress and high cognitive load with the biometrical markers. The composite of the biometrical data is a promising approach to the OOWs’ cognitive load quantification, although to develop a mathematical-based model of the machine learning algorithm, which recognizes an OOWs’ high workload, enough biometrical sample data (training data) must be provided. Thus, a full-scale experiment will be conducted.

5. Conclusions

This study attempts to represent a cross-section of the cognitive load measurement during a simulated navigational task with an experimental proposal. The simulated navigational task is approaching the pilot station trough the northern Adriatic traffic separation zone with moderate traffic density. The experiment assumes that the OOW’s biometrical parameters change when the OOW faces a disturbance factor—the sound of a fire alarm—during collision avoidance. The disturbance factor in on real ships takes many forms; e.g., all sorts of navigational alarms and environmental conditions to the latest disturbance factor in the form of distraction from surfing on social media sites. For the present experiment, we chose an alarm indicating a fire in the engine room, which, we assumed, would induce stress and raise the cognitive load of experienced participants. The BVP, HR, and EDA are primary biometrical observables collected from a multi-sensor wristband on the OOW’s wrist. The experiment reveals measurable biometrical responses, and even with a limited number of biometrical sensors indicates the state of stress and high cognitive load. The median of HR during collision avoidance at normal conditions is 79.05 BPM. At disturbance factor simulation, the HR rises for 5.6% to the 84.65 BPM. The highest rate, as expected, is at zero visibility task, where HR raised for 13.67% to 92 BPM. The results are encouraging for future research into cognitive load quantification.

We are aware that the present sample is small and generalization at this phase of the experiment is challenging. Additionally, the results (and temptation for generalization) can be attributed to the homogenous sample, where participants share a similar response due to navigational challenge. The methods were chosen on purpose. We aimed to test a wide range of biometric markers so that studies in the future can further elaborate on those that have previously proved useful. Thus, for a full-scale experiment, we prepare a combined quantitative/qualitative approach supported with a machine learning algorithm that will be developed to recognize the state of cognitive load of the participant during the simulation. The next step would be to transfer measurement into reality.