EEG-Based Performance-Driven Adaptive Automated Hazard Alerting System in Security Surveillance Support

Abstract

Automated vision-based hazard detection algorithms are being rapidly developed to provide hazard alerts for construction workers. However, these alerting systems often apply a fixed low-beta alerting threshold, which can cause excessive false alarms, followed by distractions and human distrust in automation. In this study, we propose a novel adaptive automated hazard alerting system capable of adjusting alert threshold levels based on environmental scenarios and workers’ hazard recognition performance evaluated using a wearable electroencephalogram (EEG) sensor system. We designed a hazard recognition experiment consisting of multiple hazardous scenarios and acquired behavioral data and EEG signals from 76 construction workers. We used the linear ballistic accumulator model to decompose hazard recognition into several psychological subcomponents and compared them among different scenarios. Subsequently, our proposed strategy includes clustering of participants’ hazard recognition performance levels based on latent profile analysis, wavelet transform of EEG signals, transfer learning for signal classification, and continual learning to improve the robustness of the model in different scenarios. The results show that the proposed method represents a feasible worker-centered adaptive hazard alerting approach. The anticipated system can be leveraged in a real-world wearable headset application that aims to promote proactive hazard intervention and facilitate human trust in automated hazard alerting technologies.

1. Introduction

Safety surveillance is critical for proactive management during construction to mitigate possible hazards ahead of time [1]. Several security surveillance systems relying on closed-circuit televisions [2], stereo-vision cameras [3], unmanned aerial vehicles [4,5], and LiDAR [6] have been developed to provide remote solutions for project monitoring. However, operators who monitor these captured image data for hazard inspection face several inherent cognitive challenges, such as bias [7], fatigue [8], complacency [9], stress [10], and distractions [11]. As a result, up to 50% of hazards are unrecognized during manual observation [12,13]. To address the limitations of current manual efforts, various state-of-the-art computer-vision (CV) technologies that create situation assessments that enable diagnosis, reasoning, and decision support have been developed [14,15,16]. Although advanced automation is now being developed and continues to become increasingly autonomous, the value of automated systems resides not in their total replacement of human operators but rather in their ability to augment operators’ capacities [17]. Thus, hybrid human–machine collaboration (HMC) systems for hazard detection have emerged, in which supporting human trust in automated alerting techniques is the key to successfully implementing the HMC strategy for construction safety improvement [18,19].

Determining the alert threshold is a critical challenge in the design of automated diagnosis and alarm systems. Most alarm systems have a low beta threshold because the costs of misses (e.g., worker injuries and fatalities, structural collapse, and electric leakage) are typically much greater than the costs of false alarms. However, a predictable issue is that this causes most alerts to be “false alarms,” which has two negative consequences. First, because it is the human who makes the final decision, frequent false alarms will make the human cross-check the raw data to ensure that the alert is indeed false, which may create distractions and lead to the expenditure of unnecessary efforts [20]. Second, and more seriously, after excessive false alarms, people may develop a “cry wolf” syndrome that might result in them responding to alerts (including those that may be true) late or ignoring them altogether [21], which may further cause catastrophic system failure and fatal accidents [22,23,24]. With the development of multiple wearable sensing devices (WSDs) as ergonomic tools for physiological monitoring to provide early warnings [25], the problem is amplified when an operator receives alerts from several independent systems. To prevent potential problems, automation must not only go beyond acting as human backup but also support the process of human trust [26,27].

Previous research has shown that multiple alert threshold settings in automated hazard diagnosis systems can improve HMC experience [28]. Thus, to address the problem of excessive false alarms and improve human trust in automation, we propose an adaptive automated diagnostic tool with alert thresholds that may change according to environmental scenarios and human operator performance during system operations to improve on-site security surveillance. Brain–computer interface (BCI) technology is highly recommended to enable direct communication between humans and automated systems [29]. Researchers in the construction domain have developed BCI applications that decode human intentions from brain activity patterns [29,30] and subsequently transform them into commands to control a robot [31].

Inspired by the intuitive communication enabled by BCI in HMC, this study sought to introduce BCI technology to infer the human operator’s hazard recognition performance from brain activities captured by EEG and thus construct a performance-driven adaptive hazard diagnosis and alarm system. Although researchers have extensively explored the use of physiological sensors to monitor operator workload and enable the machine system to react accordingly, previous studies also suggest that workload-driven adaptive systems are deficient in balancing cognitive workload savings and situational awareness elicitation. To address this dilemma, the strategy of a performance-driven adaptive alerting system is proposed, and the feasibility of an EEG-based BCI system that aims to enable intuitive communication of hazard recognition performance levels is explored. Our proposed strategy also overcomes previous BCI prediction models’ low robustness and plasticity in handling changing scenarios by adding a generalization module enabled by continual learning. The main novelty and contributions of this study are summarized as follows:

• A detailed comparative study was conducted on EEG-enabled adaptive systems in the field of construction. The results revealed that a performance-driven adaptive system can work more robustly than the commonly proposed workload-driven system for HMC hazard inspection.
• To examine the feasibility of the proposed EEG-based performance-driven adaptive aiding system, a case study was conducted to verify the hypothesis that an EEG-based BCI can reliably distinguish brain activation patterns elicited by participants with high, medium, and low hazard recognition performance levels.
• To the best of our knowledge, this study is the first to investigate how continual learning can improve the skill transfer between scenarios in EEG-based BCI systems.

The remainder of this paper is organized as follows. Section 2 describes previous research on the concept and implementation of adaptive automation. Section 3 introduces the experimental setup and the methodology. Section 4 and Section 5 present and discuss the results of the data analysis. Finally, Section 6 outlines the conclusions and identifies directions for future research.

2. Related Work

2.1. Conceptualizing Adaptive Automation

To promote human trust and calibrate appropriate reliance on automation, two conceptually attractive strategies–adaptable/adaptive automation–have been proposed in the field of HMC [32]. The first emphasizes giving human operators the choice to invoke or remove higher levels of automation by self-monitoring their capacity to perform tasks, which helps operators adapt better to the functional characteristics of the working system [33]. Following this idea, appropriate operator selection and training are needed to improve human understanding of the automation principle [34]. However, a prominent relevant concern is the tendency of humans to be overconfident and inaccurate in the subjective estimates of their performance and workload [35]. It was also reported that this adaptable approach could only improve the overall system performance to a limited extent [36]. Therefore, it is suggested that “for a viable future, technology must adapt to the human, which underwrites the necessity of human factors science” [36]. Adaptive automation, proposed in the 1990s, provides a new form of HMC by allowing automated systems to automatically adapt to humans as a function of the environmental state, human state, or task performance [37,38].

2.2. Workload-Driven vs. Performance-Driven Adaptive Automation

In recent years, various WSDs have been developed that can be used to track workers’ physical activities [39,40], locomotion [41,42,43], and health-related physiological data indicating fatigue [44], vigilance and attention [45], mental workload [11], stress [46], and emotions [47] and provide early warning signs of safety issues to construction workers to mitigate health risks and safety hazards on construction sites [48,49]. Furthermore, mental-state monitoring of workers has been used to construct adaptive joint HMC systems that include a wearable biosensor for assessing workers’ psychological conditions such that the automated system can adjust its working style accordingly to facilitate human trust in automation [27,50]. In an adaptive alerting system, the alert threshold levels should be adjusted automatically according to the status of the operators. More specifically, when the operator is detected as working in a suboptimal state (e.g., high workload), adaptive aiding is triggered [51], which is also known as a “human-in-the-loop” approach in the literature [52]. A series of auxiliary systems has been developed based on real-time, EEG-based workload measurements [18]. For example, Liu et al. developed a brainwave-driven HMC paradigm in which robots continuously monitor workers’ cognitive load and adjust their operating speed accordingly [53]. Compared to a manual condition or one where adaptive aiding is provided randomly, physiological adaptive automation of workload tracking has proven to significantly improve targeting performance [27].

Although previous studies have appropriately characterized EEG features for workload-driven HMC systems, some studies have suggested that automated systems may not be embraced by several construction workers. For example, Shayesteh and Jebelli found that instead of reducing, participants’ cognitive load increased when collaborating with an autonomous robot [54]. Previous studies also indicated a trade-off dilemma between cognitive workload and situational awareness, the two safety-critical variables for human operators that are significantly influenced when operators use automated aiding systems [55,56]. According to [57], a decrease in task workload also reduces situational awareness. For instance, drivers reported that driving with adaptive cruise control (ACC) was less effortful than driving manually [58]; however, driving with ACC may reduce driver vigilance and increase driver distraction [59]. Such a dilemma between cognitive workload savings and operators’ engagement elicitation results in workload-driven adaptive alerts that are less reliable for security surveillance support [60].

As the relationships between cognitive state and hazard recognition performance and human trust in automation are still vague [61,62], and the purpose of such a joint system is to improve the overall hazard recognition performance, this study proposed implementing a direct performance-driven adaptive automated alert system, wherein alert threshold levels automatically decrease on inferring that the human operator cannot effectively recognize the hazardous scenario. This performance-driven approach has been proven capable of achieving a good balance between the cognitive workload and situational awareness. For example, Parasuraman et al. investigated the driving scenario and reported that, compared with static automation, adaptive aiding was associated with both reduced workload and increased situational awareness [63].

Previously, without real-time physiological measures, this idea of performance-driven adaptive automation was impossible in practice in the absence of overt behavioral output and ground truth during hazard inspection. In recent years, several studies have explored the possibility of using EEG to predict future cognitive performance [64,65]. This leads to the hypothesis that an EEG-enabled BCI can be established to predict construction hazard recognition performance. However, there is no direct evidence of whether EEG signals from individuals with high, medium, or low hazard recognition performance levels can be differentiated with classification accuracy exceeding the chance level. Therefore, further research is necessary to investigate the potential feasibility of applying such an EEG-based BCI to furnish a performance-driven adaptive aiding system for hazard inspections.

2.3. Transfer Learning and Continual Learning for EEG Signal Classification

To establish a reliable BCI system, robust feature extraction and classification are critical. For feature extraction from EEG signals, studies on WSDs in construction typically select time-, frequency-, or time–frequency-domain features of EEG signals as inputs to machine learning classifiers, such as linear discriminant analysis, support vector machines, and k-nearest neighbor [66,67]. Owing to the high dimensionality of EEG features, methods of feature dimensionality reduction (such as principal component analysis and t-distributed stochastic neighbor embedding) have been used to learn more discriminative EEG features [67,68]. For classification, automated machine learning approaches have been utilized to tune hyperparameters for optimized classifiers [67]. Although impressive performance has been achieved by these machine learning classifiers, the manual effort required to select appropriate EEG features and tune hyperparameters in these streamlined studies could discourage researchers from further exploring wearable EEG-based systems for other applicable scenarios, as it poses high technical barriers with expertise requirements regarding both EEG signals and an understanding of engineering tasks. In addition, recent years have witnessed a blooming investigation of construction safety from neuroergonomics perspectives [66]; however, the conversion of cognition-wise findings to WSDs in practical use still has a long way to go, particularly considering the dynamic nature of real sites compared with that of confined laboratory task settings.

To overcome these difficulties, researchers have turned to an end-to-end analysis approach for EEG signal classification enabled by deep learning instead of handcrafting EEG features and deploying traditional classifiers. Their findings demonstrate exceptional classification performance for EEG signals provided by a convolutional neural network (CNN) in various cognitive tasks [69,70]. More recently, the paradigm of transfer learning has been explored to take advantage of well-established CNN models for image classification and has been proven to perform well in classifying EEG signals [71]. Although multiple cognitive tasks have been explored, few studies have considered construction hazards as visual stimuli. Thus, this study sought to investigate the effectiveness of employing transfer learning to classify the brain signals induced by various construction hazards. Furthermore, to transform the time-series EEG signals into inputs in a two-dimensional image format that can be fed into CNNs, this work takes inspiration from previous studies wherein wavelet transformation was conducted to transform raw EEG signals into the time-frequency domain to obtain temporal-spectral EEG representations [72,73].

Another advantage of our proposed approach is that we consider the fact that construction workers are faced with changing scenarios when on duty and the fact implied by previous studies that the human brain activates differently in response to different hazardous scenarios [30,67]. Although previous researchers have made great efforts to train classifiers to classify EEG signals in an attempt to achieve as high an accuracy as possible, their efforts have mainly been refined to a predesigned task setting [66,67]. When deploying the proposed EEG-based WSDs from the laboratory to the real world, the robustness and precision of the model predictions are still in question. This is because these models are incapable of adapting to changing scenarios. In fact, prediction models can forget most of the knowledge learned from previous tasks after training on subsequent tasks, which is known as catastrophic forgetting [74,75]. Previously proposed static models cannot meet the requirements of a real-world adaptive diagnosis aiding system, as they cannot perform robustly when changing scenarios are presented over time. In particular, when attempting extremely high accuracy in classification, we occasionally run into the stability–plasticity dilemma, where extreme stability makes it difficult for the model to learn sequential tasks. In contrast, excessive plasticity can cause the forgetting of previously learned information [76,77]. A possible solution to this dilemma is to repeat the training process using an extended dataset that includes both previous and current data. Nevertheless, repeated training on larger datasets is computationally intensive. To the best of our knowledge, this study is the first to emphasize this issue of data distribution shift for a real-world EEG-based adaptive aiding system and propose an approach to facilitate the prediction model to achieve a good balance between stability and plasticity in a resource-efficient manner via continual learning strategies.

3. Methodology

3.1. Participants

Seventy-six construction workers with normal or corrected-to-normal vision participated in this experiment. All workers were recruited through the real estate management office at Tsinghua University and worked on campus construction sites. One participant was excluded from the analysis because his EEG signals contained excessive artifacts. Five participants were considered unreliable in the validation test (see “Stimuli and Experimental Protocol”). The final sample consisted of 70 male participants (13.2 ± 9.7 years of work experience, range = 0.5–35 years; age = 42.2 ± 9.7 years, range = 21–60 years; all Chinese). All participants signed an informed consent form before participation and received RMB 100 as monetary compensation. This study was approved by the Department of Civil Engineering at Tsinghua University.

3.2. Stimuli and Experimental Protocol

All stimuli (images) were retrieved from an in-use construction safety management platform, a repository for safety reports from numerous projects [78]. The experiment consisted of 60 pairs of construction scenes, each having two opposite conditions (hazardous or safe).

The participants were required to complete a hazard recognition task: they viewed images of real-world construction scenes displayed on a computer screen and judged whether they were hazardous or safe (Figure 1a). Prior to the official experiment, the participants were instructed on the experimental procedure and completed 10 trials to familiarize themselves with the task. The stimuli used in the practice session were different from those used in the official session. In the official experiment, 120 images were presented in randomized order. To alleviate fatigue, a one-minute break was imposed every 30 trials, during which participants were instructed to sit back and relax with their eyes closed. Finally, a validation session was conducted, in which participants responded to 30 trials randomly selected from the previous 120 trials. The consistency of responses to the same stimulus was checked, and participants with an inconsistency rate above 50% were excluded from the data analysis. Five participants were excluded from this study.

The procedural details of each trial are shown in Figure 1b. Each trial began with a fixation cross that appeared for 500 ms. Thereafter, an image depicting a construction scene was presented for a maximum of 3000 ms, followed by a blank screen for 500 ms. Subsequently, a response screen was shown, during which the participant was required to report his judgment of the construction scene seen before by pressing the corresponding key on the keyboard (“0” for safe and “1” for hazardous). No time limit was set for the response screen. On average, the official experimental session lasted approximately 14 min.

3.3. EEG Data Recording and Preprocessing

A 32-channel electrode cap (Neuroscan system in China and Brain Products in the United States, according to a 10–20 system) was used to record EEG signals at a sampling rate of 250 Hz. The EEG was referenced to all the electrodes. The impedance of each electrode was carefully adjusted and maintained at less than 20 kΩ before data recording. Offline analysis of EEG data was conducted using the FieldTrip toolbox [79] in MATLAB (version R2019a, MathWorks, Inc., Natlick, MA, USA). The EEG signals were initially treated using band-pass filtering (0.1–40 Hz). Following previous EEG-based real-time system designs in the construction field [80], independent component analysis was conducted to remove the possible artifacts (heart rate, respiration responses, eye movements, etc.) from the EEG data. The corrected data of each trial were segmented into an (–200, 1000 ms) epoch (0 ms denotes the stimulus onset) with a 200 ms pre-stimulus baseline correction. Subsequently, epochs with values exceeding ±100 μv for any electrode were rejected to avoid possible artifact contamination.

3.4. Data Analysis

Figure 2 shows a flowchart of the study methodology.

Table 1. Descriptions of the three hazard types.

Hazard Type	Descriptions of Hazards	Condition	Example Image
Electric leakage	Overhead power lines, unprotected electrical panels, unclosed electrical compartments, etc.	Hazardous
		Safe
Lack of edge protection	Slips, trips, and falls from a height (elevator hole, stair edge, suspended platform, etc.) without edge protection.	Hazardous
		Safe
Structural instability	Unstable temporal structures, such as scaffolding, formwork, and shoring.	Hazardous
		Safe

lists the three types of hazards investigated. The electricity-related hazards comprised 28 trials, falling-related hazards comprised 38 trials, and the structure-related hazards comprised 14 trials. We selected these three hazard types because they are reported to be the most common precursors to onsite accidents [81].

3.4.1. Linear Ballistic Accumulator Model

To gain deeper insight into the cognitive mechanisms of recognizing various hazards, computational modeling of response time (RT), which allows the decomposition of RT into several different psychological functions involved in the hazard recognition process, could be conducted. Previous studies have applied the accumulated model to various cognitive processes [82,83]. Although previous studies have typically indexed cognitive differences by mean differences in RT between various hazard types, RT contains a richer amount of information reflecting, for example, information-processing efficiency and response conservativeness. To dissociate a single RT into different subprocesses involved in hazard recognition response decisions, we utilized a linear ballistic accumulator model (LBA) [84]. In LBA, response time is considered the time participants take to accumulate evidence of hazardousness toward a decision threshold for characterizing the scene as hazardous or safe. The hazard recognition process is described using four parameters [85]: drift rate (v), representing the speed of evidence accumulation, that is, efficiency in information processing; the upper limit of the starting point distribution (A), representing the upper limit of the amount of existing evidence at the start of evidence accumulation (which varies across trials); threshold (b), representing the amount of evidence above which a “hazardous” decision is made; and non-decision time (psi), representing the time taken to encode stimuli and execute a response. With these parameters, the decision time is defined as the distance between the starting point and threshold divided by the drift rate (see Figure 3), whereas RT is composed of the decision and non-decision times (psi).

The LBA model parameters were estimated using the RSTAN package [86] written in R (version 4.0.2), which analyzes behavioral data using Bayesian inference. To avoid possible bias, only the trials with correct responses were included in the analysis. In the RSTAN package, the parameters are estimated using the Hamiltonian Monte Carlo algorithm [87], which uses the no-U-turn sampler to sample posterior distributions with correlated parameters [88]. In this study, the Hamiltonian Monte Carlo algorithm (iteration = 4000, warmup = 2000, and thinning = 1) was used to obtain the posterior distribution of each LBA parameter for each hazard type. Four Hamiltonian Monte Carlo chains were run to satisfy the Gelman–Rubin criteria for convergence ([89], with Rˆ close to 1).

3.4.2. Latent Profile Analysis

In this study, we used latent profile analysis (LPA) to segment participants into subgroups with three hazard recognition performance levels (“High”, “Medium”, and “Low”). LPA assumes that people can be categorized with varying degrees of probability into different configurable profiles of personal attributes, which has received growing interest in occupational behavior research in recent years [90]. The tidyLPA R package [91] was used for the LPA analysis. Participants’ recognition accuracies for different types of hazards were used as categorical latent variables. LPA posterior probabilities were used to segment the participants into the corresponding subgroups. Subgroups were named based on the probability of correct responses in all trials for each hazard type.

3.4.3. Time-Frequency Analysis of EEG Signals

The EEG signals were transformed into time-frequency maps [92] in an image format for the following classification analyses. Time-frequency analyses of the EEG signals were based on a wavelet transform approach [93]. The analyses were performed using the FieldTrip toolbox [79] in MATLAB. Following a previous study [94], the time-frequency representations of EEG data were generated using a Hanning taper to achieve a compromise between time and frequency resolutions. A 500 ms sliding window with 50 ms time steps was applied. Each epoch was transformed in the frequency domain, and the time-frequency representation of the EEG signals was computed for each trial of each participant.

3.4.4. Transfer Learning for EEG Signal Classification

As the EEG signals were transformed into images in the previous step, we applied the transfer learning paradigm to conduct signal classification. The concept of transfer learning is as follows. For a classification problem, for example, C_a, we train the model on a set of data (e.g., D_s). Now, for a different classification problem, such as C_b, we do not have to train the model from scratch; rather, we can use the model trained on D_s and apply the learned knowledge to problem C_b. Thus, we take advantage of the existing knowledge. In this study, we selected one of the most popular CNN models, ResNet18, for the classification. The model, trained on approximately 1.2 million images from the ImageNet database [95], has shown impressive performance in several challenging circumstances. Following this, the ImageNet database became the source domain in which the model was pretrained, while the time-frequency maps of the EEG signals were the target domain.

To train the ResNet18 model, we followed the customary approach described in [96]. The top fully connected layer of ResNet18 was replaced with a new fully connected layer of 512 activation units, followed by a final layer with a LogSoftmax activation function to output the prediction. In the fine-tuning process, the early layers were fixed. We trained only the newly introduced layers using the Adam optimizer [97] with a learning rate of 0.003. This was because the earlier layers of the CNNs generated more generic features that could be used despite the data distribution, while the higher-level layers of the CNN devoted more representational power to features that were more specific to differentiating between the EEG time-frequency maps induced by three performance-level subgroups of participants. [98]. To evaluate the classification performance, we split the real data into training and validation subsets with a split ratio of 80%:20%. To avoid overfitting, the training was stopped as soon as the loss on the validation set did not decrease in three epochs.

3.4.5. Continual Learning Strategies

To measure catastrophic forgetting, we first considered per-task baselines, that is, the results of a model trained independently for each task. To check the effect of continual learning, we considered the naïve baseline, which was fine-tuning across tasks. The model was first trained on Task A and then on Task B, starting from the previously learned parameters. The difference between fine-tuning and continual learning is that fine-tuning assumes different tasks without much consideration of the source performance. In contrast, continual learning does not forget the source domain while learning a target domain.

Continuous learning consists of two broad families of methods: rehearsal and regularization. The first assumes memory and access to explicit previous knowledge (instances), and the second only has access to compressed knowledge such as previously learned parameters. In the rehearsal approach [99], the model was first trained on Task A; subsequently, the parameters were fine-tuned through batches obtained from a dataset containing a small number of examples of Task A and the training set of Task B. Training examples for Task A were selected through uniform sampling. The main disadvantage of the rehearsal approach is that it requires a large storage capacity to preserve raw samples or representations learned from the previous task. In contrast, the regularization approach consolidates past knowledge using additional loss terms that slow down the learning of the important weights used in the previously learned task. The most notable regularization-based approach is the elastic weight consolidation (EWC) [100].

In this study, the continual learning circumstance was domain-incremental learning, where the task structure remained consistent but the input distribution changed across sequential tasks. Task identity was unknown at the time of testing, and the model was only needed to solve the current task. This corresponds to a real-world adaptive aiding system that learns to operate in various scenarios without specifying the scenario.

4. Results

4.1. Behavioral Descriptive Statistics

Figure 4 shows the mean accuracy and RTs for the two conditions for each hazard type. Differences across different types of hazards were checked using an analysis of variance (ANOVA) test. The results revealed that the accuracies of electricity and edge protection-related hazards were significantly higher than those of structural hazards (66.05% ± 9.87% for electricity, 64.36% ± 8.92% for edge protection, and 58.53% ± 12.18% for structures; F = 10.04, p < 0.001). There was no significant difference in RTs across the three hazard types (F = 0.12, p = 0.88).

4.2. Analyses with LBA Modeling

To probe the cognitive differences in recognizing different types of hazards, 1714 correct trials for hazardous stimuli across all participants (hazard “Electric leakage” (EL): 640; hazard “Lack of Edge protection” (LEP): 781; hazard “Structural instability” (SI): 293) were submitted to LBA modeling. A satisfactory convergence was found for all the estimated parameters according to the Gelman–Rubin statistic: all Rˆ = 1.00. Figure 5 and Figure 6 show the posterior distributions of the four parameters.

The normality of the posterior distributions of the four parameters for each hazard type was tested using the Kolmogorov–Smirnov test (p > 0.05). Furthermore, the differences in parameters across the three types of hazards were checked using ANOVA, and significant differences were found in all four parameters (p < 0.001). Post hoc tests were conducted using Tukey’s HSD, and significant differences were found in any pair of hazard types for each of the four parameters (p < 0.05).

4.3. Analysis with LPA

The LPA analysis separated the participants into three performance-level subgroups. These were named “high”, “medium”, and “low”, with the first subgroup having the highest probability of accurately recognizing all hazards and the last having the lowest probability of accurately performing the tasks (see Figure 7).

4.4. EEG Classification Performance

For practical applications, the BCI system should achieve a classification accuracy of at least chance level (=0.33). When trained independently on each scenario, the mean accuracy over three runs for EL, LEP, and SI is 0.692, 0.699, and 0.655, respectively. All task-specific models were performed above the chance level.

Table 2. Mean accuracy over three runs of each sequential learning setup.

Strategy	Sequential Setup	Accuracy
Naïve	EL→LEP	0.687
	EL→SI	0.74
	LEP→EL	0.682
	LEP→EL	0.714
	SI→EL	0.677
	SI→LEP	0.684
Rehearsal	EL→LEP	0.697
	EL→SI	0.662
	LEP→EL	0.684
	LEP→EL	0.648
	SI→EL	0.649
	SI→LEP	0.701
EWC	EL→LEP	0.697
	EL→SI	0.66
	LEP→EL	0.684
	LEP→EL	0.66
	SI→EL	0.684
	SI→LEP	0.678

Note: EL denotes the hazard “Electric leakage”, LEP denotes the hazard “Lack of edge protection”, and SI denotes the hazard “Structural instability”.

summarizes the mean classification accuracies of all the training strategies over the three runs. Although neural networks can provide good performance for individual tasks, learning multiple tasks sequentially remains a considerable challenge for deep learning. We observed a slight forgetting phenomenon between EL and LEP; the naïve baseline, tested on LEP after fine-tuning on EL, achieved lower accuracy in the first task compared to being trained independently in both directions (0.687 vs. 0.699; 0.682 vs. 0.692). In contrast, SI introduced an asymmetric synergetic effect; EL or LEP exposure helped the model improve SI, achieving results that exceeded those obtained with the task-specific model (from 0.655 to 0.74 and 0.714, respectively); however, the effect was not symmetric as the accuracies of EL and LEP did not increase when SI was learned first.

As the EEG-based BCI interacts with real-world environments that change the data distribution and catastrophic forgetting does exist, as presented above, the BCI needs to be adaptive to changing hazardous scenarios. Regarding whether continual learning helped improve the model’s plasticity, the results in Table 2 reveal that the order of tasks plays an important role. Although models that used continual learning strategies performed slightly better than the naïve approach between EL and LEP in both directions, neither EWC nor rehearsal yielded any improvement over the naïve baseline when tested on SI after being fine-tuned on the other two hazards. However, after exposure to SI, the model that used continual learning forgot less than the naïve baseline in two setups: EWC and rehearsal performed better than the naïve baseline on EL and LEP, respectively.

5. Discussion

5.1. Environmentally Determined Adaptive Aiding

Suitable alert thresholds for automated diagnostic systems are expected to vary depending on the external hazardous scenarios. According to [101], selecting an appropriate threshold involves a trade-off between the miss and false alarm rates. Although both misses and excessive false alarms degrade trust and adversely affect performance [102], further research suggests that the degree of difficulty of the task, rather than the type of error, influences the trust level. More specifically, trust has been found to degrade, particularly when automation misses or provides a false alarm when detecting a target that the operator perceives to be easily identifiable. However, trust in automation increases when the target is perceived as being difficult to identify [103]. For this purpose, we investigated the perceived difficulty of workers recognizing different types of construction hazards.

Although no significant differences were observed in RTs across different hazard types, further component decomposition of RT based on LBA revealed that participants adopted different sub-processes to correctly recognize various hazards. In particular, recognizing LEP was accompanied by the highest decision threshold, which was reflected in the highest k and A values. Falls from height (FFH) account for most accidents and fatalities in construction [104] and remain a pervasive problem worldwide [105]. This study further revealed a possible cause of FFH, as participants demonstrated the highest tolerance for LEP. The high decision threshold for LEP may also explain why previous studies reported that most cognitive resources, as indicated by stronger brain activation, were required to recognize falling hazards [106]; that is, participants are more prone to underestimate the risk of FFH, so stronger evidence needs to be collected to support their endorsement of LEP as hazardous. To protect workers from FFH, installing barricades or edge protection is one of the most common practices. According to the Workplace Safety and Health Council, barricades are required for all building edges and edges of excavations, holes, floor openings, and roofs at construction sites [107]. Despite these policies, missing barricades remain a serious problem in construction. It has been found that the lack of guardrails, handrails, barriers, and edge protection accounts for approximately one-third of fall-related accidents [108]. Therefore, from the perspective of developing effective hazard alerting techniques, researchers in the CV domain are recommended to advance the algorithm for the detection of missing barricades, as in [109,110], to mitigate falling risks that are most likely to be underestimated by the construction workers themselves.

The results also showed that participants were most cautious in recognizing SI, as indicated by significantly lower k and A parameters. The most conservative attitude toward SI could most likely be attributed to workers’ unfamiliarity with structure-related hazards and their concerns about the serious consequences of misses [111,112]. Another notable piece of evidence comes from the finding that participants demonstrated significantly lower accuracy in recognizing SI compared to other types of hazards, which collectively indicates that the most lacking knowledge/mental representation/schema regarding hazard types for construction workers are structure-related hazards. Thus, it is highly desirable to develop structure-specific automated diagnostic systems, such as advancements in building information model-driven safety planning support for scaffolds [113] and temporary structures [114]. In addition, for integrated hazard alerting systems, when detected to be in structure-related scenarios, the alert threshold is supported to adaptively lower down to better support human operators, as they perceive the most difficulty in this scenario.

5.2. Continuously Monitoring Performance from EEG Signals

EEG-based WSDs are among the most important inventions for improving workplace safety and health. In fact, it has the potential to completely change the way construction workers interact with machine vision and robotic systems. This is because a novel and effortless communication pathway is established, by which workers’ brain-activation patterns are continuously analyzed to detect abnormal performance status, so that machine systems can react accordingly to best support human partners in HMC task settings. In this study, we specifically investigated a hazard recognition task to improve safety surveillance at construction sites. We proposed a wearable EEG sensor system that can predict construction workers’ performance level during hazard recognition; thus, when workers are detected at a low-performance level, the alert threshold of automated diagnostic systems could adaptively lower to avoid possible misses; on the other hand, when workers demonstrate good performance for hazard detection, the alert threshold for an assisted aiding system could raise to avoid possible false alarms that would introduce distractions. Therefore, the proposed EEG-based adaptive automated hazard alarm system is expected to result in more sustainable collaboration between workers and the automated hazard diagnostic system, as it supports proactive hazard prevention and human trust in automation. The proposed module can be integrated into the backend system of smart helmets designed for construction workers [53] and into existing hazard confirmation platforms reinforced by a multiagent checking mechanism [78].

As identified in the results, when the hazardous scenarios changed, the performance prediction model exhibited catastrophic forgetting. As researchers have published evidence that EEG-based BCI can decode hazard-type information from brain activation patterns [30,67], our findings lend significant credence to the fact that the shift in data distribution introduced by different hazard scenarios should be considered when designing EEG-based WSDs for real practice. Compared with the traditional naïve approach, which requires an extension of the training dataset, continual learning enables the model to implicitly compress the knowledge learned from previous tasks into the model structure. Thus, it can shorten the training time and save training resources when training on subsequent tasks. Our results also showed that when scenario change occurred between the electricity- and falling-related hazards, continual learning performed better than the naïve method to mitigate forgetting. However, when structure-related hazards were involved, the situation became complicated. Being exposed to EL or LEP earlier could improve the classification accuracy of SI, whereas exposure to SI first would result in more forgetting when tested on EL and LEP. This indicated that the brain activation patterns induced by SI were quite different from those induced by EL and LEP and that the model’s capacity to distinguish individuals’ performance levels in recognizing electricity- and falling-related hazards could be transferred to that of recognizing structure-related hazards, but the transferability was asymmetric. We may further imply that the human knowledge for recognizing EL and LEP reflected in brain activation patterns has an implicit overlap with the knowledge for recognizing SI, whereas recognizing SI requires certain specific domain knowledge or cognitive subprocesses that are not shared with recognizing EL and LEP. This assertion regarding human knowledge transfer between recognizing different hazard types needs to be consolidated by further cognitive evidence, but an interesting direction emerges in that the implicit knowledge transfer between recognizing different hazardous scenarios in humans and machine vision could be investigated, compared, and referenced in future studies.

6. Conclusions and Future Work

As an increasing number of CV-based hazard alarm systems have been developed, it is necessary to consider whether these alarm systems would introduce excessive warnings and adversely affect human hazard recognition performance owing to being distracted and potentially causing distrust in automation. In this study, we proposed a wearable EEG sensor system that could predict hazard recognition performance from construction workers’ brain activities, which would drive an adaptive adjustment of the alert threshold of the automated hazard diagnostic system to cultivate a more sustainable collaboration with its assisted individuals. Our approach is innovative in terms of its hierarchical structure, which considers both the cognitive differences in environmental scenarios and the adaptation to the shift in the distribution of induced EEG signals introduced by the scenario change. This design outweighs previous static models, which are incapable of adapting to changing scenarios in terms of robustness and plasticity. We tested our proposed approach using a case study, wherein 76 construction workers were involved in an experiment consisting of multiple hazard recognition tasks. We used LBA for computational modeling of participants’ RT in correct trials to reveal subtle cognitive differences in recognizing different hazardous scenarios. The results showed that the cognitive differences were particularly reflected in the decision threshold, as participants held the least conservative attitude toward falling hazards, followed by electricity hazards, and were most cautious about structural hazards. This has important implications for the ability of automated hazard diagnostic systems to implement differentiated alarming strategies, considering the perceived difficulty of their human partners in different hazard types.

In addition, a preliminary analysis was conducted to investigate the feasibility of an EEG-based BCI system for acquiring real-time hazard recognition performance. We used the transfer learning paradigm to classify the EEG signals elicited by participants from clusters of high-, medium-, and low-hazard recognition performance levels based on task performance in all scenarios. This analysis revealed that an EEG-based BCI is a highly promising solution for predicting task performance with approximately 70% accuracy when trained independently on individual on-site scenarios. A distinctive contribution of this study is that we investigated whether catastrophic forgetting exists in a scenario change for an adaptive WSD-based aiding system targeted for real practice and how to mitigate the possible forgetting phenomenon. We proposed a novel strategy that takes advantage of recent advancements in deep learning as we converted raw time-series EEG signals into a two-dimensional time-frequency domain based on a wavelet transform and, thus, exploited a well-established CNN model for signal classification and explored two mainstream strategies for continual learning to make the prediction model capable of continuously learning and adapting over time. We believe that our methodology has many applications, particularly in the design of smart helmets with implanted electrodes for neurophysiological sensing. The proposed hierarchical structure also paves the way for automated hazard detection techniques to evolve from domain-specific to integrated, thus enabling them to be more likely to be embraced in daily use.

This study had three limitations that can be addressed in future research. First, this study investigated only three types of construction hazard. This is because the investigated electricity leakage, lack of edge protection, and structural inability are the most common precursors of on-site accidents, based on the prior literature. However, we encourage researchers to investigate additional hazard types in the future. Second, although continual learning helped in some task settings, it did not boost performance in all training sub-sequences. Thus, an interesting question for future research is why the scenario order has a substantial influence on continual learning results. To answer this question, we can probe deeper into the working principles of the human brain to prevent catastrophic forgetting and how it can serve as an inspiration source for CV algorithms to learn continuously among multiple tasks [115]. Finally, future algorithms should consider individual differences. In this study, the LBA model was fitted to trials across all participants to enable a large sample size. However, if a large number of observations were available, the parameters could be estimated for each individual participant.