1. Introduction
Fatigue is an important factor affecting the safe operation of nuclear power plants [
1]. A large number of accidents at nuclear power plants around the world are caused by human error [
2] and fatigue is one of the main factors leading to human errors [
3]. Not only can fatigue reduce a person’s performance, it can also affect health. The main control room of a nuclear power plant has gradually changed from traditional simulated control to digital control, while the operator’s task in the control room of the nuclear power plant has gradually shifted to monitoring tasks [
4,
5]. The main task of the operator in the main control room of a nuclear power plant is to monitor and analysis graphs, data and relevant parameters on multiple monitors over long periods of time. Their continuous attention and monotonous vigilance task lead to high mental load. Operators are easily fatigued by high mental workloads [
6]. For the purposes of this article, fatigue refers to mental fatigue.
So far, methods of monitoring operator fatigue-related include task performance, subjective evaluation and physiological indicators [
7,
8,
9]. The physiological indicators are objective data reflecting the fatigue state of an operator. Physiological indicators to detect fatigue include invasive and non-invasive detection. Invasive detection of fatigue is based on physiological signals stimulated by hormone levels, such as electroencephalogram (EEG), electrooculogram (EOG) and electrocardiogram (ECG). Our work is to combine blink rate (BR), number of frames closed in a specified time (PERCLOS) [
10], average mouse velocity (AMV) and average value of mouse velocity (AOV) features in a non-invasive detection way to avoid interfering with the operator’s normal work. On the one hand, these four indicators can be acquired in a non-invasive way. This approach avoids interfering with the operator’s normal work; on the other hand, the combination of using the camera to capture changes in the eye, and to augment it with changes in the velocity of the mouse, and then combining these two types of feature sets, shows a significant improvement in fatigue recognition compared to using a single eye feature.
Blink rate and PERCLOS features are used to analyze and detect fatigue in automobile driving [
11], aircraft driving [
12] and human–computer interaction [
13]. Eye aspect ratio (EAR) has been successfully used in operator fatigue detection in automated control systems [
14,
15]. First, the EAR values are obtained to generate the features associated with the eyes. Then, these features are used for supervised learning [
11]. Using convolutional neural network, after a large number of eye opening and closing pictures are trained, a model for detecting eye opening and closing is obtained. Then, fatigue is determined according to predetermined rules and algorithms [
16]. At present, when using eye-related features for non-invasive detection, the methods to obtain the eyelid closure state are roughly the two mentioned above. One is to calculate the eye-width ratio using facial landmark and determine the eye opening and closing by a predetermined threshold. The other uses a trained neural network to judge the opening and closing of the eyes directly from images. Then, the features are extracted again based on the opening and closing state of the eyes, such as blink rate or PERCLOS. Finally, the fatigue detection is implemented using the recognition fatigue algorithm, or blink rate and PERCLOS threshold size are used to determine fatigue according to predetermined rules. The above two single feature set methods are affected by facial images, lighting, motion blur or head deflection during the task.
Mental fatigue is a complex and individualized phenomenon that affects psychological, physiological and behavioral aspects [
17]. Determining fatigue using a single indicator is unreliable. Few studies have been conducted to detect the fatigue status of operators in the main control room of a nuclear power plant based on the fusion of multiple indicators by machine learning [
18]. In [
19], mouse, facial features and eye gaze are used to recognize attention. EEG, ECG and EOG were selected for fatigue detection in [
20]. It is fully proved that the fatigue detection performance of more features is better than that of fewer features. The dynamically changing features of the mouse are also currently used for the detection of physiological states associated with people, such as stress [
21,
22], mood [
23,
24], attention [
19,
25] or fatigue [
26]. Changes in mouse velocity and distance are used to detect worker fatigue during human–computer interaction. This method has been shown to be able to quantify the fatigue state of workers for a long time without any interference in work [
26]. At present, the dynamic features of mouse are mainly used for the detection of physiological states such as emotion and attention, and there are few studies on fatigue. Our goal is to achieve real-time and high-precision non-invasive fatigue status detection.
The research in this paper is focused on machine learning. In [
27], it is summarized that the models used for the implementation of fatigue detection techniques include mathematical model-based, rule-based and machine-learning-based implementation methods. The current detection method used for main control operators in nuclear power plants is mainly through statistical analysis [
7,
8,
9]. In [
28] a machine learning clustering approach is used to evaluate operator fitness for duty based on operator fatigue status, but it cannot be used to detect fatigue in real time. Deep learning of complex models requires a lot of time for recognition. Traditional machine learning has low equipment requirements and is capable of achieving fast fatigue status recognition. We need to detect the level of operator fatigue in real time. We choose support vector machines (SVM), K-Nearest Neighbors (KNN) and random forests (RF) as machine learning algorithms to identify fatigue [
13,
29]. The use of machine learning for fatigue detection has become very widespread. The data used for training and testing in the existing studies are eye-related and mouse-related features.
Before supervised learning, it is necessary to segment the sample data and determine the fatigue level and labeled data. These sample data with fatigue level labels are used for training and evaluation algorithms [
30]. The methods of data labelling can be broadly classified as follows, using subjective assessment methods [
10,
31], time-on-task phases [
32] and trends in physiological changes [
33]. The data labeling method of a single indicator is always unconvincing. Subjective evaluation depends on the ideas of the evaluators themselves. The time-on-task phases can easily confuse samples between two neighboring time-on-task phases and the trend of physiological changes cannot truly reflect the differences between different users. In 2017, Toeplitz Inverse Covariance-Based Clustering (TICC) was proposed [
34]. The TICC clustering method cannot only find repeated patterns in the dataset, but also explain the clustering results. TICC was successfully used to cluster fatigue sensitive indicators in [
30]. These fatigue-sensitive indexes include eye index, task performance index and subjective rating index. In nuclear power plants and many other human–computer interaction tasks, they use features such as blink rate or PERCLOS to detect and analyze fatigue. Blink rate and PERCLOS were shown to reflect the level of operator fatigue [
9,
35,
36]. In [
37,
38], there is a negative correlation between mouse velocity change and operator fatigue. Previous studies have labeled fatigue levels into two categories: fatigue and non-fatigue. Fatigue is a process with complex physiological and psychological changes and accumulation. Fatigue occurs when cognitive load accumulates to a certain level [
9,
39,
40,
41]. Multi-level fatigue labeling helps improve our understanding of operator fatigue, providing a basis for developing techniques to prevent operators’ fatigue operation in an MCR.
In order to study the real time fatigue detection, an appropriate time window means a lot. We label data first and then start supervised learning. A suitable sliding window yields better results for our supervised learning to extract features. In [
17] different feature extraction windows were tried. In our study different performance was achieved by comparing different feature windows and the overlap rate of the windows.
The structure of our paper is composed of five main sections, and the remaining sections are organized as follows.
Section 2 presents the methodology and identification framework.
Section 2 describes the simulation experiments and data collection.
Section 3 presents the results.
Section 4 provides the discussion and conclusion. The framework of our study is shown in
Figure 1.
4. Discussion and Conclusions
It is found that TICC can objectively determine the fatigue level by using the change trend of subjective evaluation indicators (KSS and SSS scales), eye indicators (BR and PERCLOS) and mouse indicators (AMV and AOV within the sampling time). In the human–computer interaction environment of the main control room of a nuclear power plant, the results show that several widely used classification algorithms can perform fatigue detection with different feature sets. All have achieved good results. In particular, KNN shows excellent performance in various feature sets and the accuracy rate is above 85%. We find through experiments that, in a sliding window of 60 s and a sliding step of 10 s, the accuracy of the three algorithms is relatively high. Our model can basically identify the fatigue state of the operator in about 10 s, except for the initial 60 s data.
In comparing the performance of the eye and mouse features we find that previous studies have illustrated that eye-related features have very promising recognition performance in fatigue detection, but our research proves that the combination of mouse features and eye features can detect fatigue more accurately.
From the clustering results, it can also be obtained that the subjective scores of operators increase. Blink frequency and PERCLOS also increase. The movement velocity (AMV and AOV) of the mouse is reduced instead. It is confirmed that the characteristic changes of eyes and mouse are consistent with the change trend of fatigue in [
9,
30,
37].
Our research is making efforts to solve the problem of real-time detection of operator fatigue in the main control room of nuclear power plants and help develop an operator fatigue detection method for nuclear power plants. However, we still need to do some further work. First, our study does not consider the effect of different simulation tasks on the data, so there may be different physiological and behavioral changes for fatigue states in different experimental scenarios, such as accident scenarios and normal start-stop pairs of scenarios. Second, our experimenters are all graduate students. Although our subjects all spent more than eight hours learning, there is still some difference compared to real plant operators.
By comparing the results of this experiment, we can understand that multiple features are more effective than a single feature. Possible future work will be to extract more features manually or to use more non-invasive devices to obtain fatigue indicators. The results and methodology of this study lay the groundwork for further explorations into mental fatigue in NPP MCR environments and could be applied to develop a reliable in MCR fatigue detection system in restricted NPP.