A Human-Adaptive Model for User Performance and Fatigue Evaluation during Gaze-Tracking Tasks

Abstract

Eye gaze interfaces are an emerging technology that allows users to control graphical user interfaces (GUIs) simply by looking at them. However, using gaze-controlled GUIs can be a demanding task, resulting in high cognitive and physical load and fatigue. To address these challenges, we propose the concept and model of an adaptive human-assistive human–computer interface (HA-HCI) based on biofeedback. This model enables effective and sustainable use of computer GUIs controlled by physiological signals such as gaze data. The proposed model allows for analytical human performance monitoring and evaluation during human–computer interaction processes based on the damped harmonic oscillator (DHO) model. To test the validity of this model, the authors acquired gaze-tracking data from 12 healthy volunteers playing a gaze-controlled computer game and analyzed it using odd–even statistical analysis. The experimental findings show that the proposed model effectively describes and explains gaze-tracking performance dynamics, including subject variability in performance of GUI control tasks, long-term fatigue, and training effects, as well as short-term recovery of user performance during gaze-tracking-based control tasks. We also analyze the existing HCI and human performance models and develop an extension to the existing physiological models that allows for the development of adaptive user-performance-aware interfaces. The proposed HA-HCI model describes the interaction between a human and a physiological computing system (PCS) from the user performance perspective, incorporating a performance evaluation procedure that interacts with the standard UI components of the PCS and describes how the system should react to loss of productivity (performance). We further demonstrate the applicability of the HA-HCI model by designing an eye-controlled game. We also develop an analytical user performance model based on damped harmonic oscillation that is suitable for describing variability in performance of a PC game based on gaze tracking. The model’s validity is tested using odd–even analysis, which demonstrates strong positive correlation. Individual characteristics of users established by the damped oscillation model can be used for categorization of players under their playing skills and abilities. The experimental findings suggest that players can be categorized as learners, whose damping factor is negative, and fatiguers, whose damping factor is positive. We find a strong positive correlation between amplitude and damping factor, indicating that good starters usually have higher fatigue rates, but slow starters have less fatigue and may even improve their performance during play. The proposed HA-HCI model and analytical user performance models provide a framework for developing an adaptive human-oriented HCI that enables monitoring, analysis, and increased performance of users working with physiological-computing-based user interfaces. The proposed models have potential applications in improving the usability of future human-assistive gaze-controlled interface systems.

1. Introduction

The human–computer interface (HCI) based on physiological interaction, also known as physiological computing [1], is a very important research area in computer science. This type of interface goes beyond the typical human–computer interaction. Physiological interfaces incorporate human body characteristics into their functionality. Physiological features of the human body could be determined as any quantitative data of a physiological nature that are recorded from the human [2]. The concept of physiological interaction or physiological computing systems (PCSs) encompasses such well-known paradigms as brain–computer interface (BCI) [3], neural computer interface (NCI) [4], gaze-tracking interface [5], etc.

The initial focus of the PCS-based interfaces was on people with disabilities, since their condition often requires an alternative mode of communication [6,7,8]. Recently, we can observe an increasing number of applications that primarily focus on healthy users. BCI games and entertainment application, for instance, are expected to constitute a large market of potential users (both healthy and disabled) [9]. Although eye tracking has been known as a useful research utility, recent studies reveal that gaze tracking gives a more challenging experience to the PC game players [10]. One of the main reasons why PCS-based interfaces are more often used in entertainment applications is the growing number of consumer-grade electronic devices for physiological signal scanning. For a long time, systems were bulky, expensive, and lab-oriented. Recently more consumer-affordable devices based on physiological computing and eye tracking emerged in the market (e.g., Tobii eye trackers for gaze tracking, Emotiv EPOC+ for BCI applications, MYO gesture control armband for electromyography (EMG)-based control) [11]. The gaze-tracking systems have become more user-friendly and significantly cheaper [12]. However, in many cases, higher affordability has been achieved at the expense of accuracy [13]. The primary purpose of such PCS-based communication and control systems is to enable alternatives or enhance methods to control user interfaces. Physiological computing systems are suitable for work and home activities. These can sustain work productivity and entertain. For disabled people who cannot move their hands and (or) legs, it can improve their quality of life. Systems that solve this problem are called assistive systems [14]. PCS is widely used in areas where manual human control is employed, but additional control is also required (e.g., car drivers, plane pilots, etc.), as well as in other domains like marketing research and advertisement testing, prosthetics, rehabilitation, psychology, etc. One of the major usability problems of these systems is the decrease in performance due to mental and physical fatigue. The control of interfaces based on PCS is a rather demanding task since a user has to carry out often unnatural activity, which results in high cognitive and physical load. The performance of a user controlling this kind of interface varies due to training, emerging fatigue, or change in mental state. Mental and physical fatigue has negative impact on performance, while training affects performance positively.

We propose the concept and model of the adaptive human-oriented HCI to enable the monitoring, analysis, and increase in the performance of users working with physiological-computing-based user interfaces. We perform the analysis of the existing HCI models related to physiological computing; carry out the analysis of the existing human performance models; develop an extension to the existing physiological models to allow for the development of adaptive user-performance-aware interfaces; and adapt the performance models for EMG-based HCI and gaze-tracking-based HCI.

Our contribution is as follows. First, the extension of the biocybernetic loop concept, called the human-assistive HCI (HA-HCI) model, is proposed. The novelty of the proposed model lies in two aspects: (i) the aforementioned model is derived from the concept of the biocybernetic loop, but it is more specific in the sense that it provides a practical framework for user interface design; (ii) the proposed model incorporates the performance evaluation in the human–computer communication process.

The paper is organized as follows. In Section 2, we discuss the related studies. In Section 3, we present an overview of the PCS paradigms which enable adaptability and an analysis of physiological signals suitable for fatigue estimation, analyzing the works on fatigue detection in different scientific domains. In Section 4, we describe the proposed human-assistive HCI (HA-HCI) model and its evaluation using game performance metrics. Section 5 describes the design of the application based on the proposed model. Section 6 describes the experimental results for adaptation of the HA-HCI model. Finally, Section 7 provides discussion on notable observations and limitations, and Section 8 presents the conclusions.

2. Overview of Related Works

Gaze tracking is the process of locating the point of a person’s gaze [15]. Applications of gaze tracking are numerous and include cognitive studies [16], medical research (ophthalmology, neurology, and related areas) [17] for diagnosing diseases such as Attention Deficit Hyperactivity Disorder (ADHD), Schizophrenia, Parkinson’s and Alzheimer’s disease, psychological evaluation [18], website usability [19] and advertisement studies [20], consumer product research [21], driver fatigue and drowsiness detection [22], reading studies [23], evaluation of image quality [24], learning process assessment [25], and human–computer interfaces for impaired [26].

Analyzing gaze fixation is important because it can provide insights into fundamental cognitive processes, diagnose and monitor neurological and developmental disorders, improve marketing research, and enhance visual assessments. For example, Hooge et al. [27] investigated the impact of fixation and saccade selection rules on the distribution of fixation durations in eye-tracking data. The researchers analyzed eye-tracking data of different qualities and used seven classification algorithms to determine the role of selection rules in merging and selecting fixation candidates. They found that for good-to-moderate-precision data, the choice of classification algorithm was not critical, as long as the selection rules were followed, which included selecting saccades with amplitudes greater than 1.0∘ and fixations with a duration longer than 60 ms. The authors suggest that researchers should report whether they used selection and their parameter values due to the importance of selection in the analysis of eye-tracking data. Altemir et al. [28] aimed to assess the evolution of gaze stability throughout life during short and long fixational tasks. The study included 259 subjects aged between 5 months and 77 years, who underwent a complete ophthalmological assessment. The results showed that gaze stability improved with age from 5 months to 30 years, while fixations tended to be longer. The study reported normative data of gaze stability and duration of fixations for every age group, and the authors suggested that currently available technology can increase the accuracy of visual assessments. Masedu et al. [29] investigated differences in visual exploration patterns between toddlers with ASD and those with typical development using an eye-tracking paradigm. The study included 18 children with Autism Spectrum Disorder (ASD) and 18 with TD, and gaze-tracking data were collected by showing a human face together with other objects. The gaze fixation sequence was modelled with a Markov chain model, obtaining transition probabilities between areas of interest (AOIs). The results showed that the transition between AOIs could differentiate between the studied groups.

Gaze tracking has been adopted to enhance road safety and reduce accidents by alerting drivers to potentially dangerous situations. Shah et al. [30] propose an advanced driver assistance technique that utilizes a real-time gaze-tracking system to obtain and communicate the gaze information of the driver. The authors developed a benchmark image dataset consisting of head poses and horizontal and vertical direction gazes of the driver’s eyes and used the You Only Look Once (YOLO-V4) face detector and Inception-ResNet-v2 CNN model for accurate detection and estimation of head pose directions and eye directions. The results showed high accuracy, with the head pose detection system achieving an average accuracy of 91%, and the eye gaze estimations achieving an RMSE of 2.68 for vertical and 3.61 for horizontal eye gaze. Yuan et al. [31] propose a knowledge-based solution for driver gaze tracking that does not require calibration, which is often a tedious and time-consuming process. The method is based on the domain prior to typical driver gaze patterns and self-calibrates in real-time during naturalistic driving scenarios. The method can extract relevant driver status features gradually and update estimation parameters periodically, and the results show that the auto-calibrated gaze estimation method can achieve automatic gaze calibration for gaze tracking during in-the-wild on-road driving without requiring cooperation from the driver. Ledezma et al. [32] propose a model-based gaze-tracking system that was tested on a suitable-features driving simulation environment using a Kinect v2.0 sensor. The developed advanced driver assistance (ADAS) uses gaze information to determine if the driver is looking at the road with their full attention, and alerts the driver only in case of distraction. The results are promising, with hit ratios between 96.37% and 81.84%. Khan and Lee [33] review various eye- and gaze-tracking techniques and their applications in ADAS. The paper discusses the acquisition of driver’s eyes and gaze data and the algorithms used to process this data and explains how the data related to a driver’s eyes and gaze can be used in ADAS to reduce losses associated with road accidents occurring due to visual distraction of the driver. The authors also present a discussion on the required features of current and future eye and gaze trackers. Naqvi et al. [34] used a deep-learning-based gaze detection method that considered driver head and eye movements, along with a near-infrared (NIR) camera sensor. This approach does not require initial user calibration and has shown better accuracy than previous methods. Lee et al. [35] suggested using a CNN to detect the emotions of drivers by analyzing images of their faces captured by thermal and NIR light sensors. To test their approach, the researchers conducted an experiment using their own database and found that their method was better at identifying aggressive or relaxed driving than previous methods. Finally, Naqvi et al. [36] proposed a method to detect aggressive driving based on changes in driver gaze and facial emotions using NIR camera sensors and a driving game simulator. The method extracts face, eye, and lip images using Dlib and uses convolutional neural networks to detect changes in gaze and emotions. The proposed method achieves high classification accuracy and outperforms previous methods.

Fatigue is described as extreme tiredness resulting from mental or physical exertion or illness [37]. It is common for almost every human physical activity. While controlling a personal computer (PC) in conventional ways, a user experiences fatigue after a relatively long period of time. Fatigue while controlling a PC or any other digital device using a human–machine interface based on physiological computing usually emerges much faster. Fatigue effects in the EMG-based interfaces are usually concerned with tension of specific muscles which are responsible for muscle control. Users of the eye-tracking-based human–machine interfaces are usually affected by fatigue caused by eye muscle tension or even tiredness related with continuously looking at a PC screen and a low blinking rate. In the field of BCI, a user encounters mental fatigue because control of BCI applications requires significant mental concentration. User fatigue results in the decrease in performance and accuracy of system control, so that a user can perform high-quality control only for a relatively short period of time (measured in minutes or hours). To expand the time of high-quality control in the human–machine interface, intelligent user interfaces (UIs) are developed or, if possible, multimodal interfaces are applied. The training effect opposes the fatigue effect. Therefore, the period of high-quality control might be expanded by performing consistent training. Concepts of fatigue and training are common for physiology research. The analytical models of sport athletes’ performance, which encompass the muscular fatigue and training components, were proposed by Banister et al. [38] in the 1980s and elaborated later.

Several studies have investigated the use of eye-tracking technology to detect the effects of fatigue and mental fatigue on human performance in different fields, including radiology, transportation, and cognitively demanding tasks. Pershin et al. [39] use AI-based metrics to predict fatigue-related changes in radiologists’ image-reading patterns. Li et al. [40] propose a four-phase framework that analyzes spatial and temporal gaze patterns to assess vigilance levels in traffic controllers. Bafna-Rührer et al. [41] explore the feasibility of mental fatigue detection using smooth-pursuit movements in an eye-interactive task. Lin et al. [42] suggest that eye movement behavior can be employed to recognize visual fatigue more sensitively than other methods. Tseng et al. [43] show that smartphone gaze is significantly impaired with mental fatigue. The authors tracked the onset and progression of fatigue, suggesting that it could provide a digital biomarker of mental fatigue. Lohr et al. [44] propose a method for detecting eye fatigue using fixation data that can be implemented on mobile and wearable devices, making it possible to react to fatigue in various environments. Craye et al. [45] present a multi-modal approach for detecting driver fatigue and distraction based on sensor data collected in a driving simulator and show that their approach can achieve high accuracy. Sommer et al. [46] compare the accuracy of different fatigue-monitoring technologies (FMT) for detecting fatigue in driving simulations and find that FMT devices perform acceptably at low temporal resolution (>20 min) but have large errors when estimating fatigue at high temporal resolution. Finally, Suzuki et al. [47] propose a model for detecting mental fatigue in natural viewing situations and an automated feature selection method to make the model robust to the target’s age, showing that their model improves detection accuracy compared to previous studies. Overall, these papers suggest that eye-tracking technology is a promising tool for detecting the effects of fatigue and mental fatigue in various contexts. We summarize the previous approaches in

Table 1. Comparison of related works on fatigue recognition using gaze tracking.

Ref.	Strengths	Weaknesses
Băiașu and Dumitrescu [22]	Detects drowsiness	Only frontal face images are used. The images are not captured in real-life setting
Pershin et al. [39]	Suggested an information gain metric blending reading time, speed, and coverage	The study was highly specific and used chest X-ray images
Li et al. [40]	Measured comprehension time as a proxy for vigilance	Used simple reaction time test
Lin et al. [41]	Measured accuracy of gaze fixation	Used just one minute of gaze data
Bafna-Rührer et al. [42]	Analyzed as characteristics of smooth-pursuit eye movements	High level of false positives
Tseng et al. [43]	Used gaze fixation of circular stimulus and measured accuracy	Used just a few minutes of gaze data
Lohr et al. [44]	Measured fatigue as accuracy of fixation on target	Assumes the user is not fatigued initially
Craye et al. [45]	Used gaze tracking as one of inputs in multimodal system	Only analyzes eye opening/closing
Sommer et al. [46]	Uses electro-oculogram	Temporal resolution is low
Suzuki et al. [47]	Captures cognitive fatigue	The study used only 10 min of gaze-tracking data

. Nowadays, this research has not lost its relevance. Moreover, it can be applied in new research areas, such as PCS, multimodal interfaces, BCI, and NCI.

3. Foundation of Physiological Computing Systems

3.1. Biofeedback

In the first treatise on cybernetic theory and communication as well as control in biological and mechanical systems by Wiener in 1948, the theory of feedback is of core importance. The concept of feedback here relies on the recognition that the controller of the system can control an appointed variable if it can access information about that variable. Starting in the 1960s, the perception of humans as able to exercise conscious influence over apparently unconscious physiology was seriously dealt with and it was discovered that feeding back physiological information to a subject ensured successful physiological control [48]; this process is called feedback. This helps people to increase their awareness of, trust in, and control over their physiological processes as well as to reduce stress [49]. Recent computer-based systems for presentation of physiological signals are to carry out two separate applications, namely clinical biofeedback and hands-free human–computer interaction (HCI). The same signal pre-processing and presentation requirements apply for both applications. A general scheme of biofeedback is presented in Figure 1.

The general workflow of biofeedback starts with recording physiological signals. Afterwards, these signals are quantified and processed to produce suitable representation of the signals for specific application. Signal analysis as well as its representation are strongly related with application (e.g., in medical applications, sampled signals are usually represented in a complex form as time series, whereas in PC games, representation of the signals is simplified and can be transformed to colors, emoticons, sounds, etc.). A user receives the feedback as determined by the application.

The concept of the biocybernetic loop originates from a closed-loop control and communication model. The feedback loop consists of three stages: collection, analysis, and translation. The specific processes involved in each stage depend on the type of system. In the first stage, the system collects data using sensors worn by the user. The second stage involves quantifying the data and identifying any artifacts. This is done in real time using an analysis algorithm, which determines if any data are irrelevant or incorrect. For the analysis stage, much attention is paid to certain cognitive aspects. For EMG and BCI systems (e.g., where the cortex helps to capture motor functions), the biocybernetic loop aims at changing physiological signal patterns into a certain computer command [50]. The biocybernetic loop encompasses biofeedback. The recording of physiological signals and providing feedback to the user are common stages of the biocybernetic loop. However, a biocybernetic loop is a more complex paradigm. It has additional stages: classification and adaptation. The classification stage aims to classify physiological signals to the interface control commands. In the adaptation stage, functionality or appearance of the system can be modified based on classification results or direct real-time measurement of psychophysiology [48].

Other physiological systems count on the accurate recognition of subconscious psychological states to detect changes in emotional states [51]. The adaptive controller is responsible for converting real-time physiological signals into computer commands, and therefore, for pattern-matching algorithms, adaptive control is direct. For biocybernetic adaptation, the role of the controller changes. These systems are developed for promoting positive states and forbidding undesirable ones. The impact between the user and system changes since biocybernetic control is for shaping and manipulating the state of the subject. If the user faces great mental workload, the system interferes to reduce workload and keep the situation stable. If the system user experiences failure, the system might either offer help or adapt itself to reduce the challenge. Certain change in a human–computer duo seeing the computer as a partner or team player as opposed to a servant-like system is the net result of the closed-loop design. At different levels of HCI, the functions of the biocybernetic loop are different. Biocybernetic adaptation alters parameters of the interactions (e.g., game levels) [52,53] or intervenes into the system actively (e.g., offers help) [54]. Each component of the biocybernetic loop is explained below.

1. Inference. The stage’s primary objective is to link the desired psychological state to a physiological metric. A psychophysiological concept is established that best represents the desired psychological state (e.g., a condition of high cognitive burden), and physiological measurements that define the most valid operationalization of that psychological state are chosen. Choosing sensor technologies and signal-processing techniques that are suited for field use and deliver high signal fidelity is critical at this point. The selection of inference model characteristics is the most important factor in the loop’s success. If the physiological indicators are not sensitive and reliable enough to capture the psychological concept, the inference model cannot establish a clear relationship between user state and system function [55].

2. Classification. At this level, the identification of the psychophysiological state in real time or near real time is of interest. If the loop is to work dynamically, it is critical that the information given from this step be up to date. As a result, the selection of the classification algorithm becomes critical at this stage. The classifier must be capable of processing and categorizing data in a fast and accurate way. The cost of misclassifying user answers must be carefully examined since the classifier eventually feeds forward judgements into the adaptation engine, shaping the efficacy of system adaptation in response to user behavior [55].

3. Adaptation. At this point, the psychophysiological reaction has already been measured and categorized. The categorization findings are utilized to determine the type of modifications to be applied at the interface. As a result, adaptation is concerned with the application of the loop’s governing rule set, specifically, what actions should be conducted at the interface in response to categorization results about the user’s state [55].

4. Interaction. The adaptation process is a type of interaction between the user and the system. From psychophysiological inference through categorization and adaptation, the form of adaptation will impact user perceptions of system efficacy. To instill user confidence, it must be properly developed to give timely and meaningful action or feedback at the interface [39].

3.2. Fatigue in HCI

Fatigue as the result of professional sport activities and fatigue resulting from the application of HCI based on physiological signals are similar in their nature. In both fields, mental and physical fatigue occur (see

Table 2. Comparison of fatigue in sports and HCI.

Criterion	Fatigue in Sports	Fatigue in HCI
Origins of fatigue	Mental/Physical	Mental/Physical
Temporal scale	Months/week/days	Hours/minutes
Detection methods	Physiological signals/Subjective tests/ Objective tests (performance)/Analytical training—fatigue models	Physiological signals/ Subjective tests/ Performance-based approaches
Environmental conditions	High physical activity and considerable strain	Low physical activity and low or medium strain

).

From the perspective of time, under time training–fatigue models applied in sports, the effects of fatigue come out in the temporal space of months, weeks, and days [56]. Meanwhile, a decrease in fatigue-induced performance of HCI based on physiological signals can be expected in hours or even minutes. It should be emphasized that in sports, fatigue can also occur in a relatively short period of time (for example, in sports such as the sprint, which requires a lot of explosive power), but in this analysis the time scale is determined based on the practice of existing training–fatigue models. The environmental conditions affecting athletes and HCI based on physiological signals are significantly different. Professional athletes train methodically, consistently for a long time. During exercise, physical activity is very high. This training can eventually lead to mental fatigue. Meanwhile, users of HCI based on physiological signals operate under conditions of relatively low physical activity, though control of certain systems using physiological signals has its own specifics. Here, the type and strength of fatigue greatly depends on a specific input signal. Using EMG-based HCI involves muscle tiredness that is of the same nature as during exercise training, but this fatigue is localized in the body where the EMG signal is generated. BCI systems cause mental fatigue because this type of interface does not involve any physical activity. The use of a gaze-tracking interface results in visual fatigue, which occurs due to muscular fatigue around the eye and a slight flicker; thus, it is a distinct form of muscle fatigue [57]. In addition, mental fatigue occurs in both the use of EMG-based HCI and the gaze-tracking interface in the long run. In terms of fatigue detection methods, both domains share similar approaches. Fatigue detection using physiological signals, subjective tests, and objective tests by their nature are similar in sports and HCI, but in terms of implementation they can differ from each other. However, analytical training–fatigue models are typical only in a professional sports domain. Most physiological measurements used for fatigue detection in sports have an equivalent in HCI. The HRV measurements are used in both a broad and very similar context in both areas [57].

The application of the EMG signal for the detection of local muscle fatigue has long been known in sports workouts (as in rehabilitation and ergonomics). Hence, various devices for monitoring muscle fatigue have been developed [58,59]. This equipment operates precisely when muscle tiredness is determined by isometric muscle contractions, but during dynamic contractions, measurement accuracy is questionable due to the movement of the electrode. EMG-based HCI control is dominated by dynamic contractions, which cause muscle tiredness to occur relatively quickly. As a result, other methods of the EMG signal analysis are used to detect muscle fatigue in the HCI field. Upper-limb power-assist exoskeletons are constantly exposed to muscle fatigue caused by dynamic muscle contractions. To solve this problem, complex methods are used to measure and analyze several EMG features (e.g., root mean square, mean power frequency, and spectral features) at the same time [60]. The EDA, EEG, and EOG signals are mainly used for detection of mental fatigue. In the HCI field these signals are most used to detect driver fatigue [61,62].

3.3. Combination of Biocybernetic Loop and Performance Evaluation

Biofeedback is a part of the biocybernetic loop that describes how psychophysiological data from the user are captured, analyzed, and converted to a computer control in real life [61]. It helps to achieve the adaptive communication between a user and a system. However, the user in this context is described as an unstable system member, since it is affected by many internal and external factors [61]. The wide variety of these factors results in the description of adaptive communication only on a very high level of abstraction.

Although the biocybernetic loop provides some abstract description of the adaptive interaction between a user and a system, it still faces limitations in some domains in the integration of physiological sensors, signal processing, and communication between physiological systems and applications [62]. Moreover, the biocybernetic loop lacks practical system development frameworks, which would facilitate the integration of the biofeedback loop to a specific application. The specific realization of the biofeedback loop highly depends on a user and an application. The application may define the level of instability of the user, since user state measurement depends on the specific system design.

The analysis of performance models based on impulse and response revealed that human performance can be defined by fitness and fatigue factors. A combination of impulse and response models with the biocybernetic loop may result in approximation of user states since user performance would be defined only by two factors (training and fatigue). Therefore, this extension of the biocybernetic loop could lead to a more detailed specification of the adaptive user interface and its interaction with the human. However, impulse and response models lack validation in the HCI domain; therefore, it is of high interest to test those models in the HCI domain and possibly extend the concept of the biocybernetic loop by including the performance models. The inclusion of performance models to the biocybernetic loop is even more reasonable because fatigue factor, which is the key factor in human performance modelling, is of the same origin despite the domain in which it occurs.

4. Human-Assistive HCI Model

A human-assistive HCI (HA-HCI) model is applied for users who can control only one modality of input (e.g., in BCI, the input modality is the brain wave signal, and in some cases it is the only input channel). An intelligent layer of the system monitors the input channel. The user receives feedback on physical and (or) mental burden, which is assessed throughout the performance evaluation method. As a recovery activity, feedback is supplied to the user, assisting the user in regaining lost performance. Following this technique, the system may be further controlled.

4.1. Structure of the Model

The structure of the HA-HCI model is as follows (see Figure 2):

• The interaction layer sets communication between the user and the system. It has two components: input channel and feedback activity.

  1.1

  The input channel represents the input modality that is used for control of the system.

  1.2

  Feedback represents the response of the system when fatigue effects appear.

• The intelligent layer is a central component of the model responsible for coordination of other components and decision-making processes.
• The performance evaluation procedure is responsible for performance evaluation of the user using the system.
• The control layer represents application-specific actions to control the system.

The human-assistive HCI model consists of several layers:

• Interaction layer. This layer provides tools of communication and control of the system. It is divided into two blocks: the input channel and feedback activity. The input channel is responsible for capturing an input modality which is presented in the model as an input channel. Feedback activity is a specific response of the system when the intelligent layer triggers a decreased level of performance. The purpose of this activity is to help the user relax and recover from mental and (or) physical fatigue. The type of feedback can be visual, auditory, tactile, or somatosensory.
• Intelligent layer. This layer is responsible for decision-making processes. Each time the user sends an input signal to the system, a decision must be made whether the signal should be converted to a control command, or a recovery activity should be provided to the user. The features of the signal, which represent fatigue, depend on the type of input modality. The extraction of these features is made in an intelligent layer. Afterwards, the extracted features are sent to a performance evaluation procedure, which returns feedback as an estimate of current performance level. The features of performance can also be received from the control layer as specific metrics of application (e.g., accuracy of user control, input speed, information transfer rate, etc.). Then, the decision is made whether the user should keep controlling the system or the fatigue is too high, and the recovery activity should be activated. Furthermore, the classification of a signal to determine the specific control command of application is also made in the intelligent layer.
• Performance evaluation procedure. This serves as a tool for quantitative assessment of user performance. The performance itself may depend on fatigue and training aspects of a specific user. The aforementioned procedure is application-specific and may vary from sophisticated fatigue feature extraction and classification techniques to a threshold function, which takes as an argument certain performance parameters. The output of this procedure is an estimate of performance level. The initial performance model can be pre-defined and, if necessary, modified online.
• Control layer. The control layer determines specific actions which are used to control the application. The application area is wide; technically it encompasses almost any digital device that can receive at least one input modality of any human-suitable form and can provide at least one output modality of any human-suitable form.

4.2. Control in the Proposed Model

From the point of view of the HA-HCI model, a user is also a part of the system. The user can send input commands and get feedback from the system (see Figure 3). System control tasks can be executed using input modalities, which are determined by the system design. No distinction is made between traditional input modalities (e.g., mouse, keyboard, joystick, etc.) and alternative ones like physiological-computing-based (e.g., EEG, EMG, EOG, gaze tracking, etc.) or NUI-based (e.g., human gestures) modalities. The user provides input to the system, which later is pre-processed in the input channel.

The concept of “single channel” in the HA-HCI model does not necessarily mean that the user is able to control the system via one input mode. The input channel can receive one unified set of input modalities. A unified set means that input modalities are undetachable from each other in terms of control. For example, a mouse and a keyboard are undetachable from each other in many cases, because one set of control commands are covered by the mouse, and another set by the keyboard. If one input mode cannot cover all control commands, one cannot consider it as an independent input mode.

The input channel is responsible for pre-processing a task. Not all input types may need pre-processing (e.g., a mouse and a keyboard); in such cases, this task is simply skipped. The most pre-processing is needed when system input is based on a physiological signal; then, signal sampling and filtering are usually applied. Further analysis of the pre-processed input data is then made in the intelligent layer.

The intelligent layer triggers feedback activity when the performance of system control decreases. In general, feedback activity is every activity which helps a user to recover performing abilities. In terms of sensing ability, feedback activity can be classified into (i) sensory feedback activity and (ii) hidden feedback activity.

• Sensory feedback activity can be sensed by the user. The feedback type can be visual, auditory, tactile, or somatosensory. The main purpose of any type of sensory feedback activity is to help a user regain performing abilities. Typical examples of such feedback are a GUI change due to an increased level of fatigue or inserts of relaxing music during the control process.
• Hidden feedback activity cannot be directly sensed by the user. In this case, the user can feel improvement of the interface performance or other metrics but cannot sense it. A typical example is the adjustment of control parameters (e.g., dwell time adjustments in gaze-tracking interfaces).
• In terms of how feedback activity is included into a control–feedback loop, it falls into (i) interruptible and (ii) uninterruptible feedback activity.
• Interruptible feedback activity interrupts the control process of the system. In this case, control of the system is disabled, and the user is instead stimulated by a relaxing activity.
• Uninterruptible feedback activity does not disable the control process. It is carried out simultaneously. The adjustment of control parameters is also a proper example to demonstrate this kind of feedback.

The intelligent layer is the most complex component of HA-HCI (see Figure 4). It is responsible for (i) pre-processed data classification to the control commands, (ii) user performance extraction, and (iii) decisions when feedback activity should be triggered.

• Pre-processed data classification to determine control commands. This procedure is common for PCS. The complexity of the classification approach depends on the application. Physiological signal classification may require sophisticated pattern recognition methods (e.g., artificial neural networks, SVM, etc.). In some cases, additional feature extraction must precede classification to reduce the dimension of the data (e.g., PCA). In simple solutions, input data can be transformed to control commands by applying a threshold function. Some interface types do not require classification at all (e.g., the gaze-tracking interface provides point of gaze). Therefore, data classification is optional in this model.
• User performance feature extraction is an important process in HA-HCI. The extracted performance features are used in the performance evaluation procedure as input arguments. Therefore, the intelligent layer and performance evaluation procedure are strongly related. Since performance is usually affected by user fatigue and training factors, the feature extraction tends to search for features in the input signal that are related with user fatigue. To extract features from input data, one may need to link a physiological measure to a specific fatigue state. Karran calls this process inference [55]. Another way to estimate the performance features is to use pre-set application-specific performance metrics of the control layer. Performance metrics like accuracy and input speed are common for many systems and those metrics are strongly related with fatigue because those metrics decrease in the presence of fatigue. A combined approach, extracting fatigue features from both input data and performance metrics, may increase accuracy, but it is a more complex approach.
• Decisions regarding when feedback activity should be triggered depend on the performance evaluation procedure. The performance evaluation procedure returns the performance estimate to the intelligent layer. The performance estimate can be a numeric value or pre-defined user state. To activate the trigger when the performance estimate is a numeric value, a threshold or sigmoid function can be used. When a pre-defined user state is an indicator, the intelligent layer should recognize this state and execute the necessary actions.

The performance evaluation procedure defines the means of performance measurement in the specific system. It can be a set of logic rules, mathematical equations or complex dynamic structures like Kalman filters and artificial neural networks (ANNs). A performance model can be passive or adaptive. A passive performance model is a pre-defined analytical model which does not change its behavior during the control process. An adaptive performance model changes over time and can be optimized during control process (e.g., using Kalman filter or other optimization algorithms [63,64,65,66]).

The control layer represents the logic of application. It receives control commands from the intelligent layer. These control commands are used to control the main application. The control layer also returns performance metrics, which can be used for evaluation.

4.3. Human Performance Modelling Using Impulse–Response Models

According to this model, any training session will have both a fitness-building effect and a fatigue-inducing effect. The total performance is defined as the sum of fitness and fatigue. In this model, fitness has a positive impact on performance, while fatigue has a negative impact. This statement is defined in a simple mathematical expression as follows:

$$P=F_{fittness}+F_{fatigue},$$

(1)

where P is performance, $F_{fittness}$ is fitness, and $F_{fatigue}$ is fatigue.

This model is based on the empirical observation that at the start of the training fatigue has a high amplitude, which decreases fast. At the same time, fitness has a lower amplitude, which decays slower than fatigue. The performance peak is the point where the difference of fitness and fatigue is the smallest.

The damped harmonic oscillation (DHO) model is used to describe daily physical performance capacity in team sports [67]. The rationale for using this model is based on chronobiology research, in which cosinor-based rythmometry is a common approach [68,69]. This model represents long-term day-to-day variation in physical performance capacity. The model is represented as a product of a damped simple sine wave and an exponential resistance component.

$$SingleDP{C}_n=-T{L}_{n}sin\left(\frac{2\pi t}{T}+\pi \right)e^{\frac{-t}{\theta }},$$

(2)

where DPC is the performance capacity on day $n$, $T{L}_n$ is the sum of all training loads of the day, $t$ is the elapsed time (in days) since the training day, $\theta$ is the damping parameter in arbitrary units, $T$ is the time period of one oscillation (in days).

Since professional sports training is a long-term matter, it is expedient to evaluate cumulative DPC, which is the sum of single DPCs till day n.

$$CumulativeDP{C}_n={{\displaystyle \sum }}_{i=1}^{n}DP{C}_{{\left(n-i\right)}_i},$$

(3)

where $n$ is the number of training days.

Kolossa et al. [69] proposed a linear fitness and fatigue model with Kalman filtering, which allows us to improve prediction by combining the last model state and indirect measurements. The model is a transformation of the well-known 3-time-constant fitness–fatigue [70] model to a linear, time-variant state-space model.

$$x_{k+1}=A_k{x}_k+B_k{u}_k+v_k$$

(4)

where $x$ is a state vector:

$$x=\left(\begin{array}{c}{x}_1\\ x_2\end{array}\right)$$

(5)

where $x_1$ is the fitness rate and $x_2$ the fatigue rate; $A_k$ is the system matrix with exponential decay rates in the diagonal, where the decay rates from the fitness and fatigue model proposed by Busso et al. [70] are as follows:

$$A_k=\left(\begin{array}{cc}{e}^{\frac{-1}{{\tau }_1}}& 0\\ 0& e^{\frac{-1}{{\tau }_2}}\end{array}\right)$$

(6)

$B_k$ is a time-varying input matrix:

$$B_k=\left(\begin{array}{c}{e}^{\frac{-1}{{\tau }_1}}\\ c_2\left(k\right)·e^{\frac{-1}{{\tau }_2}}\end{array}\right)$$

(7)

It includes $c_2\left(k\right)$, the training effect factor on the fatigue component, in addition to two exponential decay rates. This is how it is defined:

$$c_2\left(k\right)=c_3{{\displaystyle \sum }}_{j=1}^{k}u\left(j\right)e^{\frac{-\left(k-j\right)}{{\tau }_3}}$$

(8)

The system states cannot be accessed directly under this approach. They can only be established using indirect $y_k$ readings.

$$y_k=C_k{x}_k+n_k$$

(9)

where $n_k$ denotes observation noise (often Gaussian), and $C_k$ is the amount of each state component on the measurement. The following is how $C_k$ is defined:

$$C_k=\left(c_1-1\right)$$

(10)

where $c_1$—$c_3$ are the weighting factors and τ1—τ3 are time constants.

The capacity to adjust the predicted model state live is the major advantage of a linear, time-variant state-space model with Kalman filtering. As a result, as compared to standard fitness–fatigue models, the tolerance for measurement errors is larger.

4.4. Gaze Performance Metrics

Dwell time is the amount of time a gaze fixation stays on an object, and it can distinguish between accidental, visual search, and intentional gazes during tasks. The duration of a fixation corresponds to the brain’s processing activity, and it is predicted that dwell time will increase as a result of fatigue.

Previous studies [71,72,73,74] have used point of gaze (POG) accuracy. POG accuracy measures the distance between the center of the target and the point where the eyes are aligned when fixating on the target of visual attention. The point where the eyes are aligned is known as the point of gaze. Fatigue is expected to decrease POG accuracy.

The Fatigue Threshold (TF) [41,75] is a value that is calculated using an empirical formula. The formula depends on the average spatial accuracy of the eye tracker, denoted as θavg. The threshold can scale with noisier signals, A, the difference in fixation qualitative score (FQlS) [76] between the first fatigued group of data and the initial FQlS, and μ, which is the mean spatial accuracy of the data.

$$TF=\frac{A\times {\theta }_{avg}}{\mu }$$

(11)

Average spatial accuracy [41,77] is calculated during calibration by finding the mean gaze point, G_i, for each calibration point, P_i, and then calculating the average distance in degrees, θ_i, between each calibration point and gaze point.

To calculate the average spatial accuracy, the mean gaze point is determined for each calibration point during calibration. The average distance in degrees, θ_i, between each calibration point P_i and gaze point G_i is then calculated.

$${\theta }_{acc}=\frac{1}{n}{{\displaystyle \sum }}_{i=1}^n\left|P_i-G_i\right|$$

(12)

5. Design and Evaluation of PCS Application Based on HA-HCI Model

5.1. Architecture

The gaze-based control architecture used in HCI is based on the HASCM model (Figure 5) and relies on eye movements as an input channel. To keep users engaged and motivated, gamification techniques [78] are used. An adaptive dwell time is used as an initial recovery activity, which is adjusted based on the rate of typing errors detected by the system’s intelligent layer. This study also includes a more complex training and fatigue model that uses the accuracy of the sight landing position as input. The intelligent layer of the system is responsible for mapping gaze on a PC screen, detecting typing errors, and providing feedback to the user. To evaluate fatigue, an error rate threshold function is used, which determines the number of errors that can be made before the recovery activity is initiated. The specific threshold value is set by the user. The system workflow is as follows: the user enters text using eye movements, and the system monitors the number of unwanted selections or errors. When the error threshold is reached, the dwell time is increased. Conversely, when the user reaches a defined number of intentional selections, the dwell time is decreased.

The present version of the developed gaze-controlled Pac-Man game [79] also extends the original game by the means of control mode. Eye movement control mode is introduced. The user can control the game by either eye movements or a keyboard. Since the primary task of the system is to play the game via eye movements, the keyboard control is introduced only after certain fatigue indicators emerge.

The HASCM framework for an eye-controlled game is presented in Figure 5. This application of the HASCM consists of the components listed below.

• Multimodal interaction layer. It describes the means of communication and feedback. The user can use one of the following input channels: (1) eye movements and (2) keyboard control. The eye movement control is established via Tobii Eye Tracker 4C. Both input channels are switched alternately based on the supervision of the intelligent layer. The component of the input channel selector is responsible for switching the input channels and informing the user of which input channel is active at the moment.
• Intelligent layer. It is responsible for analyzing the input channel parameters and making decisions related with switching between input channels. The control using eye movements is a more demanding activity, which leads to fatigue more prominently. However, it is the primary control mode of the presented game; thus, the prolonged usage of it is of interest. The relation between the eye movement parameters and fatigue is not clear enough; therefore, it is the research focus of this study. The keyboard control is enabled when the eye movement parameters indicate fatigue. It is basically a layover of the eye movement. Keyboard control is terminated after a defined period.
• DHO-based performance model. This model is chosen since it has demonstrated promising results in modelling training effects on physical performance capacity [64]. It is investigated further in the following sections.
• Eye-controlled game. The idea of the game is based on a well-known Pac-Man game, which is a type of maze chase game. We implemented a version of the game in which a player must move in the maze horizontally or vertically and collect pills. The desired eye movements are made by navigating in the maze (Figure 6). The alternating vertical and horizontal movements of the eyes are the important part of therapy that were demonstrated to improve eyesight [80] and treat amblyopia [81] and eye movement disorder.

5.2. Subjects and Setup

The gaze-tracking experiment was carried out with 20 volunteers (age range 19–42, mean age 28 ± 5.3 SD). The study used a convenience sampling technique, which is a non-probability sampling method. Convenience sampling involves selecting participants who are readily available and willing to participate in the study. The recruitment process involved posting advertisements in campus buildings and emailing potential participants directly. The respondents were university students and academic lecturers. All participants were healthy with no self-reported history of eye-related diseases. All subjects signed an informed consent form, and the Helsinki Declaration was followed. The respondents were instructed to play the game for 15 min in a timed mode. The Tobii Eye Tracker 4C gadget was utilized to collect eye movements and control the main character of the game based on the player’s gaze. See Figure 7 of an individual playing the game.

6. Results

For modelling eye performance in the spelling system based on gaze tracking, Banister et al.’s model was used. Accuracy based on a gaze landing position was taken for fatigue evaluation. The results of gaze velocity measurements before playing the game (pre-task) showed a significant variability between the subjects ranging from 195 ms to 372 ms (Figure 8a). The gaze velocity measurements were repeated after playing the game and demonstrated a noticeable drop in average gaze velocity (Figure 8b). However, this drop was not observed across all subjects. In fact, some subjects showed a slight improvement in gaze velocity, thus confirming the positive effect of learning.

The results of this experiment reveal that Banister et al.’s model [38] fits well for evaluating user performance in the gaze spelling task. The Principal Component Analysis (PCA) of the data collected during the experiment suggests that fatigue effects in this case appear faster than training effects and have a major impact on performance. However, the experiment was executed in a relatively short period of time; thus, one can assume that training effects could have a higher impact in the long term.

The analysis of gaze error vs. angle of approach showed an increase both in number and magnitude of errors while executing gaze-controlled tasks (Figure 9) when observed after playing the game (Figure 9b) as compared to pre-game measurements (Figure 9a). We found that as time went on, saccade velocity decreased and the spatial distribution of gazes became less focused on the horizontal (i.e., W-E) and vertical (i.e., N-S) axes of the game, indicating a decrease in both the ability to follow the game and accuracy in control.

Figure 10 shows the difference in accuracy of gaze movements used as part of game control. The game requires using vertical and horizontal movements during movements of the game character in the maze. Deviation from horizontal or vertical moves decreases the speed of control and leads to errors and eventually to a lost game. Note that spatial accuracy decreases after subjects become fatigued while playing the game.

The fatigue effects in the PC game based on eye tracking were measured for a longer period compared to previous research. For this purpose, the DHO model of training and fatigue was applied. The reason for choosing this model is the wavy nature of the data collected during the PC game experiment. The DHO model is suitable for describing the data which reflect both long-term fatigue and training effects and short-term recovery of performance. The DHO model validity is determined with sufficient even–odd reliability (r = 0.82) (Figure 11). However, the DHO model showed high variability in terms of deferent user control [78]. To validate the claim, we use the comparison operator:

$$C\left(\Delta t\right){\displaystyle \sum }_{\Delta t}\left[v\left(t+\Delta t\right)<\left(t\right)\right]$$

(13)

where $\left[.\right]$ is the Iverson bracket operator, $v$ is saccade velocity, $\Delta t$ is time difference.

Then, we calculated the linear regression of $C$ with respect to $\Delta t$ as $C=b_0+b_1\Delta t$ for all subjects. The decrease in saccade velocity was confirmed for all participants with $b_1=-0.53 \pm 0.09$ and a mean correlation of 0.98 [78].

The training–fatigue model [38] fits well the empirical data gathered during the gaze spelling task. However, it failed to fit the data obtained from the PC game based on eye tracking. Therefore, in this application, the DHO model was employed, since it showed a significantly better result. The experiments with the gaze spelling system and the PC game based on gaze tracking differed in terms of duration. This implies that Banister et al.’s [38] model is possibly suitable for describing the performance in the short term, as the DHO model deals well the performance in the long term. Some research suggests that users begin to sense the fatigue after 13 min of using a gazetracking system [47]. However, to prove this assumption on the performance models, more experiments are required. We also noticed large variability of user parameters, suggesting the need for personalization in physiological-computing-based HCI. Training and fatigue models like Banister’s or DHO cannot be generalized for the whole population. On the contrary, they can be applied for the specification of individual users and even can serve as a performance classification tool, where users are classified into learners and fatiguers.

7. Discussion

7.1. Discussion on Performance in Assistive Systems

The human-assistive HCI model provides a framework for the development of human–machine interfaces based on physiological computing. Using the human-assistive HCI model, user interfaces based on the performance characteristics of physiological interaction can be designed. The aim of this model is to enhance the time of continuous accurate control of the human–machine interface based on physiological computing.

Typically, one of the biggest application areas of user interfaces based on various physiological signals is the systems used by disabled people [82,83]. In general, such assistive technologies can be applied everywhere (both at work and home) in our everyday life to increase the quality of our life, sustain work productivity, as well as provide entertainment [84]. Using such a system, the user starts feeling fatigue relatively fast; therefore, performance of the system control decreases as well. The performance of a user depends on individual characteristics. The proposed model aims to adapt the interface to individual user needs and abilities and helps to prolong the time of effective usage of a certain interface [85]. Moreover, some specific performance evaluation methods, suitable for interfaces based on eye tracking and EMG recording, have been adapted from other domains such as sports informatics [86,87]. In PCS, physiological signals (e.g., EEG, EOG, EMG, etc.) are applied for interface control. This way of interface control allows disabled people to communicate with others and control their digital devices and their environment.

When using assistive interfaces based on PCS systems, performance can decrease radically. Many factors have an impact on this, but the most important is fatigue [88,89]. In solving the problem, fatigue is simply bypassed by developing a sophisticated user interface and multimodal solutions (e.g., hybrid BCI). However, it must be acknowledged that fatigue effects in similar systems are unavoidable. Often users can control only one input modality; therefore, after fatigue appears and accuracy of the system control decreases, user’s motivation decreases as well. Another major factor of the interface control performance is training (learning) [90]. The training factor has a positive impact on interface control performance. The training aspect increases constantly while the user uses the interface. Evaluation and prediction of the system control performance in real time would be a natural way to solve the motivation problem.

Athlete performance models used in sport training could be applied to predict fatigue and training effects in PCS-based interfaces. Human-assistive HCI models rely on a biocybernetic loop and HI principles, though their novelty lies in the performance assessment and prediction element included in the system design. Hence, it is a priori accepted that fatigue and training effects will occur using assistive interfaces based on PCS. Though the proposed models originate from the concept of a biocybernetic loop, the representation of the aforementioned models focuses on practical aspects of the development of PCS-based user interfaces. In general, human-assistive HCI models provide a design framework for the PCS-based user interfaces. The distinction between the concept of a biocybernetic loop and human-assistive HCI models also lies in the performance measurement methods. Performance or user states in a biocybernetic loop are measured based on biophysiological characteristics of the user. In a human-assistive HCI model, performance can be measured directly from biophysiological signals generated by the user, or indirect measures related to the specific application can be applied (e.g., accuracy, error rate, information transfer rate, etc.). Thus, the proposed models are suitable for but not limited to the development of the PCS-based user interfaces. Fatigue in the field of sport training and physiology is widespread for researchers [91,92]. Both subjective and objective research methods are applied. In addition, the monitoring of human physiological signals is often used to detect fatigue. Furthermore, impulse–response models in the physiological domain elegantly include fatigue factor in mathematical performance models. Although a mathematical impulse–response model abstracts the human performance to two factors (fitness and fatigue), it has still been validated in many studies regarding the performance estimation of athletes [91,92].

The generation of physiological signals for system control sometimes involves the same muscles (e.g., in EMG-based interfaces) as in an intensive physical activity (e.g., athlete training). This implies that fatigue is of the same nature in both domains. In fact, the analysis carried out in this paper shows that fatigue is estimated using similar methods both sports and HCI. Fitness factor is also relevant to more than just human physiology. A more general term for fitness is training. The importance of training can be seen in many fields where permanent exercise is required, one of which is the use of certain software. The assumptions suggest that impulse–response models are worth testing in PCS.

The aim of the integration of impulse and response models to PCS interface design is to ensure the adaptiveness of the interface. The common approach of the adaptivity in PCS is the biocybernetic loop. Therefore, the obvious way to introduce the training–fatigue models in PCS is by including these models into the biocybernetic loop. This extension of the biocybernetic loop could lead to a more detailed description of the adaptive system and its interaction with the user. The analytical model of athlete performance proposed by Banister et al. [38] was adapted to the PCS research area. Based on this analytical model and experimental results, the analytical performance model for a speller based on eye tracking has been derived. The derived model of eye tracking performance could be applied to develop human-assistive interface systems. The analytical DHO model, applied to evaluate the performance of sports athletes, was adapted to evaluate the performance of users in the context of a PC game based on eye tracking. This model is suitable for long-term analysis of performance dynamics. Findings show that this model effectively describes long-term fatigue and training effects and short-term recovery of user performing abilities.

7.2. Limitations

The main drawback of the model lies in variability of performance estimation metrics. The CMJ metric used in sports training is suitable to evaluate athletes’ performance or fatigue level in many sports [93]. However, to objectively evaluate performance in the HCI domain, no similar metrics are distinguished. Performance assessment methods in the HCI and PCS areas are greatly dependent on the scope and the type of input modalities used for control. The arguments of mathematical performance functions based on training and fatigue will also vary from one system to another. Using the same system does not change the analytical model for different users, but the parameters of this model will be different for different users. Hence, every user must necessarily undergo trial testing before using a certain system to find the right model or, at least, suitable parameters.

After using the system for a longer period, the user trains himself; thus, controlling the system becomes smoother. For the same reason, the performance evaluation procedure can also change. Then the need for re-optimization of the analytical model emerges. This problem might be solved by adjusting the model online as proposed in [69].

7.3. Recommendations

General recommendations for the HCI designers when developing human-assistive interfaces are formulated as follows:

• Analyze requirements for user performance introduced by the specific domain of application and the developed system.
• Analyze the communication modalities used by the system and any user-related effects on performance, such as those introduced by fatigue.
• Adopt the Banister or DHO model presented in this dissertation for the developed HCI of the system. The choice of the analytical performance models is not limited to the models presented in this dissertation.
• Implement a biocybernetic feedback loop to allow the adaptability of the HCI characteristics depending on human performance when working with the system in real time.
• Evaluate usability of the interface and test with users in a real-world environment.

7.4. Theoretical Implications

This study has several theoretical implications that contribute to our understanding of the human fatigue effect during gaze-controlled tasks; they are as follows:

• Theoretical foundations: The study has helped to establish a theoretical foundation for understanding human fatigue recognition. It has identified key factors that influence fatigue, such as sleep deprivation, circadian rhythm disruption, and workload, and has shown how these factors can affect cognitive and physical performance. This study has also demonstrated that fatigue can have both subjective and objective components, with subjective experiences of fatigue often not correlating with objective performance measures.
• Multidisciplinary perspective: The study has drawn on insights from multiple disciplines, including psychology, neuroscience, physiology, and engineering. This multidisciplinary approach has helped to build a more comprehensive understanding of fatigue and has led to the development of more effective methods for detecting and measuring fatigue.
• Technology development: The study has contributed to the development of new technologies for detecting and monitoring fatigue. For example, wearable sensors and mobile apps have been developed that can track physiological indicators of fatigue, such as heart rate variability and skin conductance. These technologies have the potential to improve safety in high-risk industries, such as transportation and healthcare, by providing real-time feedback to workers and alerting them when they are at risk of fatigue-related errors.

7.5. Managerial and Practical Implications

The study has practical and managerial implications, which are discussed below:

• Occupational safety: The findings of the study have significant implications for occupational safety. Human fatigue is a critical factor in many workplace accidents and incidents. By developing an accurate and reliable model for recognizing human fatigue, managers can take proactive measures to prevent accidents and ensure the safety of workers.
• Workforce management: The study provides a valuable tool for managers to monitor employee fatigue levels and make informed decisions about scheduling, workload, and resource allocation. This can improve productivity, reduce absenteeism, and enhance employee well-being and job satisfaction.
• Training and education: The study highlights the importance of educating employees and managers about the risks of fatigue and the importance of recognizing and managing it. By providing training and education on this topic, organizations can promote a culture of safety and well-being.
• Human resources management: The study underscores the need for human resource managers to consider fatigue when designing job roles, selecting candidates, and managing performance. By taking fatigue into account, organizations can ensure that employees are appropriately matched to their roles and have the necessary support and resources to manage fatigue effectively.
• Healthcare: The study has implications for healthcare providers who are responsible for diagnosing and treating fatigue-related conditions. By improving our understanding of the physiological and behavioral signs of fatigue, healthcare providers can develop more effective interventions to manage fatigue and its associated health risks.

8. Conclusions

Mental and physical fatigue is the primary factor in the decrease in performance abilities in physiological computing systems (PCSs). Despite the significance of the fatigue factor, previous research in the PCS domain was conducted only in a fragmented manner and lacked a complex approach to the fatigue problem. On the other hand, fatigue research in the sports training domain is of high interest and is far more advanced. Since the nature of fatigue in both sports training and PCS is similar, the approaches of fatigue estimation and prediction known in the sports domain could be adopted in PCS. The proposed human-assistive HCI model describes the interaction between a human and PCS from the user performance perspective. The main novelty is the performance evaluation procedure, which interacts with the standard UI components of the PCS and describes how the system should react to loss of productivity (performance). The applicability of the human-assistive HCI models has been demonstrated by the design of an eye-controlled game.

The analytical user performance model developed by Banister et al. is applicable for the evaluation of training and fatigue effects in using the gaze-tracking-based spelling system. To validate the model, the accuracy of gaze landing in performing a text entry task was analyzed for seven subjects. The analysis results were fitted to Banister et al.’s model. The most accurate model reached good fitness results (R2 = 0.9027, RMSE = 0.0098, SSE = 0.0005); however, user performance variability is high, and it is greater than intra-user variability owing to learning and fatigue effects. According to PCA analysis, two factors can explain intra-user variability: weariness (73% of variation) and learning (17% of variance). Because learning is slower than tiredness and has less of an influence on results, the time-to-peak value is less than the time-to-initial performance. As a result, it is recommended that time-to-initial performance be used as an estimate of rest time.

An analytical user performance model based on damped harmonic oscillation (DHO) is suitable to describe variability in performance of a PC game based on gaze tracking. The validity of the DHO model fitting has been tested using odd–even analysis, which has shown a strong positive correlation (0.82 ± 0.08). To categorize players based on their skills and abilities, individual characteristics established through the damped oscillation model can be used. The results of our experiments show that players can be classified as learners or fatiguers based on their damping factor. The amplitude and damping factor show a strong positive correlation, indicating that good starters tend to have faster fatigue rates, while slow starters have less fatigue and may improve their performance. A temporal and directional analysis of saccade velocity indicates that saccade velocity and gaze movement accuracy tend to decrease due to eye fatigue over the course of the game, since linear regression models demonstrate negative trends for each subject.