Next Article in Journal
Dual-Branch Discrimination Network Using Multiple Sparse Priors for Image Deblurring
Previous Article in Journal
Beamforming Design for Cooperative Intelligent Reflecting Surface-Assisted mmWave Communication
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Effect of Visually Induced Motion Sickness from Head-Mounted Display on Cardiac Activity

1
Industry-Academy Cooperation Team, Hanyang University, Seoul 04763, Korea
2
Center for Bionics, Korea Institute of Science and Technology, 5 Hwarang-ro 14-gil, Seongbuk-gu, Seoul 04763, Korea
3
Department of Biomedical Engineering, Hanyang University, Seoul 04763, Korea
4
Department of HY-KIST Bio-Convergence, Hanyang University, Seoul 04763, Korea
*
Author to whom correspondence should be addressed.
Sensors 2022, 22(16), 6213; https://doi.org/10.3390/s22166213
Submission received: 3 July 2022 / Revised: 14 August 2022 / Accepted: 15 August 2022 / Published: 18 August 2022
(This article belongs to the Topic Virtual Reality, Digital Twins, the Metaverse)

Abstract

:
Head-mounted display (HMD) virtual reality devices can facilitate positive experiences such as co-presence and deep immersion; however, motion sickness (MS) due to these experiences hinders the development of the VR industry. This paper proposes a method for assessing MS caused by watching VR content on an HMD using cardiac features. Twenty-eight undergraduate volunteers participated in the experiment by watching VR content on a 2D screen and HMD for 12 min each, and their electrocardiogram signals were measured. Cardiac features were statistically analyzed using analysis of covariance (ANCOVA). The proposed model for classifying MS was implemented in various classifiers using significant cardiac features. The results of ANCOVA reveal a significant difference between 2D and VR viewing conditions, and the correlation coefficients between the subjective ratings and cardiac features have significant results in the range of −0.377 to −0.711 (for SDNN, pNN50, and ln HF) and 0.653 to 0.677 (for ln VLF and ln VLF/ln HF ratio). Among the MS classification models, the linear support vector machine achieves the highest average accuracy of 91.1% (10-fold cross validation) and has a significant permutation test outcome. The proposed method can contribute to quantifying MS and establishing viewer-friendly VR by determining its qualities.

1. Introduction

Virtual reality (VR) using head-mounted displays (HMDs) has become increasingly popular for professional and entertainment purposes and contributed to technological advancement and increased economic activity [1,2]. VR technology has been used in many areas, such as military training simulations [3], training or education in medical procedures [4], architecture [5], manufacturing [6], entertainment [7], and gaming [8]. VR technology can provide an experience that is impossible in the real world and deep immersion [9]. However, the devices trigger motion sickness (MS), including visual fatigue, nausea, anxiety, and disorientation, in some users [10]. Visually induced motion sickness (VIMS), motion sickness disorder (MSD), and VIMS disorder are defined as vestibular disorders; however, MS can be experienced by anyone [11]. Approximately 33% of the population is highly susceptible [12], and at least 59% of the population has reported experiencing MS [13]. Many developers have attempted to improve software and hardware; however, the issue of MS remains [14]. Consequently, MS is a major obstacle in the popularization and development of the VR industry [15]. Thus, studies on minimizing MS are necessary, which can contribute to improving the VR user experience and friendliness. To solve the problem of MS, reliable measurement methods for quantitatively assessing MS should be established [2,16].
Many previous studies have attempted to understand and interpret MS from the perspectives of postural instability [17]; vestibular function [18]; eye movement [19]; the autonomic nervous system, such as cardiac activity, electrodermal activity, and skin temperature [1,20]; the central nervous system, such as electroencephalogram (EEG) oscillations [21] and functional magnetic resonance imaging [22]; and pupillary rhythms [23]. However, when using machine learning, most studies do not distinguish MS patients from healthy controls. Some studies have attempted to classify the level of MS severity, as shown in Table 1, using the following criteria: EEG oscillations [24,25,26,27,28,29,30,31], heart–brain connectivity [2], vision technology [16], and multimodal data fusion [32,33,34,35,36,37]. Most studies attempting to classify MS are based on brain activity and multimodal data fusion. These approaches have significant disadvantages, such as the measurement burden of sensor attachment and low usability compared to cardiac activity. Despite these limitations, the method developed in a previous study has the advantage of acquiring significant and important data. Thus, fields that require precise measurement of phenomena related MS can be at an advantage. However, a simple method should be developed to assess the sensitivity of MS, because there are fields that require rapid and easy MS measurement or monitoring. Thus, we propose developing a method that enables the simple and convenient assessment of MS based on a single measurement of cardiac activity.
Physiologically, heart rate variability (HRV) indicates the interaction of autonomic, intrinsic, and humoral influences on heart rate [38]. The HRV spectrum for assessing autonomic balance is divided into very low frequency (VLF, 0.0033–0.04 Hz), low frequency (LF, 0.04–0.15 Hz), and high frequency (HF, 0.15–0.4 Hz), which correspond to sympathetic activity, a mixture of sympathetic and parasympathetic activities, and parasympathetic activity, respectively [39,40]. Previous studies have reported that the development of MS is strongly correlated with sympathetic and parasympathetic activities [41,42]. In addition, the time domain indices of cardiac activity, such as heart rate, the standard deviation of normal to normal (SDNN), and the proportion of successive RR intervals (pNN50), have been reported to be associated with MS [43,44]. In multimodal data fusion (see Table 1), a few studies have considered the heart response intended to be used in this study to measure the MS symptom. These studies used heart rate [32,34,35,37], R-peak amplitude [32], SDNN [35], and HRV index (i.e., HRV amplitude, LF, HF and LF/HF ratio) [34] as indicators of cardiac activity. This study intends to consider a new indicator, normalized HRV, which includes cardiac features used in previous studies.
To this end, this study proposes a novel method for classifying MS caused by VR content from HMDs based on cardiac features. The cardiac features measured in both before and after VR viewing conditions are compared with 2D conditions using a statistical analysis method called analysis of covariance (ANCOVA); moreover, these features are analyzed using a partial correlation with a simulator sickness questionnaire (SSQ). Herein, MS (VR condition) is distinguished from the normal state (2D condition) using statistically significant cardiac features based on various classifiers, such as linear discriminant analysis (LDA), K-nearest neighbors (KNN), decision tree (DT), and linear support vector machine (LSVM). In addition, a real-time system is developed to monitor MS.

2. Methods

2.1. Participants

Thirty participants volunteered to participate in this study. Two participants were excluded from the experiment due to premature termination caused by severe levels of VIMS, resulting in a final sample of 28 participants (14 females and 14 males). Participants’ ages ranged from 21 to 34 years (Mean = 26.9 years, Standard deviation = 3.5 years). All participants had normal or corrected-to-normal vision and no family or medical history of cardiovascular disease. Informed written consent was obtained from each subject prior to the experiment, and they were instructed to get a full night’s rest and to abstain from cigarettes, alcohol, and caffeine for 24 h prior to the experiment. This research complied with the tenets of the Declaration of Helsinki and was approved by the Institutional Review Board of Sangmyung University (No. BE2018-46).

2.2. Experimental Stimuli and Procedure

VR content “Ultimate Booster Experience” (GexagonVR, Saint Petersburg, Russia, 2016) was used in this experiment to cause VIMS through the Oculus Rift S VR HMD device (Oculus VR Inc., Menlo Park, CA, USA). This study aimed to choose VR stimuli that can sufficiently cause the MS state, and the VR content “Ultimate Booster Experience” with the highest SSQ score among the five VR contents was chosen through a pre-test of 20 subjects. None of the participants had previously experienced viewing VR content (i.e., Ultimate Booster Experience) by using an HMD. The VR stimulus in the experiment contained content including a giant swing, bungee jump, air balloon, eagle flight, and rocket mode.
This study employed a “within-subject” design to compare the viewer’s cardiac activity in response to the VR contents under the 2D (non-VIMS) and VR conditions (VIMS). The participants sat on a comfortable armchair in an electrically shielded room. They were instructed to experience the VR content in 2D (viewing distance of 1 m) and VR versions using a 27-inch LED monitor (LG Electronics Inc., Seoul, Korea) and the HMD device. The scenes in the two versions were identical. Participants experienced the VR content using either the 2D or the HMD version of the VR content for 12 min. The experiment was conducted simultaneously for two days. The order of tasks (i.e., 2D and VR versions) was count-balanced randomly to minimize sequence/order effects by repeated measures design (that is, VR and 2D versions on the first and second days, respectively, or vice versa). A resting-state was provided for 5 min before and after viewing tasks, and electrocardiogram (ECG) signals were measured. The ECG signals were measured using the lead-1 method using three electrodes placed on the left collarbone (ground, black lead), left (VIN+, red lead), and right arms (VIN-, white lead), see Figure 1B. To obtain optimal electrode response, the surface of the skin where the electrode was to be placed was abraded with a clean, dry cloth. The ECG wireless sensor (ECG BioNomadix) to measure and transmit signals was placed on the stomach using a band. The experiment was conducted in an electrically shielded room to minimize the risk of external interference, which can affect ECG measurements. Except for the necessary user interface, all equipment was located outside the room using a connection cable. In addition, the participants were asked to self-report (a four-point scale, 0 to 3) subjective MS using an SSQ [45] before and after the viewing tasks. Subsequently, cardiac activity and subjective ratings before and after the viewing tasks were compared. The experimental procedure and environment are illustrated in Figure 1.
The SSQ employed is widely used to describe and assess users’ levels of MS symptoms and useful in VR studies. This SSQ comprises 16 items related to the symptoms of MS and is categorized into three non-mutually exclusive factors: (1) nausea (N), which comprises general discomfort, increased salivation, sweating, nausea, difficulty concentrating, stomach awareness, and burping; (2) oculomotor responses (O), which include general discomfort, fatigue, headache, eyestrain, difficulty focusing, difficulty concentrating, and blurred vision; and (3) disorientation (D), which comprises difficulty focusing, nausea, fullness of the head, blurred vision, dizziness (eyes open), dizziness (eyes closed), and vertigo. The total SSQ score is calculated using Equation (1) based on the three factors, where the values of N, O, and D are defined by summing the subjective rating values (4-point scale, 0–3) of each questionnaire for nausea, oculomotor responses, and disorientation, respectively. In general, a five-point scale is used for the single stimulus subjective tests as recommended in ITU-T REC. P.913 [46], but this work performed a subjective rating using the 4-point scale (0, 1, 2, 3) according to guidelines in a previous study [45]. Examples of SSQ are shown in Appendix A.
Total   SSQ   score = N + O + D × 3.74
N   score = N × 9.54 ,   O   score   =   O   × 7.58 ,   D   score = D × 13.92

2.3. Data Acquisition and Signal Processing

ECG signals were recorded at a sampling rate of 500 Hz using Bionomadix BN-ECG2 units and an MP160 amplifier system (Biopac Systems Inc., Goleta, CA, USA). The ECG signals were recorded using circular disposable Ag/AgCl electrodes (11 mm diameter, pregelled, 40 mm foam electrode-EL501, Biopac Systems, Goleta, CA, USA) laced using the Lead-I method. ECG signals were then processed to extract features related to cardiac activity in the following steps: (1) Acquired ECG signals are preprocessed using a band-pass filter with a pass band of 5 Hz to 15 Hz [47] to minimize the effect of muscle artifact, 60 Hz interference, baseline wander, and T-wave interference [48]. (2) R-peak was detected from the preprocessed ECG signals using the “QRS detection algorithm” [48], and the R-peak to R-peak intervals (RRIs) were calculated using the interval between R-peak to R-peak. The detected R-peaks were filtered as normal to normal (NN) intervals, and the criterion for NN intervals was defined as 600–1200 ms based on previous work [49]. Data from RRI that did not meet this criterion were excluded from the analysis. (3) The standard deviation of SDNN was obtained by calculating the standard deviation across normal RRIs [50]. (4) The pNN50 (%) was calculated from the percentage of adjacent RR intervals that differ by more than 50 ms [51]. (5) Successive RRIs were resampled at 4 Hz to convert to time series data and are then analyzed in the HRV spectrum using the fast Fourier transform (FFT, Hanning window technique). (6) The HRV spectrum was categorized into frequency bands for a VLF band ranging from 0.0033–0.04 Hz (sympathetic activity) and an HF band ranging from 0.15–0.4 Hz (parasympathetic activity); moreover, the power for each frequency band was extracted [40,52]. (7) ln VLF and ln HF (that is, normalized HRV) were calculated using the natural log (assuming ln) from the power spectrum values (VLF and HF powers). (8) The normalized HRV plot was categorized into nine zones (that is, Zones 1 to 8 and a reference zone) to assess the autonomic balance of the sympathetic and parasympathetic nervous systems (SNS and PNS) [40,53], as shown in Figure 2. All signal processing and data analyses were performed using MATLAB (2020b, Mathworks Inc., Natick, MA, USA).

2.4. Statistical Analysis

This study follows a “within-subject” design for 2D and VR viewing conditions. Moreover, ANCOVA was applied in this study because a paired t-test between after-viewing conditions is not able to consider the viewers before the state. ANCOVA can assess dependent variables of post-viewing states between the two conditions by considering the pre-viewing state baseline as a covariate [2,54]. Statistical significance at the 95% significance level (p > 0.05) was controlled by the Bonferroni correction as a conservative test to protect against Type 1 errors caused by multiple comparisons based on the number of each hypothesis (i.e., α = 0.05/n) [55,56]. For this study, the statistically significant levels of SSQ and cardiac features were set to 0.0125 (N, O, D, and total SSQ scores resulting in α = 0.05/4) and 0.0083 (heart rate, SDNN, pNN50, ln VLF, ln HF, and ln VLF/ln HF ratio resulting in α = 0.05/6), respectively. The effect size to verify the practical significance was analyzed using the partial eta-squared value (ηp2) corresponding to an F-test. The standard values for the practical significance of 0.01, 0.06, and 0.14 are generally regarded as a small, medium, and large, respectively [57]. A partial correlation was used to analyze the correlation between the SSQ scores and cardiac features (post-viewing condition), considering the pre-viewing condition as a covariate [29]. Correlation coefficients of 0.00–0.09, 0.10–0.39, 0.40–0.69, 0.70–0.89, and 0.90–1.00 represent negligible, weak, moderate (good), strong, and very strong correlations, respectively [58]. All statistical data analyses were conducted using IBM SPSS Statistics 21.0 for Windows (SPSS Inc., Chicago, IL, USA).

2.5. Classification

This study used four basic machine-learning algorithms (LDA, KNN, DT, and LSVM) to verify the features. Optimization results for each classification method were obtained with 10-fold cross-validation by using parameter optimization for classifiers in the classification learner, a MATLAB toolbox. The options for the optimizer are as follows: optimizer, Bayesian optimization; acquisition function, expected improvement per second plus; iterations, 100; training time limit, no; and validation, tenfold cross-validation. The accuracy, recall, precision, F1 score, and receiver operating characteristic (ROC) curve, with the area under the ROC curve (AUC) as the metric, were analyzed to determine the performance of each classification. Additionally, a permutation test was conducted to determine the confidence of the classifiers. The permutation data were repeatedly classified for 10,000 iterations for each classifier (2020b, MathWorks Inc., Natick, MA, USA).

3. Results

3.1. SSQ Scores

All SSQ scores (i.e., N, O, D, and total SSQ) in the VR viewing condition are higher than those in the 2D viewing condition. As seen in Figure 3, the ANCOVA analysis reveals a significant difference in post-viewing condition for N, O, D, and total SSQ scores with the pre-viewing condition as a covariate (N score: F1,53 = 75.948, p < 0.001, with a large effect size (ηp2 = 0.589); O score: F1,53 = 94.215, p < 0.001, with a large effect size (ηp2 = 0.640); D score: F1,53 = 91.157, p < 0.001, with a large effect size (ηp2 = 0.632); total SSQ score: F1,53 = 192.424, p < 0.001, with a large effect size (ηp2 = 0.784)).

3.2. Cardiac Activity

For the VR viewing condition, the heart rate, SDNN, pNN50, and ln HF are lower than those in the 2D viewing condition. Moreover, the ln VLF and ln VLF/ln HF ratios are larger than those under the 2D viewing conditions. As seen in Figure 4, the ANCOVA analysis reveals a significant difference in post-viewing condition for SDNN, pNN50, ln VLF, ln HF, and ln VLF/ln HF ratio with the pre-viewing condition as a covariate (SDNN: F1,53 = 15.244, p < 0.001, with a large effect size (ηp2 = 0.223); pNN50: F1,53 = 11.212, p < 0.0083, with a large effect size (ηp2 = 0.175); ln VLF: F1,53 = 37.031, p < 0.001, with a large effect size (ηp2 = 0.411); ln HF: F1,53 = 56.352, p < 0.001, with a large effect size (ηp2 = 0.515); ln VLF/ln HF ratio: F1,53 = 50.513, p < 0.001, with a large effect size (ηp2 = 0.488)]. Additionally, the heart rate shows no significant result [F1,53 = 2.456, p < 0.001, with a small effect size (ηp2 = 0.044)).

3.3. Normalized HRV (Autonomic Balance)

The autonomic balance was plotted in nine zones of two dimensions using ln VLF (x-axis) and ln HF (y-axis) values for each 2D and VR viewing condition. Considering the results, the 2D condition pattern is stabilized before and after viewing, and the 3D condition pattern can be observed to be significantly changed. As seen in Figure 5, the autonomic balance is mainly located within the reference zone prior to the 2D viewing condition and almost remains within that reference zone without significant changes after viewing. However, in the VR viewing condition, the autonomic balance moves consistently from within the reference zone before viewing to almost into Zone 5 after viewing. From these results, VR viewing destabilizes the autonomic balance and shifts it to a generally activating sympathetic and deactivating parasympathetic nerve.

3.4. Correlation Analysis

Partial correlation is present between the total SSQ score and significant cardiac features in the post-viewing condition, with residual covariates in the pre-viewing condition. Figure 6 shows the plot of residuals for the total SSQ score and cardiac features with linear regression lines. The correlation coefficients between the total SSQ scores and cardiac features (i.e., heart rate, SDNN, pNN50, ln VLF, ln HF, and ln VLF/ln HF ratio) in the post-viewing condition are significantly different (SDNN: r = −0.381, p < 0.01; pNN50: r = −0.377, p < 0.01; ln VLF: r = 0.653, p < 0.001; ln HF: r = −0.711, p < 0.001; ln VLF/ln HF ratio: r = 0.677, p < 0.001). Additionally, note that the heart rate is not significantly different (r = −0.160, p > 0.05).

3.5. Classification Performance and Permutation Test

In this study, the 2D and VR viewing conditions are classified. The features for the classification are only five statistically significant indicators (SDNN, pNN50, ln VLF, ln HF, and ln VLF/ln HF ratio). The heart rate is not included in the classification features. The optimized hyperparameters are as follows: the data for all methods are standardized; the number of neighbors for KNN is 28, and Euclidean distance is applied; the split criterion for DT is maximum deviance reduction, and the maximum number of splits is 2; the box constraint level for LSVM is 0.0706. The accuracies, recalls, precisions, f1 scores, and AUCs of the four classifiers (LDA, KNN, DT, and LSVM) are listed in Table 2. The ROC curves for all classifiers are shown in Figure 7. All the classifiers are significant (p < 0.0001) in the permutation test. Figure 8 shows the permutation test’s accuracy distributions for all the classifiers (10,000 iterations).

3.6. Real-Time System to Monitor MS

The developed real-time system for assessing MS comprises a Wireless ECG Amplifier (Bionomadix BN-ECG2 units), a power supply (MP160), and a personal computer for analysis. It can classify an MS or normal state in a VR environment using an HMD device. This system monitors user behavior via webcam and VR scenes and confirms the results of cardiac features in both the time and frequency domains. In addition, the results depending on time for assessing MS can be confirmed using a sliding bar for each time log. The system was developed using MATLAB App designer (2020b, Mathworks Inc., Natick, MA, USA), and signal processing is performed using the MATLAB toolbox (2020b, Mathworks Inc., Natick, MA, USA), as shown in Figure 9. A real-time system was constructed to record the onset-trigger for the start of the HMD device using user datagram protocol communication to synchronize with the computer for MATLAB. As shown in Figure 10, the proposed method for classifying the MS state involves the processes of signal measurement and synchronization, pre-processing, feature extraction, and classification.

4. Discussion

This paper proposes a quantitative method for measuring MS based on cardiac activity and a real-time system for assessing the same. This study confirms whether 2D and VR viewing conditions cause MS in subjects based on SSQ scores. The subjects in the VR viewing condition demonstrably experience MS but not in the 2D viewing condition, because the SSQ scores show a statistically significant difference between the 2D and VR viewing conditions. This study yields four significant findings: (1) The cardiac features, such as SDNN, pNN50, ln VLF, ln HF, and ln VLF/ln HF ratio, show a significant difference between the 2D and VR viewing conditions based on the ANCOVA analysis. (2) Considering the normalized HRV, the autonomic balance is located within the reference zone in the normal state. However, it moves almost into Zone 5 in MS. (3) The correlation coefficients between the total SSQ score and significant cardiac features are in the range of −0.377 to −0.711 (for SDNN, pNN50, and ln HF) or 0.653 to 0.677 (for ln VLF and ln VLF/ln HF ratio), indicating significant negative or positive correlations, respectively, in the range of weak to strong. (4) Among the algorithms for classifying MS, the highest average accuracy, recall, precision, F1 score, and AUC of MS classification results using the LSVM achieved 91.1, 96.4, 87.1, 91.5, and 0.96 (10-fold cross validation), respectively. The ROC curve is a useful tool when predicting the probability of a binary classification. In general, skillful models reveal curves that bow up to the top left of the plot. When the curve is close to the point (0,1), it is evaluated to be the perfect model. In addition, recall and precision are defined as the ratio of correctly classified positive samples to a total number of classified positive samples (either correctly or incorrectly) and the ratio between the numbers of positive samples correctly classified as positive to the total number of positive samples. Recall and precision are metrics used to assess the performance of classification algorithms since accuracy alone is not sufficient to understand, and improving precision typically reduces recall and vice versa. A good classification model needs to strike the right balance between precision and recall [59]. F1 score is used by combining precision and recall to obtain a balanced classification model. The proposed LSVM-based classification model was confirmed to have good performance. The permutation test generates a null distribution by calculating the accuracy of the classifier on 10,000 different permutations from the experimental data set, and the permutation-based p-values are used to assess the competence of a classifier. This study confirmed that the classification model for the MS reveals a significant result in the permutation test (10,000 iterations, p < 0.0001).
A significant difference can be observed in the cardiac activity results of individuals with subjective symptoms of MS. In this study, both the SNS’s activation and the PNS’s deactivation are determined from the normalized HRV indices in subjects that experience MS. The normalized HRV features have been demonstrated to best reflect the SNS and PNS activities [60,61]. The findings of this study are consistent with those of previous studies in the following aspects: (1) SNS activation (i.e., increasing VLF or LF power) [1,62,63,64] and (2) PNS deactivation (i.e., decreasing HF power) [1,58,61]. Physiological and morphological connections exist between the vestibular and autonomic nervous systems (i.e., the SNS and PNS) [65,66]. Conflicting inputs of afferent signals from visual, vestibular, and somatosensory signals cause MS, and MS typically transmits vestibuloautonomic responses in humans [67]. The activation of SNS can be interpreted as a defensive reaction against the sensation of nausea [68] and is closely related to physiological stress [69]. In a study combining HRV and functional magnetic resonance imaging, brain regions of the medial prefrontal cortex (MPFC) were reported to correlate significantly with the PNS activation during moderate and strong nausea. The MPFC, known to reflect automatic modulatory regions, may have an excitatory influence on the autonomic control regions. MS is closely associated with the switch from the inhibitory to excitatory influences on the cardiovagal outflow [70]. In addition, SDNN reflects both the sympathetic and parasympathetic influences, and pNN50 corresponds to the parasympathetic neural regulation from heart activity [71]. This study verifiably demonstrates that the SDNN and pNN50 show a significant difference between the MS and normal states, and the results of this study agree with those of previous studies [44,72]. Moreover, cardiac-related features are verified to be closely related to the phenomenon of MS and are significant indicators for quantitative assessment of MS.
In addition, many previous studies have reported that MS is strongly associated with the cognitive load caused by inconsistencies or conflicts among different types of sensory information (i.e., visual and motor) [2,16,73,74,75]. Sensory conflict is a well-known theory that explains MS, and it can be caused by a mismatch or inconsistency between the visual and vestibular senses during HMD usage [76]. MS can lead to distortions and delays in visual information processing by the brain due to a mismatch between sensory information; moreover, it can be interpreted as consuming excessive neural resources for processing the massive visual information in VR compared to that in 2D [2,16]. Cardiac activity is strongly associated with cognitive load. The heart transmits sensory information related to its activities to the brain based on afferent pathways, and this phenomenon is known as heartbeat-evoked potentials (HEPs) [1,77]. Cognitive processing in the brain is influenced by the changes in heart rhythm via the afferent and efferent pathways between the heart and brain [77,78,79]. Our previous work demonstrated that increasing cognitive load is correlated with an irregular pattern of heart rhythms (that is, increasing SDNN and pNN50), increasing the activation of SNS, and decreasing the activation of PNS [40]. In previous studies, significant results of cardiac activity have also been confirmed as an indicator of cognitive load [80,81,82,83]. Thus, the findings in this study related to changes in cardiac activity can suggest that the cognitive load can be interpreted as conflict among sensory inputs and processing large amounts of visual information. In addition, a quantitative method and real-time system for monitoring MS are developed in this study. Many previous studies have attempted to perform quantitative measurements using EEG, HEP, vision responses, and multi-modal fusion. These studies reported that MS can be classified with 74.7% to 99.1% accuracy [2,16,24,25,26,27,28,29,30,31,32,33,34,35]. In this study, a competitive classification accuracy (91.1%) of MS compared to previous studies was achieved using a single measurement. In addition, the approach used in previous studies has high complexity and disadvantages for measuring or evaluating MS, such as the measurements requiring sensor attachments, as well as low usability compared to cardiac activity. The proposed method also has the advantage of minimizing limitations such as complex and expensive equipment, inconvenience, and the burden of sensor attachment, which can be further minimized by replacing the implemented ECG sensor with photoplethysmography, compared to approaches using EEG or multimodal fusion, as in past studies. The method proposed in this study cannot be judged to have improved performance or contribution compared previous studies, but there will be differences in potential fields of application based on the advantages and disadvantages of each measurement method. For example, EEG or other features can be used primarily for precise measurements of MS because they have the advantage of measuring more important and significant data. We believe that the proposed method will have a contribution that can be applied to fields requiring fast and convenient measurement of MS based on usability, which is simple to measure. In addition, the symptoms of MS are known to be caused by various factors, such as user characteristics (i.e., viewing time, age, gender), viewing environment (i.e., gaze angle, fixation, field of view), device, and VR content (i.e., resolution). The relationship between the MS state and these causal factors should be verified to minimize the sensitivity of the MS. To verify each condition’s effect on the MS, the development of a methodology for quantitatively evaluating the MS state should be preceded. Thus, the proposed method can contribute to improving the VR experience for viewers by minimizing MS.
This study has several limitations. (1) This study selected an SSQ questionnaire to perform the subjective measurement of MS, because the SSQ questionnaire is the best-known test for the subjective measurement for MS. However, traditional SSQ questionnaires were developed for measuring cybersickness and may differ significantly from today’s VR experience. In future studies, recently revised questionnaires need to be considered (e.g., virtual reality sickness questionnaire [84]), including the traditional SSQ questionnaire. (2) Previous studies related to MS have often reported conflicting results. For example, after experiencing the phenomena of MS, one study reported an increase in HR [36], whereas another identified a decrease in HR [37]. This study confirmed a decrease in HR consistent with the previous study [37], but it showed no statistically significant difference between 2D and HMD. These discrepancies in study conclusions may be attributed in part to experimental and procedural differences [85]. Alternatively, they could be attributable to differences in VR content or HMD device. Thus, it cannot be determined that the proposed method can measure all of the MS symptoms caused by various conditions. This limitation needs to be validated via future studies.

5. Conclusions

This paper proposes a method for assessing MS caused by watching VR content on an HMD using cardiac features. This study demonstrates a significant difference in cardiac features between the MS and normal states, and the classification performance achieves an average accuracy of 91.1%. In addition, a real-time system was developed for continuously monitoring MS using significant cardiac features. The proposed method can quantify the level of MS severity and determine the optimal viewing factors of a VR environment (i.e., viewer characteristics, viewing condition, VR content factors, and HMD device factors) to minimize the symptoms of MS. This research will consequently help improve the viewing environment and lead to the establishment of viewer-friendly VR. However, since MS is strongly correlated with various factors, such as age and gender [86], further research should be conducted to increase the generality of the classifiers in subjects of different ages and genders.

Author Contributions

S.P.: conceptualization, methodology, data curation, validation, writing—original draft; J.H.: investigation, visualization, data curation, software; L.K.: conceptualization, writing—review and editing, supervision. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by the Challengeable Future Defense Technology Research and Development Program (912911601) of Agency for Defense Development in 2020.

Institutional Review Board Statement

This experimental study was approved and reviewed by the Institutional Review Board (approval number: BE2018-46) of the Sangmyung University.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The datasets generated and/or analyzed during the current study are not publicly available due to confidentiality agreements but are available from the corresponding author on reasonable request.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table A1. Simulator sickness questionnaire [45]. Instruction: Circle how much each symptom below is affecting you right now.
Table A1. Simulator sickness questionnaire [45]. Instruction: Circle how much each symptom below is affecting you right now.
SSQ ItemNone: 0, Slight: 1, Moderate: 2, Severe: 3
General discomfortNone□Slight□Moderate□Severe□
FatigueNone□Slight□Moderate□Severe□
HeadacheNone□Slight□Moderate□Severe□
EyestrainNone□Slight□Moderate□Severe□
Difficulty focusingNone□Slight□Moderate□Severe□
Increased salivationNone□Slight□Moderate□Severe□
SweatingNone□Slight□Moderate□Severe□
NauseaNone□Slight□Moderate□Severe□
Difficulty concentratingNone□Slight□Moderate□Severe□
Fullness of HeadNone□Slight□Moderate□Severe□
Blurred visionNone□Slight□Moderate□Severe□
Dizziness with eye openNone□Slight□Moderate□Severe□
Dizziness with eye closedNone□Slight□Moderate□Severe□
VertigoNone□Slight□Moderate□Severe□
Stomach awarenessNone□Slight□Moderate□Severe□
BurpingNone□Slight□Moderate□Severe□

References

  1. Setiowati, N.O.; Wijayanto, T.; Trapsilawati, F. Identifying Cybersickness When Wearing a Head-Mounted Display Through Heart Rate Variability Data. IOP Conf. Ser. Mater. Sci. Eng. 2020, 885, 012069. [Google Scholar] [CrossRef]
  2. Park, S.; Kim, L.; Kwon, J.; Choi, S.J.; Whang, M. Evaluation of Visual-Induced Motion Sickness from Head-Mounted Display Using Heartbeat Evoked Potential: A Cognitive Load-Focused Approach. Virtual Real. 2021, 8, 1–22. [Google Scholar] [CrossRef]
  3. Hidalgo, M.; Harris, S.; Boland, W.; Halfman, T.; Johnston, J.; Hillyer, T.; Patton, D.; Elliott, L. Training Capabilities Assessment in Support of Enhanced Military Training: Comparing Head-Mounted Displays. In Lecture Notes in Networks and Systems International Conference on Applied Human Factors and Ergonomics; Springer: Berlin/Heidelberg, Germany, 2021; pp. 11–18. [Google Scholar] [CrossRef]
  4. Deib, G.; Johnson, A.; Unberath, M.; Yu, K.; Andress, S.; Qian, L.; Osgood, G.; Navab, N.; Hui, F.; Gailloud, P. Image guided percutaneous spine procedures using an optical see-through head mounted display: Proof of concept and rationale. J. NeuroInterv. Surg. 2018, 10, 1187–1191. [Google Scholar] [CrossRef] [PubMed]
  5. Flotyński, J.; Englert, A.; Nowak, A.; Walczak, K. An Architecture for Distributed Explorable HMD-Based Virtual Reality Environments. In Advances in Intelligent Systems and Computing International Conference on Information Systems Architecture and Technology; Springer: Berlin/Heidelberg, Germany, 2020; pp. 38–47. [Google Scholar] [CrossRef]
  6. Malik, A.A.; Masood, T.; Bilberg, A. Virtual Reality in Manufacturing: Immersive and Collaborative Artificial-Reality in Design of Human-Robot Workspace. Int. J. Comput. Integr. Manuf. 2020, 33, 22–37. [Google Scholar] [CrossRef]
  7. Zhang, Q.; Ban, J.S.; Kim, M.; Byun, H.W.; Kim, C.H. Low-Asymmetry Interface for Multiuser VR Experiences with Both HMD and Non-HMD Users. Sensors 2021, 21, 397. [Google Scholar] [CrossRef] [PubMed]
  8. Seok, K.H.; Kim, Y.; Son, W.; Kim, Y.S. Using Visual Guides to Reduce Virtual Reality Sickness in First-Person Shooter Games: Correlation Analysis. JMIR Serious Games 2021, 9, e18020. [Google Scholar] [CrossRef] [PubMed]
  9. Servotte, J.-C.; Goosse, M.; Campbell, S.H.; Dardenne, N.; Pilote, B.; Simoneau, I.L.; Guillaume, M.; Bragard, I.; Ghuysen, A. Virtual Reality Experience: Immersion, Sense of Presence, and Cybersickness. Clin. Simul. Nurs. 2020, 38, 35–43. [Google Scholar] [CrossRef]
  10. Carnegie, K.; Rhee, T. Reducing Visual Discomfort with HMDs Using Dynamic Depth of Field. IEEE Comput. Graph. Appl. 2015, 35, 34–41. [Google Scholar] [CrossRef] [PubMed]
  11. Cha, Y.H.; Golding, J.F.; Keshavarz, B.; Furman, J.; Kim, J.S.; Lopez-Escamez, J.A.; Magnusson, M.; Yates, B.J.; Lawson, B.D.; Staab, J.F.; et al. Motion Sickness Diagnostic Criteria: Consensus Document of the Classification Committee of the Barany Society. J. Vestib. Res. 2021, 31, 327–344. [Google Scholar] [CrossRef] [PubMed]
  12. Murray, J.B. Psychophysiological Aspects of Motion Sickness. Percept. Mot. Skills 1997, 85, 1163–1167. [Google Scholar] [CrossRef] [PubMed]
  13. Merhi, O.; Faugloire, E.; Flanagan, M.; Stoffregen, T.A. Motion Sickness, Console Video Games, and Head-Mounted Displays. Hum. Factors 2007, 49, 920–934. [Google Scholar] [CrossRef] [PubMed]
  14. Liao, C.-Y.; Tai, S.-K.; Chen, R.-C.; Hendry, H. Using EEG and Deep Learning to Predict Motion Sickness Under Wearing a Virtual Reality Device. IEEE Access. 2020, 8, 126784–126796. [Google Scholar] [CrossRef]
  15. Gálvez-García, G.; Aldunate, N.; Bascour-Sandoval, C.; Barramuño, M.; Fonseca, F.; Gómez-Milán, E. Decreasing Motion Sickness by Mixing Different Techniques. Appl. Ergon. 2020, 82, 102931. [Google Scholar] [CrossRef] [PubMed]
  16. Park, S.; Mun, S.; Ha, J.; Kim, L. Non-contact Measurement of Motion Sickness Using Pupillary Rhythms from an Infrared Camera. Sensors 2021, 21, 4642. [Google Scholar] [CrossRef] [PubMed]
  17. Litleskare, S. The Relationship Between Postural Stability and Cybersickness: It’s Complicated—An Experimental Trial Assessing Practical Implications of Cybersickness Etiology. Physiol. Behav. 2021, 236, 113422. [Google Scholar] [CrossRef] [PubMed]
  18. Wurthmann, S.; Naegel, S.; Roesner, M.; Nsaka, M.; Scheffler, A.; Kleinschnitz, C.; Holle, D.; Obermann, M. Sensitized Rotatory Motion Perception and Increased Susceptibility to Motion Sickness in Vestibular Migraine: A Cross-Sectional Study. Eur. J. Neurol. 2021, 28, 2357–2366. [Google Scholar] [CrossRef] [PubMed]
  19. Nam, Y.; Hong, U.; Chung, H.; Noh, S.R. Eye Movement Patterns Reflecting Cybersickness: Evidence from Different Experience Modes of a Virtual Reality Game. Cyberpsychol. Behav. Soc. Netw. 2022, 25, 135–139. [Google Scholar] [CrossRef] [PubMed]
  20. Smyth, J.; Birrell, S.; Woodman, R.; Jennings, P. Exploring the Utility of EDA and Skin Temperature as Individual Physiological Correlates of Motion Sickness. Appl. Ergon. 2021, 92, 103315. [Google Scholar] [CrossRef]
  21. Jang, K.M.; Kwon, M.; Nam, S.G.; Kim, D.; Lim, H.K. Estimating Objective (EEG) and Subjective (SSQ) Cybersickness in People with Susceptibility to Motion Sickness. Appl. Ergon. 2022, 102, 103731. [Google Scholar] [CrossRef] [PubMed]
  22. Carvalho, G.F.; Mehnert, J.; Basedau, H.; Luedtke, K.; May, A. Brain Processing of Visual Self-Motion Stimuli in Patients with Migraine: An fMRI Study. Neurology 2021, 97, e996–e1006. [Google Scholar] [CrossRef] [PubMed]
  23. John, B. Pupil Diameter as a Measure of Emotion and Sickness in VR. In Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications, Denver, CO, USA, 25–28 June 2019; pp. 1–3. [Google Scholar]
  24. Yu, Y.-H.; Lai, P.-C.; Ko, L.-W.; Chuang, C.-H.; Kuo, B.-C.; Lin, C.-T. An EEG-Based Classification System of Passenger’s Motion Sickness Level by Using Feature Extraction/Selection Technologies. In Proceedings of the 2010 International Joint Conference on Neural Networks (IJCNN), Barcelona, Spain, 18–23 July 2010; IEEE Publications: Piscataway, NJ, USA, 2010; pp. 1–6. [Google Scholar]
  25. Wei, C.-S.; Ko, L.-W.; Chuang, S.-W.; Jung, T.-P.; Lin, C.-T. EEG-Based Evaluation System for Motion Sickness Estimation. In Proceedings of the 5th International IEEE/EMBS Conference on Neural Engineering, Cancun, Mexico, 27 April–1 May 2011; Volume 2011, pp. 100–103. [Google Scholar]
  26. Ko, L.-W.; Lee, H.-C.; Tsai, S.-F.; Shih, T.-C.; Chuang, Y.-T.; Huang, H.-L.; Ho, S.-Y.; Lin, C.-T. EEG-Based Motion Sickness Classification System with Genetic Feature Selection. In Proceedings of the IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB), Singapore, 16–19 April 2013; IEEE Publications: Piscataway, NJ, USA, 2013; pp. 158–164. [Google Scholar]
  27. Mawalid, M.A.; Khoirunnisa, A.Z.; Purnomo, M.H.; Wibawa, A.D. Classification of EEG Signal for Detecting Cybersickness Through Time Domain Feature Extraction Using Naïve Bayes. In Proceedings of the International Conference on Computer Engineering, Network and Intelligent Multimedia (CENIM), Surabaya, Indonesia, 26–27 November 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 29–34. [Google Scholar]
  28. Pane, E.S.; Khoirunnisaa, A.Z.; Wibawa, A.D.; Purnomo, M.H. Identifying Severity Level of Cybersickness from Eeg Signals Using cn2 Rule Induction Algorithm. In Proceedings of the International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS), Bangkok, Thailand, 21–24 October 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 170–176. [Google Scholar]
  29. Jung, H.; Kim, H.S.; Kim, J.Y.; Sun, J.M.; Ahn, J.S.; Ahn, M.J.; Park, K.; Esteller, M.; Lee, S.H.; Choi, J.K. DNA Methylation Loss Promotes Immune Evasion of Tumours with High Mutation and Copy Number Load. Nat. Commun. 2019, 10, 4278. [Google Scholar] [CrossRef] [PubMed]
  30. Li, X.; Zhu, C.; Xu, C.; Zhu, J.; Li, Y.; Wu, S. VR Motion Sickness Recognition by Using EEG Rhythm Energy Ratio Based on Wavelet Packet Transform. Comput. Methods Programs Biomed. 2020, 188, 105266. [Google Scholar] [CrossRef] [PubMed]
  31. Liu, R.; Cui, S.S.; Zhao, Y.; Chen, X.; Yi, L.; Hwang, A.D. VIMSNet: An Effective Network for Visually Induced Motion Sickness Detection. Signal Image Video Process. 2022, 1–8. [Google Scholar] [CrossRef]
  32. Dennison, M., Jr.; D’Zmura, M.; Harrison, A.; Lee, M.; Raglin, A. Improving Motion Sickness Severity Classification Through Multi-modal Data Fusion. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications; International Society for Optics and Photonics: Bellingham, WA, USA, 2019; Volume 110060T. [Google Scholar]
  33. Li, Y.; Liu, A.; Ding, L. Machine Learning Assessment of Visually Induced Motion Sickness Levels Based on Multiple Biosignals. Biomed. Signal Process. Control 2019, 49, 202–211. [Google Scholar] [CrossRef]
  34. Oh, S.; Kim, D.K. Machine-Deep-Ensemble Learning Model for Classifying Cybersickness Caused by Virtual Reality Immersion. Cyberpsychol. Behav. Soc. Netw. 2021, 24, 729–736. [Google Scholar] [CrossRef]
  35. Recenti, M.; Ricciardi, C.; Aubonnet, R.; Picone, I.; Jacob, D.; Svansson, H.Á.R.; Agnarsdóttir, S.; Karlsson, G.H.; Baeringsdóttir, V.; Petersen, H.; et al. Toward Predicting Motion Sickness Using Virtual Reality and a Moving Platform Assessing Brain, Muscles, and Heart Signals. Front. Bioeng. Biotechnol. 2021, 9, 635661. [Google Scholar] [CrossRef]
  36. Dennison, M.S.; Wisti, A.Z.; D’Zmura, M. Use of physiological signals to predict cybersickness. Displays 2016, 44, 42–52. [Google Scholar] [CrossRef]
  37. Garcia-Agundez, A.; Reuter, C.; Becker, H.; Konrad, R.; Caserman, P.; Miede, A.; Göbel, S. Development of a classifier to determine factors causing cybersickness in virtual reality environments. Games Health J. 2019, 8, 439–444. [Google Scholar] [CrossRef]
  38. Appelhans, B.M.; Luecken, L.J. Heart Rate Variability and Pain: Associations of Two Interrelated Homeostatic Processes. Biol. Psychol. 2008, 77, 174–182. [Google Scholar] [CrossRef]
  39. Reyes del Paso, G.A.; Langewitz, W.; Mulder, L.J.; van Roon, A.; Duschek, S. The Utility of Low Frequency Heart Rate Variability as an Index of Sympathetic Cardiac Tone: A Review with Emphasis on a Reanalysis of Previous Studies. Psychophysiology 2013, 50, 477–487. [Google Scholar] [CrossRef]
  40. Park, S.; Won, M.J.; Mun, S.; Lee, E.C.; Whang, M. Does Visual Fatigue from 3D Displays Affect Autonomic Regulation and Heart Rhythm? Int. J. Psychophysiol. 2014, 92, 42–48. [Google Scholar] [CrossRef] [PubMed]
  41. Lin, C.T.; Lin, C.L.; Chiu, T.W.; Duann, J.R.; Jung, T.P. Effect of Respiratory Modulation on Relationship Between Heart Rate Variability and Motion Sickness. In Proceedings of the 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, USA, 30 August–3 September 2011; pp. 1921–1924. [Google Scholar] [CrossRef]
  42. Karjanto, J.; Yusof, N.M.; Abd Fathul Hakim Zulkifli, J.T.; Delbressine, F.; Rauterberg, M. Frequency-Domain Analysis of Heart Rate Variability in Passenger’s Motion Sickness Using Fast Fourier Transform and Autoregressive Modeling. Proc. Mech. Eng. Res. Day 2020, 2020, 8–9. [Google Scholar]
  43. Sugita, N.; Yoshizawa, M.; Tanaka, A.; Abe, K.; Chiba, S.; Yambe, T.; Nitta, S. Quantitative Evaluation of Effects of Visually Induced Motion Sickness Based on Causal Coherence Functions Between Blood Pressure and Heart Rate. Displays 2008, 29, 167–175. [Google Scholar] [CrossRef]
  44. Martin, N.; Mathieu, N.; Pallamin, N.; Ragot, M.; Diverrez, J.-M. Automatic Recognition of Virtual Reality Sickness Based on Physiological Signals; IBC: Amsterdam, Netherlands, 2018. [Google Scholar]
  45. Kennedy, R.S.; Lane, N.E.; Berbaum, K.S.; Lilienthal, M.G. Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. Int. J. Aviat. Psychol. 1993, 3, 203–220. [Google Scholar] [CrossRef]
  46. ITU-T. ITU-T P.913: Methods for the Subjective Assessment of Video Quality, Audio Quality and Audiovisual Quality of Internet Video and Distribution Quality Television in Any Environment. Available online: https://www.itu.int/rec/T-REC-P.913/en (accessed on 21 June 2021).
  47. Vastarouchas, C.; Kapoulea, S.; Psychalinos, C. ECG Signal Acquisition for the Pan-Tompkins Algorithm Using Current-Mirror Filters. In Proceedings of the IEEE International Conference on Electronics, Circuits and Systems (ICECS), Monte Carlo, Monaco, 11–14 December 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 317–320. [Google Scholar]
  48. Pan, J.; Tompkins, W.J. A Real-Time QRS Detection Algorithm. IEEE Trans. Biomed. Eng. 1985, 32, 230–236. [Google Scholar] [CrossRef]
  49. Kim, J.; Cho, K.; Kim, Y.-K.; Lim, K.-S.; Shin, S.U. Study on Peak Misdetection Recovery of Key Exchange Protocol Using Heartbeat. J. Supercomput. 2019, 75, 3288–3301. [Google Scholar] [CrossRef]
  50. Wang, H.-M.; Huang, S.-C. SDNN/RMSSD as a Surrogate for LF/HF: A Revised Investigation. Modell. Simul. Eng. 2012, 2012, 16. [Google Scholar] [CrossRef]
  51. Siepmann, M.; Krause, S.; Joraschky, P.; Mück-Weymann, M.; Kirch, W. The Effects of St John’s Wort Extract on Heart Rate Variability, Cognitive Function and Quantitative EEG: A Comparison with Amitriptyline and Placebo in Healthy Men. Br. J. Clin. Pharmacol. 2002, 54, 277–282. [Google Scholar] [CrossRef]
  52. Park, S.; Won, M.J.; Lee, D.W.; Whang, M. Non-contact Measurement of Heart Response Reflected in Human Eye. Int. J. Psychophysiol. 2018, 123, 179–198. [Google Scholar] [CrossRef]
  53. McCraty, R.; Watkins, A. Autonomic Assessment Report: A Comprehensive Heart Rate Variability Analysis. In Heart Math Research Center Reports; Institute of Heart Math: Boulder Creek, CA, USA, 1996. [Google Scholar]
  54. Keselman, H.J.; Huberty, C.J.; Lix, L.M.; Olejnik, S.; Cribbie, R.A.; Donahue, B.; Kowalchuk, R.K.; Lowman, L.L.; Petoskey, M.D.; Keselman, J.C.; et al. Statistical Practices of Educational Researchers: An Analysis of Their ANOVA, MANOVA, and ANCOVA Analyses. Rev. Educ. Res. 1998, 68, 350–386. [Google Scholar] [CrossRef]
  55. Dunnett, C.W. A Multiple Comparison Procedure for Comparing Several Treatments with a Control. J. Am. Stat. Assoc. 1955, 50, 1096–1121. [Google Scholar] [CrossRef]
  56. Johnson, J.L.; Slentz, C.A.; Ross, L.M.; Huffman, K.M.; Kraus, W.E. Ten-Year Legacy Effects of Three Eight-Month Exercise Training Programs on Cardiometabolic Health Parameters. Front. Physiol. 2019, 10, 452. [Google Scholar] [CrossRef] [PubMed]
  57. Huck, S.W.; Cormier, W.H.; Bounds, W.G. Reading Statistics and Research; Harper & Row: New York, NY, USA, 1974. [Google Scholar]
  58. Schober, P.; Boer, C.; Schwarte, L.A. Correlation Coefficients: Appropriate Use and Interpretation. Anesth. Analg. 2018, 126, 1763–1768. [Google Scholar] [CrossRef] [PubMed]
  59. Buckland, M.; Gey, F. The relationship between recall and precision. J. Am. Soc. Inf. Sci. 1994, 45, 12–19. [Google Scholar] [CrossRef]
  60. Arlt, J.; Jahn, H.; Kellner, M.; Ströhle, A.; Yassouridis, A.; Wiedemann, K. Modulation of Sympathetic Activity by Corticotropin-Releasing Hormone and Atrial Natriuretic Peptide. Neuropeptides 2003, 37, 362–368. [Google Scholar] [CrossRef] [PubMed]
  61. Chen, J.L.; Chiu, H.W.; Tseng, Y.J.; Chu, W.C. Hyperthyroidism Is Characterized by Both Increased Sympathetic and Decreased Vagal Modulation of Heart Rate: Evidence from Spectral Analysis of Heart Rate Variability. Clin. Endocrinol. 2006, 64, 611–616. [Google Scholar] [CrossRef]
  62. Lin, C.L.; Jung, T.P.; Chuang, S.W.; Duann, J.R.; Lin, C.T.; Chiu, T.W. Self-Adjustments May Account for the Contradictory Correlations Between HRV and Motion-Sickness Severity. Int. J. Psychophysiol. 2013, 87, 70–80. [Google Scholar] [CrossRef]
  63. Malińska, M.; Zużewicz, K.; Bugajska, J.; Grabowski, A. Heart Rate Variability (HRV) During Virtual Reality Immersion. Int. J. Occup. Saf. Ergon. 2015, 21, 47–54. [Google Scholar] [CrossRef]
  64. Tu, M.Y.; Chu, H.; Lai, C.Y.; Chiang, K.T.; Huang, C.C.; Chin, H.C.; Wen, Y.H.; Chen, C.L. Effect of Standardized Yelling on Subjective Perception and Autonomic Nervous System Activity in Motion Sickness. Int. J. Environ. Res. Public Health 2021, 18, 12854. [Google Scholar] [CrossRef]
  65. Money, K.E. Motion Sickness. Physiol. Rev. 1970, 50, 1–39. [Google Scholar] [CrossRef]
  66. Previc, F.H. Do the Organs of the Labyrinth Differentially Influence the Sympathetic and Parasympathetic Systems? Neurosci. Biobehav. Rev. 1993, 17, 397–404. [Google Scholar] [CrossRef]
  67. Ohyama, S.; Nishiike, S.; Watanabe, H.; Matsuoka, K.; Akizuki, H.; Takeda, N.; Harada, T. Autonomic Responses During Motion Sickness Induced by Virtual Reality. Auris Nasus Larynx 2007, 34, 303–306. [Google Scholar] [CrossRef] [PubMed]
  68. Himi, N.; Koga, T.; Nakamura, E.; Kobashi, M.; Yamane, M.; Tsujioka, K. Differences in Autonomic Responses Between Subjects with and Without Nausea While Watching an Irregularly Oscillating Video. Auton. Neurosci. 2004, 116, 46–53. [Google Scholar] [CrossRef]
  69. Sloan, R.P.; Shapiro, P.A.; Bagiella, E.; Boni, S.M.; Paik, M.; Bigger, J.T., Jr.; Steinman, R.C.; Gorman, J.M. Effect of Mental Stress Throughout the Day on Cardiac Autonomic Control. Biol. Psychol. 1994, 37, 89–99. [Google Scholar] [CrossRef]
  70. Kim, J.; Napadow, V.; Kuo, B.; Barbieri, R. A Combined HRV-fMRI Approach to Assess Cortical Control of Cardiovagal Modulation by Motion Sickness. In Proceedings of the 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, USA, 30 August–3 September 2011; pp. 2825–2828. [Google Scholar] [CrossRef]
  71. Markov, A.; Solonin, I.; Bojko, E. Heart Rate Variability in Workers of Various Professions in Contrasting Seasons of the Year. Int. J. Occup. Med. Environ. Health 2016, 29, 793–800. [Google Scholar] [CrossRef] [PubMed]
  72. Chandana, S.; Purnima, B.R.; Vittal, P.R. Classification of Individuals Based on Autonomic Response to Virtual Gaming. J. Comp. Theor. Nanosci. 2020, 17, 4385–4393. [Google Scholar] [CrossRef]
  73. Lin, C.T.; Tsai, S.F.; Ko, L.W. EEG-Based Learning System for Online Motion Sickness Level Estimation in a Dynamic Vehicle Environment. IEEE Trans. Neural Netw. Learn. Syst. 2013, 24, 1689–1700. [Google Scholar] [CrossRef] [PubMed]
  74. Chang, Y.K.; Alderman, B.L.; Chu, C.H.; Wang, C.C.; Song, T.F.; Chen, F.T. Acute Exercise Has a General Facilitative Effect on Cognitive Function: A Combined ERP Temporal Dynamics and BDNF Study. Psychophysiology 2017, 54, 289–300. [Google Scholar] [CrossRef]
  75. Mittelstaedt, J.M.; Wacker, J.; Stelling, D. VR Aftereffect and the Relation of Cybersickness and Cognitive Performance. Virtual Real. 2019, 23, 143–154. [Google Scholar] [CrossRef]
  76. Chung, W.; Barnett-Cowan, M. Influence of Sensory Conflict on Perceived Timing of Passive Rotation in Virtual Reality. Multisens Res. 2022, 1, 1–23. [Google Scholar] [CrossRef]
  77. McCraty, R.; Atkinson, M.; Tomasino, D.; Bradley, R.T. The Coherent Heart-Brain Interactions, Psychophysiological Coherence, and the Emergence of System-wide Order. Integr. Rev. Transdiscipl. Transcult. J. New Thought Res. Prax. 2009, 5, 10–115. [Google Scholar]
  78. Hansen, A.L.; Johnsen, B.H.; Thayer, J.F. Vagal Influence on Working Memory and Attention. Int. J. Psychophysiol. 2003, 48, 263–274. [Google Scholar] [CrossRef]
  79. Park, S.; Won, M.J.; Lee, E.C.; Mun, S.; Park, M.C.; Whang, M. Evaluation of 3D Cognitive Fatigue Using Heart-Brain Synchronization. Int. J. Psychophysiol. 2015, 97, 120–130. [Google Scholar] [CrossRef] [PubMed]
  80. Fuentes-García, J.P.; Villafaina, S.; Collado-Mateo, D.; de la Vega, R.; Olivares, P.R.; Clemente-Suárez, V.J. Differences Between High vs. Low Performance Chess Players in Heart Rate Variability During Chess Problems. Front. Psychol. 2019, 10, 409. [Google Scholar] [CrossRef]
  81. Guo, J.; Dai, Y.; Wang, C.; Wu, H.; Xu, T.; Lin, K. A Physiological Data-Driven Model for Learners’ Cognitive Load Detection Using HRV-PRV Feature Fusion and Optimized XGBoost Classification. Softw. Pract. Exp. 2020, 50, 2046–2064. [Google Scholar] [CrossRef]
  82. Solhjoo, S.; Haigney, M.C.; McBee, E.; van Merrienboer, J.J.G.; Schuwirth, L.; Artino, A.R., Jr.; Battista, A.; Ratcliffe, T.A.; Lee, H.D.; Durning, S.J. Heart Rate and Heart Rate Variability Correlate with Clinical Reasoning Performance and Self-Reported Measures of Cognitive Load. Sci. Rep. 2019, 9, 14668. [Google Scholar] [CrossRef]
  83. Urrestilla, N.; St-Onge, D. Measuring Cognitive Load: Heart-Rate Variability and Pupillometry Assessment. In Proceedings of the Companion Publication of the 2020 International Conference on Multimodal Interaction, Online, 25–29 October 2020; Association for Computing Machinery: New York, NY, USA, 2020; pp. 405–410. [Google Scholar]
  84. Kim, H.K.; Park, J.; Choi, Y.; Choe, M. Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment. Appl. Ergon. 2019, 69, 66–73. [Google Scholar] [CrossRef] [PubMed]
  85. Uijtdehaage, S.H.J. The Effects of Scopolamine and Methscopolamine on Vection-Induced Motion Sickness; The Pennsylvania State University Press: University Park, PA, USA, 1991. [Google Scholar]
  86. Golding, J.F. Motion Sickness Susceptibility. Auton. Neurosci. 2006, 129, 67–76. [Google Scholar] [CrossRef] [PubMed]
Figure 1. (A) Experimental procedure and (B) environment.
Figure 1. (A) Experimental procedure and (B) environment.
Sensors 22 06213 g001
Figure 2. Zone definitions of normalized HRV. (A) Zone 1: High parasympathetic/low sympathetic. (B) Zone 2: High parasympathetic/normal sympathetic. (C) Zone 3: High dual autonomic tone. (D) Zone 4: High sympathetic/normal parasympathetic. (E) Zone 5: High sympathetic/low parasympathetic. (F) Zone 6: Normal sympathetic/low parasympathetic. (G) Zone 7: Low sympathetic and parasympathetic. (H) Zone 8: Low sympathetic/normal parasympathetic.
Figure 2. Zone definitions of normalized HRV. (A) Zone 1: High parasympathetic/low sympathetic. (B) Zone 2: High parasympathetic/normal sympathetic. (C) Zone 3: High dual autonomic tone. (D) Zone 4: High sympathetic/normal parasympathetic. (E) Zone 5: High sympathetic/low parasympathetic. (F) Zone 6: Normal sympathetic/low parasympathetic. (G) Zone 7: Low sympathetic and parasympathetic. (H) Zone 8: Low sympathetic/normal parasympathetic.
Sensors 22 06213 g002
Figure 3. Representation of SSQ scores for MS between the 2D and VR conditions based on the ANCOVA test (*** p < 0.001). (A) SSQ items of nausea, oculomotor responses, and disorientation. (B) Total SSQ score.
Figure 3. Representation of SSQ scores for MS between the 2D and VR conditions based on the ANCOVA test (*** p < 0.001). (A) SSQ items of nausea, oculomotor responses, and disorientation. (B) Total SSQ score.
Sensors 22 06213 g003
Figure 4. Representation of cardiac activity for MS between the 2D and VR conditions based on the ANCOVA test (*** p < 0.001 and ** p < 0.0083). (A) RRI (heart rate). (B) SDNN. (C) pNN50. (D) ln VLF. (E) ln HF. (F) ln VLF/ln HF ratio.
Figure 4. Representation of cardiac activity for MS between the 2D and VR conditions based on the ANCOVA test (*** p < 0.001 and ** p < 0.0083). (A) RRI (heart rate). (B) SDNN. (C) pNN50. (D) ln VLF. (E) ln HF. (F) ln VLF/ln HF ratio.
Sensors 22 06213 g004
Figure 5. Comparison of the autonomic balance (normalized HRV) before and after the (A) 2D and (B) VR viewing conditions, structured by predefined zones of distribution.
Figure 5. Comparison of the autonomic balance (normalized HRV) before and after the (A) 2D and (B) VR viewing conditions, structured by predefined zones of distribution.
Sensors 22 06213 g005
Figure 6. Results of the partial correlation analysis between the total SSQ score and cardiac features (red dotted line, p < 0.001). (A) Heart rate. (B) SDNN. (C) pNN50. (D) ln VLF. (E) ln HF. (F) ln VLF/ln HF ratio.
Figure 6. Results of the partial correlation analysis between the total SSQ score and cardiac features (red dotted line, p < 0.001). (A) Heart rate. (B) SDNN. (C) pNN50. (D) ln VLF. (E) ln HF. (F) ln VLF/ln HF ratio.
Sensors 22 06213 g006
Figure 7. ROC curves for 10-fold cross validation according to the four classifiers (LDA, KNN, DT, and LSVM).
Figure 7. ROC curves for 10-fold cross validation according to the four classifiers (LDA, KNN, DT, and LSVM).
Sensors 22 06213 g007
Figure 8. Results of the distributions for the four classifiers for the permutation test (p < 0.0001). (A) LDA. (B) KNN. (C) DT. (D) LSVM.
Figure 8. Results of the distributions for the four classifiers for the permutation test (p < 0.0001). (A) LDA. (B) KNN. (C) DT. (D) LSVM.
Sensors 22 06213 g008
Figure 9. Real-time system for assessing MS using cardiac activity. (A) User monitor cam. (B) VR scene. (C) ECG raw signals and detecting the R-peaks. (D) HRV by FFT. (E) Results of the cardiac time domain indices (i.e., SDNN and pNN50). (F) Results of the cardiac frequency domain indices (i.e., ln HF, ln VLF, and ln VLF/ln HF ratio). (G) Results of the nine zone in autonomic balance by the normalized HRV. (H) Sliding bar in each time log. (I) The binary decision for MS.
Figure 9. Real-time system for assessing MS using cardiac activity. (A) User monitor cam. (B) VR scene. (C) ECG raw signals and detecting the R-peaks. (D) HRV by FFT. (E) Results of the cardiac time domain indices (i.e., SDNN and pNN50). (F) Results of the cardiac frequency domain indices (i.e., ln HF, ln VLF, and ln VLF/ln HF ratio). (G) Results of the nine zone in autonomic balance by the normalized HRV. (H) Sliding bar in each time log. (I) The binary decision for MS.
Sensors 22 06213 g009
Figure 10. Flowchart for the proposed method of classifying motion sickness state (two-class).
Figure 10. Flowchart for the proposed method of classifying motion sickness state (two-class).
Sensors 22 06213 g010
Table 1. Summary of MS measurement literature.
Table 1. Summary of MS measurement literature.
MeasurementPlatformContentParticipantsClassification PerformancePaper
EEGalpha (8–13 Hz) and theta (4–7 Hz) bands360-degree VR-based dynamic 3D environmentdriving simulation7 subjects95%
(5-fold cross-validation)
[24]
alpha (8–12 Hz) band360-degree projectiondriving simulation6 subjects86.92%
(leave-one-out cross-validation)
[25]
delta (0.1–3 Hz), theta (4–7 Hz), alpha (8–13 Hz), beta (13–20 Hz), and gamma (21–50 Hz) bands360-degree projectiondriving simulation6 subjects80.7%
(cross-validation)
72.1%
(test data set)
[26]
alpha and beta bandsHMDmirror edge game9 subjects83.8%
(cross-validation)
[27]
theta (4–8 Hz), beta (12–30 Hz), and alpha (8–12 Hz) bandsHMDmirror edge game9 subjects88.9%
(3-fold cross-validation)
[28]
theta (4–8 Hz), alpha (8–12 Hz), low-beta (12–16 Hz), high-beta
(16–25 Hz), and gamma (25–45 Hz) bands
360-degree videoVR video11 subjects99.12%
(5-fold cross-validation)
[29]
delta, theta, alpha, and beta bands (1–30 Hz)HMDVR scene of road18 subjects79.25%
(10-fold cross-validation)
[30]
parallel-feature extraction and feature attention modulesVR vehicle-driving simulatordriving simulation8 subjects96.7%
(leave-one-out cross-validation)
[31]
HEPfirst and second componentsHMDNo Limits 2 Roller Coaster Simulation48 subjects (train: 28 and test: 20)96.4%
(10-fold cross-validation)
87.5%
(test data set, 20 subjects)
[2]
Visionpupil size changeHMDultimate booster experience47 subjects (train: 24 and test: 23)90%
(10-fold cross-validation)
80%
(test data set, 23 subjects)
[16]
Multi-modalEEG, ECG, RSP, EGG, and postural sway HMDvirtual space station environment20 subjects95%
(10-fold cross-validation)
[32]
EEG, center of pressure, head and waist motion trajectories projection screenvisual streaming and car driving video20 subjects91.1%
(10-fold cross-validation)
[33]
ECG and RSPHMDroller coaster video20 subjects96.48%
(cross-validation)
[34]
EEG, EMG, and ECGBioVRSea (VR goggles)rough sea scenario28 subjects74.7%
(10-fold cross-validation)
[35]
stomach activity, EOG, and RSPHMDvirtual environment game20 subjects77.8%
(cross-validation)
[36]
ECG, EOG, RSP, and EDAHMDVR contents66 subjects82%
(cross-validation)
[37]
Table 2. Performance of the different types of classifiers for the 2D and VR viewing conditions.
Table 2. Performance of the different types of classifiers for the 2D and VR viewing conditions.
AccuracyRecallPrecisionF-1 ScoreAUC
LDA85.789.383.386.20.94
KNN87.592.983.988.20.93
DT87.578.695.786.30.83
LSVM91.196.487.191.50.96
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Park, S.; Ha, J.; Kim, L. Effect of Visually Induced Motion Sickness from Head-Mounted Display on Cardiac Activity. Sensors 2022, 22, 6213. https://doi.org/10.3390/s22166213

AMA Style

Park S, Ha J, Kim L. Effect of Visually Induced Motion Sickness from Head-Mounted Display on Cardiac Activity. Sensors. 2022; 22(16):6213. https://doi.org/10.3390/s22166213

Chicago/Turabian Style

Park, Sangin, Jihyeon Ha, and Laehyun Kim. 2022. "Effect of Visually Induced Motion Sickness from Head-Mounted Display on Cardiac Activity" Sensors 22, no. 16: 6213. https://doi.org/10.3390/s22166213

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop