Is an ADHD Observation-Scale Based on DSM Criteria Able to Predict Performance in a Virtual Reality Continuous Performance Test?

Featured Application

This study showed that only the DSM 5 criteria referring to inattention symptoms were able to significantly predict performance in the variables measured by a Continuous Performance Test based on Virtual Reality.

Abstract

The Diagnosis of Attention Deficit/Hyperactivity Disorder (ADHD) requires an exhaustive and objective assessment in order to design an intervention that is adapted to the peculiarities of the patients. The present study aimed to determine if the most commonly used ADHD observation scale—the Evaluation of Attention Deficit and Hyperactivity (EDAH) scale—is able to predict performance in a Continuous Performance Test based on Virtual Reality (VR-CPT). One-hundred-and-fifty students (76% boys and 24% girls) aged 6–16 (M = 10.35; DT = 2.39) participated in the study. Regression analyses showed that only the EDAH subscale referring to inattention symptoms, was a statistically significant predictor of performance in a VR-CPT. More specifically, this subscale showed 86.5% prediction-accuracy regarding performance in the Omissions variable, 80.4% in the Commissions variable, and 74.5% in the Response-time variable. The EDAH subscales referring to impulsivity and hyperactivity were not statistically significant predictors of any variables in the VR-CPT. Our findings may partially explain why impulsive-hyperactive and the combined presentations of ADHD might be considered as unique and qualitatively different sub-categories of ADHD. These results also highlighted the importance of measuring not only the observable behaviors of ADHD individuals, but also the scores in performance tests that are attained by the patients themselves.

1. Introduction

ADHD is a common, chronic, and impairing neuropsychiatric disorder, with worldwide prevalence rates ranging from 5% to 7% among the school-age population [1]. ADHD is characterized by a persistent behavioral pattern associated with inattention, overactivity (or hyperactivity), and difficulty in controlling impulses, leading to three presentations: the combined presentation, the predominantly inattentive presentation, and the predominantly impulsive-hyperactive presentation (hereafter I/H) [2]. This disorder relates to significant impairments at home (in family adaptation) and at school (low academic performance) [3]. Additionally, among the long-term consequences of having ADHD symptoms, we could indicate a higher probability of being unemployed, drug abuse, or being imprisoned [4].

In this sense, latent deficits in ADHD are manifested through observable symptoms described in the DSM-5 manual [2], which have been included in different observational scales (completed by teachers or parents). For instance, some of the best known and most widely used observational scales are the following: (1) the Evaluation of Attention Deficit and Hyperactivity (EDAH scale) [5]; (2) the Behavior Assessment System for Children (BASC) [6]; (3) the Child Behavior Checklist (CBCL) [7]; and (4) the Conners’ scales [8]. However, the use of these instruments as the sole assessment measure has been harshly criticized because assessment depends on the subjectivity of the observer [9]. This implies an important limitation in the evaluation of ADHD, because terms like “restlessness”, or “being a clueless person” could be interpreted differently depending on the person who evaluates a particular case. For example, when parents complete an observational scale, those having more than one child often evaluate the different items by comparing with their other children.

For this reason, some other widely used tests in the assessment of ADHD are the Continuous Performance Tests (CPT) (based on participants’ performance), aimed at detecting problems such as deficits in monitoring and updating information in working memory, in inhibiting undesired responses or avoiding to pay attention to irrelevant stimuli and shifting attention away between activities [10]. Among these, Conners’ CPT [8], the Children Sustained Attention Task [11], the Integrated Visual and Auditory Test [12], and the Test of Variables of Attention [13] are the most widely used tests in the assessment of ADHD symptoms. These tests are based on the current models about the etiology of ADHD, which state that the dysfunction in executive processes is one important pathway to understanding this disorder [14,15].

CPTs provide different variables, which are associated to the phenotypic behavior of ADHD students [16]. More specifically, a high number of omission errors and the presence of lengthy response times are thought to relate to inattention deficit. On the contrary, a high number of commission errors and higher levels of variability in their patient’s responses might indicate the presence of impulsive/hyperactive symptoms. In this sense, the profile obtained for each participant is useful in the differential diagnosis of ADHD and its clinical presentations [17]. However, CPT are also criticized as having low ecological validity, since ADHD symptoms do not always occur in a controlled environment, which differs considerably from real-life conditions [18,19]. Thus, various authors [20,21,22] consider the inclusion of Virtual Reality in the CPT (VR-CPT) as a solution that would allow a significant increase in ecological validity. VR-CPT offers the possibility of carrying out assessments in more realistic conditions, including distractors present in typical classrooms (i.e., a classmate who speaks to the subject during the execution of the task, a teacher knocking on the door or the sound of an ambulance passing near the window) [23]. This allows clinicians to know in depth how distractors influence attention capacity, as well as what type of distractors interfere significantly in the performance of children and adolescents. Namely, it is possible to measure the influence of the distractors according to the sensory modality in which they are presented. Moreover, data provided by VR-CPTs are more useful in designing an interventional plan than those obtained with a traditional CPTs, which do not provide any information as to the patients´ behavior in daily-life contexts [17].

These findings have been taken into consideration in the current protocols about the assessment of ADHD, which recommend the correct administration of the following diagnosis tools: (1) a structured or semi-structured interview; (2) an observational scale based on DSM criteria; and (3) a CPT, in order to contrast the results and verify the presence of ADHD symptoms [24,25].

Taking all this into consideration, the present study aims at analyzing whether the data collected by the EDAH scale might partially explain the results obtained by a VR-CPT called AULA Nesplora. This objective allows to measure the degree of congruence between what third parties observe and the patient´s own performance will thus be measured, resulting in an important innovation: Although there are some studies that analyze the relationship between performance in a CPT and current and retrospective symptoms in adults and children [16,18], no studies so far analyze the capacity of an observation scale (based on DSM criteria) to predict performance in the variables of VR-CPTs.

2. Materials and Methods

2.1. Participants

The present study made use of a non-probabilistic clinical sample composed of 150 children with ADHD (76% boys and 24% girls) aged 6 to 16 (M = 10.35; SD = 2.93) and with an average IQ of 109.82 (SD = 22.53). Participants have been diagnosed with the combined presentation of ADHD by neuropsychiatrists, according to the Diagnostic and Statistical Manual of Mental Disorders [2].

2.2. Procedure

The study obtained previous approval by the Ethical Committee of the Principality of Asturias (reference: CPMP/ICH/135/95, code: TDAH-Oviedo), and all instructions from the protocol were performed according to institutional guidelines and laws.

Firstly, a member of the research group contacted with local hospitals and clinical services serving children and adolescents diagnosed with ADHD (more particularly, the combined presentation of ADHD). Contact with these services was initially made by phone, and, later, a face-to-face meeting was held with those professionals who agreed to participate in the project [26].

Secondly, different meetings with families/parents were organized in order to explain the main objectives of the present project. Having given previous written consent for the study, the parents completed the observational scale about the Evaluation of Attention Deficit and Hyperactivity Disorder (EDAH scale) [5], which is based on DSM criteria of ADHD symptoms [2]. Then, the children and adolescents performed the Virtual Reality Continuous Performance Test (VR-CPT), called Aula Nesplora CPT. The evaluations were conducted in a laboratory and lasted for 1 h. A member of the research group was always present during the evaluation process, in order to supervise the administration of the tests. Finally, the parents were informed by clinicians about the results obtained in both tools.

2.3. Instruments

Considering the objectives of the present study, the tests used are described below:

The EDAH Scale [5], which was completed by families (the children´s parents). This scale consists of 20 items about symptoms related to Attention Deficit and Hyperactivity/Impulsivity Disorder. It differentiates between ADHD and control groups, as well as between ADHD presentations. The following variables were included in the present study: EDAH-AD (score in the items that measure Attention Deficit), EDAH-I/H (score in Impulsivity/Hyperactivity items), and EDAH-ADHD (the sum of attention deficit plus Impulsivity/Hyperactivity symptoms). The reliability of the instrument, using Cronbach´s Alpha, was 0.74 in the current sample.

AULA Nesplora [23] is a VR-CPT, which evaluates attention, impulsivity, processing speed, and motor activity in children and adolescents aged between 6 and 16. The task is performed in a virtual reality environment, which is shown through Three-Dimensional (3D) glasses equipped with motion sensors and headphones. The virtual environment presented through the glasses is like a standard school classroom. The participant takes the perspective of a student sitting at one of the desks and facing the chalkboard. Head movements (which are related to motor activity) are detected by sensors located in the glasses, since the software updates the field of vision, giving the participant the feeling of actually being in a classroom.

The test consists of three parts, which are gradually explained by a virtual teacher. The first part aims to immerse the participant in a virtual reality environment. More specifically, this task consists of visually locating balloons and popping them. The first part only aims at immersing participants in the virtual reality environment, by visually locating balloons and popping them and, therefore, performance in it is quite irrelevant and the results from this part are not provided by the test. The second task is based on the “x-no” paradigm (traditionally known as “no-go”) in which the participant must press a button when he or she does not see or hear the stimulus “apple”. This task mainly measures attentional levels, so children or adolescents with inattention problems are expected to make a lot of omission errors in this part. Finally, the last task is based on an “x” paradigm (or “go”): Participants are asked to press a button whenever they see or hear the number “seven”. This task aims to measure the inhibitory control capacity, so it is expected that patients with impulsive-hyperactive problems commit a high number of commission errors. Moreover, it is also convenient to highlight that in each part (Go task Vs. No go task) appears different types of distractors (Visual Vs. Auditory distractors) and this offers the possibility of comparing the results from each part in the presence or absence of distractors. This benefit supposes an important innovation in the evaluation of ADHD symptoms because it allows getting a diagnosis with more ecological validity. Moreover, the increase of ecological validity has been shown to be more effective in the diagnosis of ADHD in comparison to other Traditional CPTs, which offer similar variables but without considering the presence of the distractor and with less levels of ecological validity [26].The completion of the test takes approximately 20 min.

To sum up, the variables provided by this test do not differ from those of other CPTs regarding attention deficit and hyperactivity/impulsivity measurements (Omissions, Commissions, Response Time). However, they enhance this information, relating these measurements to sensory modality (visual vs. auditory), presence/absence of distractors, task type (go vs. no-go) and adding a new index called motor activity. Cronbach´s Alpha in this sample was 0.78.

2.4. Data analysis

This study examined the discriminant value of the subscales of EDAH in predicting performance in VR-CPT. The descriptive statistics for the variables under study were analyzed, paying special attention to skewness and kurtosis. Following the criterion of Kline [27], the maximum scores accepted for skewness and kurtosis were limited to a range of 3–10. The results thus allowed us to perform parametric analyses.

In this sense, three regression models were carried out in order to verify the discriminant values of EDAH subscales in predicting the scores in Omissions, Commissions and Response Time provided by a VR-CPT. Percentile scores were used in order to control the effect of age and gender.

SPSS 24 [28] was used in the analysis of data, having p < 0.05 as the criterion for reaching statistical significance.

3. Results

As shown in

Table 1. Descriptive Statistics for VR-CPT variables and EDAH Subscales.

	M	SD	Asymmetry	Kurtosis
Omissions	62.11	25.29	−0.377	−0.924
Commissions	58.03	29.05	−0.229	−1.149
Response Time	49.47	29.10	0.126	−1.169
EDAH. I/H	82.55	16.87	−1.664	3.851
EDAH.AD	82.51	15.38	−1.821	5.468
EDAH.CD	80.40	15.94	−1.584	3.853

Note.M = Mean; SD = Standard Deviation; EDAH.H = Items of EDAH scale referred to Hyperactive symptoms; EDAH.AD = Items of EDAH scale related to Attention Deficit; EDAH. CD = Items about Conduct Disorder symptoms.

and according to the Kline (2011) criteria, it was found that the variables had a normal distribution.

Once the descriptive statistics were analyzed, the three regression models were conducted. The first regression model (

Table 2. Regression models to predict performance in the VR-CPT variable.

Independent variables: EDAH Scale	Dependent Variables: VR-CPT Variables
	Omissions	Commissions	Response Time
EDAH.AD β (t)	0.411 (2.765 **)	0.615 (3.162 **)	0.782 (3.558 **)
EDAH. H β (t)	0.240 (1.254)	−0.049 (−0.208)	−0.451 (0.652)
EDAH.CD β (t)	0.256 (1.256)	0.334 (1.342)	0.334 (1.342)
R2	0.865 ***	0.804 ***	0.745 ***

Note. β = Standardized beta coefficient; t = Student t coefficient; R² = variance explained; EDAH.AD = Items of EDAH scale related to Attention Deficit; EDAH.H = Items of EDAH scale referred to Hyperactive symptoms; EDAH. CD = Items about Conduct Disorder symptoms.; ** p < 0.01;*** p < 0.001.

) was statistically significant predictor of the omissions obtained in the VR-CPT, F(3, 148) = 318.220, p < 0.001. The second regression model was also statically significant for predicting the commissions obtained in the VR-CPT, F(3, 148) = 198.177, p < 0.001. Similarly, the third regression model was significant in the prediction of response time variable from VR-CPT, F(3, 148) = 144.804, p < 0.001.

These results indicated that in the regression model for the prediction of scores in the omission variable obtained in VR-CPT, only the score obtained in the Attention Deficit subscale from EDAH was statistically significant. Likewise, regarding the model for predicting the performance of a commission variable from VR-CPT, only the Attention Deficit subscale from EDAH was, again, a statistically significant variable. The same pattern was repeated in the last regression model (for predicting the Response time variable), because only the Attention Deficit subscale was a significant independent variable.

4. Discussion

The present study supports the utility of the inattention subscale (belonging to EDAH scale) in predicting a patient´s performance in a VR-CPT (more specifically, AULA Nesplora). These results coincide with previous investigations, which suggested a strong relationship between the presence of inattention symptoms and a significantly high number of omission errors and slow response time [20,21,22,23].

Additionally, the results also showed that the remaining subscales from the EDAH scale (EDAH subscale, related to hyperactive symptoms, and EDAH subscale, related to conduct disorder symptoms) did not significantly predict any particular variable from the VR-CPT. These findings are in the line of previous studies [29], which discussed the difficulty of determining what type of symptoms of ADHD are most dominant. The fact that the EDAH subscale referred to hyperactive symptoms did not predict that any variable from VR-CPT could be due to the fact that CPT are solely based on measuring different Executive Functions (EF). In this sense, there are some clinical studies that support the view that EF deficits, although found in many individuals in groups of children and adolescents who suffer ADHD, are not a necessary feature of ADHD and, therefore, the EDAH subscales based on DSM criteria (and, more particularly, the Hyperactive subscale) are not taking them into account [30,31]. Similarly, this study also resulted in another unexpected finding, since inattention symptoms were capable to predict an 80.4% of performance in the commission variable from the VR-CPT. This result could be partially explained by the fact that EF deficits are mainly related to inattentive rather than impulsive-hyperactive symptoms [32]. Considering the present findings, the following question is posed: is ADHD a single diagnostic category or is it better to talk about two different disorders? Children with the inattentive presentation of ADHD frequently show non-specific attention problems, which are associated with deficient sensory processes, poorly focused attention and less accurate information processing. Understandably, these problems mainly lead to learning disabilities [33]. However, children with predominantly impulsive-hyperactive or combined presentations of ADHD do not have general attention problems like those mentioned in the previous case. These subtypes are more associated with memory retrieval problems, disruptive behavior, and peer rejection [15,33].

Similarly, the results also suggested the importance of carrying out an objective assessment of ADHD, not only considering the symptoms of ADHD contained in DSM-5 manual, but also taking into account the patient’s own performance in a CPT, in order to contrast the two different types of measures (symptoms collected by observational scale and variables collected by a CPT). As many protocols recommend [24,25], it is highly relevant to use several assessment tools with the same patient, to ensure the objectivity of the diagnosis process. Moreover, including a CPT based on Virtual Reality increases the ecological validity of the patient´s evaluation and, at the same time, brings out the possibility of analyzing how distractors affect their daily life [20,22].

Therefore, the results obtained in the present study may be useful in guiding clinicians get an objective and reliable assessment of the ADHD symptomatology. However, it is important to highlight some limitations of the study that should be considered in future research lines. In this sense, it would be convenient to include a control group, in order to analyze whether the evidence obtained is maintained. Another important limitation of this study relates to the ADHD sample, as it consists of children and adolescents who have been clinically diagnosed as presenting a combined presentation of ADHD. In this sense, it might also be positive to include the remaining two presentations of ADHD in the ADHD group (that is: the predominantly inattentive presentation and the predominantly impulsive-hyperactive presentation), so as to observe possible differences. Hence, we would have the possibility to compare performance in interesting variables, like motor activity, depending on the ADHD presentation. This would allow us to check whether the inattention presentation presents the lowest level of motor activity and, by contrast, whether the impulsive-hyperactive presentation obtains the expected highest level for this same variable.