The Test–Retest Reproducibility, Standard Error Measurement, and Smallest Real Difference of Dynamic Balance Tests in Chronic Stroke Survivors

Abstract

Background: A dynamic balance assessment of stroke patients is a very important factor in their rehabilitation process. The aim of this study was to investigate the test–retest reproducibility and smallest real differences in the dynamic balance tests of chronic stroke survivors. Methods: This was a methodological study that fifty-three chronic stroke survivors participated in. Dynamic balance tests were performed using the figure of eight walk test, four square step test, and step test. These tests and measurements were performed twice with 1-week intervals and intra-class correlation coefficients (ICC) were used to investigate their test–retest reliability. To determine the size of the measurement error, the standard error of measurement (SEM) and smallest real difference (SRD) were used. The SEM was used to compare the test–retest reproducibility. Results: The test–retest reliability of the figure of eight walk test, four square step test, and step test was confirmed to be high, with an ICC = 0.93~0.99. The SEMs were 2.21 and 2.25 for the figure of eight walk test and four-square step test, respectively. As for the step test (paretic and non-paretic side), the SEMs were 0.68 and 0.70, respectively, which were <10% of the average score. The SRDs of the figure of eight walk test, four square step test, and step test (paretic and non-paretic side) were 6.10, 6.21, and 1.87 and 1.93, respectively. This showed a reliability of <20% of the highest score. The SRDs (%) of the figure of eight walk test, four square step test, and step test (paretic and non-paretic sides) were 28.9%, 21.9%, and 27.32% and 24.43%, respectively. The SEM for all the tests was <30% and the level of measurement error was acceptable. The ICCs of the test–retest reliability of the figure of eight walk test, four square step test, and step test were identified as high, and the test–retest reproducibility was verified by the low values of the SEM and SRD. Conclusions: The figure of eight walk test, four square step test, and step test might be useful tools for clinicians to examine the changes in dynamic balance over time in the rehabilitation of chronic stroke survivors.

1. Introduction

Impaired balance in stroke survivors is one of the main factors that inhibits their functional recovery [1]. Balance recovery is a principal factor in stroke rehabilitation and this implies a functional recovery corresponding to rehabilitative interventions [2,3]. Thus, it is important to select valid and reliable tools that can test the balance of stroke survivors. In addition, these tools need to be easily administered in a clinical setting and their test–retest reproducibility should be sufficiently supported.

Reproducibility is defined as the extent to which a certain value is consistent with the previous value taken, a generic term associated with the concepts of concordance rate and reliability [4]. Test–retest reproducibility is determined by the intra-class correlation coefficient (ICC), which quantifies the relative reliability and is stated in numbers without units [5,6]. Nonetheless, high ICC values do not provide information on the measurement errors during repeated tests. The size of intra-rater disagreement is unknown and the comprehensive characteristics of the reproducibility cannot be reflected [6]. Random measurement errors exist in all tests, however, there are many limitations in analyzing the tests’ results if this cannot be quantified. To augment this, the absolute reliability indices of the standard error measurement (SEM) and smallest real difference (SRD), also called the minimal detectable changes, are used to determine the size of a random measurement error [7,8].

The SEM determines the amount of error of any measurement tool caused by random variation [5]. This reflects the minimum error threshold for individual patients at a 95% confidence level [9,10]. Conversely, the SRD is a standard value by which clinicians or researchers can estimate the smallest actual change that a measurement tool can detect after an intervention. Thus, SEM and SRD are used to detect changes in subjects over time in clinical settings and research [11].

In the clinical setting, various tools have been developed to test the dynamic balance of stroke survivors. The dynamic balance level of stroke patients is one of the factors that determines their functional level and can be a criterion for setting goals for rehabilitation. The figure of eight walk test (F8WT), four square step test (FSST), and step test (ST) have been verified for inter-rater and intra-rater reliability and validity [12,13,14]. These tests have the advantage of being able to quickly and easily examine the dynamic balance of stroke survivors. However, only test–retest reliability has been reported for these tests, while SEM and SRD have not been investigated.

Therefore, the aim of the present study was to investigate the test–retest reproducibility of dynamic balance tests, specifically the F8WT, FSST, and ST, in a larger cohort of chronic stroke survivors, in order to assess the patients with an easier and more convenient evaluation method in their rehabilitation procedures.

2. Materials and Methods

2.1. Participants

Fifty-three chronic stroke survivors, who were recruited through the bulletin board of a rehabilitation hospital, participated in the study. The volunteers were screened using the inclusion criteria: hemiplegic stroke as diagnosed by a neurologist, a Mini-Mental Status Examination (MMSE) score of >24, the ability to walk >10 m with or without assistive devices, no receptive aphasia, and no lower motor neuron diseases or musculoskeletal diseases. The exclusion criteria were as follows: the use of medications that could affect neuromuscular control or provoke drowsiness, severe conditions that require medical care (e.g., uncontrolled blood pressure or angina), a psychological condition, or refusal to be involved in this study. There were 61 subjects who applied for the study, 3 of whom were discharged from the hospital, 2 of whom were rejected, and 4 of whom stopped the evaluation process for personal reasons, resulting in a total of 52 who completed the study process.

2.2. Procedures

This was a methodological study. The basic information about the participants, such as sex, age, disease duration, etiologic type, affected side, and MMSE score, were collected through medical charts or brief interviews. A summary of the general information of these participants is provided in

Table 1. Basic information of the participants.

Basic Information	Mode (%) or Mean (Standard Deviation)
Sex (male/female)	36 (69.2)/16 (30.8)
Age (year)	60.67 (0.73)
Disease duration (months)	28.65 (0.64)
Etiologic type (infarction/hemorrhage)	33 (63.5)/19 (36.5)
Affected side (left/right)	23 (44.2)/29 (55.8)
MMSE (score)	26.06 (0.72)

. The participants were then tested using the F8WT, FSST, and ST and were re-tested 7 days later to examine the concordance rate [11]. When an examiner repeatedly performs dynamic balance tests and assigns scores, bias may occur due to the results of previous evaluations; thus, only five participants were evaluated in one session of the procedure of this study. In addition, contact between the examiner and subject through intervention and examination was prevented from occurring until the next evaluation.

2.3. Tests and Measurements

2.3.1. Figure 8 Walk Test (F8WT)

The F8WT was used to examine the participants’ gaits in both straight and curved paths. It measured the time taken to walk from the middle of two cones spaced 5 feet apart, turn left counterclockwise, and move toward the remaining cone clockwise [15]. The time it took to return to the starting point with two feet from the point of the first step was measured with a stopwatch. The inter-rater, intra-rater, and test–retest reliabilities of the F8WT in the stroke survivors were reported as an ICC = 0.94~0.99 [14].

2.3.2. Four Square Step Test (FSST)

Four cylindrical rods that were 90 cm in length and had diameters of 1 cm were configured in a cross-shaped square on the floor. The participants were instructed to step over the rods as fast as possible into each square, with both feet touching the space and facing forward in the following order from the starting position: 2, 3, 4, 1, 4, 3, 2, and 1 (clockwise and counterclockwise). The time measurement was started when the first step was taken and ended when the last foot touched space 1 [16]. If the participant touched the bar with their feet, deviated due to a loss of balance, could not place two feet in one space, or needed assistance, the test was considered a failure and was performed again. A demonstration, when necessary, was performed prior to the test. For the chronic stroke survivors living in the community, the intra-rater reliability of the FSST was reported as an ICC = 0.83 and the inter-rater reliability had an ICC = 0.99 [13].

2.3.3. Step Test (ST)

A 7.5 cm high footrest was placed in front of the participant, with one lower extremity (more affected side) on the ground, while the opposite lower extremity (less affected side) was raised and lowered repeatedly, as fast as possible, for 15 s. Both the more affected and less affected sides were scored by recording the number of steps taken within the 15 s. The STs’ inter-rater reliability was reported as an ICC = 0.99 and its intra-rater reliability was recorded as an ICC = 0.98~0.99 [12].

2.4. Statistical Analysis

All the data were analyzed for normality using the Shapiro–Wilk test. A frequency analysis and descriptive statistics were used to analyze the basic information of the participants.

For the test–retest reliability of the F8WT, FSST, and ST, the type (1, 1) for intra-rater reliability was used. The ICC was classified as follows: ≤0.25 poor; 0.26–0.49 low; 0.50–0.69 moderate; 0.70–0.89 high; and 0.90–1.00 very high [17]. The SEM was calculated as the standard deviation*√1–ICC to quantify the measurement error. To determine the score for the actual changes (improved or worsened) at a 95% confidence interval, the SRD was calculated as 1.96*SEM*√2 [18]. When the SEM was <10% of all the test–retests [19] and the SRD was <20% of the highest score [20], they were considered acceptable, as smaller values indicate a higher reliability.

The SRD (%) was used to compare the random measurement errors between two measurements. It could also be used to the compare test–retest reproducibility among the tests [21], which was calculated by dividing the mean of all the measurements from both sessions and multiplying it by 100%. The SRD (%) was considered acceptable if the mean value was <30% [22].

To determine whether there was a systematic error in the test–retests, the difference between the means of the two test values was analyzed using the paired t-test. Bland–Altman plots and 95% confidence intervals were used to estimate the pattern of inconsistency in the measurements [17]. The results were considered significant at p < 0.05 and a statistical analysis was performed using SPSS version 23.0 (IBM Corp., Armonk, NY, USA).

3. Results

The test–retest reliability of F8WT, FSST, and ST (more affected and less affected sides) had an ICC = 0.93~0.99 (95% confidence interval), and the test–retest concordance rate was high (

Table 2. Summary statistics of the dynamic balance tool for the participants.

Tests and Measures	Mean (SD) Median (Q1~Q3) Range (Min~Max)		Mean Difference (SD)	ICC (2,1) (95% CI)	SEM	SRD (SRD %)	p-Value
	1st Test	2nd Test
F8WT (s)	21.21 (15.70) 16.76 (11.65~24.85) 6.83~80.99	20.94 (16.10) 15.37 (11.45~24.97) 7.09~90.01	0.27 (2.48)	0.98 (0.97~0.99)	2.21	6.10 (28.9)	0.43
FSST (s)	28.52 (22.17) 20.84 (15.02~35.63) 5.50~130.01	27.97 (23.11) 20.29 (14.87~30.77) 5.12~140.07	0.54 (2.36)	0.99 (0.99~0.99)	2.25	6.21 (21.9)	0.10
ST (score) More affected side	6.61 (2.66) 6.5 (4.25~9) 1~16	6.82 (6.50) 6.5 (5~8.75) 0~16	−0.2 (0.91)	0.95 (0.92~0.97)	0.68	1.87 (27.82)	0.12
ST (score) Less affected side	7.78 (2.66) 8 (6~9) 2~16	8.01 (2.77) 8.5 (6~9) 3~15	−0.23 (0.98)	0.93 (0.88~0.96)	0.70	1.93 (24.43)	0.09

Abbreviations: SD, standard deviation; Q1: first quartile; Q3: third quartile; F8WT: Figure-8 walk test; FSST: Four square step test; ST, Step test; ICC, Intra-class correlation coefficient; CI, confidence interval; and SEM, standard error.

). The SEMs were 2.21 and 2.25 for the F8WT and FSST, respectively. For the ST (more affected and less affected sides), the SEMs were 0.68 and 0.70, respectively, which were <10% of the average score. The SRDs were 6.10 and 6.21 for the F8WT and FSST, respectively. Regarding the ST (more affected and less affected sides), the SRDs were 1.87 and 1.93, respectively, which were <20% of the highest score. The SRDs (%) of the F8WT and FSST were 28.9% and 21.9%, respectively, and those of the more affected side and less affected side were 27.32% and 24.43%, respectively. The SRD (%) for all tests was <30% and the level of measurement error was acceptable. In addition, there was no significant difference in the mean test–retest scores of all the tests (p > 0.05). In the Bland and Altman scatter plot, the limit of agreement range was −4.59 to 5.13 for the F8WT (Figure 1), −4.08 to 5.16 for the FSST (Figure 2), and −1.99 to 1.57 and −2.15 to 1.69 for the ST (more affected and less affected sides), respectively. The variation in the mean difference (range of error) between all the tests was within the acceptable range.

4. Discussion

The present study investigated the test–retest reproducibility of dynamic balance tests on chronic stroke survivors. In this study, the test–retest reliability of the F8WT had an ICC = 0.98, similar to the result of an ICC = 0.97 reported in a previous study [14]. Although their test–retest reliabilities were not reported in other studies, the ICCs of the FSST and ST (more affected and less affected sides) in the present study were 0.99, and 0.95 and 0.93, respectively. All three tests for dynamic balance assessed in this study were found to have an excellent test–retest reproducibility.

Since ICC does not indicate the level of measurement error, it cannot interpret the influence of diversity between subjects [23]. In addition, there are limitations to using ICC to explain reproducibility, as there is the issue of the learning effect when ICC is measured repeatedly [6]. SEM and SRD estimate the size of the random measurement error caused by chance variation in the score change that occurs when the same test is repeatedly measured. Additionally, they are used to determine if the measured value is systematically kept constant within a 95% confidence level [5,9]. Furthermore, SEM is a meaningful statistical value that can estimate changes in the test scores in cohort studies [8].

The SEMs and SRDs of the F8WT, FSST, and ST have not been reported in previous studies. In the present study, the SEMs of the F8WT, FSST, and ST (more affected and less affected sides) were 2.21, 2.25, and 0.68 and 0.70, respectively, which were all <10% [19]. Thus, the SEM values of the dynamic balance tests in the present study were determined to be within reasonable levels. This indicates that the F8WT, FSST, and ST are sensitive assessment tools that can determine the actual change in the dynamic balance of chronic stroke survivors.

SRD is an important index for clinicians to determine the degree to which subjects can change (improvement or worsening), with a 95% confidence interval [7,10]. The results of this study showed that the SRDs were 6.10, 6.21, and 1.87 and 1.93 for the F8WT, FSST, and ST (more affected and less affected sides), respectively. Acceptable results were found to be <20% of the highest score [20]. SRD was used to determine whether the subject change was statistically significant [10]. According to the results of this study, the SRD was not affected by measurement error and chance variation, and the actual subjects’ F8WTs and FSSTs could be significantly improved by approximately 6 s or less, while the ST (more affected and less affected sides) could be improved by approximately two points or more. In addition, since there was no significant distinction in the mean difference between the F8WT, FSST, and ST, no systematic error was found between the test–retests. Moreover, the SRDs (%) were 28.9% and 21.9% for the F8WT and FSST, respectively, and for the ST (more affected and less affected sides), they were 27.32% and 24.43%, respectively. The level of measurement error was acceptable [22].

In this study, the F8WT, FSST, and ST were visualized using the Bland–Altman method to estimate the magnitude of the inter-rater agreement [17]. In the limits of the agreement range, the F8WT was 9.72 (−4.59~5.13), FSST was 9.24 (−4.08~5.16), and ST (more affected and less affected sides) was 3.56 (−1.99~1.57) and 3.84 (−2.15~1.69), respectively. The variation (range of error) in the mean difference between the tests was acceptable. This implies that the difference between the mean value and means between examiners were not related, and there was no systematic error as it existed in the 95% confidence interval [24]. The F8WT, FSST, and ST were found to have a high reproducibility for the dynamic balance measurements of chronic stroke survivors. Reliable tests require high ICC values and low SEM and SRD values [25], which the tests in this study possessed, indicating a sufficient reproducibility.

Chronic stroke survivors experience various levels of limitations in independent and functional activities due to damage to the central nervous system. Interventions related to rehabilitation focus on improving the patient’s balance ability for independent activities; it is important to properly set short-term and long-term goals, because rehabilitation is rarely short-lived. Therefore, it is imperative to reset the contents and goals of interventions based on a clear evaluation of the patient’s balance ability during their rehabilitation. The dynamic balance tests used in this study can be considered to be a useful evaluation tool that can be easily accessed in clinical practice because there are no restrictions on time and space and no special equipment is required.

This study has several limitations. First, although the dynamic balance tests evaluated in this study are not evaluation tools for stroke patients only, only chronic stroke survivors were targeted to minimize the potential recovery effects in the test–retest agreement rate. Second, relatively mild or moderate stroke survivors who had the ability to walk >10 m with or without assistive devices were included. However, since this study investigated the reproducibility of the dynamic balance test for stroke patients, only stroke patients who could walk independently were included. Third, since the dynamic balance tests used in this study gave points for each evaluation item, a low confidential interval may be issued in point estimates of agreement. In a future study, an investigation into the reproducibility of F8WT, FSST, and ST will be needed for acute (within 6 months) stroke survivors.

5. Conclusions

In conclusion, it was found that the F8WT, FSST, and ST had high test–retest agreement rates, no systematic bias, and low SEMs and SRDs for detecting actual changes. Therefore, the F8WT, FSST, and ST can be useful tools for clinicians and researchers to observe or assess the changes in the dynamic balance of chronic stroke survivors in rehabilitation procedures.