Test–Retest Repeatability of Human Gestures in Manipulation Tasks

Abstract

The importance of performance excellence and operator’s safety is fundamental not only when operators perform repetitive and controlled industrial tasks, but also in case of abrupt gestures due to inattention and unexpected circumstances. Since optical systems work at frequencies that are too low and they are not able to detect gestures as early as possible, combining the use of wearable magneto-inertial measurement units (MIMUs) with the adoption of deep learning techniques can be useful to instruct the machine about human motion. To improve the initial training phase of neural networks for high classification performance, gesture repeatability over time has to be verified. Since the test–retest approach has been poorly applied based on MIMUs signals in a context of human–machine interaction, the aim of this work was to evaluate the repeatability of pick-and-place gestures composed of both normal and abrupt movements. Overall, results demonstrated an excellent test–retest repeatability for normal movements and a fair-to-good test–retest repeatability for abrupt movements. In addition, results suggested important information about the application of deep learning to identify the types of movements: the test showed how to improve reinforcement learning for the identification of onset gestures, whereas the retest allowed for defining the time necessary to retrain the network.

1. Introduction

The concept of collaborative robotics, introduced by Industry 4.0 and reinforced by Industry 5.0, involves different levels of human–machine interaction. The most demanding case in terms of safety occurs if humans and robots simultaneously share the same workspace [1]. Two different approaches can be exploited to guarantee safety, the power and force limiting (PFL) and the speed and separation monitoring (SSM). PFL combines methods of passive safety design with an energy control of mobile robotic components, limiting force, torques and velocity. On the contrary, SSM imposes a protection distance between the robot and the operator, without specifying any limitations for the robotic system. In detail, when the operator is inside the shared workspace, SSM approach implies an online control of the robot to generate alternative paths without collisions. In this case, systems for the human motion tracking must be adopted to monitor the minimum distance. Moreover, algorithms for the collision avoidance have to be implemented to update the robot trajectory in real-time.

Many research works [2,3,4,5] have already exploited different technologies to track human motion and recognize human gestures in case of repetitive and controlled industrial tasks such as assembly [6] and pick-and-place [2]. However, the operators can perform abrupt gestures different from normal movements due to inattention and unexpected circumstances not directly related to the job task. When these situations occur, it is essential to detect abnormal gestures at their onset, avoiding collisions between the human and the machine while maintaining a high efficiency of execution [7]. Some research works have already studied the characteristics of abrupt movements, the methods for identifying their occurrence and their effects on the performance of task execution [8,9,10,11]. However, attention devoted to optimizing the detection of abrupt movements as early as possible remains scant. Optical systems are not adequate for this purpose because they work at too low frequencies and they are not able to detect abrupt gestures as early as possible. The problem of frequency can be solved by adopting wearable magneto-inertial measurement units (MIMUs), which collect data from the triaxial accelerometer, gyroscope and magnetometer embedded in each sensor [12]. In addition, MIMUs also overcome other limits typical of optical systems such as occlusion, encumbrance, a limited range of acquisition volume, heavy weight and high cost. In contrast, the sensor fusion process performed for the estimation of MIMUs’ orientation can produce problems related to the drift phenomenon and to ferromagnetic disturbances affecting magnetometer readings. Instead, the problem of the early detection of abrupt movements typical of optical systems can be approached through deep learning techniques, which are useful to extract and learn features directly from raw data and hence to instruct the machine about human motion [13,14,15]. To guarantee a high classification performance, deep learning neural networks need an initial training phase. Considering this scenario, repeatability of the gesture over time has to be verified to understand whether and after how long it is necessary to retrain the network.

Therefore, a consistency evaluation referring to both the degree of correlation and the correspondence between measurements has to be assessed through a test–retest. The estimation of test–retest repeatability based on MIMUs’ signals has been approached in different contexts such as the classification of construction activities in the industrial field [16], the biomechanical 3D investigation of gait [17], or the improvement of sport training load quantification [18]. However, since this approach has been poorly applied in the context of human–machine interaction, the aim of this work was to evaluate the test–retest repeatability in a typical industrial gesture. In detail, twenty-six participants performed a pick-and-place task composed of normal and regularly paced movements interspersed with abrupt movements caused by randomly generated visual or acoustic alarms. Since the present work was thought for the industrial context and specifically for identifying suitable training and testing modalities of deep learning techniques, the time between test and retest sessions (between 20 and 65 days) was higher than the one typical of other literature studies [18,19]. Results represent the first step for the optimization of a procedure enabling the real-time recognition of human activities and hence the improvement of human–machine interaction in terms of safety and effectiveness when abrupt movements occur.

2. Materials and Methods

2.1. Participants, Experimental Set-Up and Protocol

Twenty-six healthy participants (fourteen males and twelve females) with no musculoskeletal or neurological diseases were recruited for the experiment. Twenty-three subjects were right-handed; three subjects were left-handed. Their main anthropometric data (mean ± standard deviation) are reported in the following: age = 27.7 ± 2.3 years, BMI = 22.2 ± 2.0 kg/m², forearm length = 0.29 ± 0.01 m, and upper arm length = 0.34 ± 0.01 m. The study was conducted in accordance with the Declaration of Helsinki and informed consent was obtained from all involved subjects.

Two wireless MIMUs (Opal^TM APDM, Portland, OR, USA) containing tri-axial accelerometer (range ± 200 g), gyroscope (range ± 2000 deg/s) and magnetometer (range ± 8 Gauss) were exploited during the test. As shown in Figure 1a, these units were positioned on participants’ right (RFA) and left (LFA) forearms, aligning their x-axes with the longitudinal axes of the corresponding human segments. The communication between MIMUs and a PC was guaranteed via Bluetooth. The proprietary software Motion Studio^TM V2R, Version 3 (APDM, Portland, OR, USA) was used to acquire data at 200 Hz.

A typical industrial task of pick-and-place was realized through the workstation illustrated in Figure 1b. It was composed of a table, a chair, a box containing 30 golf balls, and four stations: S0, S1, S2 and S3. Due to the heterogeneity of the tested group in terms of anthropometric characteristics, it was necessary to adapt the experimental set-up to each participant. To this purpose, a board with thirty holes (diameter of 6 cm) was realized to choose the more appropriate distances for S0, S1 and S2 stations. Moreover, a second board with one hole (diameter of 6 cm) corresponding to the S3 station was added to the set-up at a fixed height of 30 cm from the table and an adjustable distance from participants.

The task was the same proposed in a previous pilot study [11]. In detail, it was composed of 30 pick-and-place gestures identified as normal movements. Each of these gestures consisted of picking a ball at a time from the box and placing it into a specific hole corresponding to one of the four stations. Participants were asked to place the balls in the holes following a lighting sequence of green LEDs positioned near each station. In addition, during the task, sudden visual alarms or acoustic alarms were randomly produced by lighting red LEDs near each station or switching on a sound buzzer, respectively.

In both cases, participants were asked to perform an abrupt movement as fast as possible, placing the ball inside the hole corresponding to the activated red LED or vertically extending the arm in case of the sound buzzer [11]. An Arduino Nano microcontroller (Arduino, Italy) with the following characteristics was exploited to generate both the visual and acoustic alarms: processor = ATmega328, clock speed = 16 MHz, and operating voltage = 5 V. The periodic lighting of the green LEDs (every 3 s = 20 bpm) and the random lighting of the red LEDs or of the buzzer were controlled by a code written through an integrated development environment. The Arduino microcontroller was also used to send a voltage trigger of 5 V to Opal sensors for systems synchronization.

The experimental protocol involved two different sessions, the test and the retest. Both sessions were performed under the same conditions by all subjects with an intercurrent period between 20 and 65 days (42.2 ± 11.2 days) depending on the subject. In detail, two main groups can be identified based on a threshold of 45 days. Fourteen participants (four males and ten females) performed the retest after less than 45 days from the test: age = 27.8 ± 2.3 years, BMI = 21.9 ± 2.3 kg/m², forearm length = 0.28 ± 0.01 m, and upper arm length = 0.33 ± 0.01 m. Twelve participants (ten males and two females) performed the retest after at least 45 days from the test: age = 27.6 ± 2.5 years, BMI = 22.5 ± 1.5 kg/m², forearm length = 0.30 ± 0.01 m, and upper arm length = 0.35 ± 0.02 m. In both sessions, each participant repeated the pick-and-place task in three different modalities: (i) with the right hand and with the trunk frontal with respect to the table (rFR); (ii) with the left hand and with the trunk frontal with respect to the table (lFR); (iii) with the left hand and with the trunk lateral with respect to the table (lLA). Inside each repetition, four random alarms were emitted: two visuals and two acoustics.

2.2. Data Analysis

Data analysis was conducted with custom Matlab^® (MathWorks, Natick, MA, USA) routines. Raw data of linear accelerations were processed to obtain the accelerations’ magnitude after removing the gravitational acceleration. Single gestures were extracted from the entire acquisition by means of windows with a length of three seconds, which was imposed by the regular pace of the green LEDs at 20 bpm. In addition, gestures were distinguished between abrupt and normal movements based on the occurrence of the visual and acoustic alarms. Subsequently, values of acceleration root mean square (RMS) were estimated for each signal considering all the windows within the same movement type. Accordingly, the analysis differentiated movements into three groups: normal movements, abrupt movements caused by a visual alarm and abrupt movements generated by the acoustic buzzer. Since the application of the Shapiro–Wilk test (2-tails, significance level: α = 0.05) demonstrated a non-normal distribution of data, a non-parametric statistical analysis was performed on acceleration RMS values with the following tests:

• Mann–Whitney U test (2-tails, significance level: α = 0.05) to verify the eventual presence of differences between males and females;
• Mann–Whitney U test (2-tails, significance level: α = 0.05) to verify the eventual presence of differences among the three test modalities (rFR, lFR, lLA);
• Wilcoxon test (2-tails, significance level: α = 0.05) to investigate the eventual presence of statistical differences among normal movements; visual, abrupt movements; and acoustic, abrupt movements, for each modality.

As suggested in the literature [20], the test–retest modality can be applied to study the repeatability of the proposed task over time. Test–retest repeatability was evaluated using SPSS^® V22.0 (IBM, Armonk, NY, USA) statistical package. Accordingly, in this analysis, the acceleration of RMS values of test and retest were used for the estimation of repeatability. A common measure of repeatability is represented by the intraclass correlation coefficient (ICC), which is calculated as the ratio between the true variance and the total variance of the variable of interest [21]. This dimensionless index assumes values inside the range [−1, 1], highlighting a higher repeatability the closer it reaches 1. In general, two slightly different models of ICC are adopted as measures of relative repeatability of a single measurement [22]. The ICC (3,1) does not consider the effect of any systematic bias as part of the measurement error. On the contrary, the ICC (1,1) assumes all within-subject variability to be a part of the measurement error. These two models produce the same result if no systematic error is present [19]. Another common measure of repeatability is represented by the coefficient of variation (CV), which is the percentage ratio between the standard deviation and the mean of a value distribution [23]. The repeatability improves the lower the CV.

In this work, the test–retest repeatability was evaluated in four conditions:

• Considering all participants (all);
• Excluding the outliers, automatically identified as participants with acceleration RMS values exceeding 1.5 times the interquartile range above the 75th quartile or below the 25th quartile (no_o);
• Considering only the participants who performed the retest after less than 45 days from the test (u_45);
• Considering only the participants who performed the retest after at least 45 days from the test (o_45).

Hence, for each of these four conditions, the test–retest repeatability was assessed through ICC (1,1), ICC (3,1) and CV estimated for each group of movements (normal; abrupt, visual alarm; abrupt, acoustic alarm) and for each modality (rFR, lFR, lLA).

3. Results

Since the acceleration RMS values of males and females did not produce statistically significant differences for both test and retest of all modalities (p-value > 0.05), the analysis was conducted without any gender distinctions. On the contrary, the three modalities (rFR, lFR, lLA) showed statistically significant differences (p-value ≤ 0.05). Hence, the following analyses continued keeping the three modes separated. Figure 2 represents the comparison of averaged RMS values for the three modalities separately for the three types of movements. Single asterisks are related to 0.01 < p-values ≤ 0.05, whereas double asterisks represent p-values ≤ 0.01.

Statistically significant differences among the three types of movements (normal; visual, abrupt; acoustic, abrupt) were found for both test and retest of the three modalities (p-value ≪ 0.001). Accordingly, the following repeatability analysis was conducted considering the three types of movements separately. Figure 3 shows RMS values of forearm acceleration magnitude recorded during the rFR modality, as an example. The three groups of movements (normal; abrupt, visual alarm; abrupt, acoustic alarm) are compared. The scatterplot is with test values on the x-axis vs. retest values on the y-axis. Each point is related to one participant.

Figure 4 represents the outliers identification based on the interquartile range. The upper and lower values of the colored bands correspond to 1.5 times the 75th quartile and 1.5 times the 25th quartile, respectively. Figure 4 is composed of nine panels related to the combination of the three types of movements (rows) and the three modalities (columns). Each panel shows the scatterplot of acceleration RMS obtained with test values on the x-axis and the difference between test and retest values on the y-axis. Each participant is represented by a single point. Outliers are the elements not included in the colored band.

Test–retest repeatability based on the evaluation of ICC and CV is shown in

Table 1. ICC (3,1), ICC (1,1) and CV results of normal movements.

		ICC (3,1)	Lower Limit 95% CI for ICC (3,1)	Upper Limit 95% CI for ICC (3,1)	ICC (1,1)	Lower Limit 95% CI for ICC (1,1)	Upper Limit 95% CI for ICC (1,1)	CV (%)
rFR	all	0.37	−0.02	0.66	0.34	−0.05	0.63	11.31
	no_o	0.37	−0.02	0.66	0.34	−0.05	0.63	11.31
	u45	0.81	0.51	0.94	0.82	0.55	0.94	5.64
	o45	−0.20	−0.68	0.40	−0.27	−0.71	0.32	17.92
lFR	all	0.52	0.17	0.75	0.52	0.18	0.75	9.85
	no_o	0.64	0.33	0.82	0.65	0.35	0.83	8.60
	u45	0.77	0.43	0.92	0.78	0.47	0.93	6.43
	o45	0.09	−0.49	0.61	0.11	−0.46	0.62	13.83
lLA	all	0.63	0.33	0.82	0.64	0.34	0.82	8.54
	no_o	0.72	0.45	0.87	0.72	0.47	0.87	7.54
	u45	0.75	0.38	0.91	0.75	0.39	0.91	6.50
	o45	0.44	−0.15	0.80	0.47	−0.09	0.81	10.92

,

Table 2. ICC and CV results of abrupt movements generated by the visual alarm.

		ICC (3,1)	Lower Limit 95% CI for ICC (3,1)	Upper Limit 95% CI for ICC (3,1)	ICC (1,1)	Lower Limit 95% CI for ICC (1,1)	Upper Limit 95% CI for ICC (1,1)	CV (%)
rFR	all	0.34	−0.05	0.64	0.36	−0.02	0.65	19.08
	no_o	0.34	−0.05	0.64	0.36	−0.02	0.65	19.08
	u45	0.53	0.03	0.82	0.54	0.06	0.82	15.00
	o45	0.14	−0.45	0.64	0.18	−0.40	0.66	23.33
lFR	all	0.45	0.08	0.71	0.46	0.10	0.72	16.31
	no_o	0.51	0.16	0.75	0.52	0.18	0.76	15.44
	u45	0.62	0.15	0.86	0.64	0.20	0.87	12.86
	o45	0.11	−0.48	0.62	0.15	−0.42	0.64	20.33
lLA	all	0.46	0.09	0.71	0.43	0.06	0.70	16.04
	no_o	0.71	0.43	0.87	0.68	0.39	0.85	12.22
	u45	0.25	−0.30	0.68	0.27	−0.27	0.68	17.71
	o45	0.77	0.37	0.93	0.69	0.25	0.90	14.08

and

Table 3. ICC and CV results of abrupt movements generated by the acoustic alarm.

		ICC (3,1)	Lower Limit 95% CI for ICC (3,1)	Upper Limit 95% CI for ICC (3,1)	ICC (1,1)	Lower Limit 95% CI for ICC (1,1)	Upper Limit 95% CI for ICC (1,1)	CV (%)
rFR	all	0.57	0.25	0.78	0.58	0.25	0.78	13.42
	no_o	0.52	0.17	0.76	0.53	0.19	0.76	12.96
	u45	0.62	0.16	0.86	0.59	0.12	0.85	15.64
	o45	0.53	−0.03	0.84	0.55	0.02	0.84	10.83
lFR	all	0.50	0.14	0.74	0.51	0.16	0.74	13.65
	no_o	0.78	0.55	0.90	0.78	0.71	0.95	10.34
	u45	0.69	0.27	0.89	0.71	0.32	0.89	13.64
	o45	0.08	−0.50	0.60	0.11	−0.46	0.62	13.67
lLA	all	0.61	0.30	0.81	0.62	0.31	0.81	13.19
	no_o	0.77	0.54	0.89	0.77	0.54	0.89	10.71
	u45	0.68	0.26	0.89	0.70	0.31	0.89	14.14
	o45	0.34	−0.26	0.75	0.30	−0.28	0.73	12.08

. ICC (3,1) and ICC (1,1) results are reported in the tables, including their 95% confidence intervals. For each type of movement, the three trials are investigated considering all the subjects altogether (all), subjects after the removal of outliers (no_o), subjects who performed the retest after less than 45 days from the test (u_45) and subjects who performed the retest after at least 45 days from the test (o_45). Since for both normal and abrutp_visual movements no outliers were identified, repeatability results are coincident when considering all participants (all) and participants without outliers (no_o).

4. Discussion

Since abrupt movements are randomly caused by involuntary reactions, their accelerations are greater than those of normal movements, as shown by the comparison between the green and the other two clusters presented in Figure 3. Moreover, abrupt movements generated by the acoustic alarm (blue cluster) produce greater accelerations than abrupt movements generated by the visual alarm (red cluster). This trend can be explained considering the specific protocol of this study, because the movement required in response of an acoustic alarm was wider than the one associated with a visual alarm. Overall, the distribution of the three groups follows the principal diagonal, suggesting a repeatability between test and retest RMS values (Figure 3).

The definition of a band based on the interquartile range of RMS values allowed the identification of a number of outliers between 0 and 3 for each combination of the three types of movements and the three modalities (Figure 4). As reported in Table 1, Table 2 and Table 3, the removal of outliers (no_o) before the evaluation of test–retest repeatability did not produce a significant improvement with respect to the analysis involving all participants (all). On average, it is possible to note a minimum increase in both ICC (3,1) (normal +0.07, abrupt_visual +0.10, abrupt_acoustic +0.13) and ICC (1,1) (normal +0.07, abrupt_visual +0.10, abrupt_acoustic +0.11). Coherently, CV values exhibit a slight reduction (normal −0.75%, abrupt_visual −1.56%, abrupt_acoustic −2.08%). In industrial scenarios, real-time should be a key feature for performance optimization. Indeed, an online risk assessment is suitable to evaluate the biomechanical load in repetitive efforts [24] or manual material handling [25] to build a warning system for the prevention of musculoskeletal disorders [26] and to improve assembly workstations [27].

Considering the scenarios of human–machine interaction, a real-time exchange of information between the operator and the robot improves both the interaction [28,29] and the robotic control [30,31]. Accordingly, the similarity of repeatability with and without the outliers (all vs. no_o) represents a positive aspect to approach real-time conditions. Indeed, the possibility of avoiding the post-elaboration of data guarantees the reduction in time.

Another important aspect highlighted from the results is the influence of the time occurring between test and retest sessions. Indeed, when this time was < 45 days (u45), the test–retest repeatability was higher with respect to the analysis related to a test–retest time ≥ 45 days (o45). On average, a general strong increase can be noticed in both ICC (3,1) (normal +0.67, abrupt_visual +0.13, abrupt_acoustic +0.35) and ICC (1,1) (normal +0.68, abrupt_visual +0.14, abrupt_acoustic +0.35). Coherently, an evident reduction involved CV values (normal −8.03%, abrupt_visual −4.06%). The only exception is represented by abrupt movements generated by the acoustic alarm with an average increment of CV (+2.27%) from u45 to o45 despite a strong increase in both ICCs.

In industrial scenarios, deep learning techniques can be involved to provide the robot with important information about the human motion in real-time. Consequently, these results suggest updating the network training within around 45 days. Indeed, as reported in the literature [19], repeatability can be considered poor if ICC < 0.4, fair-to-good if 0.4 ≤ ICC < 0.75 and excellent if ICC ≥ 0.75. Observing results related to u45 condition, average values of ICCs are 0.78 for normal movements, 0.48 for abrupt movements generated by the visual alarm and 0.68 for abrupt movements generated by the acoustic alarm. As expected, normal movements are characterized by an excellent repeatability because they are repetitive and predictable. Accordingly, when abrupt gestures occur, the involved higher accelerations cause lesser controlled movements and hence produce lower ICC values. In particular, even if both values are fair-to-good, the lighting of visual alarms occurs in different stations, whereas the lighting of the sound buzzer always results in the same abrupt movement. Consequently, ICC values related to the abrupt_acoustic movements are higher than the ones associated with abrupt_visual movements. In general, these values of ICCs also highlight the suitability of MIMUs to conduct this analysis.

The main limitation of this study is represented by the specific nature of the tasks which the participants are required to perform, despite being one of the most common types of tasks in the industrial field. In this regard, current activities also consist of investigating other typical industrial tasks from other MIMUs positioned on upper arms and trunks. In order to have a wider and more homogeneous sample of subjects, the same experimental campaign should be extended to around 100 participants. Considering the importance of unexpected situations in terms of safety in human–machine interaction, current efforts also aim to exploit deep learning techniques to distinguish between normal and abrupt movements. Moreover, great attention is dedicated to the achievement of real-time processes.

5. Conclusions

The present study aimed at evaluating the test–retest repeatability of a typical industrial pick-and-place gesture composed of normal, regularly paced movements and abrupt movements caused by randomly generated visual or acoustic alarms. Acceleration RMS values obtained from MIMUs on forearms were exploited for conducting this analysis. The proposed study is the first step in optimizing real-time human activity recognition, and hence, in improving human–machine interaction in terms of safety and effectiveness. Overall, results demonstrated an excellent test–retest repeatability for normal movements and a fair-to-good test–retest repeatability for abrupt movements. In detail, the removal of outliers did not produce significant improvements in repeatability, whereas the reduction in time between test and retest sessions increased repeatability indices. Considering the industrial scenarios of human–machine interaction included in this work, the general test–retest repeatability presented important information about how to optimize deep learning techniques for the improvement of abrupt movement detection in real-time.