A Hand Motor Skills Rehabilitation for the Injured Implemented on a Social Robot

Featured Application

The system described in this work is intended to be applied to hand motor skill rehabilitation and recovery.

Abstract

In this work, we introduce HaReS, a hand rehabilitation system. Our proposal integrates a series of exercises, jointly developed with a foundation for those with motor and cognitive injuries, that are aimed at improving the skills of patients and the adherence to the rehabilitation plan. Our system takes advantage of a low-cost hand-tracking device to provide a quantitative analysis of the performance of the patient. It also integrates a low-cost surface electromyography (sEMG) sensor in order to provide insight about which muscles are being activated while completing the exercises. It is also modular and can be deployed on a social robot. We tested our proposal in two different facilities for rehabilitation with high success. The therapists and patients felt more motivation while using HaReS, which improved the adherence to the rehabilitation plan. In addition, the therapists were able to provide services to more patients than when they used their traditional methodology.

1. Introduction

Rehabilitation of the brain- and motor-injured is an important task. These kinds of challenged individuals have reduced motion in their muscles as a result of an accident, acquired diseases, or birth conditions. However, they can improve their motor skills by following a rehabilitation plan. Nonetheless, foundations such as ADACEA, which is a Spanish-based organization for the brain- and motor-injured, are usually under founded and short staffed, and sometimes they cannot provide the required rehabilitation services. In addition, there is a lack of an established protocol to quantitively evaluate a patient’s performance on rehabilitation exercises.

So far, the therapists of ADACEA carry out exercises with patients one by one, and do not have any quantitative method to evaluate the patients’ performance. The patients have the rehabilitation sessions scheduled, but since these kinds of foundations are usually short staffed, as mentioned before, the patients do not execute the amount of rehabilitation sessions they should. In addition, the evaluation is purely qualitative. In this sense, the therapists take notes about the performance and significant events such as pain or unusually poor execution of the exercises based on the therapists’ experience. These eventualities negatively impact the way they follow the rehabilitation process and what they take away from it.

Having this need in mind, we developed HaReS, a hand rehabilitation system, which is a system for motor rehabilitation. By using HaReS, patients can perform rehabilitation exercises on their own, without requiring the presence of a therapist. Furthermore, HaReS automatically grades the performance of users, so therapists have a quantitative measurement of patients’ performance. This way, the therapists can provide an optimized and enhanced service to more patients.

HaReS is composed of a surface electromyography (sEMG) sensor and a hand-tracking device that allows to record and provide quantitative measurements and feedback of the rehabilitation sessions that a patient is performing. The exercises are set up by a therapist for each user. In addition, HaReS can take advantage of a social robot to show the exercises and to interact with patients. Thus, enhancing their adherence and engaging them. The system is modular, so it is able to work without the sensors or the robot. Apart from the robot, the rest of the system is fairly low cost. It is important to note that it is executed on a desktop or laptop computer, so all components are connected to it. The hand-tracking device, the sEMG sensor, and the social robot are all connected to the computer, which is in charge of running HaReS. In this sense, the robot is not executing HaReS, nor are the sensors connected to it. The role of the robot within the HaReS framework is to display the system in its built-in screen and interact with the patient using gestures, lights, and its speech capabilities.

The main contributions of this paper are the following:

• A hand rehabilitation system that integrates a series of exercises for the motor- and brain-injured;
• Use of low-cost sensors for hand tracking and muscle signal monitoring;
• An automatic and quantitative evaluation of the exercises to be analyzed by the therapists;
• Optional use of a social robot to improve patient adherence to rehabilitation

The rest of the paper is structured as follows. First, some related work to this matter are presented in Section 2. Then, Section 3 describes the proposal in detail. Next, the evaluation we performed to validate the system is given in Section 4. Finally, Section 5 states the conclusion of this work and future research directions.

2. Related Works

As stated in [1], conventional exercise programs follow the Bobath [2,3] or Brunnstrom [4] concepts, as well as the proprioceptive neuromuscular facilitation (PNF) principles [5]. On the one hand, Bobath’s method emphasizes the reduction of enhanced muscle tone before facilitating active movements by means of cutaneous and proprioceptive stimuli applied to the region of the target muscles. On the other hand, Brunnstrom’s approach and PNF use maximal innervation of intact or less-paretic muscle groups to produce irradiation effects in more severely paretic synergistic muscle groups. Several studies have been carried out with some of these techniques [6,7,8,9,10]. However, none of these studies presented a suitable control group, so the benefit was difficult to extrapolate from spontaneous recovery. Additionally, they presented a lack of exercise strategies for the hand muscles.

In [1], the authors investigated the effect of a specific training program focused on the performance of the basic movement parameters of the hand. They paid special attention to the identical repetition of movements, whose long-term benefits have been demonstrated in other studies with animals [11,12]. The patient scores for the rehabilitation exercises were calculated according to the Rivermead Motor Assessment [13], a widely used technique to measure motor ability in stroke patients, and 24 out of 27 patients showed an important improvement according to this metric. In this study, they introduced control groups to ensure the impact of the specific repetitive motor training in motion improvement.

In order to evaluate muscles’ progress during the rehabilitation process, some rehabilitation exercises take advantage of surface electromyography (sEMG) electrodes [14,15,16,17]. These wearable electrodes enable the professional in charge to analyze the muscular activities and muscular strength of a patient during each exercise [18,19]. In some rehabilitation methods, with aim of having an interactive system and motivating patients with visual and audio feedback, during the process, sEMG analysis is incorporated into a 3D game [20,21], virtual reality (VR) [22], and augmented reality (AR) [23]. In addition, there are algorithms that take sEMG signals as the input in order to compute [24] different low-level movements of the muscles. Specifically, this paper introduces a method to perform movement identification of the upper limb: Abduction, adduction, flexion, extension, and abduction followed by arm to the front.

In recent years, researchers have focused on robot-assisted neurorehabilitation in order to improve the performance of rehabilitation exercises. According to [25,26], two main approaches have been used to design hand rehabilitation devices: End-effector and exoskeleton robots.

End-effector-based robots are usually employed to simulate activities of daily living (ADLs), to train the hand and, eventually, the wrist function by only interacting with the distal segments of the fingers, that is, the fingertips or sometimes the middle phalanxes. The rest of the arm is not controlled by the robot, which may result in patients using undesired compensatory strategies, so it is common to provide a weight support in order to reduce the muscle fatigue produced by the distal limb [27,28,29,30,31,32,33]. These types of robots are prepared for hands of different sizes, so they can be easily adjusted for different patients.

On the contrary, an exoskeleton consists of a mechanical structure that is mounted in parallel with the limb of the user in order to provide assistance, so each degree of freedom is locally aligned with the corresponding human joint [16,17,34,35,36,37,38,39,40]. The majority of hand exoskeletons aim to restore grasping function by helping patients open the hand or provide force augmentation to hold objects. They do not limit the natural joint movements of patients, but their design is far more complex and the adaptation process for new users is more difficult.

Several clinical studies have studied the efficacy of these types of assisted rehabilitation strategies and have concluded that they are effective in reducing the motor impairments of stroke victims, and that they improve the ability to perform activities of daily living [25,41].

Most of these rehabilitation exercises require the presence of professional therapists that help and supervise the results of this process. However, the lack of medical resources, the difficulties in visiting these professionals daily, and the challenges to transfer this clinic technology to a home environment make the rehabilitation process much harder for patients when they are at home.

Recent works have focused on at-home rehabilitation exercises in order to provide these tools outside of a clinic environment. In [42], the authors developed a multisensor system for rehabilitation and interaction with people with motor and cognitive disabilities. In the case of [43], PHAROS was proposed, an interactive robot system that recommends and monitors physical exercises at home, designed for the rehabilitation of chronic diseases. More recently, [44] proposed an augmented reality platform to engage and supervise rehabilitation sessions at home using low-cost sensors. This platform also stores a user’s statistics and allows therapists to tailor the exercise programs according to their performance. The results presented in these research works suggest that these platforms improve the results and adhesion to rehabilitation exercises.

However, there is a lack of this type of research specialized in hand motor rehabilitation at home that take into account the particularities and needs of its exercises and evaluations.

Hand pose estimation methods are a really interesting topic, and thus, there are several different state-of-the-art approaches. For instance, there are pure vision-based methods [45,46] that take as input images of hands and compute the hand pose. Nonetheless, these systems lack enough accuracy and are very computationally demanding, requiring a powerful graphics processing unit. There are also algorithms that process electromyography (EMG) signals [47] to predict static hand poses. These kind of methods are also unsuitable because they do not provide fine hand tracking, but rather, a range of fixed poses. Hand tracking can also be achieved by inertial measure unit sensors [48], but it is also limited to a range of specific hand movements. The methods reviewed so far rely on computational models to perform predictions. Nonetheless, there are also hardware-based methods to perform hand tracking. For instance, motion capture systems [49,50] could be used, but they are expensive and intrusive, as they must be worn by the users. After carefully considering the different approaches to perform hand tracking, we decided to include the Leap Motion controller [51] in HaReS. This device is low cost and is not intrusive, as it is placed in the desk and the user does not need to wear it or any specific marker. In addition, its error ranges from $0.2$ to $1.2$ mm [52], reportedly being accurate and robust enough for a range of different applications, including robot manipulation [53] and dataset creation [54].

3. Architecture of HaReS, the Hand Rehabilitation System

As introduced before, HaReS is a system for hand motor rehabilitation. It is composed of pieces of hardware and software. Regarding the hardware, HaReS uses an sEMG sensor in order to track the activation of the muscles during the rehabilitation session. Then, a hand-tracking device is also used to provide the user’s hand poses. The rehabilitation exercises are set up on a software app, which record sthe data of the sessions, including sEMG profiles and hand poses. The records are available for the therapists to review remotely and/or in the future. In addition, some quantitative measurements are given by the system as a potential indicator of the user’s performance.

These functionalities enable a better evaluation of the improvement the user experiences by completing the rehabilitation exercises. In addition, the system is modular, so any of the pieces can be removed at any time. Regardless, it is worth mentioning that the devices we used to create HaReS are fairly low cost. Finally, HaReS can be deployed on a social robot for further improving patients’ experience and motivation. A diagram of the complete architecture of HaReS is shown in Figure 1.

At this point, it is worth noting that the use of a robot is completely optional, as we intended HaReS to be low cost and fully modular. The system can be deployed on a desktop or laptop computer, with no robot involved. We would also like to state that the chosen sensors, such as Leap Motion, are accurate, whist keeping cost at bay. For instance, hand tracking based on cameras is possible [45,46], but lacks enough accuracy and requires much more computation power.

The components of the system are detailed in the following subsections.

3.1. sEMG Sensor

Electromyography (EMG) is a technique used in clinical processes to study the electrical activity produced by skeletal muscles. The signals are captured by a series of electrodes that are placed in the muscle’s belly. Surface EMG (sEMG) is a non-invasive type of EMG, and the electrodes should be positioned on the skin surface over the muscles to record the signals. Although this superficial type of EMG is less accurate than intramuscular EMG (invasive type), it is a better fit for our purpose. sEMG electrodes are user-friendly and they are accurate enough to capture muscle activity during exercise.

The sensor of choice for HaReS esd based on the BITalino project. This device aims to provide a framework for science education, research, and prototyping. It includes several sensors and actuators such as electrocardiogram (ECG), electrodermal activity (EDA), electroencephalogram (EEG) and, of course, electromyogram (EMG). The sEMG sensor of the BITalino is bipolar, which means that it has two electrodes for sensing the electrical activity of the muscle and one as a reference. The measuring range is $\pm 1.64$ mV and the sampling rate is up to 4000 Hz.

As suggested in [55], the optimal sample rate for EMG pattern recognition is between 400 and 500 Hz, so we used 500 Hz as the sampling rate for the data acquisition process that involved the BITalino sensor. To study the finger and wrist movements, the electrodes were placed on the flexor carpi radialis. In addition to the electrodes’ placement, we were able to receive data from contraction of the flexor digitorum, and for the reference, we placed an electrode on the elbow bone.

This sensor is used by HaReS to record the sEMG activity of a desired muscle. Given these data, along with the synchronized motion data captured by a hand-tracking sensor, an experienced therapist could detect if the patient is triggering the correct muscles during the performance of a certain task, thus having more insight about whether the rehabilitation program is improving the patient’s skills. Figure 2 shows some sample hand poses with the corresponding sEMG data.

3.2. Hand-Tracking Sensor

Another important feature of HaReS is the ability to review the user’s hand poses while performing the rehabilitation exercises. There are different approaches to tackle this problem, but we aimed for two important features. First, it had to be low cost so that the system could be implemented by anyone. Next, it also had to be non-invasive so that it would not interfere with the already limited motor skills of the patients.

Thus, the sensor of choice for hand tracking was the Leap Motion controller. This device yields an infrared-based stereo setup, which allows to compute the position and orientation of the bones in a hand, namely, the user’s hand pose. Leap Motion sends data at 115 Hz and is precise enough to be considered ground truth [54]. Some sampless ass provided by this sensor can be seen in Figure 3.

This functionality is used by HaReS to record the user’s hand poses during the exercises. It provides therapists with more data to improve the evaluation of the rehabilitation program and to know whether patients are performing the exercises correctly.

3.3. Social Robot

The social robot within the HaReS framework is used for displaying information and interacting with the patient. The inclusion of this robot is merely intended to improve the motivation of patients and to enhance their experience while using HaReS.

In this regard, the social robot of choice was the Pepper robot. Pepper is an humanoid-like robot that focuses on interaction. It yields LEDs on its face and has arms that enable it to show emotions and engage with user interaction. It also has a touchscreen built-in tablet that allows to display HaReS.

We used the Pepper robot within the HaReS framework to run the software on its tablet, to cheer up and encourage patients, and to provide feedback using its arms, face, and speech capabilities.

3.4. Rehabilitation Exercises

As stated before, HaReS has a range of rehabilitation exercises. These exercises were jointly designed with therapists of ADACEA (Alicante, Spain), which is a foundation for the rehabilitation of acquired brain injury-affected individuals.

So far, for the motor rehabilitation of the hands, they divide the exercises in two categories. The first category focuses on strengthening the muscles on the hand and on low-level and fine motion. This part of the rehabilitation ensures that patients are able to perform the correct movements. The second part focuses on functional skills that ensure patients learn how to correctly interact with everyday objects. The tasks designed by the therapists of ADACEA and implemented in HaReS cover the first part entirely and provide entry-level exercises to cover the functional part. It is important to note that therapists must set up the exercise schedule and the settings for each exercise for each user.

The rehabilitation exercises integrated in HaReS are detailed in the following subsections.

3.4.1. Copy Pose

In this exercise, the patient must copy a static hand pose for a configurable amount of time. The goal hand poses are recorded using the Leap Motion device by the therapists. The same device is used for processing the patient’s hand pose whilst performing this exercise. A tolerance threshold is applied in order to consider that a pose is correct.

As explained before, the Leap Motion controller provides the tridimensional position of each joint in a hand, taking advantage of this to measure the users’ performance. In this case, we computed the mean euclidean distance to each fingertip from the user’s pose to the goal pose. In order to make the system tolerant to different users, the hands were normalized so that they all had the same size. This is also adequate for measurements, as it enables a global framework of comparison that is not user-dependent.

An image depicting a subject executing this exercise can be seen on the leftmost image of Figure 4.

3.4.2. Copy Pose Dynamic

This exercise is similar to the one described in Section 3.4.1. However, in this case, the poses are dynamic, namely, the user must copy a specific movement instead of a fixed pose. The dynamic hand poses the patient must replicate are prerecorded by the very same hand-tracking device that is used in the HaReS system, namely, the Leap Motion controller. The therapists can thus add new dynamic poses to the task with ease so that the patients can replicate them.

To quantitatively rank the similarity between the goal movement and the user movement, we computed dynamic time warping (DTW) [56]. Applied to this scope, DTW measures the similarity between the trajectories of two 3D points that are shifted on the time dimension, namely, it isolates the time factor measuring only the similarity of two trajectories. As the original implementation is very computationally demanding, we used an approximation of the algorithm, which is much faster, as described in [57].

We applied DTW to the trajectories of the fingertips between the goal movement and the user movement and averaged them. An exercise is considered correct if this value is under a configurable tolerance threshold.

The rightmost image of Figure 4 depicts a subject carrying out one of these exercises.

3.4.3. Follow the Path

This exercise consists of following a specific path with the index fingertip of the hand, namely, the user must place the finger on an array of subgoals arranged on the screen sequentially. As providing visual feedback within the display of a tridimensional position is ambiguous, the paths are simplified to be 2D. Thus, the paths are defined in a plane that is perpendicular to the ground.

In this case, the time the user takes to reach all of the subgoals is measured. As explained before, the euclidean distance is computed between the subgoal and the 2D fingertip position, and if the distance is lower than a threshold, the user is considered to have reached that subgoal.

The leftmost image of Figure 5 depicts a subject carrying out one of these exercises.

3.4.4. Grab Object

Unlike the exercises described in the sections above, which are focused on strengthening the muscles on the hand and on low-level and fine motion, this exercise also introduces a functional factor as well. In this exercise, an object is displayed to the patient, who performs the movement required to grab the object. As there is a range of different ways to grab a certain object, no quantitative measure is provided for these exercises.

As the sessions are recorded, they provide the therapists with evidence about the patient’s skills, and whether their grabbing methods harm other muscles due to unnatural movements induced by the condition of the patient.

A subject carrying out one of these exercises can be seen in the rightmost image of Figure 5.

3.4.5. Free Style

This task does not have a fixed aim, but instead records the patients’ hand movements for the therapists so that they can review them later. This option enables the therapist to ask for certain exercises that are not considered by HaReS and still provide valuable insight about the user rehabilitation program.

4. Evaluation

In order to evaluate the value of HaReS, we conducted complete pilot tests in two different facilities for motor and cognitive rehabilitation. One was carried out in the ADACEA foundation, as mentioned before (five patients with a range of different levels of hand motor limitation and of different ages, and two therapists), and the second one was performed in a local nursing home (five patients with a range of different levels of hand motor limitation, aged 72–79, and two therapists). The complete pilot test included two sessions of two hours with the robot in each facility at the beginning and at the end of the pilot test. During the remaining time, they used HaReS without the social robot, namely, deployed on a desktop computer. In total, the pilot test was carried out within a timeframe of four months.

We divided the evaluation section into the two following subsections. First, we conducted a survey for both patients and therapists, the results of which are shown in Section 4.1, in order to gain insight into the overall benefits of HaReS and the facets that are hard to measure, such as the motivation factor. The next part of the evaluation showed more qualitative aspects of the pilot experiment, such as the evolution of the scores per activity included in HaReS. This matter is shown in Section 4.2.

4.1. Qualitative Evaluation

As mentioned before, surveys were handed to the hosts of the experiments in order to qualitatively evaluate HaReS as a whole system. The survey covered a range of different aspects, from the usability to the impact in the rehabilitation. The results are shown in

Table 1. Questions and corresponding scores of the survey conducted in two different facilities for motor and cognitive rehabilitation.

ID	Question	Score
1	The exercises implemented in HaReS overall helped to improve the motor skills of the patients	4
2	The exercises implemented in HaReS helped to improve the adherence at home to the rehabilitation plan	5
3	The exercises implemented in HaReS effectively cover a common rehabilitation plan	3
4	The hand poses provided by the Leap Motion controller are accurate enough	4.5
5	The HaReS interface is comprehensive and easy to use	4
6	HaReS is a motivating factor for the therapists	5
7	HaReS is a motivating factor for the patients	5
8	The use of a social robot is critical in terms of motivation for the therapists	2
9	The use of a social robot is critical in terms of motivation for the patients	4
10	HaReS helped the therapists to provide a better service	4
11	HaReS helped the therapists to provide service to more patients	3

. The score was averaged among the answers of both organizations and were in the range of 1–5, 1 being totally disagree and 5 being fully agree.

Regarding the results of the survey, the therapists and patients agreed that HaReS helped to improve the motivation of the two groups. Therapists remarked that HaReS indeed helped to improve the motor skills of the patients at the same level of the traditional rehabilitation methodology, and also stated that the proposal helped to provide services to more patients at the same time. For the patients, the use of a social robot was a motivation factor that improved the user experience and engagement. Nonetheless, the therapists stated that the robot had no effect in their motivation, nor on the results of the rehabilitation plan. Regarding the accuracy of the hand poses, the therapists stated that it was acceptable, and that it seldom lost track of the hands or depicted incorrect poses.

In addition to the survey, the therapists also offered valuable feedback. The comments included that HaReS could benefit from more exercises. They also suggested to somehow involve gamification to further improve the adherence to the rehabilitation plan and to enhance the appeal of HaReS. Finally, they proposed the use of a virtual reality setup that provides a better immersion of the patients in order to further improve their experience and the efficiency of the functional exercises.

As a conclusion, we can mention that HaReS indeed helped in the motivation of both therapists and patients, and that HaReS helped to optimize the time that the therapists used to provide services to more patients at the same time.

4.2. Quantitative Evaluation

Here, we show the quantitative results obtained by the patients during the four-month timeframe in which the pilot experiment was conducted. Specifically, we show the quantitative measures of each activity included in HaReS. First, as explained before, the copy pose activity measures the mean euclidean distance from the user’s fingertip to the goal. Then, the copy pose dynamic activity measures the mean DTW from the user’s fingertip to the goal. Finally, the follow the path activity measures the time the user takes to reach all of the goals. The results are shown in Figure 6, Figure 7 and Figure 8.

As per the experiments shown, the patients of the ADACEA foundation experienced a significant improvement over the timeframe in which they used HaReS in all tasks. The patients of ADACEA are people that have suffered an accident or a sudden condition. Thus, their hand motor limitation was acquired and, in most cases, reversible. This means that they were expected to improve over the three months in which the experiment was conducted. Nonetheless, the results for the patients of the nursing home were different. They barely experienced any improvement. This is likely due to all of them being elderly people, with motor limitations due to their age. Note that the expected goal of HaReS in this case is to stop further degradation of their muscles.

5. Conclusions and Future Work

In this paper, we introduced HaReS, a hand skill rehabilitation software for the motor- and brain-injured. HaReS takes advantage of low-cost sensors, such as a hand-tracking device and sEMG sensors, to help quantitatively and automatically evaluate the performance of a rehabilitation plan, and helps to follow the evolution of a patient. As there is no need for a therapist to be present during the rehabilitation sessions, it helps to provide services to more patients.

The evaluation we carried out in two different rehabilitation facilities indicated that HaReS effectively helped both therapists and patients with the rehabilitation plan in terms of both adhesion and improvement of the motor skills. In addition, they both found HaReS a motivating factor that engaged users in the rehabilitation exercises.

As for future work, we plan to involve a full machine learning method for estimating users’ hand poses that enables to provide accurate prediction whilst keeping the computation cost at bay. In addition, as the therapists stated, it is important that the rehabilitation exercises have a goal, such as grasping an object or completing a puzzle, rather than aimlessly repeating a movement. In this regard, we plan to implement the idea developed here using a virtual reality environment so that patients can engage cognitively even further. In addition, we plan to use the data gathered by HaReS to train machine learning methods that can automatically predict whether the user is improving or not by reviewing their history, or to state whether a user is prone to suffer any hand motor or cognitive disease such as Parkinson’s. We plan to also involve computational models [58,59,60] to produce EMG signals that would enable to provide an automatic analysis to assist therapists to understand the sEMG signals provided by HaReS.

Finally, we want to also remark that due to the COVID-19 constraints, we were unable to access more patients or more rehabilitation facilities. We plan to carry out a proper set of experiments once we are able to freely access these people again.