Rehabotics: A Comprehensive Rehabilitation Platform for Post-Stroke Spasticity, Incorporating a Soft Glove, a Robotic Exoskeleton Hand and Augmented Reality Serious Games ^†

Abstract

Spasticity following a stroke often leads to severe motor impairments, necessitating comprehensive and personalized rehabilitation protocols. This paper presents Rehabotics, an innovative rehabilitation platform incorporating a multi-component design for the rehabilitation of patients with post-stroke spasticity in the upper limbs. This system incorporates a sensor-equipped soft glove, a robotic exoskeleton hand, and an augmented reality (AR) platform with serious games of varying difficulties for adaptive therapy personalization. The soft glove collects data regarding hand movements and force exertion levels when the patient touches an object. In conjunction with a web camera, this enables real-time physical therapy using AR serious games, thus targeting specific motor skills. The exoskeleton hand, facilitated by servomotors, assists patients in hand movements, specifically aiding in overcoming the challenge of hand opening. The proposed system utilizes the data collected and (in combination with the clinical measurements) provides personalized and refined rehabilitation plans and targeted therapy to the affected hand. A pilot study of Rehabotics was conducted with a sample of 14 stroke patients. This novel system promises to enhance patient engagement and outcomes in post-stroke spasticity rehabilitation by providing a personalized, adaptive, and engaging therapy experience.

1. Introduction

Stroke is one of the leading causes of long-term disability globally, frequently resulting in upper limb spasticity. Spasticity is characterized by increased muscle tone, stiffness, exaggerated tendon reflexes, involuntary activation of the muscles and resistance to passive movement [1], which significantly affects patients’ quality of life [2]. Rehabilitation, including physical therapy and occupational therapy, is a key part of recovery after a stroke. However, the amount of time that post-stroke patients typically spend in the hospital working on their upper limb rehabilitation is inadequate for achieving a full recovery of function [3,4].

Traditional physiotherapy, while crucial for recovery, often encounters challenges such as suboptimal patient engagement and lack of personalization. Serious games have demonstrated a significant enhancement in patient motivation [5]. Also, innovative technologies, such as robotic exoskeletons and augmented reality (AR), have shown promising results in rehabilitating patients with motor impairments [6]. This maximizes the results when paired with the presence of a therapist (as in traditional physiotherapy), allowing for immediate adjustment and personalization of therapy based on the patient’s performance and feedback. Nevertheless, in telerehabilitation, while immediate therapist feedback might not be possible, the use of advanced technologies like AR can enable real-time adaptation and personalization of therapy protocols. The concept of serious games in rehabilitation is not new. In 2009, Burke et al. determined the principles of game design suitable for upper limb stroke rehabilitation, and several games were developed and showcased based on these principles [5]. Based on this premise, the challenge of customizing games for rehabilitation purposes, as stated in [7], lies in rehabilitation complexity, as it often demands human involvement. For instance, research on game usability has been conducted where assessment involved interviewing both rehabilitators and their patients [4]. In a proposal by Rabin et al. [8], the impact of serious games on motor control was studied using a serious game that progressively increases in difficulty for upper-limb rehabilitation. The primary issue with this approach is the adjustment of game parameters. Hocine et al. [7] detailed a universal technique for adjusting the difficulty of games intended for upper-limb rehabilitation, focusing on modifying pointing tasks and creating game levels. González-González et al. [9] proposed a recommender system which is able to provide the user with personalized games according to their history and skills. Also, devices such as KINECT have been used to adjust the difficulty levels using a fuzzy system [10].

To enhance motor functions, exoskeleton hand rehabilitation robots have been frequently applied, mimicking human limbs in their design, while being attached to the patient at various locations and having joint axes that align with those of the human joints. Moreover, they can be anchored to a table or be either mobile or fixed relative to the patient’s body [11]. Long-term therapy also includes a significant financial burden of the patients involved, as well as the necessity for patients to physically attend therapy sessions. This contributes to diminished enthusiasm, potentially leading to a decline in quality of life.

Taking the above into account, these challenges highlight the need for innovative systems that can support both therapists and patients during the rehabilitation process. Robotic systems, for instance, can alleviate the physical strain experienced by therapists while facilitating more intensive and extended training periods for patients. This can expedite recovery and enable the more effective execution of repetitive action exercises [6,12]. In this paper, we propose an innovative platform, “Rehabotics”, that addresses these limitations by combining a soft glove for data collection, a robotic exoskeleton hand, and AR serious games, thereby offering a comprehensive, patient-centric, and engaging approach to post-stroke spasticity rehabilitation.

The paper is organized as follows: Section 2 describes the components and methodology involved in the proposed system. In Section 3, we present the results from our pilot study. Finally, Section 4 includes a discussion section, reflecting on the study’s results, highlighting the potential of the Rehabotics system to enhance rehabilitation outcomes while suggesting areas for future research and addressing the limitation of the study.

2. Materials and Method

The proposed system is composed of multiple components, each serving a distinct function in the rehabilitation process. Rehabotics’ architecture can be divided in 3 layers: the patient layer, the cloud layer and the clinician layer (Figure 1). The patient layer consists of the AR system encompassing a set of 7 serious games, the soft glove embedded with a series of sensors, the robotic exoskeleton hand and a data acquisition module. The collected data from the patient layer are transmitted to the cloud layer, which serves as the system’s computational core. The cloud layer is essentially the brain of the system, conducting data analysis and decision-making tasks. It also includes a patient registry which stores the profiles of all patients using the system, along with their therapy progress, performance, and demographic data. The statistical analysis tools facilitate a more detailed analysis of these data, generating insights about patient performance trends and therapy effectiveness. The clinician layer incorporates a patient management system and reminder functionality through a web application. Clinicians can use the management system to schedule and keep track of their patient’s sessions. They can also set reminders for themselves in case a patient is underperforming, ensuring timely interventions. This layer continues to serve as the interface for clinicians to monitor patient progress and validate system recommendations.

This three-layer architecture was deployed to enhance the Rehabotics system’s effectiveness, making it not just a therapeutic tool, but also a management system for post-stroke spasticity rehabilitation. As such, the patient layer ensures active patient engagement, the cloud layer offers data analysis capabilities, and the clinician layer provides comprehensive clinical management and oversight functionality.

2.1. System Design and Components

2.1.1. Soft Glove

A crucial component of our system is the soft glove worn by the patient during therapy sessions (Figure 2a). This glove is equipped with multiple sensors that continuously monitor and record the patient’s hand movements and force exertion levels, capturing the position, velocity, and orientation of the hand in real-time. These sensors work together to provide the status of the hand’s movements in real time, the amount of force applied from the patient and the bending of each finger during various activities (Figure 2b).

This soft glove is based on CaptoGlove [13], which is embedded with an array of sensors. It includes a bending sensor on each finger measuring the bending per finger independently. Furthermore, it includes pressure sensors located on the fingertips which assess the amount of pressure being applied. The final sensors, located on the back of the hand, comprise an inertial measurement unit that incorporates a tri-axial gyro, tri-axial accelerometer, tri-axial magnetometer, and a single barometer. These components work together to provide readings on velocity, orientation, and the effects of gravitational forces. The glove can work wirelessly using a Bluetooth Low Energy (BTLE) module (Nordic^® Semiconductor, Trondheim, Norway), transmitting real-time data with a sampling rate from 30 to 35 Hz.

The data collected by the soft glove are used in several ways. Firstly, the data from the bending sensors are used to help the clinician assess the patient’s hand’s range of motion (ROM), both passively (passive ROM) and actively (active ROM). During passive ROM measurement, the joint movement is induced by the clinician, while in active ROM measurement, the patient exerts an effort to open their hand using their functional hand [14]. Secondly, the data from the pressure sensors are used to quantify the spasticity of the upper hand using the Modified Ashworth Scale (MAS) [15] and track the patient’s progress over time. By analyzing changes in movement patterns and force exertion levels using the pressure sensors, the system can objectively measure improvements in the patient’s motor skills and the effectiveness of the therapy. This provides valuable insights for both the patient and the therapist, enabling them to understand the recovery progress made and adjust the rehabilitation plan if necessary.

The integration of the soft glove’s clinical assessment capabilities further enhances the personalization and effectiveness of rehabilitation therapy. As such, by providing continuous, objective, and comprehensive data on the patient’s hand movements, experts can tailor the therapy to the individual’s specific needs and monitor their progress in a quantitative way, an exceedingly challenging task if only traditional physiotherapy is applied.

2.1.2. Robotic Exoskeleton Hand

The robotic exoskeleton hand (Figure 3) represents a significant part of our integrated system designed for post-stroke spasticity rehabilitation. This component is designed to provide physical assistance to the patient during periods of increased spasticity, facilitate the execution of exercises and enhance the overall effectiveness of the therapy.

The exoskeleton hand is installed with 5 servomotors, each connected to the fingers through a metallic line. This configuration allows the extension of the patient’s fingers, providing the physical assistance necessary for various rehabilitation exercises. When the system detects that the patient is struggling to open their hand, the servomotors are activated to pull the metallic lines, effectively assisting the extension of the fingers, even when their spasticity level is high. Specifically, for patients suffering from hand spasticity, a common issue is the inability to fully open the hand due to increased muscle tone. This problem can impede the patient’s ability to perform exercises effectively and limit the potential for functional recovery. To address this, our robotic exoskeleton hand is designed to actively assist with the hand-opening movement.

Moreover, the force exerted by the servomotors is adjustable, allowing for a personalized level of assistance, based on the patient’s current capabilities and progress. The data collected by the soft glove and the patient’s MAS score are used to determine the optimal force level, ensuring that the patient receives the right amount of assistance at any given time.

2.1.3. AR Serious Games Platform

The concept of “serious games” refers to games designed as interactive tools to facilitate the rehabilitation process, while promoting engagement and enhancing motivation. Serious games serve a dual purpose. Firstly, they transform the traditionally mundane physiotherapy exercises into an engaging and enjoyable experience, increasing patient motivation and compliance. Secondly, the games act as an objective, quantitative tool for assessing the patient’s progress. By tracking the patient’s performance in the games, the system can measure improvements in motor skills over time, providing an additional layer of data for clinical decision support. In this study, the main challenge was accurate hand position control, due to the inertial sensors’ limitations in capturing the absolute position of the hand in space (related to noise and integration errors from acceleration and angular velocity) [16]. To address this obstacle, a web-camera was used to translate the recorded hand movements into virtual hand movements within the game environment, using OpenCV and MediaPipe libraries.

The AR Serious Games Platform incorporates a set of Sollerman games, including Wallet, Key, Jar, Screwdriver, Draw, and Balloon Pop, some of which are presented in [17], along with the box and block test (BBT) [18]. These games are designed to engage patients in exercises that mimic real-world activities, enhancing the functional relevance of the exercises. All the Sollerman games scores—except for Balloon Pop—are measured according to the Sollerman Hand Function Score. In addition, the box and block test is measured on a scale from 0 to 150, depending how many blocks the patient successfully moves to the other side, scoring 1 point for each block. In Balloon Pop, the patient scores 10 points for each balloon they successfully pop. During the execution of these games, the AR platform provides real-time feedback on performance, encouraging patients to strive for continuous improvement.

A web service interface is used to transfer the data from the AR platform to an endpoint of the backend in cloud layer. For each therapy session, the score for each game and several of the additional game parameters are transmitted. Accompanying metadata ensure accurate patient identification. The games are designed with customized parameters which can affect the difficulty of the exercise. The type of game, the exercise duration, the number of repetitions and the difficulty levels are not static, but rather are adjusted according to the performance of the patient. This performance is tracked using the data collected by the web-camera and the soft glove, thus creating a specific profile for each patient.

2.1.4. Rehabotics Personalized Profile

One of the most important factors is the initialization of the patient’s profile. In order to first create a profile, MAS, passive and active ROM are measured from the clinical assessments using the soft glove and stored for each patient, while gamification begins with the same initial session which includes one repetition of each Sollerman game and the BBT.

2.2. Usage Scenario

A sample case scenario is as follows: a patient visits a physiotherapy clinic. The clinician enters the patient’s details into the Rehabotics platform. The clinician then fits the soft glove onto the patient’s hemiplegic hand and proceeds to assess the level of spasticity using the MAS, in addition to the passive and active ROM. Following this, the soft glove is removed, and the clinician assesses the hand functionality utilizing the BBT. Upon completion of the assessment, the robotic exoskeleton hand is fitted onto the patient’s hand. The clinician subsequently initiates a stretching program specifically tailored to the patient, conducting 3 sets of 10 repetitions on the flexors of the patient’s hand. After the stretching procedure is complete, the exoskeleton hand is removed. At this point, the clinician opens the serious games platform. The patient then proceeds to engage with all the games available, playing each serious game once.

3. System Evaluation

To demonstrate the efficiency of the proposed system, Rehabotics was applied to post-stoke patients for rehabilitation. As such, the participants included in the pilot study consisted of 14 post-stroke patients (mean age, 66.16 ± 10.03 years). In order to assess the reliability and stability of Rehabotics, this study’s focus was directed towards the AR BBT. To accomplish this, we applied a test–retest validation method, which involved administering the same test to the same group of subjects under identical conditions, repeating the process after a one-week interval.

Participants were asked to interact with the AR BBT under controlled conditions. The patients were seated in an ergonomic chair with the web camera positioned at a fixed distance of 50cm, aligned perpendicular to them. The room was well lit to ensure the production of optimal results for the tracking system. Throughout this process, a physiotherapist was present to assist the patients if any difficulty was encountered. The same group of participants was retested under identical conditions after 1 week using the same configuration. Furthermore, the Pearson correlation coefficient was employed as part of the test–retest method to assess the consistency and reliability of the AR BBT. Our findings revealed a correlation coefficient of 0.97, indicating an exceptionally strong positive correlation. The mean score of the initial test was 8.36, with a standard deviation of 1.34. The second test indicated a slight increase in the mean score to 8.5, with a standard deviation of 1.3. These results suggest that the AR BBT generates consistent results over time, and is an effective tool to evaluate the motor function of patients; however, it is important to note that these findings are based on a pilot study with a relatively small sample size. Nevertheless, the preliminary results of the proposed system display encouraging results.

4. Discussion

While our AR platform and the use of the box and block test have shown promising results, we recognize that our system has multiple other components that require further exploration. Specifically, two key components of our intervention, the soft glove and the exoskeleton robotic hand, need additional research. The soft glove, with its ability to gather data about the patient’s hand movements, holds significant potential for contributing to a deeper understanding of patient progress and the effects of our therapy. To this end, the data generated by this tool can be further investigated, especially on how it could be utilized to personalize and optimize patient therapy. The exoskeleton robotic hand, on the other hand, could provide assistance to patients with severe spasticity who struggle to initiate movements. While it shows promise, this component has not been pilot tested with stroke patients. Future work will require comprehensive testing and validation to ensure safety, efficacy, and patient comfort.

Furthermore, despite the reliability of the AR BBT for assessing motor function in patients (as demonstrated by the pilot study), there are additional serious games within the AR Serious Games platform that were not assessed. Each game is likely to provide unique challenges and may offer different insights into patient capabilities and progress. Additional gamification features would potentially enable a more comprehensive understanding of each patient’s motor capabilities and progress, supporting more personalized and effective therapeutic interventions.

The possibility for the incorporation of clinical decision support into the Rehabotics platform has not escaped our notice. As such, an autoregressive model to generate clinical rules for physiotherapy could be considered ideal, since these models predict future values based on a weighted sum of past observations [19]. The model can then be trained on a larger dataset collected from both healthy individuals (control group) and stroke patients. In our system, this model could analyze the data collected through the glove and the serious games output, thus creating a patient’s profile to suggest the ideal serious games for rehabilitation. The clinician could then evaluate the suggested therapy session and can choose modifications (sequence of games, adjusting the number of repetitions, or altering the difficulty level) based on their clinical judgment. Moreover, the participant’s profile will evolve over time, based on the patient’s progress, ensuring that the therapy remains challenging yet achievable, fostering patient engagement and motivation.

In conclusion, Rehabotics is an innovative stroke rehabilitation platform that integrates various technologies to provide a comprehensive solution and high levels of customization, rarely seen in traditional physiotherapy. As a whole, Rehabotics represents a promising step towards revolutionizing stroke rehabilitation, promising significant improvements in patient outcomes with further research and refinement.