Telehealth Using PoseNet-Based System for In-Home Rehabilitation

Abstract

The increasing cost of healthcare services is accelerating the development of the telehealth system to fulfill the necessity of delivering an efficient and cost-effective remote healthcare services. Moreover, the ageing of the global population and the disruption of the COVID-19 pandemic are creating a rapid rise of demand for healthcare services. This includes those who are in need of remote monitoring for chronic conditions through rehabilitation exercises. Therefore, this paper presents a telehealth system using PoseNet for in-home rehabilitation, with built-in statistical computation for doctors to analyze the patient’s recovery status. This system enables patients to perform rehabilitation exercises at home using an ordinary webcam. The PoseNet skeleton-tracking method is applied to detect and track the patients’ angular movements for both elbows and knees. By using this system, the measurement of the elbow and knee joint angles can be calculated and recorded while patients are performing rehabilitation exercises in front of the laptop webcam. After the patients complete their rehabilitation exercises, the skeleton results of four body parts will be generated. Based on the same actions performed by patients on selected days, the doctors can examine and evaluate the deviation rate of patients’ angular movements between different days to determine the recovery rate.

1. Introduction

Telehealth refers to the delivery of health-related services including medical care, health education and health information services over a distance via the use of digital information and communication technologies such as computers, laptops and mobile devices [1]. Telehealth was created to overcome the physical barriers in offering healthcare services to patients and caregivers without in-person visits and to reduce the travelling challenges for patients in rural areas [2]. On top of this, rapid ageing of the global population has brought unprecedented and escalating challenges to the healthcare services. According to data provided by the World Health Organization (WHO), the proportion of the global population over the age of 60 will rise from 12% to a staggering 22%, almost doubling from 2015 to 2050 [3]. These elderly, especially those with chronic conditions, are not fit to travel long distance for healthcare services. Therefore, telehealth plays a crucial role in the seamless delivery of an efficient and cost-effective remote healthcare services for these people. In addition, the current COVID-19 pandemic has also induced the rapid adoption of telehealth and instigated adaptation and reform in the delivery of healthcare services [2]. Studies have shown that 90% of patients and parents felt safer using a telehealth virtual appointment compared to an in-person visit during the COVID-19 pandemic [4]. The functionality and scope of the telehealth system are rapidly increasing and well received by patients and clinicians.

By leveraging the advantage of telehealth to facilitate remote healthcare services, in-home rehabilitation offers diverse healthcare services to individual households. This covers a range of activities, from assessing the patient’s functional ability in his/her environment to specific supervised treatment or exercises [5]. In most countries nowadays, in-home rehabilitation services are strongly supported by the government. The number of patients applying for such services has grown significantly, adopting family assessment and occupational therapy to enhance the performance of daily activities [6]. It can be effectively used in physical therapy, such as postoperative knee rehabilitation [7,8]. As a result, in-home rehabilitation offers a lot of benefits for patients via the use of a telehealth system.

Traditionally, patients perform rehabilitation exercises in hospitals or rehabilitation centers. In Malaysia, rehabilitation services are offered by both governmental and non-governmental agencies [9]. Some challenges in these provisions include the lack of coordination among these agencies and severe service barriers for patients in rural areas with poor transportation, or for whom driving is unsuitable because of their medical condition. One way to address this issue is to hire a home therapist who visits the patient’s house, but not all patients can afford the higher cost. An alternative solution is to offer in-home rehabilitation services via the use of a telehealth system.

Previously, a marker-based system was used for in-home rehabilitation to detect patient’s movements by placing markers on their bodies. However, the cost of these marker-based systems is high [10]. An example of a low-cost marker-based system such as Perceptron Neuron is priced at USD$1499 (approximately RM 6000) in Malaysia [11]. According to the Malaysian Department of Statistics, the average monthly household income of the bottom 40% of the population (the B40 group) is below RM 4849, while the average monthly household income of the middle 40% of the population (the M40 group) ranges from RM 4850 to RM 10,959 [12]. Hence, Perceptron Neuron is obviously beyond the reach of the B40 and M40 household income groups in Malaysia. On top of this, professional skill is required to place the markers correctly on the bodies [13]. If the markers are misplaced, it might lead to an inaccurate detection. In addition, it is also a time-consuming process and patients might experience some discomfort when many markers are attached to the body for long hours during the rehabilitation procedure [13]. Due to the high cost of the marker-based system, a more affordable system using Microsoft Kinect was introduced in 2010 [10,14,15,16,17,18,19,20,21,22,23]. Most importantly, this system eliminates the need to place markers on the body. After the discontinuation of Microsoft Kinect in the fall of 2017, another skeleton recognition technology, the PoseNet, which uses TensorFlow.js library, emerged for movement detection [15]. Unlike Microsoft Kinect, patients are not required to purchase the sensor or attach markers to their body. PoseNet can detect patient’s movement with just an ordinary webcam attached to the laptop.

In this paper, a PoseNet-based telehealth system for in-home rehabilitation is introduced. The system produces a set of statistical computations based on the patient’s angular movements for comparison and evaluation by the doctors. This enables doctors to conduct in-depth analysis. On top of this, the system offers the potential to fill in the gaps by: (1) using an ordinary webcam with internet connection for rehabilitation exercises; (2) providing access for both patient and doctor to the skeleton results in a timely manner as soon as the patient has completed the rehabilitation exercises; and (3) displaying the percentage of deviation rate of the patient’s skeleton results for different days, for the doctor to further analyze the patient’s recovery status. Our aim is to assist the patient in performing rehabilitation exercises at home without increasing their financial burden and travelling challenges. This is especially useful for elderly patients with limited mobility.

During in-home rehabilitation, the patient needs to stand in front of the laptop webcam to perform the rehabilitation exercises. PoseNet will detect the posture and record the angular movements of both elbows and knees. The patient can obtain the skeleton data after completing the rehabilitation exercises for evaluation. The patient needs to continue with the same rehabilitation exercises for the selected number of days so that the doctors can monitor their rehabilitation postures and examine the deviation rate of the patient’s angular movements to determine their recovery rate. They can then receive advice or recommendations from the doctors. As a result, the patient can improve their postural movements based on the doctor’s prescriptions. Hence, it is possible to implement an in-home rehabilitation procedure using the PoseNet-based telehealth system, especially for low-income patients in Malaysia.

The rest of the paper is organized as follows. First, Section 2 reviews some of the related studies that deliver similar systems. Next, the proposed telehealth system is elaborated in Section 3. The methodology for the functionality test is presented in Section 4. Section 5 clarifies the discussions of the results obtained. Finally, the conclusion is discussed in the last section.

2. Related Work

Various rehabilitation systems or applications have been currently developed, such as the virtual reality and motion-based games [16]. As a result, these applications have attracted great attention and interest among researchers. These applications motivate patients to move their body parts more frequently and offer patients a sense of accomplishment through an interactive user interface. Patients will need to follow the step-by-step rehabilitation exercises as guided by a 3D avatar or by following the recorded instructional video by the professional physician.

Anton et al., Yu et al., Chiang et al., Lin et al. and Su et al. [16,17,18,19,20] presented a Kinect-based rehabilitation system by integrating different approaches. Research in [16,17,18] focuses on integrating a 3D avatar for rehabilitation exercises. The system matches the patients’ movements with the benchmark movements of a 3D avatar to determine the accuracy of the posture. The authors in [16] determine the accuracy of the posture by measuring the differences between the 3D angle of the joints and limbs. After the comparison, the system will display five different messages, namely “Execute the movement (has reached the initial position and has to execute the movement)”, “Execute posture (very far from reaching the next posture)”, “Cool, cool (about to reach the position)”, “You’re close (very close to the position)”, “Correct (posture is correct)”. The authors of [17,18] adopt the dynamic time warping (DTW) approach to calculate the optimal warping distance in order to measure the similarity between two motion sequences. The optimal warping distance is then converted to a performance score (0–100%) to show the similarity between the patient’s and the 3D avatar’s movement.

Research in [19,20] adopts a recorded instructional video to guide patients to perform rehabilitation exercises at home. The patients need to follow the exercises as guided by a physician in the recorded instructional video. After completing the rehabilitation exercises, joint distance is calculated in [19] based on the captured patient’s skeleton keypoints and the law of cosine is applied to calculate the angle differences between patients’ movement and the standard movement of the physician. A quantitative score is then provided by dividing the score into four levels of comments, namely “Try it again”, “Good Work”, “Excellent” and “Perfect”. On the contrary, the research in [20] integrated DTW with Euclidean distance and fuzzy logic to convert the optimal warping distance to three performance levels, namely “Bad”, “Good” and “Excellent”, to determine the accuracy of the posture.

Four sets of the Kinect-based rehabilitation system were presented in [21,22,23,24]. These research papers were published based on a project that was designed to focus on patients who usually suffer from movement disorders and neurological disorders that require in-home rehabilitation. The system in [21] presents a virtual trainer for patients to imitate movements during the rehabilitation exercises. Based on the 2D keypoints obtained from the hands and legs of patients, the joint angle is calculated by using the law of cosine. The calculated joint angle is then used to compare with the benchmark joint angle of the virtual trainer to evaluate the deviation of the patients’ movement angle. On the other hand, researchers in [22,23,24] designed a video game for rehabilitation exercises with the aim of increasing the motivation of patients to perform in-home rehabilitation. These three papers offer several rehabilitation games that require the use of virtual movements to control the games. The patients need to move their bodies and perform a series of specific movements to achieve the goal. This is also beneficial for patients because they exercise their bodies more often while playing. Researchers in [22,23] implement games based on different difficulty levels to monitor patients’ progress. The paper in [24] measured and recorded the patients’ angular movement during game play. After the patients have completed the games, an angular movement graph will be generated, enabling the physiotherapist to monitor whether the patients achieve the accuracy of the required movement.

The aforementioned papers use Kinect sensors as the input device to detect the skeleton movements of the patients in their rehabilitation systems. The patients need to purchase a Kinect sensor to use these systems. Alternatively, Adolf et al. [25] developed a rehabilitation system, OpenPose, which uses a computer vision approach to detect the keypoints and skeleton movements of the patients. This has created an opportunity for in-home rehabilitation system using only a laptop or mobile camera, eliminating the need to purchase any additional device or sensor. When the patients perform rehabilitation exercises, they only need to stand in front of their laptop or mobile camera to allow OpenPose to detect their skeleton movements. After the patients have completed the rehabilitation exercises, the angles are computed between the body parts using the captured data of the patients’ skeleton keypoints. An angular movement graph will be generated for the physiotherapist to perform manual assessment by comparing with the angular movements performed by the professional trainer to monitor the accuracy of the patients’ movements. Despite its lower cost, manually examining the patients’ posture through an angular movements graph may be tedious and susceptible to fatigue error.

On the other hand, Hassan et al. [26] proposed a PoseNet-based rehabilitation system to assist patients who undergo physiotherapy exercises. This system utilizes the PoseNet to detect and extract the skeleton keypoints of patients from the 30-s live exercise video. A 3D avatar performs the exercise to guide the patient in the correct postures. After recording, the PoseNet model will extract the patient’s skeleton keypoints. The score is then calculated by using the DTW approach to measure the similarity between the keypoints of the patients and the benchmark keypoints performed by the experts. Finally, the patient’s score (0–100%) will be provided based on all the exercise movements along with a short feedback statement to advice on incorrect performance, such as, “Work on your left elbow”, “Work on your right wrist” and “Good job”.

In summary, the existing in-home rehabilitation systems come with a few problems as follows: (1) A special-purpose hardware such as Kinect is required; (2) There is no automated comparison of the angular movements performed by patients on different days; (3) There is no visualization result and automated statistical computation to assist in the analysis and evaluation process of the patient’s performance. Hence, a telehealth system using PoseNet-based system for in-home rehabilitation is proposed with the following advantages: (1) Using PoseNet with an ordinary webcam to detect the patients’ angular movements incurs no additional expenditure on special-purpose equipment; (2) An automated comparison of the patients’ angular movements on different days is built into the system; (3) A visualization result and statistical variables such as mean, standard deviation and the percentage deviation rate of the patients’ angular movements are automatically calculated so that the doctors can analyze the patients’ recovery status directly with more in-depth analysis.

3. Proposed System

In this section, the design and implementation of the proposed telehealth service using a PoseNet-based system for in-home rehabilitation is explained. The system architecture is presented, followed by the explanation on the computation of the body keypoints. Finally, the statistical computation for the patient’s performance on different days is presented.

3.1. System Architecture

Figure 1 shows the system architecture of the proposed telehealth system. The system consists of three different modules namely, the account management module, the virtual appointment module and the prescription module. These modules are used by the patients, doctors and system administrators.

In the proposed system, the admins are responsible for maintaining the account of both patients and doctors. The patient can make virtual appointments with the doctor or perform the rehabilitation exercises, as shown in the bottom part of Figure 1. A patient can first schedule a virtual appointment with their doctor to discuss their physical condition. They will receive an appointment email notification when the doctor confirms the appointment schedule. During the scheduled appointment date and time, the patient will conduct a video call with the doctor. After discussing with the doctor, they can start the rehabilitation exercises (step 1). The proposed system will then calculate the patient’s angular movements (step 2) and store them into the database (step 3). After completing the rehabilitation exercises, a skeleton result (angular movement graph) will be generated, enabling the patient (steps 4 and 5) to examine their angular movements through the graph. The doctors can then access the patient’s data (step 6) to evaluate and compare the skeleton results on different selected days (step 7) to provide some advice or prescriptions to the patient (step 8). Finally, the patient can check the prescriptions and make adjustments to their subsequent exercises (step 9).

The skeleton detection method used to deploy the proposed system is the PoseNet. It is a machine learning model that can be applied for real-time human pose estimation [27,28]. By running PoseNet with Tensorflow.js library, anyone with camera-equipped devices such as laptops, PCs and smartphones can experience this technology directly from a web browser. The patient simply needs to stand in front of the device camera. The system will then track all the skeleton points of the patient. The PoseNet can detect a single pose or multiple poses from the images or videos within the browser. For our proposed system, only single-pose detection is implemented.

On accessing our proposed system, the patient firstly needs to log in to the web-based telehealth system, as shown in Figure 2a. The patient then needs to select one of their preferred doctors (Figure 2b) before performing rehabilitation exercises. Subsequently, the patient can click the start button in Figure 2c to perform the rehabilitation exercises. The system will start the 5-s preparation countdown timer before starting the video capture for the rehabilitation exercise. After completing the rehabilitation exercises, the skeleton result is instantly generated for patient and doctor.

Figure 3 shows the skeleton detection processes during the rehabilitation exercise. First, the patient needs to connect his/her laptop to the internet connection to gain access to the web-based system. When the patient starts the rehabilitation session, their live video is recorded through the webcam and passed as an input to the PoseNet model. When the PoseNet model detects a human pose in the video frame, it extracts the keypoints of the patient by using the 2D coordinate (x, y) to indicate each detected body joint from every video frame. The extracted keypoints are used to calculate the distances and angles of both elbows and knees. When the predefined duration ends, the system will stop the recording and store all four angles from both elbows and knees as an array structure into the database. The patient and doctor can directly access the skeleton results. Based on the same set of actions performed by the patient during the few selected days, the doctor can examine the skeleton result by comparing and evaluating the deviation rate of the angular movements to determine the patient’s recovery rate. Based on this, the doctor can offer advice or write a prescription for the patient.

In order for the doctor to obtain the patient’s skeleton result, they first need log into the web-based telehealth system from the login page shown in Figure 4a. The system will then direct the doctor to the dashboard shown in Figure 4b. By selecting the All patients option at the My Patients (Skeleton) sidebar of Figure 4c, the doctor can view a pre-recorded skeleton list for all the patients. The doctor can then select one of the patient records by clicking the View patient’s skeleton result/add patient’s prescription orange button shown in Figure 4c. The skeleton result for that patient is generated instantly. From here, the doctor can compare the current and previous skeleton result by selecting two dates, as illustrated in Figure 4d. The doctor can then examine the patient’s angular movements through the graph with the help of the built-in statistical computation shown in Figure 4f. After that, the doctor can advise or write a prescription for both elbow and knee movements of the patient. The advice and prescription from the doctor are directed to the specific patient in the format as illustrated in Figure 4e for any adjustment they should apply in their rehabilitation exercises.

Next, the administrators manage the accounts of patients and doctors, moderate the patients’ appointment requests and verify the identities of all patients and doctors. After log into the web-based telehealth system (Figure 5a), the system will then direct the administrators to the dashboard shown in Figure 5b. By selecting Doctors and Patients option at the sidebar shown in Figure 5b, the administrators can view a list of doctors and patients. To manage the accounts for doctor or patient, the administrators can perform CRUD (create, read, update, delete) operations as illustrated in Figure 5c,d respectively. Besides, the administrators can also manage the patients’ virtual appointment by selecting the Patient Appointment option at the sidebar shown in Figure 5b to get a list of all patients’ appointment shown in Figure 5e. To delete patient’s appointment, the administrator can click the Cancel Appointment yellow button shown in Figure 5e.

3.2. Body Keypoints Computation

PoseNet can detect a total of 17 keypoints from different parts of the human body [27,28]. Each key point is represented by its 2D coordinate (x, y), as shown in Figure 6. During a rehabilitation session, the skeleton detection with 17 keypoints is displayed on the body. In the proposed system, only 12 keypoints are used to detect shoulders, elbows, wrists, hips, knees and ankles of the patient.

Figure 7 shows the angles for four body parts, namely the left hand, right hand, left leg and right leg, respectively. Each body part comprises three different keypoints, namely $\mathrm{Point} \mathrm{A}$ (shoulders), $\mathrm{Point} \mathrm{B}$ (elbows) and $\mathrm{Point} \mathrm{C}$ (wrists) for both hands and $\mathrm{Point} \mathrm{A}$ (hips), $\mathrm{Point} \mathrm{B}$ (knees) and $\mathrm{Point} \mathrm{C}$ (ankles) for both legs. Hence, both hands and legs comprise 12 keypoints in total. Each set of three keypoints in the four body parts are allocated with 2D coordinates, $\mathrm{Point} \mathrm{A} \left(\mathrm{x}1, \mathrm{y}1\right)$, $\mathrm{Point} \mathrm{B} \left(\mathrm{x}2, \mathrm{y}2\right)$ and $\mathrm{Point} \mathrm{C} \left(\mathrm{x}3, \mathrm{y}3\right)$, respectively, forming the triangle shown in Figure 8. Next, the corresponding distances for Point A to Point B, Point B to Point C and Point A to Point C of the 12 keypoints are calculated for each video frame by using the Euclidean distance.

Since each hand and leg is composed of three different keypoints, the angles of the key-points are computed at $\mathrm{Point} \mathrm{B}$, which corresponds to both the elbow and knee positions of the patient. Since the flexion and extension exercises are usually performed on the elbows and knees, the computed angles will reflect the movement correctness to determine the performance of the flexion and extension exercises. After obtaining the distances from both hands and legs, the angles θ of both elbows and knees are calculated by substituting the value of $i$, $j$ and $k$ into the $arccos()$ function.

Finally, the skeleton result is generated for both patient and doctor to examine the angular movements. Meanwhile, the Mean, $\mu$ and Standard Deviation, $\sigma$ are computed to display the average of the angular movements and to show how much the angles in a distribution spread out from one another. A low value of $\sigma$ indicates that the angular movements from each frame are clustered closely around the $\mu$, whereas a high value of $\sigma$ indicates the angular movements are spreading to a wider range.

3.3. Statistical Computation

After the patient has performed the same rehabilitation exercises on the selected days, the doctor can compare the patient’s angular movements between the different days by referring to the Mean Absolute Deviation (MAD), Mean Squared Error (MSE) and Mean Absolute Percentage Error (MAPE), as defined in Equations (1)–(3). These statistical computations are very useful to assist in the evaluation process because they offer a more-in-depth analysis metrics for the doctor to write their prescription.

MAD is used to calculate the average distance between the current and previous skeleton results for each angle point of each frame, whereas the MSE is used to calculate the average squared difference. Lastly, the MAPE displays the percentage deviation rate of the current skeleton result as compared to the previous skeleton result. These three formulae are commonly used in measuring the forecasting performance [29,30,31]. MSE is more sensitive to errors than MAD because all forecast errors are squared, which causes the value of MSE to always be greater than MAD. Therefore, the forecast error in MSE becomes more pronounced. However, the major shortcoming of MAD and MSE is that they do not consider the magnitude of the actual value [29]. As a result, the forecast error does not reflect the accuracy of the angular movements. Hence, MAPE is applied to address this issue as it takes into account the magnitude of the actual values. It sums up all the absolute errors for each period with respect to actual values and then expresses the average error in term of generic percentage. This is useful for reporting purposes in our proposed system as it can be used to determine a patient’s recovery rate. This is because the average accumulated error rate in MAPE corresponds to the patient’s angular data in each frame as compared to the actual values. The lower values of MAD, MSE and MAPE signify a more accurate model, which indicates that the postural movements performed by the patient are relatively better.

The low values of MAD and MSE show that the average and squared error of the patient’s angular movements in each frame is small as compared to the actual data from the physician’s movements. Similarly, a low value of MAPE indicates that the average accumulation error percentage of the patient’s angular movements in each frame has a small deviation rate. Most importantly, the MAPE plays a crucial role because the doctor can further analyze the patient’s recovery status through the percentage deviation rate to determine whether the patient’s condition has improved.

$$MAD=\frac{{{\displaystyle \sum }}_{t=1}^N\left|A_t-F_t\right|}{N}$$

(1)

$$MSE=\frac{{{\displaystyle \sum }}_{t=1}^N{\left(A_t-F_t\right)}^2}{N}$$

(2)

$$MAPE=\frac{{{\displaystyle \sum }}_{t=1}^N\left|\frac{A_t-F_t}{A_t}\right|}{N} \mathrm{\times } 100\%$$

(3)

where $A_t$ is the actual value from the patient’s movement, $F_t$ is the value from the benchmark movement, and $N$ is the total number of frames.

4. Methodology

In order to validate the functionalities of the automated statistical computation to assist in the analysis and evaluation process of the patient’s performance in the proposed system, several functional tests were executed. One subject of 22 years old male and one subject of 20 years old female were invited to participate in the functional tests. Both subjects were required to stand in front of the laptop webcam to perform the rehabilitation exercises, as shown Figure 9. A series of benchmark poses were selected from the well-known KIT dataset [32] to represent the ground-truth skeleton-angular movements to be compared to those movements performed by both subjects. The KIT Whole-Body Human Motion Database is a large-scale dataset that offers a data corpus of 2615 different movements, focusing not only on mere whole-body motions but also on whole-body actions (this study makes use only of the motions).

Since the invited subjects did not have any previous experience in rehabilitation exercises, two functional tests were conducted. The purpose of conducting these two tests was to compare the different postures of hand and leg movements performed by both subjects to those from the KIT dataset. Both subjects were required to follow the selected movements from the KIT dataset for two selected days. The first video clip guides the subjects to demonstrate both arms lifted at 90 degrees and to bend to the front for a duration of 30 s, as illustrated in Figure 10. This set of movements helps to enhance the body posture and strengthen the muscles of the shoulders and arms. The second video clip guides the subjects to perform the half squatting action for 20 s, as shown in Figure 11. This set of movements increases the stability of the hip abductor, thus stabilizing the pelvis and offering good balancing training. In order to gain a fair comparison between the benchmark poses and subjects’ poses, both subjects performed the set of movements in the first and second tests for 30 and 20 s respectively, following the durations of the original benchmark movements.

5. Results and Discussion

Figure 12 and Figure 13 show the skeleton detection in both of the video clips of the benchmark poses. The benchmark angular movements from the KIT dataset were calculated.

Figure 14a,e show the posture of the left elbow as performed by both subjects in the first functional test. The dotted region of the angular movements graph shows the obvious angular differences between Day 1 (red line) and Day 2 (black line) as compared to the benchmark angular movements from the KIT dataset (blue line). Subsequently, the MAPE values of the left elbow movements as performed by subject 1 are shown in

Table 1. Mean, Standard Deviation, MAD, MSE and MAPE for functionality test 1.

			Angular Movement (KIT Dataset)	Subject’s Angular Movement			${\mathit{A}}_{\mathit{t}}$ = KIT Angular Movement, ${\mathit{F}}_{\mathit{t}}$ = Subject’s Angular Movement
				Day 1	Day 2		Deviation Rate (Day 1)	Deviation Rate (Day 2)
Subject 1 (Male)	Left Elbow	Mean, $\mu$	143.04	140.83	142.83	MAD	24.27	21.81
		Standard Deviation, $\sigma$	41.09	43.06	41.49	MSE	1834.36	1403.42
						MAPE	51.71%	34.99%
	Right Elbow	Mean, $\mu$	142.25	138.13	141.1	MAD	26.78	22.7
		Standard Deviation, $\sigma$	42.59	46.37	44.34	MSE	1715.62	1399.89
						MAPE	35.75%	28.55%
	Left Knee	Mean, $\mu$	175.46	171.83	172.58	MAD	3.62	2.88
		Standard Deviation, $\sigma$	1.35	0.64	1.05	MSE	15.63	10.86
						MAPE	2.06%	1.64%
	Right Knee	Mean, $\mu$	174.83	175.14	176.52	MAD	1.91	2.87
		Standard Deviation, $\sigma$	1.23	2.25	2.72	MSE	6.3	10.25
						MAPE	1.09%	1.64%
Subject 2 (Female)	Left Elbow	Mean, $\mu$	143.04	141.08	142.52	MAD	13.95	9.2
		Standard Deviation, $\sigma$	41.09	42.41	40.74	MSE	673.99	375.88
						MAPE	19.68%	11.01%
	Right Elbow	Mean, $\mu$	142.25	139.98	140.55	MAD	16.02	10.22
		Standard Deviation, $\sigma$	42.59	44.74	44.48	MSE	815.41	384.23
						MAPE	19.76%	13.58%
	Left Knee	Mean, $\mu$	175.46	171.99	172.08	MAD	3.49	3.39
		Standard Deviation, $\sigma$	1.35	0.95	0.82	MSE	15.03	13.79
						MAPE	1.98%	1.92%
	Right Knee	Mean, $\mu$	174.83	178.26	176.52	MAD	3.46	1.89
		Standard Deviation, $\sigma$	1.23	0.48	2.38	MSE	13.79	5.42
						MAPE	1.98%	1.08%

with 51.71% (Day 1) and 34.99% (Day 2), respectively. On the contrary, subject 2 performed better in terms of the posture of the left elbow as compared to subject 1 on Day 1 and Day 2. Hence, the MAPE values of the left elbow performed by subject 2, shown in Table 1 with 19.68% (Day 1) and 11.01% (Day 2), are significantly lower than those of subject 1. In spite of this, there is slight improvement in the posture of the left elbow performed by both subjects on Day 2 as compared to Day 1, which is reflected in the decreased MAPE values.

For the right elbow shown in Figure 14b,f, subject 1 shows obvious angle differences in the dotted region of the angular movements graph for both Day 1 and Day 2 due to the incorrect performance of the right-arm-lifting posture. These differences are reflected in the MAPE values of Table 1 for the right elbow in Day 1 and Day 2 of 35.75% and 28.55%, respectively. Subject 2 still performed relatively well for the posture of the right elbow, similar to that of the left elbow. Hence, the MAPE values of the right elbow movement of subject 2 are comparatively lower as shown in Table 1 with 19.76% (Day 1) and 13.58% (Day 2), respectively. Meanwhile, both subjects’ angular movements are consistent for both knees, as shown in Figure 14c,d,g,h, hovering between 170 to 180 degrees. Since both legs simply stand still in this sequence of poses, the angular movements of left and right knees are expected to maintain consistently low MAPE values.

On the contrary, the angular movements graph in the second functional test performed by both subjects show significant angle differences, which are highlighted by the dotted line portion in Figure 15a,e. These are also reflected in the left elbow of

Table 2. Mean, Standard Deviation, MAD, MSE and MAPE for functionality test 2.

			Angular Movement (KIT Dataset)	Subject’s Angular Movement			${\mathit{A}}_{\mathit{t}}$ = KIT Angular Movement, ${\mathit{F}}_{\mathit{t}}$ = Subject’s Angular Movement
				Day 1	Day 2		Deviation Rate (Day 1)	Deviation Rate (Day 2)
Subject 1 (Male)	Left Elbow	Mean, $\mu$	150.96	148.17	148.37	MAD	14.94	14.25
		Standard Deviation, $\sigma$	21.68	25.07	25.09	MSE	630.94	616.6
						MAPE	12.61%	12.31%
	Right Elbow	Mean, $\mu$	154.04	152.75	153.06	MAD	18.21	18.72
		Standard Deviation, $\sigma$	23.04	30.19	28.63	MSE	1041.18	1045.65
						MAPE	15.23%	16.75%
	Left Knee	Mean, $\mu$	167.14	167.64	167.35	MAD	7.85	7.15
		Standard Deviation, $\sigma$	12.35	12.31	12.56	MSE	125.68	106.07
						MAPE	4.85%	4.46%
	Right Knee	Mean, $\mu$	164.57	164.84	165.3	MAD	7.78	7.78
		Standard Deviation, $\sigma$	10.79	10.59	11.16	MSE	132.22	161.8
						MAPE	4.94%	4.92%
Subject 2 (Female)	Left Elbow	Mean, $\mu$	150.96	151.02	148.51	MAD	15.6	13
		Standard Deviation, $\sigma$	21.68	20.65	26.84	MSE	689.02	543.08
						MAPE	13.53%	10.73%
	Right Elbow	Mean, $\mu$	154.04	149.75	155.36	MAD	23.57	16.11
		Standard Deviation, $\sigma$	23.04	34.41	26.53	MSE	1649.98	988.74
						MAPE	19.41%	14.49%
	Left Knee	Mean, $\mu$	167.14	167.63	168	MAD	10.27	5.34
		Standard Deviation, $\sigma$	12.35	12.05	12.05	MSE	231.15	66.6
						MAPE	6.45%	3.27%
	Right Knee	Mean, $\mu$	164.57	165.82	166.01	MAD	8.58	6.32
		Standard Deviation, $\sigma$	10.79	10.47	10.73	MSE	145.14	103.96
						MAPE	5.44%	4.01%

with MAPE values of 12.61% (day 1) and 12.31% (day 2) for subject 1 and 13.53% (day 1) and 10.73% (day 2) for subject 2. Subsequently, subject 2 shows significant improvement for the posture of right elbow from Day 1 to Day 2 with decreased MAPE values (19.41% to 14.49%) as compared to subject 1. The angular movements graph for both knees performed by both subjects shows minor variations for both days, as illustrated in Figure 15c,d,g,h with MAPE values ranging from 4% to 7%. Moreover, the movements for both knees performed by both subjects on Day 2 are improving based on the decreasing MAPE values. The low MAPE values of the left and right knees also indicate that both subjects had followed the half squatting pose closely compared to the benchmark pose from the KIT dataset.

In summary, both functional tests have demonstrated the PoseNet-based system for in-home rehabilitation with the following validations: (1) using PoseNet with an ordinary webcam to detect the patients’ angular movements; (2) an automated comparison of the patients’ angular movements on different days is instantly available for patients and doctors after the rehabilitation exercise; (3) visualization graph and statistical computation is automatically generated for doctors to examine the patients’ recovery rate with a more in-depth analysis. Although the functionality of the skeleton detection is validated to offer the angular movement graph and statistical computation for the doctor’s evaluation, there are two limitations that need to be considered in the proposed system. First, the patient needs to ensure that no other human is visible within the webcam view during the rehabilitation session as this will lead to inaccurate skeleton detection. Second, the patient needs to ensure that his/her laptop is placed following the recommended standing distance as specified in the system setup (Figure 9). This will ensure that the webcam can capture the patient’s entire body in the camera view for accurate analysis.

6. Conclusions

This paper presents a telehealth service using the PoseNet-based system for in-home rehabilitation using an ordinary webcam. The proposed system offers benefits for patients in performing daily rehabilitation through remote monitoring. This is especially useful for those elderly patients with limited mobility and chronic condition. The system is also useful to reduce the travelling challenges for those patients from rural areas who need to perform daily rehabilitation exercises. Using an ordinary webcam for rehabilitation is very promising because it enables the patients to adopt in-home rehabilitation without additional cost.

The proposed system is validated to operate well and is able to detect the variation of the subject’s movements in both functional tests. Since PoseNet is completely free and can successfully capture all the skeleton points, PoseNet may be a more economical solution than the Microsoft Kinect for in-home rehabilitation system, helping to reduce the patient’s financial burden. In addition, the built-in statistical computation plays a crucial role in the rehabilitation procedure to assist the doctors in performing deeper analysis in determining the patient’s recovery rate. The functional tests also demonstrated the usefulness of the statistical results for the subjects to monitor their progress on different days.

Having completed functional tests, the system is ready for clinical trials that will be carried out in collaboration with the rehabilitation centers to evaluate the proposed telehealth system for actual in-home rehabilitation practices. The clinical trials will further investigate the effectiveness of the PoseNet-based in-home rehabilitation to meet the patients’ needs while also maintaining patient safety, study integrity, and data quality. Subsequent future work includes introducing a 3D avatar that can demonstrate the rehabilitation exercises more vividly and provide guidance on how to perform them more accurately. On top of this, background music can be added to make the rehabilitation exercise more appealing and relaxing for patients.