Workout Detection by Wearable Device Data Using Machine Learning

Abstract

There are many reports that workouts relieve daily stress and are effective in improving mental and physical health. In recent years, there has been a demand for quick and easy methods to analyze and evaluate living organisms using biological information measured from wearable sensors. In this study, we attempted workout detection for one healthy female (40 years old) based on multiple types of biological information, such as the number of steps taken, activity level, and pulse, obtained from a wristband-type wearable sensor using machine learning. Data were recorded intermittently for approximately 64 days and 57 workouts were recorded. Workouts adopted for exercise were yoga and the workout duration was 1 h. We extracted 3416 min of biometric information for each of three categories: workout, awake activities (activities other than workouts), and sleep. Classification was performed using random forest (RF), SVM, and KNN. The detection accuracy of RF and SVM was high, and the recall, precision, and F-score values when using RF were 0.962, 0.963, and 0.963, respectively. The values for SVM were 0.961, 0.962, and 0.962, respectively. In addition, as a result of calculating the importance of the feature values used for detection, sleep state (39.8%), skin temperature (33.3%), and pulse rate (13.2%) accounted for approximately 86.3% of the total. By applying RF or SVM to the biological information obtained from the wearable wristband sensor, workouts could be detected every minute with high accuracy.

1. Introduction

There are many reports that workouts relieve daily stress and are effective with respect to improving mental and physical health [1,2,3]. Daily exercise provides a stable supply of serotonin and endorphins, promotes the production of brain cells, and relieves fatigue. Fatigue caused by workouts also leads to improved sleep quality [4,5,6,7,8,9,10,11,12,13].

In recent years, it has been mentioned that due to the stress and anxiety associated with the spread of the novel coronavirus, there has been a reduction in workout time and an increase in sedentary time [14,15,16,17,18]. Given the lack of exercise and sedentary lifestyle brought about by COVID-19, it is thought that maintaining body acceleration through workouts could lead to improved mental health. Regarding the correlation between exercise and mental health, the mechanism by which exercise affects the psychological state of humans has not yet been elucidated, and the pathogenesis of psychiatric disorders and the development of therapeutic methods have been investigated. However, there are limitations to studies using model animals. Therefore, it has been suggested that the analysis of long-term biological information obtained from humans under daily activities is also important for elucidating the mechanism of the onset of psychiatric disorders, which have been on the rise in recent years [19]. In today’s stressful world, effective intervention measures for mental health, such as workouts, are expected to become increasingly important.

Currently, the analysis of long-term biological information from humans under daily activities is attracting attention as a way to predict the occurrence and prognosis of diseases, drug efficacy, and so on. Additionally, practical applications such as electrocardiograms, pulse waves, and body acceleration obtained from wearable sensors are used. There is a need for rapid and simple analysis and evaluation methods for living organisms. In general, biometric data recorded by wearable devices are managed and analyzed by smartphone applications, and individual psychosomatic state characteristics are evaluated. Wearable sensors on the market mainly manage data and stream real-time data by directly integrating sensors and applications, and users need to use specific applications.

These applications estimate calorie consumption based on personal information such as a person’s height, weight, sex, and age. The automatic detection of the presence or absence of workouts is often based on the time, duration, and intensity of activities. Applications are personalized and provide guidance for incorporating workouts into one’s daily lifestyle based on personal exercise patterns and trends. In contrast, the workout detection algorithm is usually a black box and has the drawback that common and frequent workouts cannot be detected. For example, in the case of a wristband-type sensor, we have confirmed from preliminary experiments that it is difficult to detect aerobic exercises without wrist movement, such as cycling. Therefore, there is still little research on how well a workout can actually be detected by analyzing biological information obtained from sensors.

Therefore, we investigated whether it is possible to detect workouts by applying machine learning to pulse, activity level, and skin temperature data obtained from wearable sensors in daily life.

2. Method

2.1. Experimental Protocol

The data were measured using the wristband wearable sensor shown in Figure 1 (Silmee W22, manufactured by TDK) [20]. The subject was a healthy female (40 years old), received an explanation from the experimenter in advance, and after being fully satisfied with the content, agreed to participate in the experiment.

Biological conditions, such as the number of steps taken, amount of activity, activity type, sleep position, sleep state, skin temperature, pulse, UV level, and amount of conversation, were measured starting on 1 January 2022. Data were recorded intermittently for approximately 64 days and stopped at the end of April 2022. The biological information described above was recorded every minute; the workouts adopted for exercise were yoga and 57 workouts were recorded. The duration of each workout was one hour, and the subject had a habit of working out approximately four times per week.

The wristband-type sensor used in the experiment is waterproof and can be worn even when sweating due to workout. The continuous operating time is approximately 10 days, the sensor size is approximately 52 mm × 24.5 mm × 13.5 mm, and the weight is approximately 26 g. There are four types of sensors inside the device: pulse sensor, acceleration sensor, UV sensor, and temperature sensor (skin temperature). For pulse wave detection, volume pulse wave is measured with a green LED at a sampling frequency of 20 Hz. Acceleration is measured at 20.5 Hz from a 3-axis accelerometer and can be evaluated up to a range of ±4 G. UV sensor measures the intensity of the UV rays hitting the device and calculates the UV index value. Temperature sensor uses a digital semiconductor to measure skin temperature of wrist in the range of −10 °C to 45 °C. In addition, conversation time is detected from voice capture by the built-in microphone. With these sensors, it is possible to maintain the measurement accuracy of biometric information in daily life.

2.2. Data Analysis and Machine Learning Algorithms

The biometric data collected by the sensor were converted to a CSV file using special software (SilmeePro Wx, manufactured by TDK) and output. The operating system used was Windows 10 (64-bit). Machine learning was performed in Python (3.6.5) with Spyder (editor, 3.2.8) obtained from Anaconda, an open data science platform. We used scikit-learn (0.20.0) as the machine learning library.

The random forest (RF), support vector machine (SVM, kernel: rbf), and K-nearest neighbor (KNN) algorithms were used as classifiers for machine learning. The number of steps, amount of activity, active type, sleeping position, sleep state, skin temperature, mean pulse rate, UV level, and amount of conversation were used as feature values, and these feature values were extracted every minute. As data pre-processing, if even one of these feature values was missing, the data at that time were regarded as missing and deleted. Since 3416 min of workout data with the measurement accuracy maintained were extracted, the number of nine types of feature values for workout, awake (activities other than workout), and sleep was also adjusted to 3416.

Detection of workout was evaluated using k-fold cross-validation method. In this study, k = 4 because 75% of the 3416 min of data was training data and 25% was validation data. For the data set, 9 features of workout, wakefulness, and sleep were randomly shuffled and these were prepared for 854 min for each segment.

Table 1. Hyperparameters of classifiers for machine learning.

Classifier	Hyperparameter
RF	n_estimators	16, 32, 64, 128, 256, 512, 1024
	max_depth	16, 32, 64, 128, 256, 512, 1024
	criterion	gini
SVM	C	0.001, 0.01, 0.1, 1, 10, 100, 1000
	gamma	0.001, 0.01, 0.1, 1, 10, 100, 1000
KNN	n_neighbors	3, 5, 7, 9, 11, 13, 15, 17, 19, 21

n_estimators: number of trees; max_depth: depth of tree; n_neighbor: K value.

shows the hyperparameters used for machine learning. RF’s “n_estimators” and “max_depth” are number of trees and depth of tree, respectively. KNN’s “n_neighbor” is K value. Other parameters are default values. These hyperparameters are changed by grid search, and the optimal recall, precision, and F-score (harmonic mean of recall and precision) in each segment are calculated by the k-fold cross-validation method. The detection accuracy of the classifier was defined as mean value of 4 segments. Here, recall, precision, and F-score are given by the following Equations (1) to (3), where TP, FP, and FN are the true positive, false-positive, and false negative, respectively. In addition, in order to investigate how much the 9 features contribute to detection in RF, features importance was calculated using the importance function of scikit-learn for each segment.

Recall = TP/(TP + FN)

(1)

Precision = TP/(TP + FP)

(2)

F-score = 2 × Recall × Precision/(Recall + Precision)

(3)

3. Results

Table 2. Detection accuracy for each classifier.

Classifier	Recall	Precision	F-Score
RF	0.962 ± 0.023	0.963 ± 0.020	0.963 ± 0.021
SVM	0.961 ± 0.023	0.962 ± 0.022	0.962 ± 0.023
KNN	0.886 ± 0.117	0.893 ± 0.094	0.886 ± 0.106

The values of detection accuracy are mean and standard deviation of 4 segments.

shows the mean and standard deviation of detection accuracy for each classifier. The detection accuracy of RF and SVM was high, and the recall, precision, and F-score values when using RF were 0.962, 0.963, and 0.963, respectively. In addition, these values of SVM were 0.961, 0.962, and 0.962, respectively. In contrast, the detection accuracy of KNN was 0.886, 0.893, and 0.886, respectively, which was about 7–8% lower than those of RF and SVM.

Table 3. Confusion matrix for random forest and SVM.

Classifier: RF		Results of Detection
State		Workout	Awake	Sleep
Actual	Workout	3156	260	0
	Awake	131	3285	0
	Sleep	0	0	3416
Classifier: SVM		Results of detection
State		Workout	Awake	Sleep
Actual	Workout	3167	249	0
	Awake	140	3276	0
	Sleep	2	5	3409

The above confusion matrix is the sum of the confusion matrices calculated from each segment.

shows the sum of the confusion matrices of workout, awake, and sleep calculated from each segment in RF and SVM. Using RF, of the 3416 min of workout data, 260 min were falsely detected as awake. Similarly, 131 min of Awake data was misdetected as workout, and sleep was correctly detected in all cases. SVM was more prone to misdetect sleep than RF. Figure 2 shows the importance of features calculated using the feature importance function of scikit-learn in RF. Importance was expressed as the mean and standard deviation of 4 segments. Sleep state (39.8%), skin temperature (33.3%), and pulse rate (13.2%) accounted for approximately 86.3% of the total.

4. Discussion

Improving physical activity is an effective means for promoting health. In this study, we clarified that it is possible to detect workout time with a high accuracy, approximately 96.2%, under daily activities by RF and SVM using data collected from wristband wearable sensors without using existing applications.

The effective early detection of diseases, care prevention, and health promotion programs are expected to be provided through the use of wearable devices and data. The effectiveness of wearable devices has already been verified in gait support for Parkinson’s disease patients, rehabilitation for chronic stroke patients, and care for chronic diseases [21]. These directly contribute to the improvement of human quality of life (QOL). Although prospective comparative studies using machine learning for clinical evaluation of wearable sensor data are emerging [22], there are only a few studies that identify the workout. As the prevention of age-related frailty is a universal issue, the detection and prediction of exercise duration in long-term monitoring data will become important in the future. Conventional monitoring of training load has been mathematically complex, but simpler methods, such as the evaluation of derived variables, have been demonstrated [23]. In addition, continuously measured physical activity data using smartwatches has become widely applied to everything from the effects of gait training on functional performance to swimming motion solution [24]. However, there are still few studies to identify workout time in a simple and accurate manner, and this study has the potential to be applied to the objective evaluation of behavioral change and rhythm disorders.

In previous research, Bai et al. (2021) reported on wristband-type wearable sensors such as the Fitbit Charge 2 and Fitbit Alta (Fitbit Inc., San Francisco, CA, USA), and Apple Watch 2 (Apple Inc., Cupertino, CA, USA). Moderate-to-vigorous physical activity (MVPA) could not be detected [25]. In a consumer activity meter accuracy evaluation, 48 subjects wore a Yamax pedometer (Yamasa Tokei Keiki Co., Ltd., Tokyo, Japan), an ActiGraph GT3X+ (ActiGraph LLC, Pensacola, FL, USA), and a Polar H7 (Polar Electro Oy, Kempele, Finland) chest strap at the same time, and the number of steps taken, heart rate, and acceleration data under 24-hour daily activities were acquired and compared. As a result of evaluating the degree of agreement of the measurements, the number of steps taken and heart rate could be estimated reasonably, while the estimation of MVPA had a mean absolute error rate ranging from 45% to a maximum of 90%, suggesting that MVPA was significantly underestimated. In addition, there were variations in the accuracy of thresholds and algorithms for estimating active time and MVPA from biological information obtained from wearable sensors, and it was noted that it is necessary to continue to evaluate the effectiveness of devices.

The machine learning algorithm in this study adopted k-fold cross-validation (k = 4) and used 2562 min out of 3416 min of long-term data for training data. In addition, it is specialized machine learning that detects one’s own behaviors from one’s own learning data, and it is thought that the detection accuracy of not only Workout but also Awake can improve because the characteristics and habits of one’s own behaviors are reflected in learning. Wearable sensors are usually used by a single person to estimate the health, sleep, activity, and other conditions of that person, and we expect to compute predictions specific to the characteristics of a single person based on biological information. Additionally, this study used nine types of feature values for machine learning. The sleep state is an index that determines whether the subject is asleep or awake; it is evaluated in two stages and calculated using three-axis acceleration. The accurate detection of the pulse interval becomes difficult when the movement of the living body becomes intense, and the measurement accuracy of the pulse decreases and may be missing value. The wristwatch-type wearable device used this time has no defects in the surface temperature and three-axis acceleration, which are features that can be measured relatively easily compared to the pulse rate. These features are considered to be useful indices for detection of workouts from actions in daily life. In contrast, since the algorithm used in this study requires a large amount of training data, it is necessary to shorten the training data length for practical application. Additionally, yoga exercise was adopted as a workout, but it is necessary to verify whether or not various kinds of workouts can be detected by increasing the number of subjects.

Although the characteristics of the data used in the field of biometric information processing are fewer than those of log data collected from the internet, much biometric information is being collected with the progressive improvements of sensors. Camargo et al. (2021) [26] reported on the development and optimization process of a combined moving mode classifier and environmental parameter estimator using machine learning and wearable sensors. This paper shows that the system can estimate human gait with high accuracy. Gurchiek et al. (2019) [27] reviewed 46 papers that use regression algorithms to accurately estimate biomechanical time series in daily life from wearable sensor data, and found that improving estimation performance requires the use of complementary physics-based and machine learning techniques to develop open-source algorithms that use complementary physics-based and machine learning techniques to improve estimation performance. In this study, it was clarified that workouts are a phenomenon with characteristics that are relatively easy to capture, and workouts can be detected every minute by using RF or SVM. Workouts have a great impact on the mind and body, and the WHO recommends at least 150 to 300 min of physical activity per week to improve one’s health [28]. Therefore, if MVPA can be detected, it would be considered useful for the promotion of health and prevention of the need for long-term care. In this study, the possibility of detecting MVPA is also suggested because the workout can be detected with high accuracy every minute. Currently, the amount of data in open databases of wearable sensors is small due to various problems, including the difficulty of anonymous processing, but it is expected to expand in the future, as in the case of ECG databases. In the future, it is expected that the characteristics of each age and sex will be clarified by comparing with such databases.

5. Conclusions

In this study, we attempted to detect workouts from nine types of biological information, such as the number of steps taken, activity level, and pulse, obtained from wearable sensors using RF, SVM, and KNN. Consequently, we clarified that not only workouts but also awake and sleep periods can be detected with high accuracy every minute by using RF or SVM without using a dedicated application developed for wearable sensors. Sleep state, skin temperature, and pulse rate among the features used in this study are useful indices for the detection of actions in daily life. For future studies, it is necessary to shorten the training data. In addition, we will increase the number of subjects and verify whether workouts other than yoga exercise can be detect with the biological information.