Proposal of a Novel Framework in Korea for a Total Safe-Care Fitness Solution in the COVID-19 Era

Abstract

Physical activity is a crucial factor for maintaining not only physical health status, but vast amounts of research have shown its link with better mental health. Supporting the use of gyms for the safety of its practitioners is vital in the new norm and living with COVID-19. Therefore, in this study we show research supporting the development of a framework for a Total Safe-Care Fitness Solution based on a multimodal COVID-19 tracking system integrating computer vision and data from wearable sensors. We propose a framework with three areas that need to be integrated: a COVID-19 vaccine and health status recognition system (QR code scan prior to entry to the gym, and physiological signals monitored by a smart-band and a health questionnaire filled in prior to entry to the gym); an accident detection system (video and smart-band based); and a gym-user digital tracking system (CCTV and smart-band based). We show the proposed architecture for the integration of these systems and provide practical tips on how to implement it in testbeds for feasibility testing. To the best of our knowledge, this is the first proposed COVID-19 tracking system of use in gyms that includes a predictive model for accident detection for safer exercise participation through health monitoring.

1. Introduction

With the new variant of the COVID-19 virus Omicron emerging with the most mutations (about 50) of any of the variants so far, it is reported to be the most transmissible [1]. Therefore, it is more important now to have effective measures, such as mask wearing, self-isolation and quarantining if infected, as well as contact tracing and rapid testing are all needed to prevent the spread of the disease [2]. A study, which modelled the effect of 20% and 40% contact tracing combined with rapid testing, showed the significant slowdown in daily infection rate from 10 per 1000 people to two or three per 1000, respectively [2]. Furthermore, it is currently estimated that with an unrestricted pandemic infection approximately 60% of the US would result in 1.4 million deaths (primarily in the older population), costing about USD 6 trillion (equivalent to 30% of the US GDP). These data suggest that even a small reduction in the spread of COVID-19 can generate substantial savings [3]. However, the cost of an extensive shutdown of non-essential businesses is estimated to cost USD 7 trillion [3]. Therefore, the use of more effective contact tracing can help reduce the spread so that extensive and expensive (economically and socially) shutdowns of any non-essential businesses can be minimized.

As a COIVD-19 carrier can be infectious without symptoms, it is vital to be able to know who they have been in contact with prior to their positive COVID-19 test results. Contact tracing is useful to reveal the people that may have been infected due to close contact with the positive carrier for a period of up to 14 days prior to infection [4]. As it is a challenge for people to recall each person they met, accurate contact tracing is difficult. To estimate the probability of disease transmission, a risk-score for each of the contacts which includes various factors, i.e., meeting venue (indoors/outdoors), duration, proximity between people, number of people meeting, etc. is required [5]. As contact tracing is usually performed manually with health officials having to conduct interviews with the infected, automatic methods to help track their position and record who they encountered would be very useful [4].

With the COVID-19 pandemic continuation, there has been a vast amount of research looking at the various ways to prevent or ‘flatten the curve’ [4,5]. It is well known that to maintain optimum health, i.e., a strong immunity, physical activity is of paramount importance [6]. Even though home exercise is an effective way to maintain a high level of cardiovascular fitness and muscle strength, people find it challenging to maintain motivation to train hard on their own [7,8]. Even via online mediums such as Zoom, Noom, Ufit, etc., it is difficult to replace the experience of going to the gym [7]. In recent years, the increase in membership numbers shows this to be true even as more and more home training methods are increasing [9]. Recent home training methods such as Tonal, Mirror, Tempo Studio, and Echelon Reflect have a price ranging from approx. USD 2000 to 3000, excluding membership fees, annually. The features of these systems include online personal training features, a web camera for interaction with live instructors, and even 3D motion sensors to help the user maintain the correct posture during the exercises, which is ideal for beginners who need constant reminders to maintain safety during the exercises. These devices also pair via Bluetooth to measure and record heart rate for more accurate exercise prescription and help the user observe their heart’s health status.

From a recent survey in the Republic of Korea of 3000 adults that regularly attended the gym, there was an injury rate of 64.3% due to lifting weights that are too heavy (45.9%), slipping on the floor (23.9%), bumping into people (12.1%), an injury upon landing after a jump or dynamic movement (1.5%), and hitting into a piece of fitness equipment (7.6%) [10]. Most notably, even if a gym user received an injury, they received treatment advice not from their doctor or a medical specialist, but mostly from their friends, other gym members, and people they lived with [10]. This is concerning as proper medical treatment or appropriate first aid can promote a quicker recovery and help the users to stay safer. With the popularity of FDA-approved smart-band and smart watches, such Fitbit, the Samsung Galaxy watch, and the Apple Watch, monitoring of the heart rate is possible in real-time with feedback easily provided by the watch or the users smart phone [11,12,13]. Based on the accelerometers, gyroscopes, and heart rate monitor in smart phones and smart watches, they have been shown to accurately detect and recognize heart rate disorders, such as ventricular tachycardia [11], atrial fibrillation [12], and even falls and accidents [13]. Currently, there is no known safety monitoring system for gym users to prevent and detect accidents that occur in the gym.

Therefore, in this study, we outline the development of a framework for a Total Safe-Care Fitness Solution based on a multimodal COVID-19 tracking system, integrating video analysis and wearable sensors to be used in fitness centers for the safety of clients working out. The proposed framework aims to integrate three systems to help make participating in exercise in the COVID-19 era. The first is the COVID-19 vaccine and health status recognition system to record and control gym users’ potential status to spread COVID-19. The second is a CCTV camera and smart phone data-based accident detection system. The third is a UWB-based tracking system to record a gym user’s location to estimate their COVID-19 exposure risk due to their location movement route and recorded timeline in relation to any gym user that has been diagnosed with COVID-19.

2. Related Work

2.1. COVID-19 Vaccine Recognition System

There are many issues about having a COVID-19 vaccine passport ‘green pass’-type systems, such as the evolving scientific literature supporting and against their use, data privacy, discrimination against people who cannot have the vaccine and people who choose not to receive the vaccine [14]. Additionally, the legal concerns of implementing a COVID-19 vaccine pass has been questioned [15]; however, in the Republic of Korea, the entrance to most public places, such as restaurants, health clubs, etc., require the patron to show proof of vaccination, primarily by the COVID-19 vaccine app (COOV), which was made by the Korean Centers for Disease and Control. Currently worldwide, there are many blockchain-based digital applications, i.e., VaccineGuard, WIShelter, SafePass, DigiLocker, Covi-pass, Digital Health Pass, etc., on smart phones which show the details of a person’s COVID-19 vaccine, even providing a QR code that can be scanned for verification [15]. For data security and privacy of the users, blockchain technology is the most common method for the digital COVID-19 passes [15], but they can be a vulnerable way of keeping the personal identifiable information encrypted if they are connected with contact-tracing apps [16].

During times of national emergency, such as any infectious disease spreading, article 76-2 of the constitution is in effect to allow the Minister of Health to collect personal data to track the history of confirmed patients without a court order. To help protect concerns about privacy and surveillance, an additional paragraph was added to Article 34-2 of the Infectious Disease Control and Prevention Act [17] on the 29th of September 2020 which states: “personal information determined to be irrelevant to the prevention of infectious diseases by Presidential Decree, such as sex, age, and so on, shall be excluded”. Furthermore, article 34-2 is also in effect and the Minister of Health must disclose all relevant information to the public regarding any administrative measures in a response to epidemic situations. For the government to limit the number of people entering public and private buildings and facilities likes gyms, Article 49 of the Infectious Disease control and prevention act is used by the Korean government for implementing administrative public health measures.

Currently, there are no known gym-user tracking systems that screen gym users for COVID-19 vaccine status and their health status, i.e., temperature and any symptoms related to COVID-19. As COVID-19 symptoms can be used to indicate whether a person may have the disease, we expect that screening gym users for both their vaccine and health status will help ensure safer gym use.

2.2. Accident and Safety

Accidents and injuries that occur in the gym are multifaceted and it is reported to be caused by factors such as poor exercise technique, low fitness levels, cardiovascular disease, low strength, hard and slippery surfaces, age, and lack of warm up and exercise preparation [18] (Montalvo et al., 2017). Muscle imbalance and poor exercise techniques can exacerbate any joint related issues and can cause muscle strains, tears, and even ligament ruptures, especially if there is an overzealous gym user [1]. As a result, it has been highly recommended by Montalvo [18] and associates that gym users should be extra careful with their technique and should improve their skill, strength, and flexibility if they want to prevent shoulder injuries. Due to the paramount importance of correct exercise posture to prevent injury and to perform effective and safe exercises, there are numerous works published in the biomechanics literature and research examining how exercise posture stresses and strains bones, ligaments, and muscles [19,20,21]. This has led to the recent developments of having real-time camera-based systems (2D and depth cameras) [22,23] and IMU-based systems to provide feedback during exercise [24,25].

Cardiac arrest and stroke can be a major area of concern, as an ageing population tends to have more time free time after retirement and wants to increase their levels of fitness. In North America and Europe, the overall incidence of exercise-induced cardiac arrest varies between 500 to 1000 per year [26,27,28]. Although the frequency of injuries due to falls during indoor sports activities is not high, with only 1 fall occurring in 2018 in Korea [10], it can cause severe injuries, such as concussions and fractures, especially in females and older cohorts who suffer from lower bone density.

With the recent advancements in wearable technology, it is becoming more and more convenient to be able to continuously monitor important health-related metrics, such as blood pressure and electrocardiogram (ECG), which can synchronize with smart apps that can easily be used by medical professionals and health enthusiasts [29,30]. This has led to the recent developments of having real-time camera-based systems (2D and depth cameras) [22,23] and IMU-based systems to provide feedback during exercise [24,25]. According to Forbes [31], Apple, Huawei, and Samsung are the three most popular smart watches in a market estimated be about USD 487 billion and that has grown 7.5% from the year 2020 to 2021, with an expected growth of approximately 8% this year. Due to this constant increase, the number of smart watch users will continue to increase making the integration of these devices to safety monitoring systems essential.

2.3. Gym-User Digital Tracking System

Recent research shows that regular gym users, even with the loss of a person in their family, see exercising in a gym as a priority in their life to maintain health and wellness [32]. There are many systems based on different technologies that are used to track users and objects in different environments, such as RFID-based indoor tracking for the elderly [33], smart phones for indoor tracking [34], and RSSI-based real-time user location tracking for indoor and outdoor environments [35]. For sporting situations, trackers have been focused on wheelchair sports [36], basketball athletes’ movements [37], a UVB tracking system (WIMU Pro, Spain) for general indoor exercises [38], and a computer vision-based tracking system for indoor and outdoor sports [39]. However, these systems were mostly developed for reasons of safety for the elderly and for the development of training strategies in sporting situations [40,41]. There are currently no known systems that have been developed that are used for the tracking of people in a gym with the objective of estimating their risk of COVID-19 transmission.

3. Proposed System Framework

The main objective of the Total Safe-Care Fitness Solution was to help ensure the safety of gym users through the integration of the following three systems:

• A COVID-19 vaccine recognition system (QR code scan prior entry to gym).
• An accident detection system (video and smart watch based).
• A gym-user digital tracking system (CCTV, UWB and smart-band based).

The expected benefits from the integration of these systems will be to help gym users feel safer to be able to work out in the gym, and if there is a COVID-19 case, the tracking will help to identify any possible transmissible cases. With the gym smart monitoring system, any accidents that occur will be recorded and the authorities, i.e., the personal trainer in the gym and emergency services, will be informed. From the big data collected using the gym user’s position and their posture (video analysis) in collaboration with the screening data (health questionnaire and COVID-19 vaccine status), we will develop predictive algorithms to help reduce the probability of accidents and COVID-19 transmission in the gym. To prevent the transmission of COVID-19, we propose a real-time tracking system that provides feedback in the form of audio if two or more gym users are within 1.5 m of each other.

3.1. COVID-19 Vaccine Recognition System (QR Code Scan Prior Entry to Gym)

In the Republic of Korea, KakaoTalk is one of the most popular apps, similar in function to What’s App©; however, in addition to its main function of communication, it provides the ability to use it to verify your identity through a QR code for entry in restaurants, hotels, etc., log in information on various websites (via a coded message), send coupons for use in shops, such as a gift card in Starbucks, play games, conduct online banking, and much more. Recently, one’s COVID-19 vaccination status can be verified using the application as it is now linked to the official COVID-19 vaccine application of the Republic of Korea (COOV). We propose not only using this QR code to check on one’s COVID-19 vaccination status, but also to link the use of the gym with a health questionnaire to ensure the safety of all attending the gym. The gym user’s identity is required by law in Korea by the Korean Centers of Disease Control (KCDC) so as to be able to track the spread of people with COVID-19. On entry to any business, the KCDC enquires about their identity, the time they spent there, whether they have been abroad within the last 14 days, and whether they have a fever or any respiratory-related symptoms. With KakaoTalk (Figure 1) having access to huge amounts of personal data, the data for financial services and medical data will be encrypted to the highest available standard and stored separately to maximize protection in case of any security breach. Only when an epidemiological investigation (Figure 2) is requested shall the data be transferred to the KCDC for tracking, and every four weeks, the recorded data related to the visit of each place will be automatically deleted [10].

The COVID-19 vaccine administration authentication-based access management system verifies if the user has the vaccine or not. Currently, if users do not have a vaccine, then they are not granted access to public areas. The QR code is linked to the “Electronic Access List” (Figure 3) which stores the information for four weeks and then automatically destroys it after 28 days pass. When a user scans the QR code into the “Facility Manager” app, the user’s QR code, business information, and visit time are saved to the Ministry of Health and Welfare, and the authentication result is delivered.

As this system is designed based on a progressive web application (PWA), if there is an application programming interface (API) service that can connect member information and a suitable messenger service, such as Kakao, then the system can be implemented in other countries with any other messenger service mainly used in that country. Additionally, the messenger service can be linked and used once the service provider, i.e., the messenger service, as the ability to register the member in the system and link them to the required health information provider, such as the American Center for Disease Prevention and Control (CDC) or the European CDC.

COVID-19 Vaccine Certification Access Management System Design

We propose the following way to access the COVID-19 vaccine for verification (Figure 4). For this system, the Kakao Cloud System protects and manages data based on the Enterprise Endpoint Encryption (E3) system, which is an end-to-end encryption-based data protection system for enterprises (ISO 27001, 27017, 27018, 27701, 27799 certification and CSA Star certification).

For the pre-screening of gym users, each person has to fill out the Korea Centers for Disease Control and Prevention COIVD-19 symptoms-related questionnaire, and the Physical Activity Readiness Questionnaire (Par-Q), with additional physiological data, will be collected using the smart watch to monitor heart rate, etc. For the Korea Centers for Disease Control and Prevention COIVD-19 symptoms-related questionnaire, if anyone answers yes, then the person will not be able to enter the gym until they receive a negative PCR result, valid for 72 h. For the Par-Q questionnaire, if anyone answers yes, then they need to receive medical approval prior to exercising. Additionally, the Par-Q answers can highlight if there is a gym user that requires extra caution and needs to be carefully monitored while working out, i.e., someone with high blood pressure, diabetes, or any other disease or illness that can be affected by exercise.

The Korea Centers for Disease Control and Prevention COIVD-19 symptoms questions are:

• Do you have a fever above 37.5°?
• Do you have any respiratory symptoms (cough, phlegm, sore throat, runny nose) within the past 14 days?
• Do you have a muscle pain, chill, vomiting and diarrhea, headache within the past 14 days?
• Have you been subject to self-isolation within the past 14 days as a result of overseas visits?
• Have you been in contact with confirmed patients, close contacts, and self-quarantine patients within the past 14 days?

3.2. Accident Detection System (Video and Smart-Band Based)

Based on the work of previous studies, we propose an accident detection based on both vision-based analysis for tracking the person’s movement characteristics, and a smart watch to measure their physiological status. To prepare this system, we plan a step-by-step procedure as follows to extract the requirements.

(1) Perform detailed literature research, including national safety authority reports and published research articles about the safety requirements of indoor sporting facilities.

(2) Examine the types of situations that are most prone to cause injury in different parts of Korea, urban and suburban. Literature, brainstorming, questionnaires, and interviews (Delphi and in-depth group and one-on-one) with sports-science experts and stakeholders will be carried out to extract any practical methods of preventing injury.

(3) Based on the legal requirements (from step 1) and the users’/stakeholders’ requirements (from step 2), we will develop a CCTV and smart watch integrated system. This system will then be tested in two testbeds in Korea, the capital and the second largest city. For the system to be successful in the field, it will have to be user friendly, easy to operate, and have a team which will install it so that the gym staff and owners will be able to use it conveniently.

(4) After testing the system in the field, a questionnaire and in-depth interviews will be carried out to check for the usability and feasibility of the system. Additionally, a SWOT analysis will then be performed to further improve the system and to look for any user difficulties/issues.

3.2.1. Details of the Development of Accident Monitoring System

The system consists of three main parts: pre-processing, integration of the multimodal data, and the analysis module (Figure 5). The pre-processing data is designed to gather all the data and convert it into supported formats, such as XML, JSON, and Query. The multimodal data processing platform has three main functions carried out by the controller, scheduler, and manager with an analysis module to control and apply the data analysis.

The scheduler is designed to monitor and inform the data stored in a database that can be processed in real-time to enable the analysis module work. It will be designed to provide multi-thread functions for real-time applications, and the API Manager function will support selective and complex requests for individual data sources (IoT, text, image, video, etc.) in the analysis module (Figure 6).

3.2.2. Details of the Development of the Dangerous Situation Recognition System

In the following, we present a system for the recognition of dangerous/risky situations and the development of an application of a multi-user human pose estimation system based on multiple cameras. Human pose estimation from multiple cameras, using OpenPose is well used, and, if done correctly, can provide excellent validity and reliability [41,42,43]. A multiple camera system will be used to record the gym user’s movement characteristics as shown in Figure 7. Then, the movement (OpenPose) data and their physiological data (smart watch) will be analyzed using machine learning and convolutional neural network techniques to classify the behavior of the gym user.

3.2.3. Application of Deep Learning to Classify Human Movement Patterns

We plan to apply a deep-learning model for indoor sports to classify user behavior pattern recognition based on human movement using 3D skeleton data extracted by OpenPose (Figure 8). Prior to set up in a gym, we will record a few popular weight training movements, such as the squat, lunge, bench press, etc., with a depth camera while collecting physiological data using a smart-band. The movement and physiological data will be synchronized using a customized program. Big data analytic methods will be used to classify movement data into safe and unsafe techniques. Additionally, the movement data and physiological data from the smart watch and the user’s medical history will be used to help predict any risky situations, by informing the gym facility’s manager of the potential risk associated with the particular user. For example, a user with high blood pressure will have their heart rate monitored and feedback will be provided if their heart rate rises too much. Additionally, if a user has a heart arrhythmia, tachycardia, or bradycardia [11,12,13], the smart watch will communicate a warning to the smart watch and the staff at the gym will be notified. Additionally, if the user falls, trips, bumps into another user or equipment then an alarm signal will be sent to the gym staff.

3.3. Gym-User Digital Tracking System

The proposed tracking system is based on the integration of three technologies: UWB, computer vision techniques from CCTV, and the smart watch (Figure 9). Each system has advantages and disadvantages, such as cameras are not able to observe users continuously if their view is blocked or occluded with another user. Therefore, by using the three systems together, we can overcome any shortcomings of a single system. Through the use of base stations, Wi-Fi, Bluetooth, and ZigBee, we plan to have Ultra-Wide Band Real-time Location System (UWB RTLS)—an ultra-precise positioning technology—operating at a frequency between 3.2 GHz to 4.8 GHz, which is an accuracy of about 3–5 cm [44].

Through the application development of a user-dependent COVID-19 tracking map (Figure 10), it will be possible to estimate the user’s COVID-19 exposure risk due to their location movement route and recorded timeline. Additionally, the distance between each of the gym users will be calculated, and this will help determine if the users had contact, i.e., were within 1.5–2 m of a user who had COVID-19. Real-time feedback will be given to the users by a vibration of the smart watch if they are too close (contact within 2 m) to another user. This data will help the Korean Centers for Disease Control when past tracking of COVID-19 patients is needed (Figure 10 and Figure 11).

4. Conclusions

This study highlights the architecture and details about how to improve the safety of gym users in a post-COVID-19 period through health monitoring. The Total Safe-Care Fitness Solution is based on the integration of the following three systems: a COVID-19 vaccine recognition system, an accident detection system, and a gym-user digital tracking system. Even if the COVID-19 pandemic is slowly becoming an endemic, the health monitoring technology for the safety of gym users will be applicable to help prevent injuries. Additionally, the system will be expandable to be able to monitor, detect, and help prevent injuries through pre-screening, and the continuous monitoring of gym users. Screening of health-related issues, for example cardiac-related risks, may be mitigated by the constant monitoring of gym users through the developed technologies and systems. Finally, the location information will be used to help improve the designs of gym facilities for more efficiency and a user-friendly layout.