Next Article in Journal
Analysis of Factors Contributing to the Increase in 7Be Activity Concentrations in the Atmosphere
Previous Article in Journal
Associations of Central Sensitization-Related Symptoms with Low Back Pain-Related Factors and Work Status in Caregivers
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Measuring Mobile Phone Application Usability for Anticoagulation from the Perspective of Patients, Caregivers, and Healthcare Professionals

Department of Pharmacy, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung 833, Taiwan
School of Pharmacy, Kaohsiung Medical University, Kaohsiung 807, Taiwan
Department of Neurology, Kaohsiung Chang Gung Memorial Hospital and Chang Gung University College of Medicine, Kaohsiung 833, Taiwan
Center for Mitochondrial Research and Medicine, Kaohsiung Chang Gung Memorial Hospital and Chang Gung University College of Medicine, Kaohsiung 833, Taiwan
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2022, 19(16), 10136;
Submission received: 18 July 2022 / Revised: 15 August 2022 / Accepted: 15 August 2022 / Published: 16 August 2022
(This article belongs to the Section Health Care Sciences & Services)


Oral anticoagulants (OAC) are recommended for preventing stroke and systemic embolism in atrial fibrillation. Proper use is imperative for maximizing anticoagulation therapy’s effectiveness and safety. In preparation for the implementation of a smartphone-based SmartMed app (application) aiming to promote patient self-management, medication adherence, and data collection for patients on anticoagulation therapy, its usability assessment can ensure the value of OAC app development and adoption. We evaluated the SmartMed app’s usability using the System Usability Scale (SUS) and the app-specific domain of the Mobile App Rating Scale (MARS) for its perceived impact on taking OAC regularly. We recruited 25 OAC users and their home caregivers and 59 healthcare professionals, including pharmacists, nurses, and cardiac surgeons from one medical center and one regional hospital in Taiwan. All participants (n = 84) thought the SmartMed app was useful, with mean SUS and MARS scores of 81.49 (±14.42) and 4.65 (±0.49), respectively. Usability evaluation revealed that fewer experiences with smartphone apps and different healthcare professionals (pharmacists versus nurses or cardiac surgeons) were associated with lower SUS scores and perceived impact. Throughout the evaluation process, the SmartMed app’s design was considered helpful from multiple stakeholders’ perspectives. Further ongoing mobile technology supports are necessary to establish the SmartMed app’s effectiveness.

1. Introduction

Cardiovascular diseases (CVD) are the leading cause of death globally. According to the World Health Organization (WHO), CVD statistics, approximately 17.9 million people died from CVDs in 2019 [1]. Similarly, heart attack and ischemic stroke ranked as the second (84.2/100,000 person) and fourth (51.6/10,000 person) leading causes of death in Taiwan, respectively [2]. Oral anticoagulants (OAC) are indicated for patients with atrial fibrillation (AF) and valvular heart diseases and for secondary prevention of ischemic stroke with cardioembolic etiology. Despite the availability of effective preventive medications, nonadherence to anticoagulants is associated with CVD morbidity, mortality, and health-related quality of life [3]. It is imperative to tailor individual patients’ preventive medicines to achieve the best effectiveness and acceptance. Underanticoagulation may cause a higher risk of thromboembolic events, while overanticoagulation is related to a greater risk of bleeding [4,5]. Moreover, both drug-drug and drug–food interactions may affect the therapeutic range of anticoagulation and cause potentially adverse events [6]. Therefore, it is necessary to educate patients taking anticoagulants regarding the medical information details and importance of proper adherence. Collecting information directly from patients can improve communication and prevent healthcare providers from underestimating the severity of the disease symptoms and adverse drug reactions.
Emerging mobile-based health applications (mHealth apps) via smartphones are widely used in patients with cancer [7,8,9], chronic kidney disease (CKD) [10], diabetes [11], chronic obstructive pulmonary disease (COPD) [12], etc. for education, symptom tracking, personal alerts and reminders, and so on and have been demonstrated to support traditional healthcare delivery and promote self-management. mHealth apps for AF demonstrated improved patient knowledge and behavior, medication adherence, and quality of life [13,14]. Existing mHealth apps supporting anticoagulant therapy for AF have largely focused on providing decision support to clinicians and early detection of AF to patients [15,16,17]. Limited smartphone apps have been developed focused on anticoagulants as a recent narrative review of 12 studies focused on warfarin [18]. It remains unclear what types of additional functionalities support, such as mHealth apps via smartphones, will be both effective and sustainable for healthcare delivery to facilitate patient self-management to optimize anticoagulant adherence and disease management. Additionally, some experts believe that too much information can cause confusion for the patient and interfere with medication adherence. It is also thought that communication barriers between healthcare providers and patients with low health literacy may influence self-management promotion for prevention and disease management [19]. Self-management experiences and patient attitudes and perceptions toward mHealth apps are very limited in Asian populations. Evidence is lacking on the mHealth apps perceived as useful in promoting patient self-management among healthcare professionals (HCP). The current study aimed to explore the usability of an evidence-based mobile smartphone app for anticoagulation (SmartMed app) from the patients, patients’ home caregivers, and HCP perspectives to establish an explicit understanding of the potential value added for patients in the future and support from HCP in obtaining value from mHealth apps.

2. Methods

2.1. Study Overview

The smartphone-based app and cloud-based system included six-tiered modules of the Smart Medication (SmartMed) app to better meet the care pathway for delivering tailored information to OACs users [4,5]. The contents of the SmartMed app were developed and maintained by a team of clinical pharmacists and physicians with specialties of cardiology and neurology from Kaohsiung Chang Gung Memorial Hospital (CGMH). The SmartMed app was initiated by the Department of Pharmacy, Kaohsiung CGMH and funded by the Ministry of Science and Technology in Taiwan.
The study participants were invited from the development site—Kaohsiung Chang Gung Memorial Hospital (CGMH), a medical center—and the regional hospital, Chia-Yi CGMH from the network of CGMH in Taiwan [20]. Kaohsiung CGMH was a tertiary medical center with 2720 beds, Chia-Yi CGMH was district hospital with 1351 beds, and both were located in Southwest Taiwan. A minimum number of at least 12 participants in both patient and/or home caregivers and HCP groups was applied in this usability study considering human mHealth app interaction [21].
Figure 1 shows the evaluation process. The usability of the SmartMed app was evaluated from the perspective of the patient, patient proxy (or home caregiver), and HCP using the 10-item System Usability Scale (SUS) [22] and a specific domain of the Mobile App Rating Scale (MARS) for the perceived impact of the app [23]. This study was conducted between April and September 2021.

2.2. Key Features of the Evaluation Program

The SmartMed app was designed for OAC users (warfarin, dabigatran, rivaroxaban, apixaban, and edoxaban) to enhance patient medication adherence and engagement with obtaining self-management knowledge of medication use, laboratory, and special diet monitoring. In addition, the SmartMed app was expected to assist patients in better communicating with health professionals across different specialties and hospitals for tracking medication uses and management of disease symptoms. It included three-tier data components based on the current guidelines recommendation for anticoagulation therapy (i.e., medication use, testing results, and adding data), health calendar, education materials, additional links to making appointments (prescription refills, medical visits), and a link to upload electronic health information (Figure 2).
To avoid information bias, study participants were requested to complete five prespecified tasks to experience and evaluate usability: (1) scan the prescription QR codes to add medication use information into the SmartMed app (domain of medication use) and schedule a reminder for taking medication (domain of health calendar); (2) input laboratory results, such as INR for warfarin users or eGFR for kidney function (domain of testing results); (3) calculate daily consumption of vitamin K-rich vegetables (domain of testing results) [24]; (4) experience the graphical panel, which encompassed medication uses, laboratory results (e.g., INR), and vitamin-K-rich vegetable consumptions for self-monitoring (Figure 3); and (5) patient education program as suggested by the global clinical guidelines for OAC treatment and management [4,5]. The patient’s education materials included an interactive game, audio, and an informative booklet for OAC.

2.3. Participant Recruitments

2.3.1. Patient and Patient Proxy

Patients with chronic OAC therapy and their family caregivers were invited from an outpatient setting. Patient’s proxy (e.g., family caregivers) were recruited if the patient had a family caregiver to support their daily tasks, such as taking medications. A patient proxy was broadly defined as a family or nonfamily informal caregiver who assisted the older patient in healthcare-related activities in daily life. A well-trained assistant introduced the OACs app and followed the user manual in a participant-familiar language (e.g., local dialect) to help the focus group of patients and their family caregivers to gain a comprehensive understanding. Patients’ or their family caregivers’ perspectives were evaluated on a one-on-one basis.

2.3.2. Healthcare Professionals

A study aim was to explore healthcare professionals (HCP) recommendation and support to enhance self-management skills of patients with chronic anticoagulation therapy—in particular, warfarin use. We invited pharmacists who were responsible for patient education, providing drug information, and the quality of pharmaceutical care monitoring. Post-graduate physicians, cardiovascular surgeons, and nurses were invited to evaluate the SmartMed app based on their experiences and primary duty of patient care with valve surgery and warfarin treatment. HCP participants installed the app on their mobile phones and used the app during the testing session. Their perspective was evaluated in a group training and evaluation process.

2.4. Evaluation Procedure

A self-reported smartphone use experience survey included sociodemographic data (age category, sex, and Android/iOS system of smartphone) and frequency of app use before the evaluation process (Figure 1). A trained pharmacist held a video demonstration to guide the participants’ first access on the five tasks on the OAC app. The app’s user manual was provided for patients, home caregivers, and HCP at the beginning of the evaluation process.
The first test of usability was performed at the end of video demonstration to provide a preview of the feasibility of the evaluation process. After the hands-on operation for the five prespecified tasks, the second test of usability and expected impact of the app on medication adherence were evaluated (Figure 1). The whole evaluation process per person was between 40 and 50 min.

2.5. Study Outcome

The usability measurement of the SUS, which contained 10 questions, was the most widely used standardized questionnaire for assessing perceived usability with multiple dimensions, which included efficiency, effectiveness, and satisfaction. Each item was measured on a 5-point Likert scale that ranged from 1 (strongly disagree) to 5 (strongly agree) (Table 1). The total SUS scores ranged from 0 to 100, and a higher score represented better usability [22].
The Mobile App Rating Scale (MARS) is a multidimensional, validated instrument for assessing the quality of mHealth apps [23,25]. The MARS includes five dimensions (engagement, functionality, aesthetics, information quality, and subjective) and one app-specific dimension to assess the perceived potential impact of the app on the user’s awareness, knowledge, attitudes, intentions to change, and actual change in the target health behavior [23]. The SmartMed app targeted specific patients treated with chronic disease for prevention or management and not for general wellness of health. Thus, only the app-specific section of the MARS (six items) was adopted, and the target behaviors for all the items were contextualized as “taking anticoagulant regularly” in our questionnaire. All items were rated on a 5-point scale that ranged from 1 (strongly disagree) to 5 (strongly agree).

2.6. Statistical Analyses

Participant’s data were deidentified. Data were presented as numbers and percentages, mean and standard deviation (SD), median and interquartile range (25th, 75th percentile). Differences among comparison groups were assessed using analysis of variance or t or chi-squared tests. A paired t-test was used to assess the difference between Test 1 and 2 SUS scores to assess the rater’s internal consistency. Both the SUS and app-specific domain of the MARS scores differences between the groups were analyzed using a generalized linear model. All statistical analyses were performed using SAS 4.0 (Cary, NC, USA). A two-sided p-value of <0.05 was considered statistically significant.

3. Results

3.1. Participant Characteristics

A total of 14 patients, 11 family caregivers, and 59 HCP completed the usability evaluation of the SmartMed app. The participants’ main characteristics are shown in Table 1. Of these, 75% of HCP were younger than 50 years, whereas more than half of the patients and caregivers were older than 50 (p = 0.0161). For the mobile app usage frequency, 92.86% patients reported ≥8 times per week (i.e., every day), followed by 72.73% family caregivers and 57.63% HCP. Most patients (60%) underwent warfarin therapy.

3.2. Usability of the SmartMed App

Table 2 presents the overall score and each item of the SUS. The mean total score was 74.14 (±13.27) at Test 1 and 82.63 ± 15.81 at Test 2.
The initial SUS score was higher in the patient group (81.49 ± 14.42) than other groups (HCP: 72.00 ± 14.10, caregivers: 76.36 ± 9.96). For Test 2, HCP rated a significantly higher score compared to Test 1 score (p < 0.0001). However, patients and their homecare caregivers ratings of Tests 1 and 2 of SUS remained consistent.

3.3. Perceived Impact of the SmartMed App

The results of the app-specific MARS score are presented in Table 3. The overall mean scores (full score = 6) were 4.40 and 4.68 in Test 1 and Test 2, respectively. Increased MARS score at Test 2 (compared to Test 1) was shown in the HCP and patient groups (HCPs: 4.34 vs. 4.71, p < 0.0001; patients: 4.62 vs. 4.71, p = 0.0401). The app demonstrated comparable scores for all domains of app-specific MARS score and was slightly superior regarding “knowledge” (item 2).

3.4. Factors Associated with Healthcare Professionals’ Evaluation of the SmartMed App

The associations between the characteristics of HCP and the SUS as well as the app-specific MARS score are displayed in Table 4.
Mean SUS score from pharmacists in CY-CGMH (non-medical center) were higher than pharmacists who worked in K-CGMH (medical center) (β = 12.88, 95% CI = 4.66–21.09, p = 0.0028). Both nurses and physicians gave lower SUS scores compared to pharmacists in K-CGMH (nurse: β = −30.02, 95% CI = −45.60–−14.44, p = 0.0003; physician: β = −20.26, 95% CI = −34.62–−5.90, p = 0.0067). App-specific MARS scores were comparable among pharmacists who worked in different hospitals (K- and CY-CGMH) (β = 0.1, 95% CI, −0.14–0.34, p = 0.4245). Compared to pharmacists in K-CGMH, slightly lower app-specific MARS scores were rated by nurses and physicians (nurse: β = −1.20, 95% CI = −1.66–−0.74, p < 0.0001; physician: β = −1.00, 95% CI = −1.42–−0.58, p < 0.0001).
Apart from different healthcare professionals (nurses and physicians versus pharmacists), lower-frequency smartphone apps users tended to be associated with lower SUS scores and perceived impact.

4. Discussion

4.1. Principle Findings

The current study reported that patients, patient proxies, and HCP had similar positive opinions regarding the OAC smartphone app regarding its usability and perceived effectiveness in helping patients take anticoagulant regularly. This is the first patient-centered mHealth app developed for promoting self-management, medication adherence, and data collection for patients on anticoagulation therapy with Mandarin language in Taiwan.
The current study yielded several factors that affected the level of usability of the SmartMed app. Limited experiences with smartphone apps and unfamiliarity with anticoagulation therapy may lead to underestimation of the usability of the SmartMed app. For instance, most participants rated low for “I need to learn a lot of things before I could get going with the app” and “I think that I would need the support of a technical person to be able to use this app,” which suggested that both patients and their homecare givers could use the app without difficulties. However, three caregivers reported seldom use (<4 times/week), whereas 92% of the patients used the app frequently (>8 times/week). This might explain why caregivers rated the app with lower SUS scores than the patients themselves (76.59 vs. 80.54). Among health professionals, SUS Test 2 scores were higher than those for Test 1, which suggests that the SmartMed app is easy to use and that hands-on operation experiences changed the health professional’s perceptions of app.
Although a moderate and positive correlation between age and usability of medical mobile device has been reported [26], user’s age was not associated with levels of usability in the present study participants. For instance, most HCP respondents were younger than 50 years, in contrast to both patient and caregiver groups; the mean scores of the SUS at Tests 1 and 2 were not different across participant groups. The usability between the patients and their caregivers in our study was comparable with the findings from previous studies in cancer and geriatric populations [27,28].
HCP-related factors associated with patient self-management promotion are not well known. Unlike pharmacists, who rated the OAC app with high usability, nurses and physicians had the opposite opinion. The SUS scores for non-pharmacist HCP (51.67 ± 1.44 for nurses and 63.13 ± 21.35 for physicians) were below the average SUS score of 68 [29,30]. The diverse findings and perspectives among patients and physicians were consistent with previous studies [31,32]. In our healthcare system, physicians are usually too busy to offer comprehensive medication information for patients in outpatient settings. The features and functionalities of the patient-centered OAC app were primary developed based on pharmacists’ experiences with patient counseling and medication review to identify drug–drug and drug–food interactions and dosing adjustments [33]. This may possibly explain the low usability score rated by non-pharmacist HCP.
The usability or value of mHealth techniques assessment was multidimensional. Six factors have been suggested to influence the value of mHealth intervention: the process of service delivery, organizational structure and professionals involved, cost of different treatments, burden of staff for delivering the intervention, lifestyle habits of the population, and local preferences on the quality of patient care [34]. The present study highlighted a need to develop consensus from different experts to empower patients’ self-management of health and medication uses (i.e., the attributes of health information) using the mHealth technology. Future mHealth-integrated care models should involve stakeholders, particularly multidisciplinary health professionals involved in the patient care process.
Despite the context and features, formative evaluations should be conducted to test design assumptions. Although the mHealth apps have the potential to save money through the improvement of treatment adherence and behavior changes, the cost-effectiveness of mHealth app remains unclear [35,36]. Despite the resource intensity in the development stage, the study team also realized a need for timely mobile technology support for the SmartMed app end-users after implementation to facilitate ongoing accessible and effective OAC care. From the developer’s perspective, ongoing cost was therefore necessary to improve health service delivery and engagement of patient and clinical outcomes through mobile communication novelties as well as the demand of security and privacy. Validation for evaluating usability with different language versions is necessary to promote the mHealth accessibility in different countries. The clinical value of different medication categories involves the indication for patients using the app of what the significant medicine-related features in daily care that must be added into the app are.

4.2. Strengths and Limitations

This was the first study on usability evaluation for OAC mHealth app from the perspectives of patients on anticoagulation therapy, family caregivers, and multidisciplinary HCP. The study design included various HCP, which helped us be more open regarding the development of an integrated care model to promote patient self-management and shared decision making. The pharmacists enrolled in the study included those from hospitals located in different geographic areas of Taiwan (rural versus non-rural cities). Pharmacists in non-rural cities gave higher SUS scores, which indicated that the OAC app could be used in different hospital levels.
There were some potential limitations of this study. First, the number of participants among HCP was proportionally different, which might limit the generalization of the findings in non-pharmacist HCP. Second, patients and their home caregivers who enrolled in this study were not new OAC users. Baseline knowledge and medication adherences of these respondents were not collected. Some respondents may be used to taking OAC regularly and be familiar with all health information related to their disease, medication, and diet, while some may not. The usability results may not be extrapolated to new users of anticoagulation. In contrast, the usability from these respondents may reflect what patient-related characteristics can benefit from the adoption of the mHealth app. Concerning that usability may be biased by end-users’ various response rates over a period of time; the short evaluation duration (<1 h) may not be sufficient to elucidate the true value of the usability of the SmartMed app among the study participants. Last, we provided an open-ended question in the scoring stage of this study (“Please give us your opinion or comment on this patient-oriented app”). Patients’ motivation to use mHealth app and strategies for user engagement with the mHealth app were reported by some nurses and pharmacists. Future investigations to identify factors associated with adoption and long-term engagement with the SmartMed app are warranted to investigate the promise of the mHeath app in daily care.

5. Conclusions

The patient-centered SmartMed app was considered usable and to have potential for maintaining medication adherence on anticoagulation therapy from different stakeholder perspectives, which included the end-users—patients and their home caregivers. Further research is warranted to establish the app’s effectiveness on improving self-management ability and prevention of thromboembolism and major bleeding in a real-world setting.

Author Contributions

Conceptualization, C.-C.C., C.-H.S., C.-C.W., T.-K.L. and C.-N.H.; methodology, S.-W.W. and C.-N.H.; data curation, S.-C.T.; formal analysis, S.-W.W. and C.-N.H.; writing—original draft preparation, S.-W.W. and C.-N.H.; writing—review and editing, all authors. All authors have read and agreed to the published version of the manuscript.


This work was supported by the Taiwan Ministry of Science and Technology (MOST 107-2622-E-182A-001-CC3, 109-2622-E-182A-001-CC3) and Kaohsiung Chang Gung Memorial Hospital (CMRPG8M0881 to C.-N.H.).

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and was approved by the Institutional Review Boards (IRB) of the Chang Gung Medical Foundation at Taipei, Taiwan (permit number: 201505009B0C505).

Informed Consent Statement

Written informed consent has been obtained from the patients involved in the study. Informed consent was waived for healthcare professionals since participants were temporary.

Data Availability Statement

The data analyzed during this study are not permitted to be made publicly available.


We appreciated Ya-En Lin, who participated in the study team to develop the module of patient education program in the SmartMed app. We thank all the participants in the focus groups and individuals who facilitated the recruitment of the participants and data collection for this study.

Conflicts of Interest

The authors have no conflict of interest to declare.


  1. WHO. Cardiovascular Diseases (CVDs). Available online: (accessed on 8 July 2021).
  2. Ministry of Health and Welfare, Taiwan ROC. Leading Cause of Death in 2019. Available online: (accessed on 1 March 2022).
  3. Holbrook, A.; Schulman, S.; Witt, D.M.; Vandvik, P.O.; Fish, J.; Kovacs, M.J.; Svensson, P.J.; Veenstra, D.L.; Crowther, M.; Guyatt, G.H.J.C. Evidence-based management of anticoagulant therapy: Antithrombotic therapy and prevention of thrombosis: American College of Chest Physicians evidence-based clinical practice guidelines. Chest 2012, 141, e152S–e184S. [Google Scholar] [CrossRef] [PubMed]
  4. Li, Y.-H.; Wang, Y.-C.; Wang, Y.-C.; Liu, J.-C.; Lee, C.-H.; Chen, C.-C.; Hsieh, I.-C.; Kuo, F.-Y.; Huang, W.-C.; Sung, S.-H. 2018 Guidelines of the Taiwan Society of Cardiology, Taiwan Society of Emergency Medicine and Taiwan Society of Cardiovascular Interventions for the management of non ST-segment elevation acute coronary syndrome. J. Formos. Med. Assoc. 2018, 117, 766–790. [Google Scholar] [CrossRef] [PubMed]
  5. Steffel, J.; Collins, R.; Antz, M.; Cornu, P.; Desteghe, L.; Haeusler, K.G.; Oldgren, J.; Reinecke, H.; Roldan-Schilling, V.; Rowell, N. 2021 European Heart Rhythm Association practical guide on the use of non-vitamin K antagonist oral anticoagulants in patients with atrial fibrillation. EP Eur. 2021, 23, 1612–1676. [Google Scholar] [CrossRef] [PubMed]
  6. Wiggins, B.S.; Dixon, D.L.; Neyens, R.R.; Page, R.L.; Gluckman, T.J. Select drug-drug interactions with direct oral anticoagulants: JACC review topic of the week. J. Am. Coll. Cardiol. 2020, 75, 1341–1350. [Google Scholar] [CrossRef] [PubMed]
  7. Kapoor, A.; Nambisan, P.; Baker, E. Mobile applications for breast cancer survivorship and self-management: A systematic review. Health Inform. J. 2020, 26, 2892–2905. [Google Scholar] [CrossRef] [PubMed]
  8. Hou, I.C.; Lin, H.Y.; Shen, S.H.; Chang, K.J.; Tai, H.C.; Tsai, A.J.; Dykes, P.C. Quality of Life of Women After a First Diagnosis of Breast Cancer Using a Self-Management Support mHealth App in Taiwan: Randomized Controlled Trial. JMIR Mhealth Uhealth 2020, 8, e17084. [Google Scholar] [CrossRef] [PubMed]
  9. Nabi, J.; Cone, E.B.; Vasavada, A.; Sun, M.; Kilbridge, K.L.; Kibel, A.S.; Berry, D.L.; Trinh, Q.D. Mobile Health App for Prostate Cancer Patients on Androgen Deprivation Therapy: Qualitative Usability Study. JMIR Mhealth Uhealth 2020, 8, e20224. [Google Scholar] [CrossRef]
  10. Singh, K.; Diamantidis, C.J.; Ramani, S.; Bhavsar, N.A.; Mara, P.; Warner, J.; Rodriguez, J.; Wang, T.; Wright-Nunes, J. Patients’ and Nephrologists’ Evaluation of Patient-Facing Smartphone Apps for CKD. Clin. J. Am. Soc. Nephrol. 2019, 14, 523–529. [Google Scholar] [CrossRef]
  11. Pamungkas, R.A.; Usman, A.M.; Chamroonsawasdi, K.; Abdurrasyid. A smartphone application of diabetes coaching intervention to prevent the onset of complications and to improve diabetes self-management: A randomized control trial. Diabetes Metab. Syndr. 2022, 16, 102537. [Google Scholar] [CrossRef]
  12. Hardinge, M.; Rutter, H.; Velardo, C.; Shah, S.A.; Williams, V.; Tarassenko, L.; Farmer, A. Using a mobile health application to support self-management in chronic obstructive pulmonary disease: A six-month cohort study. BMC Med. Inf. Decis. Mak 2015, 15, 46. [Google Scholar] [CrossRef]
  13. Lane, D.A.; McMahon, N.; Gibson, J.; Weldon, J.C.; Farkowski, M.M.; Lenarczyk, R.; Watkins, C.L.; Dilaveris, P.; Caiani, E.G.; Potpara, T.S. Mobile health applications for managing atrial fibrillation for healthcare professionals and patients: A systematic review. Europace 2020, 22, 1567–1578. [Google Scholar] [CrossRef]
  14. Guo, Y.; Chen, Y.; Lane, D.A.; Liu, L.; Wang, Y.; Lip, G.Y.H. Mobile Health Technology for Atrial Fibrillation Management Integrating Decision Support, Education, and Patient Involvement: mAF App Trial. Am. J. Med. 2017, 130, 1388–1396.e1386. [Google Scholar] [CrossRef]
  15. Tran, K.; Leblanc, K.; Valentinis, A.; Kavanagh, D.; Zahr, N.; Ivers, N.M. Evaluating the Usability and Perceived Impact of an Electronic Medical Record Toolkit for Atrial Fibrillation Management in Primary Care: A Mixed-Methods Study Incorporating Human Factors Design. JMIR Hum Factors 2016, 3, e7. [Google Scholar] [CrossRef]
  16. Stephan, L.S.; Dytz Almeida, E.; Guimaraes, R.B.; Ley, A.G.; Mathias, R.G.; Assis, M.V.; Leiria, T.L. Processes and Recommendations for Creating mHealth Apps for Low-Income Populations. JMIR Mhealth Uhealth 2017, 5, e41. [Google Scholar] [CrossRef]
  17. Ayyaswami, V.; Padmanabhan, D.L.; Crihalmeanu, T.; Thelmo, F.; Prabhu, A.V.; Magnani, J.W. Mobile health applications for atrial fibrillation: A readability and quality assessment. Int. J. Cardiol. 2019, 293, 288–293. [Google Scholar] [CrossRef]
  18. Jang, I. A Systematic Review on Mobile Health Applications’ Education Program for Patients Taking Oral Anticoagulants. Int. J. Environ. Res. Public Health 2021, 18, 8902. [Google Scholar] [CrossRef]
  19. Boonstra, M.D.; Reijneveld, S.A.; Westerhuis, R.; Tullius, J.M.; Vervoort, J.P.; Navis, G.; de Winter, A.F.J.P.E.; Counseling. A longitudinal qualitative study to explore and optimize self-management in mild to end stage chronic kidney disease patients with limited health literacy: Perspectives of patients and health care professionals. Patient Educ. Couns. 2022, 105, 88–104. [Google Scholar] [CrossRef]
  20. Shao, S.C.; Chan, Y.Y.; Kao Yang, Y.H.; Lin, S.J.; Hung, M.J.; Chien, R.N.; Lai, C.C.; Lai, E.C.C. The Chang Gung Research Database—A multi-institutional electronic medical records database for real-world epidemiological studies in Taiwan. Pharmacoepidemiol. Drug Saf. 2019, 28, 593–600. [Google Scholar] [CrossRef]
  21. Caine, K. Local standards for sample size at CHI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA, 7–12 May 2016; pp. 981–992. [Google Scholar]
  22. Lewis, J.R. The System Usability Scale: Past, Present, and Future. Int. J. Hum. –Comput. Interact. 2018, 34, 577–590. [Google Scholar] [CrossRef]
  23. Stoyanov, S.R.; Hides, L.; Kavanagh, D.J.; Zelenko, O.; Tjondronegoro, D.; Mani, M. Mobile app rating scale: A new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015, 3, e27. [Google Scholar] [CrossRef]
  24. Chang, C.H.; Wang, Y.W.; Yeh Liu, P.Y.; Kao Yang, Y.H. A practical approach to minimize the interaction of dietary vitamin K with warfarin. J. Clin. Pharm. Ther. 2014, 39, 56–60. [Google Scholar] [CrossRef]
  25. Terhorst, Y.; Philippi, P.; Sander, L.B.; Schultchen, D.; Paganini, S.; Bardus, M.; Santo, K.; Knitza, J.; Machado, G.C.; Schoeppe, S. Validation of the mobile application rating scale (MARS). PLoS ONE 2020, 15, e0241480. [Google Scholar] [CrossRef]
  26. Chaniaud, N.; Megalakaki, O.; Capo, S.; Loup-Escande, E. Effects of User Characteristics on the Usability of a Home-Connected Medical Device (Smart Angel) for Ambulatory Monitoring: Usability Study. JMIR Hum Factors 2021, 8, e24846. [Google Scholar] [CrossRef]
  27. Quinn, C.C.; Staub, S.; Barr, E.; Gruber-Baldini, A. Mobile Support for Older Adults and Their Caregivers: Dyad Usability Study. JMIR Aging 2019, 2, e12276. [Google Scholar] [CrossRef]
  28. Loh, K.P.; Ramsdale, E.; Culakova, E.; Mendler, J.H.; Liesveld, J.L.; O’Dwyer, K.M.; McHugh, C.; Gilles, M.; Lloyd, T.; Goodman, M.; et al. Novel mHealth App to Deliver Geriatric Assessment-Driven Interventions for Older Adults With Cancer: Pilot Feasibility and Usability Study. JMIR Cancer 2018, 4, e10296. [Google Scholar] [CrossRef]
  29. Sauro, J.; Lewis, J.R. Quantifying the User Experience: Practical Statistics for User Research, 2nd ed.; Morgan Kaufmann: Cambridge, MA, USA, 2016. [Google Scholar]
  30. Lewis, J.R.; Sauro, J. The Factor Structure of the System Usability Scale. In International Conference on Human Centered Design; Springer: Berlin/Heidelberg, Germany, 2009. [Google Scholar]
  31. Mrklas, K.J.; Barber, T.; Campbell-Scherer, D.; Green, L.A.; Li, L.C.; Marlett, N.; Miller, J.; Shewchuk, B.; Teare, S.; Wasylak, T.; et al. Co-Design in the Development of a Mobile Health App for the Management of Knee Osteoarthritis by Patients and Physicians: Qualitative Study. JMIR Mhealth Uhealth 2020, 8, e17893. [Google Scholar] [CrossRef]
  32. Barber, T.; Sharif, B.; Teare, S.; Miller, J.; Shewchuk, B.; Green, L.A.; Marlett, N.; Cibere, J.; Mrklas, K.; Wasylak, T.; et al. Qualitative study to elicit patients’ and primary care physicians’ perspectives on the use of a self-management mobile health application for knee osteoarthritis. BMJ Open 2019, 9, e024016. [Google Scholar] [CrossRef]
  33. Schnipper, J.L.; Kirwin, J.L.; Cotugno, M.C.; Wahlstrom, S.A.; Brown, B.A.; Tarvin, E.; Kachalia, A.; Horng, M.; Roy, C.L.; McKean, S.C.; et al. Role of Pharmacist Counseling in Preventing Adverse Drug Events After Hospitalization. Arch. Intern. Med. 2006, 166, 565–571. [Google Scholar] [CrossRef]
  34. Jurkeviciute, M.; Van Velsen, L.; Eriksson, H.; Lifvergren, S.; Trimarchi, P.D.; Andin, U.; Svensson, J. Identifying the Value of an eHealth Intervention Aimed at Cognitive Impairments: Observational Study in Different Contexts and Service Models. J. Med. Internet Res. 2020, 22, e17720. [Google Scholar] [CrossRef]
  35. Hazel, C.A.; Bull, S.; Greenwell, E.; Bunik, M.; Puma, J.; Perraillon, M. Systematic review of cost-effectiveness analysis of behavior change communication apps: Assessment of key methods. Digit. Health 2021, 7, 20552076211000559. [Google Scholar] [CrossRef]
  36. Luo, X.; Xu, W.; Ming, W.-K.; Jiang, X.; Yuan, Q.; Lai, H.; Huang, C.; Zhong, X. Cost-Effectiveness of Mobile Health–Based Integrated Care for Atrial Fibrillation: Model Development and Data Analysis. J. Med. Internet. Res. 2022, 24, e29408. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Study design and evaluation process.
Figure 1. Study design and evaluation process.
Ijerph 19 10136 g001
Figure 2. Main features of SmartMed app (KCGMH, Kaohsiung Chang Gung Memorial Hospital).
Figure 2. Main features of SmartMed app (KCGMH, Kaohsiung Chang Gung Memorial Hospital).
Ijerph 19 10136 g002
Figure 3. Example of the SmartMed App features.
Figure 3. Example of the SmartMed App features.
Ijerph 19 10136 g003
Table 1. Characteristics of study participants.
Table 1. Characteristics of study participants.
N of PatientHCP
(n = 59)
Patient (n = 14)Family Caregiver (n = 11)p Value *
Age group, year, n(%) 0.0161
Sex, n (%) 0.1161
APP usage frequency/week, n (%) 0.2189
System of mobile phone, n (%) 0.1522
Healthcare professionals, n (%) -
Pharmacist in K-CGMH3636(61.02)----
Pharmacist in CY-KCGMH1616(27.12)----
Patient’s age, year, mean (SD) --53.30(10.29)68.25(24.73)0.0831
OAC use, n (%) 0.1044
Drug Frequency, n (%) -- 0.6299
Once daily (QD)5--2(15.38)3(27.27)
Twice daily (BID)19--11(84.62)8(72.73)
* p value was based on chi-squared test with Fisher exact teat; OAC: oral anticoagulation; K: Kaohsiung (medical center); CY-CGMH: Chia-Yi Chang Gung Medical Hospital (regional hospital).
Table 2. Usability measure by SUS.
Table 2. Usability measure by SUS.
ItemContentOverall (n = 84) HCP (n = 59) Patient (n = 14) Family Caregiver
(n = 11)
Test 1Test 2p ValueTest 1Test 2p ValueTest 1Test 2p ValueTest 1Test 2p Value
1I think that I would like to use this product frequently4.14(0.75)4.39(0.68)0.00044.02(0.72)4.44(0.65)<0.00014.43(0.76)4.36(0.74)0.33564.45(0.69)4.18(0.75)0.1921
2I found the product unnecessarily complex2.14(0.95)1.83(0.98)0.00212.27(0.95)1.75(0.98)<0.00011.57(0.76)1.86(0.95)0.30192.27(0.90)2.27(1.01)1.0000
3I thought the product was easy to use4.04(0.80)4.37(0.77)0.00023.92(0.81)4.36(0.76)0.00024.43(0.76)4.50(0.76)0.33564.18(0.60)4.27(0.90)0.7560
4I think that I would need the support of a technical person to be able to use this product2.36(1.14)2.05(1.13)0.00392.63(1.13)2.20(1.19)0.00141.57(0.94)1.43(0.85)0.43461.91(0.70)2.00(0.89)0.7961
5I found the various functions in this product were well integrated4.21(0.71)4.40(0.75)0.00414.15(0.71)4.41(0.75)0.00314.36(0.74)4.43(0.85)0.58304.36(0.67)4.36(0.67)1.0000
6I thought there was too much inconsistency in this product1.90(0.77)1.48(0.69)<0.00012.12(0.74)1.51(0.73)<0.00011.36(0.50)1.43(0.65)0.33561.55(0.82)1.36(0.50)0.1669
7I imagine that most people would learn to use this product very quickly.3.99(0.83)4.24(0.80)0.00743.80(0.84)4.22(0.83)0.00034.64(0.50)4.36(0.93)0.10394.18(0.60)4.18(0.40)1.0000
8I found the product very awkward to use1.82(0.79)1.46(0.75)<0.00011.92(0.74)1.51(0.82)0.00021.36(0.84)1.21(0.43)0.33561.91(0.83)1.55(0.69)0.1669
9I felt very confident using the product4.18(0.75)4.46(0.70)0.00014.05(0.70)4.44(0.73)<0.00014.79(0.58)4.71(0.61)0.33564.18(0.87)4.27(0.65)0.5884
10I needed to learn a lot of things before I could get going with this product2.01(0.88)1.75(0.93)0.00972.20(0.88)1.85(1.01)0.01551.29(0.61)1.36(0.63)0.33562.00(0.77)1.73(0.65)0.0816
Odd items (1, 3, 5, 7, 9)15.56(3.07)16.87(2.96)<0.000114.93(3.11)16.86(3.12)<0.000117.64(2.24)17.36(2.71)0.485716.36(2.38)16.27(2.45)0.8461
Even items14.76(3.41)16.43(3.36)<0.000113.87(3.24)16.19(3.61)<0.000117.86(2.54)17.71(2.79)0.710015.36(2.98)16.09(2.30)0.3328
Total score74.14(13.27)81.49(14.42)<0.000172.00(14.10)82.63(15.81)<0.000181.07(8.42)80.54(11.01)0.720276.36(9.96)76.59(9.17)0.9187
Table 3. App-specific MARS for perceived impact.
Table 3. App-specific MARS for perceived impact.
ItemContentOverall(n = 84) Healthcare Professional(n = 59) Patient (n = 14) Family Caregiver (n = 11)
Test 1Test 2p ValueTest 1Test 2p ValueTest 1Test 2p ValueTest 1Test 2p Value
1Awareness: This app is likely to increase awareness of the importance of [taking anticoagulant regularly]4.45(0.65)4.68(0.58)<0.00014.38(0.69)4.68(0.63)0.00024.64(0.63)4.79(0.43)0.16484.45(0.52)4.55(0.52)0.3409
2Knowledge: This app is likely to increase knowledge/understanding of [taking anticoagulant regularly]4.43(0.63)4.74(0.49)<0.00014.40(0.62)4.76(0.50)<0.00014.57(0.76)4.86(0.36)0.16484.27(0.65)4.45(0.52)0.3409
3Attitudes: This app is likely to change attitudes toward improving taking anticoagulant regularly4.39(0.64)4.64(0.65)0.00104.33(0.66)4.68(0.68)0.00134.64(0.63)4.64(0.63)1.00004.27(0.65)4.45(0.52)0.3409
4Intention to change: This app is likely to increase intentions/motivation to address [taking anticoagulant regularly]4.42(0.64)4.68(0.54)<0.00014.33(0.66)4.73(0.52)<0.00014.71(0.47)4.64(0.50)0.58304.45(0.69)4.45(0.69)ND
5Help seeking: Use of this app is likely to encourage further help seeking for [taking anticoagulant regularly] (if bleeding or thrombus symptoms occurred)4.32(0.68)4.65(0.55)<0.00014.27(0.71)4.69(0.53)<0.00014.43(0.65)4.64(0.50)0.18944.36(0.67)4.45(0.69)0.3409
6Behavior change: Use of this app is likely increase/decrease [taking anticoagulant regularly]4.40(0.60)4.69(0.54)<0.00014.32(0.62)4.75(0.51)<0.00014.71(0.47)4.71(0.47)-4.36(0.67)4.36(0.67)ND
mean score (SD)4.40(0.54)4.68(0.49)<0.00014.34(0.58)4.71(0.50)<0.00014.62(0.42)4.71(0.41)0.04014.36(0.53)4.45(0.49)0.0816
ND: no difference; p value was based paired t-test.
Table 4. Factors associated with healthcare professional’s evaluation of the app.
Table 4. Factors associated with healthcare professional’s evaluation of the app.
SUSApp-Specific MARS
nMean(SD)β(95% CI)p ValueMean(SD)β(95% CI)p Value
Pharmacist (K-CGMH)3681.81(12.81)Reference 4.80(0.34)Reference
Pharmacist (CY-CGMH)1695.16(5.44)12.88(4.66–21.09)0.00284.94(0.15)0.10(−0.14–0.34)0.4245
Age group, years
≤351886.67(15.97)Reference 4.89(0.30)Reference
Female3582.79(15.63)Reference 4.70(0.50)Reference
Smartphone app uses/week, number of times
<1976.39(20.62)Reference 4.69(0.52)Reference
System of mobile phone
Android3283.83(16.60)Reference 4.71(0.57)Reference
K (Kaohsiung)-, CY (Chia-Yi)-Chang Gung Memorial Hospital located in a rural city (medical center) and non-rural city (regional hospital), respectively.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Wang, S.-W.; Chiou, C.-C.; Su, C.-H.; Wu, C.-C.; Tsai, S.-C.; Lin, T.-K.; Hsu, C.-N. Measuring Mobile Phone Application Usability for Anticoagulation from the Perspective of Patients, Caregivers, and Healthcare Professionals. Int. J. Environ. Res. Public Health 2022, 19, 10136.

AMA Style

Wang S-W, Chiou C-C, Su C-H, Wu C-C, Tsai S-C, Lin T-K, Hsu C-N. Measuring Mobile Phone Application Usability for Anticoagulation from the Perspective of Patients, Caregivers, and Healthcare Professionals. International Journal of Environmental Research and Public Health. 2022; 19(16):10136.

Chicago/Turabian Style

Wang, Shih-Wei, Chun-Chi Chiou, Chien-Hao Su, Cheng-Chih Wu, Shu-Chen Tsai, Tsu-Kung Lin, and Chien-Ning Hsu. 2022. "Measuring Mobile Phone Application Usability for Anticoagulation from the Perspective of Patients, Caregivers, and Healthcare Professionals" International Journal of Environmental Research and Public Health 19, no. 16: 10136.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop