Next Article in Journal / Special Issue
Need for UAI–Anatomy of the Paradigm of Usable Artificial Intelligence for Domain-Specific AI Applicability
Previous Article in Journal
A Literature Survey of How to Convey Transparency in Co-Located Human–Robot Interaction
Previous Article in Special Issue
Comparison of Using an Augmented Reality Learning Tool at Home and in a Classroom Regarding Motivation and Learning Outcomes
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Developing Usability Guidelines for mHealth Applications (UGmHA)

Information Technology Department, Faculty of Computing and Information Technology, King AbdulAziz University, Jeddah 21589, Saudi Arabia
*
Author to whom correspondence should be addressed.
Multimodal Technol. Interact. 2023, 7(3), 26; https://doi.org/10.3390/mti7030026
Submission received: 31 January 2023 / Revised: 20 February 2023 / Accepted: 24 February 2023 / Published: 28 February 2023

Abstract

:
Mobile health (mHealth) is a branch of electronic health (eHealth) technology that provides healthcare services using smartphones and wearable devices. However, most mHealth applications were developed without applying mHealth specialized usability guidelines. Although many researchers have used various guidelines to design and evaluate mHealth applications, these guidelines have certain limitations. First, some of them are general guidelines. Second, others are specified for mHealth applications; however, they only cover a few features of mHealth applications. Third, some of them did not consider accessibility needs for the elderly and people with special needs. Therefore, this paper proposes a new set of usability guidelines for mHealth applications (UGmHA) based on Quinones et al.’s formal methodology, which consists of seven stages starting from the Exploratory stage and ending with the Refining stage. What distinguishes these proposed guidelines is that they are easy to follow, consider the feature of accessibility for the elderly and people with special needs and cover different features of mHealth applications. In order to validate UGmHA, an experiment was conducted on two applications in Saudi Arabia using UGmHA versus other well-known usability guidelines to discover usability issues. The experimental results show that the UGmHA discovered more usability issues than did the other guidelines.

1. Introduction

In the recent era, the exponential evolution of technology is accelerating world development by making our life easier, more flexible and more effective. This development has been involved in the healthcare field in presenting electronic health (eHealth) technology. eHealth can enhance healthcare services using the internet and wireless technologies, usually through traditional desktops [1]. All over the world, countries are adopting eHealth services to improve the safety, standards and quality of their healthcare systems.
In relation to eHealth, a specific and essential part must be considered, called mobile health (mHealth) technology [2]. The concept of mHealth was defined and coined by Robert Istepanian in 2004 [3]. mHealth moves healthcare services from traditional desktops into mobile technologies, such as smartphones and wearable devices [4,5]. The key features of mHealth applications are the ability to monitor chronic disease by patients and facilitate communication between patients and doctors [6].
Despite the valuable benefits of mHealth applications, they can harm a patient’s life and health if they do not work as expected, such as erroneous results or wrong actions, whether by patients or clinical staff [7]. To the best of our knowledge, most mHealth applications are developed without using standard and specialized mHealth usability guidelines [8]. A set of usability guidelines is an evaluation method used to determine the extent to which a mHealth application’s interface is safe, efficient and usable by different kinds of people [9,10]. Accordingly, mHealth applications need to be regulated by applicable usability guidelines in order to assess their efficiency and safety for patients.
Many kinds of studies used various existing guidelines to design and evaluate mHealth applications; however, there are three main problems with these guidelines, which are:
  • Some of the guidelines are general, which means they are not specified for mHealth applications, such as Nielsen’s principles [11]. Since mHealth applications are intended to affect the health and the body of people, they need to have more rigorous and restricted guidelines.
  • Some of the guidelines are specified for mHealth applications; however, they cover only a few features of mHealth applications, such as:
    (a)
    Xcertia [12], which covered access to personal health information and notification features, while other features, such as self-monitoring, were not covered.
    (b)
    HE4EH [6], which covered self-monitoring, health goals and tips and biometric measurement features, while other features, such as consultations, were not covered.
    (c)
    Telemedicine recommendations [13], which covered consultations and booking appointment features, while other features, such as self-monitoring, were not covered.
  • Some of the guidelines did not consider accessibility features for the elderly and people with special needs, such as [11,13]. Accessibility is important to ensure that the application is accessible to people of different ages and needs.
In this study, our goal is to develop a comprehensive set of usability guidelines for mHealth applications, titled (UGmHA), that satisfy the following criteria:
  • Simple to follow and understandable by designers, developers and evaluators.
  • Accessible to all people, including elderly people and people with special needs.
  • Specialized in designing and evaluating mHealth applications that can cover different features. The following are examples of features gathered from different studies:
    • Self-monitoring for chronic diseases [14,15].
    • Online consultations [14,16].
    • Sharing data with health care providers (HCP) [14,15].
    • Booking appointments [16].
    • Biometric measurements [14].
    • Health goals and tips [14,15].
    • Notifications and reminders [14].
    • Access to personal health information [15].
This study has two main objectives. The first objective is to develop UGmHA using the systematic and formal methodology of Quinones et al. [17]. The second objective is to validate UGmHA by comparing UGmHA against the Nielsen and Xcertia guidelines, which are well-known and global guidelines. The experiment was conducted in Saudi Arabia by two authors of the paper using two local applications, which are named “Sehhaty” and “Sokry”.
The rest of the paper is organized as follows. Section 2 presents a background and literature review of mHealth-related guidelines as well as common development approaches and validation methods for new usability guidelines. Section 3 illustrates how UGmHA was developed and validated. Section 4 shows the results of the development and validation of UGmHA. Section 5 discusses the results obtained from the development and validation of UGmHA. Finally, our conclusions and future work are provided in Section 6.

2. Literature Review

This section presents the background information related to eHealth, mHealth and usability. We also review the existing guidelines related to mHealth applications, common development approaches and validation methods for new usability guidelines.

2.1. Electronic Health (eHealth) and Mobile Health (mHealth)

The term eHealth was established by Mitchell in 1999 [18] and is a broad term that integrates healthcare and technology to improve healthcare efficiency and reduce costs. The main objectives of eHealth are to improve patient safety and provide accurate diagnostics and appropriate treatment [19]. eHealth offers many different services and functionalities, such as electronic health records (EHR), archiving, electronic prescribing, order-entry systems and computerized decision support systems (CDS) [20]. Therefore, countries worldwide have adopted eHealth services to improve the standards and quality of their healthcare systems.
mHealth is a specific branch of electronic health (eHealth) technology. The “mHealth” terminology was coined and established by Robert Istepanian in 2004 [3]. mHealth can be defined as collecting and processing health data using smartphones, tablets and wearable devices [5]. mHealth applications are crucial since they enable a patient to monitor their own chronic disease and activities, support behavior changes for patients, improve their lifestyle and facilitate communication between patients and doctors [6,21]. Moreover, they provide doctors with greater mobility to help them care for their patients from different areas [5].

2.2. Usability and Guidelines

Usability is a key feature of a successful system because it is essential to create a well-designed and highly usable system [22,23]. According to ISO 9241-11 standard, usability is defined as “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [24].
Usability can be implemented and evaluated effectively using usability guidelines. The usability guidelines have the advantage that they can be applied early in the design and used to evaluate and discover usability issues before application release [25]. Nielsen’s principles are the first and best-known usability guidelines, which were introduced by Nielsen and Molich in 1990 [26]. These guidelines are considered traditional and general for most user interfaces [27,28].
However, there is evidence in the literature confirming that general guidelines are not appropriate to design and evaluate domain-specific applications and may overlook essential elements that need to be considered in specific applications [27,28,29,30]. Moreover, Hermawati et al. [28] indicated that any set of usability guidelines should depend on the specific requirements of the user, task and environment of use. This means that any change in these requirements may introduce new features and issues of usability that are not considered by general guidelines. Therefore, it is crucial to develop mHelath-specific guidelines to ensure that usability features related to mHealth applications are identified.
Usability guidelines may include checklist items that work as a guide to evaluate domain-specific applications [17]. The checklist items can add more details and specifications for usability guidelines to make them easily tailored to the specific features of an application [31]. Without using checklist items, there is the potential for the wrong association of usability issues to the corresponding guideline, missed domain-specific usability issues and unreliable findings due to the lack of evaluator consensus [32].
Several guidelines have been used by studies for the design and evaluation of mHealth applications.
Nielsen’s principles [11] comprises ten usability guidelines, which are:
  • Visibility of system status.
  • Match between system and the real world.
  • User control and freedom.
  • Consistency and standards.
  • Error prevention.
  • Recognition rather than recall.
  • Flexibility and efficiency of use.
  • Aesthetic and minimalist design.
  • Help users recognize, diagnose and recover from errors.
  • Help and documentation.
Despite Nielsen’s principles being globally well-known for designing and evaluating most user interfaces, they are general and not specialized for distinct features of mHealth applications [27,28]. Furthermore, accessibility features for the elderly and people with special needs are not considered. However, several studies used Nielsen’s principles as a basis for generating new usability guidelines by modifying or adding new guidelines or checklist items of domain-specific applications [27].
Roy et al. [33] discussed the framework of the Xcertia guidelines, which were founded by leaders of the healthcare industry, such as the Association of American Medical Colleges, U.S. Food and Drug Administration (FDA) and Healthcare Information and Management Systems Society (HIMSS). The Xcertia guidelines aim to fulfill two primary purposes, which are assessing how an mHealth application is designed to be safe and optimizing mHealth applications to be used by specified users within a specified environment [33].
As shown in Figure 1, Xcertia includes five key workgroups: privacy, security, content, usability and operability. Each workgroup includes a set of guidelines, and each guideline can be measured by a set of performance requirements or checklists. Nonetheless, this research will focus only on the “usability” workgroup because it is within our scope. The “usability” workgroup includes ten guidelines with checklist items for each guideline. Below are the guidelines of the “usability” workgroup of Xcertia:
  • Visual design.
  • Readability.
  • App navigation.
  • Onboarding.
  • App feedback.
  • Notifications, alerts and alarms.
  • Help resources and troubleshooting.
  • Historical data.
  • Accessibility.
  • Ongoing app evaluation.
Xcertia guidelines are developed mainly for mHealth applications and the “usability” in Xcertia considers some of the accessibility features for people with special needs. However, despite Xcertia guidelines being developed mainly for mHealth applications, most of the usability guidelines are still general and do not evaluate specific features of mHealth applications, such as self-monitoring, consultations, health advice and booking appointments.
Khowaja et al. [6] presented a modified set of specialized guidelines for mHealth applications called (HE4EH). HE4EH includes 25 guidelines with checklist items. The guidelines are:
  • Visibility of system status.
  • User control and freedom.
  • Match between system and real world.
  • Consistency and standards.
  • Error prevention.
  • Help users recognize, diagnose and recover from errors.
  • Recognition rather than recall.
  • Flexibility and efficiency of use.
  • Aesthetic and minimalist design.
  • Help and documentation.
  • Privacy.
  • Skills.
  • Pleasurable interaction.
  • Accessibility.
  • Compatibility between different platforms.
  • Minimized human–device interaction.
  • Physical interaction and ergonomics.
  • Readability and layout.
  • Non-interrupting app information visualization.
  • Content.
  • Display.
  • Navigation.
  • Interactivity.
  • Behavior change.
  • Self-monitoring.
However, the study only focused on self-monitoring of diabetes and blood glucose levels without considering hypertension, obesity, etc. [6]. Moreover, since the scope of the study was about self-monitoring and behavioral-change features, the other features related to mHealth applications, such as consultations and booking appointments, were not covered. Furthermore, the study included more than 16 guidelines, while it is recommended to keep the number of usability guidelines between 10 and 16 [17].
Aldekhyyel et al. [13] evaluated the usability of mHealth applications with telemedicine features deployed during the COVID-19 pandemic in Saudi Arabia. The mHealth applications were “Sehha”, “Cura” and “Dr. Sulaiman Alhabib”, which were evaluated using Nielsen’s ten principles with a five-point severity rating scale (SRS). Then, the study provided usability design recommendations for each application based on the discovered usability issues. However, since the scope of the study was about telemedicine applications, it focused only on consultations, appointments, etc., without considering other mHealth-specific features, such as self-monitoring and health advice. Moreover, accessibility features for the elderly and people with special needs were not covered.
Al-Razgan et al. [34] converted and adapted some of the mobile phone usability guidelines and recommendations to develop modified usability guidelines for mobile launchers designed for elderly people. The study included 13 guidelines, which are classified into three sections as shown below:
  • Look and Feel
    • Make elements on the page easy to read.
    • Easy recognition and accessibility.
    • Make clickable items easy to target and hit.
    • Use language and culture appropriate for the elderly; minimize technical terms.
  • Interaction
    • Provide clear feedback on actions.
    • Provide preferable gestures for the elderly.
    • Provide the elderly with information on launcher/elderly status.
    • Use conventional interaction items.
    • Ergonomic design.
  • Functionality
    • Provide functions that reduce elderly memory load.
    • Elderly does not feel lost or stuck (elderly control and freedom).
    • Prevent errors from occurring.
    • Provide necessary information and settings.
However, despite the study being mainly about mobile launchers’ usability for elderly people, we can make use of the guidelines to add the accessibility features for elderly people.
Arachchi et al. [35] integrated different learning theories and usability guidelines to develop an eLearning module for people with intellectual disability needs. The guidelines considered the specific abilities and context of learners, such as attention, mobility, cognitive capacity, learning ability and ability to read and write. The study incorporated nine guidelines with checklist items. The guidelines are:
  • Minimised distractions.
  • Easy to operate.
  • Readability and visualization.
  • Consistency.
  • Feedback.
  • Navigability.
  • Personalizing.
  • Easy to understand and relevant.
  • Learner-friendly.
However, despite the study being mainly about eLearning usability for intellectual disabilities, which is a part of special needs, we can make use of its guidelines to add accessibility features to support people with special needs.

2.3. Common Approaches for Developing New Usability Guidelines

Some common approaches are used to develop domain-specific usability guidelines, such as methodologies, literature review, usability problems, existing guidelines, interviews, theories and mixing processes [27]. However, the methodologies approach attracts the interest of most studies since it is the only formal approach that follows a systematic and clearly defined process. On the other hand, all other approaches are considered informal since their activities and processes are performed in a non-systematic way [27].
Different methodologies can be followed to generate a new set of usability guidelines, such as the Rusu methodology [10], which consists of six stages. This methodology is the most popular one [27,28] and is used by several studies, such as [36,37,38,39]. These studies emphasized the importance of the methodology to facilitate the generation of domain-specific guidelines. Nevertheless, Rusu et al.’s methodology [10] has been modified and adapted by the same authors with the support of usability experts and researchers to generate a new and examined methodology, here called Quinones et al.’s methodology [17]. Quinones et al.’s methodology consists of seven main stages, which are [17]:
  • Exploratory stage: to collect existing studies related to the main topics of the research, such as general or related usability guidelines, principles, application features and attributes.
  • Descriptive stage: to highlight the most important information of the previously collected studies to formalize the main concept of the research.
  • Correlational stage: to determine the characteristics that the usability guidelines of the specific domain should have based on traditional guidelines.
  • Selection stage: to keep, adapt or discard the existing sets of usability guidelines that were selected in the descriptive stage.
  • Specification stage: to formally specify the set of proposed guidelines using a standard template.
  • Validation (experimental) stage: to test the proposed guidelines against traditional ones through experiments.
  • Refinement stage: to refine the proposed guidelines based on the validation-stage results.
The advantage of employing Quinones et al.’s methodology is that it provides a standard template, which is used to explain more details about an individual guideline, including the ID, priority, name, definition, explanation, application features, examples, benefits, checklists and usability attributes. Moreover, it is possible to perform as many iterations as needed for all or some stages to improve the usability guidelines or the performance of experiments used for the validation stage [17].

2.4. Common Methods for Validating New Usability Guidelines

Three types of validation methods are proposed in Quinones et al.’s methodology. The first type is recommended by the methodology, and the other two types are for obtaining additional feedback as shown below: [17]:
  • Guideline evaluation: to check the proposed guidelines against traditional or specialized guidelines in terms of the number of discovered usability issues (The recommended method).
  • Expert judgment: using a questionnaire that assesses expert and evaluator perceptions of the proposed usability guidelines (To receive additional feedback).
  • User testing: to obtain users’ opinions about the proposed set of usability guidelines (To receive additional feedback).
However, the first type offers more opportunities to provide in-depth information about the effectiveness of the usability guidelines [17,28]. The comparison can be achieved by using a between-subject or within-subject study [40]:
  • Between-subject: each participant is exposed to only one condition of testing.
  • Within-subject: each participant is exposed to more than one of the conditions of testing.
The obtained results can be analyzed using some of the following criteria [17]:
  • Number of the discovered usability issues.
  • Number of discovered specific usability issues.
  • Number of severe usability issues.
  • Number of critical usability issues.
The majority of studies have applied the comparison method to validate their guidelines. However, the number of evaluators, compared guidelines and applications vary between the studies as shown in Table 1.

3. Materials and Methods

This section presents the methodology used to develop and generate UGmHA guidelines as well as the methods used to validate the resulting UGmHA guidelines.

3.1. Developing UGmHA

In the previous section, the literature review showed different guidelines used to evaluate mHealth applications. Some of these guidelines are general but used by various studies to evaluate mHealth applications, such as Nielsen’s principles [11]. On the other hand, some of the guidelines are used mainly to evaluate mHealth applications, such as Xcertia [12], HE4EH [6] and telemedicine mobile applications [13].
Each related work provides a diverse wealth of information and ideas around designing and evaluating mHealth applications. However, the guidelines and principles have limitations as described in Section 2. Inspired by previous work, we developed a set of comprehensive usability guidelines for mHealth applications (UGmHA), consisting of guidelines and checklist items generated from various studies and existing guidelines.
In order to develop our proposed guidelines effectively and efficiently, we employed the formal methodology of Quinones et al. [17], which was explained in Section 2. This methodology involves seven main stages, which were applied to fit within the context of our situation and research goals. The first five stages (exploratory, descriptive, correlational, selection and specification) were implemented by the first author of this paper and supervised by the other authors, while the sixth stage (validation) was performed by the first and third authors of the paper.

3.1.1. Stage 1: Exploratory Stage

In this stage, a systematic literature review was performed to collect information related to mHealth applications, consisting of application features, usability attributes, guidelines and recommendations. Since our study aims to make the applications accessible to different kinds of people, the reviewed literature is also related to elderly people and people with special needs.
In order to perform the systematic review, the method presented by Kitchenham and Charters [45] was employed using the following keywords:
  • For features: (“mHealth applications” OR “mobile health applications”) AND “features”.
  • For attributes: (“mHealth applications” OR “mobile health applications”) AND “attributes”.
  • For guidelines and recommendations: (“mHealth applications” OR “mobile health applications”) AND (“usability guidelines” OR “usability principles” OR “usability heuristics” OR “design recommendations”).
  • For accessibility guidelines: (“mHealth applications” OR “mobile health applications”) AND “accessibility guidelines”.
All keywords were used to search for relevant information using multiple databases, including MDPI, IEEE, Springer Link, JAMIA and JMIR.
The inclusion criteria for this study were: (1) studies related to mHealth applications; (2) studies related to the accessibility guidelines; (3) studies with a clear description of features, attributes, guidelines or recommendations; (4) studies that are publicly available; and (4) studies written in English. The exclusion criteria were (1) studies that were unavailable as a full-text; (2) studies not specified for mHealth applications or accessibility guidelines; (3) studies that did not provide enough information about targeted topics; and (4) survey papers.
As shown in Figure 2, a total of 54 studies were reviewed from databases, including features, attributes and guidelines related to mHealth applications and accessibility. We excluded eight studies due to the duplication of information. Then, we applied the inclusion and exclusion criteria based on the title, abstract and full text; a total of 34 studies additional were excluded. Finally, 12 studies were included for the goal of our study, consisting of 3 studies for features, 3 for attributes and 6 for guidelines.
The obtained information from this stage is shown below:
  • Features: these features are related to mHealth applications: [14,15,16], which will be discussed in the next stage.
  • Usability attributes, which include:
    ISO attributes (effective, efficiency and satisfaction) [46].
    Nielsen’s attributes (learnability, efficiency, memorability, errors and satisfaction) [47].
    PACMAD attributes, which combine both attributes of ISO and Nielsen in addition to the cognitive load attribute (effective, efficiency, learnability, memorability, errors, satisfaction and cognitive load) [48].
  • Usability guidelines and design recommendations, which include:
    Usability guidelines related to mHealth applications, which are Nielsen [11], Xcertia [12], HE4EH [6] and telemedicine mobile application recommendations [13].
    Usability guidelines related to elderly people [34].
    Usability guidelines related to people with special needs [35].
Section 2 presents more details regarding guidelines and design recommendations relevant to our study.

3.1.2. Stage 2: Descriptive Stage

In this stage, we highlighted and selected the essential information from the collected literature review in the previous stage.
  • For features: we gathered all features from [14,15,16] and summarized them to obtain a list of integrated features. Additionally, we added two more features related to our research, which were accessibility to elderly people and accessibility to people with special needs. Below are the selected features:
    • Booking appointments [16].
    • Consultation with HCP via text, voice messages and video calls [14,16].
    • Sharing data with HCP [14,15].
    • Access to personal health information [15].
    • Self-monitoring of chronic disease [14,15].
    • Allowing uploading and viewing of biometric measurements [14].
    • Graphic display of patient’s information [14].
    • Set health goals and treatment plan [14,15].
    • Track health progress [15].
    • Reminders and notifications [14].
    • Health tips and motivation [14,15].
    • Sharing health data with friends [15].
    • Interactive prompts [14].
    • Earn rewards [15].
    • Bluetooth technology connection [14].
    • Accessibility to elderly people.
    • Accessibility to people with special needs.
  • For usability attributes: we selected only the PACMAD [48] attributes since they combine attributes of both ISO [46] and Nielsen [47]. Below are the selected attributes and their definition based on PACMAD [48].
    Effectiveness: the ability to produce the desired outputs.
    Efficiency: the reduction of wasted materials, such as time, money and effort.
    Satisfaction: the fulfillment of the user’s needs and desires.
    Learnability: a user can learn how to use the application easily.
    Memorability: a user can remember how to use the application after a while.
    Errors: minimizing the user’s error rate while using the application.
    Cognitive Load: the ability to use a mobile application while doing daily activities.
  • For usability guidelines: we classified the studies into the main guidelines, which will be used as basic guidelines for the next stages, and checklist items, which help to make the main guidelines more specific to the mHealth applications:
    Main guidelines: we selected all of Nielsen’s principles [11] as well as the “usability” workgroup of Xcertia guidelines [12]. The reason for selecting these two guidelines is that Nielsen’s principles are well-known guidelines that are used to evaluate most user interfaces. Moreover, Xcertia is specified to evaluate mHealth applications in general and developed by authorized organizations in the US.
    Checklist items: we selected [6,12,13], which are related to mHealth applications. Additionally, we selected the guidelines related to elderly people [34] and people with special needs [35].
On the other hand, we discarded all other workgroups of Xcertia [12], which were “privacy”, “security”, “content” and “operability” since they are out of the scope of this study. Moreover, we discarded some checklist items in [6] that are very specific to diabetes and cannot be generalized to all chronic diseases, and we also discarded checklist items in [34,35] that cannot be used for mHealth applications. The results of this stage are shown in Table 2.

3.1.3. Stage 3: Correlational Stage

In this stage, the selected features, attributes and the main usability guidelines were matched as shown in Table 3. The purpose of this stage is to identify the specific features and attributes of mHealth applications that need to be evaluated by the main guidelines [17]. The process of matching should be controlled by features, which means that we chose to match the attributes that can measure a specific feature. Then, we chose the main guidelines (from Xcertia and Nielsen) that evaluate attributes with a feature as shown in Figure 3.
Let us take the booking appointments feature as an example: the most suitable attributes and guidelines can be matched with:
  • Attributes:
    Effectiveness: to measure if the appointment is booked at the right date and time selected by the user.
    Efficiency: to measure if this feature saves users’ time and effort without the need to go to the clinic.
  • Guidelines:
    Visibility of system status in Nielsen and App feedback in Xcertia: to evaluate booking appointments with the effectiveness attribute if the application provides feedback to the user regarding the right date and time of the appointment.
    Notifications and alarms in Xcertia: to evaluate booking appointments with the efficiency attribute if the application notifies a user after an appointment is booked. Thus, the users’ time and effort to remember the appointment are minimized.
    Historical data in Xcertia: to evaluate booking appointments with the efficiency attribute if the application saves previous and upcoming appointments. Thus, the users’ time and effort to call the clinic to ask for their appointments are minimized.
Another example, is the earn rewards feature, which can be measured by the satisfaction attribute because, when the user earns rewards for a good action or habit, they will be more satisfied with the application. However, there is no matched guideline from Nielsen and Xcertia to evaluate this feature (which will be discussed in the next stage).

3.1.4. Stage 4: Selection Stage

In this stage, an evaluation and determination were made for each main guideline identified in the previous stage [17]:
  • Keep: without any change if the guideline is clear and correctly matched to the specific feature of mHealth applications.
  • Adapt: if the guideline needs a change to be more related to the mHealth applications.
  • Eliminate: if the guideline is redundant or not relevant to the features of mHealth applications.
  • Create: if a new guideline is required to evaluate the specific feature of the mHealth applications that the main guidelines cannot evaluate.
In this sense, we adapted all of Nielsen’s guidelines as well as some of the Xcertia guidelines, including onboarding, notifications, historical data and accessibility by adding more checklist items from different studies to be more specific to mHealth applications. Furthermore, we kept the ongoing app evaluation guideline in Xcertia. On the other hand, we eliminated the other Xcertia guidelines, which were visual design, readability, app navigation, app feedback and help resource and troubleshooting because we considered them and their checklist items as parts of Nielsen’s usability guidelines.
Therefore, we merged the checklists of each eliminated guideline in Xcertia with similar guidelines in Nielsen. For example, we found that the checklist items of visual design guideline in Xcertia satisfy the definition of Nielsen’s consistency and standards, recognition rather than recall and error prevention guidelines. Hence, the visual design guideline was eliminated, and its checklist items were moved to the corresponding guideline in Nielsen. This process was applied to all eliminated guidelines in Xcertia. However, F12 and F14 features were not covered by any of the selected main guidelines.
That is why we needed to create a new guideline (Interactive and motivations) that evaluates these features. After that, we determined the applicability of each guideline to identify its importance, whether (1) useful, (2) important or (3) critical based on the guideline definition and checklists. The results of this stage are described in detail in Table 4, Table 5 and Table 6.

3.1.5. Stage 5: Specification Stage

In this stage, the new set of usability guidelines was formally defined. Quinones et al. [17] recommended keeping the number of usability guidelines between 10 and 16 because it is difficult to employ many guidelines in practice and recommended using checklists to add more details to the guidelines. Therefore, we integrated the guidelines of Nielsen [11] and Xcertia [12] to form the main guidelines of UGmHA and keep them within the recommended range.
However, since Nielsen’s guidelines are general and most of Xcertia guidelines and checklist items are not specific to the features of mHealth applications, we adapted and extended them by adding more checklist items from different studies, which were [6,13,34,35]. The reasons for adding these checklist items are to make the main guidelines more specific to the features of mHealth applications and support the involvement of accessibility features for the elderly and people with special needs. The obtained results of this stage are 16 usability guidelines as shown in Table 7 in addition to 154 checklist items classified by the main guidelines.
The resulting template of this stage will be discussed in Section 4.

3.2. Validating UGmHA

3.2.1. Stage 6: Validation Stage

The validation stage was divided into three phases as suggested by Quinones et al.’s methodology [17]:
  • Guideline evaluation phase: check the new set of UGmHA against Nielsen and Xcertia guidelines regarding the number of identified usability issues and severity.
  • Expert judgment phase: evaluate the usefulness, efficiency and effectiveness of UGmHA guidelines by questionnaire or interview. Then, the results will be used in the seventh stage (refining stage) of Quinones et al.’s methodology [17].
  • User testing phase: enhance the usability of an existing application using refined guidelines of UGmHA. Then, perform user testing for the enhanced application.
However, the first phase of validation (guideline evaluation) will be presented in this paper, while the second and third phases (expert judgment and user testing) are ongoing work that will be published as soon as they are finished. Below are the details of the first phase of the validation.
  • Experiment Design:
    Participants: The evaluators who participated in this experiment are two of the authors of this research. The first evaluator was Mrs. Eman Nasr, a master’s student in the Faculty of Computing and Information Technology (FCIT) at King Abdulaziz University with a medium knowledge of usability evaluation. The second evaluator was Dr. Duaa Sinnari, an assistant professor in FCIT at King Abdulaziz University with high experience in usability evaluation and human–computer interaction (HCI).
    Guidelines: UGmHA was tested against Nielsen’s principles and the Xceria guidelines. The reason for selecting these two guidelines is that UGmHA is based mostly on Nielsen’s and Xcertia guidelines and because Nielsen’s principles are well-known guidelines and Xcertia is specified to evaluate mHealth applications.
    Applications: We selected the “Sehhaty” and “Sokry” applications. Figure 4 shows the home screen of (a) the first application (Sehhaty) and (b) the second application (Sokry) on the Android operating system in the Arabic language. The reasons for selecting these two applications are because they are simple to use and to achieve diversity between the selected applications based on governmental/ private, functionality and popularity as shown in Table 8.
  • Experiment Procedure: To assess the performance, each of the evaluators evaluated both applications individually in a within-subject study that compared the UGmHA, Xcertia and Nielsen guidelines. First, the evaluators evaluated the “Sehhaty” and “Sokry” applications individually using Nielsen’s guidelines and wrote down the discovered usability issues for each application. Second, the assessments of each evaluator using Nielsen’s guidelines were grouped together to generate a single list of usability issues for each application.
    Third, the evaluators worked together to rate each usability issue in the generated list based on the Severity Rating Scale (SRS). The SRS has five points (0–4) that can be used to prioritize and estimate which usability issues are important to be fixed as shown in Table 9 [51]. Then, the same aforementioned processes were repeated using Xcertia followed by the UGmHA guidelines. From this experiment, we produced six lists of usability issues, which are:
    • Using Nielsen’s guidelines: list of usability issues of the “Sehhaty” application.
    • Using Nielsen’s guidelines: list of usability issues of the “Sokry” application.
    • Using Xcertia guidelines: list of usability issues of the “Sehhaty” application.
    • Using Xcertia guidelines: list of usability issues of the “Sokry” application.
    • Using UGmHA guidelines: list of usability issues of the “Sehhaty“ application.
    • Using UGmHA guidelines: list of usability issues of the “Sokry” application.

3.2.2. Stage 7: Refining Stage

In this stage, the expert and user feedback obtained from stage 6 (validation stage) are used for the refining [17]. The process of refining is described as follows:
  • Document the changes that should be made to the guidelines.
  • Define the guidelines to be created, refined or deleted.
  • Iterate and repeat some stages again, if necessary.
However, since the experts’ judgment and user testing in the validation stage are future work. Accordingly, the refining stage is future work for this study.

4. Results

This section presents the results of developing and validating the UGmHA guidelines.

4.1. Results of UGmHA Development

The standard template of Quinones et al. [17] can contain different elements. Still, it depends on the researcher to determine whether to use the complete template or choose the needed elements [52]. Therefore, we decided to select the ID, priority, guideline name, guideline definition, application features and checklist items as described in Table 10. Appendix A shows a detailed description of each guideline in UGmHA.

4.2. Results of the UGmHA Validation

After receiving the usability results from the sixth stage (validation stage) described in Section 3, we defined the following findings:

4.2.1. Number of Usability Issues among the Three Guidelines

As depicted in Table 11, the numbers of usability issues discovered in the “Sehhaty” application were 73 using UGmHA, 22 using the Xcertia guidelines and 17 using Nielsen’s guidelines. On the other hand, the numbers of usability issues discovered in the “Sokry” application were 95 using UGmHA, 28 using the Xcertia guidelines and 25 using Nielsen’s guidelines.

4.2.2. Severity of Usability Issues among the Three Guidelines

As shown in Table 12 and Figure 5, the number of issues with a “Catastrophic” rating was the highest using all guidelines for both applications. Except for Nielsen’s guidelines, on “Sehhaty”, the highest number of issues were rated with a “Major” rating. However, there were no issues with the “no problem” rating for all guidelines on both applications.
For the UGmHA guidelines, as shown in Figure 6, the most unapplied guidelines in “Sehhaty” were accessibility with 11 issues and then user control and freedom and aesthetic and minimalist design with 9 issues, while the most unapplied guidelines in “Sokry” were the visibility of system status and accessibility with 13 issues and then user control and freedom with 11 issues.
However, the least unapplied guidelines in “Sehhaty” were error diagnosis and recovery with one issue, while the least unapplied guideline in “Sokry” was match between system and real world with one issue.

5. Discussion

This section presents the discussion of results obtained from developing and validating the UGmHA guidelines.

5.1. UGmHA Development Discussion

If we compare the resulting guidelines of UGmHA with the Nielsen and Xcertia guidelines, we find that the number of guidelines in UGmHA (16 guidelines) is greater than the number of guidelines in Nielsen (10 guidelines) and Xcertia (10 guidelines). The reason is that UGmHA integrated Nielsen and Xcertia by adapting and eliminating their guidelines through Quinones et al.’s methodology [17] as well as created a new guideline that covers features that Nielsen and Xcertia did not cover. Table 13 shows the UGmHA guidelines with the corresponding guidelines in Nielsen and Xcertia.
Furthermore, the number of checklist items in UGmHA (154 items) is greater than the number of checklist items in Xcertia (60 items), while there are no checklist items in Nielsen. As mentioned before, Nielsen’s guidelines are general, and a small number of checklist items in Xcertia guidelines are specific to the features of mHealth applications.
Therefore, we adapted and extended the guidelines by adding more checklist items that cover specific features of mHealth applications and features of accessibility for the elderly and people with special needs. The checklist items were extracted from the following studies:
  • HE4EH [6], which covers self-monitoring, biometric measurements, etc.
  • Telemedicine of mHealth applications [13], which covers consultation, booking appointments, etc.
  • Elderly people [34], which covers the feature of accessibility for elderly people.
  • People with special needs [35], which covers the feature of accessibility for people with special needs.
This leads to the main contribution of UGmHA by including more guidelines, checklist items and mHealth-specific checklist items than Nielsen and Xcertia as shown in Table 14.

5.2. UGmHA Validation Discussion

From the first experiment of UGmHA validation, we conclude that UGmHA discovered more usability issues than Xcertia and Nielsen for both applications. This is because UGmHA includes more guidelines and checklist items than the others. Moreover, the usability issues discovered in the “Sokry” application are more than those in the “Sehhaty” application for the three guidelines, which means that UGmHA can perform like Nielsen and Xcertia to determine which application is more usable.
For the severity of usability issues, we found that UGmHA found more severe usability issues than Xcertia and Nielsen. Again, this is because UGmHA includes more checklist items and more mHealth-specific checklist items.
For the UGmHA guidelines, we found that the highest number of issues were with the accessibility guideline in both applications. The reason is that the specifications of special needs and elderly people were not considered in the design, such as color preferences, font sizes and the number of words and sentences. This highlights the importance of the accessibility guideline and its checklist items that determine to what extent the application is accessible. For the newly created guideline Interactivity and motivations, we found several usability issues in both applications. This also determines to what extent the applications motivate users to take action and achieve a specific goal.
However, we could not use the Ongoing app evaluation guideline in the experiment since it is up to the application team. Therefore, we will leave it up to the expert judgment phase to acquire their opinion and suggestion about it. In the near future, we plan to provide design recommendations for “Sehhaty” or “Sokry” based on the discovered usability issues and, then, to provide these recommendations to the application team to take their opinion.
One of the challenges we faced was that we could not perform phases 2 and 3 of the validation stage. The reason is that we contacted the user experience (UX) team of the “Sehhaty” application to provide their opinion regarding UGmHA and discovered usability issues, they were willing to cooperate with us; however, right now, they are busy with government projects. Therefore, we postponed the implementation of phases 2 and 3 for a few months in future work.

6. Conclusions and Future Work

With the evolution of mobile applications that have become a supportive technology for different fields, mobile health (mHealth) applications are important to improve the efficiency of healthcare delivery using mobile devices, such as smartphones and wearable devices.
mHealth allows patients to monitor and track their health data and allows communication between patients and their physicians without meeting face to face. However, mHealth applications can harm users if they are not working as intended. Although some studies have discussed general and specific mHealth guidelines, there are certain limitations to them.
In this research, we proposed comprehensive usability guidelines for mHealth applications (UGmHA) that address some of the limitations of previous studies and cover various features related to mHealth applications. The proposed guidelines consist of sixteen (16) guidelines with 154 checklist items built from global and well-known guidelines, such as Nielsen and Xcertia.
In order to validate the effectiveness of the UGmHA guidelines, an experiment was conducted by measuring the performance of the proposed guidelines through comparing the outcomes of the UGmHAto worldwide guidelines. This comparison was applied to two mHealth applications in Saudi Arabia, “Sehhaty” and “Sokry”, to determine which guidelines could identify more usability issues. The results showed that the proposed guidelines discovered more usability issues compared with the others. Thus, UGmHA can assist expert evaluators, designers and developers in designing or evaluating mHealth applications by measuring the usability issues that can influence the user experience of mHealth applications.
For future work, further steps will be conducted to validate and refine UGmHA, which include:
  • Generating design recommendations based on the discovered usability issues for “Sehhaty” or “Sokry” and obtaining the application team’s opinion.
  • Expert judgment by reviewing guidelines by user experience (UX) experts and acquiring their advice and comments to perform the Refining stage.
  • User testing by enhancing the usability of an existing application using refined guidelines of UGmHA and, then, performing user testing for the enhanced application.

Author Contributions

Conceptualization, E.N., W.A. and D.S.; methodology, E.N.; validation, E.N. and D.S.; formal analysis, E.N. and D.S.; resources, E.N.; writing—original draft preparation, E.N.; writing—review and editing, E.N., W.A. and D.S.; supervision, W.A. and D.S.; project administration, W.A. and D.S. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:
eHealthElectronic health
mHealthMobile health
UGmHAUsability guidelines for mHealth applications
FDAFood and Drug Administration
HIMSSHealthcare Information and Management Systems Society
SRSSeverity rating scale
HCPHealthcare provider

Appendix A. Final Usability Guidelines for mHealth Applications (UGmHA)

This appendix includes sixteen (16) proposed usability guidelines for mHealth applications (UGmHA). The description and details of each guideline are provided based on Quinones et al.’s template [17]. The template consists of the guideline ID, guideline name, guideline definition, covered application features based on the provided checklist and guidance checklist to evaluate each guideline as shown in Table A1, Table A2, Table A3, Table A4, Table A5, Table A6, Table A7, Table A8, Table A9, Table A10, Table A11, Table A12, Table A13, Table A14, Table A15 and Table A16.
Table A1. Descriptive details of the first guideline.
Table A1. Descriptive details of the first guideline.
IDMHP1
Priority(3) Critical
NameVisibility of system status (Application Feedback)
DefinitionEnsure that the application keeps users aware of what is going on in the application by including appropriate feedback on user actions.
FeaturesBooking appointments, consultation with HCP via text, voice messages and video calls; sharing data with HCP; access to personal health information; self-monitoring of chronic diseases; allowing uploading and viewing of biometric measurements; set health goals and treatment plans; track health progress; Bluetooth technology; and accessibility by elderly people.
Checklists
  • Does the application clearly identify the patient’s name or ID [12]?
  • Are feedback messages displayed in a predictable and consistent area within the interface in order to be noticed by the patient [12]?
  • Does successful feedback appear when an action is completed, such as a successful profile update or appointment booking [12]?
  • Is there a confirmation message for critical actions, such as appointment cancellation or data deletion [34]?
  • Is the time between actions and results appropriate for the user’s cognitive processing (not too slow or too fast) [6,12]?
  • Are several redundant identifiers—such as color, iconography, labeling and additional text—used to denote vital or safety-related information [12]?
  • Do feedback messages that are not urgent or not related to medical safety risks avoid interfering with the application’s operations [12]?
  • Do ongoing processes (greater than 20 s) utilize ongoing feedback, such as a progress bar or show the time remaining to complete the downloading process [6,12]?
  • Is the application feedback appropriate and understandable by elderly people [34]?
  • Does the application add real-time updates that signify the available clinics (currently online) vs. unavailable clinics (offline) [13]?
  • For self-monitoring, does the application acknowledge the performance of the user’s recorded behavior and the goals achieved [6]?
  • For self-monitoring, does the application inform users if the readings will vary depending on the tool used, method or time for taking measurements [6]?
  • If the application uses sensors, are users informed for how many days this sensor works [6]?
  • For a consultation session, does the application show a confirmation message to the user before starting the session [13]?
  • For a consultation session, does the application show a confirmation message before closing the session [13]?
  • For a consultation session, does the application show a timer and wait time to see a doctor or consultation duration [13]?
Table A2. Descriptive details of the second guideline.
Table A2. Descriptive details of the second guideline.
IDMHP2
Priority(2) Important.
NameMatch between the system and the real world (Metaphor).
DefinitionEnsure that the application speaks the users’ language, employs terminology and concepts that the user is familiar with and arranges information in a natural and logical sequence.
FeaturesAccessibility by elderly people and accessibility by people with special needs.
Checklists
  • Is the application speaking the native language of the country that it is intended for [12]?
  • Is written content free of jargon, acronyms and text that may not be understandable to users with no clinical knowledge [12]?
  • Does the application use terminology that is readable and understandable by users and avoid medical terminology [35]?
  • Does the application use terms that older adults are familiar with [34]?
  • Are the terms used in the application consistent through all screens [13]?
  • If there are services or options used in the application that seem to be the same, but they are actually different, does the application provide a description for these options or services [13]?
Table A3. Descriptive details of the third guideline.
Table A3. Descriptive details of the third guideline.
IDMHP3
Priority(2) Important.
NameUser control and freedom (Navigation).
DefinitionEnsure that the application has proper navigation and provides an “emergency exit” to smoothly leave an unwanted state.
FeaturesBooking appointments, consultation with HCP via text, voice messages and video calls; access to personal health information; self-monitoring of chronic diseases; allowing uploading and viewing of biometric measurements; and accessibility by elderly people.
Checklists
  • Are users able to easily recognize where they are in the application and how to navigate to various destinations [12]?
  • Are the number of taps, swipes or screens required for the navigation minimized [12]?
  • Does the application provide reversible actions, such as allowing the user to return to previous pages [11,12]?
  • Does using the back button always take the patient back to the page they were on before [34]?
  • Does the application include clear instructions and emergency exits to leave an unwanted state [34]?
  • Is the main menu provided in order to navigate easily within the application [12,34]?
  • Is the main menu located on the top and left-hand side [12]?
  • Are menu options labeled intuitively [12]?
  • Does the navigation bar exist on every page in the application [34]?
  • Does the navigation text clearly identify the destination [34]?
  • Is the search service for information in the application provided, such as consultations, the name of the doctor and historical information [13,34]?
  • Are the user’s search results presented in a proficient way (for instance, in alphabetical order, relevancy order or last updated order) [13,34]?
  • Is the search box easily recognizable and accessible [13,34]?
  • Does the size of the search box fit on the screen [13,34]?
  • For appointment scheduling, is the patient able to choose among physician’s specialties [13]?
  • For appointment scheduling, is the patient able to see a physician’s detailed information (such as name, education experiences and achievements [13]?
  • Is a patient able to view past consultations and upcoming appointments [13]?
  • Is a patient able to view the prescription and investigations ordered by their consulted physicians [13]?
  • Is a patient able to view files, measurements and data shared with their HCP [13]?
  • Is there any indicator for users where they can find their history of measurements [13]?
Table A4. Descriptive details of the fourth guideline.
Table A4. Descriptive details of the fourth guideline.
IDMHP4
Priority(2) Important.
NameConsistency and standards.
DefinitionEnsure that the application’s elements are consistent and follow the standard design of the application.
FeaturesAccessibility by elderly people and accessibility by people with special needs.
Checklists
  • Is the text aligned right or left based on the direction of the local language [12]?
  • Is the application’s logo positioned on the top center, top right or top left [12]?
  • Is the main menu indicated by a three-bar “hamburger” icon, which is a familiar icon to the users [12]?
  • Is the text hierarchy maintained, so that larger font sizes represent headings and smaller font sizes represent paragraph text [12]?
  • Are all screens designed and elements maintain the same format [34,35]?
  • Does the application use consistent font sizes and colors across all screens [35]?
  • Are the titles, menus and forward, backward and save buttons positioned consistently [35]?
  • Do error messages follow traditional visuals, such as bold and red text? [12]?
  • Do the elements in the application screen fit the normal posture of the hand and finger [34]?
Table A5. Descriptive details of the fifth guideline.
Table A5. Descriptive details of the fifth guideline.
IDMHP5
Priority(3) Critical.
NameError prevention.
DefinitionEnsure that the application prevents problems from occurring by careful design.
FeaturesBooking appointments, self-monitoring of chronic diseases, allowing uploading and viewing of biometric measurements, accessibility by elderly people and accessibility by people with special needs.
Checklists
  • Does the application minimize the possibility of data entry error by providing users with selectable options rather than demanding text entries, such as selecting a doctor’s specialty to book an appointment [12]?
  • Does the application inform users of data entry requirements, such as password needs letters, numbers and symbols [12]?
  • Is an early error message displayed by the application when unacceptable letters or symbols are entered [13]?
  • For self-monitoring, is a user informed of the requirements, best conditions, usual time and frequency of testing the biometric measurements [6]?
  • For self-monitoring, is a user informed through the application of the desired and normal test results [6]?
Table A6. Descriptive details of the sixth guideline.
Table A6. Descriptive details of the sixth guideline.
IDMHP6
Priority(2) Important.
NameRecognition rather than recall (Memory).
DefinitionEnsure that the application minimizes the user’s memory load.
FeaturesBooking appointments, consultation with HCP via text, voice messages and video calls; access to personal health information; self-monitoring of chronic diseases; accessibility by elderly people; and accessibility by people with special needs.
Checklists
  • Does the application support recognition over recall by keeping important information on screen rather than requiring users to remember it [12]?
  • Is the application providing support to remember functions easily [34]?
  • Are the icons or images on a button easy to guess what it does? [34]?
  • Are items placed in recognizable places? [34]?
  • Does the application group similar functions together, such as group consultations for specific specialties together [34]?
Table A7. Descriptive details of the seventh guideline.
Table A7. Descriptive details of the seventh guideline.
IDMHP7
Priority(1) Useful.
NameFlexibility and efficiency of use (Efficiency).
DefinitionEnsure that the application provides an accelerator that shortcuts some actions and allows users to customize the application based on their needs and preferences.
FeaturesConsultation with HCP via text, voice messages and video calls; sharing data with HCP; access to personal health information; self-monitoring of chronic diseases; accessibility by elderly people; and accessibility by people with special needs.
Checklists
  • Does the application provide a flexible and usable interface to operate in both landscape and portrait orientations [12]?
  • Does the application provide clear and readable content across different sizes of mobile screens and operating systems [12]?
  • Are users allowed to customize the application based on their preferences, such as colors and font size [12,34]?
  • Are the application’s default settings easy to use for the elderly [34]?
  • For a consultation session, is a patient able to disable the camera and mute voice through consultation [13]?
  • For a consultation session, does the application support text messaging through consultation [13]?
  • Is a patient able to attach and send files to the physician through consultations [13]?
  • Does self-monitoring of chronic diseases address each individual’s specific educational and behavioral requirements [6]?
  • For self-monitoring, does the application provide a facility to set easy tasks and increase the difficulty level until the target goal is achieved [6]?
Table A8. Descriptive details of the eighth guideline.
Table A8. Descriptive details of the eighth guideline.
IDMHP8
Priority(1) Useful.
NameAesthetic and minimalist design (Design).
DefinitionEnsure that the application does not contain useless or irrelevant information. Ensure that the visual design adheres to the contrast, repetition, alignment and proximity rules.
FeaturesAccess to personal health information, accessibility by elderly people and accessibility by people with special needs.
Checklists
  • Does the application follow contrasting colors, such as a white background and dark text for easy readability [12,34,35]?
  • Does the application minimize the use of extraneous text, graphics and animation to avoid distracting the patients or cluttering the screen [12,34,35]?
  • Are the critical elements to the application functionality and content understandability placed in a recognizable position, such as above the scroll line to minimize the opportunity for missed information? [12,34]?
  • Does the application include multimedia, graphics, images and icons to make the content clear and descriptive [35]?
  • Does the application indicate actionable elements and differentiate interactive elements from non-selectable content [12,34]?
  • Is the content organized around big ideas with a holistic approach [35]?
  • Do the options/information follow logical sequences [34]?
  • Is the amount of text reduced (only essential information presented) [34]?
  • Does the application use small information chunking or lists and tables to promote learnability and memorability when huge amounts of data must be displayed [12,34,35]?
  • Does the application incorporate spacing allowances between lines to allow for breathability and readability [12,35]?
  • Are the labels described clearly [34]?
  • Are the buttons large enough to see the image or text on them [34]?
  • Is the button size suitable for finger touch [34]?
  • Is there enough space between the buttons to avoid accidentally pressing numerous or incorrect buttons [34]?
Table A9. Descriptive details of the ninth guideline.
Table A9. Descriptive details of the ninth guideline.
IDMHP9
Priority(3) Critical.
NameError diagnosis and recovery (Recovery).
DefinitionEnsure that the application expresses the error messages in plain language (with no codes), accurately describes the issue and positively suggests a solution.
FeaturesAccessibility by elderly people and accessibility by people with special needs.
Checklists
  • Are error messages clear, concise and explain the problem [12,34]?
  • Are error messages informing users of required corrective actions in a clear and understandable way [12,34]?
  • Are error messages understandable by the elderly and people with special needs [34,35]?
Table A10. Descriptive details of the tenth guideline.
Table A10. Descriptive details of the tenth guideline.
IDMHP10
Priority(1) Useful.
NameHelp and documentation (Help).
DefinitionEnsure that the application incorporates clear help and troubleshooting tools to assist users when necessary.
FeaturesSelf-monitoring of chronic diseases, interactive prompts, accessibility by elderly people and accessibility by people with special needs.
Checklists
  • Does the application include a help section that compiles all the patient-assistance information [12]?
  • Are help pop-ups for simple information and links for complex information embedded in the application when patients may be likely to need them [12]?
  • Does the application use graphics or videos to enhance or replace text for complicated assistance instructions [12]?
  • Does the application provide customer services, such as e-mail and online chat [12]?
  • Are experienced users allowed to bypass step-by-step walkthroughs or detailed instructions [12]?
  • Does the application give easy-to-follow steps and avoid text-heavy paragraphs for instructional information [12]?
  • For self-monitoring, does the application provide instructions on how to perform a specific behavior [6]?
  • For self-monitoring, is the behavior of an expert provided in an application that shows a person in a video for how to correctly perform a test or do a behavior [6]?
Table A11. Descriptive details of the eleventh guideline.
Table A11. Descriptive details of the eleventh guideline.
IDMHP11
Priority(3) Critical.
NameNotifications, alerts and alarms (Notifications).
DefinitionEnsures that the application’s notifications, alerts and alarms take both safety and usability into account to notify users when their attention is required. Notifications (general reminders for users). Alerts (non-urgent signs meant to catch user attention). Alarms (urgent signs for safety-critical messages).
FeaturesBooking appointments, consultation with HCP via text, voice messages and video calls; self-monitoring of chronic diseases; allowing uploading and viewing of biometric measurements; reminders and notifications; and interactive prompts.
Checklists
  • Do safety-critical alarms send out redundant user signals [12]?
  • Are users required to accept alarms before proceeding with other tasks [12]?
  • Is it possible for users to opt out of non-critical notifications and alerts [12]?
  • Are users notified if an app deletes data after a certain period of time [12]?
  • Are the volume and vibration of audible notifications customizable [12]?
  • Are notifications, alerts and alarms stored as historical data for future reference [12]?
  • For consultations, is a user informed through sound alerts that the physician is present in the session [13]?
  • For consultations, is the user notified via alerts that the consultation’s session will finish if no response is received from the user [13]?
  • For self-monitoring, is a user notified through a sound alarm of high and low measurements and how far their measurements are from the target range [6]?
Table A12. Descriptive details of the twelfth guideline.
Table A12. Descriptive details of the twelfth guideline.
IDMHP12
Priority(2) Important.
NameOnboarding.
DefinitionEnsure that the application facilitates launching, registering and preparing for first time use.
FeaturesAccess to personal health information.
Checklists
  • Does the application provide a launch screen that clearly states its name and objectives? [12]?
  • Does the application give the user intuitive choices for initial use, such as preferred language [12]?
  • Does the application provide opportunities for users either to access detailed application instructions or immediately begin working with the application [12]?
  • Does the application allow users to bypass personal data entry if it is not critical to the application’s functionalities [12]?
  • For the data needed for the application functionality, does the application mark it as mandatory [12]?
  • For applications with complex onboarding processes, is the data entered stored so that users can recover and avoid reentry if they are disconnected during the onboarding [12]?
  • When setup is complete, does the application provide options for a walkthrough or tutorial on app use [12]?
  • Does the application bypass onboarding for returning users [12]?
  • Does the application allow users to update the data entered through onboarding on the “Profile” or “Setting” page [12]?
Table A13. Descriptive details of the thirteenth guideline.
Table A13. Descriptive details of the thirteenth guideline.
IDMHP13
Priority(3) Critical.
NameHistorical data (History).
DefinitionEnsure that the application stores historical data that allow users to access, read and understand these data easily.
FeaturesBooking appointments, consultation with HCP via text, voice messages and video calls; access to personal health information; self-monitoring of chronic diseases; allowing uploading and viewing of biometric measurements; graphic display of patient’s information; and track health progress.
Checklists
  • For large data history, are historical data sortable and filterable based on certain categories [12]?
  • Are historical data displayed in a suitable way to the patients and customized depending on the data type (For example, the daily collected historical data can be represented in alphabetical order so that a patient has easy access to the most recent data collected) [12]?
  • Are patients informed if the application has a limited amount of data storage [12]?
  • For self-monitoring, does the application show results and reading history as a graph in order to be readable and understandable for patients [6]?
  • Does the application show a clear and readable graph and charts for historical biometric measurements [6]?
  • For self-monitoring, does the application inform patients of how many hours and days the results are stored and displayed [6]?
  • For self-monitoring, is a review of previous goals provided in the application [6]?
Table A14. Descriptive details of the fourteenth guideline.
Table A14. Descriptive details of the fourteenth guideline.
IDMHP14
Priority(3) Critical.
NameAccessibility.
DefinitionEnsure that the application is usable by all users, including elderly people and people with special needs.
Featuresaccessibility by elderly people and accessibility by people with special needs.
Checklists
  • Does the application provide accessibility for people with special needs [12]?
  • Does the application use text alternatives or captions for multimedia, such as images or videos [12]?
  • Is the data entry process easy for the elderly and people with special needs [34,35]?
  • Does the application allow a variety of input modalities, such as gestures, handwriting and speech recognition [12]?
  • Does the application provide a validation of the entered input modalities [12]?
  • Does the application facilitate one-handed use [12,34]?
  • Does the application consider customization of colors to be suitable for visual impairments, such as avoiding red and green colors [12]?
  • Is it possible to increase the font size [34]?
  • Does the application consider screen reader accessibility [12]?
  • Does the application include less than four colors to not distract people with intellectual disabilities [35]?
  • Are content written in large font sizes, such as larger than 14 points [35]?
  • Are short sentences employed in the application <7 words [35]?
  • Are the minimum number of sentences employed in the application <4 for each screen [35]?
  • When changes are made to facilitate accessibility, does the content preserve functioning and readability [12]?
  • Are the menus and buttons large enough for the elderly and those with special needs to navigate [34,35]?
  • When the text size is increased, do buttons and icons enlarge [34]?
  • Does the application allow to enable tap gestures for elderly people [34]?
  • Do gestures work correctly and smoothly [34]?
Table A15. Descriptive details of the fifteenth guideline.
Table A15. Descriptive details of the fifteenth guideline.
IDMHP15
Priority(2) Important.
NameOngoing app evaluation (Evaluation).
DefinitionEnsure that the application undergoes iterative evaluation and follows a user-centered design.
FeaturesImportant to design and evaluate all features.
Checklists
  • Are the characteristics of the target users studied before application envelopment [12]?
  • If the application is intended for use in a clinical environment, does it consider how the app may realistically fit into the clinical workflow [12]?
  • Does the application meets known usability guidelines [12]?
  • Does the application use tools for failure modes or fault-tree analysis to determine which user tasks can cause risks for patients [12]?
  • IS user testing conducted iteratively with the target end users of the application [12]?
  • Is a final summative (validation) test conducted to ensure that the application can be used safely and successfully [12]?
Table A16. Descriptive details of the sixteenth guideline.
Table A16. Descriptive details of the sixteenth guideline.
IDMHP16
Priority(1) Useful.
NameInteractivity and motivations (Interactivity).
DefinitionEnsure that the application motivates users and allows for communication between patients to share their experiences.
FeaturesConsultation with HCP via text, voice messages and video calls; sharing data with HCP; self-monitoring of chronic diseases; health tips and motivation; sharing health data with friends; interactive prompts; and earning rewards.
Checklists
  • For consultations, is the patient provided a satisfaction survey at the end of the consultation [13]?
  • For self-monitoring, does the application ask patients to assess the relationship of self-monitoring for specific diseases with exercise, food, medications and stress [6]?
  • For self-monitoring, is social support provided by allowing users to communicate and share their experience of how they change their behavior to offer help to patients [6]?
  • For self-monitoring, are contingent rewards in terms of praise or encouragement given that are explicitly linked to the achievement of a specified goal [6]?
  • For self-monitoring, are opportunities for social comparison provided by allowing users to view the performance of non-expert users [6]?
  • For self-monitoring, is motivational interviewing supported in-app by asking the user to provide self-motivating statements to minimize the change resistance [6]?
  • Are users invited to share content and provide feedback about their experiences [6]?

References

  1. Risling, T.; Martinez, J.; Young, J.; Thorp-Froslie, N. Evaluating patient empowerment in association with eHealth technology: Scoping review. J. Med. Internet Res. 2017, 19, e329. [Google Scholar] [CrossRef] [PubMed]
  2. Dicianno, B.E.; Parmanto, B.; Fairman, A.D.; Crytzer, T.M.; Yu, D.X.; Pramana, G.; Coughenour, D.; Petrazzi, A.A. Perspectives on the evolution of mobile (mHealth) technologies and application to rehabilitation. Phys. Ther. 2015, 95, 397–405. [Google Scholar] [CrossRef] [Green Version]
  3. Istepanian, R.S.; Jovanov, E.; Zhang, Y. Guest editorial introduction to the special section on m-health: Beyond seamless mobility and global wireless health-care connectivity. IEEE Trans. Inf. Technol. Biomed. 2004, 8, 405–414. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. World Health Organization. mHealth: New Horizons for Health Through Mobile Technologies; World Health Organization: Geneva, Switzerland, 2011. [Google Scholar]
  5. Barutçu, S. mHealth apps design using quality function deployment. Int. J. Health Care Qual. Assur. 2019, 32, 698–708. [Google Scholar] [CrossRef] [PubMed]
  6. Khowaja, K.; Al-Thani, D. New checklist for the heuristic evaluation of mHealth apps (HE4EH): Development and usability study. JMIR mHealth uHealth 2020, 8, e20353. [Google Scholar] [CrossRef]
  7. Cook, V.E.; Ellis, A.K.; Hildebrand, K.J. Mobile health applications in clinical practice: Pearls, pitfalls, and key considerations. Ann. Allergy Asthma Immunol. 2016, 117, 143–149. [Google Scholar] [CrossRef]
  8. Larson, R.S. A path to better-quality mHealth apps. JMIR mHealth uHealth 2018, 6, e10414. [Google Scholar] [CrossRef]
  9. Andrade, F.; Nascimento, L.; Wood, G.; Calil, S. Applying heuristic evaluation on medical devices user manuals. In Proceedings of the World Congress on Medical Physics and Biomedical Engineering, Toronto, ON, Canada, 7–12 June 2015; Springer: Cham, Switzerland, 2015; pp. 1515–1518. [Google Scholar]
  10. Rusu, C.; Roncagliolo, S.; Rusu, V.; Collazos, C. A Methodology to establish usability heuristics. In Proceedings of the ACHI 2011: The Fourth International Conference on Advances in Computer-Human Interactions, Guadeloupe, France, 23–28 February 2011; IARIA: Wilmington, NC, USA, 2011; pp. 59–62. [Google Scholar]
  11. Nielsen, J. Ten Usability Heuristics. 2005. Available online: http://www.nngroup.com/articles/ten-usability-heuristics (accessed on 5 November 2021).
  12. Xcertia mHealth App Guidelines. 2019. Available online: https://www.himss.org/sites/hde/files/media/file/2020/04/17/xcertia-guidelines-2019-final.pdf (accessed on 7 November 2020).
  13. Aldekhyyel, R.N.; Almulhem, J.A.; Binkheder, S. Usability of Telemedicine Mobile Applications during COVID-19 in Saudi Arabia: A Heuristic Evaluation of Patient User Interfaces. Healthcare 2021, 9, 1574. [Google Scholar] [CrossRef]
  14. Donevant, S.B.; Estrada, R.D.; Culley, J.M.; Habing, B.; Adams, S.A. Exploring app features with outcomes in mHealth studies involving chronic respiratory diseases, diabetes, and hypertension: A targeted exploration of the literature. J. Am. Med. Inform. Assoc. 2018, 25, 1407–1418. [Google Scholar] [CrossRef] [Green Version]
  15. Lazard, A.J.; Brennen, J.S.B.; Belina, S.P. App Designs and Interactive Features to Increase mHealth Adoption: User Expectation Survey and Experiment. JMIR mHealth uHealth 2021, 9, e29815. [Google Scholar] [CrossRef]
  16. Shati, A. Mhealth applications developed by the Ministry of Health for public users in KSA: A persuasive systems design evaluation. Health Inform. Int. J. 2020, 9, 1–13. [Google Scholar] [CrossRef]
  17. Quiñones, D.; Rusu, C.; Rusu, V. A methodology to develop usability/user experience heuristics. Comput. Stand. Interfaces 2018, 59, 109–129. [Google Scholar] [CrossRef]
  18. Mitchell, J. From Telehealth to e-Health: The Unstoppable Rise of e-Health; Department of Communications, Information Technology and the Arts: Canberra, Australia, 1999. [Google Scholar]
  19. Rooij, T.v.; Marsh, S. eHealth: Past and future perspectives. Pers. Med. 2016, 13, 57–70. [Google Scholar] [CrossRef] [PubMed]
  20. Black, A.D.; Car, J.; Pagliari, C.; Anandan, C.; Cresswell, K.; Bokun, T.; McKinstry, B.; Procter, R.; Majeed, A.; Sheikh, A. The impact of eHealth on the quality and safety of health care: A systematic overview. PLoS Med. 2011, 8, e1000387. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  21. Prado, L.; Carpentier, C.; Préau, M.; Schott, A.M.; Dima, A. mHealth Apps for Self-Management of Chronic Conditions in France: What Is Out There? In MEDINFO 2019: Health and Wellbeing e-Networks for All; IOS Press: Amsterdam, The Netherlands, 2019; pp. 1970–1971. [Google Scholar]
  22. Markus, M.L.; Keil, M. If we build it, they will come: Designing information systems that people want to use. MIT Sloan Manag. Rev. 1994, 35, 11. [Google Scholar]
  23. Tan, W.s.; Liu, D.; Bishu, R. Web evaluation: Heuristic evaluation vs. user testing. Int. J. Ind. Ergon. 2009, 39, 621–627. [Google Scholar] [CrossRef]
  24. International Organization for Standardization. Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs); ISO: Geneva, Switzerland, 1992. [Google Scholar]
  25. Bevana, N.; Kirakowskib, J.; Maissela, J. What is usability. In Proceedings of the fourth International Conference on HCI, Stuttgart, Germany, 1–6 September 1991; pp. 1–6. [Google Scholar]
  26. Nielsen, J.; Molich, R. Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Seattle, WA, USA, 1–5 April 1990; pp. 249–256. [Google Scholar]
  27. Quiñones, D.; Rusu, C. How to develop usability heuristics: A systematic literature review. Comput. Stand. Interfaces 2017, 53, 89–122. [Google Scholar] [CrossRef]
  28. Hermawati, S.; Lawson, G. Establishing usability heuristics for heuristics evaluation in a specific domain: Is there a consensus? Appl. Ergon. 2016, 56, 34–51. [Google Scholar] [CrossRef]
  29. Vieira, E.A.O.; Silveira, A.C.D.; Martins, R.X. Heuristic evaluation on usability of educational games: A systematic review. Inform. Educ. 2019, 18, 427–442. [Google Scholar] [CrossRef]
  30. Paz, F.; Paz, F.A.; Pow-Sang, J.A.; Collantes, L. Usability heuristics for transactional web sites. In Proceedings of the 2014 11th International Conference on Information Technology: New Generations, IEEE, Las Vegas, NV, USA, 7–9 April 2014; pp. 627–628. [Google Scholar]
  31. Leverenz, T. The Development and Validation of a Heuristic Checklist for Clinical Decision Support Mobile Applications. Ph.D. Thesis, Wichita State University, Wichita, KS, USA, 2019. [Google Scholar]
  32. Chattratichart, J.; Lindgaard, G. A comparative evaluation of heuristic-based usability inspection methods. In CHI’08 Extended Abstracts on Human Factors in Computing Systems; Association for Computing Machinery: New York, NY, USA, 2008; pp. 2213–2220. [Google Scholar]
  33. Roy, B.; Call, M.; Abts, N. Development of usability guidelines for mobile health applications. In Proceedings of the International Conference on Human–Computer Interaction, Orlando, FL, USA, 26–31 July 2019; Springer: Cham, Switzerland, 2019; pp. 500–506. [Google Scholar]
  34. Al-Razgan, M.S.; Al-Khalifa, H.S.; Al-Shahrani, M.D. Heuristics for evaluating the usability of mobile launchers for elderly people. In Proceedings of the International Conference of Design, User Experience, and Usability, Heraklion, Crete, Greece, 22–27 June 2014; Springer: Cham, Switzerland, 2014; pp. 415–424. [Google Scholar]
  35. Arachchi, T.K.; Sitbon, L.; Zhang, J. Enhancing access to eLearning for people with intellectual disability: Integrating usability with learning. In Proceedings of the IFIP Conference on Human–Computer Interaction, Mumbai, India, 25–29 September 2017; Springer: Cham, Switzerland, 2017; pp. 13–32. [Google Scholar]
  36. Sanz, F.; Galvez, R.; Rusu, C.; Roncagliolo, S.; Rusu, V.; Collazos, C.A.; Cofré, J.P.; Campos, A.; Quiñones, D. A set of usability heuristics and design recommendations for u-learning applications. In Proceedings of the Information Technology: New Generations: 13th International Conference on Information Technology, Las Vegas, NV, USA, 11–13 April 2016; Springer: Cham, Switzerland, 2016; pp. 983–993. [Google Scholar]
  37. Campos, A.; Rusu, C.; Roncagliolo, S.; Sanz, F.; Gálvez, R.; Quiñones, D. Usability heuristics and design recommendations for driving simulators. In Proceedings of the Information Technology: New Generations: 13th International Conference on Information Technology, Las Vegas, NV, USA, 11–13 April 2016; Springer: Cham, Switzerland, 2016; pp. 1287–1290. [Google Scholar]
  38. Gale, N.; Mirza-Babaei, P.; Pedersen, I. Heuristic guidelines for playful wearable augmented reality applications. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play, London, UK, 5–7 October 2015; pp. 529–534. [Google Scholar]
  39. da Hora Rodrigues, K.R.; Teixeira, C.A.C.; de Almeida Neris, V.P. Heuristics for assessing emotional response of viewers during the interaction with TV programs. In Proceedings of the Human–Computer Interaction. Theories, Methods, and Tools: 16th International Conference, HCI International 2014, Heraklion, Crete, Greece, 22–27 June 2014; Proceedings, Part I 16. Springer: Cham, Switzerland, 2014; pp. 577–588. [Google Scholar]
  40. Charness, G.; Gneezy, U.; Kuhn, M.A. Experimental methods: Between-subject and within-subject design. J. Econ. Behav. Organ. 2012, 81, 1–8. [Google Scholar] [CrossRef]
  41. Kuparinen, L.; Silvennoinen, J.; Isomäki, H. Introducing usability heuristics for mobile map applications. In Proceedings of the International Cartographic Conference, Dresden, Germany, 25–30 August 2013; International Cartographic Association: Bern, Switzerland, 2013. [Google Scholar]
  42. Munoz, R.; Chalegre, V. Defining virtual worlds usability heuristics. In Proceedings of the 2012 Ninth International Conference on Information Technology-New Generations, IEEE, Las Vegas, NV, USA, 16–18 April 2012; pp. 690–695. [Google Scholar]
  43. Moraes, M.C.; Silveira, M.S. How am I? Guidelines for animated interface agents evaluation. In Proceedings of the 2006 IEEE/WIC/ACM International Conference on Intelligent Agent Technology, IEEE, Hong Kong, China, 18–22 December 2006; pp. 200–203. [Google Scholar]
  44. Muhanna, M.; Masoud, A.; Qusef, A. Usability heuristics for evaluating Arabic mobile games. Int. J. Comput. Games Technol. 2022, 2022, 5641486. [Google Scholar] [CrossRef]
  45. Keele, S. Guidelines for Performing Systematic Literature Reviews in Software Engineering; Technical Report, Ver. 2.3; EBSE: Durham, UK, 2007. [Google Scholar]
  46. Iso, W. 9241-11. Ergonomic requirements for office work with visual display terminals (VDTs). Int. Organ. Stand. 1998, 45, 22. [Google Scholar]
  47. Usability 101: Introduction to Usability. 2012. Available online: https://www.nngroup.com/articles/usability-101-introduction-to-usability/ (accessed on 8 March 2021).
  48. Harrison, R.; Flood, D.; Duce, D. Usability of mobile applications: Literature review and rationale for a new usability model. J. Interact. Sci. 2013, 1, 1. [Google Scholar] [CrossRef] [Green Version]
  49. Sehhaty Application. 2022. Available online: https://play.google.com/store/apps/details?id=com.lean.sehhaty (accessed on 6 December 2022).
  50. Sokry Application. 2022. Available online: https://play.google.com/store/apps/details?id=com.sokry (accessed on 6 December 2022).
  51. Severity Ratings for Usability Problems. 1994. Available online: https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/ (accessed on 5 September 2021).
  52. Quiñones, D.; Rusu, C. Applying a methodology to develop user eXperience heuristics. Comput. Stand. Interfaces 2019, 66, 103345. [Google Scholar] [CrossRef]
Figure 1. Xcertia workgroups.
Figure 1. Xcertia workgroups.
Mti 07 00026 g001
Figure 2. Flowchart for the systematic literature review of UGmHA.
Figure 2. Flowchart for the systematic literature review of UGmHA.
Mti 07 00026 g002
Figure 3. Matching process for features, attributes and guidelines.
Figure 3. Matching process for features, attributes and guidelines.
Mti 07 00026 g003
Figure 4. (a) The home screen of the first application. (b) The home screen of the second application.
Figure 4. (a) The home screen of the first application. (b) The home screen of the second application.
Mti 07 00026 g004
Figure 5. Severity of issues in the “Sehhaty” and “Sokry” applications among the three guidelines.
Figure 5. Severity of issues in the “Sehhaty” and “Sokry” applications among the three guidelines.
Mti 07 00026 g005
Figure 6. Number of usability problems found by UGmHA.
Figure 6. Number of usability problems found by UGmHA.
Mti 07 00026 g006
Table 1. Review of the validation process in different studies.
Table 1. Review of the validation process in different studies.
RefValidation ProcessValidation Result
[41]Four evaluators evaluated an application by comparing Nielsen guidelines against the authors’ guidelines.The authors’ guidelines were more effective than Nielsen’s since they discovered more usability issues (19 issues, 6 of them were severe) than Nielsen’s guidelines (15 issues, 5 of them were severe).
[42]Six evaluators evaluated two applications by comparing Nielsen guidelines against the authors’ guidelines.The author’s guidelines worked better than Nielsen’s since they discovered more usability issues (App1: 43, App2: 45) than Nielsen’s guidelines (App1: 28, App2: 24)
[43]Two evaluators evaluated four animated agents through new guidelines.The results showed which agent’s design was better.
[44]Eighteen evaluators evaluated two Arabic applications by comparing the authors’ guidelines against Nielsen’s guidelines and game usability principles (two applications and three guidelines).The author’s guidelines worked better than the others since they discovered more usability issues (Game1: 13, Game2: 12) than Nielsen (Game1: 6, Game2: 5) and the game usability principles (Game1: 10, Game2: 8).
Table 2. The information highlighted in the descriptive stage.
Table 2. The information highlighted in the descriptive stage.
TopicCollected InformationSelected Information
FeaturesFeatures provided by [14]Features of these studies are combined and
Features provided by [15]summarized in Table 3.
Features provided by [16]
Accessibility to elderly peopleSelected.
Accessibility by people with special needsSelected.
AttributesUsability attributes proposed by ISO standard [46] (three attributes): effective, efficiency and satisfactionUnselected.
Usability attributes proposed by Nielsen [47] (five attributes): learnability, efficiency, memorability, errors and satisfactionUnselected.
Usability attributes proposed by Harrison [48] (seven attributes): effective, efficiency, learnability, memorability, errors, satisfaction and cognitive loadSelected because it combines both attributes of ISO and Nielsen in addition to the cognitive load.
Existing guidelinesNielsen’s guidelines (10 guidelines) [11]The set of Nielsen’s principles are selected.
Xcertia guidelines (five workgroups: privacy, security, content, usability and operability) [12]The only selected workgroup is “usability” and discards all others since our focus is only on usability.
HE4EH (25 guidelines) [6]Select some checklists that cover the feature of self-monitoring in general and discard checklists that cannot be generalized to all chronic diseases.
Telemedicine mobile application design recommendations [13]All recommendations are selected.
Elderly people guidelines (13 guidelines) [34]Select the checklists that can be used for mHealth applications and discard the checklists that are specific to the mobile launchers.
Intellectual disabilities guidelines (31 guidelines) [35]Select the checklists that can be used for mHealth applications and discard the checklists that are specific to the eLearning systems.
Table 3. Matching among features, attributes and guidelines.
Table 3. Matching among features, attributes and guidelines.
No.FeaturesAttributesNielsenXcertia
F1Booking appointment [16]Efficiency and effectivenessNH1XU5, XU6, UX8
F2Consultation with HCP via text, voice messages and video calls [14,16]Efficiency and effectivenessNH1XU5
F3Sharing data with HCP [14,15]Memorability and efficiencyNH1, NH6XU5
F4Access to personal health information [15]Memorability, error, efficiency and effectivenessNH1, NH5, NH6, NH9XU4, XU5, XU8
F5Self-monitoring of chronic disease [14,15]Learnability, cognitive load, efficiency and effectivenessNH1, NH6XU6, XU8
F6Allowing uploading and viewing of biometric measurements [14]Memorability, error, efficiency and effectivenessNH1, NH3, NH6XU3, XU6, XU8
F7Graphic display of patient’s information [14]Memorability, satisfaction, effectiveness and efficiencyNH2, NH4, NH6, NH8XU1, XU2, XU8
F8Set health goals and treatment plan [14,15]Satisfaction, learnability, cognitive load, effectivenessNH1, NH2XU2
F9Track health progress [15]Memorability, efficiency and cognitive loadNH1XU5, XU6, XU8
F10Reminders and notifications [14]Memory and cognitive loadNH6XU6
F11Health tips and motivation [14,15]Satisfaction and effectivenessNH2XU2
F12Sharing health data with friends [15]Satisfaction and learnability--
F13Interactive prompts [14]LearnabilityNH10XU6, XU7
F14Earn rewards [15]Satisfaction--
F15Bluetooth technology connection [14]Efficiency and effectivenessNH1XU5
F16Accessibility to elderly peopleSatisfaction, efficiency, effectiveness and memorability-XU9
F17Accessibility by people with
pecial needs
Satisfaction, efficiency, effectiveness and memorability-XU9
Table 4. Guideline selection process (Nielsen) [11].
Table 4. Guideline selection process (Nielsen) [11].
IDGuideline NameActionCovered FeaturesApplicability
NH1Visibility of system statusAdapt *F1–F6, F8, F9, F15(3) Critical
NH2Match between system and the real worldAdapt *F2, F7, F8, F11(2) Important
NH3User control and freedomAdapt *F1, F4, F5, F6(2) Important
NH4Consistency and standardsAdapt *F7, F11(2) Important
NH5Error preventionAdapt *F5, F6(3) Critical
NH6Recognition rather than recallAdapt *F4, F5(2) Important
NH7Flexibility and efficiency of useAdapt *F3, F4, F5, F8(1) Useful
NH8Aesthetic and minimalist designAdapt *F4, F7(1) Useful
NH9Help users recognize, diagnose and recover from errorsAdapt *F1, F4(3) Critical
NH10Help and documentationAdapt *F13(2) Important
* Add more checklist items related to mHealth applications from different studies.
Table 5. Guideline selection process (Xcertia) [12].
Table 5. Guideline selection process (Xcertia) [12].
IDGuideline NameActionCovered FeaturesApplicability
XU1Visual designEliminate (the checklists were moved to Nielsen’s NH4, NH5 and NH6 guidelines)F7-
XU2ReadabilityEliminate (the checklists were moved to Nielsen’s NH8 guideline)F7, F8, F11-
XU3App navigationEliminate (the checklists were moved to Nielsen’s NH3 guideline)F6-
XU4OnboardingAdapt *F4(2) Important
XU5App feedbackEliminate (the checklists were moved to Nielsen’s NH1 guideline)F1–F4, F9, F15-
XU6Notifications, alerts and alarmsAdapt *F1, F5, F6, F9, F10, F13(3) Critical
XU7Help resource and troubleshootingEliminate (the checklists were moved to Nielsen’s NH10 guideline)F13-
XU8Historical dataAdapt *F1, F2, F4–F7, F9(2) Important
XU9AccessibilityAdapt *F16, F17(3) Critical
XU10Ongoing app evaluationKeep (no change for this guideline)F1–F17(2) Important
* Add more checklist items related to mHealth applications from different studies.
Table 6. Guideline selection process (new created guidelines).
Table 6. Guideline selection process (new created guidelines).
IDGuideline NameActionCovered FeaturesApplicability
N1Interactivity and MotivationsCreate (to cover F12 and F14 features since they are covered neither in Nielsen’s nor in the Xcertia guidelines.)F12, F14(1) Useful
Table 7. The set of proposed usability guidelines for mHealth applications (UGmHA).
Table 7. The set of proposed usability guidelines for mHealth applications (UGmHA).
IDGuideline Name
MHP1Visibility of system status
MHP2Match between system and the real world
MHP3User control and freedom
MHP4Consistency and standards
MHP5Error prevention
MHP6Recognition rather than recall
MHP7Flexibility and efficiency of use
MHP8Aesthetic and minimalist design
MHP9Error diagnosis and recovery
MHP10Help and documentation
MHP11Notifications, alerts and alarms
MHP12Onboarding
MHP13Historical data
MHP14Accessibility
MHP15Ongoing app evaluation
MHP16Interactivity and motivations
Table 8. Details of the “Sehhaty” and “Sokry” applications.
Table 8. Details of the “Sehhaty” and “Sokry” applications.
SehhatySokry
Government/ PrivateA government mHealth application that belongs to the Ministry of Health (MOH) in
Saudi Arabia.
A private mHealth application.
FunctionalityProvides different services, such as booking appointments, children’s vaccines, immediate consultations to make video and audio calls with doctors, COVID-19 services and reviewing health data, including insurance and registered information in MOH.Helps users manage their diabetes by recording blood sugar readings, meals, exercises and medications. In addition, provides health advice related to diabetes.
PopularityPopular with the people since it is connected with health records of MOH and considered an essential application for people in Saudi Arabia (+10 million downloads) [49].Less popular to the people (+50,000 downloads) [50].
Table 9. Severity Ranking Scale (SRS) [51].
Table 9. Severity Ranking Scale (SRS) [51].
RatingDescription
0I do not agree that this is a problem at all.
1Cosmetic problem only. Need not be fixed unless extra time is available in the project.
2Minor usability problem. Fixing this should be given low priority.
3Major usability problem. Important to fix so it should be given high priority.
4Usability catastrophes. Imperative to fix this before the product can be released.
Table 10. Descriptive elements for the UGmHA guidelines.
Table 10. Descriptive elements for the UGmHA guidelines.
IDGuideline ID
Priority(3) Critical, (2) Important or (1) Useful.
NameName of the guideline that resulted from the integration of Nielsen [11] and Xcertia [12].
DefinitionIdentify the guideline and its purpose.
FeaturesThe selected features covered by the guideline.
ChecklistsThe checklist items selected from [6,12,13,34,35] to add more details to the guideline and to make it more related to the features of mHealth applications and accessible to the elderly and people with special needs.
Table 11. Numbers of usability issues of both applications based on the three guidelines.
Table 11. Numbers of usability issues of both applications based on the three guidelines.
SehhatySokryTotal
UGmHA7395168
Xcertia222850
Nielsen172542
Table 12. Severity of usability issues of the “Sehhaty” and “Sokry” applications.
Table 12. Severity of usability issues of the “Sehhaty” and “Sokry” applications.
SehhatySokry
RatingUGmHAXcertiaNielsenUGmHAXcertiaNielsen
No Problem0 (0%)0 (0%)0 (0%)0 (0%)0 (0%)0 (0%)
Cosmetic7 (10%)2 (9%)0 (0%)6 (6%)2 (7%)0 (0%)
Minor10 (14%)1 (5%)0 (0%)17 (18%)4 (14%)2 (8%)
Major17 (23%)6 (27%)12 (71%)26 (27%)10 (36%)9 (36%)
Catastrophic39 (53%)13 (59%)5 (29%)46 (48%)12 (43%)14 (56%)
Total732217952825
Table 13. Relation of the UGmHA guidelines with Nielsen and Xcertia.
Table 13. Relation of the UGmHA guidelines with Nielsen and Xcertia.
IDUGmHACorresponding Guideline (Nielsen)Corresponding Guideline (Xcertia)
MHP1Visibility of system statusVisibility of system statusApp feedback
MHP2Match between system and the real worldMatch between system and the real world-
MHP3User control and freedomUser control and freedomApp navigation
MHP4Consistency and standardsConsistency and standardsVisual design
MHP5Error preventionError preventionVisual design
MHP6Recognition rather than recallRecognition rather than recallVisual design
MHP7Flexibility and efficiency of useFlexibility and efficiency of use-
MHP8Aesthetic and minimalist designAesthetic and minimalist designReadability
MHP9Error diagnosis and recoveryError diagnosis and recovery-
MHP10Help and documentationHelp and documentationHelp resource and troubleshooting
MHP11Notifications, alerts and alarms-Notifications, alerts and alarms
MHP12Onboarding-Onboarding
MHP13Historical data-Historical data
MHP14Accessibility-Accessibility
MHP15Ongoing app evaluation-Ongoing app evaluation
MHP16Interactivity and Motivations--
Table 14. Guidelines and checklist items of UGmHA, Xcertia and Nielsen.
Table 14. Guidelines and checklist items of UGmHA, Xcertia and Nielsen.
UGmHAXcertiaNielsen
Number of guidelines161010
Number of checklist items154600
Number of mHealth-specific checklist items4460
Number of accessibility checklist items1870
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Nasr, E.; Alsaggaf, W.; Sinnari, D. Developing Usability Guidelines for mHealth Applications (UGmHA). Multimodal Technol. Interact. 2023, 7, 26. https://doi.org/10.3390/mti7030026

AMA Style

Nasr E, Alsaggaf W, Sinnari D. Developing Usability Guidelines for mHealth Applications (UGmHA). Multimodal Technologies and Interaction. 2023; 7(3):26. https://doi.org/10.3390/mti7030026

Chicago/Turabian Style

Nasr, Eman, Wafaa Alsaggaf, and Doaa Sinnari. 2023. "Developing Usability Guidelines for mHealth Applications (UGmHA)" Multimodal Technologies and Interaction 7, no. 3: 26. https://doi.org/10.3390/mti7030026

Article Metrics

Back to TopTop