Implementation and Evaluation of a Mobile Retinal Image Acquisition System for Screening Diabetic Retinopathy: Study Protocol

Abstract

Screening diabetic retinopathy, a major cause of blindness, is time-consuming for ophthalmologists and has some constrains in achieving full coverage and attendance. The handheld fundus camera EyeFundusScope was recently developed to expand the scale of screening, drawing on images acquired in primary care and telescreening made by ophthalmologists or a computer-aided diagnosis (CADx) system. This study aims to assess the diagnostic accuracy of the interpretation of images captured using EyeFundusScope and perform its technical evaluation, including image quality, functionality, usability, and acceptance in a real-world clinical setting. Physicians and nurses without training in ophthalmology will use EyeFundusScope to take pictures of the retinas of patients with diabetes and the images will be classified for the presence or absence of diabetic retinopathy and image quality by a panel of ophthalmologists. A subgroup of patients will also be examined with the reference standard tabletop fundus camera. Screening results provided by the CADx system on images taken with EyeFundusScope will be compared against the ophthalmologists’ analysis of images taken with the tabletop fundus camera. Diagnostic accuracy measures with 95% confidence intervals (CIs) will be calculated for positive and negative test results. Proportion of each category of image quality will be presented. Usability and acceptance results will be presented qualitatively.

1. Introduction

1.1. Background and Rationale

Diabetic retinopathy affects 34.6% of patients with diabetes and is the leading cause of blindness worldwide [1,2,3]. The disease is asymptomatic in early stages [4,5], so it is important to screen individuals at risk of developing this condition. Retinal image acquisition is highly technical and requires specialized technicians managing tabletop retinal cameras—large devices that need to be fixed and calibrated onto a table in a healthcare setting [6,7]. Then, all images are analysed by an ophthalmologist, and of those, only 11% show referable diabetic retinopathy, meaning that almost 90% of screened patients have mild or no signs of diabetic retinopathy and no diabetic macular oedema [8].

Despite the implemented screening programmes in different countries [9,10,11,12], less than half of individuals with diabetes receive retinal screenings [5,13,14,15]. Constrains related to lack of healthcare professionals, infrastructure, and costs have been reported [16]. Moreover, patients’ attendance rates range between 61% and 88.9% [17], for reasons such as lack of awareness or misinformation regarding the need for eye screening, not receiving screening invitation, and being unable to drive because of the effect of eye drops for pupil dilation [17,18]. This represents an important loss of opportunity to diagnose early and treat diabetic retinopathy [18], which can, ultimately, be responsible for patients’ vision loss and blindness [18]. The World Health Organization reinforces the need for screening of all individuals with diabetes [19]. As more and more patients are diagnosed with diabetes and as the human lifespan increases, healthcare services throughout the world need to adapt quickly to ensure that screenings are delivered [20].

In many diabetic retinopathy screening programmes, fundus images are acquired with tabletop fundus cameras and transferred electronically to a reading centre to be analysed by an ophthalmologist [6]. This telemedicine approach enables to overcome ophthalmologists’ shortage, as eye technicians—not ophthalmologists—perform image acquisition. However, this screening modality relies on patients’ uptake to screening appointments. Opportunistic screenings when patients visit their doctors may be an alternative to increase patient uptake. Moreover, evidence from research points out that diabetic retinopathy screening intervals could be personalized to each patient, based on an individual risk assessment [21,22,23]. Rather than inviting all patients with diabetes to an annual screening, intervals can be extended for some patients [21,22,23]. A reorganization of screening programmes will be needed, and a tabletop fundus camera located in a centralized healthcare unit for rigid scheduled screenings may not suit the new screening model; instead, a handheld fundus camera, permanently available in each primary healthcare unit and possible to operate by non-ophthalmologists and non-eye technicians, may be more appropriate. These healthcare professionals would be able to screen patients with diabetes when they attend a regular medical appointment, reaching the patients who would fail to attend a scheduled screening. Handheld fundus cameras—smaller and easier to transport than tabletop fundus cameras [6,23]—would in principle enable screenings at multiple settings: offices of family physicians, general practitioners, endocrinologists, and nurses, and at patients’ homes or community settings [8,24,25,26,27,28]. CellScope Retina (University of California Berkeley, Berkeley, CA, USA), Remidio Fundus on Phone (Remidio, Karnataka, India), D-EYE (Si14 SpA, Padova, Italy), VistaView (Volk Optical, Mentor, OH, USA), and EyeFundusScope (Fraunhofer AICOS, Porto, Portugal) are handheld fundus cameras that integrate a smartphone [29,30,31,32,33]. Other handheld fundus cameras, for example, PiCTOR Prestige (Volk Optical, Mentor, OH, USA), do not integrate a smartphone; instead, they have dedicated touchscreens and computing units [34].

Whether the images are taken with tabletop fundus cameras or handheld fundus cameras, the ophthalmologists’ workload is a challenge. Artificial intelligence (AI)-based software for diabetic retinopathy screening has the potential to automate image analysis and quickly identify pathologic features (microaneurysms, haemorrhages, and neovascularization) [34], reducing ophthalmologists’ workload. In recent years, many AI algorithms for diabetic retinopathy screening have been developed [35]. Retmarker (Retmarker Ltd., Coimbra, Portugal), iGradingM (EMIS Leeds, UK), EyeArt (Eyenuk Inc., Woodland Hills, CA, USA), and IDx-DR (IDx, LLC, Coralville, IA, USA) obtained a Conformité Européenne (CE) Mark [8], and only the latter also obtained approval from the Food and Drug Administration (FDA, Silver Spring, MD, USA) [34]. All of them are indicated for use with conventional tabletop retinal cameras [8,35]. Medios DR AI was developed to analyse images taken with a smartphone-based nonmydriatic retinal camera—Remidio Fundus on Phone (Remidio, Karnataka, India) [36]. However, its accuracy study was conducted using pupil dilation drops in 90.6% of patients [37]. Increasing the size of the pupils with eye drops increases the portion of the retina that can be seen in fundus photography, and hence increases the diagnostic accuracy for diabetic retinopathy [38,39,40]. However, ensuring good accuracy with a smartphone-based fundus camera without pupil dilation induced by eye drops is essential, since it potentially increases the coverage of the screening: eye drops is contraindicated in patients with narrow angle glaucoma, narrow filtration angle, and known hypersensitivity [41]; in addition, the effect of that medication in prohibiting driving is one of the main barriers to screening attendance [42].

The computer-aided diagnosis (CADx) system for diabetic retinopathy screening based on deep learning integrated with EyeFundusScope (Fraunhofer AICOS, Porto, Portugal) smartphone-based retinal camera was assessed on images acquired with EyeFundusScope [43] in a pilot study (in this study a commercial ophthalmoscope was connected to a smartphone) and showed high sensitivity and specificity (67% and 95% respectively) without pupil dilation drops [44]. The study was conducted with patients with known diabetic retinopathy at an ophthalmology outpatient clinic, using indirect ophthalmoscopy as the comparator. In another study, the same AI software was used to classify retinal images taken with tabletop fundus cameras [45]. Therefore, a comparison with the reference standard for screening (tabletop fundus camera) in a generalizable sample of individuals with diabetes, to assess the ability to discriminate patients with diabetic retinopathy from those without it [46] is lacking. Moreover, in that study, EyeFundusScope was operated by a highly trained technical professional [44]. However, since image acquisition with smartphone-based fundus cameras can be challenging for beginners, we need to assess the quality of images and the diagnostic accuracy of EyeFundusScope when operated by their potential future users—nurses and physicians without training in ophthalmology.

The usability of EyeFundusScope was not yet assessed among physicians and nurses without training in ophthalmology. In a previous study [47], physicians and nurses performed fundus image acquisition, and although usability measures were not collected, as usability was not the focus of that work, users expressed that EyeFundusScope was easy to use. However, subjects were healthy and young adults; in patients with difficulty to remain upright and follow instructions, the use of EyeFundusScope (or other retinal cameras) may be challenging.

Because the quality of retinal images is essential to enable proper analysis, it is important to assess whether the focus and field of view are adequate, as well as vessel visibility. Image acquisition in older people and with cataracts is difficult [48]. In the 80+ age group, technical failure rate is 41.6% without pupil dilation [48]. Furthermore, the proportion of ungradable images in people with central cataracts accounts for 57% of ungradable images, and the more the years after diagnosis of diabetes, the higher is the proportion of ungradable images [48]. Moreover, the skill of each operator influences the quality of images, and although we expect it to vary for each operator, we need to assess if these differences affect the AI classification for diabetic retinopathy [49].

New medical devices receive market authorization approval after provision of scientific evidence of safety and effectiveness. This information is generated in clinical research and is presented to the regulators acting in each country. With this study, we aim to provide that evidence for EyeFundusScope and its integrated CADx system. We will consider two scenarios of clinical use: (1) EyeFundusScope to acquire retinal images that are then classified for diabetic retinopathy by an ophthalmologist; and (2) images acquired with EyeFundusScope that are classified for diabetic retinopathy by the CADx system (Figure 1). Considerations for clinical implementation, namely real-world technical functionality of EyeFundusScope, its usability, acceptance, and integration in clinical workflow were not assessed yet and will be assessed in this study.

1.2. Objectives

1.2.1. Primary Objectives

The primary objective of this project is:

• To assess the diagnostic accuracy of using EyeFundusScope alone and along with the CADx system for diabetic retinopathy classification.

1.2.2. Secondary Objectives

The secondary objectives of this project are as follows.

• To assess the image quality and technological functioning of EyeFundusScope in real clinical settings.
• To analyse the association of patients’ clinical characteristics with test accuracy and the quality of images acquired with EyeFundusScope.
• To assess the inter and intraoperator agreement and reliability of EyeFundusScope for diabetic retinopathy classification.
• To assess the usability and acceptance of EyeFundusScope.
• To optimize the ground truth for training an AI software for diabetic retinopathy and image quality automated classification.

2. Methods

2.1. Study Design

This project comprises the following studies:

• A diagnostic test accuracy cross-sectional study, in which a sample of individuals with diabetes will be prospectively included and screened for diabetic retinopathy by EyeFundusScope (index test) and a tabletop fundus camera (reference standard test); and
• A pilot study using a handheld nonmydriatic fundus camera operated by physicians and nurses not specialists in ophthalmology in another sample of individuals—this study includes image quality, screening results agreement with these images, usability, and acceptance.

2.2. Eligibility Criteria, Study Setting, Recruitment, and Sample Size

A consecutive sample of individuals diagnosed with diabetes, aged 18–90 years, will be eligible to participate, after receiving information about the study and providing written informed consent form. Clinical researchers will approach eligible participants when they present at Hospital CUF Tejo (Lisbon, Portugal) for routine medical appointments or community assessments, and at ACeS Gondomar—ARS Norte (Porto, Portugal)—a primary healthcare unit—for diabetic retinopathy screenings. People with diabetes with the following conditions will be excluded from the study: blindness, strabismus, retinal detachment, blepharospasm, or no autonomy to remain seated—retinal imaging may be more difficult if these conditions are present: sensitivity to light due to medication, recent photodynamic therapy, or other—to avoid eye pain or discomfort—and pregnant women or breastfeeding, for ethical reasons. All eligible participants will undergo both the index test and the reference standard test. The Standards for Reporting of Diagnostic Accuracy Studies (STARD) diagram [50] will be presented, showing the number of patients assessed for eligibility and the number of non-consenting patients, as well as participants excluded, along with the respective reasons, and the number of patients screened with the index test and with the reference standard test.

The sample size (n = 578) was calculated on the basis of 95% CIs, 10% margin of error, expected sensitivity and specificity—81% and 96%, respectively [45]—and the prevalence of DR among people with diabetes of 13% [51]. Then, we inflated the sample size by an additional 25% (n = 116, total n = 462) to take account of ungradable images. Sample size calculation was performed at the subject level, since both eyes are required to obtain the reference standard result, considering the worst eye for the result for each subject.

2.3. Diagnostic Accuracy Study

2.3.1. Index Test

The index test consists in screening each participant for diabetic retinopathy using EyeFundusScope. Physicians and nurses without training in ophthalmology will complete a training session before they can start using EyeFundusScope; the study protocol and the EyeFundusScope instruction manual will be available to clinician participants during the study. The photography protocol consists of acquiring at least two retinal images of each eye—one centred on the macula and one centred on the optic disc. Acquiring images of both eyes is expected to take five minutes or less, with minimal impact on clinical workflow. Drops for pupil dilation will not be used. EyeFundusScope images will be classified for diabetic retinopathy by a panel of ophthalmologists blinded to each other. In addition, a subsample of participants will be photographed at least by two clinicians.

2.3.2. Reference Test

Tabletop fundus images will be used as the reference standard test for diabetic retinopathy, as this is the screening test in use in clinical practice and one of recommended tests by the World Health Organization [52]. The classification of images by ophthalmologists is insufficiently reliable [53,54,55], and this could bias the accuracy measures of the diagnostic test under study. To address this challenge, we will use a panel diagnosis as the reference standard: at least two ophthalmologists will independently classify images acquired with EyeFundusScope regarding diabetic retinopathy classification, blinded to the results of the index test. The ophthalmologists will also be masked to the clinical status and characteristics of participants (Appendix A,

Table A1. Sociodemographic characteristics and clinical and laboratory data.

Category	Characteristics
Sociodemographic	Age Sex
Clinical	Type of diabetes Duration of diabetes Other eye diseases: cataract, glaucoma, age-related macular degeneration, myopia (number of diopters), other Current diabetic retinopathy classification (reference standard)
Laboratory	HbA1c

). Discordances will be solved based on consensus; otherwise, a third ophthalmologist will classify those images. In this study, we will adopt a modified version of the International Clinical Disease Severity Scale (ICDSS), which is in use at the study sites and with which ophthalmologists are familiar with (

Table 1. Severity levels of diabetic retinopathy adapted from [56].

Severity Levels	Characteristics
R0	No retinopathy: any lesion related to DR 1
R1	Mild non-proliferative retinopathy: microaneurysms, retinal haemorrhages with or without any exudate excluding the definition of DME 2
R2	Moderate or severe pre-proliferative: “rosary” veins, venous loops and duplications, intra-retinal microvascular anomalies, multiple, deep, cottony exudates
R3	Proliferative: disc neo-vessels, retinal neo-vessels, vitreous or pre-retinal haemorrhage, pre-retinal fibrosis with or without traditional retinal detachment
M1	Maculopathy: Presence of exudates less than 1 DD 3 from the centre of the fovea, or circinate or grouped exudates in the macular area, or retinal thickening to less than 1 DD 3 from the centre of the fovea, or any microaneurysm or haemorrhage less than 1 DD 3 from the centre of the fovea, if associated with an AV 4 < 0.5
P0	Photocoagulated retina not needing more treatment
P1	Photocoagulated retina needing more treatment
Not classifiable	Non-informative image

¹ Diabetic retinopathy; ² diabetic macular oedema; ³ disc diameter; ⁴ venous anomaly.

) [56].

2.4. Inter and Intraoperator Agreement and Reliability

Operating a smartphone-based fundus camera requires some skills that non-specialists in ophthalmology may lack. Variability between operators of a medical device may compromise screening results [49]. Considering 3 EyeFundusScope operators, k of 0.8, k maximum amplitude of 0.2, diabetic retinopathy prevalence of 20% and 95% confidence intervals, the sample size estimate for reproducibility calculations is 200 individuals with diabetes for the study of interoperator reproducibility, and 200 individuals with diabetes for the study of intraoperator reproducibility. To assess how similar the classification of images collected by different users are to each other, at least one physician and one nurse without training in ophthalmology will acquire retinal images in a subsample of patients with diabetes (n = 200) using EyeFundusScope. Moreover, the patient may move the head or the eye during acquisition, potentially affecting the classification of diabetic retinopathy. We think it is important to assess if physicians and nurses without training in ophthalmology can acquire images in a reliable way. Variability between different image acquisitions by the same user will be measured, with the same clinician participant photographing the retina of the same patient at least three times in another subsample.

2.5. Image Quality and Technical Performance

Images acquired with EyeFundusScope will be classified by a panel of ophthalmologists regarding their quality: insufficient, usable, good, and excellent (

Table 2. Criteria for classification of image quality by ophthalmologists.

Image Quality Level	Characteristics
Excellent	Small vessels are clearly visible and sharp Absolute edge-to-edge sharpness in the optical circle The optic disc and the macula are visible, and the temporal arches are complete
Good	Small vessels are clearly visible, but not sharp Without low quality factors, besides the lack of absolute sharpness from end to end in the optical circle The optic disc and the macula are visible, and the temporal arches are complete
Usable	The field of view is partially hidden Irregular lighting with the legible region of the background image greater than 80% of the area seen Optic disc or macula are visible, and some of the vessels or retinal lesions (identified by ophthalmologists) are also visible, but not always in good contrast or with some blur
Inadequate	Large or nearly complete light/dark regions, making it impossible to observe any retinal regions Optic disc or macula are not visible Blur or totally out of focus Large artefacts (occupying more than 25% of the retinal area) Totally undiagnosed, or the image has serious quality problems and cannot be used to provide a reliable diagnosis by ophthalmologists

and Appendix B, Figure A1). This classification will be compared with the one provided automatically by a software for retinal image quality assessment [57]. This evaluation is to ensure that the images contain relevant information for observation by ophthalmologists. The use of EyeFundusScope in real clinical settings can throw up unexpected challenges; all potential difficulties will be reported.

2.6. Usability

As retinal imaging can be technically challenging, the ease of use of EyeFundusScope by physicians and nurses without training in ophthalmology for retinal image acquisition on patients with diabetes will be evaluated in this study. The number of attempts and time to make an acceptable image will be recorded. Additionally, after all retinal image acquisitions have been performed, users will fill out the “System Usability Scale”, an instrument with 10 questions answered with 5-point Likert scale (1 = strongly disagree, 5 = strongly agree) [58]—we will use the version previously translated, culturally adapted, and validated to the Portuguese population [59]; qualitative observations (at least 50) of users operating EyeFundusScope will complement the usability study.

2.7. Technology Acceptance

A questionnaire will be conducted with physicians and nurses after their participation in image acquisition to assess their perceptions about the use of EyeFundusScope as a medical device for diabetic retinopathy screening: which errors happened, its integration in clinical workflow, and potential barriers to its implementation (Appendix C). Because a suitable test is such that is accepted by the population—Wilson and Jungner’s 5th and 6th principles of screening [52]—we will assess EyeFundusScope’s acceptance by conducting interviews, which will include the questions presented in Appendix D with 50 patients, observations of retinal image acquisitions, and taking written notes of relevant comments of patients participants during image acquisition about eventual concerns regarding this new technology (n = 20). Qualitative findings will be analysed using thematic analysis [60].

2.8. Data Collection Methods and Management

2.8.1. Data Collection Methods

Retinal images collected with EyeFundusScope will be coded and stored using a dedicated mobile application that will automatically upload the images to a server; images will then be displayed on a web platform for image classification by ophthalmologists. The time required for the clinician to acquire all images of each patient and to insert the images in the information system will be recorded. Then, each ophthalmologist will record an image score for each retinal image during its interpretation and classification for diabetic retinopathy and image quality. The time required for the ophthalmologist to interpret and classify each retinal image will be recorded automatically in the web platform and will be considered from the time the ophthalmologist starts reading the image until the time the ophthalmologist finishes documenting the classification.

Demographics and clinical and laboratory data will be collected from patients and electronic health records using a paper data collection form. Researchers will conduct in-loco observations of healthcare professionals using EyeFundusScope. All data will finally be entered into a Microsoft Excel^® datasheet.

2.8.2. Data Management

All the data collected will be encoded. Information about each participant, introduced in the mobile app, as well as images of the retina acquired with EyeFundusScope, will be sent through encrypted communication to protected servers, where it will be stored for later analysis by ophthalmologists. Data will be hosted on Fraunhofer AICOS (Porto, Portugal) servers, or the CUF Tejo hospital. As soon as the information is entered, including images of the retina, with associated ID, introduced in the mobile app, it will be accessible to researchers. Ophthalmologists will classify retinal images using a web platform that will anonymously display fundus images and some clinical information of the patient. This platform is only accessible through a login, which each ophthalmologist will receive and will require encrypted communication. This process complies with all the requirements of the General Data Protection Regulation and was subject to evaluation by the Data Protection Officer (DPO) from CUF and the CUF Department of Information Systems (DPI). Paper data entry forms and paper consent forms will be enclosed in opaque envelopes and stored in restricted access areas inside the Sponsor’s facilities. Researchers are committed to maintaining secrecy and confidentiality in all their activities. Analysis of the study data, as well as its presentation in reports, scientific publications, or other forms of publication and dissemination, will preferably be done in an aggregated form and will not allow the identification of study participants.

2.9. Monitoring

2.9.1. Data Monitoring

A data monitoring committee is not needed in this study, because the study will be performed in an adult population with a non-critical or life-threatening disease, and the medical device under investigation is well characterized and known for not harming patients.

2.9.2. Harms

The clinical investigators will collect information about adverse events of study procedures—device failures and device-related adverse events—during the study and will assess them for intensity, causality, and expectancy; then, adverse events will be reported to the Sponsor (Fraunhofer AICOS, Porto, Portugal) according to the study protocol. Additionally, the Study Monitor—as an independent representative of the Sponsor—will monitor the study to ensure that it is carried out in accordance with the protocol, the applicable regulations, and the Clinical Investigation of Medical Devices for Human Subjects—Good Clinical Practice (ISO 14155:2020) [61], performing visits to research centres.

2.9.3. Auditing

Independent regulatory authorities, namely the National Clinical Research Ethics Committee—CEIC (Lisbon, Portugal) and the National Authority of Medicines and Health Products (INFARMED I.P., Lisbon, Portugal), will eventually perform external audits. The Investigators and the Sponsor (Fraunhofer AICOS, Porto, Portugal) will collaborate on such audits, facilitating the Auditors’ access to documents, facilities, records, and other elements related to the study. Internal audits will not be performed due to close monitoring of the Study Monitor, and the low risk of damage from any deviations from the planned and/or established study protocol.

2.10. Ethics and Dissemination

The study received institutional and ethical approval from ARS Norte and Hospital Infante Santo (now Hospital CUF Tejo, Lisbon, Portugal); approval from CEIC is also expected. Important eventual protocol modifications will be communicated to research ethics committees and other relevant parties. Clinical researchers will obtain written informed consent from participants before the study. The study will be carried out in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki). The privacy rights of human subjects will be observed in all phases of the study. Information about participants will be stored using data encryption techniques. Identified data will only be available for researchers involved in data collection and will be deidentified prior to the results analysis and dissemination. CADx’s results will not be displayed on the smartphone screen, so that the nurse or the physician operator will not be influenced in their clinical decision. The participants will receive adequate care for adverse events and insurance coverage. The authors will publish the study results in scientific journals, conferences, and presentations to stakeholders, and on social media. The study protocol will be publicly available on the ISRCTN Registry.

2.11. Statistical Analysis

Demographic, clinical, and laboratory characteristics of the study population will be reported using descriptive statistics using absolute frequencies (n) and relative frequencies (%) for categorical variables. Median and percentiles or mean and standard deviation will be used for continuous variables when appropriate.

Results of the screening for diabetic retinopathy with EyeFundusScope (index test) will be compared to those of the tabletop fundus camera (reference standard) by calculating sensitivity, specificity, predictive values, and likelihood ratios. A cross-tabulation of results of index test by the results of the reference test will be reported, as well as non-classifiable images due to insufficient image quality. The reliability and interobserver agreement between ophthalmologists (reference standard) will be calculated to address some expected variability in the ophthalmologists’ classification using kappa statistics and overall and specific observed agreement.

The association between the classification for diabetic retinopathy and image quality using retinal images acquired with EyeFundusScope will be presented using association measures, namely odds ratios with 95% confidence intervals (OR [95% CI]). Exploratory association between image quality classified by ophthalmologists and the software for automatic image quality classification will be calculated in each image and fixation point obtained using descriptive statistics and association measures. Association between diabetic retinopathy classification by ophthalmologists with image quality automatic classification will be conducted using descriptive statistics and association measures.

Overall measure of retinal image quality based on standard quality assessment protocol of images acquired with EyeFundusScope, and image quality of EyeFundusScope as a function of time and number of images acquired by each operator, using descriptive statistics and association measures will be reported. Inter and intraoperator agreement for diabetic retinopathy classification results will be assessed using descriptive statistics, overall and specific agreement proportion for each category; reliability will be assessed using Cohen’ s kappa. Agreement related to image quality between the software for image quality classification and the ratings from ophthalmologists will be assessed using descriptive statistics, overall observed agreement, and Cohen’ s kappa.

We will calculate and present CIs for all statistical measures.

Proportion of individuals who are willing to undergo further annual screening with the new test will be assessed and qualitative analyses of interviews and observations will be presented. The proportion of healthcare professionals who report ease of use, the score of the “System Usability Scale” [58], qualitative analyses of the observation, the mean time required to perform retinal image acquisition with EyeFundusScope of each eye and of each patient, number of eyes in which it was not possible to acquire images, and number of technical failures of both EyeFundusScope and the information system.

Data will be analysed using IBM ^® SPSS ^® Statistics and/or R Studio ^® and the significance level will be 5%.

3. Discussion

The current study will contribute to inform the potential for implementation of diabetic retinopathy screening with EyeFundusScope and EyeFundusScope + CADx system in clinical practice. If the results of this clinical validation are found to be satisfactory, it will be possible to obtain the CE Mark as a medical device, which opens the door for its use as a mobile eye fundus camera and an automated diagnostic system based on deep learning. This new screening technology will support physicians acquiring retinal images opportunistically and that does not require the intervention of an ophthalmologist to identify referrable diabetic retinopathy. Therefore, it can increase screening coverage, and save ophthalmologists’ valuable time to in person observation and treatment of patients screened positive.

Good image quality is essential for good accuracy of screening results and, ultimately, the clinical value of the tools that we will assess in this study. A few studies have assessed the quality of images of handheld fundus cameras, with which we could compare our results. Darma and colleagues [62] reported image quality obtained with fundus images acquired without pupillary dilation with a smartphone-based fundus camera (Panoptic 11,820 (Welch Allyn, Skaneateles Falls, NY, USA) + iPhone 4 (Apple, Cupertino, CA, USA) and a dedicated handheld fundus camera (Smartscope M5 + EY3 lens; Optomed, Oulu, Finland): 2% and 50%, respectively, of images of quality of 4 (using a 5-point scale ranging from 1 = “can’t see anything” to 5 = “subtle details visible”). Those images were taken from retinas of healthy people with a mean age of 26 years old, in which fundus photograph is easier than in older people and patients with other eye diseases, such as cataracts. However, the study was conducted in 2014, and since then, smartphone technology developed significantly; therefore, we expect our results for image quality to be higher. A recent study [63] described that the fundus images acquired with a handheld fundus camera (Horus Eye-Fundus Camera; Medimaging Integrated Solution Inc., Hsinchu, Taiwan) without pupillary dilation had good (55.7%) or excellent (22.7%) image quality. The average age of the participants was 62 years old, and they had diverse retinal pathology. A direct comparison of results of other studies will not be possible because different scales for image quality are used.

Piyasena and colleagues showed that in the 80+ age group, the technical failure rates reduced from 41.6% to 16.9% following mydriasis [48]. This study concluded that the odds of having one eye ungradable increases by 2.6% (95% CI 1.6–3.7%) for each extra year since the diagnosis of diabetes, with central cataract (57%) being the major cause of ungradability [48].

Different smartphone-based retinal cameras have shown high diagnostic accuracy, with pooled sensitivity and specificity of 87% (95% CI 74–94%) and 94% (95% CI 81–98%), respectively, for any diabetic retinopathy, and 91% (95% CI 86–94%) and 89% (95% CI 56–98%) for referrable diabetic retinopathy [27]. When AI was used to classify images acquired with smartphone-based cameras, pooled sensitivity and specificity were 91% (95% CI 84–96%) and 50% (95% CI 38–62%) [27]. A direct comparison will be difficult, as in some studies images were acquired by ophthalmologists or retina specialists and/or used different grading criteria of diabetic retinopathy. However, our study will focus on enabling the validation of EyeFundusScope and for EyeFundusScope + CADx system as medical devices for diabetic retinopathy screening. We will interpret the results of our study according to the levels that would comply with the accepted standards of established national-level screening programmes: sensitivity ≥80% and specificity ≥95%) [64].

4. Conclusions

Evidence from this study of clinical validation will be useful for paving the way of obtaining a CE mark for EyeFundusScope and for EyeFundusScope + CADx system for diabetic retinopathy. Furthermore, the results of this study will inform the decision makers about the implementation of opportunistic diabetic retinopathy screenings, as well as of extended screening programmes to areas underserved by clinic infrastructure and to people with limitations in mobility. A medical device that is safe, accurate, and easy to use by physicians and nurses not specialists in ophthalmology has the potential to bring confidence to medical teams to adopt the device in their healthcare institutions.

Moreover, with this study, we anticipate challenges to local implementation and integration of these new point-of-care diabetic retinopathy screening tools, both EyeFundusScope and EyeFundusScope + CADx system. This translation from the laboratory to real clinical settings and use conditions within the target population will facilitate its future implementation into clinical practice. At the end of the study, we will have a user-ready and clinically validated medical device. Therefore, the results of this study will be relevant for patients, health professionals and decision makers.