Next Article in Journal
The Role of Magnetic Resonance Imaging (MRI) in the Diagnosis of Pancreatic Cystic Lesions (PCLs)
Next Article in Special Issue
Additional Value of PET and CT Image-Based Features in the Detection of Occult Lymph Node Metastases in Lung Cancer: A Systematic Review of the Literature
Previous Article in Journal
Prognostic Significance of Ultrasound Characteristics and Body Mass Index in Patients with Apparent Early-Stage Cervical Cancer: A Single-Center, Retrospective, Cohort Study
Previous Article in Special Issue
Radiomics and Artificial Intelligence Can Predict Malignancy of Solitary Pulmonary Nodules in the Elderly
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Diagnostics
Volume 13
Issue 4
10.3390/diagnostics13040584
diagnostics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Artificial Intelligence-Assisted Chest X-ray for the Diagnosis of COVID-19: A Systematic Review and Meta-Analysis

by
I-Shiang Tzeng
1,*,
Po-Chun Hsieh
2,
Wen-Lin Su
3,
Tsung-Han Hsieh
1 and
Sheng-Chang Chang
4
1
Department of Research, Taipei Tzu Chi Hospital, Buddhist Tzu Chi Medical Foundation, New Taipei City 23142, Taiwan
2
Department of Chinese Medicine, Taipei Tzu Chi Hospital, Buddhist Tzu Chi Medical Foundation, New Taipei City 23142, Taiwan
3
Division of Pulmonary Medicine, Taipei Tzu Chi Hospital, Buddhist Tzu Chi Medical Foundation, New Taipei City 23142, Taiwan
4
Department of Medical Imaging, Taipei Tzu Chi Hospital, Buddhist Tzu Chi Medical Foundation, New Taipei City 23142, Taiwan
*
Author to whom correspondence should be addressed.
Diagnostics 2023, 13(4), 584; https://doi.org/10.3390/diagnostics13040584
Submission received: 20 December 2022 / Revised: 25 January 2023 / Accepted: 1 February 2023 / Published: 5 February 2023
(This article belongs to the Special Issue Artificial Intelligence as a Diagnostic Tool for Lung Nodule Evaluation)
Browse Figures
Versions Notes

Abstract

:
Because it is an accessible and routine image test, medical personnel commonly use a chest X-ray for COVID-19 infections. Artificial intelligence (AI) is now widely applied to improve the precision of routine image tests. Hence, we investigated the clinical merit of the chest X-ray to detect COVID-19 when assisted by AI. We used PubMed, Cochrane Library, MedRxiv, ArXiv, and Embase to search for relevant research published between 1 January 2020 and 30 May 2022. We collected essays that dissected AI-based measures used for patients diagnosed with COVID-19 and excluded research lacking measurements using relevant parameters (i.e., sensitivity, specificity, and area under curve). Two independent researchers summarized the information, and discords were eliminated by consensus. A random effects model was used to calculate the pooled sensitivities and specificities. The sensitivity of the included research studies was enhanced by eliminating research with possible heterogeneity. A summary receiver operating characteristic curve (SROC) was generated to investigate the diagnostic value for detecting COVID-19 patients. Nine studies were recruited in this analysis, including 39,603 subjects. The pooled sensitivity and specificity were estimated as 0.9472 (p = 0.0338, 95% CI 0.9009–0.9959) and 0.9610 (p < 0.0001, 95% CI 0.9428–0.9795), respectively. The area under the SROC was 0.98 (95% CI 0.94–1.00). The heterogeneity of diagnostic odds ratio was presented in the recruited studies (I2 = 36.212, p = 0.129). The AI-assisted chest X-ray scan for COVID-19 detection offered excellent diagnostic potential and broader application.

1. Introduction

COVID-19 is a deadly pathogenic disease that results from the dissemination of coronavirus infection [1]. Acute respiratory distress syndrome, nervous system problems, organ dysfunction, or death may be caused by COVID-19 [2,3]. Thus, early recognition and prompt medical treatment of COVID-19 has become a major issue. To identify COVID-19 easily and efficiently, and to determine the prognosis, researchers have focused on researching and developing new detection methods [4]. To date, the chest X-ray is cheaper than other specialized methods. The chest X-ray is assessable from the image and has been refined during the last decade [5]. Recent evidence has indicated that the chest X-ray is a potent way of forecasting pulmonary diseases [6], respiratory diseases [7], cardiovascular diseases [8], or acute internal bleeding [9]. Cytokine storms and innate immune system overworking may trigger Acute Lung Injury (ALI) and the induction of acute respiratory distress syndrome (ARDS) related to the COVID-19 patients involved with hypertension [10]. Multiphase fibrosis, tissue stiffness, and lung function damage [11] may be caused by the hyaluronic acid (HA) molecules’ product in lung tissue, which is triggered by the cytokine storm. SARS-CoV-2 transparent cells relying on binding to the spike (S) glycoprotein of the Angiotensin-Converting Enzyme 2 (ACE2) receptor [12,13].
Therefore, a chest X-ray or computed tomography (CT) scan are recommended as first-line diagnostic tools for pulmonary involved patients [14]. Multilobar bilateral and unilateral chest X-ray, ground-glass opacity (GGO), and peripheral infiltrates on chest CT scans have been clinically proven to have a radiological role in the diagnosis of the COVID-19 disease [15,16]. For the peripheral regions of tissue, more than one lobe was found in the form of GGOs, or less nodules were found in each lobe [14,17]. The diagnosis of vascular nodules in the images of patients was difficult to attribute to the removal and large number of lung CT images and their complex and heterogeneous structures [18]. Thus, the artificial intelligence (AI) systems that assist medical imaging for screening have gained an important role in supporting decision making [19]. The AI presented an ability to change clinical decision-making; however, we should be cautious about implementing AI systems in each information system [20]. In terms of the merit of AI systems for physicians, AI systems assist as a diagnostic tool, making faster and efficient decisions. The augment ability on medical imaging in disease diagnosis is driving advances in traditional image processing and AI algorithms to retrieve diagnostic information. When diseases occur, AI provides a physician with the necessary diagnostic information required to speed up diagnosis and add precision intervention decisions. Some traditional image-assisted techniques for AI diagnosis contain contours and region progression, which provide a physician an aid with which to extract diagnostic information. Moreover, traditional models experience limited performance, customizability, and a strong reliance on in-advance computed features. AI has the ability to avoid the above limitations and to derive complex image features by importing feature semantics into classifiers [21,22,23]. In previous studies that evaluated the relationship between the use of a chest X-ray and COVID-19 incidence and death rates, it was found that the chest X-ray has not only diagnostic value but also great potential as a prediction image for clinical outcomes [24,25].
Recently, a study conducted a literature review on the diagnostic role of AI, which suggested its excellent potential and wide application relative to comparative methods, based on sufficiently sized datasets and independent testing [26]. The study found that radiographic diagnostics have sharper sensitivity than laboratory testing when compared to the numerous diagnoses of COVID-19 under development that employ AI to swiftly assess chest CT imaging [26]. Nonetheless, this study was based on a restricted review without considering a summary sensitivity and the specificity of the receiver operating characteristic curve (ROC). Hence, a more rigid multi-center study investigating the predictive role of AI-assisted chest X-ray scans for COVID-19 is guaranteed to improve our understanding of the accuracy of AI diagnostic devices.
According to the preceding study, we implemented a meta-analysis to evaluate the diagnostic value of AI-assisted chest X-ray scans to determine their accuracy in COVID-19 patients.

2. Materials and Methods

2.1. Seek Tactics and Picking Standard

We used PubMed, Cochrane Library, MedRxiv, ArXiv, and Embase to search for research published from 1 January 2020 to 30 May 2022 involving “machine learning”, “artificial intelligence”, “medical image”, “SARS-CoV-2”, and “COVID-19”, because publication related to AI was distinct from traditional therapeutic publication. Only research that considered chest X-rays to probe the usage of AI were chosen for the review. From the selected papers, the following data were excerpted: the number of datasets used for training and validation, the proportion of COVID-19 scans within the dataset, and the sensitivity, specificity, and area under curve (AUC) of the proffered manner. We also considered whether the datasets and model code were estimable. The research was then classified by imaging process: chest X-ray.
The following MeSH terms and their combinations were searched, including (Machine learning OR Artificial intelligence OR Medical image) and (SARS-CoV-2 OR COVID-19) and (AUC OR ROC OR Sensitivity OR Specificity).

2.2. Procedures

This review was performed in accordance with the preferred reporting items for systematic reviews and meta-analyses (PRISMA). Two researchers (IST and PCH) independently extracted the data from the included studies. Two researchers (IST and PCH) performed the initial screening, manually searching the results and selecting articles for full-text retrieval in the title (or abstract) review process. The opinion of the third reviewer (WLS) was considered if identification was inconsistent between IST and PCH. They determined eligibility by screening the titles and abstracts of the retrieved studies and extracted data by constructing a excel table. They validated any discrepancies and addressed concerns through discussion to achieve consensus in the extracted data. Information bias may be generated from an image data source, which also influenced the results of study. Several sources of imaging data included the type of imaging contexts (i.e., three- or two-dimensional), type of AI approach, sensitivity, and specificity extracted from recruited studies. The risk of bias was frontally evaluated by two independent reviewers (IST and PCH). Next, we reviewed each study using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) guidelines [27]. The QUADAS-2 tool was used to assess the methodological quality of the included studies [27]. The QUADAS-2 tool consisted of four key domains covering patient selection, index test, reference standard, and flow and timing. We identified the risk of bias as ’high’, ‘low’, or ‘unclear’. The result of the risk of bias for the recruited studies was presented using a plot.

2.3. Statistical Analysis

According to the patients with or without a COVID-19 diagnosis, we calculated counts of true positives (i.e., sensitivity multiply number of COVID-19 diagnoses), false positives (i.e., (1-specificity) multiply number of non-COVID-19 diagnoses), true negatives (i.e., specificity multiply number of non-COVID-19 diagnoses), and false negatives (i.e., (1-sensitivity) multiply number of COVID-19 diagnoses) of COVID-19 from the included research and calculated the pooled estimates for sensitivity and specificity and corresponding 95% CI.
The “mada” package [28] was used to assess the data to explore the pooled sensitivity and specificity and their 95% CI. The “mada” package [28] was also used to evaluate the summary receiver operating characteristic curve (SROC), which was used to calculate the AUC value. Finally, funnel plots were created to examine the publication bias of the included studies. All statistical analyses and graphical presentations mentioned above were implemented using R software (4.2.0 version, Vienna, Austria).

3. Results

3.1. Literature Selection and Quality Assessment

After evaluating and carefully screening all the studies from the databases, the literature search yielded nine eligible studies [29,30,31,32,33,34,35,36,37], including 39,603 participants (included 2976 COVID-19 cases and 36,627 non-COVID-19 individuals). The PRISMA flowchart of study selection is shown in Figure 1.

3.2. Risk of Bias Assessment

Based on the definition of the criterion standard for the detection of COVID-19 using chest X-rays assisted by AI system, data were extracted from the recruited studies for this meta-analysis. Extraction data included the first author, year of publication, the conducted study country, type of study, and number of patients. The QUADAS-2 tool was used to assess the quality and potential bias of nine studies. Four key domains covering patient selection, index test, reference standard, and flow and timing were assessed. The results of the literature quality assessment using QUADAS [27] are provided in Figure 2. Low risk bias implied confidence on the part of the literature reviewer that results represent the true diagnostic effect (such as sensitivity, specificity, and AUC). Figure 2 showed that 90% of the “overall risk of bias” item presented as having a low risk of bias.

3.3. General Characteristics

We summarize the general characteristics of all nine studies included in this meta-analysis in Table 1 and their research findings. The general characteristics of the recruited studies are shown in Table 1. We found that four studies were conducted in the United States of America and that the other five studies were conducted in different countries, respectively. Next, we included seven case-control studies and two retrospective studies. For participants in this meta-analysis, this study included 2976 COVID-19 cases and 36,627 non-COVID-19-diagnosed individuals. The sensitivity and specificity of the nine studies are presented as a forest plot in Figure 3. We also presented 95% CI of sensitivity and specificity for the nine studies in Figure 3.

3.4. Results of Pooled Estimates for Sensitivity and False Positive Rate Analysis

For the random effects model, the pooled sensitivity was 0.9472 (p = 0.0338, 95% CI 0.9009–0.9959), and the pooled specificity was 0.9610 (p < 0.0001, 95% CI 0.9428–0.9795). Two-dimensional plots were provided by the “mada” package [28]. One is the crosshair plot, and the other is the ROC ellipse plot. In Figure 4, the crosshair was conducted using an arbitrary color, which made the crosshairs wider with increased weight. Bold purple crosshairs were presented due to their biggest weight among nine studies. The SROC was generated to assess the diagnostic value, and the results revealed an AUC of 0.98 (95% CI 0.94–1.00 in Figure 5).

3.5. Deeks’ Test

We used the formulas provided by Deeks [38]. The “mada” module also performed χ2 tests to evaluate the heterogeneity of sensitivities and specificities. The null hypotheses in both cases’ heterogeneity of sensitivities and specificities were equivalent. Test results for the equality of sensitivities: X-squared = 297.3168 with p < 0.0001; test results for equality of specificities: X-squared = 581.6488 with p < 0.0001. This showed that the heterogeneity results of sensitivities and specificities were not equal in both cases.

4. Discussion

COVID-19 is a severe acute respiratory syndrome caused by the SARS-CoV-2 virus, resulting in organ exhaustion and gradual death [39]. In September 2022, COVID-19 infections and deaths worldwide reached 620 and 6.5 million, respectively, (https://www.worldometers.info/coronavirus/, accessed on 31 January 2023). This study successfully presented overwhelming evidence suggesting that the adoption of AI-based chest X-rays is a useful diagnostic tool to detect COVID-19.
There were seven case-control studies and two retrospective studies recruited in this meta-analysis due to the search strategy adopted in this study. Moreover, we focused on diagnostics of the COVID-19 epidemic rather than the novelties of AI, as clinical research studies are more widespread for other AI utilizations. Finding accessible small datasets in the public domain is a general challenge for overall research. Because COVID-19 has occurred only since late 2019, there are few images of COVID-19 patients accessible at separate institutions and in public domain datasets. Some studies have adopted similar datasets (Table 1). This is a drawback, as the algorithm trained on a specific dataset may not have the ability to execute as well when lending itself to the assorted data [40]. Simultaneously, the scarcity of external checks among the investigated studies may enhance this risk of bias.
Our evaluation of the repeatability of the results of the algorithms may have been limited owing to smaller datasets. Nevertheless, some research used datasets in public domain but neglected their clarity regarding the origin and disposition of the image data. This may cause a certain level of bias. Another review utilizing AI to interpret COVID-19 diagnoses revealed that the high level of bias in most of their papers was an effect of the small number of COVID-19 images [37]. However, small datasets are not incompatible with research on AI in COVID-19. The stated influence may be controversial. A previous study focusing on pulmonary nodules assessed with AI was based on a small sample of 186 patients [41]. Images used in the study were found in numerous public storehouses and taken from publications [36]. These images probably presented extreme and provocative cases of COVID-19 that may have been easier for the algorithm to perceive. Furthermore, some datasets were branched through vision ascent and the development of repetition. Among the nine studies, only two studies presented an independent examination with an external dataset (22%), and the two studies that adopted external validation presented average sensitivity and specificity results of 97.56% and 95.15%, respectively. The remaining seven studies without external validation presented average sensitivity and specificity values of 94.18% and 96.38%, respectively. Thus, evaluating the performance of externally checked models was superior to disregarding externally checked models [42]. A previous study revealed in another current review of this objective [43] that only 21% (13/62) assessed their algorithms on separate datasets. High performance in external testing provided solid evidence that the model may be extrapolated to another patient population. Such external validation may mitigate influence and controversy. To date, the performance in external testing confirmed that the model could be converted into clinical practice.
Some studies constructed on the sampling from large COVID-19 datasets to mitigate the uncertainty of prediction [44,45,46,47]. Then, a previous systematic review described AI-based diagnostic imaging (CT and chest X-ray) tools and showed that the performance of both CT and chest X-ray diagnostic tools may be limited by the scale of the dataset [48]. In addition, the well-balanced data had the benefit of the training of neural networks. Minor differences of in the variable class of data significantly affect the results of study [49]. Equal classes of data are also found in a previous review [50]. However, a small number of chest X-rays are still to be found in some studies conducted throughout the pandemic [51,52,53].
According to the diagnostic ability of CT, another meta-analysis conducted based on CT imaging settled as the future work. This meta-analysis extracted the diagnostic performance of AI-assisted CT-Scan for COVID-19. Some studies which based their meta-analysis on AI-assisted CT-Scan diagnostic performance for COVID-19 demonstrated an AUC from 0.95 to 0.97 [54,55]. Consequently, the AI-assisted CT-Scan for COVID-19 was associated with pneumonia based on objective investigation [56,57]. A review of recent publications proved that physicians, in detecting COVID-19, used an AI supportive system [26]. Community acquired pneumonia and lung diseases were detected with an AUC value of 0.96 using a deep learning model [58]. On the contrary, some published studies showed the fair performance (AUC values ranged from 0.732 to 0.87) of DL model-assisted COVID-19 detection [59,60,61]. However, molecular diagnostic tests, such as reverse transcription–polymerase chain reaction (RT-PCR), were still the most reliable diagnostic tool, rather than the AI-assisted CT-Scan for COVID-19 detection [62]. Because lungs are infected at the later stage of infection and present a certain level of confusion in identification through medical imaging (Figure S1), the importance of RT-PCR tests cannot be underestimated. Controversially, AI, which assisted the chest CT-Scan to classify diseases’ status, has an excellent performance of diagnoses, with AUC values ranging from 0.90 to 1.00 [36,63,64,65].
This study utilized the SROC analysis of diagnostic accuracy in the adoption of AI-assisted chest X-rays for the detection of COVID-19. In this meta-analysis, a strictly screened literature search was performed for all studies published in English between 1 January 2020 and 30 May 2022. The pooled estimates for sensitivity and specificity were 0.9472 (p = 0.0338, 95% CI 0.9009–0.9959) and 0.9610 (p < 0.0001, 95% CI 0.9428–0.9795), respectively. To evaluate the diagnostic value and prediction accuracy of using AI-assisted chest X-rays for COVID-19, the SROC curve indicated that the AUC value was 0.98 (95% CI 0.94–1.00). The results of this study suggested that the use of AI in chest X-rays has a significant value for forecasting COVID-19. Soda et al. [66] determined that the AUC of AI in chest X-rays in predicting COVID-19 was 0.63 (95% CI 0.52–0.74); nevertheless, only older adults aged above 65 years were included, and the number of subjects was not large. Another review study [67] summarized the literature of AI in chest X-rays for predicting COVID-19, albeit without synthesis evidence.
This meta-analysis verified the prediction value of AI in chest X-rays for COVID-19 patients (Table 2); however, there were some limitations and shortcomings. First, there was notable but not significant heterogeneity (I2 = 36.212, p = 0.129) on the diagnostic odds ratio in the review; the random effect results of the pooled sensitivity and specificity were 94.72 (p = 0.0338, 95% CI 90.09–99.59) and 96.10 (p < 0.0001, 95% CI 94.28–97.95), respectively. The funnel plot of the sensitivity and specificity may show a certain selection bias for recruited studies (Figure S2). However, such heterogeneity may have resulted from dissimilarity in areas, the cut-off value of AI in chest X-rays, the severity of COVID-19, and the resolution of the chest X-ray (Figure S2). Moreover, the predictive value of AI in chest X-rays was not consistent, and the predictive value of AI in chest X-rays had not been considered in the preceding studies, which may have negatively influenced the contributed sensitivity and specificity. Deep learning is a type of machine learning. Moreover, the goal of machine learning is to enable computers to think and behave independently of human input. However, deep learning is the process of teaching computers to reason by using structures inspired by the human brain. The recruited nine studies, which focus on a deep learning model (Table 1), may limit the results of this study.

5. Conclusions

In conclusion, AI can play a significant auxiliary role in utilizing chest X-rays when diagnosing COVID-19. Considering that COVID-19 is a complicated syndrome involving complex pathophysiological mechanisms, AI can assist chest X-rays though should not be considered as the single decisive signal for perceiving COVID-19. Other elements such as medical history, physical examination, and pathogenic microorganism tests should be implemented during the clinical diagnostic procedure. Future studies should consider clinical comparisons and external validation.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/diagnostics13040584/s1, Figure S1: Funnel plots for recruited 9 articles according to sensitivity and specificity. Figure S2: Medical images of COVID-19, pneumonia patients, and normal individual.

Author Contributions

Conceptualization, I.-S.T.; methodology, I.-S.T. and P.-C.H.; software, T.-H.H.; validation, I.-S.T. and P.-C.H.; formal analysis, I.-S.T.; investigation, I.-S.T. and W.-L.S.; resources, S.-C.C.; data curation, I.-S.T., P.-C.H. and W.-L.S.; writing—original draft preparation, I.-S.T.; writing—review and editing, I.-S.T. and.; visualization, I.-S.T. and T.-H.H.; funding acquisition. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by Taipei Tzu Chi Hospital; grant number TCRD-TPE-111-08 (2/2).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data used to support the findings of this study are including within the article.

Acknowledgments

We acknowledge valuables comments and suggestions from three anonymous reviewers.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Hu, B.; Guo, H.; Zhou, P.; Shi, Z.L. Characteristics of SARS-CoV-2 and COVID-19. Nat. Rev. Microbiol. 2021, 19, 141–154. [Google Scholar] [CrossRef]
  2. Wu, C.; Chen, X.; Cai, Y.; Xia, J.; Zhou, X.; Xu, S.; Huang, H.; Zhang, L.; Zhou, X.; Du, C.; et al. Risk Factors Associated with Acute Respiratory Distress Syndrome and Death in Patients with Coronavirus Disease 2019 Pneumonia in Wuhan, China. JAMA Intern. Med. 2020, 180, 934–943. [Google Scholar] [CrossRef] [PubMed]
  3. Ayoubkhani, D.; Khunti, K.; Nafilyan, V.; Maddox, T.; Humberstone, B.; Diamond, I.; Banerjee, A. Post-covid syndrome in individuals admitted to hospital with COVID-19: Retrospective cohort study. BMJ 2021, 372, n693. [Google Scholar] [CrossRef]
  4. Han, J.; Shi, L.X.; Xie, Y.; Zhang, Y.J.; Huang, S.P.; Li, J.G.; Wang, H.R.; Shao, S.F. Analysis of factors affecting the prognosis of COVID-19 patients and viral shedding duration. Epidemiol. Infect. 2020, 148, e125. [Google Scholar] [CrossRef]
  5. Sauter, A.P.; Andrejewski, J.; De Marco, F.; Willer, K.; Gromann, L.B.; Noichl, W.; Kriner, F.; Fischer, F.; Braun, C.; Koehler, T.; et al. Optimization of tube voltage in X-ray dark-field chest radiography. Sci. Rep. 2019, 9, 8699. [Google Scholar] [CrossRef] [PubMed]
  6. Berk, I.A.H.V.D.; Kanglie, M.M.N.P.; van Engelen, T.S.R.; Altenburg, J.; Annema, J.T.; Beenen, L.F.M.; Boerrigter, B.; Bomers, M.K.; Bresser, P.; Eryigit, E.; et al. Ultra-low-dose CT versus chest X-ray for patients suspected of pulmonary disease at the emergency department: A multicentre randomised clinical trial. Thorax 2022, 1–8. [Google Scholar] [CrossRef]
  7. Souid, A.; Sakli, N.; Sakli, H. Classification and Predictions of Lung Diseases from Chest X-rays Using MobileNet V2. Appl. Sci. 2021, 11, 2751. [Google Scholar] [CrossRef]
  8. Xie, Y.; Xu, E.; Bowe, B.; Al-Aly, Z. Long-term cardiovascular outcomes of COVID-19. Nat. Med. 2022, 28, 583–590. [Google Scholar] [CrossRef]
  9. Reisi-Vanani, V.; Lorigooini, Z.; Dayani, M.A.; Mardani, M.; Rahmani, F. Massive intraperitoneal hemorrhage in patients with COVID-19: A case series. J. Thromb. Thrombolysis 2021, 52, 338–344. [Google Scholar] [CrossRef]
  10. Fan, E.; Beitler, J.R.; Brochard, L.; Calfee, C.S.; Ferguson, N.D.; Slutsky, A.S.; Brodie, D. COVID-19-associated acute respiratory distress syndrome: Is a different approach to management warranted? Lancet Respir. Med. 2020, 8, 816–821. [Google Scholar] [CrossRef]
  11. Guan, W.J.; Ni, Z.Y.; Hu, Y.; Liang, W.H.; Ou, C.Q.; He, J.X.; Liu, L.; Shan, H.; Lei, C.L.; Hui, D.S.C.; et al. Clinical characteristics of coronavirus disease 2019 in China. N. Engl. J. Med. 2020, 382, 1708–1720. [Google Scholar] [CrossRef] [PubMed]
  12. Dabravolski, S.A.; Kavalionak, Y.K. SARS-CoV-2: Structural diversity, phylogeny, and potential animal host identification of spike glycoprotein. J. Med. Virol. 2020, 92, 1690–1694. [Google Scholar] [CrossRef] [PubMed]
  13. Murray, E.; Tomaszewski, M.; Guzik, T.J. Binding of SARS-CoV-2 and angiotensinconverting enzyme 2: Clinical implications. Cardiovasc. Res. 2020, 116, e87–e89. [Google Scholar] [CrossRef]
  14. Bernheim, A.; Mei, X.; Huang, M.; Yang, Y.; Fayad, Z.A.; Zhang, N.; Diao, K.; Lin, B.; Zhu, X.; Li, K.; et al. Chest CT findings in coronavirus disease-19 (COVID-19): Relationship to duration of infection. Radiology 2020, 295, 200463. [Google Scholar] [CrossRef] [PubMed]
  15. Heidinger, B.H.; Kifjak, D.; Prayer, F.; Beer, L.; Milos, R.I.; Röhrich, S.; Arndt, H.; Prosch, H. [Radiological manifestations of pulmonary diseases in COVID-19]. Radiologe 2020, 60, 908–915. [Google Scholar] [CrossRef]
  16. Gravell, R.J.; Theodoreson, M.D.; Buonsenso, D.; Curtis, J. Radiological manifestations of COVID-19: Key points for the physician. Br. J. Hosp. Med. 2020, 81, 1–11. [Google Scholar] [CrossRef]
  17. Xie, X.; Zhong, Z.; Zhao, W.; Zheng, C.; Wang, F.; Liu, J. Chest CT for typical coronavirus disease 2019 (COVID-19) pneumonia: Relationship to negative RT-PCR testing. Radiology 2020, 296, E41–E45. [Google Scholar] [CrossRef]
  18. Wong, H.Y.F.; Lam, H.Y.S.; Fong, A.H.; Leung, S.T.; Chin, T.W.; Lo, C.S.Y.; Lui, M.M.; Lee, J.C.Y.; Chiu, K.W.; Chung, T.W.; et al. Frequency and distribution of chest radiographic findings in patients positive for COVID-19. Radiology 2020, 296, E72–E78. [Google Scholar] [CrossRef]
  19. Tahir, A.M.; Qiblawey, Y.; Khandakar, A.; Rahman, T.; Khurshid, U.; Musharavati, F.; Islam, M.T.; Kiranyaz, S.; Al-Maadeed, S.; Chowdhury, M.E.H. Deep Learning for Reliable Classification of COVID-19, MERS, and SARS from Chest X-ray Images. Cognit. Comput. 2022, 14, 1752–1772. [Google Scholar] [CrossRef]
  20. Alsharif, M.H.; Alsharif, Y.H.; Chaudhry, S.A.; Albreem, M.A.; Jahid, A.; Hwang, E. Artificial intelligence technology for diagnosing COVID-19 cases: A review of substantial issues. Eur. Rev. Med. Pharmacol. Sci. 2020, 24, 9226–9233. [Google Scholar]
  21. Menze, B.H.; Jakab, A.; Bauer, S.; Kalpathy-Cramer, J.; Farahani, K.; Kirby, J.; Burren, Y.; Porz, N.; Slotboom, J.; Wiest, R.; et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging 2015, 34, 1993–2024. [Google Scholar] [CrossRef]
  22. Li, C.; Xie, Y.; Sun, J. 3D intracranial artery segmentation using a convolutional autoencoder. In Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Kansas City, MO, USA, 13–16 November 2017; pp. 714–717. [Google Scholar]
  23. Meijs, M.; Patel, A.; van de Leemput, S.C.; Prokop, M.; van Dijk, E.J.; de Leeuw, F.-E.; Meijer, F.J.A.; van Ginneken, B.; Manniesing, R. Robust segmentation of the full cerebral vasculature in 4D CT of suspected stroke patients. Sci. Rep. 2017, 7, 15622. [Google Scholar] [CrossRef] [PubMed]
  24. Jacobi, A.; Chung, M.; Bernheim, A.; Eber, C. Portable chest X-ray in coronavirus disease-19 (COVID-19): A pictorial review. Clin. Imaging 2020, 64, 35–42. [Google Scholar] [CrossRef] [PubMed]
  25. Cozzi, A.; Schiaffino, S.; Arpaia, F.; Della Pepa, G.; Tritella, S.; Bertolotti, P.; Menicagli, L.; Monaco, C.G.; Carbonaro, L.A.; Spairani, R.; et al. Chest X-ray in the COVID-19 pandemic: Radiologists’ real-world reader performance. Eur. J. Radiol. 2020, 132, 109272. [Google Scholar] [CrossRef] [PubMed]
  26. Ozsahin, I.; Sekeroglu, B.; Musa, M.S.; Mustapha, M.T.; Ozsahin, D.U. Review on Diagnosis of COVID-19 from Chest CT Images Using Artificial Intelligence. Comput. Math. Methods Med. 2020, 2020, 9756518. [Google Scholar] [CrossRef]
  27. Whiting, P.; Rutjes, A.W.; Reitsma, J.B.; Bossuyt, P.M.; Kleijnen, J. The development of QUADAS: A tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med. Res. Methodol. 2003, 3, 25. [Google Scholar] [CrossRef]
  28. Doebler, P.; Holling, H. Meta-Analysis of Diagnostic Accuracy with mada. Cran.r-project.org Web site. Available online: https://cran.r-project.org/web/packages/mada/vignettes/mada.pdf (accessed on 15 October 2022).
  29. Borkowski, A.A.; Viswanadham, N.A.; Thomas, L.B.; Guzman, R.D.; Deland, L.A.; Mastorides, S.M. Using Artificial Intelligence for COVID-19 Chest X-ray Diagnosis. Fed. Pract. 2020, 37, 398–404. [Google Scholar] [CrossRef]
  30. Zokaeinikoo, M.; Kazemian, P.; Mitra, P.; Kumara, S. AIDCOV: An Interpretable Artificial Intelligence Model for Detec-tion of COVID-19 from Chest Radiography Images. ACM Trans. Manag. Inf. Syst. 2021, 12, 1–20. [Google Scholar] [CrossRef]
  31. Keidar, D.; Yaron, D.; Goldstein, E.; Shachar, Y.; Blass, A.; Charbinsky, L.; Aharony, I.; Lifshitz, L.; Lumelsky, D.; Neeman, Z.; et al. COVID-19 Classification of X-ray Images Using Deep Neural Networks. Eur. Radiol. 2021, 31, 9654–9663. [Google Scholar] [CrossRef]
  32. Ahmed, S.; Hossain, T.; Hoque, O.B.; Sarker, S.; Rahman, S.; Shah, F.M. Automated COVID-19 Detection from Chest X-RayImages: A High Resolution Network (HRNet) Approach. SN Comput. Sci. 2021, 2, 294. [Google Scholar] [CrossRef]
  33. Kikkisetti, S.; Zhu, J.; Shen, B.; Li, H.; Duong, T. Deep-learning convolutional neural networks with transfer learning accurately classify COVID19 lung infection on portable chest radiographs. PeerJ 2020, 8, e10309. [Google Scholar] [CrossRef] [PubMed]
  34. Shibly, K.H.; Dey, S.K.; Islam, M.T.U.; Rahman, M.M. COVID Faster R-CNN: A Novel Framework to Diagnose Novel Coronavirus Disease (COVID-19) in X-ray Images. Inform. Med. Unlocked 2020, 20, 100405. [Google Scholar] [CrossRef] [PubMed]
  35. Gomes, J.C.; Barbosa, V.A.d.F.; Santana, M.A.; Bandeira, J.; Valenca, M.J.S.; de Souza, R.E.; Ismael, A.M.; dos Santos, W.P. IKONOS. An intelligent tool to support diagnosis of COVID-19 by texture analysis of X-ray images. Res. Biomed. Eng. 2020, 38, 15–28. [Google Scholar] [CrossRef]
  36. Ko, H.; Chung, H.; Kang, W.S.; Kim, K.W.; Shin, Y.; Kang, S.J.; Lee, J.H.; Kim, Y.J.; Kim, N.Y.; Jung, H.; et al. COVID-19 Pneumonia Diagnosis Using a Simple 2D Deep Learning Framework with a Single Chest CT Image: Model Development and Validation. J. Med. Internet Res. 2020, 22, e19569. [Google Scholar] [CrossRef] [PubMed]
  37. Sharma, V.; Dyreson, C. COVID-19 Screening Using Residual Attention Network an Artificial Intelligence Approach. arXiv 2020, arXiv:2006.16106. [Google Scholar]
  38. Deeks, J.J. Systematic Reviews of Evaluations of Diagnostic and Screening Tests. Br. Med. J. 2001, 323, 157–162. [Google Scholar] [CrossRef] [PubMed]
  39. Atzrodt, C.L.; Maknojia, I.; McCarthy, R.D.P.; Oldfield, T.M.; Po, J.; Ta, K.T.L.; Stepp, H.E.; Clements, T.P. A Guide to COVID-19: A global pandemic caused by the novel coronavirus SARS-CoV-2. FEBS J. 2020, 287, 3633–3650. [Google Scholar] [CrossRef]
  40. Futoma, J.; Simons, M.; Panch, T.; Doshi-Velez, F.; Celi, L.A. The myth of generalisability in clinical research and machine learning in health care. Lancet Digit. Health 2020, 2, e489–e492. [Google Scholar] [CrossRef]
  41. Krarup, M.M.K.; Krokos, G.; Subesinghe, M.; Nair, A.; Fischer, B.M. Artificial Intelligence for the Characterization of Pulmonary Nodules, Lung Tumors and Mediastinal Nodes on PET/CT. Semin. Nucl. Med. 2021, 51, 143–156. [Google Scholar] [CrossRef]
  42. Roberts, M.; Driggs, D.; Thorpe, M.; Gilbey, J.; Yeung, M.; Ursprung, S.; Aviles-Rivero, A.I.; Etmann, C.; McCague, C.; Beer, L.; et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat. Mach. Intell. 2021, 3, 199–217. [Google Scholar] [CrossRef]
  43. Zheng, C.; Deng, X.; Fu, Q.; Zhou, Q.; Feng, J.; Ma, H.; Liu, W.; Wang, X. Deep Learning-based Detection for COVID-19 from Chest CT using Weak Label. medRxiv 2020. [Google Scholar] [CrossRef]
  44. Alizadehsani, R.; Roshanzamir, M.; Hussain, S.; Khosravi, A.; Koohestani, A.; Zangooei, M.H.; Abdar, M.; Beykikhoshk, A.; Shoeibi, A.; Zare, A.; et al. Handling of uncertainty in medical data using machine learning and probability theory techniques: A review of 30 years (1991–2020). Ann. Oper. Res. 2021, 1–42, advance online publication. [Google Scholar] [CrossRef] [PubMed]
  45. Cohen, J.P.; Morrison, P.; Dao, L. COVID-19 Image Data Collection. arXiv 2020, arXiv:2003.11597. [Google Scholar]
  46. Calderon-Ramirez, S.; Yang, S.; Moemeni, A.; Colreavy-Donnelly, S.; Elizondo, D.A.; Oala, L.; Rodriguez-Capitan, J.; Jimenez-Navarro, M.; Lopez-Rubio, E.; Molina-Cabello, M.A. Improving uncertainty estimation with semi-supervised deep learning for COVID-19 detection using chest X-ray images. IEEE Access 2021, 9, 85442–85454. [Google Scholar] [CrossRef] [PubMed]
  47. Zhou, J.; Jing, B.; Wang, Z.; Xin, H.; Tong, H. SODA: Detecting COVID-19 in chest X-rays with semi-supervised open set domain adaptation. IEEE/ACM Trans. Comput. Biol. Bioinform. 2021, 19, 2605–2612. [Google Scholar] [CrossRef]
  48. Santosh, K.; Ghosh, S. COVID-19 imaging tools: How Big Data is big? J. Med. Syst. 2021, 45, 71. [Google Scholar] [CrossRef]
  49. Wei, Q.; Dunbrack, R.L. The role of balanced training and testing data sets for binary classifiers in bioinformatics. PLoS ONE 2013, 8, e67863. [Google Scholar] [CrossRef]
  50. Tabik, S.; Gomez-Rios, A.; Martin-Rodriguez, J.L.; Sevillano-Garcia, I.; Rey-Area, M.; Charte, D.; Guirado, E.; Suarez, J.L.; Luengo, J.; Valero-Gonzalez, M.A.; et al. COVIDGR dataset and COVID-SDNet methodology for predicting COVID-19 based on chest X-ray images. IEEE J. Biomed. Health Inform. 2020, 24, 3595–3605. [Google Scholar] [CrossRef] [PubMed]
  51. Chen, L.; Rezaei, T. A New Optimal Diagnosis System for Coronavirus (COVID-19) Diagnosis Based on Archimedes optimization algorithm on chest X-ray images. Comput. Intell. Neurosci. 2021, 2021, 7788491. [Google Scholar] [CrossRef]
  52. Sharifrazi, D.; Alizadehsani, R.; Roshanzamir, M.; Joloudari, J.H.; Shoeibi, A.; Jafari, M.; Hussain, S.; Sani, Z.A.; Hasanzadeh, F.; Khozeimeh, F.; et al. Fusion of convolution neural network, support vector machine and Sobel filter for accurate detection of COVID-19 patients using X-ray images. Biomed. Signal Process. Control 2021, 68, 102622. [Google Scholar] [CrossRef]
  53. Blain, M.; Kassin, M.T.; Varble, N.; Wang, X.; Xu, Z.; Xu, D.; Carrafiello, G.; Vespro, V.; Stellato, E.; Ierardi, A.M.; et al. Determination of disease severity in COVID-19 patients using deep learning in chest X-ray images. Diagn. Interv. Radiol. 2021, 27, 20–27. [Google Scholar] [CrossRef]
  54. Bai, H.X.; Wang, R.; Xiong, Z.; Hsieh, B.; Chang, K.; Halsey, K.; Tran, T.M.L.; Choi, J.W.; Wang, D.C.; Shi, L.B.; et al. Artificial intelligence augmentation of radiologist performance in distinguishing COVID-19 from pneumonia of other origin at chest CT. Radiology 2020, 296, E156–E165. [Google Scholar] [CrossRef] [PubMed]
  55. Zhang, K.; Liu, X.; Shen, J.; Li, Z.; Sang, Y.; Wu, X.; Zha, Y.; Liang, W.; Wang, C.; Wang, K.; et al. Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography. Cell 2020, 181, 1423–1433.e11. [Google Scholar] [CrossRef]
  56. Harmon, S.A.; Sanford, T.H.; Xu, S.; Turkbey, E.B.; Roth, H.; Xu, Z.; Yang, D.; Myronenko, A.; Anderson, V.; Amalou, A.; et al. Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets. Nat. Commun. 2020, 11, 4080. [Google Scholar] [CrossRef]
  57. Belfiore, M.P.; Urraro, F.; Grassi, R.; Giacobbe, G.; Patelli, G.; Cappabianca, S.; Reginelli, A. Artificial intelligence to codify lung CT in COVID-19 patients. Radiol. Med. 2020, 125, 500–504. [Google Scholar] [CrossRef]
  58. Li, L.; Qin, L.; Xu, Z.; Yin, Y.; Wang, X.; Kong, B.; Bai, J.; Lu, Y.; Fang, Z.; Song, Q.; et al. Using artificial intelligence to detect COVID-19 and community-acquired pneumonia based on pulmonary CT: Evaluation of the diagnostic accuracy. Radiology 2020, 296, E65–E71. [Google Scholar] [CrossRef] [PubMed]
  59. Wang, S.; Zha, Y.; Li, W.; Wu, Q.; Li, X.; Niu, M.; Wang, M.; Qiu, X.; Li, H.; Yu, H.; et al. A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis. Eur. Respir. J. 2020, 2, 56. [Google Scholar] [CrossRef] [PubMed]
  60. Wu, X.; Hui, H.; Niu, M.; Li, L.; Wang, L.; He, B.; Yang, X.; Li, L.; Li, H.; Tian, J.; et al. Deep learning-based multi-view fusion model for screening 2019 novel coronavirus pneumonia: A multicentre study. Eur. J. Radiol. 2020, 128, 109041. [Google Scholar] [CrossRef] [PubMed]
  61. Mei, X.; Lee, H.C.; Diao, K.Y.; Huang, M.; Lin, B.; Liu, C.; Xie, Z.; Ma, Y.; Robson, P.M.; Chung, M.; et al. Artificial intelligence-enabled rapid diagnosis of patients with COVID-19. Nat. Med. 2020, 26, 1224–1228. [Google Scholar] [CrossRef]
  62. Neri, E.; Miele, V.; Coppola, F.; Grassi, R. Use of CT and artificial intelligence in suspected or COVID-19 positive patients: Statement of the Italian Society of Medical and Interventional Radiology. Radiol. Med. 2020, 125, 505–508. [Google Scholar] [CrossRef]
  63. Wang, X.; Deng, X.; Fu, Q.; Zhou, Q.; Feng, J.; Ma, H.; Liu, W.; Zheng, C. A weakly-supervised framework for COVID-19 classification and lesion localization from chest CT. IEEE Trans. Med. Imaging 2020, 39, 2615–2625. [Google Scholar] [CrossRef] [PubMed]
  64. Singh, D.; Kumar, V.; Vaishali Kaur, M. Classification of COVID-19 patients from chest CT images using multi-objective differential evolution-based convolutional neural networks. Eur. J. Clin. Microbiol. Infect. Dis. 2020, 39, 1379–1389. [Google Scholar] [CrossRef]
  65. Javor, D.; Kaplan, H.; Kaplan, A.; Puchner, S.B.; Krestan, C.; Baltzer, P. Deep learning analysis provides accurate COVID-19 diagnosis on chest computed tomography. Eur. J. Radiol. 2020, 133, 109402. [Google Scholar] [CrossRef] [PubMed]
  66. Soda, P.; D’Amico, N.C.; Tessadori, J.; Valbusa, G.; Guarrasi, V.; Bortolotto, C.; Akbar, M.U.; Sicilia, R.; Cordelli, E.; Fazzini, D.; et al. AIforCOVID: Predicting the clinical outcomes in patients with COVID-19 applying AI to chest-X-rays. An Italian multicentre study. Med. Image Anal. 2021, 74, 102216. [Google Scholar] [CrossRef] [PubMed]
  67. Mulrenan, C.; Rhode, K.; Fischer, B.M. A Literature Review on the Use of Artificial Intelligence for the Diagnosis of COVID-19 on CT and Chest X-ray. Diagnostics 2022, 12, 869. [Google Scholar] [CrossRef] [PubMed]
Diagnostics 13 00584 g001 550
Figure 1. The PRISMA flowchart of study selection.
Figure 1. The PRISMA flowchart of study selection.
Diagnostics 13 00584 g001
Diagnostics 13 00584 g002 550
Figure 2. The risk of bias plot.
Figure 2. The risk of bias plot.
Diagnostics 13 00584 g002
Diagnostics 13 00584 g003 550
Figure 3. The forest plot of sensitivity and specificity results of the nine studies.
Figure 3. The forest plot of sensitivity and specificity results of the nine studies.
Diagnostics 13 00584 g003
Diagnostics 13 00584 g004 550
Figure 4. Pooled estimates for sensitivity and false positive rate analysis. Crosshairs wider with increased weight and colored arbitrarily.
Figure 4. Pooled estimates for sensitivity and false positive rate analysis. Crosshairs wider with increased weight and colored arbitrarily.
Diagnostics 13 00584 g004
Diagnostics 13 00584 g005 550
Figure 5. The summary operating characteristic curve and the AUC were 0.98 (95% CI 0.94–1.00). By default, the point estimate of the pair of sensitivity and false positive rate is also plotted together with a confidence region (i.e., conf.region).
Figure 5. The summary operating characteristic curve and the AUC were 0.98 (95% CI 0.94–1.00). By default, the point estimate of the pair of sensitivity and false positive rate is also plotted together with a confidence region (i.e., conf.region).
Diagnostics 13 00584 g005
Table
Table 1. The general characteristics of the recruited studies.
Table 1. The general characteristics of the recruited studies.
Study NumberFirst AuthorPublication Year CountryStudy
Type		DatasetDeep Learning ModelAll DataCOVIDNon-COVIDSensitivitySpecificity
1Borkowski [29]2020United
States of
America		Case
control		COVID-19/non-
COVID
pneumonia/
COVID-19/non-COVID pneumonia/normal		Microsoft
CustomVision		100050050010095
2Zokaeinikoo [30]2021United
States of
America		Case
control		COVID-19/non-
COVID
infection/
normal		AIDCOV using
VGG-16		5801269553299.399.98
3Keidar [31]2021IsraelRetrospectiveCOVID-
19/normal		RetNet502427360206787.192.4
4Ahmed [32]2021JapanCase
control		COVID/non-
COVID		HRNet1410410100098.5398.52
5Kikkisetti [33]2020United
States of
America		RetrospectiveCOVID/bacterial
pneumonia/
viral
pneumonia/
normal		VGG-16203144515867993
6Shibly [34]2020BangladeshCase
control		COVID/non-
COVID		Faster R-CNN19,25028318,96797.6595.48
7Gomes [35]2020BrazilCase
control		COVID-
19/bacterial and
viral pneumonia		IKONOS6320464585697.799.3
8Ko [36]2020South
Korea		Case
Control		COVID-
19/pneumonia/
normal		DarkNet-191125125100095.1395.3
9Sharma [37]2020United
States of
America		Case
control		COVID-19/non
COVID-19		Residual Att
Net		23912011910096
Table
Table 2. The characteristics and performance of diagnostic test results.
Table 2. The characteristics and performance of diagnostic test results.
Test Result
Total (n = 39,603)PositiveNegative
True conditionCOVID-19 (n = 2976)2804.79171.21
Non COVID-19 (n = 36,627)1259.078835,367.9212
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Tzeng, I.-S.; Hsieh, P.-C.; Su, W.-L.; Hsieh, T.-H.; Chang, S.-C. Artificial Intelligence-Assisted Chest X-ray for the Diagnosis of COVID-19: A Systematic Review and Meta-Analysis. Diagnostics 2023, 13, 584. https://doi.org/10.3390/diagnostics13040584

AMA Style

Tzeng I-S, Hsieh P-C, Su W-L, Hsieh T-H, Chang S-C. Artificial Intelligence-Assisted Chest X-ray for the Diagnosis of COVID-19: A Systematic Review and Meta-Analysis. Diagnostics. 2023; 13(4):584. https://doi.org/10.3390/diagnostics13040584

Chicago/Turabian Style

Tzeng, I-Shiang, Po-Chun Hsieh, Wen-Lin Su, Tsung-Han Hsieh, and Sheng-Chang Chang. 2023. "Artificial Intelligence-Assisted Chest X-ray for the Diagnosis of COVID-19: A Systematic Review and Meta-Analysis" Diagnostics 13, no. 4: 584. https://doi.org/10.3390/diagnostics13040584

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop