Next Article in Journal
Joint Methodology Based on Optical Densitometry and Dynamic Light Scattering for Liver Function Assessment
Next Article in Special Issue
On the Implementation of the Artificial Neural Network Approach for Forecasting Different Healthcare Events
Previous Article in Journal
Highlights on Genetic Polymorphism Associated with Thromboembolic Risk; Implications in Ophthalmic and Autoimmune Disorders—A Review
Previous Article in Special Issue
COVID-19 Prediction Using Black-Box Based Pearson Correlation Approach
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Multimodality Imaging of COVID-19 Using Fine-Tuned Deep Learning Models

Department of Information Systems, College of Computer and Information Sciences, Jouf University, Sakaka 72388, Saudi Arabia
Department of Information Systems, Mansoura University, Mansoura 35516, Egypt
Author to whom correspondence should be addressed.
Diagnostics 2023, 13(7), 1268;
Received: 30 January 2023 / Revised: 22 March 2023 / Accepted: 23 March 2023 / Published: 28 March 2023
(This article belongs to the Special Issue Diagnostic AI and COVID-19)


In the face of the COVID-19 pandemic, many studies have been undertaken to provide assistive recommendations to patients to help overcome the burden of the expected shortage in clinicians. Thus, this study focused on diagnosing the COVID-19 virus using a set of fine-tuned deep learning models to overcome the latency in virus checkups. Five recent deep learning algorithms (EfficientB0, VGG-19, DenseNet121, EfficientB7, and MobileNetV2) were utilized to label both CT scan and chest X-ray images as positive or negative for COVID-19. The experimental results showed the superiority of the proposed method compared to state-of-the-art methods in terms of precision, sensitivity, specificity, F1 score, accuracy, and data access time.

1. Introduction

The first confirmed COVID-19 case, according to the World Health Organization (WHO) reports, was in the central Chinese city of Wuhan on 8 December 2019. COVID-19 was reported as an epidemic in January 2020. From that date onwards, new confirmed cases were reported each day and the COVID-19 virus spread to every continent. According to the WHO, the number of confirmed COVID-19 virus cases is more than 416,614,050 and the number of deaths is more than 5,844,095 [1]. COVID-19 has become a global health crisis and the WHO has declared it a major pandemic. Understanding how the disease spreads and determining how undetected and undocumented cases contribute to the transmission of the virus are major challenges. Although COVID-19 vaccines are available on the market, there is still an inevitable demand for smart healthcare systems for the general population—and, especially, the elderly—so that the expected shortage in doctors in the health sector during the crisis can be promptly addressed. With the spread of COVID-19 to numerous countries in the world, considering the increases in the number of people infected and the number of deaths from day to day, it has become mandatory to diagnose and identify this COVID-19 virus.
All over the world, the COVID-19 virus remains a threat to the economies of countries and the health of people. It has been proven that the disease is transmitted from one person to another and, therefore, delays in discovering the disease lead to the spread of infection through interactions between the healthy and infected patients [2,3,4].
The test to verify a person’s infection with the virus is often implemented by taking samples from the patient’s throat, sputum, or nasopharynx to analyze the PCR of the viral RNA. However, one of the limitations of such tests is their low accuracy [5,6,7,8]. The diagnosis of COVID-19 based on laboratory tests is costly, laborious, time-consuming, and involves a complicated manual process [9]. Therefore, it is recommended that this type of test be replaced with chest CT images, which could be used as one type of early investigative test [3]. The challenge of laboratory testing with CT image analysis is that it demands a radiology expert and takes a significant amount of time. One solution for this challenge is to save valuable time for medical professionals by using an automated analysis system.
Thus, the early diagnosis of COVID-19 would be invaluable in containing the disease outbreak. However, as the diagnosis of COVID-19 based on laboratory tests still displays a low positive rate, as well as being costly, laborious, time-consuming, and manually complicated [9], suggestions have emerged from specialists recommending the use of radiography screening as a primary tool for checking for COVID-19. With increased numbers of infected patients on a daily basis, especially in the epidemiological setting, the bottleneck for this method is the requirement for a large number of expert X-ray specialists to interpret the CT images. Moreover, the similar and overlapping patterns of lung diseases make it difficult for radiologists to point out these slight differences [10]. As a result, there is an urgent need to develop intelligent systems to aid radiologists with fast and accurate results.
Artificial intelligence (AI) has made great strides in recent years. Deep learning and the accompanying innovations offer radiologists a chance to change the radiology scene and improve survival rates. Deep learning neural system models have been applied to a few imaging tasks to date, including image segmentation, classification, and object detection. Deep learning techniques are unique compared to classic AI strategies, which are the significant ones. The latter employ feature extraction strategies in preparation for the computation, while deep learning techniques familiarize themselves with the image information without the requirement for feature extraction. Deep learning provides promising solutions for the analysis of medical images in future applications. With the recent advances in machine learning and artificial intelligence applications, especially in the medical field and including medical image processing, artificial intelligence has become a promising tool that can change diagnosis methods. Deep learning networks, as an artificial intelligence tool, have proven successful in image classification with their unique characteristics of being able to learn image representations automatically and mapping features onto continuous vectors that are subsequently used for predictions. AI applications in radiology [11] are driven by the idea that medical images are sets of data that can be computed by a machine to extract useful information from the properties of the image [12].
Thus, this paper presents a novel technique for the detection of the COVID-19 virus early on based on X-ray and CT chest radiography image analysis using deep learning algorithms. This methodology can be basically classified as image analysis as it comprises steps such as image acquisition, image preprocessing, feature extraction, and the employment of different classifiers. This paper’s contributions can be outlined as follows: (1) We built promising, fine-tuned DL models capable of diagnosing chest X-ray and CT images that showed improvements in their precision, sensitivity, specificity, F1-scores, accuracy, and data access time. The models for COVID-19 detection were trained and their performances were evaluated using novel samples in order to contribute to the control of the epidemic. (2) The proposed system will reduce the pressure on the national healthcare budget by reducing the cost of the physical tests performed in clinical laboratories. It has been reported that the cost of a PCR test is about USD 120–130 [13]. (3) For radiologists, with the spread of COVID-19, a daily flood of CT images need to be analyzed; the proposed techniques can act as an aid for radiologists (regardless of the level of experience) by reducing not only the time required to diagnose CT scans but also the pressure on the radiologist. (4) The disease can be localized by using a Grad-CAM algorithm that visualizes the infected areas of the lungs in chest X-ray and CT images.
The remainder of the paper is laid out as follows: Section 2 contains a literature review; Section 3 presents the proposed methodology; Section 4 contains the experimental results and discussion; and, finally, conclusions are drawn in Section 5.

2. Literature Review

The attention the COVID-19 virus has received from researchers has led to an enormous number of publications. In this section, most of the recently developed systems applying deep learning techniques to COVID-19 detection are explored. Barstugan et al. [14] proposed a coronavirus classification technique for CT images based on machine learning methods. The dataset, consisting of patients from Italy, one of the most endemic regions, comprised 618 images, including 219 images from COVID-19 patients, 224 images from influenza-A viral pneumonia patients, and 175 images from healthy cases. Chowdhury [15] trained four convolutional neural networks (CNNs) to classify CT images into two classes: normal and COVID-19 pneumonia. Linda et al. [9] proposed COVID-net, which is a deep CNN that can recognize and identify the COVID-19 disease from CT images. Shuai et al. [16] adapted the Inception transfer learning model to detect COVID-19 in CT images. Ghoshal et al. [17] presented a Bayesian deep learning classifier to estimate model uncertainty using the transfer learning method with COVID-19 X-ray images. The proposed system differs from the others in tracking disease progression through a deep analysis of the periodical changes in the CT images from the same patient, making it possible to modify treatment and help achieve improved results for patients. It also utilizes an efficient parameterized transfer learning model and smart data augmentation.
Rahimzadeh et al. [18] proposed a linked CNN dependent on the Xception and ResNet50V2 models to characterize COVID-19 cases from chest X-rays. The created framework utilized a dataset that contained 180 images of COVID-19 patients, 6054 images of pneumonia patients, and 8851 images of typical individuals. For each of the eight preparation stages, 633 pictures were selected. The test results exhibited 99.56% precision with 80.53% of COVID-19 cases. Fan et al. [19] proposed a multi-kernel attention network to analyze chest X-ray images from COVID-19 patients. Their model has three stages: a feature extraction stage followed by two parallel multi-kernel-size attention modules and, finally, the classification stage. The experimental results demonstrated improved performance in COVID-19 detection and an accuracy of 98.2%.
Loey et al. [20] presented a generative adversarial network (GAN) using deep learning to analyze COVID-19 from chest X-rays. Their study utilized the three pre-prepared models AlexNet, GoogleNet, and RestNet18. Apostolopoulos et al. [21] presented an exchange learning technique with a CNN for the analysis of COVID-19 cases using chest X-rays. The framework can detect COVID-19 images using five main models: VGG19, Inception, MobileNet, Xception, and Inception-ResNetV2. VGG19 was chosen as the fundamental learning model and it showed 93.48% accuracy. To detect COVID-19 patients, Panwar et al. [22] presented a binary image classification task. The input data were classified using a fine-tuned VGG model. Mishra et al. [23] used deep CNN-based image classification models to differentiate COVID-19 instances using chest CT scan images. Song et al. [24] employed a linear classifier to extract semantic features from CT scans. Jaiswal et al. [2] used a DenseNet201-based deep transfer learning (DTL) model to identify patients with COVID-19. The proposed model uses its own training weights to extract features from the ImageNet dataset. Silva et al. [25] proposed CovidNet, an efficient, voting-based technique for analyzing COVID-19 patterns in CT images.
Allioui et al. [26] proposed a multi-agent deep learning model for enhancement of COVID-19 CT image segmentation. Their proposal was based on multi-agent deep reinforcement learning (DRL), which utilizes a modified version of the Deep Q-Network. Khan et al. [27] proposed a COVID-19 detection method for CT images using deep learning, entropy-controlled optimization, and parallel feature fusion techniques. Their method mainly depends on the AlexNet and VGG16 models. The features are extracted and fused using a parallel positive correlation approach. Then, the entropy-controlled firefly optimization method is employed to select the optimal features. Their best achievement was an accuracy rate of 98%. Rehman et al. [28] proposed a framework for the detection of COVID-19 disease and 14 other types of chest diseases. They employed a convolutional neural network architecture with a soft-max classifier. Then, transfer learning was applied in order to extract deep features, which provided results similar to classic machine learning classification methods. Guo et al. [29] studied COVID-19 diagnosis from chest CT scans via an ensemble learning method based on ordinal regression. Their proposal relies on multi-binary, neuron stick-breaking, and soft label techniques. Mukherjee et al. [30] implemented an advanced deep network architecture with two CT image datasets. The authors utilized the transfer learning strategy with custom-sized input tailored to each type of deep architecture in order to improve the performance. Their best models achieved an average accuracy of 99.4%.
Nasiri and Hasani [31] proposed a method for diagnosing coronavirus disease from X-rays. They used the DenseNet169 deep neural network (DNN). The extracted features were then used as input for the Extreme Gradient Boosting (XGBoost) algorithm to perform the classification task. They achieved accuracy up to 99.78%. Ullah et al. [32] developed an effective COVID-19 detection technique using the Shufflenet CNN by employing three types of images; i.e., chest radiographs, CT scans, and ECG trace images. Nasiri and Alavi [33] proposed a pretrained network named DenseNet169 to extract features from X-ray images. Analysis of variance (ANOVA) was employed as a feature selection method to reduce the computation and time complexity. Then, the selected features were classified with Extreme Gradient Boosting (XGBoost). Their proposed method reached 98.72% accuracy for two-class classification and 92% accuracy for multiclass classification.

3. Proposed Methodology

When building the diagnostic DL model, the chest X-ray and CT images were initially collected. In this study, as illustrated in Figure 1, a publicly available SARS-CoV-2 CT scan dataset was used [34]. The dataset contained 1252 CT scans positive for COVID-19 and 1230 CT scans from non-infected individuals. Another chest X-ray dataset [35] with 6939 sample images was also considered in this study, which included three classes (COVID-19, normal, and pneumonia) with 2313 samples for each category. Section 4.1 provides a detailed description of the datasets utilized. Preprocessing, one of the basic phases in DL learning, is responsible for resizing images to fit the deep learning model. Other processes are also performed to prepare the images for the next phase, such as data augmentation to select the more diverse, more robust datasets to train the model; image grayscale conversion; and image binarization.
Transfer learning is an inevitable step for networks with sparse data (a few hundred or thousand images). Transfer learning is applied to a vast, pretrained network of millions of images. There are two main techniques for applying transfer learning: feature extraction and fine-tuning. For the first technique, only some of the newly added layers are updated and improved during the training phase. In contrast, for the second technique, the weights for all layers are updated, optimized, and customized for the new classification problem. In general, fine-tuning is more effective than the feature extraction technique. Fine-tuning DL models (EfficientB0, VGG-19, DenseNet121, MobileNetV2, etc.) requires extensive resources and time. Initially, the convolution layers learn low-level features and, as the network grows, mid/high-level features are learned. With fine-tuning, these trained low-level features are retained, while the high-level features are trained for new classification problems. In this study, five residual blocks were used: the input, two convolution layers, a max-pooling layer, and an output layer. Subsequently, fine-tuning transfer learning was employed for the first four head layers of the network. The trainable parameters were adjusted along with the supplemented soft-max activation function, which consisted of two or three output neurons relating to binary or three-way classification. Algorithm 1 summarizes the working steps for our DL model.
Algorithm 1: DL Model Working Steps
Input: τ1Dataset containing SARSCOV2 CT—scans
   τ2Dataset containing chest X—ray images
   αLearning rate
   βBatch size
Output: ωCNN final weights
1: Set train and test data sizes
2: Calculate train class weights
3: Feed in a base model (IMAGENET weights)  // EfficientB0, VGG-19, DenseNet121, EfficientB7, or MobileNetV2
4: Generate a new model (transfer learning of low layers)
5: Set model’s top layers  // average pooling, flatten, dense, dropout
6: Set the initial hyperparameters: α, β, ω
7: Train the base models and store the final weights (ω)
8: While (stopping condition not reached) do
9:  Move forward and calculate cross-entropy E c = y l o g p + 1 y log 1 p
10:  Move backward and update the optimizer
11: EndWhile
In this section, the architectures for the CNNs and the transfer learning approach are described. Although CNNs are more similar to vanilla neural networks, the convolution operation is carried out in more than one layer [36]. A simple neural network layer is presented in Equation (1).
z 1 = g ( W 1 a 0 + b 1 )
where z 1 is the current layer; a 0 is the first or input layer; W 1 represents the weights for the first layer; and b 1 is the bias. For instance, for the VGG19 Conv layer [37] in Equation (2), for each channel of x, there is a corresponding channel in the first filter of W C 1 . Equation (3) illustrates the output of the final layer.
z i , j , k 1 = x W c 1 i , j , k + b k , 1 1
z i , j , k 1 = l , m , n = 1 3 W c l , m , n , k 1 a ( i + l 1 , j + m 1 , n ) 0 + b k , 1 1
where i, j, and k correspond to the row, column, and channel for z 1 , respectively; l, m, and n refer to the row, column, and channel number for the filter, respectively; and k denotes the filter being used for the present epoch.
Figure 2 depicts the general convolution operation carried out by the CNNs, which comprised input, convolution, pooling, fully connected, and output layers. The chest X-ray and CT scan dataset images were fed into the input layer.
Figure 3 illustrates an example of the convolution operation with a 6 × 6 matrix using a stride of 2 and a 3 × 3 filter. The stride value defines the moving filter window of the input matrix. The pooling layer, which comes after the convolutional layer, is responsible for reducing the network computational loss; it is a fully connected layer where all neurons receive their inputs from the flattened form of the previous convolutional layer, as illustrated in Figure 4. An example of a flattening operation is depicted in Figure 5. In our study, some of the popular pooling functions considered were the average, L2 norm, minimum, and maximum functions. In addition, the output layer depends on the number of categories required to train the DL models. In our experiments, two different datasets were utilized: the CT scan dataset, which has a binary classification of COVID and non-COVID; and the chest X-ray dataset, which has a triple classification of COVID-19, normal, and pneumonia classes.
The proposed DL models consider the pretrained weights, which help in learning COVID-19 cases. Three main steps follow: in the first step, the training and test datasets for CT scan or chest X-ray images are prepared. Here, the first CT scan dataset was divided into a training set and testing set, and the training data samples were used to learn the utilized models. The split ratio for the training and testing sets was 978:274 for the COVID-19 class and 1006:223 for the non-COVID class, as presented in Table 1. Further, for the second chest X-ray dataset, there was an approximately equal distribution between COVID-19, normal, and pneumonia classes, with 1850 for training and 463 for testing, as reported in Table 2.
In the second step, the base model and the new model are generated. Here, five main models with weights pretrained with ImageNet were used as the base models. The experiments were run many times with the intention of reaching the most suitable hyperparameters, which, in turn, would provide the best results. Table 3 summarizes the hyperparameters for the different DL models used in this study, and Table 4 presents the characteristics of the DL model architectures used in our experiments. Finally, in the third step, the trained weights are updated and then stored. Hence, once the forward propagation is completed, the binary cross-entropy loss function (Equation (4)) is calculated for the output layer.
E c = y l o g p + 1 y log 1 p
where y denotes the true value, and p denotes the probability predicted by the model. Then, when the backpropagation process occurs, it counts the number of changes in the weights. Traveling forward and backward is called one epoch, and during one epoch one sample from the dataset is passed per batch size (BS).

4. Experimental Results and Discussion

4.1. Description of Datasets

In this study, two different open-access sources were used as our basic experimental datasets. The CT scan dataset [34] had a total number of 2481 CT images divided into 1229 normal cases and 1252 COVID-19 patients, whereas the chest X-ray dataset [35] had 6939 sample images consisting of three classes (COVID-19, normal, and pneumonia), with 2313 samples used for each category. Figure 6 shows samples from the CT scan dataset and Figure 7 shows samples from the chest X-ray dataset. Next, the database was split into training and testing sets. The details for the training and testing samples are shown in Table 1 and Table 2 for the CT scan and chest X-ray datasets, respectively, with the results of the different models displayed in the subsequent tables. Table 1 describes the splitting strategy used for the training and testing sets in the experiment for the CT scan dataset. The 80–20 training–testing ratio was adopted in our experiments. The COVID-19 class had 1013 images for training and 239 images for testing, while the normal class had 971 images for training and 258 for the testing set. Table 2 describes the splitting strategy used for the chest X-ray dataset. This dataset had an equal distribution, with 1850 images for training and 463 images for testing in each class. Data augmentation techniques were employed here to effectively increase the number of training samples. The images were augmented through cropping, noising, brightness modifications, contrast modifications, and random flipping.

4.2. Performance Metrics

In order to measure the performance of the proposed DL models, Equations (5)–(9) were used.
S e n s i t i v i t y = TP TP   +   FN  
S p e c i f i t y = T N T N + F P  
P r e c i s i o n = T P T P + F P  
A c c u r a c y = T P + T N T P + F P + T N + F N    
F 1 = 2 R e c a l l P r e c i s i o n   R e c a l l + P r e c i s i o n    

4.3. Results for DL Models

Table 5 reports the results obtained for the five baseline DL models (EfficientB0, VGG-19, DenseNet121, EfficientB7, and MobileNetV2) in the classification task for both the CT scan and chest X-ray datasets. The maximum values are marked in red and underlined. Moreover, Table 5 also reports the run time in seconds for the training and testing sets. The minimum measured values are also marked in red and underlined. As presented in the table, for the CT scan dataset, all five models provided an average accuracy greater than 95%, while for the chest X-ray dataset, all models provided an average accuracy greater than 95% except for the EfficientB7 model, which had 89.04% average accuracy.
For the CT scan dataset, EfficientB0 achieved the highest results (99%) in terms of precision and the F1-score, while VGG-19 and DenseNet121 attained the lowest average scores (96%). The MobileNetV2 model achieved the highest averages (99.18% and 99.19) for sensitivity and accuracy, respectively, while EfficientB7 achieved the highest score (99.74%) in terms of specificity. In contrast, VGG-19 attained the lowest averages for sensitivity, specificity, and accuracy. MobileNetV2 achieved the best results in terms of training and testing run times, with 117.43 s for the training run time and 0.77 s for the testing run time, respectively. In general, the MobileNetV2 model can be considered the superior model in comparison to the other four models.
For the chest X-ray dataset, DenseNet121 achieved the highest average precision, F1-score, specificity, and accuracy with 99.57%, 99.56%, 99.78%, and 99.71% respectively. The EfficientB0 model achieved the highest average sensitivity with 99.77%. The VGG-19 model achieved the lowest average values for all metrics. In general, the DenseNet121 model can be considered the best model, despite MobileNetV2 having the lowest training run time. For more details, see Appendix A.
Table 6 presents the detailed results for the two categories (COVID-19 vs. non-COVID) obtained with the five mentioned DL models with regard to precision, sensitivity, specificity, F1-score, and accuracy. As can be observed in the table, all models achieved accuracy greater than 95%. Regarding the COVID-19 class, the highest values are underlined and marked in red, while the non-COVID class is underlined and marked in green. For the COVID-19 category, the EfficientB0 model was the best in terms of precision and F1-score. However, EfficientB7 achieved the highest sensitivity and specificity, while MobileNetV2 achieved the highest accuracy. Regarding the non-COVID class, the EfficientB0 model was also the best in terms of precision and F1-score. The MobileNetV2 model achieved the highest sensitivity, specificity, and accuracy. For more details, see Appendix A.
Table 7 presents the detailed results for the three-category (COVID-19 vs. normal vs. viral pneumonia) dataset. The highest values for the COVID-19 class are underlined and marked in red, the highest values for the normal class are underlined and marked in green, and, finally, the highest values for the viral pneumonia class are underlined and marked in blue. As can be observed in the table, for the COVID-19 category, DenseNet121 achieved the highest precision, sensitivity, specificity, F1-score, and accuracy with 98.93%, 99.46%, 100%, 99.45%, and 99.63%, respectively. DenseNet121 also achieved the highest values for the normal class for all metrics with 100%, 99.89%, 99.78%, 100%, and 99.92%, respectively. Furthermore, for the viral pneumonia class, it achieved the best results in terms of precision, specificity, and accuracy with 99.78%, 99.89%, and 99.56%, respectively. However, MobileNetV2 surpassed DenseNet121 in terms of the F1-score and sensitivity, achieving 99.35% and 98.91%, respectively. For more details, see Appendix A.
Figure 8 illustrates the 80–20% confusion matrixes for the five models with the CT scan dataset, showing support of 274 and 223 for the COVID-19 and normal classes, respectively. The best model, MobileNetV2, misclassified only 2 out of 274 of the COVID-19 images as normal. Moreover, 2 out of 223 of the normal images were also misclassified. In contrast, the worst model, VGG-19, misclassified ten images in the COVID-19 class as normal. Figure 9 illustrates the 80–20% confusion matrixes for the five models with the chest X-ray dataset, showing support of 457, 463, and 462 for the normal, COVID-19, and viral pneumonia classes, respectively. As can be observed, DenseNet121 was the best model because it classified all COVID-19 images (463 images) accurately. Classifying COVID-19 images accurately with no errors is a remarkable result, as the control is spread among individuals. Moreover, only one normal image was misclassified as pneumonia, and five pneumonia images were misclassified as COVID-19. However, this is less severe than misclassifying COVID-19 images. It can be observed that the DenseNet121 model was not confused with respect to COVID-19 images.
Figure 10 shows the receiver operating characteristic (ROC) curves for the true-positive rate (TPR) vs. false-positive rate (FPR) for the normal, viral pneumonia, and COVID-19 chest X-ray dataset images. Furthermore, the values of the area under the curve (AUC) are shown in the figure for each category. In addition, Figure 11 shows the ROC curves for the COVID-19 and non-COVID CT scan dataset images. Figure 12 illustrates the results of applying the Grad-CAM algorithm to cover the chest X-ray and CT scan dataset images with heat maps. In the chest X-ray images, the class activation mapping was undertaken by concentrating on particular portions of the normal, viral pneumonia, and COVID-19 classes. In the CT scan images, we applied the Grad-CAM algorithm to COVID-19 and non-COVID classes. In general, in the normal images in both datasets, there was not any kind of opacity that distinguished normal patients from other patients. As depicted in Figure 12, there were no significant localized regions in normal images. In other classes, our models demonstrated the capability to detect the localized regions in the heat maps generated.

4.4. Comparative Study

Table 8 presents a comparison of the best proposed DL model and the state-of-the-art method [38] that obtained the best results in the literature for the CT scan dataset. The model proposed here surpassed the state-of-the-art method in terms of sensitivity, specificity, and accuracy. Moreover, the authors of [38] relied on a deep feature fusion stage extracted from the deep features of AlexNet, GoogleNet, ShuffleNet, and ResNet-18. Therefore, their fusion demands an enormous number of parameters, which, in turn, requires excessive processing time.
Table 9 presents a comparison of the best proposed DL model and the state-of-the-art methods for the chest X-ray dataset. The worst results, such as for the model from [34], were obtained with models pretrained for feature extraction instead of with the transfer learning strategy. In general, the results demonstrate the superiority of the proposed model for the chest X-ray three-class classification task, and a remarkably higher average accuracy result of 99.32% was achieved in the case of the DenseNet121 model. Fine-tuning with a moderate number of layers and parameters contributed to these results appreciably. Similarly, dimension reduction promotes faster learning, which was reflected in the short training time (2906.43 s).

5. Conclusions

In this study, two primary benchmark datasets for CT and X-ray images were used. All images were enhanced and preprocessed as part of the basic DL learning phase. COVID-19 images were classified as positive or negative using a set of fine-tuned transfer learning models. A set of deep learning models were trained and tested in this research study. For the CT scan dataset, all five models provided average accuracy greater than 95%, whereas, for the chest X-ray dataset, all models provided average accuracy greater than 95% except EfficientB7, which achieved 89.04% accuracy. Compared to the methods in the literature, the results show that MobileNetV2 surpassed the best method in terms of sensitivity, specificity, and accuracy, with a training run time of 117.43 s and testing run time of 0.77 s. In addition, DenseNet121 achieved the highest precision, specificity, F1-score, and accuracy for X-ray images with 99.57%, 99.78%, 99.56%, and 99.71% respectively. In the future, a prediction approach based on a combination of these DL models will be considered in order to improve the results. In addition, more complicated and larger datasets will be used for training to assess the robustness of the proposed approach. Moreover, using CT scans for COVID-19 detection may bring extra radiation to patients. In the future, we will consider the radiation dose issue in our proposed models.

Author Contributions

Methodology, A.S.; Software, S.A.E.-G.; Validation, S.A., S.A.E.-G. and A.S.; Writing—original draft, A.S.; Writing—review & editing, S.A. All authors have read and agreed to the published version of the manuscript.


This research was funded by Jouf University, grant number CV-35-41.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data used to support the findings of this study are available at: (15 March 2022) for the CT dataset and: (15 March 2022) for the chest X-ray dataset.


The authors extend their appreciation to the Deanship of Scientific Research at Jouf University for funding this work through research grant no. CV-35-41.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Two-class experiment results.
Table A1. EfficientB0.
Table A1. EfficientB0.
IterationPrecisionSensitivitySpecificityF1-ScoreAccuracyTrain Time (S)Test Time (S)
Table A2. VGG-19.
Table A2. VGG-19.
IterationPrecisionSensitivitySpecificityF1-ScoreAccuracyTrain Time (S)Test Time (S)
Table A3. DenseNet121.
Table A3. DenseNet121.
IterationPrecisionSensitivitySpecificityF1-ScoreAccuracyTrain Time (S)Test Time (S)
Table A4. EfficientB7.
Table A4. EfficientB7.
IterationPrecisionSensitivitySpecificityF1-ScoreAccuracyTrain Time (S)Test Time (S)
Table A5. MobileNetV2.
Table A5. MobileNetV2.
IterationPrecisionSensitivitySpecificityF1-ScoreAccuracyTrain Time (S)Test Time (S)
Three-class experiment results.
Table A6. EfficientB0.
Table A6. EfficientB0.
IterationPrecisionSensitivitySpecificityF1-ScoreAccuracyTrain Time (S)Test Time (S)
Table A7. VGG-19.
Table A7. VGG-19.
IterationPrecisionSensitivitySpecificityF1-ScoreAccuracyTrain Time (S)Test Time (S)
Table A8. DenseNet121.
Table A8. DenseNet121.
IterationPrecisionSensitivitySpecificityF1-ScoreAccuracyTrain Time (S)Test Time (S)
Table A9. EfficientB7.
Table A9. EfficientB7.
IterationPrecisionSensitivitySpecificityF1-ScoreAccuracyTrain Time (S)Test Time (S)
Table A10. MobileNetV2.
Table A10. MobileNetV2.
IterationPrecisionSensitivitySpecificityF-ScoreAccuracyTrain Time (S)Test Time (S)


  1. World Health Organization (WHO). Coronavirus Disease (COVID-19) Pandemic. Available online: (accessed on 17 February 2022).
  2. Jaiswal, A.; Gianchandani, N.; Singh, D.; Kumar, V.; Kaur, M. Classification of the COVID-19 infected patients using DenseNet201 based deep transfer learning. J. Biomol. Struct. Dyn. 2021, 39, 5682–5689. [Google Scholar] [CrossRef] [PubMed]
  3. He, X.; Yang, X.; Zhang, S.; Zhao, J.; Zhang, Y.; Xing, E.; Xie, P. Sample-Efficient Deep Learning for COVID-19 Diagnosis Based on CT Scans. medRxiv 2020. [Google Scholar] [CrossRef]
  4. Cengil, E.; Çınar, A. The effect of deep feature concatenation in the classification problem: An approach on COVID-19 disease detection. Int. J. Imaging Syst. Technol. 2022, 32, 26–40. [Google Scholar] [CrossRef] [PubMed]
  5. Chu, D.K.; Pan, Y.; Cheng, S.M.; Hui, K.P.; Krishnan, P.; Liu, Y.; Ng, D.Y.M.; Wan, C.K.C.; Yang, P.; Wang, Q.; et al. Molecular diagnosis of a novel coronavirus (2019-nCoV) causing an outbreak of pneumonia. Clin. Chem. 2020, 66, 549–555. [Google Scholar] [CrossRef] [PubMed][Green Version]
  6. Zhang, N.; Wang, L.; Deng, X.; Liang, R.; Su, M.; He, C.; Hu, L.; Su, Y.; Ren, J.; Yu, F.; et al. Recent advances in the detection of respiratory virus infection in humans. J. Med. Virol. 2020, 92, 408–417. [Google Scholar] [CrossRef] [PubMed]
  7. Khan, S.H.; Sohail, A.; Khan, A.; Lee, Y.-S. COVID-19 detection in chest X-ray images using a new channel boosted CNN. Diagnostics 2022, 12, 267. [Google Scholar] [CrossRef]
  8. Aboutalebi, H.; Pavlova, M.; Shafiee, M.; Sabri, A.; Alaref, A.; Wong, A. Covid-net cxr-s: Deep convolutional neural network for severity assessment of covid-19 cases from chest x-ray images. Diagnostics 2022, 12, 25. [Google Scholar] [CrossRef]
  9. Wang, L.; Wong, A. Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest radiography images. arXiv 2020, arXiv:2003.09871. [Google Scholar] [CrossRef]
  10. Agrawal, T.; Choudhary, P. FocusCovid: Automated COVID-19 detection using deep learning with chest X-ray images. Evol. Syst. 2021, 13, 1–15. [Google Scholar] [CrossRef]
  11. Qure, A.I. AI assistance for Accelerated Healthcare. Available online: (accessed on 14 March 2023).
  12. Nasiri, H.; Kheyroddin, G.; Dorrigiv, M.; Esmaeili, M.; Nafchi, A.R.; Ghorbani, M.H.; Zarkesh-Ha, P. Classification of COVID-19 in chest X-ray images using fusion of deep features and LightGBM. In Proceedings of the 2022 IEEE World AI IoT Congress (AIIoT), Seattle, WA, USA, 6–9 June 2022; pp. 201–206. [Google Scholar]
  13. Wilhelm, J.; Pingoud, A. Real-time polymerase chain reaction. Chembiochem 2003, 4, 1120–1128. [Google Scholar] [CrossRef]
  14. Barstugan, M.; Ozkaya, U.; Ozturk, S. Coronavirus (COVID-19) Classification using CT Images by Machine Learning Methods. arXiv 2020, arXiv:2003.09424. [Google Scholar]
  15. Chowdhury, M.E.; Rahman, T.; Khandakar, A.; Mazhar, R.; Kadir, M.A.; Mahbub, Z.B.; Islam, K.R.; Khan, M.S.; Iqbal, A.; Al Emadi, N.; et al. Can AI help in screening Viral and COVID-19 pneumonia? arXiv 2020, arXiv:2003.13145. [Google Scholar] [CrossRef]
  16. Shan, F.; Gao, Y.; Wang, J.; Shi, W.; Shi, N.; Han, M.; Xue, Z.; Shen, D.; Shi, Y. Lung infection quantification of covid-19 in ct images with deep learning. arXiv 2020, arXiv:2003.04655. [Google Scholar]
  17. Ghoshal, B.; Tucker, A. Estimating uncertainty and interpretability in deep learning for coronavirus (COVID-19) detection. arXiv 2020, arXiv:2003.10769. [Google Scholar]
  18. Rahimzadeh, M.; Attar, A. A modified deep convolutional neural network for detecting COVID-19 and pneumonia from chest X-ray images based on the concatenation of Xception and ResNet50V2. Inform. Med. Unlocked 2020, 19, 100360. [Google Scholar] [CrossRef]
  19. Fan, Y.; Liu, J.; Yao, R.; Yuan, X. COVID-19 detection from X-ray images using multi-kernel-size spatial-channel attention network. Pattern Recognit. 2021, 119, 108055. [Google Scholar] [CrossRef] [PubMed]
  20. Loey, M.; Smarandache, F.; Khalifa, N.E.M. Within the lack of chest COVID-19 X-ray dataset: A novel detection model based on GAN and deep transfer learning. Symmetry 2020, 12, 651. [Google Scholar] [CrossRef][Green Version]
  21. Apostolopoulos, I.D.; Mpesiana, T.A. Covid-19: Automatic detection from X-ray images utilizing transfer learning with convolutional neural networks. Phys. Eng. Sci. Med. 2020, 43, 635–640. [Google Scholar] [CrossRef][Green Version]
  22. Panwar, H.; Gupta, P.; Siddiqui, M.K.; Morales-Menendez, R.; Bhardwaj, P.; Singh, V. A deep learning and grad-CAM based color visualization approach for fast detection of COVID-19 cases using chest X-ray and CT-Scan images. Chaos Solitons Fractals 2020, 140, 110190. [Google Scholar] [CrossRef]
  23. Mishra, A.K.; Das, S.K.; Roy, P.; Bandyopadhyay, S. Identifying COVID19 from chest CT images: A deep convolutional neural networks based approach. J. Healthc. Eng. 2020, 2020, 8843664. [Google Scholar] [CrossRef]
  24. Song, J.; Wang, H.; Liu, Y.; Wu, W.; Dai, G.; Wu, Z.; Zhu, P.; Zhang, W.; Yeom, K.W.; Deng, K. End-to-end automatic differentiation of the coronavirus disease 2019 (COVID-19) from viral pneumonia based on chest CT. Eur. J. Nucl. Med. 2020, 47, 2516–2524. [Google Scholar] [CrossRef]
  25. Silva, P.; Luz, E.; Silva, G.; Moreira, G.; Silva, R.; Lucio, D.; Menotti, D. COVID-19 detection in CT images with deep learning: A voting-based scheme and cross-datasets analysis. Inform. Med. Unlocked 2020, 20, 100427. [Google Scholar] [CrossRef] [PubMed]
  26. Allioui, H.; Mohammed, M.A.; Benameur, N.; Al-Khateeb, B.; Abdulkareem, K.H.; Garcia-Zapirain, B.; Damaševičius, R.; Maskeliūnas, R. A multi-agent deep reinforcement learning approach for enhancement of COVID-19 CT image segmentation. J. Pers. Med. 2022, 12, 309. [Google Scholar] [CrossRef] [PubMed]
  27. Khan, M.A.; Alhaisoni, M.; Tariq, U.; Hussain, N.; Majid, A.; Damaševičius, R.; Maskeliūnas, R. COVID-19 case recognition from chest CT images by deep learning, entropy-controlled firefly optimization, and parallel feature fusion. Sensors 2021, 21, 7286. [Google Scholar] [CrossRef] [PubMed]
  28. Rehman, N.-u.; Zia, M.S.; Meraj, T.; Rauf, H.T.; Damaševičius, R.; El-Sherbeeny, A.M.; El-Meligy, M.A. A self-activated cnn approach for multi-class chest-related COVID-19 detection. Appl. Sci. 2021, 11, 9023. [Google Scholar] [CrossRef]
  29. Guo, X.; Lei, Y.; He, P.; Zeng, W.; Yang, R.; Ma, Y.; Feng, P.; Lyu, Q.; Wang, G.; Shan, H. An ensemble learning method based on ordinal regression for COVID-19 diagnosis from chest CT. Phys. Med. Biol. 2021, 66, 244001. [Google Scholar] [CrossRef]
  30. Mukherjee, H.; Ghosh, S.; Dhar, A.; Obaidullah, S.M.; Santosh, K.; Roy, K. Deep neural network to detect COVID-19: One architecture for both CT Scans and Chest X-rays. Appl. Intell. 2021, 51, 2777–2789. [Google Scholar] [CrossRef]
  31. Nasiri, H.; Hasani, S. Automated detection of COVID-19 cases from chest X-ray images using deep neural network and XGBoost. Radiography 2022, 28, 732–738. [Google Scholar] [CrossRef]
  32. Ullah, N.; Khan, J.A.; El-Sappagh, S.; El-Rashidy, N.; Khan, M.S. A Holistic Approach to Identify and Classify COVID-19 from Chest Radiographs, ECG, and CT-Scan Images Using ShuffleNet Convolutional Neural Network. Diagnostics 2023, 13, 162. [Google Scholar] [CrossRef] [PubMed]
  33. Nasiri, H.; Alavi, S. A novel framework based on deep learning and ANOVA feature selection method for diagnosis of COVID-19 cases from chest X-ray Images. arXiv 2021, arXiv:2110.06340. [Google Scholar] [CrossRef]
  34. Angelov, P.; Soares, E.A. SARS-CoV-2 CT-scan dataset: A large dataset of real patients CT scans for SARS-CoV-2 identification. medRxiv 2020, 2020, 4. [Google Scholar]
  35. Asraf, A. Covid19 Penumonia Normal Chest Xray PA Dataset. Mendeley Data 2021. [Google Scholar] [CrossRef]
  36. Goodfellow, I.; Bengio, Y.; Courville, A. Deep Learning; MIT Press: Cambridge, MA, USA, 2016. [Google Scholar]
  37. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
  38. Ragab, D.A.; Attallah, O. FUSI-CAD: Coronavirus (COVID-19) diagnosis based on the fusion of CNNs and handcrafted features. PeerJ. Comput. Sci. 2020, 6, e306. [Google Scholar] [CrossRef] [PubMed]
  39. Singh, D.; Kumar, V.; Kaur, M. Classification of COVID-19 patients from chest CT images using multi-objective differential evolution–based convolutional neural networks. Eur. J. Clin. Microbiol. Infect. Dis. 2020, 39, 1379–1389. [Google Scholar] [CrossRef] [PubMed]
  40. Ozturk, T.; Talo, M.; Yildirim, E.A.; Baloglu, U.B.; Yildirim, O.; Acharya, U.R. Automated detection of COVID-19 cases using deep neural networks with X-ray images. Comput. Biol. Med. 2020, 121, 103792. [Google Scholar] [CrossRef]
  41. Haque, K.F.; Abdelgawad, A. A deep learning approach to detect COVID-19 patients from chest X-ray images. AI 2020, 1, 418–435. [Google Scholar] [CrossRef]
  42. Ouchicha, C.; Ammor, O.; Meknassi, M. CVDNet: A novel deep learning architecture for detection of coronavirus (COVID-19) from chest x-ray images. Chaos Solitons Fractals 2020, 140, 110245. [Google Scholar] [CrossRef]
Figure 1. Overall architecture of the proposed methodology.
Figure 1. Overall architecture of the proposed methodology.
Diagnostics 13 01268 g001
Figure 2. General convolution operation carried out by CNNs.
Figure 2. General convolution operation carried out by CNNs.
Diagnostics 13 01268 g002
Figure 3. An example of a convolution operation with a resulting 4 × 4 feature map, filter size of 3, and stride of 2.
Figure 3. An example of a convolution operation with a resulting 4 × 4 feature map, filter size of 3, and stride of 2.
Diagnostics 13 01268 g003
Figure 4. An example of a max-pooling operation with 2 × 2 window size.
Figure 4. An example of a max-pooling operation with 2 × 2 window size.
Diagnostics 13 01268 g004
Figure 5. An example of a flattening operation.
Figure 5. An example of a flattening operation.
Diagnostics 13 01268 g005
Figure 6. Sample COVID-19 and non-COVID CT images.
Figure 6. Sample COVID-19 and non-COVID CT images.
Diagnostics 13 01268 g006
Figure 7. Sample COVID-19, normal, and pneumonia images.
Figure 7. Sample COVID-19, normal, and pneumonia images.
Diagnostics 13 01268 g007
Figure 8. The 80–20% confusion matrixes for the five models with the CT scan dataset.
Figure 8. The 80–20% confusion matrixes for the five models with the CT scan dataset.
Diagnostics 13 01268 g008
Figure 9. The 80–20% confusion matrixes for the five models with the chest X-ray dataset.
Figure 9. The 80–20% confusion matrixes for the five models with the chest X-ray dataset.
Diagnostics 13 01268 g009
Figure 10. ROC curves for the TPR vs. FPR for chest X-ray dataset images.
Figure 10. ROC curves for the TPR vs. FPR for chest X-ray dataset images.
Diagnostics 13 01268 g010
Figure 11. ROC curves for the TPR vs. FPR for CT scan dataset images.
Figure 11. ROC curves for the TPR vs. FPR for CT scan dataset images.
Diagnostics 13 01268 g011
Figure 12. Grad-CAM sample results for COVID-19, normal, and viral pneumonia classes in the chest X-ray dataset and COVID-19 and non-COVID classes in the CT scan dataset.
Figure 12. Grad-CAM sample results for COVID-19, normal, and viral pneumonia classes in the chest X-ray dataset and COVID-19 and non-COVID classes in the CT scan dataset.
Diagnostics 13 01268 g012
Table 1. Frequency of training and testing images in the first dataset (CT scan).
Table 1. Frequency of training and testing images in the first dataset (CT scan).
Training set9781006
Testing set274223
Table 2. Frequency of training and testing images in the second dataset (chest X-ray).
Table 2. Frequency of training and testing images in the second dataset (chest X-ray).
SplitCOVID-19NormalViral Pneumonia
Training set185018501850
Testing set 463463463
Table 3. Values for the hyperparameters used in the experiments.
Table 3. Values for the hyperparameters used in the experiments.
Number of epochs100
Batch size 64
Train–test split ratio 80–20
Optimizer Adam
Learning rate 1 × 10−3
Dropout 0.5
Table 4. Characteristics of the DL model architectures used in the experiments.
Table 4. Characteristics of the DL model architectures used in the experiments.
ModelDefault Input SizeCustom SizeNo. of LayersNo. of Parameters
EfficientB0224 × 22464 × 6423704,384,248
VGG-19224 × 22464 × 641920,159,382
DenseNet121224 × 22464 × 6412107,305,622
EfficientB7224 × 22464 × 6481364,765,158
MobileNetV2224 × 22464 × 645302,592,662
Table 5. Average classification results obtained for both the CT scan and chest X-ray datasets.
Table 5. Average classification results obtained for both the CT scan and chest X-ray datasets.
Model/Metric PrecisionSensitivitySpecificityF1-ScoreAccuracyTrain Time (S)Test Time (S)
CT-Scan Dataset
Chest X-Ray Dataset
Table 6. Classification results of the five DL models for the two-category (COVID-19 vs. normal) CT scan dataset.
Table 6. Classification results of the five DL models for the two-category (COVID-19 vs. normal) CT scan dataset.
Table 7. Classification results for the five DL models for the three-category (COVID-19 vs. normal vs. viral pneumonia) chest X-ray dataset.
Table 7. Classification results for the five DL models for the three-category (COVID-19 vs. normal vs. viral pneumonia) chest X-ray dataset.
Viral pneumonia99.34%97.18%99.67%98.25%98.84%
Viral pneumonia97.40%97.40%98.69%97.40%98.26%
Viral pneumonia99.78%98.91%99.89%99.35%99.56%
Viral pneumonia88.96%85.49%94.67%87.20%91.60%
MobileNetV2COVID-19 97.46%99.56%98.69%98.50%98.98%
Viral pneumonia99.56%97.61%99.78%98.58%99.05%
Table 8. The results obtained with the CT scan dataset compared to state-of-the-art methods.
Table 8. The results obtained with the CT scan dataset compared to state-of-the-art methods.
Silva et al. [25]/2020 EfficientNet family 93.51%79.59%93.98%86.19%87.60%Low accuracy
Singh et al. [39]/2020Multi-objective differential evolution (MODE) model N/A90.50%90.50%N/A93.00%Low accuracy
Panwar et al. [22]/2020VGG19 modelN/A94.04%95.84%N/A95.00%High computational complexity
Jaiswal et al. [2]/2021DenseNet12196.29%96.29%96.21%96.29%96.25%Use of a dataset with a limited number of samples
Dina Ragaband Omneya Attallah [38]/2020FUSI-CAD99.00%99.00%99.00%99.00%99.00%High computational complexity
This work MobileNetV298.00%99.18%99.18%98.00%99.19%Does not use hyperparameter optimization (HPO) algorithms
Table 9. The results obtained for the chest X-ray dataset compared to the state-of-the-art methods.
Table 9. The results obtained for the chest X-ray dataset compared to the state-of-the-art methods.
Authors/YearMethodPrecisionSensitivitySpecificityF1-ScoreAccuracy Drawbacks
Ozturk et al. [40]/2020DarkCovidNet 89.96%85.35%N/A87.37%87.02%Low accuracy
Apostolopoulos and Mpesiana [41]/2020VGG-1992.85% N/A 93.48%Low accuracy
Agrawal et al. [10]/2021FocusCovid95.60%95.20%N/A95.20%95.20%Use of a dataset with a limited number of samples
Cengil et al. [4]/2022AlexNet + EfficientNet-B0 + N2ASNetLarge+ Xception and SVM 99.80%98.60%99.90%99.19%95.70%High computational complexity
Ouchicha et al. [42]/2020CVDNet96.72%96.84%N/A96.68%96.69%High computational complexity
Nasiri and Alavi [31]/2022DenseNet169 +ANOVA + XGBoost99.0293.1810097.5798.72Sensitivity to the number of features selected by the ANOVA
Nasiri et al. [12]DenseNet169 +MobileNet +LightGBM95.1295.2097.1695.6098.54Dependence on only the patient’s chest X-ray
This workDenseNet12199.00%98.98%99.49%98.99%99.32%Does not use hyperparameter optimization (HPO) algorithms
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Almuayqil, S.; Abd El-Ghany, S.; Shehab, A. Multimodality Imaging of COVID-19 Using Fine-Tuned Deep Learning Models. Diagnostics 2023, 13, 1268.

AMA Style

Almuayqil S, Abd El-Ghany S, Shehab A. Multimodality Imaging of COVID-19 Using Fine-Tuned Deep Learning Models. Diagnostics. 2023; 13(7):1268.

Chicago/Turabian Style

Almuayqil, Saleh, Sameh Abd El-Ghany, and Abdulaziz Shehab. 2023. "Multimodality Imaging of COVID-19 Using Fine-Tuned Deep Learning Models" Diagnostics 13, no. 7: 1268.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop