Artificial Intelligence-Assisted RT-PCR Detection Model for Rapid and Reliable Diagnosis of COVID-19

Abstract

With the spread of SARS-CoV-2 variants with higher transmissibility and disease severity, rapid detection and isolation of patients remains a critical step in the control of the pandemic. RT-PCR is the recommended diagnostic test for the diagnosis of COVID-19. The current study aims to develop an artificial intelligence (AI)-driven COVID-19 RT-PCR detection system for rapid and reliable diagnosis, facilitating the heavy burden of healthcare workers. A multi-input deep convolutional neural network (DCNN) is proposed. A MobileNetV2 DCNN architecture was used to predict the possible diagnostic result of RT-PCR fluorescence data from patient nasopharyngeal sample analyses. Amplification curves in FAM (ORF1ab and N genes, SARS-CoV-2) and HEX (human RNAse P gene, internal control) channels of 400 samples were categorized as positive, weak-positive, negative or re-run (unspecific fluorescence). During the network training, HEX and FAM channel images for each sample were simultaneously presented to the DCNN. The obtained DCNN model was verified using another 160 new test samples. The proposed DCNN classified RT-PCR amplification curves correctly for all COVID-19 diagnostic categories with an accuracy, sensitivity, specificity, $F_1$-score, and AUC of the model reported to be 1. Furthermore, the performance of other pre-trained well-known DCNN models was also compared with the MobileNetV2 model using 5-fold cross-validation, and the results showed that there were no significant differences between the other models at the 5% significance level; however, the MobileNetV2 model outperformed others dramatically in terms of the training speed and fast convergence. The developed model can help rapidly diagnose COVID-19 patients and would be beneficial in tackling future pandemics.

1. Introduction

The ongoing severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, local outbreaks of Middle East respiratory syndrome (MERS) and Ebola virus disease (EVD), as well as emerging pan-drug resistant bacterial pathogens represent a major public health concern. Rapid and accurate detection and identification of pathogens is therefore of major significance for the timely management of patients and prevention of the spread of outbreaks. Molecular assays are routinely used for the detection of human pathogens in clinical microbiology laboratories due to their high sensitivity and specificity. To date, coronavirus disease 2019 (COVID-19) has been responsible for over 6.4 million deaths worldwide [1]. With the acceleration of the COVID-19 pandemic due to emerging SARS-CoV-2 immune escape variants with higher transmissibility and disease severity, timely detection and isolation of COVID-19 patients remain a critical step in the control of the pandemic.

Real-time reverse-transcription polymerase chain reaction (RT-PCR) tests have been extensively exploited worldwide since the beginning of the pandemic. These tests detect the SARS-CoV-2 viral RNA signatures to confirm the presence of SARS-CoV-2 in a patient nasopharyngeal sample. RT-PCR combines PCR amplification chemistry with the detection of amplified products via fluorescent probes for determining the presence of pathogen-specific genetic material [2]. The analysis of RT-PCR data is traditionally performed by estimating the threshold cycle (Ct) at which the exponential phase of the fluorescence signal crosses a baseline threshold. Determination of a positive or negative RT-PCR result is based on appropriate setting of threshold. Stochastic variations such as noise in the fluorescence signal may affect the sigmoidal function in RT-PCR assays. Besides threshold determination, amplification artifacts during the amplification process may lead to false positivity in a given sample. Non-specific amplification may result from primer dimerization or due to cross-reaction with non-target genetic material leading to non-specific fluorescence signals in the patient sample [3]. Therefore, analysis of RT-PCR fluorescence data by experienced personnel is fundamental for the accurate diagnosis of COVID-19 patients.

Rapid tracking, identification and isolation of SARS-CoV-2-infected individuals remains a critical step for the implementation of early interventions and curbing the spread of the pandemic. Considering the large volume of patient samples and burden of intensive laboratory testing, the use of deep learning for automated identification and classification of RT-PCR assay results represents a time- and labor-efficient technology. Artificial intelligence (AI) algorithms have been widely used in diagnosis, clinical decision making, patient management, social control, surveillance, therapeutics and vaccine development [4]. In terms of diagnosis, the most explored area of AI for COVID-19 was AI-assisted diagnosis through chest CT imaging using a convolutional neural network (CNN). A number of studies have proposed the use of deep learning for accurate and sensitive diagnosis and have shown AI models to achieve a higher diagnostic performance compared to radiologists [5,6,7,8,9]. The use of different machine learning models to classify patients into mild and severe cases of COVID-19 based on multivariate blood testing has also been previously shown [10]. Alternatively, AI has been considered as an emerging diagnostic tool in clinical microbiology laboratories. Application of AI-based testing has been proposed to improve the turnaround time, quality and cost of microbiological tests and has been applied to automated urine analysis [11] and the detection of bacteria [12,13,14,15], protozoa and parasites [16,17,18,19,20] in clinical samples via image analysis, as well as to antimicrobial susceptibility testing [21,22,23,24].

In the current study, we propose to use a deep convolutional neural network (DCNN) as an automated real-time COVID-19 diagnosis system to classify various RT-PCR fluorescence signals as positive, weak positive, negative or inconclusive. Implementation of the proposed high-performance AI system is applicable to all RT-PCR assays for pathogen detection in diagnostic laboratories and will be highly beneficial in tackling the future pandemics.

2. Materials and Methods

2.1. Dataset

The present study was carried out in the DESAM Institute COVID-19 PCR Laboratory at Near East University in Cyprus. RT-PCR tests were performed using the Diagnovital SARS-CoV-2 Multiplex Real Time PCR Kit (A1, Life Sciences, Istanbul, Turkey) and the Bio-Speedy SARS CoV-2 Double Gene RT-PCR Kit (Bioeksen, Istanbul, Turkey) which target SARS-CoV-2 N1 and ORF1ab genes in the FAM channel and human RNAse P gene (internal control) in the HEX channel. The average daily number of RT-PCR tests performed in the laboratory was 1275.

There were four classes: positive (P), weak positive (WP), negative (N), and re-run (RR). For a sample to be interpreted as positive, the amplification curve should be sigmoidal and the Ct value should be ≤32. The expected Ct value for a weak positive result was 32–35. A Ct value of >35–40 or no amplification was considered negative. Any non-specific non-sigmoidal amplification was interpreted as a re-run sample. Each of the four classes had two input images, FAM and HEX, as illustrated in Figure 1, which were both used to predict the diagnosis classes of COVID-19. These were [$458\times 566\times 3$] pixel color images obtained for 560 patients. Each class contained 140 samples (each sample contained FAM and HEX images of a patient) to ensure equal contributions during the network training. This dataset was divided into three sets: training, validation, and test. These sets consisted of 240, 160, and 160 patient samples. The training and validation sets were used to train the DCNN, and the obtained network model was verified using an unseen test dataset for the final performance evaluation.

2.2. Deep Convolutional Neural Networks

A DCNN can extract image features automatically instead of extracting features explicitly, such as in traditional machine learning classifier approaches. This ability can be achieved using convolutional layers. A typical deep network consists of several convolutional layers, fully connected dense layers, pooling operations, normalization operations, and residual connections. Deep networks represent a very powerful modelling approach and are able to learn large-scale images that are well proven in the ImageNet challenge [25].

2.3. Multi-Input Image DCNN Model

An overview of the proposed framework is shown in Figure 2. The system received FAM and HEX images simultaneously, and the images were subsequently normalized in the range of [−1, 1]. Both normalized images were then presented to the pre-trained deep network model in parallel. The deep network architecture used in this study was MobileNetV2 [26]. This deep network model is computationally the cheapest and one of the fastest architectures with 53 deep layers and approximately a total of 3.5 M parameters in comparison with other ImageNet deep neural network architectures such as ResNets [27,28], Inceptions [29], and VGGs [30]. The first version of the MobileNet architecture [31] introduced a depthwise separable convolution layer that required a lower number of network connection weights in comparison to the use of a traditional convolution layer.

Depthwise separable convolution achieves this by performing depthwise and pointwise convolutions separately for image feature filtering. The performance of MobileNet was slightly improved in the second version of the network by adding the idea of expanding (decompression) and shrinking (compression) the number of channels of the input features before and after depthwise convolution, which is also known as a bottleneck block. During expansion, a modified version of the rectified linear unit activation function (ReLU6) was used. In contrast, no activation function is used (i.e., a linear function is used) when shrinking the channel number at the end of the block; therefore, the relevant information extracted from the input data is not lost. In addition, the number of deep layers was increased in the second version of the MobileNet model. Therefore, to prevent the network from the vanishing gradient problem [32], a residual connection (also known as a skip connection) was added to the bottleneck layer to forward a copy of the input features to the end of this layer before the input features passed through the layer. It is important to note that the residual-connection-based bottleneck block is only performed whenever the number of channels of the input and output of the bottleneck block is the same.

Training this type of large network from scratch, which relies on high runtime memory and multi-processing tasks on the graphics processing unit (GPU), is a difficult task, particularly on personal computers, which have limited processing power. To overcome this difficulty, the existing knowledge gained from the ImageNet database was transferred to the classification problem of COVID-19 RT-PCR diagnosis, which is also known as transfer learning. During the training of the proposed system, the weights of the MobileNetV2 model were frozen to extract only image features. The feature extraction network, MobileNetV2, and its bottleneck block are shown in Figure 3.

There were 345,600 deep features for each image yielded by the MobileNetV2 network. Both deep feature vectors were concatenated before they were connected to the fully connected layer. This layer is also known as the classification layer, in which the relationship between the deep features and classes is learned. Only one dense layer with 40 hidden nodes was used in the proposed framework. The output of the hidden nodes was computed using the standard rectified linear unit (ReLU) activation function. The outputs from the dense layer were first normalized (i.e., batch normalization) and then connected to the softmax layer that outputs four possible classes for the final decision. To determine the predicted class for the HEX and FAM images of the current sample, the prediction of each output node was computed, and the maximum probability among these outputs was selected, as given in (1):

$$\widehat{y}\left(\left[X_{FAM}^{\left(j\right)},X_{HEX}^{\left(j\right)}\right]\right)=\underset{i\in [1,4]}{argmax}\left(a_i\right),$$

(1)

where $\widehat{y}$ denotes the class label of the maximum predicted probability output, $X^{\left(j\right)}$ indicates the $j^{th}$ image data, $a_i$ is the softmax activation output of the corresponding output node as given in (2), and $z_i$ is the sum of the weighted activation outputs of the dense layer for the corresponding softmax output node i.

$$a_i=\frac{e^{z_i}}{{\sum }_{j=1}^4{e}^{z_j}}$$

(2)

To train the connection weights of the proposed DCNN model, a categorical cross-entropy loss function is used, as shown in (3). A more sophisticated version of the backpropagation learning algorithm called Adam optimization was used to compute the gradient of the loss function with respect to the network weights, where the gradients of the softmax layer were backwardly propagated throughout the first trainable layer of the network. Furthermore, dropout regularization was added to the training for the network to become less prone to overfitting. As a result, the learning algorithm distributes the weight balance to all the network’s connection weights to avoid relying on specific connection weights.

$$ϵ({\mathbf{y}}_i,{\mathbf{a}}_i)=\sum _{j=1}^4{y}_{ij}·ln\left(a_{ij}\right)$$

(3)

where $\mathbf{y}$ indicates the one-hot vector of the $i^{th}$ target class, which consists of setting all elements to zero except that the column corresponding to the target class is set to one.

Table 1. Training parameters of the multi-input image DCNN model.

Parameter	Description
Optimization algorithm	Adam optimizer
Learning rate	0.01
Batch size	64
Number of epoch	100
Dropout rate	0.5

lists all training parameters of the DCNN. In this study, the Keras framework [33], which is built on top of Tensorflow, was used to implement the DCNN models. All experiments and analyses were performed using Google Colab Pro, which is equipped with an NVIDIA Tesla P100 16 GB GPU computing processor with 3584 CUDA cores.

2.4. Performance Evaluation Metrics

To assess the performance of the proposed DCNN’s predictions, a $4\times 4$ confusion matrix was generated using actual and predicted classes of the network. For each class, true positive (TP), true negative (TN), false positive (FP) and false negative (FN) were computed separately. Various statistical analyses were carried out as follows: accuracy ($ACC$), $F_1$ score, Matthews’s correlation coefficient ($MCC$), true positive rate ($TPR$) and true negative rate ($TNR$). These statistical tests were computed for each class in order to identify the classification performance of the DCNN on individual classes. It is important to note that a value of the statistical tests close to 1 implies how well the performance of the network is. However, the result of the $MCC$ is the most important feature, and is considered to highlight the performance of the DCNN [34]. The equations of the statistical assessments are given in Equations (4)–(8) respectively. Furthermore, ROC (receiver operator characteristic) curves were plotted for each class and the average AUC (area under the curve) value was computed in order to highlight the achievement level of the model that distinguishes between the positive and negative classes.

$$AC{C}_i=\frac{T{P}_i+T{N}_i}{T{P}_i+T{N}_i+F{P}_i+F{N}_i}$$

(4)

$$F_{1i}=\frac{2T{P}_i}{2T{P}_i+F{P}_i+F{N}_i}$$

(5)

$$MC{C}_i=\frac{T{P}_{i}T{N}_i-F{P}_{i}F{N}_i}{\sqrt{(T{P}_i+F{P}_i)(T{P}_i+F{N}_i)(T{N}_i+F{P}_i)(T{N}_i+F{N}_i)}}$$

(6)

$$TP{R}_i=\frac{T{P}_i}{T{P}_i+F{N}_i}$$

(7)

$$TN{R}_i=\frac{T{N}_i}{T{N}_i+F{P}_i}$$

(8)

3. Experimental Results

The proposed PCR-DCNN model for COVID-19 detection demonstrated fast convergence during the training process, in which the model fully converged at epoch 6, as shown by the accuracy and loss training curves in Figure 4. The final obtained model fits the data well, which can be clearly seen in the loss curves of both the training and validation data (see Figure 4b), where both curves converge with each other through the training process and also yield a very low loss (i.e., near zero). The performance of the obtained model was verified using an unseen test dataset for the final generalization performance. As shown in the generated confusion matrix in Figure 5, all four classes were predicted with 100% accuracy, and all the statistical analyses ($ACC$, $F_1$ score, $MCC$, $TPR$, and $TNR$) were reported to be 1 for each class prediction. Importantly, the rejection success of the network where the $TNR$ outcome is 1 indicated that the network did not have any indecision when predicting the diagnostic class of the corresponding input HEX and FAM images of the patients. Similar results were also obtained when analyzing the ROC curve given in Figure 6, in which the proposed model separated the positive and false classes accurately; hence, the AUC value was 1.

Furthermore, the MobileNetV2 model was compared with other well-known pre-trained models. In this comparison, 5-fold cross-validation was carried out by combining both training and validation sets, that is, a total of 400 patient samples; thus, samples were divided into five equal sets, where each set contained 80 patient samples. It is important to note that each split set also contained the same number of input images from each class, that is, 20 patient input images from each diagnosis class composed of one set of 5-fold cross-validation process. This ensures that the trained models are less prone to overfitting [35]. The current DCNN model was trained 5 times by leaving one split set (or fold) out to test the performance of the model while using the other four sets for model training. In every training session, new parameters of the model were initialized. Finally, the average $ACC$, $F_1$ score, $MCC$, $TPR$, $TNR$ and $AUC$ were reported for the corresponding model. The resultant confusion matrices obtained with 5-fold cross-validation for each model are shown in Figure 7. As can be seen clearly in the models’ confusion matrix, the models predicted a few incorrect classes, mostly the class of negative diagnoses when the samples were actually positive results in COVID-19. This is because when the viral load in the patient sample is low, the magnitude of the fluorescence signal can be weak in the FAM channel, which causes confusion between negative and positive classes; however this situation can be overcome when more of this type of sample data is presented to the network during training. For each pre-trained model,

Table 2. Performance of the pre-trained models obtained from 5-fold cross-validation on the COVID-19 PCR dataset.

Model	$\mathit{A}\mathit{C}\mathit{C}$	${\mathit{F}}_1$	$\mathit{M}\mathit{C}\mathit{C}$	$\mathit{T}\mathit{P}\mathit{R}$	$\mathit{T}\mathit{N}\mathit{R}$	$\mathit{A}\mathit{U}\mathit{C}$
MobileNetV2	0.9975 ± 0.0050	0.9950 ± 0.0100	0.9935 ± 0.0129	0.9950 ± 0.0100	0.9983 ± 0.0033	0.9997 ± 0.0006
VGG16	0.9925 ± 0.0073	0.9850 ± 0.0146	0.9807 ± 0.0186	0.9850 ± 0.0146	0.9950 ± 0.0049	0.9983 ± 0.0031
VGG19	0.9950 ± 0.0100	0.9899 ± 0.0202	0.9874 ± 0.0252	0.9900 ± 0.0200	0.9967 ± 0.0067	0.9982 ± 0.0037
ResNet50	0.9962 ± 0.0075	0.9925 ± 0.0151	0.9904 ± 0.0191	0.9925 ± 0.0150	0.9975 ± 0.0050	1.0000 ± 0.0000
DenseNet201	0.9975 ± 0.0050	0.9950 ± 0.0100	0.9935 ± 0.0129	0.9950 ± 0.0100	0.9983 ± 0.0033	0.9998 ± 0.0005
EfficientNetB7	0.9963 ± 0.0050	0.9925 ± 0.0100	0.9903 ± 0.0129	0.9925 ± 0.0100	0.9975 ± 0.0033	0.9984 ± 0.0032
NASNetLarge	0.9962 ± 0.0075	0.9923 ± 0.0153	0.9902 ± 0.0195	0.9925 ± 0.0150	0.9975 ± 0.0050	1.0000 ± 0.0000
InceptionResNetV2	0.9962 ± 0.0075	0.9925 ± 0.0151	0.9904 ± 0.0191	0.9925 ± 0.0150	0.9975 ± 0.0050	1.0000 ± 0.0001

shows the average performance measure and standard deviation obtained from 5-fold cross-validation. Here, the highest $MCC$ and $F_1$-score of the model were considered as the best model. Consequently, although the performances of all models were similar, the performances of MobileNetV2 and DenseNet201 were slightly better than those of the other models.

The loss values (i.e., Equation (3)) of each model on each patient’s input images (HEX and FAM) that are presented to the models were computed for 160 patients (i.e., test set). Then, to compare the two models’ mean absolute loss values, a t-test was performed where the null hypothesis to be tested was that both mean values were equal. As expected, because the results of the models in Table 2 are very close, the loss values of almost all models were from populations with equal means, so the null hypotheses are not rejected, as shown in Figure 8, where the value 0 indicates that the null hypothesis is not rejected. However, when comparing the VGG19 model against InceptionResNetV2, MobileNetV2, and VGG16, the null hypotheses were rejected, indicating that the mean loss of those models was significantly different at the 5% significance level. Thus, the InceptionResNetV2, MobileNetV2, and VGG16 models are better than the VGG19 model because their mean loss values are smaller than those of the VGG19 model.

Further analysis of the training execution time of the pre-trained models was performed. The results indicated that although all the pre-trained models performed with the same accuracy, the MobileNetV2 architecture outperformed dramatically in terms of the training speed and fast convergence.

Table 3. Total training execution time of the pre-trained models on the PCR COVID-19 dataset.

Model	Execution Time (s)
MobileNetV2	515
VGG16	2030
VGG19	2518
ResNet50	1656
DenseNet201	1497
EfficientNetB7	3433
NASNetLarge	1774
InceptionResNetV2	1741

lists the execution times of the training processes obtained for each model. Each model was trained using 240 patients’ input images (each patient had HEX and FAM images), and the current model was validated using 160 unseen patient data during a total of 100 training epochs, which resulted in the total execution time of the models.

4. Discussion

The global accelerated course of the pandemic necessitates improved and automated platforms in healthcare systems overloaded with mass testing. The interpretation and analysis of RT-PCR fluorescence data require trained personnel, which represents a heavy burden on laboratory staff. Therefore, an AI-assisted RT-PCR detection model represents a valuable system to reduce the workload in healthcare systems and provide a quicker response to the pandemic.

In the current study, a model trained with a DCNN was developed as a COVID-19 RT-PCR diagnosis classification system using fluorescence data from patient nasopharyngeal samples. As RT-PCR data are prone to background noise, the global shape of the fluorescent signal should always be examined by an experienced molecular biologist. This image analysis performed in clinical laboratories is particularly suited for deep convolutional networks. The AI-based detection system developed in this study was able to recognize variations in the curve patterns of the PCR graphs and automatize RT-PCR diagnosis of COVID-19. With the proposed model, false-positive results due to unspecific fluorescence are consequently avoided.

In a previous study by Alouani et al. [36], the authors employed a deep learning model, qPCRdeepNet, which uses a deep convolutional neural network to analyze the fluorescent readings obtained during SARS-CoV-2 RT-PCR in order to improve test specificity. The DCNN model was proposed to detect stochastic variations (i.e., noise) in the fluorescence signal present in the obtained RT-PCRs, which can prevent correct threshold determination and consequently lead to errors in the qualitative interpretation of the results. A high performance of the qPCRdeepNet was reported in the detection of false positive results [36]. Alternatively, Lee and colleagues described a deep learning model trained with the long short-term memory (LSTM) network, in which raw data of fluorescence values measured in each 40 cycles of SARS-CoV-2 RT-PCR were used. In this study, the authors reported a high negative screening performance of the developed model if various other information such as patient’s clinical characteristics, blood test results and chest CT imaging information was available. The models trained with raw RT-PCR data were confirmed to shorten the diagnosis time of RT-PCR [37].

To the best of our knowledge, this is the first study in literature to develop an AI-based detection and classification system for COVID-19 RT-PCR diagnosis using the fluorescent signal and amplification curves. The developed model can be implemented in any clinical laboratory using a web interface by exporting PCR curves from the RT-PCR instruments. Implementation of such AI-driven diagnosis of SARS-CoV-2 will be invaluable for minimizing the time required for the evaluation of the PCR tests and will reduce human intervention in laboratory practice.

This study has some limitations. The proposed network is trained to separate the FAM and HEX images based on the shape of fluorescence data and the corresponding Ct values of each class. The training data in the current study were based on the Ct values recommended by the manufacturer of RT-PCR kits used in this study. One limitation of the model may be the use of different RT-PCR kits with varying cutoff Ct values by users in different laboratories, which may cause the failure of the acquired model to separate the classes properly. Therefore, new images must be obtained using alternative kits in order to train the existing model to obtain the competence of distinguishing the new data.

5. Conclusions

In this study, a multi-input RT-PCR amplification image-based MobileNetV2 deep neural network was proposed for the classification of COVID-19 laboratory diagnoses into four possible classes: positive, weak positive, negative, and re-run. Instead of training such a large deep neural network, the knowledge gained from the ImageNet challenge database was transferred to the classification problem of RT-PCR-based COVID-19 detection. In summary, the proposed multi-input DCNN-based system received both HEX and FAM fluorescent signal images from a patient sample and extracted deep features from both images. Consequently, these features were concatenated and presented to the classification layer of the proposed system. One of the advantages of the MobileNetV2 deep network is that it is computationally inexpensive in comparison with other ImageNet pre-trained deep network models, as the duration of the training process for the images from 400 patients (both training and validation datasets) takes only 515 s to be trained for 100 epochs. The performance of the proposed DCNN system was evaluated statistically, and the proposed RT-PCR COVID-19 detection deep network demonstrated robust and reliable predictions; consequently, all assessment metrics were reported as consistent results.