Effectiveness of Deep Learning Models for Covid-19 Detection on

Real-life Data
1
Dhyey Patel, 2 Kavish Mehta, 3Aparna Menon,4Sara Parveen
1,2, 3, 4
Electrical and Computer Engineering Department, University Of Waterloo, Waterloo, Ontario, Canada

Abstract— Respiratory illnesses represent a formidable
challenge to the medical community, with COVID-19's
emergence in 2020 presenting a particularly alarming threat to
lung health. This development has underscored the pressing need
for more effective methods of studying and diagnosing the
disease. In recent years, Artificial Intelligence (AI) techniques
have been increasingly employed for this purpose. However, the
real-world efficacy of these methods remains a crucial
consideration. The present study seeks to evaluate the accuracy
of AI models on real-world data and improve their performance
through the use of an image augmentation technique, i.e., canny
edge detection. Specifically, we aim to measure the effectiveness
of these models in identifying and diagnosing respiratory
diseases using real-world data, while also exploring how image
augmentation techniques can enhance their accuracy. Through
this research, we hope to shed light on the feasibility and
potential utility of AI-based approaches to respiratory disease
diagnosis and treatment, as well as highlight areas where future
improvements are needed.
Index Terms—COVID-19, Deep Learning, Convolutional
Neural Networks, Image Augmentation, Canny Edge Detection,
Transfer learning
I. INTRODUCTION
The COVID-19 pandemic caused by the SARS-CoV-2
virus has resulted in the need for effective and efficient
diagnostic methods.The predominant method used to
diagnose COVID-19 currently is the RT-PCR (Reverse
Transcription Polymerase Chain Reaction) test. its
limitations in terms of time and resources have led to the
exploration of alternative methods that are more
convenient and accurate. Methods using Deep Learning
models for this purpose have been one of the most
explored alternatives.
In this context, our review paper [1] explored the use
of Deep learning techniques for identifying respiratory
diseases. Our study compared the effectiveness of using
Chest X-Rays(CXRs) and Computerized Tomography
Scans(CT-Scan) as the primary datasets and evaluated
widely-used Convolutional Neural Network(CNN)
models for image segmentation and identification. During
our review, we came across a study by Silva P et al. [2], in
which they applied Deep learning methods to real-life
data and tried to measure the effectiveness of their model.
While we appreciate the cross-data evaluation approach
used by them, we found that the results they achieved
were suboptimal, leaving room for improvement.
Therefore, our research aims to build upon their existing
work.
In contrast to Silva P et al. 's implementation of only
two architectures based on Efficientnet-B0 [3], we
evaluate the performance of other widely-used models
such as ResNet50_v2 [4], AlexNet [5], ChexNet [6], and
MobileNetV2 [7] for this purpose.
In the course of our study, we were inspired by the
work of S K T Hwa et. al. [8] and Shou-Ming Hou et.
al.’s [9] who used advanced Canny edge detection
techniques on CXRs to improve the efficiency of deep
learning models. In our work we investigate the impact of
using Canny edge detection [10] on the efficiency of deep
learning models trained on CT-scans. Our detailed
explanation of our implementation, motivations, findings,
and conclusions can be found in the upcoming sections.
II. DATASET
In this section, we will discuss the datasets that we
used for this study. We used the same datasets as Silva P
et al. [2] in their work. Dataset-1 (SARS-CoV-2 CT-scan
dataset [11]) was used to divide in train and validation sets
respectively. Whereas Dataset-2 (COVID-CT dataset [12])
was used for the test purpose.
A. Distribution of Data
As discussed earlier, Dataset-1 [11] was used for
training and validation purposes, whereas Dataset-2 [12]
was used for testing. Dataset-1 consisted of a total of 2482
CT-scan images and Dataset-2 consisted of a total of 746
CT-scan images. Table-1 summarizes how the data was
split for training, validation and testing.
Table 1: Summary of Data distribution
Dataset Purpose Covid Non-Covid
Training 1000 1000
Dataset-1
[11] Validation 252 230
Dataset-2 [12] Testing 349 397
 In Dataset-2, 349 CT-scan images are of Covid
infected patients and 397 CT-scan images of non-infected
patients. The disparity observed in the count of non-Covid
cases between our work and the study conducted by Silva
P. et al. can be attributed to a temporal difference in the
data sources. Specifically, it is likely that the source
utilized in our analysis had been updated since the time of
Silva P. et al.'s study, leading to variations in the reported
figures for non-COVID patients.
B. Variety in Test Data
Dataset-2, as reported in reference [12], has been
curated by compiling CT-scans from diverse sources to
provide a comprehensive range of data for model testing.
The dataset has been designed with a view to simulate
real-life data that may be encountered by the model in
practical settings. To this end, the dataset has been created
to incorporate several self-induced errors such as textual
information or highlight on images, variations in size, and
changes in contrast, as depicted in Figure-1 respectively.
(1) (2)
(3)
Figure-1: Images from Dataset-2 [12] showing the variety
in the test dataset (Images might be scaled down for
convenience)
III. MOTIVATION AND METHODS
In this section, we will discuss the methodologies
adopted in our research and the driving factors for those.
Upon reviewing Silva P et. al. 's [2] research, we
determined that additional preprocessing techniques could
enhance model accuracies when analyzing real-life data.
Furthermore, we concluded that basing some findings on a
single model is inadequate, and we should include other
widely used models in our analysis. Therefore, drawing
inspiration from SKT Hwa et. al.'s, Shou [8]-Ming Hou et.
al.’s [9] paper and our own review paper [1], we
incorporated canny edge detection as a preprocessing
technique and evaluated the results using several other
Convolutional Neural Network (CNN) architectures such as
ResNet50_v2 [4], AlexNet [5], ChexNet [6], and
MobileNetV2 [7].
A. Pre-processing Motivation
A.1 Canny Edge Detection [10]
In medical diagnostics, it is necessary to have good
accuracy in a model. One of the most reliable edge
detection techniques developed to date is the Canny edge
detection algorithm, which is highly precise and rigorously
defined. This method has gained recognition for its ability
to provide consistent and accurate edge detection results.
SKT Hwa et. al. 's [8] and Shou-Ming hou et. al.’s [9]
research validates the above statement and motivated us to
use canny edge detection for our research.
In figure-2 we can see a CT-scan image and its
corresponding canny edge processed image.
(1) (2)
Figure-2: (1) Original CT-scan image [12] (2) Canny-edge
processed image
By utilizing this pre-processing technique, we can
extract the essential edge-based features from a CT-scan
and eliminate any extraneous features that may have been
introduced during the image extraction process. This is
particularly relevant in real-world scenarios where CT-scan
images from diverse sources may be susceptible to varying
degrees of quality defects.
Although the original form of Canny edge detection
is not without its imperfections, there are several
techniques for improving it, as demonstrated by the
research of SKT Hwa et. al. and Shou-Ming Hou et. al..
Nevertheless, utilizing Canny edge detection in its original
form has yielded enhanced accuracy for machine learning
models.
B. Modeling Motivation
In the course of our research, we endeavored to
address the primary concerns noted in Silva P. et al.'s [2]
work pertaining to the vanishing gradient and bulky nature
of the models. To mitigate these issues, we employed four
distinct models: ResNet50_v2 [4], AlexNet [5], ChexNet
[6] and MobileNetV2 [7].
B.1 ResNet50_v2
ResNet50_v2 [4], which includes 48 convolution
layers, 1 MaxPooling layer, and 1 Average Pooling layer,
was selected as our base model. This model was trained on
Imagenet [13] weights through transfer learning. The
architecture features a convolutional layer with a 7x7
kernel size, 64 kernels with a stride of size 2, and a Max
Pooling layer forming a single unique layer. The following
layers include 1x1, 64, 3x3, 64, and 1x1, 256 kernels
repeated three times, 1x1, 128, 3x3, 128, and 1x1, 512
kernels repeated four times, and 1x1, 256, 3x3, 256, and
1x1, 1024 kernels repeated six times. Nine layers of 1x1,
512 kernels with two of 3x3, 512, and 1x1, 2048 kernels
were replicated three times, totaling nine layers. The final
layer consists of a fully connected layer with 1000 nodes
and a softmax function. Skip connections were employed
to address the vanishing gradient issue by providing
alternative paths for the gradient to converge with three
shortcut connections.
B.2 AlexNet
AlexNet [5] is a smaller model and also the
architecture uses ReLU (rectified linear unit) activation
functions in the hidden layers which helps with vanishing
gradient problem and dropout regularization to prevent
overfitting.
AlexNet has five convolutional layers, starting with
96 filters of 11×11 size, a stride of 4 pixels, and no
padding. The other layers have different filter sizes,
numbers of filters, and padding sizes. The model also has
three fully connected layers, with the first two having 4096
neurons and the last one having 1000 neurons. The output
layer uses softmax activation to predict class probabilities.
So, we can see that the size of the model is very small
as compared to other models and yet we are able to rectify
the errors that were present in Silva P et al.’s paper [2].
B.3 MobileNetV2
MobileNetV2 [7] uses a building block called the
"inverted residual with linear bottleneck". This building
block consists of three consecutive layers: a 1x1
convolutional layer that expands the number of channels, a
depthwise separable convolutional layer that reduces the
spatial dimensions, and another 1x1 convolutional layer
that compresses the number of channels back to the
original size.
The output of the building block is obtained by
adding the input feature maps to the output feature maps,
which creates a "shortcut" connection that improves the
information flow through the network and helps to avoid
the vanishing gradient problem.
Towards the conclusion of the network, there is a
layer for global average pooling, followed by a dropout
layer, and ultimately a fully connected layer that utilizes
the softmax activation function.
B.4 ChexNet
ChexNet [6] is a convolutional neural network (CNN)
architecture that was specifically designed for chest X-ray
image analysis and diagnosis of 14 different chest diseases.
The ChexNet architecture uses a modified version of the
Google Inception v3 [14] CNN architecture and consists of
121 layers. Due to its significant results on the X-ray
images, we decided to test it on the CT-scans as well.
ChexNet uses a 121-layer CNN architecture with skip
connections, which allows for better information flow
through the network and faster convergence during
training. The bulk of the ChexNet architecture consists of a
series of Inception modules. An Inception module is a
group of convolutional layers with different filter sizes and
types that are concatenated together to form a single
output. The Inception modules help ChexNet extract
features at different scales and levels of abstraction.
IV. RESULTS
Table 2: Summary of Dataset-2 [9]
Model Train Val Train Val
Accuracy Accuracy Loss Loss
ResNet50 96.78% 90.59% 11.19% 21.37%
AlexNet 91.12% 90.25% 30.81% 37.63%
MobileNet 90.90% 82.76% 35.78% 39.45%
ChexNet 88.70% 81.50% 47.30% 60.05%
After training all the four models, ResNet50v2 gave
the most satisfactory results on the validation set without
overfitting with validation accuracy of 94.59% as well as
96.78% accuracy on training data.
A. Comparison
In our research, we have evaluated the performance of
four different neural network models, namely ResNet50_v2
[4], AlexNet [5], ChexNet [6] and MobileNetV2 [7]. Our
primary objective was to classify a disease accurately,
therefore, the accuracy of the models was of utmost
importance, even though we also aimed to increase the
speed of execution by using smaller models.
We found that smaller-sized AlexNet and
MobileNetV2 models resulted in lower accuracy, and thus,
it was necessary to use a well-generalized model. We chose
ResNet50_v2 as our primary model due to its higher
accuracy and the presence of skip connections that reduced
complexity and prevented over-fitting. The ResNet50_v2
model converged easily to a straight line with minimal
validation and training loss as the number of epochs
increased, as demonstrated by Figure-3.
Figure-3: ResNet50_v2 Accuracy vs Epochs
In contrast, ChexNet showed quick convergence in the
initial epochs with high accuracy but then oscillated
between local maximum and local minimum, which was
due to its high complexity and variance, as illustrated in
Figure-4. Therefore, we concluded that ResNet50_v2 was
the most suitable model for our study.
Figure-4: ChexNet Accuracy vs Epochs
B. Confusion Matrix
Figure-5: Confusion Matrix of ResNet50_v2
ResNet50_v2 has been identified as the most stable
model as per the discussion above and it can be seen from
the confusion matrix in Figure-5. The total accuracy of the
model is calculated to be 62.46%. Further analysis of the
model reveals its precision, recall, and F1-score to be
0.5831, 0.6934, and 0.6335, respectively. The obtained
precision and recall values indicate that the model has a
higher true positive rate than false positive rate, which is
promising. However, the F1-score suggests that there is
scope for improvement in balancing precision and recall,
which may further enhance the overall performance of the
model.
V. CONCLUSION
From this research, we can conclude that the efficiency
of Deep Learning models on real-life images is not up to the
mark in medical diagnostics related issues. There can be
many irregularities in the real-life data which the model
might have never seen before, and hence leading into
ambiguous results.
A. Limitations
During the research, potential challenges related to the
deployment of the model in an open environment were
identified. Dataset-2 [12] contains images with text or
highlighted areas that may negatively affect the model's
accuracy. Additionally, CT scans of patients with diseases
other than COVID-19 may interfere with the model's
predictions. Some images that produce a blurry output after
canny edge detection may also be misclassified.
Furthermore, the model's inability to recognize patterns
outside of its learned dataset may lead to incorrect
predictions in the future.
B. Future advancement possibilities
This section will explore potential avenues for future
research and development to build on the findings
presented. The literature consulted for the Canny Edge
[8][9] method suggests that an improved version of the
Canny Edge algorithm could potentially enhance the
accuracy of the model.
Additionally, rather than solely testing the model with
error-inducing data, there is the possibility of training the
model using such data to enable better generalization of the
model. Also an ensemble of all these models may work
more efficiently than each model individually.
VI. REFERENCES
[1] Patel, D., Mehta, K., Menon, A., Parveen, S. (2023).
Covid-19 detection using medical images of the
human thoracic region with the help of deep learning
models. University Of Waterloo
[2] Silva P, Luz E, Silva G, Moreira G, Silva R, Lucio D,
Menotti D. COVID-19 detection in CT images with
deep learning: A voting-based scheme and
cross-datasets analysis. Inform Med Unlocked.
 2020;20:100427. doi: 10.1016/j.imu.2020.100427. 248-255). Ieee.
Epub 2020 Sep 14. PMID: 32953971; PMCID:
PMC7487744. [14] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., &
Wojna, Z. (2016). Rethinking the inception
[3] Mingxing, T., & Le Quoc, V. E. Rethinking Model architecture for computer vision. In Proceedings of
Scaling for Convolutional Neural Networks. arXiv. the IEEE conference on computer vision and pattern
May 2019. arXiv preprint arXiv:1905.11946. recognition (pp. 2818-2826).
[4] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep
residual learning for image recognition. In
Proceedings of the IEEE conference on computer
vision and pattern recognition (pp. 770-778).
[5] Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017).
Imagenet classification with deep convolutional
neural networks. Communications of the ACM, 60(6),
84-90.
[6] Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H.,
Duan, T., ... & Ng, A. Y. (2017). Chexnet:
Radiologist-level pneumonia detection on chest x-rays
with deep learning. arXiv preprint arXiv:1711.05225.
[7] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., &
Chen, L. C. (2018). Mobilenetv2: Inverted residuals
and linear bottlenecks. In Proceedings of the IEEE
conference on computer vision and pattern
recognition (pp. 4510-4520).
[8] Hwa, S. K. T., Bade, A., & Hijazi, M. A. (2020,
November). Enhanced Canny edge detection for
Covid-19 and pneumonia X-Ray images. In IOP
Conference Series: Materials Science and Engineering
(Vol. 979, No. 1, p. 012016). IOP Publishing.
[9] Hou, S. M., Jia, C. L., Hou, M. J., Fernandes, S. L., &
Guo, J. C. (2021). A Study on Weak Edge Detection
of COVID-19’s CT Images Based on Histogram
Equalization and Improved Canny Algorithm.
Computational and Mathematical methods in
medicine, 2021.
[10] Canny, J. (1986). A computational approach to edge
detection. IEEE Transactions on pattern analysis and
machine intelligence, (6), 679-698.
[11] Soares, E., Angelov, P., Biaso, S., Froes, M. H., &
Abe, D. K. (2020). SARS-CoV-2 CT-scan dataset: A
large dataset of real patients CT scans for
SARS-CoV-2 identification. MedRxiv, 2020-04.
[12] Zhao, J., Zhang, Y., He, X., & Xie, P. (2020).
Covid-ct-dataset: a ct scan dataset about covid-19.
[13] Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., &
Fei-Fei, L. (2009, June). Imagenet: A large-scale
hierarchical image database. In 2009 IEEE conference
on computer vision and pattern recognition (pp.