ORIGINAL RESEARCH • THORACIC IMAGING

Deep Learning to Determine the Activity of Pulmonary

Tuberculosis on Chest Radiographs
Seowoo Lee, MD  •  Jae-Joon Yim, MD  •  Nakwon Kwak, MD  •  Yeon Joo Lee, MD  •  Jung-Kyu Lee, MD  • 
Ji Yeon Lee, MD  •  Ju Sang Kim, MD  •  Young Ae Kang, MD, PhD  •  Doosoo Jeon, MD, PhD  • 
Myoung-jin Jang, PhD  •  Jin Mo Goo, MD, PhD  •  Soon Ho Yoon, MD, PhD
From the Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital, 101 Daehak-ro, Chongno-gu, Seoul
03080, Korea (S.L., J.M.G., S.H.Y.); Division of Pulmonary and Critical Medicine, Department of Internal Medicine, Seoul National University College of Medi-
cine, Seoul, Korea (J.J.Y., N.K.); Division of Pulmonary and Critical Care Medicine, Department of Internal Medicine, Seoul National University Bundang Hos-
pital, Seongnam, Korea (Y.J.L.); Division of Pulmonary and Critical Care Medicine, Department of Internal Medicine, Seoul Metropolitan Government–Seoul
National University Boramae Medical Center, Seoul, Korea (J.K.L.); Division of Pulmonary and Critical Care Medicine, Department of Internal Medicine, Na-
tional Medical Center, Seoul, Korea (J.Y.L.); Division of Pulmonary and Critical Care Medicine, Department of Internal Medicine, Incheon St. Mary’s Hos-
pital, College of Medicine, The Catholic University of Korea, Incheon, Korea (J.S.K.); Division of Pulmonary and Critical Care Medicine, Department of Internal
Medicine, Severance Hospital, Yonsei University College of Medicine, Seoul, Korea (Y.A.K.); Department of Internal Medicine, Pusan National University Yang-
san Hospital, Pusan National University School of Medicine, Yangsan, Korea (D.J.); Medical Research Collaborating Center, Seoul National University Hospi-
tal, Seoul, Korea (M.J.J.); Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul, Korea (J.M.G.); and Department of Radiol-
ogy, UMass Memorial Medical Center, Worcester, Mass (S.H.Y.). Received January 9, 2021; revision requested March 9; revision received May 14; accepted June 3.
Address correspondence to S.H.Y. (e-mail: yshoka@gmail.com).
Supported by Research Program 2019 funded by the Seoul National University College of Medicine Research Foundation.

Conflicts of interest are listed at the end of this article.

Radiology 2021; 301:435–442  •  https://doi.org/10.1148/radiol.2021210063  •   Content codes:

Background:  Determining the activity of pulmonary tuberculosis on chest radiographs is difficult.

Purpose:  To develop a deep learning model to identify active pulmonary tuberculosis on chest radiographs.

Materials and Methods:  Chest radiographs were retrospectively gathered from a multicenter consecutive cohort with pulmonary tu-
berculosis who were successfully treated between 2011 and 2017, along with normal radiographs to enrich a negative class. The
pretreatment and posttreatment radiographs were labeled as positive and negative classes, respectively. A neural network was trained
with those radiographs to calculate the probability of active versus healed tuberculosis. A single-center consecutive cohort (test set
1; 89 patients, 148 radiographs) and data from one multicenter randomized controlled trial (test set 2; 366 patients, 3774 radio-
graphs) were used to test the model. The area under the receiver operating characteristic curve (AUC) was used to evaluate the per-
formance of the model and of the four expert readers.

Results:  In total, 6654 pre- and posttreatment radiographs from 3327 patients (mean age 6 standard deviation, 55 years 6 19;
1884 men) with pulmonary tuberculosis and 3182 normal radiographs from as many patients (mean age, 53 years 6 14; 1629
men) were gathered. For test set 1, the model showed a higher AUC (0.83; 95% CI: 0.73, 0.89) than one pulmonologist (0.69;
95% CI: 0.61, 0.76; P , .001) and performed similarly to the other readers (AUC, 0.79–0.80; P = .14–.23). For 200 randomly se-
lected radiographs from test set 2, the model had a higher AUC (0.84) than the pulmonologists (0.71 and 0.74; P , .001 and .01,
respectively) and performed similarly to the radiologists (0.79 and 0.80; P = .08 and .06, respectively). The model output increased
by 0.30 on average with a higher degree of smear positivity (95% CI: 0.20, 0.39; P , .001) and decreased during treatment (base-
line, 3 months, and 6 months: 0.85, 0.51, and 0.26, respectively).

Conclusion:  A deep learning model performed similarly to radiologists for accurately determining the activity of pulmonary tubercu-
losis on chest radiographs; it also was able to follow posttreatment changes.
© RSNA, 2021

Online supplemental material is available for this article.

M ycobacterium tuberculosis is one of the top 10 causes

of death worldwide and remains a major public
health problem. According to the World Health Organi-
zation global tuberculosis report (1), 10 million people
were estimated to be infected with tuberculosis in 2020.
Sputum smear microscopy after Ziehl-Neelsen staining
and microbiologic culture are mandatory tests for the
diagnosis and treatment monitoring of pulmonary tu-
berculosis. Polymerase chain reaction assay and T-cell–
based tests are used for diagnosis but not for treatment
monitoring (2,3). Alongside microbiologic confirma-
tion, chest radiography is the first choice of initial imag-
ing study as a triage tool, diagnostic aid, and screening
This copy is for personal use only. To order printed copies, contact reprints@rsna.org
method for detecting tuberculosis (4). Chest radiographs
with a visual scoring system provide a high sensitivity
(up to 98%) but suffer from low specificity (76%) for
culture-confirmed pulmonary tuberculosis (4), and there
is substantial variability across readers. The reader must
also have sufficient experience (5).
A deep neural network can outperform human experts
in detecting radiographic abnormalities of pulmonary tu-
berculosis from normal chest radiographs. Hwang et al (6)
showed state-of-the-art classification performance for screen-
ing active pulmonary tuberculosis with a deep learning
algorithm, with area under the receiver operating charac-
teristic curves (AUCs) between 0.98 and 1.00. However,
Deep Learning to Determine the Activity of Pulmonary Tuberculosis on Chest Radiographs
Abbreviations
AUC = area under the receiver operating characteristic curve, IQR =
interquartile range
Summary
A deep learning model was able to determine the activity of tuberculo-
sis on chest radiographs, reflecting bacilli burden.
Key Results
N In two test sets that included radiographs depicting active and
healed tuberculosis (test set 1, n = 148; test set 2 subset, n = 200),
a deep learning model (area under the receiver operating charac-
teristic curves [AUCs], 0.83 and 0.84, respectively) differentiated
active from healed tuberculosis on radiographs, with comparable
performance to that of expert readers (AUCs, 0.69–0.80 [P , .001
to P = .23] and 0.71–0.80 [P , .001 to P = .08]).
N A higher degree of sputum smear positivity increased model out-
put (odds ratio, 1.36 per 0.1 increase in the disease activity score),
which decreased during treatment by 0.37 per month on average
(P , .001).
the networks were not trained with chest radiographs in treated
patients with tuberculosis who had varying residual fibrotic ab-
normalities (7). Thus, little is known regarding how current net-
works would perform in countries with a high burden of tuber-
culosis. These networks would be most useful in these countries
due to the shortage of trained imaging professionals. In these
countries, patients with treated or spontaneously healed tuber-
culosis are relatively common, and the frequency of radiologic
sequelae is up to 90% (8). In addition, immigrants or asylum-
seekers from high- and low-incidence countries pose concerns
for tuberculosis control. For immigration control, the initial
screening and monitoring of active tuberculosis primarily de-
pends on chest radiographs (9). In such situations, assessing the
radiographic activity of pulmonary tuberculosis is more relevant
than differentiating patients with active pulmonary tuberculosis
from healthy individuals.
The purpose of this study was to develop a deep neural net-
work to determine the activity of pulmonary tuberculosis on
chest radiographs and to compare its performance with that of
imaging professionals. In addition, we sought to determine if
this method could be used to follow the therapeutic response on
chest radiographs.
Materials and Methods
This retrospective study was approved by the institutional review
boards of the participating hospitals with a waiver of the require-
ment for informed consent from each patient.
Study Sample
We retrospectively searched a consecutive cohort of patients
with pulmonary tuberculosis who were successfully treated at
six Korean hospitals (hospitals A–F) between 2011 and 2017
(Fig 1, Table 1). We applied the following eligibility criteria:
(a) patients who were confirmed to have active pulmonary
tuberculosis with use of sputum microscopy, culture, or poly-
merase chain reaction test; (b) the treatment outcome was
“cured” or “completed” according to the definitions from the
436 radiology.rsna.org  n  Radiology: Volume 301: Number 2—November 2021
World Health Organization guidelines; and (c) available pre-
and posttreatment chest radiographs.
Pretreatment chest radiographs were obtained within the pe-
riod between 1 month before the initiation of antituberculosis
medication and 1 week after the initiation. Posttreatment radio-
graphs were obtained within the period between 1 week before
the last medication and 1 month after the last medication. We
labeled the pretreatment radiographs as the positive class and
the posttreatment radiographs as the negative class. We included
both posteroanterior and anteroposterior chest radiographs. We
excluded a total of 1127 patients; exclusion criteria are shown in
Table E1 (online).
To enrich the set of radiographs that did not have any ra-
diologic sequelae with healed tuberculosis, we further collected
a similar number of normal chest radiographs from hospital A
from the same period. A lack of radiographic abnormalities was
confirmed with corresponding CT scans within 1 month of the
radiographic examination. All of these radiographs were labeled
as the negative class.
Designing and Training the Deep Neural Network
After compiling 9836 chest radiographs from 6509 patients, we
randomly assigned radiographs for training and validation at a
ratio of 7:3 (Table 2; Table E2 [online]). EfficientNet, which was
introduced by Tan and Le (10), was adopted as a base feature ex-
tractor. The network was built and trained by using open-source
software (TensorFlow, version 1.11.0; Keras, version 2.2.4) (Ap-
pendix E1 [online]). The output of the neural network was a
value between 0 and 1 for the probability of active tuberculosis.
The network was implemented on a public website (version 1.0;
https://radiologist.app/activetb).
Validation of the Neural Network
We applied the model to two temporal test sets. For test set 1, we
included 89 patients with the same eligibility criteria from hospital
A in 2018 (Table E3 [online]), providing a total of 148 radio-
graphs: 80 pretreatment radiographs and 68 posttreatment
radiographs. For test set 2, we included a total of 3774 con-
secutive radiographs (349 pretreatment, 296 posttreatment, and
3129 in-treatment radiographs) from 366 patients from a ran-
domized controlled trial from February 2014 through January
2017 (11) at three of the participating centers (hospitals A–C)
and another hospital (hospital G) (Table 2; Table E4 [online]).
We excluded patients enrolled in that trial from the training and
validation sets and test set 1 and assigned them to the test set 2
to avoid patient overlap.
Comparison with Expert Human Readers
Two pulmonologists dedicated to managing tuberculosis
(reader 1, J.J.Y., and reader 2, N.K., with 26 years and 10 years
of experience, respectively) and two expert thoracic radiologists
(reader 3, S.H.Y., and reader 4, J.M.G., with 15 years and 30
years of experience) participated as human readers. The read-
ers could access information on each patient’s age and sex but
were blinded to disease activity information. They consecu-
tively and independently evaluated the 148 chest radiographs
from test set 1 and 200 randomly selected pre- and posttreat-
 Lee et al

Figure 1:  Study diagram.

Table 1: Demographic Characteristics of Patients Treated for Pulmonary Tuberculosis in the Training and Validation Data Sets

Characteristic Total Hospital A Hospital B Hospital C Hospital D Hospital E Hospital F

No. of patients 3327 550 513 430 595 580 659
Sex
 Male 1884 321 274 266 339 331 353
 Female 1443 229 239 164 256 249 306
Age (y)* 55 6 19 63 6 18 55 6 19 52 6 19 54 6 19 54 6 19 53 6 19
Note.—Unless otherwise specified, data are numbers of patients.
* Data are means 6 standard deviations.

Table 2: Number of Chest Radiographs in the Training, Validation, and Test Data Sets

Negative Class

Data Set Total Positive Class (Pretreatment) Posttreatment Healthy

Training and validation data set 9836 3327 3327 3182
  Training data set 6883 2328 2328 2227
  Validation data set 2953 999 999 955
Test set 1 148 80 68 …
Test set 2* 3774 349 296 …
  Pre- and posttreatment only 645 349 296 …
  Randomly selected for human reviewers 200 100 100 …
* Test set 2 included 3129 radiographs obtained during treatment.

Radiology: Volume 301: Number 2—November 2021  n  radiology.rsna.org 437

Deep Learning to Determine the Activity of Pulmonary Tuberculosis on Chest Radiographs

Table 3: Areas under the Receiver Operating Characteristic Curve for the Deep Learning Model and Human Readers

Human Readers*

Data Set Model Reader 1 Reader 2 Reader 3 Reader 4

Test set 1 (n = 148) 0.83 (0.76, 0.89) 0.69 (0.61, 0.76) 0.79 (0.71, 0.85) 0.80 (0.73, 0.86) 0.79 (0.71, 0.85)
[P , .001] [P = .16] [P = .23] [P = .14]
Test set 2 (n = 645)† 0.82 (0.79, 0.85) … … … …
Randomly selected radiographs 0.84 (0.78, 0.89) 0.71 (0.64, 0.77) 0.74 (0.67, 0.80) 0.79 (0.72, 0.84) 0.80 (0.73, 0.85)
from test set 2 for human [P , .001] [P = .001] [P = .08] [P = .06]
reading (n = 200)
Note.—Data in parentheses are 95% CIs. P values in brackets indicate the result of comparison between neural network and each reader.
* Readers 1 and 2 were pulmonologists; readers 3 and 4 were thoracic radiologists.
†
Pre- and posttreatment radiographs only.

ment chest radiographs from test set 2. There was a 1-month

interval between readings of the two data sets. They evaluated
pulmonary tuberculosis activity on each radiograph with use of
a five-point ordinal scale derived from modified definitions of
active and inactive tuberculosis from the literature (12): defi-
nitely active tuberculosis, probably active tuberculosis, equivo-
cal activity, probably healed tuberculosis, and definitely healed
tuberculosis (Fig E1 [online]).

Statistical Analysis
The area under the curve at receiver operating characteristic
analyses was used to evaluate the performance of the deep neural
network and human readers in test sets 1 and 2. The empirical
method suggested by DeLong et al (13) was used to compare re-
ceiver operating characteristic curves with each other. The degree
of agreement across human readers was assessed with the Cohen
k coefficient. We evaluated the relationship between the neural
network score and the readers’ grading and across the readers by
using Spearman correlation coefficients.
For test set 2 analyses, we assessed the correlation between
the network-driven logit-transformed disease activity score and
the degree of smear positivity on sputum smear test (Appendix
E1 [online]) results by using a linear mixed model with pa-
tients as a random effect to account for multiple measurements
per patient. The association of the activity score with the degree
of smear positivity was evaluated for an ordinal logistic regres-
sion model with use of generalized estimating equation analysis
to account for multiple measurements per patient. The linear
mixed model was also used to evaluate whether the activity
score decreased with treatment.
P , .05 was considered indicative of statistically significant
difference. Statistical analyses were performed by using SAS ver-
sion 9.4 (SAS Institute), SPSS version 23 (IBM), and MedCalc
version 19.2.6 (MedCalc Software).
Results
Characteristics of the Study Sample
A total of 6654 pre- and posttreatment chest radiographs from
3327 patients with pulmonary tuberculosis (mean age 6 standard
deviation, 55 years 6 19; 1884 men) were included in this study
(Fig 1, Table 1), along with 3182 normal chest radiographs in
Figure 2:  Graph shows receiver operating characteristic curves of the
neural network model and human readers in test set 2.
3182 patients from hospital A (mean age, 53 years 6 14; 1629
men) for training and validation.
Performance of the Deep Neural Network
For the test set 1, which consisted of 148 chest radiographs, the
network had an AUC of 0.83 (95% CI: 0.76, 0.89; P , .001)
(Table 3; Fig E2 [online]). The two pulmonologists had AUCs of
0.69 (95% CI: 0.61, 0.76; P , .001) and 0.79 (95% CI: 0.71,
0.85; P , .001), and the two radiologists had AUCs of 0.80
(95% CI: 0.73, 0.86; P , .001) and 0.79 (95% CI: 0.71, 0.85;
P , .001). The neural network showed higher performance than
reader 1 (P , .001), but we did not find evidence of a difference
in its performance from that of the other human readers (P =
.16, .23, and .14). The model’s and readers’ AUCs for smear-
positive cases were 0.89 and 0.83–0.93, respectively, and those
for smear-negative cases were 0.82 and 0.64–0.76. The Cohen
k values between pairs of pulmonologists and radiologists were
0.28 and 0.33, respectively, on a five-point scale. The k values
were 0.53 for the two pulmonologists and 0.54 for the two radi-
ologists when the results were dichotomized as active or healed
tuberculosis (Table E5 [online]). The Spearman correlation coef-

438 radiology.rsna.org  n  Radiology: Volume 301: Number 2—November 2021

 Lee et al

ficients between the network and each reader were between 0.59 .001). The neural network showed higher performance than
and 0.73, and those across the readers were between 0.55 and the pulmonologists (P , .001 and P = .001, respectively), but
0.80 (Table E5 [online]). we did not find evidence of a difference in performance be-
For test set 2, which comprised 645 chest radiographs when tween the model and the thoracic radiologists (P = .08 and
considering pre- and posttreatment radiographs only, the neu- P = .06, respectively) (Table 3, Fig 2). The Cohen k values be-
ral network’s AUC was 0.82 (95% CI: 0.79, 0.85; P , .001). tween pairs of pulmonologists and radiologists were 0.20 and
For 200 randomly selected chest radiographs from test set 2, the 0.33, respectively, on a five-point scale. The k values were 0.33
AUC was 0.84 (95% CI: 0.78, 0.89; P , .001) (Table 3, Fig 2). for the two pulmonologists and 0.58 for the two radiologists
For the same data set, the pulmonologists had AUCs of 0.71 when the results were dichotomized as active or healed tuber-
(95% CI: 0.64, 0.77; P , .001) and 0.74 (95% CI: 0.67, 0.80; culosis (Table E5 [online]). Spearman correlation coefficients
P , .001), and the radiologists had AUCs of 0.79 (95% CI: between the network and each reader were between 0.54 and
0.72, 0.84; P , .001) and 0.80 (95% CI: 0.73, 0.85; P , 0.76, and those across the readers were between 0.64 and 0.79
(Table E5 [online]).
Disease activity scores of 0.15 and 0.82 deter-
mined active pulmonary tuberculosis at sensitivity
and specificity of 95%, respectively, in test set 2.
The cutoff at 95% sensitivity provided a specific-
ity of 48.5% (95% CI: 37.1, 60.2) in test set 1
and 26.0% (95% CI: 21.3, 31.3) in test set 2. The
cutoff at 95% specificity produced a sensitivity
of 40.0% (95% CI: 30.0, 51.0) in test set 1 and
49.6% (95% CI: 44.3, 54.8) in test set 2. For ra-
diographs that the network misclassified in test set
2, the readers’ AUCs were below 0.3 (95% sen-
sitivity setting) and 0.5 (95% specificity setting),
indicating that on the radiographs misclassified by
the model, it was also difficult for human experts to
determine the disease activity (Table E6 [online]).
Model-derived disease activity scores according to
the Likert-based human rater assessment are sum-
Figure 3:  Box and whisker plot shows positive correlation between disease activity score and marized in Table E7 (online).
microscopic grade of Mycobacterium sputum smear test (negative, no bacilli detected on smear;
Relationship between the Disease Activity Score
6 [ie, equivocal positive], one to two bacilli found on 300 microscopic fields; 11, one to nine
and Sputum Smear Test
bacilli found per 100 fields; 21, one to nine bacilli found per 10 fields; 31, one to nine bacilli
per field; 41, more than nine bacilli found per field). The Spearman correlation coefficient of the
The median disease activity scores were 0.36 (in-
two variables was 0.34 (P , .001). Linear regression showed a sputum grade coefficient of 0.083
terquartile range [IQR], 0.15–0.74), 0.72 (IQR,
and a y-intercept of 0.53 (P , .001). Center lines represent median values; upper and lower bor-
0.44–0.83), 0.82 (IQR, 0.37–0.94), 0.70 (IQR,
ders of the boxes indicate 25th and 75th percentile values; upper and lower ends of vertical dotted
0.55–0.90), 0.85 (IQR, 0.66–0.98), and 0.89
lines show the 25th and 75th percentiles 6 1.5 interquartile range; dots represent outlier values.
(IQR, 0.81–0.99) for the negative
(no bacilli detected on smear), equiv-
Table 4: Temporal Changes in Model Output during Antituberculosis Treatment
ocal positive (one to two bacilli found
Month of on 300 microscopic fields), 11 (one
Treatment No. of Radiographs Mean Model Output* Median Model Output† to nine bacilli found per 100 fields),
0 783 0.72 6 0.28 0.85 (0.52–0.96) 21 (one to nine bacilli found per 10
1 1083 0.69 6 0.30 0.83 (0.43–0.95) fields), 31 (one to nine bacilli per
2 525 0.63 6 0.32 0.72 (0.31–0.93) field), and 41 (more than nine bacilli
3 384 0.52 6 0.31 0.51 (0.21–0.85) found per field) degrees of sputum
4 390 0.42 6 0.28 0.36 (0.17–0.66) smear positivity, respectively (Fig 3;
5 333 0.37 6 0.26 0.29 (0.16–0.55) Table E8 [online]). The linear mixed
6 294 0.33 6 0.23 0.26 (0.15–0.46) model showed that the log odds of
7 155 0.34 6 0.25 0.25 (0.13–0.49) the disease activity score increased by
8 21 0.27 6 0.21 0.18 (0.12–0.39) 0.30 on average (95% CI: 0.20, 0.39;
9 4 0.15 6 0.11 0.11 (0.09–0.22) P , .001) when the degree of spu-
tum smear positivity became higher.
Note.—Model output was a value between 0 and 1 for the probability of active tuberculosis.
The generalized estimating equation
* Data are means 6 standard deviations.
analysis of the ordinal logistic regres-
†
Data in parentheses are interquartile ranges.
sion model showed that the odds of

Radiology: Volume 301: Number 2—November 2021  n  radiology.rsna.org 439

Deep Learning to Determine the Activity of Pulmonary Tuberculosis on Chest Radiographs

having a higher degree of the sputum smear test increased by
1.36 (95% CI: 1.22, 1.50; P , .001) per 0.1 increase in the
disease activity score. The analysis also showed that the odds of
having a lower degree of sputum smear positivity increased by
1.94 (95% CI: 1.63, 2.30; P , .001) per 1-month elapse of the
antituberculosis treatment.
Disease Activity Scores during Antituberculosis Treatment
In test set 2, the median disease activity scores were 0.85 (IQR,
0.52–0.96), 0.51 (IQR, 0.21–0.85), 0.26 (IQR, 0.15–0.46) and
0.11 (IQR, 0.09–0.22) at 0, 3, 6, and 9 months, respectively,
since the start of treatment (Table 4). The linear mixed model
showed that the disease activity score gradually decreased by 0.37
on average (95% CI: 0.35, 0.39;
P , .001) on the logit scale
of disease activity score per 1
month of treatment duration
(Figs 4–6).

Discussion
Several deep learning ap-
proaches have been reported to
differentiate active pulmonary
tuberculosis from normal lungs
on chest radiographs (6,14,15),
but this study differs from prior
research by introducing radio-
graphs that showed postin-
flammatory sequelae, including
fibrotic changes, granulomas,
and volume loss of the lung. We
Figure 4:  Box and whisker plot shows a temporal change in disease activity score (on log-odds scale) during antituber- collected only one image pair
culosis treatment. Center lines represent median values; upper and lower borders of the boxes indicate 25th and 75th per- per patient (ie, one radiograph
centile values; upper and lower ends of vertical dotted lines show the 25th and 75th percentiles 6 1.5 interquartile range; showing active tuberculosis and
dots represent outlier values.

Figure 5:  Representative images in a 44-year-old woman with pulmonary tuberculosis in test set 1 (A) before treatment, (B) after treatment, and (C) with magnified
view of the lung lesion. (A) The model was able to accurately determine tuberculosis activity, producing a high disease activity score of 0.96. Pulmonologists interpreted
the radiograph as definitely active tuberculosis, and radiologists interpreted it as probably active tuberculosis. (B) After 6 months of antituberculosis medication, the disease
activity score decreased to a low value of 0.09. Pulmonologists interpreted the radiograph as probably healed tuberculosis and definitely healed tuberculosis, and radiolo-
gists interpreted it as probably healed tuberculosis and equivocal activity. Human experts mostly accurately determined tuberculosis activity, but there were some discrepan-
cies on posttreatment chest radiographs. (C) The model detected clustered tiny nodular opacities with fuzzy margins and suggested a high score of active tuberculosis. On
the other hand, the model did not respond to tiny nodular opacities with clear margins.

440 radiology.rsna.org  n  Radiology: Volume 301: Number 2—November 2021

 Lee et al

Figure 6:  Representative consecutive images in a 53-year-old man during treatment for pulmonary tuberculosis at (A) 7 days (disease activity score, 0.94), (B) 49
days (disease activity score, 0.68), (C) 140 days (disease activity score, 0.38), and (D) 175 days (disease activity score, 0.18) since the start of antituberculosis treatment.
Disease activity scores calculated by the model decreased gradually during antituberculosis treatment. Heat maps (right panels) also showed a gradual decrease in inten-
sity and territories of active lung lesions.

one showing healed tuberculosis) in thousands of consecutive
multicenter patients to encompass the full spectrum of initial ra-
diologic manifestations and sequelae in the data set while avoid-
ing similar or redundant images. Notably, the training data set
comprised a 7-year consecutive cohort from six hospitals; thus,
the data set included the full spectrum of typical and atypical
radiographic findings of active and healed pulmonary tuberculo-
sis that accompanied intrathoracic extrapulmonary tuberculosis
(eg, tuberculous pleurisy) and underlying lung disease (eg, inter-
stitial lung abnormalities).
This deep learning network may be advantageous in countries
with a high burden of tuberculosis where spontaneously healed
or previously treated patients with tuberculosis are prevalent.
Those countries usually suffer from low incomes and a shortage
of expert imaging professionals (1). It is important to triage
patients with suspected tuberculosis at chest radiography into
those who should be tested bacteriologically or with the Xpert
MTB/RIF assay (4), and any radiographic abnormalities are
generally regarded as a positive result, given that tuberculosis can
sometimes manifest atypically (4). If a deep learning network
can accurately differentiate active tuberculosis from healed tu-
berculosis on radiographs, then this will provide more efficient
triage for testing in limited-resource settings by decreasing un-
necessary testing in individuals less likely to have active tubercu-
losis (Table E9 [online]).
Another potential application of the network is to monitor
and determine the treatment response of intractable mycobacte-
rial diseases. Specifically, the disease activity score was correlated
with the grade of the sputum smear and decreased with the
duration of treatment. Those findings are in accordance with
previously reported results for the visual severity of pulmonary
tuberculosis on radiographs (16). Multidrug- or extensively
drug-resistant pulmonary tuberculosis typically requires long-
term treatment. Current proxy markers for treatment success
are time to sputum culture conversion and culture conversion
status at 2 months or 6 months (17,18). However, these mark-
ers have some limitations in either sensitivity or specificity, and
confirming negative conversion takes 2 and 8 weeks in liquid
and solid culture media, respectively. Given the finding that
lower network-derived disease activity scores were associated
with a lower bacilli burden, monitoring the activity score on
chest radiographs during treatment may supplement current
prognostic markers. The same approach can be attempted to
monitor the treatment response of nontuberculous mycobacte-
rial lung diseases that share similar radiographic findings with
tuberculosis (19), for which objective quantitative tools for
monitoring during treatment are scarce.
This study had limitations. First, the network was validated
with a limited quantity of retrospectively collected data. Also,
the data for test set 2 were obtained from three of the four hos-
pitals where the training data were collected. Furthermore, the
addition of normal radiographs to negative cases may inflate the
performance of the network. Second, the network could not
separate patients with healed tuberculosis from healthy individu-
als. Radiographic abnormalities in active tuberculosis can com-
pletely resolve in up to 60% of patients with treated tuberculosis
(20), and posttreatment fibrotic sequelae may not develop if
lesions are healed before forming necrosis (21). Differentiating
patients with healed tuberculosis from healthy individuals on
radiographs would be more difficult than differentiating active
tuberculosis from healed tuberculosis, and further study is war-
ranted for this task. Third, despite the decrease in the disease

Radiology: Volume 301: Number 2—November 2021  n  radiology.rsna.org 441

Deep Learning to Determine the Activity of Pulmonary Tuberculosis on Chest Radiographs
activity score during treatment, a considerable overlap existed.
Physicians would be needed to assess treatment response or tu-
berculosis activity in a comprehensive evaluation of symptomatic
improvement, score change, and sputum or culture conversion.
Fourth, our cohort did not include patients with human immu-
nodeficiency virus and tuberculosis (22), so the performance of
the network is unknown in those patients. Fifth, a high disease
activity score with the network may result from parenchymal
abnormalities of other pathogens in patients with healed tuber-
culosis, such as nontuberculous mycobacteria or aspergillosis
(7,12). Sixth, we did not apply lung segmentation, which might
improve the network’s performance. Seventh, human readers and
our network tried to differentiate active tuberculosis from healed
tuberculosis on a single radiograph. Human readers will differ-
entiate them more straightforwardly if previous radiographs are
available for comparison, and our network did not consider this.
Eighth, expert readers did not have any instructions for a harmo-
nized interpretation of radiographs, and this might have resulted
in limited agreements across readers.
In conclusion, the deep neural network was able to determine
the activity of tuberculosis on chest radiographs, reflecting ba-
cilli burden and changes after treatment. The network may help
radiologically triage patients with active tuberculosis by exclud-
ing healed tuberculosis in high-burden countries and may assist
in monitoring the activity of mycobacterial diseases that require
long-term treatment.
Author contributions: Guarantors of integrity of entire study, S.L., Y.J.L., S.H.Y.;
study concepts/study design or data acquisition or data analysis/interpretation, all au-
thors; manuscript drafting or manuscript revision for important intellectual content,
all authors; approval of final version of submitted manuscript, all authors; agrees to
ensure any questions related to the work are appropriately resolved, all authors; litera-
ture research, S.L., N.K., Y.J.L., J.Y.L., J.S.K., D.J., S.H.Y.; clinical studies, S.L., J.J.Y.,
N.K., Y.J.L., J.K.L., J.Y.L., J.S.K., D.J., J.M.G., S.H.Y.; experimental studies, N.K.;
statistical analysis, S.L., J.J.Y., N.K., J.S.K., M.J.J., S.H.Y.; and manuscript editing,
S.L., J.J.Y., N.K., Y.J.L., J.S.K., Y.A.K., J.M.G., S.H.Y.
Disclosures of Conflicts of Interest: S.L. Activities related to the present article:
will receive stock options from Medical IP for the potential commercialization of the
model presented in the article. Activities not related to the present article: disclosed no
relevant relationships. Other relationships: disclosed no relevant relationships. J.J.Y.
disclosed no relevant relationships. N.K. disclosed no relevant relationships. Y.J.L.
disclosed no relevant relationships. J.K.L. disclosed no relevant relationships. J.Y.L.
disclosed no relevant relationships. J.S.K. disclosed no relevant relationships. Y.A.K.
disclosed no relevant relationships. D.J. disclosed no relevant relationships. M.J.J.
disclosed no relevant relationships. J.M.G. Activities related to the present article: is
member of Radiology editorial board. Activities not related to the present article: re-
ceived or will receive research grants from Infinitt Healthcare, Dongkook Lifescience,
and LG Electronics. Other relationships: disclosed no relevant relationships. S.H.Y.
Activities related to the present article: will receive stock options from Medical IP for
the potential commercialization of the model presented in the article. Activities not re-
lated to the present article: is the chief medical officer of Medical IP and receives stock
options as compensation. Other relationships: disclosed no relevant relationships.
References
1. World Health Organization. Global tuberculosis report 2020. Geneva,
Switzerland: World Health Organization, 2020.
2. Friedrich SO, Rachow A, Saathoff E, et  al. Assessment of the sensitivity
and specificity of Xpert MTB/RIF assay as an early sputum biomarker of
response to tuberculosis treatment. Lancet Respir Med 2013;1(6):462–470.
442 radiology.rsna.org  n  Radiology: Volume 301: Number 2—November 2021
3. Clifford V, He Y, Zufferey C, Connell T, Curtis N. Interferon gamma release
assays for monitoring the response to treatment for tuberculosis: a system-
atic review. Tuberculosis (Edinb) 2015;95(6):639–650.
4. World Health Organization. Chest radiography in tuberculosis detection:
summary of current World Health Organization recommendations and
guidance on programmatic approaches. Geneva, Switzerland: World Health
Organization, 2016.
5. Balabanova Y, Coker R, Fedorin I, et al. Variability in interpretation of chest
radiographs among Russian clinicians and implications for screening pro-
grammes: observational study. BMJ 2005;331(7513):379–382.
6. Hwang EJ, Park S, Jin KN, et al. Development and validation of a deep
learning-based automatic detection algorithm for active pulmonary tuber-
culosis on chest radiographs. Clin Infect Dis 2019;69(5):739–747.
7. Kim HY, Song KS, Goo JM, Lee JS, Lee KS, Lim TH. Thoracic sequelae
and complications of tuberculosis. RadioGraphics 2001;21(4):839–858;
discussion 859–860.
8. Ali MG, Muhammad ZS, Shahzad T, Yaseen A, Irfan M. Post tuberculosis
sequelae in patients treated for tuberculosis: an observational study at a ter-
tiary care center of a high TB burden country. Eur Respir J 2018;52:PA2745.
9. Aldridge RW, Zenner D, White PJ, et al. Tuberculosis in migrants moving
from high-incidence to low-incidence countries: a population-based cohort
study of 519 955 migrants screened before entry to England, Wales, and
Northern Ireland. Lancet 2016;388(10059):2510–2518.
10. Tan M, Le Q. EfficientNet: Rethinking Model Scaling for Convolutional
Neural Networks. In: Kamalika C, Ruslan S, eds. Proceedings of the 36th
International Conference on Machine Learning. Proceedings of Machine
Learning Research: PMLR Proceedings of Machine Learning Research,
2019; 6105–6114.
11. Lee JK, Lee JY, Kim DK, et al. Substitution of ethambutol with linezolid
during the intensive phase of treatment of pulmonary tuberculosis: a pro-
spective, multicentre, randomised, open-label, phase 2 trial. Lancet Infect
Dis 2019;19(1):46–55.
12. Nachiappan AC, Rahbar K, Shi X, et al. Pulmonary tuberculosis: role of ra-
diology in diagnosis and management. RadioGraphics 2017;37(1):52–72.
13. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under
two or more correlated receiver operating characteristic curves: a nonpara-
metric approach. Biometrics 1988;44(3):837–845.
14. Khan FA, Majidulla A, Tavaziva G, et  al. Chest x-ray analysis with deep
learning-based software as a triage test for pulmonary tuberculosis: a pro-
spective study of diagnostic accuracy for culture-confirmed disease. Lancet
Digit Health 2020;2(11):e573–e581.
15. Pasa F, Golkov V, Pfeiffer F, Cremers D, Pfeiffer D. Efficient deep network
architectures for fast chest x-ray tuberculosis screening and visualization. Sci
Rep 2019;9(1):6268.
16. Ralph AP, Ardian M, Wiguna A, et al. A simple, valid, numerical score for
grading chest x-ray severity in adult smear-positive pulmonary tuberculosis.
Thorax 2010;65(10):863–869.
17. Kurbatova EV, Cegielski JP, Lienhardt C, et al. Sputum culture conversion
as a prognostic marker for end-of-treatment outcome in patients with mul-
tidrug-resistant tuberculosis: a secondary analysis of data from two observa-
tional cohort studies. Lancet Respir Med 2015;3(3):201–209.
18. Rajpurkar P, Irvin J, Bagul A, et  al. MURA Dataset: Towards Radiolo-
gist-Level Abnormality Detection in Musculoskeletal Radiographs. arX-
iv:1712.06957 2017. https://arxiv.org/abs/1712.06957. Published 2017.
Accessed July 19, 2020.
19. Koh WJ, Kwon OJ, Lee KS. Nontuberculous mycobacterial pulmonary dis-
eases in immunocompetent patients. Korean J Radiol 2002;3(3):145–157.
20. Menon B, Nima G, Dogra V, Jha S. Evaluation of the radiological sequelae
after treatment completion in new cases of pulmonary, pleural, and medias-
tinal tuberculosis. Lung India 2015;32(3):241–245.
21. Long R, Maycher B, Dhar A, Manfreda J, Hershfield E, Anthonisen N. Pul-
monary tuberculosis treated with directly observed therapy: serial changes in
lung structure and function. Chest 1998;113(4):933–943.
22. Lee CH, Hwang JY, Oh DK, et al. The burden and characteristics of tuber-
culosis/human immunodeficiency virus (TB/HIV) in South Korea: a study
from a population database and a survey. BMC Infect Dis 2010;10(1):66.