ROC Curve Homework

There were 600 cases from a defined population that had been subjected to some diagnostic Test
A, Test B and Test C.
In Test A, 120 people test result was positive and 480 people test result was negative. Suppose
that 100 actually positive cases and 500 actually negative cases were ultimately found in the
population studied and that the diagnostic test to be evaluated yielded 70 true positive (TP)
decisions, 30 false negative (FN) decision, 450 true negative (TN) decisions and 50 false positive
(FP) decisions.
In Test B, 60 people test result was positive and 540 people test result was negative. Suppose
that 100 actually positive cases and 500 actually negative cases were ultimately found in the
population studied and that the diagnostic test to be evaluated yielded 40 TP decisions, 60 FN
decision, 480 TN decisions and 20 FP decisions.
In Test C, 70 people test result was positive and 530 people test result was negative. Suppose
that 100 actually positive cases and 500 actually negative cases were ultimately found in the
population studied and that the diagnostic test to be evaluated yielded 45 TP decisions, 55 FN
decision, 475 TN decisions and 25 FP decisions.

Based on the materials learned in the class,

1) Please first summarize these data in separated decision matrix and calculate Accuracy
for Test A, Test B and Test C.
2) Please compare the accuracy of Test A, B and C and tell which test is best and which test
is worst?
3) If accuracy measure cant help completely answer question 2), please draw simple ROC
curves to tell.
Answers:
1)
TestAresult
Positive(T+) Negative(T) TotalActualStates
Actualresult
Positive(D+)
Negative(D)
TotalAtestresults

Calculated indices

70
50
120

30
450
480

100
500

TPF=70/100=0.7
FNF=1-TPF=0.3
FPF=50/500=0.1
TNF-1-FPF=0.9
P(D+)=100/600=0.17, P(D-)=1-P(D+)=0.83
Accuracy=TPF*P(D+)+TNF*P(D-)=0.7*0.17+0.9*0.83=0.866

TestBresult
Positive(T+) Negative(T)
Actualresult
Positive(D+)
Negative(D)
TotalBtestresults

40
20
60

TotalActualStates

60
480
540

100
500

Calculated indices
TPF=40/100=0.4
FNF=1-TPF=0.6
FPF=20/500=0.04
TNF-1-FPF=0.96
P(D+)=100/600=0.17, P(D-)=1-P(D+)=0.83
Accuracy=TPF*P(D+)+TNF*P(D-)=0.4*0.17+0.96*0.83=0.865

TestCresult
Positive(T+) Negative(T)
Actualresult
Positive(D+)
Negative(D)
TotalBtestresults

45
25
70

55
475
530

TotalActualStates
100
500

TPF=45/100=0.45
FNF=1-TPF=0.55
FPF=25/500=0.05
TNF-1-FPF=0.95
P(D+)=100/600=0.17, P(D-)=1-P(D+)=0.83
0.865
Accuracy=TPF*P(D+)+TNF*P(D-)=0.45*0.17+0.95*0.83=0.893

2)
all theon
accuracy
values are very
close to
each other,
is veryChard
to tell whether
which
is better.
2)As
Based
the calculated
indices,
accuracy
ofitTest
is highest,
0.893,
sotest
Test
C is best

among
the three tests; however, it is very hard to which is better between Test A and Test B, because the
accuracy of Test A, 0.866 is very close to the accuracy of Test B, 0.865.
3)

ROCCurve
1

1,1

0.9
0.8
0.7

0.1,0.7

TPF

0.6
0.5

TestB

0.4

0.04,0.4

TestA

0.3
0.2
0.1
0
0

0.2

0.4

0.6

0.8

FPF

Based on the ROC curve, the area under Test A is much larger than the one under Test B. So,
Test A is better than Test B.