Diagnostic models for computing

measurement uncertainty in blood bank

screening tests
Paulo Pereira, James Westgard, Pedro Encarnação, Gracinda de Sousa

2nd World Congress of Health Research, Viseu

Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Introduction
The European Union regulation for blood banks does not require the evaluation of
measurement uncertainty in virology screening tests. It is required when screening
tests are accredited by ISO 15189 guideline complying with the principles of the
Guide to the Expression of Uncertainty in Measurement (GUM). However, GUM is
intended for numerical quantities, not for ordinal quantities where the result is
expressed by a positive/negative results. Blood banks with ISO 15189 accredited
screening tests could evaluate the uncertainty on cutoff point fulfilling GUM or
consider an alternative (to GUM) model focused on the probability of results.
Objectives
This article discusses and proposes alternative (to GUM) models intended for
screening tests validation, i.e., clinical sensitivity, clinical specificity, area under
receiver operating characteristic (ROC) curve, where the outputs could be viewed as
an alternative uncertainty result concept.

© Paulo Pereira 2/20

Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
When diagnostic models of probability should be an alternative measurement
uncertainty model?
A screening test output is an ordinal quantity, expressed by a binary result
(positive/negative). The mathematical models for measurement uncertainty calculus
fulfilling the Guide to the Expression of Uncertainty in Measurement (GUM) can be
applied uniquely to numerical results, not to ordinal (or nominal) results where the
uncertainty is expressed by probabilities (alternative to GUM and VIM concept).
When it should be determined?
When a blood establishment screening laboratory employs a state of the art
evaluation method for screening tests focused on binary results probabilities and its
intended use, i.e., post-transfusion safety/risk.
When a laboratory implements the “risk based thinking” concept fulfilling (DIS) ISO
9001:2015 “risk based thinking” concept.
When a laboratory accredits a test according to ISO 15189 (or ISO/IEC 17025).
© Paulo Pereira 3/20
Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Arising of false results caused by bias effect on binary results
 Seroconversion window period (the most critical bias component):false negative
results
 Prozone effect: false negative results
 Effects of preanalytical variables: false positive and false negative results
 Effects of disease: false positive and false negative results
 Effects of drugs: false positive and false negative results
 Effects of herbs and natural products: false positive and false negative results

© Paulo Pereira 4/20

Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Methods
a) Seroconversion window period
 Measurement of trueness
 Synonymous of “seroconversion sensitivity”
 “The window period for a test designed to detect a specific disease (particularly
an infectious disease) is the time between first infection and when the test can
reliably detect that infection”(Le T, Bhushan V, Vasan N. First aid for the USMLE
step 1. 20th ed. New York (NY): McGraw-Hill Medical; 2009)
 Typically, the window period is described as the number of days between the day
of infection and the first positive result
 Theoretically, a patient with a longer window period has a higher chance of false
negative results, i.e., when the patient is a blood donor there is a high post-
transfusion risk

© Paulo Pereira 5/20

Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Methods
a) Seroconversion window period: classical measurement

Pre-seroconversion period (window period) Post-seroconversion period

Cutoff True positive results

(1.00 s/co)
Infection False negative results

0 2 4 6 8 10 12 14 16 17 18 19 20

Days © Paulo Pereira 6/20

Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Methods
a) Seroconversion window period: measurement considering an indeterminate results
zone (metrological concept of “grey-zone”)
Pre-seroconversion period (window period) Post-seroconversion period

Hypothetically, the use of the indeterminate

zone decreases the window period, with no
impact on post-transfusion risk, since blood
components of donors with indeterminate
Cutoff True positive results results must be rejected
(1.00 s/co)
Indeterminate results
Infection False negative results

0 2 4 6 8 10 12 14 16 17 18 19 20

© Paulo Pereira 7/20

Days
Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Methods
a) Diagnostic accuracy models when the diagnosis is known
 The Bayesian probabilities reflect true positive results and true negative results as
uncertain (chance of false negative and false positive results).
 The critical measurements taken from contingency 2x2 table are the clinical
sensitivity se[%]=[TP/(TP+FN)]·100, and the clinical specificity
sp[%]=[TN/(FP+TN)] ·100.
 The measurement of clinical sensitivity and clinical specificity are only possible
when patients and healthy subjects samples are available (e.g., panels, samples
from diagnosed persons, and blood donors with a past history of negative
results, respectively). The probabilities (absolute values, such as 100%) can be
only associate to the samples tested; they cannot be associated with the patients
or healthy persons populations!

© Paulo Pereira 8/20

Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Methods
a) Diagnostic accuracy models when the diagnosis is known
 The blood establishment screening laboratory must measure the 95%
confidence interval (CI) to infer to 95% of the patients and healthy subjects
population of blood establishment geographic area. The 95% interval is
analogous to the expanded uncertainty interval from GUM approach.

© Paulo Pereira 9/20

 When it is measured a probability using an
Practicability and relevance of GUMabsolute approach forthecomputing
value, probability of uncertain
measurement uncertainty in blood bank screening
results in patientstests
or healthy subjects
2x2 contingency table for diagnostic accuracysamples is expressed, respectively, by false
negatives or false positives ratio, e.g.,
Diagnostic accuracy criteria
se=99.9% with false negative results=0.1%
Candidate assay Positive Negative
Total
results (Disease, D=1) (Non disease, D=0)

False-positive
Positive True-positive
results (FP) TP + FP
(Y=1) results (TP)
α-error

False-negative
Negative True-negative
results (FN) FN + TN
(Y=0) results (TN)
β-error

Total TP + FN FP + TN N

© Paulo Pereira 10/20

Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Methods
b) Agreement when comparator is other than diagnostic accuracy criteria
 The agreement uniquely represent the agreement of candidate test results with a
comparative test
 If the comparative test has a worst diagnostic accuracy performance than the
candidate test, the agreements analysis will have a biased output!!!
 The critical measurements taken from 2x2 table are the overall agreement
OA[%]=[(a+d)/n]·100, the positive agreement PA[%]=[a/(a+c)]·100, and the
negative agreement NA[%]=[d/(b+d)] ·100.
 Positive agreement and negative agreement cannot be confused, respectively,
with sensitivity and specificity!

© Paulo Pereira 11/20

Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Methods
b) Agreement when comparator is other than diagnostic accuracy criteria
 This model should be an alternative to diagnostic accuracy only when patients
and healthy subjects samples are unavailable. The probabilities can be only
associate to the comparative test results; they cannot be associated with the
patients or healthy persons!
 The blood establishment screening laboratory must measure the 95%
confidence interval (CI) to infer to 95% of the comparative test positive and
negative results samples. The 95% interval is analogous to the expanded
uncertainty interval from GUM approach.

© Paulo Pereira 12/20

 Analogous
Practicability and relevance of GUM approach for to the diagnostic accuracy
computing
measurements, the uncertainty of
measurement uncertainty in blood bank screening tests
agreement absolute values have an
Uncertain results measurement: false negative and false associated
positive agreement
uncertainevaluation
ratio
Comparative assay
Candidate assay Positive Negative
Total
results (X=1) (X=0)

Positive b
a a+b
(Y=1) α-error

Negative c
d c+d
(Y=0) β-error

Total a+c b+d n

© Paulo Pereira 13/20

Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Methods
c) Evaluation of uncertain results due to effects of sensitivity, false positive results,
and cutoff value
 The graphic displays the area under the curve; the area for a specific cutoff value
is according to the ratio of true positive results (sensitivity) and the ratio of false
positive results (1-specificity).
 The change of cutoff value will have an effect on area under curve.
 There some mathematical models published for the measurement of area under
curve; a parametric estimation is expressed published by Metz (1978)
AUCp=FPF(c)∙TPF(c), and a non-parametrical estimation is expressed by the
Mann-Whitney statistical test (1947) AUCnp=U/n1 ∙n0.

© Paulo Pereira 14/20

Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Methods
c) Evaluation of uncertain results due to effects of sensitivity, false positive results,
and cutoff value
 The area result  [0.50, 1.00].
 Area under curve evaluation [Hosmer (2013)]: if AUC[0.50, 0.70[ the
discrimination is poor, if AUC[0.70, 0.80[ the discrimination is acceptable, if
AUC[0.80, 0.90[ the discrimination is excellent, and if AUC[0.90, 1.00] the
discrimination is outstanding.
 The 95% interval of area under curve is analogous to the expanded uncertainty
interval from GUM approach.

© Paulo Pereira 15/20

Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Receiver operating characteristic (ROC) curve and area under the curve (AUC) 0 t=FPFc 1

Perfect assay Imperfect but useful assay Perfectly useless assay

Probability density

Disease
D1
Non disease Disease Non disease Disease Non disease
D0 D1 D0 D1 D0

1 1 1
ROCt
ROCt

AUC=1.00 AUC=0.95 AUC=0.50

Sensitivity (TPF)

Sensitivity
ROCt

AUC=1,000 AUC=0,975 AUC=0,500

0 0 0
0 1-Specificity (FPF) 1 0 1 0 1

© Paulo Pereira 16/20

Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Results
The measurement uncertainty intervals were taken from results of a single anti-
hepatitis C virus immunoassay on a Portuguese blood establishment.
Window Diagnostic accuracy Agreement
AUC
period Sensitivity Specificity Overall Positive Negative
Absolute value 97 days 100% 100% 99.6% 100% 99.6% 1.00
95% CI Not applicable 88.3-100% 98.5-100% 98.0-99.9% 97.9-100% 97.8-99.9% 0.99-1.00

The results show the relevance of alternative to GUM concepts for the evaluation of
post-transfusion risks components. They answer to chance questions, as well as to a
measurement trueness question unanswered by GUM models which are
inapplicable to ordinal quantities/binary results not dealing with probabilities. The
seroconversion window period, such as the 95% CIs represent chances of uncertain
results/false results arising from human biological variation as well as from the
analytical sensitivity of tests (biased).
© Paulo Pereira 17/20
Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Measurement uncertainty
determination

Measurement of
trueness or Probability
Measurement of probabilitty?
trueness Are patients and
Yes healthy subjects
No
sample available?

Diagnostic accuracy
Seroconversion
models when the Agreement of results
window period
diagnosis is known

Diagnostic accuracy
when cutoff varies?
Overall agreement,
Window period No positive agreement &
Yes Sensitivty & negative agreement
Receiver operating specificity
characteristic curve &
area under curve
Sensitivity, specificity & area under curve
Reported window period & probability of results on
patients and healthy subjects samples and populations © Paulo Pereira 18/20
Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
Conclusions
The measurement uncertainty of screening immunoassays results can be
determined by alternative (to GUM) models. The presented models allow the
laboratory to claim the performance requirements for the measurement uncertainty
and these determinations could be regularly reviewed.

© Paulo Pereira 19/20

Practicability and relevance of GUM approach for computing
measurement uncertainty in blood bank screening tests
References
Bureau International des Poids et Mesures (2012) JCGM 200 International vocabulary of metrology - Basic
and general concepts and associated terms. JCGM, Sèvres
Bureau International des Poids et Mesures (2008) JCGM 100 Evaluation of mesurement data - Guide to the
expression of uncertainty in measurement (GUM 1995 with minor corrections). JCGM, Sèvres
S L R Ellison and A Williams (2012) Eurachem/CITAC guide: Quantifying uncertainty in analytical
measurement. 3rd ed.
International Organization for Standardization (2012) ISO 15189 Medical laboratories - Requirements for
quality and competence. ISO, Geneva
Clinical and Laboratory Standards Institute (2004) EP5-A2 Evaluation of precision performance of
quantitative measurement methods, 2nd ed Approved guideline. The Institute, Wayne (PA)
European Federation of National Associations of Measurement, Testing and Analytical Laboratories
(EUROLAB) (2007) Technical Report No.1/2007 MU revisited: Alternative approaches to uncertainty
evaluation. The Federation, Paris
Nordtest (2012) TR 537 Handbook for calculation of MU in environmental laboratories, 3rd ed. The
Organization, Espoo

© Paulo Pereira 20/20

 http://www.ipst.pt/
paulo.pereira@ipst.min-saude.pt

This presentation is intended to be used on this conference only.

The use outside this conference can only occur with the consent of the author.