Vaccine 26 (2008) 6462–6465

Contents lists available at ScienceDirect

Vaccine
journal homepage: www.elsevier.com/locate/vaccine

The effect of misclassification on evaluating the effectiveness of

influenza vaccines
Kotaro Ozasa ∗
Epidemiology for Community Health and Medicine, Kyoto Prefectural University of Medicine, Graduate of Medical Science, Japan

a r t i c l e i n f o a b s t r a c t

Article history: Misclassification is a measurement error and can be considered a type of information bias. Misclassi-
Available online 23 June 2008 fication can occur at both exposure and outcome levels. Nondifferential misclassification causes only a
dilution effect leading to underestimation, whereas differential misclassification can have more compli-
Keywords: cated and serious consequences. To avoid nondifferential diagnosis misclassification, it is necessary to
Influenza vaccine use highly specific diagnostic examinations or criteria such as virus detection to exclude ‘false positive’
Effectiveness
cases, and to limit the observation period to an intensive epidemic period if using less specific diagnostic
Misclassification
criteria such as symptoms of influenza-like illness (ILI) or absence from school or workplace. To avoid
Sensitivity
Specificity
differential diagnosis misclassification, vaccinated and unvaccinated groups must be equally scrutinized,
and such scrutiny is more important than the specificity of diagnosis. So, passive findings from patients
with influenza at clinics can cause complicated differential misclassification despite use of highly spe-
cific diagnostic procedures because vaccinated and unvaccinated patients may participate differently.
Also important is standardization of diagnostic procedure that vaccination anamnesis does not influence
diagnosis of influenza, or examination of the influence. Exposure misclassification would mainly under-
estimate vaccine effectiveness in most situations. Consequently, misclassification of diagnosis, especially
differential misclassification, affects evaluation of influenza vaccine effectiveness.
© 2008 Elsevier Ltd. All rights reserved.

1. Introduction that leads to an underestimation of the strength of the association

Misclassification is defined as the erroneous classification of an
individual, a value, or an attribute into a category other than that
to which it should be assigned [1]. Misclassification is thought of as
a measurement error, and can be considered a type of information
bias [2]. Misclassification can occur at both exposure and outcome
levels. An example of a misclassification of exposure is an incorrect
recall of vaccination by a patient, while an example of a misclas-
sification of outcome is a diagnosis of influenza-like illness (ILI)
using fever and/or upper respiratory symptoms that might result
in exclusion of some true influenza cases or the inclusion of other
diseases.
Nondifferential misclassification indicates that the probability
of misclassification is the same for all study groups, while differen-
tial misclassification indicates that the probability differs between
groups [1]. When outcome misclassification is not affected by expo-
sure (e.g., diagnosis of influenza is independent of whether the
subject was vaccinated or not), the misclassification is nondiffer-
ential. Nondifferential misclassification results in a ‘diluting effect’
between exposure and outcome [2]. Similarly, when exposure mis-
classification is not associated with evaluation of outcome, it is also
nondifferential.
In contrast, when an exposure misclassification is associated
with outcome evaluation, or when outcome misclassification is
associated with exposure, such differential misclassification may
either over- or underestimate the association between exposure
and outcome [2]. The direction and magnitude of the effect may be
difficult to evaluate. For example, vaccinated people may visit doc-
tors more readily when feeling ill, and consequently may be more
likely to be diagnosed as having influenza than non-vaccinated
people. From the viewpoint of outcome, patients with influenza
may more correctly recall about their vaccination in a retrospec-
tive study than those without influenza. Thus, any misclassification
may considerably influence evaluation of influenza vaccine effec-
tiveness.
2. Sensitivity/specificity of diagnosis and nondifferential
misclassification of diagnosis

Sensitivity is defined as the proportion of sick people classi-

∗ Tel.: +81 75 251 5789; fax: +81 75 251 5799. fied as sick according to given criteria/screening among the whole
E-mail address: kozasa@koto.kpu-m.ac.jp. sick people [2], and refers to the probability that a sick individ-

0264-410X/$ – see front matter © 2008 Elsevier Ltd. All rights reserved.
doi:10.1016/j.vaccine.2008.06.039
 K. Ozasa / Vaccine 26 (2008) 6462–6465 6463

Table 1
Sensitivity and specificity

People with disease (1000) People without disease (9000) Total (10,000)

Positive to disease criteria True positive (TP) (800) False positive (FP) (450) 1250
Negative to disease criteria False negative (FN) (200) True negative (TN) (8550) 8750

Sensitivity = TP/(TP + FN) = 80%; specificity = TN/(TN + FP) = 95%; positive predictive value (PPV) = TP/(TP + FP) = 64%.

Table 2
Influence of sensitivity/specificity on evaluation of effectiveness of vaccine: underestimation of effectiveness due to nondifferential misclassification of outcome

Comparison groups True number of True relative Observed number of patients with influenza diagnosed by criteria with Observed
patients with influenza risk (RR) sensitivity of 80% and specificity of 95% RR

Vaccinated 1000 50 50 × 0.80 + 950 × 0.05 = 87.5

0.25 0.44
Unvaccinated 1000 200 200 × 0.80 + 800 × 0.05 = 200

ual will be classified as sick (Table 1). Specificity is defined as
the proportion of healthy people classified as healthy according
to given criteria/screening among the whole healthy people, and
refers to the probability that a healthy individual will be classi-
fied as healthy. Low sensitivity and/or low specificity of diagnostic
procedures causes nondifferential misclassification of diagnosis [2].
Positive predictive value (PPV) is defined as the proportion of truly
sick people among those classified as sick according to the given
criteria/screening, and refers the probability that criteria/screening
predicts the sickness [2]. PPV depends on prevalence of sickness
in study population whereas sensitivity and specificity are inde-
pendent of the prevalence. For example, according to criteria with
sensitivity of 80% and specificity of 95%, PPV is 64% when preva-
lence of sickness is 10% in study population (Table 1), while it is
14% when the prevalence is 1%.
An example of underestimation of effectiveness of influenza
vaccine due to low sensitivity/specificity (nondifferential misclas-
sification) is shown in Table 2. The observed number of patients
with influenza (87.5) among vaccinated people can be calculated
from the true number of patients and non-patients (50 and 950),
and sensitivity and specificity of diagnostic criteria (0.80 and 0.95).
The number among unvaccinated people is calculated by the same
method (200). The effectiveness of influenza vaccination is pre-
sented as a ratio of rate/proportion of patients in vaccinated group
relative to the rate/proportion in unvaccinated group (relative risk;
RR). Although true RR is (50/1000)/(200/1000) = 0.25, the observed
RR is (87.5/1000)/(200/1000) = 0.44. This difference shows the dilu-
tion effect of nondifferential misclassification.
Table 3 shows this dilution effect for various sensitivities and
specificities. No effect is detected (RR = 1) when using a diag-
nostic criteria with a sensitivity of 0.5 and a specificity of 0.5
(diagnosis based only upon flipping a coin). Specificity is more
important than sensitivity in determining the true RR. It means
that exclusion of ‘false positive’ cases is important. Diagnostic
criteria with a high specificity are likely to limit the number of
patients who are diagnosed as having influenza according to the
criteria, so a large initial study population is necessary. How-
ever, use of highly sensitive diagnostic criteria to gather a larger
number of patients results in increase of nondifferential misclassi-
fication.
A randomized double-blind placebo-controlled trial by Govaert
et al. (Table 4) [3] adopted several influenza diagnosis criteria:
serological diagnosis, symptomatic diagnosis by a family physician
using the International Classification of Health Problems in Pri-
mary Care (ICHPPC-2) criteria, and diagnosis based on self-reported
symptoms (mailed questionnaire) using the criteria of ICHPPC-2
and the Dutch Sentinel Stations (DSSs). Serological diagnosis, which
had the highest specific criteria, showed the highest effectiveness
of influenza vaccine (RR = 0.50), while family physician diagnosis
showed the second (RR = 0.53), and self-reporting, which had the
lowest specificity, showed the lowest effectiveness (RR = 0.69 and
0.83). The probability of ‘false positive’ cases as well as ‘false neg-
ative’ cases was similar for both vaccine and placebo groups; this
was believed to be due to the double-blind vaccine/placebo inocula-
tion methodology. Thus, misclassification derived from diagnostic
procedures was nondifferential in that trial.
3. PPV and nondifferential misclassification of diagnosis
A high PPV of influenza among study subjects reduces the
proportion of ‘false positive’ cases. It reduces nondifferential mis-
classification and leads to a precise evaluation of effectiveness.
Govaert et al. [3] found that vaccine effectiveness was more clearly
observed in the period of epidemic elevation (between week 49 in
1991 and week 6 in 1992) (Fig. 1) than over the whole study period
(Table 4). This restriction increased the PPV of influenza for each
criteria, and consequently reduced the nondifferential misclassifi-
cation. Limiting study period is also useful when using less specific
diagnostic criteria such as symptoms of ILI or absence from school

Table 3
Dilution of relative risk (RR) according to sensitivity and specificity

Sensitivity (%) Specificity

100% 99% 90% 50%

100 0.25 (true RR) 0.29 0.52 0.88

90 0.25 0.29 0.54 0.90
80 0.25 0.29 0.56 0.92
Fig. 1. Weekly influenza incidence in the Netherlands during the 1991–1992
50 0.25 0.32 0.67 1.00
influenza season, as reported by Govaert et al. [3].
6464
Table 4
Modification in effectiveness of influenza vaccination according to different criteria
Influenza or influenza-like Vaccine group no. (%) (n = 927)
illness according to
Serology 41 (4)
Family physician 17 (2)
DSS criteria 62 (7)
ICHPPC-2 criteria 108 (12)
Cited and reconstructed from Ref. [3].
or workplace. Study in a large outbreak season will be desired
because high prevalence of target disease increases the PPV.
In conclusion, to avoid a nondifferential diagnosis misclassifica-
tion, it is necessary to use highly specific diagnostic examinations
such as virus detection, to limit the observation period to an inten-
sive epidemic period, and to be conducted in a large outbreak
season.
4. Differential misclassification of diagnosis
Differential misclassification results in either over- or under-
estimation of true association. Table 5 shows examples; the
upper: relatively excess ‘false positive’ cases in vaccinated group
leads underestimation, and the lower: those in unvaccinated
leads overestimation. Relatively excess ‘false negative’ cases in
vaccinated or unvaccinated groups will lead inverse effect, respec-
tively.
In a cohort study, vaccinated people may more readily visit clinic
and more likely to report ILI symptoms to physician than unvacci-
nated people because they may be more health conscious and/or
because the clinic is where they were vaccinated. This leads to an
increased number of influenza patients in vaccinated group, caus-
ing a differential diagnosis misclassification and underestimation
of vaccine effectiveness. In contrast, if unvaccinated people fear ILI-
like symptoms because they were not vaccinated, or if vaccinated
people think those symptoms indicate other diseases since they
were vaccinated, these tendencies would result in overestimation
of the vaccine effect.
Physician behavior can also cause a differential diagnosis mis-
classification. A physician may be more likely to perform a virus
detection test on unvaccinated compared to vaccinated patients,
resulting in a greater dropout of vaccinated patients with influenza,
and hence an overestimation of vaccine effectiveness. The reverse
tendency would result in an underestimation. Therefore, standard-
ization of diagnostic procedures independent of patient vaccination
anamnesis is essential. If not, physicians’ diagnostic procedure
should be examined whether anamnesis of influenza vaccination
affected the diagnostic test or diagnosis itself.
Thus, passive findings from patients with influenza who
visit clinics would result in complicated differential misclassi-
fication. Especially, any information cannot be obtained from
people who did not visit clinics; whether they had influenza
or other diseases, died, or moved out of study area. These
dropouts reduce the validity of study. Therefore, potential patients
Table 5
Examples of influence of differential misclassification of diagnosis
Comparison groups True number of True relative risk (RR)
patients with influenza
Vaccinated 1000 50
0.25
Unvaccinated 1000 200
Vaccinated 1000 50
0.25
Unvaccinated 1000 200
K. Ozasa / Vaccine 26 (2008) 6462–6465
Placebo group no. (%) (n = 911) Relative risk for whole period Relative risk for epidemic period
80 (9) 0.50 0.39
31 (3) 0.53 0.40
89 (10) 0.69 0.41
129 (14) 0.83 0.74
with influenza must be equally scrutinized both in vaccinated
and unvaccinated cohort using the same defined criteria. This
design element is more important than any others such as
using highly specific examinations or criteria such as virus
detection, which reduces nondifferential misclassification. In a
recent prospective study in Japan [4], there was no description
how investigators surveyed their all cohort members, except for
devices (self-report questionnaire, phone call, or by mail). Thus,
ILI patients determined in this study were derived from those
who visited clinics, so that the results might be seriously dis-
torted by diagnostic differential misclassification as well as other
biases.
In a case–control study, which has been widely used in rou-
tine use, the similar problems would occur. Case–control study
requires ascertaining all cases, or a representative sample of all
cases, occurring in that population over the study period [5]. This
assumption is not consistent with differential misclassification of
diagnosis, derived from patient behavior and physicians’ diagnos-
tic procedure. The latter can be evaluated, but the former is hardly
examined.
In conclusion, selective loss of patients with influenza from
the vaccinated group leads to overestimation of the vaccine effect,
whereas loss from the unvaccinated groups leads to underestima-
tion. Therefore, to avoid differential misclassification of diagnosis,
vaccinated and unvaccinated groups must be equally scrutinized
with the same criteria. This equal observation is more important
than specificity of diagnosis. In a study using passive findings from
influenza patients at clinics, standardization of diagnostic proce-
dures independent of patient anamnesis of vaccination is required,
as well as examination whether diagnosis of influenza might be
affected by the patients anamnesis.
5. Misclassification of exposure
When evaluating the effect of vaccines, exposure means ‘vac-
cination’, and measurement is based on medical records or
self-reporting. While using medical records is the most valid
method, this cannot always be achieved. Self-reporting may cause
misclassification due to incorrect recall by the study subjects.
Nondifferential misclassification of exposure will dilute the effec-
tiveness of vaccine, and differential one may lead either over- or
underestimation. In most situations in evaluating effectiveness of
influenza vaccine, the effect of exposure misclassification will be a
dilution.
Misclassification of diagnosis Observed number of Observed RR
patients with influenza
+20 70
0.33
+10 210
0 50
0.21
+30 230
 K. Ozasa / Vaccine 26 (2008) 6462–6465
6. Conclusion
Misclassification of a diagnosis, especially differential misclassi-
fication, seriously affects the results of studies on influenza vaccine
effectiveness. The following conditions must prevail in order to
avoid such misclassification: equal scrutiny of outcomes in both
vaccinated and unvaccinated groups and diagnosis of influenza
being independent of patient vaccination status, to ascertain all
cases or representative sample of influenza in the study popula-
tion; and highly specific diagnostic criteria and high prevalence of
influenza in the study population, to minimize ‘false positive’ cases.
6465
References
[1] Last JM, editor. A dictionary of epidemiology. 4th ed. Oxford: Oxford University
Press; 2001.
[2] Rothman K. Introduction to modern epidemiology. 2nd ed. Chestnut Hill, MA:
Epidemiology Resources Inc.; 1990. p. 24–9, 57–60.
[3] Govaert TM, Thijs CT, Masurel N, Sprenger MJ, Dinant GJ, Knottnerus JA. The
efficacy of influenza vaccination in elderly individuals: a randomized double-
blind placebo-controlled trial. JAMA 1994;272:1661–5.
[4] Kawai N, Ikematsu H, Iwaki N, Satoh I, Kawashima T, Tsuchimoto T, et al. A
prospective, internet-based study of the effectiveness and safety of influenza
vaccination in the 2001–2002 influenza season. Vaccine 2003;21:4507–13.
[5] Rodrigues LC, Smith PG. Use of the case–control approach in vaccine evaluation:
efficacy and adverse effects. Epidemiol Rev 1999;21:56–72.