TEACHER’S CORNER

(wileyonlinelibrary.com) DOI: 10.1002/pst.1659 Published online 3 December 2014 in Wiley Online Library

The disagreeable behaviour of the

kappa statistic
Laura Flight and Steven A. Julious*
It is often of interest to measure the agreement between a number of raters when an outcome is nominal or ordinal. The kappa
statistic is used as a measure of agreement. The statistic is highly sensitive to the distribution of the marginal totals and can
produce unreliable results. Other statistics such as the proportion of concordance, maximum attainable kappa and prevalence
and bias adjusted kappa should be considered to indicate how well the kappa statistic represents agreement in the data. Each
kappa should be considered and interpreted based on the context of the data being analysed. Copyright © 2014 John Wiley &
Sons, Ltd.

Keywords: kappa statistic; concordance; agreement; PABAK

1. INTRODUCTION
Table I. Interpretations
It is often of interest to measure the agreement between a num- of the kappa statistic
ber of raters when an outcome is nominal or ordinal. The kappa
Kappa Agreement
statistic, first proposed by Cohen [1], is a measure of agreement
and is frequently used. This method, however, has a number of < 0.20 Poor
limitations. 0.21  0.40 Fair
This Teacher’s Corner article gives a description of the statis- 0.41  0.60 Moderate
tic and important related concepts, followed by a motivational 0.61  0.80 Good
example of how the limitations affect the usefulness of kappa in 0.81  1.00 Very Good
practice.
chance denoted by pe [1]. Consequently, Cohen proposed the
2. THE KAPPA STATISTIC kappa statistic:
po  pe
The measurement of agreement often arises in the context of D . (1)
1  pe
reliability, where the concern is less about whether there is an
This estimates the proportion of agreement between raters after
association or correlation between the classifications of two raters
removing any chance agreement. The kappa statistic can be inter-
but more about whether the two raters agree [2]. Bloch and Krae-
preted using the scale taken from Altman [4] given in Table I.
mer [3] suggest that agreement is a distinct kind of association
Shrout [5] gives more conservative interpretations of the statis-
describing how well one rater’s classification agrees with another
tic, for example, values in the range 0.41–0.60 represent only
or how well a single rater’s classifications at one time point agrees
fair agreement. Throughout, the Altman scale of interpretation
with their classifications at another time point.
is used.
Throughout this article, the main focus is on inter-rater agree-
ment, where it is assumed the objects to be classified are inde-
pendent, the raters make their classifications independently and 2.2. Weighted Kappa Statistic
the categories are independent [1]. The issues raised can also be When considering multiple categories to classify objects, ordering
generalised to intra-rater agreement. is important. Cohen’s kappa (Equation 1), however, assumes the
disagreement between different categories is equally weighted,
2.1. Cohen’s Kappa Statistic and the ordering of categories is not important [6]. If disagree-
ments are thought to have varying consequences, a weighted
The kappa statistic was proposed by Cohen in the context of two
kappa can be calculated. This method applies a weight vij to dis-
raters classifying objects into two categories. The proportion of
agreements in the ith row and jth column of the data table. Larger
concordance po is the number of times different raters classify
objects into the same category divided by the total sample size
(N). This is the observed agreement between the raters and can Medical Statistics Group, University of Sheffield, Sheffield, England
be used as a simple estimate of agreement. *Correspondence to: Professor Steven Julious, Medical Statistics Group, ScHARR,
Cohen (1960) suggests this estimate alone is not sufficient University of Sheffield, 30 Regent Court, Regent Street, Sheffield, England, S1 4DA
as it is necessary to account for agreement expected by E-mail: s.a.julious@sheffield.ac.uk
74

Pharmaceut. Statist. 2015, 14 74–78 Copyright © 2014 John Wiley & Sons, Ltd.
L. Flight and S. A. Julious

Feinstein and Cicchetti [7] highlight the following issues they

Table II. Data with term ‘paradoxes’ of the kappa statistic:
two raters categoris-
ing into two cate- (1) For high values of concordance, low values of kappa can
gories be recorded.
(2) Asymmetric, imperfectly imbalanced tables have a higher
1 2 Total kappa than perfectly imbalanced and symmetric tables.
1 a b g1
By examining Equation 1 for the kappa statistic, it is evident
2 c d g2
that its value is dependent on the proportion of agreement
Total f1 f2 N expected by chance pe . The smaller the value of pe , the larger the
kappa statistic will be. The larger the pe , the smaller the kappa is.
Feinstein and Cicchetti illustrate that the value of pe is depen-
dent on the distribution of the marginal totals. This is evident
weights apply greater penalties to disagreements with larger con- when rewriting the formula for pe as:
sequences [2]. The determination of these weights should be
made prior to the collection of data [6]. .f1 g1 C f2 g2 /
The formula for a weighted kappa statistic is given by, pe D . (3)
N2
P
vij poij
w D 1  P , (2) The values f1 , f2 and g1 , g2 are the marginal totals for rater 1 and
vij peij
rater 2, respectively.
where poij is the proportion of observed classifications in the i, jth The first paradox occurs when there is symmetrical imbal-
cell, and peij is the proportion of expected classifications. The peij ance in the vertical and horizontal marginal totals. The sec-
are found by multiplying the ith row total by the jth column total ond paradox occurs if the imbalance is asymmetrical or imper-
and dividing by the total sample size N. fectly symmetrical.

3.1. The Maximum Attainable Kappa, Ämax

3. THE PARADOXES
The following definitions (Box 2) are given to help explain some
of the issues arising with the kappa statistic and are used in the
context of Table II.
Throughout this article, the special dichotomous (22) case
is considered where no weightings are applied. The principles
discussed can be generalised to the (r  r) case.
Feinstein and Cicchetti also highlight that it is not only the mag-
nitude of kappa that is affected by the marginal totals but also
the maximum possible value of the statistic [7]. Cohen notes
that kappa can only reach the maximum value of one when the
off-diagonal elements (b, c) in Table II are equal to zero [1]. For
this to occur, the marginal values must be identical. Hence g1 D f1
and g2 D f2 , resulting in a perfectly symmetrical table.

75

Pharmaceut. Statist. 2015, 14 74–78 Copyright © 2014 John Wiley & Sons, Ltd.

The maximum attainable kappa, max , is calculated using,
poM  pe
max D , (4)
1  pe
where poM is estimated by,
     
g1 f 1 g2 f2 gr fr
poM D min , C min , C : : : C min , (5)
N N N N N N
for an r  r table.
Sim and Wright [2] interpret max as reflecting the extent to
which the rater’s ability to agree is constrained by pre-existing fac-
tors that result in unequal marginal totals. It is helpful to report
this statistic alongside Cohen’s kappa, as it can illustrate how a low
value might be a consequence of the marginal totals rather than
poor agreement.
3.2. Prevalence and Bias
Byrt et al. [8] describe the kappa statistic as being affected by both
the prevalence and bias between raters . Prevalence in this con-
text is the probability with which a rater classifies an object as
‘one’ in the study sample, say. This relates to the balance of the
table, with low prevalences giving a balanced table.
Bias is concerned with the frequency at which raters choose
a particular category and affects the symmetry of the table. A
non-biased table will be symmetrical as the raters do not differ
in their frequency of choosing category one. The extent of bias
and prevalence can be evaluated using the Prevalence and Bias
indexes. Prevalence and bias are used along with symmetry and
balance to explain when the paradoxes of kappa can occur.
4. PREVALENCE AND BIAS ADJUSTED KAPPA
If after examining the symmetry and balance of a table and esti-
mating the prevalence index (PI) and the bias index (BI) it is clear
that the kappa is likely to be influenced by the distribution of
the marginal totals, it is possible to calculate an adjusted kappa
statistic. Byrt et al. (1993) define a statistic that adjusts for preva-
76
Copyright © 2014 John Wiley & Sons, Ltd.
L. Flight and S. A. Julious
lence and bias (PABAK) [8]. This statistic is estimated by replacing
the diagonal elements of the table (a, d) by their average n D
.a C d/
and the off-diagonal elements (b, c) by their average
2
.b C c/
m D . Substituting these values into Equations 1 and 3
2
and using N D aCbCcCd, the formula for PABAK can be reduced
to:  
2n
 0.5
N
PABAK D D 2po  1 (6)
.1  0.5/
5. MOTIVATING EXAMPLE
In this section, a number of examples are used to illustrate
how the kappa statistic should not be solely relied upon when
assessing agreement.
In the example that is the motivation for this paper, there
are N D 261 students who are categorised by two indepen-
dent assessors as either ‘one’ or ‘two’. It is important to establish
whether there is agreement between assessors. If agreement is
poor it would be necessary to use additional assessors or to have
the grading scheme amended.
5.1. Unadjusted Kappa
Data for this example are given in Table III. Using Feinstein and
Ciccetti’s definitions, this is an example of a symmetrically imbal-
anced table, as f1 and g1 are large, and f2 , g2 are small. The propor-
tion of the objects in class one for both assessors is clearly greater
than 0.5. There is a high probability that an assessor will classify
a student as one and hence, high prevalence (PI=0.628). The fre-
quencies for which the assessors choose a particular category are
similar, and so there is low bias (BI=0.234).
The proportion of concordance po D 0.682 indicates good
agreement between the two assessors. The kappa statistic, on
the other hand is  D 0.038, with pe D 0.360. This suggests
little agreement beyond that expected by chance between the
assessors. The interpretation of agreement varies substantially
depending on the summary statistic chosen.
Pharmaceut. Statist. 2015, 14 74–78
L. Flight and S. A. Julious

0.4
Table III. Data with
two assessors categoris-

0.3
ing into two categories
(N D 261)

0.2
1 2 Total

Kappa
1 171 72 243

0.1
2 11 7 18
Total 182 79 261

0.0
Considering the imbalance in the table and the high PI value, 0.65 0.70 0.75 0.80 0.85 0.90 0.95
it is likely that the kappa statistic is being influenced by the Proportion in g1
prevalences and hence, distribution of the marginal totals. The
proportion of students in the marginal g1 is 0.931 and f1 is 0.697. Figure 1. Kappa statistic plotted against the proportion of students rated as
g1
The maximum attainable kappa is one:one . /
N

   
243 182 18 79
min , C min ,  0.670
261 261 261 261
max D D 0.292. Table IV. Perfectly sym-
1  0.670 metrical (N D 261)
(7)
This low value indicates that there is unlikely to ever be strong 1 2 Total
agreement between the assessors when considering the inter-
pretations in Table I, a consequence of factors influencing their 1 100 51 151
marginal totals. 2 51 59 110
As an illustration of how the prevalences and the distribution Total 151 110 261
of the marginals influences kappa, in Table III, the number of stu-
dents in the off-diagonal cells is fixed, and the remaining students
are split evenly between the diagonal cells. One student is then
moved from cell ‘two:two’ to ‘one:one’, hence, the prevalence of Table V. Imperfectly
each assessor categorising a student as one is increased. The over- asymmetrical (N D 261)
all concordance, however, remains constant as the proportion
1 2 Total
of students awarded the same grade by each assessor does not
change. The proportion of concordance is constant where as the 1 60 101 161
kappa statistic falls. 2 1 99 100
Although the proportion of concordance and kappa are not Total 61 200 261
strictly comparable as concordance does not account for agree-
ment expected by chance, the large differences in their values
indicate that the kappa may not be behaving as expected. Table V (PI=0.149, BI=0.383) also has po D 0.609, however, this
The proportion of students in the marginal g1 and kappa are table is asymmetrical. One assessor categorises more students in
calculated as one student is moved as described. These two statis- category one and fewer students in category two when compared
tics are plotted in Figure 1 to demonstrate how there is a change with the second assessor. There is a difference in the distribu-
in kappa despite no change in the concordance and arguably tion of the marginal totals of the assessors. This suggests less
the agreement. The vertical line on the plot marks the propor- agreement than in the symmetric Table IV, as intuitively asses-
tion of students in g1 for Table III, and the horizontal line marks sors who agree will have similar marginal distributions [7]. The
the value of the kappa statistic for this table. The kappa statis- kappa statistic for Table V, however, is larger  D 0.391. This value
tic even falls below zero when the proportion in the marginal g1 indicates greater agreement despite the lack of agreement in the
approaches 0.95. This negative kappa is interpreted by Viera and marginal totals.
Garrett (2005) as less than chance agreement [9].
Based on the motivational example, scenarios are imagined 5.2. Adjusted Kappa
with a sample size of 261 where symmetry and imbalance impact
on kappa. If a table has symmetry and is perfectly imbalanced, In the motivating example, the PABAK D 0.364, much greater
such as the hypothetical example in Table IV, this indicates some than the unadjusted kappa. The high prevalence and imbalance
agreement between the assessors. The PI value is 0.157, a low in the table is decreasing the kappa statistic, which can lead to
value reflecting the distribution of students in the marginals for incorrect inferences about the agreement between the two raters.
both assessors being balanced. The BI score of 0 is a consequence Varying the prevalences in Table III by moving a student from
of the perfect symmetry of the table, g1 D f1 and g2 D f2 . The two:two to one:one as before, the PABAK statistic remains con-
proportion of concordance for this table is 0.609, which further stant as the proportion of objects in the marginal g1 changes.
indicates good agreement, however, the kappa is 0.189. Using the This illustrates how the adjusted kappa is less influenced by
interpretations in Table I, this suggests poor agreement. prevalence and the distribution of the marginals.
77

Pharmaceut. Statist. 2015, 14 74–78 Copyright © 2014 John Wiley & Sons, Ltd.

6. DISCUSSION
The motivational and hypothetical examples presented demon-
strate some of the ways a kappa statistic can behave in an unex-
pected and disagreeable way. Feinstein and Cicchetti attribute
these problems to the distribution of the marginal totals in a table
and the affect this has on the proportion of agreement expected
by chance, pe [7].
Byrt et al. describe the properties of the data in terms of preva-
lences and bias [8]. The Bias Index and the Prevalence Index are
proposed as ways of measuring how bias and prevalence might
influence the marginal totals and impact on the interpretation
of kappa.
The motivational example demonstrates how the kappa statis-
tic can mask high levels of agreement seen when considering
only the observed proportion of agreement. The large discrepan-
cies between these two values can result in misleading inferences
being made if relying solely on the kappa statistic.
The hypothetical examples show how it is not feasible to
use standardised interpretations of kappa such as those given
in Table I. When comparing two tables, a higher kappa value
might actually be associated with a table that has intuitively
less agreement.
Interpretations of the kappa statistic should be made on a case
by case basis after considering all the characteristics of the data.
These include the proportion of concordance and the symme-
try and balance of the marginal totals. The PI, BI and maximum
attainable kappa max should also be calculated so appropriate
interpretations can be made.
Alternatives such as the prevalence and bias adjusted kappa
(PABAK) can be calculated to facilitate comparisons with the
unadjusted kappa to assess how sensitive the statistic is to the
distribution of the marginal totals [8]. It is unlikely that comparing
kappa statistics from different tables and data will be of use due
to these sensitives.
7. CONCLUSION
In this Teacher’s Corner article an example is used to describe
the kappa statistic. Although a well known and frequently used
method for estimating levels of agreement, the kappa statistic has
serious limitations. It is highly sensitive to the distribution of the
marginal totals and can produce unreliable results.
The kappa statistic should be used with caution, and inferences
made should account for these limitations. Kappas from different
data are unlikely to be directly comparable, and a generic scale
78
Copyright © 2014 John Wiley & Sons, Ltd.
L. Flight and S. A. Julious
of interpretation should be used with these limitations in mind.
It is recommended, when reporting the kappa statistic, to also
report the proportion of concordance, PABAK, PI, BI and max
to allow a fully informed judgement about the usefulness of
kappa regarding the level of agreement seen [10]. Each kappa
should be considered and interpreted based on the context
in hand.
Acknowledgements
This report is an independent research arising from an NIHR
Research Methods Fellowship, RMFI-2013-04-011 Goodacre, sup-
ported by the National Institute for Health Research. The views
expressed in this publication are those of the author(s) and not
necessarily those of the NHS, the National Institute for Health
Research, the Department of Health or the University of Sheffield.
We would like to thank two anonymous reviewers for their
valuable comments which greatly improved the paper.
REFERENCES
[1] Cohen J. A coefficient of agreement for nominal scales. Educational
and Psychological Measurement 1960; 20:37–46.
[2] Sim J, Wright CC. The kappa statistic in reliability studies: use, inter-
pretation, and sample size requirements. Physical Therapy 2005;
85:257–268.
[3] Bloch DA, Kraemer HC. 2  2 Kappa coefficients: measures of agree-
ment or association. Biometrics 1989; 45:269–287.
[4] Altman DG. Practical statistics for medical research. Chapman and
Hall: London, 1991, pp. 404.
[5] Shrout PE. Measurement reliability and agreement in psychiatry.
Statistical Methods in Medical Research 1998; 7:301–317.
[6] Cohen J. Weighted kappa: nominal scale agreement with provision
for scaled disagreement or partial credit. Psychological Bulletin 1968;
70:213–220.
[7] Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The
problems of two paradoxes. Journal of Clinical Epidemiology 1990;
43:543–548.
[8] Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. Journal of
Clinical Epidemiology 1993; 46(5):423–429.
[9] Viera A, Garrett JM. Understanding interobserver agreement:the
kappa statistic. Family Medicine 2005; 37(5):360–3.
[10] Chen G, Faris P, Hemmelgarn B, Walker RL, Quan H. Measuring
agreement of administrative data with chart data using prevalence
unadjusted and adjusted kappa. BMC Medical Research Methodology
2009; 9:5. DOI: 10.1186/1471-2288-9-5.
Pharmaceut. Statist. 2015, 14 74–78