Available online at www.sciencedirect.

com

International Journal of Nursing Studies 46 (2009) 141–142

www.elsevier.com/ijns
Commentary
Interrater reliability and the kappa statistic:
A comment on Morris et al. (2008)
Jan Kottner *
Centre for Humanities and Health Sciences, Department of Nursing Science, Charité Universitätsmedizin Berlin,
Charitéplatz 1, 10117 Berlin, Germany
Received 26 March 2008; received in revised form 1 April 2008; accepted 1 April 2008

Keywords: Interrater reliability; Kappa

Establishing interrater reliability of instruments is an
important issue in nursing research and practice. Morris
et al.’s (2008) paper highlights the problem of choosing the
appropriate statistical approach for interrater reliability data
analysis and the authors raise the important and relevant
question how to interpret kappa-like statistics like Cohen’s
(k) or weighted kappa (kw).
It is true that the often called ‘chance corrected’ k was
frequently criticised because its value is dependent on the
prevalence of the rated trait in the sample (‘base rate
problem’). Consequently, even if two raters nearly or exactly
agree, k-coefficients are nearly or equal to 0 if the prevalence
of the rated characteristic is very high or very low. This
objects the natural expectation that interrater reliability must
be high as well. However, this is neither a limitation nor a
‘‘main drawback’’ (p. 646). In fact it is a desired property,
because k-coefficients are classical interrater reliability
coefficients (Dunn, 2004; Kraemer et al., 2002; Landis
and Koch, 1975). In the classical test theory, reliability is
defined as the ratio of variability between subjects (or
targets) to the total variability. The total variability is the
sum of subject (target) variability and the measurement error
(Dunn, 2004; Streiner and Norman, 2003). Consequently, if
the variance between the subjects is very small or even zero
the reliability coefficient would be near zero as well. There-
fore, reliability coefficients do not only reflect the degree of
agreement between raters, but also the degree to which a
* Tel.: +49 30 450 529 054; fax: +49 30 450 529 900.
E-mail address: jan.kottner@charite.de.
measurement instrument can differentiate among indivi-
duals. The only reason to apply a particular instrument to
a measurement situation is to differentiate between indivi-
duals (Streiner and Norman, 2003). If this very instrument is
not able to detect any differences, one should rather question
the instrument than the statistic (Shrout, 1998). Morris et al.
(2008) were unable to calculate kw in 18 out of 63 variables,
because of the ‘‘large number of ‘constants’ in the data’’ (p.
646). In other words, both participating experienced staff
nurses were unable to detect any variability among 30 clients
when nearly one-third of the I-NMDS variables were
applied. This is an important finding. Users must be careful
when applying these 18 I-NMDS items, because the degree
of interrater reliability is still unknown. That means a certain
degree of relative precision of these items could not be
demonstrated. It is questionable to apply such a measure-
ment, when it was already expected that variability of the I-
NMDS variables would be very low (p. 647).
Every interrater reliability coefficient is unavoidably
linked to the population to which the instrument is applied
(Kraemer, 1979; Streiner and Norman, 2003). Low interrater
reliability coefficients are caused either by a lack of agreement
between raters or by non-identified differences between the
rated subjects. Nevertheless the strength of interrater relia-
bility coefficients is that it is an indicator for the quality and the
clinical value of observations characterising individuals or
subjects (Kraemer, 1979; Shrout, 1998).
For further discussions on this topic it would be helpful to
strictly differentiate between interrater reliability and inter-
rater agreement. High proportions of interrater agreement

0020-7489/$ – see front matter # 2008 Elsevier Ltd. All rights reserved.
doi:10.1016/j.ijnurstu.2008.04.001
142 J. Kottner / International Journal of Nursing Studies 46 (2009) 141–142
are important but kappa-like statistics provide information
about the clinical value of the ratings. Finally, calculated
proportions of overall agreement are also dependent on
sample characteristics. If the prevalence of a category is
very high or very low it is likely that the overall proportion of
agreement is inflated. To overcome this ‘limitation’ certain
indices of specific agreement seem to be very helpful (Fleiss
et al., 2003; Szklo and Nieto, 2007).
Conflict of interest
None.
References
Dunn, G., 2004. Statistical Evaluation of Measurement Errors:
Design and Analysis of Reliability Studies, second ed. Hodder
Arnold, London.
Fleiss, J.L., Levin, B., Paik, M.C., 2003. Statistical Methods for
Rates and Proportions, third ed. Wiley, New Jersey.
Kraemer, H.C., 1979. Ramifications of a population model for k as a
coefficient of reliability. Psychometrika 44 (4), 461–472.
Kraemer, H.C., Periyakoil, V.S., Noda, A., 2002. Kappa coefficients
in medical research. Stat. Med. 21 (14), 2109–2129.
Landis, J.R., Koch, G.G., 1975. A review of statistical methods in the
analysis of data arising from observer reliability studies (part I).
Statistica Neerlandica 29, 101–123.
Morris, R., MacNeela, P., Scott, A., Treacy, P., Hyde, A., O’Brien, J.,
Lehwaldt, D., Byrne, A., Drennan, J., 2008. Ambiguities and
conflicting results: The limitations of the kappa statistic in
establishing the interrater reliability of the Irish nursing mini-
mum data set for mental health: a discussion paper. Int. J. Nurs.
Stud. 45 (4), 645–647.
Shrout, P.E., 1998. Measurement reliability and agreement in psy-
chiatry. Stat. Methods Med. Res. 7 (3), 301–317.
Streiner, D.L., Norman, G.R., 2003. Health Measurement Scales: A
Practical Guide to their Development and Use, third ed. Oxford
University Press, Oxford.
Szklo, M., Nieto, F.J., 2007. Epidemiology Beyond the Basics,
second ed. Jones and Barlett Publishers, Sudbury.