See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/6515620

Breaking the first law of informatics: The Quality and

Outcomes Framework (QOF) in the dock

Article  in  The Journal of Innovation in Health Informatics · February 2006

DOI: 10.14236/jhi.v14i3.625 · Source: PubMed

CITATIONS READS

36 351

2 authors, including:

Simon de Lusignan
University of Oxford
691 PUBLICATIONS   11,491 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Integrate Gastroenteritis Project View project

MOTION – Morphological Computation of Perception and Action View project

All content following this page was uploaded by Simon de Lusignan on 20 September 2015.

The user has requested enhancement of the downloaded file.

Informatics in Primary Care 2006;14:153–6
Editorial
Breaking the first law of informatics:
the Quality and Outcomes Framework
(QOF) in the dock
Simon de Lusignan MSc MD MRCGP
Senior Lecturer, Primary Care Informatics, Department of Community Health Sciences, St George’s,
University of London, London, UK
Chris Mimnagh MBChB MSc
Medical Director, Knowsley PCT, Huyton, UK
Introduction
van der Lei stated ‘the first law of informatics’:1
Data shall be used only for the purpose for which they
were collected.
And the collateral:
If no purpose was defined prior to the collection of data,
then the data should not be used.
Information technology makes aggregation of data a
relatively easy task on a regional or national basis. One
good example of this is the data used to evaluate
performance on the Quality and Outcomes Frame-
work (QOF).2 These are data taken from nearly every
general practice in the UK to measure achievement
of quality targets in chronic disease management.
Evidence-based quality targets have been set for the
management of chronic disease and quality points are
awarded for achievement of these targets. Results for
every practice and their individual quality scores,
are publicly available from The Information Centre.3
Simply having these data so readily available makes it
tempting to draw conclusions from them in breach of
the first law of informatics. This editorial explores the
pros and cons of using data recorded for one purpose
but subsequently analysed to support objectives not
stated at the time of collection.
We first examine the charge that QOF data, col-
lected for managerial purposes, are now being used to
draw conclusions about the quality of care and the
defence of this approach.
# 2006 PHCSG, British Computer Society
The case for the prosecution:
QOF breaks the first law of
informatics
The Quality Prevalence and Indicator Database (QPID)4
has scant regard for the first law of informatics. It
states:
It [QPID] meets the need for additional information
requirements arising from the introduction of the Quality
and Outcomes Framework in general practice ...
And makes the assumption:
... a set of QPID tables, providing summaries of 2004/05
QOF information that should meet most users’ require-
ments.5
The assumption is that reuse of clinical data recorded
on a general practice electronic patient record can
readily be used for other purposes. Previously primary
care clinicians, working in the context of the ten-minute
consultation, rarely recorded the complete dataset as
part of clinical care. The existence of financially rewarded
quality points has now subtly changed the purpose of
data recording. Data recorded for a financial purpose
is now considered usable for other purposes. Re-
searchers are starting to comment on associations
between QOF data and factors such as deprivation.
They make the assumption that acknowledgement of
these data’s potential shortcomings makes this breach
of the first law acceptable.6
Although the first law of informatics is 15 years old,
it is widely supported in the informatics literature. The
idea that data for health service management could be
154 S de Lusignan and C Mimnagh
derived from routinely recorded clinical data was an
idea promoted in the 1998 NHS Information Strategy
Information for Health.7 Some disrespectfully suggested
that the idea that management data can be generated
as a by-product of clinical care was ‘Burns’ folly’.
Informaticians highlighted the problems of the
multiple reuse of information collected from routine
patient care and noted that the strategy had under-
estimated the magnitude of change required.8 There
is a need to move from different patterns of data
recording in primary and secondary care, between
clinical and non-clinical staff, and create a systemic
approach to recording data.
The arguments about taking limited data and
drawing conclusions about the quality of care by using
apparent measures of outcome are well-rehearsed within
the hospital setting, where the use of star ratings has
been roundly criticised.9 A wider range of factors con-
tribute directly towards quality. The quality of the
building and team (structures), organisational pro-
cesses to ensure quality (such as training, appraisal,
clinical audit), as well as other outcomes which should
include patient satisfaction and quality of life, may be
as important as easy-to-measure data. Case mix will
also make a significant contribution to outcome.
Although the authors presented the case for a move
from measuring outcomes to monitoring process, we
see the fundamental flaw in the hospital star rating
system as breaking the first law of informatics.
There are many good reasons why QOF data are
not readily usable for other purposes. These have been
summarised by Bingham.10 For example:
. QOF data cannot identify the age–sex distribution
of the practice list and therefore standardise preva-
lence rates
. QOF data do not identify the number of patients
suffering from comorbidity of chronic diseases; you
cannot correlate the presence of one risk factor or
disease with another except within the same target
population
. QOF data do not show the level of exception coding
for all indicators nor the precise exception codes
used
. the denominator (population size) and numerator
for diseases (national prevalence day) are assessed
on different days11
. not all practices participated in the QOF and some
practices that opted out of the standard General
Medical Services contract – choosing a Personal
Medical Services (PMS) contract instead – may have
negotiated different local contracts.11
These shortcomings greatly weaken the ability of QOF
data to be used in the ways that are proposed.
The case for the defence: we all
use routinely collected data
The case for the defence is that this is a bad and
outmoded law, honoured in the breach and to be
disregarded.
Routinely collected data are being used more and
more for quality improvement, health service plan-
ning and research and this is an international
phenomenon.12 The use of routinely collected data
is growing and health informaticians cannot sit, like
Canute, saying that this particular tide should not
come in.
We must learn how to overcome any limitations
of these data. Primary care informatics has been
characterised as:13
The scientific study of data, information and knowledge
and how to use them to improve health and primary
medical care ...
Primary care informaticians should be developing
techniques for overcoming these problems. This can
only be achieved through informaticians encourag-
ing multidisciplinary teamwork involving clinicians,
managers and technologists.
We should be able to measure and adjust for any
perverse incentives associated with data recording.
For example, if an elderly man got chest pain walking
up a steep hill that went when he rested, historically, in
the pre-QOF world of general practice, the authors
would have labelled him as having ‘Angina’. Since
QOF, this person would be labelled ‘Chest pain’ until
all the investigations had been done and the diagnosis
confirmed by a cardiologist.
There are already benefits from using data for other
purposes. Services such as ‘Dr Foster’ are making
routinely collected data more available14 and by
identifying ‘frequent fliers’ (high users of NHS ser-
vices) are using routinely collected data to influence
decisions about the nature of healthcare provision.15
The verdict
The case is finely balanced. Those who flout the first
law have such faith in QOF data that they over-claim
what these data infer, but fail to take account of the
complexity of primary care, incentives related to its
recording and the incompleteness of these data. Is it
possible that its ready availability and novelty is
seducing researchers and editors alike?
However, even if there are insufficient ‘health warn-
ings’ about these data’s weaknesses, is this a reason
not to use them? The challenge for the informatics

community is to dissect the detail of each indicator
and to identify where it is reliable and where there are
limitations.
Information technology allows high entropy of
data; larger and larger volumes of data are becoming
more widely available, yet its presentation may be
chaotic, making its value limited or uncertain. QOF
data about every practice can be readily downloaded
from the internet. We need to learn the lessons of how
to draw appropriate conclusions from these data
before unbridled access to routine data occurs via
the new Secondary Uses Service,16 which aims to
‘provide timely, pseudonymised patient-based data
and information for purposes other than direct clin-
ical care ...’.
Our judgement is that only analyses that take into
account all the socio-technical issues, the complexity
and contextual issues associated with a set of data can
ever overcome the first law of informatics. Informa-
ticians should be describing the detail of the strengths
and weaknesses of the current technical specifications
in the pages of this journal.
A second law of informatics?
There may be a second law of informatics:
Information technology allows the entropy of clinical
data, but caution is required when proceeding, having
intentionally disregarded the initial premise for the data
recording. The first law of informatics can be broken given
a sufficiently limited dataset for which all a priori variables
are known.
Only specific targeted reuse of data will prove possible.
The a priori variables required to correctly interpret
data include an understanding of the socio-technical
context and the complexity of the clinical care envir-
onment in which the data were recorded and the input
of a front-line professional who understands what the
data meant at the time of recording. Other than when
all the factors related to a small dataset are identified,
the first law of informatics will continue to apply.
Reprocessing of entire clinical datasets and expecting
to infer meaning from them will remain illusory.
ACKNOWLEDGEMENTS
We are grateful for Professor Johan van der Lei’s
helpful comments.
Breaking the first law of informatics 155
REFERENCES
1 van der Lei J. Use and abuse of computer-stored medical
records. Methods of Information in Medicine 1991;30:79–
80.
2 British Medical Association. Quality and Outcomes
Framework (new GMS Contract). February 2003. www.
bma.org/ap.nsf/Content/investinggpchap3
3 The Information Centre. www.ic.nhs.uk
4 The Information Centre. QPID (Quality Prevalence and
Indicator Database). www.ic.nhs.uk/psu/services/QOF/
5 The Information Centre. QPID (Quality Prevalence and
Indicator Database) Project: introductory notes for poten-
tial users. www.ic.nhs.uk/psu/services/QOF/qpidintro.
pdf/file
6 Strong M, Maheswaran R and Radford J. Socioeconomic
deprivation, coronary heart disease prevalence and
quality of care: a practice-level analysis in Rotherham
using data from the new UK general practitioner Quality
and Outcomes Framework. Journal of Public Health
(Oxford) 2006;28:39–42.
7 Burns F. Information for Health: an information strategy
for a modern NHS 1998–2005. London: Department of
Health, 1998. www.dh.gov.uk/assetRoot/04/01/44/69/
04014469.pdf
8 Rector A. Terminology, codes, and classifications in
perspective: the challenge of re-use. British Journal of
Healthcare Computing and Information Management
2000;17:20–3.
9 Lilford R, Mohammed MA, Spiegelhalter D and Thomson
R. Use and misuse of process and outcome data in
managing performance in acute medical care: avoiding
institutional stigma. The Lancet 2004;363:1147–54.
10 Bingham P. The limitations of Quality and Outcomes
Framework (QOF) data. ERPHO (Eastern Region Public
Health Observatory) Briefing No. 3, 14 July 2005. www.
erpho.org.uk/download.aspx?urlid=12597&urlt=1
11 The Information Centre. Quality and Outcomes Frame-
work. Background information. 2005. www.ic.nhs.uk/
services/qof/background/index_html#notes
12 de Lusignan S and van Weel C. The use of routinely
collected computer data for research in primary care:
opportunities and challenges. Family Practice 2006;23:
253–63.
13 de Lusignan S. What is primary care informatics? Journal
of the American Medical Informatics Association 2003;
10:304–9.
14 The Information Centre and Dr Foster. Dr Foster Intel-
ligence. www.drfoster.co.uk
15 Curtis P. ‘Frequent Flyers’ cost the NHS £2.3bn a year.
Guardian Society, 13 February 2006. society.guardian.
co.uk/healthmapping/story/0,,1708515,00.html
16 NHS Connecting for Health. Secondary Uses Service (SUS).
www.connectingforhealth.nhs.uk/delivery/programmes/
sus
 156 S de Lusignan and C Mimnagh

ADDRESS FOR CORRESPONDENCE

Simon de Lusignan
Senior Lecturer, Primary Care Informatics
Department of Community Health Sciences
St George’s, University of London
6th Floor, Hunter Wing
Cranmer Terrace
London SW17 0RE
UK
Tel: +44 (0)20 8725 5661
Fax: +44 (0)20 8725 8584
Email: slusigna@sgul.ac.uk

Accepted June 2006

View publication stats