CRITICAL

APPRAISAL
“I read a book twice as fast as anybody else.
First, I read the beginning, and then I
read the ending, and then I start in the
middle and read toward whatever end I like
best.”

Gracie Allen
DON’T BELIEVE EVERYTHING
YOU READ,

But DON’T be sceptical

Give every paper the benefit of

the doubt
Why Critical appraisal?
 Scenario
You are in a busy clinic, A patient shows you a
newspaper clipping, reviewing a recent article from a
national medical journal that warns against the use of
HRT because of links with breast cancer. You assess
her risk of breast cancer as low, but she declines
HRT.
When you discuss with her the results of an article
showing that postmenopausal use of oestrogen
reduces the risk of IHD, she counters with another
article that concludes that cardiovascular mortality is
increased in oestrogen users!
 Why appraisal?
 Information overload
 ‘’Internet printout / newspaper’’
syndrome
 Can deal with Drug reps
 Peer review
 Systematic review
 Better practice
HOW DO WE CRITICALLY
APPRAISE A PAPER?
 Validity } 
 Reliability }    11 QUESTIONS
 Applicability} 
 A1. validity
1. Did the study ask a clearly focused question
in terms of:
 Population
 Intervention
 Outcome
2. Randomised?
3. Were all the subjects who entered the trial
properly accounted for at its conclusion?
 YES?  continue Appraising

 NO?  not worth reading

 A2. Validity
Are there any differences between the
two groups which could explain the
differences between them? (age, sex,
social class)
 Consider:
4. Blinding
5. Were the groups similar at the start of
the trial?
6. Apart from the intervention, were the
groups treated equally?
 B. What are the results?
(RELIABILITY)
7. How large was the difference between
the two groups?

8. How precise was the estimate of the

treatment effects?
C. Will the results help locally?
(APPLICABILITY)

9. Can the results be applied to your local

population?

10. Were all the important outcomes

considered?

11. Are the benefits worth the harms and

costs?
 A1. validity
1. Did the study ask a clearly focused question
in terms of:
 Population
 Intervention
 Outcome
2. Randomised?
3. Were all the subjects who entered the trial
properly accounted for at its conclusion?
 Population:
 Residence, Age, Gender, Occupation
 How were they recruited?
 Who was included/excluded

⇓
SELECTION BIAS!
Exclusion of people with mild / severe
disease  misleading conclusions about
treatment effect
 Intervention:
 Is dose adequate?
 Is duration adequate?
 What is the comparison? Placebo?
 outcome
 What are the outcomes?
 measured over what period?
 Is the outcome measure valid?
 Systematic BIAS
 Anything that erroneously influences
the conclusions about groups and
distorts comparisons
 Fig 1 Sources of bias to check for in a randomised controlled trial

Greenhalgh, T. BMJ 1997;315:305-308

Copyright ©1997 BMJ Publishing Group Ltd.

 Other BIAS
 Attrition bias

 Publication bias
 Language bias
 MD bias
 Randomisation
 Good:
 random number generator
 random number tables
 Bad:
 Alternate days / months
 Date of birth
Were all the subjects who
entered the trial properly
accounted for at its
conclusion?
Original and Proposed Revised Versions of
CONSORT Flow Chart

Egger, M. et al. JAMA 2001;285:1996-1999.

Copyright restrictions may apply.

Egger, M. et al. JAMA 2001;285:1996-1999.
"intention to treat" analysis
were patients analysed in the groups to
which they were randomised?
A2. Are there any differences between
the two groups which could explain the
differences between them? (age, sex,
social class)
 Consider:
4. Blinding
5. Were the groups similar at the start of
the trial?
6. Apart from the intervention, were the
groups treated equally?
 Blinding
 Participants
 Health care workers
 Study personnel
Were the groups similar at the
start of the trial?

 Baseline demographics
Apart from the intervention,
were the groups treated equally?
 Frequency of being seen
 Depth of examination
 Blood tests etc.

⇓
PERFORMANCE BIAS
 B. What are the results?

7. How large was the difference between

the two groups?

8. How precise was the estimate of the

treatment effects?
 Results:

 ARR
 RRR
 ARR
 ARR = event rate in placebo – event rate
in treatment group
 Egin a study of a drug to prevent MI 18.5%
on placebo had MI & 9% on treatment had
MI 
ARR = 18.5 - 9 = 9.5%
 RRR
 RRR = event rate in placebo – event rate in treatment group
event rate in placebo

= 18.5 – 9 = 9.5 = 51%

18.5 18.5
How precise was the estimate of
the treatment effects?
 P Value
 The probability that the results
observed in a study could have occurred
by chance
 <0.05 statistically significant
 The smaller the better
Confidence intervals
 95% CI: range
covered by 2SD
above & below mean
 CI = mean +/- 1.96SD
 99% CI = mean +/-
3SD
C. Will the results help locally?

9. Can the results be applied to local population?

10. Were all the important outcomes

considered?

11. Are the benefits worth the harms and

costs?
9. Can the results be applied to
local population?
 Are there are any differences between
the participants in the trial and the
local population that would make it
impossible to apply the results locally?
 Any variable? Eg:
 Hollywood stars
 all participants from inland Europe

 Migrants etc.
10. Were all the important
outcomes considered?

 Has the paper answered the original

research question and whether any
other important outcomes have been
highlighted or missed out
example
 Symptoms
 ADL
 Hospitalisation
 Adverse effects
11. Are the benefits worth the
harms and costs?
 NNT
 the number of patients you need to treat to
prevent one additional bad outcome

 NNT = 1 .
ARR
= 1/ 9.5% = 10
 A drug with a NNT of 60
 Bd dose
 £40 per dose
 Tratment for 12 months
 Total cost = 40 x 2 x 360 x 60 = £1.7M
to save one life
 WOULD YOU PRESCRIBE IT?
NO
 Drug with NNT of 3
 OD dose
 £1 per dose
 Treatment for 6 months
 Total cost = 1 x 180 x 3 =£540 to save
one life
 WOULD YOU PRESCRIBE IT?
Causes DVT in 30% !!
NO
 Scenario
You are in a busy clinic, A patient shows you a
newspaper clipping, reviewing a recent article from a
national medical journal that warns against the use of
HRT because of links with breast cancer. You assess
her risk of breast cancer as low, but she declines
HRT.
When you discuss with her the results of an article
showing that postmenopausal use of oestrogen
reduces the risk of IHD, she counters with another
article that concludes that cardiovascular mortality is
increased in oestrogen users!