How to Incorporate NonRandomized Studies in

Cochrane Reviews of Patient

Safety
Scientific Evidence for Healthcare Quality and Patient Safety
19th Cochrane Colloquium/VI International Conference on Patient
Safety
Joint Plenary Session
Madrid, Spain
October 20, 2010

13 Including non-randomized
studies
Barney Reeves, Julian Higgins, John Deeks, George
Wells

Information/Education
1. Gordon M, Findley R., Educational interventions to improve handover in health
care: a systematic review. Med Educ. 2011 Nov;45(11):1081-9.
2. McCulloch P, Rathbone J, Catchpole K., Interventions to improve teamwork and
communications among healthcare staff. Br J Surg. 2011 Apr;98(4):469-79.
3. Rabl LI, stergaard D, Mogensen T., Outcomes of classroom-based team
training interventions for multiprofessional hospital staff. A systematic review.
Qual Saf Health Care. 2010 Dec;19(6):e27.
4. Fischer SH, Tjia J, Field TS., Impact of health information technology
interventions to improve medication laboratory monitoring for ambulatory
patients: a systematic review. J Am Med Inform Assoc. 2010 Nov-Dec;17(6):6316.
5. Nagpal K, Vats A, Lamb B, Ashrafian H, Sevdalis N, Vincent C, Moorthy K.,
Information transfer and communication in surgery: a systematic review. Ann
Surg. 2010 Aug; 252(2):225-39.
6. Kilner E, Sheppard LA., The role of teamwork and communication in the
emergency department: a systematic review. Int Emerg Nurs. 2010 Jul;
18(3):127-37.
7. Merin AE, van de Ven J, Mol BW, Houterman S, Oei SG. Multidisciplinary team
training in a simulation setting for acute obstetric emergencies: a systematic

Gordon 2011 - Handover

Medical Education 2011: 45: 1081

1089

Handover is the accurate, reliable communication of taskrelevant information on patients across shift changes in staff

Communication failure at handover is identified as a major

source of error within patient care (patient safety)

Objective: To determine the characteristics

of educational interventions employed to
enhance handover amongst health
professionals and to establish the
effectiveness of these interventions

Systematic Review
Steps of a Systematic Review

Clearly formulated question

Comprehensive data search

Unbiased selection and extraction

process

Critical appraisal

Analysis/synthesis of data

Systematic Review
Steps of a Systematic Review

Clearly formulated question

Comprehensive data search

Unbiased selection and extraction

process

Critical appraisal

Analysis/synthesis of data

Gordon 2011 Handover

Clearly formulated question
PICO
Population

Medical and nursing staff, including

undergraduates in in-patient medical
establishments

Interventio Any structured educational activity

n
Compariso
n
Outcome

Standard/usual strategy (not explicitly

identified)

Kirkpatricks adapted hierarchy; 4 levels of

outcome
Level 1 - reaction to intervention
Level 2a - attitudes and confidence
Level 2b - knowledge and skills
Types of studies
Level 3 - behaviour change
designs
were considered
for this
All interventional
Level 4study
- patient
outcomes
(includes patient
review
safety)
commentaries, surveys, audits, review articles not

Systematic Review
Steps of a Systematic Review

Clearly formulated question

Comprehensive data search

Unbiased selection and extraction

process

Critical appraisal

Analysis/synthesis of data

Comprehensive data search

Some points to remember
Exhaustive searching, which is recommended for RCT,
may not be justified when reviewing NRS
Do not limit search strategies by index terms for study
design

Not easy to design a restrictive search strategy for

particular NRS since filters and indexing fields to limit
searches unlikely helpful (NRS design labels not used
consistently in studies, NRS not indexed reliably by
bibliographic databases)

Comprehensive data search

Some points to remember
Exhaustive searching, which is recommended for RCT,
may not be justified when reviewing NRS
Do not limit search strategies by index terms for study
design

Not easy to design a restrictive search strategy for

particular NRS since filters and indexing fields to limit
searches unlikely helpful (NRS design labels not used
consistently in studies, NRS not indexed reliably by
bibliographic databases)

Comprehensive data search

Some points to remember
Exhaustive searching, which is recommended for RCT,
may not be justified when reviewing NRS
Do not limit search strategies by index terms for study
design

Not easy to design a restrictive search strategy for

particular NRS since filters and indexing fields to limit
searches unlikely helpful (NRS design labels not used
consistently in studies, NRS not indexed reliably by
bibliographic databases)

Gordon 2011 Handover

Comprehensive Data Search

Online databases searched (standardized search

strategy)
MEDLINE
EMBASE
CINAHL (Cumulative Index to Nursing and Allied Health
Literature)
British Nursing Index (BNI)
PsycINFO
ERIC (Educational Resource Information Centre)
British Education Index (BEI)
Cochrane Trials Database

Reference lists from included studies were searched for

further relevant studies
Online abstracts from relevant education societies,
including:
Association for the Study of Medical Education
Association for Medical Education in Europe

Systematic Review
Steps of a Systematic Review

Clearly formulated question

Comprehensive data search

Unbiased selection and extraction

process

Critical appraisal

Analysis/synthesis of data

Unbiased selection and

extraction
Some points to remember
Judging eligibility: Reviewing citations and abstracts
identified by searching will be very time consuming since

Large volume of citations identified

Needed information for eligibility may not be in title or
abstract

Data collection: In addition to data required for SR of

RCT, need:

Confounding factors considered

Group comparability on confounding factors and control
methods
Aspects of risk of bias specific for NRS
Adjusted and unadjusted effect estimates

Adjusted results: For NRS, comparisons of the raw data

are unadjusted and susceptible to confounding; need
adjusted

Unbiased selection and

extraction
Some points to remember
Judging eligibility: Reviewing citations and abstracts
identified by searching will be very time consuming since

Large volume of citations identified

Needed information for eligibility may not be in title or
abstract

Data collection: In addition to data required for SR of

RCT, need:

Confounding factors considered

Group comparability on confounding factors and control
methods
Aspects of risk of bias specific for NRS
Adjusted and unadjusted effect estimates

Adjusted results: For NRS, comparisons of the raw data

are unadjusted and susceptible to confounding; need
adjusted

Unbiased selection and

extraction
Some points to remember
Judging eligibility: Reviewing citations and abstracts
identified by searching will be very time consuming since

Large volume of citations identified

Needed information for eligibility may not be in title or
abstract

Data collection: In addition to data required for SR of

RCT, need:

Confounding factors considered

Group comparability on confounding factors and control
methods
Aspects of risk of bias specific for NRS
Adjusted and unadjusted effect estimates

Adjusted results: For NRS, comparisons of the raw data

are unadjusted and susceptible to confounding; need
adjusted

Unbiased selection and

extraction
Some points to remember
Judging eligibility: Reviewing citations and abstracts
identified by searching will be very time consuming since

Large volume of citations identified

Needed information for eligibility may not be in title or
abstract

Data collection: In addition to data required for SR of

RCT, need:

Confounding factors considered

Group comparability on confounding factors and control
methods
Aspects of risk of bias specific for NRS
Adjusted and unadjusted effect estimates

Adjusted results: For NRS, comparisons of the raw data

are unadjusted and susceptible to confounding; need
adjusted

Gordon 2011 Handover

Unbiased Selection and Extraction Process

Citations reviewed independently by each of review
authors

Agreement between review authors assessed using

Cohens kappa statistic

Potentially relevant abstracts independently reviewed

using a screening checklist and full papers obtained for
any studies that appeared to meet the inclusion criteria

Disputes resolved by consensus

Full manuscripts for included studies assessed

independently by each review author

Gordon 2011 Handover

Comprehensive Search/Selection - Results

Citations identified through
database searching
(n=780)

Additional citations from

references of included studies or
abstracts of relevant societies
(n=0)

Citations after duplicates removed

(n=298)
Citations not relevant
(n=258)
Abstracts screened using
screening checklist
(n=40)

Abstracts excluded
(n=21)

Full-text articles assessed for

eligibility
(n=19)

Full-text articles
excluded
No education = 8
Review article = 1

Studies included
(n=10)

Systematic Review
Steps of a Systematic Review

Clearly formulated question

Comprehensive data search

Unbiased selection and extraction

process

Critical appraisal

Analysis/synthesis of data

Critical appraisal - risk of bias

Some points to remember
Reporting guideline STROBE only recently developed
Often no protocol for NRS review; protocol protect against
bias

At the protocol stage, compile a list of potential confounding

factors and justify the choice, independent of studies

At the protocol stage, determine how ROB in primary studies
assessed

Assessing ROB: Difficult to develop a generic tool for

evaluating ROB for NRS since different NRS designs have
varying susceptibility to different biases

Several tools for assessment of risk of bias may be needed

Methodological information needed for ROB can be difficult
to find

Sources of Bias: As for RCT, dimensions of bias assessed

for NRS include: selection, performance, detection,
attrition, reporting pin

Critical appraisal - risk of bias

Some points to remember
Reporting guideline STROBE only recently developed
Often no protocol for NRS review; protocol protect against
bias

At the protocol stage, compile a list of potential confounding

factors and justify the choice, independent of studies

At the protocol stage, determine how ROB in primary studies
assessed

Assessing ROB: Difficult to develop a generic tool for

evaluating ROB for NRS since different NRS designs have
varying susceptibility to different biases

Several tools for assessment of risk of bias may be needed

Methodological information needed for ROB can be difficult
to find

Sources of Bias: As for RCT, dimensions of bias assessed

for NRS include: selection, performance, detection,
attrition, reporting pin

Critical appraisal - risk of bias

Some points to remember
Reporting guideline STROBE only recently developed
Often no protocol for NRS review; protocol protect against
bias

At the protocol stage, compile a list of potential confounding

factors and justify the choice, independent of studies

At the protocol stage, determine how ROB in primary studies
assessed

Assessing ROB: Difficult to develop a generic tool for

evaluating ROB for NRS since different NRS designs have
varying susceptibility to different biases

Several tools for assessment of risk of bias may be needed

Methodological information needed for ROB can be difficult
to find

Sources of Bias: As for RCT, dimensions of bias assessed

for NRS include: selection, performance, detection,
attrition, reporting pin

Critical appraisal - risk of bias

Some points to remember
Reporting guideline STROBE only recently developed
Often no protocol for NRS review; protocol protect against
bias

At the protocol stage, compile a list of potential confounding

factors and justify the choice, independent of studies

At the protocol stage, determine how ROB in primary studies
assessed

Assessing ROB: Difficult to develop a generic tool for

evaluating ROB for NRS since different NRS designs have
varying susceptibility to different biases

Several tools for assessment of risk of bias may be needed

Methodological information needed for ROB can be difficult
to find

Sources of Bias: As for RCT, dimensions of bias assessed

for NRS include: selection, performance, detection,
attrition, reporting pin

Critical appraisal - risk of bias

Some points to remember
Reporting guideline STROBE only recently developed
Often no protocol for NRS review; protocol protect against
bias

At the protocol stage, compile a list of potential confounding

factors and justify the choice, independent of studies

At the protocol stage, determine how ROB in primary studies
assessed

Assessing ROB: Difficult to develop a generic tool for

evaluating ROB for NRS since different NRS designs have
varying susceptibility to different biases

Several tools for assessment of risk of bias may be needed

Methodological information needed for ROB can be difficult
to find

Sources of Bias: As for RCT, dimensions of bias assessed

for NRS include: selection, performance, detection,
attrition, reporting pin

Critical appraisal - risk of bias

Some more points to remember
Selection Bias/Confounding: When selection bias produces
imbalances in prognostic factors associated with the
outcome then confounding occurs

Statistical methods sometimes used to counter

confounding by producing adjusted estimates
Assessment of study quality may involve making
judgments about the appropriateness of this analysis

Confounding factors: For potentials confounding factors:

list in protocol
identify those included/excluded in NRS
identify ways measured
assess balance between groups at baseline
identify ways controlled (match, stratify, model;
propensity)
Residual confounding
Confounding by indication

Critical appraisal - risk of bias

Some more points to remember
Selection Bias/Confounding: When selection bias produces
imbalances in prognostic factors associated with the
outcome then confounding occurs

Statistical methods sometimes used to counter

confounding by producing adjusted estimates
Assessment of study quality may involve making
judgments about the appropriateness of this analysis

Confounding factors: For potentials confounding factors:

list in protocol
identify those included/excluded in NRS
identify ways measured
assess balance between groups at baseline
identify ways controlled (match, stratify, model;
propensity)
Residual confounding
Confounding by indication

Critical appraisal - risk of bias

Some more points to remember
Selection Bias/Confounding: When selection bias produces
imbalances in prognostic factors associated with the
outcome then confounding occurs

Statistical methods sometimes used to counter

confounding by producing adjusted estimates
Assessment of study quality may involve making
judgments about the appropriateness of this analysis

Confounding factors: For potential confounding factors:

list in protocol
identify those included/excluded in NRS
identify ways measured
assess balance between groups at baseline
identify ways controlled (match, stratify, model;
propensity)
Residual confounding
Confounding by indication

Gordon 2011 Handover

Critical Appraisal

Quality of the studies - used 16 quality-based criteria by

Reed, Ann Intern Med 2005
16 Criteria
Literature review described
Clear objectives reported
Appropriate design
Study design reported
Comparison group used
Any randomisation
Blinding
Learner characteristics
Could the study be replicated
Resources described
Outcomes match objectives
Replicable data collection
Statistical tests used
If used, are they appropriate
Outcomes
Strength of conclusions

Gordon 2011 Handover

Critical Appraisal

Strength of the conclusions drawn by each study was

rated on a numeric scale, in line with Best Evidence
Medical Education (BEME) guidance
Measure of how well conclusions made are supported by
data presented

Disputes resolved by discussion until consensus

Results and strength of conclusions
1
2
3
4
5

No clear conclusions can be drawn. Not significant

Results ambiguous, but there appears to be a trend.
Conclusions can probably be based on the results.
Results are clear and very likely to be true.
Results are unequivocal.

Systematic Review
Steps of a Systematic Review

Clearly formulated question

Comprehensive data search

Unbiased selection and extraction

process

Critical appraisal

Analysis/synthesis of data

Analysis/Synthesis of Data
Some point to remember
What is different when including NRS?

Expect greater heterogeneity than for a SR of RCT due to

the increased potential for methodological diversity

Usually appropriate to analyze adjusted effect estimates

(i.e. analyses attempting to control for confounding)

Danger is that a large NRS of poor methodological quality

(for example based on routinely collected data) may
dominate the findings of other smaller studies at less risk
of bias

Analysis/Synthesis of Data
Some more point to remember
When pooling judged not appropriate

Studies not sufficiently homogeneous to combine then

display study results in a forest plot but suppress pooled
estimate

Can sort studies in the forest plot by feature believed to

reflect susceptibility to bias

Heterogeneity diagnostics and investigations are

worthwhile even when a judgment made that a pooled
effect estimate is not

Narrative syntheses problematic since difficult to

describe results without being selective or emphasizing
some findings over others; ideally, set out in protocol

Recommend that NRS of different study designs

should not be pooled (different designs influenced to
varying degrees by different sources of bias; different
designs should expected to differ increasing

Gordon 2011 Handover

Analysis/Synthesis of Data - Results

Types of studies:
6 before-after study
3 action-based study
1 non-randomized controlled study

Interventions wide
range
workshops
periodic lectures
small group discussions
audit and feedback
online/printed material
etc

Outcome measures wide

range
quality checklist/scores
survey KAB
questionnaires opinions
etc

Gordon 2011 Handover

Analysis/Synthesis of Data - Results

Levels of Outcomes
6 level 2a (attitudes and confidence)
3 level 2b (knowledge and skills)
1 level 3 (behaviour change)

Strength of Conclusions
3 at 2/5 (results ambiguous, but appears to be a trend)
3 at 3/5 (conclusions can probably be based on the results)
4 at 4/5 (results are clear and very likely true)

Summary of reported teaching methods and content themes

Teaching methods

Content themes:

Group sessions/lectures
Simulation
Role-play exercises
Online materials

Information management
Team working/leadership/communication
Error awareness and professional
behaviour

Systematic Review
Steps of a Systematic Review

Clearly formulated question

Comprehensive data search

Unbiased selection and extraction

process

Critical appraisal

Analysis/synthesis of data

Interpretation of Results
Some points to remember
Challenges that a result, from a NRS review, can give a
definitive answer about an intervention arise at all stages
of the review

deciding which study designs to include

searching for studies
assessing studies for potential bias
deciding whether to pool results

Adequate assessment of ROB: Biases that affect

RCT also affect NRS but typically to a greater extent;
regarding confounding, remember:

Direction of bias is unpredictable

Methods to control confounding likely varies between
studies
Residual confounding unknown and likely varies between
studies
Identify likely confounding factors which were adjusted

Interpretation of Results
Some points to remember
Challenges that a result, from a NRS review, can give a
definitive answer about an intervention arise at all stages
of the review

deciding which study designs to include

searching for studies
assessing studies for potential bias
deciding whether to pool results

Adequate assessment of ROB: Biases that affect

RCT also affect NRS but typically to a greater extent;
regarding confounding, remember:

Direction of bias is unpredictable

Methods to control confounding likely varies between
studies
Residual confounding unknown and likely varies between
studies
Identify likely confounding factors which were adjusted

Interpretation of Results
Some points to remember
Challenges that a result, from a NRS review, can give a
definitive answer about an intervention arise at all stages
of the review

deciding which study designs to include

searching for studies
assessing studies for potential bias
deciding whether to pool results

Adequate assessment of ROB: Biases that affect

RCT also affect NRS but typically to a greater extent;
regarding confounding, remember:

Direction of bias is unpredictable

Methods to control confounding likely varies between
studies
Residual confounding unknown and likely varies between
studies
Identify likely confounding factors which were adjusted

Interpretation of Results
Some more points to remember
A clue to the presence of bias is notable between-study
heterogeneity (but homogeneity does not indicate lack of
bias)

Evaluating strength of evidence: General concern

about biases in NRS, and the difficulties of attributing
causality to the observed effects

Strength of evidence provided by a SR of NRS likely

depends on meeting the challenges set out
GRADE scheme for assessing quality of a body of evidence
is recommended for use in Summary of Findings tables

Conducting a SR of NRS is more difficult and resource

intensive than a SR of RCT, and conclusions are likely
weaker and may make a relatively small contribution

Interpretation of Results
Some more points to remember
A clue to the presence of bias is notable between-study
heterogeneity (but homogeneity does not indicate lack of
bias)

Evaluating strength of evidence: General concern

about biases in NRS, and the difficulties of attributing
causality to the observed effects

Strength of evidence provided by a SR of NRS likely

depends on meeting the challenges set out
GRADE scheme for assessing quality of a body of evidence
is recommended for use in Summary of Findings tables

Conducting a SR of NRS is more difficult and resource

intensive than a SR of RCT, and conclusions are likely
weaker and may make a relatively small contribution

Interpretation of Results
Some more points to remember
A clue to the presence of bias is notable between-study
heterogeneity (but homogeneity does not indicate lack of
bias)

Evaluating strength of evidence: General concern

about biases in NRS, and the difficulties of attributing
causality to the observed effects

Strength of evidence provided by a SR of NRS likely

depends on meeting the challenges set out
GRADE scheme for assessing quality of a body of evidence
is recommended for use in Summary of Findings tables

Conducting a SR of NRS is more difficult and resource

intensive than a SR of RCT, and conclusions are likely
weaker and may make a relatively small contribution

Interpretation of Results
Some more points to remember
A clue to the presence of bias is notable between-study
heterogeneity (but homogeneity does not indicate lack of
bias)

Evaluating strength of evidence: General concern

about biases in NRS, and the difficulties of attributing
causality to the observed effects

Strength of evidence provided by a SR of NRS likely

depends on meeting the challenges set out
GRADE scheme for assessing quality of a body of evidence
is recommended for use in Summary of Findings tables

Conducting a SR of NRS is more difficult and resource

intensive than a SR of RCT, and conclusions are likely
weaker and may make a relatively small contribution

Gordon 2011 Handover

Conclusions

Paucity of research investigating educational

interventions to improve handover amongst medical and
nursing staff, but growing rapidly
Studies suggest that educational interventions can
improve handover, but small sample sizes, no long-term
retention data and possible publication bias limit
conclusion
Methodological quality of reported studies is generally
poor
Limited evidence demonstrating the transfer of skills to
the workplace and no evidence that these interventions
improve patient outcomes

Non-Randomized Studies Meeting

Ottawa, Canada

Non-Randomized Studies Meeting

Ottawa, Canada

Some types of NRS design used for

evaluating the effects of
interventions
Non-randomized study (NRS)

Non-randomized controlled trial

Controlled before-and-after study
Interrupted-time-series study
Historically controlled study
Uncontrolled longitudinal study
Cohort study
Case-control study
Cross-sectional study

Some types of NRS design used for

evaluating the effects of
interventions

Non-randomized
controlled trial

People are allocated to different interventions using methods that

are not random

Controlled before-andafter study

Observations are made before and after the implementation of an

intervention, both in a group that receives the intervention and in a
control group that does not

Interrupted-time-series
study

Historically controlled
study

Uncontrolled
longitudinal study

Cohort study

Case-control study

Cross-sectional study

s
e
r

u
t
a before and after an intervention
e
Observations at multiple time points
F
n
g
Compares a group of i
participants
receiving
s an intervention with a
l
similar group from s
the past who did not
e
e
b
D
a of individuals, usually all receiving
Observations
are made on a L
series
y
the same
and after an intervention but with no
dintervention, before
n
control
group
u
ig
St
s
Defined group ofe
(the cohort) is followed over time, to
D people
examine associations
between different interventions received and
youtcomes
subsequent
d
u people with a specific outcome of interest (cases) with
t
Compares
S
people without that outcome (controls), to examine association
between the outcome and prior exposure to an intervention

Collects information on interventions (past or present) and current

outcomes for a group of people at a particular time point, to examine
associations between outcomes and exposure to interventions

Study Design Features

Was there a comparison:

Between two or more groups of participants receiving different interventions?

Within the same group of participants over time?

Were participants allocated to groups by:

Concealed randomization?
Quasi-randomization?
By other action of researchers?
Time differences?
Location differences?
Treatment decisions?
Participants preferences?
On the basis of outcome?
Some other process? (specify)

Which parts of the study were prospective:

Identification of participants?
Assessment of baseline and allocation to intervention?
Assessment of outcomes?
Generation of hypotheses?

On what variables was comparability between groups assessed:

Potential confounders?
Baseline assessment of outcome variables?

Non-Randomized Studies Meeting

Ottawa, Canada

Non-Randomized Studies Meeting

Ottawa, Canada

Non-Randomized Studies Meeting

Ottawa, Canada

Set of checklists for NRS have been

developed based on guidance documents
from workshop

How to Incorporate NonRandomized Studies in

Cochrane Reviews of Patient
Safety

A framework for considering risk of bias

in SR including NRS

Guidance - confounding

Guidance selective outcome reporting

and risk of bias

Guidance selective analysis reporting

and risk of bias