HOW TO DO A SYSTEMATIC

REVIEW

Ferry Efendi, S.Kep., Ns., M.Sc., Ph.D

 TYPES OF RESEARCH DATA

PRIMARY SECONDARY TERTIARY

A typology of reviews: an analysis of 14 review types and
associated methodologies
How is a systematic review different from a traditional narrative review?

Traditional/narrative review Systematic review

Search strategy not explicitly stated
Studies may be “cherry-picked” by author
Prone to selection bias
No standard consensus on minimum items
May describe a broad overview of a topic
Based on author’s subjectivity and expertise
Literature gathered using explicit search
methodology (must be replicable)
Explicit inclusion & exclusion criteria
Minimises selection bias
Standard protocols i.e. PRISMA guidelines
Focuses on a specific, structured question
Based on structured, scientific methodology

Traditional narrative reviews may be at high risk of bias.

Systematic reviews aim to minimise this bias.

Systematic review vs “systematic literature review”

▪ A systematic review is a very specific type of study

— Uses particular methodology which must be explicitly reported/documented
— Follows standardised guidelines (PRISMA, PRISMA-P, etc)
— Included studies usually undergo assessment of quality and risk of bias
— It is NOT simply a literature search that is more systematic than usual

▪ Are you doing an actual systematic review?

▪ Or are you doing a "normal" literature review that you want to be systematic with?

▪ These are two different things

Why do we need systematic reviews?

▪ Managing information overload

▪ Building a comprehensive, trustworthy picture of a topic

▪ Answering what is known and what is unknown

▪ Enable informed decision-making by clinicians, policymakers, and researchers

▪ Minimise the risk of duplicating research efforts and wasting research resources
Case study – post-traumatic psychological interventions

▪ For traumatic events, psychological debriefing was standard practice until 2002

▪ Cochrane Review in 2002 identified nine studies to evaluate the effectiveness of

early single-session psychological debriefing for preventing post-traumatic stress
disorder (PTSD)

▪ Shows how systematic reviews can shine a light on the whole picture to change
standard practice

Rose, S. C., Bisson, J., Churchill, R., & Wessely, S. (2002). Psychological debriefing for preventing
post traumatic stress disorder (PTSD). The Cochrane Library.
http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD000560/full

Rose, S., Bisson, J., & Wessely, S. (2003). A systematic review of single-session psychological
interventions (‘debriefing’) following trauma. Psychotherapy and psychosomatics, 72(4), 176-
184. http://www.karger.com/article/abstract/70781
The hierarchy of evidence

Systematic
reviews,
meta-analyses
RCTs
Quality

Cohort studies

Case-control studies

Case reports
The key stages

Formulate a focused, answerable research question

Define scope and inclusion/exclusion criteria to be used

Systematically search for all relevant research (both published

AND unpublished)

Assess the quality and risk of bias for each study included in
your review

Synthesise included studies via meta-analysis (if possible)

10
What you need to get started

▪ Time!
▪ A team
— Subject experts for developing research question and screening records
— Librarian / information specialist for performing a systematic search
— Statistician / methodologist for undertaking meta-analysis
What you need to get started

WARNING: Slow and deliberate science

▪ Systematic reviews are a hefty investment of time, labour, and effort!

— Average of five authors
— Mean of 67.3 weeks from start to finish (to be published)
— Average yield of relevant items per search: 2.94% (e.g. only 1 relevant study
per 30 search results!)

Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017). Analysis of the time and workers needed to
conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ open,
7(2), e012545.
Before you start – is a review actually needed?

▪ There is a huge amount of duplication and research waste being done

▪ Solution – check PROSPERO before you start your systematic review!
▪ Don’t forget to register YOUR protocol when it’s completed

Moher, D., Booth, A., & Stewart, L. (2014). How to reduce unnecessary duplication: use PROSPERO. BJOG: An International Journal of Obstetrics & Gynaecology, 121(7), 784-786.

Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2012). The nuts and bolts of PROSPERO: an international prospective register of systematic reviews. Systematic Reviews, 1(1), 2.
HANDS ON: searching PROSPERO (3 minutes)

Can you find existing systematic reviews on your topic, or related to your topic?

▪ Search PROSPERO
Tip: Don’t overcomplicate it – just chuck in a few keywords
https://www.crd.york.ac.uk/PROSPERO/
Before you start – develop a plan (protocol): PRISMA-P!
other options: please check JBI, CASP etc
PRISMA-P checklist
http://www.prisma-statement.org/documents/PRISMA-P-checklist.pdf
Before you start – develop a plan (protocol): PRISMA-P!

▪ PRISMA-P elaboration and explanation (this is your step-by-step guide!)

— http://www.bmj.com/content/349/bmj.g7647
20
Formulating your research question

21
Formulating your research question

▪ Quantitative research question? Go with PICO(S)

— C (comparison intervention) and O (outcomes) often omitted from search

strategy, but included for developing inclusion/exclusion criteria

▪ Qualitative? Try SPICE, SPIDER, or others.

Formulating your research question: PICO(S)

Patient / population Patients with heart failure

P
Intervention Telemedicine
I
Comparison Conventional clinical assessment and
C intervention treatment

Outcome(s) Functional outcome, as measured by

O modified Rankin Score (mRS)

Study type Randomised controlled trials

S

How effective and safe is telemedicine for patients with heart failure,
compared to conventional clinical assessment and treatment?
Formulating your research question: SPIDER (qualitative)

S Sample group Pre-school, primary, and secondary students

P Phenomenon of Interest Mindfulness programs

I
D Design Quasi-experimental design (QED)
(specific methodology)

E Evaluation Socio-emotional outcomes

Behavioural outcomes
Academic outcomes
R Research type Mixed methods
(e.g. mixed methods,
quant, or qual)

Do mindfulness programs improve the academic, behavioural, and socio-

emotional functioning of primary and secondary students?
Formulating your research question: SPICE (qualitative)

S Setting (where?) n/a (global – all countries)

P Perspective (for who or what?) Legumes

I Intervention (phenomenon of Elevated CO2

interest)

C Comparison No elevation in CO2

E Evaluation Seed mass

(outcome) Germination
Seed vigour

What is the effect of climate change on the seed quality of legume crops?
Formulating your research question: PICo (qualitative)
HANDS ON: Formulating your research question (3 minutes)

1. Describe your research question of interest in one or two sentences.

2. Break down your research question into PICO/SPIDER format

▪ No hard and fast rules

▪ It’s just a guide to help you conceptually

▪ You can leave some parts blank! (usually the C and O of PICO…)
Selecting sources to search

28
Selecting sources to search – health sciences databases

▪ MEDLINE

▪ Cochrane Library

▪ CINAHL

▪ PsycINFO

▪ EMBASE

▪ SCOPUS
29
Selecting sources to search – databases outside health sciences

▪ Education: ERIC, AEI Informit (Australian)

▪ Sociology: Sociological Abstracts, Anthropology Plus, Informit/FAMILY (Australian), ProQuest

▪ Legal: AGIS Informit (Australian), CINCH, HeinOnline, WestLaw

▪ Biosciences/life sciences: Biological Abstracts (BIOSIS), CAB Abstracts, ScienceDirect

▪ Multidisciplinary: Scopus, Web of Science, ProQuest, ScienceDirect, Informit

▪ Humanities: JSTOR, ProQuest Central, and many others! (ask our Humanities librarians)

30
QUICK TEST: Selecting sources to search

Which scholarly databases would you search for the topic below?

What do we know about the experiences of patients with depression undergoing

cognitive-behavioural therapy?
QUICK TEST: Selecting sources to search

Which scholarly databases would you search for the topic below?

What do we know about the experiences of patients with depression undergoing

cognitive-behavioural therapy?

▪ MEDLINE
▪ PsycINFO
▪ EMBASE
▪ CINAHL
▪ Cochrane Library
▪ Scopus / Web of Science
How many databases do I need to search?

• Broadly speaking, 3-4 minimum

• Depends on your discipline area and topic – how long is a piece of string?
Designing your search strategy

34
Designing your search strategy – logic grid

CONCEPT 1 CONCEPT 2 CONCEPT 3

Depression Cognitive-behavioural therapy Qualitative studies

What do we know about the experiences of patients with

depression undergoing cognitive-behavioural therapy?
How is searching different in systematic reviews?

▪ Highly comprehensive, highly sensitive

▪ Searches both published AND unpublished literature
▪ Uses a combination of subject headings (e.g. MeSH) AND keywords
▪ Considers all synonyms and alternative search terms
▪ Careful translation of search strategy across databases
▪ Accurately documented search strategy
▪ Search must be replicable
Identifying your search terms

It's all about synonyms (and antonyms!) Substance-related disorders

Substance abuse

▪ How might others describe/articulate your

Substance addiction
concepts?
▪ What terminology is used internationally? Substance dependence
(e.g. Australia vs US)
Substance-induced organic mental
▪ Different disciplines, different words disorders
Drug habituation
▪ Has the terminology changed over time?
Drug use disorder

Drug abuse
Identifying your search terms

▪ Consult your fellow subject experts and colleagues

▪ Browse existing relevant articles – scan title, abstract, and subject headings to
extract candidate search terms

▪ Find Cochrane Reviews similar to your topic – examine full detailed search
strategies (documented in Appendix, usually)

▪ Text mining and word frequency analysis – use PubMed ReMiner (Google it!)

40
Identifying your search terms – subject headings

Golden rule: use both keywords AND subject headings

What’s the difference?

▪ Keywords: words and phrases found in titles and abstracts

— Scattergun approach. Highly sensitive. Low precision.

▪ Subject headings: human-curated terms, organised into a hierarchical

thesaurus of categories
— More precision. BUT – not all records have subject headings, and
human curation is imperfect.
43
Developing your search strategy – subject headings
Designing your search – adding subject headings to your logic grid

CONCEPT 1 CONCEPT 2 CONCEPT 3

Depression Cognitive-behavioural Qualitative studies
therapy
exp Depression/ exp Cognitive Therapy/ exp Qualitative Research/
exp Mood Disorders/ exp Behavioral Therapy/ exp Interview/
HANDS ON: using subject headings (4 minutes)

1. Go to MEDLINE or PUBMED
2. Pick ONE concept from your topic. Find TWO subject heading terms to use.

▪ You may not find any subject headings for your concept at all – that’s fine!

▪ Use the scope note to find out more and explore other candidate search terms

▪ Click on the subject heading itself to explore the thesaurus tree

▪ Would you ‘explode’ your subject heading, or not? Why or why not?
Designing your search strategy – combining your search terms

Boolean operator Effect Total results Function

AND Narrows your search Fewer Explore relationship

between different
concepts

OR Broadens your Greater Group similar terms

search together

Warning: you can use NOT to exclude unwanted terms, however – this is an extremely potent tool to be used
very carefully. I would not recommend using it in a systematic review search strategy, as it is often misused
with the effect of accidentally excluding relevant records.

60
Database limits

▪ Limits can help to narrow down your search

— Age
— Gender
— Publication date (e.g. last 10 years)
— Language

▪ Very effective for fast, clinical searches

▪ For systematic reviews, be wary
— Be particularly cautious of limits that tend to “over-reach” such as those for
population age and human studies
▪ Always apply at the very END of your search

http://hlwiki.slais.ubc.ca/index.php/Human_studies_filter
Search filters

▪ Methodological search filters

— Randomised controlled trials (Cochrane Handbook filter)
— Qualitative studies filter
— Quantitative studies filter

▪ Subject-based search filters

— e.g. indigenous Australians, palliative care, etc
Documenting your search strategy

▪ Use PRISMA flow diagram as a guide

▪ Methods section (narrative description)
— Describe all sources/databases you’ve searched
— Describe other methods you’ve deployed (e.g. grey literature searches,
searching clinical trial registries, etc)
▪ Results section
— Display your PRISMA flow diagram
▪ Document your full search strategy in detailed syntax
— Put this in the Appendix
— Cover at least one database search strategy in full
Documenting your search strategy

Vaniyapong T, Chongruksut W, Rerkasem K. Local versus general anaesthesia for carotid endarterectomy. Cochrane Database of Systematic Reviews 2013, Issue 12. Art.
No.: CD000126. DOI: 10.1002/14651858.CD000126.pub4.
Documenting your search strategy

http://prisma-statement.org/PRISMAStatement/FlowDiagram.aspx
Exporting and organising references

▪ Organise references into folders based on which databases retrieved from

▪ Use "Find Duplicates" function in EndNote to eliminate duplicate records

▪ Document the number of records at each stage for your PRISMA Flowchart
Accessing full-text articles for screening

For records allocated to be included (or unsure), obtain full-text

1. EndNote can automate retrieval of many/most full-text PDFs

— See the library’s EndNote guide for details (or ask a librarian!)
2. Use Library Search
Software for systematic reviews & meta-analysis
Common mistakes to avoid

▪ Don't underestimate time and effort required

— Potentially more than a year
— Plan systematically (that’s what your protocol is for!)
— Avoid “shortcuts”

▪ Don't go it alone
— MINIMUM of one other individual to cross-check your search strategy and
screen records
Further resources
Further resources

Free online course:

https://www.coursera.org/learn/systematic-review
THE LANCET OF PUBLIC HEALTH
https://journals.sagepub.com/doi/10.1177/0969733015602052
https://onlinelibrary.wiley.com/doi/abs/10.1111/nuf.12243
https://onlinelibrary.wiley.com/doi/abs/10.1111/nhs.12629
 REFERENCES

1. Bonner, A., Wellard, S., Caltabiano, M. (2008). Levels of fatigue in people with ESRD living in far North
Queensland. Journal of Clinical Nursing, 17, 90–98.
2. Martens, T., & Emed, J. (2007). The experiences and challenges of pregnant women coping with
thrombophilia. Journal of Obstetric, Gynecologic, & Neonatal Nursing, 36, 55–62.
3. Roe, B., Ostaszkiewicz, J., Milne, J., & Wallace, S. (2007). Systematic review of bladder training and
voiding programmes in adults. Journal of Advanced Nursing, 57, 15–31.
4. Shreffler-Grant, J., Hill, W., Weinert, C., Nichols, E., & Ide, B. (2007). Complementary therapy and
older rural women: Who uses it and who does not? Nursing Research, 56, 28–33.
5. Zhang, A., Strauss, G., & Siminoff, L. (2007). Effects of combined pelvic muscle exercise and a support
group on urinary incontinence and quality of life of postprostatectomy patients. Oncology Nursing
Forum, 34, 47–53.
6. Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated
methodologies. Health Information & Libraries Journal, 26(2), 91-108.
7. Polit, D. F., & Beck, C. T. (2013). Study guide for essentials of nursing research: appraising evidence for
nursing practice. Lippincott Williams & Wilkins.
8. Systematic Searching for Systematic Reviews: an introduction. 2018. La Trobe University Library
THANK YOU