How to read

and critique an
article

Prof. Aida Mohey Mohamed

Community Medicine department
Alexandria University
2020
Critical appraisal
A systematic process used to identify the strengths and weaknesses
of a research article in order to assess the usefulness (clinical impact),
applicability and validity of research findings.

The most important components of a critical appraisal are:

▪ The appropriateness of the study design for the research question
▪ Careful assessment of the key methodological features of this
design.
APPROACH TO AN ARTICLE
▪It is important to approach an article in the same
systematic and organized manner.
▪It can become very scary, especially if someone does
not have a background in the research field.
▪Therefore, one should approach each article in the
same manner and keep in mind certain crucial points
that are significant in understanding and analyzing any
article.
What is the best epidemiological design to study the efficacy of an
intervention?
A. Cross-sectional study D
B. Case-control study
C. Prospective cohort study
D.Randomized controlled trial
Which is the best epidemiological design to study the accuracy of a
diagnostic test?
A. Cross-sectional study
B. Case-control study D
C. Cohort study
D.Cross sectional comparison with gold standard test
I. Introduction
• In order to dissect a journal article, one should divide it into three
main sections: the introduction, discussion, and conclusion.

• The introduction should further encompass four main elements:

▪ Article identifiers
▪ study type/PICO(T)
▪ A review of the medical literature
▪ Methods (study design).
In critical appraisal of a journal article, which of the
following is NOT an element of the introduction
section?

A. Article identifiers
B. PICO question
C. Results
D. A review of the medical literature
E. Methods (study design)
C
1. Article Identifiers
•The main article identifiers are:
•The article name (Title) [what, who, where]
•Author
•Title of publication (Journal name, year,
volume, issue, page numbers)
•This serves as an introduction to the talk that is
crucial to the presentation as a whole.
In critical appraisal of a research article, the main
article identifiers are all of the following EXCEPT:

A. The article title

B. The name and affiliation of the author(s)
C. The year of publication
D. The objective of the study
D
2. Study type
After introducing the article, it is important to state the type of study because this
sets the stage for determining the quality, reliability, and validity of the study as it
applies to the clinical domain.
Participating in a clinical study (intervention or observational) contributes to medical
knowledge. The results of these studies can make a difference in the care of future
patients by providing information about the benefits and risks of therapeutic,
preventative, or diagnostic products or interventions.
This information is important because it determines:
▪ The applicability of the study and accuracy in determining whether its results
should be taken into consideration in real practice
▪ Whether its recommendations set the stage for future development of practice
guidelines
To recognize the type of study, one should ask
three questions
1. Aim of the study
a. To simply describe a population → descriptive
b. To quantify the relationship between factors (PICO applicable) → analytic
2. If intervention analytic:
a. Randomized → randomized controlled trial (RCT)
b. Non-randomized → Quazi-experimental
3. If observational analytic, when the outcomes were
determined?
a. Some time after the exposure → prospective cohort
b. At the same time of the exposure → cross-sectional
c. Before the exposure was determined → case-control study
PICO(T) applicable studies are:
A. Descriptive studies
B. Intervention analytic studies
C. Observational analytic studies
D. B and C
E. All types of studies (analytic and non analytic)

D
PICO(T)
• The adequate (well-constructed) research question allows for the
correct definition of which information (evidence) is needed to solve
the clinical research question.
• In approaching an article, especially in the introduction, one should
always keep in mind the PICO(T) algorithm.
• The PICO strategy can be used to construct several kinds of research
questions, originated from clinical practice.
• This includes identifying the
(P) patients or population studied coupled with
(I) the intervention performed,
(C) the comparison used,
(O) the studied outcomes, and
(T) the time in which the study was conducted.
PICO(T)
• Within this paradigm, you would introduce the objective and aims of
the study.
• This method is easily applied to experimental studies; however, this
may be a bit challenging in certain studies. The PICO(T) method is
best used in analytic studies. This includes experimental analytic
(RCT or controlled trials) and observational analytic studies (cohorts,
cross-sectional, case control).
• It would therefore be difficult to apply this method, specifically the
intervention aspect of it to non-analytic studies (systematic
reviews/meta-analysis, cross-sectional surveys). Therefore, it would
only be possible to apply the P and O (patient or population) and
outcome.
In construction of a clinical question (PICO method)
it would only be possible to apply the P (patient)
and O (outcome) in which of the following study
design?
A. Randomized controlled trials
B. Cohorts studies
C. Case control studies
D. Systematic reviews/meta-analysis

D
Observational analytic
Study type Description
Cohort Cohorts are a group of
participants with a similar
characteristic that are followed
through time (either
retrospectively or prospectively)
to determine which risk
factors are associated with that
particular outcome.
Observational analytic
Study type Description
Case control A study that compares patients who
have a disease or outcome of
interest (cases) with patients who do
not have the disease or outcome
(control participants) and looks back
retrospectively to compare how
frequently the exposure to a risk
factor is present in each group to
determine the relationship between
the risk factor and the disease.
Observational descriptive
Study type Description
Cross sectional Observation of all of a population,
or a representative sample, at one
specific point in time.
A cross-sectional study is one in
which subjects are sampled
without respect to disease status
and are studied at a particular
point in time, as in a random-
sample health survey.
 Observational descriptive
Study type Description

Case series A report about a small group of

similar cases. Typically, they are the
clinical route to case definition and
recognition of disease entities and
to the formulation of hypotheses.
Experimental
Study type Description

Randomized controlled trials (RCTs) RCTs are studies that measure an

intervention’s effect by randomly
assigning individuals (or groups of
individuals) to an intervention
group or a control group.
This is the strongest study to
determine a cause–effect relation
that would potentially exist
between treatment and outcome.
Secondary research
Study type Description

Meta-analysis A subset of systematic reviews; a method

for systematically combining pertinent
qualitative and quantitative study data from
several selected studies to develop a single
conclusion that has greater statistical
power.
This conclusion is statistically stronger than
the analysis of any single study because of
▪ Increased numbers of subjects
▪ Greater diversity among subjects
▪ Accumulated effects and results
D–C–F–C-E
3. Review of the Medical Literature

• As part of the introduction, it is important to review the current medical

literature regarding a particular topic.
• This is important because:
• It sets the stage for the relevance and applicability of that study
• More importantly, it indicates to the reason why that research was
chosen.
• It is important to understand
• The available current knowledge
• Cross reference it with the current gaps in knowledge or research
• Tie the gap into the current study
• What it adds to the body of literature
4. Methods
• The most important part of the Methods section
is discussing the inclusion and exclusion criteria
that the authors used in their study.
• This, coupled with the actual methodology of
the study, can determine how valid and reliable
results are.
• For results to be accurate and sound, they
should arise from a non-biased, representative
sample that would in turn validate the positive
or negative results of a study.
• In this section, one can mention the study
design and how it was implemented.
II.
• The discussion part can be divided into two parts:
• Discussing the results of the study
• Assessing the statistical methods.
• With respect to the statistical methods, it is important to focus on
one main statistical method that the authors used and use it to
convey whether they met their objectives.
• When presenting the results of a study, it is impossible to present
every table, chart, and figure. Two to three figures should be chosen
to represent the majority of the results and must be explained
thoroughly with attention to the clinically significant results.
Remember

▪A negative study is not a bad study.

▪If a study fails to detect a relationship, then that
does not mean that the study is useless. Rather, it
presents further knowledge on the topic and rules
out certain factors that may not be associated with
it.
▪ Evidence based practice is supposed to affect clinical decision
making, but interpreting research is often difficult.
▪ Clinical interpretation is important because it includes patient
safety and efficacy.
▪ Statistically significant differences between data sets may not
always result in an appropriate change in clinical practice.
▪ Clinical research is only of value if it is properly interpreted.
▪ Make inferences from data collected using laws of probability
and statistics:
▪ Tests of significance (p-value)……statistical sig.
▪ Effect size and confidence intervals…..clinical sig.
Cohen quantified ES that have been operationally described in ranges:
<0.2 = trivial effect ; 0.2-0.5= small effect ; 0.5-0.8= large effect
Which of the following statements is INCORRECT as
regards the clinical impact of an evidence?

A. Clinical research is only of value if it is properly interpreted

as having clinical impact
B. Clinical interpretation of a research is important because
it includes patient safety and efficacy.
C. Statistically significant differences between data sets may
always result in an appropriate change in clinical
practice.
D. Clinical significance is detected by the effect size and
confidence intervals. C
 Discussion
It is important to first look at sample or population chosen.
1. Does it represent an unbiased sample (representative)
from which conclusions can be accurately and reliably
be drawn?
2. Is the sample or population comparable at baseline?
3. Is the size of the sample large enough to represent
enough cases that could provide conclusive inferences
about a specific disease?
4. In assessing sample size, one should also look to the
power of the study.
5. What is the effect size (observed association or
difference)?
The power of a study
• Statistically: The probability that the statistical test
will reject the null hypothesis when the null
hypothesis is false (i.e., the probability of not
committing a type II error, or making a false negative
decision).
• In RCT: It is the probability of the test reporting a
statistically significant effect for a real effect of a
given magnitude.
▪ The power analysis determines the appropriate
sample size of subjects needed in a study to detect
statistically significant difference with an appropriate
effect size.
▪ A small sample size limits statistical power, while
larger sample sizes provide more power to detect
statistically significant differences.
 16
P-values VERSUS Confidence intervals

▪ P-value answers the question...

▪ “Is there a statistically significant difference between the two
treatments?“

▪ The point estimate (effect size) and its confidence interval

answers the question...(quantify the effect size)
▪ "What is the size of that treatment difference?", and
▪ "How precisely did the trial determine or estimate the
treatment difference?"
 Clinical relevance/significance/impact 1

▪ CI are reported with a “point estimate” (PE) .

▪ The PE is a specific value (which maybe a sample mean, sample proportion, mean
difference, effect size (relative risk, odds ratio, NNT) but does NOT represent a “true”
value, rather, it represents the “best estimate” of the true value from the average of the
sample and should be viewed in consideration of the range of CI (degree of certainty)

▪ PE far from 0 represent more effect or difference, either positive or negative

▪ PE closer to 0, the less group difference or effect
▪ If the CI does not contain a value of 0, the results are significant.
▪ With larger ES and small CIs that do not cross 0 have the most clinical significance.
What about clinical importance (usefulness)?

▪ Clinical importance is a medical judgment, not statistical

▪ Clinicians should change practice only if they believe

▪ the study has definitively demonstrated a treatment difference
▪ that the treatment difference is large enough to be clinically
important.

▪ Depends on knowledge of:

▪ A range of possible treatments
▪ Their costs
▪ Their side effects
Example: The following finding of non-
significance in a clinical trial on 178 patients

Treatment Success Failure Total

A 76 (75%) 25 101

B 51 (66%) 26 77

Total 127 51 178

Chi-square value = 1.74 25
(p>0.1) (non–significant)
i.e. there is no statistical significant difference in efficacy between the
two treatments.

--- The observed difference is: 75% - 66% = 9%....PE

and the 95% CI for the difference is: (- 4% to 22%)

-- This indicates that compared to treatment B, treatment A has, at

best an appreciable advantage (22%) and at worst , a slight
disadvantage (- 4%).

--- This inference is more informative than just saying that the
difference is non-significant.
Reaction of investigator to results of a statistical significance test

Statistical significance

Not significant Significant

Not Happy
Practical Annoyed
important
importance of
observed effect
Important Very sad Elated
III.
▪ Assesses the relevance of the article as to whether it is
clinically significant & clinical importance even though it
may be statistically significant or not.
▪ It allows participants to analyze whether it is a feasible study
and if it can be applied in clinical practice.
▪ If so, what would the economic burden associated with it
be? There may be numerous studies that show that a
particular diagnostic modality is highly sensitive and specific
in detecting a disease, but they may be too expensive and
cumbersome to be applied as a screening tool.
▪The relevance and statistical significance of the study
can also be measured by the grade and level of
evidence that the study represents.

▪RCTs are typically the highest level of evidence, and

data derived from them tend to represent grade A
recommendations given that they are the most
unbiased, controlled, blinded studies that typically
confer a large sample size and contain data over a
long period of time.
▪ Internal validity: Did the authors rule out chance, bias, and confounding factors?
▪ Was there an external funding source? If so, what was its contribution?
▪ Not every therapeutic trial that is funded by a pharmaceutical company is deemed
“bias.” It is important to assess the funding source’s contribution. If it merely
supplied the finance but a separate group of investigators conducted and analyzed
the study, then the results of the study can represent valid data.
▪ The External Validity of the study can also be assessed by asking whether the study
is feasible:
▪ Change your practice?
▪ Add knowledge about the subject?
▪A good strategy to adopt when analyzing a study is to
come up with three strengths and three weaknesses
of the study.
▪The strengths of the study can be apparent by
referring to the overall effect and statistical or clinical
significance.
▪When assessing the weaknesses (limitations) of a
study, one can refer to specific weaknesses inherent
to the study design or to author-specific weakness
that may lie within the funding source, population
sampling, biases, non-blinding, non-randomization,
and so on.
Response bias =
tendencies for
participants to
respond
inaccurately or
falsely to
questions

(Social
desirability bias)
• Finally, you should discuss either certain questions
about the study such as whether the authors could
have used different methodological designs or
different population samples that would have
brought on more solid statistically and clinically
significant results.
• Additionally, points for future research that were not
addressed in the study.
HOW TO CRITICALLY
APPRAISE AN
ARTICLE
Critical appraisal is a systematic process used to identify the strengths
and weaknesses of a research article in order to assess the usefulness
applicability and validity of research findings.
The following questions were taken from an
article by Young et al. on how to appraise an
article and are useful to keep in mind when
reading a scientific article.

Reference: Young JM, Solomon MJ. How to Critically Appraise an Article

DISCLOSURES. Nat Clin Pract Gastroenterol Hepatol. 2009;6(2):82-91.
Ten Questions to Ask When Critically Appraising
a Research Article
1. What is the 2. Does the study 3. What type of
relevance of the
add anything research question
research question to
one's own practice?
new? is being asked?

5. Did the study 6. Was the study

4. Was the study methods address
design appropriate the most important
performed
for the research potential sources according to the
question? original protocol?
of bias?

7. Does the study 8. Were the 9. Do the data 10. Are there any
statistical analyses
test a stated performed
justify the potential conflicts
hypothesis? correctly? conclusions? of interest?
Which one of the following questions is MOST
important when critically appraising a research
article?
A. What type of research question is being asked?
B. What is the relevance of the research question to one's
own practice?
C. Was the study performed according to the original
protocol?
D. Does the study test a stated hypothesis?
E. Do the data justify the conclusions? B
The following lists serve as checklists
for critically appraising an article
based on the specific type of article.
Systemic Reviews and Meta-Analyses
1. Were all relevant studies included (i.e., was the search
comprehensive, did it exclude articles on the basis of
publication status or language, and was the potential
for publication bias assessed)?
2. Were selected articles appraised and data extracted by
two independent reviewers?
3. Was sufficient detail provided about the primary
studies, including descriptions of the patients,
interventions, and outcomes?
4. Was the quality of the primary studies assessed?
5. Did the researchers assess the appropriateness of
combining results to calculate a summary measure?
Preferred Reporting
Items for
Systematic Reviews
and Meta-Analyses
(PRISMA) flow
diagram of search
strategy and
selection of
articles, systematic
review and meta-
analysis of global
typhoid incidence,
1946-2018.
In critical appraisal of the systematic reviews and
meta analysis, all of the following items should be
addressed EXCEPT:
A. All relevant studies should be included
B. The selected articles should be appraised and data extracted by one
reviewer
C. Detailed description should be provided about the primary studies
D. The quality of the primary studies should be assessed
E. The appropriateness of combining results to calculate a summary
measure should be assessed

B
Randomized Controlled Trials
1. Was the process of treatment allocation truly random (Randomization)?
2. Would participants have been able to know or guess their treatment
allocation (allocation concealment)?
3. Were participants and researchers “blinded” to participants’ treatment
group?
4. Were outcomes assessed objectively?
5. Were participants followed up for a sufficient length of time?
6. Were all participants who were randomly allocated a treatment
accounted for in the final analysis (% drop-outs)?
7. Were all participants’ data analyzed in the group to which they were
randomly allocated (Intention to treat analysis)?
CONSORT…Consol
idated standards
of reporting trials
diagram detailing
patient flow-
through within the
RCT.
Cohort Study
1. Is the study prospective or retrospective?
2. Is the cohort representative of a defined group or
population?
3. Were all important confounding factors identified?
4. Were all important exposures and/or treatments,
potential confounding factors and outcomes measured
accurately and objectively in all members of the cohort?
5. Were there important losses to follow-up?
6. Were participants followed up for a sufficient length of
time?
Case control study
• Were the cases clearly defined (standard case definition)?
• Were the cases representative of a defined population?
• How were the controls selected and were they drawn from
the same population as the cases?
• Were study measures (exposure) identical for cases and
controls?
• Were study measures objective or subjective and is recall
bias likely if they were subjective?
Cross sectional study
• Was the study sample clearly defined?
• Was a representative sample achieved (e.g. was the
response rate sufficiently high)?
• Were all relevant exposures, potential confounding factors
and outcomes measured accurately?
• Were patients with a wide range of severity of disease
assessed?
Case series study
• Were cases identified prospectively or retrospectively?
• Are the cases a representative sample (e.g. a consecutive series of
individuals recruited from multiple centers) and similar to patients in
your practice?
• Were all relevant exposures, potential confounding factors and
outcomes measured accurately?
Study of diagnostic accuracy
• Does the sample of patients represent the full spectrum of patients
with and without the diagnosis of interest?
• Was there a comparison with an appropriate ‘gold-standard’ test?
• Did all patients receive both the test under evaluation and the same
‘gold-standard’ test?
• Were the tests performed independently with blinding of assessors
to the results of the ‘gold-standard’ test?
• Were the cut-offs that were used to classify patients as having a
positive test result clearly described?
Key points
▪ Critical appraisal is a systematic process used to identify the strengths
and weaknesses of a research article
▪ Critical appraisal provides a basis for decisions on whether to use the
results of a study in clinical practice
▪ Different study designs are prone to various sources of systematic bias
▪ Design-specific, critical-appraisal checklists are useful tools to help
assess study quality
▪ Other factors are an important part of the critical appraisal process.
these include:
▪ the importance of the research question
▪ the appropriateness of statistical analysis
▪ the validity of conclusions
▪ potential conflicts of interest
References
1. Ely JW, Osheroff JA, Chambliss ML, Ebell MH, Rosenbaum ME. Answering physicians’ clinical
questions: obstacles and potential solutions. J Am Med Inform Assoc. 2005;12(2):217-224.
2. Gorman PN, Helfand M. Information seeking in primary care: how physicians choose which
clinical questions to pursue and which to leave unanswered. Med Decis Making. 1995;15(2):113.
3. Chambliss ML, Conley J. Answering clinical questions. J Fam Pract. 1996;43(2):140-144.
4. Currie LM, Graham M, Allen M, Bakken S, Patel V, Cimino JJ. Clinical information needs in context:
an observational study of clinicians while using a clinical information system. AMIA Annu Symp Proc.
2003:190-194.
5. Young JM, Solomon MJ. How to critically appraise an article. Nat Clin Pract Gastroenterol Hepatol.
2009;6(2):81-92.
6. Wilton NK, Slim AM. Application of the principles of evidence-based medicine to patient care.
South Med J. 2012;105(3):136-143.
7. Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Evidence-Based Medicine: How to Practice
and Teach EBM. New York: Churchill Livingston; 1997.
Thank You