How to critically read a scientific report

Eva King: Learning Adviser

Email – e.king3@uq.edu.au
 Scientific reports
▪ A written account of a scientific line of inquiry (a study)

▪ Follows a set format: IMRaD

▪ Disseminated to the broader scholarly community via journals,

conferences, text books

▪ ‘Peer review’ process ensures quality control

CRICOS code 00025B
 So why the need to read critically?
▪ ‘Peer review’ process ensures quality control….or does it?
▪ Conflicts of interest (funding?)
▪ Pay-to-publish
▪ Journal ranking system
▪ Peer reviewer issues

▪ To put scientific reports to work for you

▪ Discern opportunities for further research
▪ Take part in the broader scholarly conversation – enter the debate!

▪ To learn how to conduct and communicate high quality science

CRICOS code 00025B
 Today’s goals
Understand the components of a scientific report

Understand the concept of evaluative criteria for scientific reporting

Know what questions to ask of each component, as you read

Develop some reading strategies

CRICOS code 00025B

Components of a scientific report

CRICOS code 00025B

 Scientific report structure
▪ Title & Abstract
▪ Introduction
Also
▪ Methods known
▪ Results or Findings as
▪ Discussion/conclusion ‘IMRaD’
▪ References
▪ +/-Appendices
CRICOS code 00025B
The concept of evaluative criteria for
scientific reporting

CRICOS code 00025B

 Critiquing scientific reports
▪ In most cases, reading critically is not finding faults, but identifying both
strengths and limitations

▪ As a reader, you are asking “Is this a good study?”

▪ To answer this, you need to know what constitutes a ‘good’ study

▪ The definition of a ‘good’ study is kind of like the definition of a ‘good’

car……..
CRICOS code 00025B
 It depends on the end user’s needs!

(a) (b) (c)

(a)

(d) (e) (f)

Along with other factors like:

• Ease of manufacture & repair
• Whole of life considerations
• Environmental regulations…etc
CRICOS code 00025B
 Observations Defining a ‘good’ study

Results Scientific Formulation

A good study meets the
=new
observations Knowledge
of a model or
hypothesis overarching objective of
advancing scientific
knowledge

Testing of the
model or
hypothesis

CRICOS code 00025B

There are many, many, many, many
study designs to choose from!
CRICOS code 00025B
 For example….. Observations

▪ Observational studies: (what is going on here?)

▪ Quantitative; descriptive statistical analysis of survey data Results =new
observations
Scientific
Knowledge
Formulation of
a model or
hypothesis

▪ Qualitative: phenomenological or ethnographic study

▪ Mixed methods: focus group analysis survey instrument design survey Testing of the
model or
hypothesis

▪ Modelling/theorising studies (what is the relationship between variables

here?)
▪ quantitative; inferential statistical analyses e.g. causal modelling, regression analysis
▪ qualitative; grounded theory

▪ Testing studies (does our model/theory/hypothesis hold true?)

▪ Quantitative: experimental studies in a laboratory setting
▪ Mixed methods: translational studies of the implementation of research findings in a real world
setting (e.g. healthcare, education) CRICOS code 00025B
 Evaluative criteria for studies
▪ Many, many study designs
▪ Therefore many, many evaluative criteria
▪ Study design options are constantly evolving
▪ Therefore, evaluative criteria are also evolving!
▪ There is no ‘one size fits all’…….

▪ BUT scholars in all disciplines contribute to devising schema to

assist readers to critically evaluate scientific reports
CRICOS code 00025B
 e.g. The ‘QUESTS’ dimensions
▪ Quality of the research evidence available (how
reliable is the evidence?)
▪ Utility of the evidence (can the methods be
transferred and adopted without modification?) Usually the
evidence is good
▪ Extent of the evidence in some respects,
but poor in others.
▪ Strength of the evidence
▪ Target or outcomes measured (how valid is the
evidence?)
▪ Setting or context (how relevant is the evidence?)
Adapted from: Harden, M., Grant, J., Buckley, G., Hart, I. "Beme Guide No. 1: Best Evidence Medical Education." Medical Teacher 21, no. 6 (1999):
553-62. CRICOS code 00025B
 Evaluative criteria: key points
▪ You cannot transfer the set of criteria for one type of study to a different type
of study

▪ It is a good idea to familiarise yourself with the evaluative criteria for common
types of studies in your discipline

▪ These can often be found in the leading journals in your discipline or in

relevant texts

▪ Reading lots of research reports in top journals will help you develop an
understanding of evaluative criteria and how they apply
CRICOS code 00025B
Questions to ask of each component, as
you read

CRICOS code 00025B

 Introduction
Does the author justify the study? This helps you to
establish relevance
▪ What is the problem that needs to be addressed? and potential
usefulness of the
▪ Why hasn’t this problem been addressed? (gap) study

▪ What aspect of this problem will be addressed in this study? (aim)

▪ How will this aim be fulfilled? (approach)
▪ Who will benefit from the findings?
▪ If the study hadn’t gone ahead, why would that be bad?

CRICOS code 00025B

 Methods
▪ Does the research question(s) or hypothesis align with the aim?

▪ Is the methodological approach (study design) appropriate?

▪ Are the methods (data sampling, collection and analysis

techniques) appropriate? This helps you to
evaluate the quality of
the study’s findings
▪ Was the study conducted rigorously?

▪ What are the strengths and limitations of the research process?

CRICOS code 00025B

Is the methodological approach appropriate?

Physical Natural Social

sciences sciences sciences
More easily measured Harder to measure, many variables, context-bound

▪ A great study design for one context may simply not be viable in another!
▪ What this means is……you need to know what approaches are relevant to
your discipline
▪ Read top journals, follow editorials, read methodological lit, coursework
students texts/reading lists
CRICOS code 00025B
Methods: Data sampling

CRICOS code 00025B

 Consider sampling strategy
▪ Random sample?
▪ Systematic sample?
▪ Representative sample?
▪ Convenience sample?
▪ Purposive sample?
▪ Snowball sample?
▪ Maximal variation sample?
▪ Theoretical sample?
Sampling strategy
depends on study
design
Authors should state
and justify their
sampling strategy
Failure to do this = a
limitation of the study
CRICOS code 00025B
For example, in a quantitative, observation or modelling study….

▪ Consider population to be sampled:

▪ What’s true for one group may not be true for another (limits of applicability
/ generalisability)
▪ Watch out for sampling bias (e.g. more people who have strong feelings
about a topic of study may participate in a survey and so not really be
representative of a population)
▪ Consider sample size:
▪ With very small samples, a seemingly strong pattern may in fact be a
statistical fluke (it might not be, but you can’t be sure without more data).
▪ With very large samples, even the tiniest of effects can be “statistically
significant”, but may not be of any practical significance.
CRICOS code 00025B
 Yet for a qualitative observation or modelling study:

▪ The intention may be to capture maximal variability

▪ The population under study may be difficult to access
▪ Sampling bias is recognised through researcher reflexivity
▪ A small, purposive sample that is studied in depth may yield more
meaningful findings than a large, superficially studied sample

The evaluative criteria for

sampling strategy would look
very different in this case!

CRICOS code 00025B

 Data collection
What and how data is collected has a profound impact on the
results
▪ focus group vs individual interview data
▪ Observation of participants in lab vs natural settings
▪ How phenomena are measured
▪ Observations vs participant’s perceptions
Is the data likely to yield
▪ Survey instrument design – likert scales meaningful information
▪ textual analysis – purpose of the document? for the line of inquiry?
▪ etc
CRICOS code 00025B
 Data analysis
▪ Once the data were collected, how were they analysed?
▪ Statistical analyses using SPSS
▪ Coding, thematic analysis, conceptual ordering, constant comparative
analysis etc in qualitative studies
▪ ‘plug and play’ using a theoretical framework (theory testing)
Strong studies
explain how the data
▪ Note: level of interpretation (inference) varies were analysed

▪ Readers’ capacity to critique data analysis is aided by

understanding the philosophical assumptions underpinning
the study

CRICOS code 00025B

 Results
▪ Is it clear how the results were arrived at? (link to data analysis)
▪ Are the results reliable, valid, objective and generalisable? These terms
have specific
(quant)
meanings in
▪ Are the results trustworthy, credible, reflexive and transferable? these
contexts
(qual)
▪ Are the results well presented?
▪ Do the results address the research question?
▪ To what extent?
CRICOS code 00025B
To interpret quantitative results, you need to be able to comprehend
what the reported numbers are telling you.

“The cognitive impairment of those with slow gait was apparent across multiple neuropsychological
domains: they had … poorer perceptual reasoning (r = 0.29; 95%CI: 0.22 to 0.35; p < .001), …”

Our level of
A measure of the confidence that the
strength (weak) of An estimate of
correlation is not just
the relationship the uncertainty
a statistical fluke.
between in r. The “real”
answer is likely in In this case we can be
perceptual
this interval. pretty confident that
reasoning and gait
there really is a
speed.
correlation between
the variables.
CRICOS code 00025B
 To interpret qualitative results, you need to be able to see the
links between the data and the analysis
▪ Excerpts of raw data e.g. direct
quotes from participants

▪ Explanation and illustration of

coding strategies

▪ Representation of themes and

conceptual ordering

▪ Models and diagrams

CRICOS code 00025B

 Discussion
▪ Does the author interpret the results appropriately (what do they
mean?)
▪ How are the results related to other studies?
▪ How do the results help solve the problem stated in the intro?
▪ How might the results be used?
▪ What problems are there with the results? (strengths and
limitations)
CRICOS code 00025B
Statistical significance does not necessarily imply practical significance
Statistics
𝑝 < 0.05:
Difference is “statistically significant”
𝑝 > 0.05:
Difference is “not statistically significant”
“Highly statistically significant” means we are
“very confident” there really is a difference;
the effect size tells us how big that difference
is.
Legal analogy
Mr X is guilty “beyond reasonable doubt.”
[Though note that convictions can be wrong!]
Mr X might be innocent, or he might be guilty,
but if he is guilty, there is just not enough
evidence to gain a conviction.
While we might be very confident a crime
was committed, the crime could be small or
large.
E.g. we have clear video evidence with a speed
camera that Mr X was speeding in a 40 km/h
school zone:
But was he doing 50km/h or 100 km/h?
Both absolute and relative differences are important to consider.
“The Pen Is Mightier Than the Keyboard: Advantages of Longhand Over Laptop Note Taking”
DOI: 10.1177/0956797614524581

But is it really?
“On factual-recall questions,
participants performed equally well
across conditions… However, on
conceptual-application questions,
laptop participants performed
significantly worse [8.7%] than
longhand participants, ….”

8.7% is important, it
would drop you from a 7
to a 6 or from a 6 to a 5.
But this was the relative
difference, what was the
absolute difference?
CRICOS code 00025B
The absolute difference was
only 0.52 marks out of 6!

One cannot extrapolate from

0.52/6 to 8.7/100!!

CRICOS code 00025B

Three things to consider when looking at (or providing) a set of inferential statistics.

1. How confident can we be that there really was an effect / is a

relationship? I.e. that the observed difference or correlation
was not just a random fluctuation? Check the p-value for
“statistical significance”.
2. If we are confident that there was an effect / is a relationship,
how strong or important is the effect / relationship? Check for
measures of effect size and analyses of practical / clinical
importance.
3. Finally, how confident are we in our estimate of the size of the
effect? Check for confidence intervals. If CIs are not provided,
keep in mind that the smaller the sample, the higher the
uncertainty (in general).

You should also ask questions like: Was the methodology

sound? Are there alternative explanations? Were confounds
well controlled for?
CRICOS code 00025B
 Example
 Gender bias in the marking of school maths
tests?
 NYT: “In math, the girls outscored the boys in the exam
graded anonymously, but the boys outscored the girls
By itself, a low p-value only tells
when graded by teachers who knew their names. ... The you that you can be confident
researchers concluded that in math and science, the there was an effect, not how
teachers overestimated the boys’ abilities and important that effect is.
underestimated the girls’, and that this had long-term
effects on students’ attitudes toward the subjects.”
 Critical thinking: Okay, the bias was “statistically
significant”, but how big was it?
 About a 1 mark out of 100 difference!

http://www.nytimes.com/2015/02/07/upshot/how-elementary-school-teachers-biases-can-
discourage-girls-from-math-and-science.html?_r=1&abt=0002&abg=1
CRICOS code 00025B
 Example
Suppose you are a Southern Flounder fish farmer and you come across a study that shows if you
heat your pond water to 23 ℃, your fish will grow much larger, and the result is highly statistically
significant.
Should you be excited? After all, you are paid per kg, not per fish,
so heavier fish will bring in more income.

Well, if the increase in your electricity bill from heating the pond
water is greater than your increased income from heavier fish,
then you are economically worse off!

Statistical significance does not necessarily imply practical/clinical/economic

importance. CRICOS code 00025B
 Making sense of qualitative findings
▪ The interpretation of qualitative data may seem highly subjective
▪ This is not relevant if the study is undertaken from a relativist
paradigm
▪ If the study is undertaken from a post-positivist or realist paradigm,
then researchers should take steps to strengthen their findings
▪ Triangulation of methods and/or analysis
▪ ‘Member checking’
▪ Using a theoretical framework to analyse the data
▪ Consensus between researchers/peer audit
CRICOS code 00025B
 Understanding strengths and limitations
▪ Most journals require that authors state their study’s strengths and
limitations

▪ Word limit constraints and other considerations mean that all of a

study’s strengths and limitations may not be articulated

▪ Therefore you should still make up your own mind!

▪ This gets back to understanding methodology and methodological

congruence
CRICOS code 00025B
 Example
Our study has several important strengths. It is the first to ... Why this paper
should be
and we have been able to adjust for a range of ...
considered to be
confounding and mediating factors. In addition, we have “strong” / “better”
used ... than previous work
The major limitation of this study was substantial non-
participation at 21 year follow-up. Our results would be
biased if ... However, as most studies find that ... this is What the
unlikely. In order to assess further whether those lost to methodological
follow-up produced bias in our results, we ...We found no limitations are of the
difference ... which suggests that attrition is unlikely to have research and why the
biased our findings substantively. However, given the results are still
profile of respondents and non-respondents, those who not trustworthy enough to
followed-up are likely to be at greater risk of ... publish?

Source: Mamun et al. 2007: Addiction, Vol. 102, Iss. 4, Pages: 647–
654 CRICOS code 00025B 42
Develop some reading strategies

CRICOS code 00025B

 It all begins with a sense of purpose…
Why are you reading this report?

▪ To keep up to date?

▪ To helps justify your research proposal?

▪ To introduce or frame your study?

▪ To justify & inform your methodological approach?

▪ To help you make sense of the findings of your study?

CRICOS code 00025B

 Critically reading a scientific report
▪ Requires active rather than passive reading.
▪ Means having a questioning mindset.
▪ Your capacity to should grow with time as you learn more
▪ From related papers
▪ About the strengths and weaknesses of the methodological approaches in
your discipline
▪ About inferential and descriptive statistics
▪ About qualitative and mixed-methods approaches

CRICOS code 00025B

 First things first…
1. Assess relevance to your needs:
a. After reading the title, read the abstract carefully.
b. Still seems relevant? Then have a close look at the conclusions. Stills seems relevant?
2. Scan the paper and have a close look at the tables and figures.
3. For large, multi-objective papers, maybe scan headings and sub-headings to identify which
sub-sections are relevant to your needs.
4. Read the introduction to gain context and the discussion to appraise relevance/usefulness
5. Skim the methods / methodology to identify any limiting aspects there.
6. Reading and re-reading will deepen your understanding, and become easier the more you
do it
See also: https://www.youtube.com/watch?v=EEVftUdfKtQ

CRICOS code 00025B

 Look for signals of ‘strength’
Strong papers will JUSTIFY the methodological approach, not just describe
it
The Betz questionnaire has been found to have good internal reliability, with Betz
reporting a split-half reliability of 0.92, and other studies reporting Cronbach’s alphas
of 0.72 and 0.90 (refs.). McMullan et al. (2012, p. 180) also report studies finding a test-
retest reliability of 0.87 over a two-week period, and correlations of about 0.70 with
the MARS scale. Consequently, it would appear that the Betz questionnaire has good
psychometric properties, and given its brevity and adaptation to tertiary classrooms,
and its apparently successful use in the McMullan et al. study with nursing students, it
was believed to be a good maths anxiety questionnaire for this study.”

CRICOS code 00025B

 Signals for limitations
Suggestions for future research, when clearly articulated, indicate limitations of
the current study:

... Future research should focus on two important extensions of the work presented here. First,
we need to understand the important determinants of change from overweight/obese status to
normal BMI and, in particular, the determinants of this change that are related to blood
pressure change. Randomized trials of interventions to treat obesity in childhood should aim,
and have sufficient power, to examine other outcomes in addition to weight loss. Specifically,
they should aim to examine cardiovascular disease risk factor end points, including blood
pressure, fasting lipids, and fasting glucose. Secondly, future research should examine the
relationship between … With the evolution of a number of birth cohorts, including long-term
follow-up of the MUSP study, it will be possible to determine whether the effects we report
here do extend into effects on adult cardiovascular disease risk.

Abdullah A. Mamun, et al. (2005). Hypertension, 45, 1083-1087. CRICOS code 00025B
 More signals
Claims of methodological strengths in one paper points to possible
methodological weaknesses in others.

The present study advances prior work in three ways. First, our students train
using real academic texts, rather than the highly simplified arguments often used
in previous research, which may provide the necessary challenge and motivation
to promote generalized improvements. Second, we use a reliable test of
analytical reasoning with known psychometric properties,28 which correlates
highly with real-world scholastic performance.29 Third, we include a control group,
allowing us to estimate the degree to which improvements are due to our
intervention as opposed to the generic effects of university education or
maturation.
Source: https://doi.org/10.1038/s41539-018-0038-5
CRICOS code 00025B
 Other strategies for establishing strength
Familiarise yourself with journal impact factors for your discipline,
remembering it is a relative, not an absolute numbering system
▪ ‘Nature’ journal = 42.778
▪ ‘Academic Medicine = 5.35 and yet both are top of the field!

Google Scholar can help you find citing research which may identify problems,
and strengths, and appraise the usefulness of the research.

CRICOS code 00025B

 Conclusion
Critically reading a scientific report….more than meets the eye!

▪ An understanding of the components of a scientific report

▪ An understanding of the concept of evaluative criteria for scientific reporting

▪ Knowing what questions to ask of each component, as you read

▪ Reading strategically, looking for signals of strengths and limitations

CRICOS code 00025B

 Further reading
Cooper, H., Camic, P. M., Long, D. L., Panter, A. T., Rindskopf, D., & Sher, K. J. (Eds.). (2012). APA
handbook of research methods in psychology, Vol. 1. Foundations, planning, measures, and
psychometrics. American Psychological Association.

Cooper, H., Camic, P. M., Long, D. L., Panter, A. T., Rindskopf, D., & Sher, K. J. (Eds.). (2012). APA
handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative,
neuropsychological, and biological. American Psychological Association.

H. Motulsky (2010). Intuitive Biostatistics: A nonmathematical guide to statistical thinking. (New York: Oxford
University Press).

See the statistics section of the Khan Academy https://www.khanacademy.org/math/probability for short
instructional videos on various aspects of probability and statistics.

This software is available for UQ staff and student use: https://my.uq.edu.au/information-and-

services/information-technology/software-and-web-apps/software-uq/graphpad-prism (a great source of
basic statistical concepts explanations and analysis validity (assumptions) checklists.)
CRICOS code 00025B
 Thank you for listening ☺

Dr Eva King
Learning Adviser Please feel free to email me or
make an appointment through
UQ Student Services the student services website

Acknowledgements: materials for this workshop

were drawn from the shared resources of the
learning adviser team

e: e.king3@uq.edu.au
w: https://www.uq.edu.au/student-services/
CRICOS code 00025B