Professional Documents
Culture Documents
Critically Reading A Scientific Report EK SEm 2 21
Critically Reading A Scientific Report EK SEm 2 21
Critically Reading A Scientific Report EK SEm 2 21
(a)
Testing of the
model or
hypothesis
▪ It is a good idea to familiarise yourself with the evaluative criteria for common
types of studies in your discipline
▪ Reading lots of research reports in top journals will help you develop an
understanding of evaluative criteria and how they apply
CRICOS code 00025B
Questions to ask of each component, as
you read
▪ A great study design for one context may simply not be viable in another!
▪ What this means is……you need to know what approaches are relevant to
your discipline
▪ Read top journals, follow editorials, read methodological lit, coursework
students texts/reading lists
CRICOS code 00025B
Methods: Data sampling
“The cognitive impairment of those with slow gait was apparent across multiple neuropsychological
domains: they had … poorer perceptual reasoning (r = 0.29; 95%CI: 0.22 to 0.35; p < .001), …”
Our level of
A measure of the confidence that the
strength (weak) of An estimate of
correlation is not just
the relationship the uncertainty
a statistical fluke.
between in r. The “real”
answer is likely in In this case we can be
perceptual
this interval. pretty confident that
reasoning and gait
there really is a
speed.
correlation between
the variables.
CRICOS code 00025B
To interpret qualitative results, you need to be able to see the
links between the data and the analysis
▪ Excerpts of raw data e.g. direct
quotes from participants
“Highly statistically significant” means we are While we might be very confident a crime
“very confident” there really is a difference; was committed, the crime could be small or
the effect size tells us how big that difference large.
is. E.g. we have clear video evidence with a speed
camera that Mr X was speeding in a 40 km/h
school zone:
But was he doing 50km/h or 100 km/h?
Both absolute and relative differences are important to consider.
“The Pen Is Mightier Than the Keyboard: Advantages of Longhand Over Laptop Note Taking”
DOI: 10.1177/0956797614524581
But is it really?
“On factual-recall questions,
participants performed equally well
across conditions… However, on
conceptual-application questions,
laptop participants performed
significantly worse [8.7%] than
longhand participants, ….”
8.7% is important, it
would drop you from a 7
to a 6 or from a 6 to a 5.
But this was the relative
difference, what was the
absolute difference?
CRICOS code 00025B
The absolute difference was
only 0.52 marks out of 6!
http://www.nytimes.com/2015/02/07/upshot/how-elementary-school-teachers-biases-can-
discourage-girls-from-math-and-science.html?_r=1&abt=0002&abg=1
CRICOS code 00025B
Example
Suppose you are a Southern Flounder fish farmer and you come across a study that shows if you
heat your pond water to 23 ℃, your fish will grow much larger, and the result is highly statistically
significant.
Should you be excited? After all, you are paid per kg, not per fish,
so heavier fish will bring in more income.
Well, if the increase in your electricity bill from heating the pond
water is greater than your increased income from heavier fish,
then you are economically worse off!
Source: Mamun et al. 2007: Addiction, Vol. 102, Iss. 4, Pages: 647–
654 CRICOS code 00025B 42
Develop some reading strategies
▪ To keep up to date?
... Future research should focus on two important extensions of the work presented here. First,
we need to understand the important determinants of change from overweight/obese status to
normal BMI and, in particular, the determinants of this change that are related to blood
pressure change. Randomized trials of interventions to treat obesity in childhood should aim,
and have sufficient power, to examine other outcomes in addition to weight loss. Specifically,
they should aim to examine cardiovascular disease risk factor end points, including blood
pressure, fasting lipids, and fasting glucose. Secondly, future research should examine the
relationship between … With the evolution of a number of birth cohorts, including long-term
follow-up of the MUSP study, it will be possible to determine whether the effects we report
here do extend into effects on adult cardiovascular disease risk.
Abdullah A. Mamun, et al. (2005). Hypertension, 45, 1083-1087. CRICOS code 00025B
More signals
Claims of methodological strengths in one paper points to possible
methodological weaknesses in others.
The present study advances prior work in three ways. First, our students train
using real academic texts, rather than the highly simplified arguments often used
in previous research, which may provide the necessary challenge and motivation
to promote generalized improvements. Second, we use a reliable test of
analytical reasoning with known psychometric properties,28 which correlates
highly with real-world scholastic performance.29 Third, we include a control group,
allowing us to estimate the degree to which improvements are due to our
intervention as opposed to the generic effects of university education or
maturation.
Source: https://doi.org/10.1038/s41539-018-0038-5
CRICOS code 00025B
Other strategies for establishing strength
Familiarise yourself with journal impact factors for your discipline,
remembering it is a relative, not an absolute numbering system
▪ ‘Nature’ journal = 42.778
▪ ‘Academic Medicine = 5.35 and yet both are top of the field!
Google Scholar can help you find citing research which may identify problems,
and strengths, and appraise the usefulness of the research.
Cooper, H., Camic, P. M., Long, D. L., Panter, A. T., Rindskopf, D., & Sher, K. J. (Eds.). (2012). APA
handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative,
neuropsychological, and biological. American Psychological Association.
H. Motulsky (2010). Intuitive Biostatistics: A nonmathematical guide to statistical thinking. (New York: Oxford
University Press).
See the statistics section of the Khan Academy https://www.khanacademy.org/math/probability for short
instructional videos on various aspects of probability and statistics.
Dr Eva King
Learning Adviser Please feel free to email me or
make an appointment through
UQ Student Services the student services website
e: e.king3@uq.edu.au
w: https://www.uq.edu.au/student-services/
CRICOS code 00025B