Reading and

understanding
quantitative articles
SIB 3 semester

Slobodan Kacanski
kacanski@ruc.dk
 Format

 Three times two-hour classes (Wednesdays 13:15 –

15:00)
 Mix of lectures, discussions and exercises
 Active student participation
 Written assignments (prerequisite for exam)
WHAT YOU DO IN A
METHODOLOGY COURSE
 Focus on the process of research, not the topic
 Discuss the norms of ‘good’ or acceptable work in the
field
 Understand strengths and limitations of various
methods
 Choose appropriate methods to your research
 Develop skills in using specific methods
 ‘Try out’ different methods in practice
Definitions
 Philosophical orientation-a worldview that underlies and
informs our research

 Methodology-a way of thinking about and studying

social phenomena

 Methods-techniques and procedures for gathering,

producing and analyzing data
Research and research methods
 Research methods are split broadly into
quantitative and qualitative methods

 Which you choose will depend on

1. your research questions
2. your underlying philosophy of science
3. your preferences and skill
Basic principles of research
design
Four main features of research design, which are distinct, but closely related

 Ontology: How you, the researcher, view the world and the assumptions that you make about
the nature of the world and of reality
 Epistemology: The assumptions that you make about the best way of investigating the world
and about reality
 Methodology: The way that you group together your research techniques to make a coherent
picture
 Methods and techniques: What you actually do in order to collect your data and carry out
your investigations

 These principles will inform which methods you choose: you need to understand how they fit
with your ‘bigger picture’ of the world, and how you choose to investigate it, to ensure that
your work will be coherent and effective
Four main schools of ontology
(how we construct reality)

However, none of these positions are absolutes.

They are on a continuum, with overlaps between them.

 Epistemology
i.e. the way in which you choose to investigate the world (how do we understand
it?)

Two main schools are positivism and social constructionism:

 Positivists believe that the best way to investigate the world is through
objective methods, such as observations. Positivism fits within a realist
ontology.

 Social constructionists believe that reality does not exist by itself. Instead, it
is constructed and given meaning by people. Their focus is therefore on
feelings, beliefs and thoughts, and how people communicate these. Social
constructionism fits better with a relativist ontology.
Methodology
 Epistemology and ontology will have implications for your
methodology
 Realists tend to have positivist approach
 tend to gather quantitative sources of data
 Relativists tend to have a social constructionist approach
 tend to gather qualitative sources of data
 Remember these are not absolutes! People tend to work on a
continuum  role for mixed methods and approaches
GROUP EXERCISE:
(15 min)

 1. Take some time to  2. Explain to your

think about your own group how do you
personal and think this might
professional influence your own
philosophical research and why.
orientation and beliefs
you have about the
world.
 The research process:

Theory → Hypothes(i/e)s → Concept Operationalisation →

Empirical setting and sample → Processing of data → Data
analysis → Data Interpretation
Choosing your approach
 Your approach may be influenced by your colleagues’ views, your organisation’s approach, your
supervisor’s beliefs, and your own experience
 There is no right or wrong answer to choosing your research methods
 Whatever approach you choose for your research, you need to consider five questions:
 What is the unit of analysis? For example, country, company or individual.
 Are you relying on universal theory or local knowledge? i.e. will your results be
generalisable, and produce universally applicable results, or are there local factors that will
affect your results?
 Will theory or data come first? Should you read the literature first, and then develop your
theory, or will you gather your data and develop your theory from that? (N.B. this will likely
be an iterative process)
 Will your study be cross-sectional or longitudinal? Are you looking at one point in time, or
changes over time?
 Will you verify or falsify a theory? You cannot conclusively prove any theory; the best that
you can do is find nothing that disproves it. It is therefore easier to formulate a theory that
you can try to disprove, because you only need one ‘wrong’ answer to do so.
Quantitative approaches
 Attempts to explain phenomena by collecting and analysing
numerical data
 Tells you if there is a “difference” but not necessarily why
 Data collected are always numerical and analysed using
statistical methods
 Variables are controlled as much as possible so we can
eliminate interference and measure the effect of any change
(not always?)
 Randomisation to reduce subjective bias
 If there are no numbers involved, it’s not quantitative
 Some types of research lend themselves better to quant
approaches than others
Quantitative data
 Data sources include
 Surveys where there are a large number of
respondents (esp. where you have used a Likert
scale)
 Observations (counts of numbers and/or coding
data into numbers)
 Archival (secondary) data (government data; SATs
scores etc.)
 Analysis techniques include hypothesis testing,
correlations and cluster analysis
 Concepts need measurement.
 Measuring could be done using scales (income, age etc.) or
could be done by counting them in a specific category (men,
women) or by asking questions where participants choose a
statement from the list of scores (e.g. high level of feeling
corresponds a high score etc. – Likert scale)
Dependent and independent
variable
 Dependent – Research question/hypothesis
describes, explains, or predicts changes in it. The
variable that is influenced or changed by the
independent variable

 Independent - Manipulation or variation of this

variable is the cause of change in other variables
Some definitions
VARIABLE DEVELOPMENT

ATTRIBUTE ATTRIBUTE DEVELOPED DEVELOPING

How do we construct variables?

Variable Development

Attributes Low Medium High

Values 1 2 3

Relationship
X,Y, Z notation

Z = covariate (control variable)

 e.g. Severity of illness

X = independent variable (interventions)

 e.g. Self-care symptom management

Y= dependent variable (outcome)

 e.g. Quality of life
Types of Quantitative Research
Designs
 Descriptive X? Y? Z?
 What is X, Y, and Z?

 Correlational rxy.z
 Is there a relationship between X and Y?

 Causal ΔX  ΔY?
 Does a change in X cause a change in Y?
The levels of measurement

1. Nominal
2. Ordinal
3. Interval
4. Ratio
Nominal level of measurement
 These variables consist of categories that are non-
ordered
 A simple categorical variable is binary or
dichotomous (1/0 or yes/no).
 When used as an independent variable, it is often
referred to as a “dummy” variable,
 When used as a dependent variable, the outcome of
some phenomenon is either present or not.
Ordinal level of measurement
 These variables are also categorical, but we can
say that some categories are higher than others.
For example, income tax brackets, social class,
levels of education etc.
 However, we cannot measure the distance between
categories, only which is higher or lower.
 Hence, we cannot say that someone is twice as
educated as someone else.
 Can also be used as a dependent variable.
Interval level of measurement
 Variables of this type are called scalar or index
variables in the sense they provide a scale or index
that allows us to measure between levels. We can
not only measure which is higher or lower, but
how much so.
 Distance is measured between points on a scale with
even units.
 Good example is temperature based on Fahrenheit or
Celsius.
Ratio level of measurement
 Similar to interval level variables in that it can
measure the distance between two points, but can
do so in absolute terms.
 Ratio measures have a true zero, unlike interval
measures.
 For example, one can say that someone is twice as rich
as someone else based on the value of their assets since
to have no money is based on a starting point of zero.
Measurement hierarchy
STRONGEST
RATIO

INTERVAL

ORDINAL

NOMINAL
WEAKEST
Analysing quant data

 Always good to group and/or visualise the data

initially  outliers/cleaning data
 What average are you looking for?
Mean, median or mode?
 Spread of data:
 skewness/distribution
 range, variance and standard deviation
Examples of data visualization
 Demonstration
Examples of data visualization
 Simplification
Examples of data visualization
 Full data visualization
Understanding of visualization

• We need to clarify what we

intend to visualize because the
research article is the report
that aims of informing the
others about the results of our
work.
What are you looking for?
 Trying to find the signal from the noise
 Generally, either a difference (between/within
groups) or a correlation
 Choosing the right test to use:
parametric vs non-parametric (depends what sort of
data you have – interval/ratio vs nominal/ordinal and
how it is distributed)

 Correlation does not imply causation!

Example correlations
 What is a model?
- Model is a representation or an abstraction
of reality.
-2 types of models:

a) Deterministic - if certain conditions are met, the outcome

is certain to happen

b) Probabilistic (stochastic) - if certain conditions are

met, the outcome is more or less likely to happen
 Look at the following example and determine the type
of the model
Example

How do we discuss this? Is this a model?

Descriptive statistics before
models
Simple model in SPSS
Exercise
Discuss the model – Determine what variables are significant
in this model and why
Exercise
Example of a model
• Data has to be valid and reliable:
• Reliability – is the degree to which an assessment tool produces stable
and consistent results.
• Validity - refers to how well a test measures what it is suppose to
measure.
• For primary data - we need to be sure that questions are consistent
(questionnaires and archival data).
• For secondary data - certainty in regards of trustworthiness is crucial.
Other aspects of research design
 Validity
 Reliability
 Trustworthiness
 Dependability: showing that the findings are consistent and could
be repeated
 Confirmability: a degree of neutrality or the extent to which the
findings of a study are shaped by the respondents and not
researcher bias, motivation, or interest
 Credibility: confidence in the 'truth' of the findings
 Transferability: showing that the findings have applicability in
other contexts
Interpreting test statistics
 Significance level – a fixed probability of wrongly rejecting the null
hypothesis H0, if it is in fact true. Usually set to 0.05 (95%) and 0.01
(99%).
 Null hypothesis - The null hypothesis is a statement assumed to be
true until the evidence indicates otherwise.
 p value - probability of getting a value of the test statistic as extreme
as or more extreme than that observed by chance alone, if the null
hypothesis H0, is true.
 Effect size – numerical way of expressing the strength or magnitude
of a reported relationship, be it causal or not
Example 1
 Matched users were those whose learning styles were matched with the
lesson plan e.g. sequential users with a sequential lesson plan. Mismatched
participants used a lesson plan that was not matched to their learning style,
e.g. sequential users with a global lesson plan.

 H0 – there will be no statistically significant difference in knowledge

gained between users from different experimental groups
 H1 – students who learn in a matched environment will learn significantly
better than those who are in mismatched environment
 H2 – students who learn in a mismatched environment will learn
significantly worse than those who learn in a matched environment
Interpreting test statistics

Statistical testing was carried out using a univariate ANOVA in SPSS, to

determine if there was any significant difference in knowledge gained.
Initial conjecture suggests that the mismatched group actually performed
better than the matched group.
However, the difference between the two groups was not significant
(F(1,80)=0.939, p=0.34, partial eta squared = 0.012) and hence hypotheses
1 and 2 can be rejected.
Common pitfalls
 Population specification
 Sampling
 Selection
 Non-responsive
 Measurement
Population specification
 This type of error occurs when the researcher
selects an inappropriate population or universe
from which to obtain data.
Example
 Packaged goods manufacturers often conduct surveys of
housewives, because they are easier to contact, and it is
assumed they decide what is to be purchased and also do the
actual purchasing.
Sampling
 Sampling error occurs when a probability
sampling method is used to select a sample, but
the resulting sample is not representative of the
population concern. Unfortunately, some element
of sampling error is unavoidable. This is
accounted for in confidence intervals, assuming a
probability sampling method is used.
Example
 Suppose that we collected a random sample of 500
people from the general U.S. adult population to
gauge their entertainment preferences. Then, upon
analysis, found it to be composed of 70% females.
Selection
 Selection error is the sampling error for a sample
selected by a nonprobability method.
Example
 Interviewers conducting a mall intercept study
have a natural tendency to select those respondents
who are the most accessible and agreeable
whenever there is latitude to do so.
Non-responsive
 Nonresponse error can exist when an obtained
sample differs from the original selected sample.
Example
 Telephone surveys (landline)???
Measurement
 Measurement error is generated by the
measurement process itself, and represents the
difference between the information generated and
the information wanted by the researcher.
Example
 A retail
store would like to assess customer
feedback from at-the-counter purchases.
What quant researchers worry
about
 Ismy sample size big enough?
 Have I used the correct statistical test?
 Have I reduced the likelihood of making Type I
and/or Type II errors?
 Are my results generalizable?
 Are my results/methods/results reproducible?
 Am I measuring things the right way?
Type I and II errors

 Type I error: incorrect rejection of true null

hypothesis
 when our results are significant (*) but wrong

 Type II error: incorrect retaining of false null

hypothesis
 when our results are insignificant ( ) and wrong
What’s wrong with quant
research?
 Some things can’t be measured – or be measured
accurately
 Doesn’t tell you why
 Can be impersonal – no engagement with human
behaviours or individuals
 Data can be static – snapshots of a point in time
 Can tell a version of the truth (or a lie?)
“Lies, damned lies and statistics” – persuasive power
of numbers
 Example 2
The Elmwood Tutoring Program provides one-on-one and small group
literacy tutoring to students in grades 3-5 who read below grade level.
Students at two local primary schools participate twice per week in 90-
minute literacy tutoring sessions focusing on improving students’ ability to
read aloud (oral reading fluency). Outcome data are collected by school
personnel who administer a standardized reading comprehension test to
students at the beginning and end of the school year. Test results show
trivial improvements in test scores from pretest to posttest. Assuming the
problem is with measurement and not with the program design…

What is the main problem that this case has with the measurement?
☐ Reliability ☐ Validity

Methods course, Yemen, October 2012

Briefly explain your answer.
Example 3 Simpson’s paradox
 It is possible for the (bivariate) association between two variables to be
positive, yet be negative at each fixed level of a third variable.

Example: Florida countywide data

There is a positive correlation between crime rate and education (%
residents of county with at least a high school education)!
There is a negative correlation between crime rate and education at each
level of urbanization (% living in an urban environment). Why ?
Perhaps urbanization is a common causal factor. As urbanization
increases, both crime and education increase.
 Example 4

Out of 73 case studies, 33 are drawn from CMFs, but

nearly 50% of the cases in this sub-group (14 case-
studies) come from one single article.

So is the conclusion valid or not?

 Example 5
The article from which the 14 examples are drawn, concludes
exactly the opposite of what is said by Porter-Bolland, namely
that ‘protected areas have slower deforestation rates and are
better at slowing deforestation rates than indigenous
reservations’ (Armenteras et al, 2009, p. 1415).
 What is the case?
 The empirical study is a case of e.g. land struggles in
a particular country
 In general, it is a case of post-colonial land reform and
reconfiguration of customary rights
 In the conceptual sense, it is a case of mutual
constitution of property and authority
 In the theoretical sense, it is a case of state formation
and production of public authority through the
production of rights
Case-studies
 Case-studies are presented as self-evident.
 Of what the material is a case is actually less
evident
 Discuss whether the case is specific (precise,
restricted), general (common, universal), concrete
(actual, tangible) or abstract (conceptual,
intangible)
Overall conclusions
Extrapolation
 Empirical generalization is an extrapolation
 Analytical generalization is an identification of
fundamental properties of a phenomenon
 Common error: let few informants speak on behalf
of a bigger group or present one/two incidences as
the rule or typical without data verification.
Problem – objectivity?
 Inquiry is never objective. We establish frames of
inquiry through which we understand the world
 Words like: wealthy, poor, farmer, migrant,
government are easy comprehensible
 As concepts, they require precision and are often
politically charged
 Every theory, in particularly the
predominant one in a discipline,
should be subject to scrutiny.
Who has the right to challenge it
and what should come instead ?
HOW TO INTERPRET?
DISCUSS IN YOUR GROUP…
 What attracts you to quantitative/qualitative
research?

 How do you think your personal characteristics

will enhance your ability to do good
quantitative/qualitative research?
Thank you!