An introduction on quantitative research

methods
 Group Experience Sharing
Please talk to your friends/colleagues about
your research experience
- Your research at MA and PhD level
- An inspiring research you have read
recently
 Guiding discussion questions
• What are the areas of research interests?
• What are the current discussions around the areas of
research interest and your critical reflection on them, in
relation to your research?
• What do you want to discover (in wh-question format)?
• In which methods have the answers been pointed out?
• What are the instruments for collecting data?
• Are the instruments good?
• Are the data collected accurate?
• How have the data been analyzed?
• How have the findings been discussed and intepreted?
 Quantitative Methods
Broad Strategies
• Experiments
– The purpose is to establish causal relationship
– Active manipulation of treatment variables by the researcher
– Random assignment of units to each types of treatments
– Comparison between treatment and control groups

• Quasi - Experiments
– The purpose is to establish causal relationship
– Not involve random assignment of units to each types of treatments

• Survey/Passive' (or 'non-intervention') Studies

– Study of the relationships among variables in already existing units. No
random assignment of individuals to groups and groups to treatments is
involved
Survey can be
 Cross-sectional: snapshot of a particular time
 Longitudinal
 Survey can prove association but does not prove causality 4
 Types of Research Questions
addressed by quantitative methods
• Descriptive
• Association/Corelational

• Causal
• Explanatory

5
 Descriptive Problems
• Which approaches of teaching are more frequently chosen by the students?
• What are the differences in girls’ choice of majors compared with boys?
• What are the differences in subject choice according to differences in
social status and ethnic family backgrounds?
• What are the areas that Professional Development Program for teachers
should cover?
• What is the primary school teachers’ teaching qualifications?
• How secondary school teachers were trained?
• What are the types of fake news on Corona virus?

6
• How often/What do ________________
(participants) do ________________
(variable being studied) at
________________ (research site)?
 Association
• To what extent are there any
relationships between teacher
qualifications and their performance?
• To what extent are there any
relationships between teacher
performance and student
achievement ?

8
• What is the relationship between
______________ (variable) and
____________ (variable) for
_________________ (participants)?
Eg: “How often do college students need to
use the bathroom during a test as
compared to during a normal class?”
 Causal relationships or
Assessing the ‘Impact’ of an
Intervention
• Do curriculum improvements that place
stronger emphasis on language competence
increase the students’ performance in the
final exams?
• Do co-operative learning for second-year
classrooms lead to better student outcomes?

10
• Does the ________________ (change) in
_________________ (independent
variable) produce changes (increase,
decrease, stabilize) the _______________
(a dependent variable)?
• Does ________________ (cause variable)
lead to/create _____________ (outcome
variable) in ________________ (setting)?
For example: “Does the color of a person’s
hair lead to higher grades in school?”
 Explanatory Problems
• Why do females, more likely than males,
choose to study BA and MA degrees at ULIS?
• Why are students reluctant to speak in the
English class?

13
 Individual Activity 1
Give an example of:
1. a descriptive question
2. a correlational question
3. a causal question
Go to menti.com to write your
questions from 1-3
14
 Research design
At this stage the following should be
identified:
• Source of information
– Who is appropriate to provide the necessary
information
– Characteristics of the target population
• Data collection methods

15
 Establishing the link between
information needed, source of
information and methods of data
collection

Information Source of Methods of

needed/variables information data collection
to be measured
Students’ Students Questionnaire
motivation
Students’ Teachers Interviews
motivation
16
 Link between the information
needed, sources and methods
Information needed Source Data collection
Student gender Students Questionnaire
Family characteristics parents Questionnaire

Parent attitudes Parents Questionnaire

Student attitudes Students Questionnaire

Student achievement Students Questionnaire

Student Withdrawal Students Questionnaire

17
 Questionnaires Needed

   Questionnaires for Students

   Questionnaires for Teachers
   Questionnaires for Administrators

18
 Instrumentation
• Develop/validate and pilot instruments
(test or questionnaires, rubrics, scales)

19
 Data collection and data
management
• Field work monitoring
• Entering the data into data file
• Cleaning the data

20
 Data Analysis
• Descriptive
• Corelational
• Causal

21
 Writing up the reports and
discussions
• Technical report
• Policy report
• General public

22
 Part 2

A focus on questionnaire
development
 Lead-in
• What do you think makes up Happiness
during Covid-19 pandemic?
• Go to menti.com to answer
Questionnaire design
Latent Variables and Indicators
1

3 Indicators
Latent Variable
4

6
Not directly
observable Directly observable
25
Latent Variables, Manifest Variables and a Sense of Direction

1 1

2 2

3 3
Noise
Latent Variable
4 4

5 5

6 6
A Bigger Idea
Little Ideas
26
Questionnaire design
Construct development
Step 1: Define a meaning for your construct. It will be of
narrow focus, capable of sustaining precise
measurement.
Step 2: Develop appropriate framework of
items/questions for this construct.
Step 3: Test the hypothesis that the items do indeed
imply the meaning of the construct as defined.
Step 4: Revise the items
(Barrett, 2002)

27
Questionnaire design
Clarify the concepts

 Obtain a range of definitions of the

concept (terms) through literature
review
 Decide on a definition
 Outline the dimensions of the concept

28
Questionnaire design
Specify the concepts
 This is the process of moving from the broad
to the specific, from the abstract to the
concrete.
 As a result of this process the researcher can
decide on the number, content and even the
format of indicators to measure a certain
concept.

29
Questionnaire design
How to develop indicators
1. Draft the first items from your knowledge and
the reviewed frameworks (literature)
2. Panel the items/questions/indicators

30
Questionnaire design
Example: What are the variables that belong to
listening learning strategies
•clarify the objectives of listening
•decide what kind of information to listen attentively
and what to ignore.
•check, verify or correct my understanding
• evaluate my answer
•look back to find my problem

31
Questionnaire design
Examples: Indicators to measure the family
wealth
Indicators Measurement
 
Clothes Number of clothes
Meal Number of meals a day
Lack of food Number of months lacking in food
House Status of the house
Cupboard Presence and type of cupboard
Colour TV Presence and type of colour TV
Motorbike Presence and type of a motorbike
Bike Presence and type of bike
Ceiling fan Presence and type of ceiling fans
Stand fan Presence and type of stand fans
Wall clock Presence and type of a wall clock
Lounge suite Presence and type of lounge suite
Refrigerator Presence and type of refrigerator
Thermos Presence and type of thermos
Telephone Presence and type of telephone
Video Presence and type of video

32
Questionnaire design
Indicators to measure students’ perceptions of school helpfulness
 Being knowledgeable
 Knowing how to behave
 Becoming a perfect
person
 Knowing the secret of
longevity
 Being accepted by a
tertiary institution or
technical college
 Gaining affection from
others
 Having an easy and well-
paid job
 Having a job in urban areas
 Becoming rich
 Going overseas
 Having a high position
 Being successful in further
education levels
 Having skills necessary for
life
 Knowing how to do business
 Well-rounded improvement
33
Questionnaire design
Make a table of specifications

 Includes all variables, indicators and

measurement methods
 Covers information needed to answer
research questions that requires using a
“questionnaire”

34
Questionnaire design
Example of Table of specifications
Variables Indicator Measurement
Gender
Motivation Class attendance frequency
Homework frequency
completion
Further investigation times
of lesson content
Collaboration with frequency
friends
Student achievement Score Score

35
Steps for Questionnaire design
1. Establish a table of specifications, panel and revise if necessary
2. Write the questions
1. Determine the general question content needed to obtain each of the desired
information
2. Determine the form of response for each of the questions
3. Choose the exact question wording.
4. Panel and revise the questions if necessary
3. Prepare the questionnaire layout for printing
1. Arrange the questions into an effective sequence.
2. Specify the physical characteristics of the questionnaire (paper type, number
of questions per page, etc.)
3. Panel and revise the questions and the whole questionnaire
4. Pre-testing and Pilot the questionnaire. Analyse and revise the
questions and the whole questionnaire if needed

36
Identify the research questions for which the
questionnaire is being developed

 What are the general research

questions?
 What are the specific research
questions?

37
Process of Operationalisation (1)

Start from General Information needed in

the questionnaire (sections)
 Personal characteristics
 attitudes
 achievement
 gender

38
Questionnaire design
Process of Operationalisation (2)
To more specific…

Attitudes towards teachers

• Student attitudes towards teachers’ activities
(length, topics, organization, etc.)
• Students’ attitudes towards teachers’ instruction
(voice, clarity, etc.)
• Students’ attitudes towards teachers’ methods of
feedback (frequency, types, accuracy, etc.)

Achievement
• Listening scores
39
Questionnaire design
Issues to consider when writing the factual questions

 Do the respondents have the necessary

information to answer the question?–
Knowledge, memory.
 Will the respondents provide the
information willingly? – Sensitive issues.

40
Questionnaire design
Question Wordings
 Use Simple Words
“the catalogue system is too difficult for most readers to master “
vs
“I can never find the books I want” (more direct, more appealing)
 Avoid acronyms, abbreviations, jargon and
technical terms
 Avoid ambiguous words or the words with many
meanings
Have you ever assessed your colleagues’ teaching?
 Avoid leading questions
You haven’t skipped any lessons in this semester, have you?

41
Questionnaire design
Question wordings
 Avoid double-barrelled questions
Do you write an essay weekly and monthly?
 Avoid implicit assumptions
When did you last speak in front of the public?
 Don’t overtax the respondents’
memories

42
Questionnaire design
Question Wordings

 Avoid proverbs or well-known sayings

 Avoid subjective words
 Attitude statements are good if the
respondents recognise the statements
which force them to think

43
Questionnaire design
Selection of types of questions (open or close)
 The number of respondents
 The amount and types of information needed
 The characteristics of respondents (knowledge, age,
culture, religions)
 The amount of time you have to process and interpret
the data
 Your knowledge of the issues (the extent to which you
can anticipate the range of possible answers).
 Your methods of data analysis

44
 Preparing questionnaire layout
Panelling, pre-testing and piloting
Questionnaire design
Spacing

 Allocating sufficient space for answers

 Space requirements should be
considered for :
– Open-ended questions
– Scaling questions
– Coding

46
Questionnaire design
Instructions
 General
 Section
 Question

47
Questionnaire design
General Instruction
 Reason(s) for the questionnaire
 A statement about anonymity
 The sample design - to indicate how the
respondent was chosen
 How to return the questionnaire - if it is mailed
 A contact person
 What will happen to the results
 Thanks
48
Questionnaire design
Question instructions
 How to answer the questions
 Make sure that the instructions and the
questions correspond

49
Questionnaire design
Order of the questions
 Very important
 There is no correct order
 Follow the literature review

50
Questionnaire design
Suggestions

1. Begin with easy and none-threatening questions

2. Do not begin with open-ended questions
3. Arrange questions from general to specific
4. Group questions into sections or topics
5. Use filter questions to ensure that the respondents are
answering relevant questions
6. Attitude statements are suggested to be arranged in
more or less random order
7. Keep the questionnaire as short as possible

51
Questionnaire design
Consistency of questionnaire layout

 Try to use similar format for questions

 Distinguish different instruction levels
with font, size

52
Questionnaire design
Panel and Review the table of specifications

 Do the variables cover all the information

needed for the program?
 Do the indicators cover all the dimensions of
the variable measured?

53
Questionnaire design
Panelling and Reviewing
 Who: a peer, a group of students
 What:
 Relevance of questions to the topic (check
against the table of specification)
 All the issues mentioned earlier
 Wording (Clarity of instructions, questions)
 Layout

54
Questionnaire design
Pre-testing
 Test questionnaire
– Do the respondents understand the questions?
– Are there any difficulties?
– Are there any sensitive questions?
– Is the question order appropriate
– Does the researcher understand the respondent's
response

55
Questionnaire design
Who will involved in Pre-testing
 Very small sample of the population targeted

56
Questionnaire design
How to conduct Pre-testing
 Step 1: Brief the respondents about the questionnaire
 Step 2: Researchers record the respondents’ process of
completing the questionnaire:
– through observation, video recording or audio recording to find
out signs of difficulties or distractions and timing

57
Questionnaire design
How to conduct Pre-testing

 Step 3: Debrief the respondents about the questions in

the questionnaire
 Any difficulties? Why?
 What are the easy questions? Why?
 Any suggestions for improvement?
 Step 4: Revise the questions if needed

58
Questionnaire design
Pilot
 Test the whole process
 Questionnaire
 Methods of administration and colleting the
questionnaires
 Response rate/missing data
 Item analysis
 Data analysis

59
Questionnaire design
Analysis

 Look at frequency of options in each question

 Too many “uncertain”, “don’t know” responses, too
many skipped or omitted items are bad signs in a pilot
study.
 Reliability of scale constructed
 Decision of remove or replace items of scales

60
Questionnaire design

Revise and prepare the final version

61
Part 3: Instrument Reliability
and Validity
Questionnaire design
Reliability and Validity

High Validity Reliable

Low Reliability High Reliability but low
validity
(A) (B) (C)

63
Questionnaire design
Validity
 Validity is the ability of an instrument to measure what is designed
to measure – the construct (Smith, 1991)
 Validity refers to the extent to which an empirical measure
adequately reflects the real meaning of the concept under
consideration (Babbie, 1990: 33)
 Validity is the degree to which empirical and theoretical evidence
support the interpretation of test scores and the use of the test

Thus, validity is based on the ACCURACY of the interpretation/

meaning of the items/questions

64
 ASPECTS OF VALIDITY
Classical Modern: Unified validity

• Content validity • Content

• Criterion validity • Substantial
• Construct validity • Structural
• Generalizability
• External
• Consequence
(Messick, 1989)
 QUESTIONS ON VALIDITY
• Are the contents of the questionnaire relevant to students and
the course?
• Are the items representative of the typical skills/components
of the construct?
• Are the scores given to different parts of the questionnaire in
proportion with the importance of the parts?
• Are different parts in the questionnaire related to each other
and to the construct under measurement?
• Are the results of the questionnaire comparable to similar
ones?
• Are the results used appropriately to their purposes?
 Validity check: who and how
• The researcher:
– Before and after: Document/literature analysis
– After, repetitive: Statistical techniques
• Panel of experts
– Qualitative validation
PRACTICALITY, INTERACTIVENESS,
FAIRNESS, USEFULNESS
• Bachman’s model (1990)
• Practicality
• Interactiveness
• Fairness
• Usefulness
Questionnaire design
Reliability
 Reliability is concerned with how many errors are
included in the evidence.
 If there is no error in the measurement, the same
measurement should be consistent over time and
context.
 Reliability is based in the ACCURACY of the data

69
 SOURSES OF ERRORS FOR
RELIABILITY
• The questions
• The researcher
• The respondent
• The administration conditions
 Tests of reliability
• Internal reliability
– Cronback alpha
• Test/retest reliability
– Correlation between scores of different times
of the same tests
• Rater reliability
– Correlation between different raters’ scores
 Part 3
Variables, data entrance and
descriptive data analysis
 Outline
• Categorical variables (ordinal and nominal)

• Continuous variables (interval and ratio)

• Transfer continuous variable into categorical

variable and report frequency

73
 Categorical data
• Nominal - numbers are used only as labels for
different objects within a set (e.g. gender)
• Ordinal - numbers are used to reflect the rank
order of objects within a set according to a
specific criterion
– Summated-rating or Likert scale items: strongly
agree, agree, disagree, strongly disagree
– Ranking of athletics or swimmers in Olympics
– Grouping students into mastery levels

74
 Statistics with ordinal and
cardinal numbers
• Frequency
 The OUTPUT window will show
something like this:
Statistics

psex pethnic
N Valid 7217 7213
Missing 1 5

psex pethnic

Cumulative Cumulative
Frequency Percent Valid Percent Percent Frequency Percent Valid Percent Percent
Valid BOY 3719 51.5 51.5 51.5 Valid KINH 5842 80.9 81.0 81.0
GIRL 3498 48.5 48.5 100.0 OTHER 1371 19.0 19.0 100.0
Total 7217 100.0 100.0 Total 7213 99.9 100.0
Missing omitted 1 .0 Missing omitted 5 .1
Total 7218 100.0 Total 7218 100.0

76
 Continuous data
• Interval - numbers reflect both the rank
order of objects and the extent of the
differences between them (e.g.
temperature)
• Ratio - scale has an absolute zero and
hence a ratio of scores is independent of
the units of the scale (e.g. height, weight)

77
 Summary of continuous variables

Example of Questions
1. What is the average score that the students
surveyed get?
2. What is the middle score?
3. Which is the most frequent score?
4. What is the highest score ?
5. What is the lowest score?
6. What is the range of students’ scores?
7. To what extent are the scores close to the mean?

78
 Mean and Median
• Mean (average, expected value)
– Sum observations / number of observations
• Median
– 50% subjects below and 50% subjects above

79
 Variance and Standard
deviation

 xi   
2

variance  
i n 1
Where µ is the mean, and n is the number of
observations.

standard deviation= variance

80
 Skewness and Kurtosis
• The skewness value provides an indication of the symmetry of the
distribution.
• Kurtosis provides information about the peakedness of the
distribution.
• If the distribution is perfectly normal: a skewness and kurtosis value
should be 0.
• Positive skewness (tail to the right) indicates a clustering at the
low values. Negative skewness (tail to the left)indicates a
clustering at the high values.
• Positive Kurtosis indicates that the distribution is rather peaked with
long thin tales. Kurtosis values below 0 indicates a distribution that
is relatively flat.

81
 Can we report a frequency for a
continuous variable?
trd500 TEACHER READING 500 SCORE

Cumulative
Frequency Percent Valid Percent Percent
Valid 81.05 2 .0 .1 .1
98.93
131.33
146.98
2
1
6
.0
.0
.1
.1
.0
.2
.1
.1
.3
This is frequency for variable
161.50
177.15
191.68
2
5
5
.0
.1
.1
.1
.1
.1
.4
.5
.6
trd500 (teacher reading score).
206.20
219.61
234.14
8
5
8
.1
.1
.1
.2
.1
.2
.9
1.0
1.2
As there are so many categories,
247.55
260.96
274.37
5
6
18
.1
.1
.2
.1
.2
.5
1.4
1.5
2.0
we’d better transfer this variable
into categorical variable (e.g to
288.89 15 .2 .4 2.5
302.30 31 .4 .9 3.3
316.83 50 .7 1.4 4.7

make frequency groups)

330.24 49 .7 1.4 6.1
344.76 64 .9 1.8 7.9
359.29 76 1.1 2.1 10.0
374.94 92 1.3 2.6 12.6
390.58 123 1.7 3.4 16.0
406.22 171 2.4 4.8 20.8
422.99 196 2.7 5.5 26.2
440.86 238 3.3 6.6 32.9
459.86 268 3.7 7.5 40.3
479.98 304 4.2 8.5 48.8
501.21 363 5.0 10.1 59.0
525.79 339 4.7 9.5 68.4
552.61 364 5.0 10.2 78.6
585.02 317 4.4 8.8 87.4
624.13 223 3.1 6.2 93.6
676.65 158 2.2 4.4 98.0
761.57 70 1.0 2.0 100.0
Total 3584 49.7 100.0
Missing System 3634 50.3
Total 7218 100.0

82
• Introduction to some students’ theses
 Part 5

Data analysis: inferential statistics

• SPSS introduction
• Answering the questions in file: survey
• Then start entering the data into a SPSS
file
• Several statistical techniques
– Bivariate
– T-test
– ANOVA
– Regression
Handout
 Activity 1
• File Vnsample
– Run frequency on father’s education, mother’s
education
– Run all descriptive statistics on pupils’ reading
scores
 Activity 2
Describe the distribution of the variable
measuring student reading performance
(variable: prd500)?
Comment on the ‘centre’ of the data, the
range, the ‘shape’ of the distribution and
the possibility of outliers. You might want
to use some of these technical terms in
your description
 Activity 3
File: VN sample
Recode variable: prd500 into ordinal
variable with 4 categories and report the
distribution:
Group 1: less than 300
Group 2: from 300.1-550
Group 3: 550.01 to 700
Group 4: above 700.01
 Activity 4
• Research question: Is there a difference
between length of service and staff
satisfaction
•  
• Data file: staffsurvey.sav
• Codebook staff_codebook.pdf
• Run analysis, interpret the results