Lecture 10

Questionnaire design

Learning Objectives
1. Understand that questionnaire design is difficult and why.
2. Plan the steps involved in designing an effective questionnaire.
3. Recognize how the method of data collection influences questionnaire
design.
4. Understand the types of questions and how they are used.
5. Describe the major sections of the questionnaire and how they relate to one
another.

1
 Methods to record data
• Questionnaire
– Prepared set of questions (or measures) used by respondents
or interviewers to record answers (data).
– Structured framework consisting of a set of questions and
scales designed to generate primary data.
• Interview guide
– Topics to cover, the questions to be asked, the sequence of
questions or topics, and the wording of the questions.
– No scales for measuring concepts.
• Observation guide
– Initial structure based on the conceptual framework.
– May expand to include new information that emerges during
the data collection process. 3

Steps in the design of a questionnaire

1. Initial considerations
2. Clarifying concepts
3. Determining question types, format, and sequence
4. Pretesting the questionnaire
5. Pilot testing the questionnaire
6. Administering the questionnaire

2
 Step 1. Initial considerations
• Clarify the nature of the research problem and objectives.
• Develop research questions to meet research objectives.
• Define target population and sampling frame (identify
potential respondents).
• Determine sampling approach, sample size, and expected
response rate.
• Make a preliminary decision about the method of data
collection.

Step 2. Clarifying concepts

• Ensure the concept(s) can be clearly defined.

• Select the variables/indicators to represent the concepts.

• Determine the level of measurement.

3
Step 3. Question types, format, & sequence
• Determine the types of questions to include and their order.

– Open-ended questions: NO given option.

Examples

– Closed-ended questions: given options.

Examples

Step 3. Question types, format, & sequence

• Check the wording and coding of questions.
• Decide on the grouping of the questions, the structure and
layout of the questionnaire, and the overall length of the
questionnaire.
– Questionnaire sections
– Opening questions
– Research topic questions
– Branching questions
– Classification questions

4
 Questionnaire sections
• Initial section
– Opening questions
• Middle section
– Research topic questions
– Questions: directed specifically at the topics of the research
• Final section
– Classification questions

Research topic questions

• Usually grouped into sections by topic
– The respondents can find it is easier to respond.
• Early questions can influence responses to later questions
– Question sequencing: general early, more specific later.
– Funnel approach.

10

5
 Branching questions

• Direct respondents to answer

Q1
the right questions and doing
so in the proper sequence
Yes No
Q2: Yes/No
• Enable respondents to skip
irrelevant questions or to
more specifically explain a
particular response. Q3

Q4

Q5

11

Classification questions
• Seek information of a more personal nature
– If asked early on, may affect the nature of responses to
subsequent questions or even result in nonparticipation.
– E.g.: age, income, etc.
– Being at the end does not mean classification questions are
less important
• Funnel approach
– “Is your total annual household income above or below $30,000?”
– If YES:
• “Is your total annual household income above or below $20,000?”

12

6
 Preparing and presenting good questions
• Use simple words
– Questions must be in a language familiar to the respondent.
– Avoid jargon / technical terms unless absolutely necessary.
– For any jargon/technical terms: provide definitions for all
words where misunderstandings could occur.
• Be brief
– Questions should be short and to the point.
• Avoid ambiguity
– Words that are unfamiliar to respondents or that have more
than one meaning  ambiguity
– Wording should be clear and concise.

13

Preparing and presenting good questions

• Avoid leading questions
– Leading questions imply that a particular answer is correct
or lead a respondent to a socially desirable answer.
• Avoid double-barreled questions
– Double-barreled questions include two or more issues and
make interpretation difficult and often impossible
– Example:
To what extent do you agree or disagree with the following statements:
• Macy’s employees are friendly and helpful.
• Macy’s employees are courteous and knowledgeable.

14

7
 Preparing and presenting good questions
• Be careful about question order…
– Early questions should be general in nature and later ones
more specific to minimize position bias introduced by the
order of the questions
 Questions in a logical order that is organized by topics

• Position Bias • No Position Bias

Q1: How important are flexible hours Q1: How important are flexible hours
in evaluating job alternatives? in evaluating job alternatives?
Q2: What factors are important in Q2: What factors are important in
evaluating job alternatives? evaluating job alternatives?

15

Preparing and presenting good questions

• Be careful about … and context effects
– A context effect occurs when the position of a question
relative to other questions influences the response.

16

8
 Preparing and presenting good questions
• Check questionnaire layout
– Presentation
– Spacing
– Layout

17

Preparing clear instructions

• Self-completion instructions
– Introducing and explaining how to answer a series of
questions on a particular topic
– Transition statements from one section (topic) of the
questionnaire to another
– Which question to go to next
– How many answers are acceptable: for example, “Check only
one response” or “Check as many as apply”
– Whether respondents are supposed to answer the question by
themselves or consult another person or reference materials
– What to do when the questionnaire is completed, that is, how
to return it

18

9
 Preparing clear instructions
• Interviewer-assisted instructions
– How to increase respondent participation
– How to screen out respondents who are not wanted and still
keep them happy
– What to say when respondents ask how to answer a question
– When concepts may not be easily understood, how to define
them
– When answer alternatives are to be read to respondents (aided
response) or not to be read (unaided response)
– How to follow branching or skip patterns
– When and how to probe
– How to end the interview

19

Step 4. Pretesting the questionnaire

• Determine the nature of the pretest for the preliminary
questionnaire.
• Decide which individuals and how many will review the
questionnaire.
• Ask for comments on instructions for completing
questions, the sequence of questions, and any difficulties in
responding to questions.
• Analyze initial data to identify the limitations of the
preliminary questionnaire.
• Revise questionnaire instructions, sequence, wording,
length, as needed.

20

10
 Step 5. Pilot testing the questionnaire
1. Collect data from a sample similar to the final respondents.
The size of the pretest sample is determined based on the
number of questions.
2. Assess reliability and validity of questions.
3. Refine the questionnaire as needed.
4. Revisit some or all of the preceding steps, if necessary.

21

Step 6. Administering the questionnaire

• Identify the best practices for administering the type of
questionnaire utilized.
1. By mail, including overnight delivery
2. In person
3. Over the telephone
4. Electronically online via a hosted website such as Qualtrics
or sending directly to respondents via e-mail or text message
• Train and audit field workers, if required.
• Ensure a process is in place to handle completed
questionnaires.
• Determine the deadline and follow-up methods.

22

11
 Examples of open-ended questions
1. What do you think about the Internal Revenue Service?

2. Which mutual funds have you been investing in for the

past year?

3. How are the funds you have invested in performing?

4. What do you think of airport security?

Back
23

Examples of closed-ended questions

1. Did you check your e-mail this morning?
____ Yes
____ No

2. Which countries in Europe have you traveled to in the past

six months?
____ Belgium
____ Germany
____ France
____ Holland

24

12
 Examples of closed-ended questions
3. How often do you eat at Samouel’s Greek Cuisine
restaurant?
____ Never
____ 1–4 times per year
____ 5–8 times per year
____ 9–12 times per year
____ More than 12 times per year

Back
25

13