Developing

Questionnaire Practice
Data collection methods
Interviewing

Questionnaires

Observational studies

Projective tests
OBJECTIVES IN WRITING A QUESTIONNAIRE

to elicit the information that is

required to enable the researcher to
answer the objectives of the survey.
 questionnaire assumptions
The questionnaire is tapping the concepts of interest.
The potential respondents are competent sources of
the information you need.
The respondents are willing to commit the time to
answer yourquestionnaire
All of the respondents understand your questions in
the same way.
People will follow the rules of the survey and provide
honest data.
Wilson, 2013: 32
 Sequencing the sections

Exclusion question

Screening questions

Main questionnaire
 Exclusion/ Security question
• A common, although not universal, practice is
to exclude respondents from research surveys
• The security question is usually asked as a
prompted question, with respondents shown
a list of industries and professions.
Exclusion/ Security question (2)
 Screening questions
• With all data collection methods other than
face-to-face interviewing these questions
must be asked at the beginning to ensure
eligibility
Screening questions (2)
 Main questionnaire
• Once into the main questionnaire, the writer must consider the
order in which the various topics are presented to the
respondents.
• As a rule, it is better to work from the most general topics
through to the most specific.
• It is generally advisable to start any section of the interview with
behavioural questions before going on to ask attitudes and
images
• If the interview is to include questions of a sensitive nature, then
• they should not be asked right at the beginning of the interview
 Attitudes Vs Behavioral Question

Questions about attitudes have no “true” answer.

Attitudes exist only in a person’s mind.

They can be consistent or inconsistent, clear or

unclear, but they cannot be said to be true or false

[Sudman, & Wansink, 2004]

Flow chart to plan questionnaire

{Brace, 2008: 42]

 The Social Context of Question Asking

• The precise wording of questions plays a vital

role in determining the answers given by
respondents.
• This fact is not appreciated as fully as it should
be, even in ordinary conversation
 Types of question and data

Open-ended

Closed-Ended
 Types of question and data- open or closed
Advantages Disadvantages
Useful when you do not know much Can be demanding for respondents,
about a particular topic and thus especially if you ask too broad a
cannot generate credible response question.
categories.
Useful when the list of known Typically produces many responses
responses is very long. but only a few on each topic.

Good for exploratory studies at the Takes significant coding effort.

beginning of projects.
Helpful as a follow-up to a closed Is sometimes difficult to compare
question. For example, if you ask a the results of open-ended questions
person to rate the usability of a across
Higherthe sample. Some
nonresponse rate than
Useful for getting at strong opinions closed
or topics that may have been missed questions.
by the questionnaire designer.
Efficient when you are asking Requires more time to answer.
questions that can be easily recalled
without a list
Wilson, 2013:45
 Types of question and data-
Advantages
Easier for respondents to answer than
open questions.
Easy to code and analyze.
Appropriate when you are certain that you have
covered the list of possible responses.
open or closed
Disadvantages
Respondents may feel that they have to
choose an alternative that isn’t what they
view as the “best” answer.
Some closed questions require research to
identify the appropriate response categories
Wilson, 2013:45
Likert Scales and Guttman Scales
Likert Scales and Guttman Scales
Planning and Developing a
Questionnaire
Determine
What type of Pilot test the
the purpose What general
question
Devise an explicit
questions should
structure should
data analysis questionnaire
of your data I ask? plan.
I use?
and tools
collection

Consider how to
Determine the establish trust,
Decide Evaluate your
sampling increase
appropriateness draft questions.
requirements rewards, and
reduce costs

Gather
requirements Review the
Consider your Create and
and general questions for
capabilities review
questions from relevance
stakeholders

[Wilson, 2013]
 1. Determine the purpose of your data
collection
a. Understanding user needs
b. Gathering information about particular attributes of your users,
their tasks, and environments
c. Validating design decisions
d. Understanding user attitudes or opinions
e. Comparing the attitudes of different groups
f. Gathering facts
g. Assessing product usability or satisfaction
h. Gathering information comparing competitive products
i. Eliciting knowledge from experts
j. Convincing management to do something
 2. Decide appropriateness

a. Can questionnaire data provide useful information that will

answer your design or business questions?
b. Do you have enough information on the topic to design a
useful questionnaire?
c. Do the benefits of designing a questionnaire and survey
outweigh the costs involved?
d. Do you have the resources necessary to design and
implement a questionnaire study?
e. Can you gather valid and reliable data using a
questionnaire?
f. Do you have access to a reasonable sample of respondents
who match your target audience?
 3. Gather requirements & general questions

a. Interview key stakeholders about what they

know and do not know about users and how
they use a product.
b. Brainstorm with the product team about what
they want to learn from a questionnaire study.
c. Conduct a short brainwriting session
d. Conduct a focus group to find out what issues
are important to key stakeholders and target
user groups.
 4. Consider your capabilities

experience and training to design, implement, and

analyze the data from a questionnaire study.
The design of questionnaires requires:
• background experience in many areas, such as
scale development, psychometrics (theories and
methods of psychological measurement),
• sampling, and
• content analysis.
 5. Determine the sampling requirements for
your questionnaire study
1. Specify or at least acknowledge the sampling process.
– Probability samples,).
– Haphazard samples
– Convenience sample.
– Purposive samples
– Snowball sampling
2. Compile a list of people in the target populations from
which you can choose a sample, or decide how and
where you are going to publicize the survey to recruit
your respondents.
3. Select the sample.
 6. What general questions should I ask?

a.Motivating to the respondent

b.Interpretable by multiple respondents in the
same way
c.Answerable accurately (e.g., whether
respondents can recall and report on past
behaviors)
d.Relevant to all respondents
7. What type of question structure should
I use?
Mutually Exclusive Responses
 8. Evaluate your draft questions.

Conduct an expert review

Ask a small focus

Conduct brief interviews

 9. Create a prototype and review it.

• Design a prototype questionnaire & review

the prototype against principles of
questionnaire design
• Interview a few people not closely associated
with the project
• If you have survey experts, ask them to review
the prototype
10. Review the questions for relevance

Relevance is an important attribute of

questionnaires
11. Consider how to establish trust
12. Devise an explicit data analysis plan
– Whether you need specific answer categories
– How you will code missing data.
– How you will code unusual answers.
– What method you will use for coding open-ended data
– What analyses you will do on single questions and sets of questions.
Consider the following types of analyses:
• Content analysis,
• Measures of central tendency (mean, median, mode) and variability
(standard deviation, variance).
• Outlier analysis.
• Exploratory analyses (box and whisker plots, histograms).
• Confidence intervals.
– Any hypotheses that you may have and what questions will be used to
test those hypotheses.
– Whether you want to cross-tabulate data from different questions.
 13. Pilot test the questionnaire

• Pretesting questionnaires is essential for discovering

flaws and usability issues with cover letters, the
questionnaire itself, and the method of administration
• Respondents should be encouraged to comment on
any aspect of the questionnaire, including unclear or
ambiguous questions, the completeness and clarity of
the response categories, biased
questions,terminology, legibility (Is the text size large
enough for older respondents?), sentence structure,
and threatening questions
 Types of Pilot Survey

informal pilots carried out with a small number of colleagues;

cognitive interviewing in which the questionnaire is tested
amongst respondents;
 accompanied interviewing which may be used principally to
test for interviewer and routeing errors;
large-scale pilot studies where a larger number of interviews
can be used to test for completeness of brand lists or
incidence of sub-groups;
 dynamic pilots, where question wording is changed between
interviews to test alternatives based on responses received

[Brace, 2008]
 Reliable & Valid questionnaire

1. Conduct an item analysis to determine

which questions are “good” and which are
“bad.”
2. Test the reliability of the questionnaire
3. Assess the validity of the questionnaire
4. Develop standards and norms.

[Dumas (2003) in Wilson (2013])

 Item analysis

1. create a pilot questionnaire with as many questions as

possible and present it to a sample of the target
respondents.
2. examine the results and look for questions.
3. eliminate these questions because they do not provide any
discrimination.
4. compare scores on each question against the overall score
and eliminate questions that are not highly correlated with
the overall score in order to increase the internal
consistency of the questions and reduce random error.
5. revise your questionnaire
[Dumas (2003) in Wilson (2013])
 Reliable & Valid questionnaire

1. Conduct an item analysis to determine

which questions are “good” and which are
“bad.”
2. Test the reliability of the questionnaire
3. Assess the validity of the questionnaire
4. Develop standards and norms.

[Dumas (2003) in Wilson (2013])

 Test the reliability of the questionnaire

• Ambiguous questions or response categories that

lead respondents to interpret the questions
differently
• Use of terms that are not understood by all
respondents.
• Double questions where it is not clear what part of
the double question participants are responding to.
• Differences in the administration of the
questionnaire
[Dumas (2003) in Wilson (2013])
 Assess the validity of the questionnaire
a. Content validity: Does the questionnaire include content that is
relevant to the purpose of the questionnaire?
b. Criterion validity: Do the results of the questionnaire correlate with
subsequent outcomes such as sales figures or website hits?
c. Convergent validity: Do the results for one questionnaire correlate
with the results of other questionnaires or methods that attempt to
measure the same thing?
d. Internal validity : Internal validity deals with the confidence you place
in the causal statements you make from a study.
e. External validity : External validity refers to the generalizability of
causal relationships. In a somewhat looser sense, external validity
refers to the relevance of the findings in a wider context (“the real
world”).
[Dumas (2003) in Wilson (2013])
 Develop standards and norms

A final step in the development of psychometric

questionnaires is to create norms regarding
what is good and bad.

[Dumas (2003) in Wilson (2013])

Words and Questions to Avoid in
Your Questionnaire
1. Avoid double-barreled questions
2. Avoid double negative questions
3. Avoid unnecessary repetitiveness
4. Avoid ‘‘leading’’ or ‘‘loaded’’ questions
5. Avoid biased questions

[Gideon, 2012]
 Double-Barreled Questions
Johnson and Christensen (2010) define ‘‘double
barreled’ ’questions as those that ‘‘combine
two or more issues or attitudes in
a single item’’ (p.175).

‘Do you think young children should

‘Do you think young children should havehave
accesstotocell
access cellphones
phonesand
andcredit
creditcards?’
cards?’
 Double Negative Questions

Please tell me whether you agree or disagree with

the following statement about teachers in public
schools:
Teachers should not be required to supervise
students in the halls, the lunchroom, and the
school parking lot (Converse and Presser (1986: p.
13).
 Avoid unnecessary repetitiveness

• it is also time-consuming, as it prolongs the

questionnaire.
• Repetitiveness may also be demoralizing.
• Respondents are more likely to develop fatigue
and frustration, and as a result will skip
questions or stop taking the survey all together.
• As a result, non-response error will increase,
and survey reliability will decrease
 Avoid Biased Questions
The government spends billions of dollars in foreign
aid to countries that are not always in agreement
with our foreign policy. In your opinion, should the
US government cease supporting such countries,
while spending more in rebuilding the local
economy?
• Yes, the US government should cease spending
ourmoney elsewhere.
• No, the US government should not cease from
spending our money elsewhere.
• Don’t know/no opinion.
 Sources

1. Sekaran, U. (2016). Bougie. M,” Research Methods for Business: A Skill

Building Approach”. UK: John Wiley & Sons.
2. Blumberg, B. F., Cooper, D. R., & Schindler, P. S. (2014). Business research
methods. McGraw-hill education.
3. Wilson, C. (2013). Credible checklists and quality questionnaires: A user-
centered design method. Newnes.
4. Gideon, L. (2012). Handbook of survey methodology for the social
sciences. New York, NY: Springer.
5. Brace, I. (2008). Questionnaire design: How to plan, structure and write
survey material for effective market research. Kogan Page Publishers.
6. Bradburn, N. M., Sudman, S., & Wansink, B. (2004). Asking questions: the
definitive guide to questionnaire design--for market research, political
polls, and social and health questionnaires. John Wiley & Sons.