Survey questions,

design, and
implementation
ONLY THE MOST IMPORTANT PARTS
Your intentions must be clear
What are you really trying to get out of a survey?
◦ Get clear on your concepts
◦ What level of specificity do you need?
◦ Is this the right method???

Who are you trying to reach?

◦ The population  the sample
◦ Do you want/need a probability sample?
◦ Do you have a list?

Organize and plan to carry out your intent

Social exchange theory guides questions,
design, and implementation strategy
Why do people respond?
◦ Cognition
◦ Motivation

This survey is interesting, important and easy!

Writing questions: principles
Ultimate goal:
◦ Everyone will interpret the same
◦ Everyone will be able to respond accurately
◦ Everyone will be willing to answer
Writing questions: principles
Question is for
◦ A particular population
◦ A particular purpose
◦ Placement next to another particular question
8 criteria for judging each question
1. Does the question require an answer?
2. Is there an accurate, ready-made response?
3. Can one accurately recall and report past behavior?
4. Is one willing to reveal the answer?
5. Is one motivated to answer?
6. Is understanding of the category influenced by more than words?
7. Is information being collected by more than one mode?
8. Is changing the question okay with the survey sponsor?
Writing questions: Types
Open ended question
Closed question, ordered response
Closed question, no order to response
Writing questions: guidelines
Simple words over specialized words
Avoid specificity that exceeds potential for accurate, ready-made response
Scaled questions
◦ Equal numbers of +/- categories in scale
◦ Avoid bias from unequal comparisons
◦ Always include “undecided” or “don’t know” at the end

Avoid asking respondent to say “yes” when it means “no”

Avoid excess cognitive burdens
◦ Ranking questions with basic instructions and limited choices

Ask only one question at a time

Designing survey: question order matters
Survey is “a conversation… evolves in accord with societal norms”
◦ Do not ask demographic/personal information until the end

Most salient  least salient

◦ Develop expectations and fulfill early on

First question:
◦ Should apply to everyone
◦ Should be easy
◦ Interesting!
◦ Connected to survey purpose
Designing survey: question order matters
Describe before evaluate
◦ Then group common response types

Opinion questions are not independent

◦ Order effects are real – use your judgment about what is most important
Designing survey: guide respondents
Pre-attentive and attentive processing
Written words and visual clues guide to completion
◦ Size, color, spacing
◦ Similarity
◦ Figure/ground
◦ Simplicity, symmetry, and regularity
Designing survey: guide respondents
Answer boxes on the left aligned, with one vertical column of answers
◦ Online conventions re: button vs. box for one vs. more than one

Use indentation and maintain spacing

◦ Skip patterns on paper need indentations for screened questions
◦ Online skip automatically

Number every question sequentially

◦ Online include completion bar

Maintain direction of question scales

Implementation: check the fundamentals
Pretest the survey questions
◦ Check comprehension, cognition, and time to complete

Methods
◦ Knowledgeable colleague review
◦ Test across browsers/OS
◦ “Think out loud” completion evaluation
◦ Time for a small pilot with intended audience?
Implementation: plan it out
How do you contact respondents?
How do you get the survey to them and back from them?
Do you need to remind them to respond?
How will you deal with the data?
Implementation: get it to the people
You need a list of the population especially if you are sampling
◦ Does the list contain everyone in the pop?
◦ Does the list include people not in the pop?
◦ How is the list maintained and updated?
◦ Are there duplicates on the list?
◦ Does the list have other information that can improve accuracy?

Or you need some plans for non-probability sampling

◦ Convenience
◦ “Walk-bys” – be reasonably sure they resemble the pop
◦ Purpose
◦ Key informants
◦ Snowball
◦ Ask for the next respondent
Implementation: motivate response
Respondent friendly, easy to return, multiple/personalized contacts
Salience: relate to current behavior and interests
◦ Why was I selected? Why is this useful?

Pre-notice helps esp if the surveyor is not the known organization

In-person distribution:
◦ What/why? Have a take-away and a practiced spiel
◦ Have all the materials there!
◦ Watch your bias in communicating and approaching
Implementation: don’t forget data entry
Make a codebook with uniform approach
Deal with missing and ‘no opinion’
“check all” is actually a series of yes/no (1/0) questions!

Keep track with attention to detail

You did it!
QUESTIONS?