Basic Survey Methods

Types of Surveys
Questionnaires
Group administration
Questionnaires
Mail administration
Questionnaires
Mail administration
Household drop off

Interviews
Personal
Interviews
Personal
Telephone
Selecting the Survey Method
Population Issues

Can the population be enumerated?

Is the population literate?

Are there language issues?

Will the population cooperate?

What are the geographic restrictions?
Sampling Issues

What data is available?
– Addresses
– Phone numbers

Can respondents be found?

Who is the respondent?
– Individual
– Head of household
– Any member of household
– Groups or agencies
Sampling Issues

Can all members of
population be sampled?

Are response rates likely
to be a problem?
et
he
gS ve
ly
tin ecti ely
.
Ra ti me
eff
ec
tiv
nd
Question Issues
ge eff na
na es atio
Ma urc rm
so info t.
1 e re of n
5 ag de orta . ic.
4 Ma
n
ltitu imp sks c if
u le ta spe
3 2 a m hat is ltip not
2 5 an
Sc ide w mu re
c a ge n sa
1 4 de n o
3 ma ec ti
3 to dir
2 5 ow en ide
eh wh ec
1 4 cid ork dd
3 De ew an
54 e th on
2 iz e ly ati
an v
form
1 4
Org ff ec ti in
3 ee of
2 55 tim de
ge itu
1 4 na ult ant.
3 Ma a m port
an im
2 51 Sc at is
1 4 3 w h
3
2 5
1 4
3
2
1

What types of questions can be asked?

How complex will the questions be?

Will screening questions be needed?

Can question sequence be controlled?

Will lengthy questions be asked?

Will long response scales be used?
Content Issues

Can the respondents be expected
to know about the issue?

Will respondent need to consult
records?
Bias Issues

Can social desirability be avoided?

Can interviewer distortion and
subversion be controlled?

Can false respondents be avoided?
Administrative Issues

Costs

Facilities

Time

Personnel
Interviews
The Role of the Interviewer

Locate and enlist cooperation of
respondents.

Motivate respondents to do good job.

Clarify any confusion/concerns.

Observe quality of responses.

Conduct a good interview.
Training the Interviewers

Describe the entire study.

State who is sponsor of research.

Teach enough about survey research
(should respect method and be
motivated).

Explain the sampling logic and process.

Explain interviewer bias.

“Walk through” interview.
Training the Interviewers

Explain selection procedures, including
– Reading maps
– Identifying households
– Identify respondents

Rehearse interview

Explain supervision

Explain scheduling
The Interviewer’s Kit

Official-looking 3-ring notebook

Maps

Sufficient copies of survey

Official identification

A cover letter

A phone number to call
The Interview

Opening remarks

Asking the questions

Obtaining adequate answers

Recording the response

Concluding the interview
Opening Remarks

Gaining entry -- appearance,
confidence, non-threatening

Doorstep technique -- brief, suggest
don’t ask

Introduction -- name, organization,
credentials

Explaining the study -- general, stress
confidentiality
Asking the Questions

Use questionnaire carefully, but
informally.

Ask questions exactly as written

Follow the order given.

Ask every question.

Don’t finish sentences.

Don’t leave any questions blank.
Obtaining Adequate Responses - The
Probe

Silent probe - a pause

Overt encouragement

Elaboration
– How do you mean? Anything else?

Ask for clarification

Repetition
Recording the Response

Record responses immediately

Include all probes (write them in)

Use abbreviations where possible (for
example, R=respondent; DK=don’t
know)
Concluding the Interview

Thank respondent.

Don’t be brusque or hasty.

Immediately after leaving -- write down
any notes about how the interview went.
Types of Questions
Type of Questions

Unstructured

Structured
Structured Questions
Dichotomous
Dichotomous
____Yes
____No
Male Female
Nominal
Nominal
Occupational Class:
1 = Truck driver
2 = Lawyer
3 = etc.
Ordinal
Ordinal
Rank the candidates in order of preference from

best to worst...
___ Bob Dole

___ Bill Clinton
___ Newt Gingrich
___ Al Gore
Interval
Interval
Likert Response Scale
1 2 3 4 5
strongly disagree neutral agree strongly
disagree agree
Interval
Semantic Differential
very some- neither some- very
much what what much
interesting boring
simple complex
uncaring caring
useful useless
etc.
Interval
Guttman (Cumulative) Scale
___Are you willing to permit immigrants to live in your country?
___Are you willing to permit immigrants to live in your community?
___Are you willing to permit immigrants to live in your neighborhood?
___Would you be willing to have an immigrant live next door to you?
___Would you let your child marry an immigrant?

Filter or Contingency Questions
Have you ever smoked marijuana?

yes
no
Have you ever smoked marijuana?

yes
no
If yes, about how many times have

you smoked marijuana?
once
2 to 5 times
6 to 10 times
11 to 20 times
more than 20 times

Try to avoid having more than three
levels (two jumps) for any question.

If only two levels, use graphic to jump
(for example, arrow and box).

If possible, jump to a new page.
Decisions About Question Wording
Can the question be misunderstood?


Questions about ‘nationality’ or ‘marital
status are too vague.

Questions about ‘mass media’ are too
general.

What kind of headache remedy do you
use?
– Want brand name?
– Want type? (Aspirin, pill, or capsule?)
What assumptions does question make?

If you ask what social class someone’s in,
you assume that they know what social
class is and that they think of themselves
as being in one.

Check assumption in a previous question.
Is the time frame specified?

Do you think Congress will cut taxes?

Do you think Congress will successfully
resist tax cuts?

Neither of these specifies a time frame.
How personal is the wording?


Are working conditions satisfactory or not
satisfactory in the plant where you work?

Do you feel that working conditions are
satisfactory or not satisfactory in the plant
where you work?

Are you personally satisfied with working
conditions in the plant where you work?
Is the wording too direct?

How did you feel about being in the war?
- Probably too direct

How well did the equipment hold up in the
field?

How well were new recruits trained?
Some Additional Issues...

Does question contain difficult or unclear
terminology?

Does the question make each alternative
explicit?

Is the wording objectionable?

Is the wording loaded or slanted?
Decisions About Question Content
Is the question necessary/useful?

Do you need the age of each child or
just the number of children under 16?

Do you need to ask income or can you
estimate?
Are several questions needed?

Don’t combine two issues in one...

What are your feelings towards African-
Americans and Hispanic-Americans?

What do you think of proposed changes
in benefits and hours?

Did you get all the info desired...

If you ask about earnings, might not
mention all income

If ask if they’re in favor of public TV, ask
for whom (might not mention it’s because
of Sesame Street for the kids)

Do you need more context to interpret answer...

If you ask about attitudes towards
Catholics, can you interpret this without
finding out about their attitudes towards
religion?

Need more info to determine the intensity...

If they say they support public TV, do
they watch it?

If they would be willing to have their tax
dollars spent on it?
Do respondents have the needed info?

‘Do you think Dean Rusk acted correctly
in the Bay of Pigs crisis?’ -- can’t say if
they don’t know who he is.

In TV surveys, ‘regular’ watchers can be
asked about program content, ‘never
watch’ should not be asked.
Does question need to be more specific?

How well did you like the book?
Versus...

Did you recommend the book to others?

Did you look for other books by that
author?
Is question sufficiently general?

For instance, seasonal specificity.

If you ask what they usually watch and
it’s summer, you’ll get a different answer
than in the winter.
Is question biased or loaded?

Ask about the benefits of a tax cut
without asking about the disadvantages

Ask about the disadvantages of
eliminating welfare without asking about
the benefits
Will respondent answer truthfully?

To find out age or income, give brackets.

To find out about charitable
contributions, ask how much “people you
know” typically give.

To find out what magazines they read,
offer to buy used ones.
Levels of Measurement
The Levels of Measurement

Nominal

Ordinal

Interval

Ratio
Some Definitions
Variable
Variable
Some Definitions
Variable
Variable
Attribute
Attribute Attribute
Attribute
Some Definitions
Variable
Variable Gender
Gender
Attribute
Attribute Attribute
Attribute
Some Definitions
Variable
Variable Gender
Gender
Attribute
Attribute Attribute
Attribute Female
Female Male
Male
Qualities of Variables

Exhaustive -- Should include all
possible answerable responses.

Mutually exclusive -- No respondent
should be able to have two attributes
simultaneously (for example, employed
vs. unemployed -- it is possible to be
both if looking for a second job while
employed).
What Is Level of Measurement?
The relationship of the values that are
assigned to the attributes for a variable
Relationship
Values 1 2 3
Relationship
Attributes Non Technical Supervisor Managerial
Values 1 2 3
Relationship
Variable Level of Job
Attributes Non Technical Supervisor Managerial
Values 1 2 3
Relationship
Why Is Level of Measurement
Important?

Helps you decide what statistical
analysis is appropriate on the values
that were assigned

Helps you decide how to interpret the
data from that variable
Nominal Measurement

The values “name” the attribute
uniquely.
Nominal Measurement

uniquely.

The name does not imply any
ordering of the cases.
Nominal Measurement

uniquely.

The value does not imply any
ordering of the cases, for example,
jersey numbers in football.
Nominal Measurement

uniquely.

The value does not imply any
ordering of the cases, for example,
jersey numbers in football.

Even though player 32 has higher
number than player 19, you can’t say
from the data that he’s greater than or
more than the other.
Ordinal Measurement
When attributes can be rank-ordered…

Ordinal Measurement


Distances between attributes do not
have any meaning.
Ordinal Measurement


have any meaning,for example, code
Educational Attainment as 0=less
than H.S.; 1=some H.S.; 2=H.S.
degree; 3=some college; 4=college
degree; 5=post college
Ordinal Measurement


have any meaning,for example, code
Educational Attainment as 0=less
than H.S.; 1=some H.S.; 2=H.S.
degree; 3=some college; 4=college
degree; 5=post college
Is the distance from 0 to 1 the same as
3 to 4?
Interval Measurement
When distance between attributes has

meaning…

meaning, for example, temperature (in
Fahrenheit) -- distance from 30-40 is
same as distance from 70-80

meaning, for example, temperature (in
Fahrenheit) -- distance from 30-40 is
same as distance from 70-80

Note that ratios don’t make any sense --
80 degrees is not twice as hot as 40
degrees (although the attribute values
are).
Ratio Measurement

Has an absolute zero that is meaningful
Ratio Measurement


Can construct a meaningful ratio
(fraction)
Ratio Measurement


(fraction), for example, number of
clients in past six months
Ratio Measurement


(fraction), for example, number of
clients in past six months

It is meaningful to say that “...we had
twice as many clients in this period as
we did in the previous six months.
The Hierarchy of Levels
Nominal
Nominal Attributes are only named; weakest

Ordinal

Ordinal Attributes can be ordered

Interval

Interval Distance is meaningful

Ratio

Ratio Absolute zero

Likert Scaling
Likert Scaling
Start with a large set of items that you think all

reflect the same construct.
Decide how to manage

multiple tasks.
20
Decide how to manage Decide how to manage
multiple tasks. Manage resources effectively. multiple tasks.
20 4 20
Manage resources effectively. Manage resources effectively.

4 4
multiple tasks.
20
Manage resources effectively.

4
Work quickly and
effectively under
Decide howpressure
to manage
Work quickly and multiple tasks.
49 Work quickly and
effectively under 20 effectively under
Decide how to manage pressure pressure
multiple tasks. 49 Manage resources effectively. 49
20 4
Manage resources effectively. multiple tasks.
Work quickly and Decide how to manage
4 20 effectively under multiple tasks.
Decide how to manage Organize the work when Decide howpressure
to manage
multiple tasks. directions are not specific. multiple tasks. 20
49 Manage resources effectively.
20 39 20
Organize the work when 4 Organize the work when
directions are not specific. directions are not specific. 4
39 Manage resources effectively. Manage resources effectively.
39
4 4
multiple tasks. Work quickly and
20 effectively under
Organize the work when
pressure
Work quickly and directions are not specific. 49
effectively under 4
pressure
49 Work quickly and
effectively under
Work quickly and
pressure effectively under
Work quickly and Work
49 quickly and
effectively under effectively under pressure
Organize the work when 49
pressure pressure
49 directions are not specific. 49
39
directions are not specific. Work quickly and
directions are not specific. pressure
39 49 Organize the work when
Organize the work when Organize the work when directions are not specific.
39
directions are not specific. directions are not specific.
39 39

directions are not specific.
39
Likert Scaling
A group of judges rates each item on a

scale where:
1=strongly disagree
2=disagree
et
he
gS ely
tin c tiv .
Ra ime
effe
t ffe
ctiv
ely
nd
na
ge se na
Ma urce atio
o rm
3=undecided
1 e res f info t.
5 g o an
na ude port s. .
4 Ma ultit im ask ec
ific
a m hat is le t t sp
3 2 ltip no
2 5 an
Sc ide w e mu are
g s
4 de
c na tion
1 3 ma
3 to irec
ow nd
4=agree
2 5 eh he e
cid kw cid
1 4
D e o r de
w nd
3 the na
2 54 n ize el y a tio
a ctiv m
1 4 Org ffe for
3 ee f in
5 tim eo
2 5
ge tud .
na ulti r tant
5=strongly agree
1 4 a m
3 M a po
an im
2 51 Sc at is
1 4 3 wh
3
2 5
1 4
3
2
1
Deciding Which Items to Select
Item-total 
Throw out items that have a low
correlations correlation with the total
(summed) score across all
items.
Internal 
For each item, get the average
consistency rating for the top quarter of
judges and the bottom quarter.

Do t-test; higher t-values mean
the items are better
discriminators, so keep them.
How Does Respondent Use?

Gives a 1-to-5 (or sometimes a 0-to-4)
rating for each item on the scale.

Sum these across all items.

This is why this is sometimes called a
“summated” scale.

don’t forget to reverse any reversal
items.
Guttman Scaling
Guttman Scaling

Also known as
– Scalogram analysis
– Cumulative scaling

Purpose:
– Establish a one-dimensional continuum
– Perfectly predict item responses from total
score
– Seldom perfect in practice
Guttman Scaling
Start with a large set of items that you

think all reflect the same construct.
multiple tasks.
20
20 4 20

4 4
multiple tasks.
20

4
Work quickly and
effectively under
Decide howpressure
to manage
49 Work quickly and
20 4
to manage
20 39 20
39
4 4
pressure
effectively under 4
pressure
49 Work quickly and
effectively under
Work quickly and
49 quickly and
pressure pressure
39
39
39 39

39
Guttman Scaling
Have many judges rate each statement

with respect to whether it is favorable
with respect to the construct -- a yes/no
rating. t
ee
g Sh ely
tin c tiv ly.
Ra ee
ffe
ec
tiv e
ge tim eff and
a na r c es tion
M ou ma
es for
1 g er o f in nt.
No na ude port
a s. .
Ma ultit im ask ec
ific
a m hat is le t t sp
2 ltip no
s an
Sc ide w mu are
Ye No c age n s
de n
3 ma ect
io
to dir
s ow en
Ye No eh wh cid
e
cid ork de
De w nd
he na
s 4 et
Ye No a niz iv ely m atio
g t r
Or ffe
c fo
ee f in
s 5 tim eo
Ye ge ud
No na ltit nt.
Ma a mu or ta
p
1 an im
Ye
s Sc at is
No 3 wh
s
Ye
Guttman Scaling
Lay out a table:

Guttman Scaling
Lay out a table:

Item Item Item Item Item Item
Respondent2 7 5 3 8 ...
7 Y Y Y Y Y Y
15 Y Y Y -- Y --
3 Y Y Y Y -- --
29 Y Y Y Y -- --
19 Y Y Y -- -- --
32 Y Y -- Y -- --
41 Y Y -- -- -- --
6 Y Y -- -- -- --
14 Y -- -- Y -- --
Guttman Scaling
When sorted by row and colum
Lay out a table: will show whether there is
a cumulative scale.
7 Y Y Y Y Y Y
15 Y Y Y -- Y --
3 Y Y Y Y -- --
29 Y Y Y Y -- --
19 Y Y Y -- -- --
32 Y Y -- Y -- --
41 Y Y -- -- -- --
6 Y Y -- -- -- --
14 Y -- -- Y -- --
Guttman Scaling
When sorted by row and colum
Lay out a table: will show whether there is
a cumulative scale.
7 Y Y Y Y Y Y
15 Y Y Y -- Y --
3 Y Y Y Y -- --
29 Y Y Y Y -- --
19 Y Y Y -- -- --
32 Y Y -- Y -- --
41 Y Y -- -- -- --
6 Y Y -- -- -- --
14 Y -- -- YNote the
-- exceptions
--
Guttman Scaling

There are several statistical techniques
for examining the table to find a
cumulative scale.

Because there is seldom a perfectly
cumulative scale, we usually have to
test how good it is.

These statistics also estimate a scale
score value for each item.

For selected items (the final scale), they
simply answer whether they agree or
not (yes/no).

Since each item has a scale value, we
can add these values.

Total score is the sum of the scale items
respondent agreed to.
Thurstone Scaling
Thurstone Scaling - Methods

Equal-appearing intervals

Successive intervals

Paired comparison
Equal-Appearing Intervals
Start with a large set of items that you think all

reflect the same construct.

multiple tasks.
20
20 4 20

4 4
multiple tasks.
20

4
Work quickly and
effectively under
Decide howpressure
to manage
49 Work quickly and
20 4
to manage
20 39 20
39
4 4
pressure
effectively under 4
pressure
49 Work quickly and
effectively under
Work quickly and
49 quickly and
pressure pressure
39
39
39 39

39
Many judges individually sort these into 11 piles
where:
1 = least favorable to the concept

11 = most favorable to the concept
1 2 3 4 5 6
Manage resources effectively. Organize the work when
4 directions are not specific.
39
7 8 9 10 11
multiple tasks.
20
Work quickly and

effectively under
pressure
49
For each item, plot the distribution of pile

numbers.

numbers.
25
20
15
10
0
1 2 3 4 5 6 7 8 9 10 11

numbers.
25
20
15
10
0
1 2 3 4 5 6 7 8 9 10 11
Get the median.


numbers.
25
20
15
10
0
1 2 3 4 5 6 7 8 9 10 11
and interquartile range.

Get the median.
Choose those items that are at equal intervals

(from 1 to 11) and that have the smallest
interquartile range.
1 2 3 4 5 6 7 8 9 10 11
Decide how to manage Organize the work when Decide how to manage Organize the work when Organize the work when
multiple tasks. directions are not specific. multiple tasks. directions are not specific. directions are not specific.
20 39 20 39 39
Work quickly and Work quickly and

effectively under effectively under
Manage resources effectively. pressure Manage resources effectively. pressure
4 49 4 49
The Other Methods

Follow a specific procedure for
sorting/comparing items in terms of
the concept of interest.

Use a simple statistical analysis to
determine scale value for each item.

Select items from across the scale
range.

Respondent checks agree or disagree
for each item.

The respondent’s total score is the
average scale value of all the items
agreed to.
General Issues in Scaling
What Is a Scale?
The assignment...
What Is a Scale?
The assignment...
...of objects...
Are you willing to permit

immigrants to live in your
country?

neighborhood?
Would you let your child

marry an immigrant?
What Is a Scale?
The assignment...
...of objects... ...to numbers...

country?

neighborhood?

marry an immigrant?
What Is a Scale?
The assignment...

country?

neighborhood?

marry an immigrant?
...according to a rule...
What Is a Scale?
The assignment...

country?

immigrants to live in your Cumulative
neighborhood?

marry an immigrant?
...according to a rule...
Scale Versus Response Scale
Scale Response Scale

Is used to collect the
Results from a process
response for an item

Scale versus Response Scale
Results from a process


Results from a process.
response for an
item.
Each item on scale

has a scale value.

response for an
item.
Each item on scale Item not associated

has a scale value. with a scale value.

response for an
item.

Refers to a set of items.


response for an
item.
Refers to a set of items. Used for a single item.

Purposes of Scaling
Hypothesis testing · Is the construct or

concept a single
dimensional one?
Exploration · What dimensions
underlie some
ratings?
Scoring · For assigning values
to responses.
Dimensionality
One-dimensional constructs
Dimensionality
Height
Dimensionality
Height
Shorter Taller
Dimensionality
Height
Shorter Taller
Thirst
Dimensionality
Height
Shorter Taller
Thirst
Less thirsty More thirsty
Dimensionality
Height
Shorter Taller
Thirst
Self-esteem
Dimensionality
Height
Shorter Taller
Thirst
Self-esteem
Less More
Dimensionality
Two-dimensional construct
Dimensionality
Two-dimensional Construct
Academic ability
Dimensionality
Academic ability
Verbal
Dimensionality
Academic ability
Verbal
Worse Better
Dimensionality
Quantitative
Academic ability
Verbal
Worse Better
Dimensionality
Quantitative
Better
Academic ability
Verbal
Worse Better
Worse
Dimensionality
Three-dimensional construct
Dimensionality
Three-dimensional construct
Dimensionality
Semantic Three-dimensional construct

differential
(meaning)
Dimensionality

differential
(meaning)
Evaluation
Dimensionality

differential
(meaning)
Evaluation
Unfavorable Favorable
Dimensionality

differential
(meaning)
Potency
Evaluation
Dimensionality

differential
(meaning)
Potency
Powerful
Evaluation
Powerless
Dimensionality

differential
(meaning)
Activity Potency
Powerful
Evaluation
Powerless
Dimensionality

differential
(meaning)
Active
Activity Potency
Powerful
Passive
Evaluation
Powerless
Types of Scales
One-dimensional 
Easier to understand

Useful for
understanding
multidimensional

Many concepts may be
one-dimensional
Multidimensional 
Reflect multiple
properties
simultaneously
The Major One-Dimensional Scale Types

Thurstone

Likert

Guttman

Basic Survey Methods

Uploaded by

Copyright:

Available Formats

You might also like

Basic Survey Methods

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Basic Survey Methods

Uploaded by

Copyright:

Available Formats

Types of Surveys

Household drop off

Rank the candidates in order of preference from

___ Bob Dole

Likert Response Scale

___Are you willing to permit immigrants to live in your country?

___Are you willing to permit immigrants to live in your community?

___Are you willing to permit immigrants to live in your neighborhood?

___Would you be willing to have an immigrant live next door to you?

___Would you let your child marry an immigrant?

Have you ever smoked marijuana?

Have you ever smoked marijuana?

If yes, about how many times have

Can the question be misunderstood?

What assumptions does question make?

Is the time frame specified?

How personal is the wording?

Is the wording too direct?

Some Additional Issues...

Is the question necessary/useful?

Are several questions needed?

Are several questions needed?

Are several questions needed?

Are several questions needed?

Do respondents have the needed info?

Does question need to be more specific?

Is question sufficiently general?

Is question biased or loaded?

Will respondent answer truthfully?

Attributes Non Technical Supervisor Managerial

Variable Level of Job

Attributes Non Technical Supervisor Managerial

When attributes can be rank-ordered…

When attributes can be rank-ordered…

When attributes can be rank-ordered…

When attributes can be rank-ordered…

When distance between attributes has

When distance between attributes has

When distance between attributes has

Nominal Attributes are only named; weakest

Nominal Attributes are only named; weakest

Ordinal Attributes can be ordered

Nominal Attributes are only named; weakest

Nominal Attributes are only named; weakest

Interval Distance is meaningful

Ordinal Attributes can be ordered

Nominal Attributes are only named; weakest

Ordinal Attributes can be ordered

Nominal Attributes are only named; weakest

Ratio Absolute zero

Interval Distance is meaningful

Ordinal Attributes can be ordered

Nominal Attributes are only named; weakest

Start with a large set of items that you think all

Decide how to manage

Manage resources effectively. Manage resources effectively.

Manage resources effectively.

Organize the work when