MODULE 6: PRIMARY DATA GATHERING –
SURVEY
Source: Introduction to Survey Methodology
Sampling Error - Sampling errors stem from the
sampling method used. Easy to deal with
mathematically and can be relatively estimated
and resolved by increasing the sample size
 Researchers must initially identify their
population of interest and then clearly
define their unit of analysis and what
elements will best serve the aim of their
study.
 By using non-probability sampling, bias
will naturally be introduced into the
research, and no generalization will be
possible.
 Using probability sampling that relies on
the principle of randomness will provide
a more representative sample, one that
better reflects the target population
and thus enables generalizations from
the sample to the larger population
Non-Sampling Error - typically been seen as too
difficult to estimate, and it has been assumed
that their effect on the results would be
minimized if samples were big enough and
properly representative.
 Non-sampling errors, on the other
hand, tend to be more complex, and
they require researchers’ detailed
attention, as they may creep into each
and every stage of the data collection
 Non-sampling errors can come from a
multitude of sources—it is safe to say
they can comprise about 95% of the
TSE.
 Response Error - response error can
stem from, among other things, social
desirability, visibility, the degree of
sensitivity of a specific item, the order
of the questions, the way in which a
specific item is constructed
 Non-Response Error - non-response
errors can stem from simple refusals to
answer questions or they can come as a
result of a failure to locate participants
who were originally sampled when it
comes time to complete the study. Non-
response can be for the entire survey,
but it can also be for specific
questionnaire items.
Total Survey Error
 combined total of both sampling and
non-sampling errors, should be the
dominant paradigm for developing,
analyzing, and understanding surveys
and their results
 The sum of the squares of both the
sampling error and the non-sampling
error is equal to that of the squared
total survey error—in short, the TSE
becomes much bigger than each of its
components.
Source: Classification of Surveys
Classification Criteria
1. Who: The Population
2. What: The Topic
3. By Whom: Survey Agency and Sponsor
4. How: Survey Mode
5. When: Cross-Sections and Panels
6. Where: Regional, National, Cross-
National and International Surveys
Who: The Population
 Target Population – finite set of the
elements (usually persons) that will be
studied in a survey.
 Frame Population - set of persons for
whom some enumeration can be made
prior to the selection of the survey
 sample, i.e. who can be listed in the
sampling frame.
 Survey Population - set of people who,
if they have been selected for the
survey, could be respondents
 Unit Non-Response - failure to collect
data from units belonging to the frame
population and selected to be in a
sample.
 Response Rate – percentage of selected
units who participate in the survey
What: The Topic
 Omnibus Survey - data on a wide
variety of subjects is collected during
the same interview
 Objective Questions - home turf of
official statistics and cover issues like
labour situation, education, living
conditions, health, etc.
 Subjective Questions - collect
information on values, attitudes, and
the like
When: Cross Sections and Panels
 Cross Sections – data are collected only
once.
 Repeated Cross Sections - survey is
repeated at regular intervals
 Longitudinal Panel – the same group of
respondents is approached at regular
time intervals.
 Rotating Panel - New panel members
participate in a fixed number of waves.
A group of new panel members is
recruited for each wave, making it
possible to draw conclusions on
individual changes and on the
population at a given moment.
Where: Regional, National, Cross-National and
International Surveys
 Regional Survey - A survey among the
inhabitants of a specific community
 National Survey - When all inhabitants
of a country belong to the target
population
 International Survey - sample from
multiple countries and the target
population is the combined population
of the countries under study.
 Cross-National Survey - an independent
sample is drawn in each participating
country, and the results of the national
data files are combined afterwards into
a harmonized cross-national data file.
 Input Harmonization - the instrument is
as identical as possible in each
participating country: the same
fieldwork approach, the same survey
mode, the same questions (but
translated), and so forth.
 Output Harmonization - allows
countries to use their preferred survey
mode.
 Ex-Post Output Harmonization -
different questions can be used, or that
some countries can derive variables
from questionnaires and others from
registers, as long as the same
definitions are used.
 Ex-Ante Output Harmonization – the
questionnaire has to be identical in
each country, but that the data
collection method may differ.
Source: Questionnaire Design and Surveys
Sampling
Statistical Inference - take a random sample
from a population and then to use the
information from the sample to make
inferences about particular population
characteristics such as the mean (measure of
central tendency), the standard deviation
(measure of spread) or the proportion of units
in the population that have a certain
characteristic.
Sampling Distribution - used to describe the
distribution of outcomes that one would
observe from replication of a particular
sampling plan.
Standard Deviation - number that indicates
how much, on average, each of the values in the
distribution deviates from the mean (or center)
of the distribution.
Variance - the average squared deviations
about the mean
Confidence Interval - used to express the
uncertainty in a quantity being estimated
Sources of Errors
 The use of an inadequate frame.
 A poorly designed questionnaire.
 Recording and measurement errors.
 Non-response problems.
Sampling Techniques
1. Random Sampling - Random sampling
of size n from a population size N
2. Stratified Sampling - can be used
whenever the population can be
partitioned into smaller sub-
populations, each of, which is
homogeneous according to the
particular characteristic of interest.
3. Cross-Sectional Sampling - Cross-
Sectional Study the observation of a
defined population at a single point in
time or time interval.
4. Quota Sampling - availability sampling,
but with the constraint that
proportionality by strata be preserved
Source: Designing a Survey
1. Choice of a Topic
 Descriptive Research - merely
give the distribution of
responses of people on some
specific questions such as
satisfaction with the economy,
government, and functioning of
the democracy.
 Explanatory Research - to
determine the reasons for the
satisfaction with the
government or the popularity
of a politician (experimental or
non-experimental)
2. Choice of the Most Important
Variables
 In the case of a descriptive
study, it is directly determined
by the purpose of the study.
 In the case of an explanatory
study, it makes sense to
develop an inventory of
possible causes and to develop
from that list a preliminary
model that indicates the
relationships between the
variables of interest.
3. Choice of a Data Collection Method
 Modes of data collection differ
in their cost of data collection,
where personal interviewing is
the most expensive, telephone
interviewing is less expensive,
and mail interviewing is the
cheapest.
4. Choice of Operationalization
Operationalization - translation of the
concepts to the questions
5. Test of the Quality of the
Questionnaire
  Check on face validity
 Control of the routing in the
questionnaire
 Prediction of quality of the
questions with some
instrument
 Use of a pilot study to test the
questionnaire
6. Formulation of the Final Questionnaire
 After corrections in the
questionnaire have been made,
the ideal scenario would be to
test the new version again.
7. Choice of Population and Sample
Design
 Sampling - procedure to select
a limited number of units from
a population in order to
describe this population.
 Sampling Frame - a list of
names and addresses of
potential respondents
8. Decide About the Fieldwork
 Number of interviews for each
interviewer
 Number of interviewers
 Recruitment of interviewers:
where, when, and how
 How much to pay: per hour/per
interview
 Instruction: kind of contacts,
number of contacts, when to
stop, and administration
 Control procedures: interviews
done/not done
 Registration of incoming forms
 Coding of forms
 Necessary staff
Source: A Step-by-Step Guide to Developing
Effective Questionnaires and Survey
Procedures for Program Evaluation & Research
Survey Errors
1. Sampling Error (How representative is
the group being surveyed?)
2. Frame Error (How accurate is the list
from which respondents are drawn?)
3. Selection Error (Does everyone have an
equal chance of being selected to
respond?)
4. Measurement Error (Is the
questionnaire valid and reliable?)
5. Non-response Error (How is the
generalizability of findings jeopardized
because of subjects who did not reply?)
Steps
1. Determine the purpose
2. Decide what you are measuring
3. Who should be asked?
 “sampling error” is reduced by
using a large, random sample or
conducting a census;
 “frame error” is minimized by
making sure the list of potential
subjects is current and
accurate;
 “selection error” is avoided by
eliminating duplication from
these lists.
4. Consider the audience
 Age
 Education level
 Familiarity with tests &
questionnaires
 Cultural bias/language barrier
5. Choose an appropriate data collection
method
  Mailed
 Telephone
 Personal (face-to-face)
interview
 Web-based
6. Choose a collection procedure
 Confidential - Name or other
identifiers are used to follow-up
with nonrespondents or match
data from pre-test/posttests.
 Anonymous - Name is not
asked of respondents.
7. Choose measurement scale and scoring
 Fixed-Response
 Open-Ended (Narrative
Response)
8. Title the questionnaire
 Include a brief purpose of the
study (one sentence or phrase)
 Consider including a simple
graphic that depicts the
purpose of the evaluation or
program
9. Start with non-threatening questions
10. Include simple instructions
11. Use plain language
12. Be brief
13. Put the most important questions first
14. Make sure questions match the
measurement scale selected, and
answer categories are precise
 Make sure answer choices
correspond to the questions,
both in content and grammar.
 Use exact numbers when
possible (instead of Frequently,
Rarely)
 Explain the “rule” being used
with clear instructions and to
apply the rule consistently
throughout the questionnaire.
 Define time frames if necessary.
 If you are using a continuum
scale with numbers to
represent concepts, make sure
to “anchor” at least the top and
bottom of the scale with terms
that describe meanings of the
numbers. (For example, 1 =
Low, 10 = High).
 Balance the “negative” or “low”
answer choices (both in number
and degree) with “positive” or
“high” choices on the scale.
 An even number of answer
choices doesn’t give the
respondent an easy, “middle”
choice. If you want to offer a
“neutral” or “no opinion”
choice, then do it by design, not
by accident.
15. Ask only one question at a time
16. Avoid loaded questions
17. Arrange in a logical order
18. Minimize open-ended questions
19. Provide space to tell more
Source: Writing Good Questions
Reliability - refers to the consistency in
responses across different respondents in the
same situations
Validity - refers to the extent that the measure
we are using accurately reflects the concept we
are interested in
Key Elements of Good Questions
1. Specificity - Does the information
targeted by the question match the
target of the needed information?
2. Clarity - The core vocabulary of the
survey question should be attuned to
the level of understanding of the
participants.
3. Brevity - questions should be stated in
as straightforward and uncomplicated a
manner as possible, using simple words
rather than specialized ones and using
as few words as possible to pose the
question
Common Question Pitfalls
1. Double-barrel Questions - These are
created when two different topics are
specified in the question, essentially
asking the respondent two questions in
one sentence
2. Loaded/Leading Questions - These
originate when question wording
directs a respondent to a particular
answer or position.
3. Questions with Built-In Assumptions -
Some questions contain assumptions
that must first be considered either true
or false in order to answer the second
element of the question
4. Double-Negative Questions - Questions
that include two negatives not only
confuse the respondent, but they may
also create a level of frustration
resulting in nonresponse.
Question Types
1. Open-Ended Questions
 respondent’s answer could
obviously cover many different
areas, including some the
researchers had not previously
considered.
 particularly well-suited to
exploring a topic or to gathering
information in an area that is
not well known.
2. Close-Ended Questions
 format of closed-ended
questions is more defined, and
has been standardized to a
greater extent than the open-
ended style
 allow researchers to provide
greater uniformity to the
responses and to easily
determine the consensus on
certain items, but only on those
items that were specified by the
answers provided.
Source: TIPS FOR DEVELOPING SURVEY
INSTRUMENTS/QUESTIONNAIRES
Instrument Types
 Pre-Test/Post-Test - used when you
would like to observe whether a desired
change occurred as a result of your
efforts.
 Focus Groups - small, deliberately
chosen groups of people that are
interviewed together to observe
reactions to the topics you propose.
 Surveys - sets of standardized questions
(questionnaire) that are administered to
selected individuals or groups of
individuals.
Question Types
  Close-Ended Questions - those that list
pre-set answers for respondents
 Open-Ended Questions - those that list
pre-set answers for respondents
 Scales - social science research
technique used to measure the
qualitative aspects of the group of
people you need information from.
 Likert Scales - type of scale that asks
respondents to indicate the level that
they agree or disagree (generally, from
‘strongly agree’ to ‘strongly disagree’)
about a statement.
Measurement Concepts
 Validity - refers to whether or not your
survey (or other form of measure) is
actually measuring what it is supposed
to
 Reliability - when your survey (or
survey question) produces consistent
Survey Instructions
results when used to measure the same
thing over and over.
 Bias - unfair preferences or dislike of
something.
Survey Design Matrix
1. Indicator data/question to be
answered
 This will most likely come from
Rating Scales
the indicator you are reporting
against or perhaps a question
your team or the client wants to
know.
2. Information Source
 Who can answer the question?
3. Sampling
 Will we be able to ask everyone
to answer the question? If not,
how will we choose?
4. Data Collection Mode
 Face-to-face? Through email?
The internet? Phone?
5. Data Collection Period
 After the exchange ends? After
a workshop session is
completed?
6. Survey Questions
 What question(s) will your
survey ask to answer the
question?
7. Survey Testing
 What information did pre-
testing the survey reveal? Do
we need to clarify wording?
Were any questions not
answerable?
 Who will use the information
 How the information will be used
 Whether the responses will be
anonymous
 The approximate time the survey will
take to complete
 Assure respondents of anonymity and
confidentiality (if applicable)
1. Numeric rating scales - where
respondents are asked to rate a topic
based on a set of numbers;
2. Graphic rating scales - which look at
behaviors or performance (i.e.,
leadership, teamwork, performance);
3. Descriptive graphic rating scale - where
respondents are asked to place a mark
along a line that depicts one extreme to
the other.