POLISCI 2325F

Research Design in Political Science

Lecture Notes
Professor Turgeon
Fall 2023, Tuesday, 3:30pm - 5:30pm

WEEK 11 (NOVEMBER 14):

SURVEYS

Chapter 18 - Surveys:

Survey → A method for collecting information from a sample of individuals by asking

questions in order to construct quantitative descriptors of attributes of the larger population
from which the sample is drawn.
- Surveys can be categorised by… (Types of Surveys):
a) The mode in which they are applied;
- Ex. paper, telephone, in-person, and online.
b) Their breath or scope; and
- Ex. cross-sectional versus longitudinal.
c) The population that they survey.
- Ex. Expert and Non-Expert.

1. Expert Surveys → surveys of individuals with a specialised knowledge on a particular

subject related to the theme of the survey.
- These experts may include academics, researchers, and professionals (ie. lawyers,
judges, curators, businessmen, government officials, etc.).
- Experts are not chosen through random sampling, but rather through non-random,
purposive sampling.

2. Non-Expert Surveys → Surveys of individuals that do not have specialised knowledge on

a particular subject.
- More common than expert surveys.
- Type of survey that most people think of when they hear “survey.”
- Respondents for non-expert surveys may be chosen through either non-random or
random techniques.
- Only random sampling, however, is likely to yield samples that are
representative of the population from which they are drawn.
- Researchers can collect…
a) Subjective information on people’s opinions, attitudes, perceptions, desires,
etc; and
b) Objective information on people’s age, income, education, religion, and
ethnicity.

Utility of Surveys:
- Surveys are unparalleled in their ability to provide comparable and generalizable
data about the cognitions, emotions, and behaviours of individuals.
 - Cogitions → the mental processes of acquiring knowledge and
understanding through thought, experience, and the senses.
- Also refers to the products of these processes (ex. Judgements and
Evaluations).
a) Comparability → Surveys facilitate comparisons across individuals for 2 reasons:
1. Every respondent is asked the same questions in the same order;
2. Most survey questions are close-ended.
- Close-Ended Questions → Questions with defined response
categories.
- May be ordered or continuous (ex. Agreement versus Age).
- May be dichotomous or multichotomous (ex. Yes/No versus
Race/Ethnicity).
- Facilitate comparisons across respondents by standardising
the respondents’ answers.
b) Generalizability:
- Surveys have a high potential to yield generalizable results because surveys
are amenable to random sampling and large-N designs:
1. Random Sampling Techniques → Facilitate generalizability because
they reduce the odds that the survey participants are systematically
different in some way from the larger population from which they are
drawn and are purported to represent.
2. Random Sampling LARGE-N Design → Increases the likelihood that a
survey will be representative of the larger population that it calais to
represent because when the number of observation in a study is large,
there is a greater potential for a range of views representing this
population to be captured than when it is small.
- Threats to Generalizability:
1. Low-response rates;
2. Survey attrition;
- Ex. Participants dropping out of longitudinal surveys after
several rounds.
3. Interviewer bias; and
- Ex. Observer Bias.
4. Social desirability effect.
- Ex. Guinea Pig Effect.
- Strategies researchers can adopt to limit the threats to generalizability:
a) Reducing the burden that surveys place on respondents;
b) Ensure that respondents perceive a personal benefit in completing the
survey;
c) Protecting the privacy of the participants; and
d) Sending reminders.

Survey Questionnaire Design:

- Researchers must put a great deal of thought into the construction of questionnaires
to make sure that the survey responses are comparable and generalizable, but also
that they are complete and accurate.
- Before applying the surveys to the populations of interest, researchers should
also pre-test their questions and questionnaires.
 - There are 6 rules that researchers can use to facilitate writing effective questions and
questionnaires:
1. Clarity.
2. Terminology.
3. Conciseness.
4. Demand.
5. Efficiency.
6. Sensitive Issues.

(1) Clarity:
- Survey questions have to be very clear so that the respondents interpret the
questions in the way the researcher intended them and provide accurate answers to
the question as a result.
- The questions also need to be clear so that all respondents interpret the
questions in the same way.
- If they do not, their responses will not be comparable.
- If the respondents do not think the questions are clear, they may also
skip them entirely.
- To ensure that the questions are clear, researchers ought to use ordinary, but precise
language, as well as simple and straightforward sentence constructions.
- Researchers should also avoid:
a) Double Negatives:
- Ex. “Please identify which of the following statements is not
inaccurate.”
b) Double-Barreled Questions:
- Ex. “Do you support increased spending on education and
health?”

(2) Terminology:
- Clarity demands that researchers define their terms.
- Researchers must define the terms that they use because participants may
not be familiar with them, or understand them in the same way as either the
researcher or other respondent.
- Sometimes, it is best to avoid using a term entirely.
- Ex. Instead of asking participants how democratic they think their country is,
researchers may ask respondents: “To what extent do laws in your country
make it difficult for parties to get on the ballot?”

(3) Conciseness:
- In striving to make questions clear and precise, researchers can unintentionally make
questions long and verbose.
- Verbosity can deter respondents from answering questions.
- To avoid it, researchers should eliminate redundant and unnecessary words
or phrases from their questions.
- Conciseness is important not only for individual questions, but also for surveys
overall.
 - To reduce the burden that surveys place on respondents, researchers should
ask the fewest number of questions and, more specifically, the fewest number
of open-ended questions, as possible.

(4) Demand:
- Surveys should not place high demands on respondents in terms of their time or
knowledge because such demands reduce the likelihood that respondents will
answer a question.
- A high demand in knowledge can also reduce the accuracy of the responses.
- When possible, researchers should avoid asking very specific questions to
avoid placing a high burden on respondents’ time and knowledge.

(5) Efficiency:
- Streamlining questions can also make them less taxing for respondents.
- Eliminating extraneous words (ex. “a,” “an,” and “the ''), as well as redundant
language, (ex. “added bonus,” “close proximity,” and “foreign imports”) is
important in this regard.
- Another important strategy for streamlining survey questions is to make sure
that the response categories follow a common syntax and incluse in the text
of the question any words common to all response categories.

(6) Sensitive Issues:

- Survey questions on sensitive issues present a particular challenge for researchers.
- Respondents may provide inaccurate responses to these questions because
they are embarrassed, or worried that if their responses become known their
physical safety or livelihood could be harmed.
- Or the respondents may simply be so offended by the questions that they
refuse to answer them or complete the survey.
- Researchers can adopt strategies to elicit honest responses to sensitive questions:
a) Desensitise the environment in which these questions are asked.
- Anonymity → researchers do not collect any identifying information
from the respondents.
- Confidentiality (when anonymity is not an option) → researchers only
disclose identifying information under mutually agreed upon terms.
- Creating a safe, comfortable, and non-judgemental environment for
the respondent when the surveys are applied in-person.
- Ex. By matching the demographic characteristics of the person
administering the survey to those of the respondent.
b) Desensitise the questions asked themselves by…
- Avoiding judgemental and pejorative language.
- Phrasing questions and their response categories in broad terms.
- Tapping respondents indirectly about sensitive issues by asking them
about related behaviour or traits which are likely to be true if the
person engaged in the sensitive act.
- Not as useful as having honest answers to direct questions,
but is preferable to having dishonest answers to direct
questions.
c) Researchers can also adopt a list experiment.
(c) List Experiment:
- Procedure:
- The respondents are randomly assigned to one of the two groups;
- Respondents in both groups are asked to identify from a list the number of
statements (not which ones) they agree/disagree with or apply to them (for
behaviours);
- All but one statement are the same across the two lists.
- The one that is different is related to the sensitive issue.
- All that is recorded is the number of statements a respondent
agrees/disagrees with or applies to them.
- Ex. none, one, two, …
- By comparing the average number of items from both lists, researchers can
estimate the proportion of people in the population that agrees/disagrees with
the sensitive issue or what proportion of people in the population practise the
sensitive behaviour.
- Key Element = Respondents from the same population are randomly assigned
to either the control or treatment list.
- Because respondents from the same population were randomly assigned to
receive one list or the other, there are no reasons to believe that the average
number of items that people agree/disagree with or apply to them from both
groups is different.
- Especially the larger the groups.
- Thus, any difference in the average number of items from both lists has to be
attributed to people in the treatment group including the sensitive item in their
count.
- It is NOT TRUE that all of the non-sensitive statements must be statements that
everyone in both groups would agree or disagree with.
- On the contrary → We actually do not want everyone to either agree or
disagree with all non-sensitive items or else the list does not guarantee
privacy anymore.

Example - List Experiment:

- Turgeon and Habel (2022) measured support for the use of race quotas in university
admissions in Brazil in 2017.
- The researchers were aware that white respondents might fear appearing
racially prejudiced if they did not show support for race quotas in university
admission.
- Thus, there were reasons to believe that their answer would suffer from social
desirability bias. To work around this problem, they conducted a list
experiment.
- Control List: “Now, let’s talk about some of the federal government’s programs,
actions, and undertakings. Please indicate HOW MANY of the federal government’s
programs, actions, and undertakings from the following list you approve. We do not
need to know which of these programs, actions, or undertakings, we are merely
interested in the number of items from that list that you approve. Would you say that
you approve of 1, 2, 3, 4, (5) or none of these programs, actions, or undertakings?”
1. The privatisation of airports;
 2. Programa de Aceleração do Crescimento, PAC (Large federal investment in
infrastructure program);
3. Bolsa Famılia (Brazil’s largest social welfare program);
4. Programa Mais Medicos (A program to increase the numbers of doctors, with
new doctors mostly from Cuba).
- Treatment List: “Now, let’s talk about some of the federal government’s programs,
actions, and undertakings. Please indicate HOW MANY of the federal government’s
programs, actions, and undertakings from the following list you approve. We do not
need to know which of these programs, actions, or undertakings, we are merely
interested in the number of items from that list that you approve. Would you say that
you approve of 1, 2, 3, 4, (5) or none of these programs, actions, or undertakings?”
1. The privatisation of airports;
2. Programa de Aceleração do Crescimento, PAC (Large federal investment in
infrastructure program);
3. Bolsa Famılia (Brazil’s largest social welfare program);
4. Programa Mais Medicos (A program to increase the numbers of doctors, with
new doctors mostly from Cuba);
5. The quota system for Black and Indigenous students in federal
university admissions.

Survey Administration:
- Survey Mode → means by which the survey is administered.
- May be administered either in person, on paper, by telephone, or via the
internet.
- In deciding how to administer their surveys, researchers need to consider 3 major
issues:
1. The Cost:
- In-Person > Telephone > Online.
2. The Response Rate:
- Administer survey according to the method that is easiest for the
greatest number of participants in the sample.
3. The potential for Selection Bias:
- Ie. Is there a potential for selection bias that arises from the mode of
administration?

Analysing Surveys:
- 3 things to consider when analysing surveys:
1. Response Rates:
- A high response rate = necessary for a survey to be representative of
the population from which the survey is drawn.
2. Weights:
- Sample weights are used to create more representative samples,
which can occur when response rates are lower in some groups more
than in others.
- Sample weights adjust for the over-or-underrepresentation of
certain groups of individuals by altering the weight assigned to
each observation in the sample.
3. Margin of Sampling Error:
- Indicates how much the results of a survey question may differ due to
chance compared to what would be found if the entire population was
surveyed.
- The smaller the Margin of Sampling Error = The Better!
- Margin of Sampling Error is based on the sample size and is smaller
for larger sample.