Constructing questions for interviews & questionnaires

Theory and practice in social research William Foddy

Preface
These slides include one theoretical framework for constructing questions and a small set of good and bad practices for constructing questions This is not only or complete set, but gives hints about what should you think when constructing questions.

Errors in gathering data through survey procedures are

Responders failure to understand questions an intended A lack of effort or interest on the part of responders Respondents unwillingness to admit certain attitudes or behaviours The failure of respondents memory or comprehension process in the stressed condition of the interview Interviewers failures of various kinds (e.g. the tendency to change wording, failures in presentation procedures and the adoption of faulty recording process

Examples that illustrate the inadequate of many of the questions

Factual questions sometimes elicit invalid answer: As simple as age Relationship between what respondents say and what they actually do in not always very strong: E.g. racism: Questions: Do you let a black person inside a bar? Do they really? Responders attitudes, beliefs, opinions, habits interests often seem to be extraordinary unstable Small changes in wording sometimes produce major changes in the distribution of responses Respondents commonly misinterpret questions Answer to earlier question can affect to later question Order of options changes the respondents: e.g. middle option Format of question: Open vs. close Respondents answer even if they don't know about the topic Cultural context affects

The principal assumptions that have defined the general orientation adopted by survey researcher in the past

The researcher has clearly defined the topic about which information is required Respondents have the information that researcher requires Respondents are able to access the required information under the conditions of the research situation Respondents can understand every question as intended Respondents are willing (or at least, can be motivated) to give the information Responses are more valid if they know why that is asked Answers are more valid if they are not suggested The research situation does not affect to the results The process does not affect the respondent All responses are meaningfully comparable

The traditional survey model

A carefully standardised physical stimulus (i.e. question)

Researcher/ Interviewer
A response (i.e. answer) expressed in term of a standardised format provided by the researcher

Respondent

The key issue: the comparability of aswer

The reseacher must be clear about the nature of the information required and encode a request for this format The respondent must decode this request in the way the researcher intends it to be decoded The respondent must encode an answer that contains the information researcher has requested The reseacher must decode an answer as the respondent intendedd it to be encoded

Symbolic interactioinist theory

Human beings interpret and define each other's actions. Not only simple stimulus-response Human beings can be objects of their own attention. Argue themselves, take pride themselves Conscious social behavior is intentional behavior. Situation affects. Interpreting, planning and acting are ongoing process which begin anew at every stage of a social process In situation there is a role and person wants to hold on it. Always exists In brief: social situation is constantly negotiating a shared definition of the situation; taking one another's viewpoint into account; and interpreting another's behavior

A model of symbolic interactionist view of question answer behaviour

Interviewer
Encodes question, taking into account own purposes and presumptions/knowledge about the respondent, and perceptions of the respondent's presumptions/knowledge about self

Responder
Decodes question, taking into account own purposes and presumptions/knowledge about the interviewer, and perceptions of the interviewer's presumptions/knowledge about self

Encodes question, taking into account own purposes and presumptions/knowledge about the respondent, and perceptions of the respondent's presumptions/knowledge about self

Encodes question, taking into account own purposes and presumptions/knowledge about the interviewer, and perceptions of the interviewer's presumptions/knowledge about self

The implication of the symbolic interaction theory for social research

Roles Respondents try to find a mutually shared definition of the situation Interpretation of researcher acts Clues of what kind of information researcher wants

Other
Multidimensionality. Nothing is onedimensional Level of generality in the statements Level of generality in the topic Utterance frame: descriptive, explanatory or evaluative

Defining topic properly

Interviewer
Encodes question. The need to clearly specify the information that is required. The assumption that the respondent have the required information. The assumption that the respondents can access the information

Responder

Decodes question

Decodes answer

Encodes answer

Researcher has clearly defined the required information

Defined topic
Global vs. local (29) Dimensions (economic, technical...)
Is PL good thing: Marketing vs. Technical

E.g. better social secure in cost of taxes

Researcher has clear idea what kind of information is required

Responder have required information

Hypothetical question => hypothetical Answer Willingness to answer vs. information to answer Opinions vs. facts
Grounded opinions Serious opinions

Responds are capable of verbalising information.

Not hurry, give time to answer

Formulating intelligible requests for information

Interviewer
Encodes question. Formulating an intelligible request for information. Defining key terms for respondents and virtues of simplicity and breity

Responder

Decodes question

Decodes answer

Encodes answer

The meaning of individual words

Context specific meanings
Ambiguities words Different meanings Cultural decencies

Relative difficulty of words

foreign words

The operation of unintended nuances associated with apparently similar words

Forbid vs. allow

The meaning of concepts

Lack of empirical references
How should be Difficul, unclear, wide concepts

Context decencies (often, usually etc.) Concrete: Not: often, quite often, seldom. Use 1,2,3,4,5

Structural complexities
Number of words Less is better, but not complex ones Grammatical
Asking too much in one question divide & conquer not what you like this and that!

Not too much negatives: Double is bad

Good practices
Descriptive introduction: I'd like to describe Explain before question
The addition of clauses, phrases and instructions

Anecdotes Education of the respondent affects

Contextual influences on respondents' interpretation of questions

Interviewer Responder

Decodes question Encodes question. Contextual clues that influence respondents interpretations

Decodes answer

Encodes answer

Clues afforded by either the Question itself or its components

Clues associated with the question
Some existing idea always Leading questions

Clues in components
Additional phrases Note: Vegetables e.g. spinach not associated vegetarian in general as intended

Clues in response options.

Approve or disapprove - not only approve in question Scale Pre-set response options as memory clues
May not remember other options Information range covered by the options. Who counts own averages? Abstract: no knowledge of normative levels

Response biased with number of options

Respondent favor first when reads, last when hears - vary the order Let responder tell, if no-one match

Impact of preceding questions

Influence associated with prior Q. Issues:
Does one have any knowledge? How deep is knowledge or how thorough How one interprets

Psychological need for consistent

Need to stay in one opinion

Even-handedness
Fair for all sides

Impact of preceding questions (contd)

The impact of the previous answers The impact of the overall situation
Semantics of human mind. Willingness to give information

Contextual influences on respondents' interpretation of questions

Interviewer Responder

Encodes question.

Decodes question

Encodes answer

Decodes answer

The kind of answer given is defined by the dimension of the response framework that is employed

Descriptive accounts
How to describe the situation Perspectives

Level of social generality of responses

You? Singular or plural?

Explanations
Respondent can frame an explanation in many different ways Why did you do X
Causal antecedent - what caused to do X Goal antecedent - purpose for X Enablement factors - how possible Causal consequences - what happened after X The researcher's expectations

Evaluations
Always relative Standards necessary does not exists Evaluative standard that are external to the question
Pleasant/easy flight - pilot vs. passenger

Evaluative standard that are built in to the question

Agree -disagree

Limitations of human memory

Limitations of human memory

Long term memory problems
Nobody remembers all Ask from present to past, not past to present.

Short term memory problems

Simple questions

Filters

Filters
Establishing the relevance of questions to respondent
Respondents tend to answer all the questions
I don't know enough

Don't know vs. don't have opinion Position of filters

Middle and dont know are problematic Middle category at all

subjective viewpoint to strongly (dis)agree

How strongly?

Reducing question threat

Reducing question threat

Always some threat Biased Refuse

How?

Casual, Do you happen to have? Imputation of deviance, You know everyone does. Anonymous Lessen psychological immediacy of the Q (e.g. other instead self. Numeric coding of alternatives) Decrease specificity of the information called (e.g. broad response categories) Adopt knowing so that respondents have to confirm rather than volunteer 'Kinsey' straight at eyes and ask Adopt indirect so that respond give answer without knowing Place threatening Q at the end of series

How? (contd)
Door in the face: Ask direct, if does not answer ask indirect/about e.g. salary. Ask long, dont hesitate top repeat, give time and encourage to use time The definition of Q threat

Threat causes bias

Topics that are desirable and over-reported
Be good citizen Be well informed and cultural person Have fulfilled of moral and social responsibilities

Topics that are socially undesirable and under-reported

Illness and disabilities Illegal and contra-normative behavior Financial status

Feeling of guilt or personal fears Threat associated with the nature of the relationship between interviewer and respond
Social equity Fear of political or economical sanctions

Open vs. closed question debate

Open vs. closed question debate

Coding responses to open questions Formulating response option for closed question

Evaluation of the use of open questions

Open Q doesnt suggest answer -- or does it?
Respondents use of "probes"

Indicate level of knowledge

Assumptions
will answer open Q if knows don't try to answer open Q if don't know wil answer closed question if don't know

Answers indicate the salience of the topic in the responder's mind

Is there evidence?

Avoid format effect

Not in the middle! Is it in the closed?

Evaluation of the use of open questions

Allows complex motivation
Indicates more how respondent has interpreted it Indicate motivation that have influenced respondent's orientation to the topic Indicates the frame of reference

Problems associated with probing inadequate answer

Turns to close

Problems associated with coding response!

Evaluation assumptions associated with closed questions

Answers the question in the same way meaningfully comparable? Easier to answer More easily analyzed
Interpretations

Problems associated with recording responses to closed questions

Start with general instructions Explain why to answer

Measuring attitudes

Measuring attitudes
List of respond alternatives p. 153 Define topic clearly Applicability of the topic to respondents has to be established
Dont know

Respondents has to know what sort of answers they should know

Level of generality

Specs of standards
"Strongly agree"

Measuring attitudes
Stimulus centered effects
Number of categories
7 +/- 2

Anchoring effects of he category labels

The word in the positive/negative end?

Problems in the batteries of rating scales

Meaning can alter Some category
long list. Don't know last when answering the first

Ambiguity

Checks to ensure that questions work as intended

Checks to ensure that questions work as intended

Editing rules (list p. 184) Piloting Question
Observation hard, but must be done in the beginning to ensure that questions work e.g. had to repeat?

Question testing
Rephrase the question in responder's own words Double interview
Come in the beginning

Allow aloud thinking

Tieto vai mielipide Halu vastata vai Vihjaa ett pitisi olla? vastaus olemassa?

Esim KKK

Koponentti parametrinen vs komponetin mukanaolo parametrista Mik on komponentti?

Mik on tyypillinen tuote?

Kuinka monta parametroitua komponenttia on tyypillisess toimitettavassa tuotteessa (0=ei kytet)? Miten paljon parametreja on tyypillisess parametroidussa komponentissa? Parametrien mrn vaihteluvli? Millaisia parametreja komponenteissa on? Millaisia arvoalueita parametreilla on (kokonaisluku, arvoalue, yksi kokonaisluku joukosta , yksi annetuista vaihtoehdoista, reaalilukuja, joukkoja)? Mit muita?

Strukturaalinen kompleksisuus, liian monimutkainen lause: Kun toimitatte tuotteen, tuotteessa om komponentteja. Kompoenenteilla voi olla toimituskohtaisia parametreja. Onko olemassa tllaisia komponentteja, joissa on parametreja? Montako tllist kompoenttia, jossa on parametreja, tuotteessa on?

Jos ymmrsi aluksi vrin, halu konsistenttiin ilmaisuun