Questionnaire Development Best Practices

By Brian Verdine

I created this document as a training exercise for incoming students to help avoid some of the common
pitfalls in creating questionnaires for individual lab projects. It is an attempt to compile a list of things to
ensure that people do not waste their time creating a questionnaire that produces data that is useless
for the purposes of statistical analysis or of very suspect validity or reliability.

I am VERY open to input on this list and would like to continue to flesh this out. It is by no means
comprehensive or complete. I did try to avoid suggestions that might not be generally agreed upon, but
this list was developed to serve psychology research labs, and may be more or less applicable in other
fields of study. If you do have a disagreement or an addition, please comment and I will continue to
edit/revise this posting.

Format
 For questionnaires, if the answer to a question is continuous (e.g., hours of TV watching per
day), allow participants to answer with a continuous number (online questionnaire technology
can typically be set up to limit what can be entered so that people do not enter text).
o You can always recode continuous data if you think it is advantageous to do so, but you
can’t make coded data continuous.
o Categorizing data loses information. There are times you may want to do that for some
particular reason after the fact, but having continuous data to work from is preferable if
possible.
 For non-continuous types of items, it is generally a good idea to use 5- or 7-option Likert scales
and always keep as many variables as possible on the same set of scales (this will help if you
ever want to factor analyze the data or compare across different sections of the questionnaire).
o Only deviate from this if the item cannot legitimately be created that way (e.g.,
questions where participants might need to select more than one option) or another
format is clearly more intuitive (e.g., continuous data like age, height, number of
siblings).
o Try to keep the scales size (5 or 7) constant across questions and across versions of the
questionnaire.
o Generally likert-scale items should not be worded as questions, but instead as
statements that people can agree or disagree with to a varying extent.
 Never ask more than one question within a question and number EVERY item chronologically
and uniquely (i.e., no 14a’s).
 Always consider the best way to format the questionnaire so that it will be fast and easy for
participants to fill out but also easy to extract the data from.

Question Development
 Never create an item without having a firm grasp of what you hope to learn from it.
 Never ask a question in a format that makes the data difficult to quantify or code (e.g., open-
ended prompts like, “Does your child recognize any written letters? Please describe.”)
o If it is hard to code, you will probably never use it, especially if you aren’t sure why you
are asking about it in the first place.
o If you do ask for data like this, be certain you have an idea of HOW you will use it.

http://www.scribd.com/brian_verdine
Page 1 of 2
  Make all questions so that a single question cannot reasonably be coded in multiple ways and so
that the coding will be intuitive and streamlined
o For example, “Please check all that apply. My child: 1) scribbles, 2) draws things I can
recognize, 3) plays with clay, 4) makes things from clay I can recognize.” That question
cannot be coded by entering 0,1,2,3 respectively and still have the scale make any kind
of logical sense based on the question being asked. Playing with clay and drawing
should be two separate questions or possibly each item should be asked as a yes/no
question).
 Try to write questions that are timeless unless it cannot be avoided (e.g., do not reference
specific examples for things that regularly change, like TV shows, unless it is necessary for
participants understanding of the question or your interpretation of the data).
o This will allow you to continue to use a similar version of the questionnaire over time
without having to make significant changes that would change your interpretation of
the question itself.
o Using a questionnaire across time will help give you the power necessary to truly
develop the questionnaire and get reliability/validity information about it.
 Never ask questions about continuous variables (e.g., height) in a way that forces that data into
categories (i.e., short, medium height, tall).

Other Best Practices

 You would not give a participant one trial on a cognitive test and assume it is a good
representation of their skill level. Do not plan to analyze single questions from a questionnaire
and get good data.
o This is why reliable/valid scales use multiple questions that are intended to converge on
a single construct or idea.
o Figuring out if a scale is reliable is not rocket science but it does take data. Online
outlets can help you collect a lot of data quickly without wasting time giving unreliable
scales or questions that have no variability to in-lab participants.
o Validating a scale is something best done against an in-lab measure or another scale
with established validity, but usually this is done after establishing reliability and
removing items that don’t cohere with the remainder of the scales or don’t provide
good information.
o Even if you cannot get enough data to test the reliability/validity of a set of items, at
face value it is more convincing to use multiple items in an analysis that you would
expect to measure the same thing as opposed to basing an analysis on a single item.
 Make approximately half of your items so that they are reverse coded.
 Questionnaires are not “free data” and they do have a participant burden associated with them.
Do not waste your participants’ time with questions you don’t know how you’ll use or with
questionnaires that are only loosely related to your research question.
o This is not responsible conduct of research and is a real violation of the contract you
have with human participants.
o On the other hand, questionnaires usually do provide “cheap data”, so investing the
time up front to make a good questionnaire can REALLY pay off dividends over the long
haul.

http://www.scribd.com/brian_verdine
Page 2 of 2