Pyc2616 10050590

© 2023 University of South Africa
All rights reserved
Printed and published by the

University of South Africa
Muckleneuk, Pretoria
PYC2616/1/2024-2030
10050590
InDesign
PR_Tour_Style
CONTENTS
Page
LESSON 1: Identify a suitable content domain 1
LESSON 2: Design a questionnaire 11
LESSON 3: Writing questionnaire items 29
LESSON 4: Pilot-test the questionnaire 38
LESSON 5: Evaluate reliability and validity 55
LESSON 6: Compile a manual 71
LESSON 7: Review a questionnaire 82
LESSON 8: Review a manual 96

CONTENT DOMAIN 112
CORRELATION COEFFICIENT 114
ITEM ANALYSIS 118
ITEM FORMAT 120
LAYOUT OF A QUESTIONNAIRE 127
MANUAL: PURPOSE AND STRUCTURE 132
RELIABILITY 136
SPECIFICATION DOCUMENT 140
SUITABILITY OF A QUESTIONNAIRE 141
VALIDITY 143
WRITING QUESTIONNAIRE ITEMS 147
...........
iii PYC 2616/1
...........
iv
INTRODUCTION
A word of welcome
A very warm welcome to you as you begin this module called Psychological
survey development PYC2616.
Survey research is an approach to research whereby information is collected

from a sample of individuals by asking them questions about a specific topic
or topics. It is often used in social and psychological research to explore and
describe behaviour, attitudes, etcetera. This module focuses on an aspect of
survey research, namely the development of the survey questionnaire. Topics in
the module include material on identifying the research aims and the goal of the
questionnaire, defining the target respondents, developing questions, choosing
the related question types, designing the questions and overall layout, running
a pilot study to evaluate the questions and reporting the results. Hopefully,
these will provide you with a window onto designing questionnaires, asking
the right questions in the right way and collecting the required information.
What this module is about

This module concentrates on the process followed in designing a survey
questionnaire and compiling a manual for the questionnaire. This includes the
selection of a suitable topic and of content relevant to the local context; the
writing of items that reflect sensitivity for diversity; the administration of the
questionnaire and analysis of its properties; and reporting the preceding in a
manual. In addition, you will be enabled to critically evaluate a questionnaire
and manual in terms of, among others, its relevance to a multicultural context
with regard to aspects such as: a suitable content domain, questionnaire design
for multicultural settings, language, gender and age sensitive questionnaire
items, how to pilot-test the questionnaire, how to evaluate reliability and
validity, how to compile a manual, how to review a questionnaire, and how
to review a manual.
Why study this module

As mentioned, this module introduces the skills and knowledge necessary
for questionnaire design and the gathering of psychological research data.
The module covers the principles of various techniques and their procedural
implementation and provides essential preparation for the research you will do
as a psychologist. Psychological research refers to research that psychologists
conduct to research and analyse the experiences and behaviours of individuals
or groups. All our knowledge in psychology comes from research, and research
continually refines and extends our understanding of the workings of the
human mind.
...........
v PYC 2616/1
INTRODUCTION
Contextualising the module

Questionnaire design in psychology has its genesis in psychological assessment.
This is a branch of psychology that is concerned with the development and
administration of psychological tests to quantify psychological constructs and
behaviour. Therefore, psychological assessment, through developing and
employing tools such as quesionnaires, has been able to measure and label
human behaviour. In some instances, these tools have aided in predicting
behaviour. Questionnaire design constitutes an important component in
psychology due to the wide adoption and use of psychological questionnaires
in a variety of domains such as occupational, educational, health, and economic
settings. The importance of psychological questionnaires is premised on their
ability to assess and capture human behaviour through the application of
scientific prescripts that guide psychological research. This module, therefore,
introduces you to the basic principles and techniques that are important for
designing a psychological questionnaire, focusing on aspects that enrich the
validity and reliability of a questionnaire.
Psychological questionnaires have a long history in the discipline of psychology

and have, arguably, had a notable contribution beyond the confines
of psychology. Nonetheless, the use of psychological questionnaires has
been a controversial subject in non-Western contexts due to their limited
scope and application. This is because psychological questionnaires, and
by extension psychological assessments, are Western-derived practices that
were hegemonically imposed on non-Western contexts with no regard for
contextual differences and factors that shape human thinking and behaviour
in those contexts. In psychology, culture and the environment are regarded
as important determinants of behaviour. Additionally, traditions and customs
which are grounded in one’s culture shape one’s orientation to life. This implies
that there cannot be a single orientation to life and that cultural differences
bring about varying psychologies and understandings of human behaviour.
Hence, the use of psychological questionnaires to understand behaviour in
non-Western contexts, including South Africa, sparked controversy.
The use of psychological questionnaires in South Africa cannot be divorced

from the country’s political history (Claasen, 1997). This is because these
questionnaires have often been used to perpetuate a racial agenda and to
further the interests of the white minority. Laher and Cockcroft (2014) also note
the history associated with the indiscriminate, biased and unfair use of these
psychological tools in South Africa. For instance, psychological assessments
were normed for white children and used with black children, and the low
scores obtained by black children were used as evidence to regard them as
inferior (Laher & Cockcroft, 2014). These discriminatory incidents pertaining to
the use of psychological questionnaires and assessments served as compelling
reasons for psychologists post-1994 to revolt against the apartheid practices
in terms of the manner in which psychological tools (i.e., questionnaires and
assessments) were used. This prompted the need to consider and develop tests
that are culturally sensitive, fair and unbiased, and take to account individual
profiles such as language, race, age, level of education, etc.
In this module, we draw your attention to important aspects to consider when

designing a questionnaire. We also urge you to be cognisant of the context in
...........
vi
I nt r o du c ti o n
which the questionnaire to be designed will be used. This implies that when
designing a questionnaire for research or assessment purposes, the population
characteristics should be considered in order to produce a questionnaire that
is scientifically sound, fair and unbiased, and does not infringe on the rights
of the users. This is important, especially when working in a multicultural
context such as South Africa with a rich history of discriminatory practices
of psychology emanating from the apartheid and colonial rule. Therefore,
designing a questionnaire that is psychometrically sound means that you have
to be mindful of the influence of Western approaches that inform questionnaire
development as this will help you to identify acceptable and unacceptable
practices in the development of a questionnaire. Additionally, this will also deter
you from using a one-size-fits-all approach in designing a questionnaire. Thus,
enabling you to design a questionnaire that is reliable and valid, conceptualised
from a multicultural perspective, and with the consciousness of the context in
which it will be utilised.
Decolonisation and questionnaire design in psychology

In your first year of psychology, you were introduced to decolonisation.
Decolonisation in psychology involves critiquing Eurocentric or Western
ideologies to reveal the role of power, oppression, and resistance in psychology
as well as the pervasive impacts of ‘institutional racism’ in society (Dudgeon &
Walker, 2015). Further, decolonisation encompasses learning about oppressive
systems, unwinding the impacts of these systems while reclaiming indigenous
ways of constructing knowledge, and recognising them as equal counterparts
in the formation of knowledge. In the South African context, decolonisation
(in psychology) takes shape through developing a contexualised psychology
that meets the needs of the majority of the South African population through
Africanising the curriculum, using contextually relevant research methods, and
changing the methods in which psychological assessment is conducted to rid
racial discrimination (Clay, 2017).
Decolonisation is a school of thought that seeks to reverse unjust practices

of the colonial era and bring about equitable and non-oppressive knowledge
systems (Furo, 2018). The decolonial project recognises the importance of
the co-construction of knowledge that is founded on indigenous knowledge,
experience and realities. This is a type of knowledge that acknowledges
the existence of multiple realities and is free from individuation and bias.
Colonisation took shape in many forms and was enforced economically, socially,
spiritually, educationally, etc. The discipline of psychology was not spared from
enforcing colonial and oppressive practices. For instance, in South Africa and
many parts across the globe, questionnaires were developed and used as tools
for perpetuating oppression and demeaning people through racial labelling and
classification. Additionally, research was conducted to advance the geopolitical
agenda to amass economic control and to marginalise people of colour. Such
practices were ushered and normalised mainly through colonisation and other
oppressive and orchestrated structures such as apartheid. The overall aim of
these structures was to subjugate the people of colour and characterise them
as inferior to their white counterparts while also prescribing to them how they
should think and understand the world around them.
...........
vii PYC 2616/1
INTRODUCTION
In this module, we encourage you to be cognisant of the history of psychology

pertaining to questionnaire development and psychological assessment. We
also encourage you adopt decolonial lenses when engaging in the exercises
of developing a questionnaire or conducting psychological research. This is
because questionnaire development and conducting research in psychology
has to be done through the bounds of fairness and representativity. Further,
a decolonial approach to questionnaire development and research will help
you construct knowledge that is ethical and void of discrimination and most
importantly, a knowledge that is contextually appropriate, sensitive to research
subjects, relevant, and responsive to local ills.
Bias and equivalence

During questionnaire design, strategies are employed to minimise bias and
achieve equivalence. Bias implies that differences in performance between
different cultural groups do not correspond to differences in the construct
being measured (Van de Vijver & Tanzer, 2004). The three types of bias are
defined in Table 1, and strategies to deal with bias are presented in Table 2.
Table 1: Types of bias

Type of bias Strategies
Construct bias Construct bias implies that the construct measured does
not have the same meaning across cultures. There is only a
partial overlap in definition of the construct across cultures,
or not all behaviours relevant to the construct are present
and properly sampled in each culture (Van de Vijver &
Poortinga, 1997).
Method bias Method bias is bias that is inherent to the method of
measurement or the measurement process. Sample bias
implies that the samples are not comparable due to cross-
cultural variation in sample characteristics that are associated
with the construct being measured. Instrument bias is due
to problems deriving from instrument characteristics, such
as stimulus familiarity and response styles. Response styles
refer to a systematic tendency to use certain categories of
the answering scale. Lastly, administration bias can come
from administration conditions, ambiguous instructions,
interaction between the administrator and respondents,
and communication problems.
...........
viii
Item bias An item is biased when it has a different psychological

meaning for different cultures. Individuals with the same
trait, but coming from different cultures, are not equally
likely to endorse the item (Van de Vijver & Leung, 1997).
Table 2: Strategies to reduce bias in cross-cultural assessment

Construct bias Decentering (i.e., simultaneously developing the same
instrument in several cultures)
Convergence approach (i.e., independent within culture
development of instruments and subsequent cross-
cultural administration of all instruments)
Construct bias and/ Use of informants with expertise in local culture and
or method bias language
Use samples of bilingual subjects
Use of local surveys (e.g., content analyses of free-
response questions)
Non-standard instrument administration (e.g., think
aloud)
Cross-cultural comparison of nomological networks (e.g.,
convergent/discriminate validity studies, monotrait-
multimethod studies, connotation of key phrases)
Method bias Extensive training of administrators (e.g., increasing
cultural sensitivity)
Detailed manual/protocol for administration, scoring,
and interpretation
Detailed instructions (e.g., with a sufficient number of
examples and/or exercises)
Use of subject and context variables (e.g., educational
background)
Use of collateral information (e.g., test-taking behaviour
or test attitudes)
Assessment of response styles
Use of test-retest, training and/or intervention studies
...........
ix PYC 2616/1
INTRODUCTION

Item bias Judgmental methods of item bias detection (e.g., linguistic
and psychological analysis)
Psychometric methods of item bias detection (e.g.,
differential item functioning analysis)
Error or distracter analysis
Documentation of “spare items” in the test manual which
are equally good measures of the construct as actually
used test items
Source: Van de Vijver & Tanzer (2004)
Minimising bias in cross-cultural studies usually amounts to a combination

of strategies: integrating design, implementation, and analysis procedures. A
detailed instruction on cross-cultural survey guidelines can be found at http://
ccsg.isr.umich.edu/.
What will you learn?

The module covers theory and constructs in basic measurement providing a
broad foundation for understanding the development of a survey questionnaire.
You are expected to apply your knowledge of questionnaire design in a manner
that is fit for purpose and contextually relevant, given different purposes and
different contexts. You need to be aware of cross-cultural differences in the
relevance, saliency, and social desirability of survey questions. For instance,
certain questions may not be relevant or salient for a given population and that
population may not have the information necessary to answer those questions.
In addition, questions considered inoffensive in one culture may be threatening
or taboo in other cultures. The format of presentation and assessment requires
active participation and applied critical thinking.
Course structure
The following foundational competence is embedded within the module,
and will be assessed directly or indirectly through assessment of the specific
outcomes in terms of the following criteria:
• a critical understanding of the meaning of psychological theory and

constructs in relation to measurement
• a critical understanding of the meaning of psychological measurement
• a range of assessment tasks will show that you have achieved the following
outcomes:
– identify a suitable content domain

– design a questionnaire
– writing questionnaire items
– pilot-test the questionnaire
– evaluate reliability and validity
– compile a manual
– review a questionnaire
– review a manual
...........
x
Study aids
To help you study, the following are provided:
Learning outcomes. The learning outcomes are what we want you to

achieve by the time you have finished working through the lesson. These
will help you to understand what is important. It will help you to identify
the main points you need to remember and be able to give an account
of them. The outcomes and assessment criteria for this module are:
Specific outcome Assessment criteria evidence shows that

you ...
1 Appraise the are competent to appraise the design of
formulation of questions intended for implementation in
measurement items multicultural settings when evidence shows
that
1.1 the technical correctness of the
formulation of a measurement item
is understood
1.2 the fitness for purpose of a
measurement item is understood
2 Appraise are competent to appraise psychometric
psychometric properties of measurement items when
properties of evidence shows knowledge of the different
measurement items approaches available for comparative
design for questionnaires in multiple
cultures, languages, or countries that
enables you to make informed choices for
your projects; and evidence shows that
2.1 basic measurement concepts (e.g.
observed score, mean, variance,
correlation) are understood
2.2 measurement items are appraised
correctly in terms of basic measurement
concepts
2.3 measurement items are appraised
correctly in terms of practical utility
3 Appraise are competent to appraise psychometric
psychometric properties of a questionnaire when
properties of a evidence shows that you are aware of
questionnaire cross-cultural differences in the relevance,
saliency, and social desirability of survey
questions; and that
3.1 basic psychometric constructs (e.g.
validity and reliability) are understood
3.2 questionnaires are appraised correctly
in terms of basic psychometric
constructs
3.3 questionnaires are appraised correctly
in terms of practical utility
...........
xi PYC 2616/1
INTRODUCTION
Specific outcome Assessment criteria evidence shows that

you ...
4 Appraise the intended are competent to appraise the intended
implementation of a implementation of a questionnaire when
questionnaire you demonstrate sensitivity in terms of
the relevance of the topic and purpose of
the survey, knowledge that respondents’
social reality and cultural framework shape
their perceptions and survey responses
in a variety of ways and when evidence
shows that
4.1 the technical correctness of a
questionnaire manual is appraised
correctly
4.2 the fitness for purpose of a
questionnaire is appraised correctly
Remember: These outcomes and criteria must be your “map”’ to help

you to complete this course successfully.
There are activities in each lesson. These include questions to think about,
self-tests, exercises to do, and drawings to interpret. They are designed
to help you to draw out important ideas and main points. Occasionally,
they are designed to test how carefully you have worked through the
material. We strongly urge you to complete these activities, since they
are an important aid to helping you understand the texts you will be
studying. Model answers to questions and self-tests can be found at the
end of each lesson.
We have included an introduction, establishing cultural equivalence and

resource references in each lesson. An introduction is like a guidebook
to your whole lesson. It gives background information about your topic
area and outlines all the ideas you are going to read about in the lesson.
“Cultural equivalence” refers to the possibility that interpretations of
psychological measurements, assessments, and observations are similar
if not equal across different ethno-cultural populations. You will notice
that some activities refer to resource material. You have to study resource
material in order to complete the actions successfully. One requires
resources to produce outcome products. For example, one may require
knowledge about certain issues before one can discuss these issues. A
resource reference indicates the name of the resource. The resource ma-
terial needed for producing the required outcome products is provided
in the second part of this guide.
Glossaries of certain basic, but perhaps unfamiliar terms, are provided.

Read these before you begin the readings so that you will have an idea
of what the terms mean when you encounter them.
...........
xii
How to approach a lesson

You should
(1) start by quickly skimming through the material in the lesson.

(2) read the material carefully, make notes and do the “activities” – make
sure you can give an account of the work.
(3) do the self-test at the end of each section and check your answers.
(4) go back to that section of the work, study the material again and make
sure that you have grasped it if your answers were wrong.
It is important to know where you are going before you start studying. By
studying this module, we want to get to a certain point and help you to focus
on this goal. We must plan the route and make sure that we take the correct
paths to lead you there. In academic terms, we call the destination (end goal/
place where we want to go to) the outcomes of the module. The road we travel
to get there is called assessment criteria. Like information boards next to the
highway, these criteria help us to make sure that we are on the right path, and
they also show us how far we have to travel. If the criteria state that there are
eight check points on this road and we can make ticks against seven of those
(because we know the contents, can apply them to our own situation and are
able to lead a debate about this subject), we know that we are nearly reaching
our goal – just one more issue to learn, interpret and implement in our context
and we will have completed the task.
Assignments and examination format

This module uses scenario-embedded multiple-choice questions. Multiple-
choice questions are questions that consist of a problem statement and two or
more potential responses. This is an open-book online examination therefore
this kind of multiple-choice question tests more than knowledge. It tests
thinking skills and insight. It requires respondents (i.e. students) to use all the
information provided in the potential responses and to think about how these
responses relate to each other.
Rank order scaling questions allow a certain set of behaviours of individuals or

groups to be ranked based on a specific attitude or characteristic. An average
ranking is calculated for each answer choice, allowing you to quickly evaluate
all responses from very important to less important (rank order questions).
Rating scales are often used to measure the direction and intensity of attitudes.
What is the nature of the examination?

The students will sit for a timed online assessment, which means they will
be required to complete the assessment online on a particular date and time.
How will the assignments and the examination be assessed

The outcomes and criteria will act as guides to the lecturer and the students
in this regard. We know where we want to go (the outcomes) and we know
which road to take to get to our destination (the criteria), so we will use these
...........
xiii PYC 2616/1
INTRODUCTION
to check our progress. The assignment questions as well as the examination

questions will be in line with the outcomes and criteria. The assessment covers
the entire syllabus. Assignment 01 together with Assignment 02 provides a
good indication of the nature and the format of the examination paper you
will complete.
Practical advice on writing an assignment and the examination

Study the scenario carefully before you consider the discussion questions.
Always unpack an assessment scenario in terms of the following:
• the assessment context

• the concern raised in the assessment context
• the owner of the concern
• the consultant
• the intention of the consultant
• the test user
• the intention of the test user
• the group targeted for assessment
• the questionnaire
• the intention of the questionnaire
Feedback
You will receive detailed feedback on Assignments 01 and 02. You will therefore
be able to evaluate your learning progress and prepare for the examination by
taking heed of the feedback written by the lecturer. Answers to the questions
as well as discussions of the questions are provided. The key to each question
is indicated, that is, the option that provides the correct answer. The reason
why the indicated key is the most correct answer is explained. In some cases,
the discussion also indicates why the other options are incorrect. Check your
answers and see where you make mistakes. Then go back to the sections in
your study guide.
Notional study hours

A module consisting of 12 credits equates to 120 notional hours. The module
is designed to take you 120 notional study hours to complete. This means that
you must use 120 hours to read through the material, familiarise yourself with
the contents and complete all the assignments and tasks set for the module.
These 120 hours exclude the time you prepare for and write the examination.
Thematic approach
As explained, this study guide contains the required resource material and
lessons. You must work through the lessons and complete the necessary
actions. Note that the activities follow on each other. If you complete all the
activities, you can expect to obtain the desired skills and knowledge. You
will then be able to apply the competencies you acquired in developing your
own questionnaire and writing a manual for your questionnaire to evaluate a
...........
xiv
questionnaire and manual in the examinations through answering the multiple-

choice questions. In the study guide, we guide you through a number of
activities to obtain these objectives. For each activity, you are required to
study the relevant resource material first, complete the necessary actions, and
produce the specified outcomes.
A final word
Now that you have received the above information, we hope that you feel
more confident to start with the lessons.
Practical methods in questionnaire design and assessment build your

understanding of research and practical skills for research projects, which
increase in importance throughout your studies in psychology. Once mastered,
you walk away from this module with a very useful skill in psychology.
References
Claasen, N. C. W. (1997). Cultural differences, politics and test bias in South
Africa. European Review of Applied Psychology, 47, 297–307.
Clay, A. (2017). Decolonizing psychology in South Africa. American
Psychological Association, 48(10). https://www.apa.org/monitor/2017/11/
decolonizing-psychology
Dudgeon, P., & Walker, R. (2015). Decolonising Australian psychology:
Discourses, strategies, and practice. Journal of Social and Political
Psychology, 3(1), 276–297.
Furo, A. (2018). Decolonizing the classroom curriculum: Indigenous knowledges,
colonizing logics, and ethical spaces [Doctoral dissertation, University
of Ottawa]. University of Ottawa Research. https://ruor.uottawa.ca/
handle/10393/37106
Laher, S., & Cockcroft, K. (2014). Psychological assessment in post-apartheid
South Africa: the way forward. South African Journal of Psychology,
44(3), 303–314.
Van de Vijver, FJR & Leung, K. (1997). Methods and data analysis for cross-
cultural research. Newbury Park, CA: Sage.
Van de Vijver, FJR & Poortinga, YH. (1997). Towards an integrated analysis
of bias in cross-cultural assessment. European Journal of Psychological
Assessment, 13, 29–37. doi:10.1027/1015-5759.13.1.29.
Van de Vijver, FJR & Tanzer, NK. (2004). Bias and equivalence in cross-cultural
assessment: An overview. Revue Européenne de Psychologie Appliquée/
European Review of Applied Psychology, 54, 119–135. doi:10.1016/j.
erap.2003.12.004.
...........
xv PYC 2616/1
INTRODUCTION
Overview of the research process:

(1) Research question or idea
(2) Choosing and designing of research
(3) Collecting of data
(4) Analysis of data
(5) Publication of data
Choose: qualitative/
quantitative/mixed study?
Selection of an appropriate methodological approach:
In psychological research, we have four main research methods, namely:
(1) Observation
(2) Experiment
(3) Case study
(4) Survey (questionnaires)
Consider the following:
Cross-cultural and/or
decolonisation aspects
that are affected by
your methodological
approaches?
In this module, we will be exploring QUESTIONNAIRE DESIGN, which includes
the following components:
(1) Introduction (YOU ARE HERE)
(2) Identifying a suitable content domain
(3) Designing a questionnaire
(4) Writing questionnaire items
(5) Pilot-testing a questionnaire
(6) Evaluating reliability and validity
(7) Compiling a manual
(8) Reviewing a questionnaire
(9) Reviewing a manual
(10) Resource material
...........
xvi

1 LESSON 1
1 Identify a suitable content domain
This lesson deals with the identification of a content domain for a questionnaire.
Aims
In lesson 1, you will learn the definition of a content domain in the context of
psychology and social science research and experiments. You will be given
activities with examples to further your knowledge.
Lesson
This lesson will help you to produce:
• an identified content domain

• a questionnaire purpose
The following resource material is required for this lesson:
• content domain
• suitability of a questionnaire
Establishing cultural e quivalence
A culturally appropriate measure seeks to be culturally equivalent across sub-

groups and populations by taking into account the following:
• shared norms: socially desirable behaviours (e.g. “the do’s and do nots”)
• shared beliefs: ideas or assumptions about the world
• shared values and expectations: moral standards perceived as desirable and
esteemed
In creating culturally appropriate measures, it is also essential to consider the

appropriateness of the purpose of the questionnaire. The purpose of the ques-
...........
1 PYC 2616/1

tionnaire will in turn give you ideas about the content domain, that is, all the
possible tasks, behaviours, attitudes, etcetera implied by the purpose. It is from
this content domain that you select items to make up the questionnaire.
During this process, emphasis should be on conceptual and cultural equivalence.
• Conceptual equivalence: Do people attach the same meanings to terms

and concepts?
• Cultural equivalence: Are the cultural norms, beliefs, values and expectations
the same for different populations?
A questionnaire cannot be valid unless it is reliable. An assessment can provide

you with consistent results, making it reliable, but unless it is measuring what you
are intend to measure, it is not valid. Therefore, for measurement equivalence
in the conduct of a survey across cultures, it is pivotal to establish equivalence
of the measures prior to drawing meaningful substantive conclusions about the
relative importance of culture equivalence, for example, when psychological and
work-related constructs are measured in a cross-cultural context.
1.1 INTRODUCTION
Let us first talk about characteristics of a good questionnaire and survey.
A good test has a clearly defined purpose. To define the purpose, the test
developer and the test user must be familiar with these questions: what is the
test supposed to measure; what is the domain or the content it is measuring;
who will take the test; who is the test intended for; and how will the test score
be used. Is the test designed to compare the performance of test takers to each
other or are they designed to diagnose. Different types of items and scores are
used for these different types of comparisons.
So, let us begin with an overview of experimental validity and content domain.
1.1.1 What is a content domain?

You are a psychologist at a government hospital and want to conduct a research
survey for the hospital measuring the secondary traumatic stress level of
emergency room doctors at the hospital. You need to create a questionnaire
that adequately measures secondary traumatic stress so that your research
study is valid. Secondary traumatic stress is the content domain in your study.
Content domain is the body of knowledge, skills or abilities being measured or

examined by a research study. A researcher would want a content domain to
cover all aspects of the subject area as well as be well defined and objective.
The more a content domain has these qualities, the easier it is to measure
in studies, experiments and tests. In order for the content domain to be
fully comprehensive, it must be experimentally valid.
...........
2

1.1.2 Experimental validity and content domain

Experimental validity refers to how well the experiment or test represents
facts or logic. Let us say for example, there is a depression questionnaire that
is considered highly valid because it is based on scientific evidence-based
research and data.
It is an accurate and comprehensive measurement tool of the entire content

domain of depression. The content domain of depression is expansive and can
be divided into many different segments such as stress level, life satisfaction,
constant fatigue, feelings of hopelessness or lack of enjoyment in things of
which one once took pleasure in. The depression questionnaire incorporates
the content domain of depression so well in its questions and is a widely
recognised and used tool for rating how depressed a person is. Now you can
match your content domain with a questionnaire as the measuring instrument
of choice.
This is just one example, but every experiment, test or study has a content
domain that it is trying to accurately measure.
1.2 GLOSSARY
Here are some basic terms related to the topic of identifying the content domain
for a questionnaire that you may not be familiar with, but which you need to
know. They will come up in your reading of the material in this lesson.
Concept English definition Afrikaanse definisie Sesotho definition
Questionnaire A questionnaire is a ‘n Vraelys is ’n Lenanepotso ke

Vraelys research instrument navorsingsinstrument sesebediswa sa
consisting of a wat bestaan uit ’n diphuputso sa
Lenanepotso series of questions reeks vrae (of ander lethathama la
(or other types of tipes aanwysings) dipotso (kapa
prompts) for the om inligting van ditlhohleletso tsa
purpose of gathering respondente te mefuta e meng)
information from versamel. tseo sepheo sa
respondents. tsona e leng ho
bokella lesedi ho
bankakarolo.
Topic The questionnaire Die Sehlooho sa
Onderwerp topic refers to vraelysonderwerp lenanepotso se
the design of a verwys na die bua ka mokgwa
Sehlooho questionnaire to ontwerp van ’n oo lenanepotso
gather statistically vraelys om statisties le entsweng ka
useful information bruikbare inligting oona ho bokella
about a given topic. oor ’n gegewe lesedi ka dipalopalo
onderwerp in te tsa bohlokwa ka
samel. sehlooho se itseng.
...........
3 PYC 2616/1

Focus Once you have Nadat jy die Hang ha o se

Fokus narrowed down onderwerp ingeperk o tsepamisitse
the topic, you need het, moet jy dit nog mohopolo
Tsepamiso ya to reduce it even verder inperk om sehloohong se
mohopolo further to give it fokus aan die vraelys itseng, o hloka ho
some focus for your te gee – fokus verwys se fokotsa le ho feta
questionnaire – focus na die probleemarea hore se tsamaellane
refers to the problem in die onderwerp wat le lenanepotso la
area within the topic jy geïdentifiseer het. hao – ho tsepamisa
you have identified. mohopolo ho bolela
karolo e hlokang ho
rarollwa sehloohong
seo o faneng ka sona.
Purpose The purpose of Die doel van die Sepheo sa
Doel the questionnaire vraelys bepaal watter lenanepotso ke
determines which soort betroubaarheid ho fumana hore
Sepheo type of reliability toepaslik is. Die mofuta o loketseng,
is appropriate. doel van die vraelys wa diphetho tse
The purpose of bepaal ook die tshwanang ditekong
the questionnaire inhoudsdomein en tse phetwang, ke
determines the daar moet beskryf ofe. Sepheo sa
content domain word hoe die lenanepotso se laola
and it should be vraelysitems hierdie dikahare tsa lekala
described how domein dek. leo ho fuputswang
the items of the ka lona mme se
questionnaire cover lokela ho hlalosa
this domain. kamoo dintlha tsa
lenanepotso di
akaretsang lekala
lena ka teng.
Content domain The content domain Die inhoudsdomein Tsebo kapa bokgoni
Inhoudsdomein is the universe of is die geheel van bo hlahlojwang
tasks, behaviours, take, gedrag, ke mesebetsi,
Tsebo kapa bokgoni attitudes, etcetera houdings, menahano,
bo hlahlojwang implied by the ensovoorts soos jwalojwalo,
purpose of the geïmpliseer word eo sepheo sa
questionnaire. deur die doel van die lenanepotso se e
vraelys. sisinyang.
Scope When you define Wanneer jy die Ha o hlalosa
Omvang the scope of a omvang van ’n kakaretso ya dintlha
questionnaire, you vraelys definieer, tseo lenanepotso le
Kakaretso set the “boundaries” stel jy die “grense” botsang ka tsona, o
– specifying what is – spesifiseer wat di “behela moedi”–o
included and what is ingesluit word en totobatsa hore na ho
excluded. wat uitgesluit word. kenyeleditswe eng le
hore na ke eng e sa
kenyeletswang.
...........
4

Content validity The content Die inhouds- Nepahalo ya

Inhoudsgeldigheid validity of the geldigheid van sephetho sa tekolo
questionnaire for its die vraelys vir sy ya lenanepotso ya
Nepahalo ya particular purpose spesifieke doel sepheo se itseng
sephetho sa tekolo is determined word bepaal deur e itshetlehile
by the degree to die mate waarin die boemong boo
which the items in items in die vraelys dintho tse botswang
the questionnaire verteenwoordigend lenanepotsong
are representative is van die geheel di akaretsang
of the universe of van take, gedrag of mesebetsi le
tasks, behaviours or houdings waarvoor menahano eo
attitudes that it was dit ontwerp is om te lenanepotso le
designed to measure. meet. etseditsweng yona.
1 AC TIVITIES
Before you work through the activities, I want you to read the resource
material for this lesson (content domain and suitability of a questionnaire).
Suitability of a
Content domain questionaire
Press the button Press the button
or go to page 112 in or go to page 141 in
the study guide. the study guide.
It takes about one hour to do the activities.
T ime re cord
These activities took me ........ hours.
METHOD Questionnaires are used extensively to gather various kinds of information

in the social sciences. You, yourself might have completed a questionnaire
Activity 1.1
as part of a survey. In this module, we offer you the opportunity to develop
Study the resource
material practical competencies to design and develop your own questionnaire. The
first step in compiling a questionnaire is to identify a suitable content domain.
Resource Generally speaking, ideas about the content domain for a questionnaire come
content domain from our observations and experiences of the world around us. Study the
resource on the identification of a content domain for a questionnaire before
you continue.
Activity 1.2 If you find it difficult to identify a topic for a questionnaire, this learning
Describe the experience could help you. On the other hand, you may be in the fortunate
relevant content
domain position of already knowing what you want to investigate. For example, you
may have been told by your employer to draw up a questionnaire to obtain
information on a certain topic, or you may need a questionnaire to find out
information as part of a research project, or you may just have a personal
...........
5 PYC 2616/1

interest in a topic. In this case, you can read the first three actions but you do
not need to do them.
AC T I O N 01
Make two lists, one list of the things that interest you and another of the things
that you do. These lists can be as long as you like but they should contain at
least five items in each category.
Things that interest me ... Things I do ...
1. 1.
2. 2.
3. 3.
4. 4.
5. 5.
AC T I O N 02
Rank the items on your interest list from the most interesting to the least
interesting and rank the items on your activities list (things you do) from the
things you spend most time on to the things you spend least time on.
AC T I O N 03
Inspect your rank-ordered lists. You can probably identify common themes
(that is, things that the lists have in common) from these lists. These should
give you a clearer indication of your world and the things that are important
to you. Based on your interests and activities, now make a list of things you
would like to find out or get more information on that would be of use to you
or someone else. Complete this sentence:
I would like to find out about ...
Now you are ready to identify a topic and describe the content domain for
your questionnaire.
AC T I O N 0 4
You have already identified a general theme of interest, either through completing
the first three actions or because you already knew at the outset what you
want to do. Even if your ideas are vague, write your ideas in a sentence that
begins as follows:
The general topic that I am interested in or want to investigate is …
...........
6

AC T I O N 05
Because your topic is fairly broad, you now need to narrow it down. Select a
problem area within the topic that you want to investigate and write it down.
AC T I O N 0 6
Once you have narrowed down the topic, you need to reduce it even further
to give it some focus for your investigation. Look at what you have written and
decide on some specific questions about the problem area within the topic
you have identified. Write them down.
Carefully consider the questions you have written. Do they all relate to each
other? Can you combine them into one question? Should you rather choose
one question and leave the others for separate questionnaires? When you have
made these decisions, you have the focus of your questionnaire. Note that these
questions help you to be specific about what you want to measure with your
questionnaire; they are not the final items or questions for the questionnaire,
which will only be formulated in lesson 3.
AC T I O N 07
Now that you have identified the focus for your questionnaire, you are ready
to limit the scope of the questionnaire so that you cover the content adequately
but do not ask irrelevant questions. Look at your questions (the previous action)
and write what you need to know in order to get the relevant information that
will provide the answers to your questions. For example, if one of your questions
was, “Do younger people perceive crime differently from older people?”, you
would need information on the respondents’ ages (biographical information)
and perceptions of crime.
In order to answer my questions, I need to get information on …
Now ask yourself whether each point you have written is really relevant to the
focus of your investigation and give a reason/reasons for including each one.
If the answer is “yes” and you can provide a reason for including each point,
you have successfully limited the scope of your questionnaire. If the answer is
“no”, cross out the ones that are not relevant. If there are some points that sound
interesting but you cannot provide reasons for including them, cross them out
too. This exercise should have helped you to successfully limit the scope of
your questionnaire. (Did you realise that the information you have identified
as being relevant is actually the content domain of your questionnaire? You
Activity 1.3 will select your items according to the content domain.)
Study the resource
material
Now that you have limited the scope of the topic and described your content
Resource domain, you have to make sure that a questionnaire is the right type of measuring
suitability of a instrument to use for obtaining the kind of information you want. Study the
questionnaire
resource on the suitability of a questionnaire as a measuring instrument.
...........
7 PYC 2616/1

AC T I O N 01
Activity 1.4 You need to describe the reason for choosing a questionnaire as your measuring
Evaluate the instrument. In order to decide whether a questionnaire is the most suitable
suitability of a measuring instrument for your investigation, it is important to understand why
questionnaire as
a measure for this and when questionnaires are used. Complete the following sentence:
content domain
Questionnaires are used for …
AC T I O N 02
Now you can match your content domain with a questionnaire as the measuring
instrument of choice. If you have not already done so, write the purpose of
your questionnaire and describe the content domain. To do this, you have to
formulate or describe what it is that you want to investigate and that includes
the topic you are interested in. For example, the purpose of your questionnaire
might be to investigate young people’s perceptions of how crime influences
their future plans. The content domain would be any information relevant to
this topic (of perceptions of how crime influences future plans) and the tasks,
behaviours and attitudes, etcetera, for obtaining this information. Now that
you have identified the content domain for your questionnaire, complete the
following sentence:
The purpose of my questionnaire is to …
Now go through the exercise described in your resource material again, that
is, decide whether you can use a questionnaire to obtain the information you
need (or whether you would have to administer ability tests and so on). If the
answer is yes, a questionnaire will be a suitable measuring instrument for your
investigation. If not, see if you can focus on a different aspect of your topic
that is suited to a questionnaire.
Once you have identified the content domain for your questionnaire, you can
go on to the next step of deciding what your questionnaire should look like.
In lesson 2, you will look at the questionnaire specification document.
S E L F -A SS E SS M E N T
Study the scenario carefully before you consider the self-assessment questions.
Lifestyle refers to the experiences and events in a person’s life. A style of living
is a pattern of experiences and events. It includes the principles according to
which a person lives, and it deals with the different areas of the person’s life
such as his/her home and work environments. A health and fitness centre
decided to include the option of a stress management programme as part
of the service given to their clients. Suppose you were contacted to evaluate
...........
8

the effectiveness of this programme. Before implementing the programme,

the personnel at the centre will attend the programme providing you with
an opportunity to obtain data on the effectiveness of the programme. As
the stress management programme is based on making lifestyle changes,
you decide to use the lifestyle questionnaire (LQ) as part of your study. The
LQ is a self-report questionnaire that was developed to obtain an impression
of a person’s behaviour in different areas of his or her life. You contact the
test publisher and they provide you with the questionnaire.
QUESTION 1
Pule: What is the content domain represented by the lifestyle questionnaire?
Pumulo: I would say the content domain of the LQ is factors affecting

lifestyle.
Temba: I think the content domain of the LQ is the assessment of lifestyle.
Zola: I think the content domain is stress management that includes

aspects of lifestyle.
You agree most with, second most with and least with:
(1) Pumulo, Zola and Temba

(2) Temba, Zola and Pumulo
(3) Zola, Pumulo and Temba
(4) Zola, Temba and Pumulo
(5) Temba, Pumulo and Zola
QUESTION 2
The main reason why a questionnaire is the most suitable measure for this
content domain is because it ...
A. collects facts and opinions about stress.

B. gathers factual information about lifestyle.
C. gets information for further analysis.
D. explains the relation between lifestyle and health.
E. monitors stress management.
F. measures specific abilities, like fitness.
G. includes all aspects of lifestyle.
(1) A, C, E and G
(2) B and C
(3) B, D and G
(4) All of the above.
...........
9 PYC 2616/1

QUESTION 3
Operationalising lifestyle in terms of specific areas (home environment, work
environment, nutrition and fitness, and recreational activities) helps to define
the ... of the questionnaire.
(1) content domain

(2) item format
(3) topic
(4) scope
Q U E S T I O N 1: O P T I O N 1 I S T H E K E Y – P U M U LO, ZO L A A N D
TEMBA
The content domain reflects the purpose of a questionnaire. This questionnaire
covers behaviours in different areas of a person’s life, such as personal
relationships, the work environment and physical health. These aspects all
relate to the larger content domain, which is factors influencing lifestyle. This
is why Pumulo’s answer is the best answer. If the content domain were the
assessment of lifestyle, then all the items would be concerned with various ways
or methods for assessing lifestyle. Since this is clearly not the case, Temba is
incorrect. Zola’s response is also not correct – while there are some items that
relate to stress, the focus of the questionnaire is not stress management either.
Q U E S T I O N 2: O P T I O N 2 I S T H E K E Y – B A N D C
The choice of measuring instrument depends on the purpose of the investigation
and the content to be measured. You will see in your resource material (see:
suitability of a questionnaire as a measuring instrument) that the main purpose
of a questionnaire is to gather factual information. In this instance, the aim is to
gather information about lifestyle (B) but not about stress (A). A questionnaire
also provides a way to get information for further analysis (C). This is why option
2 provides the correct answer. A questionnaire does not provide a means for
investigating a relationship between variables (D), nor does it measure specific
abilities (F). This particular questionnaire covers some aspects of lifestyle, not
all of them (G), and it does not deal with stress management (E).
Q U E S T I O N 3: O P T I O N 4 I S T H E K E Y – S CO P E
In this instance, the topic (option 3) or general theme of interest is lifestyle. The
content domain (option 1) of the questionnaire relates to some of the factors
that affect lifestyle. By identifying these factors, you have defined the scope
of your questionnaire. This is why option 4 provides the correct answer. The
actual nature of these factors or aspects will determine what kind of questions
you use (option 2).
...........
10

Lesson
2 2
2 Design a questionnaire
This lesson helps you to create a simple survey questionnaire based on principles
of effective item types and layout.
AIMS
In lesson 2, you will learn to compile a questionnaire specification document,

indicating the different types of items and scales you are going to use, the
number of items and the layout. You will be given activities with examples to
further your knowledge.
Lesson
This lesson will help you to
• understand the attributes of a well-designed questionnaire,

• adopt a framework for developing questionnaires, and
• produce a questionnaire specification document
• content domain
• item format
• layout of a questionnaire
• specification document
Psychological survey development should be culturally sensitive so that

researchers understand, acknowledge, and respect cultural differences among
research participants. Questionnaires should reflect the cultural, linguistic, geo-
graphic, racial and ethnic diversity of the population served. Cultural sensitivity
...........
11 PYC 2616/1

can create avenues around communication barriers that can lead to frustration
that can develop between the survey research, researchers, and the participants.
While culture is often described in terms of group characteristics, a multi-

dimensional perspective captures a more individualised, holistic view: Culture
is the combination of thoughts, feelings, attitudes, material traits, and behaviours
of a group of people. To be adept in our culturally loaded interactions with peo-
ple, we must be comfortable with considering our level of cultural sensitivity as
individuals and as researchers. To accomplish this, cultural sensitivity must be
an open-ended, mindful, and measurable initiative integrated into the fabric of
survey research and questionnaire design. Given that lesson 2 (design a question-
naire) is aimed at a list of the required characteristics for your questionnaire, the
process is an opportunity for all researchers to re-examine their vast, ongoing
cultural sensitivity initiative. The fact is, since cultural attributes are present in
every person at all times, perceived cultural similarities and differences shape
all of our interpersonal interactions. We are all constantly responding to what
we perceive in our co-workers, participants, friends and neighbours as their
personal cultural climate, and adjusting our conduct in a way that we judge is
suitable for successful communication.
Whether or not we are aware of it, “culture” and its influences are ever constant
in all that we do. In striving to find the richness of culture, both our own and of
the population we serve, we as researchers are able to discover ourselves, our
participants and the context of their life circumstances. In this module, there is
an increased awareness to do away with the notion that “culture” is exotic and
intrinsic to only those who are from groups other than our own. As each of us
let go of the mindset that culture is an unimportant trend, we sharpen in on an
essential and often overlooked key ingredient of effective psychological survey
development. We are able to “know” that consideration of culture and its influ-
ences is an integral ingredient to effective psychological survey development
and no survey can achieve success without a well-designed questionnaire that is
sensitive to culture and diversity, and prevent ethnic discrimination. In addition,
the relevant norms may vary across social classes or subcultures within a society.
When there is such variation in norms, the bias induced by socially desirable
responding may distort the observed associations between the behaviour in
question and the characteristics of the respondents, besides affecting estimates
of overall means or proportions. Therefore, do not allow cultural differences to
become the basis for criticism and judgments and avoid stereotypes. Mahatma
Ghandi said: “no culture could live if it attempts to be exclusive.”
2.1 INTRODUCTION
The heart of any survey research project is the survey questionnaire itself. A good
questionnaire helps to achieve the research objectives; it provides complete and
accurate information, is easy for the respondents to complete, is so designed
as to make sound analysis and interpretation possible and is brief.
The distinct steps in designing a good questionnaire are: decide on the information
required; define the target respondents; select the method(s) of reaching the
...........
12

respondents; determine question content; formulate the questions; sequence

the questions; check the questionnaire length; pre-test the questionnaire; and
develop the final questionnaire.
Although it is easy to think of interesting questions to ask people, constructing

a good survey questionnaire is not easy at all. The problem is that the answers
people give can be influenced in unintended ways by the wording of the
items, the order of the items, the response options provided, and many other
factors. At best, these influences add noise to the data. At worst, they result
in systematic biases and misleading results. Therefore, in this lesson, we
consider some principles for designing survey questionnaires to minimise
these unintended effects and thereby maximise the reliability and validity of
the respondents’ answers.
Before looking at the layout of a questionnaire, it will help to consider survey

responding as a psychological process.
Respondents must interpret the question, retrieve relevant information from

memory, form a tentative judgment, convert the tentative judgment into one
of the response options provided (e.g., a rating on a one-to-seven scale), and
finally edit their response as necessary.
Consider, for example, the following questionnaire item:
How many alcoholic drinks do you consume in a typical day?
•_____ a lot more than average

•_____ somewhat more than average
•_____ average
•_____ somewhat fewer than average
•_____ a lot fewer than average
Although this item at first seems straightforward, it poses several difficulties

for respondents. First, they must interpret the question. For example, they
must decide whether “alcoholic drinks” include beer and wine (as opposed
to just hard liquor) and whether a “typical day” is a typical weekday, a typical
weekend day, or both).
2.1.1 Design a questionnaire

To create a questionnaire that best captures the constructs you are trying to
measure, keep the following points of advice in mind.
...........
13 PYC 2616/1

2.1.2 What are you measuring?

As you create items for your questionnaire, stay focused on the constructs you
are attempting to measure. Questionnaire construct refers to the design of a
questionnaire to gather statistically useful information about a given topic.
When properly constructed and responsibly administered, questionnaires can
provide valuable data about any given subject. For example, your questionnaire
(attitude and intention questionnaire [AIQ]) is based on the theory of planned
behaviour. The theory of planned behaviour has three factors or constructs,
namely the person’s attitude towards the behaviour (to use alcohol), his/her
subjective norm concerning the behaviour and his/her perceived control with
regard to the behaviour. These constructs constitute the content domain of
the AIQ. Make sure your items are specific to your construct and could not be
interpreted as referring to related constructs (the discriminant validity problem).
Also, make sure that no important qualities of your construct are omitted by
your questionnaire. This is a concern regarding the content validity of your
measure: the degree to which your measure captures the full range of your
content domain. Criticisms of content validity are criticisms about what has
been left out of a measure.
2.1.3 Presentation and sequence of items

Again, the presentation and sequence of items can lead to unintended influences
on the respondents’ answers. For example, there is an item-order effect when
the order in which the items are presented affects people’s responses. One item
can change how participants interpret a later item or change the information
that they retrieve to respond to later items. For example, a researcher asked
university students about both their general life satisfaction and their dating
frequency. When the life satisfaction item came first, the correlation between
the two was only −.12, suggesting that the two variables are only weakly
related. However, when the dating frequency item came first, the correlation
between the two was +.66, suggesting that those who date more have a strong
tendency to be more satisfied with their lives. Reporting the dating frequency
first made that information more accessible in memory so that they were more
likely to base their life satisfaction rating on it.
The response options provided can also have unintended effects on people’s
responses. For example, when people are asked how often they are “really
irritated” and given response options ranging from “less than once a year” to
“more than once a month”, they tend to think of major irritations and report
being irritated infrequently. However, when they are given response options
ranging from “less than once a day” to “several times a month”, they tend to
think of minor irritations and report being irritated frequently.
2.1.4 Be consistent in response options

The response options in your questionnaire are the ways that your participants
can respond to the prompts you provide. Perhaps the most common response
option in psychological questionnaires is a Likert scale, which asks participants
to indicate the degree to which they agree or disagree with a statement using
a scale that typically contains five to seven response points. If possible, keep
the same response options throughout your questionnaire. If you use a 1 to 5
...........
14

scale on page 1 and a 1 to 9 scale on page 2, some of your participants are

likely to keep using the 1 to 5 scale on page 2.
2.1.5 Types of items

Survey questionnaire items are either open ended or closed ended. Open-ended
items simply ask a question and allow respondents to answer in whatever way
they want. Closed-ended items ask a question and provide several response
options that respondents must choose from.
Closed-ended items ask a question and provide a set of response options for
participants to choose from. The alcohol item mentioned above is an example,
as is the following:
How old are you?
•_____ under 18
•_____ 18 to 34
•_____ 35 to 49
•_____ 50 to 70
•_____ over 70
All closed-ended items include a set of response options from which a

participant must choose.
In the section on consistency in response options, you read about the term
Likert scale. Although this term is sometimes used to refer to almost any rating
scale (e.g., a 0-to-10 life satisfaction scale), it has a much more precise meaning.
In the 1930s, researcher Rensis Likert created a new approach for measuring
people’s attitudes (Likert, 1932). It involves presenting people with several
statements – including both favourable and unfavourable statements – about
some person, group, or idea. Respondents then express their agreement or
disagreement with each statement on a five-point scale: strongly agree, agree,
neither agree nor disagree, disagree, strongly disagree. Numbers are assigned
to each response (with reverse coding as necessary) and then summed across
all items to produce a score representing the attitude toward the person, group,
or idea. The entire set of items came to be called a Likert scale.
Thus unless you are measuring people’s attitude toward something by assessing
their level of agreement with several statements about it, it is best to avoid
calling it a Likert scale. You are probably just using a “rating scale”.
Open-ended items simply ask a question and allow participants to answer
in whatever way they choose. The following are examples of open-ended
questionnaire items:
• “What is the most important thing to teach children to prepare them for life?”
• “Please describe a time when you were discriminated against because of
your race.”
• “Is there anything else you would like to tell us about?”
Open-ended items are useful when researchers do not know how participants
might respond or want to avoid influencing their responses.
...........
15 PYC 2616/1

2.1.6 Combining question types

The question format should be appropriate to the information sought. Combining
closed- and open-ended items, rating and ranks, rather than relying on a single
type of item can increase coverage of a topic.
Including different formats can provide checks on the information and increase
the validity of the results. For example, in a survey of attitudes toward library
study areas, students were first asked an open-ended question, “What do you
like most about the library study areas?” The most common response was the
“quiet” in the library compared to the student’s residence. Further along in
the questionnaire, students were asked to rate specific aspects of the library
environment as “excellent”, “satisfactory”, or “needs improvement”.
2.1.7 Formatting the questionnaire

Effective items are only one part of designing a survey questionnaire. Firstly,
every survey questionnaire should have a written or spoken introduction that
serves two basic functions. One is to encourage respondents to participate
in the survey. Thus, the introduction should briefly explain the purpose of
the survey and its importance, provide information about the sponsor of the
survey, acknowledge the importance of the respondent’s participation, and
describe any incentives for participating.
The second function of the introduction is to establish informed consent.
Remember that this aim means describing to respondents everything that
might affect their decision to participate. This includes the topics covered by
the survey, the amount of time it is likely to take, the respondent’s option to
withdraw at any time, confidentiality issues, and so on. Written consent forms
are not always used in survey research, so it is important that this part of the
introduction be well documented and presented clearly and in its entirety to
every respondent.
The introduction should be followed by the substantive questionnaire items. But
first, it is important to present clear instructions for completing the questionnaire,
including examples of how to use any unusual response scales. Remember that
the introduction is the point at which respondents are usually most interested
and least fatigued, so it is good practice to start with the most important
items for purposes of the research and proceed to less important items. Items
should also be grouped by topic or by type. For example, items using the same
rating scale (e.g., a five-point agreement scale) should be grouped together if
possible to make things faster and easier for the respondents. Demographic
items are often presented last because they are the least interesting to the
participants but also easy to answer in the event that the respondents have
become tired or bored. Ask your personal or sensitive questions near the end
of the questionnaire. If you need to ask any personal questions or contact
information questions, save them for the end of your questionnaire. If you ask
these kinds of questions too soon, it can discourage people from answering
your questionnaire. Of course, any questionnaire should end with an expression
of appreciation to the respondent.
In lesson 3, we consider some principles of writing questionnaire items that
minimise unintended context effects and maximise the reliability and validity
of the participants’ responses. Questionnaire items should be brief, relevant,
unambiguous, specific, and objective.
...........
16

2.2 GLOSSARY
Here are some basic terms related to the topic of designing a questionnaire that
you may not be familiar with, but which you need to know. They will come
up in your reading of the material in this lesson.

Design of a The questionnaire Die vraelys is ’n Lenanepotso
questionnaire is a structured gestruktureerde ke mokgwa o
Ontwerp van ’n technique for tegniek om primêre hlophisitsweng wa
vraelys collecting primary data in ’n opname ho bokella datha
data in a survey. te versamel. Dit is eo mofuputsi a
Moralo wa It is a series of ’n reeks geskrewe ipokelletseng yona.
lenanepotso written or verbal of mondelinge Ke letooto la dipotso
questions for which vrae waarop tse ngotsweng
the respondent die respondent kapa tse botswang
provides answers. antwoorde verskaf. ka molomo
A well-designed ’n Goed ontwerpte tseo moarabi
questionnaire vraelys motiveer a di arabang.
motivates the die respondent om Lenanepotso le
respondent to volledige en akkurate radilweng hantle le
provide complete inligting te verskaf. kgothalletsa moarabi
and accurate hore a fane ka lesedi
information. le felletseng, le
nepahetseng.
Respondents A person who ’n Persoon wat op Motho ya arabang
Respondente replies to something, iets reageer, veral ntho e itseng,
especially one een wat inligting vir haholoholo ya
Baarabi supplying ’n vraelys verskaf. fanang ka lesedi la
information for a lenanepotso.
questionnaire.
Questionnaire items Questions or Vrae of teks binne die Dipotso kapa
Vraelysitems text within the vraelys wat gebruik sengolwa se ka
questionnaire used word vir verwysing hara lenanepotso
Dintho tsa for reference by the deur die individu wat se sebedisetswang
Lenanepotso individual completing die vraelys voltooi. ho tataisa motho
the questionnaire. Vraelysitems verskil ya tlatsang
Questionnaire items met betrekking lenanepotso. Dintho
vary with regard to tot hoeveel hulle tse lenanepotsong di
how much they rely staatmaak op die fapana ho ya ka hore
on the respondent’s respondent se na di itshetlehile
interpretation, or interpretasie, of bied hakae tlhalosong ya
provide alternatives alternatiewe waaruit moarabi, kapa ho
from which a ’n respondent ’n fana ka dikgetho tseo
respondent chooses antwoord kies. moarabi a kgethang
an answer. karabo ho tsona.
...........
17 PYC 2616/1

Item scaling Scales are used for Skale word gebruik vir Dilekanyetsi di
Itemskaal rating performance, prestasiegradering, sebedisetswa ho
indicating aanduiding lekanya mosebetsi,
Ho lekanya dintho preferences, and van voorkeure ho bontsha dikgetho
expressing opinions en menings en le ho hlahisa
and attitudes. houdings. maikutlo.
Layout of a The questionnaire Die vraelysuitleg Tlhophiso ya
questionnaire layout or format of -formaat sluit lenanepotso e
Uitleg van ’n vraelys includes organising die uitleg van die kenyeletsa ho
the cover page, voorblad, instruksies hlophisa leqephe la
Tlhophiso ya instructions vir die voltooiing sekoahelo, ditaelo tsa
lenanepotso for completing the van die vraelys, die ho tlatsa lenanepotso,
questionnaire, the vraagvolgorde en die tatelano ya dipotso le
question sequence algehele vraelysuitleg tlhophiso kaofela ya
and the overall in. lenanepotso.
questionnaire layout.
Questionnaire A list of the ’n Lys van die vereiste Lenane la ditlhokwa
specification the required eienskappe vir die tsa lenanepotso.
document characteristics for vraelys. Eintlik moet Ha e le hantle,
Vraelysspesifi- the questionnaire. enige inligting lesedi lefe kapa lefe
kasiedokument Basically, any rakende die werklike le potapotileng
information vraelys en waarom lenanepotso la nnete
Tokomane ya ditlhoko surrounding the die verskillende vrae le hore hobaneng
tsa lenanepotso actual questionnaire ingesluit is (of nie) dipotso tse fapaneng
and why the different ingesluit word in die di kenyeleditswe
questions were spesifikasiedokument. (kapa ha di a
included (or not) kenyeletswa) di
should be included lokela ho kenngwa
in the specification tokomaneng ya
document. ditlhoko.
2 AC TIVITIES
material for this lesson (content domain, item format, layout of a
questionnaire and specification document).
Content domain Item format

Layout of a Specification
questionaire document
...........
18

T ime re cord
METHOD A specification document is a list of the required characteristics for your

questionnaire. It defines what your questionnaire should contain in terms of the
Activity 2.1
type of items, the number of items, layout and so on, in order for the questionnaire
Study the resource
material
to do what it is supposed to do. In this lesson, you will complete activities that
give you some practice in drawing up the questionnaire specification document.
Resource These activities serve as an example to help you to compile the specification
specification document for your own questionnaire. Study the resource on the specification
document document for a questionnaire.
Activity 2.2 The first step in drawing up the questionnaire specification document is to
Decide on the item decide on the kind of items you are going to use. This is referred to as the item
format and scaling
method format. Some of the items will take the form of rating scales and you need to
decide which items lend themselves to what kind of rating scales.
Resource
Layout of a
questionnaire
Item format
Content domain
AC T I O N 01
Identify different types of items and scaling methods. When you read the
resource on the layout of the questionnaire, you will see that it is important
to have a balance of different types of questions in order to maintain the
respondents’ interest and desire to complete the questionnaire as well as to
collect all the relevant information. In the resource on item format, different
types of items (or questions) are described. Look at the following items and
in the right hand column, write down what kind of item each one is. Indicate
whether it is an open question, closed question (and of which specific type)
or a rating scale (and again of which type).
Item Type
1. Do you have a valid driver’s licence? Yes/no
2. Why do people need to have a valid driver’s licence?
3. People should have a driver’s licence (choose one answer)
for identification purposes
to prove that they can drive
in case they have an accident
...........
19 PYC 2616/1

4. Young people are good drivers. True/false

5. Good drivers are
alert - - - - - relaxed
cautious - - - - fast reactors
older - - - - - younger
6. M
ark the characteristics of good drivers from the list
below:
male
female
even tempered
fast reactions
slow and steady etcetera
7. Are you a good driver? Rate your abilities as follows
a great deal very little
12345
self-confidence ……………….
experience ……………….
knowledge of road rules ……………….
8. Good drivers are ...
9. Should the age for driver’s licences be increased to 21?
ANSWER
The item types are:
(1) Closed question, because there is a limited choice of answers (yes or no).
(2) Open question, because it requires the respondents to state their own
opinions and allows for any kind of answer.
(3) Closed question, because there is a limited choice of answers (multiple-
choice type).
(4) Closed question, because there is a limited choice of answers (true/false
type).
(5) Rating scale, semantic differential type (extreme scale points are opposing
adjectives).
(6) Closed question, because there is a limited choice of answers (checklist/
inventory).
(7) Rating scale, Likert type.
(8) Open question, because it allows the respondents to give any kind of
answer.
(9) This looks like an open question but it is actually a closed question because
it implies only a yes or no answer. It would be an open question if you
asked, “What is your opinion about increasing the driving age to 21 years?”
...........
20

AC T I O N 02
Link the item format and scaling method to the purpose and content of your
questionnaire. In lesson 1 you decided on the purpose and content domain
of your questionnaire. Now you need to decide what kind of items to use in
order to get the information you want. The uses, as well as advantages and
disadvantages of different types of items, are discussed in the resource on
item format.
To get an idea of how to link item format and scaling method to the purpose
and content of the questionnaire, let us use the example of the questionnaire
to investigate whether South Africans experience crime-related stress, referred
to in the resource on identifying the content domain for a questionnaire. In the
blank spaces below, write down what sort of questions (that is, item format)
you would use to get the information you need. The first two are filled in as
an example.
Information required Example of item type

Age Closed
(checklist or multiple choice)
Gender Closed
(check male or female)
Socio-economic status
Personal experience of crime
Levels of stress
Personal reaction to different crimes
Perceptions of the effect of crime on
daily life
To find out the respondents’ age, you might just ask an open question like “How
old are you?” but some respondents might find this offensive or insensitive,
particularly older respondents. If your questionnaire is for adults, it might be
better to offer them a choice of answers, such as age groups. For example, you
could say, “Indicate your age group by marking the applicable box below”:
Under 18 18 to 22 years 23 to 35 years 36 to 50 years 51 years and

years older
The question of gender is easier because the options are obviously limited. A
closed question will suffice.
Socio-economic status is a little trickier. If you asked an open question, like

“describe your job” or “describe where you live”, you could get a meaningless
range of answers. It is probably better to use a closed question, providing
choices of different types of occupations or income groups. For example, you
could say, “indicate your occupation group”:
...........
21 PYC 2616/1

business academic clerical or sales semi- student unemployed

management administrative skilled
Personal experience of crime can be investigated in a variety of ways. If you

just want to know whether or not respondents have had personal experience
of crime, you can merely ask a closed question with a yes/no answer. If you
want to know how much or how often they personally experienced crime,
you can use a multiple-choice item or a rating scale. If you want a general
description, use an open-ended question.
Measuring levels of stress lends itself to a rating scale because you want more
than just factual information. You could draw up a list of crimes and ask the
respondents to rate how much stress they associate with each type of crime.
For example, people might not worry much about drunkenness in public but
may be very worried about hijacking.
Descriptions of reactions to crime can be obtained with a simple open-ended
question or you might try a rating scale like a semantic differential. An open-
ended question is also suitable for the question of how crime affects people’s
daily lives.
Now look at the options you have listed and decide on the type of question
for each topic, bearing in mind that you want a balance of different types of
questions. For example, if you use an open question to measure personal
experience of crime and perceptions of the effect of crime, rather use a
multiple-choice item or a semantic differential (than another open question)
to measure personal reactions to different crimes.
The next step is to apply this process to your own questionnaire. In activity 1.2
(lesson 1), you wrote what information you need to get for your questionnaire.
Make a list of the information here (like the example you have just worked
through) and write possible question types for each. The first line has
been completed as an example.
Information required Example of item type

Biographical information Closed (checklist)
Check through your list to make sure that you have included a balance of
different types of items. Make sure that you have some closed questions, some
open questions and at least one rating scale.
Activity 2.3 The length of a questionnaire is not random. It has to be considered carefully
Decide on the total to ensure that you get the information you want but do not lose respondents
number of items because it is too long or boring. This topic is discussed in the resource called,
Resource layout of the questionnaire. The first step is to identify the extent to which each
Layout of a
content area (the information you need) needs to be covered; the second step
questionnaire is to consider the characteristics of your respondents and the time available
for testing.
...........
22

AC T I O N 01
You need to identify the coverage required for each content area. One of the
considerations in deciding on the number of items for your questionnaire is
the number of different smaller content areas or constructs covered by your
questionnaire. In the previous activity, you used the example of an investigation
of South African’s experiences of crime-related stress and for this investigation,
five content areas were identified. These were biographical information,
personal experience of crime, levels of stress, reactions to different crimes, and
the effect of crime on daily lives. You therefore need at least one item on each
of these content areas. In some cases, one item is not enough. For example,
if you want information on stress levels associated with different crimes, you
might want to use a rating scale. Rating scales do not have a fixed number of
items (or things to be rated) but for the purposes of this assignment, your rating
scale should consist of at least twelve items. It is also useful to have more than
one item dealing with the same aspect to serve as a control so that you can
see whether the respondent is answering questions consistently or not (see
the resources on “reliability” and “validity”). For example, in addition to your
rating scale, you might also have an open-ended question that deals with the
same content area.
In the previous activity, you made a list of the information you need to investigate
the topic. In this way, you have in fact already identified your smaller content
areas that make up the focus of your investigation. In the space below, write
down your list of smaller content areas and next to each one, make a note of
the coverage required. The most important particular content area may require
more than one item whereas less important areas will only require one item.
The first one has been completed as an example. You can also use the example
of the questionnaire on crime as a guideline.
Content area Coverage

Biographical information Three items
You have now identified the coverage required but you may have to modify it
because of the type of respondents who will complete the questionnaire and
the time they typically will have available.
AC T I O N 02
You should evaluate the impact of characteristics of respondents and the time
available for completing the questionnaire. Read the guidelines in length of
the questionnaire in your resource on the layout of the questionnaire. Now
write down the most important factors to take into account when deciding on
the length of a questionnaire.
...........
23 PYC 2616/1

Apply this information on the length of a questionnaire to the example discussed

in the previous exercise (South African’s experiences of crime-related stress).
Considering these guidelines and looking at the different types of questions as
well as the coverage of content areas you decided on in the previous exercises,
write down how many items you think you should include in this questionnaire
and give reasons for your answer.
I think there should be approximately … items.
Content area Number of items

Personal experience of crime
Levels of stress associated with different crimes
Personal reactions to different crimes
Perceptions of the effect of crime on daily life
Other
You could probably cover the content domain comprehensively with 21 items
(some of which may be grouped into a rating scale containing approximately 12
items). We could break down the coverage of the content areas as follows: the
first three items would be closed questions to collect biographical information
(first content area), then a filter question (closed, yes/no type) followed by an
open question on personal experience of crime (second content area), a rating
scale (consisting of 12 items) on levels of stress associated with different crimes
(third content area), a closed (multiple-choice) question on personal reactions
to crimes and an open question to serve as a control (fourth content area), an
open question on perceptions of the effect of crime (fifth content area) and
lastly an open question for any other comments the respondent may wish to
add. We therefore have five closed items, four open items and a 12-item rating
scale (total of 21 items).
If you want to involve people from all strata of society as respondents for
your questionnaire, remember to make your items straightforward. Your
respondents may not be specialists but rather the person in the street, therefore
the questionnaire should not be too long or complicated. Most people lead
busy lives and do not want to spend much time completing the questionnaire.
The average person should be able to answer these items in half an hour.
Some of the items may be answered very quickly (for example, the questions
on biographical information or the checklist), while others may take a little
longer (open-ended questions, for instance).
Now you are ready to apply this exercise to your own questionnaire. Using
the previous exercise as a guideline for your own questionnaire and looking
at the different types of items as well as the coverage of content areas you
decided on earlier, write how many items you think you would need for the
entire questionnaire. Also, list the content areas and the number of items for
each one. The first item is filled in as an example.
...........
24

I think there should be approximately … items.
Content area Number of items

Now that you have decided on the length of your questionnaire, you need to
consider the layout or the way in which your items will be presented.
Activity 2.4 First impressions are important! This means that the layout of your questionnaire
Design the layout is important for conveying the idea that it is worthwhile completing the
of the questionnaire questionnaire. Nobody wants to bother with a sloppy document that is not
Resource professional. In activity 2.4, you will be given the opportunity to work on the
Layout of a layout of your questionnaire.
questionnaire
AC T I O N 01
Provide an introduction and instructions for completing the questionnaire. By
referring to the resource on the layout of the questionnaire, write a suitable
introduction to your questionnaire.
Introduction: ……………………………………........................…………………………..
Of course you can include anything you want to in your introduction but it is
suggested that you include at least the four main points provided in the layout
of the questionnaire resource.
AC T I O N 02
Now organise your items in a logical and appropriate sequence. When you
read through the resource on the layout of the questionnaire, you will see that
the presentation and sequence of items are important. Before you decide on
your own questionnaire, let us practice on the example we worked on in the
previous section, that is, the investigation of South Africans’ experience of
crime-related stress. You have already identified approximately 21 items (in
the previous exercise) for this questionnaire. Now organise these items in a
logical sequence, using the guidelines in your resource material (presentation
and sequence of questions in the layout of the questionnaire resource). Start
by asking yourself whether there are any sensitive or leading questions and
whether you need a filter question. Once you have identified these, it makes
the organisation easier.
Sequence of the presentation of items: ……………………................……………..
Our suggestion would be to start with a broad question that orients the
respondent to the topic, such as an open-ended question on people’s perceptions
of the effect of crime in South Africa. This could be followed by the 12-item
...........
25 PYC 2616/1

rating scale for stress levels associated with different types of crime. (Can you
see that we are moving from the general to the more specific?)
We might then introduce the topic of personal experience of crime and use a
filter question followed by an open question about people’s personal experience
of crime. Personal reactions may be regarded as a sensitive topic and therefore
these items should be placed towards the end of the questionnaire. You might
want to use a closed question and an open question to serve as a sort of validity
check for this content area.
Items on biographical information (age, gender, socio-economic status, etc)

could be placed at the beginning of the questionnaire or at the end. If there
are only a few items, they would not be out of place at the beginning but if
there are a lot of items (for example, address, occupation, income, marital
status, type of dwelling and possession of a car), it would be better to put
these at the end. Lastly, provide an open question for additional information
that respondents may want to express.
Now that you have worked through the example, you are ready to tackle your
own questionnaire. Using the guidelines provided here, look carefully at the
items you identified for your own questionnaire and then provide the sequence
in which you think they should be presented.
Sequence of presentation of items: …………………...................……………………..
At this stage, you should have successfully compiled a questionnaire specification

document, indicating the different types of items and scales you are going to
use, the number of items and the layout. This provides the outline and now
you have to fill in the detail. In the next lesson, you will find guidelines about
the formulation and wording of good questions and items.
Attitude and intention questionnaire (AIQ)

Instructions
Please read the following questions carefully and indicate your response
to each question by ticking the answer that best describes how you feel
about the issues raised by these questions. Please answer all questions. Do
not spend too much time thinking about your answers. The questionnaire
should not take long to complete.
...........
26

Section 1: Biographical information

1.1 My gender is:
(a) Female […]
(b) Male […]
Section 2: Attitude towards alcohol consumption
2.1 My parents’ alcoholism had a devastating effect on our family
(a) True […]
(b) False […]
Section 3: Recreational behaviour
3.1 My opinion about participating in extramural activities is:
...............................................................................................................................
...............................................................................................................................
QUESTION 1
Ben: Do you think the instructions that the AIQ provides to respondents
are correct and complete?
Temba: The instructions provide the correct information but are incomplete.
Bets: I think the instructions provide the correct information and they
are complete.
Fauna: No, I think the information provided by the instructions is incorrect

and the instructions are incomplete.
You agree most with, second most with and least with:
(1) Temba, Bets and Fauna

(2) Fauna, Bets and Temba
(3) Temba, Fauna and Bets
(4) Bets, Temba and Fauna
(5) Fauna, Temba and Bets
QUESTION 2
Question 3.1 in the scenario is an example of which type of item format? It
is a/n ...
(1) rating scale.

(2) open question.
(3) checklist.
(4) None of the above.
...........
27 PYC 2616/1

QUESTION 3
You want to ask respondents whether they have used recreational drugs but
you realise that this may be a sensitive question. Where is the best place to
put a sensitive question like this?
(1) In a separate section at the beginning to get it out of the way.

(2) Include it in section 1.
(3) At the end of section 2.
(4) Near the end of section 3.
Q U E S T I O N 1: O P T I O N 5 I S T H E K E Y – FAU N A , T E M B A A N D
BETS
The instructions are incomplete because they do not refer to the intention of
the questionnaire and the matter of confidentiality. There is also a problem
with the information that is provided. It asks the respondents to respond to
each question by ticking an answer. Some questions cannot be ticked.
Please note that questionnaire instructions refer to how the questionnaire should
be completed. It does not refer to the instructions provided with each question.
Q U E S T I O N 2: O P T I O N 2 I S T H E K E Y
Item 3.1 is an open question as respondents are required to provide their own
“personal” answers. Closed questions (such as item 2.1) provide respondents
with a limited choice of alternate answers. In item 3.1 the respondents do not
receive a choice of alternative answers, and therefore this item is not a closed
question.
Sensitive or very personal issues should be near the end of the questionnaire.
The respondents, if asked to respond to such questions in the beginning, may
be unwilling to complete the rest of the questionnaire.
2.3 REFERENCES
Prabhu, RK. & Rao, UK. (1960). Mind of Mahatma Gandhi. India: Navajivan Trust
...........
28

Lesson
3 3
3 Writing questionnaire items
In this lesson, you will learn more about the process of writing questionnaire
items.
AIMS
In lesson 3, the focus is on the importance of effectively writing questionnaire

items. You will also get to understand why issues of language, culture and histori-
cal context are important when writing questionnaire items. You will be given
activities with examples to teach you how to apply your knowledge.
Lesson
• understand different types of questionnaire items:
(1) open-ended questions

(2) closed-ended questions
• understand components of closed-ended questions:
(1) categorical variables

(2) rating scales
• understand sociocultural considerations in writing questionnaire items:
(1) colonial history of research

(2) the importance of cultural understanding
(3) the importance of language
• writing questionnaire items
...........
29 PYC 2616/1

Establishing cultural under standing
Questionnaires should communicate the realities of the people being studied. An

understanding of the interconnection between various sociocultural, linguistic
and economic aspects and how they relate to the study is therefore necessary.
This means that before undertaking a research project the researcher needs to
ensure that they have studied the culture of the group being researched. Their
language and how the language is used within their context are important.
Questions on how they prefer to be labelled, if they prefer labels, and how they
would feel about being researched, are very critical. For example, the contesta-
tions found in literature on disability clearly illustrate the problems that often
arise when researchers define communities. In recent years, literature has been
challenged for using either of the following:
• people with disabilities

• people living with disabilities
• disabled persons
One of the ways to help resolve this is to asked the people concerned how they
would like to be referred to, thus making them the centre of your item devel-
opment. Your questionnaire should consist of items that are not foreign to the
people you are studying and the researcher should not impose a particular way
of understanding to his/her participants.
3.1 INTRODUCTION
In this lesson, you will be taken through the process of writing questionnaire
items. You will learn about the important things you need to take into
consideration when writing the items and what you need to avoid.
Questionnaire items could be defined as a series of questions developed

with an aim of data collection for research purposes. The development of
questionnaire items is the most critical stage of the survey research process
as it determines the quality of the data a researcher would get. In the field of
research there is a famous saying, “garbage in, garbage out”, which simply
means that the quality of your questionnaire items will determine the quality
of the data you receive. It is therefore important to pay careful attention when
writing questionnaire items for your research.
3.2 CONSIDERATIONS IN FORMULATING QUESTIONNAIRE ITEMS
3.2.1 Colonial history of research still shapes the field

It is important to take into consideration the history of the field of research in
a country like South Africa and how it is still relevant in the manner in which
questionnaires are designed. As a former colonised land, South Africa has been
subjected to western research strategies that have led to the dehumanisation
of black people. To this day, research in South Africa is still conducted with
...........
30

a colonial lens where only western ways of knowing are prioritised. Most
research studies still target black communities and western assumptions are
imposed in the way data is gathered.
A questionnaire is a data collection tool designed in the west and it is based

on western assumptions on how data should be collected. It is therefore
difficult to turn such a tool into something that takes into consideration
African ways of knowledge production. In order to do that, we would need
to discard the paradigm and the methodological framework from which
the instrument emanates. However, what we could try to do is to caution
researchers to be always mindful of the worldview the research tools they are
using are communicating. Researchers are trained in the western education
system; they are therefore communicating a western culture. This culture is
often subtle and has been internalised in such a way that it has become normal
and hard to pinpoint as being problematic – for example, questions such as
• What is your highest level of education?

• What is the household income?
The above questions are communicating a western viewpoint. When

questionnaires ask about one’s level of education, what is being asked is the
level of western education a person has attained – that means, has the person
been through a westernised system of formal education. This disregards the
indigenous education and other forms of education that might exist. This is
the case because the whole aim of colonialism was to replicate the coloniser’s
systems and ways of being and to inferiorise the ways of the colonised. As
a result, there is still a generation that does not expect anyone not to have
attained some form of western education.
The second example on household income also communicates a similar

western worldview as the first example. Income is constructed based on
western formulations of the capitalist remuneration system. It does not take
into account the fact that there could be other economic systems that do not
measure household subsistence in monetary terms.
These are the aspects we are urging you, as a student, to be critical of. It is
always important to ask yourself whose reality am I using to measure this
experience. The problem with the two questions is that western concepts
are used to measure experiences of people in non-western contexts and the
result is the production of knowledge which pretends to be of indigenous
people. What decolonial researchers such as Chilisa (2012) propose is to allow
indigenous people to express their experiences through their own methods.
When writing questionnaire items, think carefully about culture. It is important

to acknowledge that everyone has a culture and the education obtained
from a university imparts a culture of the world of its conception. It is also
important to remember that the culture of the education system is different
from the lived realities of the people. That means communities have a
culture of their own that researchers need to understand and must avoid
misrepresenting. This makes community involvement during the planning
stages of questionnaire development critical and will prevent making the
mistakes past researchers in Psychology, Anthropology and Sociology made
...........
31 PYC 2616/1

when using the conceptualisations of the western world when studying non-
western communities. In doing so, they distorted those communities’ cultures
and presented them as backward and resistant to change. As widely reported,
no culture can be perceived as static but changing one’s ways of doing should
not be imposed. The western education has expected other nations to assimilate
its cultures instead of evolving on their own.
3.2.2 Language
Another critical aspect, which is connected to the issue of culture mentioned
above, is the need to consider language when developing questionnaire items.
Researchers are always advised to use simple and clear language. It is important
to remember that often times your respondents will come from different walks
of life and may not be familiar with the terms that social science researchers
think are basic. Researchers must therefore avoid using jargon (language only
understood by those in one’s field of work).
It is important to always remember that a questionnaire is something that

the intended respondents have to grasp with ease. This will help you gather
quality data and thus produce reliable knowledge. As part of ensuring that
your questionnaire is as clear as possible, please avoid doing the following.
• Leading questions – these are the types of questions that can sway the
respondent in a particular direction. They are also called suggestive questions.
The respondent might get a hint on how you want them to answer the
question. For example, “Isn’t it safer being at school than at home?”
• Double-barrelled questions – these questions require more than one answer.
The problem with it is that the respondent would not know which part
of the question to respond to and the researcher would have difficulty in
analysing responses from such questions. For example: “Does the university
consider race and gender in its admission policies?” An answer could apply
to only one variable and not the combination.
• Avoid using presumptions – this applies in instances where the researcher
assigns participants into particular categories. An example of a presumptuous
question would be: “What type of cereal do you eat in the morning?” The
researcher has already assumed that the respondents eat cereal in the
morning.
• Ambiguous questions – these are the types of questions that could have
multiple meanings. The respondents could interpret the questions in many
different ways. For example, if you ask students “Are you satisfied with
the curriculum?” the students could be satisfied with some aspects of the
curriculum and dissatisfied with others. The researcher will also not reap
the intended rewards from the question as the responses will not help him/
her to work on the aspects of the curriculum that require an improvement.
It is also important to be cognisant of the sociodemographics of the country.

The majority of the people in South Africa are not English speaking yet most
research studies are formulated in the English language. This has always been
resolved through the process of translation where the researcher has to translate
every questionnaire item into the language of the respondents. In a multicultural
society like South Africa where there are eleven official languages, some studies
...........
32

would have the same questionnaire in three versions. This is to ensure that
everyone is responding in the language he or she truly understands. However,
this does not resolve the problem of the culture of the academy we introduced
in the last section. The variables being measured are still formulated with a
western lens. If the research is measuring the level of education, education will
still be based solely on western terms. It will not include indigenous definitions
and formulations. It therefore becomes important to rather use language to get
an in-depth understanding of the knowledge these languages carry.
• What can we learn from indigenous languages about indigenous ways of
being?
• What do these languages tell us about their systems?
These are the questions future studies need to help us understand if we are
to truly deal with the remnants of the colonial past embedded in the ways of
doing research.
3.3 TYPES OF QUESTIONNAIRE ITEMS

When writing a series of questions for your questionnaire you can either use
open- or closed-ended questions. Open-ended questions entail the type of
questions where respondents are given the opportunity to provide their own
responses. Researchers often use these types of questions when they are not
sure how respondents might respond to their questions. They are also used
when the researchers do not want to influence the responses. Below are a few
examples of open-ended questions:
• What is the main thing you would like to see in the new Psychology
curriculum?
• How does learning in English make you feel?
• Is there anything else you would like to suggest about curriculum
transformation?
Open-ended questions require participants to share their thoughts about a
phenomenon being studied and those thoughts would be transcribed and
coded before they are analysed.
Closed-ended questions entail a type of question where each question is
followed by a set of response options a respondent has to choose from. Unlike
open-ended questions, closed-ended questions are easy to code and analyse.
They can involve either categorical variables or quantitative variables. Examples
of categorical variables are sex, race and religious group while quantitative
variables use rating scales where responses are ordered and the respondents
can pick an option.
Example of categorical variables:
To which population group do you belong?
African
Coloured
Indian/Asian
White
...........
33 PYC 2616/1

Example of a rating scale:

The researcher presents a respondent with a statement about an idea they
want to measure and provide ranked options.
The increase of value added tax (VAT) is unfair to the poor.
(1) Strongly agree
(2) Agree
(3) Neither agree nor disagree
(4) Disagree
(5) Strongly disagree
When using the closed-ended method of questioning, the researchers are
advised to have the category “other” which should be followed by a space
to provide an explanation or an alternative response. This is done in order to
accommodate responses that fall outside the given options.
For example, there are people who may not fall in the population group
categories listed above such as the Khoi-san people; an “other” category in
such an instance would help.
3.4 GLOSSARY
Here are some basic terms related to aspects of questionnaire item writing
discussed above. They will aid your understanding of the material on this topic.

Categorical variables Variables that Veranderlikes Dintho tse ka
Kategoriese can be grouped wat in kategorieë fetohang tse
veranderlikes into categories by gegroepeer kan arotsweng
assigning a label word deur ’n etiket mekgahlelo ka ho
Dintho tse ka (qualitative, e.g. (kwalitatief, bv kleur, di kenya leibole
fetohang tsa colour, race, religion) ras, godsdiens) of ’n (mokgahlelong wa
mekgahlelo or a number. nommer daaraan toe boleng, mohl. mmala
te ken. morabe, bodumedi)
kapa nomoro.
Rating scale It is an assessment Dit is ’n Ke mokgwa wa
Waardebepaling technique where assesseringstegniek tekolo moo dikgetho
response options waar die tsa dikarabo di
Mokgwa wa are ranked in a responsopsies in fuwang maemo ka
tekanyetso systematic order. ’n sistematiese tatelano e itseng.
volgorde gerangskik
word.
Indigenous Ways of knowing Maniere van weet, Ditsela tsa ho
knowledge that are embedded wat in inheemse tsebisana meetlo le
Inheemse kennis in indigenous mense se tradisies en ditlwaelo tsa batho
people’s traditions kulture ingebed is. ba botjhaba bo
Tsebo ya dintho tsa and cultures. itseng.
botjhaba
...........
34

Indigenous A language that is ’n Taal wat inheems Puo e buuwang ke

languages native to the people aan die mense van batho ba naha e
Inheemse taal of the land. die land is. itseng.
Dipuo tsa lehae
Colonialism Invasion of and Die inval en Ho nka naha eo e

Kolonialisme forceful settlement kragdadige vestiging seng ya heno ka
in foreign lands and in vreemde lande dikgoka le ho rala
Bokolonale designing policies en die instel van dipholisi le mesebetsi
and practices that beleide en praktyke e netefatsang
ensure systematic wat die sistematiese tshebediso e mpe ya
exploitation of uitbuiting van ditlhokwa tsa baahi,
those lands’ cultural, daardie lande se botjhaba le moruo
social and economic kulturele, sosiale wa bona.
resources. en ekonomiese
hulpbronne verseker.
Western ways Ways of knowing and Kennis en begrip van Mekgwa ya ho
Westerse maniere understanding the die wêreld wat in tseba le ho utlwisisa
world that are rooted westerse kulture en mekgwa ya bophelo
Mekgwa ya ba in western cultures tradisies gewortel is. ba basweu .
basweu and traditions.
3 AC TIVITIES
material for this lesson (writing questionnaire items).
Writing
questionnaire items
Press the button
or go to page 147 in
the study guide.
AC T I O N 01
METHOD In Lesson 02 you specified the content areas for your questionnaire together
Activity 3.1
with the number of items and type of items for each content area. You have
also indicated the sequence of presentation of these items. Keeping in mind
Write items for a
questionnaire the criteria discussed in the previous activity, you now need to write the items
for your questionnaire as specified in Lesson 02.
Resource
Writing
questionnaire
items
...........
35 PYC 2616/1

AC T I O N 02
Evaluate these items. Do they all comply with the general principles for
constructing new items? If they do, then well done. If some, or all, of your items
breach one or more of the general principles, then, (a) write each ‘incorrect’
item, in its original form (b) indicate why the item is ‘bad’, and (c) reformulate
the item so that it accords with the general principles for constructing ‘good’
items.
Use the following layout as an example:
Original item This item is not good Improved version of

because … item …
AC T I O N 03
You can now compile a draft version of your questionnaire consisting of an
introduction (refer to Lesson 02) and the items you have written (and corrected
where necessary). Note that with the rating scale you need to indicate the
rating associated with each number on the questionnaire. Examples are: 1 =
disagree entirely up to 5 = agree entirely; 1 = strongly disagree up to 5 =
strongly agree; 1 = never up to 5 = always.
QUESTION 1
Below is a list of questionnaire items. Please look at them carefully and based
on what you have learnt above indicate what is problematic about the way
there are formulated.
A. Most children start their schooling at the age of six years. Were you also
six years old when you started school?
B. How far are you from the nearest hospital and how often do you visit it?
C. Are you satisfied with township life?
D. What is your favourite fizzy drink?
...........
36

QUESTION 2
Do you remember that we cautioned you to be mindful of the culture the
questionnaire items are communicating? Can you explain how a particular
culture is communicated and presented as a norm in the first two questions?
QUESTION 1
• The first question is a leading question. The respondent is led into a particular
direction.
• The second question is a double-barrelled question. It is asking two things
at once.
• The third question is ambiguous, it is too broad, and it does not ask about
specific aspects of township life.
• The fourth question is presumptuous; the researcher has assumed that the
respondent takes fizzy drinks.
QUESTION 2
• As we indicated before, formal schooling is a western method of education.
This excludes people who acquire their learning outside the schooling
system. The first question is therefore communicating a western viewpoint.
• The second question is measuring only one form of a healthcare facility,
which is a hospital. It is also communicating a western cultural point of view.
It is only asking about one form of healthcare, which is a hospital. There
could be different types of healthcare facilities that are not of western origin.
...........
37 PYC 2616/1

Lesson
4 4
4 Pilot-test the questionnaire
This lesson deals with pilot-testing the initial version of the questionnaire.
AIMS
In this pilot-testing phase of developing your questionnaire, you will learn how
to distinguish between strong and weak items. You will be guided by activities
and examples through which you will administer the questionnaire and you will
make use of item analysis procedures to revise the questionnaire.
Lesson
This lesson will help you to produce a set of items to be included in the final
version of the questionnaire.
• correlation coefficient
• item analysis
In multicultural environments such as South Africa, it is important that what you

will be researching/studying should be a true representation of the people you
are studying. It is important that when constructing or designing your question-
naire your questions should be based on the people’s lived experiences, their
culture, socio-economic status, language and how all these interact. In certain
instances, you can use a group of individuals you want to study to assist you in
the development of the questions. The questions in your questionnaire should
mimic their context and affirm their realities, meaning that your group of interest
will be at the centre of your research study and assisting you to write relevant and
relatable questions for them. In this case, it important to check and make sure
...........
38

that the concepts and words you are using will not be imposed on others or that
the concepts you are using are not foreign to other people’s language, culture
and lived experience and conditions. When the questionnaire is constructed,
you must make sure that you do not portray other people’s ways of living and
their experiences as superior to others (i.e. placing others’ way of being and
concepts at the centre and translating from those experiences for other people).
The above-mentioned information is important when piloting or testing whether

your questionnaire works – meaning will your questionnaire speak to its intended
group. The piloting phase will be done with your intended group of interest,
whether it is a group of a specific culture, gender, or age to name a few. In the
paragraph above, we mentioned that it could be beneficial, using your group
of interest for the constructing phase of the questionnaire. When you are pilot-
ing, you will be using the same people in that group of interest but individuals
who were not part of the developing phase of the questionnaire. The use of the
second group will be to confirm whether the questionnaire will be relatable to
other people within that group of interest or whether they will be able to identify
with the questions asked.
The two paragraphs above stress that it is important to do an analysis of who

is included in your sample/population/group of interest. When a multicultural
target population or sample (i.e. a sample with people from different races, cul-
tures and genders) is considered, it is especially important to think about issues
such as cultural background and first- versus second-language speakers of the
language used in the questionnaire, and the cultural lens used in developing
the questions and how it will influence your results. It is also important that you
are contextually sensitive and aware of how this might affect your results. When
you have people from the same cultural, socio-economic and language group,
it is easier to avoid such issues.
The sample used for piloting will be able to assist you in identifying biased con-
tent or items from their responses. This would help in assessing whether your
questionnaire is addressing what you want to measure. This will be a step towards
refining content and concepts in your questionnaire in order to produce good
quality information and to provide you with the best possible understanding
of the topic you are investigating.
4.1 INTRODUCTION
At this stage of the questionnaire development, you will learn how to assess
your items, by assessing the validity of your concept/content of the items of
the questionnaire in relation to its sample and with the use of certain statistical
tools (i.e. item analysis). Through this process, you will be able to distinguish
between good/strong items and weaker items. By weeding out weak items,
you get a better quality measurement of the issue you intend to investigate –
similar to focusing the lenses of a pair of binoculars. The pilot-testing of the
questionnaire also gives you an assurance that the information you gather
is actually dealing with the topic you want to investigate. You want to gain
assurance that the data you gather when you eventually administer the final
...........
39 PYC 2616/1

questionnaire is not contaminated or obfuscated by the content of the questions

and the way they were posed.
4.2 GLOSSARY
Below are brief explanations of some basic terms you will encounter in this
lesson. Acquaint yourself with their meaning in order to have a better grasp
of the learning content.

Item analysis Refers to ways in Verwys na maniere E bolela ditsela
techniques which participants waarop deelnemers tseo bankakarolo
Itemontledings- respond to individual op individuele vrae ba arabang dipotso
tegnieke questions (items) in a (items) in ’n vraelys ka bonngwe
questionnaire. reageer. lenanepotsong.
Mekgwa ya ho
hlahlobisisa ntho
Rating scale A method used to ’n Metode waar Mokgwa o
Beoordelingskaal record responses verskillende sebedisetswang
using different kategoriëe gebruik ho ngola kapa
Mokgwa wa categories to answer word om antwoorde ho nka dikarabo,
tekanyetso a question. op ’n vraag te meet. o sebedisang
mekgahlelo e
fapaneng ya ho
araba potso.
Pilot-testing Process involving the ’n Proses waardeur Tsweletso ya ho
Loodstoetsing administering of a ’n studie/vraelys op etsa phuputso/ho fa
study/questionnaire ’n klein steekproef sehlopha se senyane
Ho etsa teko ka to a small sample toegepas word lenanepotso ho bona
sehlopha se se nyane to determine its ten einde die hore morero o a
functionality. bruikbaarheid atleha.
daarvan te bepaal.
Sample A limited number of ’n Beperkte aantal Palo e
Steekproef individuals selected individue wat uit lekanyeditsweng
from a larger ’n groter populasie ya batho ba
Sampole population. geselekteer is. kgethilweng palong
ya ba bangata
Variance Indicates the extent Dui op die mate E bolela boholo
Variansie to which differences waartoe verskille ba phapano e
occur. voorkom. bonahalang.
Phapano
Item discrimination Refers to the extent Verwys na die mate Ho bolela boholo
Itemdiskriminasie in which answers to a waartoe antwoorde boo dikarabo tsa
question differentiate op ’n vraag tussen potso di fapanang ka
Ho fapana ha dintho between two or twee of meer groepe bona dihlopheng tse
more groups. verskil. pedi kapa tse fetang
moo.
...........
40

Correlation Indicates the Dui op die sterkte E bolela matla a

Korrelasie strength of a van ’n verhouding kamano dinthong tse
relationship amongst tussen twee of meer pedi kapa ho feta.
Kamano two or more items. items.
4 AC TIVITIES
It is important that you read the resource material for this lesson (correlation,
item analysis and writing questionnaire items).
Correlation Writing
coefficient Item analysis questionnaire items
Press the button Press the button Press the button
or go to page 114 in or go to page 118 in or go to page 147 in
the study guide. the study guide. the study guide.
It takes about two hours to do the activities.
T ime re cord
METHOD Constructing a questionnaire is a creative activity. Part of the act of creating

something new, whether it is a work of art or a questionnaire, is to stand
Activity 4.1
back, look at what you have done, and consider how to improve it. In lesson
Administer
3, you started this process of improvement by rewriting items that were
and revise the
questionnaire double-barrelled, vague or otherwise deficient. In the course of this lesson,
you will improve your questionnaire further by actually trying it out and
seeing how people respond to each item. In particular, you will use simple
item analysis techniques to improve the 12- item rating scale that forms part
of your questionnaire. Item analysis allows you to identify items that do not
work properly and to discard them so that you end up with a shorter, better
scale containing only the best possible items.
AC T I O N 01
You need to identify a suitable sample. You already have a draft questionnaire
and an introductory talk, but to actually administer your questionnaire (i.e., to
try it out), you must get hold of people to try it out on! Like any other sampling
situation, the sample should represent the population to which you hope to
generalise your findings.
You have already described the target population for your questionnaire (the
population that your questionnaire is intended for), but you may want to have
...........
41 PYC 2616/1

another look at what you did in lesson 1. Do you still agree with how you
described your target population then, or do you want to change anything?
Write a short description of how you now see your target population.
In an ideal world, you would be able to select your sample from the target
population using sophisticated sampling techniques such as stratified random
sampling. However, for the purposes of pilot-testing a questionnaire, researchers
rarely go to such great lengths to find a truly representative sample. Instead,
they simply get together a sample that appears to be as similar as possible to the
target population. So if your questionnaire is meant for both men and women
and for people with a university degree, try not to choose a sample consisting
only of men who have never been to university. If your questionnaire is meant
for managers, do not use a sample consisting of manual workers.
You do not need hundreds of people for pilot-testing your questionnaire. For
purposes of item analysis it is usually said that one needs at least one more
person than there are items in one’s scale. Since you want to analyse your
rating scale and there are 12 items in your rating scale, that means you need
at least 13 people. We can relax this rule a bit and say you need between 10
and 15 people in your sample.
Now give some thought as to where you will get hold of such a sample. You
could ask your friends and family to try out the questionnaire, provided they
are similar to your target population. Another good starting place is where
you work or study – maybe you could ask suitable people there to do the
questionnaire. Your church or any other organisation to which you belong
can also be a good place to find a sample. You need not administer your
questionnaire to everybody in the sample at the same time, and not everybody
in the sample need to come from the same place.
Having given the practicalities of getting hold of a sample some thought, write
some notes about the place or places where you expect to get your sample from.
Finding a sample is a matter of balancing practical issues with theoretical

requirements. In theory, you would like your sample to be representative
of the target population, but in practice that is never possible. For example,
you may want your questionnaire to be used on people of all race groups
in South Africa, but may live and work in a context where you can only get
hold of people of one race group. We do not expect you to have a perfectly
representative sample, but simply to be able to explain in what ways your
sample is less than perfectly representative of your target population. Now
write a short description of how your sample is or is not representative of your
target population. Characteristics of the sample you could mention include
home language, age, gender, educational level and socio-economic status.
...........
42

AC T I O N 02
Administer the questionnaire to the sample. If you are lucky, you will be able
to get all the people in your sample together in one place (for example, they
may be a group of school children) so that they can all do the questionnaire at
the same time. However, you may have to get people to do the questionnaire
one at a time or in small groups. In whatever way you decide to administer
the questionnaire, be sure to be ethical about what you are doing. Use the
following checklist to make certain that you are acting ethically. The answer
to each of the questions should be YES.
Ethical checklist Yes/No Notes

Respondents understand why they are being
asked to complete the questionnaire:
• It is to help you with your studies.
• It is to help you improve the questionnaire.
• They will not get their scores back.
Respondents understand that they do not have
to complete the questionnaire:
• You will not hold it against them if they
decide not to do it.
• You will not tell anybody else if they refused.
Respondents understand that their responses will
be confidential:
• Their names will not be on thequestionnaire.
• You will not show their responses to anybody
else.
When you administer the questionnaire, it is best to find a quiet place where
people will not be disturbed and to allow them as much time as they need. It is
usually a good idea if you read the introduction at the top of the questionnaire
out loud – especially if you have a group of people together. Then answer
any questions they have. While they are completing the questionnaire, glance
at what each person is doing to ensure that they have the right idea of what
is required, but do not look over people’s shoulders the whole time – some
people find it very irritating! Be available to answer any questions people may
have. Of course you cannot tell people the right answer to any question since
there is no right answer and it is up to them how they want to respond, but
you can help them where something seems unclear. Keep notes of the kind
of questions people ask and what difficulties arise – you can use these notes
later to make improvements to your questionnaire.
Summarise the notes you took while administering the questionnaire. These can
include, for example, places where the instructions were unclear, notes about
particular items that seem to be giving people trouble, and notes about any
unusual events that occurred while you were administering the questionnaire.
When you get the questionnaire back, it is a good idea to quickly scan it to see
that they have completed all of it. People often inadvertently leave out some
...........
43 PYC 2616/1

parts (for example, biographical information), and the less missing information
you have to deal with the better. If anything is missing, ask the person if he
or she would mind quickly completing that part of the questionnaire as well.
You should now have a collection of 10 to 15 completed questionnaires and

notes about what happened during the pilot administration.
AC T I O N 03
There are two ways of using a pilot study to improve your questionnaire. The
first is to use what happened during the study (for example, people asking
questions about some of the items, or people getting confused about the
instructions) to revise the questionnaire. Look again at the notes you made
while you were administering the questionnaire and then see where you can
improve the questionnaire. Were the instructions clear enough? Were there
some questions that people did not quite understand? Were there questions
that offended some people? Now write a short summary of what changes (if
any) you have decided to make to your questionnaire.
Activity 4.2 The second way of using a pilot study to improve a questionnaire is to analyse
Do an item analysis the responses people gave to each item in the questionnaire. This is known as
item analysis. Some professional test constructors use sophisticated statistical
Resource
procedures in their item analyses, but even without any statistical tools, it is
Item analysis
possible to do the two most essential types of item analysis: remove items with
Writing the too little variance and remove items that do not discriminate (see the resource
questionnaire
items on item analysis). To keep things simple, you only need to perform an item
correlation analysis on the rating scale part of your questionnaire, but it is possible to do
coefficient similar analyses on all kinds of items.
AC T I O N 01
Compile a data sheet. You should focus on the 12 items in the rating scale part
of the questionnaire. If you constructed the rating scale as suggested in previous
lessons, the possible responses to each item in the scale will have a number from
1 to 5 and people will have ticked a box corresponding to one of the numbers
for each item. For example, here is how a particular respondent completed
the first three items of a rating scale measuring “noise tolerance”.
Statement 1 2 3 4 5
Never Almost Some- Most Always
never times of the
time
I like loud music. X
I prefer quiet places. X
I enjoy noisy environments. X
The ticked options are known as item responses. This respondent’s item
responses suggest that he or she is somebody who is very tolerant of noise as
he/she always likes loud music, enjoys noisy environments most of the time
and never likes quiet places.
...........
44

In the resource on writing questionnaire items, two types of statements for

rating scales were mentioned, those where option 5 indicates a high level of
what you are measuring and those where option 5 indicates a low level of
what you are measuring. You should highlight this second type of statement
on each of the completed questionnaires in your sample before scoring the
item responses as this type of item must be “reverse-scored”. Reverse-scoring
means that for this type of item option 1 should be scored 5, option 2 should
be scored 4, option 3 should be scored 3, option 2 should be scored 4 and
option 5 should be scored 1. As you can see, item 2 in our example should
be “reverse-scored”, because if somebody says he/she never likes quiet places
he/she is, in effect, saying that he/she always likes noisy places and he/she
should therefore get a high score.
You can indicate the scores for each respondent next to their item responses.
We have done this for our example, taking into account the item that needs
to be “reverse-scored”. You will see that the scores correspond with our
description of this respondent having a high noise tolerance as he/she got the
maximum item score (5) for two of the items and a near-maximum score (4)
for another item.
Statement 1 2 3 4 5
Never Almost Some- Most Always
never times of the
time
I like loud music. X 5
I prefer quiet places
X 5
(Reverse score)
I enjoy noisy environments. X 4
It is difficult to work with item responses and scores spread out over 10 or 15
questionnaires (one for each respondent), so you need to collect all of them
together in one place. If you know how to use a computer, you could type
the scores into a spreadsheet program or a statistics package. However, if you
are not computer literate, you can use a data sheet like the following. At this
stage your data sheet should have 12 items and 15 cases. Unfortunately, we
cannot show a full 12 items and 15 cases here because the space is limited.
This example has five items and eight cases. Simply expand this table to
accommodate your 12 items by 15 cases scenario.
Items Total
Cases
1 2 3 4 5 score
1
2
3
4
5
6
7
8
...........
45 PYC 2616/1

The data sheet is divided into rows and columns – one row for each person
in your sample and one column for each item in your rating scale. There is
space on the data sheet for eight people and five items, but as your sample is
larger you have to add rows and columns to accommodate the additional items
and cases. You will notice that there is an extra column for a case number
(from 1 to 8) and total score. Case numbers are helpful for finding particular
questionnaires in case you make a mistake when writing down data, while
total score is the score a person got on the scale as a whole.
Now take a questionnaire from your pile of questionnaires. Write the number
“1” on the front of the questionnaire – this is the case number. Next, go to
the rating scale part of your questionnaire and transfer the score for each
item to the first row on the data sheet. (Remember to reverse-score where
necessary before transferring the scores.) When you have finished with the
first questionnaire the first row on the data sheet will be completely filled in
(except for the block under “total score”). Take the next questionnaire from
your pile, write the number “2” on it and fill in the second row on your data
sheet. Continue like this until you have transferred all the item scores from
the rating scales to the data sheet. Below is an example where the first three
item scores of the first case (person) who completed a “noise tolerance” scale
have been filled in. The person got item scores of 5 for item 1, 5 for item 2
and 4 for item 3.
Items Total
Cases
1 2 3 4 5 score
1 5 5 4
2
3
4
It is easy to make mistakes while transferring the item responses from the
questionnaires to the data sheets, so when you have finished doing it for
everybody in your sample, check that you have not made any mistakes. Take
two or three questionnaires from your pile and check carefully that you wrote
the item responses correctly on the data sheet. If you do not find any errors,
good! If you do, go back and check every questionnaire.
Finally, calculate each respondent’s total score on the scale and write it in the
“total score” column on the far right of the data sheet. You get the total score
by adding up the person’s item scores. Below is an example where the total
scores for the first two people in a sample have been calculated. The lowest
possible total score anybody can have on the rating scale is five (if they got
an item score of 1 for each of the five items), while the highest possible total
score is 25 (if they got an item score of 5 for each of the five items).
Items Total
Cases
1 2 3 4 5 score
1 5 5 4 3 4 21
2 3 2 2 1 3 11
3
4
...........
46

Choose two or three of your cases at random and check if you have calculated
their total scores correctly. If you find any errors, redo all the calculations.
AC T I O N 02
You are now ready to start with the actual item analysis of your rating scale
(see the resource on item analysis). The first step is to find items with too little
variance. “To vary” means to change; and items with too little variance are
those items where almost everybody in the sample gets the same item score
(i.e. almost everybody answers the item in the same way). (See the resource
on item analysis.) You want your scale to show differences between people,
for example if you have a noise tolerance scale, you want to know who has a
high noise tolerance and who does not. Items where everybody gets the same
score do not help you to see differences between people and are therefore of
no use in a scale. On the other hand, items where there is much variation (or
variance) in how people respond help you to see differences between people.
Of course most items are somewhere in between, for example one might find
the following situation with an item: nobody gets an item score of 5, a few
people get item scores of 1, 2 or 4 and most people in the sample get an item
score of 3. Would you say such an item shows enough variance?
Either “yes” or “no” could be correct as there are no absolute answers as to

when an item shows enough variance. What you can do, though, is to compare
items and decide which ones are better items in terms of the amount of variance
they show. Below are item responses from two items. Which one of the two
items shows more variance and is therefore the better item?
Items Total
Cases
1 2 3 4 5 score
1 5 1
2 3 3
3 3 2
4 3 4
5 4 1
6 3 3
7 3 4
8 3 2
By running your eye down the column of item scores under item 1 and then
the column of item scores under item 2, you should be able to see that item
2 has more variance and is therefore the better item. Most people responded
in the same way to item 1 (their item response was 3), whereas there is much
more variation in response to item 2. Item 2 may not be perfect, for example,
...........
47 PYC 2616/1

nobody had an item response of 5, but it is certainly a better item (in terms
of variance) than item 1.
There are statistical methods for showing which items in a scale show the
least variance and should therefore be discarded, but you are not expected
to calculate any statistics. Simply do the same for all the items in your scale
as you did for the two example items above – that is, run your eye down
each of the columns on your data sheet and look for items that may not have
sufficient variance. If a column contains mainly only one number (e.g. almost
all 4’s with very few other numbers), the item does not show much variance.
If a column contains a good spread of numbers (for example many 2’s, 3’s and
4’s and some 1’s and 5’s), the item shows lots of variance. If a column contains
mainly one number, but with a fair number of other numbers as well, the item
shows some variance.
In this way, you identify what looks like the worst items in your scale in that
they seem to have less variance than the other items. Now write down the
numbers of the three or four worst items in your questionnaire. Then, after
examining the way each item was worded, write why you think it may be that
the item has so little variance.
It is not always possible to explain why most people end up answering an

item in the same way (why the item has little variance), but there are a couple
of common reasons. One reason is that the item may have been too extremely
worded, for example an item such as “I become angry when I hear any sound
louder than a whisper” is likely to get the same response (NEVER) from just
about everybody. Other reasons why items may lack variance is because they
are too vague (so just about everybody goes for a “safe” response such as
SOMETIMES) or that there is a strong “socially desirable” way of responding
that most people opt for in order to look good. From looking at your items,
would you say that any of these reasons seem to apply to the items that you
decided were lacking in variance? Do not be discouraged if you cannot seem
to find a reason why a particular item has little variance. This is precisely why
you do item analysis – because people often surprise you and you cannot
always tell which items will draw sensible responses and which ones will not.
You now have a list of items that may be problematic due to a lack of variance.
Next, you will add items to the list that are problematic for a very different
reason. Then you will decide which ones to get rid of.
AC T I O N 03
To measure differences between people our items need to show some variance,
but even if the items show lots of variance, the scale may not measure anything
in particular. What if each item measures something completely different
(one item measures noise tolerance, another item measures attitude towards
mathematics, etc) so that it becomes impossible to know what the scale as a
whole measures?
You can do something to ensure that each item in the scale measures more
or less the same thing and that the items are not too divergent. This will
...........
48

guarantee that you can sensibly add each of the item responses together to get
a person’s score on the scale as a whole. How will you know if each of the
items broadly measures the same thing? One method is to check if people’s
responses on each item correspond (or “correlate”) with their responses to
the scale as a whole. The idea is that people who get a high score on an item
should also get a high score on the scale as a whole, while those who get a
low score on the item should get a low score on the scale as a whole. If an
item can “discriminate” in this way between high and low scorers on the scale
as a whole, it is a good item. One normally talks of discrimination as a bad
thing, but in this case, you want an item to discriminate between high and low
scorers because it shows that the item measures more or less the same thing
as the other items in the scale.
Suppose the following were the scores a sample of five people got for item 3
in an “optimism” scale, together with the total scores they got for the scale as
a whole. (Note that only five cases and five items are shown here.)
Items Total
Cases
1 2 3 4 5 score
1 4 53
2 1 12
3 5 49
4 4 40
5 2 16
Would you say that item 3 discriminates well between high and low scorers
on the scale as a whole? Why?
Item 3 does seem to be pretty good at discriminating between high and low
scorers. Respondents who get high item scores (4’s and 5’s) also get high total
scores (40 and over), while those who get low item scores (1 or 2) also get low
total scores (under 20). Looking whether item scores correspond with a total
score, is called item-total correlation and it is a technique often used in item
analysis. Professional questionnaire constructors usually calculate a correlation
coefficient (see the resource on correlation coefficient). A correlation coefficient
is an index of how strongly two variables are related. The idea is to establish
how strong each item-total correlation is. But here you are not expected to
do this. Instead, you can use a simple way of getting an impression of how
strongly each item correlates with the total.
Simply make a small scatterplot for each item showing the relationship between
each item and the scale total. Scatterplots are explained in the resource on the
correlation coefficient. Here is a scatterplot showing the relationship between
item 3 and the scale total for the optimism scale discussed above.
...........
49 PYC 2616/1

Each dot on the scatterplot represents a person. The dot is positioned (roughly)
next to the correct total scale score on the left and above the correct item score
at the bottom. As you can see, there were five people in the sample (five dots)
and the dots are arranged roughly in a diagonal line from bottom left to top
right. This means that there is a strong correlation between the item score and
the total score – that is the item discriminates well. It is unlikely that you will
find correlations nearly this strong, but if the dots for your sample are arranged
in a vaguely similar pattern, then you will know that there is at least some
correlation (the item discriminates). If the dots do not seem to form a pattern
at all, then there is no correlation (the item does not discriminate), and if the
line seems to go from top left to bottom right, there is a negative correlation
(the item does discriminate, but the wrong way round, so it is no good).
If your questionnaire has 12 items, you will have to draw 12 scatterplots (one for
each of the 12 items in your scale), and if you administered your questionnaire
to 15 individuals you will have 15 dots on each scatterplot (one for each person
in your sample). You will also have 12 total scores (one for each person in your
sample). Do not worry about being too precise, so long as your scatterplots
show roughly how well each item correlates with the scale total.
As was the case for item variance, there are also no absolute rules with regard
to item discrimination, but we advise you not to be too strict. In the social
sciences, correlations are often not quite as strong as one would like them
to be, and if you insist that each item should correlate very strongly with the
scale as a whole, you may end up having no items left at all! Considering all
your scatterplots (12, if you have 12 items), identify what appears to be the
worst items in your scale in terms of failure to discriminate. Now write down
the numbers of the three or four worst items. Then, after looking again at the
way each item was worded in your questionnaire, write why you think it may
be that the item discriminates so badly.
The reason why items do not discriminate is usually because they measure
something different from the other items in the scale. Sometimes one can see
why this is the case when one looks again at the wording of the item, but
sometimes it seems inexplicable and one just has to accept that it is so.
AC T I O N 0 4
...........
50

You are now ready to compile a final version of your questionnaire. You have
to make some tough choices regarding which items to keep and which to
discard from your scale. Your scale should have eight items, so if you started
with 12 items you can afford to discard four items. Study the two lists of bad
items you compiled – one list for items that do not show much variance and
the other list for items that do not discriminate well. You may find that some
items appear on both lists. Decide which four items should be discarded and
write down their numbers, together with a short justification as to why you
chose to get rid of each of these items.
There is no fixed rule about identifying your worst items. If you find that you
really cannot decide between two items (maybe one that shows little variance,
but does discriminate somewhat versus one that shows a lot of variance but
hardly discriminates at all), simply flip a coin. If you do not seem to have
enough really bad items to discard, find some that are a little less good than
the others. Getting rid of the worst items will not give you a perfect scale,
but it will almost certainly be a better scale than if you had not performed an
item analysis and just kept all the items. Most importantly, your final scale is
more coherent (the items all measure broadly similar things) than the original
scale – which means (as you will see in the next lesson) that it has a greater
degree of reliability.
Finally, taking into account the notes you made during the pilot-testing of the
questionnaire, revise the instructions to the questionnaire, change whatever
items you think need changing and remove the worst items from the rating
scale part of the questionnaire. You have now pilot-tested your questionnaire
and have used the pilot-test to construct a final version of the questionnaire.
QUESTION 1
Consider the responses below of four participants to a questionnaire consisting
of five items:
Cases Items Total

score
1 2 3 4 5
1 5 5 4 3 4 21
2 3 2 2 1 3 11
3 3 3 2 3 4 15
4 5 5 1 1 4 16
...........
51 PYC 2616/1

Select the correct statement(s):
A. Item 3 shows greater variance than item 4.

B. Item 5 is the weakest discriminator between high and low scores.
C. Items 2 and 3 have similar variances.
D. Item 1 is the best discriminator between high and low scores.
(1) A and C
(2) B and D
(3) A, B and C
(4) A,B, C and D
QUESTION 2
Based on the table above, select the correct statement(s):
A. Item 3 is a good item.

B. Item 5 is the weakest item.
(1) A
(2) B
(3) A and B
QUESTION 3
Which of the statements below do you agree with:
A. he following item should be reverse-scored if used in a questionnaire

T
that measures emotional tolerance: “I am an impatient person”.
B. The reverse-score for a response of 2 on a five-point scale (ranging from
1 to 5) is 1.
C. The scatterplot of a positive correlation between two variables is upwardly
skewed from left to right.
(1) A and B
(2) B and C
(3) A and C
(4) A, B and C
...........
52

QUESTION 4
Please study the graph below and answer the following questions.
Please choose the correct answer.
A. There is a positive correlation between the total score and item 2.

B. There is a negative correlation between the total score and item 2.
C. Item 2 discriminates well with the total score.
D. Item 2 discriminates well between high and low scores.
(1) A and B
(2) A, B and C
(3) A, C and D
(4) B and C
(5) B, C and D
Q U E S T I O N 1: O P T I O N 3 I S T H E K E Y – A , B A N D C
Items 2 and 3 have comparable spreads of numbers, which is larger than any
of the other items. They are therefore also better discriminators between high
and low scores than any of the other items. Item 5 shows the smallest variance
and is therefore the most ineffective item of the group.
Q U E S T I O N 2: O P T I O N 3 I S T H E K E Y – A A N D B
Item 3 is a good discriminator since it shows a good correspondence between
a high item score and a high score for the scale as a whole. In addition, low
item scores correspond well with low total scores.
Item 5 is the weakest item since it shows the least correspondence between
the item and total scores.
...........
53 PYC 2616/1

Q U E S T I O N 3: O P T I O N 3 I S T H E K E Y – A A N D C
Statement A is correct. The item score should be reverse-scored since agreeing
to be an impatient person reflects emotional intolerance, which should obtain
a low score on a scale that measures emotional tolerance.
Statement B is incorrect. The reverse-score for a response of 2 should be 4.

The score of 4 is the same distance as the score of 2 from 3, which serves as
the mid-point of the scoring range.
Statement C is correct. A positive correlation exists between two variables

when low scores on one variable correspond with low scores on the other
variable, and high scores correspond between the variables.
Q U E S T I O N 4: O P T I O N 3 I S T H E K E Y – A , C A N D D
The scatterplot shows that item 2 has a positive correlation, which means that
the item discriminates well with the total score and between high and low
scores. Therefore, any option with statement B is wrong as the scatterplot does
not show a negative correlation.
...........
54

Lesson
5 5
5 Evaluate reliability and validity
In this lesson it is explained how to evaluate the reliability and validity of a

questionnaire.
AIMS
In lesson 5, you will learn why a questionnaire should be reliable and valid
and how to interpret information to determine if a questionnaire meets these
requirements. You will be given activities with examples to teach you how to
apply your knowledge.
Lesson
• understand three types of reliability:
(1) split-half reliability

(2) test-retest reliability
(3) alternate form reliability
• understand three categories of validity:
(1) content validity

(2) criterion-related validity
(3) construct validity
• interpret information on the reliability and validity of a questionnaire
• correlation coefficient
• reliability
• validity
...........
55 PYC 2616/1

Reliability is not an absolute characteristic of a questionnaire. The questionnaire

is reliable for a specific population and context. This will also have an impact
on the validity of the same questionnaire, where the questionnaire is expected
to measure what it is designed to measure. If it is not designed to be reliable
for multiple population groups, it might very well also not measure what it
was intended to measure for differing groups of test takers. To ensure that the
same concepts are measured for different groups, the level of equivalence be-
tween groups needs to be determined. Bias implies that a questionnaire has a
tendency against (or for) a particular group while equivalence refers to a lack
of group-related bias.
Suppose that a questionnaire about attitudes to cultural practices was originally

designed for a group of Xhosa-speaking young adults. The reliability and va-
lidity might not be the same if the questionnaire was administered to a group
of Afrikaans-speaking young adults because it is no longer being used for the
population for whom the original reliability and validity were established.
This has serious implications when we try to generalise findings from a study
where the reliability and validity are not as accurate as it was for the original
population. It might lead to inappropriate assumptions about different cultural
groups. This happened during the pre-94 era in South Africa, where intelligence
and aptitude tests that were normed for European populations, were used incor-
rectly on black (and other) South African groups.
These are examples of where a test or questionnaire is used inappropriately,

either accidentally or on purpose, to suggest that certain facts about certain
groups are true.
5.1 INTRODUCTION
A questionnaire is designed for a specific purpose implying that the researcher
wants to obtain information on a specific concept or construct. But before the
researcher can use a questionnaire, he or she needs to evaluate the functioning
of this questionnaire in terms of two requirements. In the first place, the
following questions need to be answered:
• Will the result be consistent if the questionnaire is administered on different

occasions?
• Will similar items provide the same result?
• Do all the items in the questionnaire consistently measure the same concept
or construct?
The consistency with which the questionnaire measures a concept or

construct is called the reliability of the questionnaire. Suppose you designed
a questionnaire to do survey research on migration from rural to urban areas.
If your results are similar every time you administer the questionnaire (i.e. the
questionnaire is reliable), can you assume that you are measuring migration
...........
56

patterns? No, consistent measurement is necessary but not sufficient to ensure

that a questionnaire meets its intended purpose. That brings us to the second
set of questions:
• Do the questions adequately represent the tasks, behaviours, attitudes,

etcetera implied by the purpose?
• Do the results on the questionnaire say something about related behaviours
(now or in the future)?
• Does the questionnaire measure the intended concept or construct?
The extent to which the questionnaire measures what it claims to measure is

called the validity of the questionnaire. If a questionnaire is valid, it implies
that the scores and the conclusions based on these scores can be used for the
intended purpose of the questionnaire. Therefore, our example questionnaire
should consistently measure migration patterns to be both reliable and valid.
5.2 WHICH TYPE OF RELIABILITY IS APPROPRIATE?
5.2.1 Test-retest reliability

Will the result be consistent if the questionnaire is administered on different
occasions? If the purpose of a questionnaire requires consistency over time and
if the results of the questionnaire should be consistent over different occasions,
you would determine the test-retest reliability of the questionnaire. The method
for determining test-retest reliability is to administer the same questionnaire to
the same group of people on two consecutive occasions and to correlate the
two sets of scores that are obtained.
5.2.2 Alternate-forms reliability

Will similar items provide the same results? If the purpose of the questionnaire
requires that scores should not depend on which particular form is used and
if the results of the questionnaire should be consistent over different forms,
you would determine the alternate-forms reliability of the questionnaire.
To determine alternate-forms reliability both forms of the questionnaire are
administered to the same group of people on two consecutive occasions and
the two sets of scores that are obtained are correlated.
If alternate-forms reliability is established for a questionnaire testing memory,

the questionnaire will be given to the same test takers, but in a different format.
If words are used to test memory, and the questionnaire is given again but
in a different format, different words of similar difficulty should be used. For
example, table, apple and computer could be replaced with chair, bread and
telephone but NOT with hyperbole, paradigm and existential. The first two
sets of words are similar and will therefore be more useful in establishing
alternate-forms reliability. The last set of words is considerably more difficult,
and therefore will not be useful in attempting to establish alternate-forms
reliability.
...........
57 PYC 2616/1

5.2.3 Split-half reliability

Do all the items in the questionnaire consistently measure the same concept
or construct? This refers to the internal consistency of a questionnaire and it
can be determined by obtaining an estimate of the split-half reliability of the
questionnaire. This will indicate the degree of equivalence between two halves
of the questionnaire, that is, the extent to which the two halves measure the
same attribute. The questionnaire is administered only once to a group of
people, it is then divided into two parts and the two sets of scores that are
obtained are correlated. A correction is made to obtain the reliability for the
whole questionnaire.
5.2.4 Interpret information on the reliability

The reliability coefficient is in many instances a correlation coefficient between
two sets of data and it is therefore interpreted in the same way as a correlation
coefficient. A value close to zero indicates a weak relationship and thus a
low reliability. A value close to one indicates a strong relationship and thus a
high reliability. We saw that the nature and the purpose of the questionnaire
determine the type of reliability that will be calculated. The purpose of the
questionnaire also determines the level of reliability that could be regarded
as acceptable.
In summary
Type of reliability Forms of the Coefficient
questionnaire
Administration
sessions
Test-retest reliability One form Coefficient of stability
Two sessions
Alternate-forms Two forms Coefficient of
reliability One session equivalence
(immediate)
Alternate-forms Two forms Coefficient of stability
reliability Two sessions Coefficient of
(delayed) equivalence
Split-half reliability One form Coefficient of internal
One session consistency
5.3 WHICH CATEGORY OF VALIDITY IS APPROPRIATE?
5.3.1 Content validity

Do the questions adequately represent the tasks, behaviours, attitudes, etcetera
implied by the purpose? The content validity of your questionnaire is influenced
by how well you designed the questionnaire. The purpose of the questionnaire
determines the content domain and you include items that are representative
of this content domain. The relevant universe needs to be defined clearly and
experts in the field decide if the chosen items adequately cover the universe of
...........
58

tasks, behaviours or attitudes as defined by the developer of the questionnaire

and whether the items indeed measure these tasks, behaviours or attitudes –
that is, if the questionnaire has content validity.
5.3.2 Criterion-related validity

Do the results on the questionnaire say something about related behaviours
(now or in the future)? Depending on the purpose of a questionnaire, further
evidence of validity might be necessary. If you want to use a questionnaire
to estimate an individual’s present position with regard to something else or
to predict some future behaviour, you would determine the criterion-related
validity of the questionnaire. This refers to how well a questionnaire estimates
an individual’s position or performance on some outcome measure (or criterion).
Your questionnaire is administered to a representative group of people and
scores for the same group are obtained on the criterion (a measure of the group’s
present or future position on a relevant variable). A correlation is calculated
between the two sets of scores.
Insurance companies, for example, need to know that the questionnaires they
use have criterion-related validity. An insurance company might construct a
questionnaire to establish “overall health” in their client population. Does their
current finding of a high or low score in “overall health” say anything about
how much they will claim from the company, or, what their health will look
like a year from now?
5.3.3 Construct validity

Does the questionnaire measure the intended concept or construct? If you
want to use a questionnaire to make conclusions about a theoretical construct
that underlies the behaviours measured by the questionnaire, the construct
validity of the questionnaire would be important. For example, if you are
testing cultural stereotypes, you need to ensure that the questionnaire actually
measures cultural stereotypes as defined in the literature. As cultural stereotypes
are an abstract concept, it needs to be clearly defined and tested to see if it has
construct validity. You could compare groups who are supposed to differ in
terms of the construct. Another method is to correlate the results with results
obtained from questionnaires that measure the same or different constructs.
5.3.4 Interpret information on the validity

Content validity cannot be expressed in terms of a quantitative index. The items
of the questionnaire are evaluated to determine if these adequately cover (i.e.
are representative of) and indeed measure the tasks, behaviours and attitudes
implied by the purpose of the questionnaire. In the case of both concurrent and
predictive validity, the validity coefficient is a correlation coefficient. As you
know by now a value close to 0 indicates a weak relationship and thus a low
validity. A value close to +1 or –1 indicates a strong relationship and thus a
high validity. Note that in the case of validity coefficients, values in the middle
range could be statistically significant. It is furthermore important to note that
a validity coefficient can have a negative value if an inverse relationship is
expected between your questionnaire and the criterion. Lastly, we explained
...........
59 PYC 2616/1

that a comparison of your questionnaire with existing questionnaires is a method

to determine construct validity. This “comparison” also involves a correlation
between two sets of scores.
In summary
Category of validity Example Relevant content/
criterion/comparison
Content validity Questionnaire on Personal experiences of
experience of crime- crime
related stress Reaction to crime
Stress levels
Criterion-related Questionnaire on Managers’ rating
validity organisational commitment of commitment.
Questionnaire on a study Examination
field performance
Construct validity Questionnaire on Community
sociability involvement (high
correlation)
Mechanical interest (low
correlation)
5.4 GLOSSARY
Here are some basic terms related to the topics of reliability and validity that

Alternate-forms The correlation Die korrelasie tussen Kamano e pakeng
reliability between two twee ekwivalente tsa diforomo tse pedi
Ekwivalentevorms- equivalent forms vorms van dieselfde tse tshwanang tsa
betroubaarheid of the same vraelys. lenanepotso le le
questionnaire. leng
Ho ba le sephetho
se tshwanang
diforomong tse
fapaneng
Concurrent validity The accuracy Die akkuraatheid Bokgoni ba
Samevallende with which a waarmee ’n vraelys lenanepotso ba ho
geldigheid questionnaire can huidige gedrag kan bontsha mosebetsi
identify current identifiseer. o etswang jwale, ka
Papiso ya diphetho behaviour. nepo.
tse fumanwang
makgetlong a
fapaneng
...........
60

Consistency The constancy Die konstantheid Bokgoni ba

Konsekwentheid with which a waarmee ’n vraelys lenanepotso ba ho
Ho hlahisa sephetho questionnaire meet. lekanyetsa ka ho
se tshwanang measures. tshwana.
Construct validity The extent to which Die mate waartoe die Bokgoni ba
Konstrukgeldigheid the questionnaire vraelys die konstruk lenanepotso ba
Kgonahalo ya measures the meet wat dit ho lekanyetsa
tekanyetso construct it is veronderstel is om te mohopolo o lokelang
supposed to meet. ho lekanyetswa.
measure.
Content validity Whether the Of die items in Hore na dintlha
Inhoudsgeldigheid items in the die vraelys ’n tse lenanepotsong
Bokgoni ba ho questionnaire covers verteenwoordigende di akaretsa karolo
keneyeletsa dintlha a representative steekproef van die ya lekala leo ho
kaofela sample of the inhoudsgebied dek. fuputswang ka lona.
content domain.
Correlation The relation between Die verhouding Kamano e pakeng tsa
Korrelasie two variables. tussen twee dinomoro tse pedi
Kamano veranderlikes. tse ka fetohang.
Equivalence Having equal values, Om dieselfde Ho lekana ka
Ekwivalensie functions, etcetera. waardes, funksies, palo, mosebetsi,
Tekano ensovoorts te hê. jwalojwalo.
Predictive validity The accuracy Die akkuraatheid Tsela eo lenanepotso
Voorspellings- with which a waarmee ’n vraelys le ka nahanelang
geldigheid questionnaire can toekomstige gedrag pele mesebetsi ya
Kgonahalo ya ho predict future kan voorspel. nako e tlang.
nahanela pele ho behaviour.
ya ka matshwao a
fumanwang
Split-half reliability The correlation Die korrelasie tussen Kamano e pakeng
Halveerbetrou- between two twee ekwivalente tsa dihalofo tse pedi
baarheid equivalent halves helftes van dieselfde tse lekanang tsa
Sephetho se of the same vraelys. lenanepotso le le
tshwanang se questionnaire. leng.
arolwang habedi
Stability Remain unchanged Bly onveranderd oor Ho dula ho se na
Stabiliteit over time. tyd. phetoho ka mora
Botsitso nako e itseng.
Test-retest reliability The correlation Die korrelasie tussen Kamano pakeng
Toets- between two twee toepassings (op tsa ditsamaiso tse
hertoetsbetrou- administrations (at verskillende tye) van pedi (ka dinako
baarheid different times) of the dieslfde vraelys. tse fapaneng) tsa
Teko e phetwang same questionnaire. lenanepotso le le
ya sephetho se leng.
tshwanang
...........
61 PYC 2616/1

5 AC TIVITIES
material for this lesson (correlation coefficient, reliability and validity).
Correlation Reliability Validity

coefficient
Press the button or go to page 136 in or go to page 143 in
or go to page 114 in the study guide. the study guide.
the study guide.
T ime re cord
METHOD Once you have compiled the final version of your questionnaire (lesson 4),
there are two more requirements that your questionnaire should meet before
Activity 5.1
you can use it. The first is that the results should be reliable, implying that
Evaluate the
reliability of the the questionnaire should measure consistently. The type of reliability and the
rating scale approximate value of the reliability coefficient need to be considered when
Resource
you evaluate the reliability of your questionnaire.
Correlation
coefficient
Reliability
AC T I O N 01
You should be able to distinguish between different types of reliability. The
purpose of the questionnaire determines which type of reliability is appropriate.
Part of your questionnaire consists of a rating scale aimed at measuring a
specific concept or construct, that is, a specific attribute relevant to the topic
you are investigating. In lesson 4, you analysed the items in this rating scale
and selected those items that are the best measure of this construct for the
final version of your questionnaire. In other words, you want the items in the
rating scale to all measure the same concept or construct. But how do you
know if the items of your rating scale all measure the same thing?
To answer this question, you will check if the rating scale is internally consistent.
Study the resource on reliability and discuss the type of reliability you would
use if you want to determine the internal consistency of a rating scale (the
extent to which the items measure the same thing). Describe the method you
would use and indicate if there are any advantages or limitations to this method.
Because you are interested in the internal consistency of the rating scale in
your questionnaire, you need to obtain an estimate of the split-half reliability
of this rating scale. This will indicate the degree of equivalence between two
halves of the rating scale, that is, the extent to which the two halves measure
the same attribute. As such, it indicates the degree of relatedness of the items
in your rating scale. A limitation of this method is that the reliability coefficient
...........
62

that one obtains to some extent depends on the items included in each of the
two halves. The way in which you divide the rating scale into two equivalent
halves thus influences the value of the reliability coefficient.
AC T I O N 02
You are not expected to calculate a split-half reliability coefficient for the rating
scale in your questionnaire. We will, however, show you how to obtain some
indication of the reliability of either of two halves of the rating scale. This will
enable you to comment on the reliability of this scale.
Below is an example of the responses to the eight items of a five-point rating
scale (i.e. the respondents marked a response from 1 to 5 for each item). Only
five cases are shown here to illustrate the method.
Cases Odd items Even items

1 3 5 7 Total 2 4 6 8 Total
score score
1 3 4 4 3 14 4 4 3 2 13
2 2 4 2 3 11 4 3 2 3 12
3 4 3 2 1 10 3 3 1 1 8
4 2 2 3 1 8 2 2 1 2 7
5 2 4 4 3 13 3 3 2 4 12
We now have two sets of five scores each (the number of respondents): a set
of total scores on the odd items and a set of total scores on the even items.
The relation between these two sets of scores will give you an estimate of the
reliability of either of the two halves of the rating scale.The relation between
the two sets of scores can be visualised as follows. For each person, we take
the total score on the odd items and the total score on the even items and
where the two meet, we make a dot on a graph (see below). If we do this for
all the people in the group, we will have a scatterplot (a number of scattered
dots). We then draw a straight line resembling the shape of the scatterplot.
...........
63 PYC 2616/1

The upward slope of the line tells you that the correlation coefficient falls
between 0 and +1. Because most dots are close to the line, the correlation
coefficient is close to +1 and there is a fairly strong relation between the two
halves.
You can now repeat this process with your own questionnaire. In lesson 4,
you did an item analysis and selected eight items for the final version of your
questionnaire. Look at these eight items and divide the rating scale into two
halves by grouping the odd items (items 1, 3, 5, etc) together and by grouping the
even items (items 2, 4, 6, etc) together. On your original data sheet (consisting
of 12 items) in lesson 4, draw a line through each of the four items you are
excluding from the final version. Now re-number your items from 1 to 8 before
copying the information to the data sheet below. For each person calculate
the total score for the odd items (add up the scores on the items) and the total
score for the even items.
Odd items Even items

Cases Total Total
1 3 5 7 2 4 6 8
score score
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
You now have two sets of 15 scores each (the number of respondents): a set
of total scores on the odd items and a set of total scores on the even items.
For each person, take the total score on the odd items and the total score on
the even items and where the two meet, you make a dot on the graph below.
Do this for all the people in the group to get a scatterplot and then draw a
straight line resembling the shape of the scatterplot.
...........
64

If your scatterplot has a really undefined shape, the correlation coefficient

is close to 0 indicating a weak relation between the two halves. However, if
the line has an upward slope, you know that the correlation coefficient falls
between 0 and +1. If most dots are close to the line, the correlation coefficient
is close to +1 and there is a fairly strong relation between the two halves.
Estimate the approximate value of the correlation coefficient between the two
halves, and write it down.
People who frequently use scatterplots can become quite good at estimating
how strong a correlation is represented by a particular scatterplot. Your estimate
is unlikely to be very accurate, but since we do not actually require you to
calculate a correlation coefficient an estimate will have to do.
AC T I O N 03
In this context, the correlation coefficient is a reliability coefficient. Consider
the type of reliability and the approximate value of the reliability coefficient
and evaluate the reliability of the rating scale. Write this down.
You have studied the relation between two halves of the rating scale and
therefore have an indication of the reliability of either of the two halves. This
indicates the extent to which the halves are equivalent and thus measure the
same attribute. Based on your scatterplot and the straight line you drew, you
“guessed” the value of your reliability coefficient. The values can range from 0
to 1 with values closer to 1 indicating a more reliable rating scale. For example,
if the reliability coefficient for the rating scale on levels of stress associated
with different crimes (the example used in lessons 1 and 2) was 0.85, this
would imply that the items in this scale to a large extent measure the same
attribute, namely stress.
Note that you only have the reliability of one half of the rating scale but the
split-half reliability of the whole rating scale would probably be even higher
as a shorter questionnaire is generally less reliable than a longer questionnaire.
...........
65 PYC 2616/1

Activity 5.2 The interpretations based on the results of your questionnaire should not only
Evaluate the validity be reliable but should also be valid, that is, the questionnaire should measure
of the questionnaire
what it claims to measure. A questionnaire is valid for a specific purpose and
Resource you should gather evidence relevant to the purpose of your questionnaire to
Correlation determine its validity.
coefficient
Validity
AC T I O N 01
You should be able to distinguish between categories of validity. In lesson 1,
you stated the purpose of your questionnaire and based on this you identified
the content domain of your questionnaire. In lesson 2, you designed your
questionnaire so that the items are representative of the specified content
domain. The degree to which you succeeded in this, influences the validity
of your questionnaire. Study the resource on validity and discuss the category
of validity that is relevant to questionnaire design. Write this down so that you
can formulate your ideas clearly.
The content validity of your questionnaire is influenced by how well you

designed the questionnaire. The content domain is the universe of tasks,
behaviours, attitudes, etcetera implied by the purpose of your questionnaire.
You only included a relatively small number of these tasks, behaviours or
attitudes as items in your questionnaire. The degree to which the items in
your questionnaire are representative of the universe of tasks, behaviours and
attitudes determines the content validity of your questionnaire.
AC T I O N 02
Consider the content domain of your questionnaire (lesson 1) and the
questionnaire specification document (lesson 2) and evaluate the content
validity of your questionnaire. Write down your thoughts on this before you
read the next paragraph!
It should be clear from your definition of the content domain (that is the relevant
universe of tasks, behaviours, attitudes) whether any particular task, behaviour
or attitude forms part of this universe. The content of the items and the item
format should indeed measure the tasks, behaviours or attitudes you defined.
The number of items should furthermore ensure that the universe of all tasks,
behaviours or attitudes is adequately represented. In lesson 2, an example of
a questionnaire for an investigation of South Africans’ experience of crime-
related stress was discussed. The content areas, type of items, and number of
items were in accordance with the purpose and the content domain of this
questionnaire as identified in lesson 1.
The content validity of a questionnaire cannot be expressed in terms of a

quantitative index and it is usually evaluated by experts in the field. In addition
to your own evaluation, you can ask someone who is reasonably informed
with regard to your topic to evaluate the content validity of your questionnaire.
...........
66

The leisure activities questionnaire (LAQ) can be used for adults to obtain
information on what people do during their free time (that is, when they are
not busy with compulsory activities). The questionnaire was administered to
a representative sample of adults during the pilot-testing to determine the
reliability and validity of the questionnaire. The respondents were tested on
two occasions with two weeks between the testing occasions. The scores of 5
of the respondents for the first and second applications of the LAQ are given
in table 1.
Table 1: Five respondents’ test scores for two administrations of the LAQ
Respondent 1 2 3 4 5
First application 50 43 60 49 56
Second application 48 32 38 55 60
The respondents also completed two additional questionnaires, the sport
participation questionnaire (SPQ) and the numerical skills questionnaire
(NSQ). The results of these measurements are shown in table 2. It contains
the correlations among the questionnaires.
Table 2: Correlations between the LAQ and other questionnaires
LAQ SPQ NSQ
LAQ 1.0
SPQ 0.84 1.0
NSQ 0.12 0.18 1.0
QUESTION 1
Lerato: What does the information in table 1 tells us about the LAQ?
Susan: The information in the table shows a poor correlation between

the two sets of scores, which means the LAQ did not demonstrate
test-retest reliability.
Vusi: We should keep in mind that a poor correlation refers to criterion-

related validity and not to the reliability of the LAQ.
Ben: Actually, the information in table 1 does not reflect a poor correlation,
which means that the LAQ shows good criterion-related validity.
...........
67 PYC 2616/1

You agree most with ..., second most with and least with
(1) Susan, Vusi and Ben.

(2) Susan, Ben and Vusi.
(3) Vusi, Susan and Ben.
(4) Vusi, Ben and Susan.
(5) Ben, Susan and Vusi.
(6) Ben, Vusi and Susan.
QUESTION 2
A year programme has been introduced in a school to make the children
more aware of environmental conservation. The children will complete a
questionnaire to measure their attitude to and knowledge on issues related
to environmental awareness. This questionnaire is administered in January
and again in July and in December. Which type of reliability do you think
is appropriate and how would you interpret a value of 0.60 on this type of
reliability?
QUESTION 3
In table 2, what does the correlation between the LAQ and SPQ mean? Choose
the most correct answer.
(1) It means there is a poor relationship between the LAQ and SPQ.
(2) It means the results on the two questionnaires are related.
(3) It shows that the LAQ has concurrent validity.
(4) It shows that the LAQ has discriminant validity.
QUESTION 4
In table 2, what does the correlation between the LAQ and NSQ mean? Choose
the most correct answer.
(1) It means there is a poor relationship between the LAQ and NSQ.
(2) It means the results on the two questionnaires are related.
(3) It shows that the LAQ has concurrent validity.
(4) It shows that the LAQ has discriminant validity.
QUESTION 5
Define the content domain for the following two questionnaires: a questionnaire
to measure spelling achievement in Grade 2 and a questionnaire to measure
anxiety.
...........
68

Q U E S T I O N 1: O P T I O N 1 I S T H E K E Y – S U S A N , V U S I A N D B E N
Order the values on the first administration in ascending order:
Person 2 4 1 5 3
1st 43 49 50 56 60
Now write down the corresponding values for the second administration:
Person 2 4 1 5 3
1st 43 49 50 56 60
2nd 32 55 48 60 38
Can you see that there is little correspondence between the pattern of values
on the first administration (ascending) and that of the second administration
(a mix of high and low values with no clear pattern)? This implies that there
is a poor correlation between the two sets of scores. A correlation between
two administrations of the same questionnaire at different times refers to the
test-retest reliability of a questionnaire. Susan was therefore correct on both
issues. Vusi was correct in identifying that the correlation is poor but this does
not refer to criterion-related validity. Ben was incorrect on both issues.
Q U E S T I O N 2:
You would have been correct if you indicated that the test-retest reliability
should be examined. This questionnaire is administered to the children more
than once and should thus demonstrate consistency over time. A value of 0.60
indicates that differences in results can be expected over different occasions
(remember the values of the reliability coefficient range from 0 to 1). The value
is nevertheless not so low that it will not provide any information but we would
suggest that the school uses the results in combination with other information.
We would expect the results on the leisure questionnaire and the sports
questionnaire to be related. A value of 0.84 indicates a good correlation
(i.e. a relationship) between the LAQ and the SPQ meaning that option 2 is
correct. One of the methods to determine construct validity is to consider the
concurrent validity (the relationship between your questionnaire and another
questionnaire that measures a similar construct) and discriminant validity (a
low correlation between your questionnaire and another questionnaire that
measures a different construct). The correlation between the LAQ and the SPQ
therefore shows concurrent validity making option 3 the most correct answer.
We would not expect a relationship between the results on the leisure
questionnaire and the numerical skills questionnaire. A value of 0.12 indicates
...........
69 PYC 2616/1

a low correlation (i.e. the absence of a relationship) between the LAQ and
the NSQ meaning that option 1 is correct. The correlation between the LAQ
and the SPQ therefore shows discriminant validity, making option 4 the most
correct answer.
Q U E S T I O N 5:
You will find it easier to define the universe of material and tasks for measuring
achievement where a great deal is known about that which you want to
measure. You can compile a list of nearly all possible words that a Grade 2
child should be able to spell. You will then select words ranging in difficulty
level from this list. You should also decide on the tasks, for example, the items
could be multiple-choice items. Content validity is more difficult to assure
when a questionnaire measures a more abstract attribute. It is not possible,
for example, to provide all possible items for a measure of anxiety. You would
have to rely on the opinion of experts as to whether the items represent the
content domain.
...........
70

Lesson
6 6
6 Compile a manual
In this lesson it is explained how a manual for a questionnaire should be

structured and what content should be included.
AIMS
In lesson 6, you will learn why it is important to have a manual for a questionnaire,
how to logically structure your manual, what information about the question-
naire should be provided and in how much detail. You will be given activities
with examples to teach you how to apply your knowledge.
Lesson
• understand why a manual is needed for a questionnaire

• describe the aim and design of a questionnaire
• describe the properties of the questionnaire
• provide procedures for the administration, scoring and interpretation of the
questionnaire
• manual: purpose and structure
Relevant information on the use of the questionnaire in a multicultural and

multilingual context should be discussed in the manual.
We have a social responsibility to research topics relevant to our communities. This

will at times include the concerns of marginalised communities. When discussing
...........
71 PYC 2616/1

the purpose of the questionnaire, the possible importance for the community
we are engaging with should be mentioned.
The description of the target population should refer to the cultural and lan-
guage group/s for which the questionnaire is suitable. In a community project,
one would involve the various stakeholders to ensure that the community (the
target population) benefits as much from the co-designed questionnaire as the
researcher does.
The manual should include information on the equivalence of the question-

naire in terms of reliability and validity for the different cultural and/or language
groups specified in the target population. But even if the properties are similar
for different groups, the interpretation of the results should provide for different
epistemological standpoints including a community understanding.
6.1 INTRODUCTION
A researcher identifies the topic for his or her study based on a number of
considerations including perceived community concerns. The researcher
might decide to use an existing questionnaire in the study but he or she needs
information on this questionnaire to decide if it is suitable for the study. We
will call the researcher the user of the questionnaire and the purpose of the
manual is to provide the user with information on the following:
The aim and design of the questionnaire
• What is the aim or purpose of the questionnaire?

• Who is the target population?
• Does the design of the questionnaire ensure that it meets its aim?
The properties of the questionnaire
• How well do the items function?

• Is the questionnaire reliable and valid?
Procedures for administration, scoring and interpretation
• What are the instructions for the administration of the questionnaire?

• How does one score and interpret the results?
Can you see that the manual is a summary of all the work that you did in
lessons 1 to 5?
6.2 THE AIM AND DESIGN OF THE QUESTIONNAIRE
6.2.1 What is the aim or purpose of the questionnaire?

Remember that the intention of the user of the questionnaire and the aim of the
questionnaire are not exactly the same. A questionnaire has a clearly stated aim
and it is designed in a manner suitable to this aim. Once a questionnaire has
...........
72

been developed, it could be used in a number of different studies (i.e., by various

users each with his or her own intention). For example, the Beck depression
inventory (BDI) was originally designed to identify and diagnose individual
mental illness in clinical practice, such as depression or anxiety. However,
measures such as these have frequently become available to researchers,
who use these measures to also do survey research around the number of
depressed or anxious individuals in a particular group such as a school or a
corporate environment.
6.2.2 Who is the target population?

You now need to indicate for whom the questionnaire could be used. The
characteristics of the target population that are important are related to the aim
of the study. These could include (but are not limited to) gender, age, cultural
background, home language, educational background, socio-economic status
and geographic location. Relevant information on the use of the questionnaire
in a multicultural and multilingual context should be given if this applies. If a
general health questionnaire is, for example, designed for a population that
is able to read to a Grade 10 level, this should be indicated. It should also be
indicated that if such a questionnaire is used for populations with an education
below this level, it might represent the population in a less than accurate
manner. In other words, although the content was related to general health,
the below Grade 10 participant might not have understood all the questions
and therefore responded inappropriately.
6.2.3 Does the design of the questionnaire ensure that it meets its aim?
A brief description of the design of the questionnaire is provided in the manual.
This includes the domain of the questionnaire and an indication of how the
items cover this domain. The item types are specified and it is indicated how
a specific item format is suitable to obtain specific information. The layout of
the questionnaire is also mentioned.
6.3 THE PROPERTIES OF THE QUESTIONNAIRE
6.3.1 How well do the items function?

The procedures followed during the pilot-testing of the questionnaire include the
selection of a representative sample, a comparison of the characteristics of this
sample to those of the target population (see above) and the administration of
the questionnaire. Their results are used to analyse the items and to determine
the spread of responses to each item and the relation between each item and
the total over all the items. These results could be represented in a table and
with scatterplots. Item selection is based on the results of the item analysis.
These procedures need to be described in the manual.
6.3.2 Is the questionnaire reliable and valid?

The person who wants to use the questionnaire needs to know if the
questionnaire is reliable (i.e., there is consistency in measurement) and valid
(i.e., it measures what it claims to measure) for the target population.
...........
73 PYC 2616/1

The type of reliability (split-half, test-retest or alternate forms) needs to be

described and justified. Relevant results could be presented in table format.
The reliability coefficient is evaluated in terms of what can be regarded as an
acceptable level of reliability given the purpose of the questionnaire. Note that
for the purpose of this module, you will work with an estimate of the reliability
coefficient. If the questionnaire will be used for more than one cultural and/
or language groups, you need to show that the questionnaire is reliable for all
of these groups.
The type of validity (content, criterion-related or construct) also needs to be

described and justified in terms of the purpose of the questionnaire. You
need to explain how the evidence was gathered and relevant results could be
presented in table format. Based on this evidence, you must discuss the extent
to which the questionnaire measures what it claims to measure. Equivalence
for different cultural and/or language groups implies a lack of prediction and
construct bias.
As with the above example, when using a questionnaire to explore general

health in a population, it will be important to explore whether different
groups define “health” differently. One group might see health as a physical
problem, while another might see health as incorporating spiritual practices.
These differing views and definitions might affect how well the questionnaire
functions in terms of both reliability (does it measure health consistently) and
validity (does it measure health for various populations).
6.4 PROCEDURES FOR ADMINISTRATION, SCORING AND

INTERPRETATION
6.4.1 What are the instructions for the administration of the

questionnaire?
The instructions for administration should indicate who is allowed to administer
the questionnaire and it should refer to the situation in which the questionnaire
can be administered, the materials that will be required and how questions
should be dealt with. An introductory talk should also be given to respondents
in which the aim of the questionnaire and the purpose for which the information
is gathered are explained. The importance of the respondents’ contribution
and the information they provide should be emphasised, and confidentiality
should be ensured. If a computerised test is, for example, used in an industrial
setting, several issues need to be considered:
(1) Will all the participants be technologically advanced to work with

the computerised system? If not, what alternative arrangements are in
place, and how will they affect the outcome of the questionnaire?
(2) Will the computerised system be able to work consistently, even in the
face of power failures, etcetera?
(3) Will confidentiality be ensured, even if the questionnaire is linked to the
internet?
(4) Will those who administer the test need to have any special skills to
administer the test in a computerised manner, and how will they have to
adapt their instructions from a pen-and-paper test situation?
...........
74

6.4.2 How does one score and interpret the results?

The manual should include instructions for scoring those items in the
questionnaire that requires scoring. Sections in the questionnaire might consist
of different types of items (i.e., different scaling methods) and these will be
scored differently. In the case of items with a correct answer, each item could
score a 1 or a 0. By adding up the scores for all the items, one obtains a total
score (called the raw score) indicating how many items out of the total number
of items a person had correct. On a five-point rating scale a 1 might indicate
a respondent does not at all agree with a statement and a 5 might indicate
strong agreement with this statement. The total score indicates the person’s
attitude towards the topic under investigation. (Reverse-scoring has been
explained in lesson 4.)
Do you see that you might have a total score for more than one section of the
questionnaire? Guidelines for interpretation should then be provided for each
of these sections. In the case of right or wrong answers, a total score might
indicate the degree of knowledge on a topic and someone with a high score
knows more than someone with a low score. A total score on a rating scale
usually indicates an attitude towards a certain issue and one can distinguish
between individuals in terms of their attitude. You can therefore see that the
guidelines for the interpretation of the results are based on the aim of the
questionnaire.
In summary
Aim and design Properties Administration,

scoring and
interpretation
Aim Item analysis and Instructions for
Target population selection administration
Design Reliability Instructions for scoring

Validity Guidelines for
interpretation
6.5 GLOSSARY
Here are some basic terms related to the manual that you may not be familiar
with, but which you need to know. They will come up in your reading of the
material in this lesson.

Administration Taking responsibility Om Ho jara boikarabelo
Toepassing for organising verantwoordelikheid ba ho hlophisa le ho
and supervising te neem vir die tsamaisa ho tlatswa
Tsamaiso the completion of organisering van ha lenanepotso.
the questionnaire. en toesighouding
tydens die voltooiing
van die vraelys.
...........
75 PYC 2616/1

Intention To plan on doing Om te beplan om Ho rera ho etsa ntho

Bedoeling something; to have a iets te doen; om ’n e itseng; ho ba le
goal. doel te hê. sepheo.
Sepheo
Interpretation To explain what Om te verduidelik Ho hlalosa diphetho
Interpretasie the results on the wat die resultate van tsa lenanepotso ho
questionnaire mean die vraelys beteken ya ka sepheo seo ho
Tlhaloso given the aim of the wat betref die doel fanweng ka sona.
questionnaire. van die vraelys.
Manual The manual of the Die handleiding Manuale wa

Handleiding questionnaire gives van die vraelys lenanepotso o
information on the gee inligting oor fana ka lesedi ka
Manuale questionnaire as die vraelys asook lenanepotso le ka
well as instructions instruksies oor hoe ditaelo tsa kamoo
on how to use the om die vraelys te lenanepotso le
questionnaire. gebruik. sebediswang ka
teng.
Scoring To assign numbers in Om punte op ’n Ho fana ka dinomoro
Puntetoekenning a specific manner to spesifieke manier aan ka mokgwa o itseng
the responses on the die antwoorde op die dikarabong tse
Ho fana ka matshwao questionnaire. vraelys toe te ken. lenanepotsong.
Target population The group of Die groep mense Sehlopha sa batho
Teikenpopulasie people that the vir wie die vraelys seo lenanepotso le
questionnaire is bedoel is. etseditsweng sona.
Batho ba hlwailweng intended for.
6 AC TIVITIES
material for this lesson (manual: purpose and structure).
Manual: purpose
and structure
Press the button
the study guide.
...........
76

T ime re cord
METHOD In the previous lessons, you went through a process of developing and evaluating
a questionnaire. You now need to compile a manual in which this process
Activity 6.1
is described so that any potential user can know when and how to use
Discuss the process
of developing the the questionnaire and also have the information to determine how well the
questionnaire questionnaire will work for the intended purpose. Generally, such a manual
consists of a description of the aim and design of the questionnaire, an evaluation
Resource
of its properties, and the procedures required for administration, scoring and
Manual: purpose
and structure interpretation.
AC T I O N 01
In lesson 1, you defined the purpose of your questionnaire: you stated the aim
of the questionnaire (what it measures and how this information can be used)
and you indicated the target population (for whom it is intended). Write down
the aim statement of your questionnaire and describe the target population of
your questionnaire, clearly indicating which characteristics are important. Refer
to the suitability of the questionnaire in a multicultural and multilingual context.
In lesson 2, you described your items in terms of content and format. You
also specified the layout of the questionnaires. Use this information to give a
brief description of how the design of the questionnaire ensures that it meets
its purpose.
AC T I O N 02
In lesson 4, you administered your draft questionnaire to a group of people
who were representative of your target population. You used their results to
analyse the items and to determine the spread of responses for each item and
the relation between each item and the total over all the items. Briefly describe
how you used each of these techniques to select items for the final version of
your questionnaire. Provide an example of an item that you included and an
item that you excluded. Illustrate this with a scatterplot to show both the spread
of responses and the relation with the total score for each of the two items.
In lesson 5, you evaluated the reliability (consistency of measurement) and

validity (if the questionnaire measures what it claims to measure) of the final
version of your questionnaire for the representative group of people.
Evaluate the reliability of your questionnaire with reference to the type of

reliability and the approximate value of the reliability coefficient. Keep the
purpose of your questionnaire in mind. Write this down and use a table to
show your results. Because of the small sample size, you would probably not
...........
77 PYC 2616/1

be able to determine if the reliability is similar for different cultural and/or

language groups.
Evaluate the validity of your questionnaire with reference to the category of

validity and the evidence gathered to determine validity. Write this down and
use a table to show your results. Take note of the possibility of bias for different
groups although this does not form part of the module.
AC T I O N 03
You have written instructions for the administration of the questionnaire in
lesson 2. Write down your introductory talk and instructions for completing the
questionnaire. Check that you have referred to the following: the administrator;
the situation; the material; dealing with questions; the aim and importance;
and confidentiality.
In lesson 2, you also selected a scaling method. This can be used to assign
numerical values to the answers on the items and to score the rating scale. In
lesson 4, you had to score the rating scale in your questionnaire. In the manual,
you have to provide instructions for scoring those items in the questionnaire
that require scoring. Indicate how each item is scored and how item scores
are summed to get a total score. Write this down.
Lastly, you need to provide guidelines for the interpretation of the results of
the questionnaire. You should indicate whether the interpretation is based
on the response to each item separately, whether (as in the case of the rating
scale) the total score for a section is interpreted or whether the total score for
the questionnaire is used. Remember that the interpretation is linked to the
aim of the questionnaire (what the questionnaire measures).
Activity 6.2 The information you have is sufficient to compile a mini-manual for your
Compile a manual questionnaire. Copy the material from the previous exercises (activity 6.1)
under the following headings.
Resource
Manual: purpose AIM AND DESIGN
and structure
Aim and target population

Design of the questionnaire
PROPERTIES OF THE QUESTIONNAIRE
Item analysis and selection

Reliability
Validity
PROCEDURES FOR ADMINISTRATION, SCORING AND INTERPRETATION
Instructions for administration

Instructions for scoring
Guidelines for interpretation
Congratulations! You know have a manual for your questionnaire.

...........
78

You are a human resources (HR) officer and your company requested you
to determine how well the personnel are coping with work-related stress.
You decide to administer a number of questionnaires and one of these is the
leisure activities questionnaire (LAQ). Below is a section from the manual for
the questionnaire.
Mini manual for the leisure activities questionnaire (LAQ)
1. Aim and design
The leisure activities questionnaire (LAQ) can be used to obtain information
on what people do during their free time (that is, when they are not busy
with compulsory activities). Recreational or leisure activities are important to
ensure a balanced life and to help people cope with stress and it is important
to know if people are in fact partaking in these types of activities. The LAQ
can be used for any adult regardless of gender, education level, population
group, etcetera. There are three factors that are important when studying
leisure activities, namely the type of activity (whether it is a sport, a cultural
or a social activity), the consistency with which the activity is practiced (how
often and for how long) and the reason for doing the activity (for enjoyment,
amusement, etcetera). The LAQ consist of 60 items, 20 for each factor, that have
been developed together with representatives from various communities. This
was done because one would expect that socio-cultural factors would have
an influence on the leisure activities that people partake in and the various
possibilities had to be covered in the questionnaire. A mixture of inventories,
rating items and open questions were included.
QUESTION 1
Susan: I presume a good place to begin this discussion is the purpose of
the LAQ.
Lerato: Well, the purpose of the questionnaire clearly is to find out more
about the leisure activities that people partake in.
Ben: No, the questionnaire’s purpose is to determine if people are coping

with work-related stress.
Vusi: Actually, the purpose of the questionnaire is to assess people in

the workplace.
...........
79 PYC 2616/1

You agree most with, second most with and least with
(1) Lerato, Ben and Vusi

(2) Lerato, Vusi and Ben
(3) Ben, Lerato and Vusi
(4) Vusi, Lerato and Ben
(5) Vusi, Ben and Lerato
(6) Ben, Vusi and Lerato
QUESTION 2
The target population for the LAQ is adequately described.
(1) True, age is specified.

(2) False, it is unclear if it would be suitable for different language groups.
(3) True, gender is specified.
(4) False, it cannot be used for different population groups.
QUESTION 3
Which of the following is NOT a purpose of a manual?
(1) to describe the aim and target population of the questionnaire

(2) to provide information on the reliability of the questionnaire
(3) to provide statistics so that items can be selected
(4) to provide guidelines for the interpretation of results
Q U E S T I O N 1: O P T I O N 1 I S T H E K E Y – L E R ATO, B E N A N D V U S I
The purpose of the LAQ is to find out more about the type of activities that
people partake in, the consistency with which they practice the activities and
their reasons for doing the activities. That is, to find out more about the leisure
activities that people partake in. The intention of the HR officer is to find out
how well people are coping with work-related stress – he/she can use the LAQ
to get more information on the topic but the LAQ could also be used for other
types of projects as its main purpose is not to determine coping. The statement
that the purpose is to assess people in the workplace could be made about
many questionnaires and is not specific to the purpose of the LAQ.
The target population is described in detail (age, gender, educational level and
population group). Although age (option 1) and gender (option 3) are indeed
specified, each on its own would not be sufficient to say that the description
of the target population is adequate. Provision was made for differences in
culture during the design and option 4 is therefore incorrect. The only aspect
that remains unclear is whether the functioning of the questionnaire has been
determined for different language groups (option 2).
...........
80

Although the process followed to select items is discussed in the section dealing
with the properties of the questionnaire, the items have already been selected.
That is, the information is not provided to enable item selection.
...........
81 PYC 2616/1

Lesson
7 7
7 Review a questionnaire
In this lesson you will use everything you have learned in lessons 1, 2 and 3
to review a questionnaire.
AIMS
In lesson 7, you will learn how to use a rating scale to evaluate the extent to which
each aspect of a questionnaire meet the requirements in terms of correctness,
suitability and functionality. You will be given activities with examples to teach
you how to apply your knowledge.
Lesson
• evaluate the instructions of the questionnaire

• evaluate the characteristics of the items of the questionnaire as well as the
structure of the questionnaire
• review the functionality of the questionnaire
• content domain
• item format
• layout of the questionnaire
• suitability of the questionnaire
It should be clearly explained to the respondents who complete the question-

naire what the purpose of the questionnaire is. They should understand why
they are doing the questionnaire, how the information will be used and what
...........
82

the value is to them and their community. The items of the questionnaire should
be formulated in such a way that it is clear to all the respondents and it should
also be suitable to diverse contexts. The questionnaire will be functional for
different groups if the information on the intended content domain is gathered
for all respondents included in the target population.
7.1 INTRODUCTION
In lessons 1, 2 and 3, you designed and developed your own questionnaire.
You identified a suitable content domain, your questionnaire specification
document indicated the type of items, number of items, layout and so on
of your questionnaire, and you followed certain guidelines in writing items
for your questionnaire. In this lesson, you will apply the competencies you
acquired in developing your own questionnaire to evaluate a questionnaire
developed by someone else.
To evaluate a questionnaire you will use a rating scale that consists of four
sections. These sections correspond to the main aspects of a questionnaire. The
four main aspects one should consider when evaluating a questionnaire are
• the instructions of the questionnaire

• the characteristics of the items of the questionnaire
• the structure of the questionnaire
• the functionality of the questionnaire
7.2 QUESTIONNAIRE INSTRUCTIONS

The questionnaire should have a set of instructions that are intended for the
person who completes the questionnaire (the respondent). These instructions
may differ from questionnaire to questionnaire but should include at least
the following: (a) an indication of the purpose of the questionnaire, (b) to
what extent the information provided by the respondent will be treated as
confidential, and (c) how the respondent should complete the questionnaire.
(a) The purpose of the questionnaire: The purpose is described in the

questionnaire’s manual. The respondents (the people who complete a
questionnaire) may not understand the purpose of a questionnaire as
explained in the manual because they are not psychologists. However,
they have to know why they complete the questionnaire. In other words,
they have to understand how the information will be interpreted and
used. Therefore, the purpose of the questionnaire should be indicated in
simple terms in the instructions that are provided to the respondents. The
purpose of a questionnaire is reflected in statements that include phrases
such as: “The purpose of this questionnaire …”, “The aim is to …”, or “…
to learn how you feel about …”, etcetera.
(b) Confidentiality: Keep in mind that people reveal personal information

in their responses. They indicate how they feel or what they think about
something. Personal information is private and one should respect this. It is
...........
83 PYC 2616/1

important to indicate whether the information provided by the respondent

will be made public or whether it will be handled in a confidential manner.
If the information is to be kept confidential, the instructions should indicate
how confidentiality would be ensured. For example, confidentiality can be
assured through anonymity. A questionnaire is completed anonymously if
the respondent is not required to provide his/her name on the questionnaire
and if the questionnaire does not contain any biographical information that
could serve to identify the respondent. If a questionnaire is not completed
anonymously but the information is to be kept private, it is important to
indicate how the information will be handled. The instructions should
indicate who would have access to the information and for what reason
or purpose. See the resource on the layout of a questionnaire for more
information about confidentiality and anonymity.
(c) Instructions for completing the questionnaire: The questionnaire’s

instructions should indicate how the questionnaire is to be filled out.
These are general instructions, not the instructions provided for each item
or question. General instructions indicate what the respondent has to do
in general to complete the questionnaire. For example, the respondent
has to answer all questions, or the respondent has to use a pencil and
not an ink pen, or the respondent should not take more than 20 minutes
to complete the questionnaire.
Thus, there are three points to consider when a questionnaire’s instructions are
reviewed: the purpose of the questionnaire, the issue of confidentiality, and
the instructions for completing the questionnaire. The following rating item is
used to evaluate the instructions of a questionnaire:
The questionnaire instructions are correct if
(a) the purpose of the questionnaire is indicated correctly

(b) the matter of confidentiality is addressed correctly
(c) the general instructions for completing the questionnaire are indicated
correctly
Rate 1 if: none of the following options

Rate 2 if: a
Rate 3 if: a+b
Rate 4 if: a+b+c
7.3 ITEM CHARACTERISTICS

The items used in a questionnaire should be formulated correctly and should
be suitable. Items are formulated correctly if they are relevant to the topic,
clear and unambiguous, configured properly, not leading, and not likely to
elicit a socially desirable response. An item is suitable if the language it is
formulated in is at an appropriate level, if the respondents are likely to find it
relevant in their context, and if they possess the knowledge that is required
to respond to the item. The following rating items are used to judge the items
of a questionnaire:
...........
84

A questionnaire item is formulated correctly if
(a) the item has content and construct relevance

(b) the item is clear and unambiguous
(c) the item is configured correctly
(d) the item is not leading
(e) the item is not likely to elicit a socially desirable response

Rate 2 if: a
Rate 3 if: a+b+c
Rate 4 if: a+b+c+d+e
A questionnaire item is suitable if
(a) the respondents are likely to find the language level appropriate
(b) the respondents are likely to find the item relevant in their context
(c) the respondents are likely to possess the knowledge required to respond
correctly

Rate 2 if: a
Rate 3 if: a+b
Rate 4 if: a+b+c
7.4 QUESTIONNAIRE STRUCTURE

In the previous section, we considered the characteristics of individual
items. In this section, we look at the characteristics of the questionnaire as
a whole. Questionnaires should have sufficient items to cover the topic they
are concerned with, but they should not be too lengthy. In addition, the items
should be presented in such a way that the questionnaire
• counters response style and item bias.

• ensures efficiency – for example, a questionnaire should direct the
respondents’ ideas to obtain meaningful responses within general contexts
by grouping questions that relate to particular aspects of the topic under
consideration, and by using filter questions, and a good balance of different
question types.
• shows sensitivity towards respondents – for example, a questionnaire should
try to put respondents at ease by incorporating neutral and interesting
questions at the beginning, and forbidding, sensitive and personal questions
towards the end of the questionnaire; and questionnaires should allow the
respondents to raise opinions and voice feelings aroused by the questionnaire
by providing open-ended questions at the end of the questionnaire.
You can read more about these topics in the resources dealing with the
identification of a questionnaire’s content domain, the layout of a questionnaire,
and the writing of questionnaire items. The following rating item is used to
judge the structure of a questionnaire:
...........
85 PYC 2616/1

A questionnaire is structured correctly if
(a) the questionnaire has sufficient scope

(b) the questionnaire does not encourage a particular response style and item
bias
(c) the questionnaire does not encourage inefficient responding
(d) the questionnaire does not show insensitivity towards respondents

Rate 2 if: a
Rate 3 if: a+b
Rate 4 if: a+b+c+d
7.5 QUESTIONNAIRE FUNCTIONALITY

A further aspect to consider when evaluating a questionnaire is its functionality.
In general, questionnaires are used to obtain accurate information, provide
a standard format for recording facts, comments and attitudes, and facilitate
data processing. The functionality of a questionnaire is evaluated in the light of
the purpose of the questionnaire. Each questionnaire has an explicit purpose
in the sense that it is intended to get information from a particular content
domain or topic, and should be used for a particular target population. The
issue to be evaluated is whether the questionnaire is structured in such a
manner that it can function maximally in the light of its declared purpose. The
question is: Does the structure of the questionnaire (i.e. the kinds of items it
consists of and the sequence in which the items are presented) support the
questionnaire’s functionality (i.e. what the questionnaire could be used for,
what it is capable of) given its declared purpose (i.e. the kind of information
it is expected to deliver).
For example, suppose one wants to gather information about environmental

issues from a large number of inhabitants in order to improve the way in which
the environment is managed. The function of the questionnaire is to provide
accurate information in a standard format and to facilitate the processing
of the information. The questionnaire’s functionality would be supported
if the questionnaire consists of closed questions with true/false, yes/no, or
multiple-choice answers. If one were to use open-ended questions, it would
be difficult to extract accurate information from the large number of different
answers, and to code the information for data processing. In other words, a
questionnaire consisting of open-ended questions would limit the functionality
of the questionnaire. However, if the intention were to explore new and unique
environmental issues in a particular environment, the functionality of the
questionnaire would be supported if one were to use open-ended questions to
get the necessary information and if one were to administer the questionnaire
to a small sample of inhabitants. In other words, the structure and functionality
of a questionnaire is a function of its declared purpose. The following rating
item is used to judge the functionality of a questionnaire:
...........
86

A questionnaire is functional if
(a) the information gathered by the questionnaire concerns the intended

content domain
(b) the questionnaire is suitable for the intended target population
(c) the questionnaire produces information in a format that facilitates data
processing

Rate 2 if: a
Rate 3 if: a+b
Rate 4 if: a+b+c
7.6 IN SUMMARY
Instructions (a) the purpose of the questionnaire is indicated correctly

(b) the matter of confidentiality is addressed correctly
(c)
the general instructions for completing the questionnaire
are indicated correctly
Item formulation (a) the item has content and construct relevance
(e)
the item is not likely to elicit a socially desirable response
Item suitability (a)
the respondents are likely to find the language level
appropriate
(b)
the respondents are likely to find the item relevant in
their context
(c)
the respondents are likely to possess the knowledge
required to respond correctly
Structure (a) the questionnaire has sufficient scope
(b)
the questionnaire does not encourage a particular
response style and item bias
(c)
the questionnaire does not encourage inefficient
responding
(d)
the questionnaire does not show insensitivity towards
respondents
Functionality (a)
the information gathered by the questionnaire concerns
the intended content domain
(b)
the questionnaire is suitable for the intended target
population
(c)
the questionnaire produces information in a format that
facilitates data processing
...........
87 PYC 2616/1

7.7 GLOSSARY
Here are some basic terms related to the review of a questionnaire that you
may not be familiar with, but which you need to know. They will come up in
your reading of the material in this lesson.

Rating scale A numeric value is ’n Numeriese waarde Nomoro e filweng
Graderingskaal assigned to an aspect word toegeken aan ’n ntlha e itseng ya
of the questionnaire. aspek van die vraelys. lenanepotso.
Mokgwa wa
tekanyetso
Confidentiality The protection of Die beskerming van Ho sireletsa tsebiso ya
Vertroulikheid personal information. persoonlike inligting. dintlha tsa motho.
Sephiri
Anonymity A person’s name/ ’n Persoon se Ha ho sa tsejwe
Anonimiteit identity is unknown. naam/identiteit is lebitso la motho.
onbekend.
Ho se bolele lebitso
Functionality Being suited to serve Om gepas te wees om -loketseng ho fihlella
Funksionaliteit a purpose. ’n doel te dien. sepheo.
Ho kgona ho etsa
mosebetsi
7 AC TIVITIES
material for this lesson (content domain, item format, layout of the
questionnaire, suitability of the questionnaire and writing questionnaire
items).
Content domain Layout of a

Item format questionaire
or go to page 112 in Press the button
the study guide. the study guide. the study guide.
Suitability of a Writing
questionaire questionnaire items
...........
88

T ime re cord
METHOD The questionnaire rating scale consists of the five rating items discussed in the
introduction. These are the criteria required when you review a questionnaire.
Activity 7.1
You will now get the opportunity to apply these criteria.
Use the
questionnaire rating
scale to review a
questionnaire AC T I O N 01
Resource Study the following questionnaire:
Content domain
Layout of a SIGNIFICANT ASPECTS OF RESIDENTIAL
questionnaire
Writing AREAS QUESTIONNAIRE
questionnaire items
The significant aspects of residential areas questionnaire (SARAQ) measures
what individuals consider to be important aspects of their residential areas.
The questionnaire is intended for individuals of age 18 years and older. The
respondents should have been resident in the area for at least six months
before completing the questionnaire.
For the purpose of this questionnaire, residential areas are viewed in terms of
high density, medium density and low density areas. High density areas refer
to living units in blocks of multi-storey buildings (i.e. buildings consisting of
three or more floors). Medium density living areas contain duplex and simplex
living units. Low density living areas contain single freestanding homes. Living
density plays an important role in what people consider to be significant aspects
of their living areas. The primary aspects of any living environment are issues
concerning (a) necessity/desire, (b) comfort/ convenience, (c) recreation, and
(d) security.
Read each statement carefully and tick the relevant box for your answer. (SA
= Strongly agree, A = Agree, D = Disagree, and SD = strongly disagree)
SA A D SD
1 It is more important to be warm in winter than
to be close to the supermarket.
2 A house should be proper.
3 I want a beautiful house and a happy family.
4 I hate to spend more money on my electricity
bill than on my golfing fees.
5 Every home should have a television set.
6 When did you move into the house you are
staying in at present?
...........
89 PYC 2616/1

SA A D SD
7 The return on investment risk is much lower in

a good residential area.
8 I hate not having a garage in which to park my
car.
9 Every person needs a holiday away from home
at least once a year.
10 Not being close to a church is better than not
being far from one’s work.
AC T I O N 02
Refer to the rating scale discussed in the introduction and use the following
review sheet to review the questionnaire. Tick the number that corresponds
to the rating you assign.
1. The instructions of the questionnaire 1 2 3 4

2. The formulation of item 1 1 2 3 4
3. The suitability of item 1 1 2 3 4
22. The structure of the questionnaire 1 2 3 4
23. The functionality of the questionnaire 1 2 3 4
...........
90

Activity 7.2 Your ratings may differ from the standard set of ratings that are provided here.
Compare your In most cases you will probably agree that the answers given are better. Here
reviews to a
standard set of are the questionnaire’s standard set of ratings. The explanations for the ratings
ratings follow below.
1. The instructions of the questionnaire 1 2√ 3 4

2. The formulation of item 1 1 2 3 4√
3. The suitability of item 1 1 2 3 4√
4. The formulation of item 2 1 2√ 3 4
6. The formulation of item 3 1 2 3√ 4
9. The suitability of item 4 1 2√ 3 4
12. The formulation of item 6 1√ 2 3 4
15. The suitability of item 7 1√ 2 3 4
18. The formulation of item 9 1√ 2 3 4
20. The formulation of item 10 1 2√ 3 4
22. The structure of the questionnaire 1 2 3√ 4
23. The functionality of the questionnaire 1 2√ 3 4
1. Rating: 2
The purpose of the questionnaire is indicated (to determine what individu-

als consider to be important aspects of their residential areas). However,
the matter of confidentiality is not addressed and no instructions are
provided for completing the questionnaire.
2. Rating: 4
Item 1 is formulated correctly. The item has content/construct relevance

(refers to comfort/convenience). It is clear and unambiguous and con-
figured correctly, not leading and not likely to elicit a socially desirable
response.
...........
91 PYC 2616/1

3. Rating: 4
Item 1 is suitable. The language level is appropriate, the item is contextu-

ally relevant, and the respondents are likely to possess the knowledge
required to respond appropriately.
4. Rating: 2
Item 2 is not formulated correctly. It has content and construct relevance

(it refers to necessity, i.e. that the house should be in a particular way,
namely “proper”) but it is not clear what the statement (that a house
should be proper) means. The item is configured correctly (one can agree/
disagree with the statement). It is not leading and not likely to elicit a
socially desirable response.
5. Rating: 4
Item 2 is suitable. The respondents are likely to find the language level
appropriate and the item relevant in their context. They are also likely to
possess the knowledge required to respond.
6. Rating: 3
Item 3 is not formulated correctly. It has content and construct relevance

(it refers to what is desired, namely a beautiful house and a happy family).
It is clear and unambiguous, is configured correctly, and is not leading,
but it is likely to elicit a socially desirable response (everybody wants a
beautiful house and a happy family).
7. Rating: 4
appropriate and the item relevant in their context. They are also likely to
possess the knowledge required to respond.
8. Rating: 4
Item 4 is formulated correctly. It differentiates between two constructs,

namely recreation (golf) and necessity (electricity). It is clear and unam-
biguous and configured correctly. It is not leading and not likely to elicit
a socially desirable response.
9. Rating: 2
Item 4 may not be suitable. The respondents are likely to find the lan-
guage level appropriate. But they may not find it relevant. The item is
relevant only to those who play golf. People who do not know what the
going rate for golfing fees is do not possess the knowledge required to
respond correctly.
...........
92

10. Rating: 4
Item 5 is formulated correctly. It has content and construct relevance

(refers to necessity), is clear and unambiguous, is configured correctly, is
not leading, and not likely to elicit a socially desirable response.
11. Rating: 2
Item 5 may not be suitable. The respondents are likely to find the language
level appropriate. But they may not find it relevant. The item is relevant
only to those who can afford a television set. However, they are likely
to possess the knowledge required to respond correctly.
12. Rating: 1
Item 6 is not formulated correctly. It does not have content/construct

relevance. Although it is not unclear or ambiguous, leading and likely
to elicit a socially desirable response, it is not configured correctly. One
cannot agree/disagree with the question.
13. Rating: 4
Item 6 is a suitable question. The respondents are likely to find the

language level appropriate and the question relevant. They are likely to
possess the knowledge to answer the question correctly.
14. Rating: 4
Item 7 is formulated correctly. It has content and construct relevance. (It

refers to the security of investment.) It is clear and unambiguous, con-
figured correctly, not leading, and not likely to elicit a socially desirable
response.
15. Rating: 1
Item 7 is not suitable. Not all the respondents are likely to find the lan-
guage level appropriate, or to find the statement to be relevant in their
context (especially respondents in lower socio-economic groups who are
more likely to rent than to buy property). Furthermore, these individuals
will probably not have the knowledge required to respond correctly.
16. Rating: 4
Item 8 is formulated correctly. It has content and construct relevance

(it would be comfortable and convenient to have a garage), is clear and
unambiguous, is configured correctly, is not leading, and not likely to
elicit a socially desirable response.
...........
93 PYC 2616/1

17. Rating: 2
Item 8 may not be suitable. The respondents are likely to find the language
level appropriate. But they may not find it relevant. The item is relevant
only to those who can afford a car. However, they are likely to possess
the knowledge required to respond correctly.
18. Rating: 1
Item 9 is not formulated correctly. Although the item is clear, configured

correctly and not leading, it may elicit a socially desirable response (eve-
rybody wants a holiday once a year). Nevertheless, the biggest problem is
that the item does not show content and construct relevance. Although it
implies recreation, the reference is to a holiday away from home, which is
not a recreational aspect of the residential area where the individual lives.
19. Rating: 2
Item 9 is not suitable. The respondents are likely to find the language ap-
propriate but individuals from lower socio-economic circumstances may
not find the idea of a holiday away from home relevant in their context,
although they do have the knowledge to respond to the question.
20. Rating: 2
Item 10 is not formulated correctly. The item has content and construct
relevance (it refers to convenience), it is configured correctly, is not
leading and not likely to elicit a socially desirable response. However,
it is difficult to comprehend because it is formulated in an unclear and
ambiguous manner.
21. Rating: 4
appropriate, and the question relevant. They are likely to possess the
knowledge to answer the question correctly.
22. Rating: 3
The questionnaire has sufficient scope in the sense that it refers to all four
content/construct areas. It does not encourage inefficient responding (it
requires all items to be responded to, and the items can be completed
in the order in which they are presented). It is not insensitive towards
respondents as there are no items that may cause embarrassment. How-
ever, the questionnaire encourages a particular response style. The re-
spondents are likely to agree with most statements, leading to the left or
centre columns to be marked more often.
23. Rating: 2
The information gathered by the questionnaire concerns the intended

content domain (namely significant aspects of residential areas). It also
...........
94

produces information that facilitates data processing. However, the ques-

tionnaire is not suitable for the intended target population as half of the
questionnaire items are not fully suitable.
Activities 7.1 and 7.2 serve as self-assessment. We will, however, repeat two
of these questions to show you how we will usually ask the questions.
QUESTION 1
Rate the formulation of item 1.
Item 1 is formulated correctly if
(a) the item has content and construct relevance

(e) the item is not likely to elicit a socially desirable response
1. Rate 1 if: none of the following options

2. Rate 2 if: a
3. Rate 3 if: a+b+c
4. Rate 4 if: a+b+c+d+e
QUESTION 2
Rate the suitability of item 4.
Item 4 is suitable if
(a) the respondents are likely to find the language level appropriate
(b) the respondents are likely to find the item relevant in their context
(c) the respondents are likely to possess the knowledge required to respond
correctly
1. Rate 1 if: none of the following options

2. Rate 2 if: a
3. Rate 3 if: a+b
4. Rate 4 if: a+b+c
See the explanation in activity 7.2.
...........
95 PYC 2616/1

Lesson
8 8
8 Review a manual
In this lesson you will use everything you have learned in lessons 1 to 6 to
review a manual. The information in lesson 6 is especially important.
AIMS
In lesson 8, you will learn how to use a rating scale to evaluate the extent to which
a manual covers the required content, is logically structured and is written in
clear and comprehensible language. You will be given activities with examples
to teach you how to apply your knowledge.
Lesson
• evaluate if the purpose, properties, and instructions for utilising the

questionnaire are included in the manual
• evaluate if coherent information related to each topic is covered in a subsection
and if the topics are logically arranged
• evaluate the language used in writing the manual
• manual: purpose and structure
There are various subsections in a manual in which reference should be made

to the relevance and functioning of the questionnaire in multicultural and
multilingual contexts. For example, the value of the questionnaire for the com-
munity represented by the target population should be clear. The properties
of the questionnaire should furthermore be evaluated to ensure equivalence in
...........
96

diverse contexts. And the sociocultural context should be accounted for in the
instructions for interpretation.
8.1 INTRODUCTION
In lesson 6 you compiled a manual in which the process of developing and
evaluating your questionnaire was described. In this lesson you will apply
the competencies you acquired in writing a manual for your questionnaire to
evaluate a manual written by someone else.
To evaluate a manual, you will use a rating scale that consists of three sections.
These sections correspond to the main criteria for evaluating a manual. The
three main criteria one should consider when evaluating a manual are
• the content of the manual

• the presentation of the information
• the use of language
8.2 THE CONTENT OF THE MANUAL

The purpose of a questionnaire’s manual is to provide information for the person
who plans to use the questionnaire (that is, the questionnaire administrator).
Questionnaire administrators need to know three things: (a) whether the
questionnaire is relevant for the purpose the questionnaire administrator has
in mind, (b) whether the questionnaire will function correctly, and (c) how
the questionnaire should be used. In other words, the manual should cover
the following topics: (a) the purpose of the questionnaire (i.e. the aim and
design of the questionnaire, and the population it can be used for as well as
the suitability of the questionnaire in a multicultural and multilingual context),
(b) the properties of the questionnaire (i.e. the analysis and selection of the
questionnaire’s items, and the reliability and validity of the questionnaire for all
cultural and/or language groups for whom the questionnaire will be used), and
(c) instructions for utilising the questionnaire (i.e. instructions for administration,
scoring and interpretation). The following rating items are used to evaluate the
content of the manual:
The purpose of the questionnaire is described if
(a) the aim of the questionnaire is stated

(b) the target population is defined
(c) the design of the questionnaire is described

Rate 2 if: a
Rate 3 if: a+b
Rate 4 if: a+b+c
...........
97 PYC 2616/1

The properties of the questionnaire are discussed if
(a) the sample used to test the questionnaire is described correctly

(b) item analysis and selection procedures are described correctly
(c) the reliability of the questionnaire is discussed correctly
(d) the validity of the questionnaire is discussed correctly

Rate 2 if: a
Rate 3 if: a+b
Rate 4 if: a+b+c+d
The procedures for utilising the questionnaire are described if
(a) instructions are provided for administering the questionnaire

(b) instructions are provided for scoring the questionnaire
(c) instructions are provided for interpreting results

Rate 2 if: a
Rate 3 if: a+b
Rate 4 if: a+b+c
8.3 THE PRESENTATION OF THE INFORMATION

The content in the manual should be presented in a properly structured
manner. The text should be clustered into topical sections, which should be
presented under descriptive headings. These topical sections may be divided
into further subsections and subheadings. If a text is well structured one can
get a good idea of what the text is about by simply scanning these headings.
A note on the side: The process also works in reverse. When one writes a
text, it is a good idea to unpack one’s topic first in terms of headings and
subheadings. Once the framework of the text is unpacked in this manner, it
is a simple task to write about the subtopics specified by each heading. The
topics (and subtopics) should be presented in logical order. It is not always
easy to decide on the best order in which to present material. One would
expect a manual (which is a document describing processes) to describe the
process from beginning to end. For example, in the case of a questionnaire’s
manual, the information is not presented in a properly structured manner if
the way in which the questionnaire is to be administered is described before
the questionnaire’s purpose. A properly structured manual begins with a
description of the purpose of the questionnaire, followed by a discussion of
the questionnaire’s properties, and concluding with instructions for using the
questionnaire. The following rating item is used to evaluate the presentation
of information:
The manual is structured properly if
(a) the text is clustered into topical sections

(b) the topics are presented in logical order
(c) the information about each topic is relevant and presented coherently
...........
98


Rate 2 if: a
Rate 3 if: a+b
Rate 4 if: a+b+c
8.4 THE USE OF LANGUAGE

In addition to being properly structured, the manual should be written in clear,
precise and correct language. The text should be easy to read. This does not
mean that technical terms have to be avoided (manuals are meant to convey
technical information) but it does mean that the technical information should
be conveyed in language that is simple, straightforward, factual and objective.
If you are required to write a manual, avoid personal opinions. Stick with
factual information and convey your ideas by means of short, direct sentences.
Avoid using difficult language. Language becomes difficult when sentences
are phrased in complicated ways, or when little-known words, phrases or
expressions are used. For example, instead of saying: “The questionnaire has
been found to have a high degree of reliability”, rather say: “The questionnaire
is very reliable”. Do not use overly difficult words, or write prose that belongs in
a literary essay. Manuals should be written in simple, straightforward language
to ensure unambiguous and precise communication. However, the use of
plain, straightforward language is not an excuse for using language in a poor
or sloppy manner. Always check the text for grammar and spelling mistakes.
And remember, you have to know exactly what you want to say when you put
your ideas in writing. Sentences often are poorly formulated not because the
authors of those sentences do not know their grammar, but because they do
not know what they want to say. The following rating item is used to evaluate
the language used in writing the manual:
The manual is written properly if
(a) the text is factual and objective

(b) the text is clear and to the point, with no ambiguous and/or conflicting
statements
(c) the text is not inundated with numerous spelling, grammar and punctuation
errors

Rate 2 if: a
Rate 3 if: a+b
Rate 4 if: a+b+c
...........
99 PYC 2616/1

In summary
Purpose (a) the aim of the questionnaire is stated
Properties (a)
the sample used to test the questionnaire is described
correctly
(b)
item analysis and selection procedures are described
correctly
(c)
the reliability of the questionnaire is discussed correctly
(d) the validity of the questionnaire is discussed correctly
Procedures (a)
instructions are provided for administering the
questionnaire
(b)
instructions are provided for scoring the questionnaire
(c) instructions are provided for interpreting results
Structure (a) the text is clustered into topical sections
(c)
the information about each topic is relevant and presented
coherently
Language (a) the text is factual and objective
(b)
the text is clear and to the point, with no ambiguous and/
or conflicting statements
(c)
the text is not inundated with numerous spelling, grammar
and punctuation errors
This is the last lesson.
8.5 GLOSSARY
Here are some basic terms related to the topics of reliability and validity that

Topical According to a Volgens ’n spesifieke Ho ya ka sehlooho se
Volgens onderwerp specific topic. onderwerp. itseng.
-sehlooho
Factual Based on facts Gebaseer op feite. -itshetlehileng ka
Feitelik dintlha tsa nnete
-nnete
Ambiguous Open to more than Oop vir meer as een -hlalosehang ka
Dubbelsinnig one interpretation. interpretasie. ditsela tse ngata.
-sa hlakang
...........
100

Logical Based on reason. Gebaseer op rede. -itshetlehileng ka

Logies mabaka.
-utlwisisehang
Coherent Logical and Logies en goed -utlwisisehang le ho
Samehangend well-organised. georganiseerd. hlophiswa hantle.
Utlwisiseha
8 AC TIVITIES
material for this lesson (manual: purpose and structure).
Manual: purpose
and structure
Press the button
the study guide.
T ime re cord
METHOD The manual rating scale consists of the five rating items discussed in the
introduction. These are the criteria that are required when you review a manual.
Activity 8.1
You will now get the opportunity to apply these criteria.
Use the manual
rating scale to review
a manual
Resource AC T I O N 01
Manual: purpose Study the manual below.
and structure
MARSTON PERSONALITY QUESTIONNAIRE MANUAL
The MPQ is provided in an A4 size booklet. The booklet contains the

instrucsions for completing the questionnaire as well as the questionnaire
itself and an answer sheet. A pencil are required to mark the answers on the
answer sheet. Which can be read and scored by computer. A special computer
programme is required to score the MPQ and to compile reports of the
results. A score mask is provided for scoring the answer sheet manually. The
questionnaire takes less than 30 minutes to complete. No special skills are
required to administer the questionnaire. The administrator is not required to
perform calculations. A table is used to convert raw scores to profile scales. As
individuals complete the questionnaire independently it can be administered
...........
101 PYC 2616/1

in groups. The questionnaire were been completed without interruption

under supervision of an administrator. It is not advisable to allow individuals
to complete the questionnaire at home in their own time.
No special scoring instructions are required if the computer programme is used
to read and score the answer sheets. All calculations are done automatically
and the program offers extensive help functions to familiarise the user with
the program. The computer calculates the behaviour profile and provides and
interpretation of the profile. The interpretation is displayed on screen, and can be
printed out if required. It is good practice to provide the person who completed
the questionnaire with the printout, and to allow his/her to ask questions to
clarify the interpretation of his/her result. Instructions are printed on the score
mask to assist with manual scoring. The mask is placed over the answer sheet
to show the relationship between the answers and the behaviour factors. Each
answer count one mark for the factor associated with that answer. The marks
obtained for each factor are added together to provide totals for each of the
four factors. The scale values that correspond to these totals are read from a
table. The factor totals are expressed on a scale that ranges from 1 to 10. The
4 scale values are used to compile an individual’s behaviour profile. There
are a large number of possible combinations of the scale values of the four
factors. Because each one of these combinations produces a different behaviour
profile there are an equally large number of possible behavior profiles. The
administrator has to make sure that the respondent completes all items in the
questionnaire. The individual’s behaviour profile cannot be determined if items
have been left out. An interpretation has been written for each of the possible
profiles. The profiles and their interpretations are contained in a separate profile
manual. Therefore, to interprate a result it is merely necessary to look up the
interpretation that corresponds to the behaviour profile in question.
All person has a preferred style of behaviour. People differ with regard to
their likes and dislikes. They differ with regard to the ways in which they do
things. This is the reason why people do different jobs. One person may love
a particular job but someone else may hate that job. The Marston Personality
Questionnaire (MPQ) was designed to measure individuals’ preferred behaviour
styles in their work environments. The MPQ is intended to be used with the
Marston Job Description System (MJDS). The MJDS provides a method to
describe any job in terms of behaviour styles. In other words, once a job has
been described by the MJDS one knows what kind of behaviour style would
be required of the person who does the job. People with behaviour styles that
are similar to the behaviour style required by the job will be happier in the
job. People who does not have the behaviour style required by the job will
find the job more stressful because they have to adapt their behaviour styles
to the requirements of the job. The MPQ can be used for any working person.
It is applicable from the most basic to the most sophisticated jobs, and works
across gender, race and education levels.
The MPQ measures preferred behaviour style in terms of four basic factors.
These factors drive {the ability to get things done personally), interaction (the
abylity to work with people), management (the ability to keep systems going)
and regulation (the ability to adhere to rules and processes). The questionnaire is
designed to measure how much drive, interaction, management and regulation
an individual prefers. Each factor is measured on a ten point scale, ranging
from 1 to 10.
...........
102

The questionnaire consists of 60 multiple choice items. The four factors are
covered in each item. Each item has the following format:
If [description of work situation] I prefer to:
(a) [description of ‘drive’ action]
(b) [description of ‘interaction’ action]
(c) [description of ‘management’ action]
(d) [description of ‘regulation’ action]
For example:
If things get really hectic at work I prefer to:
(a) put my mind to it and get things done as quickly as possible
(b) consult with my colleagues
(c) dig in and maintain a steady work pace
(d) stick with company policy
To counter response biasse the sequence in which the action descriptions are
provided is varied randomly. For example, the [description of ‘drive’ action]
is not always placed first. Fifty different [descriptions of work situations] are
used to cover a wide range of work events.
A large sample of nearly nine hundred university students was used in the
development of the questionnaire. The sample came from a number of different
universities across the country and was representative of the south african
population in respects of gender, race and education levels. All students were
in their final year of study and ready to enter the job market. The original MPQ
that was used in the sample study consisted of 145 items. An item analysis
showed that 67 of these items were really good but that the remaining 78 items
did not meet the criteria to be included. A further seven of the 67 selected items
were excluded because they overlapped with [description of work situation].
Furthermore 60 items were retained for the final version of the MPQ. The
large number of good items ensures a high degree of questionnaire reliability,
and supports the questionnaire’s construct validity. The validity coefficient of
0,91 is very high for this kind of questionnaire.
– End of manual –
AC T I O N 02
Refer to the rating scale discussed in the introduction and use the following
review sheet to assess the manual. Tick the number that corresponds to the
rating you assign.
1 The manual described the purpose of the questionnaire 1 2 3 4

2 The manual described the properties of the 1 2 3 4
questionnaire
3 The manual describes the procedures for utilising 1 2 3 4
the questionnaire
4 The manual is structured properly 1 2 3 4
5 The manual is written properly 1 2 3 4
...........
103 PYC 2616/1

Activity 8.2 Your ratings may differ from the standard set of ratings that are provided here.
Compare your In most cases, you will probably agree that the answers given are better. Here
reviews to a standard are the manual’s standard set of ratings. The explanations for the ratings follow
set of ratings
below.
1 The manual described the purpose of the questionnaire 1 2 3 4√

2 The manual described the properties of the 1√ 2 3 4
questionnaire
3 The manual describes the procedures for utilising 1 2 3 4√
the questionnaire
4 The manual is structured properly 1√ 2 3 4
5 The manual is written properly 1 2 3√ 4
1. Rating: 4
The aim of the questionnaire is stated: The manual indicates that the MPQ
was designed to measure individuals’ preferred behaviour styles in their work
environments.
The target population is defined: According to the manual the MPQ can be used
for any working person, that it is applicable from the most basic to the most
sophisticated jobs, and that it works across gender, race and education levels.
The design of the questionnaire is described: The content domain of the

questionnaire is indicated as the four factors (drive, interaction, management
and regulation) constituting behaviour preference. The manual describes the
format of the items used in the questionnaire and also indicates how these
items cover the content domain.
2. Rating: 1
The manual does poorly in this regard. It meets none of the review criteria.
The sample used to test the questionnaire is not described correctly if one
views this description in light of the questionnaire’s target population. The
questionnaire’s target population is working people but the sample used to
test the questionnaire comes from a student population.
The manual mentions item analysis and selection but it does not describe
how this was done. We read that 78 items did not meet the selection criteria
but we do not know what these criteria were. Only one selection criterion is
mentioned, namely that seven items were deleted due to their overlap with
[description of work situation]. However, this has to be incorrect. All items
should have a [description of work situation] otherwise the questionnaire will
not have face and content validity. Therefore this statement may be a misprint
(perhaps the intention was to say that the items were rejected because of poor
or low overlap), or else the author of the manual simply did not know what
he/she was talking about.
...........
104

Validity and reliability: The manual mentions validity and reliability but there is
no proper description and discussion of these properties of the questionnaire.
3. Rating: 4
The manual describes the procedures required to utilise the questionnaire. It

indicates how to administer the questionnaire, how to score the responses,
and how to interpret the results.
Administration: The manual indicates that the questionnaire can be completed

individually or in groups and that it takes less than 30 minutes to complete. It
should be done without interruption and under supervision of the administrator.
No special skills are required to administer the questionnaire. Instructions
for completing the questionnaire are provided in the booklet that contains the
questionnaire and its answer sheet. A pencil is required to mark the answers
on the answer sheet.
Scoring: The questionnaire can be scored manually or by computer. A score

mask is provided for manual scoring. Instructions are printed on the score
mask. The mask is placed over the answer sheet to show the relationship
between the answers and the behaviour factors. Each answer counts one mark
for the factor associated with that answer. The marks obtained for each factor
are added together to provide totals for each of the four factors. The scale
values that correspond to these totals are read from a table. In this manner,
the factor totals are expressed on a scale that ranges from 1 to 10. Scoring
the questionnaire by computer requires a special computer programme. No
special scoring instructions are required if the computer programme is used
to read and score the answer sheets. All calculations are done automatically
and the programme offers extensive help functions to familiarise the user with
the programme.
Interpretation: The questionnaire scores can be interpreted manually or

by computer. There are a large number of possible combinations of the scale
values of the four factors. Because each one of these combinations produces
a different behaviour profile, there are an equally large number of possible
behaviour profiles. An interpretation has been written for each of the possible
profiles. The profiles and their interpretations are contained in a separate
profile manual. Therefore, to interpret a result it is merely necessary to look
up the interpretation that corresponds to the behaviour profile in question.
When computer scoring is used, the computer automatically compiles the
behaviour profile and selects the corresponding interpretation from its database.
4. Rating: 1
The manual is not structured properly. The text is not clustered into topical
sections, topics do not follow on each other in logical order, and although the
information about each topic is relevant, it is not always presented coherently.
...........
105 PYC 2616/1

Clustering into topical sections: The manual does not use any headings to
organise its content. The text should have been structured under headings
such as:
(1) the purpose of the MPQ

(2) properties of the MPQ
(3) how to use the MPQ
Each of these topical sections could have been structured further in terms of
subsections. For example:
3. how to use the MPQ

3.1 administering the MPQ
3.2 scoring the MPQ
3.3 interpreting MPQ results
The logical order of topics: The manual does not present the topical sections
in logical order. The first two paragraphs deal with the administration of the
MPQ. The purpose of the questionnaire, which should have been indicated
first, is only introduced in the third paragraph. The rest of the text is structured
logically though. The questionnaire is described in terms of its items, and this
is followed by an explanation of how the questionnaire was pilot-tested.
Relevance and coherency of topics: The information provided about the

topics (purpose, properties and utilisation of the questionnaire) is relevant.
However, the information about the utilisation of the questionnaire (processes
of administration, scoring and interpretation) is not presented in a coherent
manner. In the first two paragraphs of the manual the author jumps between
topics. An analysis of these paragraphs shows the following:
Sentences from the manual … This is about …

The MPQ is provided in an A4-size booklet. The Administration
booklet contains the instrucsions for completing the
questionnaire as well as the questionnaire itself and
an answer sheet.
A pencil are required to mark the answers on the Going from

answer sheet. Which can be read and scored administration to
by computer. scoring
A special computer programme is required to score Scoring

the MPQ and to compile reports of the results. A
score mask is provided for scoring the answer sheet
manually.
The questionnaire takes less than 30 minutes Administration

to complete. No special skills are required to administer
the questionnaire.
...........
106

The administrator is not required to perform Scoring

calculations. A table is used to convert raw scores to
profile scales.
As individuals complete the questionnaire Administration

independently it can be administered in groups.
The questionnaire were been completed without
interruption under supervision of an administrator. It
is not advisable to allow individuals to complete the
questionnaire at home in their own time.
No special scoring instructions are required if Scoring

the computer programme is used to read and score the
answer sheets. All calculations are done automatically
and the programme offers extensive help functions
to familiarise the user with the programme.
The computer calculates the behaviour profile and Going from

provides and interpretation of the profile. scoring to
interpretation
It is good practice to provide the person who completed Interpretation

the questionnaire with the printout, and to allow his/
her to ask questions to clarify the interpretation of
his/her result.
Instructions are printed on the score mask to assist Scoring

with manual scoring. The mask is placed over the
answer sheet to show the relationship between the
answers and the behaviour factors. Each answer
counts one mark for the factor associated with that
answer. The marks obtained for each factor are added
together to provide totals for each of the four factors.
The scale values that correspond to these totals are
read from a table. The factor totals are expressed on
a scale that ranges from 1 to 10. The 4 scale values
are used to compile an individual’s behaviour profile.
There are a large number of possible combinations of Interpretation

the scale values of the four factors. Because each one
of these combinations produces a different behaviour
profile, there are an equally large number of possible
behavior profiles.
The administrator has to make sure that the Administration

respondent completes all items in the questionnaire.
The individual’s behaviour profile cannot be
determined if items have been left out.
...........
107 PYC 2616/1

An interpretation has been written for each of the Interpretation

possible profiles. The profiles and their interpretations
are contained in a separate profile manual. Therefore,
to interprate a result it is merely necessary to look up
the interpretation that corresponds to the behaviour
profile in question.
5. Rating: 3
The manual is written in a factual and objective style, and the text is clear and
to the point, with no ambiguous and/or conflicting statements. Unfortunately,
the manual contains numerous spelling, grammar and punctuation errors.
Incorrect Correct
First paragraph:
instrucsions instructions
A pencil are required A pencil is required
an answer sheet. Which can be an answer sheet, which can be
questionnaire were been completed questionnaire should be completed
Second paragraph:
to allow his/her to ask to allow him/her
Each answer count Each answer counts
The 4 scale values The four scale values
behavior behaviour
interprate interpret
Third paragraph:
All person has Every person has
People who does not have People who do not have
Fourth paragraph:
drive {the ability drive (the ability
abylity ability
...........
108

Seventh paragraph:
biasse bias
Eight paragraph:
the south african the South African
Furthermore 60 items Thus 60 items
Activities 8.1 and 8.2 serve as self-assessment. We will, however, repeat two
of these questions to show you how we will usually ask the questions.
QUESTION 1
Rate the manual’s indication of the purpose of the MPQ.
The purpose of the questionnaire is described if
(a) the aim of the questionnaire is stated

(1) Rate 1 if: none of the following options

(2) Rate 2 if: a
(3) Rate 3 if: a+b
(4) Rate 4 if: a+b+c
QUESTION 2
Rate the structure of the manual.
The manual is structured properly if
(a) the text is clustered into topical sections

(c) the information about each topic is relevant and presented coherently
(1) Rate 1 if: none of the following options

(2) Rate 2 if: a
(3) Rate 3 if: a+b
(4) Rate 4 if: a+b+c
...........
109 PYC 2616/1

...........
110

RESOURCES
Content domain
Correlation coefficient
Item analysis
Item format
Layout of a questionnaire
Manual: purpose and structure
Reliability
Specification document
Suitability of a questionnaire
Validity
Writing questionnaire items
...........
111 PYC 2616/1

Content domain
HOW TO IDENTIFY THE CONTENT DOMAIN FOR A QUESTIONNAIRE

“What is a questionnaire?” A questionnaire is not some sort of official form
or a set of randomly collected questions. The function of the questionnaire is
measurement. It is an important tool for data collection. A questionnaire is a
set of questions drawn up with the aim of obtaining information on a specific
topic. It is not used to answer research questions that aim to investigate the
relation between different topics, such as the relation between personality and
anxiety. You could have a questionnaire on one topic, for example, people’s
anxiety levels, and another questionnaire that measures personality, but if you
wanted to examine the relation between these two things, you would need to
consider what kind of research design to use.
1. Identifying the focus of a questionnaire

Before you can draw up a questionnaire, you need to have a clear idea of
the purpose of the questionnaire. The purpose of the questionnaire refers to
what it intends to measure and for whom it will be used. The people who
will be asked to complete the questionnaire comprise the target population
(in other words, the group of people at whom the questionnaire is directed).
The purpose of the questionnaire will in turn give you ideas about the content
domain, that is, all the possible tasks, behaviours, attitudes, etcetera implied
by the purpose. It is from this content domain that you select items to make
up the questionnaire.
The first step is to identify the general topic of interest. For example, you may
be interested in the interface between people and technology, or in the topic of
crime, or people’s shopping habits. These are very broad topics, so you need to
narrow them down further before you can investigate them properly. The next
step is to select a problem area within that topic or one aspect of the topic that
you want to investigate. For example, if you are interested in the topic of crime,
you should select one aspect of crime, such as the impact of crime on people’s
everyday lives – that will give some focus to your investigation. However, the
problem area is still fairly broad, so now you need to reduce the general problem
to more specific questions. To continue with the example of crime, you could
reduce the general problem (crime) to more specific questions like: “What are
people’s perceptions of crime in South Africa?”, “How do people cope with
the level of crime in society?”, “Do South Africans experience crime-related
stress?” or “What are people’s perceptions of the criminal justice system?” All
these questions are relevant to the problem area, namely the impact of crime
on people’s lives, and relate to the broader topic of crime. Having identified
these questions, you have established the focus of your questionnaire. The
content domain therefore consists of the tasks, behaviours, attitudes, etcetera
related to one or more of these questions.
...........
112

2. Limiting the scope of the questionnaire

Once you have identified the focus and the related content domain for your
questionnaire, there are literally thousands of questions that you could ask.
Some of these questions will be more relevant than others. Your questionnaire
will serve no purpose if the questions are not directly related to the focus or
purpose it is designed for. Therefore, the next step is to limit the scope of the
questionnaire. (The word “scope” means the extent or range of observation
or action.) This means that you have to decide on what is relevant for your
questionnaire.
The guidelines are that you have to ask (1) enough questions to cover your
content domain adequately but (2) only questions that are directly relevant to
your content domain. Let us look at an example by taking one of the questions
posed in the previous section: “Do South Africans experience crime-related
stress?” This suggests that your questionnaire should include items or questions
dealing with how people feel about crime, whether the crime situation is
stressful for them and so on. Even though the question “Do South Africans
experience crime-related stress” is fairly specific, there are still many other
aspects that relate to it. For example, the experience of crime-related stress
may vary according to people’s personalities or mental health, or the kind of
news reporting they are exposed to, or their cultural belief systems. They might
also be experiencing stress that is caused by other factors apart from crime.
You cannot possibly ask questions that cover all these aspects adequately
and still have a useful questionnaire (see also “specification document for a
questionnaire”). So your task is to limit the scope of your questionnaire, that
is, decide what is directly relevant to the focus of your questionnaire. If the
questions do not seem relevant, your respondents might not want to complete
your questionnaire.
Here are some ideas about what you might include in your questionnaire
to investigate whether South Africans experience crime-related stress. For
example, you could ask:
• the respondents’ age, gender and socio-economic status to see if various

sections of the population experience crime differently
• whether they have had personal experience of crime
• for ratings (see “item format”) on the level of stress associated with different
crimes
• open-ended questions about descriptions of their reactions to various crimes
• how crime affects their daily lives
Once you have made a list of the aspects you want to cover, look at each
one and ask yourself: “Is this directly relevant” and “why?”. This forces you to
have a good reason why each one is included in the questionnaire. By limiting
the scope in this way, you can cover your topic adequately but still have a
questionnaire that is relevant and not too long.
...........
113 PYC 2616/1

Correlation coefficient
HOW TO USE A CORRELATION COEFFICIENT IN ITEM ANALYSIS

In psychological research, one is often interested in the possible relation
between two psychological constructs. A construct is an unobservable quality
that forms part of a theory designed to explain observable behaviour. Examples
of psychological constructs are anxiety, intelligence, stress and independence.
Researchers design and develop questionnaires to measure these constructs
and then calculate a statistic known as a correlation coefficient to tell them
more about the relation between the constructs. Here we will explain what
is meant by a relation between two constructs and we will show how this
relation can be visualised.
1. The correlation coefficient

If there is a relationship between two constructs (for example, between
emotional sensitivity and anxiety), it means that a person’s relative position
(compared to other people) on one construct bears a relation to his or her
relative position on the other construct. A positive relationship implies that
relatively high scores on one construct are associated with relatively high scores
on the other and relatively low scores on the first correspond with relatively
low scores on the second. This implies that as scores on the one construct
increase, there are increases in the scores on the other construct. A negative
relationship on the other hand implies that high scores on one construct are
associated with low scores on the other construct. If the constructs are not
related, changes on the one do not correspond with changes on the other.
The statistical relationship between two constructs is called a correlation and

the statistic used to describe this is called a correlation coefficient. (The formulas
to calculate the correlation coefficient are not covered here). The correlation
coefficient can range in value from –1,00 to +1,00. These values represent
a perfect negative (–1) and a perfect positive correlation (+1). A value close
to 0 indicates a weak relationship while 0 represents no correlation. You can
see that the numerical size of a correlation coefficient indicates the strength
of the relationship while the sign (positive/negative) indicates the direction of
the relationship.
2. The scatterplot
If you visualise the relation between two constructs, you get a clear indication of
the strength and direction of this relation. The graphic display of the correlation
coefficient is called a scatterplot. The graph consists of a horizontal line
(called a horizontal axis) and a vertical line (or vertical axis) that meet at an
angle of 90 degrees. Possible scores on the one construct appear on the one
axis and possible scores of the second construct on the other axis. Note that
...........
114

these scores are all the possible scores a person can obtain on a questionnaire
measuring this construct. For each individual you take his or her score on the
one construct and his or her score on the other construct and where the two
meet you make a dot. Because you have a group of people, you will have a
number of scattered dots (each representing a person) and therefore the name
scatterplot. If there is a perfect positive relation between two constructs (a
correlation coefficient of +1), the dots form a perfectly straight line with an
upward slope. This is shown in scatterplot a in figure 1. For a perfect negative
relation (a correlation coefficient of –1), the scores form a perfectly straight line
with a downward slope (see scatterplot b in figure 1). No relation (a correlation
coefficient of 0) between two constructs results in an undefined shape (see
scatterplot c in figure 1).
Scatterplot a
Scatterplot b
...........
115 PYC 2616/1
R E S O U R CE S
Scatterplot c
Figure 1: Scatterplots for different degrees of correlations
Exercise: If you administer two questionnaires, one measuring emotional

sensitivity and one measuring anxiety, to eight students in a study group you will
have two sets of eight scores (a score on each questionnaire for each person). The
scores for both questionnaires range from 10 to 40. Suppose the scores obtained
by the eight students for emotional sensitivity are: 15 18 26 30 19 25 28 35;
and the scores for anxiety are: 18 18 24 32 18 30 24 35. We have plotted the
dots for the first four persons: the dot for the first person is where 15 and 18
meet, etcetera. Plot the rest of the scatterplot and comment on the relationship.
Figure 2: Scatterplot of the relation between emotional sensitivity and anxiety
Comments on the exercise: The scatterplot has been completed in figure 3.

To determine the strength and direction of the relation you can draw a straight
line resembling the shape of the scatterplot. This line will have an upward slope
indicating a positive relation. Not all the dots will fall exactly on the line but
most of them will be fairly close. Therefore, although this is not a perfect relation
of +1, it seems like a fairly strong relation (possibly between 0.70 and 0.80).
...........
116
R e s o u rce s
Figure 3: Linear graph of the relation between emotional sensitivity and anxiety
3. Using correlations in item analysis

Just as we can correlate the scores a sample of people got on one questionnaire
with their scores on another questionnaire, we can also correlate their scores
on one item of a questionnaire with their score on another item. Another
possibility (and this is often used in “item analysis”) is to correlate their scores
on any particular item with their total score on a scale consisting of several
items. If the item correlates strongly with the total score, we know that it
measures more or less the same thing as the other items.
...........
117 PYC 2616/1
R E S O U R CE S
Item analysis
HOW TO DISTINGUISH BETWEEN GOOD AND BAD ITEMS

The term “item analysis” refers to a set of procedures to select the best items
for inclusion in a measuring instrument (or, conversely, to identify the worst
items to discard from an instrument). The two most commonly used criteria
used for identifying which items to keep and which to discard are item
difficulty (sometimes also referred to as item facility or item variance) and
item discrimination.
1. Item difficulty/variance
Imagine a questionnaire where everybody always gets all the right answers
or, a questionnaire where nobody ever gets any right answer. It is clear that
such a questionnaire is of no use in showing up differences between people.
Starting from this extreme example, test constructors have argued that the
ideal questionnaire would be one where about half the people who do
the questionnaire gets each of the items on the questionnaire right. Such a
questionnaire would be ideal for identifying differences between people. Item
analysis therefore involves discarding items that are too easy (almost everybody
gets them right) or too difficult (almost nobody gets them right) and keeping
items that are of medium difficulty. The difficulty index for an item is usually
calculated by dividing the number of people who gave a correct response by
the total number of people in the sample. Ideally, the difficulty index for items
in a questionnaire should be between 0.25 and 0.75 (i.e. between 25% and
75% of people get the item right) and the average difficulty index for items in
a questionnaire should be about 0.5.
When dealing with instruments where there are no right or wrong answers
– such as personality questionnaires and attitude scales – we cannot use the
concept of item difficulty in the same way, but the same principle applies.
We do not want items where everybody answers in the same way as they do
not help us to see differences between people. In this module, we will refer
to this kind of item analysis simply as item variance.
2. Item discrimination
This refers to the ability of an item to discriminate between respondents
according to whatever the measuring instrument as a whole is measuring. Items
should only be selected for the final version of the instrument if they measure
the same characteristics as the other items in the instrument – else, they cause
the instrument to lose focus so that it becomes difficult to know what it might
be measuring. Discrimination is measured by correlating each item in a scale
with the total score for the scale (see “correlation coefficient”). The higher
the correlation coefficient, the more discriminating the item. A minimum
...........
118
R e s o u rce s
correlation of 0.2 is generally required, but this can vary depending on what
kind of construct is being measured. Items with negative or zero correlations
are almost always excluded. A negative correlation could be indicative that an
item should have been reverse-scored, and before discarding such an item, it
is important to check if this is the case.
3. How many items to exclude

It is usual to discard 20% to 30% of the items from a scale as a result of an
item analysis, but this can vary greatly depending on how many “superfluous”
items are available. You could for example specify that you want a final scale
consisting of eight items – but this is a purely arbitrary choice and it is possible
to have longer or shorter scales. If one measures a very clearly defined, simple
construct, it is possible to have a scale consisting of no more than a handful
of items, but if the construct being measured is a little less clearly defined
or complex, then more items will be required.
4. Other forms of item analysis

In addition to the two most commonly used forms of item analysis (item
difficulty/variance and discrimination), there are dozens of other item analytic
techniques that are used for different purposes. Notable among these are a
range of item bias statistics, which help test constructors to identify items that
perform differently (are biased) for different groups – for example items that
unfairly discriminate against certain race or gender groups.
...........
119 PYC 2616/1
R E S O U R CE S
Item format
HOW TO IDENTIFY DIFFERENT ITEM FORMATS
1. Closed questions
Generally speaking, most questions are either “open” or “closed”. A closed
question is one that offers respondents a limited choice of alternate replies
whereas an open question allows the respondents to answer in any way
they want to. Examples of closed questions would be those that offer simple
choices, such as yes or no, or a range of alternatives, such as five different
brand names. They are considered “closed” because the respondents do not
have the freedom to answer any way they want to.
Closed questions of the yes/no type merely require respondents to tick off the
answer “yes” or “no”, according to whether they agree with the statement or
not. Here is an example of a closed question of the yes/no type:
I will vote in the December elections Yes No
Closed questions, the true/false type, are similar to the yes/no questions and
are those that require a true/false answer. For example:
I believe in the death penalty. True False

All women are potential rape victims. True False
Closed questions, multiple-choice type – an example of a closed question

requiring the respondent to mark one of a limited selection of possible answers:
In the municipal elections, I will vote for ACDP o

ANC o
DA o
IFP o
Other o
This item, which is a form of closed question, is also known as a multiple-

choice question (the respondent chooses from a number of given alternatives).
Rating scales are generally used to measure attitudes, opinions or feelings and
in a sense, these can be regarded as having a closed-item format because the
respondent has a limited range of options (ratings) to choose from.
1.1 Inventories and checklists

An inventory or checklist is essentially a list of statements or items where
people are asked to mark or tick the items that apply to them. It is also a
...........
120
R e s o u rce s
form of closed question. This is a very simple kind of question that almost all
respondents should be able to deal with successfully. You could use this kind
of question when you want to obtain straightforward information, for example,
to find out what kind of hobbies people have. If you asked the respondents to
list their hobbies, you might get thousands of different ones. It may be easier
to give a list of hobbies and ask them to mark the ones that apply to them.
Here is an example:
Please tick your free-time activities from the list below:
sport
reading
cooking
watching television
stamp collecting
gardening
creative writing
art
other
You would not use this type of question format if you wanted to measure
attitudes or opinions, or the strength of people’s feelings about an issue. It is
merely a checklist, used in the same way that you would use a checklist to
make sure that you have packed everything you need for your holiday.
1.2 Advantages and disadvantages of closed questions

An advantage of closed questions is that the set of alternative answers is
uniform and therefore makes it easier to compare people’s answers. It is also
easier to answer closed questions than to construct your own response. In
addition, closed questions are quicker to answer than open questions. Sensitive
issues are often better addressed with closed questions that provide implicitly
acceptable alternative answers. A person is more likely to respond to this kind of
question than to provide a subjective answer on a sensitive topic. For example,
respondents may be uncomfortable answering a question about abortion. If the
question offers a choice of answers, ranging from those that accept abortion
to those that oppose it, it would be easier to answer the question because the
choices indicate that each one potentially represents someone’s view. Therefore,
by implication, any one of the answers is acceptable. However, it may be more
difficult for people to write down a controversial answer themselves.
Closed questions do have other disadvantages. The main disadvantage of

closed questions is that they force the respondent to answer in terms of the
alternatives offered and nothing else. This may lead to a loss of spontaneity.
There may also be some loss of rapport if the respondents become irritated
because they feel that the choice of answers does not include one that allows
them to answer the way they want to. It is sometimes a good idea to offer an
additional option such as “other” to allow the respondent to give an accurate
response in cases where none of the alternatives applies to that particular
individual. It should be remembered that closed questions can actually direct
the respondents’ thinking along particular lines and in this way may influence
...........
121 PYC 2616/1
R E S O U R CE S
their answers to questions that follow (and this introduces bias). In addition,
the respondent who may be unsure of the best answer or misunderstands the
question, may select one of the alternatives randomly, without giving it much
thought.
2. Open questions
Open or free-response questions are not followed by any kind of choice and
the respondents provide their own answers that have to be recorded in full.
An example of an open-ended question is “What do you think of the parole

system in South Africa?” It is important to phrase the question carefully if you
want more than just a yes or no answer. If you had asked, “Do you agree with
the parole system?” the respondent could say yes or no and provide nothing
more.
The main advantage of open questions is that respondents have the freedom to
express their ideas without the restrictions of set possible answers. Respondents
may have ideas and opinions that you have not thought of and these might be
lost if you ask a closed question. However, open-ended questions invariably
elicit some irrelevant and repetitious information that cannot be used and
wastes time in processing them. Answering open-ended questions also requires
a considerable degree of language proficiency and communication skills. For
this reason, they are not suitable for people with language difficulties or low
levels of literacy.
Although open-ended questions can give you a great deal of information, the
disadvantage is that they are difficult to analyse. If you want to compare the
responses of a large group of people, it may be better to use closed questions
for which all the respondents have the same choice of answers.
3. Rating scales
It is seldom useful to use single items or questions to measure complex or non-
factual topics such as opinions, beliefs, attitudes and values. These are complex
issues that have to do with states of mind rather than with behaviour or events
in the outside world and are therefore more difficult to measure. They are
usually multifaceted and have to be approached from different angles. Single
questions dealing with such topics are open to bias and unreliability due to the
way in which the questions are worded, the format of the questions and the
effect of the context regarding the way in which the questions are interpreted
and answered. Therefore, to measure non-factual topics, the tendency is to
use rating scales.
Rating scales here refer to multi-item scales, that is, a group of items dealing
with the same topic, each item requiring a rated response. For each item in
the rating scale, respondents have to indicate the extent to which they agree
or disagree with a statement by marking a point on a numerical scale. For
...........
122
R e s o u rce s
example, you could use a rating scale to investigate attitudes toward crime
in South Africa. Respondents would be required to rate their responses (by
ticking the appropriate box) to statements such as the following:
Statement Strongly Disagree Do not Agree a Strongly

disagree a little know little agree
Crime is disrupting our lives.
Police effectively curb crime.
Ratings give a numerical value to some kind of assessment or judgement.

We can apply ratings to anything, such as rating entrants in a competition,
preferences for certain objects or differentiating characteristics. For example,
you might be asked to rate the service provided by shop assistants on a scale of
1 to 5, where 1 = very bad service, 2 = poor service, 3 = adequate service,
4 = good service and 5 = excellent service. You could also rate them in terms
of friendliness, appearance, efficiency, etcetera. Ratings are given numbers
because it is easier to work with numbers than descriptions when you have
to analyse people’s responses. You can, for example, add a person’s ratings on
a scale and compare one person’s total with another, or work out an average
for a particular group.
Although ratings are an attempt to provide an objective measure of a person’s

attitude or opinion, the main danger with ratings is the ease with which ratings
can be influenced, often by factors of which the respondent is not aware.
Ratings may be influenced, for example, by a person’s mood on the day the
questionnaire is completed, or by political events in the country at the time.
(Note that this can also be the case with other types of items.)
The following guidelines can be followed when compiling a rating scale:
(1) Define the dimension being rated. This means that you have to decide
what it is that you want respondents to rate. Each item or statement to be
rated must refer to only one thing or dimension. For example, if you ask
respondents to “rate the shop assistant’s friendliness and efficiency” on a
scale of 1 (bad) to 5 (good), you are confusing two different dimensions:
friendliness and efficiency. The respondent might think that the shop
assistant was very efficient but not at all friendly. The respondent then
does not know which rating to use.
(2) Decide on the number of ratings for the scale. You may only need three
but there may be as many as 10. It depends on what is being rated. If
you only need respondents to indicate whether they agree, are neutral or
disagree, then you only need three ratings. If, however, you feel that there
may be a greater range of opinions, then you must provide more options
(a range of five or more). For example, look at the following statements
regarding people’s reading habits:
Do you read books? Yes Sometimes No
...........
123 PYC 2616/1
R E S O U R CE S
1 2 3 4 5
How often do you read books? Never One One a One a More than
in six month week one a
months week
These two ratings give you different information. The first one gives
you an indication of whether respondents read books or not, which is
fine if that is all you want to know. It does not tell you much about the
respondents’ reading habits. The second one gives you a much better
idea of their book reading habits. From this, you can see that the number
of ratings depends on what you want to find out.
(3) Decide whether to use an even or uneven number of ratings. Many

researchers prefer an uneven number in order to have a neutral category
in the middle but the problem is that people may tend to choose the
neutral one (this is called the error of central tendency).
(4) Define the different rating categories. You must specify criteria for each
rating so that they are mutually exclusive. This means that each rating
category should mean something different so that the respondents do
not have the problem of deciding which rating category their responses
fit into. For example, it might be confusing if your rating scale has the
following options:
Agree a Agree Agree a Neutral Disagree Disagree Disagree

lot somewhat little a little somewhat a lot
There is nothing to indicate what the difference is between “agree some-

what” and “agree a little”. Here is another example: you might want to
know how many times a week a respondent does physical exercise and
you provide the following rating categories: 1 = none, 2 = two to three
times, 3 = three or more times. If the respondent exercises three times
a week, what rating will he or she choose: a 2 or a 3? Each category or
rating must specify a particular kind of response that is not described by
any other category or rating.
Attitude scales are rating scales that consist of a group of items designed to
reflect different attitudes toward the topic in question. An attitude scale is
a technique for placing people on a continuum in relation to each other, in
relative and not in absolute terms. These scales are not designed to yield subtle
insights into individual situations and should not be used for clinical purposes.
Their main function is to classify people with respect to a certain attitude. This
allows the researcher to analyse the ways in which that attitude relates to other
variables in the survey. For example, you might find that different age or socio-
economic groups have different totals on a scale measuring attitudes to crime.
There are many different types of rating scales used to measure attitudes. We
will only discuss two here, namely Likert scales and the semantic differential.
...........
124
R e s o u rce s
3.1 Likert scales

One form of attitude scaling is the Likert scale, also known as a summated scale.
Huysamen (1996:124- 125) writes as follows: “A summated attitude scale may
be described as a rating scale in which a subject indicates the extent to which
he or she agrees (or disagrees) with statements. These statements usually deal
with a particular social or political issue or institution, such as communism,
abortion, a particular political party, the Truth and Reconciliation Commission,
etcetera. Usually only the end-points of such scales are labelled. The one on
the left may be labelled ‘disagree entirely’ and the one on the right ‘agree
entirely’. Subjects mark the point that reflects the extent to which they agree
or disagree with the statement involved.”
Here is an example of some items on a Likert scale. In this case, attitudes

towards a particular political party (XX) are being investigated.
1 2 3 4 5
Statement Disagree Agree
entirely entirely
Party XX has saved our
country from disaster.
Party XX is to blame for the
problems in the country.
I identify with party XX.
The respondent marks the point that best reflects his or her attitude. The
scores for each item or statement are then added up to obtain a total score for
the scale (and this is why it is referred to as a summated scale). When using
a summated scale, it is important to ensure that the scale is uni-dimensional,
that is, all the items measure the same dimension or topic. In this example, all
the items measure attitudes toward political party XX.
You will note that the statements in this example have been phrased in both
favourable and unfavourable terms. It is important to have both favourable
and unfavourable statements so that you do not influence the respondent in
any way. For example, if all your statements were phrased in a favourable
way, the respondents may tend to answer in a favourable light even though
they may think differently.
Likert scales usually have the option of five or seven ratings. The advantage is
that a number of ratings offer greater options than just a yes/no answer. There
are possible disadvantages to using an uneven number because people may
tend to choose the midpoint and then you do not know if the person is neutral,
lukewarm, lacks knowledge (about the content domain) or lacks an attitude
toward the issue in question.
...........
125 PYC 2616/1
R E S O U R CE S
3.2 Semantic differential

The semantic differential is a type of rating scale that is used particularly in
the measurement of attitudes. A semantic differential consists of a collection
of seven-point rating scales and the scale points on each end are defined by
opposing adjectives (that is, the items are bipolar). An example of a semantic
differential is given below:
Powerful _ _ _ _ _ _ _ Powerless
Hostile _ _ _ _ _ _ _ Friendly
Dynamic _ _ _ _ _ _ _ Static
Poor _ _ _ _ _ _ _ Rich
The respondent marks the rating that expresses his or her attitude. Using the
example of political party XX again, if the respondent thinks the party is very
powerful, he or she would place a mark on the first or second space closest
to “powerful”. If he or she thinks the party is powerful but not very much so,
then he or she is more likely to choose a space in the middle. If the respondent
thinks the party is really powerless, he or she would mark the space closest
to “powerless”, and so on. In this way, a picture can be obtained of people’s
attitudes toward the particular political party in terms of various descriptors.
When you compile a semantic differential scale, do not always put the positive
extreme on one side. The location of positive and negative poles should be
random to counteract any halo effect. The halo effect refers to the tendency
for respondents not to evaluate each item individually but for their responses
to be influenced by their general feeling of like or dislike. For example, having
decided on the direction of their attitude, people may tick down the one
side rather than reading each item if the items are arranged with favourable
responses on one side and unfavourable responses on the other.
It is also important that your two descriptors define the same construct. For
example, if you want to measure the construct of mood, your descriptors
might be happy/sad or stable/unstable. However, if your descriptors are happy/
reserved, it is not clear whether you are measuring the construct of mood
(happiness) or the ability to interact easily with others (reserved). In addition,
you should take care that the two descriptors really are opposites. For example,
sad/satisfied are not really opposites because you can feel satisfied about
something while still being sad.
The semantic differential is useful when you want to obtain an idea of people’s
endorsement of certain attributes. You could, for example, ask respondents
to rate the extent of their agreement that the organisation is powerful on a
scale of 1 to 5 but you would not get information about “powerful-ness” in
relation to the opposing attribute, that is, its “powerless-ness” unless you used
a semantic differential.
...........
126
R e s o u rce s
Layout of a questionnaire
HOW TO STRUCTURE A QUESTIONNAIRE
1. Introduction and covering letter

A well-designed questionnaire with a covering letter that has a professional
appearance is more likely to be completed than a shoddy, rushed document.
The covering letter should arouse the respondents’ interest and motivate
them to complete the questionnaire truthfully. A questionnaire should have an
introduction that informs respondents about the purpose of the questionnaire,
convinces them that their participation is valued, motivates them to complete
the questionnaire, reduces their fears regarding time and inconvenience, and
assures them of confidentiality and safety. The respondent must be made aware
that all answers are valuable (and in the case of a survey questionnaire, that
there are no correct or incorrect responses). It is important that the respondents
express their opinions and experiences from their own perspective. You do
not want the respondents to give answers that they think seem to be the
right thing to say or what the researcher expects. You want them to respond
according to their perceptions of their own reality (in other words, answer as
honestly as possible).
Here are some guidelines for an introduction to a questionnaire. Consider

including
(1) the name of the person or organisation conducting the study, to establish
credibility
(2) a general statement of the objectives of the questionnaire (people are
more likely to participate when they perceive that the study’s findings
will impact on them directly)
(3) assurance that their participation is valued and confidential (and in the
case of a survey questionnaire, that the answers are neither correct nor
incorrect)
(4) some estimate of the time required to complete the questionnaire
2. Confidentiality and anonymity

All survey data should be treated as confidential, in the sense that only
the researcher will have access to them. Anonymity is not the same as
confidentiality. A questionnaire is likely to elicit more accurate information
about the respondents’ feelings when it does not require personal identification.
Because it is impersonal, the respondents have greater freedom to express
themselves without fear that their responses would be used in a way that is
not in their interests. Information of a personal nature is more easily obtained
when the respondents know that they will remain anonymous. Anonymity
can be promised only when questionnaires are going to be identified by code
...........
127 PYC 2616/1
R E S O U R CE S
rather than by personal details. Anonymity is particularly important in surveys

that involve “sensitive” topics.
It is suggested that you take steps to ensure that no identifiable information will
be published without the person’s consent. The following may be displayed
prominently on the front of the questionnaire:
The contents of this questionnaire are absolutely confidential. Your identity

will not be disclosed under any circumstances.
3. Length of the questionnaire

The length of the questionnaire depends on the topic and the degree of interest
it holds for the respondent. As a rule, a questionnaire should be brief and to the
point while providing adequate coverage of the topic. People generally do not
bother to complete long and complex questionnaires. However, they may be
more inclined to complete a long questionnaire if they have a particular interest
in the topic, or believe that their responses will directly benefit them in some
way, or think that the investigation will influence policy that will affect them.
Ideally, a questionnaire should take approximately 30 minutes to complete.

There is no set number that is right for a questionnaire. The length depends on
the type of questions that are asked. For example, anyone is likely to get tired
of answering many open-ended questions that require a great deal of writing.
If a lot of writing is required, there should be fewer questions.
The optimal length of a questionnaire also depends on the characteristics of

the respondents. Specialists in a certain field are likely to be more willing
to complete a longer questionnaire on their subject than the average person. For
people with low levels of literacy or education, it is better to keep questionnaires
short because they are likely to take longer over each question than people
who are familiar with this type of task.
It is important to make sure that each question is directly relevant to the topic.
When you have drawn up your questionnaire, look at each question again
and ask yourself whether each one is necessary. If it seems too long, maybe
the same information can be obtained by combining some of the questions
or perhaps you need to be ruthless and just take out some of the questions, as
long as you are sure that you will still get the information you need.
However, shorter is not necessarily better. You need to have thorough coverage
of your topic in order to ensure “reliability” and “validity”. Sometimes it is
necessary to include two or more questions on the same topic to see if the
respondents are consistent in their answers. The aim is to strike a balance
between a concise questionnaire and one that is inclusive enough to ensure
validity.
...........
128
R e s o u rce s
4. Presentation and sequence of questions

The order in which questions are presented is important because it can affect
the overall study. For example, a poorly organised questionnaire can confuse
respondents, introduce bias and endanger the quality of the study. However,
there is no one correct order in which questions should be presented. There
are some general guidelines:
(1) Try to avoid putting ideas into the respondents’ minds or suggesting
preferable attitudes. You might therefore start with open questions and
then introduce more structured questions at a later stage. For example, if
you are investigating people’s perceptions of the environmental impact
of nuclear power stations, you might begin with an open-ended question
such as, “What role does nuclear power play in our lives?” and then further
on in the questionnaire ask a more structured question, such as “What
kind of impact does a nuclear power station have on the environment?”
(Tick the appropriate answer.)
no impact at all
very little impact
a huge impact
If you had asked the question about nuclear power and the environment
first, this may have stimulated a particular perception based on an emo-
tional response (such as fear), for example that nuclear power stations
have a negative impact on the environment, without really thinking the
question through. The negative perception might then have influenced
the way the person responded to subsequent questions. By first asking
about the role of nuclear power, you are helping the person to think ra-
tionally about nuclear power and in this way, it is less likely that a certain
perception will be adopted toward the more emotive issues.
(2) Work from broad questions at the beginning to more specific questions
toward the end. This is called the funnel approach. In this way, you place
specific questions within a general context, which increases the likelihood
of obtaining meaningful responses. For example, in a questionnaire
about crime you might start with general questions about perceptions
of and attitudes toward crime and lead up to questions about personal
experiences or specific topics such as criminal justice or support systems
for victims of crime.
(3) It is often tempting to put a set of rather forbidding questions at the

beginning of a questionnaire, such as those relating to classifying and
personal data such as age, address, marital status and income. Even
though these questions may be necessary, they tend to be very off-putting
to respondents. Once your respondents have had the purpose of the
questionnaire explained to them, they expect some interesting questions
on the topic and instead they are bombarded with questions about their
private life. Unless there are very good reasons not to do so, it is better to
put personal data questions near the end of the questionnaire, preceded
by a short explanation such as: “To help us classify your answers and to
make comparisons, would you please tell us ...where you live”, and so on.
...........
129 PYC 2616/1
R E S O U R CE S
(4) In your questionnaire, you probably have groups of questions relating to

particular aspects of your main topic. For example, if you are investigating
the topic of crime, you might have questions that relate to certain themes,
such as personal experiences of crime (one theme or group of questions),
attitudes toward punishment (another theme or group of questions), and
perceptions of the justice system (another theme or group of questions).
You then have to decide on the order in which to present these groups
of questions. There are two main considerations: one is the logic of the
survey and the other is the likely reactions of the respondents. In general,
the questionnaire should be interesting, sensible, and non-threatening.
It is usually a good idea to start with “awareness” questions relating to
the topic in general. For example, if you are doing a survey on reading
habits, you might start with questions designed to provide information
on what newspapers or magazines the respondent has heard of. These
might be followed by “factual” questions dealing with the respondents’
own actions or behaviour, such as what papers they read or how many
library books they borrow per month. At this stage, you hope that the
respondents are finding the questions interesting and not too difficult,
which promotes co-operation. Then you might include questions on likes
and dislikes, preferences and attitudes. This also means that the format of
the questionnaire has probably moved from simple question-and-answers
to checklists, rating scales or attitude statements.
(5) When you are dealing with sensitive or very personal issues, these
should come toward the end of the questionnaire. You want to avoid
embarrassing or offending the respondents – if they are offended by
questions at the beginning of the questionnaire, they might then be
unwilling to complete the rest of the questionnaire. A well-designed
questionnaire is likely to make the respondent feel more comfortable about
answering sensitive questions that appear later on in the questionnaire.
(6) It is often useful to place one or more open-ended questions at the end
of the questionnaire to allow the respondents to express opinions or
feelings that are related to the purpose of the questionnaire but have not
been covered by the questions. You may think that you have covered all
relevant aspects of the topic but because individuals are unique, there is
always the possibility that you have not asked a question that pertains to
someone. By giving them the opportunity to add their own thoughts, you
will get better coverage of the topic. Furthermore, the respondents are
more likely to feel satisfied that answering the questions was worth the
effort. You could say something like “If you have additional information or
opinions that you would like to express, please do so here” and provide
space for the respondents to write in their comments.
5. Balance of question types

You also need to consider including a balance of different types of questions,
so that the respondent does not get bored with answering too many questions
of one type. For example, are there too many answers that just require ticking
yes/no or are there too many open-ended questions that are time-consuming
...........
130
R e s o u rce s
and require a great deal of writing? The ideal is to vary the type of questions
so that the respondents do not get bored or irritated (which may affect the
validity of their responses).
6. Filter questions
If you have a group of related questions that apply to some people but not
to others, it is generally useful to start with a filter or screening question that
excludes some respondents from answering irrelevant questions. For example,
if you have a series of questions dealing with women’s magazines, it is likely
that some of your respondents do not buy women’s magazines and therefore
several questions will not apply to them and consequently there is no point
in answering further questions about them. You could begin by asking for
factual information by, for example, asking them to indicate on a checklist
what magazines they buy. A typical filter question would then be “Have you
bought a women’s magazine in the last three months?”. If the answer is no,
the respondent is asked to skip the next few questions that deal with women’s
magazines.
...........
131 PYC 2616/1
R E S O U R CE S
Manual: purpose and structure
HOW TO COMPILE THE MANUAL OF A QUESTIONNAIRE
1. Purpose of a manual
Someone else might be interested in using your test or questionnaire. We
will refer to this person as the user of the questionnaire. You need to write a
manual for your questionnaire so that this person can see if the questionnaire
is relevant for the specific purpose he or she wants to use it for. You should
clearly state the aim of the questionnaire and describe the target population.
Relevant information on the use of the questionnaire in a multicultural and
multilingual context should be given and the cultural and language group/s for
which the questionnaire is suitable should be specified. A brief description of
the design of the questionnaire helps the user to understand how this design
ensures that the questionnaire can be used for the intended purpose.
Once the user has decided that the questionnaire is relevant, he or she would
like to know how well this questionnaire would work for the intended purpose.
To enable users to evaluate the properties of the questionnaire, you should
report the process of analysing and selecting the items as well as the reliability
and validity of the questionnaire. If the questionnaire is to be used for more
than one cultural and/or language group, equivalence for these groups in terms
of reliability and validity should also be discussed. Based on this information,
the users can decide for themselves if the properties of the questionnaire are
acceptable or if there is another questionnaire that might provide better (i.e.,
more reliable and valid) results.
You should give instructions for the administration of the questionnaire to

prepare the user with regard to aspects such as the situation in which the
questionnaire should be administered, the material needed, how to deal with
questions, and how to explain the aim of the questionnaire to the respondents.
Lastly, the manual should contain instructions for the scoring of the questionnaire
and some guidelines on how to interpret the results. Can a total score be
obtained for the questionnaire or for sections of the questionnaire and how is
an individual’s score interpreted?
2. Structure of a manual
The following outline can be used to structure the manual:
Aim and design
Aim
Target population
Design of the questionnaire
...........
132
R e s o u rce s
Properties of the questionnaire
Item analysis and item selection

Reliability
Validity
Procedures for administration, scoring and interpretation
Instructions for administration

Instructions for scoring
Guidelines for interpretation
2.1 Aim and design

You should clearly state the aim of the questionnaire in the manual. In other
words, it must be clear what the questionnaire measures and how this information
can be used. For example, the aim of a questionnaire on food expenditure can
be described as: “This questionnaire has been developed to determine what
percentage of total spending is allocated to food. It can, for example, be used
to compare food expenditure in urban and rural areas”. Another example is
an attitude questionnaire on the death penalty: “This questionnaire has been
developed to measure people’s attitude towards the death penalty. It can be
used in policy-making in the justice system”.
The aim of the questionnaire determines for whom it will be used. You should
describe those characteristics of the target population that are relevant to the aim
of the questionnaire. Remember that the user should have enough information
to be able to decide if the questionnaire is relevant for his or her purpose. The
questionnaire on food expenditure would probably be targeted at a national
sample that includes different cultures and languages. It would be more suitable
for adults. In the case of the attitude questionnaire it would be important to state
for which country this questionnaire has been developed (different countries
have different systems of law). Another important characteristic would be age
as the subject matter of this questionnaire (the death penalty) makes it more
suitable for adults.
Once the aim has been stated and the target population has been defined, a
brief description of the design of the questionnaire should be provided. The
purpose of the questionnaire determines the content domain and it should
be described how the items of the questionnaire cover this domain. The type
of items should also be indicated, for example multiple-choice items on food
expenditure where the respondent selects the relevant response from a number
of alternatives. In the case of the attitude questionnaire, the respondent might
have to rate how much he or she agrees with a statement about the death
penalty.
2.2 Properties of the questionnaire

To determine how effective the questionnaire is, you need to administer it to
a group of people who are representative of the target population. This group
of people should be described and it should be indicated to what extent
...........
133 PYC 2616/1
R E S O U R CE S
they are representative in terms of those characteristics that define the target
population. It should also be mentioned when and under which circumstances
the questionnaire was administered to them.
You should briefly describe each technique used for item analysis and you
should indicate which criteria were used to justify the inclusion or exclusion
of items in the item selection process. For example, to determine if the items
in a questionnaire all measure the same thing, one will look at the relation
between each item and the total of all the items (see also “item analysis”). A
strong relation indicates that the specific item measures the same thing as the
other items in the questionnaire. If this is important, one may decide to include
only those items with a moderate to strong relation with the total.
It is furthermore important for the user to know how reliable or consistent the
questionnaire is for the people for whom it is to be used. You should give a
brief description of the method used to determine reliability and justify why
this type of reliability was used (as opposed to the other types of reliability).
The reliability coefficient is then evaluated in terms of what can be regarded
as an acceptable level of reliability given the purpose of the questionnaire.
The reliability should be acceptable for all the cultural and/or language groups
for whom the questionnaire will be used.
A questionnaire is valid for a specific purpose and you need to identify the
category of validity (be it content validity, criterion-related validity or construct
validity) that is relevant for your questionnaire. You must explain how the
evidence was gathered to determine the validity, and based on this evidence,
you need to discuss the extent to which the questionnaire measures what
it claims to measure. In other words, you must evaluate the validity of the
questionnaire. The questionnaire needs to be equivalent for all the cultural
and/or language groups for whom the questionnaire will be used. Prediction
bias refers to a lack of equivalence in terms of criterion-related validity and
construct bias refers to a lack of equivalence in terms of construct validity.
2.3 Procedures for administration, scoring and interpretation

You should provide general instructions for administration of the questionnaire
in the manual. The person who wants to use the questionnaire might want
information on the following aspects: Who is allowed to administer the
questionnaire; the situation in which the questionnaire should be administered
– for example if it is suitable for groups or for individuals, and if people
can complete the questionnaire on their own or if supervision is needed; the
material needed such as the questionnaire itself, a pen or pencil; and how
to deal with situations such as a person asking an explanation of an item or
question. Guidelines should also be given for an introductory talk in which
the purpose of the study and the aim of the questionnaire are explained and
the issue of confidentiality of the results is mentioned. The importance of the
study needs to be conveyed and any fears the respondents might have must
be addressed.
You should provide instructions for scoring so that a numerical value can be
given to the answers on those items where scoring is relevant. In the case of
...........
134
R e s o u rce s
items with a correct answer, each item could score a 1 or a 0. By adding up

the scores for all the items, one obtains a total score (called the raw score)
indicating how many items out of the total number of items a person had
correct. Rating scales are often used with attitude questionnaires. On a five-
point rating scale a 1 might indicate a respondent does not at all agree with
a statement and a 5 might indicate strong agreement with this statement. The
total score indicates the person’s attitude towards the topic under investigation.
Note that the statements are not all worded so that agreement indicates the
same type of attitude – that is, agreement for some items might indicate a
positive attitude and for other items a negative attitude. Reverse-scoring might
therefore be necessary before the total score is calculated.
The guidelines for interpretation of the results should be based on the aim
of the questionnaire. If a questionnaire (or part of a questionnaire) measures
how well someone has mastered a course on managing household finances,
then the total score indicates the individual’s knowledge of the course content.
Someone with a high score obviously knows more than someone with a low
score. If the statements in a rating scale all indicate problems with the death
penalty as means of punishment, then a high total score could be interpreted
as that the individual does not regard the death penalty as an acceptable means
of punishment. Because respondents can agree (or disagree) to a greater or
lesser extent with the statements on a rating scale, one can distinguish between
individuals in terms of their attitude. One could also interpret an individual’s
total score by comparing it to the average total score of a group. The average
total score of a group is calculated by adding up the total scores for all the
individuals in the group and dividing the sum by the number of individuals.
...........
135 PYC 2616/1
R E S O U R CE S
Reliability
HOW TO CONSIDER THE RELIABILITY OF A QUESTIONNAIRE

The reliability of a questionnaire refers to how consistently the questionnaire
measures that which it is supposed to measure.
1. Measurement error and reliability

If I use a measuring tape to measure your height today and I use the same tape
to measure it tomorrow, I would expect to get the same result on both occasions.
It is, however, not that simple in the social sciences where researchers deal
with more abstract attributes of people such as attitudes and emotions. There
are various conditions associated with the administration of a questionnaire
which are irrelevant to its intended purpose but which might affect the results
of the questionnaire. One such condition would be the occasion on which the
questionnaire is administered. A group of people who have just attended a
workshop might be tired and in more of a hurry to complete the questionnaire
than if they have completed it after a good night’s rest. Another example of
a condition that might affect the results of a questionnaire is the sample of
items in the questionnaire. The items you include in your questionnaire could
concentrate on material that has recently been covered in the news while other
items measuring the same attribute might be less familiar.
Not only are these conditions irrelevant to the purpose of the questionnaire,
they might also differ from one administration to the next. Their effect on the
results of the questionnaire is therefore unpredictable and inconsistent. These
irrelevant conditions are called unsystematic sources of variation. To vary
means to change. The reliability of a questionnaire refers to how consistently it
measures regardless of the occasion on which it is administered or the sample
of items included. It therefore refers to the consistency of results over different
administrations involving different occasions, questionnaire forms, etcetera.
A statistical index of reliability is the reliability coefficient. The values of this

index range between 0 and 1 and it is important that you know how to interpret
the value of the reliability coefficient. A completely unreliable questionnaire
(i.e., with a large measurement error) will have a reliability coefficient close to
0 while a completely reliable questionnaire (i.e., with no measurement error)
will have a reliability coefficient of 1. Therefore, the closer the value of the
reliability coefficient to 1, the more reliable the questionnaire. One way of
determining this reliability coefficient is to calculate a correlation coefficient
(see also “correlation coefficient”).
...........
136
R e s o u rce s
2. Different types of reliability
2.1 Test-retest reliability

Do you remember that different occasions of administration represent an
unsystematic source of variation? If you want to know how consistent the results
of a questionnaire are over different occasions, you will need to determine
the test-retest reliability of the questionnaire. The method for determining the
test-retest reliability is to administer the same questionnaire to the same group
of people (who are representative of the target population) on two consecutive
occasions. The two sets of scores obtained are correlated and the correlation
coefficient represents the degree of test-retest reliability. If the second score
strongly correlates with the first score, the correlation coefficient will be close
to 1. The closer the correlation coefficient (which in this context we can also
call a reliability coefficient) is to 1, the more consistent or stable the scores
are on the questionnaire over different occasions. Test-retest reliability thus
indicates stability or consistency of scores over time.
It is important to understand that a perfect correlation does not indicate that

the second scores were identical to the first scores. However, it does imply
the same ranking of scores from the first to the second administration; that is
a person’s relative position to that of the others in the group stays the same.
Depending on the attribute being measured, you might expect people to

score somewhat higher during the second administration because of practice,
maturation, schooling or other intervening effects. The time interval between the
two administrations of the questionnaire should be at least several days to reduce
the possibility of effects such as familiarity with the type of items or respondents
remembering what their answers were during the first administration. The
interval should, however, not exceed several weeks, since real changes might
then occur in the attribute being measured.
2.2 Alternate-forms reliability

Two forms of the same questionnaire are often developed. The alternate forms
are independently constructed to meet the same specifications, that is the
content and the properties of the items are similar (see “item analysis”). Despite
the similarities, different forms of a questionnaire (and thus different samples
of items) could represent an unsystematic source of variation. If you want to
know how consistent the results of a questionnaire are over different forms,
you will obtain an estimate of the alternate-forms reliability of each form of
the questionnaire (it is the same for each form). To determine alternate-forms
reliability, both forms of the questionnaire are administered to the same group
of people (who are representative of the target population) on two consecutive
occasions. The two sets of scores obtained are correlated and the correlation
coefficient represents the degree of alternate-forms reliability of either form of
the questionnaire. The closer the correlation coefficient (or reliability coefficient)
is to 1, the greater the extent to which the forms are indeed equivalent and
thus measure the same attribute. Alternate-forms reliability is thus a measure
of equivalence.
...........
137 PYC 2616/1
R E S O U R CE S
If several days have elapsed between the two administrations (of the two forms),
alternate-forms reliability is not only a measure of the consistency of results
over different forms but it can also be used to determine the consistency of
results over different occasions or stability over time. This offers some solution
to possible memory effects experienced with test-retest reliability.
Both test-retest reliability and alternate-forms reliability involve two

administrations to the same group of people with an intervening time interval.
One could expect some intervening changes as well as real changes in the
attribute being measured to influence the reliability estimates for both types
of reliability. The difference between the two types of reliability is that with
test-retest reliability identical items are used in both administrations. However,
it has been mentioned that item-sampling differences represent a possible
source of error in the case of alternate-forms reliability. Another disadvantage
of alternate-forms reliability is that the construction of parallel forms of a
questionnaire is expensive and time-consuming and that it is difficult to produce
truly parallel forms.
2.3 Split-half reliability

When calculating test-retest reliability and alternate-forms reliability, you
are interested in the consistency of results over two administrations of the
questionnaire. Obtaining a second set of scores from the same group of people
is sometimes impractical. There are methods for estimating reliability without
developing alternate forms and without administering the questionnaire twice
to the same group of people. In one such approach, a single questionnaire is
administered only once to a representative group of people. This questionnaire
is then divided into two parts in such a manner that they may be regarded
as two parallel halves of the questionnaire. Each person has a total score on
the one half and a total score on the second half – this provides you with two
sets of scores that are then correlated. The correlation coefficient or reliability
coefficient is an estimate of the reliability of either of the two halves.
How do you divide a questionnaire into two equivalent halves? A common

method used to divide a questionnaire is to compare scores on the odd items
with scores on the even items (e.g., compare the total score for items 1, 3, 5
and 7 with the total score for items 2, 4, 6 and 8).
You now have an estimate of the reliability of each half of the questionnaire
(it is the same for each half) and not of the whole questionnaire. This is
probably an underestimation of the reliability of the full questionnaire as a
shorter questionnaire is generally less reliable than a longer questionnaire. A
correction is made to obtain the reliability of the whole questionnaire. This is
called the split-half reliability of the questionnaire and it measures the degree
of equivalence between the two halves of the questionnaire, that is, the extent
to which they measure the same attribute. As such, it reflects the consistency
of the group’s performance on the items within the questionnaire and indicates
the degree of relatedness of the items. This type of reliability is therefore
regarded as a measure of the internal consistency of the questionnaire and
the closer the split-half reliability is to 1, the higher the internal consistency
of the questionnaire.
...........
138
R e s o u rce s
3. Evaluating reliability
It is important to note that it is not correct to refer only to the reliability of a
questionnaire. The questionnaire you are dealing with should be specified
together with the representative group for whom the estimate of reliability
has been determined. The nature and purpose of the questionnaire determine
which type of reliability is appropriate and the acceptable level of reliability
also depends on the use of the questionnaire. If you are working with a
psychological test of which the results could have important consequences
for an individual, such as an intelligence test, the reliability coefficient should
be above 0.90. However, a questionnaire with a reliability coefficient of 0.70
(and even somewhat lower) can be useful if the results are used in combination
with other information about the individual or group.
...........
139 PYC 2616/1
R E S O U R CE S
Specification document
HOW TO COMPILE A SPECIFICATION DOCUMENT FOR A QUESTIONNAIRE

How do you know what a questionnaire should contain? The answer to this
question can be found in a questionnaire specification document. A specification
document is really just a list of the required characteristics for your questionnaire.
You may be familiar with the notion of specifications in relation to computers
– most software packages give an indication of the hardware specifications
required before the programme will run on any particular computer. For
example, when you buy a computer game, it usually indicates what is required
in terms of the capacity of the computer, its speed, the size of the memory,
supporting software, the soundcard, etcetera. If your computer does not meet
these specifications, the game program will probably not run properly. When
you draw up the specifications for your questionnaire, you are in effect making
a list of what it should contain in terms of type of items, number of items,
layout and so on, in order for the questionnaire to do what it is supposed to
do (that is, collect information from the chosen content domain).
Before you start compiling a questionnaire, you should have a rough idea of
the line of enquiry you wish to follow. Do you need short, factual answers
or do you want to conduct analytical research on a set of attitudes? How
many people will be involved? Are they children or adults? If adults, are they
businesspersons or the unemployed? All these questions influence the kind
of questions you will ask, the level of language you use, how complex the
questions are and so on. In this way, the purpose of the investigation, the kind
of information you want and the characteristics of the respondents (that is, the
target population) all influence the way the questionnaire is compiled, that is,
they influence the questionnaire specifications. The detailed specification of
measurement aims should be clearly related to the purpose of the research.
...........
140
R e s o u rce s
Suitability of a questionnaire
HOW TO DETERMINE THE SUITABILITY OF A MEASURING INSTRUMENT

The main purposes of a questionnaire are to (1) obtain accurate factual
information, (2) provide a standard format for recording facts, comments
and attitudes and (3) facilitate data processing. Because the purpose of a
questionnaire is essentially fact-finding, questionnaires are not designed to
“explain” anything or show causal relationships.
There are many different instruments and techniques for gathering information
from people. You can use questionnaires, standardised ability and achievement
tests, projective tests, interviews, observation, narrative discourse and a range
of different methods. The measuring instrument and approach you use depend
on the topic you have chosen and the purpose of your investigation.
Questionnaires can be used for a variety of purposes and every questionnaire

differs according to that purpose. As a research tool, the questionnaire is ideal for
collecting opinions, preferences and facts for a specific purpose from a defined
set of respondents (respondents are the people who are going to respond to
or answer the questions). For example, you would use a questionnaire if you
wanted to find out how many people use a certain product (like a brand of
washing powder), or what kind of car they prefer, or their opinions about the
death penalty. This kind of questionnaire is called a survey questionnaire.
An example of a typical use of a questionnaire is a population census or an
opinion poll.
Educational and psychological questionnaires measure knowledge, interests,

and other constructs. Some personality tests (like the 16 Personality Factor
questionnaire) are entirely made up of rating scales. A survey-type of
questionnaire is of limited use if you want to test a person’s ability in a particular
field. For example, if you want to see how good someone is at mathematics,
you would use a mathematical ability test designed specifically for that purpose,
rather than a questionnaire. Similarly, if you want respondents to reveal hidden
aspects of their personality or emotional functioning, you would choose a
projective test rather than a questionnaire.
Exercise
In the following exercise, you are given a list of topics for investigation (left
column). In the middle column, indicate whether you think a questionnaire
would be a suitable measuring instrument and in the last column, give a reason/
reasons for your answer. Sometimes you just have to rely on your common sense
to decide whether you can get the information you want with a questionnaire
or not but you should always be able to give a reason for your decision.
...........
141 PYC 2616/1
R E S O U R CE S
TOPIC YES OR NO REASON

1. Support for political parties
2. Preference for different types of
beer
3. Typing skills
4. Opinions about the parole system
5. Parenting practices
6. Effect of personality on
intelligence
Comments on exercise
(1) Yes.
You want to find out facts. Your questionnaire could include questions
about the parties’ respondents support, or ratings of various aspects of
the party and their policies, etcetera.
(2) Yes.
Again, you are looking for factual information. You might have a list of
the different beers and their qualities and ask people to indicate what
they like and why.
(3) No.
Here you are looking at a practical ability and in order to assess typing
skills, you would have to administer a practical test of typing ability. You
could use a questionnaire if you wanted facts about typing qualifications,
work experience, etcetera.
(4) Yes.
Questionnaires are frequently used to gather information about people’s
opinions on various issues.
(5) Yes.
This is a slightly tricky one. You can use a questionnaire to find out
about parenting practices if you ask, for example, about attitudes toward
punishment, about routines in the home, food provided and stimulation
available. On the other hand, if you wanted to examine the effectiveness
of certain parenting practices, you would use observation rather than a
questionnaire.
(6) Yes and No.

This is another tricky one. You could use a questionnaire to measure
aspects of personality (for example, ratings on certain dimensions of
personality) but you would need a separate test to measure intelligence,
and if you are interested in the relation between personality and intel-
ligence, you would need the right kind of research design to see how
people’s scores on the personality questionnaire relate to their scores on
the intelligence test.
...........
142
R e s o u rce s
Validity
HOW TO CONSIDER THE VALIDITY OF A QUESTIONNAIRE

The validity of a questionnaire refers to the extent to which it measures
what it claims to measure. It refers to the extent to which the scores and the
conclusions based on these scores can be used for the intended purpose of
the questionnaire. The validity of a questionnaire is therefore evaluated for a
particular application. If a questionnaire is used for more than one purpose,
evidence should be gathered of the validity for each purpose. This usually
requires numerous studies of the relationships between performance on the
questionnaire and other independently observed behaviours. There are three
categories of gathering validity evidence, namely content validity, criterion-
related validity and construct validity.
1. Content validity
The purpose of the questionnaire refers to what it intends to measure and for
whom it is to be used. This in turn determines the content domain the researcher
needs to cover. If the purpose of the questionnaire is to measure achievement
in a specific course (that is, how well a student has mastered all the objectives
of the course), the content domain is the course material and related tasks
the student should be able to perform. The purpose of a questionnaire could
also be to measure behaviour in a particular context and the content domain
would then be different possible behaviours in that context. For example, if
you want to measure behaviour in a crowd situation, the content domain you
need to cover includes different behaviours such as losing one’s inhibitions
or on the other hand becoming withdrawn.
The content domain is the universe of tasks, behaviours, attitudes, etcetera

implied by the purpose of the questionnaire (see “content domain”). The
researcher only includes a relatively small number of these tasks, behaviours
or attitudes in the questionnaire and should therefore try to select tasks,
behaviours or attitudes that are representative of the larger universe of possible
tasks, behaviours or attitudes. The content validity of the questionnaire for
its particular purpose is determined by the degree to which the items in the
questionnaire are representative of the universe of tasks, behaviours or attitudes
that it was designed to measure.
Content validity can to a great extent be guaranteed by the proper design

of the questionnaire. The relevant universe needs to be defined clearly. This
definition should be explicit enough to enable experts in the particular area
to agree on whether any particular task, behaviour or attitude forms part of
the universe. Items are chosen from this universe and the content validity of
the questionnaire is evaluated by experts in the field who must decide if the
chosen items adequately cover the universe of tasks, behaviours or attitudes
...........
143 PYC 2616/1
R E S O U R CE S
as defined by the developer of the questionnaire and whether the items

indeed measure these tasks, behaviours or attitudes. Content validity cannot
be expressed in terms of a quantitative index.
Content validity should not be confused with what is called face validity. Face
validity refers to the degree to which items in a questionnaire appear to be
relevant to what is being measured. Face validity is based on the subjective
evaluation by people who are not necessarily experts either in the particular
area or in psychometrics. If the respondents do not regard the items as relevant
(the questionnaire does not have sufficient face validity), they might be less
motivated and even unwilling to cooperate.
2. Criterion-related validity
Tests and questionnaires are often used to estimate an individual’s position or
performance on some outcome measure. This outcome measure is referred
to as the criterion. The criterion-related validity of a questionnaire is the
extent to which the scores on the questionnaire are effective in estimating an
individual’s position or performance on the relevant criterion. Two different
approaches to gathering evidence of criterion-related validity are concurrent
validity and predictive validity.
With concurrent validity, measures are obtained on the criterion at approximately

the same time as the scores on the questionnaire. The extent to which scores
on the questionnaire accurately estimates an individual’s present position
on the relevant criterion is then determined. This type of validity should be
determined if you want to use your questionnaire to identify some current
behaviour or status of individuals. For example, suppose your questionnaire
aims to measure the level of organisational commitment of employees at a
higher education institution. You could take a representative group of employees
and administer your questionnaire to them. At the same time, you would
ask their line managers to rate these employees in terms of their level of
organisational commitment (the criterion). You will then determine if scores
on your questionnaire distinguish between employees who differ in terms of
their commitment as rated by the line managers.
To evaluate predictive validity, the measures on the criterion are obtained in

the future, usually months or even years after scores on the questionnaire are
obtained. It is then determined to which extent the scores on the questionnaire
accurately predict an individual’s scores on the relevant criterion. This type of
validity should be determined if you want to use your questionnaire to predict
some future performance of individuals. For example, if your questionnaire is
on a particular university course and you want to use it to select candidates
for entrance into this course that are most likely to succeed, then you could
take a representative group of students who are applying for the course and
administer your questionnaire to them. At the end of the course, you could
obtain the students’ examination marks as measure of the criterion, which is
academic success. You will then determine how effective the scores on your
questionnaire are in predicting the students’ examination marks.
...........
144
R e s o u rce s
The most popular method to determine criterion-related validity is to calculate

the correlation between the results of the questionnaire and the measures on the
criterion (see also “correlation coefficient”). The resulting correlation coefficient
is known as the validity coefficient. The higher the validity coefficient, the
better the criterion-related validity of the questionnaire. Validity coefficients are
usually in the low to middle range of correlations and your evaluation of the
coefficient obtained for your questionnaire will depend on how you are going
to use your questionnaire. It is important to remember that the measure of the
criterion should not only be appropriate in terms of what your questionnaire
aims to measure, but it should also be reliable and valid.
3. Construct validity
A construct is an unobservable quality which forms part of a theory designed
to explain observable behaviour. For example, anxiety (a construct) is not
observable but it forms part of a theory that explains observable behaviours
such as nail biting, sleep disturbances and eating disorders. Constructs influence
behaviour and because you cannot directly observe a construct, you infer it
from the behaviour associated with that construct. If you see someone showing
the above behaviours, you may infer that this person has some anxiety.
If you design a questionnaire to measure a construct, you have to define your

construct in terms of observable behaviours. You then need to include some of
these behaviours in your questionnaire from which you can infer an estimate
of the existence of the underlying construct. Construct validity refers to the
appropriateness of these inferences about the underlying construct. You can
thus define the construct validity of a questionnaire as the extent to which
it indeed measures the theoretical construct it aims to measure. Determining
construct validity involves the findings of many studies and construct validity
cannot be expressed in terms of a single validity coefficient. There are diverse
procedures for determining construct validity, two of which will be explained.
Based on the relevant theory, you would expect groups who are supposed
to differ in terms of a construct to also obtain significantly different scores
on a questionnaire measuring this construct. If your questionnaire measures
the degree of warmth someone shows towards others (interest in people,
outgoingness, etc), then you could administer the questionnaire to a group of
social workers and to a group of research scientists as part of the validation
process. You would expect the social workers to obtain a significantly higher
average score on the questionnaire indicating that they possess a higher degree
of the construct. Such a result would provide support for the construct validity
of the questionnaire and for the theory underlying the construct.
Another way to determine construct validity is to look at the correlation

coefficients between different questionnaires. You know that a correlation
between two constructs indicates a relationship between them, that is, changes
on the one correspond with changes on the other (see also “correlation
coefficient”). If two questionnaires measure the same construct you would
expect the scores on these questionnaires to be significantly correlated. This
is referred to as convergent validity. If on the other hand two constructs were
theoretically unrelated, you would not expect a high correlation between the
...........
145 PYC 2616/1
R E S O U R CE S
scores on questionnaires measuring these constructs. This is called discriminant

validity. An example would be a questionnaire measuring sociability. One
would expect a fairly high correlation between sociability and community
involvement (a more sociable person would probably become more involved
in the community). However, one would not expect a significant correlation
between mechanical interest and sociability as sociability and mechanical
interest are theoretically unrelated.
...........
146
R e s o u rce s
Writing questionnaire items
HOW TO WRITE ITEMS FOR A QUESTIONNAIRE
1. General principles guiding the construction of good items

When constructing items, you should bear the following in mind:
(1) The items should be based on a meaningful definition or description of

what you want to measure.
(2) Constructing items is both a science and an art. It is a science in that
it requires an in-depth knowledge of one’s topic and familiarity with
the principles governing good item design. It is an art in that it requires
creativity in selecting or constructing items appropriate to the particular
context.
(3) The items should be aimed at obtaining meaningful information about the
behaviour, experiences, ideas, attitudes or perceptions of the respondents
– with a minimum of distortion.
(4) Constructing items can be an enjoyable, interesting yet frustrating and
time-consuming process. It is likely to take longer than you think. As you
will soon see, careful thought must be given to the relevance, language
level, cultural interpretations, and clarity of the items. Remember that your
respondents are doing you a favour by completing the questionnaire. It is
therefore vitally important that it is reader-friendly. Avoid items that are
humiliating, confusing, or make the respondents feel inadequate. Such
items may not only affect their motivation to respond to that particular
question but their attitude towards the entire survey.
2. General guidelines for using and modifying existing items

It is frequently recommended that researchers use well-known questionnaires, of
which the reliability and validity (see the resources on “reliability” and “validity”)
have already been established. This enables one to compare one’s findings
with those of other researchers who have used the same questionnaires. But
it is not always possible to use well-known questionnaires for your particular
target population. Nor is it always advisable. Even if you do happen to find
a questionnaire that taps the construct you intend to measure, it may not be
entirely suitable for your study. (Do not blindly assume that the items of a
published questionnaire are good simply because they were constructed by
an “expert” and the questionnaire has been found by various researchers to
be reliable and valid.) Although the items in an existing questionnaire may be
reliable and valid for Americans or Europeans (who are the favoured respondents
of much past research), they may not be suitable for South Africans who differ
considerably from those populations in terms of life experiences. Furthermore,
even if the items are “good”, the language and jargon used may not be suitable
for your particular group of respondents. If you do choose to use existing
...........
147 PYC 2616/1
R E S O U R CE S
questionnaires, you must therefore critically scrutinise each item, bearing in

mind the principles for constructing new items (which are mentioned below).
Let us assume that you want to find out whether Unisa students feel lonelier
than residential students do and that you find an existing questionnaire that
appears to contain a number of highly relevant items. However, a few are
unsuitable as they relate to loneliness in the work situation. You may choose
to eliminate these, or you may decide to modify them. For example, you may
change the question “How lonely are you at work these days?” to “How lonely
are you at university?” (Be careful: If you eliminate or modify items, you should
not report the reliability and validity estimates of the questionnaire published
by the original author as these properties might be different for a questionnaire
containing the “changed” items.)
3. Guidelines for constructing new items

You may need to construct new items for a number of reasons. For one, there
may be no existing questionnaire that taps the particular construct you intend
to investigate. If there are existing questionnaires, they may contain items that
are not entirely suitable for the respondents in your study and therefore have
to be adapted. Or you may have to eliminate a number of unsuitable items
from an existing questionnaire and replace them with new ones.
The guidelines presented below advise you on what to take into account
when constructing or adapting items. It is important to note that they are only
guidelines – each questionnaire is unique and presents its own demands and
problems.
3.1 Content and construct relevance

When constructing or adapting items you must keep clearly in mind what you
are aiming to find out. This seems to be common sense yet it is easily forgotten.
Dos and don’ts
• Do read each item and ask yourself if the item relates to your topic. Does it
tap the construct you want to measure? For example, if you want to determine
whether pupils perceive their science teacher as a “good” teacher don’t ask,
“Do you like your science teacher?” Likeability is not the same as capability.
• Don’t be tempted to ask questions that are interesting but not vital to your
research. Be ruthless in disregarding such items.
3.2 Language level

The language level of your items must match the language level of your target
population. Remember that, as a researcher, you are a relatively well-educated
person with knowledge of your topic and a good command of language and
psychological terminology. The respondents may not be as knowledgeable
or have as large a vocabulary as you have. Therefore, guard against using
language that is too academic or difficult for all of them to understand. They
...........
148
R e s o u rce s
may misinterpret items written in such language, or simply choose any answer
in order to avoid appearing stupid. This will invalidate your results.
Dos and don’ts
• Don’t use academic or technical terminology, jargon, words that are seldom
used in everyday speech, very long sentences, or complicated syntax (see
example below).
• Do phrase your items in such a way that the language level matches that of
your respondents. All the words in your items should be familiar to them.
For example, the following item may be easily understood by teachers but
not by 12-year olds: “The notion that pedagogues are habitually unfair to
scholars is preposterous”. Rather state: “Teachers are often unfair to pupils”.
• If you are not sure whether items would be easily understood by your
target population then do present them to a small group of respondents
from your target group and ask these people to “think aloud” while they
respond to them.
3.3 Contextual relevance

Remember that the same item may mean different things to groups with
different socio-economic and cultural backgrounds. The meaning of some
idiomatic phrases in American questionnaires (or even South African idiomatic
phrases) may not be clear to all groups of South African respondents. It may
also happen that items originally written in English are not clearly understood
when literally translated into other South African languages. So translations
should be checked by good translators.
Black respondents are likely to be at a disadvantage when they are required

to respond in English or Afrikaans. If it is too expensive and time consuming
to have questionnaires translated into all South African languages, at least
be sure that the questionnaire does not contain phrases that have different
connotations (implied meanings) in different cultures.
Dos and don’ts
• Do try to see the items from the respondents’ perspective.

• Do be aware of possible cross-cultural differences. For example, “family”
may be interpreted by some groups as family living in the same house
or as immediate family. Other groups may interpret “family” as including
distant relatives.
• If your target population is different from your own cultural group then do
pre-test your items on a few members of that group.
• Do try to have your items correctly translated into the language of your
respondents.
3.4 Clarity
All instructions, questions and statements in a questionnaire must be so clear
that all the respondents interpret them in the same way and understand
...........
149 PYC 2616/1
R E S O U R CE S
exactly what is required of them. No respondent should be expected to try to

fathom what a question means. Indeed, one of the main problems that may
introduce unwanted variance into the scores on a questionnaire is random or
careless choice of responses by respondents who lose motivation and fail to
give serious attention to completing the questionnaire because they are unable
to understand what items mean.
Writing clear instructions and items is not always easy. It often requires
painstaking effort, pre-tests and revisions to ensure clarity. If anything in your
questionnaire is not understood and/or misinterpreted, your results will be
useless.
Dos and don’ts
• Do avoid ambiguity. Ambiguous words or statements are those that lend

themselves to being interpreted in a number of ways. Consider the following
statements:
Visiting lecturers can help one feel less isolated.
(Does this mean that the lecturers do the visiting – or do the students?)
Boys study more than girls do.
(How would you interpret that? Do boys study more material than girls do
or do they study for a longer time?)
The following questions, although not strictly ambiguous, are imprecise
and may also be interpreted in various ways:
How many hours do you presently work over the weekend?
Is there heavy traffic in your neighbourhood?
Do you believe in family planning?
“Work”, “heavy traffic”, and “family planning” may mean different things to
different people. For example, some respondents may classify the following
as “work” whereas others may not: studying, housework, temporary em-
ployment, and charity involvement. “Heavy traffic” may be interpreted to
mean either the volume of traffic or the size of vehicles. “Family planning”
may be interpreted to mean either budgeting for the future or birth control.
It is important therefore that you choose words that have precise meanings.
• Don’t ask questions with two inherent issues. If there is more than one
issue inherent in an item the respondents will not know which to answer.
Consider these examples:
I am fully occupied and I don’t feel lonely.
I don’t feel lonely because I am fully occupied.
Do you find the psychology course informative or interesting?
For example, imagine you (a) are fully occupied (you have a full-time job
and are studying through Unisa), and (b) feel extremely lonely because you
do not have any time to socialise and/or do not have any close friends or
relatives. How would you answer the first two items above? Rather break
such questions or statements into two separate items.
• Do scrutinise any items that contain the conjunctions “and” or “or” to see
if they contain more than one possible issue.
...........
150
R e s o u rce s
• Wherever possible, don’t use negatives (especially double and multiple

negatives and those to which respondents are required to respond with
agree or disagree). Try to answer the following: Do you agree or disagree
with the following statement:
It is not unusual for me not to do assignments until the last moment.
It would be far easier to agree or disagree with the following:
I usually do my assignments well in advance.
Negatives implicit in words such as “control”, “restrict”, “forbid” and “op-
pose” also tend to cause confusion.
• Do use active rather than passive statements. Passive statements are more
difficult to understand, and therefore more difficult to respond to, than
active statements. Here is an example of a passive statement: It is believed
by students that they will be given extension by lecturers.
The following is simpler:
Students believe lecturers will give them extension.
• Do ask specific questions rather than general or vague questions. General

items may not be interpreted in the same way by everyone, and thus
produce unreliable answers. The more general the item, the more likely it
is that it will be interpreted in different ways and the more difficult it will
be to answer. But, although specific items are preferable, general items may
sometimes be necessary, as it may be impossible to ask about everything
in detail. The following are examples of general items:
Are you happy these days?
Does your husband help you?
Respondents may find it difficult to answer these items. Some may be happy
in their jobs but unhappy in their relationships. Others may be happy in
most spheres but unhappy with their health. As regards the second item,
a husband may help with the cooking but may refuse to shop or wash the
dishes.
• So do write items that are specific, simple, clear, and to the point.
3.5 Configuration
“Closed” items offer a given choice of responses. When you construct a set
of such items, be sure that the given responses are appropriate for each item.
In other words, your item should be configured correctly.
If the given choice of responses are “always”, “very often”, “sometimes”,

“seldom” or “never”, consider what a student could respond to the following
question.
(a) Have you met any other students who are studying psychology?
One could hardly respond “always” to this question.
Now suppose students are asked to respond “strongly agree”, “agree”, “uncertain”,
“disagree”, or “strongly disagree” to a set of items. Which of these responses
could they give to the following statement?
...........
151 PYC 2616/1
R E S O U R CE S
(b) I phone a lecturer if I do not understand something in the textbook.
Not all the given responses above are appropriate to this statement. “Uncertain”,
“strongly agree” and “strongly disagree” are unnecessary. All the given responses
would be more suitable to a statement such as “I believe one should phone
a lecturer whenever one doesn’t understand something in the textbook”.
Indeed “always”, etcetera would be a more suitable set of given responses to
question (b).
3.6 Prerequisite knowledge

If you construct items that ask for factual information, remember that it can
be difficult to remember events in the distant past.
Dos and don’ts
• Do limit the time frame to the immediate past (such as the last week – at
the most the last six months) when asking about what happened in the past.
If the respondents have to spend a lot of time and effort to remember what
they did or felt in the distant past they are likely to guess or give up. And
this will affect the reliability of the information they provide. For example,
it may be difficult for a student to answer the following question: “How
many hours did you spend on doing assignments last year?”
• When asking questions relating to factual information, do make sure your
respondents have the information. For example, if you ask students “Do
you agree or disagree with Unisa’s policy regarding entrance requirements?”
make sure your respondents are aware of the policy. In such a case it is
better to first ask whether they are aware of the policy.
3.7 Leading questions

Leading questions are those that influence respondents to give a particular
answer.
Dos and don’ts
• Don’t write items that encourage respondents to give a particular answer.

The following are examples of leading questions:
Discrimination in South Africa is horrific, isn’t it?
Would you agree that XX is an excellent washing powder?
These questions “influence” respondents to think of discrimination in exces-
sive terms and XX in good terms.
• Don’t give examples unless it is really necessary, as these too may influence
certain respondents. For example:
Do you use any word processing packages, such as ZZ?
Rather list all possible programs or use no examples at all.
...........
152
R e s o u rce s
4. Problems relating to response bias or response style

Response biases or styles, which are tendencies to choose a particular type
of answer over other alternatives, cause problems in research. The following
are common examples of response bias and style.
4.1 Social desirability response bias

The social desirability response bias is a tendency to choose what one believes
to be the most socially acceptable response. Some respondents may tend to do
this in order to make themselves feel good or to create a favourable impression.
This bias may operate at various levels of consciousness, ranging from an

unconscious effort to please to deliberate faking in order to impress others.
Deliberate faking is most likely to occur when the respondents have good
reason for presenting a good image of themselves: when they are fully aware of
what is being measured and for what purpose; when their identity is disclosed,
and when they are aware that their responses will affect them in some way.
For example, faking is likely to occur when the respondents are applying for
a job or for being declared fit or unfit for work. But respondents may also lie
(even to themselves) to protect their real feelings or justify their behaviour, or
because they do not want to admit their ignorance. For example, not many
people will admit to hating their children, battering their spouses, taking drugs,
or being ignorant of world affairs.
Faking can be reduced if the respondents are made aware that they are not
being individually assessed and that their responses will remain anonymous.
4.2 Response styles

A response style is a tendency to make a particular type of response, regardless
of the item content. For example, some respondents may tend to choose
extreme responses such as “strongly agree”. Some may repeatedly choose
central responses such as “uncertain”, and others may be inclined to “disagree”
irrespective of the item content.
To avoid such tendencies researchers frequently design balanced questionnaires

(see the resource on “item format” where rating scales are discussed) consisting
of items which are positively stated and items which are negatively stated.
However, one of the main problems of such “balanced” questionnaires is that
negatively stated items tend to confuse the respondents. For this reason, you
need only know the two types of positively stated items: positively stated and
negatively scored. For example, when measuring trust (a high score indicates
a high level of trust), one might use an item such as “People often let me
down” – positively stated and positively scored. For example, “I trust people”.
...........
153 PYC 2616/1

Pyc2616 10050590

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Pyc2616 10050590

Uploaded by

Copyright:

Available Formats

© 2023 University of South Africa

All rights reserved

Printed and published by the

LESSON 1: Identify a suitable content domain 1

LESSON 2: Design a questionnaire 11

LESSON 3: Writing questionnaire items 29

LESSON 4: Pilot-test the questionnaire 38

LESSON 5: Evaluate reliability and validity 55

LESSON 6: Compile a manual 71

LESSON 7: Review a questionnaire 82

LESSON 8: Review a manual 96

Survey research is an approach to research whereby information is collected

What this module is about

Why study this module

Contextualising the module

Psychological questionnaires have a long history in the discipline of psychology

The use of psychological questionnaires in South Africa cannot be divorced

In this module, we draw your attention to important aspects to consider when

Decolonisation and questionnaire design in psychology

Decolonisation is a school of thought that seeks to reverse unjust practices

In this module, we encourage you to be cognisant of the history of psychology

Bias and equivalence

Table 1: Types of bias

Item bias An item is biased when it has a different psychological

Table 2: Strategies to reduce bias in cross-cultural assessment

Type of bias Strategies

Minimising bias in cross-cultural studies usually amounts to a combination

What will you learn?

• a critical understanding of the meaning of psychological theory and

– identify a suitable content domain

Learning outcomes. The learning outcomes are what we want you to

Specific outcome Assessment criteria evidence shows that

Specific outcome Assessment criteria evidence shows that

Remember: These outcomes and criteria must be your “map”’ to help

We have included an introduction, establishing cultural equivalence and

Glossaries of certain basic, but perhaps unfamiliar terms, are provided.

How to approach a lesson

(1) start by quickly skimming through the material in the lesson.

Assignments and examination format

Rank order scaling questions allow a certain set of behaviours of individuals or

What is the nature of the examination?

How will the assignments and the examination be assessed

to check our progress. The assignment questions as well as the examination

Practical advice on writing an assignment and the examination

• the assessment context

Notional study hours

questionnaire and manual in the examinations through answering the multiple-

Practical methods in questionnaire design and assessment build your

Overview of the research process:

1 Identify a suitable content domain

This lesson will help you to produce:

• an identified content domain

The following resource material is required for this lesson:

Establishing cultural e quivalence

A culturally appropriate measure seeks to be culturally equivalent across sub-

In creating culturally appropriate measures, it is also essential to consider the

During this process, emphasis should be on conceptual and cultural equivalence.

• Conceptual equivalence: Do people attach the same meanings to terms

A questionnaire cannot be valid unless it is reliable. An assessment can provide

1.1.1 What is a content domain?

Content domain is the body of knowledge, skills or abilities being measured or

1.1.2 Experimental validity and content domain

It is an accurate and comprehensive measurement tool of the entire content

LESSON 1: Identify a suitable content domain 1

LESSON 2: Design a questionnaire 11

LESSON 3: Writing questionnaire items 29

LESSON 4: Pilot-test the questionnaire 38

LESSON 5: Evaluate reliability and validity 55

LESSON 6: Compile a manual 71

LESSON 7: Review a questionnaire 82

LESSON 8: Review a manual 96

4. Young people are good drivers. True/false