outline

1
 Objectives
 Introduction
 Data collection tool
 Designing a questionnaire
 Reliability and validity of tool
 Summary
 Reference
 Session objectives
2

After the end of this presentation you will be able to:

 Describe different type of tool used for data collection.
 Discuss different steps in designing a questionnaire.
 Differentiate reliability and validity of data collecting
tools.
 Introduction
3

Data collection is an important aspect of any type

of research study.
Inaccurate data collection can impact the results of
a study and ultimately lead to invalid results.
If data are of poor quality then it is certain that
the results of any Research analysis will be
unreliable and inaccurate.
 Introduction…
4

 Depending on the nature of the information to be

gathered, different instruments are used to conduct
the assessment.
 The various methods of data gathering involve the
use of appropriate recording forms. These are
called tools or instruments of data collection.
 Introduction…
5

Data:
Any record, descriptive account, or symbolic
representation of an attitude, event or process.
Data is the key component of a research
The key to a good research is good data
 Introduction…
6
Data collection:
It is a stage in the research process when information is
gathered through surveys, experiments, fieldwork, or indirect
methods to generate data.
It is the process of gathering and measuring information on
variables of interest to answer stated research questions, test
hypotheses, and evaluate outcomes.
 Tools of Data collection
7

In general selection of data collection instrument in

relation to variables type and objectives is based on :

 Resource required  Familiarization of

 Cost effectiveness the procedure
 Accuracy
 Acceptability of the  Relevance
method
 Timeliness
 Coverage of the method
 Question
8

What is the differences between data

collection techniques and data
collection tools?
 Differences between data collection
techniques and data collection tools
9

Data collection techniques Data collection tools

Using available information Checklist; data compilation

forms
Observation Eyes and other senses,
pen/paper, watch, scales,
microscope, etc..
Interviewing Interview guide, checklist,
questionnaire, tape recorder
Administering written Questionnaire
questionnaire
 Data collection tools
10
10
Questionnaire
 Checklist
Data compilation forms
Eyes and other senses,
 Pen/paper, watch,
Scales, microscope,.
Interview guide,
 Tape recorder, video recorder, etc…
 Questionnaire
11

It is a set of systematically structured questions used by a

researcher to get needed information from respondents.
It is simply a ‘tool’ for collecting and recording information
about a particular issue of interest.
Questionnaires should always have a definite purpose that is
related to the objectives of the research.
 Questionnaire……
12

 Questionnaires includes both:

- guides/questionnaires used in interviews and

- Self-administered questionnaires which are most

commonly used in large scale surveys .

NB: Most quantitative studies use self-administered

questionnaires.
 Questionnaire……
13
 Questionnaires are commonly used to:
 Collect factual information in order to classify people and their
circumstances.
 Gather straightforward information relating to people’s behavior
 Look at the basic attitudes/opinions of a group of people relating to a
particular issue.
 Measure the satisfaction of customers with a product or service
 Collect ‘baseline’ information which can then be tracked over time to
examine changes.
 Types of questions
14
14
 Depending on how questions are asked and
recorded we can distinguish two major possibilities:
1. Open-ended questions, (allowing for completely open
as well as partially categorized answers)
 Completely open ended question
 Partially open ended question
2. Closed questions.
 Types of questions…
15
 The choice between open-ended and close-
ended questions is based on:
 Knowledge of subject matter
 Depth of information required
 Sample size
 Length of questionnaire
 Data analysis techniques
 The amount of time to complete the research.
 1. Completely open-ended questions
16

 Open-ended questions permit free responses which

should be recorded in the respondents’ own words.
 Such questions are useful for obtaining in-depth
information on:
 Facts with which the researcher is not very familiar
 Opinions, attitudes and suggestions of informants
 Sensitive issues
 1. Completely open-ended…
17

 For example:
 ‘What is your opinion on the services provided in the ANC?’
(Explain why.)
 ‘What do you think are the reasons some adolescents in this area
start using drugs?’
 ‘What would you do if you noticed that your daughter (school
girl) had a relationship with a teacher?’
 The answers to these questions are written down as closely
as possible in the words of the respondents.
 2. Partially categorized questions
18

 In interviews questions are often asked as

open-ended questions, but to facilitate
recording and analysis, some answers can
already be pre-categorized.
 Cont…
19
For example
‘How did you become a member of the Village
Health Committee?’
 Advantages Disadvantages

O Permit unlimited number of Respondents give answers with diff.

P answers level of detail.
E
N
- Respondents can qualify and Answers can be irrelevant.
clarify responses
E
N
Can find the unanticipated Inarticulate or forgetful
D
respondents are at disadvantage.
E
D
Reveal respondents thinking coding responses is subjective
processes and tedious.
Requires more respondents
effort and time.

20
 2. Close-ended questions
21

 Closed questions have a list of possible options or

answers from which the respondents must choose.
 Closed questions are most commonly used for
background variables such as:
 Age, marital status or education,
 Although in the case of age and education you may
also take the exact values and categorize them during
data analysis.
 Close-ended question…..
22
Closed questions may be used to get the respondents to express
their opinions or attitudes by choosing rating points on a scale.
For example:

What is your opinion on the following statement:

‘Women who have induced abortion should be severely

punished.’
 Close-ended question…..
23
Advantages of closed questions:
 It saves time
Easy and quick to answer
Answers across respondents easy to compare
Answers easier to analyze on computer
Response choices make question clearer
Easy to replicate study
 Close-ended question…..
24

Disadvantages:
-Can put ideas in respondents head
- Respondents with no opinion answer anyway
-Many choices can be confusing
-Can't tell if resp. misinterpreted the question
- Fine distinctions may be lost
- Clerical mistakes easy to make Force respondents into
 simple responses
 Types of closed-ended questions
25

1. Contingency questions
 Are those questions whose relevance depends on
responses to a prior question.
 Example : Have you ever been hospitalized?
( ) Yes (Please answer questions 8.11.)
( ) No (Please skip question 8.11. Go directly to
question 12 on page 3.)
 Cont…
26
2. Two-way questions
 Include a question or statement followed by two dichotomous ,
realistic alternatives.
- yes/no, agree/disagree, like/dislike, and so on.
 Sometimes, a third alternative, “don’t know” or “no opinion,” is
included to make the choices more realistic.
 Example : Have you had a mammogram in the past 12 months?
( ) Yes ( ) No ( ) Don’t Know
 Cont…
27
3. Multiple-Choice Questions
It is among the most frequently used formats in
close-ended questions.

There are two types of multiple choice questions

- multiple-choice, single response questions

- Multiple-choice, multiple- response questions.

 Cont..
28
28

4. Ranking scale ( forced ranking scale)

Asks respondents to rank a number of items in
relation to one another.

Provides more information than multiple-choice

question (multiple selection + ranking)
 Cont….
29
Example
The following are sources where a person may obtain
knowledge about AIDS. In your situation, please
rank these sources in terms of relevance to you.
( ) Newspaper
( ) Radio
( ) T.V.
( ) Book
( ) Workplace/School
( ) Friends
( ) Relatives
( ) Family Members
( ) Other
 Cont…
30
6. Agreement Scale Questions
It presents a statement and asks respondents to
indicate their level of agreements or disagreement.

There are typically five ordinal response categories:

“strongly agree,” “agree” “undecided,”
“disagree,” and “strongly disagree,”
 Checklists
31
Checklists encompass several questions that have the

same response format.

A checklist is a two dimensional arrangement in which a

series of questions is listed along one dimension (usually

vertically) and response alternatives are listed along the
other.
Checklists are relatively efficient and easy for respondents to

understand, but because they are difficult to read orally, they

are used more frequently in SAQs than in interviews.
 Checklist…
32
Checklists can be used to systematically observe

human behavior or the condition of specific

equipment (e.g., fridge, expiring dates of medicines,
or completeness of records).
Observations can be relatively open or can be

predetermined comparisons of reality against fixed

standards.
 Checklist….
33
The objectives of the study determine the content of

a checklist.
A checklist includes all the items or points that must

be considered during an observation in the field, or

when extracting data from existing records.
 Checklist….
34
Example:
 An observation checklist for rating the cleanliness and use

of Ventilated Improved Pit Latrines (VIP) in a study on

hygiene practices
35 Tools and methods of
data collection:
Designing research instrument
 Designing research instrument
36
 Research instrument design is the process by which
a research instrument is developed.
 In order to gather useful and relevant information it is
essential that careful consideration is given to the
design of your questionnaire.
 A well-designed questionnaire requires thought and
effort, and needs to be planned and developed in a
number of stages.
 Initial considerations
37

Question content, phrasing and response format

Question sequence and layout

Pre-test and revision

Final questionnaire
 Principles of good question writing
38
 There are key principles that will help to ensure that the
questions are understood and answered as you intended.
 Write from the perspective of the person who will be answering.
 Avoid confusion
 “Avoid asking about future intentions”
 Avoid overlapping or unbalanced response options

 Limit questions to only those that will give you information

related to your project objectives.
 Questions should be considered
39

 The following questions should be considered

before designing the questionnaire:
 What exactly do we want to know, according to the
objectives and variables we identified earlier?
 Is questioning the right technique to obtain all answers,
or do we need additional techniques, such as
observations or analyses of records?
 Questions should be …
40

 Of whom will we ask questions and what techniques will

we use?
 Do we understand the topic sufficiently to design a
questionnaire?
 Are our informants mainly literate or illiterate? (If
illiterate, the use of self-administered questionnaires is out
of the question.)
 How large is the sample that will be interviewed?
 Steps in designing a questionnaire
41
 The steps apply to design ‘questioning tools’
include;
Step 1: Content
Step 2: Formulating questions
Step 3: Sequencing the questions
Step 4: Formatting the questionnaire
Step 5: Translation
Step 6: Questionnaire pre- testing
 Step 1: Content
42
 Take your objectives and variables as a starting point.
 Decide what questions will be needed to measure or (in the
case of qualitative studies) to define your variables and reach
your objectives.
 When developing the questionnaire, you should reconsider
the variables you have chosen and, if necessary, add, drop or
change some.
 You may even change some of your objectives at this stage.
 Step 2: Formulating questions
43

 Formulate one or more questions that will provide

the information needed for each variable.
 Take care that questions are specific and precise
enough so that different respondents don’t interpret
them differently.
 Step 2: Formulating questions…
44
 Check whether each question measures one thing at a time.

 For example: ‘Do you think that the war situation leads to
mental problems that require treatment by health staff?’

 We can bring it into three topics, which should be split up in

 mental problems resulting from the war
 treatment required, and
 who should provide the treatment.
 Step 2: Formulating questions…
45

 Avoid words with double or vaguely defined

meanings or that are emotionally laden.
 Concepts such as dirty (clinics), lazy (patients), or
unhealthy (food), should be omitted.
 Ask sensitive questions in a socially acceptable way.
 Step 2: Formulating questions…
46

 A common weakness in questionnaires is the

inappropriate transformation of research
questions into interview questions.
 You cannot ask informants: ‘Would a woman’s
educational status influence her health?’
 Ask the questions to yourself or to a friend and
check whether the answers you get are the type of
responses you want.
 Step 3: Sequencing the questions
47
 Begin with a non-threatening question.
 The sequence of questions must be logical for the
informant and allow, as much as possible, for a
‘natural’ conversation, even in more structured
interviews.
 At the beginning of the interview a limited number of
questions concerning ‘background variables’ may be
asked.
 Step 3: Sequencing…
48

 As informants may be reluctant to provide

‘personal’ information and may become worried
about confidentiality, you should restrict yourself to
an essential minimum.
 Start with an interesting but non-controversial
question (preferably open) that is directly related to
the subject of the study.
 Step 3: Sequencing…
49

 Pose more sensitive questions as late as possible in

the interview (e.g., questions pertaining to income,
political matters, sexual behavior, or stigma
experienced in case of stigmatizing diseases).
 Step 4: Formatting the questionnaire
50

 When you finalize your questionnaire, be sure that:

 A separate, introductory page is attached to each
questionnaire, explaining the purpose of the study,
requesting the informant’s consent to be interviewed and
assuring confidentiality of the data obtained.
 Each questionnaire has a heading and space to insert the
number, date and location of the interview.
 Step 4: Formatting…
51
 Sufficient space is provided for answers to open-ended
questions, categories such as ‘other’ and for comments
on pre-categorized questions.
 Boxes for pre-categorized answers are placed in a
consistent manner.
 You may add the name of the interviewer, to facilitate
quality control.
 Step 5: Translation
52
 If interviews will be conducted in one or more local
languages, the questionnaire should be translated in order to
standardize the way questions will be asked.
 After having it translated you should have it retranslated into
the original language by a different person.
 You can then compare the two versions for differences and
make decisions concerning the final phrasing of difficult
concepts.
 Step 5: Translation….
53

 If it is conducted in one or more local language, it

should be translated .

Afan Oromo
E.g. English (Amharic)
English
 Step 6: Questionnaire pre- testing
54
 The major challenge in questionnaire design is to make it

clear to all respondents.

 In-order to indentify and solve the confusing points, we

need to pre-test the questionnaire.

 During the pre-test: the questionnaire participants should be

randomly selected from the study population.

 Purpose of pre‐test
55
 Check the clarity of instruments (to test the data
collection instrument).
 Are questions easily understood by the study participants?
 Correction of the survey instruments before the actual
survey.
 What changes could be made to ensure the quality of the data
collection ?
 Assess the level of understanding of data collectors
and supervisors.
 Do data collectors understand the questions?
56
Tools and methods of data
collection:

Reliability and validity of data

collection tools
 Reliability
57
 Reliability: is the extent to which results are consistent over time and an accurate
representation of the total population under study.
 A data collection method or instrument is considered reliable if the same result is
obtained from using the method on repeated occasions.”

 Reliability estimates are used to evaluate:

(1) the stability of measures administered at different times to

the same individuals or using the same standard (test–retest reliability)

(2) the equivalence of sets of items from the same test (internal consistency).
 Reliability can depend on various factors:
 Types of Reliability
59
 Inter-observer reliability
To what degree are measures taken by different
raters/observers consistent?
How to test reliability?
 Consider pre-testing if different raters/observers are

giving consistent results on the same phenomenon.

 Cont..
60
 Test-retest reliability
 Is a measure consistent from one time to another?

How to test reliability?

 Consider administering the same test to the same

(or similar) sample in different occasions. But be

aware of the effects of the time gap.
 Cont…
61
 Parallel forms reliability
 Are previous tests and tools constructed in the
same way from the same content domain giving
similar results?

How to test reliability?

 Consider splitting a large set of questions into

parallel forms and measure the correlation of the

results.
 Cont..
62
 Internal consistency reliability
 Do different measures on a similar issue yield

results that are consistent?

How to test reliability?
 Consider testing a sampling of all records for

inconsistent measures.
 How to improve reliability?
63
 When constructing reliable data collection
instruments:
 Ensure that questions and the methodology are
clear
 Use explicit definitions of terms
 Use already tested and proven questioning
methods.
64

Examples of reliable measures:

1.ruler is a reliable measure of length
Examples of measures that may be unreliable
2.Asking the question "Have you been tested for
HIV?" may not yield reliable data because some
people may answer truthfully on this sensitive topic
and some may not.
65

 Lincoln and Guba (1985, p. 300) use “dependability”, in

qualitative research which closely corresponds to the notion

of “reliability” in quantitative research.

 To ensure reliability in qualitative research, examination of
trustworthiness is crucial.
 Validity
66
 Validity is the best available approximation to the
truth of a given proposition, inference of
conclusion. A measurement method or instrument is
considered valid if it measures what it intends to
measure.”
 Researchers generally determine validity by asking
a series of questions, and will often look for the
answers in the research of others.
 validity is the extent to which the interpretations of
the results of a test are warranted, which depend on
the test’s intended use (i.e., measurement of the
underlying construct).
 Types of validity
67

 Internal validity
 Internal validity is relevant in studies attempting to
establish a causal relationship, and it is only
relevant for the specific study in question.
 “Can change be attributed to a program or
intervention and not to other possible causes?”
 Cont…
68
 External validity
 It is related to generalizing. It is the degree to
which the conclusion of your study will hold for
other persons in other places and at other times.
 Cont…
69

Examples of challenges to validity

 An evaluation designed to assess the impact of

nutrition education on weaning practices is valid if

actual weaning practices are observed. An
evaluation that relies only on mothers’ reports may
find that it is measuring what mothers know rather
than what they do.
 Cont…
70
 Construct validity: This type of validity is a
judgment based on the accumulation of evidence
from numerous studies using a specific measuring
instrument.
 Content validity: This type of validity addresses
how well the items developed to operationalize a
construct provide an adequate and representative
sample of all the items that might measure the
construct of interest.
 Cont…
71
 Criterion-related validity: This type of validity
provides evidence about how well scores on the
new measure correlate with other measures of the
same construct or very similar constructs that
theoretically should be related.
 Making Sure Data Are Valid and Reliable
72
 Do the data represent the outcomes that the instrument
is supposed to measure? Are the data valid?
 A valid measure assesses what it is designed to measure,
which allows for comparison of results across studies.
 Have we ensured that our measures are reliable?
 A reliable measure produces stable responses regardless of
the data collector.
 An unreliable measure will yield varied responses
depending on differences between interviewers or data
collectors.
 Cont…
73
 Do we have the right data?
 Do we have enough data?
 To be valid for decision-making, data must answer
your questions about program outcomes and include
a sufficient number of participants to be
representative of your target population and its
various subgroups.
 Validity of data is improved when data collection
is “triangulated,” for example, when various
methodologies are used to measure the same
phenomenon, or multiple researchers conduct a
structured observation of the same phenomenon.
 Relation ship between reliability and validity
74
 Validity vs. Reliability
 Summary
75
 The quality of research depends to a large extent on
the quality of the data collection tools.
 Therefore designing good ‘questioning tools’
forms is an important and time-consuming phase in
the development of most research proposals.
 Reference
76
1. Wai-Ching Leung. How to design questionnaire. Student BMJ. 2001 June;9:187-9.

2. http://www.cochrane.es/files/Recursos/How_to_design_a_questionnaire.pdf

3. Denis Polit and etal (2003). The Nursing Research principles and method 7 th edition.

4. www.albertahealthservices.ca. Alberta Health Services – Addiction and Mental Health

Community Treatment and Support Knowledge and Strategy.

5. Getu Degu and Tegbar Yigzaw (2006): Research methodology lecture notes for health
science students. University of Gondar In collaboration with the Ethiopia Public Health
Training Initiative, The Carter Center, the Ethiopia Ministry of Health, and the Ethiopia
Ministry of Education.

6. Patricia M Hudelson. Qualitative Research for Public Health Programmes.

WHO/MNH/PSF/94.3. Geneva 1994.
 Reference…
77
6. Oppenheim, A. N. (1992) Questionnaire design, interviewing and
attitude measurement (2nd edition). London: St Martins Press.

7. Research fundamentals by Carole L. Kimberlin and Almut

G.Winterstein .Am J Health-Syst Pharm—Vol 65 Dec 1, 2008.

8. Trochim 2001. http://trochim.cornell.edu/kb/ ISYS3015

Analytical Methods for IS Professionals School of IT, The
University of Sydney
78

Thank you…!!!