Chapter 3

Research Design: Planning the Research

Chapter Objectives
By the end of this chapter students will be able to:
 Describe Research Design including
 Explain the Meaning and Need for Research Design
 Features of a good design
 Identify the different Research Designs

1
3.1. Research Design (RD) - Introduction
 RD is the entire design or plan of the research project
 It involves deciding on all aspects of the research process:
 mainly research method/approach

 overall structure for the procedures

 data collection techniques

 data analysis, and solution development

 publication outlet if possible

 RD refers to the arrangement of the conditions for collection

and analysis of data in a manner that will satisfy or achieve
the objectives of a research undertaking.
 It’s intended to enable the researcher to answer questions as

validly, objectively, accurately and economically as possible;

 It provides for the collection of relevant data with minimum

effort, time and money.

2
Cont…
 Specifically, RD decisions include:
 What is the study about? (Problem definition)
 Why is the study being made? (Justification)
 Where will the study be carried out? (Location)
 What type of data is required? (Quantitative, Qualitative,
Primary, Secondary)
 Where can the required data be found? (target population)
 What periods of time will the study include?
 What will be the sample design? (technique chosen)
 What techniques of data collection will be used?
(observation, interview, questionnaires),
 How will the data be analyzed?
 In what style will the report be prepared?
3
RD helps in many ways
 It sets up the framework for adequate tests of relationships
among study variables
 It tells what observations to make/what variables to measure,
how to make them/measure them, and how to analyze the
data;
 It suggests how many observations to make and the type of
statistical analysis to use
 facilitate the smooth sailing of the various research operations,
making research as efficient as possible yielding maximal
information with minimal expenditure of effort, time and
money.
 helps a researcher to organize his ideas in a way s/he can look
for flaws and inadequacies, get others critical evaluation and
comments on the proposed study, and achieve research
purpose and improve the reliability of the results.

4
Cont…
 Factors to be considered during preparation of an RD:
 the availability of time and finance for the research work.
 the availability, skills and staff of the researcher, (if any);
 the objective of the problem to be studied
 the nature of the problem to be studied
 Identifying the study variables
 Identifying the study subjects/experimental units
 the means of obtaining information and reasoning leading to
selection;
 Sampling
 Thus, RD is an advance planning of the relevant data
collection and analysis methods to be adopted, keeping in
view the objective of the research and the availability of
staff, time and money.
5
Features of a Good Design
 The choice of a good RD is related to the purpose of the
research problem & the nature of the problem to be studied.

6
3.2. Approaches used in RD
 Quantitative, Qualitative and Design science
 Qualitative Research
 involves the use of qualitative data, such as interviews,

documents, and participant observation data, to

understand and explain social phenomena.
 Qualitative data sources include observation and
participant observation (fieldwork), interviews and
documents and texts, and the researcher's impressions and
reactions.
 Examples of qualitative methods are action research, case

study research and ethnography.

 Quantitative research methods
 were originally developed in the natural sciences to study

natural phenomena.
Cont…
 popular data collection techniques include Questionnaire
(survey instruments), secondary data sources or archival
data, objective measures or tests.
 Examples include survey methods, laboratory experiments,
formal methods (e.g. econometrics) and numerical
methods such as mathematical modeling
 Design Science - is a problem-solving paradigm:
 seeks to create innovations that define the ideas, practices,
technical capabilities, and products through which the
analysis, design, implementation, and use of IT can be
effectively and efficiently accomplished.
 employs data collection techniques from qualiatative and
quantitative methods
 Some Comparisons
Quantitative Qualitative Design science
 Explanation,  Explanation,  Motivated to solve
prediction description problems
 Test/Modify  Build theories  Prescriptive –
theories Develop artifact
 Known  Unknown  Known variables
variables variables
 Large sample  Small sample  Small sample, case
based
 Standardized  Observations,  Observation,
instruments interviews interviews, etc
 Deductive  Inductive  Inductive
9
Cont…
Quantitative Qualitative Design science
 Statistical  Analytic  Replicable to
generalization generalization similar problems in
similar contexts
 Use numbers  Use texts  Texts, codes and
drawings
 Standard  No standard  Pragmatic validity
instrument to instrument,
validation triangulation
 Objective Truth  Subjective truth  Subjective
Truth ???
 Linear Process  Iterative  Iterative

10
Data Collection
 is a term used to describe the process of preparing and
collecting research data
 It is important to choose the right source of data and data
collection method(s) as this will allow data to be collected
that will meet the objectives of the research.
 Data Sources
 People (respondents), Databases, Computer Log files
 Documents (web documents, news…, reports …)
 Devices like camera
 All require a decision whether to use as it is or sample.
 Why sample? How? How many?
 Data collection techniques
 Depends on the type of research methods
Cont…
 Types of Data
 Primary Data: are those which are collected afresh and for the first
time and thus happen to be original in character.
 Secondary data: are those which have been collected by someone
else and which have already been passed through the statistical
process.
 Methods of Primary data collection
 Observation (Systematic viewing)
 Personal Interviews (structured or unstructured)
 Questionnaire (self administered)-
 Schedules (filled by enumerators)
 Choice of data collection largely depends upon
 The objective of research,
 Minimization of Bias or expenses
 Reduction of non-response or data error
Cont…
 Collection of secondary data
 Secondary data might be either published or unpublished.
 be careful in using 2ry data since it may be misleading.
 One has to check for
 Suitability: Is it relevant for your research problem?
 Adequacy: will you answer your questions adequately?
 Reliability- when was the data collected, who collected data,

how was the data collected?

 What is sampling?
 The act, process, or technique of selecting a suitable sample, or
a representative part of a population for the purpose of
determining parameters or characteristics of the whole
population.
 A census (complete enumeration of all the items in the
population) may not be practical and is almost never
economical.
Why Sampling instead of a census?
 Sampling can save time, money and energy.
A sample study is usually less expensive than a census study
and produces results at a relatively faster speed.
 Sampling may enable more accurate measurements for a
sample study is generally conducted by trained and
experienced investigators.
 Sampling remains the only way when
 population contains infinitely many members or
 population contains some inaccessible members.
 a test involves the destruction of the item under study
 Sampling usually enables to estimate the sampling errors
& obtain information concerning some characteristic of
the population.
Sampling
Sampling Techniques
 Probability sampling
 each unit of the population will be represented in the sample.
 each member of the population has a chance (equal in the
case of random selection) of being selected.
 ensures information is obtained from a representative sample
of the population
 sampling error can be computed
 survey results can be projected to the population
 more expensive than non-probability samples
 Probability Sampling techniques
 Simple random sampling, Systematic random sampling
 Stratified random sampling, Proportional stratified sampling
 Cluster sampling
Cont…
 Non-probability sampling
 the researcher has no way of forecasting that each member of
the population will be represented in the sample. i.e., some
members of the population have little or no chance of being
selected.
 information is obtained from a non-representative sample of
the population
 sampling error can not be computed
 survey results cannot be projected to the population
 cheaper and faster than probability
 reasonably representative if collected in a thorough manner
 Non-probability sampling techniques
 Convenience sampling, Purposive/judgment sampling
 Snowball sampling and Quota sampling
Probability Sampling Techniques
 Simple Random Sampling
 Simple random selection where every member of the
population is given an equal chance (sample size/population
size) of being selected;
 Good for homogeneous population;
 Easy when the population is small and elements are known;
 Impractical for very large populations

 Systematic Random Sampling

 Selecting elements of the population in predetermined
sequence;
 Select every kth item on a list (k= N/n)
 Randomness element is in picking up the starting point
Cont…
Systematic Sampling Procedure:
- number the units in the
population from 1 to N
- decide on the n (sample size)
that you want or need
- k = N/n = the interval size
-randomly select an integer
between
1-k
- then take every
kth unit
Cont…
 Stratified Random Sampling
 applied when the population has different layers (strata)
 population is divided on the basis of characteristic of
interest in the population (e.g. race, age, gender etc.)
 the researcher randomly samples from each one of the layers
(stratum) equally
 examples
 sampling of school children from grades 4, 5 and 6
 sampling of customers (Corporate, Residential customers,
etc)
 has a smaller sampling error than simple random sample
since a source of variation is eliminated
 ensures representativeness when proportional sampling used
Cont…
 Direct Proportional Stratified Sampling
 the sample size in each stratum is proportional to the stratum
size in the population
 Examples
 2,000 corporate customers
 400,000 residential customers
 Disproportional Stratified Sampling
 the sample size in each stratum is NOT proportional to the
stratum size in the population
 Used if
 some strata are too small
 some strata are more important than others
 some strata are more diversified than others
 When to use stratified sampling
 If primary research objective is to compare groups
Cont…
 Using stratified sampling may reduce sampling errors
 Cluster Sampling
 grouping the population into clusters (usually along
geographic boundaries) and then select members of clusters
randomly.
 less expensive but less precise than random sampling
Population Random selection of clusters

 When to use cluster sampling

 If there are substantial fixed costs associated with each

data collection location

 When there is a list of clusters but not of individual

population members
Probability Sampling Techniques Summary
Population characteristics Appropriate sampling
technique
Homogeneous members  Simple random
sampling
 Systematic random
sampling
Stratified population with  Stratified random
approximately equal in size sampling
Stratified population, strata  Proportional stratified
different in size sampling
Population with discrete  Cluster sampling
clusters with similar
characteristics
Non-Probability Sampling Techniques
 Convenience Sampling
 taking of samples that are readily available
 E.g., those that arrive on a scene by coincidence, people in my
classes, mall intercepts
 may be appropriate for some less demanding research
 may give very biased results particularly when the population is
not homogeneous.

 Purposive Sampling
 Units of the sample are chosen purposively.
 Choosing people that are “typical” of a group;
 E.g., a judgment sample of college students to secure reactions
to a new method of teaching.
 usually used in qualitative research to develop hypotheses
Cont…
 Snowball Sampling
 Selection of additional respondents is based on referrals from
the initial respondents. E.g., friends of friends
 Used to sample from low incidence or rare populations.
 Quota Sampling
 The population is divided into cells on the basis of relevant
control characteristics.
 A quota of sample units is established for each cell.
 A convenience sample is drawn for each cell until the quota is
met.
 similar to stratified sampling - elements are selected in the same
proportion as in the population but not in a random fashion.
 E.g, there are equal number of IS and CS students
 Quota sampling would choose 20 IS and 20 CS students
without any attempt to random selection
Sampling Design Process
 Steps in the sampling process
 Defining the population

 who do you want to survey?” i.e. those who have the

information sought What are their characteristics.
 Specifying the sampling unit
 Specifying the sampling frame
 (Obtaining a “list” of population - how will you reach

sample. Ex: Tele. Book, map, city directory, …)

 Specifying the sampling method
 Probability or Non- Probability Sampling

 Determining the sampling size

 Specifying the sampling plan
 Selecting the sample
Identifying a Sufficient Sample Size
 It depends on many factors like type of research …..
 To some extent, the size of an adequate sample depends on
 how homogeneous or heterogeneous the population is :

how alike or different its members are with respect to the

characteristics of research interest
 If the population is markedly heterogeneous, a larger

sample will be necessary than if the population is fairly

homogeneous
 the degree of precision with which the researcher
wishes to draw conclusions or make predictions about
the population under study
 The basic rule is, the larger the sample, the better.
 But such a generalized rule may not be too helpful to a
specific research problem.
Guidelines for selecting a sample size:
 For small populations (with fewer than 100 people or other
units), there is little point in sampling. Survey the entire
population
 If the population size is around 500, 50% of the population
should be sampled
 If the population size is around 1,500, 20% should be sampled
 Beyond a certain point (at about 5,000 units or more), the
population size is almost irrelevant, and a sample size of 400
should be adequate
 Generally, the larger the population, the smaller the
percentage (but not the smaller the number!) one needs to
get a representative sample
 Exercise: Does this work for both qualitative and quantitative ?
Cont…
 Sampling is critical to
 Internal validity: the extent to which the interpretations of

the results of the study follow from the study itself.

 External validity: the extent to which findings of a study

can be generalized to people or situations other than those

observed in the study.
Errors in Survey
 Random Sampling Error
 the sample selected is not representative of the population due

to chance
 the level of it is controlled by sample size

 a larger sample size leads to a smaller sampling error.

 Non-sampling Error:
 systematic error the level of it is NOT controlled by sample

size.
Cont…
 Two basic types of non-sampling error
 Non-response error: occurs when units selected as part

of the sampling procedure do not respond in whole or in

part
 If non-respondents are not different from those that

did respond, there is no non-response error

 Response or data error: is any systematic bias that

occurs during data collection, analysis or interpretation

 Respondent error (e.g., lying, forgetting, etc.),
 Interviewer bias
 Recording errors or Poorly designed questionnaires
3.1 Qualitative Research Design
Qualitative Research Methods
Its aim is to give a complete, detailed descriptions of the

phenomena to be studied
 Objective facts + values

Key philosophical assumption - understanding how people

make sense of their worlds and the experiences people have

Key concern: knowing or understanding from the participants’

perspectives
Key focus: understanding (rather than predicting or
controlling) social settings or social phenomena
In qualitative research, the researcher constructs knowledge

in collaboration with research participants through interaction

and reflection
 Knowledge is considered as a social construct
Cont…
 Tries to include values and motives of the actors in the
Knowledge construction Process
 Focus is to have a deeper understanding of the selected
phenomena in its holistic state
 Qualitative researchers are the primary instrument for data
collection and analysis.
 Mostly words, phrases, sentences and may include visual
images, audio and video recordings.
 Obtained from recordings of interviews, field notes of
observations, and analysis of documents as well as reflective
notes of the researcher/ the researcher’s impressions and
reactions.
 Mass of qualitative data is organised, summarised, described
and interpreted
Cont…
When to choose?
Describe the phenomena

Build a theory

 to gain new insights about a particular phenomena

 develop new concepts or theoretical perspectives
about the phenomena
 discover the problem that exists in the phenomena
Verification: to test the validity of certain assumptions,
claims, theories or generalization with the real world
Evaluation: to evaluate the effectiveness of a
particular policies, design artifacts, programs, etc.
 Define problem

Review literature Conceptual

framework/
Proposal
Collect Data

Data Analysis
Build Theory
Or
Framework
Data Interpretation/
Report Findings
Qualitative Research in IS/IT
 Used to understand user problems for design
such in diagnosing user problems and needs
 Used to collect inputs for artifact development
 Used in artifact evaluation- researchers
qualitatively evaluate a product by interviewing
and observation
 Used to uncover non-technical factors affecting
the adoption and evolution of a new software
product and other IT systems
 Used to develop theories such as HCI theory
 Common Types of Qualitative
 There are many qualitative research methods.
Research
 Case study: an empirical inquiry that investigates a
contemporary phenomenon within its real-life context,
especially when the boundaries between phenomenon and
context are not clearly evident.
 most appropriate for “how” and “why” research questions
 E.g., Why do programmers fail to document their code?
 Action research: the researcher works in close
collaboration with a group of people to improve a situation
in a particular setting.
 Ethnography: emphasis on describing and interpreting
cultural behavior.
 Grounded theory: emphasis on the generation of theory
which is grounded in or emerged from the data.
Qualitative Data Analysis
 Coding is the process of examining the raw qualitative data in the
transcripts and extracting sections of text units (words, phrases,
sentences or paragraphs) and assigning different codes or labels
so that they can easily be retrieved at a later stage for further
comparison and analysis, and the identification of any patterns.
 codes can be based on themes, topics, ideas, concepts,
terms, phrases or keywords which are found in the data
Application of Qualitative - Example
 System Development Research Process that Nunamaker, et al
(1991) proposed five stages or activities
1. construct a conceptual framework,

2. develop a system architecture,

3. analyze and design the system,

4. build the (prototype) system, and

5. observe and evaluate the system.

 The last stage explicitly includes “Develop new theories/
models based on the observation and experimentation of the
system’s usage”
 Reading Assignment:
 Read more on the Qualitative Data Analysis Techniques
3.2 Quantitative Research Methods
 Uses deductive method of knowledge acquisition
 Intends to falsify an existing theory
 Tries to generate generalizable knowledge
 Accepts objectivity of knowledge
 Uses a standard measurement instruments
 The researcher is independent in the process of knowledge construction
 Quantifies the phenomena in terms of numbers
 Intends with prediction and controlling
 Methods are like Laboratory Experiment, Survey
 The purposes of research can be categorized as:
 Descriptive (fact finding)
 Explorative (looking for patterns)
 Analytical (explaining why or how)
 Predictive (forecasting the likelihood of particular events)
Cont…
 Descriptive: seeks to accurately describe current or past
phenomena - to answer such questions as:
 What type of open source software frequently used in Ethiopia?
 What type of computers mostly imported in Ethiopia
 What are the main software tools used by end users?
 Explorative: seeks to understand psychological and
sociological phenomena to find behavioral patterns:
 How do users use the new software system?
 Why would users use the software in this way contrary to original
purpose.
 For example use of Internet for news rather work related matters
 What are the antecedent factors for IS project success?
 Analytical:seeks to explain the reasons behind a particular
occurrence by discovering causal relationships.
Cont…
 Typical questions are:
 Can network signal be improved by ….
 Do user involvement in software development and users IT skills
increase software success?
 Does Hardware performance have an influence on software failure?
 Are road related factors more important in determining accident
severity?
 Predictive: seeks to forecast the likelihood of particular
phenomena occurring in given circumstances.
 Does internet BW predict growth online business services?
 Can the new SW system increases the organization’s efficiency?
 To what extent does new software predict growth of the
organization profit?
 Thiscompares independent (IV) & dependent variable (DV).
 We can predict DV by measuring IV.
Method of Data Collection
 Questionnaire
 Online questionnaire (there are free sites)
 Computer simulation
 Databases (web sites, news …)
 Computer log files – Data like
 Attempts made by Hackers
 Attempts made to break passwords

 Questionnaire Survey
 Can be sent to large number of people, including those who live
thousands of miles away
 Researcher does not have contact with the respondents
 Helps respondent to freely express their idea than they would be in
personal interviews, especially on sensitive and controversial issue
Cont…
 Draw back of questionnaire
 Majority of people don’t return
 Returned responses may not be a true representative of the
originally selected samples; Rely on self report
Development of Questionnaire
 Questionnaires make use of
 Open ended question
 Multiple choice: Eg: What are features of user friendly software
1. Graphical interface
2. Clear navigation direction
3. Immediate feedback
4. Other specify ____________
 Yes / No
 lists and rating scales
 Rankings
 Behaviors and attitudes are complex and cannot be easily
evaluated and quantified
When to use different types of questions
 Open questions should be used when rich qualitative data is
needed that describes the respondent´s perception of their own
experience.
 Multiple choice questions are useful when there is more complexity
in the range of possible responses in discrete categories, but the
range of expected responses is still fairly limited.
 Dichotomous questions are useful in situations where you want to
force respondents to express a clear opinion or as a filter for
determining which subsequent questions are appropriate.
 A rating scale (Likert Scale) is more useful when a behavior,
attitude, or other phenomena of interest needs to be evaluated on
a continuum scale
.
Cont…
 It is designed with the following scales
1. “inadequate” to “excellent”,
2. “never” to “always” or
3. “strongly disapprove” to “strongly approve”
4. “Strongly agree” to “strongly disagree”
5. 5 4 3 2 1 or -2 -1 0 1 2

 Rank ordering can be useful when the relative preference of

respondents is sought from a relatively limited range of options.
Guideline for Questionnaire Design
 Keep it short- information that solve part of the research
problem
 Use simple, clear, unambiguous questions
Cont…
 Check for unwarranted assumptions implicit in your questions
 How many cigarette do you smoke each day?
 Good to add a choice- Ex: >25__ 25-16__ 15-5__ <5__ None __
 Word your questions in ways that don’t give clues about
preferred or more desirable responses
 What strategies have you used to try to quit smoking?
 Leads him to list strategies he did not try
 Check consistency – that leads to give contradictory answer for
two questions
 Determine in advance how you will code the response
 Keep the respondents task simple
 Provide clear instructions
 Make the questionnaire attractive and professional looking
 Conduct a pilot test- helps to validate your questionnaire
Quantitative Data Analysis
 Laboratory experiment
 Using machine learning algorithms (for NLP, DM, IR….)
 Different parameter settings and various experiments
 Statistical - two types
 Descriptive statistics
 Used to describe the phenomena with frequency, mean,
median, mode
 Inferential statistics
 used to infer about the population from the sample data
 T-test to compare means of two groups
 Analysis of Variance (ANOVA)- (for > 2 groups)
 Regression , ……
 More Readings: conducting interviews and questionnaire survey.
 Exercise: Enumerate basic differences of qualitative research
design and quantitative research design?
3.3 Design Research
Process Model
 Design science research (DSR)- a problem-solving paradigm:

seeks to creates novel or innovative IT artifacts that define

the ideas, practices, technical capabilities, and products
through which the analysis, design, implementation, and use
of IT can be effectively and efficiently accomplished.

Fig. 3.2 Design Research Process

Design Research Process
1. Awareness of a problem: recognition of a problem which can be solved
by or using new artifacts.
2. Suggestions: discussing what kind of artifact might solve the problem.
3. Development: designing and creating the artifact.
4. Evaluation: checking whether the artifact solves the problem,
analyzing its strengths and weaknesses.
 evaluated (according to the functional specification implicit or explicit in
the suggestion).
5. Conclusion: compilation of results and future aspects such as open
questions or future plans.
 Artifacts (Design research outputs) in IS Research
 Constructs: conceptual vocabulary of a problem/solution domain such as
formal langs, mathematical like petri nets
 Models: representations of existing or possible real-world systems. a set of
propositions or statements expressing relationships among constructs;
abstractions &
Cont…
 Methods: algorithms and practices to perform a specific
task
 Instantiations: implementation examples of constructs,
models or methods.
 Better theories: artifact construction
 Examples of Design Research
 Provide a unifying framework; Thoroughly explore an area
 Resolve a long-standing question
 Contradict existing knowledge
 Experimentally validate a theory
 Produce an ambitious system; Provide empirical data
 Derive superior algorithms (in IR, Machine learning..
 Develop new methodology (for software develop..)
 Develop a new tool (DM/BI, KBS, Stemmer…
Cont...
Evaluation of Design Research
Quantative and Qualitative evaluation approaches:
Analysis of the built systems
Trials in laboratory

Field trials

Commercial success

Measure of success should be defined before the

implementation
Systems should be evaluated against the defined measures

 Good research results require a careful design of

the research methodology and considerable
evaluation efforts
Exercise
 Discuss a Research design including its meaning and
significance, the features of a good research design. Is single
research design suitable in all research studies? If not, why?
 Discuss the basis of stratification to be employed in sampling
public opinion on inflation.
 Discuss how qualitative and quantitative approaches differ in
terms of various parameters such as
 the nature of reality, relationship of the researcher to the
research participants, the possibility of generalizations, the
possibility of causal linkage, and role of values in research.
 Research Qn: investigate users’ response to an interface;
what would the quantitative and qualitative approach do?
 Discuss Design research vs. Research design
 When does an artifact constitute (design) research?