Discussion Guide No.

5
Research Methods

Dr. Noly M. Mascariñas

Professor

RESEARCH DESIGN

Meaning, Purpose, and Principles

1. Research design is the plan, structure, and strategy of investigation conceived so as to (a) obtain answers to
research investigations and (2) control variance.

2. The plan is the overall scheme or program of the research. It includes an outline of what the investigator will do
from writing the hypothesis and their operational implications to the final analysis of data.

3. The structure of the research is more specific. It is the outline, the scheme, the paradigm of the operation of the
variables.

4. The strategy is also more specific than plan. It includes the methods to be used to gather and analyze the data. In
other words, the strategy implies how the research objectives will be reached and how the problems encountered in
the research will be tackled.

5. A research design has two (2) important purposes:

(a) To provide answers to research questions.

 The research design tells us what observations to make, how to make them, and how to analyze the
quantitative representations of the observations.
 An adequate design also outlines possible conclusions to be drawn from the statistical analysis.

(b) To control variance

 The main technical function of research design is to control variance.

 A research design is a set of instructions to the investigator to gather and analyze his data in certain ways.
It is, therefore, a control mechanism.
 The statistical principle behind this mechanism is: “Maximize systematic variance, control systematic
extraneous variance, and minimize error variance.”
 Systematic variance refers to the variance of the dependent variable influenced by the independent
variable.
 Control of extraneous variables means that the influences of independent variables extraneous to the
purposes of the study are minimized, nullified, or isolated.
 Error variance is the variability of measures due to random fluctuations whose basic characteristic is that
they are unpredictable. Two important determinants of error variance are: (1) individual differences among
subjects, and (2) errors of measurement.

6. There are several ways to control extraneous variance, namely:

(a) Eliminate the variable as a variable by choosing subjects so that they are homogenous as possible on that
independent variable. The problem with this approach is that we lose the power of generalization.
(b) Through randomization. If randomization was thoroughly accomplished, then the groups can be considered
statistically equal in all possible ways. This does not mean, however, that the groups are equal in all the
possible variables because by chance the groups can be unequal but the probability of their being equal is
greater than the probability of their not being equal. Hence, randomization is the best way of controlling all
possible extraneous variables.
(c) By building the extraneous variables right into the design of the study as independent variables thereby making
it possible to extract from the total variance of the dependent variable, the variance due to extraneous
variables.
(d) By matching subjects. The variable on which the subjects are matched must be substantially related to the
dependent variable or the matching is a waste of time and can be misleading. The problem with matching is
that it is difficult to find matched subjects on more than two variables.
(e) By statistical control, e.g., analysis of covariance.

Internal and External Validity

1. The research design must be constructed so as to maximize internal and external validity.

2. Internal validity asks the question: “Did, in fact, the independent variables make a difference in the dependent
variable in this specific instance?”
3. External validity asks the question: “To what populations, settings, treatment variables, and measurement variables
can this effect be generalized?”

4. There are eight (8) classes of extraneous variables which, if not controlled in the research design, may produce
effects becoming confounded with the effect of the independent variables:

(a) History, refers to the specific events occurring between the first and the second measurements in addition to
the experimental variables.

(b) Maturation, refers to processes within the respondents operating as a function of the passage of time per se
(not specific to the particular events) including growing older, growing hungrier, growing more tired, and the like.

(c) Testing, refers to the effects of taking a test upon the scores of a second test.

(d) Instrumentation, refers to changes in obtained measurement due to changes in instrument calibration or
changes in the observers or judges.

(e) Statistical regression, refers to the phenomenon operating where group have been selected on the basis of
their extreme scores.

(f) Selection, refers to biases resulting from differential selection of respondents for the comparison groups.

(g) Experimental mortality, refers to the differential loss of respondents from the comparison groups.

(h) Selection-maturation interaction, refers to the interaction effects between the aforementioned variables which
can be mistaken for the effects of the independent variables.

5. There are four (4) jeopardizing factors to external validity (or representativeness), namely:

(a) The reactive or interactive effect of pretesting, in which a pretest might increase or decrease the
respondent’s sensitivity or responsiveness to the experimental variable and thus make the results obtained for
a pre-tested population unrepresentative of the effects of the experimental variable for the un-pretested
universe from which the experimental respondents were selected.

Pretesting modifies the subject in such a way that he responds to the experimental treatment differently than will
unpretested persons in the same population.

(b) The interaction effects of selection biases and the experimental variable.

(c) Reactive effects of experimental arrangements/procedures, which would preclude generalization about the
effect of the experimental variable upon persons being exposed to it in non-experimental settings.

It means effects arising from the experimental setting which will not occur in non-experimental setting.

(d) Multiple-treatment interference refers to effects due to multiple treatments applied to the same subjects
where prior treatments influence subsequent treatments in the series because their effects are not erasable.

Research Designs

1. According to Kidder, the design of the study maybe either any of the following:

(a) Pre-experimental

 One shot case study

 One group pretest/posttest
 Static group comparison

(b) Quasi-experimental

 Interrupted time series

 Regression discontinuity
 Pretest-posttest non-equivalent control group

(c) Experimental design

 Randomized two group

 Before-after two group
 Solomon four group
 Factorial design (e.g. split plot)
 Repeated measure (One unit of analysis and several treatments)

(d) Survey research

  Static group comparison
 Panel design
 Cross-sectional pseudo-panel

(e) Participant Observation

 Complete participation
 Observer as participant
 Participant and observer
 Complete observer

2. There are over 20 possible designs based on the foregoing presentation. But the most commonly used especially
in evaluation researches are presented below. The following notations are used to diagram each design:

“R” -means random assignment

“_ _ _ _” -drawn horizontally, separating the two groups, indicates that the groups not randomly assigned,
that is, the groups are non-equivalent

“O” -indicates measurement of some kind, an observation

“X” -indicates the program to be evaluated

(a) Classical experimental research design

 The classical experimental research design is also called the true control group, pretest-posttest design.
 It is an ideal design for assessing the impact of projects because it effectively controls plausible alternative
causal factors needed to make valid causal inferences of project impact.
 It can be shown using the diagram below:

EG: R O1 X O2
CG: R O1 O2

(b) Pretest-posttest, non-equivalent groups design

 This design differs from the foregoing in that experimental and control groups are not randomly assigned.
 Because of this, the control group is a not a true control group, that is, it is not statistically equivalent to the
experimental group.
 The following diagram illustrates this design:

EG: O1 X O2
CG: O1 O2

(c) Posttest only, non-equivalent groups design

 This design is resorted to when it is not possible to collect information on the experimental and control
groups prior to the introduction of the project.
 In most cases, evaluation is not a built-in component in the project cycle and baseline data were not
collected at the inception of the project.
 The diagram below illustrates this design:

EG: X O1
CG: O2

(d) The true control group, posttest only design

 The true control group posttest only design is the same as posttest only, non-equivalent groups design only
that it is randomized.
 Using this design in evaluating development projects requires an identical entity or unit as control group.
 This can be illustrated through the following:

EG: R X O1
CG: R O2

(e) Before-and-after design

 This is one of the simplest designs for establishing causal attribution between two or more variables.
 Only the experimental group is observed before and after the intervention is introduced. Hence,
establishing causal attribution faces some difficulty.
 The before-and-after design can be shown in a diagram through the following:

EG: O1 X O2
 (f) Posttest only, one group design

 In this design, only the experimental group is observed but only after exposure to the project. To be able to
make inferences regarding project impact, measured outcome indicators are compared with some
comparable areas or groups using secondary data.
 The following diagram will illustrate this:

CG: X O2

(g) The single group time series design

 A number of observations or measurements are done prior to and after exposure to the
treatment/intervention. This enables the evaluators to validate their findings. The following diagram shows
the longitudinal character of this design.

EG: O1 O2 O3 X O4 O5 O6

Basic Research Methods

1. True Experimental Research

(a) The purpose of true experimental research is to investigate possible cause-and-effect relationships by
exposing one or more experimental groups to one or more treatment conditions and comparing the results to
one or more control groups not receiving the treatment.

(b) Important characteristics include:

 Rigorous management/control of the independent and dependent variables.

 Use of a control group as a baseline against which to compare the groups receiving the experimental
treatment.
 Random selection of subjects, random assignment of subjects to groups, and random assignment of
experimental treatments to groups.

(c) All variables of concern are held constant except the treatment variables are deliberately manipulated or
allowed to vary. Thus, the internal validity of experimental design is high.

(d) Its major disadvantages relate to external validity, namely, artificial nature of the experimental setting, demand
characteristics, and evaluation apprehension.

Demand characteristics- refer to the fact that the experimental setting may evoke
certain demands--- that is, expectations on the part of the subjects to act in
the way they think the experimenter would wish.

Evaluation apprehension- refers to the concern that a subject has about being
observed and judged while in the laboratory setting. That is, subjects may
try to present themselves in a favorable light.

2. Field Experiment

(a) In contrast to the laboratory experiment, the setting of the field experiment is a natural one and the subjects are
not generally aware that they are subjects in an experiment.

(b) The researcher who uses a field experiment to test a hypothesis is looking at behavior in its natural setting.

(c) The advantages of using a field experiment include the following:

 By focusing on behavior in a natural setting, the experimenter can be much more certain of the external
validity of his findings.
 Problems of the artificiality of the experimental setting, demand characteristics, and evaluation
apprehension are eliminated.

(d) The problems associated with this research method are as follow:

 Because the researcher is working in a complex natural setting where many events may occur
simultaneously, he must be sure that the independent variable is sufficiently visible for potential subjects to
have the opportunity to react. Otherwise, a failure to respond becomes difficult to interpret.
  The dependent variable must be selected carefully. The researcher must be able to readily observe and
reliably judge the dependent variable behavior.
 Ethical concern. Is it reasonable for the investigator to involve individuals in an experiment without their
knowledge or permission?
 Researcher has no control over majority of events in the environment. Unexpected events may reduce or
destroy the effectiveness of the manipulation.

(e) Example: In 1968, more than 4,000 passengers on the IND subway in New York City saw another passenger
fall on the floor of the train. The passengers did not know it that they were subjects in a field experiment
designed to assess helping behavior (Piliavin et al, 1969). The experimenters arranged the situation so that the
characteristics of the person needing help would vary: he was either black or white, and was either apparently
drunk or slightly disabled. After the person “fell”, the experimenters, who were also present in the subway car,
noted the number of people who offered to help, their race, and the speed of the helping response.

3. Quasi-Experimental Research

(a) The purpose of this method is to approximate the conditions of the true experiment in a setting which does not
allow the control and/or manipulation of all relevant variables.

(b) This research is characterized by methods of partial control based on a careful identification of factors
influencing both internal and external validity.

(c) The strengths of quasi-experimental research is that it allows the study of very strong variables that cannot be
manipulated or controlled by the experimenter. They also have high external validity.

(d) The weaknesses of the design have to do with the lack of control over the independent variables and the
absence of random assignment of subjects.

4. Ex-Post Facto Research

(a) It is a systematic empirical inquiry in which the scientist does not have direct control of independent variables
because their manifestations have already occurred or because they are inherently not manipulable.

(b) Inferences about relations among variables are made without direct intervention from concomitant variation of
independent and dependent variables.

(c) It involves the investigation of possible cause-and-effect relationships by observing some existing
consequences and searching back through the data for plausible causal factors. This is in contrast to
experimental design which collect its data under controlled conditions in the present.

(d) The weaknesses of ex-post facto research include the following:

 Lack of control over independent variables. Within the limits of selection, the investigator must take the
facts as he finds them with no opportunity to arrange the conditions or manipulate the variables that
influenced the facts in the first place.
 The difficulty in being certain that the relevant causative factor is actually included among the many
factors under study.
 The complication that no single factor is the cause of an outcome but some combination and interaction
of factors go together under certain conditions to yield a given outcome.
 A phenomenon may result not only from multiple causes but also from one cause in one instance and
from another cause in another instance.
 When a relationship between two variables is discovered, determining which is the cause and which is
the effect maybe difficult.
 The fact that two or more factors are related does not necessarily imply a cause-and-effect relationship.
They all simply maybe related to an additional factor not recognized or observed.

5. Survey Research

(a) It is a research method which studies large and small populations by selecting and studying samples chosen
from the populations to discover the relative incidence, distribution, and interrelations of sociological and
psychological variables.

(b) Oftentimes called sample survey.

(c) There are five basic types of surveys based on the method of obtaining information, namely:

 Personal interview is the most powerful of the five types. Interview schedules are difficult to construct,
they are time consuming, relatively costly, but there is no other method that yield the information the way
they do.
 Mail questionnaire has serious drawbacks, namely, the possible lack of response, the inability to check
the responses given, and generally poor returns, which is why it should not be used if a better method can
possibly be used and if it is to be used, it should be used in conjunctions with other techniques.
  Panel survey interviews a sample of respondents and then re-interviewed and studied at later times,
thereby, enabling the researcher to study changes in behavior and attitudes.
 Telephone surveys have little to recommend them beyond speed and low cost.
 Controlled observation

(d) The advantages of survey research are as follow:

 It has the advantage of wide scope: a great deal of information can be obtained from a large population.
 It allows for very specific formulation of issues of concern.
 Survey research information is accurate--within sampling error, of course.

(e) The disadvantages of survey research are as follow:

 Survey information ordinarily does not penetrate very deeply below the surface.
 It is demanding of time and money.
 Interviews are susceptible to interviewer bias.
 The survey interview can temporarily lift the respondent out of his own social context, which may make
the results of the survey invalid.
 It requires a good deal of research expertise. The researcher must know sampling, question and
schedule construction, interviewing, data analysis, and other technical aspects of the survey.

6. Case and Field Study

(a) The purpose of case and field study research is to study intensively the background, current status, and
environmental interactions of a given social unit---an individual, a group, an institution, or a community.

(b) Case studies are in-depth investigations of a given social unit resulting in a complete, well-organized picture of
that unit. Depending upon the purpose, the scope of the study may concentrate on specific factors or take in
the totality of elements and events.

(c) Compared to a survey study which tends to examine a small number of variables across a large sample of
units, the case study tends to examine a small number of units across a large number of variables and
conditions.

(d) Case studies are particularly useful as background information for planning major investigations in the social
sciences. Because they are intensive, they bring to light the important variables, processes, and interactions
that deserve more intensive attention. They pioneer new ground and are often the source of fruitful hypotheses
for further study.

(e) Case study data provide useful anecdotes or examples to illustrate more generalized statistical findings.

(f) Because of their narrow focus on a few units, case studies are limited in their representativeness and external
validity.

(g) Case studies are particularly vulnerable to subjective biases. The case itself maybe selected because of its
dramatic, rather than typical attributes, or because it neatly fits the researchers’ preconceptions. Subjective
interpretation tends to influence also the results of the study.

7. Archival Research

(a) It refers to the analysis of any existing records that have been produced or maintained by persons or
organizations other than the researcher. In other words, the original reason for collecting the records was not
that of the researcher’s.

(b) Archival research has two important advantages:

 It allows the researcher to test hypothesis over a wider range of time and societies than would
otherwise be possible. Demonstrating the validity of a hypothesis in a number of different cultures and
historical periods gives us considerable confidence in the validity of the hypothesis.
 It uses “unobtrusive measures”. Because the information used in archival research was originally
collected for some other purposes, there is little or no chance that demand characteristics or evaluation
apprehension will be problems for the present researcher.

(c) The major disadvantage of this method has to with difficulties associated with data availability. The researcher
is often to the mercy of those who collected the data. Sometimes, of course, creativity and ingenuity will help
the researcher to locate the kinds of data needed. In other cases, missing or inaccurate records will prevent an
adequate test of the researcher’s hypothesis. Even if the material is available, it is sometimes difficult to
categorize it in the way necessary to answer the research question. Careful methods of coding, content
analysis, and the use of complicated statistical analysis are often needed.