Chapter 8: Using Nonexperimental Research Are Random Samples Necessary?

! Time Sampling
Conducting Observational Research
! With time sampling, you scan the group for a specific period of time (e.g.,
-The observational research designs described in this chapter are purely
30 seconds) and then record the observed behaviors for the next period (e.g.,
observational in two senses:
another 30 seconds). You alternate between periods of observation and
(1) They are correlational designs and thus do not involve manipulating
recording as long as necessary. Time sampling is most appropriate when
independent variables, and
behavior occurs continuously rather than in short bursts spaced over time and
(2) all use trained researchers to observe subjects’ behaviors.
! Purely observational research has two features
when you are observing large groups of subjects engaged in complex
!Correlational designs that do not involve manipulating independent variables !Use interactions.
trained observers ! Individual Sampling
! With individual sampling, you select a single subject for obser- vation over a
given time period (e.g., 10 minutes) and record his or her behavior. Over
Developing Behavioral Categories
successive time periods, repeat your observations for the other individuals in
- Behavioral categories (also referred to as coding schemes or in animal
the observed group. Individual sampling is most appropriate when you want to
research as ethograms) include both the general and specific classes of
preserve the organization of an individual’s behavior over time rather than
behavior that you are interested in observing.
simply noting how often particular behaviors occur.
Each category must be operationally defined. For example, Underwood et al. (2004)
defined the behavioral categories for their study of social exclusion as follows: Verbal social
exclusion: “Gossiping, planning to exclude the peer, emphasizing the friendship and the ! Event Sampling
peer’s outsider status, and whispering” (p. 1545). Verbal aggression: “Mockery, sarcasm ! In event sampling, you observe only one behavior (e.g., sharing behavior)
and openly critical comments” (p. 1545). Verbal assertion: “Saying ‘shhh!’ to the actor, and record all instances of that behavior.
telling the actor to stop cheating or to stop bragging, or disputing the actor’s comments” ! Event sampling is most useful when you can clearly define one behavior as
(p. 1545). more important than others and focus on that one behavior.
-Developing behavioral categories can be a simple or formidable task. Recording physical
characteristics of the subject is a relatively simple affair. However, when recording social
! Recording
behaviors, defining behavioral categories becomes more difficult. This is because coding
socially based behaviors may involve cultural traditions that are not agreed on (e.g., coding ! Use a recording device to make a record of behavior for later review.
certain speech as “obscene”) You also could use recording devices to make a permanent record of behavior
! A behavioral category includes the general and specific classes of for later analysis.
Recording equipment has several advantages.
behavior to be observed.
First, because you have a permanent record, you can review your subjects’
! Categories must be operationally defined behavior several times, perhaps picking up nuances you might have missed in a
! Developing behavioral categories may be easy or challenging single, live observation.
! Behavioral categories must be clearly defined to avoid confusion Second, you can have multiple observers watch the recorded video
! Two methods can be used to help develop behavioral categories independently and then compare their evaluations of behavior. (Although you can
! Make preliminary observations of behavior use multiple observers for live observations, it may be disruptive to your subjects to have several
observers watching.)
! Conduct a literature search to find research that used behavioral
Finally, you may be able to hide a camera more easily than you can hide
categories
yourself. The hidden camera may be less disruptive to your subjects’ behavior
! Take the time to carefully construct your categories than an observer.
! Adjustments are more easily made before rather than during your Making video recordings of behavior does not eliminate the need to classify the
research
behaviors and to measure such aspects of the behaviors as frequencies and
durations.
Quantifying Behavior in Observational Research
-Direct behavioral observation requires that you develop ways to quantify the Evaluating Interrater Reliability
behaviors under observation. Methods used to quantify behavior in
! Multiple observers are used to increase reliability of observations
observational studies include the frequency method, the duration method, and
the intervals method.
! You must establish reliability of observations from multiple
! Frequency Method. With the frequency method you can record the observers (interrater reliability)
frequency with which a behavior occurs within a time period -Observers invariably differ in how they see and interpret behavior. Something
as simple as a different angle of view can cause a disagreement. Disagreement
! Duration Method. With the duration method, you record how long a
also may arise if you have not clearly defined your behavioral categories.
behavior lasts
Because disagreement is likely to occur to one degree or another, you must
! Intervals Method. With the intervals method, you divide the observation assess interrater reliability, which provides an empirical index of observer
period into several discrete time intervals (e.g., ten 2-minute intervals), and record agreement.
whether a behavior occurs within each interval ! Methods for evaluating interrater reliability
Extra Notes:
! Percent agreement !
-In observational research, you cannot test any hypothesis; you can just form a ! Simplest method
! (total agreements/total observations)*100(%) !
hypothesis from a observation. ! Percent agreement should be around 70%
-A hypothesis can be tested if only if variables can be manipulated.
! Percent agreement may underestimate agreement !

Recording Single Events Versus Behavior Sequences ! Cohen’s Kappa

-Researchers doing observational studies have long recorded single events
occurring within some identifiable observation period.
Bakeman and Gottman advocate looking at behavior sequences rather than at
isolated behavioral events.
As an example, consider an observational study of language development in which you
record the number of times that a parent uses language to correct a child’s behavior.
Although such data may be informative, a better strategy might be to record
those same behaviors sequentially, noting which instances of language use
normally follow one another.
-Recording such behavior sequences provides a more complete picture of
complex social behaviors and the transitions between them.
! Popular method
! Assesses the amount of agreement actually observed relative to the
amount of agreement that would be expected by chance
! A Cohen’s Kappa of .70 indicates acceptable reliability !
! Pearson Product-Moment Correlation
! Correlate ratings of multiple observers with Pearson r
! Simple and easy method to evaluate interrater reliability
! Two sets of scores may correlate highly, but may still differ markedly
We have two variables

! Observational research traditionally involves recording single ! Intraclass Correlation (ICC)

events in an observation ! Extension of Analysis of Variance logic to interrater reliability
! A powerful and flexible tool for evaluating interrater reliability
! Behavior sequences can also be recorded
We have one variabl

Volunteer Bias and Internal Validity
! If multiple observers disagree you must deal with the
disagreement
! A number of methods can be used to handle disagreements
! Average across observers
! Have observers meet to resolve disagreements
! Designate one observer as the “main observer” and the other a
“secondary observer” !
Sources of Bias in Observational Research
! Bias may be a problem in observational research
Observer bias occurs when your observers know the goals of a study or the
hypotheses you are testing and their observations are influenced by this
information.
! A number of potential sources of bias must be considered
! Observer bias occurs when knowledge of goals of research or
hypotheses influence observations
! Using a blind observer avoids observer bias: A blind observer is one who is unaware of
the hypotheses under test.
! Observers interpreting behavior rather than recording what they see.
Another source of bias in observational research arises when observers interpret what they
see rather than simply record behavior. We have all seen nature specials on television in
which a researcher is observing animals in the wild (e.g., chimpanzees). Too often, those
researchers infer intentions behind the behaviors they observe. When one chimp prods
another with a stick, for example, the researcher may record the behavior as a “playful,
mischievous attack.”
Volunteer Bias and External Validity
! Recording quantitative data involves quantifying behavior
numerically (e.g., rate behavior on a scale)
! Recording qualitative data involves making careful notes during
observation periods
!! Either or both types of data can be recorded if necessary !
Nonexperimental Approaches to Data Collection
! Naturalistic Observation: Naturalistic observation involves observing
your subjects in their natural environments without making any attempt to
control or manipulate variables
! Unobtrusive observations of subjects’ naturally occurring behavior
are made
! Advantage: Behavior not tainted by artificial setting
! Disadvantage: Purely descriptive, cannot infer causality
! Ethnography: In ethnography a researcher becomes immersed in the
behavioral or social system being studied. The technique is used primarily to study
and describe the functioning of cultures through a study of social interactions and
expressions between people and groups. You insinuate yourself within a group and
study the social structures and interaction patterns of that group from within.
Your role as a researcher is to make careful observations and record the social
structure of the group that you are studying.
! The researcher becomes immersed in the behavioral or social system
being studied
! A number of factors must be considered
! Gaining entry to field setting
! Gaining entry into group to be studied
! Making yourself “invisible” in the group
! Making observations and recording behavior
! How to analyze data
! Sociometry: Sociometry involves identifying and measuring
interpersonal relationships within a group. Sociometry has been applied to the
systematic study of friendship patterns among children and peer assessments
of teenagers solicited to deal drugs as well as other social networks and work
relationships.
! You identify and measure interpersonal
relationships within a !group
! Members of a group rate each other on
some dimension
! Can be used as a stand alone technique or
one in a wider study
! Construct a sociogram to graphically
display interpersonal relationships
Example of a sociometric scoring sheet ->
! Case History: Your research needs may require you to study in depth a
single case or just a few cases. The case history is a descriptive technique in
which you observe and report on a single case (or a few cases).
! You observe and report on a single case
! Descriptive technique that cannot be used to establish causality.
Although a case history can be useful, it does not qualify as an experimental design. In fact, a case history
is a special application of a demonstration. Because you do not manipulate independent variables, you
cannot determine the causes of the behavior observed in your case history. You can, of course, speculate
about such causes. You can even compare theories by interpreting cases from different perspectives, but
you cannot state with any certainty which perspective is superior.
! Archival Research: Archival research is a nonexperimental strategy
that involves studying existing records. These records can be historical accounts
of events, census data, court records, police crime reports, published research
articles, or any other archived information.
! You use existing records (e.g., police records) as your source of !data
! You should have specific research questions and hypotheses
! Gaining access to records may be difficult or easy, depending on
records needed
! Incomplete records may be a problem
! Like the case history method, archival research is purely descriptive.
You may be able to identify some interesting trends or correlations based
on your archival research. However, you cannot establish causal
relationships.
! Content Analysis: Use content analysis when you want to analyze a
written or spoken record (or other meaningful matter) for the occurrence of specific
categories or events (such as pauses in a speech), items (such as negative comments), or
behavior (such as factual information offered during group discussion).
Because it is difficult to content-analyze such materials in real time, you
normally use archival sources for a content analysis.
! You analyze spoken or written records for the occurrence of specific
categories of events (e.g., a word or phrase)
-Content analysis are invalidated over time and it cannot establish
causal relationships.
Performing in Content Analysis
! Clearly defined response categories are essential
! Two units of analysis
! Recording unit: Element of the material you are going to record !(e.g.,
instances of a certain word)
! Context unit: Context within which material analyzed appears
! Observers doing content analysis must be blind so that bias will
not enter the analysis !
Content Analysis: Defining Characteristics
! Used to analyze a written or spoken record for occurrence of
specific behaviors or events
! Archival sources often used as sources for data
! Appears simple, but may be complex
! Should be used within a clearly developed study, including
hypotheses to be tested
! Response categories must be clearly defined
! A method for quantifying behavior must be defined
! Materials to be analyzed should be chosen carefully to increase
generality
! Cannot be used to establish causal connections among variables
Meta-Analysis
! A meta-analysis applies statistical techniques to evaluate a body of
literature in a research area
! You can combine or compare studies
! Differs from a traditional literature review which involves summarizing
research literature
-A meta-analysis is a set of statistical procedures that allow you to combine
or compare results from different studies. Because you are making use of
existing literature, meta-analysis is a form of archival research. When you
conduct a meta-analysis, you find and analyze existing research (published and
even unpublished) so that you can make statistically guided decisions about the
strength of the observed effects of independent variables and the reliability of
results across studies. You can also do a meta-analysis of existing meta-
analyses. This technique is known as a second-order meta-analysis. In this
technique, you find as many topic- relevant meta-analyses as possible and do a
meta-analysis of their results.
Three steps involved in doing a meta-analysis
! Identifying relevant variables
! Locating research to review
! Conducting the meta-analysis
Factors to Include When Meta-Analyzing Literature
! Full reference citation
! Names and addresses of authors
! Sex of experimenter
! Sex of subjects used in each experiment
! Characteristics of subject sample (e.g., how obtained, number)
! Task required of subjects and other details about the dependent
variable !
! Design of the study (including any unusual features)
! Control groups and procedures included to reduce confoundings
! Results from statistical tests that bear directly on the issue being
considered in the meta-analysis (effect sizes, values of
inferential statistics, p values) !
Drawbacks to Meta-Analysis
! Quality of research varies across different journals !
! Research from different journals may have to be differentially weighted !
! No agreement on how research should be weighted
! Studies in a meta-analysis may used very !different methodologies
!! May not be a serious problem: Different methodologies analogous to
different subjects in a study
! Practical problems
! Incomplete information
! Imprecise information
Meta-Analysis Versus Traditional Review
! Meta-analysis yields different conclusions than traditional review in
some aspects
! More likely to detect differences
! Larger effects of variables detected
! No difference between meta-analysis and !traditional review in
evaluating methodology
! It may be better to base decisions about effects of variables on
statistical analysis than more subjective observations
QUESTIONS
1. What are the defining characteristics of observational research?
The observational research designs described in this chapter are purely
observational in two senses: (1) They are correlational designs and thus do not
involve manipulating independent variables, and (2) all use trained researchers
to observe subjects’ behaviors.
2. How are behavioral categories that are used in observational
research developed?
To develop clear, well-defined categories, begin with a clear idea about the
goals of your study. Clearly defined hypotheses help narrow your behavioral
categories to those that are central to your research questions. Also, keep your
behavioral categories as simple as possible and stay focused on your research
objectives. Avoid the temptation to accomplish too much within a single study.
3. What are the techniques used to make behavioral observations in
observational research?
Methods used to quantify behavior in observational studies include the
frequency method, the duration method, and the intervals method.
4. What is the distinction between recording single acts and behavior
sequences?
Although recording single acts may be informative, a better strategy might be
to record those same behaviors sequentially, noting which instances of
language use normally follow one another. Recording behavior sequences
provides a more complete picture of complex social behaviors and the
transitions between them.
5. What are the sampling techniques used to handle complexity when
making behavioral observations?
Three sampling techniques from which to choose are time sampling, individual
sampling, and event sampling. Recording devices are also useful for observing
behavior.
6. Why should you evaluate interrater reliability?
Because disagreement is likely to occur to one degree or another, you must
assess interrater reliability, which provides an empirical index of observer
agreement.
7. What are the techniques used to evaluate interrater reliability, and
when would each be used?
The simplest way to assess interrater reliability is to evaluate percent
agreement.
A more popular method of assessing interrater reliability than percent
agreement is Cohen’s Kappa. Unlike percent agreement, Cohen’s Kappa (K)
assesses the amount of agreement actually observed relative to the amount of
agreement that would be expected by chance.
Pearson’s product-moment correlation coefficient, or Pearson r, provides a
convenient alternative to Cohen’s Kappa for measuring interrater agreement.
You can use the intraclass correlation coefficient (rI) to assess reliability if your
observations are scaled on an interval or ratio scale of measurement.
8. How do you deal with data from multiple observers?
You can average across observers within each observation period to get a
mean; you can have observers meet and resolve any discrepancies; you can
designate one of the observers as the “main observer” and the other as the
“secondary observer.”
9. What are the sources of bias in observational research, and how
can the bias be reduced?
One source of bias that can easily be avoided is observer bias. Observer bias
occurs when your observers know the goals of a study or the hypotheses you
are testing and their observations are influenced by this information. Use a
blind observer. A blind observer is one who is unaware of the hypotheses under
test.
Another source of bias in observational research arises when observers
interpret what they see rather than simply record behavior. If you have
preserved the actual behavior in your records rather than your interpretation of
the behavior, you can always provide an interpretation later. If new evidence
suggests a different interpretation, you will have the original behavioral
observations available to reinterpret.
10. What is the difference between quantitative and qualitative data?
In qualitative approach, your data will be numbers that can be mathematically
manipulated and analyzed with available descriptive and inferential statistics.
When the data are qualitative, you cannot apply standard descriptive and
inferential statistics to your data.
11. What are the problems inherent in collecting qualitative data?
Usually, there are large amounts of raw data to deal with, and you will need
specialized computer programs to analyze qualitative verbal information.
12. Define naturalistic observation and unobtrusive observation. How
are they used to study behavior?
Naturalistic observation involves observing your subjects in their natural
environments without making any attempt to control or manipulate variables.
Unobtrusive observations are observations that do not alter the natural
behaviors of your subjects. In general, you want to prevent your subjects from
knowing that they are being observed. When it is not possible to hide
yourselves, a widely used technique is to habituate your subjects to your
presence (a fancy way of saying “letting your subjects get used to you”) before
you begin making your observations. You also can make observations
unobtrusively by abandoning direct observations of behavior in favor of indirect
measures.
13. What are some of the advantages and disadvantages of
naturalistic observation?
Naturalistic observation gives you insight into how behavior occurs in the real
world. Properly conducted naturalistic observation has extremely high external
validity. BUT, naturalistic observation can be time consuming and expensive.
Unlike some types of observation in which subjects in effect record their own
data, naturalistic observation requires you to be there, engaged in observation,
during the entire data-collecting period, which may last hours, days, or longer.
Also, getting to the natural habitat of your subjects may not be easy.
16. What is sociometry and when is it used?
Sociometry involves identifying and measuring interpersonal relationships within
a group. Sociometry has been applied to the systematic study of friend- ship
patterns among children and peer assessments of teenagers solicited to deal
drugs, as well as other social networks and work relationships.
17. How are the case history and archival research used?
The case history is a descriptive technique in which you observe and report on
a single case (or a few cases). A case is the object of study.
Archival research is a nonexperimental strategy that involves studying existing
records. These records can be historical accounts of events, census data, court
records, police crime reports, published research articles, or any other archived
information.
Like the case history method, archival research is purely descriptive. You may
be able to identify some interesting trends or correlations based on your
archival research. However, you cannot establish causal relationships.
18. What is content analysis, and what steps are taken when using it?
We use content analysis when you want to analyze a written or spoken record
(or other meaningful matter) for the occurrence of specific categories or events
(such as pauses in a speech), items (such as negative comments), or behavior
(such as factual information offered during group discussion). You must clearly
define your response categories and develop a method for quantifying behavior.
In essence, content analysis is an observational technique.
19. What is meta-analysis and what steps are involved in using it?
A meta-analysis is a set of statistical procedures that allow you to combine or
compare results from different studies. Because you are making use of existing
literature, meta-analysis is a form of archival research. To conduct a meta-
analysis, you must follow three steps: (1) identify relevant variables, (2) locate
relevant research to review, and (3) conduct the meta-analysis proper.
20. What are some of the issues facing you if you decide to do a meta-
analysis?
The issues we may face are about Assessing the Quality of the Research
Reviewed, about Combining and Comparing Studies Using Different Methods.
And there are Practical Problems. There is also a question of whether the
Results of Meta-Analysis Differ From Those of Traditional Reviews or not?

14. What is ethnography and what are the issues facing a field
ethnographer?
In ethnography a researcher becomes immersed in the behavioral or social
system being studied. The problems faced are: Observing as a Participant or
Nonparticipant, Gaining Access to a Field Setting, Gaining Entry Into the Group,
Becoming Invisible, Making Observations and Recording Data, Analyzing
Ethnographic Data.

15. How are ethnographic data recorded and analyzed?

The various recording techniques discussed previously can be applied to
ethnography. You could, for example, make copious notes during critical
interactions. When such overt note taking is not possible (especially if you have
decided to use covert entry into a group), you could instead keep scraps of
paper or index cards and jot down thoughts you will expand later. You also
could use voice-activated audio recorders or other recording devices. Another
strategy involves waiting until the end of the day when you are alone to record
your observations.
The first step in analyzing ethnographic data is to do an initial reading of your
field notes to identify any themes and hypotheses, per- haps with an eye
toward identifying themes and hypotheses overlooked. You also would
systematically extract any major topics, issues, or themes present in your field
records. The second step in analyzing ethnographic data is to code any
systematic patterns in your notes and consider doing an in-depth content
analysis (as discussed later in this chapter).
 SUMMARY
In some situations, conducting an experiment may not be When using the case history approach, you analyze an
possible or desirable. In the early stages of research or when you are interesting case that illustrates some empirical or theoretical point.
interested in studying naturally occur- ring behaviors of your subjects, Alternatively, you may compare and contrast two or more cases in
a nonexperimental approach may be best. order to illustrate such points. Archival research makes use of existing
Observational research involves observing and recording the records. You examine those records and extract data to answer
behaviors of your subjects. This can be accomplished either in the field specific research questions.
or in the lab and can use human participants or animal subjects. Content analysis involves analyzing a written or spoken record
Although observational research sounds easy to con- duct, as much (or other con- tent) for the occurrence of specific categories of events
preparation goes into an observational study as into any other study. or behaviors. As with any observational technique, you must develop
Before making observations of behavior, you must clearly define the behavior categories. During content analysis, you note and analyze
behaviors to be observed, develop observation techniques that do not recording and context units.
interfere with the behaviors of your subjects, and work out a method Meta-analysis is a family of statistical techniques that can help
of quantifying and recording behavior. you evaluate results from a number of studies in a given research
The frequency, duration, and intervals methods are three area. In contrast to a traditional literature review (in which subjective
widely accepted ways to quantify behavior in an observational study. evaluations rule), meta-analysis involves statistically combining the
In the frequency method, you count the number of occurrences of a results from a number of studies. Meta-analytic techniques tend to be
behavior within a specified period of time. In the duration method, you more objective than traditional literature review techniques.
measure how long a behavior lasted. In the intervals method, you The three steps involved in conducting a meta-analysis are
break your observation period into small time intervals and record (1) identifying relevant variables to study,
whether or not a behavior occurred within each. (2) locating relevant research to review, and
After you have decided how to quantify behavior, you must (3) actually doing the meta-analysis (comparing or combining results
make some decisions about how to record your observations. Paper- across studies). Although meta-analysis has advantages over
and-pencil data-recording sheets pro- vide a simple and, in most traditional literature reviews, there are some drawbacks.
cases, adequate means of recording behavior. In some situ- ations First, it is sometimes difficult to evaluate the quality of the research
(such as when the behavior being observed is fast paced), you should reviewed.
consider using electronic recorders rather than a paper-and-pencil Second, studies in a research area may use vastly different methods,
method. Using a recorder allows observers to keep their eyes on making comparison of results suspect.
subjects while making notes about behavior. Third, the information in published articles may be incomplete,
In addition to developing a method for quantifying behavior, eliminating potentially important studies from the analysis.
you must decide on how and when to make observations. Sometimes *
it is not possible to watch and record behaviors simultaneously *
because behavior may occur quickly and be highly complex. In such
situations, you could use time sampling or individual sampling or Sbselman & B.Tanrıverdi
automate your observations by using a video recorder.
*
In observational research, you should use multiple observers.
*
When multiple observers are used, you must evaluate the degree of
*
interrater reliability. This can be done using either percent agreement,
*
Cohen’s Kappa, intraclass correlation, or Pearson r. A Cohen’s Kappa of
*
.70 or greater or a statistically significant Pearson r of around .90 or
*
greater suggests an acceptable level of interrater reliability.
*
Nonexperimental techniques include naturalistic observation,
ethnography, case study, archival research, and content analysis. In *
naturalistic observation, you make careful, unobtrusive observations of *
subjects in their natural environment so that you do not alter their *
natural behavior. In cases in which you cannot remain unobtrusive, *
there are steps you can take to habituate your participants to your *
presence. *
Ethnography involves getting immersed in a behavioral or *
social system to be studied. The technique is best used to study and *
describe the operation of groups and the social interactions that take *
place within those groups. An ethnographic study can be run as a *
participant observation, in which the researcher actually becomes a *
member of the group, or as nonparticipant observation, in which the *
researcher is a nonparticipating observer. *
Sociometry involves identifying and measuring interpersonal *
relationships within a group. Research participants evaluate each other *
along some socially relevant dimension (e.g., friendship), and patterns *
of those ratings are analyzed to characterize the social structure of the *
group. The results of a sociometric analysis can be plotted on a *
sociogram, which graphically represents the social connections *
between participants. Sociometry can be used as a stand-alone *
research technique or as a mea- sure within a wider study.
!