Professional Documents
Culture Documents
Minggu 3 - Research Design in Qualitative Quantitative Mixed Method Bahan Bacaan
Minggu 3 - Research Design in Qualitative Quantitative Mixed Method Bahan Bacaan
Research Design in
Qualitative/Quantitative/
Mixed Methods
M ichael R. H arwell
University of Minnesota
147
148 Section III. Opportunities and Challenges in Designing and Conducting Inquiry
believed to be worth studying are described. I use analyze qualitative data; quantitative studies are,
two studies to illustrate strengths in the research among other things, studies that collect and
design, as well as opportunities to enhance the analyze quantitative data; and so on.
results, and give greater attention to mixed meth- Crotty (1998) described four key features to
ods because of their relative newness and poten- consider in research design: the epistemology
tial. Three ways that researchers can enhance the that informs the research, the philosophical
ability of research designs to better support the stance underlying the methodology in question
study of promising ideas in educational studies (e.g., post-positivism, constructivism, pragma-
are described. I conclude by arguing that mixed tism, advocacy/participatory; see Morgan, 2007),
methods offer an especially promising path toward the methodology itself, and the techniques and
using research design in ways that support rigor- procedures used in the research design to collect
ous inquiry. data. These features inform the descriptions of
research designs below.
(pp. 39–40). Thus, replicability and generalizabil- interviews and media collected during the author’s
ity are not generally goals of qualitative research. visits to India in ways that took into account his
Qualitative research methods are also described identity; Harry, Sturges, and Klingner (2005),
as inductive, in the sense that a researcher may who used the methods of grounded theory to
construct theories or hypotheses, explanations, develop a theory providing a new perspective on
and conceptualizations from details provided by a ethnic representation in special education; Brown
participant. Embedded in this approach is the (2009), who studied the perspectives of university
perspective that researchers cannot set aside their students identified as learning disabled; and
experiences, perceptions, and biases, and thus Chubbuck and Zembylas (2008), who examined
cannot pretend to be objective bystanders to the the emotional perspective and teaching practices
research. Another important characteristic is that of a White novice teacher at an urban school.
the widespread use of qualitative methods in edu- These studies reflect several important fea-
cation is relatively new, dating mostly to the 1980s, tures of qualitative research, including a focus
with ongoing developments in methodology and on discovering and understanding the experi-
reporting guidelines (Denzin, 2006). The relative ences, perspectives, and thoughts of participants
newness of this methodology also means that through various strategies of inquiry. The stud-
professional norms impacting research, includ- ies were also conducted in naturalistic settings in
ing evidence standards, funding issues, and edi- which inquiry was flexible and guided by par-
torial practices, are evolving (see, e.g., Cheek, ticipants’ comments, which in some instances
2005; Freeman, deMarrais, Preissle, Roulston, & were used to construct explanations of their
St.Pierre, 2007). Good descriptions of qualitative views and perspectives. An important feature of
methods appear in Bogdan and Biklen (2003), several of these studies is their use of elements of
Creswell (1998), Denzin and Lincoln (2005), different strategies of inquiry.
Miles and Huberman (1994), and Patton (2002).
There are several categorizations of research
Quantitative Research Methods
designs in qualitative research, and none is uni-
versally agreed upon (see, e.g., Denzin & Lincoln, Quantitative research methods attempt to
2005). Creswell (2003) listed five strategies of maximize objectivity, replicability, and general-
inquiry in qualitative research that I treat as syn- izibility of findings, and are typically interested in
onymous with research design: narratives, phe- prediction. Integral to this approach is the expec-
nomenological studies, grounded theory studies, tation that a researcher will set aside his or her
ethnographies, and case studies. Creswell also experiences, perceptions, and biases to ensure
described six phases embedded in each research objectivity in the conduct of the study and the
design that are more specific than those suggested conclusions that are drawn. Key features of many
by Crotty (1998), but still encompass virtually all quantitative studies are the use of instruments
aspects of a study: (1) philosophical or theoretical such as tests or surveys to collect data, and reli-
perspectives; (2) introduction to a study, which ance on probability theory to test statistical
includes the purpose and research questions; hypotheses that correspond to research questions
(3) data collection; (4) data analysis; (5) report writ- of interest. Quantitative methods are frequently
ing; and (6) standards of quality and verification. described as deductive in nature, in the sense that
Journals that publish qualitative methodology inferences from tests of statistical hypotheses lead
papers and qualitative research studies in educa- to general inferences about characteristics of a
tion include Qualitative Research, Qualitative population. Quantitative methods are also fre-
Inquiry, Field Methods, American Educational quently characterized as assuming that there is a
Research Journal, Educational Researcher, and the single “truth” that exists, independent of human
International Journal of Qualitative Studies in perception (Lincoln & Guba, 1985).
Education. Examples of the use of qualitative Trochim and Land (1982) defined quantitative
research designs are provided by Stage and Maple research design as the
(1996), who used a narrative design to describe
the experiences of women who earned a bache- glue that holds the research project together. A
lor’s or master’s degree in mathematics and opted design is used to structure the research, to show
to earn a doctorate in education; Gaines how all of the major parts of the research
(2005), who explored the process of interpreting project—the samples or groups, measures,
150 Section III. Opportunities and Challenges in Designing and Conducting Inquiry
example, some authors insist that a mixed meth- However, the pragmatic stance also has its crit-
ods study is any study with both qualitative and ics (Mertens, 2003; Sale et al., 2002). Dialectical
quantitative data, whereas other authors say a researchers argue that multiple paradigms are
mixed methods study must have a mixed meth- compatible and should be used, but their differ-
ods question, both qualitative and quantitative ences and the implications for research must be
analyses, and integrated inferences (Tashakkori, made clear (Greene & Caracelli, 1997). It is impor-
2009). There is also disagreement regarding tant to emphasize that a researcher who adopts a
various aspects of mixed methods, such as when dialectic stance would, other things being equal,
mixing should occur (e.g., at the point of design- draw on the same procedures for mixing as one
ing a study, during data collection, during data adopting a pragmatic stance. The issue of stances
analyses, and/or at the point of interpretation). is certainly not settled, and additional develop-
Still other authors have criticized the whole ments on this topic continue to appear in the
idea of mixed methods (Denzin, 2006; Sale, mixed methods literature (e.g., design stance of
Lohfeld, & Brazil, 2002; Smith & Hodkinson, Creswell & Plano Clark, 2007).
2005), criticism which is sometimes framed in Mixed methods designs seem especially firmly
terms of the response of advocates of a particu- rooted in the evaluation literature. An early
lar “stance” to arguments for mixing methods: important paper in this area was Greene, Caracelli,
the purist stance, the pragmatic stance, and the and Graham (1989), which highlighted five major
dialectical stance (Greene & Caracelli, 1997; purposes of (or justifications for) a mixed meth-
Johnson & Onwuegbuzie, 2004; Lawrenz & ods evaluation. One is triangulation, which exam-
Huffman, 2002). Those adopting a purist stance ines the consistency of findings, such as those
argue that mixed methods are inappropriate obtained through different instruments, and
because of the incompatibility of the worldview which might include interviews and surveys.
or belief system (paradigms) (Tashakkori & According to Green et al., triangulation improves
Teddlie, 2003) underlying qualitative and quan- the chances that threats to inferences will be con-
titative methods, i.e., qualitative and quantita- trolled. A second purpose is complementarity,
tive methods are studying different phenomena which uses qualitative and quantitative data
with different methods (Smith & Hodkinson, results to assess overlapping but distinct facets of
2005). Some purists have also raised concerns the phenomenon under study (e.g., in-class
that mixed methods designs leave qualitative observations, surveys), and a third is develop-
methods in the position of being secondary to ment, in which results from one method influ-
quantitative methods (Denzin, 2006; Giddings, ence subsequent methods or steps in the research;
2006; Yin, 2006). for example, interviews with teachers might sug-
Researchers who adopt a pragmatic stance gest that an additional end-of-year assessment be
argue that paradigm differences are independent added. A fourth purpose of a mixed methods
of, and hence can be used in conjunction with, evaluation is initiation, in which results from one
one another in the service of addressing a ques- method challenge other results or stimulate new
tion (Johnson & Onwuegbuzie, 2004; Morgan, directions for the research; for example, teacher
2007). Wheeldon (2010) summarizes this view: interviews might challenge results provided by
administrators in a school district. The fifth and
Instead of relying on deductive reasoning and last purpose is expansion, which may clarify
general premises to reach specific conclusions, or results or add richness to the findings.
inductive approaches that seek general A number of frameworks for mixed meth-
conclusions based on specific premises, ods have appeared in this literature, many of
pragmatism allows for a more flexible abductive which have built on the work of Greene et al.
approach. By focusing on solving practical (1989) (Caracelli & Greene, 1997; Creswell,
problems, the debate about the existence of 2003; Johnson & Onwuegbuzie, 2004; Lawrenz
objective “truth,” or the value of subjective & Huffman, 2002; Leech & Onwuegbuzie, 2007;
perceptions, can be usefully sidestepped. As such, Morse, 2010; Newman, Ridenour, Newman, &
pragmatists have no problem with asserting both DeMarco, 2003; Tashakkori & Teddlie, 2003).
that there is a single “real world” and that all These frameworks differ in many ways, but
individuals have their own unique interpretations they all successfully convey a sense of the large
of that world. (p. 88) number of methodological tools available to
Chapter 10. Research Design in Qualitative/Quantitative/Mixed Methods 153
researchers. However, none of these frameworks collection phases (as opposed to only collecting
has been widely adopted. qualitative or quantitative data) and the expertise
The framework of Creswell (2003) is used here needed to integrate the qualitative and quantita-
to categorize research designs in mixed methods, tive findings. Morgan (1998) suggested that the
but this choice should not be interpreted to mean sequential explanatory design is the most fre-
that it has been widely endorsed by mixed meth- quently used mixed methods approach.
ods theoreticians and practitioners. This frame- As an example, consider the quantitative study
work is described next, and studies by Howell of Howell et al. (2002). These authors were inter-
et al. (2002) and Norman (2008) are used to ested in learning if students who received vouchers
illustrate strengths in their research design, as allowing them to enroll in a private school (and
well as opportunities to enhance their results by who subsequently enrolled) showed improved
employing a mixed methods approach. In doing achievement compared to students who did not
so, I assume that different paradigms can be receive a voucher and attended a public school.
matched to answer research questions of interest, To explore this question, these authors examined
and thus I adopt a pragmatic stance. It is impor- achievement data for more than 4,000 students in
tant to emphasize that there are many complexi- three U.S. cities in which vouchers were randomly
ties not captured by these examples, including awarded via a lottery. This arrangement produced
the nature and timing of the mixing and the use a randomized design in which receiving a voucher
of elements from multiple mixed methods (yes, no) was the key independent variable, and
approaches in a given research design. student achievement was the key dependent vari-
Creswell (2003) described six somewhat over- able. No theoretical model was offered. The
lapping mixed methods research designs, referred results showed an average increase in the achieve-
to as strategies of inquiry, that guide the con- ment of African American students who received
struction of specific features of a mixed methods a voucher and subsequently enrolled in a private
study. The designs vary in whether qualitative school compared to African American students
and quantitative data are collected sequentially who did not receive a voucher, but not for any
or concurrently, the weight given one kind of other groups in the study. From a quantitative
data or another, when the mixing is done, and perspective, this study has many strengths, par-
the extent to which a theoretical perspective (e.g., ticularly random assignment, that enhance causal
post-positivism, constructivism) is present and arguments.
guides the research design. Howell et al. (2002) also commented that the
The first mixed methods approach described reasons vouchers improved the academic perfor-
in Creswell (2003) is the sequential explanatory mance of African American students and not
design, in which qualitative data are used to other students were unclear. Is this difference
enhance, complement, and in some cases follow related to a student’s view of his or her academic
up on unexpected quantitative findings. In this skills, peer related, due to motivation differences
approach, the focus is on interpreting and or strong parental support for education, the
explaining relationships among variables and student’s interactions with school staff, or other
may or may not be guided by a particular theo- factors? The Howell et al. comment suggests that
retical perspective. Quantitative data are collected a sequential explanatory research design would
and analyzed first, followed by the collection and allow these authors to search for explanations of
analysis of qualitative data, meaning that qualita- the finding that vouchers improved the academic
tive and quantitative data are not combined performance of African American students but
(mixed) in the data analysis; rather, integration not other students. For example, collecting qual-
takes place when the findings are interpreted. In itative data at the end of the study through inter-
general, results are interpreted in ways that views of a purposively sampled group of parents
usually give more weight to the quantitative of students who had and had not received a
component. The separate phases of design, data voucher could be helpful in enhancing and
collection, and reporting for qualitative and complementing the quantitative findings.
quantitative data are considered strengths, A second mixed methods approach, the
because this arrangement is relatively easy to sequential exploratory design, is essentially the
implement. The weaknesses of this approach are reverse of the sequential explanatory design,
the time and resources needed for separate data with quantitative data used to enhance and
154 Section III. Opportunities and Challenges in Designing and Conducting Inquiry
complement qualitative results. This approach is process of how academic counselors make deci-
especially useful when the researcher’s interest is sions, and used this model and a case study
in enhancing generalizability, and it may or may approach (Creswell, 1998). Norman purposively
not be guided by a theoretical perspective. sampled 6 counselors and 24 students and col-
Creswell (2003) pointed out that instrument lected data via in-depth interviews with the
construction is an example of this approach, in counselors and students that included their
that a draft of an instrument (survey, test) is observations of the advising process, and high
piloted with a small number of cases who often school and college records of students’ mathe-
provide important qualitative feedback about matics course taking and grades. Norman
their experience with the instrument, followed, reported that the counselors generally misinter-
after appropriate modifications of the instru- preted the mathematics portions of high school
ment, by using the instrument to collect quanti- transcripts, which had important implications
tative data. The quantitative results are then for advising a student on which college mathe-
used to enhance, complement, and possibly matics course to begin with. For example, a student
extend the earlier pilot results. The strengths whose transcript indicated that his or her highest
and weaknesses of this approach are similar to completed high school mathematics course was
those of the sequential explanatory design. “Integrated Mathematics IV,” a standards-based
As an example, consider the mixed methods course, was generally advised to start with a
study of Norman (2008). Norman was interested precalculus mathematics course, even though
in uncovering the perspectives of college aca- Integrated Mathematics IV is a precalculus
demic counselors when advising newly admitted course and the student should have been advised
students on the appropriate mathematics course to enroll in Calculus I. Norman also reported
with which to begin their college mathematics that counselors who looked at transcripts of
career, and integrating this information with students who completed a traditional high
quantitative information reflecting the high school mathematics curriculum, in which the
school mathematics curriculum students com- highest mathematics course completed was
pleted and their scores on a college mathematics listed as precalculus, were more likely to recom-
placement test. Norman’s work was motivated by mend that a student enroll in Calculus I. Norman
anecdotal evidence that students who completed suggested that counselor views toward standards-
a standards-based mathematics curriculum in based curricula may be related to working in a
high school were more likely to be advised to mathematics department, because mathematics
begin college with a less difficult mathematics departments have generally been quite critical of
course than comparable students who completed standards-based curricula (Roitman, 1999;
a traditional high school mathematics curricu- Schoenfeld, 2004).
lum. Standards-based curricula in high school Norman (2008) also collected and analyzed
focus on a variety of mathematics topics every quantitative data for a sample of more than
school year in a way that emphasizes problem 1,000 college freshmen that included informa-
solving and small-group work, and de-emphasizes tion on the high school mathematics curricu-
algorithmic manipulation. Traditional curricula lum they completed; their score on a college
typically focus on algebra, algorithms, and repe- mathematics placement exam; and the difficulty
tition and depend heavily on the teacher for of their first college mathematics course, which
student learning (Schoenfeld, 2004). was captured using a 4-point Likert variable
Differential advising of students as a func- (1 = a course that should have been completed
tion of the high school mathematics curricu- in high school, which is sometimes referred to as
lum they completed implies that some students a developmental course; 4 = a course whose dif-
are advised to begin their college mathematics ficulty exceeded that of Calculus I). The results
with a course they should not take (e.g., precal- of these analyses suggested that the curriculum
culus rather than Calculus I). The implication a student completed was related to his or her
of this practice is that students and colleges mathematics placement score and the difficulty
may be spending time, money, and effort on level of the student’s first college mathematics
unneeded courses. course. In particular, students who completed a
Norman (2008) adapted a theoretical decision- standards-based high school mathematics curric-
making model (Miller & Miller, 2005) to the ulum were more likely to enroll in a less difficult
Chapter 10. Research Design in Qualitative/Quantitative/Mixed Methods 155
college mathematics course compared to students with it. Here, the study could focus on discover-
who completed a traditional curriculum. ing and understanding students’ experiences with
Norman’s (2008) study reflects several exem- the advising process and its implications for their
plary features of mixed methods research, includ- college experience. A case study approach, using a
ing using a case study approach that focused on purposively chosen sample of students who
discovering and understanding the experiences, began their college mathematics course taking at
perspectives, and thoughts of counselors and stu- different difficulty levels, would be appropriate.
dents, and the presence of quantitative data high- Information obtained from interviews and stu-
lighting the possible consequences of counselors’ dent academic records could be used to inform
misunderstanding the information on a student’s the construction of a survey to be sent to a repre-
high school transcript. While the exact timing of sentative sample of students. The survey results
the quantitative and qualitative data collection in could be used to improve decision making by
Norman’s study was unclear, it appears the data counselors and to enhance generalizability.
were collected relatively close in time, suggesting A fourth mixed methods approach is the con-
that the results of one data collection did not current triangulation design, which is used when
influence the other data collection. However, the the focus is on confirming, cross-validating, or
main question focused on counselors, and col- corroborating findings from a single study.
lecting and analyzing data for counselors first, Qualitative and quantitative data are collected
intentionally using this information to guide the concurrently, such that weaknesses of one kind of
collection and analysis of quantitative data for data are ideally offset by strengths of the other
students, and integrating the two sets of findings kind. Typically, equal weight is given to the two
suggest that a sequential exploratory design kinds of data in mixing the findings, although
would have been of value. For example, informa- one kind of data can be weighted more heavily.
tion obtained from the qualitative data about The qualitative and quantitative data are analyzed
counselors’ misunderstanding of high school separately, and mixing takes place when the find-
mathematics course taking could have been used ings are interpreted. Important strengths of this
in developing questions for a student survey, such approach are the ability to maximize the infor-
as on whether students thought their counselor mation provided by a single study, for example,
misunderstood their high school course taking. when interest is in cross-validation, and a shorter
A third mixed methods approach is the data collection period compared to the sequential
sequential transformative design, in which either data collection approaches. Important weak-
qualitative or quantitative data may be collected nesses include the additional complexity associ-
first. Here, the theoretical perspective underly- ated with collecting qualitative and quantitative
ing the methodology is critical to the conduct of data at the same time and the expertise needed to
the study, and the chosen methods should serve usefully apply both methods. Discrepancies
the theoretical perspective. Once again, qualita- between the qualitative and quantitative findings
tive and quantitative data are analyzed sepa- may also be difficult to reconcile.
rately, and the findings are integrated during the In the Howell et al. (2002) study, the primary
interpretation phase. This approach is often finding was that the achievement of African
used to ensure that the views and perspectives of American students who received a voucher to
a diverse range of participants are represented or attend private school was on average higher than
when a deeper understanding of a process that is that of African American students who did not
changing as a result of being studied is sought. receive a voucher, and that this difference did not
Its strengths and weaknesses are similar to those emerge for other student groups. Adopting a con-
of the sequential explanatory design. current triangulation design could provide an
For example, the decision-making model used explanation for these findings by collecting quali-
by Norman (2008) suggested that collecting data tative data in the form of interviews with parents
for students who are affected (sometimes adversely) of students who did and did not receive a voucher,
by the recommendations provided by counselors and quantitative data in the form of student test
is important in evaluating and improving the scores and background information. This would
advising process. Norman commented that most offer an opportunity to corroborate findings
students follow the recommendation of which from this study with respect to the improved
mathematics course to take, even if they disagree achievement of African American students but
156 Section III. Opportunities and Challenges in Designing and Conducting Inquiry
not other students. For example, a plausible out- A second data-mixing strategy described by
come of concurrent data collection is that the Caracelli and Greene (1993) is typology develop-
qualitative data suggest that parents of African ment, in which the analysis of one kind of data
American students appeared to be more commit- produces a typology or set of categories that is
ted to, and enthusiastic about, their students’ used as a framework in analyzing the other kind
education in general and the voucher program in of data. In Norman’s (2008) study, analyses of the
particular, than parents of other students, and qualitative data could produce themes that allow
that this enthusiasm persisted throughout the a variable with nominally scaled categories to be
school year (Achievement tests used in this study developed, in which the categories provide an
were given at the end of the school year.). In this explanation of why participants became counsel-
instance, the qualitative and quantitative infor- ors. This variable could then be used in the quan-
mation would provide corroborating evidence titative analysis.
that the improved achievement of African A third data-mixing strategy is extreme case
American students could be attributed in part to analysis, in which extreme cases identified with
receiving a voucher and enrolling in a private one kind of data are examined with the other
school, and in part to the support, encourage- kind, with the goal of explaining why these cases
ment, and motivation of students’ parents. are extreme. For example, multilevel analyses of
The fifth approach is the concurrent nested quantitative data in Norman’s (2008) study may
design, in which qualitative and quantitative suggest that some counselors are, with respect to
data are collected concurrently and analyzed the sample of counselors, statistical outliers (e.g.,
together during the analysis phase. Greater students linked to these counselors dispropor-
weight is given to one kind of data, in the sense tionately began their college mathematics study
that one kind of data is typically embedded in with courses that should have been completed in
the other. However, there may or may not be a high school). Qualitative data could be used to try
guiding theoretical perspective. A popular appli- to explain why these counselors appeared to dis-
cation of this approach is with multilevel struc- proportionately steer students to developmental
tures (Tashakkori & Teddlie, 2003), in which college mathematics courses.
different levels or units of an organization are The fourth data-mixing strategy described by
studied. Strengths of this approach include the Caracelli and Greene (1993) is data consolidation/
shorter data collection period and the multiple merging, in which a careful review of both
perspectives embedded in the data, whereas kinds of data leads to the creation of new vari-
weaknesses include the level of expertise needed ables or data sets expressed in a qualitative or
to execute the study successfully, especially in quantitative metric. The merged data are then
mixing the qualitative and quantitative data used in additional analyses. In Norman’s (2008)
within the data analysis, and difficulties in rec- study, a review of the qualitative and quantita-
onciling conflicting results from the qualitative tive data may suggest new variables. For exam-
and quantitative analyses. ple, a review of the counselor and student data
In this design, qualitative and quantitative data may suggest constructing a variable capturing
are mixed in the analysis phase, a process that can the extent to which students assert themselves in
take many different forms (see, e.g., Bazeley, 2009; the advising process. Examples of data mixing
Tashakkori & Teddlie, 2003). Caracelli and Greene appear in Caracelli and Greene (1993); Sandelowski,
(1993) described four strategies to mix qualitative Voils, and Knafl (2009); and Tashakkori and
and quantitative data in the analysis. One is data Teddlie (2003).
transformation, in which qualitative data are Norman’s (2008) study seems ready-made for
transformed to quantitative data or qualitative a concurrent nested design, given the inclusion of
data are transformed into narrative, and the both counselors and students. A key outcome in
resulting data are analyzed. In Norman’s (2008) this study was the difficulty level of a student’s first
study, this could involve transforming (i.e., rescal college mathematics course. Quantitative evi-
ing) qualitative data in the form of interviews, dence of a relationship between a student’s high
field notes, and so on to a quantitative form that school mathematics curriculum (standards-based,
captures key themes in these data. Typically, the traditional) and the difficulty level of his or her
transformed qualitative data exhibit a nominal or first college mathematics course could be mixed
ordinal scale of measurement. with qualitative data obtained concurrently from
Chapter 10. Research Design in Qualitative/Quantitative/Mixed Methods 157
information provided by counselors. For example, This information could then be used to inform
qualitative data could be obtained using a narra- the construction of a survey that would provide
tive approach, in which counselors talked about quantitative information for a representative sam-
the events in their lives that led them to become ple of newly admitted college students to enhance
counselors and to continue to advise students. generalizability. The strengths and weaknesses of
Transforming the qualitative counselor data such this approach are similar to those of the other
that the transformed variables reflect important concurrent approaches.
themes allows this information to be directly Other examples of mixed methods studies
included (mixed) with student variables in a include Buck, Cook, Quigley, Eastwood, and
quantitative multilevel data analysis, in which Lucas (2009); Day, Sammons, and Gu (2008);
students are treated as nested within counselors Onwuegbuzie, Bustamante, and Nelson (2010);
(Raudenbush & Bryk, 2002). These analyses could and Onwuegbuzie et al. (2007). Good descrip-
explore the impact of nesting and the impact of tions of mixed methods can be found in Bazeley
counselor variables on student data and provide a (2009), Creswell (2003), Creswell and Plano
powerful explanation of how and potentially why Clark (2007), Greene (2007), Reichardt and
students enrolled in a particular college mathe- Rallis (1994), and Tashakkori and Teddlie
matics course. (2003). The relative newness of mixed methods
The sixth mixed methods approach is the con- means that journals that concentrate on pub-
current transformative design. As with the sequen- lishing mixed methods methodology papers
tial transformative design, there is a clearly defined and mixed methods studies in education are
theoretical perspective that guides the methodol- evolving. They currently include the Journal of
ogy. In this approach, qualitative and quantitative Mixed Methods Research, International Journal
data are collected concurrently and can be weighted of Multiple Research Approaches, Qualitative
equally or unequally during the integration of Research Journal, American Educational Research
findings. Qualitative and quantitative data are Journal, Educational Researcher, and Educational
typically mixed during the analysis phase. Strengths Evaluation and Policy Analysis.
include a shorter data collection period, whereas In sum, the mixed methods approach offers a
weaknesses include the need to transform data so collection of flexible research designs that seem
that it can be mixed in the analysis phase and dif- well suited to support rigorous examinations of
ficulties in reconciling conflicting results using promising ideas. The six designs of Creswell
qualitative and quantitative data. (2003) draw on the strengths of qualitative and
As an example of applying a concurrent quantitative methods to enhance inquiry in ways
transformative design, Norman (2008) reported unlikely to occur with singular applications of
evidence that some students were unhappy with these methods. Still, it is important to emphasize
the recommendations they received but felt pow- that mixed methods design continues to face a
erless to change the process. Suppose that the number of significant challenges (Bryman, 2007;
goal of the study was to develop a theory explain- Creswell, 2009; Lawrenz & Huffman, 2002;
ing this sense of powerlessness in ways that could Tashakkori, 2009).
improve the advising process. Norman’s adapta- One important challenge is resolving out-
tion of the Miller and Miller (2005) decision- standing disagreements over appropriate para-
making model includes a component that calls digms. The mixed methods literature contains
for students who are concerned about their rec- a number of “mission accomplished” statements
ommendation to be heard. This could take the implying that important philosophical differ-
form of collecting qualitative information from ences have been resolved, primarily by adopting
students reflecting their experiences with the a pragmatic stance (e.g., Carey, 1993; Creswell
advising process and their view of its impact on & Plano Clark, 2007; Haase & Meyers, 1988;
their mathematics course taking, and simultane- Tashakkori & Teddlie, 2003). Yet it is clear that
ously collecting qualitative data for a group of important differences remain (Sandelowski,
counselors. Developing a theory explaining the 2001; Yin, 2006). A related challenge is achiev-
student’s feelings of powerlessness would require ing better agreement on what characterizes a
that student and counselor responses be carefully mixed methods study and its components. For
examined and categorized to suggest themes to example, what are the components of a mixed
guide and inform the construction of the theory. methods study (research questions, design,
158 Section III. Opportunities and Challenges in Designing and Conducting Inquiry
data collection, data analyses, interpretation), study to compare learning outcomes of a treat-
when should the components be mixed, and ment group whose members receive a promising
how (and how much) should they be mixed to intervention against those of a control group
justify the label mixed methods (see, e.g., using longitudinal data. However, in the service
Bazeley, 2009; Bryman, 2007; Johnson & of developing a better understanding of the
Onwuegbuzie, 2004; Morse, 2010; Tashakkori intervention, the researcher may decide to add a
& Teddle, 2003; Wheeldon, 2010)? Perhaps a preliminary component to the study in which
single comprehensive framework outlining the single-subject methods (Kratochwill & Levin,
characteristics and components of mixed meth- 1992) will be used to examine the learning tra-
ods studies will emerge; on the other hand, it is jectories of a small number of purposively sam-
entirely possible that a crazy quilt pattern of pled participants. This would require expertise
frameworks will continue to define mixed in single-subject methodology that a researcher
methods studies. may or may not possess. Similar examples can
be constructed for qualitative and mixed meth-
ods studies.
Still, the greater challenge for many research-
The Study of Promising ers is likely to be modifying personal norms
Ideas and Research Design defining scholarship and the adequacy of a
contribution to the field. This may require a
Earlier descriptions of qualitative, quantitative, researcher to step outside the methodological
and mixed methods suggest three related ways boundaries he or she has been trained to honor
for researchers to enhance the ability of research and, in some cases, enforce, and to embrace
designs to better support the study of promis- research designs the researcher may have been
ing ideas. These represent both suggestions taught, and have taught others, are inferior.
and challenges. Embracing other methodologies in ways that
First, and most important, is for researchers cross disciplinary, funding, and publication
to consciously shed narrow definitions of lines, for example, serving as a member of an
research design and to embrace more flexible multidisciplinary research team that includes
designs that support rather than constrain the (and values) individuals with expertise in quali-
study of promising ideas. This may mean using a tative or quantitative methods, will require a
familiar research design in an unfamiliar way, certain amount of risk taking but will help to
such as substantially modifying elements of a move the field toward more flexible designs.
design, for example, the way that data are col- Second, researchers can work (or continue to
lected, analyzed, or interpreted in a qualitative work) to modify professional norms in their
study, or changing features of the intervention roles as authors, manuscript reviewers, journal
being studied or the hypotheses being tested in editors, panel members evaluating grant pro-
the middle of a quantitative study because of posals, and so forth, to allow a greater range of
preliminary evidence that a different research studies and findings to be supported. This will
direction would be more productive. It may require an artful balancing between encourag-
mean using a well-known research design to sup- ing risk taking (e.g., studies employing innova-
port exploratory rather than confirmatory work, tive interventions or emerging methods of
adopting key elements of research designs used qualitative inquiry) and the need to satisfy
in other fields, such as dosage-escalation studies standards of research design and reporting that
in medicine (Whitehead, Patterson, Webber, help to ensure the integrity of study results.
Francis, & Zhou, 2001), or abandoning the sin- Interestingly, there is growing evidence of sup-
gular use of qualitative or quantitative methods port for doing just this. Some of this evidence
in a study in favor of a mixed methods approach. appears in studies published in journals, such as
Part of the challenge of embracing more flex- the American Educational Research Journal,
ibility in research designs is technical, to the Educational Evaluation and Policy Analysis, and
extent that it requires additional expertise on the the Journal of Mixed Methods Research, that rely
part of a researcher or suggests the need to on multidisciplinary research teams. Other evi-
develop new research designs. For example, a dence is in the form of new funding programs
researcher may plan to conduct a quantitative at the U.S. Department of Education (primarily
Chapter 10. Research Design in Qualitative/Quantitative/Mixed Methods 159
supporting quantitative research), such as the and breadth of a study’s findings. Advances in
Race to the Top and Investing in Innovation, data collection, data analysis, report writing,
which focus on innovative and ambitious and standards of inquiry and verification have
approaches to educational change that draw on been rapid—and in some cases stunning—and
multiple disciplines. employing the techniques and procedures that
An especially important challenge for modi- reflect these advances should enhance the role
fying professional norms lies in the practices of of research design in supporting the study of
educational research journals that typically pub- promising ideas. For example, advances in com-
lish quantitative studies. The filters enforced by puter hardware and software have significantly
the editorial process of these journals often rein- expanded the ways that data can be collected
force existing and narrow professional norms. and analyzed, especially non-numerical data
Evidence of their success can be found in many such as words and gestures in qualitative research
quantitative meta-analyses. In a typical quantita- (Bazeley, 2009; Creswell & Plano Clark, 2007),
tive meta-analysis, a sample of published and and the accessibility of techniques to control
unpublished studies of a common phenomenon selection bias and take missing data into account in
are collected, examined, and combined—for quantitative research (What Works Clearinghouse,
example, studies that have introduced interven- 2008). Advances in standards of quality and
tions (vouchers, increased teacher professional verification enhance the transparency and
development) designed to reduce the achieve- integrity of results (see, e.g., Creswell & Plano
ment gap between African American and White Clark, 2007; Shadish et al., 2002).
students. Variables capturing key features of each
study, such as sample size, percentage of the
sample that was African American, and charac-
teristics of the interventions are developed, and Conclusion
each study is coded using these variables (Cooper,
Hedges, & Valentine, 2009). Quantitative analy- The pursuit of promising ideas in educational
ses of the resulting data typically provide evi- research sounds noble, and it is. In this spirit, the
dence of the impact of editorial practices. task of this chapter was not to reexamine or
For example, limiting the analyses to published reignite the conflict between qualitative and
studies will typically reveal that all (or virtually all) quantitative methods, nor to assess the peace-
reported at least one statistically significant result making capacity of mixed methods, but rather
(see, e.g., Borman, Hewes, Overman, & Brown, to examine the role of research design in sup-
2003; Fukkink, & de Glopper, 1998; Purdie, porting the rigorous study of ideas that are
Hattie, & Carroll, 2002), a reflection of the well- believed to be worth studying.
documented practice of publishing studies with An examination of qualitative and quantita-
statistically significant findings (Bozarth & Roberts, tive methods in education suggests that singular
1972; Hewitt, Mitchell, & Torgerson, 2008; Leech applications of these methodologies will con-
et al., 2010; Rosenthal, 1979). Other analyses will tinue to play an important role in research study-
often reveal that only a handful of research ing new ideas. Still, there is good reason to
designs were used in the pool of sampled studies. believe that mixed methods do indeed represent,
The absence of published studies that failed as Teddlie and Tashakkori (2003) argued, a ‘‘third
to find evidence of a statistically significant methodological movement’’ (p. 5), which is only
effect and the prevalence of studies employing now beginning to mature “as a well-established
similar research designs send a powerful mes- methodological alternative with agreed-on foun-
sage to researchers: Risk taking is risky, and dations, design, and practices” (p. 287).
small successes are valued more than large fail- In arguing for the value of mixed methods,
ures. This reinforces a “cookie cutter” approach Creswell and Plano Clark (2007) wondered,
to research designs and provides further evi- what would happen if
dence of the need to strengthen the connection
between educational inquiry and the study of quantitative researchers paid more attention to
promising ideas. the incredible range of hypotheses that
Third, researchers can consciously use tech- qualitative researchers have generated for them?
niques and procedures that enhance the depth And what if qualitative researchers spent more
160 Section III. Opportunities and Challenges in Designing and Conducting Inquiry
time exploring the range of phenomena that catalog of research designs available to support
quantitative researchers have sought to define rigorous inquiry. Still, change that leads to more
and test? (p. 59) researchers embracing more flexible research
designs more conspicuously depends on modify-
These are important and intriguing questions, ing norms in ways that broaden the range of stud-
and it seems clear that flexible research designs ies and findings supported in education. That
that aggressively support the study of promising change is underway is undeniable; what is less
ideas are needed to help answer them. Mixed clear is the scope and pace of the change.
methods designs seem especially well suited for In sum, mixed methods research offers an
this task. Whether mixed methods will someday especially promising path toward using
enjoy an equal partnership with qualitative and research design in ways that support rigorous
quantitative research in education or supplant examinations of promising educational ideas.
these methodologies is unclear and in many ways The time to fully embrace mixed methods designs
unimportant. What is important is expanding the has come.
meta-analysis (2nd ed.). New York: Russell Sage Greene, J. C. (2007). Mixed methods in social inquiry.
Foundation. New York: Wiley.
Creswell, J. W. (1998). Qualitative inquiry and Greene, J. C., & Caracelli, V. J. (1997). Advances in
research design: Choosing among five traditions. mixed-method evaluation: The challenges and
Thousand Oaks, CA: Sage. benefits of integrating diverse paradigms. New
Creswell, J. W. (2003). Qualitative, quantitative, and directions for evaluation (No. 74, pp. 19–32).
mixed methods approaches (2nd ed.). Thousand San Francisco: Jossey-Bass.
Oaks, CA: Sage. Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989).
Creswell, J. W. (2009). Mapping the field of mixed Toward a conceptual framework for mixed-
methods research. Journal of Mixed Methods method evaluation design. Educational
Research, 3, 95–108. Evaluation and Policy Analysis, 11, 255–274.
Creswell, J. W., & Plano Clark, V. L. (2007). Designing Greenstein, T. N. (2006). Methods of family research
and conducting mixed methods research. (2nd ed.). Thousand Oaks, CA: Sage.
Thousand Oaks, CA: Sage. Haase, J. E., & Myers, S. T. (1988). Reconciling
Crotty, M. (1998). The foundations of social research: paradigm assumptions of qualitative and
Meaning and perspective in the research process. quantitative research. Western Journal of Nursing
London: Sage. Research, 10, 128–137.
Day, C., Sammons, P., & Gu, Q. (2008). Combining Harry, B., Sturges, K. M., & Klingner, J. K. (2005).
qualitative and quantitative methodologies in Mapping the process: An exemplar of process
research on teachers’ lives, work, and and challenge in grounded theory analysis.
effectiveness: From integration to synergy. Educational Researcher, 34, 3–13.
Educational Researcher, 37, 330–342. Hewitt, C., Mitchell, N., & Torgerson, D. (2008).
Denzin, N. K. (2006). The elephant in the living room: Listen to the data when results are not
Or extending the conversation about the politics significant. British Medical Journal, 336,
of evidence. Qualitative Research, 9, 139–160. 23–25.
Denzin, N. K., & Lincoln, Y. S. (2005). Introduction. Hiatt, J. F. (1986). Spirituality, medicine, and healing.
In N. K. Denzin & Y. S. Lincoln (Eds.), The Southern Medical Journal, 79, 736–743.
SAGE handbook of qualitative research (3rd ed., Howe, K. R. (2009). Positivist dogmas, rhetoric, and
pp. 1–29). Thousand Oaks, CA: Sage. the education science question. Educational
Freeman, M., de Marrais, K., Preissle, J., Roulston, K., Researcher, 38, 428–440.
& St.Pierre, E. A. (2007). Standards of evidence Howell, W. G., Wolf, P. J., Campbell, D. E., &
in qualitative research: An incitement to Peterson, P. E. (2002). School vouchers and
discourse. Educational Researcher, 36, 25–32. academic performance: Results from three
Fukkink, R. G., & de Glopper, K. (1998). Effects of randomized field trials. Journal of Policy
instruction in deriving word meaning from Analysis and Management, 21, 191–217.
context: A meta-analysis. Review of Educational Jacob, B. A., & Lefgren, L. (2004). Remedial
Research, 68, 450–469. education and student achievement: A
Gage, N. L. (1989). The paradigm wars and their regression discontinuity analysis. Review of
aftermath: A “historical” sketch of research on Economics and Statistics, 86, 226–244.
teaching since 1989. Educational Researcher, 18, Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed
4–10. methods research: A research paradigm whose
Gaines, E. (2005). Interpreting India, identity, and time has come. Educational Researcher, 33,
media from the field: Exploring the 14–26.
communicative nature of the exotic other. Johnson, R. B., & Turner, L. A. (2003). Data
Qualitative Inquiry, 11, 518–534. collection strategies in mixed methods research.
Garner, C., & Raudenbush, S. J. (1991). Neighborhood In A. Tashakkori & C. Teddlie (Eds.), Handbook
effects on educational attainment: A multilevel of mixed methods in social and behavioral
analysis of the influence of pupil ability, family, research (pp. 297–319). Thousand Oaks, CA:
school, and neighborhood. Sociology of Sage.
Education, 64, 251–262. Kempthorne, O. (1952). Design and analysis of
Gay, L. R., Mills, G., & Airasian, P. W. (2005). experiments. New York: Wiley.
Educational research: Competencies for analysis Kerlinger, F. (1964). Foundations of behavioral
and applications (8th ed.). Upper Saddle River, research. New York: Holt.
NJ: Merrill. Kratochwill, T. R., & Levin, J. R. (1992). Single-case
Giddings, L. S. (2006). Mixed-methods research: research design and analysis: New directions for
Positivism dressed in drag? Journal of Research psychology and education. Hillsdale, NJ:
in Nursing, 11(3), 195–203. Lawrence Erlbaum.
162 Section III. Opportunities and Challenges in Designing and Conducting Inquiry
Lawrenz, F., & Huffman, D. (2002). The archipelago Education Research. Washington, DC: National
approach to mixed method evaluation. Academies Press.
American Journal of Evaluation, 23, 331–338. National Research Council. (2005). Advancing
Leech, N. L., Morgan, G. A., Wang, J., & Gliner, J. A. scientific research in education (L. Towne,
(2010). Statistical significance and evidence- L. L. Wise, & T. M. Winters, Eds.). Committee
based approach: Implications of non-significant on Research in Education. Washington, DC:
results. Journal of Non-significant Results in National Academies Press.
Education, 1, 1–12. Newman, I., Ridenour, C. S., Newman, C., &
Leech, N. L., & Onwuegbuzie, A. J. (2007). A DeMarco, G. M. P. (2003). A typology of
typology of mixed methods research designs. research purposes and its relationship to mixed
Quality & Quantity: International Journal of methods. In A. Tashakkori & C. Teddlie (Eds.),
Methodology, 43, 265–275. Handbook of mixed methods in social and
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic behavioral research (pp. 167–188). Thousand
inquiry. Newbury Park, CA: Sage. Oaks, CA: Sage.
Maruna, S. (2010). Mixed method research in Norman, K. (2008). High school mathematics
criminology: Why not go both ways? In curriculum and the process and accuracy of
A. R. Piquero & D. Weisburd (Eds.), Handbook initial mathematics placement for students
of quantitative criminology (pp. 123–140). who are admitted into one of the STEM
New York: Springer. programs at a research institution.
Maxwell, J. A. (2004). Causal explanation, qualitative Unpublished dissertation, University of
research, and scientific inquiry in education. Minnesota, Twin Cities.
Educational Researcher, 33, 3–11. O’Cathain, A. (2009). Mixed methods research in the
Mertens, D. M. (2003). Mixed methods and the health sciences: A quiet revolution. Journal of
politics of human research: The transformative- Mixed Methods Research, 3, 3–6.
emancipatory perspective. In A. Tashakkori & O’Cathain, A., Murphy, E., & Nicoll, J. (2007). Why,
C. Teddlie (Eds.), Handbook of mixed methods and how, mixed methods research is undertaken
in social and behavioral research (pp. 135–164). in health services research in England: A mixed
Thousand Oaks, CA: Sage. methods study. BMC Health Services Research, 7.
Miles, M. B., & Huberman, M. A. (1994). Available at http://www.biomedcentral
Qualitative data analysis (2nd ed.). Thousand .com/1472-6963/7/85
Oaks, CA: Sage. Onwuegbuzie, A. J., Bustamante, R. M., & Nelson, J. A.
Miller, M., & Miller, T. (2005). Theoretical (2010). Mixed research as a tool for developing
application of Holland’s theory to individual quantitative instruments. Journal of Mixed
decision-making styles: Implications for career Methods Research, 4, 56–78.
counselors. Journal of Employment Counseling, Onwuegbuzie, A. J., Witcher, A. E., Collins, K. M. T.,
41, 20–28. Filer, J. D., Wiedmaier, C. D., & Moore, C. W.
Morgan, D. L. (1998). Practical strategies for (2007). Students’ perceptions of characteristics
combining qualitative and quantitative of effective college teachers: A validity study of
methods: Applications to health research. a teaching evaluation form using a mixed-
Qualitative Health Research, 3, 362–376. methods analysis. American Educational
Morgan, D. L. (2007). Paradigms lost and Research Journal, 44, 113–160.
pragmatism regained: Methodological Patton, M. Q. (2002). Qualitative research and
implications of combining qualitative and evaluation methods (3rd ed.). Thousand Oaks,
quantitative methods. Journal of Mixed Methods CA: Sage.
Research, 1, 48–76. Pedhazur, E. J., & Schmelkin, L. P. (1991).
Morse, J. M. (2006). The politics of evidence. Measurement, design, and analysis: An
Qualitative Health Research, 16, 395–404. integrated approach. Hillsdale, NJ: Lawrence
Morse, J. M. (2010). Simultaneous and sequential Erlbaum.
qualitative mixed method designs. Qualitative Pegues, H. (2007). Of paradigm wars:
Inquiry, 16, 483–491. Constructivism, objectivism, and postmodern
Moss, P. A., Phillips, D. C., Erickson, F. D., Floden, stratagem. Educational Forum, 71, 316–330.
R. E., Lather, P. A., & Schneider, B. L. (2009). Plano Clark, V. L. (2010). The adoption and practice
Learning from our differences: A dialogue of mixed methods: U.S. trends in federally
across perspectives on quality in education funded health-related research. Qualitative
research. Educational Researcher, 38, 501–517. Inquiry, 16, 428–440.
National Research Council. (2002). Scientific research Purdie, N., Hattie, J., & Carroll, A. (2002). A review
in education (R. J. Shavelson & L. Towne, Eds.). of the research on interventions for attention
Committee on Scientific Principles for deficit hyperactivity disorder: What works
Chapter 10. Research Design in Qualitative/Quantitative/Mixed Methods 163
165
166 Section III. Opportunities and Challenges in Designing and Conducting Inquiry
out there that we can discover, or is the world a extension, the purpose of research is to contrib
product of the imagination, constructed in the ute theories or refinements of existing theories
minds of individuals and groups? Researchers to science. Theory is a kind of abstraction, a
design studies based on what they believe knowl simplification of reality that applies in similar
edge to be. The search for an objective truth circumstances and not just to the specific case at
involves a different path from the search for hand. For researchers, theories focus attention,
individual meaning or a consensus about inter limit choices, and provide explanations—
subjective meaning. characteristics that give good theories a central
In the professions, as opposed to the basic role in research design. For actors in the educa
disciplines, utility is necessary. Professionals tional environment, they are practical for the
provide service to a client based on superior same reasons.
knowledge developed from a long study of the Theories about social behavior have inherent
disciplinary research. According to Shils (1984), limits. These are identified by Thorngate (1976)
what separates academic knowledge from com and elaborated on by Weick (1979). Thorngate
mon knowledge is that academic knowledge is developed a postulate of commensurate com
developed by a rigorous methodology. Research plexity in which there are trade-offs among a
ers in a pure discipline (Biglan, 1973) attempt to theory being general, a theory being accurate,
establish truth in that discipline. Researchers in and a theory being simple. A theory cannot be all
a profession have as their purpose not to attain three simultaneously; general accurate theories
pure knowledge but rather praxis, that is, to are not simple, accurate simple theories are not
attain the best knowledge that can be applied in general, and simple general theories are not
service of their clients’ needs. To offer education accurate. Weick provides examples of each. In
as a societal good, money is spent, programs developing a research design, the theory used, or
are funded, and teachers are trained and the one the researcher is trying to develop, has
hired. If research is to inform these processes, more important effects on research design than
then it is difficult to escape from pragmatism does anything except the topic chosen for the
and positivism. research. Theory drives hypotheses. The choice
Research that advances methodology is of of a theory to use or develop reflects the
value to a field by developing better researchers, researcher’s interest in being general, simple, or
but such research is not always of direct use to a accurate and shapes the study accordingly. From
researcher’s clientele. A field that emphasizes my observations, I would suggest that educa
internal debates about philosophy, methodol tional research errs on the side of being simple
ogy, and definitions can be very lively but is in and, with luck, accurate.
danger of becoming irrelevant. Well-designed Theory is emphasized in research because it
research should deliver new understandings and provides explanation. Without meaningful
new theories. The ultimate test of its value to the descriptions of the situation—that is, without
public will not rest with internal elaboration or identifying new ideas to be understood and
with faculty members charming other faculty related to each other by theories—research
members; rather, it will be seen with improving would not move forward. Designing research
understanding, teaching, learning, and organiz that identifies the ways in which people in a
ing in a heterogeneous society. In what follows, I given situation view their worlds is a sensible
discuss some of the primary considerations that starting place for meaningful research. Without
should inform research designs. important things to be studied, no theories
would need to be developed, and a rigorous
methodology to estimate relationships based on
Theories
theory would not be necessary.
Educational researchers are interested in
finding out how one thing is related to another;
Topics and Ideas
describing a set of phenomena; and establishing
a basis on which to make claims, predictions, How does one go about selecting substantive
and explanations. Braithwaite (1955) writes that issues connected by a theory? Texts covering
the purpose of science is theory and that, by research in education or the behavioral sciences
Chapter 11. Intellect, Light, and Shadow in Research Design 167
have tended to either be mute on the question spiritual. Of these, verbal/linguistic and logical/
or provide only minimal advice (Gall, Borg, & mathematical topics have dominated educational
Gall, 2002; Kerlinger, 1973; Krathwohl, 1988), research, while the others have received little
with some improvement lately (Creswell, 2007). study or have been the province of specialized
The selection of topics for study is neither inno fields (e.g., music).
cent nor rational. It is not innocent, because The topics we choose reflect a worldview of
being selected gives a topic legitimacy, creates educational research: study the specific rather
the power of knowledge for those affected by than the general, emphasize topics related to
the topic, and creates invisibility for those linguistic and logical cognitive development,
excluded. It is not rational, because researchers and use methods that favor such studies. We
choose topics to study based on professional also favor topics that reflect social justice, cog
interests, not professional mandates, and on nitive development that leads students into
self-interest based on what researchers value, economically productive lives, and those topics
are curious about, or perceive they will profit that fit within predetermined disciplinary spe
from. What is studied in a discipline becomes cialties. If education deals with the whole stu
what is taught in the discipline. Just like what is dent, as Bowen (1977) suggests, it would
included in the curriculum is a political deci behoove researchers to look for topics that
sion as well as an educational decision, what is consider more kinds of intelligence than those
studied is not neutral; it implies that what is associated with linguistic and mathematical
studied is valuable. reasoning. Having become obsessed with cer
It is axiomatic that the most important part tainty in our research, we have eschewed broad
of any study is the choice of a topic; that is, research topics that require integrating various
research findings depend on what is being components of human intelligence in favor of
researched. A good topic, when well studied, being able to say with great certainty some
improves descriptions in a field, better explains thing of diminutive importance. Highly spe
how theories operate in the discipline, and cialized methodologies require highly refined
shows how this knowledge can be applied to topics, which provide little opportunity for
benefit clients and society as a whole. The transformational outcomes and the develop
research problem to be addressed and the state ment of big ideas.
ment of the purpose of the study focus research
activities and limit the scope of the study. If the
Choosing Topics
purpose is too broad, then the research cannot
be accomplished with reasonable effort. If the Identifying a research problem is the starting
purpose is too narrow, then the study is trivial. place for research. Research problems involve
The statement of purpose is the most important unresolved real-world conditions or situations,
sentence in a research proposal. Researchers and it is the research problem that is nested
need to avoid making Type III errors—asking between the topic and the purpose. Theoretical
the wrong question or not asking the right research problems deal with “we don’t know
question—or what I consider to be Type IV why,” descriptive research problems deal with
errors, that is, studying the wrong thing. “we don’t know what,” and practical research
Because we are well trained, we academics can problems deal with “we don’t know how.” The
justify studying practically anything. Politicians best researchers are attracted to uncertainty,
can mock this attribute of the academy, in the paradoxes, anomalies, contradictions, and ambi
past handing out “Golden Fleece” awards for guities in the field. The significance of a prob
research that seemed to be fleecing the taxpayer. lem is often based on the way in which it inter
Rather than focusing exclusively on what we do sects with theoretical uncertainty and practical
study, it is important to recognize what we choose importance.
not to study. Nearly 30 years ago, Gardner (1983) Probably the best way of finding a problem to
identified eight areas of intelligence: verbal/ study is to do extensive reading in an important
linguistic, logical/mathematical, visual/special, topical area and find out what is poorly under
physical/kinesthetic/intrapersonal/social, musi stood. Many articles contain sections that give
cal, intrapersonal/emotional, naturalistic, and suggestions for future research, and many have
168 Section III. Opportunities and Challenges in Designing and Conducting Inquiry
glaring shortcomings that suggest a problem out the research? Only the third question is
should be reexamined from a different perspec related to methodology.
tive. Some researchers choose a methodology
and then try to find a problem to match it. This
Different Approaches to Research
approach creates an unnecessary constraint on
what is to be studied, and the foolishness of this There are many ways in which to study the
sequence cannot be emphasized enough. The same topic, and these produce different results.
cart does not pull the horse. Mitroff and Kilmann (1978) describe four
Other researchers are told what to study by a approaches to research based on a Jungian
superior, such as an advisor, a provider of analysis of our predispositions to approach deci
resources, an admired scholar in the field, or a sion making and obtain information, similar to
coinvestigator. So long as the relationship is not the Myers-Briggs Type Indicator tests (Myers-
exploitative, joining an ongoing research agenda Briggs, 1962). The four approaches include the
has the clear advantage of the person being clear following: the scientist, the conceptual theorist,
about what he or she will study. It has the disad the conceptual humanist, and the individual
vantage of not learning how to define one’s own humanist. Given the topic college student reten-
research problem. Typically, a study involves the tion, consider the way in which each methodol
following iterative process: Approach a topic of ogy is chosen based on the researcher’s interests,
interest, read in the area, write a brief statement is different from the other methods, and pro
of purpose, discuss this statement with others, duces different results. Descriptive phrases below
reflect, read, write, talk, and so on, until a com are taken from tables in Mitroff and Kilmann on
pelling problem with a realistic scope has been the pages indicated.
identified and clearly expressed. For the scientist, the approach should be
A hot topic is one where there is a lot of objective, causal, cumulative, and progressive,
interest and a great likelihood of getting a emphasizing reliability and external validity and
project funded, finding others to participate in separating the scientist from the observed. It
the study, and publishing the results. Good aims at precise, unambiguous empirical knowl
topics allow young scholars to demonstrate edge using strict logic (Mitroff & Kilmann, 1978,
their research skills and increase the likeli p. 34). The norms of this approach are known as
hood of getting further support for their the CUDOS:
research and publications in their fields. For
better or worse, it usually means becoming Communism, indicating that scientific
more specialized. With luck, a good topic is knowledge is common property;
one the researcher loves. Here, love is a pas Universalism, indicating that scientific knowledge
sionate attachment to the research and an should be independent of the personality of the
enjoyment of the research process. This attach individual scientist;
ment should not be confused with a lack of
Disinterestedness, such that the scientist should
objectivity, but it involves caring about an observe what happens and not advocate a theory
increased understanding of a topic and a will or experimental outcome; and
ingness to put forward the effort that results in
influential research. Organized
After deciding what to study and why such a Skepticism, where scientists should be critical of
study is worthwhile, the final part of designing their own and others’ ideas. (Merton, 1942/1973,
research is to decide on the processes by which p. 269)
the research is to be accomplished. Many people,
in thinking about research design, think that An example of the scientific study of reten
they need only be concerned with research tion would be an organizational experiment
methodology (Kerlinger, 1973, cited in Daniel, based on the hypothesis that higher-achieving
1996). The following three questions always students are more likely to remain enrolled in
affect research design: (1) What will the college. Students would be randomly assigned to
researcher study? (2) Why is the research a treatment group or a control group. In the
important? (3) How will the researcher carry treatment group, students would participate in a
Chapter 11. Intellect, Light, and Shadow in Research Design 169
retention program, such as developing study better understand how the issues facing students
skills, but otherwise would have experiences no contribute to their leaving, and tries to alleviate
different from those of the control group. After those conditions. The purpose is to increase the
a given time period, the researcher would find overall retention rate with the belief that stu
out whether the retention rate for students who dents who complete college lead richer lives.
participated in the retention program was sig An individual humanist addresses inquiry as
nificantly different from that for students who a personal, value-constituted, interested, and
did not. This information would be used to sup partisan activity, engaging in poetic, political,
port or negate the hypothesis. acausal, and nonrational discourse in pursuit
The conceptual theorist is involved with of knowledge. Intense personal knowledge
research that is impersonal, value-free, disinter and experience are highly valued, aiming to help
ested, imaginative, and problematic, involving this person to know himself or herself uniquely
multiple causation, purposeful ambiguity, and and to achieve her or his own self-determina
uncertainty. The theorist is interested in the tion. The logic of the unique and singular has
conflict between antithetical imaginative theo mythical, mystical, and transcendental over
ries, comprehensive holistic theories, and ever tones that operate as counternorms to the
expanding research programs to produce con CUDOS (Mitroff & Kilmann, 1978, p. 95). A
flicting schemas using dialectical and indetermi retention study from this perspective would try
nate logics (Mitroff & Kilmann, 1978, p. 56). A to develop a detailed understanding of a single
theorist conducting retention research would student in the full context of his or her life. It
provide at least two theoretical explanations of could take the form of an “N of 1” case study: a
retention behavior, use survey methods to gather phenomenological inquiry into who the student
information, analyze the data using statistics, is, what the student finds at school, and how
find out whether the data supported one theory staying or leaving school would be better for this
more than the other, and use that information to particular individual.
make more theories. Much of the empirical The purpose of presenting these four
research reported in educational journals is a perspectives—and many more can be imagined—
combination of the scientist and theorist— is to illustrate that there is no best way in which
theory guiding social science inquiry into to study a topic. Different kinds of studies make
educational structures and processes. different kinds of assumptions about what is
The conceptual humanist (although I find important to know, serve different needs for
social humanist to be a more accurate descrip different people involved in the studies, and
tion) approaches research as a value-constituted, produce different kinds of outcomes. The four
interested activity that is holistic, political, and perspectives were presented in what was once
imaginative; where multiple causations are pres considered the normative order of acceptability:
ent in an uncertain and problematic social envi science-based research, theory development,
ronment; and with a deep concern for humanity. action research, and phenomenology. One may
This approach recognizes the importance of be no more correct than the others. Some are
the relationship between the inquirer and the more acceptable to certain audiences than to
subject and has the aim of promoting human others, and each produces a particular out
development on the widest possible scale. The come that favors some stakeholders more than it
normative outcomes of such research would be does others.
economic plenty, aesthetic beauty, and human
welfare. Similar to an action researcher, the
Methodology and the Scientific Approach
social humanist prefers small-group dynamics
where both the inquirer and the participants Methodology is often considered the core of
learn to know themselves better and work research design. Kerlinger (1973, cited in Daniel,
together to improve the situation (Mitroff & 1996) described as one of the research myths the
Kilmann, 1978, p. 76). A retention researcher idea that research design and research methods
using this approach could develop an ongoing were synonymous, even though many research
program of action-oriented ethnographic ers held this view. Methodology is the tool used
research studies, where the researcher comes to to accomplish part of the study, specifically, how
170 Section III. Opportunities and Challenges in Designing and Conducting Inquiry
to obtain and analyze data. It is subservient to In the same way that social science research
choosing an important topic to study, matching has imitated the natural sciences, educational
the research problem and the methodology, and research has imitated social science research.
knowing what the results mean and how they Research in these areas may be separated more
can be applied. To do good research, the meth by the topic studied than by the rigor of the
odology used should be appropriate for the methodology. As with social science research
problem addressed. This is a necessary condi in general, educational research might pre
tion but not a sufficient one. An elegantly ana tend a level of control so as to carry out an
lyzed data set that was composed of ambiguously experiment. Researchers give themselves sol
measured data that addressed a question of triv ace that “other things are equal,” or “other
ial importance is not likely to enter the annals of effects are random,” or “spuriousness is not a
great research. problem,” and proceed as if the social world
Educational research is part of the social sci were simple and understandable in the same
ence research tradition, a tradition that was way that the traditional world of the pure sci
influenced by research in the natural sciences. ences can be.
The natural sciences use the scientific method to
solve research problems or support a perspec
A Traditional Approach to Designing
tive. The method contains a series of sequential
Educational Research
steps similar to the following: Identify a prob
lem, gather information from the literature Graduate programs in education typically are
about this question, develop a hypothesis in the arranged in departments that reflect academic
context of a theory, collect data related to the subspecialties such as history of education, soci
hypothesis, analyze the data, and draw a conclu ology of education, anthropology of education,
sion related to the truthfulness of the hypothesis counseling psychology, experimental psychol
and correctness of the theory. ogy, higher education, school administration,
Scientists, as logical purists, build argu and curriculum and instruction. Most of these
ments on falsifiability and the law of the programs require a course in research design
excluded middle. This law states that A and appropriate for their field. My discussion
not-A cannot exist simultaneously. But if A revolves around quantitative and qualitative
stands for “this program helps students to approaches to research, terms that reify catego
learn” and not-A stands for “this program does ries that are themselves overlapping and arbi
not help students to learn,” then both can be trary, but are commonly used to describe
true, as in the case of aptitude–treatment inter research courses.
actions; that is, a treatment could be effective A simple way of looking at a proposal is to see
for a high-aptitude student but not effective for how it answers the three questions posed earlier:
a low-aptitude student. If both are true, then What is this study about? Why is the study
the law of the excluded middle is violated and important? How will the researcher conduct the
falsifiability cannot be demonstrated. This sit study? The study itself must cover these three
uation is problematic for scientific research questions and answer two additional questions:
in education. What did the researcher find? What do the find
Education is not a scientifically based process, ings mean? Research designs revolve around a
partly because the term education is ideological limited number of elements. Their exact use and
and idiosyncratic, much different from the term exposition vary depending on the particular
temperature. At best, scientific research can shed approach taken. The purpose and promise of
light on narrowly defined educational behaviors, these elements have been identified and dis
and researchers can hope for—but cannot cussed by a number of textbooks such as those
guarantee—a cumulative effect. When a govern of Gall and colleagues (2002) and Creswell
ment policy assumes that education is equivalent (2007). These texts identify many of the issues
to improving the score on a test, the society will facing researchers, especially those who are new
not have a moral compass and will not be edu to the process.
cated. Feyerabend (1993) holds the view that if Although not suitable for all studies, well-
we do not separate scientific research and the designed quantitative research usually ad-
state, as we have separated the church and the dresses the areas presented in the following
state, irreparable harm will be done. outline:
Chapter 11. Intellect, Light, and Shadow in Research Design 171
The structure of qualitative studies is more be analogous to the study of turnover in work
idiosyncratic than the more formal structure of organizations, and the literature in one area can
traditional quantitative studies. In qualitative be used to reinforce the literature in the other
research, the sample is the study, and the reasons area (Bean, 1980). The use of literature in quan
for selecting the sample need to be emphasized. titative studies, however, can differ substantially
Most researchers, when approaching a topic from the use of literature in qualitative studies.
they care about, have tentative hypotheses about In a quantitative study, the literature is used
what causes what or predispositions to thinking to identify the importance of the dependent
that the world operates according to certain variable, relevant independent variables, and
principles that also apply in this area. People theories that bind these factors together, to jus
bias their observations based on their experi tify the use of statistical procedures and to pro
ence. All of us know that we have certain biases, vide a context for the discussion. In qualitative
and we can try to counter those biases in our studies, a premium is placed on the ability to see
research by looking for evidence that directly what is before the researcher. Our ability to
contradicts what we expect. In the unconscious, observe is both heightened and diminished by
there is a second set of biases of which, by defini our prior knowledge and expectations (Bean,
tion, we are not aware. Sometimes peer readers 1997). It is heightened by making ourselves
can help the researcher to discover what is miss aware of important details to observe, and it is
ing or what is inappropriately under- or over diminished because we focus only on those
emphasized in the study. details. Due to the preconceived notions of the
After analyzing the data in a quantitative researcher, those factors actually influencing the
study, the researcher presents the findings. respondents’ world might not be identified.
Typically, it is rather straightforward, because When the literature shapes the way in which we
the data to be gathered and the analyses pro view the world, what is actually before us is
posed for the data were specified in the proposal replaced by what we expect to see.
for the study. For qualitative researchers, the A review of the literature, as a stand-alone
data, the findings, and the method might not be section summarizing research in the topical
distinct. The narrative that presents selected area, makes little sense. The literature, as a com
questions and answers can represent findings pendium of related information, should be used
based on data that came from the method by to advance arguments related to the importance
which questions were developed. As the previ of the subject. It should identify topical areas
ous sentence suggests, it is a convoluted process. that are either well or poorly understood, iden
The presentation might revolve around respon tify and describe relevant theories, identify and
dents’ experiences and understandings, a chro describe appropriate methodologies to study the
nology of events, or themes supported by topic, describe dependent and independent
respondents’ statements. In ethnographic stud variables if relevant, provide definitions, and
ies, the descriptions of lives in context can provide a context to discuss the findings from
stand on their own (Lawrence-Lightfoot, 1995). the study.
Thick descriptions (Geertz, 1973) might pro Google Scholar, Dissertation Abstracts
vide greater insight into the education of a stu International, ERIC Documents, the ISI Web of
dent in a school than would an analysis of Knowledge, and the proceedings of relevant
variables. In most studies, some analysis of the professional organizations all can be used to
descriptions is expected. This predisposition is access current research. A condemning retort is
part of the legacy of pragmatism; researchers that the literature in a study is dated. This phrase
in education are expected to identify how has some interesting subtexts. The first assump
knowledge gained from the study can improve tion is that the most recent research is the best
professional practice. research and that previous research is irrelevant.
In designing a study, the background, con A second assumption is that all research is of
text, importance of the topic, and presumed limited generalizability over time so that if it is
practical value of the study come from the lit older than, say, 5 years, it is irrelevant. In either
erature written about the topic or analogous case, dated research is of marginal value. By
literatures in similar fields. For example, the study extension, the research that a person is currently
of college student retention can be considered to conducting is also of marginal value because it
Chapter 11. Intellect, Light, and Shadow in Research Design 173
will be useful for only 5 years. This planned linear modeling (GLIM) and hierarchical linear
obsolescence of research becomes a justification modeling (HLM).
for a frenzied increase in the rate of publication The increase in complexity is associated with
and is counterproductive in terms of identifying an increase in agitated exchanges between statis
important and durable ideas in the field. ticians about whose method is correct. An
Literature should not be weighed nor consid improved methodology has not been matched
ered dated after 5 years. by these studies becoming more influential in
In traditional quantitative research, the topic policy making or practice (Kezar, 2000). The
contains the dependent variable, and the factors debate is sometimes invisible to the public, tak
associated with it identify the independent vari ing place between the author of a piece of
ables that have been found to have important research and the consulting editors who review
effects on the dependent variable. In studies that the research, and sometimes it appears in jour
are not codifications—that is, not extensive nals such as the Educational Researcher.
reviews of the literature for the heuristic pur
pose of organizing what is known about a
Data
topic—citing the literature should be done for
the purpose of building an argument, not sim Quantitative studies require data that can be
ply to show familiarity with the canon. used in statistical analyses. The sources of data
Since the 1960s, the number of statistical can vary widely—historical documents, govern
analyses available for researchers to include in mental records, organizational records, inter
their designs has increased dramatically. Five views, standardized surveys, questionnaires
commercial statistical packages bear the initials developed as part of the research protocol for a
SAS, SPSS, BMDP, GLIM, and HLM. The devel particular study, unobtrusive measures, obser
opment of these statistical packages has allowed vations, participant observation, and so on. The
ever more complex analyses to be performed. quality of the research depends on the quality of
National data sets from the National Center for the data analyzed; data analysis has only a sec
Educational Statistics (NCES) and other sources ondary influence.
have provided the opportunity to bring order to The quality of the data varies greatly. Good
vast amounts of data. research design requires that the researcher
For the description of large-scale phenom understand the strengths and weaknesses of the
ena, these data sets can be very valuable. For data. Historical data can reflect the biases and
analyzing the causes of behavior, however, the ideological preferences of those who recorded it.
attempt to gain a broad vision masks individual People who provide data can intentionally dis
or small-group differences. Longitudinal studies tort it to put themselves in a better light, for
almost always suffer from decay; that is, mea example, reporting that they had higher grades
sures may differ from year to year and respon than they actually did. Survey data might come
dents drop out of the study. So the comparisons from a biased sample reflecting only the experi
from year to year might not be the result of what ences of high–socioeconomic status respondents.
people report; rather, they might be the result of Questions in a survey might be ambiguously
changes in who is doing the reporting. written, or a single item might contain two
The availability of data and the means to questions with different answers, for example,
analyze them raised the level of expectation “How satisfied are you with your salary and
in some journals that such analyses should fringe benefits?” Survey data that require a
be the norm. What is certain is that during forced-choice response might not represent the
the past 50 years, the sophistication of analyses real interests of the respondent. A respondent
has increased. The literature shifted from might have no opinion on most of the questions
normed surveys that reported frequencies, to and refuse to answer them. Other respondents
chi squares, to analyses of variance (ANOVAs) might not want to reveal personal information
and simple correlations, to factor analysis and and so might misrepresent their actual incomes,
multiple regression, to causal modeling with whether they have ever plagiarized, or how
ordinary least squares path analysis, to maxi much they use drugs or alcohol. Although the
mum likelihood used in linear structural rela questionnaire is not missing any data, the data
tions (LISREL) modeling, and to generalized provided might be intentionally inaccurate.
174 Section III. Opportunities and Challenges in Designing and Conducting Inquiry
In other cases, respondents might not under Milgram, 1974) and medical research where
stand the questions, might not care about the participants were given diseases or intentionally
answers given, or might become fatigued while denied treatment (Jones, 1993).
filling out the questionnaire, so that the accuracy The bureaucratic response to these ethical
of the responses is different for the beginning violations was to create rules that would include
and the end of the questionnaire. A well-written everybody doing any kind of research that
question should reflect one bit of information involved contact with living people. Bureaucratic
about the respondent unambiguously and reli actors, evaluating research they are not conduct
ably, and the answer to the question should ing themselves, become the gatekeepers of ethi
match observable facts. cal behavior. This responsibility is misplaced;
It is acceptable to use problematic data if the researchers themselves should be responsible for
analyst understands and acknowledges the prob protecting the interests of participants in their
lems that exist in the data. For example, a data set studies. I am not naïve enough to think that all
might not be random but might be completely researchers are ethical or that institutional
representative of one subgroup in the population review boards (IRBs) or protection of human
studied. The bias in this sample can make con subjects committees will go away. The problem
clusions drawn about the well-represented group is that ethical judgments about research have
accurate, but the conclusions would not apply to been made extrinsic to the research process.
the whole population. Although not representa Researchers need to design research that does
tive, the data might be useful to see whether a just what the committees want—to protect the
hypothesized relationship exists at all, that is, as participants of a study from harm. If researchers
a test of theory. are not socialized to provide these protections,
Data gathered from face-to-face interviews IRBs might not help. The enforcement system
for qualitative research have the potential to used, which involves taking federal support away
yield a gold mine of insights into the people’s from ethical researchers because they happen to
lives and situations. There is no substitute for be at an institution where one person did not
prolonged and focused conversations between comply with the guidelines, is a collective pun
trusted parties to discover what is important to ishment which is itself unethical. IRBs have the
the interviewees and how respondents under enormous power of being able to block research,
stand key elements in their own lives. When and the potential for abusing power must be
mishandled, interview data can reflect what the kept under constant scrutiny.
interviewees think the interviewers want to hear, For qualitative researchers especially, com
normatively appropriate responses, the fears and plying with a written informed consent form
biases of the interviewees, and the fears and can damage the trust required to conduct a
biases of the interviewers. Data flaws become study. The study of any group that dislikes
limitations of the study for which the only authority is made impossible, or at least less reli
response is to caution the reader that the results able, by asking participants at the outset to sign
are far from certain. a form that says, “You should know that this
researcher does not intend to hurt you.” A jour
nalist and an ethnographer can conduct and
Ethics and Institutional Review Boards
publish identical studies. However, the journal
Before proceeding with an examination of ist needs no informed consent from those who
research methods, there are some ethical and are interviewed for the story, whereas the eth
legal considerations that have obtrusively en- nographer at a research institute needs IRB
tered the development of a research protocol. In permission to ask the same questions. The jour
line with designing research to be useful, it nalist is protected by freedom of speech, whereas
should also be designed to be ethical. The most academic freedom, according to IRB rules, pro
obvious ethical problems arise when a research vides no such protection for the researcher.
procedure causes harm to those who are asked While much of the antagonism between
or forced to participate in the process. There are IRBs and the faculty involve what are seen as
several well-known cases of abuse, including nuisance parameters, as hurdles to be jumped,
psychological studies where participants were put IRB guidelines constitute a direct attack on
in unusually stressful situations (Baumrind, 1964; academic freedom. When a faculty member has
Chapter 11. Intellect, Light, and Shadow in Research Design 175
to get permission to “do research as he or she at that point that their probabilistic, relative
sees fit,” there is no academic freedom. There nature comes into sharpest focus. (p. 116)
are three traditional threats to academic free
dom: economics, religion, and politics. One Does what works 90% of the time for the
could argue that IRBs are simply a new form of participants in a study work for one particular
the economic threat, being that failure to fol teacher in one particular class dealing with one
low IRB guidelines will result in the loss of particular subject? Tutoring generally helps stu
substantial federal funding for research at the dents to learn how to read, but for a student who
institution. Too often, I fear, these offices dis is acting out against authority, and who views
place their goals of protecting human subjects the tutor as an authority figure, tutoring might
and replace them with the goal of making prevent the student from learning to read.
researchers follow their rules. The direct and As a “reductionistic fallacy” (Lincoln &
opportunity costs to institutions in wasting Guba, 1985, p. 117), generalization simplifies
faculty time is real, but the greater problem is decision making and simultaneously reduces
researcher self-censorship—not doing research the understanding of the particular. Teachers
because of the fear that it will not be approved operate in particular environments, and the
by the IRB. Academic freedom is the most cen findings from a “scientific” study with a high
tral value in American higher education, and degree of generalizability do not ensure a pro
anything that puts it at risk needs greater justi gram’s utility in a given classroom. The purpose
fication than IRBs have provided for their of scientific research is to eliminate uncertainty
behavior. Regardless of what happens to IRBs, so that the operator can predict and control the
research should be designed to protect every future. Applied to education, this goal is not a
one, to benefit the participants in the research, research norm or a teaching norm but rather a
and to protect society from ignorance. political norm.
provides no direction. A researcher has some (Parnes, 1992). However, the content of the steps
control, but not complete control, over deciding is not well understood. If it were, then everyone
what research to conduct. A researcher has lim would be creative and have plenty of excellent
ited control, or no control, over how research is problems around which to design research.
funded, how it is evaluated, and how it is used. To be creative, as opposed to simply novel, a
The shadow of research appears when one con researcher should be well-versed in substantive
fronts the lack of creativity in research and psy knowledge of the topic and the limitations of
chological barriers to the free flow of ideas. It the chosen methodology. Creativity has at its
occurs when a researcher encounters difficulties foundation a sense of play—of suspending nor
related to the disciplinary research environment mal constraints so as to see new patterns, possi
and the primary and secondary social environ bilities, or connections. Play is usually an “idea
ments associated with the research. non grata” in a workaholic environment, although
the hermeneutical philosopher Godamer con
sidered play to be an expression of great serious
The Loss of Creativity
ness (Neill & Ridley, 1995). Play is characterized
If the world were static, then creativity would by just those things that are likely to lead a
not be necessary; what worked in the past would researcher into creative work, including taking
continue to work in the future. In a dynamic risks, testing new ideas in safety, avoiding rigidity,
social world existing in a turbulent ecology, the and suspending judgment (Schwartzman, 1978).
generation of new ideas is necessary for survival. A risk-averse, judgmental, assessment-oriented
In the natural world, mutation is a random pro environment focused on short-term gains will
cess, and selection occurs where the fit to the have a negative effect on creativity. If proposals
natural environment of the new form has advan are assessed by published criteria, then how can
tages over existing forms. In the social world, new projects that do not fit established criteria
creativity is the source of variation and must be funded? We live in a judgment-rich environ
be present before selection can take place. ment, where we have been socialized for years
Without creativity in identifying problems to be into viewing work as something that will be
addressed or methods to be used, a field of study graded. Peer reviews, editorial reviews, adminis
would atrophy. trative reviews, and granting agency reviews
If research has a core more important than occur regularly. Faculty work can be assessed on
anything else, it is creativity. Without creativity, an annual basis, with the expectation of prod
researchers would only repeat themselves. ucts in hand making the time frame for com
Without creativity, the questions we pose, the pleting work within a year or less. In graduate
starting place for research design, would be end schools, students are steered out of creative
lessly repetitive. Creativity allows the clientele of projects because such projects are too risky. It is
researchers—be they the public, practitioners, unfortunate when research is designed not out
or other researchers—to bring new ideas into of the possibility of success but rather out of the
their intellectual or practical lives. They can fear of failure. In the process, creativity—researchers’
agree or disagree with each other’s findings. best friend and asset—is shunted to the rear.
They can find fault in their methodologies. But Academic reproduction (Bourdieu, 1984/1988;
the new ideas remain as work to be examined, Bourdieu & Passeron, 1990) ensures repro
understood, enacted, selected, and retained for duction, not evolution. Creativity suffers in the
use (Weick, 1979). current context of conducting research, and
Getzels and Csikszentmihalyi (1976) describe producing valuable new understandings is ever
problem finding as being at the heart of the more difficult.
creative process. Educational researchers who
invent the best problems have the greatest
Fear and the Researcher’s Ego
chance of contributing to their fields. A good
problem implies the way in which it should be There are a number of personal factors that
studied. A superb methodology will not make affect research design. Morgan (1997) describes
up for a poor research problem. Structured pro “psychic prisons” as a metaphor for the ways
cesses for becoming more creative that empha in which our imaginations become trapped.
size steps to be followed have been identified Whatever neuroses we have can emerge in the
Chapter 11. Intellect, Light, and Shadow in Research Design 177
task of developing research. Researchers can fix There is a competitive aspect to designing
ate on certain ideas, repress others, idealize research. Instead of a “best knowledge for the
states, or project their own views on the data. A discipline” model, it involves “I got there first,”
frequently occurring form of projection occurs “I’m right and you’re wrong,” “I win the argu
when the conclusions of a research study are not ment,” “My theory is right,” “I got the grant and
connected to the data. Researchers project their you didn’t,” “My university is ranked higher than
beliefs onto the data, concluding what they your university,” and the like. These are the con
wanted to conclude before they began conduct cerns of the ego, extrinsic to the creation of
ing the study. knowledge, casting a shadow on research. From
Fear has an unacknowledged influence on the point of view of designing research to dis
research design that manifests itself in a variety cover knowledge, it is bizarre that information is
of ways. The first is internal censorship. Certain not shared. From the point of view that research
topics and methods are never given serious con is not about improving knowledge but rather is
sideration because to do so would be to invite about supporting the ego, making a name for
trouble, at least in the minds of the researchers. oneself, and providing research overhead to
For example, during the 1970s, many people one’s institution, it makes perfect sense. The
did not consider qualitative research to be an impulse is to design research in order to win
appropriate form of educational research. Fear some imaginary (or real) competition, not
ing rejection by colleagues, granting agencies, because it is vital to the field.
advisers, or editors, researchers steered them
selves away from the use of qualitative research.
Disciplinary Norms and Groupthink
It was not surprising that much of the emphasis
of Lincoln and Guba’s (1985) Naturalistic The kind of study a researcher can conduct
Inquiry is a justification for, and not an explana depends on the development of the field. Mature
tion of, this kind of study. Researchers engaged fields, such as the arts and sciences, medicine,
in self-censorship by avoiding Black studies, and engineering (Parsons & Platt, 1973), have a
women’s studies, GLBT (gay, lesbian, bisexual, long tradition of theories and methods that are
transgendered) studies, and the study of emo thought to be appropriate to use when conduct
tional aspects of organizational behavior ing research. Cultural studies, critical theory,
(Fineman, 2000). and other postmodern approaches have pre
Fear also underlies what has been called the ferred methods that keep other disciplines vital
“imposter syndrome” (Harvey & Katz, 1985), by challenging their traditions. Research norms
where researchers might fear that they are fakes. become institutionalized through accepting
This problem can show up in an obsessive need papers for professional meetings and publica
to review the literature because a researcher tion. A disciplinary language develops, and a
“doesn’t know enough yet.” A researcher might kind of parochialism develops in citation pat
fear not being professional enough or not being terns: Cite from journals in the field only.
thorough enough and might analyze data in an Disciplines within education and in the profes
endless pattern of trivial changes. This fear is a sions have become ever more specialized.
pathology, not a motivator. Research followed suit and led the way to disci
Research can also be conducted in service to plinary specialization.
the ego, not the discipline, where the researcher Research design reflects this specialization in
is driven by the extrinsic value of research. This topic and method. Specialization can have the
drive results in designing research for maximum advantage of accuracy and the disadvantage of
visibility, regardless of substance. Finding the triviality. Researchers who venture outside the
smallest publishable unit in a data set inflates norms can be transformational if they are lucky
one’s résumé but clutters journals. The ego or can be ignored or ridiculed if they are not.
thrives on high levels of productivity. The disci New ideas are sometimes blocked by the disci
pline thrives on high levels of quality. The cur plinary equivalent of groupthink. Groupthink,
rent research market defines what is acceptable, first described by Janis (1972), includes many
and clever marketing may be more important to factors that limit creativity and risk taking,
one’s ego than a quiet but long-term contribution including sharing stereotypes that guide the
to the field. decision, exerting direct pressure on others,
178 Section III. Opportunities and Challenges in Designing and Conducting Inquiry
maintaining the illusion of unanimity and Simple solutions are often wrong. Random
invulnerability, and using mind guards to pro ization means that individual student differ
tect the group from negative information. ences will not be a factor in the research and that
Groupthink is more than a norm; it is an all kinds of students can expect to benefit
exclusionary process designed to protect the equally from the program. The results are
group from outside influence. Groupthink in designed to mask individual differences to see
research can limit the topics studied and the whether the program worked for the majority. It
methodology used. The long period during works if the mean of the criterion variable for
which editors silenced the voices of women; the treatment group is significantly higher than
African Americans; and the gay, lesbian, bisex the mean of the control group. Like randomiza
ual, and transgendered community in education tion, means are designed to mask individual
is one example. Another currently exists among differences. Berliner (2002) makes the point that
those in education who support only “scientific there is a “ubiquity of interactions” and that a
research” (National Research Council, 2002). program could have remarkable positive effects
Berliner (2002) suggests that the problem is not on a small segment of the treated population,
one of science but rather one of politics and none of which would be discovered by this
money. Those who label good research in educa research design. A program could benefit gifted
tion as “scientific” are stuck in groupthink, as are students, African American students, girls, ath
those who consider the research method as letes, or special needs students in a manner
essential and all else as trivial. When methodol invisible to scientific methods.
ogy precedes identifying the problem to be stud Much of the contentiousness about educa
ied, groupthink wins and research suffers. tional research design centers on whether the
research is scientific, a desiderata identified by
the National Research Council’s (NRC) publica
Methodology and Methodological
tion of Scientific Research in Education (2002).
Correctness
The debate revolves around using scientific
At the extreme, the result is “methodological methodologies to examine educational issues.
correctness,” a term I coin as a play on “political The NRC’s position is generally supported by
correctness.” It is associated with taking oneself some (Feuer, Towne, & Shavelson, 2002; Slavin,
very seriously and is related to academic funda 2002) and cautioned against or rejected by oth
mentalism, where skepticism is replaced by ers (Berliner, 2002; Erickson & Gutierrez, 2002;
dogma. Methodological correctness means that Olson, 2004; St.Pierre, 2002). Research design,
the purpose of research is to optimize method research funding, and politics are intercon
ology. It is an example of goal displacement, nected (Burkhardt & Schoenfeld, 2003). The
where the purpose of research is no longer to Obama administration has done much to restore
find out something important but rather to use the importance of scientific knowledge in policy
method flawlessly. The hegemony of methodol making, but one can never assume that such a
ogists in determining the value of research has a change is permanent.
chilling effect on exploring new approaches to A research article, like the tip of an iceberg,
research, on studying topics not studied previ contains only a small percentage of the informa
ously, and on studying topics that do not lend tion that the author encountered in the study.
themselves to study using preferred methods. Given this situation, research becomes an enact
Institutionalized methodological correctness ment of the map–territory relationship, that is,
takes the form of guidelines, where if the guide the relationship between the object studied and
lines are not followed, the result is funding not the symbol for that object—the research report
being given or results not being taken seriously. (Bateson, 2000). How complete does the symbol
The U.S. Department of Education has provided need to be to represent some objective reality?
A User-Friendly Guide, one that is not “friendly” Borges (1998), in his story “On Exactitude in
at all, that can be summarized as follows: The Science,” provides a fictional example of an
only rigorous evidence that can be used to eval empire that was so enamored of mapmaking
uate an educational intervention comes from that the cartographers were encouraged to make
research using randomized controlled trials maps that were larger and more accurate. In the
(Institute of Education Sciences, 2003). end, they made a map that was so detailed, it
Chapter 11. Intellect, Light, and Shadow in Research Design 179
needed to be exactly the same size as the land it published. The imposter syndrome is ever-present:
described. As a map, it was perfectly useless. “How do I, as an editor, make decisions that will
In this case, greater accuracy and greater make me look like I know what I’m doing?” The
methodological correctness diminished utility. ordinary response is risk aversion: “If I don’t take
Bateson (2000) argues that maps are useful not chances, I’m least likely to look like an imposter.”
because they are literal representations but Editors are likely to reject methodologically
rather because they are in some way analogous flawed research in favor of methodologically
to reality. Research provides a map, an analog of correct research. Imaginatively flawed research,
reality. If Bateson is right, then it might be more research whose consequences are trivial for the
appropriate to design and evaluate research not discipline or practitioners, can be published if the
on the basis of how correct the methodology is methods are correct but with predictable disdain
or how literally it represents reality, but rather from the public (Kezar, 2000). I have heard of no
on how useful it is for understanding and acting cases where an editor has written an author say
in our environments. ing, “The ideas in this article are so compelling
that I’m going to publish it even though it con
tains obvious methodological flaws.” Editorial
The Primary Research Audience
referees work at the pleasure of the editor, and if
Research design is affected by the primary they are to be retained, they work in line with the
research audience for the study. For doctoral editorial vision. Reviewers are often shown the
students, the primary audience is their advisors comments of other referees so that they can
and other members of their research committees. compare their responses. Feedback provides an
For faculty, the primary audience is journal and implicit pressure to conform.
publishing house editors and grantors. Refereed The upward drift in methodology can be con
journal editors are the gatekeepers of much of sidered paradoxical. To get published, authors
the research that is published, which in turn use sophisticated methodologies. The newer and
influences what is taught and who is tenured and more complex the method is, the fewer the peo
promoted at schools that value research. ple who will be able to evaluate the article, and
Recognizing this power, a researcher responds the fewer the practitioners who will be able to
to the real or imagined preferences for topic or understand the research and judge whether using
method of this audience. The obvious way of the results would be beneficial. In attempting to
finding editorial preferences is to read the jour gain the approval of other researchers, a researcher
nal, see what is being published, and use a similar might not care whether an article advances prac
approach to one’s own study. Doctoral students tice in the field. Good research can do both; some
would be prudent to read dissertations directed publications do neither.
by a prospective dissertation advisor to see what
these preferences actually are. This situation
The Secondary Research Audience
begs the question, should these gatekeepers set
the research agenda? Editors of research journals It is a desirable state when secondary research
usually have been successful researchers in their audiences—other researchers, practitioners, and
fields and have published widely. The advisory the public—are more important than the pri
board that hires an editor increases a journal’s mary ones. From an altruistic perspective, it is
prestige by hiring the most prestigious editor it for these audiences that the research is con
can find. The editor then seeks out other suc ducted. Research should be designed to be useful
cessful researchers in the field and brings them to the discipline and to advance theoretical or
on board. This selection procedure produces a empirical understanding of what is happening in
conservative bias: It rewards what has worked some area of education. Does this research pro
in the past. vide new ideas, new understandings, and new
One model for the editorial process is that practices that advance the ways in which profes
reviewers have had long experience in the field sionals and practitioners in the field can serve the
and make prudent judgments about what stud public good? An affirmative answer would justify
ies will advance educational practice or knowl the use of public and philanthropic resources in
edge. Another model views editorial decisions as pursuit of educational knowledge. Good research
being on show because what editors approve is should benefit everybody.
180 Section III. Opportunities and Challenges in Designing and Conducting Inquiry
A measure of the value of research is not just To design research is to make a map, an anal
acceptability to the editorial process, that is, the ogy of what happens in the world. Research
merit indicated by its publication; rather, the design depends on what is being studied and
impact of a piece of research on theory or prac what the researcher wants to find out. The dou
tice becomes the ultimate measure of its value ble entendre of “wants to find out” is intentional.
or worth. Research that does not meet at least The researcher wants to find out something
minimal methodological acceptability is not about, say, how to improve literacy rates in rural
published and does not become available to its areas. The researcher also wants to find out that
potential audience. Assuming that it does reach a his or her hypothesis is true, for example, that
larger audience, does it affect future research in tutoring improves literacy.
the field? The choice of the topic focuses the endeavor.
A well-designed study should include, at the The choice of method limits what can be discov
end of the article, recommendations for future ered, emphasizing some possibilities and elimi
research, but in practice, these recommenda nating others. Each choice involves trade-offs.
tions tend to focus on narrow methodological Each methodology chosen should, if done well,
concerns, such as improving a questionnaire supply some beneficial information. There is
and using a different sample. The implicit rec one best way in which to find out something
ommendation for future researchers is that they extremely simple, such as the mean length of
continue to advance the theoretical orientation time it takes students to memorize a list of spell
of the line of research. A second concluding sec ing words. As the question addressed becomes
tion should deal with the practical applications broader and more complex, it can be studied
of the study. The form of the application is along using a variety of designs. There is no best way
these lines: “If your educational world is similar in which to study education; each approach
to the one in which this study was conducted, emphasizes some things and is silent on others.
here are the things you should do, based on my Political and research ideologies can drive
findings, that would improve educational prac research or be ignored.
tice and understanding in your world.” Research could be designed purely on the
Educational research can be influential not basis of curiosity if the researcher wants to know
because of its quality but rather because the find something. The methodology is likely to be
ings confirm what policy makers already believe. emergent as the researcher plays with the topic,
This situation is distressing because it means that thinking of it without preconception; delighting
an excellent study will affect policy only if policy in possibility; and creating an ongoing dialogue
makers want it to affect policy. When two studies with the topic, methods, other researchers in the
are excellent but lead to opposite conclusions, field, the persons being studied, and so on.
policy makers lose confidence in the research and Research can also be designed around extrin
return to intuition to set policy. The politics of sic reasons: “How can I make myself famous,
educational research seems to be one of its promoted, tenured, or rich on the basis of my
salient features (Cooper & Randall, 1999). research?” For that research, the researcher
should let money and disciplinary popularity
lead the endeavor. For research to affect policy,
Conclusion one should follow the money out of governmen
tal or other granting agencies and heed their
The reporting of research can be viewed as sto guidelines for topics and methods. Research
rytelling, as part of a mythic process of identify should be designed to meet their expectations
ing who we are. In storytelling, we seek to using methods they prefer. An effective presen
remember the past, invent the present, and envi tation of the results might demand that they be
sion the future (Keen & Valley-Fox, 1989). presented in the most simple or most mystifying
Research can be viewed as a similar process in forms.
remembering the past by examining the litera Designing research for altruistic purposes,
ture; inventing the present by conducting the to benefit humanity, is more complicated,
study and describing the findings; and envision because what benefits one group might not ben
ing the future where this research influences efit another. Any discovery can have wonderful
thought, policy, and practice. unanticipated consequences. Basic research has
Chapter 11. Intellect, Light, and Shadow in Research Design 181
grand possibilities, but the environment must In the near future, methodological correctness
thrive on patience and failure—on trying many will likely maintain its salience. I would expect
new things that do not work to find the few that humanistic and aesthetic values will be
that do. Such environments are rare. Research neglected in research in the face of issues of
designed to solve well-defined problems—applied social justice and pragmatism. Capitalistic ele
research—can also benefit humanity. Other ments related to the costs of education and the
applied research is intended to profit the patent ways in which the education system provides a
holder. Research designed to provide an educa suitable labor force for the nation’s economy
tional environment that will save humanity will likely be emphasized. Whatever work we
should get top billing, but who could agree on do or we neglect, our research must refresh
what that research would be? our intellect.
Keen, S., & Valley-Fox, A. (1989). Your mythic Education Research. Washington, DC: National
journey. New York: Putnam. Academies Press.
Kerlinger, F. (1973). Foundations of behavioral Neill, A., & Ridley, A. (Eds.). (1995). The philosophy
research (2nd ed.). New York: Holt, Rinehart & of art. New York: McGraw-Hill.
Winston. Olson, D. R. (2004). The triumph of hope over
Kezar, A. (2000). Higher education research at the experience in the search for “what works”: A
millennium: Still trees without fruit? Review of response to Slavin. Educational Researcher,
Higher Education, 23, 443–468. 33(1), 24–26.
Krathwohl, D. R. (1988). How to prepare a research Parnes, S. J. (1992). Source book for creative problem
proposal: Guidelines for funding and dissertations solving. Buffalo, NY: Creative Foundation Press.
in the social and behavioral sciences (3rd ed.). Parsons, T., & Platt, G. M. (1973). The American
Syracuse, NY: Syracuse University Press. university. Cambridge, MA: Harvard University
Lawrence-Lightfoot, S. (1995). I’ve known rivers: Press.
Lives of loss and liberation. New York: Penguin Schwartzman, H. (1978). Transformations: The
Books. anthropology of children’s play. New York:
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic Plenum.
inquiry. Beverly Hills, CA: Sage. Schwartzman, H. (1993). Ethnography in
Merton, R. K. (1973). The normative structure of organizations (Qualitative Research Methods
science. In N. W. Storer (Ed.), The sociology of Series, No. 27). Newbury Park, CA: Sage.
science (pp. 267–278). Chicago: University of Shils, E. (1984). The academic ethic. Chicago:
Chicago Press. (Original work published 1942) University of Chicago Press.
Milgram, S. (1974). Obedience to authority. Slavin, R. E. (2002). Evidence-based education
New York: Harper & Row. policies: Transforming educational practice
Mitroff, I. I., & Kilmann, R. H. (1978). and research. Educational Researcher, 31(7),
Methodological approaches to social science. San 15–21.
Francisco: Jossey-Bass. St.Pierre, E. A. (2002). “Science” rejects
Morgan, B. (1997). Images of organization (2nd ed.). postmodernism. Educational Researcher, 31(8),
Thousand Oaks, CA: Sage. 25–27.
Myers-Briggs, I. (1962). Manual for the Myers-Briggs Thorngate, W. (1976). In general vs. it all depends:
Type Indicator. Princeton, NJ: Educational Some comments on the Gergen–Schlenker
Testing Service. debate. Academy of Management Journal, 25,
National Research Council. (2002). Scientific research 185–192.
in education (R. J. Shavelson & L. Towne, Eds.). Weick, K. (1979). The social psychology of organizing
Committee on Scientific Principles for (2nd ed.). Reading, MA: Addison-Wesley.