Professional Documents
Culture Documents
Ann Brown
Ann Brown
Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless
you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you
may use content in the JSTOR archive only for your personal, non-commercial use.
Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at .
http://www.jstor.org/action/showPublisher?publisherCode=taylorfrancis. .
Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed
page of such transmission.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.
Taylor & Francis, Ltd. is collaborating with JSTOR to digitize, preserve and extend access to The Journal of
the Learning Sciences.
http://www.jstor.org
THE JOURNALOF THE LEARNINGSCIENCES,2(2), 141-178
CopyrightO 1992, LawrenceErlbaumAssociates,Inc.
Design Experiments:Theoretical
and Methodological Challengesin
Creating Complex Interventions
in Classroom Settings
Ann L. Brown
Universityof California-Berkeley
DESIGN EXPERIMENT
Contributions Cl
to LearningTheory Te
Cu
TA
Engineering a Working
Environment
Output
Assessment of the right things
Accountability
Metacognition
READING COMPREHENSION
COMMUNITIESOF LEARNERS
TABLE1
Changes in Classroom Philosophy
IntentionalLearning
Role TraditionalClassroom Environment
Students Passiverecipientsof Studentsas researchers,teachers,
incominginformation and monitorsof progress
Teachers Didacticteaching Guideddiscovery
Classroommanager Modelof activeinquiry
Content Basicliteracycurriculum Thinkingas basic literacy
Lowervs. higherskills
Contentcurriculum Contentcurriculum
Breadth Depth
Fragmented Recurrentthemes
Fragmented Explanatorycoherence
Fact retention Understanding
Computers Drill and practice Tools for intentionalreflection
Programming Learningand collaboration
Assessment Fact retention Knowledgediscoveryand utilization
Traditionaltests Performance
Projects
Portfolio
METHODOLOGICALISSUES IN EVALUATING
COMPLEXINTERVENTIONSTUDIES
100.
80,
SQ
a.
E 60
a
x
us
w
Simple
c. Advanced
- 40
o
o
2
40
0
0
9 months
Explanations
80 -
Impasse Driven
CL
i-
Resolving Inconsistencies
Deeper Mechanisms
60
40 "
0
0 0
0 20
0.
0
0.
Unit 1 Unit 2 Unit 3
9 Months
To givejustoneexampleof thismixedapproach,considerourtreatment
of contentknowledge acquisition. Wegiveshortanswerquizzeson a pretest
andposttestbasison all unitsof coveredcontent.An exampleof suchdata
is shownin Figure4. Herethe performance of approximately 60 fifth-and
sixth-grade children in the experimental researchclassrooms is compared
witha partialcontrolgroup(PCont)thatwastreatedexactlythesameasthe
researchclassroomforthe firstsemester(Unit1), andthentaughtenviron-
mentalscienceby theirregularscienceteacher.Thepartialcontrolgroup
hadexactlythesameaccessto books,videos,computers, andso forthasdid
theresearch classrooms. As canbe seen,thetwogroupsdidnotdifferfrom
eachotherin Unit 1, wheretheywere,in effect,bothexperimental groups,
treatedexactlythe same;but the childrenin the researchclassrooms
outperformed thepartialcontrolgroupon bothUnits2 and3. A readonly
control(ROC)group,whoreadthe keymaterialsbutdidno research,did
poorlythroughout.This traditionaluse of staticpretestsand posttests,
combinedwithappropriate controldata,providesus withclearevidenceof
the effectivenessof the interventionand is easy to sharewith school
personnelas wellas fellowscientists.
A richerpictureof knowledgeacquisition,however,comesfrommore
in-depthprobingof a subsetof students.Forexample,weconsiderstudents'
responsesto clinicalinterviewsandtransfertests. Studentsdifferin their
levelof understanding andin theconfidence withwhichtheyholdopinions;
in the clinicalinterviewit is the skilledinterviewer'sjob to posequestions
that are sensitiveto the bandwidthof competencewithinwhicheach
individualstudentcan navigate(Brown& Reeve,1987).Thisis a formof
dynamicassessment(Campione& Brown, 1987, 1990)that we use to
measureemergent competence in general(seeCampioneet al., in press,for
dynamic assessment of computeruse in the communityof learners).The
point is to map the zone of proximaldevelopment (Vygotsky,1978),the
region competence a studentcantraversewithandwithoutaid.
of that
To mapthiswindowof opportunity for learning,theinterviewer raisesa
seriesof keyquestionsconcerning, forexample,thefoodchain,adaptation,
and so forth. First,the interviewer elicitsbasic expositoryinformation
(Whatdoesthestudentknowaboutphotosynthesis?). If thestudentcannot
answer,theinterviewer provideshintsandexamplesas necessary to testthe
student'sreadinessto learnthat concept.If, however,the studentseems
initiallyknowledgeable, the experimenter mightquestionthatunderstand-
ingby introducing counterexamples to thestudent's beliefs(Isa mushroom
a plant?Whataboutyeast?),and,againif appropriate, he or shemightask
the studentto engagein thoughtexperimentsthat demandnovel uses of the
information.For example,whena studenthas sortedpicturesperfectlyinto
herbivoresand carnivores,and provideda good descriptionof the differ-
ence, he or she may be asked, "What would happen on the African plain if
KNOWLEDGE
TEST. UNITI
80
POST
70 POST
60
W 50
0
Q 40
30
20
10
RESEARCH PCont FCC
GROUP
KNOWLEDGE
TEST,UNITII
80
PFE
POST
70
60
0
ch 50
40
30
RESEARCH PCont FC
GROUP
KNOWLEDGE
TEST,UNITIII
80 FIGURE4 Pre- and posttest
short answerknowledgetests for
70 POST three curriculumunits of envir-
onmental science. The data are
60 from "RestructuringGradeSchool
w 50 Learning Environmentsto Pro-
mote Scientific Literacy"in Re-
40 structuringLearning:Analysisand
Recommendationsof the Council
30
of Chief State School Officersby
A. L. Brownand J. C. Campione,
20
in press,Washington,DC:Council
10 of Chief State School Officers.
RESEARCH PCont FDC Copyright by Ann L. Brown.
GROUP Adaptedby permission.
158
DESIGNEXPERIMENTS 159
there were no gazelles or other meat for cheetahsto eat? Could they eat
grain?" Some students are surprisinglyuncertain about this, and may
suggestthat cheetahscould eat grainundercertaincircumstances,although
they would not live happily. Some even entertaina criticalperiodhypoth-
esis-that the cheetah could change if it were forced to eat grain from
infancy, but once it reachedadolescence,it would be too set in its ways to
change.Onlya few invokenotionsof form and function,suchas properties
of the digestive tract, to support the assertion that cheetahs could not
changewithin their life span. These extensionactivitiesof thoughtexperi-
ments and counterexamplesare far more revealingof the currentstate of
students' knowledge than their first unchallengedanswers, which often
providean overly optimisticpictureof their knowledge.
For example, Katy, a sophisticatedseventh grader, gave a textbook-
perfect descriptionof photosynthesisthat would certainlybe taken as an
indicationthat she fully understoodthe basic mechanisms.She was then
asked, "What would happen if there were 30% less sunlight?"Katy's
responseneverincludedthe criticalinformationthat as plants make food
with the sun'senergy,seriousreductionin the availabilityof sunlightwould
disrupt the entire food chain-no sun, no plants, no food! Instead she
concentratedon light to see with:
That would kill off the plants, beetles, and, um, nocturnalthings would be
OK.Thedayturnalthings- snakes,rabbits,hares- wouldbe alright,couldbe
nocturnal.But the dayturnalthingswouldneed sunlightto see- couldn'tfind
theirfoodinthedarkandwouldeventually
starveto death.Hawkswouldalso
die out, but owls are nocturnal-would be able to see at night and, um,
raccoonswould probablybe near the top of the food chain.
POST1
POST2
4
Co
z
O
w
O
P-u
UJ
z
~Iz
1
TAUGHT TAUGHT/NOVEL NOVEL ALLTYPES
GROUPS
FIGURE5 Use of taughtversusnovel biologicalprinciplesin the designan animal
transfertest.
Settingasidepurelymethodologicalissues,thereis anotherimportantset of
concernsthat face the designerof educationalinterventions.Thereis a long
history of attemptsto reconstructlearningand work environments.Con-
temporaryattempts cannot be viewed in an historical vacuum. In this
sectionI look at lessonsto be learnedfrom some of the most famousdesign
experiments.I referto these as: (a) The HawthorneEffect (Roethlisberger
& Dickson, 1939)-the nature of control in multifacetedinterventions,
(b) The DeweyEffect- the romanticizationof the processof discovery,and
(c) The Reality Principle-the problem of the shelf life of successful
interventions.
But note that this illusion of control, or real control, is one of the things
I want to happen in my classrooms. I want students to act as consultants,
to be coinvestigators of their own learning (Scardamalia & Bereiter, 1983),
to take charge of their own learning environment to the extent possible
(Brown, 1985). On the ground that the best defense is a good offense, I
argue that, redefined in this way, the Hawthorne effect is exactly what I am
aiming for in my classrooms. The Hawthorne experiments have a great deal
to say about worker and student responsibility and satisfaction.
The Hawthorne effect also has a great deal to say about learning in
complex social environments. Consider an interesting point: Why was the
intervention perceived to be a failure; why does it serve as a cautionary tale
in secondary sources? Actually, only the experimenters thought it was a
failure. Management was delighted with the improved esprit de corps,
productivity, reduced drop-out rate, and so forth. The experimenters'
feeling of failure is illustrated nicely in the following quotation
(Roethlisberger & Dickson, 1939):
The difficulty, however, went much deeper than the personal feelings of
failureof the investigators.Theywereentertainingtwo incompatiblepointsof
view. On the one hand, they weretryingto maintaina controlledexperiment
in which they could test for the effects of single variableswhile holding all
other factorsconstant.On the otherhand, theyweretryingto createa human
situationwhichremainedunaffectedby theirown activities.It becameevident
that in human situationsnot only was it practicallyimpossibleto keep all
other factors constant, but trying to do so in itself introducedthe biggest
change of all; in other words, the investigatorshad not been studyingan
ordinaryshopsituationbut a sociallycontrivedsituationof theirown making.
With this realization,the inquirychangedits character.No longerwere the
investigatorsinterestedin testing for the effects of single variables.In the
place of a controlled experiment,they substitutedthe notion of a social
situation which needed to be describedand understood as a system of
interdependent elements. (p. 185)
166 BROWN
that they can discover new commonalties for themselves. As Bruner argued
(1969), education:
CONCLUSION
ACKNOWLEDGMENTS
An earlier version of this paper was presentedat the meetings of the
AmericanEducationalResearchAssociation, Chicago, 1991 in a sympo-
sium entitled"Onparadigmsand methods:What do you do whenthe ones
you know don't do what you want them to?"
DESIGN EXPERIMENTS 175
REFERENCES
Brown, A. L., & Campione, J. C. (1979). The effects of knowledge and experience on the
formation of retrieval plans for studying from texts. In M. M. Gruneberg, P. E. Morris, &
R. N. Sykes (Eds.), Practical Aspects of Memory (pp. 378-384). London: Academic.
Brown, A. L., & Campione, J. C. (1986). Psychological theory and the study of learning
disabilities. American Psychologist, 41(10), 1059-1068.
Brown, A. L., & Campione, J. C. (1990). Communities of learning and thinking, or a context
by any othername. HumanDevelopment,21, 108-125.
Brown, A. L., & Campione, J. C. (in press). Restructuring grade school learning environments
to promotescientificliteracy.In Restructuring
learning:Analysisand recommendations
of
the Council of Chief State School Officers. Washington, DC: Council of Chief State School
Officers.
Brown, A. L., Campione, J. C., Reeve, R. A., Ferrara, R. A., & Palincsar, A. S. (1991).
Interactive learning and individual understanding: The case of reading and mathematics. In
L. T. Landsmann(Ed.), Culture,schoolingandpsychologicaldevelopment(pp. 136-170).
Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Brown, A. L., Campione, J. C., Webber, L. S., & McGilly, K. (1992). Interactive learning
environments - a new look at assessment and instruction. In B. R. Gifford & M. C.
O'Connor (Eds.), Future assessment:Changing views of aptitude, achievementand
instruction (pp. 121-213). Boston: Academic.
Brown, A. L., & French, L. A. (1979). The zone of potential development: Implications for
intelligence testing in the year 2000. Intelligence, 3, 253-271.
Brown, A. L., & Kane, M. J. (1988). Preschool children can learn to transfer: Learning to
learn and learning from example. Cognitive Psychology, 20, 493-523.
Brown, A. L., & Palincsar, A. S. (1982). Inducing strategic learning from texts by means of
informed, self-control training. Topics in Learning and Learning Disabilities, 2(1), 1-17.
Brown, A. L., & Palincsar, A. S. (1987). Reciprocal teaching of comprehension strategies: A
natural history of one program for enhancing learning. In J. D. Day & J. G. Borkowski
(Eds.), Intelligenceand exceptionality:New directionsfor theory,assessment,and instruc-
tional practice (pp. 81-132). Norwood, NJ: Ablex.
Brown, A. L., & Palincsar, A. S. (1989). Guided, cooperative learning and individual
knowledge acquisition. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays
in honor of Robert Glaser (pp. 393-451). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Brown, A. L., & Reeve, R. A. (1987). Bandwidths of competence: The role of supportive
contexts in learning and development. In L. S. Liben (Ed.), Development and learning:
Conflict or congruence? (pp. 173-223). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Bruner, J. S. (1963). The process of education. Cambridge, MA: Harvard University Press.
Bruner, J. S. (1969). On knowing: Essays for the left hand. Cambridge, MA: Harvard
University Press.
Campione, J. C. (1989). Assisted assessment: A taxonomy of approaches and an outline of
strengthsand weaknesses.Journalof LearningDisabilities,22, 151-165.
Campione, J. C., & Brown, A. L. (1987). Linking dynamic assessment with school achieve-
ment. In C. S. Lidz (Ed.), Dynamic assessment (pp. 82-115). New York: Guilford.
Campione, J. C., & Brown, A. L. (1990). Guided learning and transfer: Implications for
approaches to assessment. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Shafto (Eds.),
Diagnosticmonitoringof skill and knowledgeacquisition(pp. 141-172). Hillsdale, NJ:
Lawrence Erlbaum Associates, Inc.
Campione, J. C., Brown, A. L., & Connell, M. L. (1988). Metacognition: On the importance
of understanding what you are doing. In R. I. Charles & E. A. Silver (Eds.), Research
agendafor mathematicseducation:The teachingand assessingof mathematicalproblem
solving (pp. 93-114). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Campione, J. C., Brown, A. L., & Jay, M. (in press). Computers in a community of learners.
DESIGNEXPERIMENTS 177
In E. De Corte, M. Linn, H. Mandl, & L. Verschaffel (Eds.), Computer-based learning
environments and problem solving. (NATO ASI Series. F: Computer and Systems Series).
Berlin: Springer.
Campione, J. C., Rutherford, M., Gordon, A., & Brown, A. L. (in press). Communities of
learners: Zones of development for those at risk. In N. C. Jordan & J. Goldsmith-Phillips
(Eds.), Learningdisabilities:New directionsfor assessmentand intervention.Needham
Heights, MA: Allyn and Bacon.
Chi, M. T. H. (1976). Short-term memory limitations in children: Capacity or processing
deficits?Memoryand Cognition,4, 559-572.
Chi, M. T. H. (1978). Knowledge structures and memory development. In R. S. Siegler (Ed.),
Children's thinking: What develops? (pp. 73-96). Hillsdale, NJ: Lawrence Erlbaum
Associates, Inc.
Cohen, D. (1989). Teaching practice: Plus ga change ... In P. Jackson (Ed.), Contributing to
educationalchange: Perspectiveson researchand practice (pp. 27-84). Berkeley,CA:
McCutchan. (Also published by the National Center for Research on Teacher Education.
Michigan: Michigan State University, 88-3, September 1988.)
Cole, M., & Bruner, J. S. (1971). Cultural differences and inferences about psychological
processes. American Psychologist, 26, 867-876.
Collins, A. (in press). Toward a design science of education. In E. Scanlon & T. O'Shea (Eds.),
New directionsin educationaltechnology.New York:Springer-Verlag.
Collins, A., & Stevens, A. (1982). Goals and strategies of inquiry teachers. In R. Glaser (Ed.),
Advances in instructional psychology (Vol. 2, pp. 65-119). Hillsdale, NJ: Lawrence
Erlbaum Associates, Inc.
Cremin,L. A. (1961). Thetransformation
of theschool:Progressivismin Americaneducation
1876-1957. New York: Knopf.
Cuban, L. (1984). How teacherstaught: Constancyand change in Americanclassrooms,
1890-1980. New York: Longman.
Cuban,L. (1986).Teachersandmachines:Theclassroomof technologysince1920.NewYork:
Teachers College Press.
Cuban, L. (1990). Reforming again, again and again. Educational Researcher, 13, 3-13.
Darwin, C. (1877). A biographical sketch of an infant. Mind, 2, 286-294.
DeLoache, J. S., Brown, A. L., & Kane, M. J. (1985, April). Young children's error correction
procedures in manipulative play. Paper presented at the meeting of the Society for Research
in Child Development, Toronto.
Dewey, J. (1900). The school and society. Chicago: University of Chicago Press.
Dewey, J. (1901). The situation as regards the course of study. Journal of Proceedings and
Addressesof theFortiethAnnualMeetingof theNationalEducationAssociation,332-348.
Dewey, J. (1929). My pedagogical creed. Washington, DC: Progressive Education Association.
Dewey, J. (1931). The way out of educational confusion. Cambridge, MA: Harvard University
Press.
Dow, P. (1991). Schoolhouse politics. Cambridge, MA: Harvard University Press.
Fish, S. (1980). Is there a text in this class? The authorityof interpretivecommunities.
Cambridge, MA: Harvard University Press.
Flavell, J. H. (1971). First discussant's comments: What is memory development the
development of? Human Development, 14, 272-278.
Flavell, J. H., & Wellman, H. M. (1977). Metamemory. In R. V. Kail, Jr. & J. W. Hagen
(Eds.),Perspectiveson thedevelopmentof memoryandcognition(pp. 3-33). Hillsdale,NJ:
Lawrence Erlbaum Associates, Inc.
Gentner, D. (1983). Structure-mapping: A theoretical framework for analogy. Cognitive
Science,7, 155-170.
Gentner, D. (1989). The mechanisms of analogical learning. In S. Vosniadou & A. Ortony
178 BROWN