classifying the purposes of research
Description, justification and clarification: a
framework for classifying the purposes of research
in medical education
David A Cook,1 Georges Bordage2 & Henk G Schmidt3
CONTEXT Authors have questioned the degree to
which medical education research informs practice
and advances the science of medical education.
OBJECTIVE This study aims to propose a framework
for classifying the purposes of education research and
to quantify the frequencies of purposes among
medical education experiments.
METHODS We looked at articles published in 2003
and 2004 in Academic Medicine, Advances in Health
Sciences Education, American Journal of Surgery, Journal of
General Internal Medicine, Medical Education and
Teaching and Learning in Medicine (1459 articles).
From the 185 articles describing education experi-
ments, a random sample of 110 was selected. The
purpose of each study was classified as description
(ÔWhat was done?Õ), justification (ÔDid it work?Õ) or
clarification (ÔWhy or how did it work?Õ). Educational
topics were identified inductively and each study was
classified accordingly.
RESULTS Of the 105 articles suitable for review, 75
(72%) were justification studies, 17 (16%) were
description studies, and 13 (12%) were clarification
studies. Experimental studies of assessment methods
(5 ⁄ 6, 83%) and interventions aimed at knowledge
and attitudes (5 ⁄ 28, 18%) were more likely to be
1
Division of General Internal Medicine, Mayo Clinic College of
Medicine, Rochester, Minnesota, USA
2
Department of Medical Education, University of Illinois at
Chicago, Chicago, Illinois, USA
3
Department of Psychology, Erasmus University, Rotterdam, The
Netherlands
Correspondence: David A Cook, Division of General Internal
Medicine, Mayo Clinic College of Medicine, Baldwin 4-A, 200 First
Street SW, Rochester, Minnesota 55905, USA.
Tel: 00 1 507 538 0614; Fax: 00 1 507 284 5370;
E-mail: cook.david33@mayo.edu
128 ª Blackwell Publishing Ltd 2008. MEDICAL EDUCATION 2008; 42: 128–133
clarification studies than were studies addressing
other educational topics (< 8%).
CONCLUSIONS Clarification studies are uncommon
in experimental studies in medical education. Studies
with this purpose (i.e. studies asking: ÔHow and why
does it work?Õ) are needed to deepen our under-
standing and advance the art and science of medical
education. We hope that this framework stimulates
education scholars to reflect on the purpose of their
inquiry and the research questions they ask, and to
strive to ask more clarification questions.
KEYWORDS *education, medical; *research, biomedical; health
knowledge, attitudes, practice; clinical competence; evidence-based
medicine; teaching ⁄ methods; review [publication type].
Medical Education 2008: 42: 128–133
doi:10.1111/j.1365-2923.2007.02974.x
INTRODUCTION
Medical education research has witnessed substantial
growth in recent years,1,2 yet there are indications
that such research may not be informing educational
practice. For example, a recent best evidence medical
education (BEME) review found weak evidence
supporting several conditions for effective high-fidel-
ity simulation but noted that few or no strong studies
could be found.3 Cook has argued that virtually all
existing research into web-based learning has little
meaningful application.4 On a more general basis,
Norman noted that curriculum-level research is
hopelessly confounded and fails to advance the
science.5,6
What is the solution to this seemingly disappointing
state of affairs? Authors have, in recent years, called
for greater use of theory and conceptual frame-
works,7–9 focus on relevant topics,10 more meaningful

Overview
What is already known on this subject
Despite recent growth in medical education
research, there are indications that such
research may not be informing educational
practice.
What this study adds
We propose a framework for classifying the
purposes of education research, namely,
description, justification and clarification.
Description studies address the question: ÔWhat
was done?Õ Justification studies ask: ÔDid it
work?Õ Clarification studies ask: ÔWhy or how did
it work?Õ
Clarification studies are uncommon among
medical education experiments.
Suggestions for further research
More clarification studies are needed to deepen
our understanding and advance the art and
science of medical education.
outcomes,11–13 and improvements in research meth-
ods and reporting.6,14–17 This is sound advice, but
research in medical education still suffers from a
more fundamental and apparently widespread prob-
lem, namely, the lack of a scientific approach. By this
we are not implying the absence of scientific study
designs and methods, for many studies employ
rigorous designs. Rather, we have noted that many
research studies in medical education fail to follow a
line of inquiry that over time will advance the science
in the field. Research that has advanced our knowl-
edge of medicine, as well as our knowledge in other
fields such as biology, chemistry and astrophysics, is
characterised by a cycle of observation, formulation
of a model or hypothesis to explain the results,
prediction based on the model or hypothesis, and
testing of the hypothesis, the results of which form
the observations for the next cycle. Not only are the
steps in this cycle – otherwise known as the scientific
method – important, but the iterative nature of the
process is essential because the results of one study
provide a foundation upon which subsequent studies
can build. It is through this process that we have
come to understand how congestive heart failure
develops, how penicillin kills bacteria, and why the
ª Blackwell Publishing Ltd 2008. MEDICAL EDUCATION 2008; 42: 128–133
Description, justification and clarification
planets revolve around the sun. More importantly,
this information allows us to predict ways to treat
congestive heart failure, design new antibiotics and
send rockets to the moon.
Unfortunately, this approach seems relatively
uncommon in medical education research. Instead,
medical educators focus on the first step (observa-
tion) and the last step (testing), but omit the middle
steps (model formulation or theory building, and
prediction) and, perhaps more importantly, fail to
maintain the cycle by building on previous results.
This in turn limits our ability to predict which
instructional methods and assessments will enhance
our educational efforts. This could be the underlying
cause of recent laments regarding the paucity of
useful education research. To better understand this
problem and identify potential solutions, it is helpful
to consider the purposes of research, which we have
divided into 3 categories: description; justification,
and clarification.
Description studies focus on the first step in the
scientific method, namely, observation. Examples
would be a report of a novel educational intervention,
a proposal for a new assessment method, or a
description of a new administrative process. Such
reports may or may not contain outcome data, but by
definition they make no comparison.
Justification studies focus on the last step in the
scientific method by comparing one educational
intervention with another to address the question
(often implied): ÔDoes the new intervention work?Õ
Typical studies might compare a problem-based
learning (PBL) curriculum with a traditional
curriculum, compare a new web-based learning
course with a lecture on the same topic, or compare a
course to improve phlebotomy skills with no inter-
vention. Rigorous study designs such as randomised
trials can be used in such studies but, without prior
model formulation and prediction, the results may
have limited application to future research or
practice.
Clarification studies employ each step in the scientific
method, starting with observations (typically building
on prior research) and models or theories, making
predictions, and testing these predictions. Confirma-
tion of predictions supports the proposed model,
whereas results contrary to predictions can be just as
enlightening – if not more – by suggesting ways in
which the theory or model can be modified and
improved. Such studies seek to answer the questions:
ÔHow does it work?Õ and ÔWhy does it work?Õ Such
129
D A Cook et al
research is often performed using classic experi-
ments, but correlational research, comparisons
among naturally occurring groups and qualitative
research can also be used. The hallmark of clarifica-
tion research is the presence of a conceptual frame-
work or theory that can be affirmed or refuted by the
results of the study. Such studies will do far more to
advance our understanding of medical education
than either description or justification studies alone.
Indeed, description and justification studies, by
focusing on existing interventions, look to the past,
whereas clarification studies illuminate the path to
future developments.
Schmidt, during the 2005 meeting of the Association
for Medical Education in Europe, presented the
results of a systematic survey of the literature on
PBL.18 From the 850 studies identified, representing
a variety of study designs, he noted 543 (64%)
description studies, 248 (29%) justification studies,
and only 59 (7%) clarification studies. To build on
these findings, we sought to determine the frequen-
cies of study purposes across a variety of educational
topics, using systematic methods and focusing on
recent literature. To address this aim, we conducted a
limited but systematic review targeting experimental
studies in medical education. The present study was
an extension of a larger systematic review of the
quality of reporting of medical education experi-
ments.17 In light of SchmidtÕs findings, we hypothes-
ised that the proportion of clarification studies would
be low. Given the experimental nature of the studies
sampled, we further anticipated that most study
purposes would fall into the justification or
clarification categories.
METHODS
We selected the 4 leading journals that focus exclu-
sively on medical education research (Academic Med-
icine, Advances in Health Sciences Education, Medical
Education and Teaching and Learning in Medicine) and
2 specialty journals – 1 for surgery and 1 for
medicine – that frequently publish education
research (the American Journal of Surgery and the
Journal of General Internal Medicine). We systematically
searched these journals for reports of experiments
published in 2003 and 2004. We defined experiments
broadly as any study in which researchers manipulate
a variable (also known as the treatment, intervention
or independent variable) to evaluate its impact on
other (dependent) variables, and included all studies
that met this definition, including evaluation studies
with experimental designs. Abstract-length reports
130 ª Blackwell Publishing Ltd 2008. MEDICAL EDUCATION 2008; 42: 128–133
were excluded. Based on a review of titles and
abstracts of the 1459 articles contained in the 6
journals during the 2-year study period, 185 experi-
ments were identified. Of these we selected a random
sample, stratified by journal, of 110 articles for full
review. Full text review by pairs of researchers
revealed that 5 studies were not experiments, leaving
105 articles for final analysis. For details of this
search, including justifications for journal selection
and sample size, see Cook et al.17
Each full article included in the study sample was
reviewed independently by 2 authors (DAC and GB)
and the study purpose was classified as description,
justification or clarification using the definitions in
Table 1. When more than one purpose was identified
in a given article, only the highest level was reported.
Studies were also classified according to the topic of
medical education addressed by the study. Topics
were identified inductively by reviewing each study to
identify themes. Disagreements were resolved by
discussion and consensus was reached on final
ratings. Inter-rater agreement was determined
using intra-class correlation coefficients (ICCs).19
Table 1 Purposes of medical education research studies
Description
Describes what was done or presents a new conceptual
model. Asks: ÔWhat was done?Õ There is no comparison group.
May be a description without assessment of outcomes, or
a Ôsingle-shot case studyÕ (single-group, post-test only
experiment)
Justification
Makes comparison with another intervention with intent
of showing that 1 intervention is better than (or as good as)
another. Asks: ÔDid it work?Õ (Did the intervention achieve the
intended outcome?). Any experimental study design with a
control (including a single-group study with pre- and
post-intervention assessment) can do this. Generally lacks a
conceptual framework or model that can be confirmed or
refuted based on results of the study
Clarification
Clarifies the processes that underlie observed effects.
Asks: ÔWhy or how did it work?Õ Often a controlled
experiment, but could also use a case–control, cohort or
cross-sectional research design. Much qualitative research
also falls into this category. Its hallmark is the presence of a
conceptual framework that can be confirmed or refuted by
the results of the study
 Description, justification and clarification

Chi-square test or FisherÕs exact test were used for
subgroup comparisons. Analyses were performed with
SAS 9.1 (SAS Institute, Cary, NC, USA).
RESULTS
The most frequent study purpose was justification (75
studies [72%]), followed by description (17 [16%])
and clarification (13 [12%]). Inter-rater agreement
for these classifications was 0.48, indicating
moderate agreement according to Landis and
Koch.20 All disagreements were resolved by
consensus.
Five topics of medical education were inductively
identified: instruction aimed at knowledge and atti-
tudes (28 articles); instruction for practice-based
learning and clinical diagnosis (history and
examination, diagnosis, evidence-based medicine [39
articles]); instruction for clinical interventions
(counselling, prescribing, procedural skills [19
articles]); teaching and leadership (13 articles), and
evaluations of assessment methods (6 articles). The
purposes of studies varied significantly across
educational topics (P = 0.0002) (Table 2). Notably,
experimental studies of assessment methods (5 ⁄ 6,
83%) and interventions aimed at knowledge and
attitudes (5 ⁄ 28, 18%) were more likely to be
clarification studies than were studies addressing
other educational topics (< 8%).
There were wide differences in study purposes across
journals, but these differences were not statistically
significant (P = 0.08) (Table 3).

Table 2 Frequencies of purposes of experimental studies according to educational topic

Knowledge ⁄ Clinical Clinical Teaching and

†
attitude diagnosis* interventions leadership Assessment‡
n = 28 n = 39 n = 19 n = 13 n=6 Total
n (%) n (%) n (%) n (%) n (%) n (%)

Description 4 (14) 9 (23) 1 (5) 2 (15) 1 (17) 17 (16)

Justification 19 (68) 29 (74) 17 (90) 10 (77) 0 (0) 75 (72)
Clarification 5 (18) 1 (3) 1 (5) 1 (8) 5 (83) 13 (12)

Significant difference (P = 0.0002) for comparison of purposes across all topics

* Clinical diagnosis: instruction for practice-based learning and clinical diagnosis (history and physical examination, diagnosis, evidence-based
medicine)
 
Clinical interventions: instruction for clinical interventions (counselling, prescribing, procedural skills)
à
Assessment: evaluations of assessment methods

Table 3 Frequencies of purposes of experimental studies according to journal of publication

Adv Health Teach Learn

Sci Educ Am J Surg Acad Med J Gen Int Med Med Educ Med
n=5 n=7 n = 23 n = 15 n = 37 n = 18
n (%) n (%) n (%) n (%) n (%) n (%)

Description 1 (20) 0 (0) 2 (9) 2 (13) 10 (27) 2 (11)

Justification 2 (40) 6 (86) 15 (65) 13 (87) 24 (65) 15 (83)
Clarification 2 (40) 1 (14) 6 (26) 0 (0) 3 (8) 1 (6)

No significant difference (P = 0.08) for comparison of purposes across all journals

Adv Health Sci Educ = Advances in Health Sciences Education; Am J Surg = American Journal of Surgery; Acad Med = Academic Medicine; J
Gen Int Med = Journal of General Internal Medicine; Med Educ = Medical Education; Teach Learn Med = Teaching and Learning in Medicine

ª Blackwell Publishing Ltd 2008. MEDICAL EDUCATION 2008; 42: 128–133 131
D A Cook et al

lates reflection on existing research and encourages

DISCUSSION
researchers to ask questions that will clarify our
understanding. It would be a misuse of the classifi-
As hypothesised, experimental studies in medical
cation if it were the sole criterion by which to grade
education were mainly conducted for justification or
existing research. Likewise, although we do imply
clarification purposes compared with descriptions
some degree of value judgment attached to research
(84% versus 16%). Furthermore, as in SchmidtÕs
among these purposes, it would be inappropriate to
study on PBL,18 clarification studies were uncommon:
reject a given study solely because it represented
only 12% of the experiments in our sample of studies
description or justification. There will always be a role
asked ÔHow?Õ or ÔWhy?Õ questions. If this trend
for such research as long as each piece of research
continues, the science of medical education is
contributes to a growing body of knowledge that
destined to advance very slowly. To alter this trend, we
advances the science of medical education.
call for greater use of theoretical or conceptual
models or frameworks in the conception and design
This study was limited to experimental studies
of medical education studies.
published in 6 journals in a 2-year period. By requiring
the reporting of a dependent variable, we effectively
While calling for more clarification research, we also
excluded a large number of description studies, which
acknowledge the continued need for some descrip-
probably explains why we found far fewer description
tion studies. There will always be truly innovative
studies than Schmidt found in his analysis of the PBL
ideas and interventions that merit description and
literature.18 We also cannot comment on the preva-
dissemination to the education community, laying
lence of study purposes among non-experimental
the groundwork for future innovation and research.
research as this was excluded from our sample. Future
Likewise, we anticipate a continued role for
reviews, including non-experimental designs and a
justification studies, particularly when looking at
sample accrued from more journals and ⁄ or a longer
higher-order outcomes (behavioural and patient- or
period of time, will help draw a more complete picture
practice-oriented outcomes) or when evaluating
of the situation. Inter-rater agreement was modest, but
novel interventions. However, it must be recognised
final ratings were the result of consensus. Several
that these studies have a limited impact on our
disagreements occurred in classifying single-group
understanding of the teaching and learning process
pre-test ⁄ post-test studies as description or justification,
and in turn have limited influence on future practice
and this point is now clarified in our definition of
and research. We are not arguing for the disappear-
justification research (Table 1).
ance of description or justification studies, but for a
better balance21 among the 3 purposes so that a
In conclusion, we offer a framework for the classifi-
much larger proportion of research addresses clari-
cation of the purposes of education research and
fication questions. Furthermore, we suggest that
demonstrate that clarification studies are uncommon
description and justification studies should build on
in experimental studies in medical education.
previous clarification studies and sound conceptual
Although the concepts are not new – indeed, they are
frameworks.
as old as the scientific method – we hope that this
framework stimulates education scholars to reflect on
Evaluations of assessment methods were classified as
the purpose of their research and the questions they
clarification more often than other educational
ask, identify and pursue a sustained line of inquiry
topics, although the sample was small. The reason for
(programmatic research), and ask more clarification
this is unclear, but probably reflects a tendency
questions. Such research will deepen our under-
toward explicit conceptual frameworks for
standing and advance the art and science of medical
experimental research of assessment methods.
education.
Although this study focused on experimental
research, not all clarification research needs to be
Contributors: DAC and GB planned and executed the study
experimental. Clarification research can be carried described here. HGS provided the conceptual framework.
out using non-experimental designs (e.g. correla- All authors were involved in the drafting and revising of the
tional studies, comparisons among naturally manuscript.
occurring groups, and qualitative research). Acknowledgements: the authors thank T J Beckman, Mayo
Clinic College of Medicine, for his assistance in the
This classification of purposes of studies in medical preliminary review process.
education will be most useful inasmuch as it stimu- Funding: none.

132 ª Blackwell Publishing Ltd 2008. MEDICAL EDUCATION 2008; 42: 128–133
 Description, justification and clarification

Conflicts of interest: GB chairs the international editorial trainee assessment and satisfaction. Med Educ
board of Medical Education. We are not aware of other 2001;35:331–6.
conflicts of interest. 12 Shea JA. Mind the gap: some reasons why medical
Ethical approval: none. education research is different from health services
research. Med Educ 2001;35:319–20.
13 Chen FM, Bauchner H, Burstin H. A call for outcomes
REFERENCES research in medical education. Acad Med 2004;79:955–
60.
14 Wolf FM. Methodological quality, evidence, and
1 Harden RM, Grant J, Buckley G, Hart IR. BEME Guide
research in medical education (RIME). Acad Med
No. 1: best evidence medical education. Med Teach
2004;79 (10 Suppl):68–9.
1999;21:553–62.
15 Hutchinson L. Evaluating and researching the effec-
2 Dauphinee WD, Wood-Dauphinee S. The need for
tiveness of educational interventions. BMJ
evidence in medical education: the development of
1999;318:1267–9.
best evidence medical education as an opportunity to
16 Carney PA, Nierenberg DW, Pipas CF, Brooks WB,
inform, guide, and sustain medical education research.
Stukel TA, Keller AM. Educational epidemiology:
Acad Med 2004;79:925–30.
applying population-based design and analytic
3 Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon
approaches to study medical education. JAMA
D, Scalese RJ. Features and uses of high-fidelity medical
2004;292:1044–50.
simulations that lead to effective learning: a BEME
17 Cook DA, Beckman TJ, Bordage G. Quality of reporting
systematic review. Med Teach 2005;27:10–28.
of experimental studies in medical education: a
4 Cook DA. The research we still are not doing: an
systematic review. Med Educ 2007;41:737–45.
agenda for the study of computer-based learning. Acad
18 Schmidt HG. Influence of research on practices in
Med 2005;80:541–8.
medical education: the case of problem-based learning.
5 Norman GR. Research in medical education: three
Paper presented at the Annual Meeting of the Associ-
decades of progress. BMJ 2002;324:1560–2.
ation for Medical Education in Europe, Amsterdam,
6 Norman G. RCT = results confounded and trivial: the
The Netherlands, September 2005.
perils of grand educational experiments. Med Educ
19 Shrout PE, Fleiss JL. Intraclass correlations: uses in
2003;37:582–4.
assessing rater reliability. Psychol Bull 1979;86:420–8.
7 Bligh J, Parsell G. Research in medical education:
20 Landis JR, Koch GG. The measurement of observer
finding its place. Med Educ 1999;33:162–3.
agreement for categorical data. Biometrics 1977;33:159–
8 Prideaux D, Bligh J. Research in medical education:
74.
asking the right questions. Med Educ 2002;36:1114–5.
21 Albert M, Hodges B, Regehr G. Research in medical
9 Dolmans D. The effectiveness of PBL: the debate con-
education: balancing service and science. Adv Health Sci
tinues. Some concerns about the BEME movement.
Educ Theory Pract 2007;12:103–15.
Med Educ 2003;37:1129–30.
10 Wolf FM, Shea JA, Albanese MA. Toward setting a
research agenda for systematic reviews of evidence of Received 13 February 2007; editorial comments to authors 2 July
the effects of medical education. Teach Learn Med 2007; accepted for publication 15 October 2007
2001;13:54–60.
11 Prystowsky JB, Bordage G. An outcomes research per-
spective on medical education: the predominance of

ª Blackwell Publishing Ltd 2008. MEDICAL EDUCATION 2008; 42: 128–133 133