Professional Documents
Culture Documents
Learning Curves in Health Professions Education.14
Learning Curves in Health Professions Education.14
Learning Curves in Health Professions Education.14
Abstract
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
Learning curves, which graphically A typical learning curve is made up of a and competency-based approaches to
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 03/31/2024
show the relationship between learning measure of learning (y-axis), a measure instruction. Additionally, instructors can
effort and achievement, are common of effort (x-axis), and a mathematical use learning curve information to more
in published education research but are linking function. At the individual level, accurately target educational resources
not often used in day-to-day educational learning curves make manifest a single to those who most require them.
activities. The purpose of this article is person’s progress towards competence
to describe the generation and analysis including his/her rate of learning, the The learning curve approach requires
of learning curves and their applicability inflection point where learning becomes a fine-grained collection of data that
to health professions education. The more effortful, and the remaining will not be possible in all educational
authors argue that the time is right for distance to mastery attainment. At the settings; however, the increased use of
a closer look at using learning curves— group level, overlaid learning curves an assessment paradigm that explicitly
given their desirable properties—to show the full variation of a group of includes effort and its link to individual
inform both self-directed instruction by learners’ paths through a given learning achievement could result in increased
individuals and education management domain. Specifically, they make overt learner engagement and more effective
by instructors. the difference between time-based instructional design.
L earning curves graphically show the The concept of the learning curve holds more effectively. We first describe the
relationship between learning effort potential value far beyond a narrow anatomy of a learning curve and its
(e.g., repetitions or time spent; see also psychological application. For a number useful properties for educators. We then
“Validity of the x-axis: Measurement of of reasons, the time is right for educators discuss validity, which is an important
effort, experience, or practice” below) to take a fresh look at the use of learning consideration for all assessments but is
and the resultant learning outcomes.1–3 curves in the health professions. First, especially complicated for learning curves
The concept has become so ubiquitous and perhaps most compelling, the as they entail three separate components,
that educators often use the term when recent emphasis by the Accreditation each of which requires separate
describing learning processes (e.g., “a Council for Graduate Medical Education consideration. We subsequently describe
steep learning curve”) without necessarily (ACGME) on pairing competency how to graph learning curves, followed
referring to a specific figure. Underlying assessments with developmental by an outline of an approach to their
the learning curve and its text descriptors milestones naturally favors individualized quantitative and qualitative evaluation
are fundamental psychological truths that competency-based metrics,5 potentially or analysis. We discuss possible health
have important educational implications: including learning curves. Second, professions applications for learning
Practice improves performance, and learning curves aptly represent the curves, highlighting their usefulness
good practice improves performance learning acquired through deliberate in instruction and education research.
even more; sufficient practice leads to practice, which is increasingly recognized Finally, we end with a brief review of
high levels of achievement; and the most as an effective instructional strategy.6 potential pitfalls.
dramatic learning occurs early in the Third, the longitudinal collection of
learning process.2,4 increasingly granular, learner-specific
information in computer databases Anatomy of a Learning Curve
facilitates tracking learner growth We show a generic learning curve in
Please see the end of this article for information
about the authors.
over time, a process that is naturally Figure 1. In general, the relationship
represented as a learning curve.7 Finally, between learning (or performance) and
Correspondence should be addressed to Martin V. multilevel data modeling, an analysis effort (or, in Figure 1, deliberate practice)
Pusic, Division of Education Quality and Analytics,
Institute for Innovations in Medical Education, NYU technique that enables quantitative is not linear. Classically, the learning
School of Medicine, 545 First Ave., Suite 6P, New analysis of learning curves, has graduated curve can take the form of an ogive or
York, NY 10016; telephone: (212) 263-2053; e-mail: from a specialized research technique to S-shape indicating that the learning rate
martin.pusic@nyumc.org.
one accessible to educators.8 (or slope) varies as the location of the
Acad Med. 2015;90:1034–1042. subject (hereafter “learner”) changes
First published online March 20, 2015
The purpose of this article is to enable during the learning process.1,3,9 As an
doi: 10.1097/ACM.0000000000000681
health professions educators to enhance example, consider learning to interpret a
Supplemental digital content for this article is
available at http://links.lww.com/ACADMED/A266 their teaching, assessment, and research pediatric ankle radiograph wherein the
and http://links.lww.com/ACADMED/A267. activities by using learning curves task is to look for a possible fracture.1
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Article
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Article
Academic year
80 77/78 78/79 79/80 80/81 81/82 82/83 83/84 84/85 85/86 86/87 87/88
70 R R R R
R R
R 5 6
R
6 6
Percent correct 60 6 6
5 5 6 6 6
4 6
4 5
5
50 4 4
5 5
5
5
5
3 4
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
3 4
40 3
4
4
4 4 4
3
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 03/31/2024
3 3 3
30 2 3
3 3 3
2 2
2
2
20 2 2
2 2
2 2
1 1 1 1
10 1 1
1
1 1 1
1
0
Figure 2 Maastricht progress tests. Average correct scores for each class or cohort per year (dashed lines) on successive progress tests from 1977 to 1988.
Solid line (R) represents average scores of national reference groups of recently graduated doctors. Adapted with permission from Van der Vleuten CPM,
Verwijnen GM, Wijnen WHFW. Fifteen years of experience with progress testing in a problem-based learning curriculum. Med Teach. 1996;18:103–110.
being modeled. For example, in Figure 2, of learning effort. We have chosen this some index of variability. Just as a
the amount of time in medical school admittedly general term in preference simple group average score on a test
is used as the measure for the Dutch to experience, practice, time, or repetition is incomplete without a measure of
medical students. Using time alone, because these other more specific terms variance (e.g., the standard deviation),
especially in a clinical setting where not can be context-bound and do not a learning curve intended to reflect the
all time “units” are equal, can result in necessarily include the notion of active performance of a group would ideally
problems or invalid results. In learning engagement (i.e., one might repeat an include both an average curve and some
electrocardiogram (ECG) interpretation, activity endlessly, but unless an effort representation of variance. Figure 3
a given resident will likely learn more is made to learn, this will not lead to shows the previously published results
about ECG interpretation during some learning or, therefore, a learning curve). of 18 postgraduate trainees learning to
rotations (e.g., cardiology) than in others interpret ankle radiographs by serially
(e.g., dermatology), and if months- Measurement of association interpreting 234 cases and receiving
in-training is used as the measure, the Besides valid measures of learning and immediate feedback.1 Specifically, Panel
learning curve will show confounded effort, a third element is required to A shows the 18 individual learning
changes in performance. Therefore, create a meaningful learning curve: curves, Panel B provides the average
whenever possible, elapsed time should the linking function. This is defined as learning curve across all 18 learners, and
be replaced with measures that more the mathematical equation that links Panel C illustrates the mean curve for the
accurately reflect the construct—namely, the x-variable (effort) to the y-variable group, along with the 95% confidence
time actually spent in the learning activity (learning or performance). The validity intervals (CIs).1 Note, however, that the
being modeled (content evidence). For of the linking function depends on the CIs speak only to the range of group-level
ECGs, this measure might be counts of fit of the psychometric relationship curves (i.e., if we repeated the study, 95%
the number of ECGs viewed. between the learning and effort and the of the group-level average curves would
extent to which potential confounders be expected to fall within this range) and
Learning curves are frequently used can be taken into account.3 Linking not the variability in individual curves
to represent the process of deliberate functions will be described in detail that comprise those group averages.
practice, wherein the practice or below in the section entitled Analysis of Finally, Figure 3, Panel D presents all
repetition of a skill or maneuver always Learning Curves. 18 learning curves, superimposed. This
includes feedback; that is, each exposure representation shows that, although the
comes with comments or reactions from overall curve can be seen to nicely follow
an advanced instructor or guide.2 This Graphing Learning Curves a learning curve pattern, the paths of
type of practice is effortful, incorporates Having considered the validity of the data individual learners vary considerably.
expert-level feedback, and is available necessary to generate a learning curve, let Note that an overall curve bounded
over an extended time period.2 For us now turn to the task of generating a by a 95% CI would not completely
deliberate practice, a valid measure will visual representation of learning curves. represent this variability because the
require both a means to accurately define Key considerations include how to CI speaks only to the range of possible
and capture the number of repetitions represent group means and individual average group-level curves and not the
and a way to ensure that feedback is variation, and how to represent multiple variability of the individuals within the
available to the learner. dimensions of a learning task. group. Thus, to represent the learning
curve variance, we recommend plotting
Throughout the rest of this article, we First, group learning curves should overlaid individual learning curves, as in
will refer to the x-axis as the measure represent both a group mean and Figure 3, Panel D.
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Article
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 03/31/2024
Figure 3 Learning curves for 18 residents interpreting pediatric ankle radiographs, with performance (cumulative accuracy) plotted against a
measure of effort (the number of cases completed up to 234). Panel A shows the individual curves with the horizontal boundary line representing
a competency threshold. Panels B and C show the group average curve, respectively, without and with the 95% confidence intervals (CIs). Panel D
shows the individual curves overlaid. Variability between individuals is best conveyed with the overlaid curves and not a 95% CI which speaks only to
the statistical uncertainty of the group average estimate. Panel A is adapted from Pusic M, Pecaric M, Boutis K. How much practice is enough? Using
learning curves to assess the deliberate practice of radiograph interpretation. Acad Med. 2011;86:731–736.
For situations in which valid learning x-axis and learner cohort are the same in their improvements in accuracy,
curves are available for several for all three stacked curves, but each whereas confidence varies considerably
dimensions of a learning task (e.g., y-axis represents a different dimension over time.
different domains of knowledge or of the learning task.17 This display
skill, such as accuracy, speed, and allows a comparison of the respective
confidence), stacking the learning developmental stages in the learning Analysis of Learning Curves
curves can yield unique insights.16 curve. For example, Figure 4 shows Learning curves can be computed for one
Figure 4 shows such a multidimensional that speed continues to improve even individual or aggregated across many.
learning curve. In this example, the after students in the group have slowed In general, a descriptive analysis is used
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Article
Group-level analyses
Averaging results across a group of learners
results in a group-level learning curve
(Figure 3, Panel B); however, the group
average learning curve relationship holds
for relatively few of the individual learners.
The utility of the group learning curve
lies in the information that it provides to
the educator designing the intervention
or assessment. Figure 3, Panel D shows
the learning curves of all 18 radiology
learners superimposed, along with the
group-level learning curve. The group-level
learning curve shows the type of negative
exponential relationship (progressively
decreasing slope) that we would expect on
the basis of theories of deliberate practice.10
If modeled using nonlinear regression
techniques, we can derive estimates of
parameters that would be useful to an
educator who has been charged with
optimizing the learning of a group of
Figure 4 A multidimensional learning curve showing group averages for resident physicians practicing learners. On average, the learners start with
radiograph interpretation over 234 trials. Note that the participants plateau, in terms of accuracy, an accuracy of 50%. From the slope, we see
after about 150 trials but that their time-per-case continues to decrease through the last case. Their
that they learn maximally over the first 100
confidence in their ratings seems to start relatively high, decrease initially, and then rebound.
repetitions, after which learning becomes
less rapid. The asymptote for this learning
at the individual level, although some allowing group-level data to inform intervention (an online case bank) appears
quantitative analyses are possible. At the estimates for individual learners. to be at a cumulative accuracy of 72%.1
group level, statistical modeling using a
linking mathematical function is possible, Individual-level analyses Linking equations
assuming the group comprises a sufficient The hallmark of individual-level learning For the majority of situations in which
number of learners. Finally, multilevel curves is their marked variability. a health professions educator would
modeling techniques hold promise in Consider again the learning curves use a learning curve, the raw learning
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Article
resulting in remediation.
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 03/31/2024
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Article
Figure 7 Traditional time-based learning compared with competency-based learning. Two residents are learning radiograph interpretation by
practicing cases with feedback. Each resident improves with deliberate practice, but they start at different initial levels and progress at different rates,
even though they are both at the same training level. In a time-based curriculum (left panel), time/effort is fixed while terminal competency is allowed
to vary, and so, if each is assigned 150 cases, they finish with considerably different levels of competence. In a competency-based curriculum (right
panel), each resident spends the amount of time required to achieve a given level of competence so that while the time/effort necessary to achieve
competence varies considerably, the terminal competency standard is consistent.
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Article
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.
Article
Acknowledgments: The authors wish to thank the medical education with deliberate practice of learning to assessment for learning. Med
members of the Research on Medical Education yield better results than traditional clinical Teach. 2011;33:478–485.
Outcomes unit of the Program for Medical education? A meta-analytic comparative review 23 Regehr G. It’s NOT rocket science:
Education Innovation and Research at New York of the evidence. Acad Med. 2011;86:706–711. Rethinking our metaphors for research in
University Langone School of Medicine for their 7 Cook DA, Andriole DA, Durning SJ, Roberts health professions education. Med Educ.
helpful reviews of this manuscript. NK, Triola MM. Longitudinal research 2010;44:31–39.
databases in medical education: Facilitating 24 Dreyfus SE, Dreyfus HK. A Five-Stage Model
Funding/Support: The radiology studies1,17 the study of educational outcomes over of the Mental Activities Involved in Directed
described in this paper were carried out with time and across institutions. Acad Med. Skill Acquisition. Berkeley, Calif: University
the support of Medical Education Research 2010;85:1340–1346. of California, Berkeley; 1980.
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
Grants from the Royal College of Physicians and 8 Liang K-Y, Zeger SL. Longitudinal data 25 Ericsson K. The acquisition of expert
Surgeons of Canada. analysis using generalized linear models. performance as problem solving. In: Davidson
Biometrika. 1986;73:13–22. J, Sternberg R, eds. The Psychology of
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 03/31/2024
Other disclosures: None reported. 9 Son LK, Sethi R. Adaptive learning and Problem Solving. 1st ed. Cambridge, England:
the allocation of time. Adapt Behav. Cambridge University Press; 2003:44–83.
Ethical approval: The authors of all the original 2010;18:132–140. 26 Green ML, Aagaard EM, Caverzagie KJ,
studies1,17 described obtained ethical approval to 10 Singer JD, Willett JB, eds. Longitudinal et al. Charting the road to competence:
conduct their work. Data Analysis: Modeling Change and Event Developmental milestones for internal
Occurrence. 1st ed. New York, NY: Oxford medicine residency training. J Grad Med
Previous presentations: Some of this content University Press; 2003. Educ. 2009;1:5–20.
was previously presented at the Association of 11 Kane MT. Current concerns in validity 27 Hodges B. The shifting discourses of
American Medical Colleges Research in Medical theory. J Educ Meas. 2001;38:319–342. competence. In: Hodges BD, Lingard L, eds.
12 Cook DA, Beckman TJ. Current concepts The Question of Competence: Reconsidering
Education Annual Meeting, November 2009,
in validity and reliability for psychometric Medical Education in the Twenty-First
Boston, Massachusetts, and at the Pediatric
instruments: Theory and application. Am J Century. London, UK: Cornell University
Academic Societies Annual Meeting, May 2009,
Med. 2006;119:166.e7–166.e16. Press; 2012:1–32.
Baltimore, Maryland. 13 Downing SM. Validity: On meaningful 28 Brydges R, Butler D. A reflective analysis of
interpretation of assessment data. Med Educ. medical education research on self-regulation
M.V. Pusic is assistant professor, Emergency 2003;37:830–837. in learning and practice. Med Educ.
Medicine, and director, Division of Education Quality 14 Van der Vleuten CPM, Verwijnen GM, 2012;46:71–79.
and Analytics, New York University Langone School Wijnen WHFW. Fifteen years of experience 29 Quirk M. Metacognitive capabilities. In:
of Medicine, New York, New York. with progress testing in a problem- Quirk ME, ed. Intuition and Metacognition
based learning curriculum. Med Teach. in Medical Education. 1st ed. New York, NY:
K. Boutis is associate professor and pediatric
emergency physician, Department of Pediatrics,
1996;18:103–110. Springer; 2006:23–35.
Hospital for Sick Children, University of Toronto, 15 American Educational Research Association. 30 Kornell N, Metcalfe J. Study efficacy and the
Toronto, Ontario, Canada. Standards for Educational and Psychological region of proximal learning framework. J Exp
Testing. Washington, DC: American Psychol Learn Mem Cogn. 2006;32:609–622.
R. Hatala is associate professor, Department of Educational Research Association; 1999. 31 Guadagnoli M, Morin MP, Dubrowski
Medicine, University of British Columbia, Vancouver, 16 Dinçler S, Koller MT, Steurer J, Bachmann LM, A. The application of the challenge point
British Columbia, Canada. Christen D, Buchmann P. Multidimensional framework in medical education. Med Educ.
D.A. Cook is professor, Medicine and Medical analysis of learning curves in laparoscopic 2012;46:447–453.
Education; director, Online Learning Development sigmoid resection: Eight-year results. Dis 32 Pusic MV, Kessler D, Szyld D, Kalet A,
and Analysis, Center for Online Learning, Mayo Colon Rectum. 2003;46:1371–1378. Pecaric M, Boutis K. Experience curves as an
Clinic College of Medicine; and consultant, 17 Boutis K, Pecaric M, Seeto B, Pusic M. Using organizing framework for deliberate practice
Division of General Internal Medicine, Mayo Clinic, signal detection theory to model changes in emergency medicine learning. Acad Emerg
Rochester, Minnesota. in serial learning of radiological image Med. 2012;19:1476–1480.
interpretation. Adv Health Sci Educ Theory 33 Motulsky H, Christopoulos A. Approach
Pract. 2010;15:647–658. to comparing models. In: Motulsky H,
References 18 Altman DG. Practical Statistics for Medical Christopoulos A, eds. Fitting Models to
1 Pusic M, Pecaric M, Boutis K. How much Research. London, UK: Chapman & Hall/ Biological Data Using Linear and Nonlinear
practice is enough? Using learning curves to CRC; 1991. Regression. 1st ed. Oxford, UK: Oxford
assess the deliberate practice of radiograph 19 Bienkowski M, Feng M, Means B. Enhancing University Press; 2004:134–151.
interpretation. Acad Med. 2011;86:731–736. teaching and learning through educational 34 Song HS, Kalet AL, Plass JL. Assessing
2 Ericsson KA. Deliberate practice and data mining and learning analytics: An issue medical students’ self-regulation as aptitude
acquisition of expert performance: A general brief. Washington, DC: U.S. Department of in computer-based learning. Adv Health Sci
overview. Acad Emerg Med. 2008;15:988–994. Education, Office of Technology; October Educ Theory Pract. 2011;16:97–107.
3 Ramsay CR, Grant AM, Wallace SA, 2012. http://tech.ed.gov/wp-content/ 35 Ellaway RH, Pusic M, Yavner S, Kalet AL.
Garthwaite PH, Monk AF, Russell IT. uploads/2014/03/edm-la-brief.pdf. Accessed Context matters: Emergent variability in an
Statistical assessment of the learning curves January 2, 2015. effectiveness trial of online teaching modules.
of health technologies. Health Technol 20 Grigg OA, Farewell VT, Spiegelhalter DJ. Use Med Educ. 2014;48:386–396.
Assess. 2001;5:1–79. of risk-adjusted CUSUM and RSPRT charts 36 Ellaway RH, Pusic MV, Cameron T, Galbraith
4 Ritter FE, Schooler LJ. The learning curve. for monitoring in medical contexts. Stat R, Farrell J. Developing the role of big data
In: Smelser NJ, Baltes PB, eds. International Methods Med Res. 2003;12:147–170. and analytics in medical education. Med
Encyclopedia of the Social and Behavioural 21 Holzhey DM, Jacobs S, Walther T, Mochalski Teach. 2014;36:216–222.
Sciences. 1st ed. Oxford, UK: Elsevier Science M, Mohr FW, Falk V. Cumulative sum failure 37 Jonassen D. Objectivism versus constructivism:
Ltd.; 2002:8602–8605. analysis for eight surgeons performing Do we need a new philosophical paradigm?
5 Nasca TJ, Philibert I, Brigham T, Flynn TC. The minimally invasive direct coronary Educ Technol Res Dev. 1991;39:5–14.
next GME accreditation system—Rationale and artery bypass. J Thorac Cardiovasc Surg. 38 Dweck CS, Chiu C, Hong Y. Implicit theories
benefits. N Engl J Med. 2012;366:1051–1056. 2007;134:663–669. and their role in judgments and reactions:
6 McGaghie WC, Issenberg SB, Cohen ER, 22 Schuwirth LW, Van der Vleuten CP. A word from two perspectives. Psychol Inq.
Barsuk JH, Wayne DB. Does simulation-based Programmatic assessment: From assessment 1995;6:267–285.
Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.