Professional Documents
Culture Documents
ebook download Practical Guide to the Evaluation of Clinical Competence 2nd Edition Eric S. Holmboe - eBook PDF all chapter
ebook download Practical Guide to the Evaluation of Clinical Competence 2nd Edition Eric S. Holmboe - eBook PDF all chapter
http://ebooksecure.com/product/ebook-pdf-practical-program-
evaluation-theory-driven-evaluation-and-the-integrated-
evaluation-perspective-2nd-edition/
https://ebooksecure.com/download/therapists-guide-to-clinical-
intervention-the-1-2-3s-of-treatment-planning-ebook-pdf/
https://ebooksecure.com/download/allergic-and-immunologic-
diseases-a-practical-guide-to-the-evaluation-diagnosis-and-
management-of-allergic-and-immunologic-ebook-pdf/
http://ebooksecure.com/product/ebook-pdf-handbook-of-practical-
program-evaluation-4th-edition/
(eBook PDF) Astronomy A Beginner's Guide to the
Universe 8th by Eric Chaisson
http://ebooksecure.com/product/ebook-pdf-astronomy-a-beginners-
guide-to-the-universe-8th-by-eric-chaisson/
http://ebooksecure.com/product/ebook-pdf-the-purposeful-argument-
a-practical-guide-2nd-edition/
https://ebooksecure.com/download/ballwegs-physician-assistant-a-
guide-to-clinical-practice-ebook-pdf/
http://ebooksecure.com/product/ebook-pdf-hydroponics-a-practical-
guide-for-the-soilless-grower-2nd/
http://ebooksecure.com/product/ebook-pdf-the-abcs-of-cbm-second-
edition-a-practical-guide-to-curriculum-based-measurement/
Any screen.
Any time.
Anywhere.
Activate the eBook version
e.
of this title at no additional charge.
Expert Consult eBooks give you the power to browse and find content,
view enhanced images, share notes and highlights—both online and offline.
4 Click “Redeem”
5 Log in or Sign up
6 Go to “My Library”
Place Peel Off
It’s that easy! Sticker Here
i
This page intentionally left blank
Practical Guide to the Evaluation
of Clinical Competence
2nd Edition
Copyright © 2018 Eric Holmboe, Richard Hawkins and Steven Durning, Published by Elsevier Inc. All rights
reserved.
For chapter 2 (Dr. Brian Clauser): Copyright © 2018, NBME. Published by Elsevier Inc. All Rights Reserved.
No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical,
including photocopying, recording, or any information storage and retrieval system, without permission in writing
from the publisher. Details on how to seek permission, further information about the Publisher’s permissions poli-
cies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing
Agency, can be found at our website: www.elsevier.com/permissions.
This book and the individual contributions contained in it are protected under copyright by the Publisher (other
than as may be noted herein).
Notices
Knowledge and best practice in this field are constantly changing. As new research and experience broaden
our understanding, changes in research methods, professional practices, or medical treatment may become
necessary.
Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using
any information, methods, compounds, or experiments described herein. In using such information or methods
they should be mindful of their own safety and the safety of others, including parties for whom they have a profes-
sional responsibility.
With respect to any drug or pharmaceutical products identified, readers are advised to check the most current
information provided (i) on procedures featured or (ii) by the manufacturer of each product to be administered,
to verify the recommended dose or formula, the method and duration of administration, and contraindications.
It is the responsibility of practitioners, relying on their own experience and knowledge of their patients, to make
diagnoses, to determine dosages and the best treatment for each individual patient, and to take all appropriate
safety precautions.
To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liabil-
ity for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise,
or from any use or operation of any methods, products, instructions, or ideas contained in the material herein.
Names: Holmboe, Eric S., editor. | Durning, Steven J., editor. | Hawkins,
Richard E., editor.
Title: Practical guide to the evaluation of clinical competence / [edited by]
Eric S. Holmboe, Steven J. Durning, Richard E. Hawkins.
Description: 2nd edition. | Philadelphia, PA : Elsevier, [2018] | Includes
bibliographical references and index.
Identifiers: LCCN 2016048388 | ISBN 9780323447348 (pbk. : alk. paper)
Subjects: | MESH: Clinical Competence | Educational Measurement--methods |
Education, Medical, Graduate--standards | Competency-Based
Education--methods
Classification: LCC R837.A2 | NLM W 18 | DDC 616--dc23 LC record available at https://lccn.loc.gov/2016048388
Assessment of health professionals across the continuum stay true to that philosophy by adding more supplemental
of medical education and practice is essential for advanc- material and new chapters on assessing clinical reasoning in
ing high-quality and safe care for patients and the public. the workplace, work-based procedural assessment, and feed-
Assessment of clinical competence is a core element of back. All other chapters have undergone extensive revision
professionalism and underlies effective professional self- to be up-to-date and practical.
regulation; it is essential for fulfilling our professional obli- The three of us have spent much of our professional lives
gation to assure the public that the graduates of medical thinking, learning, and then teaching about assessment.
education training programs are truly prepared to enter the Like many of you, much of our initial learning was through
next stage of education and/or practice. Despite substan- trial and error, occurring as a result of being assigned posi-
tial attention to the quality and safety of healthcare over tions of responsibility in determining the competence of
the past 20 years, major deficiencies and concerns persist students and residents in internal medicine. We have also
in healthcare fields. The transformation of medical educa- had the privilege to work within national organizations
tion, and the education of all healthcare professionals, is involved in the assessment of physicians across the contin-
appropriately seen as part of the solution. Effective assess- uum. Assessment is not routinely seen by physicians and
ment is a vital component of this transformation. First and other health professionals as a welcome activity, especially
foremost, medicine is a service profession. As medical edu- when it comes from an external entity. Yet without assess-
cators, it is vital we develop and use high-quality assessment ment feedback is almost impossible and continuous profes-
methods and systems in order to fulfill a primary obligation sional growth is difficult. We hope by sharing part of our
to the public and patients we serve. Furthermore, effective own journey through this textbook we can help the reader
assessment provides the necessary data for robust feedback address important assessment challenges they are facing in
and guidance to support professional growth and develop- their own work context and also contribute to larger con-
ment. Learners are entitled to no less; without assessment versations around assessment as a mechanism to improve
and feedback the attainment of mastery, the ultimate goal healthcare quality and safety.
of outcomes-based education, is nearly impossible. The primary purpose of this book is to provide a practi-
It has been nearly 10 years since the publication of the cal guide to developing assessment programs using a sys-
first edition of this book, and much has changed during this tems lens. No single assessment method is sufficient to
period. Competency-based medical educational (CBME) determine something as complex as clinical competence.
models are now being implemented to varying degrees across Educators will need to develop programs of assessment by
the globe in an effort to drive better outcomes of education choosing the optimal combination of methods, based on
and by extension healthcare. The philosophical underpin- the best evidence available, for their local context. This book
nings of CBME are informing curricular and program- has been organized around the various assessment methods
matic assessment changes, accreditation and certification and instruments and how individuals with responsibilities
approaches, and the credentialing of healthcare profes- for assessment can apply these methods and instruments
sionals. CBME has highlighted the importance of leverag- in their own setting. We have provided an overview of key
ing more traditional methods of assessment while creating educational theories where applicable to help the reader
substantial pressure and defining the need to advance other understand how best to use the assessment method and its
methods of assessment, especially in the workplace. Fully purpose. Each chapter provides information on the strengths
implemented, CBME frameworks embrace holistic and and weaknesses of the assessment method, along with infor-
constructivist approaches to assessment; successful assess- mation about specific tools. Many chapters provide examples
ment programs will need to incorporate a diverse range of of assessment instruments along with suggestions on faculty
educational and assessment theories and methods. development and effective implementation of the assess-
We are pleased to be able to share changes and advances ment method. Each chapter also contains an annotated
in assessment that have occurred since 2008. Many readers bibliography of helpful articles for additional reading.
let us know that one of the main benefits of the first edition The first chapter provides an overview of basic assess-
was the practical suggestions in each chapter that could be ment principles with a focus on the rise and impact
implemented in training programs. We have attempted to of competency-based approaches to achieve outcomes.
v
vi Preface
Chapter 2 provides a useful primer on key theories The final three chapters help the reader “put it all together.”
and aspects of psychometrics, a discipline that remains Portfolios, covered in Chapter 14, offer a comprehensive
essential to effective assessment. Chapter 3 explores the approach to supporting an assessment program. The chapter
evolving approaches to the use of rating scales, a com- provides practical advice on how to design and implement
mon component of assessment forms and surveys, high- portfolios. Chapter 15 provides a systematic approach to
lighting the importance of appropriate frameworks and working with the dyscompetent learner, i.e., the learner in
anchors. Direct observation in the workplace, especially difficulty. These learners require an assessment program and
of clinical skills, is the focus of Chapter 4 with multiple systematic approach using multiple assessment methods.
practical suggestions on how to better prepare faculty The final chapter, Chapter 16, covers the important role of
in this essential assessment skill. Chapter 5 explores the programmatic evaluation as part of an effective educational
assessment of clinical skills with standardized patients, program. Newer concepts and approaches to programmatic
another form of direct observation in controlled settings. assessment are provided.
Chapter 6 provides an extensive overview on the effective Effective assessment requires a multifaceted approach
use of the traditional written, standardized tests of medical using a combination of assessment methods. This is the
knowledge and clinical reasoning, still an essential part of rationale behind the organization and design of this book.
an assessment program. However, the need for high-quality Effective assessment also depends upon collaboration among
assessment of clinical reasoning in the workplace has grown a team of faculty and other educators; thus any change to an
in importance with the recognition of the persistent and assessment system must include not only buy-in from oth-
pernicious problem of diagnostic and therapeutic errors in ers, but also the investment to train educators to use assess-
clinical practice. This is the focus of Chapter 7, a new chap- ment methods and tools effectively. In a CBME system, this
ter for this edition. Another new addition, Chapter 8, covers must also include the learners as “active agents” in their own
the assessment of procedural competence in the workplace, learning and assessment. Interprofessional faculty, program
another growing area of interest for medical educators in an leaders, and learners need to work together to co-create and
era of patient safety concerns. co-produce assessment to maximize educational, and ulti-
Chapter 9 addresses the importance of assessing evi- mately, clinical outcomes.
dence-based practice, an essential competency in a time of It is essential to remember the true assessment instru-
rapidly expanding medical knowledge and growing use of ment is the individual using it, not the instrument itself.
clinical decision support at the point of care. Chapter 10 has Assessment tools are only as good as the individual using
been extensively revised and now focuses on the multiple them. If done well, assessment can have a profoundly posi-
ways to assess performance in clinical practice using quality tive effect on patients, learners, and faculty. That has not
and safety measures. The growing use of these measures is changed since 2008 and likely never will. Nothing can be
now an established part of medical practice across the globe. more satisfying than knowing each and every one of your
Chapter 11 provides guidance on the effective use of multi- graduates is truly ready to move to the next career level.
source feedback, an approach essential to patient-centered The public expects no less, and we should expect no less
care and interprofessional practice. from ourselves. In that spirit, we welcome comments from
Chapter 12 is a complement to Chapter 5, covering the you, the reader, on how we can improve upon this book.
growing field of simulation outside standardized patients. Eric S. Holmboe
Simulation, depending on the discipline, should increas- Steven J. Durning
ingly become a standard component of an assessment pro- Richard E. Hawkins
gram. Chapter 13 is a new chapter on practical approaches
to feedback. This chapter was added because no assessment
system can be fully effective without robust feedback.
Contributors
vii
viii Contributors
In memory of my incredibly supportive parents, Dr. Much love and gratitude to my mother, Jacqueline
Kenneth C. and Mrs. Bette M. Holmboe. Hawkins, and my partner, Margaret Jung, for their support
All my love and appreciation to my wife and best friend, and encouragement.
Eileen Holmboe, and my two amazing children who bring Richard E. Hawkins
so much joy, Ken and Lauren.
Eric S. Holmboe
Dedication
ix
This page intentionally left blank
Contents
6 Using Written Examinations to Assess Medical 15 The Learner With a Problem or the Problem
Knowledge and Its Application, 113 Learner? Working With Dyscompetent
David B. Swanson and Richard E. Hawkins Learners, 288
William Iobst and Eric S. Holmboe
7 Assessing Clinical Reasoning in the
Workplace, 140 16 Program Evaluation, 303
Eric S. Holmboe and Steven J. Durning Richard E. Hawkins and Steven J. Durning
xi
Video Contents
xii
1
Assessment Challenges in the Era of
Outcomes-Based Education
ERIC S. HOLMBOE, MD, MACP, FRCP, OLLE TEN CATE, PHD,
STEVEN J. DURNING, MD, PHD, AND RICHARD E. HAWKINS, MD, FACP
1
2 CHA P T ER 1 Assessment Challenges in the Era of Outcomes-Based Education
competencies as a foundational outcomes framework for The primary purpose of this second edition is to pro-
educational programs.7–11 vide practical guidance to educators and program leaders
In 1978, McGaghie and colleagues described a ratio- on the “front lines” for building and implementing better
nale for an approach to medical education founded on the programs and systems of assessment using the best evidence
acquisition of defined competencies. “The intended output and information available. Assessment is fundamental and
of a competency-based programme,” they wrote, “is a health essential for effective learning and for achieving both desired
professional who can practise medicine at a defined level of educational and clinical outcomes. CBME is part of the lat-
proficiency, in accord with local conditions, to meet local est phase on what should be a continuous commitment to
needs.”8 Educational leaders and policymakers worldwide improve educational programs and by extension the qual-
produced multiple reports lamenting that medical educa- ity and safety of care patients and populations receive. This
tion systems were not producing physicians with the abili- introductory chapter will present an overview of the drivers
ties needed to meet the complexities of modern practice, of change in the assessments used during clinical education,
leading to the realization that reforms in undergraduate, frameworks for such assessment, criteria for choosing assess-
graduate, and continuing medical education were urgently ment methods, elements of an effective faculty development
needed. In the United States, several recent reviews call effort, and the new concepts of competencies, milestones,
attention to the inadequate preparation of our graduates to and entrustable professional activities now being used to
practice effectively in our evolving health care systems.12–14 facilitate change and improvement in medical education.
This context and other factors ultimately led to the devel- Before moving on to fundamental issues of assessment in a
opment of competency frameworks in several countries as CBME world, we will first review some key definitions and
part of initiatives to implement competency-based medical elements of CBME.
education (CBME) to achieve better educational and clini-
cal care outcomes. The first iteration of the Canadian Medi-
cal Education Directions for Specialists (CanMEDS) Roles Outcomes and Competency-Based Medical
by the Royal College of Physicians and Surgeons of Can- Education
ada was produced in 1996.15,16 Recognizing similar needs
and issues, the Accreditation Council of Graduate Medical A focus on the educational process has now shifted to an
Education, the American Board of Medical Specialties, the emphasis on what a physician is able to actually do at the end
Institute of Medicine, the General Medical Council of the of training and at important junctures during the training
United Kingdom, the Royal Australasian College of Sur- process. Competencies have become a primary mechanism
geons, the Dutch College of Medical Specialties, and other for defining the educational outcomes. Outcomes-based
national professional entities produced competency frame- education starts with a specification of the competencies
works.17–21 Two key features of these competency projects expected of a physician, and these requirements drive the
stand out. One is a redefinition of the doctor to include content and structure of the curriculum, the selection and
many more important and relevant abilities and constructs deployment of teaching and learning methods, the site of
beyond medical knowledge and technical skill that had been training, and the nature of the teachers. Assessment plays a
dominating training in the previous decades. The other fea- central role in determining whether students and residents
ture is the intention to better monitor doctors in training have actually achieved the competencies that have been
and to ensure they meet predefined competency standards specified and whether the educational program has been
upon graduation to unsupervised practice.7,22 efficacious. CBME highlights the importance of integrating
Since the publication of the first edition of this book curriculum and assessment; they should not be independent
in 2008, a number of major reports and initiatives have activities but rather inform each other as part of an overall
sought to move CBME toward broader implementa- educational system and program of assessment. This change
tion. The International CBME Collaborators, a group of in thinking and the need to assess the diverse competencies
medical educators and leaders convened by the Royal Col- of the physician have been important factors in the develop-
lege of Physicians and Surgeons of Canada, produced a ment of new methods of assessment, especially work-based
series of articles on the history, concepts, and challenges assessments covered in detail throughout this book.
to implementation of competency-based medical educa- CBME is an outcomes-focused approach to and philoso-
tion, including needed changes to assessment, across the phy of designing the explicit developmental progression of
continuum of medical training.15,16,23–25 In the same year, health care professionals to meet the needs of those they
Frenk and a group of international leaders published an serve. Among its fundamental characteristics (Box 1.1) is
influential position paper in The Lancet on the need to a shift in emphasis away from time-based programs based
accelerate transformation in medical education, grounded solely on exposure to experiences such as clinical rotations
in the principles of CBME.6 Finally, on the 100th anni- in favor of an emphasis on needs-based graduate outcomes,
versary of the Flexner report (1910), the Carnegie Founda- authenticity, and learner-centeredness.11,26 As defined
tion released recommendations for medical education that by Frank and colleagues, CBME is “an outcomes-based
embraced many of the key principles and goals of CBME.9 approach to the design, implementation, assessment, and
All of these reports have highlighted the critical need for evaluation of medical education programs, using an orga-
better assessment. nizing framework of competencies.”11 Although outcomes
CHAPTER 1 Assessment Challenges in the Era of Outcomes-Based Education 3
• BOX 1.1 Fundamental Characteristics of an instructor. Clinical skill and judgment were tested using
Competency-Based Medical Education an oral examination that often required the student to go
to the bedside, gather patient information, and present it
Graduate outcomes in the form of achievement of predefined along with a diagnostic list and treatment plan to one or
desired competencies are the goals of competency-based
more examiners who asked questions. Because these were
medical education (CBME) initiatives. These are aligned with the
roles graduates will play in the next stage of their careers. the only generally accepted methods available, they were
These predefined competencies are derived from the needs of applied to most assessment problems even if they were not
patients, learners, and institutions and organized into a coherent completely suitable to the task. That may have been accept-
guiding framework. able in a time when supervisors had much more control over
Time is a resource for learning, not the basis of progression
the health care process and had natural checks of everything
of competence (i.e., time spent on a ward is not the marker of
achievement). learners reported. Over the past decades health care has
Teaching and learning experiences are sequenced to facilitate become too complex to warrant this type of “on-the-fly,” ad
an explicitly defined progression of ability in stages. hoc approach. For example, lengths of stay in hospitals have
Learning is tailored to the learner’s individual progression in dropped dramatically and faculty have multiple competing
some manner.
responsibilities.
Numerous direct observations and focused feedback
contribute to effective learner development of expertise. From that point to the present, there have been exten-
Assessment is planned, systematic, systemic, and integrative. sive changes in the way assessment is conducted. Meth-
ods have proliferated, as have the requirements for their
appropriate use. Much progress has been made in the
assessment of medical knowledge with a variety of written
are now the primary driver, that does not mean educational and computer-based techniques offering reliable and valid
structures and processes are not important. The famous results (see Chapter 6). In the last few decades, consid-
Donabedian equation for quality, Structure × Process = Out- erable gains have been made in defining and enhancing
comes, highlights that good outcomes depend on effective the psychometric qualities of objective structured clinical
structures and processes.27 However, we are also learning examinations (OSCEs), particularly related to their use
that the relationship between structure and process is quite in high-stakes examinations (see Chapter 5). However,
complex and nonlinear in its actual execution.28 Chapter 16 assessment in the context of learners caring for patients
provides helpful guidance on how to embrace complexity as in clinical units (i.e., wards, operating theater, ambulatory
part of program design and evaluation. Assessment is a criti- clinic) has lagged to some degree, especially in the areas of
cal part of the complex interaction between structure and clinical skills, interprofessional teamwork, and quality and
process in an educational program. safety of care.24,30
Assessment is an essential activity (i.e., process) that can Equally important, the methods that have been devel-
be used to demonstrate outcomes of interest. This is not a oped to support clinical education often rely on faculty
new insight—assessment has always been critically impor- who are inexperienced in their use, do not share common
tant in any educational endeavor. However, the problems standards or shared mental models of the competencies of
with assessment in medical education, and in general all of importance, and have not been trained to apply them in a
health professions education, have been long-standing and consistent fashion. In addition, faculty now experience sub-
persistent, such as lack of direct observation of learner per- stantial time pressures, more learner and patient handoffs,
formance and meaningful feedback, overreliance on testing higher degrees of comorbidity among hospitalized patients,
for assessment of medical knowledge, lack of attention to and increasing personal clinical responsibilities. Perhaps
other essential competencies that address our graduates’ more concerning are recent findings that one of the prin-
abilities to function effectively in our health care systems cipal drivers of faculty assessment relates to their own clini-
such as interprofessional teamwork and quality improve- cal skills, with a number of studies highlighting important
ment, and ineffective use of assessment methods and tools deficiencies in practicing physician clinical skills such as
by faculty, to name a just a few. In this introductory chapter, medical interviewing, physical examination, and communi-
we will first explore fundamental issues in assessment, fol- cation skills.31,32 Finally, many of the faculty are also being
lowed by recent attempts to more effectively operationalize asked to assess and judge competencies, such as care coor-
competencies through milestones and entrustable profes- dination, patient safety, and use of information technology,
sional activities, and then close with the importance of cre- areas in which they themselves were never formally trained.
ating a program of assessment. Throughout this chapter we Compounding this state of affairs has been the lack of effec-
will refer the reader to other chapters in the book to help the tive faculty development approaches and models to address
reader create and revise their own program of assessment. these new clinical and educational methods.33
logic can help educators develop assessment programs that and programs need to build in ongoing evaluation of their
meet public, patient, and learner needs.34 Many programs assessment activities. (See Chapter 16.)
globally are implementing curricular changes that embrace
competencies and outcomes, supported by improvements Technology
in technology, psychometrics, and evolving work-based
assessment approaches that increasingly incorporate more Over the past 50 years, the availability of increasingly
qualitative techniques and systematic judgment. sophisticated technology has changed the testing of medi-
cal knowledge and judgment in fundamental ways.43,44 The
Accountability and Quality Assurance introduction of the computer heralded an era of large-scale
testing by encouraging the use of multiple-choice ques-
The movement to competency-based medical education tions (MCQs), the answers to which could be scanned by
has been accompanied by significant efforts to enhance machine, turned into scores, and then reported in an effi-
the accountability of physicians.3 Motivated by the need cient and objective fashion.
to improve quality and safety, and in part by high-pro- More recently, the intelligence of the computer has
file cases such as those involving Michael Swango in the improved assessment in two ways:
United States and Howard Shipman in the United King- 1. On the one hand, it has enabled the application of signif-
dom in the 1990s, the public has continued to pressure icant psychometric advances to the assessment of medical
medicine to increase its level of oversight and eliminate the knowledge. Specifically, the computer’s intelligence has
“bad apples.”35,36 Medical educators are also more keenly improved efficiency by allowing the selection of ques-
aware that too many trainees graduate with substantial tions that are targeted to the ability of particular examin-
deficiencies in foundational knowledge and clinical skills ees. Sequential testing and adaptive testing permit gains
and more recently have become aware of deficiencies in in efficiency and precision.
competencies important to succeed in our health care sys- 2. On the other hand, it has improved the assessment of
tems.12–14,37 Effective quality assurance depends on robust higher cognitive abilities, including clinical reasoning, by
assessment programs and is critically important to ensure permitting the use of interactive item formats that more
that graduates of medical education programs are truly closely simulate the types of judgments physicians need
ready for promotion to the next stage and ultimately unsu- to make in practice. (See Chapter 6.)
pervised practice. Promoting trainees who lack compe- Although the impact of technology on assessment of
tence erodes, if not destroys, the trust between the medical clinical skills has been slower to develop, advances in simu-
profession and the public. lation and computer technology have led to the develop-
ment of approaches and tools that recreate aspects of the
Quality Improvement Movement clinical encounter with considerable fidelity. These methods
have a growing impact on assessment, especially in the area
At the same time, there has been a variety of efforts focused on of procedural skills, where mastery models are beginning to
continuously improving the quality of health care.4,27,38–41 gain traction.45–48
These efforts have relied on methods devised by workers in Finally, technology, especially through smartphone and
the field of quality management science and, in some cases, tablet applications, is beginning to change the way assess-
used successfully in industry for over 60 years to drive con- ment data is obtained and processed. For example, tools
tinuous improvement in health care and now increasingly designed for assessment through direct observation are
in medical education programs. Central to quality improve- increasingly being converted into smartphone applica-
ment is assessment—it is very hard to improve without tions.46,47 Learning management systems, increasingly used
meaningful measurement and data. It offers a means of by programs, are also beginning to incorporate mobile apps
identifying those whose overall performance is well below into their platforms.49 These portable applications hold sub-
standard and also identifying areas for improvement for stantial promise to reduce the data collection burden while
those who are generally performing adequately, helping to guiding the assessment activity of the faculty to attend to
drive the continuous quality improvement process. These critical competencies.
developments have helped to fuel the creation of several
new methods of assessment and to increase the use of other Psychometrics
methods already available. For example, the milestones
initiative, an attempt to better describe competencies in At the same time that the technology has improved, there
narrative, developmental terms in the United States, uses have been significant advances in psychometrics, the basic
the principles of continuous quality improvement as part science of assessment. Classical test theory, prominent from
of its foundation to improve graduate medical education. the turn of the 20th century, has gradually given way to
The milestones initiative can be viewed through the lens measurement models based on strong assumptions about
of “action- or practice-based research” to learn and develop test items and examinees. The family of item response the-
evidence over time.42 There is no single “holy grail” of ory models now makes it possible to produce equivalent
assessment. All assessments have strengths and weaknesses, scores even when examinees take tests made up of different
CHAPTER 1 Assessment Challenges in the Era of Outcomes-Based Education 5
questions.50 They also support the computer-based admin- Group process, commonly through entities called clini-
istration of examinations that are tailored to the ability level cal competency committees, has also become an impor-
of individual test-takers; this allows tests to be shortened by tant part of the assessment process and programs. Effective
as much as 40%.51 The ability to shorten tests has cost and group process can lead to better judgments around com-
validity implications; less test material exposure decreases petence.57–59 Finally, qualitative research techniques have
the likelihood that future examinees are familiar with exam- been shown to have value in judging aggregate assessment
ination content.52 Generalizability theory makes it possible information, such as that contained within a portfolio (see
to identify how much error is associated with different fac- Chapter 14). Again, a rigorous approach to application
ets of measurement (e.g., raters, patients).53 Based on this of qualitative research techniques and principles helps to
information, assessments can be prospectively designed to enhance the reliability and validity of judgments.60–62
make the best use of resources, such as faculty time, while
maintaining the reliability of the results. Framework for Assessment
In addition to these major developments, there have
been a number of other advances. For example, there are As methods of assessment have proliferated, so has the
a variety of systematic methods available for setting stan- need to use them efficiently and to combine them into a
dards on tests and for identifying when test questions are system of assessment. Developing, implementing, and
biased against particular groups of examinees.2,54,55 Test sustaining effective systems for the assessment of clinical
development methods have gotten better, as have the means competence in medical school, residency, and fellowship
for judging whether particular items are working properly. programs require consideration of what competencies need
Overall, these advances have improved both the quality and to be assessed, how to best assess them, and the level of the
efficiency of assessment. trainee being assessed. Consequently, a three-dimensional
framework for structuring an assessment system can help
Qualitative Assessment and Group Process medical educators make better judgments about learner
development. Along the first dimension are the competen-
Although advances in psychometrics have clearly helped to cies that need to be assessed, along the second is the level of
improve assessment in medical education and will remain assessment required, and along the third is the trainees’ stage
a core science for assessment, many have noted limita- of development.
tions of the traditional psychometric approach in today’s
complex clinical and educational environment.56 Often Dimension 1: Competencies
referred to as “qualitative” or “narrative” assessment, use
of the written word has grown in importance. For exam- As shown in Table 1.1, there are several schemes for describ-
ple, many of the new smartphone apps contain natural ing the knowledge, skills, and attributes of the physi-
language processing capability that allow for the capture cian.16–19 The CanMEDS model, which was developed and
of narrative assessment and feedback through dictation. recently updated by the Royal College of Physicians and
Milestones, discussed in more detail later, are more robust Surgeons in Canada, describes the competencies in terms
narrative descriptors of stages of development, bringing of the roles of a physician. Good Medical Practice, which
both quantitative and qualitative aspects of measurement was created by the General Medical Council in the United
more closely together.48 Kingdom, describes the elements of good practice. In the
TABLE
1.1 The Competencies of Physicians as Described by Four Organizations
ABMS, American Board of Medical Specialists; ACGME, Accreditation Council for Graduate Medical Education; CanMEDS, Canadian Medical Education Direc-
tions for Specialists; GMC, General Medical Council (UK); IOM, Institute of Medicine.
6 CHA P T ER 1 Assessment Challenges in the Era of Outcomes-Based Education
United States, two influential groups developed a set of Shows how. Although trainees may know and know
core competencies. The Accreditation Council for Gradu- how, they may not be able to integrate these skills into a
ate Medical Education (ACGME) and the American Board successful performance with patients. Consequently, certain
of Medical Specialties (ABMS) adopted six general compe- assessment methods require the trainee to show how they
tencies in 2001. These competencies consist of the educa- perform with patients. For example, a standardized patient
tional outcomes framework for residency and fellowship presenting with an ethical challenge would offer the trainee
training, as well as maintenance of certification programs an opportunity to “show how” he or she would respond to a
throughout a physician’s career in the United States. The professionalism challenge.
Institute of Medicine (IOM) has recommended five core Does. No matter how good traditional assessment meth-
skills, or competencies, that create a framework for evaluat- ods become, there remains the concern that what happens
ing performance and stimulating the reform of education. in a controlled testing environment does not generalize
They are intended to improve professional education and directly or predict what happens in practice. The highest
practice with a goal of enhancing the safety and quality of level of Miller’s pyramid therefore focuses on methods that
health care. Although there are some differences among the provide an assessment of routine performance. For example,
schemes, there is also significant overlap in these descrip- the development and use of a critical incident system, such
tions of a physician. as the one currently used in some medical schools, offers
These competencies are intended as the first step in iden- an assessment of what students actually do in terms of
tifying key educational outcomes that should inform the professionalism.
learning objectives, assessment, and curriculum of graduate Miller’s pyramid is a useful framework for considering
training programs, adapted to the content, education, and differences and similarities among assessment methods.
practice of the particular specialty/subspecialty. As we will However, the fact that it is a pyramid might imply to some
see later, milestones and entrustable professional activities that methods addressing the higher levels are better, or con-
(EPAs) are concepts, specified and adapted by specialties, versely that the larger area occupied by the base of the pyra-
that can facilitate the implementation of competency-based mid implies that knowledge assessment is most important.
programs. The data produced by the assessment of these Instead, superior methods are those best aligned with the
competencies serve as a basis for judging the quality of the purpose of the assessment. For example, if an assessment
trainees and their training, as well as supporting the con- of foundational medical knowledge is needed, a method
tinuous improvement of both. associated with that level (e.g., multiple-choice questions)
is likely better than a method associated with another level
Dimension 2: Levels of Assessment (e.g., standardized patients). Recently Cruess and colleagues
argued to add “Is” to the top of the pyramid to recognize the
The multifaceted nature of the competencies makes it appar- importance of professional formation, but it is not yet clear
ent that no single method could provide a sufficient basis for where this fits into an assessment program.64
making judgments about students or residents. In an orga-
nized approach to this problem, Miller proposed a classifi- The Cambridge Model
cation scheme that stratifies assessment methods based on As physicians near the end of training and enter practice,
what they require of the trainee. Often referred to as Miller’s external forces come to play a very large role in performance.
pyramid, it is composed of four levels: knows, knows how, The Cambridge Model, a variation on Miller’s pyramid,
shows how, and does.63 proposes that performance in practice (the highest level
of the pyramid) is influenced by two large forces beyond
Miller’s Pyramid competence.65 Systems-related factors, such as government
Knows. This is the lowest level of the pyramid and it con- programs, clinical microsystems (i.e., the clinical units
tains methods that assess what a trainee “knows” in an area where learners care for patients), institutional care delivery
of competence. Forming the base of the pyramid, knowl- practices, patient expectations, and guidelines, among other
edge represents the foundation upon which clinical com- factors, strongly influence what physicians do. Similarly,
petence is built. An MCQ-based examination composed of factors related to the individual physician such as state of
questions focused on ethics and principles of patient con- mind, physical and mental health, and relationships with
fidentiality would provide an assessment of what a trainee peers and family have a significant effect. Consequently,
“knows” about professionalism. assessment becomes more difficult because it is harder to
Knows how. To function as a physician, a good knowl- disentangle the effects of the context (e.g., context speci-
edge base is necessary but insufficient. It is important to ficity; see Chapter 7) of care from the competence of the
know how to apply this knowledge in the acquisition of individual physician. Here, a focus on health care processes
data, the analysis and interpretation of findings, and the and outcomes as a measure of what a physician “does” can
development of management plans. For example, a method provide a robust assessment of a physician’s ability to inte-
that poses a moral dilemma, asks trainees to reason through grate multiple competencies within a complex social con-
it, and evaluates the sophistication of their moral thinking text. However, processes and outcomes are still impacted by
would provide a “knows how” assessment of professionalism. system factors that can affect patient preferences and thus
CHAPTER 1 Assessment Challenges in the Era of Outcomes-Based Education 7
impact the measurement of processes of care. Finally, avail- to learners who are in the competence or proficiency stages.
ability of specific services may also impact outcomes. It is important to realize that learners are typically at differ-
ent stages depending on the content and context of the task
Dimension 3: Assessment of Progression being assessed. For example, a resident may be seen as pro-
ficient in working up a patient with chest pain but be at the
Acquiring competence is not an overnight process. Train- advanced beginner level in counseling a patient regarding
ees progress through a series of stages that begin in under- end-of-life care. Likewise, many students achieve compe-
graduate medical education and continue throughout their tence with regard to medical knowledge, or perhaps com-
careers. Educators must be able to recognize when a trainee munication skills, before they acquire the same level in more
has attained sufficient knowledge, skills, and attitudes to challenging systems-based practice domains such as care
enter the next stage, and this requires appropriate standards coordination or cost-conscious care delivery. Ultimately,
and benchmarks for the transition. Hubert and Stuart Drey- work-based assessment will need to predominate, especially
fus have created a developmental model of learning appli- for ongoing professional development in both training and
cable to the health professions that proposes five stages of practice. Educators need to recognize this developmental
educational development (Table 1.2).17,66 sequence when designing an assessment system, and it will
The characteristics of learners and the steps they must be critical to ensure that the chosen method is suitable to
go through to acquire competence will change over the five the task.
stages of development. Necessarily, the methods of assess-
ment applied at each developmental level will likely also Criteria for Choosing a Method
evolve. For example, at the level of the novice, an MCQ-
based knowledge test might be most appropriate, but a stan- Decisions about which method of assessment to use in a
dardized patient–based examination might be better suited particular circumstance have traditionally rested on validity
TABLE
1.2 The Stages of Learning as Proposed by Dreyfus
From Dreyfus HL: On the Internet. Thinking in Action Series. New York, Routledge, 2001.
8 CHA P T ER 1 Assessment Challenges in the Era of Outcomes-Based Education
and reliability. Validity is the degree to which the inferences particular method fits into the overall system for assessment.
based on the results of an assessment are correct. Valid infer- The same method can (and arguably should) be used to
ences regarding a particular test score or assessment result assess more than one competency. For example, peer assess-
depend on the reliability of these outcomes, and reliability ment can provide a measure of both professionalism and
is a component in more “modern” concepts of validity such interpersonal skills. Likewise, two different methods can
as those Kane and Messick discuss in Chapter 2 (Chapter be used to capture information on the same competency,
2 provides more detail about validity and psychometric thereby increasing confidence in the results. For example,
theory). patient care can be assessed using both the mini-CEX (clini-
For purposes of assessment in medical education, van der cal evaluation exercise) and monthly ratings by attending
Vleuten added educational effect, feasibility, and accept- physicians.
ability as factors to be considered in choosing a method of Educational effect, catalytic effect, feasibility, and accept-
assessment. This combination of factors is often referred to ability are not easily quantifiable, nor is the relationship
as the utility index and represented by the equation Valid- among methods of assessment in a system. However, these
ity × Reliability × Educational Effect × Cost Effectiveness × factors plus reliability and validity should be weighed inter-
Acceptability = Utility.67 Utility is a useful concept as pro- actively when considering selection of a particular method.
grams choose and implement assessment methods. It is also
important to note that utility is a multiplicative construct— Elements of Effective Faculty Development
if any one of the terms, or variables, is zero, then utility by
definition is zero. Faculty members play a particularly critical role in assess-
In terms of educational effect, van der Vleuten and ment in the clinical setting because it is often based on
Schuwirth argue that trainees will work hard in preparation observation. And by faculty we mean any health profes-
for an assessment.61 Consequently, the method should direct sional, at a minimum, who participates in an assessment
them to study in the most relevant way. For example, if an system. Recall that Miller placed “does,” meaning the care of
educational objective is for trainees to know the differential actual patients, at the tip of the pyramid. Envision the pyra-
diagnoses for a particular chief complaint, then assessment mid as a spear and at the tip of that spear is patients. Using
using extended matching questions will likely induce better this metaphor helps faculty appreciate the central role of
learning than assessment based on standardized patients. observation in both ensuring trainee competence (at a mini-
Feasibility is the extent to which an assessment method mum) and guaranteeing that patients receive high-quality,
is affordable and efficient. Although high-fidelity simula- safe care in the context of training.32 Most important is the
tions might be a good way to assess procedural competence, fact that the actual measurement instrument is the faculty,
the use of a method such as direct observation of proce- not the assessment tool. We cannot emphasize this enough
dural skills (DOPS), which is based on faculty observa- throughout the book that assessment in the workplace is
tion, is likely to be more feasible in most graduate training essential and relies on informed, expert judgment.
settings.68 Assessment methods and tools are only as good as the
Acceptability is the degree to which the trainees and fac- individuals using them. Although there has been substan-
ulty believe that the method produces valid results. This fac- tial progress in creating many new methods and tools,
tor will influence motivation of faculty to use the method significantly less attention has been paid to the develop-
and enhance the trainees’ trust of the results. It is important ment of approaches to training faculty in how to use them
that educational leaders not underestimate trainee knowl- most effectively. This omission continues to occur despite
edge and understanding of assessment and their ability to repeated studies over time demonstrating significant prob-
participate in decisions regarding assessment practices. lems with the quality of faculty assessments.31,70–72 Chapter
More recently, an international group of assessment 4 will cover in greater detail key issues in observation and
experts led by Norcini updated the concept of utility.69 rater cognition. There are three significant reasons faculty
Validity, acceptability, and educational effect were retained training is urgently needed.
as separate categories, and for validity the importance of First, to perform quality assessment, faculty members
coherence (a body of evidence that hangs together to support must possess sufficient knowledge, skill, and attitudes in
the results for a specific purpose) was highlighted. Reliabil- the competency targeted by the assessment. For example,
ity was essentially split into two new categories: reproduc- the decline of clinical skills teaching in the workplace was
ibility and consistency (i.e., repeatability) and equivalence noted by George Engel73 in 1976 and has resulted in many
(assessment yields equivalent results across space and time). of today’s educators failing to acquire a high level of clinical
Catalytic effect was added to highlight the important role skills needed for effective care and teaching. This likely lim-
of assessment in driving future learning forward (through its the degree to which they can validly assess clinical perfor-
assessment date and feedback). Finally, the last new category mance, and recent research adds evidence to the importance
was feasibility, namely, that assessment should be practical, of the faculty’s own underlying clinical skills.31
realistic, and sensible.69 Second, competencies will evolve and change over time.
In addition to factors highlighted in two versions of cri- Witness the birth of the competencies of practice-based
teria for good assessment, it is important to consider how a learning and improvement and systems-based practice, and
CHAPTER 1 Assessment Challenges in the Era of Outcomes-Based Education 9
more recently the change of the manager role in CanMEDS Overview of Assessment Methods
to leader.74 The majority of faculty today during their own Traditional Measures
training never received any formal instruction in many of
the competencies and subcompetencies now needed for Traditional measures will continue to play an important
modern practice. Many faculty learn new knowledge and role in the assessment of clinical proficiency (see Chapters
skills alongside their trainees.33 5 and 6).
Finally, assessment is a core tenet of professionalism for Specifically, written methods such as MCQs and stan-
medical educators. Too often, faculty members view it as dardized patients will be foundational components of
someone else’s job, especially when a negative performance assessment programs for the near future, especially in
appraisal is involved (see Chapter 15). Faculty development undergraduate medical education. All of these methods can
reinforces the importance of assessment and provides medi- be improved and work on each must continue.
cal educators the opportunity to develop common standards
for performance. Methods Based on Observation
To make effective use of the methods of assessment, edu-
cational institutions must commit the necessary resources Even though assessment has been woven through the basic
for faculty development. However, too often faculty devel- science curriculum, historically it has not been as well inte-
opment translates into a project or a brief workshop. If grated with clinical education (see Chapters 3, 4, 7, and 11).
faculty development is to be truly successful, medical Nonetheless, assessment methods based on the obser-
educators need to embrace new strategies that embed fac- vation of routine encounters in the clinical setting offer a
ulty development in real-time teaching and clinical activi- rich and feasible target for assessment. Continued refine-
ties. For example, Hemmer and colleagues embed faculty ment of the methods themselves is needed, as is faculty
frame-of-reference training into formal evaluation sessions development, which is a key to their successful use. Further-
for students.75 Faculty development, like quality improve- more, the opportunity for educational feedback as part of
ment and maintenance of competence, must become a these methods is probably as important as their assessment
continuous process and be appropriately rewarded. As potential.
noted earlier, the quality and safety of patient care depend
on it. Simulation
Medical educators must also end their quest for the holy
grail of assessment, the perfect rating form imbued with Improvements in technology have spurred the development
special powers to solve all measurement needs. Assessment is of a series of simulators that recreate reality with high fidel-
hard work and requires a multifaceted approach. Landy and ity (see Chapters 5 and 12). The use of simulation in assess-
Farr, in a landmark article in the performance appraisal field ment is growing but much of the technology still remains
over 35 years ago, pleaded with researchers to redirect devel- expensive, and several developments are needed before
opment efforts from a search for the perfect rating form widespread adoption and use. Researchers will need to con-
to training the assessors.76 Researchers in this field subse- tinue to focus on identifying appropriate scoring methods,
quently developed numerous rater training approaches that optimizing the generalizability of scores, and ensuring their
can lead to better assessments. Chapter 4 provides guidance relevance to performance in practice.80 Particularly in the
on a number of practical faculty training methods. area of procedural skills, however, these methods will offer
Milestones and EPAs, described later, require special the ability to test under a variety of conditions without con-
consideration. Using EPAs and milestones for curriculum cern for harm to patients. Some evidence is accruing that
development and assessment requires a shift in thinking mastery-based approaches combined with simulation-based
by faculty and an infrastructure to support new assessment deliberate practice can translate into improved patient care
practices. Both individual faculty and committees must get and outcomes.45,81,82 Educators will confront difficult deci-
acquainted with and become experienced in entrustment sions requiring them to balance the cost, variable fidelity
decision making for EPAs and its conditions.77 Training in of individual simulation methods, and potential risks to
the dimensions to be used in assessment and in the criteria patients (and trainees) in making decisions regarding how
for decisions is needed, and specific tools related to EPA- best to assess procedural skills.83
based assessment, such as video recording, are now being
developed. If anything, sufficient and adequate supervi- Work
sion and feedback are key to entrustment decisions, which
requires longitudinal mentorship.78,79 This does not neces- The assessment of physicians’ performance at work (mostly
sarily mean huge investments in time for mentoring, but the “does” level of Miller’s pyramid) is the area of assessment
does mean an efficient use of any encounter that mentors undergoing the most change and development (see Chap-
and mentees have, for the benefit of learning. Group process ters 3, 4, 7, 8, 9, 10, 11, and 14). Although learners may try
will also likely enhance the effectiveness of milestones and to “perform” when under direct observation (“show how”),
EPAs as part of an assessment system, and faculty will need most learners acclimate quickly, and even if what the faculty
training in effective group process.57 observe is “best behavior,” there is still much utility in the
10 CHA P T ER 1 Assessment Challenges in the Era of Outcomes-Based Education
assessment and in ensuring the patient receives safe, effec- national systems of assessment,29 we provide some back-
tive, patient-centered care.31 The day-to-day performance ground in this chapter to help guide the reader in evaluat-
of physicians is being used increasingly in the settings of ing and exploring these concepts for their own assessment
continuous quality improvement and physician account- program.
ability. Assessment in this context is a matter of identifying
the basis for the judgments (e.g., outcomes, process of care), Milestones
deciding how the data will be gathered, and avoiding threats
to validity and reliability (e.g., patient mix, patient com- The ACGME competency framework has always been
plexity, attribution, and numbers of patients).84 The patient inspired by the five “Dreyfus stages of development of skill,”
is also playing a much greater role in work-based assess- including novice, advanced beginner, competent, proficient,
ment, predominantly through patient experience surveys.85 and expert, first described in 1986,17,26 but only several years
In addition, patient-reported outcome measures (PROMs) later it was suggested to actually superimpose the stages as
are being increasingly used by health systems to judge func- milestones on the framework.26 Milestones were adopted to
tional outcomes for patients (see Chapter 10). Although facilitate the assessment of learners in the workplace and
substantial research is now occurring in quality and safety facilitate curricular change.86 They are concrete behavioral
measures, patient experience surveys, and PROMs, much descriptions aligned with the five developmental steps to
work remains to be done. However, given that this is ulti- assist faculty in the assessment of medical trainees using a
mately what patients and the public most care about, edu- logical trajectory of professional development within com-
cational programs need to embrace work-based assessments petencies and subcompetencies. Developed as benchmarks
as part of an overall assessment program. for effective assessment, ACGME milestones were written
for all US postgraduate medical disciplines and published
New Directions in Assessment in the Journal of Graduate Medical Education in March 2013
and March 2014.87 Specialty milestones are the framework
Implementation of competency-based medical education programs use for semiannual reports on resident progress.
models has been very challenging for many programs across Table 1.3 shows, as an example, one of the 21 milestone
the educational continuum.11,29 One reason has been the sets of the pediatric competencies.88 In 2014, all specialties
difficulty in translating the language and concepts of com- had described milestones for their programs,87 and every
petencies into educational practices and assessments. As US resident must now be regularly evaluated against all
a result, two new approaches, milestones and EPAs, have competencies of the specialty using these milestones. Early
arisen and continue to evolve as mechanisms to potentially research using national data for all emergency medicine and
facilitate more effective implementation of outcomes-based internal medicine programs in the United States demon-
education using competency frameworks. Although both of strate encouraging findings for some aspects of validity.89,90
these newer approaches are grounded in robust educational Milestones also have been reported to be helpful for earlier
theory, it is important for the reader to recognize that we identification of residents having difficulty, for better feed-
are in the very early days in determining the utility, includ- back to residents and fellows, and for development of better
ing validity, and impact of both milestones and EPAs on assessment approaches, and to be a useful framework for
educational and clinical outcomes. Although early research faculty development.91
is encouraging, much work remains to be done. However, In the 2015 edition of CanMEDS, milestones were
given that both milestones and EPAs are becoming part of also introduced and defined as “descriptions of the abilities
TABLE
1.3 Example of ACGME Milestone Descriptions with One of the 21 Competencies of Pediatric Training
Competency: Demonstrate humanism, compassion, integrity, and respect for others; based on the characteristics of an empathetic
practitioner
Level 1 Level 2 Level 3 Level 4 Level 5
Sees the patients Demonstrates compassion Demonstrates consis- Is altruistic and goes Is a proactive
in a “we versus for patients in selected tent understanding of beyond responding to advocate
they” framework situations (e.g., tragic patient- and family- expressed needs of on behalf of
and is detached circumstances, such as expressed needs and patients and families; individual
and not sensitive unexpected death) but has a desire to meet those anticipates the human patients,
to the human a pattern of conduct that needs on a regular needs of patients and families, and
needs of the demonstrates a lack of basis; is responsive in families and works to groups of
patient and sensitivity to many of the demonstrating kindness meet those needs as part children in need
family needs of others and compassion of skills in daily practice
expected of a trainee or physician at a defined stage of pro- EPAs have now been identified for many graduate medi-
fessional development” of each of the “enabling competen- cal education programs including obstetrics/gynecology,
cies” under the seven CanMEDS competency roles, to guide pediatrics, internal medicine, family medicine, psychia-
learners and educators in determining whether learners are try, hematology and oncology, and pulmonary and critical
“on track.”15 care.96–102 An example of an EPA is conducting an uncompli-
cated delivery. This activity, performed by family physicians
Entrustable Professional Activities and obstetrics-gynecology specialists, needs to be entrusted
to a trainee at some point in his or her training, as the
The concept of EPAs was introduced in 2005.92 After a trainee eventually will need to conduct it without supervi-
publication in Academic Medicine in 2007,93 it has attracted sion. It requires specific knowledge, skills, and behaviors;
substantial attention among postgraduate programs in the proficiency is acquired through training; and it is directly
United States, Canada, and other countries. Since then, observable and reflects competencies. As this activity par-
EPAs have been proposed for numerous programs and con- ticularly reflects the CanMEDS roles of medical expert,
stitute an emerging basis for judging readiness for resident communicator, and collaborator, it exemplifies how EPAs
entry in the United States and Canada.94 A most recent integrate competencies. Other examples of EPAs are provid-
elaborate description of the concept and how to use it in ing preoperative assessment, managing care of patients with
workplace training and assessment can be found in a Guide acute common diseases across multiple care settings, provid-
99 of the Association for Medical Education in Europe.49 ing palliative care, managing common infections in nonim-
An EPA can be defined as a unit of professional practice munosuppressed and immune-compromised populations,
that can be fully entrusted to a trainee as soon as he or conducting a family education session about schizophrenia,
she has demonstrated the necessary competence to execute conducting a risk assessment, serving as the primary admit-
this activity unsupervised. In contrast with competencies, ting pediatrician for previously well children suffering from
they are not a quality of a trainee, but a part of work that common acute problems, pharmacologically managing an
must be done. Table 1.3 shows a typical competency— anxiety disorder, providing end-of-life care for older adults,
not related to a specific task—whereas an EPA would be a and offering office-based counseling in developmental and
concrete task that requires that and often other competen- behavioral pediatrics. A comprehensive set of EPAs should
cies. More specifically defined, EPAs are part of essential cover the core of a profession. Each EPA should be described
professional work in a given context—they must require well and include, next to an informative title, specifications
adequate knowledge, skill, and attitude, generally acquired and limitations; a listing of required competencies; elabo-
through training; must lead to recognized output of pro- ration of required experience, knowledge, and skills; sug-
fessional labor; should usually be confined to qualified gestions for assessment; and an expiration date showing
personnel; should be independently executable; should be when the practitioner should no longer be assumed to be
executable within a time frame; should be observable and competent in the EPA after a period of non-practice.94 See
measurable in the trainee’s process and outcome, leading to Appendix 1.2.
a conclusion (“well done” or “not well done”); and should Linked to the EPA construct is the purpose of entrustment
reflect one or more of the competencies to be acquired (see decision making. This process serves to acknowledge ability,
Appendix 1.1).92 provide permission to act with limited supervision, and enable
Much of the work done in health care can be captured duties in health care practice. True competency-based medi-
by tasks or responsibilities that must be entrusted to indi- cal education grants certification as soon as competence is
viduals. EPAs require a practitioner to possess and inte- adequately demonstrated, irrespective of the time in training,
grate multiple competencies simultaneously from several
domains, such as content expertise, skills in collaboration, TABLE
communication, management, and so forth. Conversely, 1.4 Overview of EPAs–Competencies Matrix
each domain of competence is relevant to many different
activities. Combining competencies (or domains of com- EPA EPA EPA EPA EPA EPA
petence) and EPAs in a matrix reveals which competencies 1 2 3 4 5 6
in particular a trainee must achieve before being trusted Competency 1 ● ● ● ●
to perform an EPA.94 The two-dimensional matrix in
Competency 2 ● ● ●
Table 1.4 provides specifications that are helpful for assess-
ment and feedback, for individual development, and for Competency 3 ● ● ● ●
* Patient care
** Medical knowledge
Provide
telephone Interpersonal and
** communication skills
advice and
manage-
ment of * Systems-based practice
patients
Practice-based learning
**
and improvement
* Professionalism
direct oversight
supervision only
and this requires a personalized and flexible approach to train- • BOX 1.2 Five Levels of Supervision and
ing programs. EPAs allow for making entrustment decisions Permission
for separate units of professional practice, resulting in a more
gradual, legitimate participation in professional communi- 1. Be present and observe, but not permitted to perform the
ties of practice103 rather than a full license to practice on the entrustable professional activity (EPA).
2. Be permitted to act under direct, pro-active supervision,
last day of training.94 Certification for EPAs is not a dichoto- present in the room.
mous process. As trust increases, the level of supervision can 3. Be permitted to act under indirect, re-active supervision,
decrease. A model of five levels of supervision, entrustment, readily available to enter the room.
and permission has been proposed for postgraduate training, 4. Be permitted to act without qualified supervision in the
shown in Box 1.2.93,104 vicinity; with distant supervision or clinical oversight;
basically acting unsupervised.
5. Be permitted to supervise junior trainees regarding the EPA.
Combining Milestones and Entrustable
Professional Activities
Although the implementation of milestones and EPAs, on top 4 can be viewed as passing the threshold that allows for clinical
of competencies, may feel to critics like another burden for oversight only. It does not qualify a trainee to stop developing
programs and individual teachers,105 some authors have sug- but would allow for a formal recognition of ability, permission,
gested combining both. Eric Warm, program director of the and duty to enact the EPA, sometimes called a Statement of
University of Cincinnati internal medicine residency training Awarded Responsibility (STAR)93 or a summative entrustment
program, has simply equated the five milestone levels of com- decision (Table 1.5).
petencies (see Table 1.3) with the five supervision levels of EPAs Given this alignment, an example may be given. Suppose
(Box 1.2). Faced with the need to regularly report on milestones a pediatric residency program has an EPA called “Provide
for all residents, he asks clinicians to estimate the trainees’ readi- telephone advice and management of patient” (taken from
ness for direct supervision, indirect supervision, or unsuper- Jones and colleagues109). In the EPAs–competencies matrix,
vised practice. This serves efficiency and conceptual elegance. it has been determined that the most important domains
To take this approach one step further, the Dreyfus model,66 the of competence are medical knowledge, interpersonal and
broadly used RIME model (reporter-interpreter-manager-edu- communication skills, and practice-based learning and
cator106 [see Chapter 3]), the milestones approach,107 and levels improvement. Let us assume that for each of these domains,
of supervision can all be aligned as shown in Fig 1.1. The model milestones have been described. A trainee must be assessed
can be extended with more detailed representations of behavior to determine whether indirect supervision (i.e., not with a
and supervision,66,108 but the core idea is that of alignment of supervisor present in the room) is justified. If the trainee
frameworks. Moving from milestone or supervision level 3 to meets the expected behavior at milestone level 3 in all of
CHAPTER 1 Assessment Challenges in the Era of Outcomes-Based Education 13
TABLE
1.5 Alignment of Various Models of Development
Appropriate Level
Milestone Dreyfus Model Transition to of Supervision and
Level Stages Learner Behavior RIME Stages Practitioner Permission
1 Novice Doing what is told, Reporter Introduction to clinical Observation, no
rule driven practice enactment
2 Advanced Comprehension Reporter/Interpreter Guided clinical practice Act under direct,
beginner proactive supervision
3 Competent Application to Interpreter/Manager Early independence Act under indirect,
common practice reactive supervision
4 Proficient Application to Manager/Educator Full unsupervised practice Clinical oversight
uncommon practice
5 Expert Experienced clinician Educator Aspirational growth after Provide supervision to
graduation others
the three most relevant domains, that decision seems justi- Entrustable Professional Activities Across
fied. If the trainee does not yet show the behavior or skill the Continuum and Nested Entrustable
expected at level 3 in any one of the competencies, there will Professional Activities
have to be more close supervision. In the terminology of the
RIME model, the learner would be evaluated as an adequate Activities can be small or large. There is no easy answer to
interpreter and beginning manager. Table 1.3 shows this the “right” breadth of EPAs and consequently to the num-
relationship.49 ber of EPAs. If the question is “What is the scope of respon-
The model can also be used in reverse order. Clini- sibility that is covered when an EPA is entrusted to a trainee
cal educators may start with an intuitive gut feeling that a for indirect supervision?” then clearly big differences can
trainee is ready for indirect supervision, based on his or her arise depending on the stage of training of the trainee in
experience in various settings. Then a quick check of the question. The first EPA that may be entrusted to a junior
important competency domains may confirm this and the medical student could be “measuring blood pressure.” If we
conclusion drawn that the trainee meets milestone level 3 of consider this a unit of professional practice or activity that
relevant competencies. Warm and colleagues reported suc- one can trust a trainee to complete without being checked
cess in organizing regular assessments of all internal medi- by a supervisor, then it is a true EPA (Fig. 1.2).
cine residents of a large Cincinnati program by scoring on Clearly, however, at a later stage this responsibility is part
an entrustment–supervision scale, assuming alignment with of a full standard physical examination that is a more logical
milestones scales.110 activity for entrustment for advanced medical students. The
full standard physical examination, in turn, can be included
Entrustable Professional Activities – in a broader EPA of a standard outpatient consultation that
Competencies – Skills also includes the history. In technical terminology, smaller
EPAs are nested within larger EPAs.49
Although EPAs are units of work and competencies are Among the Utrecht University undergraduate EPAs, one
descriptors of personal qualities and abilities, in common is “the clinical consultation,” to be entrusted to any medical
language educators tend to call “physical examination” a graduate before graduation for indirect supervision. This is
competency. Strictly, not the physical examination itself, a relatively broad EPA, as it requires neurologic, ENT, gyne-
but the ability to perform a physical examination is the cologic, psychiatric, and other history and physical exami-
skill and is a feature of the learner or professional. And it nation skills. In the Utrecht curriculum, students are to be
would be correct to say that that skill, on a more detailed entrusted with a focused “ENT clinical consultation” at an
level, requires manual skills, visual skills, auditory skills, earlier stage, and likewise for other specialties during a dedi-
and even time management and communication skills. If a cated clerkship. Only in the final year do all these smaller
learner possesses these skills, or competencies if one would EPAs lead to full trust in the broad EPA of “the clinical
call them such, he or she may be granted the trust to do consultation,” to be signed off separately in a subinternship
the physical examination without supervision. Simply put, for indirect supervision.
health professionals require an integrated set of abilities For EPA-based evaluation, it is therefore adequate to
(i.e., competencies) to effectively execute the clinical activ- design EPAs for a particular course within the educational
ity (i.e., EPA). continuum, for example, EPAs for undergraduate education
14 CHA P T ER 1 Assessment Challenges in the Era of Outcomes-Based Education
Language: French
NOS FRÈRES
FAROUCHES
5e Édition
GALLIMARD
Copyright by Librairie Gallimard, Paris
I
MŒURS DE RAGOTTE
Elle est si naturelle que, d’abord, elle a l’air un peu simple. Il faut
longtemps la regarder pour la voir.
A l’école.
Elle est allée à l’école huit mois, chez ce vieil ours de Varneau.
On payait trente sous par mois et, en hiver, chaque élève
apportait le matin sa bûche.
Il y avait deux partis en classe : les écriveux et ceux qui
n’écrivaient pas. Ses sœurs ont eu le temps d’apprendre. Comme
elle était l’aînée, elle a dû tout de suite se mettre au ménage avec sa
mère, et elle n’a rien appris.
Elle connaît la lettre P, la lettre J et la lettre L, parce que ces
lettres lui ont servi à marquer le linge de ses petits, qui s’appellent
Paul, Joseph et Lucienne. Elle reconnaît aussi le chiffre 5, on ne sait
pas pourquoi.
Elle ne peut rendre la monnaie que sur dix sous. Par exemple, si
on lui achète un sou de lait, elle redoit neuf sous. A partir de dix
sous, elle s’embrouille, et elle aime mieux dire :
— Vous me paierez une autre fois !
Elle ne sait pas encore que le timbre des lettres est à deux sous.
Louée.
Mariée.
Elle s’est mariée en sabots ; elle avait acheté des souliers neufs,
mais par crainte de les salir, elle ne voulait les mettre que pour faire
son entrée à l’église. Arrivée sous le porche, elle voit que sa mère,
qui devait les porter à la main, ne les a pas.
— Et mes souliers, maman ?
— Ha, ma fille, je les ai oubliés ; ils sont sous l’armoire, mon
enfant !
Il fallut bien aller jusqu’au chœur avec les sabots qui tapaient le
moins possible sur les dalles.
Amour.
En ménage.
Philippe ne lui donne jamais un sou. Il fait sa vie de son côté, elle
fait la sienne du sien. Loin de se plaindre, elle blâme certaines
femmes :
— Il y en a, dit-elle, qui gardent le porte-monnaie et qui ne
remettent de l’argent à leur homme que vingt sous par vingt sous.
Moi, je ne pourrais pas.
Toutefois, elle pense qu’à la rigueur la femme peut vivre sur son
homme, et même le mari sur sa femme : c’est compagne et
compagnon ! Mais un père et une mère ne doivent pas rester à la
charge de leurs enfants. Dès qu’elle ne pourra plus, aidée de son
principal ou seule, faire sa vie, elle voudra mourir.
— Dans un ménage, dit-elle, quand il pleut sur l’un, il fait mou sur
l’autre.
Ce qui veut dire que, si l’un gagne des sous, l’autre en profite.
Elle ne dépense pas dix francs par an à son entretien, et dans les
vieilles culottes qu’on passe à Philippe et qu’il use, elle trouve
encore de bonnes pièces pour se faire des chaussons tout neufs.
Elle se chauffe mal, si elle ne voit pas le feu ; elle aime les beaux
feux de bois dont la braise ardente fait pleurer des larmes cuites ;
mais elle trouve que rien ne vaut le gentil feu d’une paire de sabots
qu’elle a portés, qu’elle brûle quand ils ne sont plus mettables, et
qu’elle regarde flamber, toute songeuse.
Elle a pris d’abord le tub pour un ciel de lit et elle finit par trouver
que ces boules, que le monsieur appelle des haltères, pourraient
servir à écraser le sel.
C’est une des dernières paysannes qui ne veulent pas accepter
certains progrès et qui s’arrêtent et se baissent n’importe où.
— Quand je suis allée à Moulins, chez une cousine, comme
j’avais un petit besoin, elle m’a mise dans une chambre, oui, toute
seule, dans une vraie chambre ! Oh ! que j’avais peur ! je serais
morte si on était entré.
Elle croit que nous sommes très riches, et si quelqu’un lui disait
que nous avons au moins mille francs, ça ne l’étonnerait pas.