See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/317487733

Competency-based medical education and continuing professional

development: A conceptualization for change

Article  in  Medical Teacher · June 2017

DOI: 10.1080/0142159X.2017.1315064

CITATIONS READS

22 618

6 authors, including:

Ford Bursey Denyse Richardson

Memorial University of Newfoundland University of Toronto
33 PUBLICATIONS   908 CITATIONS    46 PUBLICATIONS   1,893 CITATIONS   

SEE PROFILE SEE PROFILE

Jason R Frank
Royal College of Physicians and Surgeons of Canada
132 PUBLICATIONS   4,750 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Emergency Medicine in Canada View project

Effective M&M Rounds for Quality Improvement View project

All content following this page was uploaded by Jason R Frank on 20 September 2017.

The user has requested enhancement of the downloaded file.

 Medical Teacher

ISSN: 0142-159X (Print) 1466-187X (Online) Journal homepage: http://www.tandfonline.com/loi/imte20

Competency-based medical education and

continuing professional development: A
conceptualization for change

Jocelyn Lockyer, Ford Bursey, Denyse Richardson, Jason R. Frank, Linda Snell,
Craig Campbell & on behalf of the ICBME Collaborators

To cite this article: Jocelyn Lockyer, Ford Bursey, Denyse Richardson, Jason R. Frank, Linda
Snell, Craig Campbell & on behalf of the ICBME Collaborators (2017) Competency-based medical
education and continuing professional development: A conceptualization for change, Medical
Teacher, 39:6, 617-622, DOI: 10.1080/0142159X.2017.1315064

To link to this article: http://dx.doi.org/10.1080/0142159X.2017.1315064

Published online: 09 Jun 2017.

Submit your article to this journal

Article views: 398

View related articles

View Crossmark data

Citing articles: 2 View citing articles

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=imte20

Download by: [74.12.189.243] Date: 19 September 2017, At: 18:15

 MEDICAL TEACHER, 2017
VOL. 39, NO. 6, 617–622
http://dx.doi.org/10.1080/0142159X.2017.1315064

Competency-based medical education and continuing professional

development: A conceptualization for change
Jocelyn Lockyera, Ford Burseyb, Denyse Richardsonc,d, Jason R. Frankd,f, Linda Snelld,e and Craig Campbelld,g;
on behalf of the ICBME Collaborators
a
Cumming School of Medicine, University of Calgary, Calgary, Canada; bFaculty of Medicine, Memorial University of Newfoundland, St
John’s, Canada; cDepartment of Medicine, University of Toronto, Toronto, Canada; dRoyal College of Physicians and Surgeons of Canada,
Ottawa, Canada; eCentre for Medical and Department of General Internal Medicine, McGill University, Montreal, Canada; fDepartment of
Emergency Medicine, University of Ottawa, Ottawa, Canada; gDepartment of Medicine, University of Ottawa, Ottawa, Canada

ABSTRACT
Competency-based medical education (CBME) is as important in continuing professional development (CPD) as at any other
stage of a physician’s career. Principles of CBME have the potential to revolutionize CPD. Transitioning to CBME-based CPD
will require a cultural change to gain commitment from physicians, their employers and institutions, CPD providers, profes-
sional organizations, and medical regulators. It will require learning to be aligned with professional and workplace standards.
Practitioners will need to develop the expertise to systematically examine their own clinical performance data, identify per-
Downloaded by [74.12.189.243] at 18:15 19 September 2017

formance improvement opportunities and possibilities, and develop a plan to address areas of concern. Health care facilities
and systems will need to produce data on a regular basis and to develop and train CPD educators who can work with phys-
ician groups. Stakeholders, such as medical regulatory authorities who are responsible for licensing physicians and other
standard-setting bodies that credential and develop maintenance-of-certification systems, will need to change their para-
digm of competency enhancement through CPD.

Introduction
The advent of competency-based medical education
(CBME) is transforming postgraduate medical education to
an outcomes-driven education and assessment model
(Englander & Carraccio 2014). Key elements of the CMBE
approach include a focus on enhanced competencies with
the goal of improving patient outcomes; the use of mile-
stones and entrustable professional activities (EPAs) to pro-
vide a structure for teaching, learning, and assessment; and
the use of time as a resource for learning rather than as a
proxy for progression of competence. Although CBME prin-
ciples have yet to be widely adopted in continuing profes-
sional development (CPD), we posit that they are just as
important after residency – a phase of professional practice
that can span 30 or 40 years – as they are during residency
training, and that significant changes are needed in our
approach to CPD to ensure that practising physicians main-
tain competence throughout their careers.
In this paper, we explore the rationale for CPD reform
and describe key aspects of CBME – particularly, competen-
cies, milestones, and EPAs. We consider the key elements
that would facilitate a transition to a CBME-CPD framework
and an expanded role for the assessment of competence
and performance in the workplace. We suggest educational
activities to support the continued learning of physicians in
a CBME environment and provide examples of workplace-
based assessment along with evidence of effectiveness.
Practice points
 Competency-based medical education (CBME) is
as important in the continuing professional devel-
opment (CPD) phase of a physician’s career as it is
in undergraduate and postgraduate medical
education.
 Practising physicians will need to develop the
expertise to systematically examine their own clin-
ical performance data, identify performance
improvement opportunities and possibilities,
develop a plan to address areas of concern, and
demonstrate outcomes of learning and assess-
ment within their scope of practice.
 Health care institutions will need to develop and
train personnel who can work with physician
groups to provide data on a regular basis related
to meaningful clinical work.
 CPD provider organizations will need to transition
from providing accredited learning activities to
seeing their mission as facilitating learning and
assessment and negotiating resources to support
physician learning.
 Medical regulatory authorities and professional
colleges will need to review and revise their cur-
rent expectations for planning, demonstrating,
and reporting on CPD outcomes.

CONTACT Jocelyn Lockyer lockyer@ucalgary.ca University of Calgary, Dept of Community Health Sciences, Floor 3, TRW Building, 3280 Hospital
Dr NW, Calgary. AB, Canada. T2N 4Z6
ß 2017 Informa UK Limited, trading as Taylor & Francis Group
 618 J. LOCKYER ET AL.
We recognize that assessment activities and educational
interventions are interdependent and interactive: assess-
ment activities can also be educational interventions, par-
ticularly when feedback is an integral part of those
activities. We conclude by summarizing the implications
for individual learners, health care facilities and systems,
CPD provider organizations, and medical regulatory
authorities.
The rationale for change in CPD
Numerous studies point to deficiencies in physician know-
ledge and actions that compromise quality of care and
patient outcomes (Drotar 2009; Erasmus et al. 2010;
Nieuwlaat et al. 2013). Although research findings show
that physicians who participate in formal CPD activities are
more likely to provide better care than their peers who do
not participate (Goulet et al. 2013; Wenghofer et al. 2014),
it is also recognized that there are limitations to the impact
of formal group learning on behavior change and patient
Downloaded by [74.12.189.243] at 18:15 19 September 2017
outcomes (Forsetlund et al. 2009). There is also an increas-
ing recognition that physicians identified as practising at a
suboptimal level tend to be those who are isolated
(whether by age or by virtue of working in a solo practice),
are practising outside the scope of their expertise, and do
not hold certification congruent with their scope of practice
(Choudhry et al. 2005; Grace et al. 2014). Stakeholders in
health care have looked to the profession to address these
issues through improvements to CPD.
Continuing medical education has a vital role to play
in ensuring that physicians do not rely solely on their
own discretion to determine whether they are sufficiently
prepared to deliver care in an area of practice.
Maintaining competence is challenging, given the known
limitations to physicians’ ability to self-assess (Davis et al.
2006; Sargeant et al. 2010; Lockyer et al. 2011) coupled
with the limited feedback and data available to them to
identify areas for improvement. CBME in CPD has the
potential to address this lack of awareness and to use
identified gaps to guide ongoing learning and subsequent
changes in clinical practice. We should also bear in mind
that the many transitions that physicians make over the
course of their careers can have deleterious effects. At
transition points it is critical for physicians to assess
whether their competence remains aligned to their evolv-
ing scope of practice (Lockyer et al. 2014, 2015). Although
entry into practice is the transition point that appears to
have garnered the most attention, physicians inevitably
make several transitions in succeeding years in response
to new technology (which requires skill development),
geographic relocation (which requires adaptation to new
cultural and community norms), or the refocusing of a
career in a new direction to meet personal or community
needs. Some physicians who reduce working hours or
leave practice may later face reentry challenges.
Eventually, physicians transition out of practice, sometimes
in recognition of limited ability or a desire to try some-
thing new (Lockyer et al. 2014, 2015). Preparing for a tran-
sition to a nonclinical role or to retirement has
implications for quality of care and for physician well-
being and requires specific competencies and awareness
that can be supported by the CBME model.
Applying CBME to CPD
In describing CBME, it is important to recognize the consid-
erable work that has gone into defining the components of
CBME and fostering consistency in the use of its vocabu-
lary. CBME itself is described as “an approach to preparing
physicians for practice that is fundamentally oriented to
graduate outcome abilities and organized around compe-
tencies derived from an analysis of societal and patient
needs. It de-emphasizes time-based training and promises
greater accountability, flexibility, and learner centeredness.”
(Frank et al. 2010a, p. 636). Because work in CBME began
within the formalized, predominately postgraduate, medical
education system, this definition rightly acknowledges the
outcomes as graduate abilities. However, since it is widely
accepted that physicians’ skills for lifelong learning are of
paramount importance to maintaining and enhancing com-
petence, we posit that the extension of CBME to CPD is
both natural and necessary.
CBME is based on three principles of learning, as follows:
(1) education must be based on the health needs of the
populations served; (2) the primary focus of education and
training should be the desired outcomes for learners rather
than the structure and process of the educational system;
and (3) the formation of a physician is a continuous pro-
gression of expertise and should be seamless across the
continuum of education and practice (Frank et al. 2010b;
Carraccio et al. 2016). In CBME, time is a resource for learn-
ing, not the organizing framework of educational achieve-
ment. CBME is organized around competencies, milestones,
and EPAs. Table 1 defines and illustrates these terms using
a clinical example.
CBME is oriented toward achieving progression in
expertise throughout practice. Physicians who have com-
pleted their formal training maintain and enhance the com-
petence they attained by the time of initial certification. For
those adding to their proficiency or mastery (for example,
in handling more complex cases), CBME is still relevant.
Being able to reference the milestones or EPAs defined in a
CBME approach can help guide professionals who are striv-
ing to acquire new competencies in response to patient
needs or personal interest in the ever-changing field of
medical practice. However, in comparison with undergradu-
ate and postgraduate medical education, CPD has been
more individualized and less structured with respect to
oversight and the role of assessment and feedback.
Nonetheless, the core CBME concept of demonstrating
competencies through the assessment of milestones or
EPAs is easily aligned with the goals of CPD.
Key elements required to support a competency-
based CPD model for physicians
Currently, the terms revalidation, recertification, maintenance
of competence, and maintenance of licensure are used to
describe systems related to affirming the continuing profes-
sional development of physicians. These systems have been
developed in different countries, primarily in response to
public concerns regarding the failure of medical regulation
(United Kingdom), the need to address the quality-of-care
gap (United States), and the need for the profession to
demonstrate commitment to competent performance in
practice (Canada). In their own way, each of these systems
 MEDICAL TEACHER 619

Table 1. Definitions and examples of competencies, milestones, and entrustable professional activities (EPAs) in continuing professional development (CPD).
Domain
Competency
An observable ability of a health pro-
fessional, integrating multiple com-
ponents such as knowledge, skills,
values, and attitudes (Frank et al.
2010a).
Milestone
A defined, observable marker of an
individual’s ability along a develop-
mental continuum (Englander et al.
2017).
EPA
An essential task of a discipline (pro-
fession, specialty or sub-specialty)
that an individual can be trusted to
perform without direct supervision
in a given health care context, once
sufficient competence has been
demonstrated (Englander et al.
2017).
Key competence at certification
Performs procedures (e.g. central
line insertion) in a skillful and
safe manner, adapting to
unanticipated findings or chang-
ing clinical circumstances.
Milestone achievement at initial
certification
Demonstrates ability to insert a
central line in a skillful and
safe manner.
EPAs at initial certification
Adequately communicates with
and manages patients during
and after the successful perform-
ance of central line insertion.
Examples in CPD
Revised competencies since certifi-
cation
Uses ultrasound as an adjunctive
tool in the planning and per-
formance of central line insertion
in practice.
Milestone achievement since certifi-
cation
Routinely performs central line
insertion using ultrasound.
EPA achievement since certification
Selects and adequately communi-
cates with and manages patients
during and after the successful
performance of central line inser-
tion using ultrasound.
Enhanced expertise in practice
Uses multiple ultrasound
approaches and strategies to
insert central lines in stable and
unstable patients.
Milestone achievement for
enhanced expertise
Is able to teach the use of ultra-
sound in central line insertion
to others.
EPA for enhanced expertise
Is able to personally demonstrate
and teach all aspects of prepar-
ing and performing central line
insertion using a range of
approaches using ultrasound.

Moving from a system that simply requires demonstra-

Downloaded by [74.12.189.243] at 18:15 19 September 2017

tion of participation in learning and assessment activities to

a competency-based model for physicians that involves
feedback and assessment will require a significant cultural
SCOPE OF PERFORMANCE shift. New components will be required, including the
PRACTICE DATA following:

CPD LEARNING 1. A broader list of competencies. A CBME approach to

CYCLES
COMPETENCY
FRAMEWORKS*
*Examples:
CanMEDS 2015;
ACGME
Figure 1. The interrelationship of competency-based CPD, competency
frameworks, and scope of practice.
is designed to promote greater transparency and account-
ability and to maintain the public’s trust. Although expecta-
tions vary from one country to another, the common goal
is to ensure that physicians can demonstrate their commit-
ment to lifelong learning by engaging in CPD activities and
using multiple strategies and tools to regularly assess their
competence and performance and improve the quality of
care they provide. As illustrated in Figure 1, within a com-
petency-based CPD structure there are at least three ele-
ments that influence the development of multiple, iterative
learning cycles: the competency framework, which provides
the foundation that guides reflection on the competencies
that must be maintained in general; scope of practice,
which stimulates the identification of practice-specific
needs based on a review of the physician’s professional
roles, responsibilities, and practice context; and perform-
ance data, which identify areas for improvement. Guided
by these elements, physicians plan and implement a series
of learning cycles and reflect on the outcomes achieved in
an iterative way.
CPD would entail a philosophical shift in favor of
developing data capture and reporting mechanisms for
all competencies – not just those pertaining directly to
clinical expertise. For example, in Canada this would
require attention to all the CanMEDS 2015 Roles,
including not only Medical Expert, but also
Communicator, Collaborator, Leader, Health Advocate,
Scholar, and Professional. In the United Kingdom,
attention would focus on safety and quality, communi-
cation, partnership and teamwork, and maintenance of
trust. Although these attributes are currently espoused
in the various systems, there are varying degrees to
which they need to be demonstrated or reported to
professional and/or regulatory organizations.
2. Progression of expertise in practice. A delineation of
competencies, milestones, and EPAs for practice will be
needed. Although such delineations exist to some
extent now, most of the focus has been on entry to
practice rather than the lifespan of independent prac-
tice. Work is required within each specialty to delineate
EPAs and milestones within the context of practice.
3. Alignment with scope of practice. A CBME approach
would require that learning experiences be undertaken
in alignment with a physician’s scope of practice to
demonstrate outcomes of learning for practice, includ-
ing areas of competence.
4. Practice-based learning support. Enhanced learning
in practice would benefit from assistance such as
coaching and peer feedback to develop and imple-
ment plans of learning anchored in the physician’s
work.
5. Multi-level assessment systems. The creation of
assessment systems of and for learning will be needed
(see Lockyer et al. 2017, in this issue), encompassing
the performance of individuals, groups, or teams in the
 620 J. LOCKYER ET AL.
workplace as well as assessments of knowledge and
skills demonstrated in various contexts, such as online,
in the laboratory, and in classroom settings.
6. Reporting infrastructure. The creation of reporting
structures such as e-portfolio systems would enable
physicians to track and to demonstrate how they are
planning and maintaining their competence.
Transforming learning activities for next-
generation CPD
In designing learning activities for practising physicians,
we should not forget the evidence that traditional con-
tinuing medical education can improve professional prac-
tice and health care outcomes for patients (Forsetlund
et al. 2009). Educational meetings or group learning activ-
ities that use mixed interactive and didactic formats and
focus on outcomes that are perceived as serious may
increase the effectiveness of educational meetings on
changing physician behaviors or improving patient out-
comes. However, educational meetings alone are not likely
Downloaded by [74.12.189.243] at 18:15 19 September 2017
to be effective for changing complex behaviors
(Forsetlund et al. 2009). A recent examination of Canadian
accredited CPD activities showed they were not designed
to promote clinical behavior change because their focus
was on remembering and understanding information
instead of preparing physicians to put knowledge into
practice by analyzing information, evaluating new evi-
dence, and planning learning activities that lead to behav-
ior change (Legare et al. 2015). Designing CPD that is
interactive, practice-based, and longitudinal is likely to
yield better outcomes (Marinopoulos et al. 2007;
Forsetlund et al. 2009).
Physicians and educators are adopting and promoting
new approaches to learning, recognizing that changing
behavior is a complex process affected by physician charac-
teristics and systemic factors (Greenhalgh et al. 2004).
Recognizing this, some new approaches are being pro-
moted on the merits of a theoretical or research base that
suggests their potential to make a positive impact on out-
comes. Examples of innovative approaches include the
following:
 Learning activities in small groups that meet regularly.
For family physicians, practice-based small groups,
guided by a facilitator who assists the group to work
through practice-based cases, drawing upon evidence-
based guidelines and other materials, have demon-
strated their effectiveness (Herbert et al. 2004; Armson &
Wakefield 2013).
 Rounds and educational programs focused on a specific
group of physicians who work together. Rounds may be
based on actual clinical situations or on performance
data from chart reviews, administrative databases and
morbidity and mortality reviews; the objective is to
facilitate physician discussion of their own (personal) or
group data along with an examination of systems that
may need improvement. In this regard, tumor boards
have demonstrated their effectiveness (Lamb et al.
2013). Similarly, interprofessional learning activities that
bring the team together to discuss solutions and collab-
orative care have been shown to be effective in some
circumstances (Reeves et al. 2013).
 Simulation activities that enable the physician to prac-
tise procedures and skills with task trainers, mannequins,
and standardized patients, as well as participation in
team-based activities with feedback (Boet et al. 2014;
McGaghie et al. 2014).
 Informed self-assessment and self-directed learning
strategies that enable learners to scan their environment
for new ideas, evidence that is being integrated into
practice guidelines, and practices that have been dis-
carded as being ineffective. This may also include asking
questions and seeking answers on a more routine basis
as well as using e-Portfolios to record educational activ-
ities and questions, reflect on practice, and demonstrate
competence (Lockyer et al. 2011; Gordon & Campbell
2013).
Enhancing assessment activities
Assessment in the CPD context is changing, as evidence of
assessment is increasingly required in revalidation and
maintenance-of-competence frameworks (Shaw & Armitage
2005; Shaw et al. 2009; Sklar 2016). This trend aligns with
the CBME approach to assessment, which can be character-
ized as ongoing assessment of learning and for learning,
making frequent use of multiple methods to assess a range
of competencies (see Harris et al. 2017 and Lockyer et al.
2017, in this issue). Current work on the development of
assessment tools for CBME recognizes that assessment tools
based on psychometric principles are inadequate in this
environment (Hodges 2013; Whitehead et al. 2015). It is
also recognized that new tools must demonstrate evidence
of validity and reliability as established through logical and
empirical approaches. Feasibility, acceptability, and equiva-
lence with other assessment tools are also important.
Lastly, assessment tools should have an educational impact
and a catalytic effect whereby the assessment motivates
those who participate to prepare in a fashion that has edu-
cational benefit and the outcomes of the assessment spur
future learning (Norcini et al. 2011).
Novel approaches are also needed to ensure the suc-
cessful implementation of new assessment programs, par-
ticularly those intended to be delivered in the workplace
and integrated within work flow. Funding systems and
strategies will require buy-in from employers and govern-
ments; physician dues are unlikely to be enough to support
the development and implementation of such high-quality
programs. The programs cannot be an excessive burden on
the physician or a hurdle that is too difficult to cross. As
noted earlier, learning activities such as simulation are both
an educational intervention and an assessment approach
that provides feedback which physicians can use to
improve their practice. Other approaches that can provide
physicians with ongoing assessment and feedback may
include the following:
 Audit and feedback: Audit and feedback generally lead
to small but potentially important improvements in pro-
fessional practice. Their effectiveness seems to depend
on baseline performance and how the feedback is pro-
vided (Ivers et al. 2012, 2014). Although audits can
involve fairly simple reviews of medical records and the
comparison of recorded information to standards, more
complex interventions are also being developed.

For example, in North America the Aligning and
Educating for Quality Initiative program of the
Association of American Medical Colleges supports the
alignment of data sources (e.g. financial, quality, referral,
utilization, and patient-centered outcomes research)
with educational programing to improve the quality of
patient care (Davis et al. 2013).
 Multi-source feedback: This type of assessment provides
physicians with data about observable behaviors from
the perspective of patients, medical colleagues or peers,
and coworkers (e.g. nurses, pharmacists, and technical
staff). This type of assessment program has demon-
strated evidence of validity and is supported by small to
large effect-size differences with respect to physician
and surgeon performance across different clinical and
nonclinical domain measures (Al Ansari et al. 2014).
 Direct observation of procedures or clinical care with
feedback: This approach is used by surgeons developing
advanced skills, such that a surgeon already trained in
the procedure joins them in the operating room or
remotely to provide feedback (Birch et al. 2007).
Downloaded by [74.12.189.243] at 18:15 19 September 2017
Implications and conclusions
Moving toward a CBME approach within CPD systems will
require a cultural shift and resources to effect the changes
needed at multiple levels. Individual practitioners should
remain responsible and accountable for their lifelong learn-
ing and need to be more systematic in looking at clinical
performance data, identifying performance improvement
potential, and aligning learning activities to address compe-
tence in these areas. At the level of the health care facility
and health care system, attention will have to be paid to the
development and training of personnel who can work with
physician groups to provide data on a regular basis that is
related to meaningful clinical work. CPD provider organiza-
tions need to see themselves as facilitators of learning and
to be able to navigate and negotiate resources. Medical
regulatory authorities responsible for licensing physicians, as
well as colleges that provide credentials through mainten-
ance of certification systems, will need to change their cur-
rent expectations with regard to how learning outcomes are
reported and how they demonstrate competence.
Disclosure statement
The authors report no conflicts of interest. The authors alone are
responsible for the content and writing of the article. Resources and
secretariat support for this project were provided by the Royal College
of Physicians and Surgeons of Canada.
Notes on contributors
Jocelyn Lockyer, PhD, is Senior Associate Dean of Education, and
Professor, Department of Community Health Sciences, Cumming
School of Medicine, University of Calgary, Calgary, Canada.
Ford Bursey, MD, is Professor of Medicine (Gastroenterology), Faculty
of Medicine, Memorial University of Newfoundland, St John’s, Canada.
Denyse Richardson, MD, is Associate Professor in the Department of
Medicine, University of Toronto, and a Clinician Educator with the
Royal College of Physicians and Surgeons of Canada, Canada.
Jason R. Frank, MD, is Director, Specialty Education, Strategy and
Standards in the Office of Specialty Education at the Royal College of
MEDICAL TEACHER 621
Physicians and Surgeons of Canada, and the Director of Educational
Research & Development in the Department of Emergency Medicine,
University of Ottawa, Canada.
Linda Snell, MD, is Professor of Medicine and Core Faculty member,
Centre for Medical and Department of General Internal Medicine,
McGill University, Canada, and Senior Clinician Educator, Royal College
of Physicians and Surgeons of Canada, Canada.
Craig Campbell, MD, is Director, Continuing Professional Development,
Royal College of Physicians and Surgeons of Canada and Associate
Professor, Department of Medicine, University of Ottawa, Canada.
References
Al Ansari A, Donnon T, Al Khalifa K, Darwish A, Violato C. 2014. The
construct and criterion validity of the multi-source feedback process
to assess physician performance: a meta-analysis. Adv Med Educ
Pract. 5:39–51.
Armson H, Wakefield J. 2013. Expanding the horizons of practice-based
small-group learning: what are we learning? Educ Prim Care.
24:153–155.
Birch DW, Asiri AH, de Gara CJ. 2007. The impact of a formal mentor-
ing program for minimally invasive surgery on surgeon practice and
patient outcomes. Am J Surg. 193:589–591.
Boet S, Bould MD, Fung L, Qosa H, Perrier L, Tavares W, Reeves S,
Tricco AC. 2014. Transfer of learning and patient outcome in simu-
lated crisis resource management: a systematic review. Can J
Anaesth. 61:571–582.
Carraccio C, Englander R, Van Melle E, ten Cate O, Lockyer J, Chan MK,
Frank JR, Snell LS. International Competency-Based Medical
Education Collaborators. 2016. Advancing competency-based med-
ical education: a charter for clinician-educators. Acad Med.
91:645–649.
Choudhry NK, Fletcher RH, Soumerai SB. 2005. Systematic review: the
relationship between clinical experience and quality of health care.
Ann Intern Med. 142:260–273.
Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier
L. 2006. Accuracy of physician self-assessment compared with
observed measures of competence: a systematic review. JAMA.
296:1094–1102.
Davis NL, Davis DA, Johnson NM, Grichnik KL, Headrick LA, Pingleton
SK, Bower E, Gibbs R. 2013. Aligning academic continuing medical
education with quality improvement: a model for the 21st century.
Acad Med. 88:1437–1441.
Drotar D. 2009. Physician behavior in the care of pediatric chronic ill-
ness: association with health outcomes and treatment adherence.
J Dev Behav Pediatr. 30:246–254.
Englander R, Carraccio C. 2014. From theory to practice: making
entrustable professional activities come to life in the context of
milestones. Acad Med. 89:1321–1323.
Englander R, Frank JR, Carraccio C, Sherbino J, Ross S, Snell L; ICBME
Collaborators. 2017. Toward a shared language for competency-
based medical education. Med Teach. 39:582–587.
Erasmus V, Daha TJ, Brug H, Richardus JH, Behrendt MD, Vos MC, van
Beeck EF. 2010. Systematic review of studies on compliance with
hand hygiene guidelines in hospital care. Infect Control Hosp
Epidemiol. 31:283–294.
Forsetlund L, Bjørndal A, Rashidian A, Jamtvedt G, O’Brien MA, Wolf F,
Davis D, Odgaard-Jensen J, Oxman AD. 2009. Continuing education
meetings and workshops: effects on professional practice and
health care outcomes. Cochrane Database Syst Rev. CD003030.
Frank JR, Mungroo R, Ahmad Y, Wang M, De Rossi S, Horsley T. 2010a.
Toward a definition of competency-based education in medicine: a
systematic review of published definitions. Med Teach. 32:631–637.
Frank JR, Snell LS, ten Cate O, Holmboe ES, Carraccio C, Swing SR,
Harris P, Glasgow NJ, Campbell C, Dath D, et al. 2010b.
Competency-based medical education: theory to practice. Med
Teach. 32:638–645.
Gordon JA, Campbell CM. 2013. The role of ePortfolios in supporting
continuing professional development in practice. Med Teach.
35:287–294.
Goulet F, Hudon E, Gagnon R, Gauvin E, Lemire F, Arsenault I. 2013.
Effects of continuing professional development on clinical
 622 J. LOCKYER ET AL.
performance: results of a study involving family practitioners in
Quebec. Can Fam Physician. 59:518–525.
Grace ES, Wenghofer EF, Korinek EJ. 2014. Predictors of physician per-
formance on competence assessment: findings from CPEP, the
Center for Personalized Education for Physicians. Acad Med.
89:912–919.
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. 2004.
Diffusion of innovations in service organizations: systematic review
and recommendations. Milbank Q.. 82:581–629.
Harris P, Bhanji F, Topps M, Ross S, Lieberman S, Frank JR, Snell L,
Sherbino J; ICBME Collaborators. 2017. Evolving concepts of assess-
ment in a competency-based world. Med Teach. 39:603–608.
Herbert CP, Wright JM, Maclure M, Wakefield J, Dormuth C, Brett-
MacLean P, Legare J, Premi J. 2004. Better Prescribing Project: a
randomized controlled trial of the impact of case-based educational
modules and personal prescribing feedback on prescribing for
hypertension in primary care. Fam Pract. 21:575–581.
Hodges B. 2013. Assessment in the post-psychometric era: learning to
love the subjective and collective. Med Teach. 35:564–568.
Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French
SD, O’Brien MA, Johansen M, Oxman AD. 2012. Audit and feedback:
effects on professional practice and healthcare outcomes. Cochrane
Database Syst Rev. CD000259.
Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O’Brien MA, French
SD, Young J, Odgaard-Jensen J. 2014. Growing literature, stagnant
Downloaded by [74.12.189.243] at 18:15 19 September 2017
science? Systematic review, meta-regression and cumulative analysis
of audit and feedback interventions in health care. J Gen Intern
Med. 29:1534–1541.
Lamb BW, Green JS, Benn J, Brown KF, Vincent CA, Sevdalis N. 2013.
Improving decision making in multidisciplinary tumor boards: pro-
spective longitudinal evaluation of a multicomponent intervention
for 1,421 patients. J Am Coll Surg. 217:412–420.
Legare F, Freitas A, Thompson-Leduc P, Borduas F, Luconi F, Boucher A,
Witteman, Jacques A. 2015. The majority of accredited continuing
professional development activities do not target clinical behavior
change. Acad Med. 90:197–202.
Lockyer J, Armson H, Chesluk B, Dornan T, Holmboe E, Loney E, Mann K,
Sargeant J. 2011. Feedback data sources that inform physician self-
assessment. Med Teach. 33:e113e20.
Lockyer J, Silver I, Oswald A, Bullock G, Campbell C, Frank JR, Taber S,
Wilson I, Harris KA. 2014. The continuum of medical education. In:
KA Harris, JR Frank, editors. Competence by design: re-shaping
Canadian medical education [Internet]. Ottawa: Royal College of
Physicians and Surgeons of Canada; [cited 2015 Dec 31]. Available
from: www.royalcollege.ca/portal/page/portal/rc/common/documents/
educational_initiatives/rc_competency-by-design_ebook_e.pdf.
View publication stats
Lockyer J, Horsley T, Zeiter J, Campbell C. 2015. Role for assessment in
maintenance of certification: physician perceptions of assessment.
J Contin Educ Health Prof. 35:11–17.
Lockyer J, Carraccio C, Chan MK, Hart D, Smee S, Touchie C, Holmboe
ES, Frank JR; ICBME Collaborators. 2017. Core principles of
assessment in competency-based medical education. Med Teach.
39:609–616.
Marinopoulos SS, Dorman T, Ratanawongsa N, Wilson LM, Ashar BH,
Magaziner JL, Miller RG, Thomas PA, Pokopowicz GP, Qayyum R,
et al. 2007. Effectiveness of continuing medical education. Evid Rep
Technol Assess (Full Rep). 149:1–69.
McGaghie WC, Issenberg SB, Barsuk JH, Wayne DB. 2014. A critical
review of simulation-based mastery learning with translational out-
comes. Med Educ. 48:375–385.
Nieuwlaat R, Schwalm JD, Khatib R, Yusuf S. 2013. Why are we failing
to implement effective therapies in cardiovascular disease? Eur
Heart J. 34:1262–1269.
Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R,
Galbraith R, Hays R, Kent A, Perrott V, et al. 2011. Criteria for good
assessment: consensus statement and recommendations from the
Ottawa 2010 Conference. Med Teach. 33:206–214.
Reeves S, Perrier L, Goldman J, Freeth D, Zwarenstein M. 2013.
Interprofessional education: effects on professional practice and
healthcare outcomes (update). Cochrane Database Syst Rev.
CD002213.
Sargeant J, Armson H, Chesluk B, Dornan T, Eva K, Holmboe E, Lockyer
J, Loney E, Mann K, van der Vleuten C. 2010. The processes and
dimensions of informed self-assessment: a conceptual model. Acad
Med. 85:1212–1220.
Shaw K, Armitage M. 2005. Supporting revalidation: methods and evi-
dence. Clin Med (Lond). 5:460–464.
Shaw K, Cassel CK, Black C, Levinson W. 2009. Shared medical regula-
tion in a time of increasing calls for accountability and transparency:
comparison of recertification in the United States, Canada, and the
United Kingdom. JAMA. 302:2008–2014.
Sklar DP. 2016. What happens after medical school? Current controver-
sies about licensure, maintenance of certification, and continuing
professional development. Acad Med. 91:1–3.
Wenghofer EF, Marlow B, Campbell C, Carter L, Kam S, McCauley W,
Hill L. 2014. The relationship between physician participation in
continuing professional development programs and physician in-
practice peer assessments. Acad Med. 89:920–927.
Whitehead CR, Kuper A, Hodges B, Ellaway R. 2015. Conceptual and
practical challenges in the assessment of physician competencies.
Med Teach. 37:245–251.