Evaluation of Learners

Effective Techniques and Strategies

Jennie Hsu-Lumetta, M.D.

March 16, 2005
1
09/15/21
“Clinical Learning is
Experience Examined…”

Ende and Davidoff, 1992

2
09/15/21
Goals and Objectives
1. Define the purpose of evaluations.
2. Identify the steps to providing effective
evaluations.
3. List some potential barriers/errors to
providing effective evaluation .

3
09/15/21
What Will Be Covered. . .
1. Definition, Purposes, Types of Evaluation
2. Pre-Evaluation Activities
3. Completing the Evaluation
4. Writing Good Comments
5. Discussing Evaluation
6. RIME Model for Clinical Evaluation

4
09/15/21
 Definition of Evaluation

To determine the value or worth of a

Program,
Product,
Performance
Process

5
09/15/21
Evaluation of Learners

The process by which the teacher

assesses the learners’
knowledge, skills, and attitudes,
based on criteria related to
educational goals.

6
09/15/21
 Why Evaluate?

Past: Assess performance

Present: Improve performance
Future: Identify future needs

7
09/15/21
Uses of Medical Student Evaluations

Use Purpose Audience

Past Assessment Clerkship

Director

Present Improvement Student

Future Dean’s Letter Dean

8
09/15/21
Uses of Resident Evaluations
Use PurposeAudience

Past Assessment Program

Director
Present Improvement Resident

Future Program Fellowship,

Director’s Letter Employers

9
09/15/21
What to Evaluate?

Attitude
Skills
Knowledge

10
09/15/21
ASK’s
Attitude, Skills, Knowledge
Competency includes the following:
 Attitude: (as observed via behaviors)
Prepared/Punctuality/Proactive
Follow-up with patient/educational issues
 Skills – context specific
 Knowledge
– Has the information
– Able to analyze and synthesize information
– Able to apply it effectively and efficiently

11
09/15/21
ACGME Competencies
1. Patient Care
2. Medical Knowledge
3. Practice-Based Learning and
Improvement
4. Interpersonal and Communication
Skills
5. Professionalism
6. Systems-Based Practice
12
09/15/21
ACGME Competencies
Patient Care that is compassionate, appropriate, and
effective for the treatment of health problems and the
promotion of health

Medical Knowledge about established and evolving

biomedical, clinical, and cognate (e.g. epidemiological
and social-behavioral) sciences and the application of this
knowledge to patient care

13
09/15/21
ACGME Competencies
Practice-Based Learning and Improvement that
involves investigation and evaluation of their own
patient care, appraisal and assimilation of scientific
evidence, and improvements in patient care

Interpersonal and Communication Skills that result in

effective information exchange and teaming with
patients, their families, and other health professionals

14
09/15/21
ACGME Competencies
Professionalism, as manifested through a commitment to
carrying out professional responsibilities, adherence to
ethical principles, and sensitivity to a diverse patient
population

Systems-Based Practice, as manifested by actions that

demonstrate an awareness of and responsiveness to the
larger context and system of health care and the ability to
effectively call on system resources to provide care that is
of optimal value

15
09/15/21
ACGME - Evaluation of Residents
 use of dependable measures to assess 6
areas of competency
 mechanisms for providing regular and
timely performance feedback to residents
 a process involving use of assessment
results to achieve progressive
improvements in residents' competence
and performance
16
09/15/21
 Types of Evaluation

1. Formative Evaluation
2. Summative Evaluation

17
09/15/21
Types of Evaluation
1. Formative Evaluation
 Assess a learner’s progress toward
educational goals
 Throughout the teaching process
 Basis for feedback for the learners and to
make teaching adjustments as needed to
learner to reach learning goals

18
09/15/21
Types of Evaluation
1. Summative Evaluation
 Final assessment of a learner
 End of teaching/learning experience
 Used to judge a learner’s mastery of the
educational goals

19
09/15/21
How to Evaluate?

Assessment/Evaluation
 Formative (ongoing)

Feedback

Evaluation
 Summative (end of rotation)

20
09/15/21
 Elements of
Effective Assessment
 Evaluation Prerequisites
 Making Good Comments
 Aligning Comments with Scaled Questions
 Discussing the Evaluation with the Learner

Source: David Rubin, MD and James Woodruff, MD. Teaching

Skills for the Medical Educator Workshop. University of
Chicago, March 23, 2002 21
09/15/21
 Effective Assessment 1:
Evaluation Prerequisites

 Understand expectations of the learner

 Understand the evaluation criteria
 Understand the survey instrument
 Have appropriate expectations
 Provide on-going feedback
 Provide opportunities for observation

22
09/15/21
Key components of Evaluation

1. Observation of the Learners

2. Questioning

23
09/15/21
Key components of Evaluation :
Observation of the Learners

The teacher can observe learner

performance for direct
evaluation of knowledge, skills,
and attitudes.

24
09/15/21
Key components of Evaluation :
The One minute Observation

 Explain the purpose of the observation to the

learner
 Explain how the observation will occur
 Select one skill for observation
 Inform the patient of what will take place
 Observe for a brief period without interrupting
 Leave the room and give immediate feedback
 Use the information gain to plan your teaching
 Repeat to observe learner’s skills over time

25
09/15/21
Key components of Evaluation :
Questioning of the Learners

1. Types of Questions
2. Levels of Questions

26
09/15/21
Key components of Evaluation :
Types of Questions

1. Open-ended
 What?
 How?
 Why?
2. Closed-ended
 Do?
 Can?
 Have?
27
09/15/21
Key components of Evaluation :
Level of Questions

1. Recall
2. Analysis/Synthesis
3. Application

28
09/15/21
 Effective Assessment 2:
Completing the Form
 Write narrative comments first
 Comments are most important!!
 Grades are a “blunt instrument”
Dr. Rubin, U of Chicago Dept. of Medicine

 Give specific examples

 Keep comments to focused behaviors
 Connect comments to ratings
 Check for internal consistency
29
09/15/21
 Sources of Comments
 Direct observation of learner
 Written records
 Responses to questions
 Homework assignments
 Interactions with healthcare team, patients
 Learner self-evaluation
 Feedback/Midpoint evaluation
– Interventions with the learner

– Learner’s response to interventions

30
09/15/21
 Effective Assessment 3:
Writing Effective Comments

 Strengths and weaknesses

 Areas of concern
 Suggestions for improvement

 Ensure internal consistency

 Focus on behaviors not personalities
31
09/15/21
 Effective Assessment 4:
Discuss with Learner
 Set a time
 Allow enough time for discussion
 Have the learner perform a self-assessment
 Explain your rationale
 Emphasize changes in behavior and performance that leads to
desired competency
 Suggest areas for future improvement
 Give evaluation in oral and written forms

*Adding feedback element increases the

usefulness of the evaluation and avoids surprises

32
09/15/21
Avoid rating errors:
 The “halo/horn” effect:
– based on reports of prior performance
 Restriction range:
– using the same rating for all components of
the evaluation
 Rating nonperformance attributes:
– using behavior attributes when rating
nonbehavioral qualities
– Evaluate performance ONLY

33
09/15/21
RIME Model for Clinical Evaluation
 Developed by Dr. Louis Pangaro
 Model for student evaluation that was
credible and reliable
 Student performance -> Progression of
4 developmental steps
 Research shows that it is predictive of
future performance

34
09/15/21
RIME Model for Clinical Evaluation
4 Stages of Clinical Development:

Reporter
Interpreter
Manager
Educator

35
09/15/21
RIME Model for Clinical Evaluation
 Reporter
– collect and report data (by the M3 year)
 Interpreter
– interpret history, PE findings and lab results, create &
prioritize a problem list and differential diagnosis (by end
of M3 year)
 Manager
– select the most appropriate diagnostic and treatment
options (by 2nd & 3rd residents)
 Educator
– identify and address knowledge gaps (senior
residents)

36
09/15/21
Proficient
Reporter Interpreter
 Interview & History
•Interprets basic test data
 Physical examinations
•Integrates medical
 Demonstrates adequate
medical knowledge knowledge with findings
 Generates adequate •Generates several reasonable
problem list
explanations
 Demonstrates respect
for patients •Can discuss issues & process
 Consistently reliable & of informed consent
honest
37
09/15/21
Advanced
Manager
 Develop diagnostic & therapeutic plans
 Makes decisions using benefit/risk analysis
 Uses evidence-based approach
 Performs as a reliable member of the team
 Manages relationships with patients

38
09/15/21
Outstanding
Educator
 Demonstrates leadership
 Demonstrates self-directed learning
 Exhibits critical reading skills
 Demonstrates teaching skills

39
09/15/21
Summary for Effective Evaluators:
 Understand learner expectations
 Understand the evaluation instrument:
– E-value, RIME model
 Include thoughtful comments
 Connect comments to rating scales
 Have multiple opportunities for assessing
performance
 Evaluate performance ONLY
 Discuss evaluation with the learner

40
09/15/21
Special thanks:
 Nancy F. Barrett
 Carol Packard

41
09/15/21
Case
You sit down with your student on her last day of the
rotation although you’ve never been comfortable
giving evaluations. Knowing evaluations are
supposed to be balanced, you try to list several
positive things and then list problems. When you
mention that the student’s cardiac patient who spent
an extra day in the hospital because she didn’t track
down a stress test result in time on Friday before the
cardiology department closed, she angrily bursts out
how difficult it is on a team where no one will give
her any help. She also says that you haven’t spent as
much time reviewing write-ups with her as other
residents.
42
09/15/21