Assessing Core Competencies Us in Go Sce

Session SES76
Assessing Core Competencies

with an Objective Structured
Clinical Examination
Matthew W. Short, MD, FAAFP
Director, Transitional Year Program
John A. Edwards, MD, MPH, FAAFP
Director, Intern Training
Madigan Army Medical Center
Learning Objectives
Review the goal of the ACGME Outcome Project
Understand the role of an OSCE as an
assessment tool for core competency evaluation
Learn how to design, organize, and perform an
OSCE within a training program
Discuss how an OSCE can provide valuable
outcome data to improve overall program
performance
ACGME Outcome Project
ACGME
Outcome Project
Long-term initiative to emphasize outcome
assessment in the accreditation process
Identify learning objectives related to the core
competencies
Use dependable and objective methods to
assess resident attainment of objectives
Use outcome data to facilitate continuous
improvement of resident and program
performance
The ACGME Outcome Project: An Introduction.
Accreditation
Council for Graduate Medical Education. Rev. 2005
Outcome Project
History
Identified 6 core competencies in Feb 99
Created an Assessment Toolbox of
recommended tools to assess outcomes
Currently in Phase 3 of the outcome
project (7/2006 6/2011)
As part of the accreditation process, programs
must provide evidence that they are making
data-driven improvements.
Accreditation
Programs are expected to phase-in
assessment tools that provide useful and
increasingly valid, reliable evidence that
residents achieve competency-based
educational objectives.
Accreditation
Programs are expected to show evidence
of how they use educational outcomes
data to improve individual resident and
overall program performance.
Accreditation
The Role of an OSCE
OSCE
Definition
Objective Structured Clinical Examination
First described by Harden in the British
Medical J ournal in 1975
Direct observations in clinical simulations
Multiple stations performing various tasks
USMLE step 2 Clinical Skills Examination
Turner JL, Dankoski ME. Objective structured clinical exams: a
critical review. Fam Med. 2008 Sep;40(8):574-8.
OSCE
Definition
Used in med schools and residency to
assess:
History and physical exam skills
Patient communication
Breadth of knowledge, differential diagnosis
Treatment plan and clinical judgment
Ability to document findings
ACGME and ABMS Joint Initiative. Toolbox of Assessment Methods.
Objective Structured Clinical Examination. Version 1.0, Summer 2000.
OSCE
Evaluation Tool
ACGME considers it a useful tool to
evaluate ACGME core competencies
Toolbox of assessment methods
Use of OSCE, SP, and simulations/models
Rate as most desirable or next best method in
assessing most required skills within the 6
ACGME core competencies
ACGME and ABMS Joint Initiative. Toolbox of Assessment Methods.
Objective Structured Clinical Examination. Version 1.0, Summer 2000.
ACGME Competencies:
Suggested Best Methods for Evaluation
Toolbox of Assessment Methods. ACGME/ABMS Joint initiative. Version 1.1. September 2000
Ratings are 1 = the most desirable; 2 = the next best method; and, 3 = a potentially applicable method.
ACGME Competencies:
ACGME Competencies:
The Madigan OSCE
Objective 1
Objective:
Create and use an OSCE to evaluate interns
on the six ACGME core competencies before
and after internship to provide educational
outcomes data that may improve individual
resident and overall program performance.
Objective:
To determine if the intern OSCE prior to
internship is more predictive than prior
academic performance in identifying potential
deficiencies in ACGME core competencies
throughout the intern year.
Objective 2
Methods
OSCE 2006-07
Methods
Eight 12 minute stations chosen by program directors
Stations peer reviewed by at least four physicians
58 interns from 10 medical specialties tested
OSCE before and after internship 2006-2007
Collectively evaluated the six ACGME core
competencies
Use of staff and SP volunteer graders
Use of simulation:
Standardized patients
Human patient simulators
Computer literature search
OSCE Stations
Station 1: Death notification- Stand pt (SP)
Interpersonal & communication skills (ICS)
Station 2: Abdominal pain- SP
Patient care (PC), Medical knowledge (MK), ICS,
Professionalism (Pro), Systems-based practice
(SBP)
Station 3: Suture skills- Simulation (Sim)
PC, SBP
Station 4: Transfusion consent- SP
PC, MK, ICS, Pro
OSCE Stations
Station 5: Wellness visit history- SP
PC, ICS, Pro, SBP
Station 6: Altered mental status- Sim
PC, MK, ICS, Pro, SBP
Station 7: Literature search- Computer
Practice-based learning & improvement
Station 8: Chest pain/cardiac arrest- Sim
PC, MK, SBP
Methods
OSCE 2006-07
Evaluation form examples
INTERPERSONAL AND COMMUNICATION SKILLS
D
i
d

N
o
t

P
e
r
f
o
r
m
N
e
e
d
s

I
m
p
r
o
v
e
m
e
n
t
A
v
e
r
a
g
e
A
b
o
v
e

A
v
e
r
a
g
e
G
o
o
d
E
x
c
e
l
l
e
n
t
Greets patient respectfully and introduces self
0 pts 1 pts 2 pts 3 pts 4 pts 5 pts
PROFESSIONALISM
D
N
P
N
I
AA
A
GE
Presents self professionally through posture and patient interaction
SYSTEMS-BASED PRACTICE
D
N
P
N
I
AA
A
GE
Orders pertinent labs based on differential diagnosis
PATIENT CARE
N
o
Y
e
s
Checks responsiveness
0 pts 5 pts
MEDICAL KNOWLEDGE
N
o
Y
e
s
Differential diagnosis- appendicitis
0 pts 5 pts
OSCE Organization
8 interns per two hour block for 8 stations
12 minute stations with 3 minutes to rotate
32 residents per eight hour day
Minimum of 8 staff and 4 patient volunteers
Interns arrive 30min early for instructions
Standardize stations (instructions for staff)
Adhere to time schedule, whistle and timer
Resident feedback
Evaluation forms, grading, and laptops
Station 1:
Death Notification
Simulation type: Standardized family member
You are the intern on call in the ICU and called to
the ED to evaluate a patient being coded
79yo female on ASA and Coumadin arrived to the
ED comatose with a large volume intracranial
hemorrhage and herniation. Coded for 15 min and
died.
Patients daughter arrives after code and you are
told to tell her of her mothers death.
Daughter is in a private room, body behind curtain.
D
i
d

N
o
t

P
e
r
f
o
r
m
N
e
e
d
s

I
m
p
r
o
v
e
m
e
n
t
A
v
e
r
a
g
e
A
b
o
v
e

A
v
e
r
a
g
e
G
o
o
d
E
x
c
e
l
l
e
n
t
Greets family member respectfully
Appropriately introduces self and title
Appropriately offers family member a seat
Maintains calm and concerned affect
Maintained appropriate eye contact
Used appropriate physical contact
Used appropriate pace, volume & tone of speech
Appeared comfortable with silence
Uses appropriate body language (face family, sit close)
Avoids medical jargon and vague descriptions
Expresses sympathy and shows compassion
Allowed the bereaved to express emotion
Gives family time to talk and ask questions
Offers family member a tissue
Describes status of patient on arrival to ED to present
Asks family what they understand of the events
Able to explain sequence of events in layman's terms
Used appropriate language to tell family patient was dead
Answers all questions in a straight forward manner
Asks the family if they wish to view body
Tells family what they will see when viewing body
Tells family okay to touch / hold body
Asks if they want to be alone with body
Prepares the body for family viewing
Offers support services (chaplain, DSW, counselor)
Communication skills
Content issues
Follow-up
Station 2:
Abdominal Pain
Simulation type: Standardized patient
17yo female with RLQ abdominal pain for 8
hrs (Standardized patient)
Perform history and physical
List differential and initial orders on sheet
Given results for UA, CBC, and Quant HCG
after 10 min, then outline additional
management plan
Patient evaluates intern PRO and ICS
PATIENT CARE
N
o
Y
e
s
MEDICAL KNOWLEDGE
N
o
Y
e
s
D
i
d

N
o
t

P
e
r
f
o
r
m
N
e
e
d
s

I
m
p
r
o
v
e
m
e
n
t
A
v
e
r
a
g
e
A
b
o
v
e

A
v
e
r
a
g
e
G
o
o
d
E
x
c
e
l
l
e
n
t
Identifies patients chief complaint as abdominal
pain Ectopic pregnancy Seats patient and sets stage appropriately
Asks patient age Appendicitis
Asks about duration of pain Tubo-ovarian abscess
Asks about location of pain Pelvic inflammatory disease or salphingitis Smiles and uses eye contact when greeting
Asks about quality of pain Ovarian Torsion Does not appear impatient during interaction
Asks about timing of pain Crohn's disease or other ileitis or IBD Is non-confrontational during visit
Asks about radiation of pain Nephrolithiasis Uses an open, inviting posture
Asks about severity of pain using a pain scale Cystitis Interviews patient at eye level and is non-threatening
Asks about exacerbating factors Adnexal cyst Puts the patient at ease with a friendly demeanor
Asks about alleviating factors Endometritis Shows interest and concern throughout patients history
Endometriosis Assesses patient understanding
Fever or chills Solicits patient questions and addresses concerns
Nausea or vomiting Requests IV access Identifies purpose of visit & makes smooth transition into history
Anorexia or weight loss Gives IVF Initially uses open ended questions to allow patient to tell story
Bowel habits (diarrhea or constipation) Treats pain with appropriate IV medication Paraphrases patient story to show interest and understanding
Blood in stool (hematochezia or melena) Gives anti-nausea medication Uses patient history to guide differential and appropriate studies
Vaginal bleeding Quantitative HCG Does not ask the same questions throughout the visit
Dysuria CBC Does not ask questions about history already provided
CVA pain Urinalysis PROFESSIONALISM
D
N
P
N
I
AA
A
GE
Urine culture Addresses the patient in a respectful way
Current medications Genprobe
Tobacco use
Medication allergies Obtains blood type Does not talk down to or belittle the patient
Sexual history Obtains antibody screen Has fluent, understandable speech that flows
Pregnancy status Orders pelvic ultrasound Presents self as a competent physician through verbal communication
Menstural history Mentions giving Rhogam if Rh neg Presents self professionally through posture and patient interaction
Prior pregnancy history Mentions getting GYN consult if US shows ectopic Sensitive to sexual lifestyle of patient
Use of contraception SYSTEMS-BASED PRACTICE
D
N
P
N
I
AA
A
GE
Use of IUD Directs patient with closed ended questions to ensure time efficiency
STD history Effectively directs patients through the visit to ensure timeliness
Surgical history Limits visit to primary or immediate concerns
Cost effective care by ordering pertinent labs
Washes hands prior to patient exam Cost effective care by ordering pertinent diagnostic tests
Requests vital signs
Utilizes reasonable generic or inexpensive medications without compromising
care
Auscultates lungs Uses consultation services only when necessary
Auscultates heart
Inspects abdomen
Auscultates abdomen
Palpates abdomen
Checks for hepatosplenomegally
Checks for rebound
Checks Rovsings sign
Checks iliopsoas sign
Checks obturator sign
Requests results of a rectal examination
Requests results of a pelvic examination
Intern name:____________________________________
Is respectful to the patient throughout the visit
History of present illness
Asks about review of systems
Asks about past medical, surgical, family, social
Physical exam
Differential diagnosis includes the following
Initial patient management includes the following
Management after positive pregnancy test
Comments:
Shows empathy toward the patient concerns
D
i
d

N
o
t

P
e
r
f
o
r
m
N
e
e
d
s

I
m
p
r
o
v
e
m
e
n
t
A
v
e
r
a
g
e
A
b
o
v
e

A
v
e
r
a
g
e
G
o
o
d
E
x
c
e
l
l
e
n
t
Seats patient and sets stage appropriately
Shows empathy toward the patient concerns
Smiles and uses eye contact when greeting
Does not appear impatient during interaction
Is non-confrontational during visit
Uses an open, inviting posture
Interviews patient at eye level and is non-threatening
Puts the patient at ease with a friendly demeanor
Shows interest and concern throughout patients history
Assesses patient understanding
Solicits patient questions and addresses concerns
Identifies purpose of visit & makes smooth transition into history
Initially uses open ended questions to allow patient to tell story
Paraphrases patient story to show interest and understanding
Uses patient history to guide differential and appropriate studies
Does not ask the same questions throughout the visit
Does not ask questions about history already provided
PROFESSIONALISM
D
i
d

N
o
t

P
e
r
f
o
r
m
N
e
e
d
s

I
m
p
r
o
v
e
m
e
n
t
A
v
e
r
a
g
e
A
b
o
v
e

A
v
e
r
a
g
e
G
o
o
d
E
x
c
e
l
l
e
n
t
Addresses the patient in a respectful way
Is respectful to the patient throughout the visit
Does not talk down to or belittle the patient
Has fluent, understandable speech that flows
Presents self as a competent physician through verbal communication
Presents self professionally through posture and patient interaction
Sensitive to sexual lifestyle of patient
Station 3:
Suture Skills
Simulation type: Pigs feet
19yo soldier stabbed in right thigh
Choose correct suture, create sterile field,
describe anesthetic, irrigation, and
debridement.
Demonstrate deep and superficial wound
closure with instrument and hand ties.
Intern asked a series of questions during
station about anesthetic, and post-procedure
precautions and instructions.
N
o
Y
e
s
N
o
Y
e
s
N
o
Y
e
s
N
o
Y
e
s
N
o
Y
e
s
N
o
Y
e
s
Demonstrates proper technique for laceration repair
D
i
d

N
o
t

P
e
r
f
o
r
m
N
e
e
d
s

I
m
p
r
o
v
e
m
e
n
t
A
v
e
r
a
g
e
A
b
o
v
e

A
v
e
r
a
g
e
G
o
o
d
E
x
c
e
l
l
e
n
t
11. Uses proper technique to drive needle
N
o
Y
e
s
12. Proper technique for fascial closure with a simple stitch
13. Proper technique for an instrument tie
N
o
Y
e
s
14. Proper technique for simple interrupted stitch
15. Proper technique for hand tie
N
o
Y
e
s
16. Proper technique for interrupted vertical mattress stitch
17. Proper technique for interrupted horizontal mattress stitch
N
o
Y
e
s
N
o
Y
e
s
N
o
Y
e
s
N
o
Y
e
s
N
o
Y
e
s
N
o
Y
e
s
N
o
Y
e
s
20. Before, because the patient is in considerable pain (initial history)
What anesthetic would you use?
22. Lidocaine or Lidocaine with epinephrine
Questions asked during the procedure
When would you NOT use anesthetic with epinephrine?
23. Ear
24. Nose
18. Normal saline
How would you irrigate the wound?
19. One liter NS bottle with pressure
Would you give anesthetic before or after cleaning the wound? Why?
6. Uses all three betadine swabs
7. Uses swabs in circular motion from wound edges outwards
8. Applies sterile drape over laceration
What would you use to irrigate the wound?
9. Appropriately loads needle on needle driver
10. Uses tissue forceps for repair
PATIENT CARE
1. Selects proper suture for deep layer (vicryl on taper needle)
2. Selects proper sutuer for skin closure (ethilon or prolene on cutting needle)
Patient preparation
Select appropriate suture for deep layer and skin closure
3. Uses sterile technique to place suture on tray
4. Uses sterile technique to don gloves
5. Uses betadine for sterile prep 25. Digits
26. Penis
Why do you NOT use epinephrine in these areas?
27. Causes vasoconstriction and possible ischemia
What are the advantages of using anesthetic with epinephrine?
28. Decreases bleeding due to vasoconstriction
29. Keeps lidocaine in wound longer
Would you place a dressing on the wound after closure or leave open to
the air?
In how many days would you remove the skin sutures?
33. 7-14 days
In how many days would you remove the deep sutures?
30. Place dressing
When should the patient remove the dressing?
31. 2 days
When can the patient shower after this laceration repair?
21. Through wound, because this would decrease further pain
Would you inject anesthetic through the wound or through the skin
surface? Why?
36. Was sterile technique maintained throughout the entire procedure?
34. Never, absorbable sutures
What would you do if the patient returns to your clinic in 5 days with
evidence of a wound infection and pus coming from the wound?
35. Open wound
32. 2 days
Station 4:
Transfusion Consent
Patient needs transfusion of 2 units of
PRBCs for symptomatic anemia.
Intern must consent the patient.
Patient asks intern standardized questions.
Intern completes written quiz after consent.
Performance and oral/written answers to
questions are graded.
PATIENT CARE
D
id

N
o
t

P
e
r
f
o
r
m
N
e
e
d
s

I
m
p
r
o
v
e
m
e
n
t
A
v
e
r
a
g
e
A
b
o
v
e

A
v
e
r
a
g
e
G
o
o
d
E
x
c
e
lle
n
t
MEDICAL KNOWLEDGE
N
o
Y
e
s
D
id

N
o
t

P
e
r
f
o
r
m
N
e
e
d
s

I
m
p
r
o
v
e
m
e
n
t
A
v
e
r
a
g
e
A
b
o
v
e

A
v
e
r
a
g
e
G
o
o
d
E
x
c
e
lle
n
t
Appropriately explains the need for a blood transfusion to the patient Greets patient respectfully and introduces self
Appropriately explains the use of a consent form Seats patient and sets stage appropriately
Appropriately explains the transfusion procedure to the patient Stop the transfusion Shows empathy toward the patient concerns
Explains the benefits of this procedure to the patient Evaluate the patient Smiles and uses eye contact when greeting
Discusses infectious risks of transfusion to the patient Call the transfusion service Does not appear impatient during interaction
Discusses risk of transfusion reactions to patient Order new crossmatch Is non-confrontational during visit
States symptoms they may have from the transfusion Order DAT Uses an open, inviting posture
Discusses risk of death with transfusion Order blood cultures Consents patient at eye level and is non-threatening
Properly obtains signature on consent form Order urine for free hemoglobin Puts the patient at ease with a friendly demeanor
Requests appropriate transfusion modality based on clinical history and
laboratory data Shows interest and concern during interaction
MEDICAL KNOWLEDGE
N
o
Y
e
s
Acute hemolytic transfusion reaction Presents an understandable reason for transfusion to patient
Specifically discusses HIV as a risk Febrile non-hemolytic transfusion reaction Assesses patient understanding
Specifically discusses Hepatitis C as a risk Sepsis Solicits patient questions and addresses concerns
Obtains patient's blood type Transfusion related acute lung injury Good eye contact and listening skill
Crossmatches blood products Delayed hemolytic transfusion reaction Terminates visit appropriately
Chooses PRBC's rather than whole blood for transfusion PROFESSIONALISM
D
N
P
N
I
AA
A
GE
Thrombotic Thrombocytopenic Purpura (TTP) Addresses the patient as siror maamduring the visit
Heparin-induced Thrombocytopenia (HIT) Is respectful to the patient throughout the visit
1 in 2 million Idiopathic Thrombocytopenic Purpura (ITP)- relative Does not talk down to or belittle the patient
Normal platelet count Has fluent, understandable speech that flows
1 in 1 million
Presents self as a competent physician through verbal
communication
Transfusion-Related Acute Lung Injury- noncardiogenic
pulmonary edema caused by the passive transfer of donor HLA
antibodies. Occurs typically 2-6 hours after transfusion of any
plasma-containing blood product (typically FFP)
Presents self professionally through posture and patient
interaction
Appropriately answers yes Is courteous despite difficult patient encounter
To prevent graft vs. host disease (the ONLY indication)
Does not appear confused when asked medical questions by
patient
3-4%
Assesses patient understanding of management plan at end of
visit
Platelets
Tells the patient he or she does not know the answer to his
question
PRBCs CMV negative or CMV-save
Tells the patient that he or she will find the answer to their
questions prior to the transfusion
Irradiated
Questions from the Quiz
The patient you just consented received 100 cc of the transfusion and
the nurse pages you to tell you the patient's temperature went from
98.6F to 101.6F. What do you do?
What is the differential diagnosis for a fever after transfusion?
What are the contraindications to platelet transfusion?
Describe what " TRALI" is?
What are the indications for blood product irradiation?
What is the appropriate transfusion modality for a pancytopenic
patient with a history of acute lymphoblastic leukemia who has
spontaneous epistaxis?
How much is my hematocrit expected to rise after transfusion?
Will you be giving me whole blood or PRBCs?
Questions from the patient
What is the risk of HIV from a single blood transfusion?
What is the risk of Hepatitis C from a single blood transfusion?
Could I develop a fever during the blood transfusion?
Station 5:
Wellness Visit History
65yo female presents for a wellness visit
Intern is responsible for obtaining a
preventive history on the patient.
At the end of the visit, intern writes down the
labs, radiographs, consults, and counseling
he/she recommends for the patient.
PATIENT CARE N
o
Y
e
s
PATIENT CARE N
o
Y
e
s
INTERPERSONAL AND COMMUNICATION SKILLS D
i
d

N
o
t

P
e
r
f
o
r
m
N
e
e
d
s

I
m
p
r
o
v
e
m
e
n
t
A
v
e
r
a
g
e
A
b
o
v
e

A
v
e
r
a
g
e
G
o
o
d
E
x
c
e
l
l
e
n
t
1. Identifies reason for visit (routine wellness exam) 1. Greets patient respectfully and introduces self
2. Asks about any other concerns 39. Yearly flu shot 2. Seats patient and sets stage appropriately
40. Pneumovax 3. Shows empathy toward the patient concerns
3. FH of MI in 1st deg relative (Male<55, Female<65) 41. Tetanus (Td) 4. Smiles and uses eye contact when greeting
4. Cigarette smoking 5. Does not appear impatient during interaction
5. Hypertension (>140/90 or antihypertensive medication) 42. Current medical problems 6. Is non-confrontational during visit
6. Hyperlipidemia 43. Medical history 7. Uses an open, inviting posture
7. Diabetes mellitus 8. Interviews patient at eye level and is non-threatening
9. Puts the patient at ease with a friendly demeanor
8. Chest pain 44. Fasting lipid panel (USPTF) 10. Shows interest and concern throughout patients history
9. Shortness of breath 45. Screen for DM (USPTF) 11. Identifies purpose of visit & makes smooth transition into history
10. Abdominal pain 12. Uses patient history to guide differential and appropriate studies
11. Blood in stool 46. Mammogram (USPTF) 13. Does not ask questions about history already provided
12. Fever/chills 47. Bone density scan (USPTF) 14. Terminates visit appropriately
13. Night sweats
14. Weight loss 48. Colon cancer screening (FOBT/FIT, Fsig, BE, Colo- USPTF) 1. Addresses the patient as maamduring the visit
49. Offer counseling for weight loss (USPTF) 2. Is respectful to the patient throughout the visit
3. Does not talk down to or belittle the patient
15. Last pap smear 50. Tobacco cessation (USPTF) 4. Has fluent, understandable speech that flows
16. History of abnormal pap smears 51. Discusses chemoprevention with daily ASA use (FH MI- USPTF) 5. Presents self as a competent physician through verbal communication
17. Vaginal bleeding 52. Dietary counseling (USPTF) 6. Presents self professionally through posture and patient interaction
18. Family history of gynecologic cancers 53. Exercise
54. Safety 1. Effectively directs patients through the visit to ensure timeliness
19. Last mammogram 55. No future paps d/t hysterectomy (ACS, USPTF, ACOG) 2. Orders pertinent labs for a wellness visit
20. History of abnormal mammogram 3. Orders pertinent diagnostic tests for visit
21. Family history of breast cancer 4. Uses consultation services only when necessary
22. Performance of self breast exams
23. Current breast concerns
24. Last colon cancer screening
25. Personal history of polyps
26. Family history of polyps
27. Family history of colon cancer
28. Bone density scan
29. Lipid panel
30. Fasting glucose
31. Thyroid
32. Alcohol
33. Drug use
34. Diet
35. Exercise
36. Abuse in home
37. Depression
38. Safety belts
Reviews cardiac risk factors
Asks a pertinent review of symptoms
Asks about pertinent cancer screening history
Gynecologic
Breast
Colon
Asks about recent labs and studies
Asks about lifestyle concerns
Asks about immunization status
Asks about medical history
Management plan
Orders pertinent labs
PROFESSIONALISM
Comments:
Orders pertinent studies
Orders appropriate consultations
Provides appropriate counseling
Station 6:
Altered Mental Status
Simulation type: Human patient simulator
Nurse calls intern to ward to evaluate an
agitated and confused patient.
Intern must use medical record and nurse
interaction to gain history when patient does
not respond well.
After evaluation, the intern lists a differential
diagnosis; recommends labs, tests, studies;
and outlines a management plan.
PATIENT CARE
N
o
Y
e
s
MEDICAL KNOWLEDGE
N
o
Y
e
s
MEDICAL KNOWLEDGE
N
o
Y
e
s
1. Assesses Airway, Breathing, and Circulation
2. Patient's age 1. Hepatic encephalopathy 35. Chemistry panel (electrolytes, glucose, creatinine)
3. Reason for admission 2. Hypoxia 36. Ca, Mg, Phos
4. Baseline mental status during admission 3. Hypercapnea 37. Liver function tests
5. Past medical history 4. Uremia 38. Complete blood count
6. Recent surgeries 39. Tox screen
7. Recent hospital procedures 5. Urinary tract 40. Alcohol level
8. Current inpatient medications 6. Pneumonia 41. Urinalysis and culture
9. Medication allergies 7. Meningitis 42. Blood cultures
10. History of drug or alcohol use 8. Encephalitis 43. HIV
11. Reviews most recent labs 9. Bacteremia/septicemia 44. LP and CSF studies
45. ESR
12. Headache 10. Medications 46. Ammonia
13. Fevers 11. Alcohol 47. Thyroid studies
14. Chills 12. Drugs 48. Troponin or cardiac enzymes
15. Shortness of breath 13. Psychosis 49. ABG
16. Chest pain
17. Urinary complaints 14. Hypernatremia 50. EKG
18. Neurologic changes 15. Hyponatremia 51. Chest X-ray
19. Agitation 16. Hyperglycemia 52. Head CT
20. Delusions 17. Hypoglycemia 53. Consider EEG if work-up negative
18. Hypercalcemia
21. Obtains full set of vital signs 19. Hypophosphatemia 54. Calls supervising resident or staff
22. Checks alertness and orientation 55. Transfers patient to a monitored setting
23. Looks for signs of trauma or fall 20. Hyperthyroidism
24. Checks pupils 21. Hypothyroidism
25. Assesses for focal neurologic deficits 22. Adrenal insufficiency
26. Auscultates heart
27. Auscultates lungs 23. Primary CNS neoplasm
D
i
d

N
o
t

P
e
r
f
o
r
m
N
e
e
d
s

I
m
p
r
o
v
e
m
e
n
t
A
v
e
r
a
g
e
A
b
o
v
e

A
v
e
r
a
g
e
G
o
o
d
E
x
c
e
l
l
e
n
t
24. Metastatic neoplasm
1. Greets nurse respectfully and introduces self
2. Communicates effectively with the nurse 25. Chronically ill patient
3. Verbals orders are stated clearly 26. Vitamin deficiency (B12/folate)
PROFESSIONALISM
D
N
P
N
I
AA
A
GE
1. Is respectful to the patient throughout the visit 27. Post-ictal
2. Maintains composure despite difficult patient encounter
3. Is courteous despite difficult patient encounter 28. Concussion
4. Appropriately receptive to suggestions from others 29. Intracranial bleed
5. Overall behavior conveys competence
30. Arrhythmia
31. Acute myocardial infarction
1. Orders only pertinent labs based on differential
2. Orders only pertinent diagnostic tests based on differential 32. Stroke
3. Uses consultation services only when necessary 33. Vasculitis
34. Hypertensive encephalopathy
N
I
D
N
P
SYSTEMS-BASED PRACTICE E G A
A
A
Management plan
Labs
Studies
Other
Neoplasm
Nutritional deficiency
Seizures
Comments:
Trauma
Cardiac
Vascular
Asks the nurse about the patient's history
Asks nurse pertinent review of systems
Physical examination
Differential diagnosis for acute mental status change
Organ Failure
Infection
Ingestion, withdrawl, psych
Metabolic and electrolytes
Endocrine
Station 7:
Literature Search
Simulation type: Computer search
Two clinical scenarios: Otitis media and UTI
Intern starts at Madigan library home page
Can use any internet resources to find two
resources, articles, abstracts to answer the
question.
Intern then discusses and compares articles
during staff evaluation.
PRACTICE-BASED LEARNING
D
i
d

N
o
t

P
e
r
f
o
r
m
N
e
e
d
s

I
m
p
r
o
v
e
m
e
n
t
A
v
e
r
a
g
e
A
b
o
v
e

A
v
e
r
a
g
e
G
o
o
d
E
x
c
e
l
l
e
n
t
N
o
Y
e
s
Appears familiar with the internet
Opens appropriate windows
Able to navigate between web pages
Typing is functionally fluid
Demonstrates adequate computer skills and familiarity
Able to efficiently surf the web to answer a clinical question
Is familiar with web-based medical search engines
Locates references that address the clinical question
Finds the most evidence-based answer to question
Appropriately comments that article has the best study design
Matches an appropriate number of PICOs
Chooses the better of the two articles when asked
J ustifies why one article is better than the other
Identifies the strengths of article #1
Identifies the weaknesses of article #1
Identifies the strengths of article #2
Identifies the weaknesses of article #2
Subjective Evaluation
Start time: ____________
Stop time: ____________
Total number of websites tried: ____________
Answers to post-search questions
PRACTICE-BASED LEARNING
Objective Evaluation
Uses clinical information for search (PICO)
Correct patient- child <2yo with otitis
Correct intervention- antibiotics
Correct comparison- no antibiotics, observation
Correct outcome- Improved clinical course
Correct outcome- Lack of complications
Correct outcome- lack of side effects
Correct outcome- pain relief >2 days
Demonstrates familiarity with web-based search engines
Uses Google
Uses Up to Date
Uses MEDLINE (OVID or PubMed)
Uses MEDLINE with pre-programmed limits yielding high quality studies (clinical queries
Uses EMBASE
Uses PrimeEvidence
Uses InfoPOEMS
Uses database of systemic reviews (Cochrane)
Uses database of clinical guidelines
Uses MD Consult
Original search yielded manageable results
Broadened search to yield more results
Effectively used limits to narrow results
Uses Other (List name:___________________________)
Uses key components in search strategy
Uses BOTH patient AND intervention from clinical question
Uses EITHER patient OR intervention from clinical question
Comments:
SNAP Article
RCT on antibiotic use in children with otitis
Review of articles
Appropriately prioritizes articles
Changed database to more appropriate database
Found appropriate article
Cochrane "Antibiotics for acute otitis media in children"
AAP Clinical Practice Guideline
Adapts strategy
Station 8:
Chest Pain and Vfib
Simulation type: Human patient simulator
67yo male presents to ED with chest pain
Only info on the patient is a sheet of paper in
his wallet with PMH, Meds, Allergies
Intern manages chest pain, then Vfib
Later given initial labs showing hyperkalemia
Intern must circle abnormal lab value and list
all treatments starting with most pertinent for
this patient
PATIENT CARE
N
o
Y
e
s
N
o
Y
e
s
1. Check responsiveness
2. Assess airway (normal airway)
3. Assess breathing (normal respirations)
4. Assess circulation (pulse present)
5. Measure vital signs
6. Auscultate lungs
7. Auscultate heart
8. Re-evaluate responsiveness (pt does not respond)
9. Assess airway (normal airway)
10. Assess breathing (no respirations)
11. Assess circulation (no pulse)
12. Calls for help (calls code)
D
i
d

N
o
t

P
e
r
f
o
r
m
N
e
e
d
s

I
m
p
r
o
v
e
m
e
n
t
A
v
e
r
a
g
e
A
b
o
v
e

A
v
e
r
a
g
e
G
o
o
d
E
x
c
e
l
l
e
n
t
1. Orders only pertinent labs based on differential diagnosis
2. Orders only pertinent diagnostic tests based on differential
3. Appropriately uses a blood pressure cuff
4. Appropriately uses PPV to ventilate patient
5. Correctly applies pads for monitoring and defibrillation
6. Appropriately uses machine to defibrillate patient
30. PO Kayexelate
31. Inhaled beta-agonist/nebs
32. Dialysis
26. First treats with IV calcium
27. IV Insulin/Glucose
28. IV Diuretics
21. Starts or requests chest compressions
22. Treats rhythm initially with defibrillation
29. IV sodium bicarbonate
23. Uses Epinephrine 1mg for persistent VF
24. Repeats defibrilation
Hyperkalemia
25. Recognizes hyperkalemia as a lab abnormality
17. Calls supervising resident or staff
Patient goes into cardiac arrest
19. Recognizes VF rhythm on cardiac monitor
20. Starts or requests positive pressure ventilation
13. Treats with sublingual nitroglycerin
14. Treats with a beta-blocker
15. Treats with morphine
16. Considers anticoagulants/lytics
9. Identifies ST depression in leads I, aVL, V2- reciprocal changes
10. Recognizes as inferior/lateral wall ischemia
11. Diagnoses an acute ST elevation MI in context of sx and findings
12. Treats with chewed aspirin
MEDICAL KNOWLEDGE
Patient has chest pain
1. Requests IV access
2. Provides oxygen
Comments:
18. Considers cardiology consult/ Cardiac Cath
Chest pain
Ventricular fibrillation
3. Places patient on a cardiac monitor
4. Obtains EKG
5. Correctly interprets 12 lead EKG
6. Obtains Troponin (cardiac enzymes) at a minimum
7. Orders a portable CXR
8. Identifies ST elevation in leads II, III, aVF, V5, and V6
OSCE Results
Results
OSCE 2006-07
58 interns (38 men, 20 women)
Ages 25-44, average age 29
41 allopathic, 17 osteopathic
Primary care (35), surgical (13), TY (10)
EM (n=8)
FM (n=8)
GS (n=5)
IM (n=10)
Neuro (n=3)
OB (n=4)
Ortho (n=2)
ENT (n=2)
Peds (n=6)
TY (n=10)
Improvement seen in all competency scores
Scores reported as percent of total available points received
OSCE Pre and Post Competency Scores
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
Score
Competency

Incoming Score 49.4% 54.6% 34.6% 25.5% 64.2% 53.8% 38.2%
Outgoing Score 59.9% 58.9% 48.2% 47.2% 72.6% 67.7% 65.2%
Total OSCE PC MK PBLI ICS Pro SBP
10.5%
4.2%
13.6% 21.7%
8.4%
13.9%
27.0%
Transitional Year Program
OSCE Outcome Data
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
Competency

Incoming
45.9% 51.2% 31.4% 22.5% 60.2% 49.6% 35.0%
Out going
58.6% 57.9% 48.4% 45.1% 70.3% 66.0% 59.6%
OSCE PC MK PBLI ICS PRO SBP
2006-07 OSCE TY Competency Data
12.7% 6.6%
17.0%
22.6%
10.1%
16.4%
24.6%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
Stati on

Incoming 72.5% 46.5% 68.1% 30.3% 39.5% 28.3% 22.5% 55.8%
Out going 74.6% 56.2% 69.8% 48.5% 52.7% 36.1% 45.1% 74.1%
Deat h Abd Pn Sut ur e
Blood
Consent
Wellness
Ment al
St at us
Lit er at ur e
Chest
Pain
2006-07 OSCE TY Station Data
2.1%
9.7%
1.8%
18.1%
13.3%
7.8%
22.6%
18.2%
Changes to TY
Program
Abdominal pain evaluation
Inpt general surgery rotation changed to clinic
Call with SOD assisting with ED evaluations
Wellness visit
FM rotation now four weeks of clinic, no inpt
OB/GYN rotation now 2wks GYN clinic & 2wks L&D
Suture skills
Increased procedure time on gen surgery clinic
Increased procedure time on family medicine
TY resident suture training at simulation center
As part of the accreditation process, programs
must provide evidence that they are making
data-driven improvements.
(ACGME Outcome Project)
OSCE Challenges
and Limitations
OSCE Challenges
OSCE takes time, money, and volunteers
OSCE takes organization and planning
Location and use of simulation centers
Tailor OSCE to type of program
OSCE not the only way to achieve outcomes
Limitations
Practice-effect bias using same OSCE
Blinding of evaluators to residents
Conclusion
A well designed OSCE can:
Evaluate competency performance of incoming
interns or residents
Assess resident performance for interval
improvement
Provide outcome measures for your program
Assist in evaluating program effectiveness
Lead to continuous program improvement
Be a useful tool to meet requirements of the
ACGME Outcome Project
Questions?

Assessing Core Competencies Us in Go Sce

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Assessing Core Competencies Us in Go Sce

Uploaded by

Copyright:

Available Formats

Session SES76

Assessing Core Competencies

You might also like