Professional Documents
Culture Documents
Evaluation Considerations For EHR-Based Phenotyping
Evaluation Considerations For EHR-Based Phenotyping
Evaluation Considerations For EHR-Based Phenotyping
Casey Overby, Chunhua Weng, Krystl Haerian, Adler Perotte Carol Friedman, George Hripcsak Department of Biomedical Informatics Columbia University
common disease
Adverse drug events Responder types
High PPV
(Kho et al. JAMIA 2012; Kullo et al. JAMIA 2010; Ritchie et al. AJHG 2010; Denny et al. Circulation 2010; Peissig et al. JAMIA 2012)
Pattern of liver injury based on liver enzyme elevations No tests to confirm drug causality (some assessment tools exist) High PPV may be challenging
Why study?
DILI accounts for up to 15 % of acute liver failure cases in the
U.S., of which 75% requires liver transplant or lead to death Most frequent reason for withdrawal of approved drugs from the market
Lack understanding of underlying mechanisms of DILI Computerized approaches can reduce the burden of
Case definition
Case definition
Case definition
Case definition
Case definition
Lessons learned
18,423
yes
no Exclude Exclude no
Consider chronicity no
13,972
yes Exclude
no
2,375
yes
1.
a.
yes
yes
yes
2.
Caused by a drug
a. New drug (A2) b. Not by another disease (D)
C4. Bilirubin >= 2x ULN yes
1,264
D. Other diagnoses?
560
no
Ref: Aithal, G.P., et al. Case Definition and Phenotype Standardization in Drug-induced Liver Injury. Clin Charmacol Ther. 2011 Jun; 89(6): 806-15
no
yes
Exclude
Exclude
Reviewer 1
TP: 27 FP: 42 NA: 30 PPV: TP/(TP+FP) = 27/(42+27) = 39%
20
Reviewer 2
20
20
20
Reviewer 3
20
Reviewer 4
Qualitative results
Perceptions of evaluation approach effectiveness: Differences between evaluation platforms Visualizing lab values Availability of notes Discharge summary vs. other notes
Perceptions of benefit of results (themes in FPs): Babies Patients who died Overdose patients Patients who had a liver transplant
Lesson learned: Whats correct for the algorithm may not be correct for the case definition
Are we measuring what we mean to measure? Case definition: liver injury due to medication, not by another disease Many FPs were transplant patients Patients correct for the algorithm, but liver enzymes elevated due to procedure Revision: exclude transplant patients
Lessons learned
Lessons learned
Lesson learned: Evaluator effectiveness influences ability to drawing appropriate inferences about algorithm performance
How does our evaluation approach influence performance
estimations?
Moderate agreement among algorithm reviewers, so interpretation of
PPV unclear
Revision: Improve evaluator approach
Lessons learned
300
200
100
20120315
20120320
20120325
20120330
20120405
20120410
20120415
20120420
20120426
20120501
20120506
20120511
20120517
20120522
20120527
20120601
20120607
20120612
20120617
20120622
20120628
20120703
20120708
400
200
20120315
20120320
20120325
20120330
20120405
20120410
20120415
20120420
20120426
20120501
20120506
20120511
20120517
20120522
20120527
20120601
20120607
20120612
20120617
20120622
20120628
20120703
20120708
review
20120315
20120320
20120325
20120330
20120405
20120410
20120415
20120420
20120426
20120501
20120506
20120511
20120517
20120522
20120527
20120601
20120607
20120612
20120617
20120622
20120628
20120703
20120708
Lessons learned
20120719
20120719
20120719
Summary of findings
Lessons learned from applying our evaluation framework Whats correct for the algorithm may not be correct for the case definition (Are we measuring what we mean to measure?) Evaluator effectiveness influences ability to draw appropriate inferences about algorithm performance Demonstrated that our evaluation framework is useful Informed improvements in algorithm design Informed improvements in evaluator approach Likely more useful for rare conditions Demonstrated EHR phenotyping algorithm development is an iterative process Complexity of the algorithm may influence
Acknowledgments
Dr. Yufeng Shen - Serious Adverse Event Consortium collaborator eMERGE collaborators
Mount Sinai (Drs. Omri Gottesman, Erwin Bottinger, and Steve Ellis) Mayo Clinic (Drs. Jyotishman Pathak, Sean Murphy, Kevin Bruce, Stephanie Johnson,
Jay Talwalker, Christopher G. Chute, Iftikhar J. Kullo) Northwestern (Dr. Abel Kho) Vanderbilt (Dr. Josh Denny)
DILIN collaborator
UNC-CH (Dr. Ashraf Farrag)
Columbia Training in Biomedical Informatics (NIH NLM #T15 LM007079) The eMERGE network U01 HG006380-01 (Mount Sinai)
Questions?
Quantitative results
Demonstration study
Qualitative results
Perceptions of evaluation approach effectiveness: Differences between evaluation platforms Visualizing lab values Availability of notes Discharge summary vs. other notes
Perceptions of benefit of results (themes in FPs): Babies Patients who died Overdose patients Patients who had a liver transplant