Clinical Prediction Rule

in Emergency Medicine
新光紀念醫院急診醫學科
Special Lecture
May 17 2011
CGMH 陳冠甫
kfchen@cgmh.org.tw

Monday, May 16, 2011 1

 Contents
• What is CPR?
• Why is it important for us to do CPR studies
• Development of CPR
• Validation of CPR
• Implementation of CPR
• Progress in CGMH
• Potential collaboration

Monday, May 16, 2011 2

 Contents
• What is CPR?
• Why is it important for us to do CPR studies
• Development of CPR
• Validation of CPR
• Implementation of CPR
• Progress in CGMH
• Potential collaboration

Monday, May 16, 2011 3

 What is CPR?

• Terminology
• Clinical Prediction Rule
• Clinical Prediction Model
• Clinical Decision Rule
• Risk scores

Monday, May 16, 2011 4

 Prediction models vs.
Diagnostic tests
Diagnostic test Prediction model
Can be single test/device or Almost always combo of
combo of predictors predictors

May involve models Always involves models

No limit on predictability Predictability is limited

May need validation Always needs validation

Affected by dz prevalence Affected by dz prevalence

Applied to symptomatic pop Can be applied to all

Context and use usually fairly Can be used for many

specific. purposes and settings.

Monday, May 16, 2011 5

 CPR in MedCalc

Monday, May 16, 2011 6

 CPR in MedCalc

Monday, May 16, 2011 7

 Examples of CPR
ABCD2 score NEXUS II CT-Head rule
Alvarado score Ottawa Knee rule
Blatchford score PERC
Canadian C-spine rule PSI
Canadian CT Head rule Ranson’s score
CAP PIRO score Rockall score
CATCH rule San Francisco syncope rule
CHADS2 score SIRS
CHA2DS2-VASc Score SOFA score
CURB-65 score TIMI score
Geneva score for PE VAP PIRO score
HAS-BLED score Well score for DVT
NEXUS C-spine rule Well score for PE

Monday, May 16, 2011 8

 Examples of CPR
Neuro ID
ABCD2 score CAP PIRO score
Trauma CURB-65 score
Canadian C-spine rule PSI
Canadian CT Head rule SIRS
CATCH rule SOFA score
NEXUS C-spine rule VAP PIRO score
NEXUS II CT-Head rule CV
Ottawa Knee rule CHADS2 score
CHA2DS2-VASc Score
GI
Geneva score for PE
Alvarado score
HAS-BLED score
Blatchford score
PERC
Ranson’s score
San Francisco syncope rule
Rockall score
TIMI score
Well score for DVT & PE

Monday, May 16, 2011 9

 Niche for CPR?
Clinical decision making
High clinical stakes
Achieve cost savings /s compromising patient care
ID high risk persons
for preventive interventions
for clinical or epidemiological studies

Medical/biologic insight
Patient/family planning purposes

Monday, May 16, 2011 10

 Any evidence based benefit?

Pneumonia Severity Index*

> 50,000 patients with CAP
Safely increase % patients treated as outpatient
Reduce Length of Stay
but not Reduce Quality of Life

*Aujesky et al, CID 2008

Monday, May 16, 2011 11

 Why is it important?

Different population
Disappointing accuracy
Need for updating & impact analysis

Monday, May 16, 2011 12

 Contents
• What is CPR?
• Why is it important for us to do CPR studies
• Development of CPR
• Validation of CPR
• Implementation of CPR
• Progress in CGMH
• Potential collaboration

Monday, May 16, 2011 13

 Development of CPR
Derivation

Validation Validation Validation

1* 2* 3*

Impact analysis

Monday, May 16, 2011 14

 Development of CPR
• Published articles: 6,744 in 1995 -> 15,662 in 2005*
• Mainly development
• Lack of validation & impact analysis

• Begin: constructing a list of potential predictors

• Then: examine a group of patients and determine the factors
• Statistical analysis
• Regression
• Recursive partitioning analysis
• Discriminant analysis
• Neural network
• Random forest *Toll el al. JCE 2008

Monday, May 16, 2011 15

 Development of CPR
Regression
• Example: SBP = 33 + 1.45 * DBP
250
200
150
POSTSBP
100 50
0

0 50 100 150
POSTDBP

POSTSBP Fitted values

Monday, May 16, 2011 16

 Development of CPR
Regression
• ABCD2 for TIA
• Hazard of TIA = A + B + C + D + D
• A = Age > 60 years -> 1 point
• B = SBP > 140 or DBP > 90 mmHg -> 1 point
• Clinical feature
• Unilateral weakness -> 2 point
• Speech disturbance /s weakness -> 1 point
• Duration of symptoms
• > 60 minutes -> 2 point
• 10 < 59 minutes -> 1 point
• Diabetes -> 1 point

Monday, May 16, 2011 17

 Development of CPR
Regression
• ABCD2 for TIA*
• Hazard of TIA

*Rothwell et al. Lancet 2005

Monday, May 16, 2011 18

 Development of CPR
Regression
• ABCD2 for TIA
• Hazard of TIA = 2.57Age + 9.67BP + 6.61Clinical1 +
2.59Clinical2 + 6.17Duration1 + 3.08Duration2 +
4.39DM

Monday, May 16, 2011 19

 Development of CPR
Regression
• ABCD2 for TIA
• Hazard of TIA = 2.57Age + 9.67BP + 6.61Clinical1 +
2.59Clinical2 + 6.17Duration1 + 3.08Duration2 +
4.39DM
• Hazard of TIA = Age + Hypertension + 2*Clinical1 +
Clinical2 + 2*Duration1 + Duration2 + DM

Monday, May 16, 2011 20

 Development of CPR
Recursive partitioning analysis
• Canadian C-Spine Rule

Monday, May 16, 2011 21

 Development of CPR
Recursive partitioning analysis
*Stiell, JAMA 2001

• Canadian C-Spine Rule

Monday, May 16, 2011 22

 Development of CPR
Recursive partitioning analysis
• San Francisco Syncope Rule

Monday, May 16, 2011 23

 Development of CPR
Recursive partitioning analysis
• San Francisco Syncope Rule

*Quinn et al, AEM 2004

Monday, May 16, 2011 24

 Development of CPR
Recursive partitioning analysis
• Non-parametric, tree based
• “Partitioning” subjects by selecting variables “recursively”
• Strengths 1

• No assumption for distribution

2 3
• Can deal with high dimensional dataset
• Sophisticated methods for missingness 4 5 6 7

• Unaffected by outliers, collinearities

• Weakness 8 9 10 11

• Poor in modeling linear structure 12 13

• Not probabilistic model, no confidence interval

Monday, May 16, 2011 25

 Prediction Tree for Viral Load in Pediatrics
High VL 33
Admission 73 40%
Low VL 49
Discharge 69

Age > 18m/o Age < 18m/o

33% High VL 23 High VL 10 83%

Low VL 47 Low VL 2

Age < 5.5 Age > 5.5

12% High VL 4 High VL 19 51%

Low VL 29 Low VL 18
Normal ANC
Abnormal ANC

50% High VL 13 High VL 6 55%

Low VL 13 Low VL 5

No underlying illness Any underlying illness

38% High VL 6 High VL 7 70%
Low VL 10 Low VL 3

Chen et al, in preparation

Monday, May 16, 2011 26

 Development of CPR
• Discriminant analysis

Monday, May 16, 2011 27

 Development of CPR
• Discriminant analysis
• Neural network Neural Networks
Outcome indicator variables
Y1 Y2

Z1 Z2 Z3 Z4 Z5 Derived variables

X1 X2 X3 X4 !!!!!!! Xp-1 Xp

Original predictors

Monday, May 16, 2011 28

 Development of CPR
• Discriminant analysis
• Neural network
• Random forest

Monday, May 16, 2011 29

 Development of CPR
Cox/Logistic Logic k-Nearest CART Neural Support
Reg. Reg. Neighbors Networks Vector
Machines
Non-parametric 0/-1 -1 1 1 0 1

Model selection -1 1 NA 1 -1 1
tied to model utility
Interpretability 0 1 -1 1 -1 -1

Mixed predictor 1 -1 -1 1 -1 -1
type
Can force predictor 1 1 1 -1 1 1
inclusion
Handling missing -1 -1 -1 1 -1 -1
data
Robustness to -1 ? 1 1 -1 -1
predictor outliers

Scale: -1 = poor, 0 = fair, 1 = good

Monday, May 16, 2011 30

 Development of CPR
Questions:
• How were predictors chosen and defined
• How were study subjects selected
• Was the sample size adequate
• including number of outcome events
• Were all important predictors present
• Does the rule make clinical sense

Monday, May 16, 2011 31

 Development of CPR
Guidelines
• Ratio of predictors to patients
• Methods of predictors selection
• Weight assignment
• Shrinkage of coefficients to prevent over-fitting
• Estimate the potential by internal validation

Monday, May 16, 2011 32

 Development of CPR
• Disappointing accuracy
• Example: Children with FUO
• AUC of derivation study: 0.76, AUC of validation study: 0.57

• Reason
• Inadequately developed
• Difference between derivation & validation population (generalizability)
• Different definition of predictor and outcome variables
• Should Only include good reliability (inter-observer variability)
• Case-mix: EuroSCORE study
• Fewer individuals in validation studies
to detect substantial
• At least 100 events and 100 nonevents changes in accuracy (for
example, a 0.1 change in c-
statistic) with 80% power
Monday, May 16, 2011 33
 Contents
• What is CPR?
• Why is it important for us to do CPR studies
• Development of CPR
• Validation of CPR
• Implementation of CPR
• Progress in CGMH
• Potential collaboration

Monday, May 16, 2011 34

 Hierarchy evidence of CPR

Level I

Level II

Level III

Level IV

Monday, May 16, 2011 35

 Hierarchy evidence of CPR can be
used in a
wide variety of
settings with
confidence that they
can change clinician
behavior and improve
patient outcomes

Rules can be used in

various settings with
confidence in their accuracy

Clinicians may consider using with

caution and only if patients in the study
similar to those in your clinical setting

Rules that need further evaluation before they

can be applied clinically

Monday, May 16, 2011 36

 Hierarchy evidence of CPR
At least one
prospective
validation in different
population & one impact
analysis, demonstrating
change in clinician behavior
with beneficial consequences.

Demonstrated accuracy in either

one large prospective study /c
broad spectrum of patients/clinicians,
or validated in several smaller settings
differ from one another.

Validated in only one narrow

prospective sample.

Derived but not validated or only validated in

split samples, large retrospective databases, or
by statistical techniques.

Monday, May 16, 2011 37

 Hierarchy evidence of CPR
Level I:
Rules that can be used in a wide variety of settings with confidence that
they can change clinician behavior and improve patient outcomes
At least one prospective validation in a different population and one impact
analysis, demonstrating change in clinician behavior with beneficial
consequences.
Level II:
Rules that can be used in various settings with confidence in their
accuracy
Demonstrated accuracy in either one large prospective study including a
broad spectrum of patients and clinicians, or validated in several smaller
settings who differ from one another.
Level III:
Rules that clinicians may consider using with caution and only if patients
in the study are similar to those in your clinical setting
These rules have been validated in only one narrow prospective sample.
Level IV:
Rules that need further evaluation before they can be applied clinically
These CPRs have been derived but not validated or have only been validated
in split samples, large retrospective databases, or by statistical techniques.

Monday, May 16, 2011 38

 Evaluation stage of CPRs

Stage 0: Pick up a good CPR to use:

e.g. Sepsis/syncope/pneumonia
Stage 1: Validate it by retrospectively
review in one setting, modify if performance
sub-optimal
Stage 2: Validate it “narrowly” by
prospective review in one setting

Monday, May 16, 2011 39

 Evaluation stage of CPRs

Stage 3: Validate it “broadly” by prospective

review in varied setting with wide spectrum
of patients and physicians
Stage 4: Narrow impact analysis of CPR
used as decision rule
Stage 5: Broad impact analysis of CPR as
decision rule

Monday, May 16, 2011 40

 Validation of CPR
Reasons:
• Due to chance?
• different set of predictors will emerge

• Statistical methods: bootstrap

• Idiosyncratic ?
• fail in a new setting

• Feasibility ?
• succeeds in theory but fails in practice

Monday, May 16, 2011 41

 Validation of CPR
Questions:
• P’t chosen in unbiased fashion?
• Wide spectrum of severity of disease?
• Blinded assessment of criterion standard for all?
• Explicit & accurate interpretation of variables?
• 100% follow-up?

Monday, May 16, 2011 42

 Validation of CPR
Derivation

Types of validation
• Temporal Validation
1*
Validation
2*
Validation
3*

• Geographical
• Domain Impact analysis

Types of impact analysis

• Randomized cluster/community trials
• Before-after analysis

Monday, May 16, 2011 43

 Validation of CPR

Predictive power

• Sensitivity and specificity

• Confidence interval

• Likelihood ratios, or as absolute or relative risks.

• ROC (receiver operative characteristics)

Monday, May 16, 2011 44

 Validation of CPR
• How many has been validated in Asian?
• ABCD2: China, Hong-Kong

• Alvarado: Singapore

• Blatchford: Hong-Kong

• CURB-65: Taiwan (EM-CGMH)

• PSI: Taiwan (IM-NCKU), Malaysia, South Korea

• SOFA: China, South Korea,

• TIMI: China

Monday, May 16, 2011 45

 Impact analysis of CPR
Change in behavior ?
• intuitive estimation vs CPR?
• Comparing physician’s vs. Rule’s prediction
• Face validity
• User friendly?
!
HIS/EPR
!
CDSS (clinical decision support system)
• practical barriers (litigation!)

Monday, May 16, 2011 46

Monday, May 16, 2011 47
 Contents
• What is CPR?
• Why is it important for us to do CPR studies
• Development of CPR
• Validation of CPR
• Implementation of CPR
• Progress in CGMH
• Potential collaboration

Monday, May 16, 2011 48

 CPRs @ CGMH
Stage0:
• 依照急診現狀所需,挑選適合本院的CPR
• 如高盛行率但無有效診斷工具之疾病
Stage1:
• 挑選一個適合的院區做chart review,訂定一個可接受的標
準(如>80% sensitivity, >0.8 AUC)。
• 若是不達這樣的標準可以考慮modify或重新derive一個
CPR,
• 可以使用在本Stage得到的database來做derivation,但由
於可能有missing data問題,要考慮prospectively取得新的
database來做驗證(validation)。

Monday, May 16, 2011 49

 CPRs @ CGMH
Stage2:
• 在上個stage得到的CPR, 做prospective validation。
• 著重的亦是single setting的validation,
• 結果也將用來決定是否繼續往下一個stage進行,或是需要重新
modify/derive。
• 將CPR 整合至CDSS(clinical decision support system)中,使用
不論是現在使用的急診醫囑系統或是Mars-T來prospectively收
案。
Stage3:
• 做multi-centers且prospective的收案來 validate這個CPR。
• 預定訂定好各院區合理且有統計學效力的sample size來收案。

Monday, May 16, 2011 50

 CPRs @ CGMH
Stage4:
• 在單(北)院區進行教育性seminar
• 以before-after的方式,來收取醫師們對於這樣一個CPR是
否會影响(impact)其臨床practice的收案分析。
• 應分成南北院區而不是各別院區。
Stage5:
• 在另一個院區(南)進行impact analysis

Monday, May 16, 2011 51

 CPRs @ CGMH
• Focused group
長庚!"#$#!%"&'()*#!+#,$&-.")&/0'-1研究小組

長庚急診$教研小組$$$$$$$ CV2
./(0林,詹9陳冠甫 ./0張

:&;<-=->-?*$ !"#$小組 $
陳冠甫 陳冠甫

%&'()($$$$$$$$ %*+,-'&$$$ "+&12-+)3$ 456

578$./0張
陳冠甫 ./0林 ./0詹 ./0翁

578$./0蘇 ./(0陳$9$林 ./0吳s ./0劉 ./0陳

Monday, May 16, 2011 52

 CPRs @ CGMH
• Focused group
• Shared information

Monday, May 16, 2011 53

 CPRs @ CGMH
• Focused group
• Shared information
• Resource integration
• IT: HIS information extraction

Monday, May 16, 2011 54

 CPRs @ CGMH
• Focused group
• Shared information
• Resource integration
• IT: HIS information extraction
• Database management: Access + SQL + VBA
• Novel MARS-T system
• Collaboration with faculty in CGU
• NSC project: “Comparative Effectiveness Research
Initiative in Clinical Study”

Monday, May 16, 2011 55

Monday, May 16, 2011 56
 CPRs @ CGMH + SKH ?

• Impact analysis
• Larger prospective development/validation
• Longitudinal & cross-sectional collaboration

Monday, May 16, 2011 57

 Questions?
kfchen@cgmh.org.tw

Monday, May 16, 2011 58