O c tober 26-30, 2003 Salt Lake City, Utah

Posterior Predictive Check What, When and How?

Joga Gobburu Team Leader, Pharmacometrics, OCPB/CDER/FDA

My remarks today do not necessarily reflect the official views of FDA

Oct, 2003

Gobburu, FDA

Posterior Predictive Check What, How and When?

Joga Gobburu Team Leader, Pharmacometrics, OCPB/CDER/FDA

Oct, 2003

Gobburu, FDA

Based on work jointly performed with Pravin Jadhav, MS Graduate Student VCU, Fellow at FDA John Lawrence, PhD Senior Biostatistician CDER, FDA
Oct, 2003 Gobburu, FDA 4

Model checking approaches mostly assess the descriptive ability of models. Adherence to mechanistic reasoning is the only rational approach to successful extrapolation.

Oct, 2003

Gobburu, FDA

Initial Model Selection

Visual inspection
Observed versus predicted

Log-likelihood ratio testing (LRT)

Covariate selection

Reliability checking
Bootstrap, Log-likelihood profiling

Mechanistic reasoning
Prior knowledge
Oct, 2003 Gobburu, FDA 6

Final Model(s) Checking

Qualitative
Visual inspection Comparison of unexplained variability

Quantitative
Internal, External, Cross-validation Posterior Predictive Check
All checks are subjective at some level
Oct, 2003 Gobburu, FDA 7

Posterior Predictive Check

A method to check if the posited model should be excluded because it fails to provide a reasonable summary of the data at hand

Oct, 2003

Gobburu, FDA

History of PPC
Guttman I. The use of the concept of a future observation in goodness-of-fit problems. J.R.Statist.Soc. B, 83-100, 1967. Rubin DB. Bayesianly justifiable and relevant frequency calculations for the applied statistician. Ann.Statist.12, 11511172, 1984. Meng X-L. Posterior predictive p-values. Ann. Statist.,22, 1142-1160, 1994. Gelman A, Meng X-L, Stern H. Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6:733-807, 1996. Gobburu, Holford et al. Simplified posterior predictive check (sPPC) for model qualification. ASCPT, 2001. Yano, Beal and Sheiner. Evaluating pk/pd models using the posterior predictive check. J.PKPD, 8:171-192, 2001.
Oct, 2003 Gobburu, FDA 9

Posterior Predictive Check

Estimation Step

Simulation Step

Evaluation Step

Oct, 2003

Gobburu, FDA

10

Posterior Predictive Check

Estimation Step
Estimate by fitting model to yobs

Simulation Step
Simulate yrep using and model

Evaluation Step
Compare yrep and yobs
Oct, 2003 Gobburu, FDA 11

Modeling Approaches
Least Squares (LS) or Maximum Likelihood (ML) Estimation
Standard 2-Stage
WinNonlin, ADAPT, Others (below)

1-Stage
NONMEM, WinNonmix, Splus, SAS, Kinetica

Bayesian Estimation
WinBUGS / PKBUGS
Oct, 2003 Gobburu, FDA 12

Posterior Distribution
Posterior distribution accounts for the parameter uncertainty and prior knowledge
Not considered today
600

ML estimate

ML estimate ignores this uncertainty/prior

ML widely used Reasonable for rich data
Oct, 2003 Gobburu, FDA

0 0

200

400

10

CL
Simplified PPC (sPPC)
13

Modeling Approaches
Least Squares (LS) or Maximum Likelihood (ML) Estimation
Standard 2-Stage
WinNonlin, ADAPT, Others

1-Stage
NONMEM, WinNonmix, Splus, SAS, Kinetica

Bayesian Estimation
WinBUGS / PKBUGS
Oct, 2003 Gobburu, FDA 14

Model Predictions
Concentration, ug/L 1000 800 600 400 200 0 0
Oct, 2003

y Observed pred y Population

...yipred Individual

obs

4 Time, h

6
15

Gobburu, FDA

Estimation Step
Population parameters
CL V = 2 CL 2 V 2
pred

Individual parameters
conditional on and yobs
CL1 V1 CL V i = 2 2 ... ... CLi Vi

Mean

Variance

[y ] = [y , y ...y ]
1 2 j
Oct, 2003

[y ]
ipred
Gobburu, FDA

y11, y12...y1j y21, y22...y2 j = ... yi1, yi 2 ...yij

16

Simulation Step
Population based simulations
Generate yrep independent of yobs Allow checking models generalizability Removed from Estimation step

Individual based simulations

Generate yrep conditional on yobs Allow checking models descriptive ability
Do not assess generalizability

Are proximal to Estimation Step

Oct, 2003 Gobburu, FDA 17

Simulation Step
Population based simulations
Generate yrep independent of yobs Allow checking models generalizability Removed from Estimation step

Individual based simulations

Generate yrep conditional on yobs Allow checking models descriptive ability
Do not assess generalizability

Are proximal to Estimation Step

Oct, 2003 Gobburu, FDA 18

Simplified PPC
Test statistics do not depend on T(yobs) T(yrep) Evaluation

yobs
Model Estimation
Oct, 2003

yrep
Model Simulation
19

Gobburu, FDA

Simplified PPC
Discrepancy variables depend on T(yobs,) T(yrep,)

yobs
Model
Oct, 2003 Gobburu, FDA

ypred

yrep
Model

20

Evaluation Step
Select meaningful ____ or T(y*,) T(y*)
T(y*) T(y*,)

C( t), C(t)
SSE = ( y
pred

y*)

Select a criteria

* = obs or rep

Visual inspection (Posterior) predictive p-value (ppc) Probability of equivalence (peqv) Probability of rejecting H0 by KS (pks)
Oct, 2003 Gobburu, FDA 21

Visual Inspection
s

0 20 40 60 80 100

May not be a bad first cut

Obvious deviations can be noted

T yo ,) (

Centered around ypred

0 20 40 60 80 100
T(yrep,) Impractical to plot all reps Similar to ypred versus yobs plot
22

Oct, 2003

Gobburu, FDA

Predictive P-Value
2000

T(yobs)

1 Nrep pc rep obs p = I(T(y ) T(y ) Nrep i=1

I() = 1 Simulation under the H0 yrep = yobs

500

1000

1500

T(yrep)
-4 Oct, 2003 -2 0 2 Gobburu, FDA 4
23

Predictive P-Value
2000

T(yobs) I() = 0

500

1000

Rep 1 2 100

I() 0 1 1

1500

Average I() is ppc

T(yrep)
-4 Oct, 2003 -2 0 2 Gobburu, FDA 4
24

Predictive P-Value
ppc = 1
2000

T(yobs)

Model always over-predicts OK or not is subjective

T(yrep)

0 -4 Oct, 2003

500

1000

1500

-2

2 Gobburu, FDA

25

Predictive P-Value
ppc = 0
2000

T(yobs)

Model always under-predicts OK or not is subjective

T(yrep)

0 -4 Oct, 2003

500

1000

1500

-2

2 Gobburu, FDA

26

Predictive P-Value
2000

ppc = 0.5 T(yobs)

Model ~ Data

1500

0.2 < ppc < 0.8 may be OK

T(yrep)
-4 Oct, 2003 -2 0 2 Gobburu, FDA 4
27

500

1000

Predictive P-Value
Discrepancy variables
Depend on , relative to ypred Cannot be compared across models
ppc is almost uninformative E.g.: How extreme is SSE of rep relative to obs? Both use ypred, hence scaled in a way

Test statistics
Do not depend on , not relative to ypred Can be compared across models
E.g.: How extreme is the obs C(t) relative to rep?
Oct, 2003 Gobburu, FDA 28

Probability of Equivalence
peqv 1 Nrep T( yrep ) = I(0.80 T(yobs ) 1.25) Nrep i=1

Closer to true model, higher the peqv Low peqv flags a poor model feature
acceptability is subjective, but informed

Oct, 2003

Gobburu, FDA

29

Probability to Reject H0
H0 : T ( y ) = T( y )
rep obs

1 Nrep p ks = I(p 0.05) Nrep i=1

Closer to true model, lower the pks High pks flags a poor model feature
acceptability is subjective, but informed

Oct, 2003

Gobburu, FDA

30

sPPC versus LRT

I test whether a particular covariate (CL in males 10% ) is important or not using LRT. I conduct PPC and find that the models with and without the covariate describe the data similarly. So?
LRT (assuming performed correctly) tests statistical relevance. Depends on sample size and effect size. PPC tests practical relevance. 10% may not need different sampling design in future trials.
Could be vice-versa
Oct, 2003 Gobburu, FDA 31

Simulated Example
Original Data (yobs)
50 sets 1-cmt model, iv bolus, CL(males)=5CL(females) Rich sampling: 0.5, 1, 1.5, 2, 4 h

PPC
True and false (males=females) model fitted to yobs 200 replicates (yrep) simulated for each model & set T(y*,) and T(y*) calculated ppc, peqv and pks determined

Oct, 2003

Gobburu, FDA

32

Simulated Clearances
Frequency, % 10 15 20 25

CL(male) = 5 CL(female)

0 0.0
Oct, 2003

0.5

1.0 Clearance, L/hr

1.5
33

Gobburu, FDA

Gender Significant? Log-Likelihood Ratio

30

Fre que ncy , %

10

20

80
Oct, 2003

100 LLR
Gobburu, FDA

120

140
34

95% Prediction Intervals

Observed concentration, mg/L 0 50 100 150 200

True Model No Covariate Model

1
Oct, 2003

2 Time, hr
Gobburu, FDA

4
35

Predictive P-Value
10 15 20

Fre que ncy , %

0.0

0.4 0.8 pPPC

0 0.0

10

15

0.4 0.8 pPPC

? Model

? Model

Oct, 2003

Gobburu, FDA

36

Predictive P-Value
10 15 20

Fre que ncy , %

0.0

0.4 0.8 pPPC

0 0.0

10

15

0.4 0.8 pPPC

True Model

No Covariate Model

Uniform distribution expected Uniform distribution not expected For =0.05, 5% below 0.05
Oct, 2003 Gobburu, FDA 37

peqv, pks
T(yobs) = C(t=0.5,4.0)

100 80 60 40 20 0 0 2 4 Time, h 6 8

Oct, 2003

Concentration

Gobburu, FDA

38

Probability of Equivalence
0.5 h

Fre que ncy , %

10

20

0 5 0.94

15

0.96

0.98 pEQV

1.00

0.96 0.98 pEQV

True Model

No Covariate Model

Oct, 2003

Gobburu, FDA

39

Probability of Equivalence
4.0 h

Fre que ncy , %

15

0 5

0 0.35

20

40

0.65

0.75 pEQV

0.45 pEQV

True Model

No Covariate Model

Oct, 2003

Gobburu, FDA

40

Power to Reject H0
0.5 h
60

Fre que ncy , %

0 20

0.0

0.02 pKS

0.04

0 0.0

40

80

0.002 0.004 pKS

True Model

No Covariate Model

Not p-values, but fraction times p<0.05

Oct, 2003 Gobburu, FDA 41

Power to Reject H0
4.0 h

Fre que ncy , %

40

80

0 10 0.0

30

0.0

0.002 0.004 pKS

0.2 pKS

0.4

True Model

No Covariate Model

Not p-values, but fraction times p<0.05

Oct, 2003 Gobburu, FDA 42

When to use (s)PPC?

Potential uses
When informative priors used (& data are spare)
Even in ML, when some parameters are fixed

When model intended for designing trials

Model should at least regenerate the original data
Especially when distributions are truncated

Oct, 2003

Gobburu, FDA

43

When to use PPC?

Application to Drug Development
Gelman et al
Reaction times in (non)schizophrenics (Bayesian)

Gobburu et al
Absorption sub-models for an analgesic (ML) Influence of age on renal CL in neonates (Bayesian)

Publications/Presentations
Limited to visual inspection (ML)

Regulatory Submissions
Typically, 95% PI determined using yrep and compared to yobs
Oct, 2003 Gobburu, FDA 44

Conclusions
PPC: What, How and When? Test statistic versus Discrepancy variable Choice of test statistic important
Challenging

Interpretation of ppc, pks and peqv Uncertainty and Priors not considered Need for further research in PPC
Oct, 2003 Gobburu, FDA 45

O c tober 26-30, 2003 Salt Lake City, Utah