New Calibration and Validation

Standards for Travel Demand Modeling

presented to
12th TRB Transportation Planning Applications Conference, May
2009, Houston, TX

presented by
Robert G. Schiffer, AICP
Thomas F. Rossi
Cambridge Systematics, Inc.

Study prepared for

Florida Department of Transportation

January 12, 2009

Transportation leadership you can trust.
Presentation Overview

Background
Literature Review
Recommended Calibration and
Validation Guidelines and Standards
• LRTP Models with Transit
• Other Model Applications

Calibration and Validation

Best Practices
Guidelines for Model Application
Next Steps

2
Background

Follow-up to Phase I Study on model parameters

• Phase I Final report is available for downloading at
− http://www.fsutmsonline.net/images/uploads/mtf-files/FSUTMS-
Cube_Parameters.pdf

Phase II Study on calibration standards included four

subtasks
• Literature Review
• Model Calibration/Validation Guidelines and Standards
• Best Practices for Model Calibration/Validation
• Documentation – 1) Calibration and Validation Standards;
2) Best Practices for Model Validation; 3) Guidelines for
Model Application

3
Background (continued)

Calibration versus validation

− Calibration – process where models are adjusted to simulate
or match observed travel behavior in the study area
− Validation – procedure used to adjust models to simulate
base-year traffic counts and transit ridership figures
Standards versus guidelines/benchmarks
− Standards – desirable accuracy levels for comparing estimated
versus observed metrics
− Benchmarks – documented statistical ranges from literature
review, model outputs, NHTS, etc.
Purposes of validation process
− Level of comfort to planners, agency staff, and elected officials
− Evidence that model is accurate enough for specific application
− Accounts for errors in observed comparative data

4
Literature Review
Checklist of Available Validation Standards from Literature – Trip Generation

60+ documents
Statistic Standard Benchmark Document(s) Cited

Population/Employment Ratio 40-60% Iowa DOT Peer Review (39)

reviewed Person Trips/Person

Person Trips/Person (Urban)

3.64 – 3.87

2.54
Validation and Reasonableness (14)

University of Wisconsin (16): Kentucky Statewide

Model/NPTS

• Specific models Person Trips/Person (Rural) 2.57

University of Wisconsin (16): Kentucky Statewide
Model/NPTS

Person Trips/HH 8.5 – 10.5 University of Tennessee (59)

− Technical reports Person Trips/HH 6.8 – 12.4 Validation and Reasonableness (14), NCHRP 365 (15)

Calibration and Adjustment 6) – population sizes: 50-

Person Trips/DU 14.1/14.5/11.8/7.6

− Model outputs
100/100-250/250-750/750k+

NCHRP 365 (15) – population sizes: 50k-200k/200k-

Person Trips/DU 9.2/9.0/8.6/8.5
500k/500k-1M/1M+

• Reference reports Vehicle Trips/DU

Resident/Commercial
9.15 VTRC (29)

78.5%/21.5% VTRC (29)

Neighborhood Trips

− Federal Person Trips/Employee   1.29 – 1.40 Validation and Reasonableness (14)

agencies/TMIP TAZs/Population

Person Trips/TAZ
 

 
1 TAZ/1k Population

25k or less
Iowa DOT Peer Review (39)

Iowa DOT Peer Review (39)

− State DOT Percent Trips by Purpose – HBW*

Percent Trips by Purpose – HBNW

18% – 27%

47% – 54%
University of Tennessee (59)

University of Tennessee (59)

guidelines and Percent Trips by Purpose – NHB 22% – 31% University of Tennessee (59)

standards Percent Trips by Purpose – HBW* 17% – 23% Validation and Reasonableness (14), NCHRP 365 (15)

Percent Trips by Purpose – HBNW 52% – 60% Validation and Reasonableness (14), NCHRP 365 (15)

Prepared ranges of Percent Trips by Purpose – NHB

Unbalanced
 

0.90-1.10
23% – 25%

 
Validation and Reasonableness (14), NCHRP 365 (15)

Validation and Reasonableness (14)

acceptability
Attractions/Productions

5
Recommended Guidelines and Standards
Checking Input Data

Socioeconomic data
• Visual comparisons
• Statistical comparisons
• Regionwide comparisons
(below) County Census Data NERPM Data

− Persons per   2000 2003

Percent
Difference 2000 2005
Percent
Difference
DU (or HH) Duval 779,618 817,480 5% 762,674 810,493 6%

− Employment/ Clay 141,671 157,502 11% 139,036 167,020 20%

population ratio Nassau 57,903 61,625 6% 56,897 64,695 14%

St Johns 124,458 142,869 15% 120,738 150,084 24%

− Autos/DU (or HH) Total 1,103,650 1,179,476 7% 1,079,345 1,192,292 10%

Benchmarks/Settings

Statistic Low High

Regionwide Persons/DU (or HH) 2.0 2.7

Regionwide Employment/Population Ratio 0.45 0.75

Regionwide Autos/DU (or HH) 1.75 2.10

Approximate Population/TAZ 6
N/A 3,000
Recommended Guidelines and Standards
Checking Input Data (continued)

Highway network data

Transit network data

Highway and transit speed data

• Logical hierarchy
• Balance highway and transit

Terminal times
• Logical hierarchy
• Phase I Report

7
Recommended Guidelines and Standards
Trip Generation
Benchmarksa

Aggregate trip rates Statistic

Person Trips/TAZ
Low

N/A
High

15,000

Person Trips/Person 3.3 4.0

• Person trips/TAZ Person Trips/DU (or HH) 8.0 10.0

HBW Person Trips/Employee 1.20 1.55

• Person trips/person a
Generally excludes nonmotorized trips; including motorized trips could increase
person trips per DU up to 11.5.

• Person trips/DU (or HH)

• HBW person trips/employee

Total unbalanced attractions versus productions

by purpose
• Preferred +/-10%; acceptable in some instances +/-50%

Percent external-external trips by zone/station

• Great variation expected (4-21 percent range documented)

8
Recommended Guidelines and Standards
Trip Generation (continued)

Percent trips by purpose

Benchmarks
Statistic Low (Percent) High (Percent)
Percent Trips by Purpose – HBW 12 24
Percent Trips by Purpose – HBSH 10 20
Percent Trips by Purpose – HBSR 9 12
Percent Trips by Purpose – HBSC 5 8
Percent Trips by Purpose – HBOa 14 28
Percent Trips by Purpose – HBNWb 45 60
Percent Trips by Purpose – NHBc 20 33
a
HBO includes a variety of special trip purposes depending on the model (e.g., airport, college, and shop).
b
HBNW accounts for all home-based trip purposes except HBW.
c
NHB includes combined purposes for NHB Work and NHB Nonwork, where appropriate.

9
Recommended Guidelines and Standards
Trip Distribution
Benchmarks

Statistic Low High

Average trip length by purpose Average Trip Length – HBW (minutes) 12 35

Average Trip Length – HBSH (minutes) 9 19

Trip length frequency Average Trip Length – HBSR (minutes) 11 19

distributions by purpose Average Trip Length – HBSC (minutes) 7 16

Average Trip Length – HBOa (minutes) 8 20

Coincidence ratios by purpose – Average Trip Length – NHBb (minutes) 6 19

measures the percent of area Average Trip Length – IE (minutes) 26 58

that coincides for two trip Statistic Standards

Mean Trip Length, Observed Total Trips +/-3%

length frequencies Trip Length Frequency Distribution +/-5%
versus Observed
Percent of Total Trips
8% Coincidence Ratios by Purposec 70%
Coincidence Ratio = 0.82 a
HBO includes a variety of special trip purposes, depending
Estimated (ATL = 18.2 Min)
on the model (e.g., airport, college, and school).
6% b
NHB includes combined purposes for NHB Work and NHB
Observed (ATL = 18.9 Min)
Nonwork, where appropriate.
c
Some lower coincidence ratios have been deemed
4% acceptable for trip purposes that had relatively few trips
and therefore higher error rates.

2%

0%
0 10 20 30 40 50 60
Travel Time (in Minutes) 10
Recommended Guidelines and Standards
Trip Distribution (continued)

Percent intrazonal trips by purpose

Map-based (“node-point”) charts
• Zone-based
• Number of trips
• Trip productions/attractions
by purpose
Benchmarks

Statistic Low High

Percent Intrazonal – HBW 1% 4%

Percent Intrazonal – HBSH 3% 9%

Percent Intrazonal – HBSR 4% 10%

Percent Intrazonal – HBSC 10% 12%

Percent Intrazonal – HBOa 3% 7% a

HBO includes a variety of special trip purposes, depending
Percent Intrazonal – NHB b
5% 9% on the model (e.g., airport, college, and school).
b
NHB includes combined purposes for NHB Work and NHB
Percent Intrazonal – Total Trips 3% 5% Nonwork, where appropriate.
Standards

Statistic Acceptable Preferable

Percent Intrazonal, Observed Total Trips +/-3% +/-5%

11
Recommended Guidelines and Standards
Mode Choice
Zero-Vehicle One-Vehicle Two-Vehicle Three-Vehicle
Mode Households Households Households Households

Walk 5,000 6,000 4,000 3,000

Mode split targets (ideal) Bike 2,000 1,000 500 200

Drive Alone - 130,000 350,000 200,000

• Trip purpose Shared Ride 2 Persons 6,000 15,000 20,000 10,000

Shared Ride 3 Persons 1,000 2,000 4,000 2,000

• Mode Local Bus, Walk 6,000 7,000 4,000 1,000

Local Bus, PNR - 500

2,000 500
• Auto ownership level Local Bus, KNR - 200

Express Bus, Walk 1,000 1,000 1,000 500

• Geographic subarea Express Bus, PNR - 2,000 4,000 2,000

Express Bus, KNR - 200 500

LRT, Walk 500 1,000 400

LRT, PNR
- 300 500
LRT, KNR

Polk

2000 1990 Census 2000

Trips Percent of Trips Percent of Number Percent of

 Trip Allocation By Mode Trips Trips Trips

HBW

Drive Alone 244,414 79.69% 188,259 80.47% Drive Alone 84.14%

One Passenger 51,465 16.78% 38,212 16.33% Carpool 15.07%

Two+ Passenger 9,637 3.14% 7,023 3.00%    

Total Transit 1,206 0.39% 465 0.20% Transit 0.78%

12
Recommended Guidelines and Standards
Mode Choice (continued)

Mode splits by observed calibration targets

Total area transit trips, estimated
versus observed
Transit trips between districts
• Tabular comparisons (CTPP)
• Desire lines
Mean trip length, estimated transit trips versus observed
Standards

Statistic Low High

Total Area Transit Trips versus Observed +/-1% +/- 2%

Transit Trips between Districts Compare model trip table against CTPP or HH survey

Mean Trip Length Transit Trips versus Observed +/-5% +/-15%

Mode Splits by Observed Calibration Targets +/- 2% +/- 2%

Elasticity of Demand with Respect to LOS Variables -0.6 -0.1

13
Recommended Guidelines and Standards
Trip Assignment
Standards

Statistic Acceptable Preferable

Volume-over-count Freeway Volume-over-Count +/- 7% +/- 6%

ratios Arterial Volume-over-Count

Collector Volume-over-Count
+/- 15%

+/- 25%
+/- 10%

+/- 20%

Frontage Road Volume-over-Count +/- 25% +/- 25%

Freeway Peak Volume-over-Count 75% of links @ +/-20%; 50% of links @ +/-10%

+/-1 lane percent error Major Arterial Peak Volume-over-Count 75% of links @ +/-30%; 50% of links @ +/-15%

(calculated based on Assigned VMT-over-Count Areawide +/-5% +/-2%

FDOT LOS Handbook) Assigned VHT-over-Count Areawide

Assigned VMT-over-Count by FT/AT/NL

+/-5%

+/- 25%
+/-2%

+/- 15%

Assigned VHT-over-Count by FT/AT/NL +/- 25% +/- 15%

Aggregate VMT Standards

Statistic Acceptable Preferable

• VMT/HH (60-75) Percent Error – LT 10,000 volume (2L road) 50% 25%

Percent Error – 10,000-30,000 (4L road) 30% 20%

• VMT/person (24-32) Percent Error – 30,000-50,000 (6L road) 25% 15%

Percent Error – 50,000-65,000 (4-6L 20% 10%

freeway)
• VMT/commercial Percent Error – 65,000-75,000 (6L freeway) 15% 5%
vehicle (3-25%) Percent Error – GT 75,000 (8+L freeway) 10% 5%

14
Recommended Guidelines and Standards
Trip Assignment (continued)
Standards

Statistic Acceptable Preferable

Screenline volume-over-count RMSE – LT 5,000 AADT 150% 45%

RMSE – 5,000-9,999 AADT 45% 35%

• By volume and location RMSE – 10,000-14,999 AADT 35% 27%

RMSE – 15,000-19,999 AADT 35% 25%

RMSE by volume group RMSE – 20,000-29,999 AADT 27% 15%

RMSE – 30,000-49,999 AADT 25% 15%

Transit assignment validation RMSE – 50,000-59,999 AADT 20% 10%

RMSE – 60,000+ AADT 19% 10%

Benchmarks
RMSE Areawide 45% 35%
Statistic Low High

Estimated-over-Observed Transit Trips +/- 9% +/- 3%

Standards

Statistic Acceptable Preferable

Acceptable Error – Transit Screenlines +/-20% +/-10%

Transit Ridership – <1,000 Passengers/Day +/-150% +/- 100%

Transit Ridership – 1k-2k Passengers/Day +/- 100% +/- 65%

Transit Ridership – 2k-5k Passengers/Day +/- 65% +/- 35%

Transit Ridership – 5k-10k Passengers/Day +/- 35% +/- 25%

Transit Ridership – 10k-20k Passengers/Day +/- 25% +/- 20%

Transit Ridership – >20,000 Passengers/Day +/- 20% +/- 15%

15
Calibration and Validation Best Practices
Steps in Model Validation and Calibration Process

Iterative process
Secondary
Institutional
Initiate Inventory Data
Framework
Collection

Process must acknowledge Primary

Data
Data
Checking
Collection
• Availability of behavioral
data (or lack thereof) Model Model
Estimation Implementation

• Variability in data accuracy Iterate

Model Model
• Regional issues to be Calibration Validation NO

evaluated with the model

Model Satisfactory
• Need for future year Application Results?

sensitivity testing
YES

Must validate each step (i.e., Continual

Model
Maintenance,
not just assignment) Application

16
Calibration and Validation Best Practices
Guidance on Validation and Adjustment
Matching base year statistics is not
sufficient to say model is validated
Validation and Reasonableness Checks

Ideal combination of skills

Estimation Calibration Validation Application
• Local area knowledge

• Familiarity with sources for Iterate Iterate Iterate

transferable parameters

• Understanding of what
constitutes acceptable results

• Experience with cause and effect of

model adjustments

• Quality and availability of data

17
Calibration and Validation Best Practices
Special Validation Considerations
Statewide Model Statewide
Statewide Model Model
Requirements by Study
Area
Urban Study
study type Urban
Model
Study
Area
Urban
Model
Study
Area
Model Area
Area 2
Area Area 1 Urban
Model
• FTA New Starts Area

projects Study Area Study Areas Study Area

Within One Within Two Outside Urban
• Subarea and corridor Model Area Model Areas Model Areas

validation Example of Coding Penalties for HOV Lanes

• Site impact studies General Purpose Lanes

HOV Lanes

Other validation practices General Purpose Lanes

• Adjustment of congested Example of Coding Prohibitors for Proper Access

speeds Would not

cross traffic to
• Use of special generators take loop ramp
Cannot cross
median to access
• Transferable parameters driveway (centroid
connector)

• Impact of new paradigms Red lines represent penalized or prohibited movements.

18
Guidelines for Model Applications
Stability of Model Parameters
DeKalb County Comparisons of Daily VMT
Vehicle Miles Traveled (1000s)
Model Year 0 5,000 10,000 15,000 20,000 25,000 30,000

Refined 2000 19,099

Static versus dynamic Refined 2030 E+C

Comprehensive Moderate Existing (A)
Comprehensive Moderate Focused (B)
24,255
23,982
23,855

parameter settings
Comprehensive High Existing (C) 29,524
Comprehensive High Focused (D) 29,833
Trends Moderate Existing Pattern (E) 24,046
Trends High Existing Pattern (F) 29,743
New Visions Moderate Focused (G) 24,520

• Generally “locked down” New Visions High Focused (H)

Final w/Moderate & Focused Growth 23,872
30,628

during validation DeKalb County Comparisons of PM Peak Period V/C Ratio

PM Peak Period Volume-to-Capacity Ratio

• Limited trend data to back Model Year

Refined 2000
0.0 0.1 0.2 0.3 0.4

0.60
0.5 0.6 0.7 0.8 0.9 1.0

up values for future years Refined 2030 E+C

Comprehensive Moderate Existing (A)
Comprehensive Moderate Focused
0.59
0.77

0.59
Comprehensive High Existing (C) 0.77

History has shown changes

Comprehensive High Focused (D) 0.78
Trends Moderate Existing Pattern (E) 0.61
Trends High Existing Pattern (F) 0.79

in…
New Visions Moderate Focused (G) 0.63
New Visions High Focused (H) 0.83
Final w/Moderate & Focused Growth 0.60

• Nonhome-based trip rates DeKalb County Comparisons of Average Travel Time

Average Travel Time (in minutes)

Model Year 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

• Trip length frequencies Refined 2000

Refined 2030 E+C
19.0
26.2
Comprehensive Moderate Existing (A) 20.6

• Auto occupancy rates

Comprehensive Moderate Focused 20.9
Comprehensive High Existing (C) 22.7
Comprehensive High Focused (D) 24.2
Trends Moderate Existing Pattern (E) 21.1
Trends High Existing Pattern (F) 23.8

• BPR curves New Visions Moderate Focused (G)

New Visions High Focused (H)
21.0
22.1
26.1
Final w/Moderate & Focused Growth

19
Guidelines for Model Applications
Typical Model Applications and Guidelines
Developed list of • MPO LRTP Updates
14 different • Comprehensive Plans
• SIS/FIHS Planning
model
• Campus Master Plans
application types • Concurrency Applications
and relevant • Development of Regional Impacts (DRIs)
model guidance • Congestion Management Systems
unique to each • Air Quality and Climate Change
(e.g., forecasting • Corridor Studies
external trips for  Corridor Feasibility Studies;
MPO LRTP  FTA New Starts/Small Starts
Updates, etc.) Applications;
 Project Development and Environment
(PD&E) Studies;
Acknowledge  Interstate Master Plans;
uncertainty in  Interchange Justification/Modification
forecasting Reports (IJR/IMR); and
process  Toll Feasibility Studies.

20
Guidelines for Model Applications
Model Application Checks

 Review logic of demographic forecasts at region

and subarea
 Generate color-coded plots of highway network
characteristics
 Compare base and future year trip productions and
attractions by purpose at the regional and subarea level
 Compare base and future year trip distribution patterns
 Review logic of changes in mode splits resulting from
scenario testing that would seemingly benefit one mode
over another
 Compare traffic estimates on specific corridors and
screenlines between base and future years and build
and no-build conditions

21
Next Steps
FDOT is conducting peer review of Final Report

Validation checklist and new standards being tested to

• Review/comment on recent validation studies

• Ratchet up expectations for ongoing validation studies

Acceptable Range of Values
Accuracy Recommended Comparisons and
Model Step Model Statistic to Evaluate Low High Standard Calculation Methods/Comments

Input Data Socioeconomic Data Cube, GIS Visual and Statistical Document checks for households and
Comparisons/Checks employment

  Persons/DU (or HH) 2.00 2.70 N/A NHTS > 2.46 FL – 2.59 U.S.

  Employment/Population Ratio 0.35 0.75 N/A  

  Autos/DU (or HH) 1.75 2.10 N/A  

  Approximate Population/TAZ N/A 3,000 N/A Recommendations from TAZ White

Paper

  Highway Network Data Cube, GIS visual and statistical Check hwy network, prohibitors, tolls,
comparisons/checks paths

  Highway Speed Data ensure logical hierarchy by AT/FT/NL; survey Will provide acceptable ranges in sep.
chk table

  Transit Network Data chk access links; chk routing against GIS data Checks for transit network, access,
paths

22