Calculating Statistics:

Concentration Related
Performance Goals
James W. Boylan
Georgia Department of Natural Resources

PM Model Performance Workshop

Chapel Hill, NC
February 11, 2004

Outline
Performance Statistic
Standard Bias and Error Calculations

Model Performance Goals for PM

Speciated Bias and Error Goals
Relative Proportions Goals

Performance Metrics

Equation

1
MB
N
1
ME
N

Mean Bias (g/m3)

Mean Error (g/m3)
Mean Normalized Bias
(%)
(-100% to +)
Mean Normalized Error
(%)
(0% to +)
Normalized Mean Bias
(%)
(-100% to +)
Normalized Mean Error
(%)
(0% to +)
Mean Fractional Bias (%)
(-200% to +200%)
Mean Fractional Error (%)
(0% to +200%)

C
i 1
N

C
i 1

1 N C m Co

MNB
N i 1
Co
N

NMB

C
i 1

1
MFB
N

i 1

Co

1
MNE
N

i 1

C m Co
Co

C
i 1

C
i 1

1
MFE
N

Co

Co C m

NME

C m Co

Co

Co

i 1

C m Co

i 1 Co C m

Example

North Carolina 77
Georgia Tech 88
GT showed a positive bias of 11 points
NB = 14.3%
FB = 13.3%

Performance Metrics
Mean Normalized Bias and Error
Usually associated with observation-based minimum
threshold
Some components of PM can be very small making it
difficult to set a reasonable minimum threshold value
without excluding a majority of the data points

Without a minimum threshold, very large normalized

biases and errors can result when observations are
close to zero even though the absolute biases and
errors are very small
A few data points can dominate the metric

Overestimations are weighted more than equivalent

underestimations

Performance Metrics
Normalized Mean Bias and Error
Biased towards overestimations

Mean Fractional Bias and Error

Bounds maximum bias and error
Gives additional weight to underestimations and
less weight to overestimations

Example Calculations
Model
g/m3
)

Obs.
g/m3
)

MB

NMB
(%)

g/m3
)

0.05

1.0

-0.95

-95

-180.95

+0.95

+95

+180.95

1.0

0.05

+0.95

+1900

+180.95

+0.95

+1900

+180.95

1.0

0.01

+0.99

+9900

+196.04

+0.99

+9900

+196.04

0.683

0.353

+0.33

+3901.7

+65.3

0.96

3965.0

186.0

+93.4

MNB
(%)

MFB
(%)

ME
g/m3
)

NME
(%)

272.9

MNE
(%)

Mean Normalized Bias and Error

Most biased and least useful of the three metrics

Normalized Mean Bias and Error

Mean Fractional Bias and Error
Least biased and most useful of the three metrics

MFE
(%)

Speci
es

SAMI Model Performance

Summary

#
Obs

Mea
n
g/m

MB
g/m

NMB
(%)

MNB
(%)

MFB
(%)

ME
g/m

NME
(%)

MNE
(%)

MFE
(%)

SO4

134

6.71

-1.16

-17.3

1.1

-22.7

2.48

37.0

55.1

50.2

NO3

134

0.63

-0.30

-47.6

6.8

-73.6

0.52

81.8

112.8

107.2

NH4

134

2.70

-1.25

-46.4

-27.4

-57.4

1.43

53.1

61.6

70.0

NH4
Bi

134

1.44

0.01

0.4

34.2

-2.6

0.62

42.9

70.4

44.4

ORG

132

3.41

-0.27

-7.8

15.8

-6.0

1.37

40.4

53.8

43.9

EC

132

0.56

-0.05

-8.6

15.1

-12.7

0.27

48.3

61.9

50.4

Soils

135

0.55

0.25

46.2

171.6

21.9

0.57

102.9

207.4

72.5

PM2.5

130

17.05

-4.79

-28.1

-9.1

-28.8

6.8

39.8

48.9

47.6

PM10

130

23.44

-5.21

-22.2

-6.2

-21.0

9.18

39.1

44.2

43.5

PMC

126

6.98

-0.48

-6.9

43.9

7.8

3.86

55.2

78.7

54.1

bext

132

133.1

-27.91

-21.0

-10.2

-23.7

43.70

32.8

40.0

40.4

Proposed Performance Goals

Based on Mean Fractional Error (MFE) and Mean
Fractional Bias (MFB) calculations
Performance goals should vary as a function of
species concentrations
More abundant species should have a MFE +50% and
MFB 30%
Less abundant species should have less stringent
performance goals

Goals should be continuous functions with the

features of:
Asymptotically approaching +50% MFE and 30% MFB
when the concentrations (mean of the observed and
modeled concentrations) are greater than 2.5 g/m3

Approaching +200% MFE and 200% MFB when the

concentrations (mean of the observed and modeled
concentrations) are extremely small

Proposed Mean Fractional

Error and Bias Goals
0.5 ( Co C m )

MFE 150e

0.75 g / m 3

50

0.5 ( Co C m )

MFB 170e

0.5 g / m 3

30

Example Calculations
Species X

Model
g/m3)

Obs.
g/m3)

FB (%)

FE (%)

Day 1 Site A

2.0

1.0

+66.7

+66.7

Day 1 Site B

1.0

2.0

-66.7

+66.7

Day 2 Site A

1.0

0.4

+85.7

+85.7

Day 2 Site B

0.5

1.5

-100.0%

+100.0%

Average

1.125

1.225

-3.6%

79.8%

Average CO + CM = 0.5*(1.125 + 1.225) = 1.175

MFE performance goal for Species X = 81.3%
MFB performance goal for Species X = 46.2%

Mean Fractional Error

Mean Fractional Error Goal

Speciated Fine PM Performance

200
150
100
50
0
0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

Average Concentration (g/m3)

8.0

Mean Fractional Bias

Mean Fractional Bias Goal

Speciated Fine PM Performance

200
100
0
-100
-200
0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

Average Concentration (g/m3)

8.0

Mean Fractional Error

SAMI 6 Episodes
Speciated Fine PM Performance

200

Nitrate
Soils

150
Elem. Carbon
Ammonium (Sulfate)

100

Organics

PMC

PM2.5

PM10

50
Ammonium (Bisulfate)

Sulfate

0
0.0

5.0

10.0

15.0

20.0

Average Concentration (g/m3)

25.0

Mean Fractional Bias

SAMI 6 Episodes
Speciated Fine PM Performance

200

Soils
Elem. Carbon

100

Ammonium (Bisulfate)
PMC

PM2.5

PM10

0
Organics

-100

Sulfate

Ammonium (Sulfate)
Nitrate

-200
0.0

5.0

10.0

15.0

20.0

Average Concentration (g/m3)

25.0

Mean Fractional Error

VISTAS July 1999 Episode

Speciated Fine PM Performance

200

Nitrate
Soils

150
Organics
Elem. Carbon

100

PM2.5

PM10

PMC

50
Sulfate

Ammonium (Sulfate)

0
0.0

5.0

10.0

15.0

20.0

Average Concentration (g/m3)

25.0

Mean Fractional Bias

VISTAS July 1999 Episode

Speciated Fine PM Performance

200

Elem. Carbon
PM2.5

Soils

100

PM10

Ammonium (Sulfate)

0
Sulfate

-100

PMC
Nitrate

Organics

-200
0.0

5.0

10.0

15.0

20.0

Average Concentration (g/m3)

25.0

Mean Fractional Error

VISTAS January 2002 Episode

Speciated Fine PM Performance

200

Soils

150
Nitrate
Elem. Carbon

100

PM2.5

PMC

50

PM10

Organics
Ammonium (Sulfate)

0
0.0

2.0

4.0

Sulfate

6.0

8.0

10.0

Average Concentration (g/m3)

12.0

Mean Fractional Bias

VISTAS January 2002 Episode

Speciated Fine PM Performance

200

Soils
PM2.5

Elem. Carbon

100

Ammonium (Sulfate)

PM10

Organics

0
Sulfate

-100
Nitrate

PMC

2.0

4.0

-200
0.0

6.0

8.0

10.0

Average Concentration (g/m3)

12.0

Relative Proportions (RP) PERF

Goals
EPA draft guidance (2001)
For major components (i.e., those observed to
comprise at least 30% of measured PM2.5), we
propose that the relative proportion predicted for
each component averaged over modeled days with
monitored data agrees within about 20% of the
averaged observed proportion. For minor observed
components of PM, we suggest a goal that the
observed and modeled absolute proportion of each
minor component agree within 5%.

1 N
Bias
N i 1

N C
Cm component
o component
1

C
N
Co Total

1
i
m Total

0.2 RP%

(5%)

Example Calculation
Calculating component proportions based on
concentrations averaged over multiple days
can hide poor model performance
Observed RP (%)

Modeled RF (%)

Day 1

50%

95%

Day 2

50%

95%

Day 3

50%

5%

Day 4

50%

5%

Average

50%

50%

Relative Proportions for SAMI

4.2% 4.1%

4.3%

4.7%

SO4

6.8%

ORG

2.8%

NH4
10.8%

NO3

12.3%
47.1%

50.5%

25.6%

Observed

EC
Soils

26.7%

Simulated

Relative Proportions for SAMI

Relative Proportions Evaluation
15

Elem. Carbon

Bias (%)

10

Soils

Nitrate

Organics

Sulfate

Ammonium (Bisulfate)

-5
-10
-15

10

20

30

40

50

Relative Proportion of Fine PM (%)

60

Proposed Relative Proportions

Performance Goals
Propose to use an equation that accounts for the
day-to-day variability of species relative
proportions:

1
Error
N

Cmcomponent

i 1

Cm

Total

RP 30%, Error 10%

RP 15%, Error 5%
RP 15% - 30%, Error [RP]/3

Cocomponent
Co

Total

Proposed Relative Proportions

Performance Goals
Relative Proportions Evaluation

Error (%)

15
10

Soils

Nitrate

Organics
Sulfate

Ammonium (Bisulfate)
Elem. Carbon

10

20

30

40

50

Relative Proportion of Fine PM (%)

60

Concluding Remarks
Recommended performance values are
model goals, not model criteria
Failure to meet proposed performance goals
should not prohibit the modeling from being
used for regulatory purposes
Help identify areas that can be improved upon
in future modeling

If performing episodic modeling,

performance evaluation should be done
on an episode-by-episode basis
If performing annual modeling,
performance evaluation should be done
on a month-by-month basis

Concluding Remarks (cont.)

As models mature, performance goals can
be made more restrictive by simply:
Adjusting the coefficients in the MFE and MFB
goal equations
Lowering the relative proportion error goals

Q: Is there a need for performance goals

for gaseous precursors or wet deposition
species?
One-atmosphere modeling system
If not, still should be evaluated to help identify
potential problems with PM model performance

Questions?