Meteorol. Appl. 9, 307–315 (2002) DOI:10.

1017/S1350482702003043

Decision-making from probability forecasts based

on forecast value
Kenneth R. Mylne The Met Office, London Road, Bracknell, Berks RG12 2SZ UK

A method of estimating the economic value of weather forecasts for decision-making is described. This
method has recently been used for user-oriented verification of probability forecasts, but is here applied
to aid forecast users in optimising their decision-making from probability forecasts. Value may be
calculated in the same way for either probability forecasts or deterministic forecasts, and thus provides
the user with a direct comparison of the value of each in terms of money saved, which is more relevant
than most standard verification scores. The method is illustrated using site-specific probability forecasts
generated from the ECMWF ensemble prediction system and deterministic forecasts from the ECMWF
high-resolution global model. It is found that for most forecast events and for most users the probability
forecasts have greater value than the deterministic forecasts from a higher resolution model.

1. Introduction Forecasts (ECMWF) (Molteni et al. 1996; Buizza and

A weather forecast, however skilful, has no intrinsic
value unless it can be used to make decisions that bring
some benefit to the end user. Conventionally, a
weather forecast provider supplies the user with a best
estimate of whether a defined event will occur (e.g.
whether wind speed will or will not exceed 15 ms–1), or
of a value for a measurable parameter (e.g. maximum
wind speed = 18 ms–1). Decision-making is often based
on whether a defined event is expected to occur or not.
For example, the owner of a small boat may decide to
seek shelter when the forecast wind speed exceeds
15 ms–1.
The nature of atmospheric predictability is such that
there is frequently a significant uncertainty associated
with such deterministic (best estimate) forecasts.
Forecast uncertainty can be expressed in many ways,
either qualitatively or quantitatively, and where pro-
vided this information can aid the decision-maker who
understands the impact of a wrong decision. However,
uncertainty is most commonly estimated subjectively;
such estimates are often inconsistent, and may be
affected by factors such as the forecaster ‘erring on the
safe side’, which may not lead to optimal decision-mak-
ing for specific users. Over the years several objective
methods of generating probability forecasts have been
developed to take account of uncertainty. Probability
forecasts have long been generated using MOS (model
output statistics) where a deterministic model forecast
is augmented with a probability distribution based on a
previously observed distribution of model errors (e.g.
Glahn and Lowry 1972; Carter et al. 1989). More
recently there has been rapid development of ensemble
prediction systems (EPS) such as those operated by the
European Centre for Medium Range Weather
Palmer 1998) and the US National Centers for
Environmental Prediction (NCEP) (Toth and Kalnay
1993). Output from an EPS is normally in the form of
probability forecasts, and there is growing evidence
(e.g. Molteni et al. 1996; Toth et al. 1997) that these
have greater skill than equivalent deterministic fore-
casts based on single high-resolution model forecasts,
particularly on medium range time-scales. To make use
of the additional information content of probability
forecasts, the decision-maker needs to know how to
respond to a forecast such as ‘There is a 30% probabil-
ity that the wind speed will exceed 15 ms–1.’ This paper
describes a technique which estimates the economic
value of a probability forecast system based on verifi-
cation of past performance, and uses it to determine a
user’s best decision-making strategy. The value of
deterministic forecasts can be calculated in the same
way, and this allows a direct comparison of the utility
of probability and equivalent deterministic forecasts in
terms which are relevant to the user. The method is
illustrated using probability forecasts generated from
the ECMWF EPS and comparing them with forecasts
from the ECMWF high-resolution deterministic
model.
2. Background to probability forecasts based on
ensembles
Uncertainty in numerical weather forecasts derives
from a number of sources, in particular uncertainty in
the analysis of the initial state of the atmosphere and
approximations in the model used to predict the future
evolution. Errors in the analysis result from observa-
tional errors, shortage of observations in some regions
of the globe, and limitations of the data assimilation

307

Kenneth R. Mylne
system. Model errors are due to essential approxima-
tions in the formulation of a model, notably the many
small-scale processes which cannot be resolved explic-
itly, and whose effect must be approximated by para-
meterisation. The non-linear nature of atmospheric
evolution means that small errors in the model repre-
sentation of the atmospheric state, whether due to the
analysis or the model, will be amplified through the
course of a forecast and can result in large errors in the
forecast. This sensitivity was first recognised by
Lorenz (1963), and was influential in the development
of chaos theory (Lorenz 1993). Gross errors in synop-
tic-scale evolution are common in medium-range fore-
casts (over 3 days), but can occasionally occur at less
than 24 hours. Errors in the detail of a forecast, such as
precipitation amounts or locations, are common even
in short-range forecasts. Ensemble prediction systems
have been developed in an attempt to estimate the
probability density function (pdf) of forecast solutions
by sampling the uncertainty in the analysis and running
a number of forecasts from perturbed analyses
(Molteni et al. 1996; Toth and Kalnay 1993). In more
recent developments, Buizza et al. (1999) have included
some allowance for model errors in the ECMWF EPS,
by adding stochastic perturbations to the effects of
model physics. Houtekamer et al. (1996) describe an
ensemble which accounts for both model errors and
analysis errors by using a range of perturbations in
both model formulation and analysis cycles. With any
of these ensemble systems, probability forecasts may be
generated by interpreting the proportion of ensemble
members predicting an event as an estimate of the
probability of that event.
A range of standard verification diagnostics are used to
assess the skill of probability forecasts. For example,
the Brier Score (see Wilks 1995), originally introduced
by Brier (1950), is essentially the mean square error for
probability forecasts of an event. ROC (relative operat-
ing characteristics), described by Stanski et al. (1989),
measures the skill of a forecast in predicting an event in
terms of hit rates and false alarm rates. Rank histograms
(Hamill and Colucci 1997, 1998) measure the extent to
which the ensemble spread covers the forecast uncer-
tainty, and can also reveal biases in the ensemble fore-
casts. However, while these diagnostics are useful to
scientists developing ensemble systems, they are of lit-
tle relevance to most forecast users. They do not tell
users how valuable the forecasts will be for their appli-
cations, nor do they answer the question of how to use
probability forecasts for decision-making.
3. Cost-loss models and economic value of
forecasts
An overview of techniques for estimating the economic
value of weather forecasts is given by Murphy (1994),
and a comprehensive review by Katz and Murphy
(1997). The method applied in this study is closely
308
14698080, 2002, 3, Downloaded from https://rmets.onlinelibrary.wiley.com/doi/10.1017/S1350482702003043 by Readcube (Labtiva Inc.), Wiley Online Library on [15/12/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
related to ROC verification, and is based on the cost-
loss situation described by Murphy (1977). It has
recently been discussed by Richardson (2000), and
is rapidly becoming accepted as a valuable tool for
user-oriented verification of probability forecasts
(e.g. Thornes and Stephenson 2001). The method
has also been applied to seasonal forecasts by Graham
et al. (2000). The aim of this paper is to present the
method in a way in which forecast providers and users,
working together, can optimise decision-making even
where the forecast probabilities may be significantly in
error.
The concept behind forecast value is that forecasts only
have value if a user takes action as a result, and the
action saves the user money. Calculation of value for
predictions of a defined event therefore requires infor-
mation on (a) the ability of the forecast system to pre-
dict the event, and (b) the user’s costs and losses asso-
ciated with the various possible forecast outcomes.
Consequently, the value depends on the application as
well as on the skill of the forecast.
3.1 Ability of the forecast system
ROC verification and the cost-loss situation are based
on categorical forecasts of a binary event, against which
a user can take protective action when the event is
expected to occur. For such an event, the ability of a
deterministic forecast system is fully specified by the 22
contingency table shown in Table 1, where h, m, f and
r give the relative frequencies of occurrence of each
possible forecast outcome.
Table 1. Contingency table of forecast performance.
Upper-case letters H, M, F, R represent the total
numbers of occurrences of each contingency, while the
lower-case letters in brackets represent the relative
frequencies of occurrences.
Event Forecast
Yes – User No – No
Protects Protective
Action
Yes Hit H Miss M
(h) (m)
Events
No False Correct
Observed
Alarm F Rejection R
(f) (r)
3.2 User costs and losses
For a user making decisions based on forecasts, each of
the four outcomes in Table 1 has an expected cost, or
loss, as given in Table 2. For convenience, it is normal
to measure all costs and losses relative to the user’s

costs for a Correct Rejection, so the ‘Normal Loss’ N
is set to zero. (Note: this assumption is not necessary,
and the method readily accounts for non-zero values of
N.) If the event occurs with no protective action being
taken, the user incurs a loss L. If the event is forecast to
occur, the user is assumed to take protective action at
cost C. In the simple cost-loss situation (Murphy
1977), this action is assumed to give full protection if
the event does occur, so the user incurs the cost C for
both Hits and False Alarms (Lm=C). In reality, protec-
tion will often not be fully effective in preventing all
losses, and the losses may be generalised by specifying
a Mitigated Loss Lm for Hits, as in Table 2.
Table 2. Contingency table of generalised user costs
and losses. Note: in the simple cost-loss situation
described by Murphy (1977), this is simplified such that
Lm = C.
Event Forecast
Yes – User No – No
Protects Protective
Action
Yes Mitigated Loss L
Loss Lm
Events
Observed No Cost C Normal
Loss N=0
For a forecast to have value it is necessary that Lm<L.
In most circumstances it would be expected that
C≤ Lm<L, but it is possible that in some circum-
stances Lm<C. For example, protective action could
involve using an alternative process which works effec-
tively in the weather conditions specified by the event,
but does not work in the non-event conditions; in this
case the cost C of a False Alarm would be high com-
pared to Lm.
The above examples assume that costs, losses and fore-
cast value are specified in monetary terms. They could,
instead, be calculated in terms of a user’s energy con-
sumption – the concept is the same. One limitation of
the system comes where a forecast is used to protect
life, owing to the difficulty in placing a cost on lives
lost.
4. Calculation of forecast value
Forecast value will be defined first for a simple deter-
ministic forecast, and the generalisation to probability
forecasts will be considered in more detail in the next
section. Value is defined as the saving a user can make
by using a forecast, so it is necessary to evaluate the
user’s mean expense, averaged over many occasions,
with and without the benefit of the forecast.
14698080, 2002, 3, Downloaded from https://rmets.onlinelibrary.wiley.com/doi/10.1017/S1350482702003043 by Readcube (Labtiva Inc.), Wiley Online Library on [15/12/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Decision-making from probability forecasts
4.1 Mean expense following forecasts
Given the information specified in Tables 1 and 2, and
assuming the user takes protective action whenever the
event is forecast, it can be expected that over a number
of forecast occasions the user will experience a mean
expense Efx of
Efx = hLm + mL + fC + rN (1)
The last term rN is normally zero, but this specifies the
generalisation to any definition of N.
4.2 Climatological reference expense
Forecast value represents the saving the user gains by
using the forecasts, and therefore requires a baseline
reference expense for comparison with Efx. If no fore-
cast is available the user has two options: either always
protect or never protect. In Murphy’s (1977) simple
cost-loss situation, where Lm=C, these options will
incur mean expenses Ecl over many occasions of C or
ōL respectively, where ō is the climatological frequency
of occurrence of the event. The user’s best choice is
always to protect if C<ō L, or C/L<ō and never to pro-
tect otherwise. Assuming the user takes the best option,
the mean climatological expense in the simple cost-loss
situation is thus given by
Ecl = min(C,ōL) (2)
For the generalised user loss matrix given in Table 2,
the mean expense of the ‘always protect’ option is given
by (1–ō)C + ōLm, and the mean expense of following
climatology is given by
Ecl = min((1–ō)C + ōLm , ōL) (3)
Fully generalising this to allow for the possibility of
N≠0 gives
Ecl = min((1–ō)C + ōLm ,(1 – ō)N + ōL) (4)
In this case the user’s best strategy is always to take
protective action if
C–N
a= < ō (5)
L + C – Lm – N
where α is the generalised cost/loss ratio introduced by
Richardson (2000). In some circumstances one of the
climatological options may not be viable for a user,
since taking protective action may involve stopping
doing their normal economic activity (e.g. a fisherman’s
protective action against strong winds may be to leave
his boat in port). The user cannot do that all the time or
he would go out of business. In this case the forecast
value should be calculated using the viable climate
option. This will be considered further in section 5.3.
309
 14698080, 2002, 3, Downloaded from https://rmets.onlinelibrary.wiley.com/doi/10.1017/S1350482702003043 by Readcube (Labtiva Inc.), Wiley Online Library on [15/12/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Kenneth R. Mylne
Fig. Fig.
1a 1b

Figure 1. Examples of relative forecast value Vr, expressed as a percentage of the value of a perfect forecast, plotted against prob-
ability threshold pt for probability forecasts of an event. The values of equivalent deterministic forecasts are shown in the
columns at the right-hand side of each graph. The best climatological option, selected following equation (4), is labelled above
the graph. Details of the forecasts used are given in section 5. Forecast lead-times are: 48h (solid line), 96h (dotted), 144h (dot-
dot-dot-dash) and 192h (dashed). The three graphs are for the same forecasts, but for different user loss functions as given in
Table 3: (a) A; (b) B; and (c) C. (Note that values for deterministic forecasts at 144h and 192h in Figure 1c are missing because
they are off the graph below –40%.)

4.3 Definition of forecast value
The value of the forecast in monetary terms, V, is the
saving the user can expect to make from following the
forecast, averaged over a number of occasions:
V = Ecl – Efx
This basic definition of value is the most relevant for a
user, except that it does not account for the cost of buy-
ing the forecast service, Cfx. The true value to the user
is therefore
service though, a user will want to consider Cfx, and a
forecast provider may use equation (7) to place an
upper bound on Cfx.
For general assessments of forecast value, it is conve-
nient to scale V relative to the value of a perfect fore-
(6) cast, in a similar fashion to the standard definition of a
skill score (see Wilks 1995). With perfect forecasts
m=f=0 and the user takes protective action only when
the event occurs. The mean expense is therefore Ep =
ōLm (or if N≠0, then Ep = ōLm + (1 – ō)N.) The relative
economic value of a forecast is then defined as:

Vu = V – Cfx = Ecl – Efx – Cfx (6) Ecl – Efx

Vr = (8)
Ecl – Ep
However, Cfx depends on the pricing policy of the
provider and cannot be estimated in general terms, so as a maximum of 1 (or 100%) for a perfect forecast sys-
for the purposes of this paper this term will be ignored tem, and is zero for a climatology forecast.
and value defined as in equation (6). In considering any

310

Fig.
1c
While Vr has a maximum of 1, there is no lower limit,
and from equations (1) and (8) it is clear that negative
values are likely when m or f, or their corresponding
user losses Lm or C, are large. Negative value simply
indicates that the forecast system does not have suffi-
cient skill to provide value to this particular user. In
this case the user’s best strategy is to follow the best cli-
matological option.
5. Decision-making and value for probability
forecasts
Many forecast users are used to making decisions from
deterministic forecasts, which give them a simple yes/no
answer on whether to take protective action against the
weather, and are uncertain how to respond to a proba-
bility forecast. Given the cost-loss information in Table
2, the probability level at which protection should be
taken can be easily related to the relative cost and bene-
fit of protection. In the simple cost-loss situation (Lm =
C), the user’s best strategy should be to protect when-
ever the probability of the event exceeds C/L; in the
more general cost-loss situation, the threshold is α, as
defined in equation (5). However, this assumes that the
forecasts perfectly represent the true probability of the
event. In reality probability forecast systems do not
14698080, 2002, 3, Downloaded from https://rmets.onlinelibrary.wiley.com/doi/10.1017/S1350482702003043 by Readcube (Labtiva Inc.), Wiley Online Library on [15/12/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Decision-making from probability forecasts
Table 3. User costs and losses in arbitrary monetary
units for the three examples of forecast value plots
shown in Figure 1. (For the meaning of the column
headings, see Tables 1 and 2.)
User N Lm L C
(CR) (H) (M) (FA)
A 0 1 5 1
B 0 1 2 1
C 0 2 10 1
produce perfectly reliable forecasts, so it would be use-
ful to find a way of identifying the ideal probability
threshold for real, imperfect probability forecasts.
Forecast value is defined above for deterministic fore-
casts. To make decisions from probability forecasts, the
user takes protective action (‘Forecast=Yes’ in Table 1)
when the probability p of the event exceeds a chosen
threshold pt. The value of the forecast therefore
depends on the choice of pt. To specify completely the
value of a probability forecast system, value must be
calculated for a range of probability thresholds and
plotted as a function of pt as shown in Figure 1. (This
use of probability thresholds is identical to that used in
ROC verification, for which hit rates (HR) and false
alarm rates (FAR) are calculated for various probability
thresholds from the contingency table in Table 1. The
appendix describes how forecast value may be evalu-
ated immediately for any forecast system for which
ROC verification is available.)
Three examples of forecast value plots are shown in
Figure 1. All are based on verification of the same fore-
casts, but the user costs and losses are different, as
shown in Table 3. The verification is based on daily
forecasts of the event ‘10 m wind speed of Beaufort
Force 5 or more’ for 41 sites in the UK, over two win-
ter seasons (December, January, February, 1998/99 and
1999/2000). These forecasts were generated and veri-
fied by the Met Office’s PREVIN system, which is
described in detail by Legg et al. (2001). Probability
forecasts were taken from the 51-member ECMWF
EPS; for comparison, equivalent deterministic forecasts
from the ECMWF high resolution (TL319) model are
also included. Results are shown for forecast lead-times
of 48, 96, 144 and 192 hours. The value of the probabil-
ity forecasts is calculated and plotted at probability
thresholds of 0, 10, 20, 30, …, 90, 100%.
5.1 Value of probability forecasts
From Figure 1a it can be seen that the value of a prob-
ability forecast is strongly dependent on pt. (The form
of this function is identical for V and Vr, since the scal-
ing of V by Ecl-Ep in equation (8) is independent of pt.)
The user’s requirement is to get maximum value from
the system, and therefore the practical value of the
311
 14698080, 2002, 3, Downloaded from https://rmets.onlinelibrary.wiley.com/doi/10.1017/S1350482702003043 by Readcube (Labtiva Inc.), Wiley Online Library on [15/12/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Kenneth R. Mylne
probability forecasts is given by the maximum of the
curve at pt= pmax. By using pmax as a decision threshold,
the user can optimise future decision-making from the
forecast system (provided only that past performance is
representative of future performance); for an imperfect
system pmax may be different from the probability
threshold expected from α, but will give the user the
maximum benefit from the available forecast informa-
tion. The form of the graphs in Figure 1 allows easy
identification of pmax for a particular user, and also
allows direct comparison of the value of the probabil-
ity forecast Vr(pmax) with the value of an equivalent
deterministic forecast.

For User A (Figure 1a), α=C/L=0.2 so pmax should be

20% for perfectly reliable forecasts. In practice, the
value is a maximum for probability thresholds of 20%
for lead-times of 144h and 192h, but 30% at 48h and
96h. The value falls negative for large thresholds.

For User B (Figure 1b), α=C/L=0.5 but maximum

value is obtained with pt of 70–80%. The only differ-
ence between users A and B is that L for a missed event
is greater for A. User A therefore benefits more by
lowering the miss rate, which is achieved by reducing pt
and thus increasing the number of occasions when pro-
tective action is taken. For User B, protective action is
relatively expensive so protection is only advisable
when the probability of the event is quite high.
A reliability diagram for the probability forecasts used
in Figure 1 is shown in Figure 2. Forecasts are cate-
gorised by the forecast probability, and the frequency
of occurrence of the event for each category plotted
against the forecast probability. For a perfectly reliable
system the points will lie along the diagonal line, but it
can be seen that for these forecasts the slope is rather
less than 45°, so that for high forecast probabilities the
event occurs less frequently than predicted. It is clearly
seen that observed frequencies of 50% occur for fore-
cast probabilities of 70–80%, in agreement with pmax
for User B. Similarly for User A, for lead-times of 48h
and 96h, the observed frequency is close to 20% when
the forecast probability is 30%. A representative relia-
bility diagram such as that in Figure 2 can be used to
calibrate forecast probabilities and produce near-per-
fectly reliable forecasts. Zhu et al. (2001) recommend
the use of calibrated probabilities so that the ideal deci-
sion threshold can be deduced directly from C/L (or
α). The method described in this paper avoids the need
for separate calibration, and allows the user (working
with the forecast provider) to obtain the maximum
benefit from any probability forecast system. In effect,
calibration is built into the identification of pmax.
Figure 2. Reliability diagram for the probability forecasts
used to generate the forecast value graphs of Figure 1. Line
types indicate the forecast lead-times as in Figure 1.
at 192h, maximum value is obtained with pmax =10%,
indicating that, with a 51-member ensemble, the user
should take protective action when only 5 ensemble
members predict the event to occur. For 96h and 144h
forecasts, the value is still increasing rapidly as the
threshold decreases to 10%, and it appears likely that
even better value could be obtained using a threshold of
2 or 3 ensemble members. For the 192h forecasts the
value is negative even at pt=10%. Again, it is possible
that a small positive benefit could be obtained from the
forecasts using a lower threshold. In this study forecast
verification was only available at 10% probability
thresholds, so it is not possible to confirm this. For
practical applications where a user has a low cost-loss
ratio α, it is important to carry out the verification for
all possible low probability thresholds down to a single
ensemble member, in order to ensure optimum use of
the ensemble information. This is likely to be particu-
larly important when considering the use of ensembles
for predicting severe weather events, when the losses
from not protecting are often high.
It must be noted that for some users the forecast value
may always be negative, even though the forecast has
skill, because the costs of misses and false alarms are
too high relative to the savings made from a correct
forecast. In this case the user’s best strategy is to either
always or never protect, whichever is better, rather
than following the forecasts.

The example of User C shown in Figure 1c is similar to
User A, except that the false alarm cost is half that of
User A. This example demonstrates the flexibility of
the system in allowing Lm ≠ C. The expected decision
threshold is α=0.111, and Figure 1c shows that, except
312
5.2 Comparison of probability and deterministic
forecasts
Also shown in Figure 1 are the values of the determin-
istic forecasts from the higher resolution TL319

ECMWF model, calculated in exactly the same way,
but with only a single yes/no boundary instead of a
range of probability thresholds. This provides a direct
comparison, in terms which are relevant to the user, of
the benefits of deterministic and probabilistic forecasts.
In the case of the 48-hour forecasts for User A the
deterministic value is almost identical to the maximum
value of the probability forecast; in every other case
shown the probability forecast has more user value
than the equivalent deterministic forecast, despite the
higher resolution of the deterministic model. In many
cases the probability forecast has positive value when
the deterministic forecast has negative value. This is
largely due to the fact that since the deterministic fore-
cast can generate probabilities of only 0 or 100%, it
cannot adapt to the requirements of different users. By
contrast, the probability forecast can use a range of
probabilities, and can be interpreted appropriately for
each user as explained above (see also Zhu et al. 2001).
For some forecast events and users the probability
forecast can have more value even at 12 or 24 hours,
although in general the benefits are greater at longer
lead-times. For longer-range forecasts there is inher-
ently more uncertainty, and a deterministic forecast is
likely to be wrong more frequently, whereas a proba-
bility forecast can account for the uncertainty by using
mid-range probabilities. Occasionally it is found that
the deterministic forecast is more valuable, even at
quite long lead-times. This occurs when the behaviour
of the deterministic model for the defined event is well-
tuned to the particular user’s losses. In any case, the
method allows the best strategy for any user to be iden-
tified, and in most cases this will be from the probabil-
ity forecasts.
14698080, 2002, 3, Downloaded from https://rmets.onlinelibrary.wiley.com/doi/10.1017/S1350482702003043 by Readcube (Labtiva Inc.), Wiley Online Library on [15/12/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Decision-making from probability forecasts
6. Practical application
Practical application of this technique for the use of
probability forecasts as decision-making tools will
require close collaboration between forecast providers
and users. The user is required to define the event to be
forecast, and also to estimate the losses expected in each
of the contingencies. The forecast provider must pro-
vide verification of the forecast system’s ability to pre-
dict the defined event. Given these data, forecast value
functions can be plotted; pmax can then be identified,
and the user takes protective action whenever the fore-
cast probability exceeds this threshold.
Although this process is simple in principle, it is less
straightforward in practice. Many decision-making
processes are more complex than a simple
‘protect’/‘don’t protect’ choice. Estimation of users’
losses is not easy, and verification data may not be
immediately available for the defined event. It has been
assumed here that the user will want to base decisions
on maximising savings over a large number of occa-
sions. Some users may place their emphasis on reducing
their exposure to large losses; however, it should be
possible to reflect this in suitable choices of the costs

5.3 Effect of the climatological reference expense

From Figure 1 it can be seen that in cases A and C the
value of the probability forecasts converges to zero at pt
=0 and in case B at pt =1. The points at pt =0 and 1 cor-
respond to the event always and never being predicted
respectively, so that V=0 at one of these points because
Efx=Ecl.

In general, the correct climatological reference option

to use will be the cheapest, as described in section 4.2.
This minimises the forecast value and avoids any risk of
over-estimating it. However, in some real situations the
user may not have that option, as described at the end
of section 4.2. For such users it would be appropriate to
choose a fixed climatological baseline for the forecast
value, rather than the minimum given by equation (4).
In Figure 3, the example of Figure 1a is shown assum-
ing that the user does not have an option to ‘always
protect’. In this case it is found that the relative value of
the forecasts is now much greater. Since it is only Ecl
which has changed in equation (8), and Ecl>Ep, the Figure 3. As Figure 1a, but with the climatological reference
absolute value V is also larger. expense restricted to the ‘never protect’ option.

313

Kenneth R. Mylne
and losses, with the result that the system will identify
a low protection threshold pmax. Despite these difficul-
ties, it is hoped that this method will help to develop
better understanding of how probability forecasts can
be used in decision-making, and lead to further work
on less idealised decision-making scenarios. Benefits
are likely to be greatest for users with large weather-
sensitivity and large potential losses, for whom the
investment in initial analysis is justified by the returns.
Forecast providers need to be aware that analysis of
some customers’ requirements may reveal that it is not
possible to provide a forecast with value, in which case
the customer is best advised not to purchase the forecast!
Conversely, Graham et al. (2000) demonstrate that the
technique may be used to gain useful information from
seasonal forecasts for which forecast skill measured by
conventional verification techniques, such as ROC, is
relatively low compared to medium-range forecasts.
7. Conclusions
A technique for estimating forecast value has been
described which allows direct comparison of the value
of probability and deterministic forecasts in terms
which are relevant to forecast users. Probability fore-
casts based on the ECMWF EPS are usually found to
have greater value than equivalent deterministic fore-
casts from a higher resolution model, particularly for
forecasts with lead-times greater than about 48 hours.
The technique may be used to apply probability fore-
casts as decision-making tools by identifying a proba-
bility threshold which maximises the forecast value.
The forecast user then takes protective action against
the event whenever the forecast probability exceeds
this threshold.
Acknowledgements
The author would like to thank Tim Legg and Caroline
Woolcock for providing the ensemble verification data
used in this study, and Mike Harrison for stimulating
the ideas behind the project. The author also thanks the
two anonymous referees for their constructive com-
ments on the first draft of this paper.
References
Brier, G. W. (1950) Verification of forecasts expressed in
terms of probability. Mon. Wea. Rev. 78: 1–3.
Buizza, R. & Palmer, T. N. (1998) Impact of ensemble size on
ensemble prediction. Mon. Wea. Rev. 126: 2503–2518.
Buizza, R., Miller, M. & Palmer, T. N. (1999) Stochastic rep-
resentation of model uncertainties in the ECMWF EPS. Q.
J. R. Meteorol. Soc. 125: 2887–2908.
314
14698080, 2002, 3, Downloaded from https://rmets.onlinelibrary.wiley.com/doi/10.1017/S1350482702003043 by Readcube (Labtiva Inc.), Wiley Online Library on [15/12/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Carter, G. M., Dallavalle, J. P. & Glahn, H. R. (1989)
Statistical forecasts based on the National Meteorological
Center’s numerical weather prediction system. Wea. and
Forecasting 4: 401–412.
Glahn, H. R. & Lowry, D. A. (1972) The use of model out-
put statistics (MOS) in objective weather forecasting. J.
Appl. Meteorol. 11: 1203–1211.
Graham, R. J., Evans, A. D. L., Mylne, K. R., Harrison, M. S.
J. & Robertson, K. B. (2000) An assessment of seasonal
predictability using atmospheric general circulation mod-
els. Q. J. R. Meteorol. Soc. 126: 2211–2240.
Hamill, T. M. & Colucci, S. J. (1997) Verification of Eta-RSM
short-range ensemble forecasts. Mon. Wea. Rev. 125:
1312–1327.
Hamill, T. M. & Colucci, S. J. (1998) Evaluation of Eta-RSM
ensemble probabilistic precipitation forecasts, Mon. Wea.
Rev. 126: 711–724.
Houtekamer, P. L., Lefaivre, L, Derome, J., Ritchie, H. &
Mitchell, H. L. (1996) A system simulation approach to
ensemble prediction. Mon. Wea. Rev. 124: 1225–1242.
Katz, R. W. & Murphy, A. H. (eds.) (1997) Economic Value
of Weather and Climate Forecasts. Cambridge University
Press, Cambridge.
Legg, T. P., Mylne, K. R. & Woolcock, C. (2001) The use of
medium-range ensembles at the Met Office. I: PREVIN. A
system for the production of probabilistic forecast infor-
mation from the ECMWF EPS, to appear in Meteorol.
Appl.
Lorenz, E. N. (1963) Deterministic nonperiodic flow. J.
Atmos. Sci. 20: 130–141.
Lorenz, E. N. (1993) The Essence of Chaos. UCL Press Ltd,
London, 227pp.
Molteni, F., Buizza, R., Palmer, T. N. & Petroliagis, T. (1996)
The ECMWF ensemble prediction system: methodology
and validation. Q. J. R. Meteorol. Soc. 122: 73–119.
Murphy, A. H. (1977) The value of climatological, categorical
and probabilistic forecasts in the cost-loss situation. Mon.
Wea. Rev. 105: 803–816.
Murphy, A. H. (1994) Assessing the economic value of
weather forecasts: an overview of methods, results and
issues. Meteorol. Appl. 1: 69–74.
Richardson, D. (2000) Skill and relative economic value of the
ECMWF ensemble prediction system Q. J. R. Meteorol.
Soc. 126: 649–667 (January, Part B).
Stanski, H. R., Wilson, L. J. & Burrows, W. R. (1989) Survey
of Common Verification Methods in Meteorology. WMO
WWW Tech Report No 8, WMO TD No. 358.
Thornes, J. E. & Stephenson, D. B. (2001) How to judge the
quality and value of weather forecast products. Meteorol.
Appl. 8: 307–314.
Toth, Z. & Kalnay, E. (1993) Ensemble forecasting at the
NMC: the generation of perturbations. Bull. Amer.
Meteor. Soc. 74: 2317–2330.
Toth, Z., Kalnay, E., Tracton, S. M., Wobus, R. & Irwin, J.
(1997) A synoptic evaluation of the NCEP ensemble, Wea.
and Forecasting 12: 140–153.
Wilks, D. S. (1995) Statistical Methods in the Atmospheric
Sciences: An Introduction, International Geophysics Series,
Vol. 59, Academic Press.
Zhu, Y., Toth, Z, Wobus, R., Richardson, D. & Mylne, K.
(2001) On the economic value of ensemble-based weather
forecasts, forthcoming in the Bulletin of the American
Meteorological Society.
 14698080, 2002, 3, Downloaded from https://rmets.onlinelibrary.wiley.com/doi/10.1017/S1350482702003043 by Readcube (Labtiva Inc.), Wiley Online Library on [15/12/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Decision-making from probability forecasts
Appendix. The link between forecast value and
H+M
ROC verification ō = (A4)
H+M+F+R
The contingency table in Table 1 is the same as that
used in ROC verification (see Stanski et al. 1989). ROC Hence, combining equations (A1), (A3) and (A4):
expresses the forecast performance in terms of a hit rate
(HR) and false alarm rate (FAR) defined as: h = ōHR (A5)
H
HR = (A1) Similarly:
H+M
m = ō(1– HR) (A6)
F
FAR= (A2) f = (1 – ō)FAR (A7)
F+R
r = (1 – ō)(1 – FAR) (A8)
where H,M,F and R are as defined in Table 1.
In ROC verification of probability forecasts HR and
To evaluate Efx from equation (1) we require values of FAR are evaluated for a range of probability thresholds
h, m, f and r. By definition, the frequency of hits h is in exactly the same way as described in this paper for
forecast value. Thus equation (1), and hence the fore-
H
h= (A3) cast value, are easily evaluated for any forecast for
H+M+F+R which ROC verification data are available and the
climatological frequency of the event, ō, is known.
and the climatological frequency of the event is

315