Technical ■ Electricity pricing

Divide and conquer: forecasting power

via automatic price regime separation
In this article Michael Baron, Michael Rosenberg and Nikolai Sidorenko derive a
relatively simple and tractable model for pricing electricity in deregulated markets.
Power prices are predicted based on the mathematical description of regular and
spike regimes. Different regimes in price time series are automatically separated by
means of change-point algorithm

– and negative values of γ (m) and δ (ω) for other m and ω (see table 1).

T
he prominent feature of the dynamics of electricity prices is high
volatility in relation to other commodities, and occasional price In order to estimate and eliminate these effects, we use indicator
spikes that last from a few hours to a few days. Various factors functions χm and ψω , defined as:
lead to such behaviour – a lack of electricity storage, localised
production and regional consumption. 1 if time t is during month m
Here we use the change point algorithm of sequential analysis to detect χm (t ) = 
0 otherwise
different regimes in power prices. Upon detecting the onset of spikes, it
automatically classifies them and produces the spike distribution. The and
algorithm then uses a particular calibration scheme to describe the regular
1 if time t falls on day ω of the week
– as well as the spike – regimes. This allows for calculation of expected ψ ω (t ) = 
distribution of power prices based on historical price time series. 0 otherwise
The results of the analysis predict power prices for any time interval
in the future. They can be used for calculating forward curves as well as for m = 1,...,12 and w = 1,...,7. They play the role of ‘dummy variables’.
asset, contract and financial valuations. The price distributions allow for Integrating both sides of equation 1 and simplifying both summations,
numerical simulations where analytical solution is not feasible. Here we we obtain the following log-linear model:
describe the model in detail.
ln(P t ) = α + βt + γ(m t ) + γ(ω t ) + X t . (2)
Model
The model proposed by Baron, Rosenberg, and Sidorenko (2001) reflects Parameters α, β, γ (1),..., γ (12), and δ (1),...,δ (7) may vary from one
the multi-regime behaviour of electricity prices (one ‘regular’ regime and region to another, although the slope β is usually small in value and the
many ‘spike’ regimes), the general trend, basic seasonal and weekly same for different regions in the country. Thus they should be estimated
variations, and their effect on spikes. separately for each market. Behaviour of the noise variable Xt varies
Due to the dramatically different behaviour of electricity prices during different regimes of the process.
during different regimes, the process cannot be adequately described by
a single equation. A more realistic model is a ‘multi-state process’ that Regular (inter-spike) regime
consists of alternating spikes and inter-spike segments separated by Between the spikes, Xt forms an autoregressive process,
‘change points’ (see figure 1). Analysis and calibration of such models
requires a sensitive statistical tool to detect the change points and X t = φX t −1 + σε t (3)
identify the homogeneous segments, which can then be explored
separately. The subject of this article is to elaborate on the use of this with respect to a normalised Gaussian white noise sequence, εt , reflecting
mechanism for efficient power pricing. dependence of electricity prices observed at different times. According to
Analysis of data in different US energy markets supports the equation 3, detrended log prices Xt have a multi-variate normal
following model for electricity prices Pt between the spikes: distribution, with mean 0, covariance matrix S, elements computed as

dPt 12 7 Σ ij = σ 2φ i− j (1 − φ2 )
= βdt + dX t + ∑ γ (m)dχm (t )+ ∑ δ (ω )dψ ω (t )
Pt m=1 ω =1
, (1) and the joint density
−k 2 −1 2
where t is time in days, m is the month, w is the day of the week and Xt f (x ) = (2π ) (det Σ ) exp {− x Τ Σ −1 x 2}. (4)
represents random noise. Parameters γ (1),...,γ (12) and δ (1),...,δ (7)
represent effects of 12 months of the year and seven days of the week on Equation 2 reflects the general trend of electricity prices and their
power prices. For any m=1,...,12 and w=1,...,7, γ (m)+ δ (w) is the average seasonal and weekly variations. Equation 3 determines a sensible
deviation of power prices from the general trend during month m and day correlation structure, where correlation between electricity prices on two
ω . Higher power prices in summer and lower power prices during different days gets weaker as the time between them increases.
weekends are reflected by positive values of γ (m) and δ (ω) for m=5,...,9 –
that is, May through September – and ω =1,...,5 – Monday through Friday 1 See Brockwell and Davis (1991), chapter 3.

70 I March 2002
Spikes
Spikes are always relatively short, but they result in very significant
Figure 2: Log prices, fitted values and weights
variations of electricity prices both within each spike and between the
spikes (figure 1). Hence, different spikes are generated by different 7 Log price
distributions. In other words, the underlying multi-state process has one Fitted value
regular mode and many spike modes. 6
Weight
In contrast to the regular regime, during spikes the price of electricity
increases drastically above the general trend, so that residuals Xt can no 5
longer be assumed to have zero mean. Moreover, the mean (detrended)
log price m= E (X t ) differs from one spike to another. Hence a Bayesian

$/MWh
4
model with a suitable prior distribution of m is appropriate.
The Bayesian approach treats the spike parameter m as a random 3
variable. When a spike starts at a random time, a new value of m is
generated from its prior distribution. Then a segment of Xt is generated 2
from the normal distribution, with mean m. During the next spike, a
sample is also generated from the normal distribution, but it will have a 1
different m.
We model by a normal disribution with parameters m and t, where m, 0

0
0

0
0
0

0
0
0
0
50
0

0
20

50
25

55
35
15

45

65
60
40
10

30
in turn, has a conjugate normal prior distribution with ‘hyper-
parameters’ q and h. Using a conjugate distribution guarantees that the Days (from January 1, 1998)
posterior distribution of m belongs to the same class as its prior
distribution. Therefore, it is also normal. Hyper-parameters refer to the
entire process, and are common for all spikes. Specifically, q is the
average of all mean prices during spikes, and h controls variations
Table 1: Estimated monthly effects
between spikes. On the contrary, t determines the variance within spikes.
m 1 2 3 4 5 6 7 8 9 10 11 12
Ÿ
Transitions g (m) (.10) (.23) (.05) (.01) .07 .07 .43 .24 .03 (.07) (.16) (.21)
Duration of each phase and transitions from one phase to another are
governed by a Markov chain with the transition probability matrix
Calibration
 1 − p (t ) p (t ) Calibration of the model requires several steps: detrending, segmentation
Π= . (5)
 q 1 − q and the final parameter estimation for each regime. First, we estimate the
trends in equation 2. The time series model – equation 3 – is then fitted
Here, p(t) is the probability for a spike to start during day t, and q is to the detrended log prices.
the probability for a spike to end during any day. Thus, a regular-regime We then construct a sensitive sequential change-point detection
day is followed by another regular-regime day with probability (1 – p(t)), algorithm and apply it repeatedly to detect all change points and separate
and the spike day is followed by another spike day with probability (1–q). different phases of the process. This, finally, allows the estimation of
Spikes are more likely during a peak season, but not likely at weekends. parameters for each phase.
Therefore, p(t) varies in time and depends on the day t. At the same time,
durations of spikes are hardly predictable based on t, mt and wt, so that q can Detrending
be assumed as constant. According to this model, durations of each spike, Standard least-squares methods find such estimates of parameters α, β,
Y, is a random variable with geometric distribution P(Y=y)=q(1–q)y–1 for γ and δ in equation 2 that minimise the sum of squared differences
y=1,2,..., with the average duration of (1/q) days. between
∧ ∧
log
∧
prices
∧
ln(P t ) and the corresponding fitted values
∧
ln(P t ) = α + βt + γ(m t ) + δ(ω t ) .
Here, this method appears inappropriate, because the results would be
Figure 1: Power prices for east NY region, 1998–99 significantly influenced by the presence of spikes. Therefore, we employ
the method of weighted least squares (Draper and Smith (1981)),
450 reducing weights for the most influential observations – prices during
spikes. Low weights assigned to extreme observations practically
400
eliminate the effect of spikes on estimates.
350 The weights are depicted in the bottom of figure 2. This figure also
300 shows the original log prices in the New York Power Pool (NYPP) east
$/MWh

250 region and the estimated trend. The trend clearly shows increasing
electricity prices during summer months and decreasing prices every
200
seven days at weekends.
150 Implementation of the weighted least-squares method results in the
Ÿ Ÿ -4
Ÿ
100 estimates
Ÿ
a = 2.9785, b = 2.27◊10 , d = 0.0549 on weekdays, and
50 d = –0.1373 at weekends. The monthly effects are summarised in table 1.
Ÿ
Note that the computed value of b implies 0.0227% increase of prices
0
in one month or 0.27277% increase of prices during a year. However, the
0

0
0

0
0
0

0
0
0
0
50

0
0

20

50
25

55
35
15

45

65
60
40
10

30

slope appears to be statistically significant in 1998–1999. All the

Days (from January 1, 1998) interaction terms are not significant with p-values between 0.3 and 0.9,
thus justifying the model shown in equation 2.

www.eprm.com I 71
Technical ■ Electricity pricing

is different from that of a change point that marks its end. The main
Figure 3: Detrended log prices and the Cusum process difference is that during a spike the probability of a change point on any
day is q, constant for all spikes. One can estimate q as the inverse of the
Cusum average spike duration and use it in the geometric distribution of spike
9
process durations. This determines the prior distribution of end-of-spike change
8 points that will be used in the Bayesian change-point detection scheme.
7
Log prices On the contrary, during a regular phase, the probability of a change
point – that is, transition to the spike phase – depends on the expected
6 frequency of spikes at that time. Spikes are more frequent in summer,
5 and they hardly ever start during weekends. Therefore, in this case, the
$/MWh

prior distribution of change points involves a number of factors that can

4
be estimated only after the separation step. A non-Bayesian algorithm is
3 required for this type of transition.
2
Detecting the start of spikes
1 Among non-Bayesian algorithms, the ‘cumulative sums’ (cusum)
0 algorithm is efficient for the sequential detection of change points. For
each n=2, 3,..., it tests the hypothesis H0 that there is no change point
–1
0 100 200 300 400 500 600 among the first n observations against the alternative Ha that a change
Days (from January 1, 1998) has occurred. The likelihood ratio statistics for this test are given by:

f (X1 ,..., Xk )g (Xk +1 ,..., Xn )

λ n = max ln , (6)
Fitting a time series k ∈{1 ,...,n−1} f (X1 ,..., Xn )
Accurate estimation of f and s in equation 3 is possible only after the
inter-spike phases are separated from the spikes. But the separation where f and g represent densities before and after the change point,
scheme requires at least some information about the distribution of respectively. A change point is declared detected when ln exceeds a
prices during the inter-spike regime. chosen threshold, h, that forces one to reject H0 in favour of Ha.
Preliminary estimation is possible based on the electricity prices during Unlike standard settings, our situation is featured by (a) unknown
the shoulder seasons, because no spikes occurred during those months in parameters and during spikes, and (b) auto-correlation during the
1998–99. Fitting an autoregressive model – equation 3 – to detrended log regular regime. Hence, a modified cusum scheme is needed.
Ÿ Ÿ
prices results in the following estimates: f =0.3958 and s =0.1824. The Problem (a) is resolved by replacing unknown parameters of f and g
method of estimation is given in Brockwell and Davis (1991). in equation 6 with their maximum likelihood estimates, calculated for
each value of k. This results in:
Separation of regular and spike phases in power prices
(k −n) 2  n  (k −n) 2
exp − ∑ (X j − Xkn ) 2skn
2
The following algorithm detects all the change points that separate g (Xk +1 ,..., Xn )= (2πskn
2
)
2
 = (2πskne)
2

different phases of the process. It is driven by a sequential scheme that  j =k + 1  ,

searches for the first change point until it is detected, estimates the
detected change point, searches for the second change point based only where
Xkn = ∑ k +1 X j (n − k )
n
on the data after the first change point, then detects the second change
point, and so on.
The nature of a change point that marks the beginning of a new spike and
= ∑ k +1 (X j − Xkn ) (n − k ) .
2 n 2
skn
Figure 4: One-year ahead forecast of spot prices
Problem (b) compels us to use the joint density (equation 4) of the
Predictive entire vector x of detrended log-prices observed until time k.
densities for
each day of The cusum process, ln, is shown in figure 3. Large values of ln imply
the year
0.1
transitions from one regime to another, distinguishing them from simply
0.1
random fluctuations. When ln exceeds the chosen threshold, a stopping
rule is activated and the start of a new spike is reported.
0.08 Threshold h can be chosen based on the desired probabilities of type
0.05 I and type II errors p I and p II . For example, one can choose h=(1–p II )/p I,
as in Govindarajulu (1987), section 2.2, guaranteeing the desired
0.06 probability of type I error p I and of type II error p II. In our problem, p I
0 is the probability of a false alarm, and p II is the probability of failing to
Feb
detect the start of a spike.
Apr 0.04
Jun Detecting the end of spikes
As explained above, here we use the geometric (q) prior distribution of
Aug 0.02
spike durations that follows from transition probabilities determined by
Oct 80 equation 5. Again, we are looking for an optimal stopping rule – the first
60
Dec 40 0 time t when collected observations X1,...,Xt provide sufficient evidence
20 that a change point has occurred. In contrast with the cusum, here we
0 Price ($/MWh)
construct such a stopping rule based on a sequence of Bayes tests.

72 I March 2002
 The x-axis represents the month. For example, x=0.5 on January 15,
Table 2: Characteristics of spike and inter-spike periods x=1 on January 31, and x=12.0 on December 31. The y-axis is the range
of possible electricity prices. The vertical axis represents the predictive
Total 1998 1999 density of electricity prices. For each day of the year x, the area below the
Number of spikes 11 5 6 graph between prices y1 and y2 is the probability that the actual price is
Number of days with spikes 24 9 15 in the range between y1 and y2.
The surface is not smooth, because the prices are forecast to drop
Mean spike duration (days) 2.1818 1.800 2.500
slightly every seven days, during weekends. Moreover, the densities are
Mean interspike period 15.8889 21.2500 11.600 shifted to the right for the peak months, especially during summer,
Mean spike effect (q) 1.4704 1.1873 1.7063 thereby predicting increasing electricity prices.
Summer months are marked by a higher variability of prices, reflected
Within-spike variance (t) 0.1323 0.1386 0.1284
by a wider shape of densities.
Between-spike variance (h) 0.2944 0.1103 0.3253
P{spikeÆcontrol} 0.4583 0.5556 0.4000 Conclusions
P{controlÆspike} 0.0629 0.0471 0.0862 The sequential change-point algorithm has demonstrated that it can
successfully detect different regimes in power prices and partition the
power price process into regular and spike regimes. The subsequent
calibration procedure and econometric analysis allowed us to come up
The Bayes test rejects the no-change hypothesis H0 in favour of Ha with the predictive price density. This density included the so called ‘fat
when the posterior probability of a change point at or before t=n , based tails’ that contain the probability of spikes, which are typical for a given
on X1,...,Xn only, price process.
Overall, this approach allows for automatic calculation of the expected
P {ν ≤ n| X1 ,..., Xn }= distribution of power prices based on historical price time-series. Results
∑ of our analysis produce future power price distributions that are needed
n k
k =1
(1 − q) f (X1 ,..., Xk )g (Xk +1 ,..., Xn )
n−1 k
for valuing generation assets, retail contracts and trading positions. These
∑
n
k =1
(1 − q) f (X1 ,..., Xk )g (Xk +1 ,..., Xn )+ f (X1 ,..., Xn )(1 − q) q results also allow for generation of Monte Carlo scenarios of future
electricity prices. EPRM
exceeds (1–p I ) (Baron (2001), section 2). Then, the corresponding
stopping rule reports a change point at the first time t when H0 is Michael Baron is associate professor of
rejected. statistics at the University of Texas, Dallas
e-mail: mbaron@utdallas.edu
Estimation of parameters of each phase
Before the described segmentation scheme is implemented and all the Michael Rosenberg is manager of retail operations
phases of the processes are separated, it is impossible to know exactly at TXU Energy Trading, Dallas, Texas
what portion of data should be used to estimate the parameters of each e-mail: mrosenberg@txu.com
segment. Therefore, the obtained estimates may not be accurate.
Now, after a set of change points is obtained and segments are Nikolai Sidorenko is senior quantitative analyst
determined, we can re-estimate the parameters of each regime. at TXU Energy Trading, Dallas
Ÿ Ÿ Ÿ
Parameters of equation 3 are estimated as f =c1/c0 and s =(c0 (1– f 2))1/2, e-mail: nsidore1@txu.com
where c0 and c1 are lag 0 and lag 1 sample auto-covariances of inter-spike
log prices. BIBLIOGRAPHY
The spike hyper-parameters are estimated according to the empirical Baron, M, Bayes stopping rules in a change-point
Bayes approach (Berger (1985), section 4.5). All sample means are model with a random hazard rate, Sequential Analysis,
calculated during different spikes. The average of these sample means 20, pages 147–163, 2001
estimates q, whereas their sample variance estimates h. Within-spike
variance is estimated by the ‘pooled sample variance’ of all the detrended Baron, M, Rosenberg, M, and Sidorenko, N, Automatic
log prices during spikes. spike detection for the modelling and prediction of
Accurate parameter estimates allow more accurate estimation of the electricity pricing, Energy & Power Risk Management,
detected change points. We can now re-estimate each change point based October 2001
on the two adjacent segments only. This re-estimation scheme may be
iterated until the set of estimated change points stabilises. It usually takes Berger, JO, Statistical Decision Theory, Springer-Verlag,
less than 10 iterations for the algorithm to converge. New York, 1985

Results Brockwell, PJ and Davis, RA, Time Series: Theory and

Applied to the electricity prices shown in figure 1, the described scheme Methods, Springer-Verlag, New York, 1991
detected 11 spikes during 700 days. The estimated parameters and other
characteristics are summarised in table 2. Draper, NR and Smith, H, Applied Regression Analysis,
Wiley, 2nd edition, New York, 1981
Application of results: prediction of power prices
The proposed model with estimated parameters can be used to forecast Govindarajulu, Z, The Sequential Statistical Analysis of
the price of electricity for any given day. Predictive densities are depicted Hypothesis Testing, Point and Interval Estimation, and
in Figure 4. The surface shows predictive densities for the full year, one Decision Theory, American Sciences Press,
year ahead, based on the described model and estimated parameters. Columbus, Ohio, 1987

www.eprm.com I 73