Professional Documents
Culture Documents
Measuring Forecast Performance of ARMA
Measuring Forecast Performance of ARMA
Olanrewaju. I. Shittu
Department of Statistics, University of Ibadan Nigeria
E-mail: oi.shittu@mail.ui.edu.ng
Abstract
The classical approach to modelling economic series is to apply the Box – Jenkins
approach of ARMA or ARIMA depending on whether the series is stationary or non-
stationary. If such series exhibits long memory property, forecast values based on ARIMA
model may not be reliable. This studies therefore focused on measuring forecast
performance of ARIMA(p,d,q) and ARFIMA (p,d,q) models for stationary type series that
exhibit Long memory properties. The UK Pound/US Dollar exchange rate data were
analysed by OX 5.1 package using the Root mean Square forecast Error (RMSFE) and
Mean Absolute Percentage Forecast Error (MAPFE) as measurement criteria.
The ARFIMA model was found to be better than ARMA model as indicated by
model diagnostic tools. The estimated forecast values from ARFIMA model is more
realistic and closely reflect the current economic reality in the two countries as indicated by
the forecast evaluation tools. The results are in agreement with Kwiatkowski et.al.(1992)
and Boutahar, M. (2008).
1. Introduction
Let a process { X t } , t = 1,..., T be a stochastic process generated by zero mean ARIMA process as
Φ ( B ) X t = Θ ( B ) ε t where Φ ( B ) = 1 − φ1B − ... − φ p B p , Θ ( B ) = 1 − θ1B − ... − θ p B p are polynomials in B;
φi ( i = 1,..., p ) and θi ( i = 1,..., q ) are the autoregressive and moving average parameters respectively and
ε t is a white noise series with variance σ ε2 . If { X t } , t = 1,..., T is an integrated series I ( d ) for
, we term such series a ‘short range’ dependence process which can be modelled using
the Autoregressive Integrated Moving Average (ARIMA(p,d,q)) Box and Jenkins (1960). However, if
d is not an integer i.e d < 1 , we term the series ‘long range’ dependence process or fractionally
integrated series of order da. The overall pattern of the fractionally integrated series is characterized by
Measuring Forecast Performance of ARMA and ARFIMA Models: An Application to
US Dollar/UK Pound Foreign Exchange Rate 168
a sequence of ‘long swings’ around a fairly stable average. Such behaviour is suggestive of long
memory in which temporal dependence falls off very slowly over time Mills(2007).
Researchers are used to modelling macro-economic time series that display long memory
property with ARMA or ARIMA method that yields poor forecast values. Series exhibiting long
memory property can be better modelled using the most appropriate model, the Autoregressive
Fractional Moving Avarage (ARFIMA) model.
The ARFIMA ( p, d , q ) model was introduced by Granger and Joyeux (1980). Since then there
has been great strides in the estimation of long memory modelling, Granger and Joycux (1080); Hoskin
(1981); Sowel (1992); Geweke and Portter-Hudack(1993) and Mayoral (2007). Doornik and Ooms
(1999) showed that it is possible to model long memory series of any length using Extended Maximum
Likelihood (EML) estimation method. They tested the efficiency of the method by applying a
parametric bootstrap estimation on UK and US inflation and got a promising result.
This paper therefore seek to measure the forecast performance ARFIMA and the ARIMA
models using the Root mean Square forecast Error (RMSFE) and Mean Absolute Percentage Forecast
Error (MAPFE) as the measurement criteria. The analysis will be carried out with the OX Programme
5.1 following the Doornik and Ooms (2001, 2004).
The remaining part of this paper is structured as follows: Section 2 covers the theory and
methods; the data used is given in section 3, the results an conclusions are given in sections 4 and 5
respectively.
Γ ( k + d ) Γ (1 − d )
=
Γ ( d ) Γ ( k − d + 1)
Γ (1 − d )
≈ k 2 d −1
Γ (d )
≈ mk 2 d −1 (2)
In general, X t may be generated by zero mean ARMA process, Φ ( B ) X t = Θ ( B ) ε t where ε t
is a white noise with variance, σ ε2 , Φ ( B ) = 1 − φ1B − ... − φ p B , Θ ( B ) = 1 − θ1B − ... − θ p B are
p p
polynomials in B and φi ( i = 1,..., p ) and θi ( i = 1,..., q ) are the parameters of the autoregressive and
moving average parameters respectively. It then follows that the general ARFIMA ( p, d , q ) model is
Φ ( B ) Δd ( X t − μ ) = Θ ( B ) ε t (3)
where Δ = 1 − B .
The autocovariance function (ACF) of a stationary ARMA process with mean μ is given as
γˆ ( k ) = E ⎡⎣( X t − μ )( X t +k − μ ) ⎤⎦ . This defines the matrix of the joint distribution of ( X1 ,..., X T )′ . So,
Sowell (1992) gives an algorithm for the computation of ACF of an ARFIMA process as,
q p
γˆk = σ ε2 ∑ ∑ψ iδ j A ( d , p + i − k , ρ j ) ,. k = 0,..., T − 1 (4)
i =− q j =1
q ⎡ p p
⎤
where ψ i = ∑ θ sθ s − i , δ j−1 = ρ ⎢∏ (1 − ρ k ρ j )∏ ( ρ j − ρ m ) ⎥ ; and
s= i ⎣ k =1 m≠ j ⎦
Γ (1 − 2 d )( d )h
A ( d , h, ρ ) = ⎡⎣ ρ 2 p F ( d + h,1;1 − d + h; ρ ) + ρ 2 p F ( d − h,1;1 − d − h; ρ ) − 1⎤⎦ .
⎡⎣ Γ (1 − d ) ⎤⎦ (1 − d )h
2
exogenous variables, zt yielding residuals, et . Denote the partial sum series of et by St , then, the zero
order KPSS statistic is,
−1
⎛ T ⎞ T
k0 = ⎜ T ∑ et2 ⎟ ∑ St2 (6)
⎝ t =1 ⎠ t =1
The approximate critical values are then given by KPSS (1992) in E-views 4.1. Previous Monte
Carlo simulations have shown that best sample results of the test are obtained using both the automatic
bandwidth selection procedure and the quadratic spectral kernel (Hobijn et. al., 1998).
= T (T + 2 ) ∑ ⎣
K
≈ χ 2 (K − l) (7)
k =1 T −k
where K − l = k − p − q and p + q does not include the constant term.
Forecasts of ARFIMA and ARMA models are computed for both in-sample and out-sample
values. The optimal forecasts values are then evaluated using the mean squared forecast error (MSFE)
defined as,
( )
2
1 h+ s ˆ
MSFE = ∑ Xt − Xt (8)
h + 1 t =s
the root mean square forecast error (RMSFE) defined as:
( )
h+ s 2
1
RMSFE = ∑ Xˆ t − X t (9)
h + 1 t =s
and the mean absolute percentage forecast error (MAPFE) is also given as,
100 h + s Xˆ t − X t
MAPFE = ∑ ˆ
h + 1 t =s Xt
where t = s ,1 + s ,..., h + s and the actual and predicted value for corresponding t values are denoted
by Xˆ t and X t respectively. (10)
The smaller the values of RMSFE and MAPFE, the better the forecasting performance of the
model.
171 Olanrewaju. I. Shittu and Olaoluwa Simon Yaya
3. Data Used
To compare the forecast performances of the models the monthly seasonally adjusted US dollar-UK
pound exchange rate data covering from January 1971 to December 2008 (T = 456) is used. These
data were obtained from IMF International financial Statistics and which we believe is large enough to
model the fractional integrated ARMA model.
4. Results
The examination of the time plot (Figure 1) of the US/UK exchange rate shows that there is some kind
of non-stationarity and possibly nonlinearity in the observed data. Real depreciation of UK pound
during the beginning of the 1971 to mid-1976 are noticed. This later appreciates until it reaches it peak
in 1980 after which there is depreciation. There is generally slow movement of this series from January
1992 to December 2008. The series demonstrates a sequence of long swing around a slowly declining
average such that the behaviour is suggestive of ‘long memory’.
Figure 1: Time Plot of UK Pound to US Dollar Exchange Rate Jan. 1991-Dec. 2008
3.0 3.0
2.5 2.5
2.0 2.0
1.5 1.5
1.0 1.0
1972 1974 1976 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008
Assuming the order of integration d is an integer, the ADF test was carried out on the series as
shown in Table 1. The test was unable to detect the non-stationarity in the series at 1%, 5% and 10%
level of significance because the test was develop on the assumption that d must be an integer (Dickey
and Fuller, 1979).
t-Statistic Prob.*
ADF test Statistic -1.863899 0.6715ns
Test critical values 1% level -3.978355
5% level -3.419729
10% level -3.132483
Following from the stationarity test using ADF, the observed autocorrelation function (ACF)
and the partial autocorrelation (PACF) plots indicated 2 significant spikes at 5% in lag 1 and 3 and this
suggest ARMA (p, 0) model. Using the AIC and BIC criteria, the best model is found to be ARMA (4,
0). The estimate of parameters; standard errors and the p-values are given in table 3 below:
Measuring Forecast Performance of ARMA and ARFIMA Models: An Application to
US Dollar/UK Pound Foreign Exchange Rate 172
Table 5: ARMA Model Estimation for the UK Pound/US Dollar Exchange Rate
From the estimates of the ARMA (4, 0) model above, all the parameter estimates are significant
at 5% level. It appears to be a good model fit as indicated by the R 2 coefficient.
Due to inability of ADF unit root test to detect non-stationarity in the series, we then carried out
long memory test following Kwiatkowski et al. (1992) and using the Ox programme 5.1. The result is
shown in Table 2.
The null hypothesis of fractional difference is accepted at 1%, 5% and 10% levels of
significance after first fractional difference.
Table 2: Fractional Integration test for the differenced UK Pound/US Dollar Exchange Rate
LM-Statistic
KPSS test Statistic 0.044217ns
Test critical values 1% level 0.216
5% level 0.146
10% level 0.119
A two dimensional grid values of ( p, q ) was set up with maximum values ( p, q ) = ( 8,8) and a
search over all constituent models was undertaken using the AIC and the probability of significance of
the parameter estimate are used to select the best fitting model. The best model is the zero mean
ARFIMA ( 3, d ,0) model with parameters as shown in Table 4 written as
Δ d X t = φ1Δ d X t −1 + φ2 Δ d X t − 2 + φ3Δ d X t −3 + ε t
Table 3: ARFIMA Model Estimation for the UK Pound/US Dollar Exchange Rate
A 95% confidence interval for d is estimated as 0.48311 < d < 0.50820 . This confirms the result
of Kwiatkowski et. al. (1992) that UK Pound/US Dollar exchange rate exhibits long memory as
detected in the test. The coefficient of multiple determination, R 2 is 0.99 indicates the fitness of the
model. Further tests show the absence of autocorrelation, heteroscedasticity and non-normality in the
residual error of the model. All these indicate the goodness of fit of this model.
Model comparison indicates that the estimated R2 are very close. The ratio of the residual
variance of the two models is obtained as σ ARFIMA
2
σ ARMA
2
= 0.9838 < 1 . This indicates an improvement
of ARFIMA model over the ARMA counterpart.
Figure 2: Plots of Residual, Actual and Fitted Observations for the ARMA Model
3.0
2.5
2.0
.2 1.5
.1 1.0
.0
-.1
-.2
72 74 76 78 80 82 84 86 88 90 92 94 96 98 00 02 04 06
In terms of forecasts, ARFIMA gives better estimates based on the estimated RMSFE and
MAPFE values for the two models. The estimation ARFIMA model has reported smaller parameter
values for these estimates.
Measuring Forecast Performance of ARMA and ARFIMA Models: An Application to
US Dollar/UK Pound Foreign Exchange Rate 174
The result of the out-of- sample forecasts for 2009:01-2009:12 shows that till the end of the
year, US Pound will continue to depreciate while US dollar appreciates in value. Following the daily
UK Pound/US Dollar exchange rates, as published in “The Punch” of March 8, 2009, it was reported
that £212.88 was sold for $150 in the capital market and this left the exchange rate as 1.4192. As at
yesterday March 28, 2008,
and it was also published that £221 was sold for $151.30 and this left the exchange rate as 1.3873.
Prior to availability of monthly exchange rate summary, these figures are enough to make our
justification about the foreign market.
5. Conclusion
The time plot of the UK Pound/US Dollar exchange rates covering the period January 1971 until
December 2008 shows that US and UK have experienced stable economies over the years due to the
slow movement of the series. Non-stationarity of the series was rejected at all levels by Dickey- Fuller
test but KPSS test confirms the existence of long memory and the estimated integrated value, d =
0.495654 falls within the range of specified value for long memory series.
175 Olanrewaju. I. Shittu and Olaoluwa Simon Yaya
The estimates of the parameters of the ARFIMA ( 3, d ,0) and ARMA( 4,0) models have been
found to be highly significant at 5% level. The ARFIMA ( 3, d ,0) model is better both in terms of the
residual variance of the model. This confirms our observation from the time plot. The estimated
forecast values from ARFIMA model is more realistic and closely reflect the current economic reality
in the two countries as indicated by the forecast evaluation tools. The results are in agreement with
Kwiatkowski et. al. (1992) and Boutahar, M. (2008).
The results show that even though some series may appear to be stationary and supported by
ADF test, they could still exhibit the characteristics of long memory process as indicated in our result.
It is therefore recommended that efficient data exploratory exercise is very crucial before
carrying out time series data analysis of any kind. This would reveal all the hidden characteristics of
the series which could assist in the choice of the appropriate model that would yield optimal forecast
values from the chosen model.
Acknowledgement
This paper is partly funded by Senate research grant number SRG/FSC/2006/2A of the University of
Ibadan, Nigeria.
References
[1] Box, G. E. P. and Jenkins, G. M. (1976). Time Series Analysis: Forecasting and Control,
Revised Edition, Holden-Day.
[2] Boutahar, M. et. al. (2008). “A fractional Integrated Exponential STAR Model Applied to the
US Efective Exchange Rates”, GREQAM, pp. 1-27.
[3] Brillinger ((1981): “Time series Data Analysis and Theory” Expanded Edition, McGraw-Hill,
Inc. New York
[4] Chattfield, C. ((1975): “The Analysis of Time Series”. An introduction, New York, Chapman
and Hall.
[5] Chong, T. T. L. (2006):”The Polynomial Aggregated AR (1) model”, Econometrics Journal,
9:98-122.
[6] Dickey, D.A. and Fuller, W. A. (1979). “Distribution of the Estimators for Autoregressive Time
Series with a Unit Root,” Journal of the American Statistical Association, 74, 427–431.
[7] Doornik, J. A. and Ooms, M. (2006): “A Package for Estimating, forecasting and Simulating
ARFIMA models: ARFIMA package 1.04 for Ox”, Nuffield College, Oxford, pp 1-30.
[8] Geweke, J. And Potter-Hudak, S. (1993): “The Estimation and Application of Long memory
Time series models”, Journal of Time Series Analysis, 4: 221-238.
[9] Granger, C. W. J. And Joyeux, R. (1980): “An Introduction to the Long Memory time series
Models and Fractional Differencing,” Journal of Time Series Analysis, 1, 15-29.
[10] Hinich, M. J, and Chong, T. (2007): “A Class Test for fractional Integration”, Studies in
Nonlinear Dynamics and Econometrics, 11(2): 1-22.
[11] Hosking, J. R. M. (1981). “Fractional Differencing”, Biometrika 68, 165–176.
[12] Hosking, J.R.M. (1996): “Asymptotic distributions of the Sample mean, Autocovariances, and
autocorrelations of long-memory time series,” Journal of Econometrics, 73, 261-284.
[13] Hurvich, C. M. And Ray, B. K. (1995): “Estimation of the Memory Parameter by non-
stationary and non- invertible fractionally integrated processes”, Journal of Time Series
Analysis, 16: 17-41.
[14] Kwiatkowski, D, Phillips, P. C., Schmidt, P. and Shin, Y. (1992). “Testing the Null Hypothesis
of Stationary against the Alternative of a Unit Root,” Journal of Econometrics, 54, 159-178.
[15] Ljung, G. M. and G. E. P. Box (1978). “On a Measure of Lack of Fit in Time Series Models,”
Biometrika, 65, 297–303.
Measuring Forecast Performance of ARMA and ARFIMA Models: An Application to
US Dollar/UK Pound Foreign Exchange Rate 176
[16] Mayoral, L. (2007): “Minimum Distance Estimation of Stationary and non-Stationary ARFIMA
Processes”, Journal of Econometrics, 10: 124-148.
[17] Mills, T. C. (2007): “Time Series Modelling of Two millennia of Northern Hemisphere
Temperatures: Long memory or shifting trend?”, Journal of Royal Statistical Society, 170(a):
83-94.
[18] Ooms, M. And Doornik, J. (2004): “Inference and Forecasting for Fractional Autoregressive
Moving Average Models, with Application to US and UK Inflation”, Studies in Nonlinear
Dynamics and Econometrics, 8(2): 1-25.
[19] Pfaff, B (2004): urca “Unit root and Co integration Test for Time Series” R package, version
0.6-1.
[20] Rao, B.B (1994): “Co integration for Applied Economists” The Macmillan Press Ltd. London,
UK.
[21] Sowell (1992): “Maximum Likelihood Estimation of Stationary univariate Fractionally
Integrated time series Models,” Journal of Econometrics 53: 165-188.