Yule-Walker Equations

in matrix form

PRACTICAL TIME SERIES ANALYSIS

THISTLETON AND SADIGOV
Objectives

 Rewrite Yule – Walker equations in matrix form

for AR(p) processes
AR(p) process

𝑋𝑡 = 𝜙0 + 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + ⋯ 𝜙𝑝 𝑋𝑡−𝑝 + 𝑍𝑡

where

𝑍𝑡 ~ 𝑁𝑜𝑟𝑚𝑎𝑙(0, 𝜎𝑍2 )
Note that, if we take expectation from both sides of the model

𝑋𝑡 = 𝜙0 + 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + ⋯ 𝜙𝑝 𝑋𝑡−𝑝 + 𝑍𝑡

We get

𝜇 = 𝜙0 + 𝜙1 𝜇 + 𝜙2 𝜇 + ⋯ 𝜙𝑝 𝜇

Subtract side by side

𝑋𝑡 − 𝜇 = 𝜙1 (𝑋𝑡−1 −𝜇) + 𝜙2 (𝑋𝑡−2 − 𝜇) + ⋯ 𝜙𝑝 (𝑋𝑡−𝑝 −𝜇) + 𝑍𝑡

If 𝑋𝑡 = 𝑋𝑡 − 𝜇, then 𝐸 𝑋𝑡 = 0, and

𝑋𝑡 = 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + ⋯ 𝜙𝑝 𝑋𝑡−𝑝 + 𝑍𝑡

AR(p) process with 𝜇 = 0

𝑋𝑡 = 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + ⋯ 𝜙𝑝 𝑋𝑡−𝑝 + 𝑍𝑡

where

𝑍𝑡 ~ 𝑁𝑜𝑟𝑚𝑎𝑙(0, 𝜎𝑍2 )
Yule –Walker equations

Autocorrelation function obeys

𝜌 𝑘 = 𝜙1 𝜌 𝑘 − 1 + 𝜙2 𝜌 𝑘 − 2 + ⋯ + 𝜙𝑝 𝜌(𝑘 − 𝑝)

for 𝑘 ≥ 1, 𝜌 0 = 1 and 𝜌 𝑘 = 𝜌(−𝑘) for 𝑘 < 0.

Lets write them for 𝑘 = 1,2, … , 𝑝.

Yule Walker equations

ρ k = ϕ1 ρ k − 1 + ϕ2 ρ k − 2 + ⋯ + ϕp ρ(k − p)

For 𝑘 = 1,2, … , 𝑝,

𝜌 1 = 𝜙1 𝜌 0 + 𝜙2 𝜌 −1 + 𝜙3 𝜌 −2 + ⋯ + 𝜙𝑝 𝜌(1 − 𝑝)
𝜌 2 = 𝜙1 𝜌 1 + 𝜙2 𝜌 0 + 𝜙3 𝜌 −1 + ⋯ + 𝜙𝑝 𝜌(2 − 𝑝)
𝜌 3 = 𝜙1 𝜌 2 + 𝜙2 𝜌 1 + 𝜙3 𝜌 0 + ⋯ + 𝜙𝑝 𝜌(3 − 𝑝)

⋮
𝜌 𝑝 − 1 = 𝜙1 𝜌 𝑝 − 2 + 𝜙2 𝜌 𝑝 − 3 + 𝜙3 𝜌 𝑝 − 4 + ⋯ + 𝜙𝑝 𝜌(1)
𝜌 𝑝 = 𝜙1 𝜌 𝑝 − 1 + 𝜙2 𝜌 𝑝 − 2 + 𝜙3 𝜌 𝑝 − 3 + ⋯ + 𝜙𝑝 𝜌(0)

Recall 𝜌 𝑘 = 𝜌(−𝑘).
i.e., 𝜌 −1 = 𝜌 1 , 𝜌 −2 = 𝜌 2 , … , 𝜌 2 − 𝑝 = 𝜌 𝑝 − 2 , 𝜌 1 − 𝑝 = 𝜌 𝑝 − 1
𝜌 −𝑘 = 𝜌(𝑘)

𝜌 1 = 𝜙1 𝜌 0 + 𝜙2 𝜌 1 + 𝜙3 𝜌 2 + ⋯ + 𝜙𝑝 𝜌(𝑝 − 1)
𝜌 2 = 𝜙1 𝜌 1 + 𝜙2 𝜌 0 + 𝜙3 𝜌 1 + ⋯ + 𝜙𝑝 𝜌(𝑝 − 2)
𝜌 3 = 𝜙1 𝜌 2 + 𝜙2 𝜌 1 + 𝜙3 𝜌 0 + ⋯ + 𝜙𝑝 𝜌(𝑝 − 3)

⋮
𝜌 𝑝 − 1 = 𝜙1 𝜌 𝑝 − 2 + 𝜙2 𝜌 𝑝 − 3 + 𝜙3 𝜌 𝑝 − 4 + ⋯ + 𝜙𝑝 𝜌(1)
𝜌 𝑝 = 𝜙1 𝜌 𝑝 − 1 + 𝜙2 𝜌 𝑝 − 2 + 𝜙3 𝜌 𝑝 − 3 + ⋯ + 𝜙𝑝 𝜌(0)
𝜌 0 =1

𝜌 1 = 𝜙1 + 𝜙2 𝜌 1 + 𝜙3 𝜌 2 + ⋯ + 𝜙𝑝 𝜌(𝑝 − 1)
𝜌 2 = 𝜙1 𝜌 1 + 𝜙2 + 𝜙3 𝜌 1 + ⋯ + 𝜙𝑝 𝜌(𝑝 − 2)
𝜌 3 = 𝜙1 𝜌 2 + 𝜙2 𝜌 1 + 𝜙3 + ⋯ + 𝜙𝑝 𝜌(𝑝 − 3)

⋮
𝜌 𝑝 − 1 = 𝜙1 𝜌 𝑝 − 2 + 𝜙2 𝜌 𝑝 − 3 + 𝜙3 𝜌 𝑝 − 4 + ⋯ + 𝜙𝑝 𝜌(1)
𝜌 𝑝 = 𝜙1 𝜌 𝑝 − 1 + 𝜙2 𝜌 𝑝 − 2 + 𝜙3 𝜌 𝑝 − 3 + ⋯ + 𝜙𝑝
Matrix form: Yule- Walker equations
𝜌(1) 1 𝜌(1) 𝜌(2) 𝜌(𝑝 − 1) 𝜙1
𝜌(2) 𝜌(1) 1 𝜌(1) ⋯ 𝜌(𝑝 − 2) 𝜙2
𝜌(3) 𝜌(2) 𝜌(1) 1 𝜌(𝑝 − 3) 𝜙3
=
⋮ ⋮ ⋱ ⋮ ⋮
𝜌(𝑝 − 1) 𝜌(𝑝 − 2) 𝜌(𝑝 − 3) 𝜌(𝑝 − 4) 𝜌(1) 𝜙𝑝−1
⋯
𝜌(𝑝) 𝜌(𝑝 − 1) 𝜌(𝑝 − 2) 𝜌(𝑝 − 3) 1 𝜙𝑝

𝑏 𝑅 𝜙

𝑏 = 𝑅𝜙

𝑅−1 𝑏 = 𝜙
Sample ACF 𝑟𝑘 ≈ 𝜌(𝑘)
𝑟1 1 𝑟1 𝑟2 𝑟𝑝−1 𝜙1
𝑟2 𝑟1 1 𝑟1 ⋯ 𝑟𝑝−2 𝜙2
𝑟3 𝑟2 𝑟1 1 𝑟𝑝−3 𝜙3
⋮ =
⋮ ⋱ ⋮ ⋮
𝑟𝑝−1 𝑟𝑝−2 𝑟𝑝−3 𝑟𝑝−4 𝑟1 𝜙𝑝−1
𝑟𝑝 𝑟𝑝−1 𝑟𝑝−2 𝑟𝑝−3 ⋯
1 𝜙𝑝

𝑏 𝑅 𝜙

𝑏 = 𝑅𝜙

𝑅−1 𝑏 = 𝜙
Matrices 𝑅 and 𝑅
 These matrices are symmetric matrices
 They are positive semidefinite matrices
 All eigenvalues are nonnegative
 Inverses of these matrices exist

1 𝑟1 𝑟2 𝑟𝑝−1 1 𝜌(1) 𝜌(2) 𝜌(𝑝 − 1)

𝑟1 1 𝑟1 ⋯ 𝑟𝑝−2 𝜌(1) 1 𝜌(1) ⋯ 𝜌(𝑝 − 2)
𝑟2 𝑟1 1 𝑟𝑝−3 𝜌(2) 𝜌(1) 1 𝜌(𝑝 − 3)
and
⋮ ⋱ ⋮ ⋮ ⋱ ⋮
𝑟𝑝−2 𝑟𝑝−3 𝑟𝑝−4 𝑟1 𝜌(𝑝 − 2) 𝜌(𝑝 − 3) 𝜌(𝑝 − 4) 𝜌(1)
𝑟𝑝−1 𝑟𝑝−2 𝑟𝑝−3 ⋯ ⋯
1 𝜌(𝑝 − 1) 𝜌(𝑝 − 2) 𝜌(𝑝 − 3) 1

 𝑏 = 𝑅 𝜙 has a unique solution

Example – AR(2)
We (will) estimate coefficients of the model

𝑋𝑡 = 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + 𝑍𝑡

by first finding 𝑟1 , 𝑟2 using acf() routine, then solving the system of

equations

𝑟1 1 𝑟1 𝜙1
𝑟2 =
𝑟1 1 𝜙2
Example – AR(3)
We (will) estimate coefficients of the model

𝑋𝑡 = 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + 𝜙3 𝑋𝑡−3 + 𝑍𝑡

by first finding 𝑟1 , 𝑟2 , 𝑟3 , using acf() routine, then solving the system of

equations

𝑟1 1 𝑟1 𝑟2 𝜙1
𝑟2 = 𝑟1 1 𝑟1 𝜙2
𝑟3 𝑟2 𝑟1 1 𝜙3
What We’ve Learned

 Matrix form of Yule – Walker equations

 Howto estimate the coefficients of an AR

process using Yule-Walker equations
 Estimating model
parameters – AR(2)
Simulation

PRACTICAL TIME SERIES ANALYSIS

THISTLETON AND SADIGOV
Objectives

 Estimate variance of a white noise in a simulated

AR(2) processes
 Estimate coefficients of a simulated AR(2)
process using Yule-Walker equations in matrix
form
AR(2) process (with mean zero)

𝑋𝑡 = 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + 𝑍𝑡
where

𝑍𝑡 ~ 𝑁𝑜𝑟𝑚𝑎𝑙 (0, 𝜎𝑍2 )

We simulate this process for

1 1
𝜙1 = , 𝜙2 = , 𝜎𝑍 = 4
3 2
Yule –Walker equaitons
We estimate coefficients of the model

𝑋𝑡 = 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + 𝑍𝑡

by first finding 𝑟1 , 𝑟2 using acf() routine, then solving the system of

equations

𝑟1 1 𝑟1 𝜙1
𝑟2 =
𝑟1 1 𝜙2
 𝜎𝑍 Estimation
Since

𝑋𝑡 = 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + 𝑍𝑡
We have

𝑉𝑎𝑟 𝑋𝑡 = 𝜙12 𝑉𝑎𝑟 𝑋𝑡−1 + 𝜙22 𝑉𝑎𝑟 𝑋𝑡−2 + 2𝜙1 𝜙2 𝐶𝑜𝑣 𝑋𝑡−1 , 𝑋𝑡−2 + 𝜎𝑍2

Thus

2𝜙1 𝜙2 𝛾 1
𝜎𝑍2 =𝛾 0 1− 𝜙12 − 𝜙22 − = 𝛾 0 1 − 𝜙12 − 𝜙22 − 2𝜙1 𝜙2 𝜌1
𝛾 0
Since

𝜌1 1 𝜌1 𝜙1
𝜌2 =
𝜌1 1 𝜙2

we have

𝜌1 = 𝜙1 + 𝜌1 𝜙2

𝜌2 = 𝜙1 𝜌1 + 𝜙2
 1 − 𝜙12 − 𝜙22 − 2𝜙1 𝜙2 𝜌1

= 1 − 𝜙12 − 𝜙1 𝜙2 𝜌1 − 𝜙22 − 𝜙1 𝜙2 𝜌1

= 1 − 𝜙1 𝜙1 + 𝜌1 𝜙2 − 𝜙2 𝜙1 𝜌1 + 𝜙2

= 1 − 𝜙1 𝜌1 − 𝜙2 𝜌2
𝜎𝑍 Estimation cont.

Thus,

𝜎𝑍2 = 𝛾 0 [1 − 𝜙1 𝜌1 − 𝜙2 𝜌2 ]

Yule –Walker estimator

𝜎𝑍2 = 𝑐0 [1 − 𝜙1 𝑟1 − 𝜙2 𝑟2 ]
Simulation
 Number of data points, 𝑛 = 10000

Routines that we use:

 arima.sim() # simulating
 plot() # plotting the series
 acf() # autocorrelation function
 matrix(,m,n) # matrix with dimensions m by n
 solve(R,b) # finds the solution to Rx=b
Code details
 sigma=4
 phi[1:2]=c(1/3,1/2)
 n=10000
 set.seed(2017)
 ar.process=arima.sim(n, model=list(ar=c(1/3,1/2)), sd=4)
 ar.process[1:5]

4.087685, 5.598492, 3.019295, 2.442354, 5.398302

Code details cont.
 r[1:2]=acf(ar.process, plot=F)$acf[2:3]

𝑟[1] = 0.6814103
𝑟[2] = 0.7255825

R=matrix(1,2,2) # matrix of dimension 2 by 2, with entries all 1's.

1 1
𝑅=
1 1
 R[1,2]=r[1] # only diagonal entries are edited

 R[2,1]=r[1] # only diagonal entries are edited

1 𝑟1
𝑅=
𝑟1 1
Code details cont.

 b=matrix(r,2,1)# b- column vector entires from r

𝑟[1] 0.6814103
𝑏= =
𝑟[2] 0.7255825
Code details cont.

 solve(R,b)

0.3490720
0.4877212

 phi.hat=matrix(c(solve(R,b)[1,1], solve(R,b)[2,1]),2,1)

𝜙1 = 0.3490720
𝜙2 = 0.4877212
Code details cont.

 c0= acf(ar.process, type='covariance', plot=F)$acf[1]

 var.hat= c0*(1-sum(phi.hat*r))
 par(mfrow=c(3,1))
 plot(ar.process, main='Simulated AR(2)')
 acf(ar.process, main='ACF')
 pacf(ar.process, main='PACF')
Results
 𝜙1 ≈ 𝜙1 = 0.3490720
 𝜙2 ≈ 𝜙2 = 0.4877212
 𝜎𝑍2 ≈ 𝜎𝑍2 = 16.37169

Actual Model

𝑋𝑡 = 0. 3 𝑋𝑡−1 + 0.5𝑋𝑡−2 + 𝑍𝑡 , 𝑍𝑡 ~ 𝑁(0,16)

Fitted model

𝑋𝑡 = 0.3490720 𝑋𝑡−1 + 0.4877212 𝑋𝑡−2 + 𝑍𝑡 , 𝑍𝑡 ~ 𝑁(0,16.37169)

What We’ve Learned

 Estimating
model parameters of a
simulated AR(2) process using Yule-Walker
equations in a matrix form
 Estimating model
parameters – AR(3)
Simulation

PRACTICAL TIME SERIES ANALYSIS

THISTLETON AND SADIGOV
Objectives

 Estimate model parameters of a simulated AR(3)

process using Yule-Walker equations in matrix
form
AR(2) process (with mean zero)

𝑋𝑡 = 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + 𝜙3 𝑋𝑡−3 + 𝑍𝑡

where

𝑍𝑡 ~ 𝑁𝑜𝑟𝑚𝑎𝑙 (0, 𝜎𝑍2 )

We simulate this process for

1 1 7
𝜙1 = , 𝜙2 = , 𝜙3 = ,𝜎 = 4
3 2 100 𝑍
Yule –Walker equaitons
We (will) estimate coefficients of the model

𝑋𝑡 = 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + 𝜙3 𝑋𝑡−3 + 𝑍𝑡

by first finding 𝑟1 , 𝑟2 using acf() routine, then solving the system of

equations

𝑟1 1 𝑟1 𝑟2 𝜙1
𝑟2 = 𝑟1 1 𝑟1 𝜙2
𝑟3 𝑟2 𝑟1 1 𝜙3
𝜎𝑍 Estimation

Yule – Walker estimator for 𝜎𝑍2

𝜎𝑍2 = 𝑐0 (1 − 𝜙𝑖 𝑟𝑖 )
𝑖=1
 Results (set.seed(2017))
 n= 100000
 𝜙1 ≈ 0.3381245
 𝜙2 ≈ 0.4984999
 𝜙3 ≈ 0.06849712
 𝜎𝑍2 ≈ 15.979

Actual Model

𝑋𝑡 = 0. 3 𝑋𝑡−1 + 0.5𝑋𝑡−2 + 0.07𝑋𝑡−3 + 𝑍𝑡 , 𝑍𝑡 ~ 𝑁(0,16)

Fitted model

𝑋𝑡 = 0.3381245 𝑋𝑡−1 + 0.4984999 𝑋𝑡−2 + 0.06849712 𝑋𝑡−3 + 𝑍𝑡 ,

𝑍𝑡 ~ 𝑁(0,15.979)
What We’ve Learned

 Estimating
model parameters of a
simulated AR(3) process using Yule-Walker
equations in a matrix form
 Parameter estimation:
Recruitment

PRACTICAL TIME SERIES ANALYSIS

THISTLETON AND SADIGOV
Objectives

 To fit an AR(p) model to recruitment (number of new fish) for a

period of 453 months ranging over the years 1950-1987.
 Use Yule-Walker equations in matrix form to estimate parameters of
the fitted model
rec {astsa}

 Recruitment (number of new fish) for a period of 453 months ranging

over the years 1950-1987.
 Monthly time series
 Source: “astsa” package

Shumway, R.H. and Stoffer, D.S. (2000)

Time Series Analysis and its Applications
With R examples
Third Edition
Springer
ACF and PACF
The parsimony principle

 Choose ‘simplest explanation that fits the evidence’

 Simplest of competing theories is to be preferred
 PACF ⟹ AR(2)
 Yule-Walker equations in matrix form
Code
 ar.process=rec-mean(rec)

𝑋𝑡 − 𝜇

 p=2
 Yule-Walker equations: 𝑅 𝜙 = 𝑏
 Sample autocorrelation coefficients, vector 𝑟
for(i in 1:p+1){
r[i-1]<-acf(ar.process, plot=F)$acf[i]
}
Matrix 𝑅
1 𝑟1 𝑟2 𝑟𝑝−1
𝑟1 1 𝑟1 ⋯ 𝑟𝑝−2
𝑟2 𝑟1 1 𝑟𝑝−3
⋮ ⋱ ⋮
𝑟𝑝−2 𝑟𝑝−3 𝑟𝑝−4 𝑟1
𝑟𝑝−1 𝑟𝑝−2 𝑟𝑝−3 ⋯
1

Realize

𝑅 𝑖, 𝑗 = 𝑅𝑖𝑗 = 𝑟 𝑖−𝑗
 R=matrix(1,p,p) # matrix of dimension p by p, with entries all 1's.

 for(i in 1:p){
for(j in 1:p){
if(i!=j)
R[i,j]=r[abs(i-j)]
}
}
 # b-column vector on the right

b=matrix(,p,1)# b- column vector with no entries

for(i in 1:p){
b[i,1]=r[i]
}

 # solve(R,b) solves Rx=b, and gives x=R^(-1)b vector

phi.hat=NULL
for(i in 1:p){
phi.hat[i]=solve(R,b)[i,1]
}
Model

𝑋𝑡 − 𝑥 = 𝜙1 (𝑋𝑡−1 −𝑥) + 𝜙2 (𝑋𝑡−2 − 𝑥) + ⋯ + 𝜙𝑝 (𝑋𝑡−𝑝 −𝑥) + 𝑍𝑡

Thus

𝑋𝑡 = 𝜙0 + 𝜙1 𝑋𝑡−1 + 𝜙2 𝑋𝑡−2 + ⋯ + 𝜙𝑝 𝑋𝑡−𝑝 + 𝑍𝑡

where
𝑝

𝜙0 = 𝑥(1 − 𝜙𝑖 )
𝑖=1
 𝑝=2

Fitted model is

𝑋𝑡 = 7.033036 + 1.331587 𝑋𝑡−1 − 0.4445447 𝑋𝑡−2 + 𝑍𝑡

𝑍𝑡 ~ 𝑁𝑜𝑟𝑚𝑎𝑙 (0,94.17131)
 What We’ve Learned

 Fittingan AR(p=2) model to Recruitment

(number of new fish) from ‘astsa’ package
using Yule-Walker equations in matrix form
 Parameter estimation:
Johnson&Johnson (AR
attempt)

PRACTICAL TIME SERIES ANALYSIS

THISTLETON AND SADIGOV
Objectives

 To fit an AR(p) model to Quarterly earnings (dollars) per Johnson &

Johnson share 1960–80.
 Use Yule-Walker equations in matrix form to estimate parameters of
the fitted model
JohnsonJohnson {datasets}

 Quarterly earnings (dollars) per Johnson & Johnson share 1960–80.

 Quarterly time series
 Source: “astsa” package

Shumway, R.H. and Stoffer, D.S. (2000)

Time Series Analysis and its Applications
With R examples
Third Edition
Springer
Transformation
Log-return a time series {𝑋𝑡 }

is defined as

𝑋𝑡
𝑟𝑡 = log = log 𝑋𝑡 − log(𝑋𝑡−1 )
𝑋𝑡−1

In R,

𝑑𝑖𝑓𝑓(log )
The parsimony principle

 Choose ‘simplest explanation that fits the evidence’

 Simplest of competing theories is to be preferred
 PACF ⟹ AR(4)
 Yule-Walker equations in matrix form
 𝑝=4
Fitted model is

𝑟𝑡 = 0.079781 − 0.6293492 𝑟𝑡−1 − 0.5171526 𝑟𝑡−2 − 0.4883374 𝑟𝑡−3 + 0.2651266 𝑟𝑡−4 + 𝑍𝑡

𝑍𝑡 ~ 𝑁𝑜𝑟𝑚𝑎𝑙 (0, 0.01419242)

where

𝑋𝑡
𝑟𝑡 = log
𝑋𝑡−1
 What We’ve Learned

 Fitting
an AR(p=4) model to log-return of
Johnson & Johnson quarterly earnings
from ‘astsa’ package using Yule-Walker
equations in matrix form