Multivariate Singular Spectrum Analysis: Andreas Groth

Multivariate Singular Spectrum Analysis
Andreas Groth
Laboratoire de Météorologie Dynamique

Ecole Normale Supérieure, Paris
Oldenburg, 17.03.2010
Univariate SSA Multivariate SSA Monte-Carlo (M)SSA Conclusions
Motivation
Given
• Short data sets
• Different scales in space and time
• Contain observation errors
• Repetition not always possible (maybe under control of parameters)
Wanted
• Distinguish between regular deterministic behavior “cycles” and
irregular behavior “noise”
• Extract a skeleton of the underlying system
• Understand and model the underlying mechanism
Multivariate Singular Spectrum Analysis Andreas Groth, LMD, ENS, Paris 2/27
Motivation
U.S. macroeconomic data from the Bureau of Economic Analysis — Trend residuals after Hodrick-Prescott detrending
Outline
1 Univariate Singular Spectrum Analysis (SSA)

General idea of time-delayed embedding
Combination with Principal Component Analysis (PCA)
Properties and interpretation
2 Multivariate Singular Spectrum Analysis (M-SSA)

Extraction of shared behavior
Reconstruct skeleton of attractor
3 Verification by Monte Carlo (M)SSA

Problem of short and noisy observations
Alternative hypotheses of oscillation-like fluctuations
4 Conclusions
Univariate Singular Spectrum Analysis (SSA)



4 Conclusions
Problem • Underlying dynamical systems ẏ = F (y) is unknown

• We obtain a scalar measurement x(t) with t = 1 . . . N
from this system x(t) = H(y(t))

Idea • Build a new M -dimensional time series from x(t) and
its lagged copies
x(t) = (x(t), x(t + 1), . . . , x(t + M − 1))

t = 1...N − M + 1
• These delay coordinates x share key topological

properties with y (Whitney 1936, Mañé 1981, Takens 1981 ...)

Idea • Build a new M -dimensional time series from x(t) and
its lagged copies
x(t) = (x(t), x(t + 1), . . . , x(t + M − 1))

t = 1...N − M + 1
• These delay coordinates x share key topological

properties with y (Whitney 1936, Mañé 1981, Takens 1981 ...)
Question I How to extract those properties from short and noisy

time series?
Idea Apply PCA to x(t) in order to extract the entire attractor

of the (nonlinear) system
(Broomhead and King, 1986)

However An original motivation to interpret the results in terms of

attractor dimensions fails even in relatively simple cases!
(Vautard and Ghil, 1989)


Anyway The idea to reconstruct the skeleton of the attractor, e.g.

the most robust, albeit unstable limit cycles by means of
PCA remains promising (Vautard and Ghil 1989, Ghil and Vautard
1991)


Anyway The idea to reconstruct the skeleton of the attractor, e.g.

the most robust, albeit unstable limit cycles by means of
PCA remains promising (Vautard and Ghil 1989, Ghil and Vautard
1991)
Singular Spectrum Analysis

• Time-delayed embedding of scalar time series + PCA
Eigenvalues and eigenvectors
1 Delay embedding of
x(t) → x(t) = (x(t), x(t + 1), . . . , x(t + M − 1)) ∈ RM
x(t) → x(t) = (x(t), x(t + 1), . . . , x(t + M − 1)) ∈ RM
2 Estimate covariance matrix C of x (of size M × M ) with elements

N −|i−j|
1 X
cij = x(t)x(t + |i − j|)
N − |i − j|
t=1
x(t) → x(t) = (x(t), x(t + 1), . . . , x(t + M − 1)) ∈ RM

N −|i−j|
1 X
cij = x(t)x(t + |i − j|)
N − |i − j|
t=1
3 Determine eigenvalues and eigenvectors (λk , ek ) of C

Cek = λk ek k = 1...M
|
E CE = Λ in matrix notation
x(t) → x(t) = (x(t), x(t + 1), . . . , x(t + M − 1)) ∈ RM

N −|i−j|
1 X
cij = x(t)x(t + |i − j|)
N − |i − j|
t=1

|
I λk equals the variance of x in the direction of ek
x(t) → x(t) = (x(t), x(t + 1), . . . , x(t + M − 1)) ∈ RM

N −|i−j|
1 X
cij = x(t)x(t + |i − j|)
N − |i − j|
t=1

|
I λk equals the variance of x in the direction of ek
I Sum of all eigenvalues equals the total variance of the original time
series x
Spectrum of eigenvalues
• Eigenvalues 1–4 explain 50% of

variance
• Separate signal from noise
• Find a “break” point in the
spectrum of eigenvalues
BUT Such a “break” point is not
obvious for complex time series!
Principal components and reconstructed components
• Project x onto eigenvectors gives principal components (PC)
M
X
Ak (t) = x(t + j − 1)ek (j)
j=1
M
X
Ak (t) = x(t + j − 1)ek (j)
j=1
• Reconstruct that part of x that belongs to certain eigenvectors
M
1 X
rk (t) = Ak (t − j + 1)ek (j)
M
j=1
gives reconstructed components (RC)
M
X
Ak (t) = x(t + j − 1)ek (j)
j=1
• Reconstruct that part of x that belongs to certain eigenvectors
M
1 X
rk (t) = Ak (t − j + 1)ek (j)
M
j=1
gives reconstructed components (RC)

I Especially, sum of all RCs gives original time series!
• RCs 1–4 explain already 50% of

total variance
• RCs 1–2 and 3–4 form
oscillatory pairs

• Stepwise reconstruction of the dynamical behavior
• Reconstruct ghost limit cycles of dynamical systems (albeit unstable,

the orbit is visited by system’s trajectory)
(Kimoto and Ghil, 1993; Ghil and Yiou, 1996)

SSA Wavelet analysis

Analyzing function Eigenvectors ek Mother wavelet Ψ
Basis function ek data-adaptive Ψ chosen a priori
PM R
Decomposition j=1 x(t + j − 1)ρk (j) x(t)Ψ(t − b/a)dt
Scale M a
Epoch t b
(after Ghil et al., 2002)
SSA Wavelet analysis

Analyzing function Eigenvectors ek Mother wavelet Ψ
Basis function ek data-adaptive Ψ chosen a priori
PM R
Decomposition j=1 x(t + j − 1)ρk (j) x(t)Ψ(t − b/a)dt
Scale M a
Epoch t b
(after Ghil et al., 2002)
• Eigenvectors are symmetric/antisymmetric

• Eigenvectors can form pairs, analog to sine/cosine-pairs in Fourier
analysis
• Eigenvectors can be considered as FIR-filters (Finite Impulse
Response), which extract a narrow frequency band of x
I Therefore, assign a frequency to each eigenvector

Eigenvectors — Data-adaptive filter
Advantages
• The basis of decomposition is not chosen a priori
• This gives more flexibility to the analysis
• However, requires more post analysis in order to interpret and
understand the outcome
Multivariate Singular Spectrum Analysis (M-SSA)



4 Conclusions
Motivation
U.S. macroeconomic data from the Bureau of Economic Analysis — Trend residuals after Hodrick-Prescott detrending
• Multivariate SSA is a natural extension of SSA

(Broomhead and King, 1986b)
• It extract principal patterns in time as well as space

Starting point Multivariate time series
X(t) = (x1 (t), x2 (t), . . . , xD (t)), t = 1...N

Starting point Multivariate time series
X(t) = (x1 (t), x2 (t), . . . , xD (t)), t = 1...N
Covariance matrix of all pairs of time series

 
C1,1 C1,2 . . . C1,D
 C2,1 C2,2 . . . C2,D 
Ce = 
 .. .. .. 
 . . Cd,d0 . 
CD,1 CD,2 . . . CD,D
of size D M × D M
Multivariate SSA — “Same” procedure as before
Eigenelements: Determine eigenvalues and eigenvectors of C

e
Ce ek ,
e ek = λk e k = 1...DM
Principal components: Project X(t) onto e

ek
D X
X M
Ak (t) = xd (t + j − 1)edk (j)
d=1 j=1
| {z }
SSA
Reconstructed components: Combine PCs with e

ek
M
1 X
rkd (t) = Ak (t − j + 1)edk (j)
M
j=1
• RCs 1–2 form oscillatory pair with period of 5.2 years

• During recessions this cycle explains most of the time series’
variance* (Groth et al., 2010)
(* Local variance fraction, Plaut and Vautard 1994)

Verification by Monte Carlo (M)SSA



4 Conclusions
Warning
• Time series is short and noisy
• Only a single observation is available
• Experiment cannot be repeated (maybe under control of parameters)
GDP
Warning
• Time series is short and noisy
• Only a single observation is available
• Experiment cannot be repeated (maybe under control of parameters)
GDP
Random walk (detrended)
Surrogate data
Generate similar time series without oscillations but otherwise close to x
1 Bad: White noise → far from x, obviously rejected
2 Better: Red noise, here in terms of AR(1) process → Closer to x
z(t) = a z(t − 1) + σξ(t) ξ white noise
Surrogate data
Generate similar time series without oscillations but otherwise close to x
1 Bad: White noise → far from x, obviously rejected
2 Better: Red noise, here in terms of AR(1) process → Closer to x
z(t) = a z(t − 1) + σξ(t) ξ white noise
Monte Carlo SSA

Measure resemblance of z with x (Allen and Smith, 1996)
1 Estimate covariance matrix of z: Cz

2 Project onto eigenvectors of x: Λz = E| Cz E
3 Compare statistics on diagonal elements of Λz with original
eigenvalues Λ
Advantages of multivariate analysis

SSA of GDP M-SSA of 5 time series


• Single time series cannot be distinguished from red noise

• Eigenvalues of GDP lie within natural variability of red noise
• Multivariate time series have significant oscillatory pair that exceeds
red noise
• M-SSA helps to reveal ghost limit cycle
Conclusions
Advantages
• SSA and M-SSA help to distinguish between regular deterministic
behavior “cycles” and irregular behavior “noise”
• Data-adaptive method with no a-priori basis functions
• Extracting of a so-called “statistical dimension”
• Reconstruct skeleton of attractor
• M-SSA helps to reveal ghost limit cycles with higher confidence than
univariate analysis
Conclusions
Advantages
• SSA and M-SSA help to distinguish between regular deterministic
behavior “cycles” and irregular behavior “noise”
• Data-adaptive method with no a-priori basis functions
• Extracting of a so-called “statistical dimension”
• Reconstruct skeleton of attractor
• M-SSA helps to reveal ghost limit cycles with higher confidence than
univariate analysis
Verification of results
• Check robustness of pairs by changing embedding dimension M
• Apply statistical significance tests, Monte Carlo (M)SSA
• Do not consider (M)SSA as stand-alone method ⇒ Apply additional
methods (Fourier, Wavelet) and their tests
References
Review Ghil et al. (2002): “Advanced spectral methods for climatic time series,”
Reviews of Geophysics, 40(1), 1–41.
SSA Broomhead & King (1986a): “Extracting qualitative dynamics from
experimental data,” Physica D, 20(2-3), 217–236.
M-SSA Broomhead & King (1986b): “On the qualitative analysis of experimental
dynamical systems,” in Nonlinear Phenomena and Chaos, ed. by S. Sarkar,
pp. 113–144. Adam Hilger, Bristol, England.
MC-SSA Allen & Smith (1996): “Monte Carlo SSA: Detecting irregular oscillations in
the Presence of Colored Noise.” Journal of Climate, 9, 3373–3404
I Ghil & Vautard (1991): “Interdecadal oscillations and the warming trend in
global temperature time series,” Nature, 350(6316), 324–327.
I Plaut & Vautard (1994): “Spells of Low-Frequency Oscillations and Weather
Regimes in the Northern Hemisphere,” Journal of the Atmospheric Sciences,
51(2), 210–236.
I Vautard & Ghil (1989): “Singular spectrum analysis in nonlinear dynamics,
with applications to paleoclimatic time series,” Physica D, 35(3), 395–424.

Multivariate Singular Spectrum Analysis: Andreas Groth

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Multivariate Singular Spectrum Analysis: Andreas Groth

Uploaded by

Copyright:

Available Formats

Multivariate Singular Spectrum Analysis

Laboratoire de Météorologie Dynamique

1 Univariate Singular Spectrum Analysis (SSA)

2 Multivariate Singular Spectrum Analysis (M-SSA)

3 Verification by Monte Carlo (M)SSA

Univariate Singular Spectrum Analysis (SSA)

1 Univariate Singular Spectrum Analysis (SSA)

2 Multivariate Singular Spectrum Analysis (M-SSA)

3 Verification by Monte Carlo (M)SSA

General idea of time-delayed embedding

Problem • Underlying dynamical systems ẏ = F (y) is unknown

General idea of time-delayed embedding

Problem • Underlying dynamical systems ẏ = F (y) is unknown

x(t) = (x(t), x(t + 1), . . . , x(t + M − 1))

• These delay coordinates x share key topological

General idea of time-delayed embedding

Problem • Underlying dynamical systems ẏ = F (y) is unknown

x(t) = (x(t), x(t + 1), . . . , x(t + M − 1))

• These delay coordinates x share key topological

Question I How to extract those properties from short and noisy

Combination with Principal Component Analysis (PCA)

Idea Apply PCA to x(t) in order to extract the entire attractor

Combination with Principal Component Analysis (PCA)

Idea Apply PCA to x(t) in order to extract the entire attractor

However An original motivation to interpret the results in terms of

Combination with Principal Component Analysis (PCA)

Idea Apply PCA to x(t) in order to extract the entire attractor

However An original motivation to interpret the results in terms of

Anyway The idea to reconstruct the skeleton of the attractor, e.g.

Combination with Principal Component Analysis (PCA)

Idea Apply PCA to x(t) in order to extract the entire attractor

However An original motivation to interpret the results in terms of

Anyway The idea to reconstruct the skeleton of the attractor, e.g.

Singular Spectrum Analysis

Eigenvalues and eigenvectors

x(t) → x(t) = (x(t), x(t + 1), . . . , x(t + M − 1)) ∈ RM

Eigenvalues and eigenvectors

x(t) → x(t) = (x(t), x(t + 1), . . . , x(t + M − 1)) ∈ RM

2 Estimate covariance matrix C of x (of size M × M ) with elements

Eigenvalues and eigenvectors

x(t) → x(t) = (x(t), x(t + 1), . . . , x(t + M − 1)) ∈ RM

2 Estimate covariance matrix C of x (of size M × M ) with elements

3 Determine eigenvalues and eigenvectors (λk , ek ) of C

Eigenvalues and eigenvectors

x(t) → x(t) = (x(t), x(t + 1), . . . , x(t + M − 1)) ∈ RM

2 Estimate covariance matrix C of x (of size M × M ) with elements

3 Determine eigenvalues and eigenvectors (λk , ek ) of C

Eigenvalues and eigenvectors

x(t) → x(t) = (x(t), x(t + 1), . . . , x(t + M − 1)) ∈ RM

2 Estimate covariance matrix C of x (of size M × M ) with elements

3 Determine eigenvalues and eigenvectors (λk , ek ) of C

• Eigenvalues 1–4 explain 50% of

Principal components and reconstructed components

• Project x onto eigenvectors gives principal components (PC)

Principal components and reconstructed components

• Project x onto eigenvectors gives principal components (PC)

• Reconstruct that part of x that belongs to certain eigenvectors

gives reconstructed components (RC)

Principal components and reconstructed components

• Project x onto eigenvectors gives principal components (PC)

• Reconstruct that part of x that belongs to certain eigenvectors

gives reconstructed components (RC)