Chapter 5

Control Theory for

Discrete-Time Systems

5.1 Minimal Variance Control

Difference Equations
Let the process be described by the ARMAX (AutoRegressive Moving Average
model with eXogenous inputs) model

y(t) + a1 y(t − 1) + · · · + any y(t − ny ) = b1 u(t − 1) + · · · + bnu u(t − nu )

+ e(t) + c1 e(t − 1) + · · · + cne e(t − ne ). (5.1)

Introduce the Equation (5.1) can be written in the form

y(t) = G(q)u(t) + H(q)e(t), (5.2)

where {e(t), t ∈ T } is a white noise stochastic process, and

B(q) C(q)
G(q) = , H(q) =
A(q) A(q)

with

A(q) = 1 + a1 q −1 + · · · + any q −ny ,

B(q) = b1 q −1 + · · · + bnu q −nu
C(q) = 1 + c1 q −1 + · · · + cne q −ne .

103
104CHAPTER 5. CONTROL THEORY FOR DISCRETE-TIME SYSTEMS

Transfer Functions
The process is descried by the time invariant dynamical system with one input
with a time delay and one output, subject to disturbances
Bp∗ (q −1 )
y(t) = u(t − k) + v(t)
A∗p (q −1 )
If it is further assume that the the disturbance v(t) is a stationary normal process
with rational strictly positive spectral density, it can be represented as
Cd∗ (q −1 )
v(t) = λ
A∗d (q −1 )
where {e(t), t ∈ T } is a sequence of independent and identically distributed normal
(0, 1) random variables and Cd and Ad are polynomials. The polynomials Ad and
Cd can always be chosen to ave their zeros inside the unit circle.
The system can be represented by the equation
Bp∗ (q −1 ) Cd∗ (q −1 )
y(t) = u(t − k) + λ e(t) (5.3)
A∗p (q −1 ) A∗d (q −1 )
To simplify the analysis, the model (5.3) is rewritten to a slightly different form,
namely,
A∗ (q −1 )y(t) = B ∗ (q −1 )u(t − k) + λC ∗ (q −1 )e(t)
where A, B and C are polynomials defined by A = Ap Ad , B = Bp Ad and C =
Cd Ap . Put
A(z) = z n + a1 z n−1 + · · · + an
B(z) = b0 z n + b1 z n−1 + · · · + bn
C(z) = z n + c1 z n−1 + · · · + cn

Solutions
Theorem 5.1 Consider the process described by
A∗ (q −1 )y(t) = B ∗ (q −1 )u(t − k) + λC ∗ (q −1 )e(t) (5.4)
where {e(t), t ∈ T } is a sequence of independent normal (0, 1) random variables.
Let the polynomial C(z) has all its roots inside the unit circle. The minimum
variance control law is then given by
B ∗ (q −1 )F ∗ (q −1 )u(t) = −G∗ (q −1 )y(t)
5.1. MINIMAL VARIANCE CONTROL 105

where the polynomials F and G of order k − 1 and n − 1, respectively, are defined

by the identity

C ∗ (q −1 ) = A∗ (q −1 )F ∗ (q −1 ) + q −k G∗ (q −1 ) (5.5)

The regulation error of the optimal system is a moving average of order k

y(t) = λF ∗ (q −1 )e(t)
= λ [e(t) + f1 e(t − 1) + · · · + fk−1 e(t − k + 1)]

Proof.
G∗ (q −1 ) B ∗ (q −1 )F ∗ (q −1 )
y(t + k) = λF ∗ (q −1 )e(t + k) + ∗ −1
y(t) + u(t) (5.6)
C (q ) C ∗ (q −1 )

Remark 5.2 A comparion with the solution of the prediction problem shows that
the term
G∗ (q −1 ) B ∗ (q −1 )F ∗ (q −1 )
y(t) + u(t)
C ∗ (q −1 ) C ∗ (q −1 )

of (5.6) can be interpreted as the k-step prediction of y(t + k) based on y(t), y(t −
1), . . . and that the control error equals the k-step prediction error.

Remark 5.3 C ∗ (z)

5.1.1 Non-Minimum Phase Systems

Assume that the system is governed by

A(q)y(t) = B(q)u(t − k) + λC(q)e(t). (5.7)

Assume that B(z) can be factored as

B(z) = B1 (z)B2 (z)

where B1 is of degree n1 and has all roots inside the unit circle and B2 is of degree
n2 and has all roots outside the unit circle.
The minimal variance control strategy is

G∗ (q −1 ) q k G(q)
u(t) = − y(t) = − y(t) (5.8)
B ∗ (q −1 )F ∗ (q −1 ) B(q)F (q)
106CHAPTER 5. CONTROL THEORY FOR DISCRETE-TIME SYSTEMS

where F and G are polynomials of degree k − 1 and n − 1 defined by the identity

(5.5). Introducing (5.8) into (5.7) yields
 
B(q)G(q)
A (q) + y(t) = λC(q)e(t) (5.9)
B(q)F (q)

Let q n+k−1 operate on the identity (5.5) and use the definition of reciprocal poly-
nomials. We find

q k−1 C(q) = A(q)F (q) + G(q) (5.10)

Equations (5.9) and (5.10) now give

q k−1 B(q)C(q)y(t) = λB(q)C(q)F (q)e(t).

The transfer function of the closed-loop system is thus

Y (z) λB(z)C(z)F (z)

= k−1 (5.11)
E(z) z B(z)C(z)

where there are k − 1 modes with poles at the origin, n modes with poles at the
roots of B, and n modes with poles at the roots of C. Furthermore, the factor BC
cancels in (5.11). This implies that the modes associated with BC are unobservable
from the output y, or that the corresponding state variables are not controllable
from the input e. Hence, the input might excite all the modes associated with the
solutions of the characteristic equation

z k−1 B(z)C(z) = 0.

This is not a serious matter if the modes are stable. However, if some modes are
unstable, it is possible to get infinitely large errors. This situation will occur if
the polynomial BC has roots outside or on the unit circle. It follows from the
representation theorems for stationary random processes that C can always be
chosen to have zeros inside or on the unit circle. As far as C is concerned the only
critical case would be if C has a zero on the unit circle. The polynomial B will
have zero outside the unit circle if the dynamics of the system is non-minimum
phase.
There are many ways to obtain control laws for non-minimum phase systems.
One possibility is to proceed as follows.
When solving the identity (5.5), the following identity is used

C ∗ (q −1 ) = A∗ (q −1 )F ∗ (q −1 ) + q −k B2∗ (q −1 )G∗ (q −1 ), (5.12)

5.1. MINIMAL VARIANCE CONTROL 107

that is,

q n2 +k−1 C(q) = A(q)F (q) + B2 (q)G(q)

instead of (5.10). Going through the arguments used when deriving Theorem 5.1,
the control law is given by

G∗ (q −1 ) q k G(q)
u(t) = − =− y(t) (5.13)
B1∗ (q −1 )F ∗ (q −1 ) B1 (q)F (q)

which gives the control error

y(t) =λ [e(t) + f1 e(t − 1) + · · · + fk−1 e(t − k + 1)

+fk e(t − k) + · · · + fk+n2 −1 e(t − k − n2 + 1)] . (5.14)

The control law (5.13), which is not optimal, gives an error with the variance
2
var[y] = min var[y] + λ2 fk2 + · · · + fk+n
 
2 −1
.

The transfer function of the closed-loop system then becomes

Y (z) λB1 (z)C(z)F (z)
= n2 +k−1 (5.15)
E(z) z B1 (z)C(z)
It now follows from the definition of B1 and the assumption that C has no roots
on the unit circle, that all modes are stable.
A comparison of (5.11) and (5.15) shows that the control law (5.13) brings the
roots of B2 (z) to the origin. There are other control strategies which instead of
move the roots of B2 to arbitrary positions inside the unit circle.

5.1.2 Sensitivity Analysis

Assume that the system is governed by

Ao (q)y(t) = Bo (q)u(t − k) + λo Co (q)e(t), (5.16)

but that the control law

q k G(q)
u(t) = − y(t) (5.17)
B(q)F (q)
is calculated under the assumption that the system model is

A(q)y(t) = B(q)u(t − k) + λC(q)e(t),

108CHAPTER 5. CONTROL THEORY FOR DISCRETE-TIME SYSTEMS

where the coefficients of A, B and C differ slightly from those of Ao , Bo and Co .

Let us investigate what happens if the system (5.16) is controlled with the control
law (5.17). Introducing (5.17) into (5.16) yields
 
Bo (q)G(q)
Ao (q) + y(t) = λo Co (q)e(t) (5.18)
B(q)F (q)
From (5.10), (5.18) now gives
n o
q k−1 Bo (q)C(q) + [Ao (q)B(q) − A(q)Bo (q)] F (q) y(t) = λo B(q)Co (q)F (q)e(t)

The closed-loop transfer function is thus

λo B(z)Co (z)F (z)
H(z) =
z k−1 B o (z)C(z) + [Ao (z)B(z) − A(z)Bo (z)] F (z)

The solutions of the characteristic equation

q k−1 Bo (q)C(q) + [Ao (q)B(q) − A(q)Bo (q)] F (q) = 0

must be within the unit circle. The variance of the output is then given by
I
1 dz
E y2 = H(z)H(z −1 )
 
2πj z
If the optimal system is sensitive to parameter variations, the control law (5.13)
can be used to make the closed-loop system less sensitive to parameter variations.
Assume that the system is governed by the model (Ao , Bo , Co , λo ) but the con-
trol law is calculated from the model (A, B, C, λ) with slightly different parameters.
The equation describing the controlled system then becomes

[Ao (q)B1 (q)F (q) + Bo (q)G(q)] y(t) = λo B1 (q)Co (q)F (q)e(t)

Example 5.4 Consider a system governed by

y(t) + ao y(t − 1) = u(t − 1) + bo u(t − 2) + e(t) + co e(t − 1).

with ao = −0.7 and co = 0.95. The minimum variance control strategy for this
system is obtained from a = ao , b = 0.99, c = co . Use the identity (5.5)

1 + cq −1 = 1 + aq −1 + q −1 g.

Hence, g = c − a = 1.65. The minimal variance strategy is

g 1.65
u(t) = − −1
y(t) = − y(t) (5.19)
1 + bq 1 + 0.99q −1
5.2. LINEAR QUADRATIC CONTROL 109

Using this control law, the output is given by

q 2 + (b + co ) q + bco
y(t) = e(t)
q 2 + (ao − a + b + c)q + ao b − abo + bo c
q 2 + 1.94q + 0.9405
= 2 e(t)
q + 1.94q + 1.65bo − 0.693
The closed-loop system is stable for 0.9897 ≤ bo ≤ 1.026.
To obtain a control law which is not optimal but which is less sensitive for
variations in the bo -parameter, we can use the control law (5.13). The identity
(5.12) gives
1 + cq −1 = (1 + aq −1 )(1 + f q −1 ) + q −1 (1 + bq −1 )g.
Hence,
a−c 3267 a−c 231
f =b = = 0.9666, g = −a = = 0.6834
a−b 3380 a−b 338
and the control law (5.13) becomes
g 2310
u(t) = − y(t) = − y(t) (5.20)
1 + f q −1 3380 + 3267q −1
It follows from (5.14) that the control strategy (5.20) thus gives the variance
22097689
var [y] = = 1.9343
11424400
Compare with the minimal variance 1.
Assuming that the control strategy is evaluated from a model whose parameter
bo differs from the value b of the system, we find the control error is
q 2 + (co + f )g + co f
y(t) = λ e(t)
q 2 + (ao + f + g)q + ao f + bo g
The closed-loop system is stable for 0.9169 ≤ bo ≤ 2.4532. The control law (5.20)
is much less sensitive to variations in the parameter bo than the optimal control
law (5.19).

5.2 Linear Quadratic Control

5.2.1 Problem Formulation
Consider a system governed by the stochastic state-space equation
x(t + 1) = Φx(t) + Γu(t) + v(t) (5.21)
110CHAPTER 5. CONTROL THEORY FOR DISCRETE-TIME SYSTEMS

where t ∈ T , x is an n × 1 state vector, u is a p × 1 vector of control variables,

v(t), t ∈ T is a seuqnece of independent normal random variables with zero mean
and covariance

cov [v(t), v(t)] = Rv .

It is assumed that v(t) isindependent of x(t) and the initial state x(t0 ) is normal
with

E [x(t0 )] = m
cov [x(t0 ), x(t0 )] = R0

The performance of the system is characterized by the expected loss

N −1
" #
X
T T T
E [J] = E x (N )QN x(N ) + x (t)Qx x(t) + u (t)Qu u(t)
t=t0

To complete the probem statement, we must also specify the admissible control
strategy. For a stochastic control problem, it is very important to specify the data
which is available for determining the control action.

Complete State information

If the output signal at time t will give the exact value of the state vector

y(t) = x(t)

this situation is referred to as complete state information.

Incomplete State Information

y(t) = θx(t) + e(t)

YtT = y T (t0 ), y T (t0 + 1), . . . , y T (t)

 

5.2.2 Preliminaries
Lemma 5.5 Let x be normal with mean m and covariance R. Then

E xT Sx = mT Sm + trSR
 
5.2. LINEAR QUADRATIC CONTROL 111

Consider the system (5.21).

Lemma 5.6 Assume that the difference equation

−1 T
S(t) = ΦT S(t + 1)Φ + Qx − ΦT S(t + 1)Γ Qu + ΓT S(t + 1)Γ

Γ S(t + 1)Φ
(5.22)

with the initial condition

S(N ) = QN (5.23)

has a solution which is positive semidefinite for t0 ≤ t ≤ N . Let the matrix L be

defined by
−1 T
L(t) = Qu + ΓT S(t + 1)Γ

Γ S(t + 1)Φ (5.24)

Then
N −1
T T T
X
x (N )QN x(N ) + x (t)Qx x(t) + u (t)Qu u(t)
t=t0

N −1 h i
T T T
X
= x (t0 )S(t0 )x(t0 ) + [u(t) + L(t)x(t)] Γ S(t + 1)Γ + Qu [u(t) + L(t)x(t)]
t=t0

N −1 n o
T T
X
+ v (t) S(t + 1) [Φx(t) + Γu(t)] + [Φx(t) + Γu(t)] S(t + 1) + S(t + 1) v(t)
t=t0

5.2.3 Complete State Information

Theorem 5.7 Consider the system (5.21). Let the admissible control strategies
be such that u(t) is a function of x(t). Assume that (5.22) with initial condition
(5.23) has a solution S such that S is positive semidefinite and Qu + ΓT SΓ is
positive definite for all t. Then, there exists a unique admissible control strategy

u(t) = −L(t)x(t) (5.25)

where the matrix L is defined in (5.24), which minimizes the expected loss. The
minimal value of the expected loss is

N
X −1
T
min E [J] = m S(t0 )m + trS(t0 )R0 + trS(t + 1)Rv (t)
t=t0

Proof. in the case of

112CHAPTER 5. CONTROL THEORY FOR DISCRETE-TIME SYSTEMS

5.2.4 Incomplete State Information

Lemma 5.8 Let E [·|y] denote the conditional mean given y. Assume that the
function f (y, u) = E [J(x, y, u)|y] has a unique minimum with respect to u ∈ U for
all y ∈ Y. Let u∗ denote the value of u for which the minimum is achieved. Then

min E [J(x, y, u)] = E [J(x, y, u∗ )] = Ey {E [J(x, y, u)|y]}

u(y)

where Ey denotes the mean value with respect to the distribution of y.

Theorem 5.9 Consider the system (5.21). Let the admissible control strategies
be such that u(t) is a function of Y(t). Assume that (5.22) with initial condition
(5.23) has a solution S such that S is positive semidefinite and Qu + ΓT SΓ is
positive definite for all t. Then, there exists a unique admissible control strategy

u(t) = −L(t)x̂(t) = −L(t)E [x(t)|Yt−1 ] (5.26)

where the matrix L is defined in (5.24), which minimizes the expected loss. The
minimal value of the expected loss is
N
X −1
min E [J] = mT S(t0 )m + trS(t0 )R0 + trS(t + 1)Rv (t)
t=t0
N
X −1
trP (t)LT (t) ΓT S(t + 1)Γ + Qu L(t)
 
+
t=t0

where P is the conditional covariance

P (t) = cov [x(t)|Yt−1 ] .