Adama Science and Technology

University

PCE6101 Linear Systems

Theory

ASTU
Department of EPCE
 Adama Science and Technology University School of Electrical Engineering and Computing
Department of Electrical Power and Control Engineering
Course Code EPCE6101
Course Title Linear systems theory
Degree Program MSc in Electrical Engineering (Control Engineering)
Credits 3
Contact Hours/week Lecture Tutorial P ra c tic e /L a b o ra to ry
2 3 -
Course Instructor (s)
Address Building: Room: Tel: +251 E-mail:

Course Objectives:
Upon successful completion of the course, students will be able to
 apply linear algebra knowledge
 get the knowledge on linear process dynamics.
 mathematically model linear dynamic process
 stabilize the linear systems.
 design feedback controller and state estimator.
 Realize linear systems
Course Description/Course Contents

 Review of Linear Algebra: Matrix Operators; Norms; Characteristic Values and Vectors; Modal
Matrix; Similarity Transformation; Jordan Canonical Form;
 State Space Representation of Continuous and Discrete Systems in the Time and Frequency
domains;
 Computation of the transition Matrix; Numerical Computation of the transition Matrix;
 The Adjoint System.
 The solution of the State-Space Equation (Continuous and Discrete);
 Stability of Linear Systems;
 Controllability and Observability;
 Minimum Realization;
 State feedback and estimator design; Pole-Assignment and model matching.

Advanced Topics in Linear Systems Theory:

NB ! Latest/recent developments regarding the specified course applications can be incorporated.
Pre-requisites None
Semester First year –I semester
Status of Course General Mandatory
Teaching & Learning methods Lecture supported by tutorial, assignment, term paper
Measurement Value/Mark (%)
Class-work 10
Assignments/ 10
Assessment/Evaluation Test (s)/Quiz (s) 20
Mini-project/ term paper 10
Examination (final) 50
Total 100
Minimum 80% during lecture and tutorial sessions is required except
Attendance Requirements
for some unprecedented mishaps
Textbooks and References
1. Chi-Tsong Chen, “Linear System Theory and Design”, 3rd edition, saunders college
publishing, 1999.
2. Kemin Zhou, with John Doyle, Essentials of Robust Control, Prentice-Hall, 1998.
 Chapter 1: Review of Linear Alge-
bra
1.1 Matrices
(Definition) Matrix
(1) Square matrix
• Matrix with the same number
of rows and columns.
• Also denoted as A= [aij].
 a11 a12  a1n 
a  a2 n 
 21 a22
A
    
 
 an1 an 2  ann  (German Mathematician and sci-
entist, 1777~1855)
 1.1 Matrices
(2) Symmetric matrix
8 0 5 

A  0 2 12  
• Matrix where aij= aji
5 12 4 
is satisfied

• Sum of a matrix A and its transpose matrix AT is al-

ways symmetric, B= A+AT.
3 2 5  3 0 1  6 2 6 
B  0 3 2    2 3 9    2 6 11 
1 9 1  5 2 1  6 11 2 
 1.1 Matrices
(3) Skew-symmetric matrix 0 2 1

A   2 0 4 
• Matrix where aij= -aji
 1 4 0 
• Diagonal elements are aii= 0
• The difference between a matrix A and AT is always
skew-symmetric, B= A-AT.
3 2 5  3 0 1   0 2 4 
    
B  0 3 2    2 3 9    2 0 7  
1 9 1  5 2 1  4 7 0 
A AT
 1.1 Matrices
(Definition) Cofactor, Cij
The determinant of a matrix whose ith rows and jth
columns are removed from a square matrix A= [aij] is
called aij's cofactor and is expressed as Cij= (-1)i+j|A(i|j)|

1 2 3 1 2 3

A  4 5 6  C22  (1) 2  2 4 5 6  (1 0  1 3)
1 1 0
1 1 0 
 3
 1.1 Matrices
(Definition) Determinant, det(A) or |A|
The determinant of a square matrix A can be obtained
using the cofactor expansion of the ith row or jth col-
umn.
|A|= ai1Ci1+ai2Ci2+...+ainCin

= a1jC1j+a2jC2j+...+anjCnj
| A | 0  C21  5  C22  0  C23
 2 3 4 
 5  (2  24)  110
A   0 5 0 
| A | 4  C13  0  C23  1 C33
 6 5 1   4  (0  30)  (10  0)  110
 1.1 Matrices
(Definition) Adjoint matrix
It is defined as transpose of the matrix consisting of co-
factor Cij, and is expressed as follows:
T
 C11 C12  C1n 
C 
 C2 n 
 21 C22
Adj( A) 
    
 
Cn1 Cn 2  Cnn 
 1.1 Matrices
For example
11 5 6
C11  (1)  (5  0  1 6)  6
1 2 3 1 0

A  4 5 6  4 6
1 2
C12  (1)  (4  0  1 6)  6
1 0
1 1 0 
1 3 4 5
C13  (1)  (4  1  1 5)  1
1 1

2 3 1 3 1 2
C21    3, C22   3, C23   1
1 0 1 0 1 1
2 3 1 3 1 2
C31   3, C32    6, C33   3
5 6 4 6 4 5
 1.1 Matrices
T T
 C11 C12 C13   6 6 1

Adj( A)  C21  
C22 C23    3 3 1  
C31 C32 C33   3 6 3
 6 3 3

6 
3 6 
 1 1 3
22 matrix  a11 a12   a22 a12 
A   Adj( A)   
 a21 a22   a21 a11 
 1.1 Matrices
(Theorem) Inverse matrix
The inverse of a square matrix A can be obtained as

1 Adj( A)
A 
| A|
 2 3
A 
 0 5 
| A | 2  5  10
5 3 1 1 5 3 
Adj( A)    A   
 0 2  10  0 2 
 1.1 Matrices
(Theorem) Properties of inverse matrix
If A and B are nn nonsingular matrices, the following
properties are satisfied.
1. A-1A= I
2. (A-1)-1= A
3. (AB)-1= B-1A-1
4. |A-1A|=|A-1||A|= 1
5. (A-1)T= (AT)-1
 1.1 Matrices
(Theorem) Partition of matrix
An nm matrix A and an mp matrix B can be parti-
tioned as follows:
 a11 a12  a1m   A1 
a   
 a2 m   A2 
 21 a22
A 
       
   
 an1 an 2  anm   An 
 b11 b12  b1 p 
 
 b21 b22  b2 p 
B    B1 B2  B p 
    
bm1 bm 2  bmp 
 1.1 Matrices
AB  A B1 B2  Bm    A B1 AB2  ABm 
 A1   A1 B 
A   A B
  2B   2 
    
   
For example  An   An B 
1 2
1 1 0   
A  , B  3 0
0 5 3  2 4 
 A1 B   4 2 
AB   AB1 AB2      
 A2 B   21 12 
 1.1 Matrices
It can be confirmed from the following results.
1  2
1 1 0     4  1 1 0     2 
AB1     3     , AB2     0   
0 5 3   2   21 0 5 3   4  12 
   
1 2
A1 B  1 1 0  3 0   4 2,
 2 4 
1 2 
A2 B  0 5 3  3 0   21 12 
 2 4 
 1.1 Matrices
(Theorem) Determinant of partitioned matrix
If F is a partitioned matrix and its inverse exits, then
the determinant of F is given by

A B 1
F    | F | | A |  | D  CA B |
C D 
1
 | D |  | A  BD C |
 1.1 Matrices
For  2 3 4 
example F   0 5 0 
 6 5 1 

The determinant of F is

A B 1
F    | F | | D |  | A  BD C |
C D 
 2 3  4  22 17
 |     6 5 |  110
0 5  0 0 5
 1.1 Matrices
(Definition) Matrix differentiation
Matrix differentiation is defined as the first, second,…
derivatives of each element.
 da11 (t ) da12 (t ) da1n (t ) 
 dt 
dt dt 
 
 da21 (t ) da22 (t ) da2 n (t ) 
d 
A(t )   dt dt dt 
dt  
     
 da (t ) dan 2 (t ) dann (t ) 
 n1  
 dt dt dt 
 1.1 Matrices
(Theorem) Properties of matrix differentiation
For A(t) and B(t) of appropriate dimensions
d d d
1. [ A(t )  B (t )]  A(t )  B (t )
dt dt dt
d d d
2. [ A(t ) B (t )]  A(t )  B (t )  A(t )  B (t )
dt dt dt
For nn matrix A(t)
d 1 d
3. A (t )   A (t )  A(t )  A1 (t )
1
dt dt
 1.1 Matrices
(Proof) 3. From A(t)A-1(t)= I and Property 2

A(t ) A1 (t )  I
d d
[ A(t ) A (t )]  A(t )  A1 (t )
1
dt dt
d 1
 A(t )  A (t )  0
dt
Rearrange the right term gives

d 1 d
 A (t )   A (t )  A(t )  A1 (t )
1
dt dt
 1.1 Matrices
d 1
Example 1: Given matrix A(t), find A (t )
dt
 e t 0
A(t )   
 t 1 
Solution: From A(t)
1 1  1 0   e t
0
A (t )  t  t    t 
e  t e   te 1 
d d e t 0   e t
0
A(t )    
dt dt  t 1   1 0 
 1.1 Matrices
Thus

d 1 d
 A (t )   A (t )  A(t )  A1 (t )
1
dt dt
 et 0    e  t 0   e t 0 
    t 
t
 te 1   1 0   te 1 
 et 0   1 0   et 0
    t   
 tet 1   e 0   (t  1)et 0 
 1.1 Matrices
(Definition) Matrix integration
Matrix integration is defined by the integral of each el-
ement.
 a (t )dt a1n (t )dt 

 11 a12 (t )dt   
 
 A(t )dt  
 a21 (t )dt a 22 (t ) dt   a2 n (t )dt 

     
 

 an1 (t )dt an 2 (t ) dt   ann (t )dt 

 1.2 Vector space and basis
(Definition) Vector space, X
The vector space X defined on the real number set, , is
a set of vectors, and the following conditions must be
satisfied.
1. For x1, x2X, x1+x2= x2+x1 X
2. For x1, x2, x3X, (x1+x2)+x3= x1+(x2+x3)X
3. There must be zero vector in X such that x+0= x
4. For xX, there must be such that x+ 0.
5. For scalar  and vector xX, there must be x
X.
6. For scalar , and vector xX, there must be
(x)= (x)X and (+)x= x+xX.
 1.2 Vector space and basis
7. For scalar  and vector x1, x2X, there must be
(x1+x2)= x1+x2X.

Examples of vector space

2 vector space
2  x  230 
X    {  |   x, y  }  {   ,
 y 134.23
 -2   -2   0   2   399.9 
-234  ,  -9  , ,  0  ,  9  , 9999.45 }
         
 1.2 Vector space and basis
3 vector space
 x
3  
X    { y  | -  x, y, z  }
 z 
n vector space
 x1 
n  
X    {   | -  x1 , x2 , , xn  }
 xn 
 1.2 Vector space and basis
(Example 2) Check whether X is a vector space or
not.
 x1 
 
X  { x2  |   x1 ,x2  }
 1 

(Solution)
 3 1.2  0  9 
e.g. X  {  2  ,  4  , , 0  , 12  }
       
 1   1  1   1 
 1.2 Vector space and basis
Taking two vectors in X and combining them yields

 x11   x12 
   
x1   x21   X , x2   x22   X
 1   1 
 x11  x21 
 
x1  x2   x21  x22   X
 2 
Therefore, it is not a vector space.
 1.2 Vector space and basis
(Definition) Linear independence and linear depen-
dence

The set of vectors x1, x2, , xnX are said to be lin-
early independent if there exist real numbers 1= 2=
= n= 0 such that 1x1+2x2++ nxn= 0. Otherwise, it
said to be linearly dependent.
 1.2 Vector space and basis
2 1 
For example, given two vec- x1 =   , x2 =  
tors, 1   2
2  1   2     0 
 x1 + x2 =   +      = 
1   2    2    0 
From the equation,
2+= 0 
+ 2= 0 
Solving these yields = = 0
 x1 and x2 are linearly independent.
 1.2 Vector space and basis
For example, given three vectors,

2 1  1 
x1 =   , x2 =   , x3 =  
1  2 0
and real constants ,  and ,
2  1  1 
 x1 + x2 +γx3 =    +   +γ  
1   2 0
 2   +γ  0 
=   =  
   2  0
 1.2 Vector space and basis
2++ = 0 
 + 2 = 0 
→ 3 = 
There are three unknown parameters but two equations.
We choose one unknown, that is,  = 1
2+= -1 
+2= 0 
Solving these gives = 1/3 0, = -2/3 0.
 x1, x2, and x3 are linearly dependent.
Note that more than 3 vectors in 2 can not be inde-
pendent.
 1.2 Vector space and basis
Example 3: Are two vectors check L.I.?
x1 = 1 2 3 , x2 = 1 1 3
T T

Solution:
1   1       0 
      
 x1 + x2 =  2  +  1    2    = 0  
 4   3   4  3   0 
From the equation
-= 0  , 2+= 0 , 4+3= 0 
To satisfy three formulas, = = 0
 x1 and x2 are linearly independent.
 1.2 Vector space and basis
(Definition) Dimension, dim(X)
The dimension of a vector space X is defined as the
maximum number of linearly independent vectors in X.

For example, the dimension of 2 is dim(2)= 2, the

dimension of n is dim(n)= n.
 1.2 Vector space and basis
(Definition) Basis
Given a set {e1, e2, , en} of n vectors in X
1. they are linear independent and
2. any vector xX is expressed by a linear combination
as
x= 1e1+2e2++nen,
then {e1, e2, , en} is called a basis.

e.g. X= {, e1, e2, , en, x,  }

x can be expressed by a linear combination of e1, e2,
, and en.
 1.2 Vector space and basis
2 1 
For example, given two vectors e1 =   , e2 =  
1  2
 2     0 
 e1 + e2 =   = 
  2  0 
Solving these gives = = 0  e1 and e2 are L.I.
Letting a vector in X as x = a b 
T

a   2   
x =   = e1 + e2 =  
b
    2  
Solving these gives = =
 Since there are nonzero  and , e1 and e2 are a basis.
 1.2 Vector space and basis
Example 4: Are e1 and e2 in X be a basis?
e1 = 1 2 3 , e2 = 1 1 3
T T

Solution:
     0
 e1 + e2 =  2    = 0 
3  3  0 
Since = = 0, e1 and e2 are L.I. but choosing a vector
x as
x = 1 5 0  X
T
 1.2 Vector space and basis
Then
     1 
  
 e1 + e2 =  2    = 5  
3  3   0 

 e1 and e2 cannot be a basis because there are

non-zero  that satisfy three equations.
In n-dimensional vector space, only n linearly in-
dependent vector set can be a basis.
 1.2 Vector space and basis
Examples of the standard basis

2 vector 1  0
e1 =   , e2 =  
space 0 1 

1  0 0
 vector
3

space e1 = 0  , e2 = 1  , e3 = 0 

0   0  1 
 1.2 Vector space and basis

(Definition) Representation
In an n-dimensional vector space X, if {e1, e2, , en} is
selected as a basis and all vectors xX can be
expressed by

x= [e1 e2  en]

Then  is called representation for a basis {e1, e2, ,

en}.
 1.2 Vector space and basis
For example, given a basis {e1, e2} and a vector x,
obtain .
0 1   2
e1 =   , e2 =   , x =  
1  2  4

2  0 1   1   2 
[e1 e2 ]β =   1 2     =  4 
4   2  

2= 2, 1+22= 4, = [0 2]T

 1.3 Vector and matrix norm
(Definition) Vector norm
Vector norm is a function with a nonnegative scalar real
value that must satisfy the following conditions:

1. x  0 for x  0 and x  0 for x  0

2.  x    x ,   
3. x1  x2  x1  x2
 1.3 Vector and matrix norm
Common vector norms
n
x1 x
j 1
j : 1-norm: column wise

n : 2-norm is the most

x
2
x 2
 i
often used norm called
Euclidean norm
i 1

x 
 max xi  max{ x1 , x2 ,..., xn }
1i  n
: Infinity-norm: row wise
 1.3 Vector and matrix norm
x = 1 5 3 7 
T
Given a vector
n
x1 x
j 1
j  max(1  5  3  7 )  16

x
2 2 2 2 2
x 2
 i  1  5  3  7
i 1

 84  9.2
x 
 max xi  max{1 , 5 , 3 , 7 }  7
1i  n
In MATL,AB. the norms just introduced can be obtained by using the
functions norm(x, 1 ), norm(x, 2 ), norm (x, inf).
 1.3 Vector and matrix norm
Quiz2: Given a vector x = 14 2 8 1 21
T
,
find the vector norms:

(1) x 1
(2) x 
 1.3 Vector and matrix norm
Example 5: Given a vector x= [x1 x2]T 2, sketch the
inequality of the three vector norms.
(1) x 1  x1  x2  1
(2) x 2
1
(3) x 
1

Solution: (1)
 1.3 Vector and matrix norm
First quadrant
x 1  x1  x2  1
x2
If x10, x20 1

x 1  x1  x2  1  0 1 x1

x2   x1  1
If x10, x2<0
x2
x 1  x1  x2  1  x1
0 1
x2  x1  1
-1
 1.3 Vector and matrix norm
x 1  x1  x2  1
x2
If x1<0, x20, 1

x 1   x1  x2  1 
-1
x1
0
x2  x1  1
If x1<0, x2<0, x2

x 1   x1  x2  1  x1
-1 0
x2  x1  1
-1
 1.3 Vector and matrix norm
Thus x2 1

x 1  x1  x2  1 x1
-1 0 1

-1

x2

2 2
(2) x 2
 ( x1  x2 ) 1 x1
-1 0 1
 1.3 Vector and matrix norm
(3) x 
 max{ x1 , x2 }  1

If x10,x20 and x1x2, x 

 max{x1 , x2 }  x1  1
If x10,x20 and x1<x2, x 
 max{x1 , x2 }  x2  1
If x10, x2<0 and x1-x2, x  max{x1 ,  x2 }  x1  1


x2 x2 x2
1 1
x1
0 1
x1 x1
0 1 0 1 -1
 1.3 Vector and matrix norm
If x10, x2<0 and x1<-x2, x

 max{x1 ,  x2 }   x2  1
If x1<0,x20 and -x1x2, x  max{ x1 , x2 }   x1  1

If x1<0,x20 and -x1<x2 , x  max{ x1 , x2 }  x2  1


x2 x2
x2 1 1

x1 x1 x1
0 1 -1 0 -1 0
-1
 1.3 Vector and matrix norm
If x1<0,x2<0 and -x1-x2 , x  max{ x1 ,  x2 }

  x1  1
If x1<0,x2<0 and -x1<-x2, x  max{ x1 ,  x2 }

  x2  1
x2 x2

-1 0
x1 0
x1

-1
 1.3 Vector and matrix norm

x2 x2 x2
1 1 1

0
x1 0
x1 0
x1
-1 1 -1 1 -1 1

-1 -1 -1

(1) x 1  1 (2) x 1 (3) x 

1
2
 1.3 Vector and matrix norm
Example 6: Given two vectors x= [x1 x2]T and x0= [x01
x02]T 2, sketch norm's inequality of the difference
between the two vectors. x  x0 2

Solution:
2 2 2
x  x0 2
 ( x1  x01 )  ( x2  x02 )  
x2
 It means the inside of a circle
with the origin (x01,x02) and a 
radius . x1
(x01,x02)
 1.3 Vector and matrix norm
(Definition) Matrix norm
A matrix norm is a nonnegative real value that must sat-
isfy the following conditions:
1. A  0,A  0 and A  0, A  0
2.  A    A ,   
3. A  B  A  B : Triangle inequality
4. AB  A  B
It is indirectly obtained from a vector norm as
Ax
A  sup  sup Ax where sup is the
x 0 x x 1 suprenum, the smallest
upper limit.
 1.3 Vector and matrix norm
(Theorem) Calculation of matrix norms
A matrix norm is calculated by
n
A 1  max(
1 j  n
a
i 1
ij ) ; Largest column absolute sum

A  λmax ( A A) T : Eigenvalue of AT A
2
n
A   max(
1i  n
a
j 1
ij ) ; Largest row absolute sum
1.3 Vector and matrix norm
 5 4 2 

A=  1 2 3  
Given a matrix,
 2 1 0 

n Absolute sum of all ele-

 A 1  max(
1 j  n
a
i 1
ij ) ments in the jth column

 max(5  1  2, 4  2  1, 2  3  0)  8
 5 1 2   5 4 2   30 24 7 
AT A=  4 2 1   1 2 3    24 21 2 
 2 3 0   2 1 0   7 2 13 
 1.3 Vector and matrix norm

AT A  λI = 0  λ  0.351, 12.56, 51.09

 A 2
 51.09  7.15
 5 4 2 
n
A=  1 2 3 
A   max(
1i  n
a
j 1
ij )  2 1 0 

 max(5  4  2,1  2  3, 2  1  0)  11

Absolute sum of all

elements in the ith row
 1.3 Vector and matrix norm

Answer
 1.3 Vector and matrix norm
Alternatively
C= [1 7 3;4 -2 -2; -2 -1 1]
CTC =[21 1 -7; 1 54 24; -7 24 14]
eig(CTC)

= 0.5185
23.1429
65.3386

C 2
 65.3386  8.083  9.434
 1.3 Vector and matrix norm
MATLAB Commands
>>A= [5 -4 2;-1 2 3;-2 1 0];
• norm(x,1) >>n1= norm(A,1)
• norm(x,2) n1 = 8
• norm(x,inf) >>n2= norm(A,2)
n2 = 7.1477
>>n3= norm(A,inf)
 5 4 2 
n3 = 11
A=  1 2 3 
 2 1 0 
 1.3 Vector and matrix norm
Applicatons of norms
Stop condition of numerical calculation
Finding the numerical solution x that satisfies the stop
condition, where = 0.1, at k = 0; x(0)= [0 0]T.
 2 x1 (k )  1 
x (k  1)  x (k )    
2(
 2 x ( k )  1) 
x (k  1)  x (k )    1 106 : stop condition
1) k= 0
 2 x1 (0)  1  0   1   0.1
x (1)  x (0)         0.1     
2(
 2 x (0)  1)   0  2
   0.2 
 1.3 Vector and matrix norm
 0.1  0  6
x (1)  x (0)  ||  
 0 || 0.3  1  10
 0.2   
2) k= 1
 2 x1 (1)  1   0.1  0.8   0.18
x (2)  x (1)        0.1    
2(
 2 x (1)  1)   0.2   1.6   0.36 
 0.18  0.1 6
x (2)  x (1)  ||  
  0.2  || 0.16  1  10
 0.36   
This procedure will be continued until the stop condition
is satisfied.
 2 x1 (k )  1 
x (k  1)  x (k )    
2(
 2 x ( k )  1) 
 1.3 Vector and matrix norm
for k=1:100
xnew= x-alpha*[2*x(1)+1;2*(x(2)-1)];
norm_test= norm(xnew-x,1);
if norm_test < eps  2 x1 (k )  1 
break x (k  1)  x (k )    
2(
 2 x ( k )  1) 
end 6
x= xnew; x ( k  1)  x ( k )    1  10
buf= [buf; k norm_test];
end
plot(buf(:,1),buf(:,2),'b','linewidth',2)
[k xnew']
0; x= [0;0]; alpha= 0.1; eps=1e-6; buf= [];
 1.3 Vector and matrix norm
x (k  1)  x (k )
0.3

0.25

0.2

0.15

0.1
 0.5
x (k ) k 58  
0.05
 1 
0
0 10 20 30 40 50 60

k
 1.3 Vector and matrix norm
Checking the stability of systems
Checking the stability of a dynamic system

 1 0  1
x (t )    x (t ), x (0)   
 0 2  1

The solution is
At
 e t
0   x1 (0) 
x (t )  e x (0)   
2t  x (0) 
 0 e   2 
Taking norm on both sides gives
 1.3 Vector and matrix norm
From the properties of norms,
x (t )  e At x (0)  e At  x (0)
The norms of e At and x(0) are
e t 0 
e At
    max{e t , e 2t }  e t
1  0 e 2t 
1
x (0) 1  1  1  2
Therefore,
x (t )  e At  x (0) 1  2e t
1
As t, ||x(t)|| 0. The system is stable.
 1.3 Vector and matrix norm
Figure shows the response.
1 0
x (0)    1 x ( )   
1 0
0.5

x2 0

-0.5

4
-1 2
1 0.5 0 -0.5 -1 0 t
x1
 1.4 Orthogonal and orthonormal
(Definition) Orthogonal and orthonormal vectors
If a set of vectors {x1, x2,…, xn} in n satisfies the fol-
lowing properties, it is said to be orthogonal or or-
thonormal where c is a constant.
c i j
Orthogonal:  xi , x j  xiT
xj  
0 i j
T 1 i j
 xi , x j  xi x j  
Orthonormal: 0 i j
Which is dot or inner product of vectors
 1.4 Orthogonal and orthonormal
For example  3  1
x1    , x2   
1 3
 x1 , x1  x1T x1  3  3  1 1  10
T
 x1 , x2  x1 x2  3  (1)  1 3  0
 x2 , x1  x2T x1  (1)  3  3  1  0
 x2 , x2  x2T x2  (1)  (1)  3  3  10

They are orthogonal but not orthonormal.

 1.4 Orthogonal and orthonormal
Gram-Schmidt process
The Gram-Schmidt process is a procedure for or-
thonormalizing a set of linearly independent vectors
{x1, x2,…, xn}.

Step 1: Set v1= x1.

Step 2: If v2= x2-av1, then v2 must be orthogonal with
v1 .

<v1, v2>= 0= <v1, x2>-a<v1, v1>

 v1 , x2   v1 , x2 
a  v 2  x2  v1
 v1 , v1   v1 , v1 
 1.4 Orthogonal and orthonormal
Step 3: If v3= x3-a1v1-a2v2, then v1, v2 must be orthogonal
with v3.

<v1, v3>= 0= <v1, x3>-a1<v1, v1>-a2<v1, v2>

<v2, v3>= 0= <v2, x3>-a1<v2, v1>-a2<v2, v2>
 v1 , x3   v2 , x3 
a1  , a2 
 v1 , v1   v2 , v2 
Therefore,
 v1 , x3   v2 , x3 
 v3  x3  v1  v2
 v1 , v1   v2 , v2 
 1.4 Orthogonal and orthonormal
Step 4: Repeat this process until the last vector is ob-
tained.
Step 5: Convert orthogonal vectors into orthonormal
vectors.
v1 v2 vn
vˆ1  , vˆ2  , ..., vˆn 
v1 2 v2 2 vn 2
 1.4 Orthogonal and orthonormal
Given two vectors x1  2 1 , x2  0 2
T T

2
Step 1: Let v1  x1   
1 
Letting v2= x2 - av1
<v1, v2>= 0= <v1, x2> - a<v1, v1>
<v1, v1>= 22+11= 5  v1 , x2  2
a 
<v1, x2>= 20+12= 2  v1 , v1  5
 0  2  2  4  1
 v2  x2  av1         
 2 5 1  5  2 
1.4 Orthogonal and orthonormal
2 4  1
v1    , v2   
1  52

v1 1 2 1  2
 vˆ1      1 
v1 2 2 1 5 
2 1  
2

v2 1 4  1 1  1
 vˆ2      2
v2 2 4 5 2
  5 
(1) 2  22
5
1.4 Orthogonal and orthonormal
Check the orthonormality.
1 2 1  1
vˆ1  1  , vˆ2  2
5  5 
1
 vˆ1 , vˆ1  ( ) 2 (2  2  1 1)  1
5
1
 vˆ2 , vˆ2  ( ) 2 ((1)  (1)  2  2)  1
5
 vˆ1 , vˆ2   vˆ2 , vˆ1 
1
( ) 2 [2  ( 1)  1  2]  0
5
 1.4 Orthogonal and orthonormal
Example 7: Obtain the orthonormal vectors from the
following three vectors.
x1  1 1 0 , x2  0 1 1 , x3  0 0 1
T T T

Solution: 1 
 v1  x1  1  Letting v2= x2 - av1
0 
<v1, v2>= 0= <v1, x2> - a<v1, v1>
<v1, v1>= 11+11= 2
0 1   1
<v1, x2>= 11= 1  v  x  av   1   1 1   1  1 
 v1 , x2  1 2 2 1   2  2 
a 
 v1 , v1  2  1 0   2 
1.4 Orthogonal and orthonormal
Letting v3= x3-a1v1-a2v2

<v1, v3>= 0= <v1, x3>-a1<v1, v1>

<v2, v3>= 0= <v2, x3>-a2<v2, v2>
 v1 , x3  0  v2 , x3  2
a1    0, a2  
 v1 , v1  2  v2 , v2  3

0  1 1

  1  1 
 v3  x3  a1v1  a2 v2   0    1    1
3 3
 1  2   1

x1  1 1 0 , x2  0 1 1 , x3  0 0 1
T T T
 1.4 Orthogonal and orthonormal
Therefore,
1  1 
v1 1   1  
 vˆ1    1   1
v1 2 12  12  0  2
   0 
 1  1
v2 2   1  
 vˆ2    1  1
v2 2 2 (1) 2  12  (2) 2  2  6
   2 
1 1
v3 3   1  
 vˆ3    1   1
v3 2 3 12  (1) 2  (1) 2  1 3
   1
 1.5 Bilinear and quadratic form
(Definition) Bilinear and quadratic functions
Given vectors x, yn and an nn matrix A, bilinear
and quadratic functions are defined as and scalar
Bilinear
n n  a11  a1n   y1 
f ( x )  xT Ay   a x y ij i j   x1  xn         
i 1 j 1  an1  ann   yn 

n n  a11  a1n   x1 
f ( x )  yT Ax   a ij yi x j   y1  yn         
i 1 j 1  an1  ann   xn 
 1.5 Bilinear and quadratic form
Quadratic
n n  a11  a1n   x1 
f ( x )  xT Ax   a x x ij i j   x1  xn         
i 1 j 1  an1  ann   xn 

 x1   y1   2 3
For example x    , y    , A   
 x2   y2   3 1
Bilinear function is
 2 3  x1 
f ( x )  y Ax   y1 y2  
T
 x 
 3 1  2 
 2 x1 y1  3 x2 y1  3 x1 y2  x2 y2
 1.5 Bilinear and quadratic form
 x1   2 3
For example x    , A   
 x2   3 1
The quadratic function for a symmetric matrix is
 2 3  x1 
f ( x )  x Ax   x1 x2  
T
 x   2 x1
2
 6 x x
1 2  x2
2

 3 1  2 
 x1   0 3
For example x    , A   
x
 2  3 0 
The quadratic function for a skew symmetric matrix is
always zero.  0 3   x1 
f ( x )  x Ax   x1 x2  
T
 x   0
 3 0   2 
 1.5 Bilinear and quadratic form
(Theorem) If the gradient operator is defined as fol-
lows, the derivative of bilinear and quadratic func-
tions are
    f ( x) 
 x   x 
 1  1 
 x     ,  x f ( x)    
   
    f ( x ) 
 xn   xn 

(1) xf(x)= x(xTAy)= Ay

(2) xf(x)= x(yTAx)= ATy

(3) xf(x)= x(xTAx)= Ax+ATx= 2Ax (If A= AT)

 1.5 Bilinear and quadratic form
(Proof) (1) For example

 x1   y1   a11 a12 
x   , y   , A   
 x2   y2   a21 a22 
 a11 a12   y1 
x Ay   x1 x2  
T
  
 a21 a22   y2 
 a11 y1  a12 y2 
  x1 x2   
 a21 y1  a22 y2 
 a11 x1 y1  a12 x1 y2  a21 x2 y1  a22 x2 y2
 1.5 Bilinear and quadratic form
xT Ay  a11 x1 y1  a12 x1 y2  a21 x2 y1  a22 x2 y2

  T 
 x ( x Ay )   a y  a y 
 T
( x Ay )   1    11 1 12 2 
x   T   a21 y1  a22 y2 
 ( x Ay ) 
 x2 
 a11 a12   y1 
     Ay
 a21 a22   y2 
Thus
x(xTAy)= Ay
 1.5 Bilinear and quadratic form
 a11 a12   x1 
(3) x Ax   x1
T
x2     
 a21 a22   x2 
 a11 x12  a12 x1 x2  a21 x1 x2  a22 x22
 T  2a11 x1  a12 x2  a21 x2 
( x Ax )   
x a x 
 12 1 21 1a x  2 a x
22 2 

 a11 a12   x1   a11 a21   x1 

       
 a21 a22   x2   a12 a22   x2 
 Ax  AT x
 1.5 Bilinear and quadratic form
Definiteness of quadratic form
Since a square matrix A can be written by the sum of
a symmetric matrix B and a skew-symmetric matrix C
as,
A= (A +AT) + (A-AT)= B+C
The quadratic form is
xTAx= xT(B+C)x= xTBx+xTCx=0 xTBx
Only the symmetrical component is involved in the
quadratic form.
 1.5 Bilinear and quadratic form
Applications of quadratic form
1) Calculate Variance of Measures
V= eTWe, where e= [x1-m, ..., xn-m]T and W is a
weighting matrix
2) Determining system stability
Lyapunov function V(x)= xTPx
3) Optimal state controller design
1  T
J 
2 0
( x Qx  uT Ru)dt
4) Checking local optima
5) Multivariable function optimization
 1.5 Bilinear and quadratic form
(Definition) Definiteness
Let A be a real symmetric matrix and x be a vector. For any x, the
sign of the quadratic form depends on A.
(1) If xTAx >0, this is positive definite (PD)
(2) If xTAx 0, this is positive semi-definite (PSD)
(3) If xTAx <0, this is negative definite (ND)
(4) If xTAx 0, this is negative semi-definite (NSD)
(5) If xTAx is positive or negative, this is indefinite (ID)- saddle
 1.5 Bilinear and quadratic form
(Theorem) Test for matrix definiteness
Method 1: Test from the sign of i= |Ai| (1  i  n)
a11 a12
1  a11 , 2  ,..., n  | A |
a21 a22

1) If 1>0, 2>0,..., n>0 PD

2) If 10, 20,..., n0 PSD
3) If 1<0, 2>0, 3<0, 4>0... ND
4) If 10, 20, 30, 40... NSD
5) Otherwise  ID
 1.5 Bilinear and quadratic form
Method 2: Test from the sign of eigenvalues i(1in)
of A
1) If i > 0  PD
2) If i  0  PSD
3) If i < 0  ND 1 2
Given A   
4) If i  0  NSD
2 3
5) Otherwise ID
1 2
1  1  0, 2   1  0
2 3
Eigenvalues: = -0.2361, 4.2361 (saddle)
A is an ID matrix.
1.5 Bilinear and quadratic form
 5 0 1 
B   0 2 2 
 1 2 3
5 0
1  5  0, 2   10  0
0 2
5 0 1
3  B  0 2 2  8  0
1 2 3
Eigenvalues : = -5.6458, -4, -0.3542
B is a ND matrix.
 1.6 Algebraic equation
Basic operation of matrix

• Multiply a row with a nonzero constant

(Ei  Ei)
• Swap two rows
(Ei  Ej)
• Multiply one row by a nonzero constant and add it to
the other row
(Ei + Ej  Ej)
 1.6 Algebraic equation
(Definition) Echelon form
If an mn matrix A satisfies the following properties, it
is called row echelon form
1) If all zero rows exist, they are placed at the bottom
of the matrix.
2) The first nonzero component in each row is one(1). 1
is called the pivot of the row.
3) If there is a pivot in both the ith and (i+1)th rows,
the pivot in the (i+1)th row is to the right of the
pivot in the ith row.
 1.6 Algebraic equation
Echelon form
0 1 9
1 2 3  1 1 5 8  0
0 1 4  , 0 0 0 1  ,  0 1 
    0 0 0
0 0 1  0 0 0 0   
0 0 0
Not echelon form
1 8 0
0 1 3  1 4 7 1  0 
0 1 4  , 0 0 0 0  ,  1 4
    0 0 1
0 0 1  0 1 0 0   
0 0 1
 1.6 Algebraic equation
Echelon form transformation
0 2 0 2 4 2
E1E2
  0 2 0
A  2 4 2  
 4 1 4   4 1 4 

0.5E1E1 1 2 1  -4E1+E3E3 1 2 1 
0 2 0 0 2 0 
   
 4 1 4  0 9 0 
 1.6 Algebraic equation

1 2 1  1 2 1 
0.5E2E2 0 1 0 9E2+E3E3
0 1 0
   
0 9 0  0 0 0 
 1.6 Algebraic equation
(Definition) Rank (A) of matrix A
The rank of an mn matrix is the number of linearly
independent rows or columns. That is, when A is
converted to a echelon form, A is the number of
nonzero rows and A  min(m,n).

-4E1+E2E2 -1/6E2E2

1 2 3 1 2 3   1 2 3
A  0 0 6  0 0 1
4 8 6    
A= 2  min(2,3)= 2
 1.6 Algebraic equation
-3E1+E2E2
E1+E3E3
1 2 1 2 
  0 0  A= 1
A3 6  
 1 2  0 0 
0 2 0 1 2 1 

A  2 4 2  0 1 0  A= 2
 
 4 1 4  0 0 0 
 1.6 Algebraic equation
(Definition) Null space, N(A)
The null space of a matrix A is defined as a set of all
vectors x that satisfy Ax= 0 and can be expressed by.

N(A)= {x| Ax= 0}

For example, 1 2
find the null space of A. A   3 6 
 1 2 
1 2 0
   x1   
Ax   3 6     0 
 x2 
 1 2  0 
 1.6 Algebraic equation
1 2 0
   x1   
Ax   3 6     0 
 x2 
 1 2  0 

Arranging this equation gives only one independent

equation
x1+2x2= 0
Since there are two unknowns, setting x2= c gives x1= -
2c. Therefore, the null space is
 2c 
N ( A)  {  |   c  }
 c 
 1.6 Algebraic equation
(Definition) Nullity, (A)
The nullity of an mn matrix A is defined as the num-
ber of independent vectors among all solution vectors
satisfying Ax= 0, which can be expressed by
(A)= n - A
1 2
For example, A   3 6 
the null space of A is  1 2 

 2c 
N ( A)  {  |   c  }
 c 
 1.6 Algebraic equation
Solution:
1 2 1 2 1 2 

A3 6     0 0 
A3 6   
 1 2   1 2   0 0 
Since A= 1,
(A)= n - A = 2-1= 1
• Rank A: Number of independent equations when
solving Ax= 0
• Nullity (A): Number of unknowns that can be deter-
mined arbitrarily when solving Ax= 0
 1.6 Algebraic equation
Alternatively;
1 2 1 2 0 
A   3 6     x1   
Ax   3 6     0 
 1 2   x2 
 1 2  0 
Arranging gives
x1+2x2= 0 , 3x1+6x2= 0 , -x1-2x2= 0 
• Number of unknowns(column numbers): n= 2
• Number of independent equations: A= 1
• If choosing one unknown, the rest can be known:
(A)= n - A= 1
 1.6 Algebraic equation
Linear algebraic equation
a11 x1  a12 x2  ...  a1n xn  b1
a21 x1  a22 x2  ...  a2 n xn  b2
 
am1 x1  am 2 x2  ...  amn xn  bm
 a11 a12  a1n   x1   b1  Ax= b
a     
 a2 n   x2   b2  Amn,
 21 a22 
           xn1 ,
    
 am1 am 2  amn   xn  bm  bm1
1.6 Algebraic equation
Note: Given an mn matrix A,
• Rank A
Numbers of independent equations,
A  min(m,n)

• Nullity) (A)
Numbers of unknown variables that can be de-
termined arbitrarily
(A)= n - A
 1.6 Algebraic equation
Condition of solution existence
Augmented matrix W= [A|b]
where Amn, xn1 , bm1
1. If A= W= n, a unique solution exits
(The numbers of unknowns and independent equations are
equal)
2. If A= W < n, infinite solutions exist
(The numbers of unknowns are greater than those of inde-
pendent equations)
3. If A W, no solution exists
(The numbers of unknowns are less than those of inde-
pendent equations)
 1.6 Algebraic equation
For example
1 2  x  2
(1)      
2 2  y  0
1 2  1 2 
A  ~  A  2
2 2  0 2 
 1 2 2  1 2 2 
W  [ A | b]    ~   W  2
 2 2 0  0 4 8

Since A= W= n= 2, a unique solution exists

 1.6 Algebraic equation
1 2  x   2
2 2  y   0
    

From the given equation,

x+2y= 2 
2x+2y= 0 

Since - gives

x= -2, y= 2,
there is a unique solution that satisfies the equation.
 1.6 Algebraic equation
For example
 x
1 2 3    1 
(2)    y   
 4 4 6 
 z   0

1 2 3  1 2 3 
A ~
4 4 6  0 4 18


1 2 3 1  1 2 3 1 
W  ~
4 4 6 0  0 4 18  4 


A= W= 2 (row rank) < n= 3, infinite solutions exist.

 1.6 Algebraic equation
 x
1 2 3    1 
 4 4 6   y   0 
  z  
 
From the given equation,
x-2y+3z= 1 
4x-4y-6z= 0 
Letting z= 0, reducing z variable and doing 4-
gives y= -2, x= -1
If choosing another z, we have other x and y, so there
are infinite solutions.
 1.6 Algebraic equation
For example
1 2  1 
   x1   
(3)  2 3     1 
 x2 
 0 1 0 
1 2  1 2  1 2  A= 2, W=3
Since A W, no
A   2    
3  ~ 0 1 ~ 0 1
solution exists.
 0 1 0 1 0 0 
1 2 1  1 2 1  1 2 1 
W  2     
3 1  ~ 0 1 1 ~ 0 1 1 
 0 1 0  0 1 0  0 0 1 
 1.6 Algebraic equation
1 2  1 
 2 3   x1   1 
  x   
 0 1  2  0 

From the given equation,

x1+2x2= 1 
2x1+3x2= 1 
-x2= 0 
Substituting  to  and  yields x1= 1 and x1= 1/2.
There is no solution that satisfies the three equations.