Department of Electronics and Communication

Engineering

Linear Algebra and Random

Processes
Subject Code - ECN-511

Course Introduction
 Course Content
1. Vector spaces, subspaces, bases and dimensions, linear dependence and
independence, vector products, orthogonal bases and orthogonal
projections,
2. Linear operators and Matrices: Eigen values and Eigen vectors,
characteristic polynomial, diagonalization, Hermitian and unitary
matrices, singular value decomposition
3. Discrete and continuous random variables: distribution and density
functions, conditional distributions and expectations, functions of
random variables, moments, sequence of random variables
4. Random process: Probabilistic structure; Mean, autocorrelation and auto-
covariance functions; Strict-sense and wide-sense stationary processes;
Power spectral density; LTI systems with WSS process as the input;
Examples of random processes - white noise, Gaussian, Poisson and
Markov Processes
 Marks Distribution
• Assignments + Quiz: 10 % + 15%
• Mid-term Exam: 35%
• End-Term Exam: 40%
S. Name of Books / Authors/ Publishers Year
No.
1. S. Axler, "Linear Algebra Done Right", 3rdEdn., Springer International 2015 2015
Publishing.
2. G.Strang, "Linear Algebra and Its Applications", 4thEdn., Cengage Learning. 2007

3. K.M. Hoffinan and R. Kunze, "Linear Algebra", 2nd Edn.,Prentice Hall 2015 2015
India.
4. A. Papoulis and S. Pillai, "Probability, Random Variables and Stochastic 2017
Processes", 4thEdn., McGraw Hill.
5. H. Stark and J.W. Woods, "Probability and Random Processes with 2001
Applications to Signal Processing", 3rdEdn., Pearson India.
 Eigen Values and Eigen Vectors

Till now, we were solving Linear Algebra Problem Statement:

n number of vectors m dimension in each vector

Now, we will focus on solving Linear Algebra Problem Statement:

Ax =  x
➢ We will still try to make it diagonal.
➢ We cannot use an elimination process that changes
Eigenvalues for the matrix

4
 Usefulness of Eigen Value

Application in ordinary differential equation

➢ An initial value problem, where we need to find v(t) and w(t)

for later times.

➢ In Matrix form:

5
➢ Note following
➢ First order equation
➢ Linear in unknowns
➢ Constant coefficients
➢ A is independent of time

➢ If above equation had one unknown instead of two, the

solution would have been simpler

➢ Even for two unknowns, intuitively we will look for solutions

with exponential dependence.
6
➢ Individually

➢ In Matrix form

➢ Substituting values of v(t) and w(t) in the given equation, we get

following two equations

7
➢ We get an eigenvalue problem

➢ If we observe, it is matrix form, known as the Eigenvalue

equation Ax =  x

➢ The eigenvalue equation involves two variables  and x.

➢  is called the eigenvalue of the matrix A
➢ x is associated eigenvector.

Our aim is to find eigenvalue and eigenvectors and to use

them

8
 The solution of Ax =  x

➢ We could re-write above equation as

( A −  I )x = 0

➢ We want a nonzero eigenvector.

➢ The aim is to create u(t) from exponentials.
➢ We are interested in only those eigenvalues for which there is a
nonzero eigenvector.
➢ The null space of must contain vectors other than zero.
In other words should be singular.

9
➢ The number  is an eigen value of A if and only if is
singular

➢ The above relation provides a characteristics equation.

➢ In our example, lets apply characteristics equation

Characteristics Polynomial

➢ The Eigenvalues will be

10
➢ The Eigenvectors will be

11
 Summary of steps involved

➢ The solution to the differential equation will be

12
➢ We can choose free parameters

➢ Find the coefficient values

➢ Complete final solution for u(t)

13
 Example

➢ Find eigenvalue and eigenvector of following matrix

➢ Find eigenvalue and eigenvector of projection matrix

14
 Eigenvalues and A

➢ The sum of the n eigenvalues equals the sum of the n diagonal

entries

➢ The product of the n eigenvalues equals the determinant of A.

12 ...n = det( A)
➢ The eigen values are on the main diagonal when A is triangular.
➢ Example:
Trace: a+d Determinant: ad-bc

15
16
 Diagonalization of a Matrix

➢ The eigenvectors are used to diagonalize a matrix.

➢ Suppose the n by n matrix A has n linearly independent
eigenvectors.
➢ If these eigenvectors are the columns of a matrix S. then

➢ The eigenvalues of A are on the diagonal of :

➢ S is called an eigenvector matrix.
➢ is called an eigenvalue matrix.

17
 Proof

Ax =  x

Split

18
➢ It is essential to keep this split in the correct order
AS = S
S −1 AS =  A = S −1S

S is invertible when eigenvectors are independent.

Remarks on Independence
1. If the matrix A has no repeated eigenvalues-the numbers
then its n eigenvectors are automatically independent.
Therefore any matrix with distinct eigenvalues can be
diagonalized.

19
2. The diagonalizing matrix S is not unique. An eigenvector x
can be multiplied by a constant and remains an
eigenvector. We can multiply the columns of S by any
nonzero constants, and produce a new diagonalizing S.
Trivial Example:

3. Other matrices S will not produce a diagonal A. Suppose

the first column of S is y. Then, the first column of SA is
λ1y. If this is to agree with the first column of AS, which by
matrix multiplication is Ay, then y must be an eigenvector:
Ay = λ1y. The order of the eigenvectors in S and the
eigenvalues in A is automatically the same.

20
4. Not all matrices possess n linearly independent
eigenvectors, so not all matrices are diagonalizable.

The standard example of a "defective matrix" is

* Algebraic multiplicity is 2. Geometric multiplicity is 1. Only

one independent eigenvector. So we can’t construct S.
21
 Examples

➢ There is no connection between diagonalizability (n independent

eigenvectors) and invariability (no zero eigenvalues).
➢ The only indication given by the eigenvalues is that diagonalization
can fail only if there are repeated eigenvalues. Even then, it does not
always fail.
➢ A = I has repeated eigenvalues 1, 1, ... , 1, but it is already diagonal!
There is no shortage of eigenvectors in that case.

22
 Positive Definiteness of a matrix

Lets review maxima and minima of a function

1. x=0, y=0
2. First derivative for having max or min
3. Second derivative to identify max or min
4. Higher terms will not define max or min but can stop from
solution to be global minima.

23
➢ Every quadratic form has a stationary point at (0,0).

➢ The function is shaped like a bowl.

➢ If the stationary point is not at (0,0), only difference would be to

use second derivative at new stationary point.

24
➢ Third derivative comes into picture when second derivative fails
to give a definite decision.
➢ This happens when quadratic part is singular.
➢ When f(x,y) is strictly positive at all points except stationary
point, (bowl goes up), it is called positive definite.

➢ In single variable, we apply second derivative, but for two

variables, we need to check Fxx, Fxy and Fyy for minimum
condition.

➢ Can we say, a>0, c>0 ensures f to be positive definite?

NO!

25
26
 Singular Case ac=b2

If a>0, Positive semidefinite

If a<0, Negative semidefinite
“Semi” means that function can be zero
Saddle Case ac<b2
The quadratic forms are called Indefinite.

27
Two variable case can be extended to multivariable case

28
➢ There are only second order terms.
➢ Its first derivative is zero.
➢ The tangent is flat. It is stationary point.
➢ We need to decide if x=0 is a minimum or a maximum or a
saddle point of the function f=xTAx.
Examples

Saddle Point

Saddle Point

29
 Test for Positive Definiteness

➢ Both eigenvalues are positive.

How?
➢ Multiplication for two eigenvalues is positive and a is positive.

Necessary and sufficient condition for symmetric Matrix A to

be positive definite:

30
 Singular Value Decomposition

➢ The SVD is closely associated with the eigenvalue-eigenvector

factorization QAQT of a positive definite matrix.
➢ The eigenvector matrix Q is orthogonal (QTQ = I) because
eigenvectors of a symmetric matrix can be chosen to be
orthonormal.
➢ The diagonal (but rectangular) matrix E has eigenvalues from
ATA, not from A!
➢ Those positive entries (also called sigma) are the singular
values of A. They fill the first r places on the main diagonal of
Σ-when A has rank r. The rest of Σ is zero.

31
➢ The columns of U (n by m) are eigenvectors of AAT, and the
columns of V (n by n) are eigenvectors of ATA. The r singular
values on the diagonal of Σ (m by n) are the square roots of the
nonzero eigenvalues of both AAT and ATA.

32
33
Example:

34
 Applications of SVD

➢ U and V do not change the length of a vector. Multiplication

by U does not change the scaling factor. How?

➢ U and V are orthogonal matrices, and SVD provides a

numerically stable solution. How?
 max
condition number =
 min
➢ A can be represented as

➢ Any matrix is the sum of r matrices of rank 1.

➢ The number of effective entries in columns can be identified.
Example: Principle component analysis.

35
 Applications of SVD

➢ The effective rank :

➢ Generally, we identify rank by independent rows and
independent columns. (Tiresome).
➢ We can count pivots, but how do we decide the effective
size of a pivot? Example:

➢ SVD provides a more stable measure of rank.

➢ The first step is to use AAT and ATA.
➢ They are symmetric and provide the same rank as A.
➢ We decide on tolerance, pick up values more than
tolerance values, and count singular values above them.

36
➢ Polar Decomposition
➢ Every non-zero complex number can be
identified as a 1 by 1 matrix.
➢ r corresponds to a positive definite matrix.
➢ eiθ corresponds to an orthogonal matrix.
➢ eiθ forms a unitary matrix, How?

➢ Proof: insert in the middle of SVD

37
➢ Q= UVT is an orthogonal matrix. How?

➢.
➢ In complex case, S becomes Hermitian instead of symmetric
and Q becomes Unitary instead of orthogonal.
➢ In the invertible case, and S, both are definite.

➢ Least Squares Solutions

38
39
 Recap (Least Squares Problem)
• Projecting b onto a subspace

• Instead of one column and one unknown, we now

have n unknowns.
Probably, there will not exist a choice of x that perfectly
fits the data b the vector b probably will not be a
combination of the columns of A; it will be outside the
column space.
The problem is to choose x so as to
minimize the error
Now, we have a look at basic subspaces:
1. All vectors perpendicular to the column space lie in
the left nullspace.
The error vector (e = b – Ax) must be in the
nullspace of AT : AT Ax = AT b

2. The error vector must be perpendicular to each

column al, ... , an of A:
 3.

Refer Slide-64
Practice Example