Module1 A Whole

Module -1 a,b (prerequisite)
• Basics of matrix
• Computational Knowledge
• Solving Linear Equations (Class 10 mode)
Module -1 a (Tools you will learn /or revisit)
• Algebra of matrices
 Singularity, condition of singularity, Symmetric, diagonal, Identity, upper triangular,
zero, Hermitian, Vector Norm, Matrix norm, Matrix condition number, Error bounds,
 Operation: Addition, Multiplication, Trace,
• Naïve Gaussian elimination: Forward and backward elimination, pseudo code,
computational complexity
• Tridiagonal Systems, Diagonal Dominance, Tridiagonal Algorithm-importance,
Examples, Gauss-Jordan Algorithm, LU decomposition in any linear equations,
Cholesky decompositions , pseudo code
Linear Equations
is called as linear Equations
Linear Equations
Is set of linear Equations

Linear Equations
In matrix form it should be
𝑥1 𝛽1
¿
𝐴1 1 ⋯ 𝐴1 𝑁
⋮ ⋱ ⋮ ⋮ ⋮
𝐴𝑁 1 ⋯ 𝐴 𝑁𝑁
𝑥𝑁 𝛽𝑁
Linear Equations
In matrix form it should be
𝑥1 𝛽1
¿
𝐴1 1 ⋯ 𝐴1 𝑁
⋮ ⋱ ⋮ ⋮ ⋮ Question: How can we solve the N
𝐴𝑁 1 ⋯ 𝐴 𝑁𝑁 dimensions equations? What will be the
𝑥𝑁 𝛽𝑁 caveats?
Linear Equations

Linear Equations

Linear Equations

Linear Equations
Solution may or may not exist, and may or may

not be unique
We will be back later

Symmetric and Transpose
Rudiment of Matrix
If , then can be called as symmetric
Triangular Matrix
A triangular matrix is a type of square matrix that has all values in the upper-right or lower-left of the matrix with the
remaining elements filled with zero values.
Symmetric and Transpose
Rudiment of Matrix
Triangular Matrix
Diagonal Matrix
A diagonal matrix is one where values outside of the main diagonal have a zero value, where the main diagonal is taken
from the top left of the matrix to the bottom right.
Identity Matrix
An identity matrix is a square matrix that does not change a vector when multiplied.
Symmetric* and Transpose
Rudiment of Matrix
Triangular Matrix
Diagonal Matrix
A diagonal matrix is one where values outside of the main diagonal have a zero value, where the main diagonal is taken
from the top left of the matrix to the bottom right.
Identity Matrix
An identity matrix is a square matrix that does not change a vector when multiplied.
Skew Symmetric* 𝑨 𝑻 =− 𝑨
Hermitian
A matrix is said to be hermitian if and only it is equal to the transpose of its conjugate matrix .
Rudiment of Matrix
Idempotent Nilpotent Orthogonal Involutory Matrix
𝑨𝑨𝑻 = 𝑰 𝑨𝟐 = 𝑰
A square matrix A of
order n is nilpotent if
and only if Ak = O for
some k ≤ n.
Hey Neo , I am
Idempotent.
I know my “power”
) very well
I am making
you Nilpotent.
Behold !!!!
Rudiment of Matrix
Idempotent Nilpotent Orthogonal Involutory Matrix Sparse Matrix
𝑨𝑨𝑻 = 𝑰 𝑨𝟐 = 𝑰 Most of the

A square matrix A of elements
order n is nilpotent if are zero
some k ≤ n.
Band Matrix
A sparse
matrix whose non-
zero entries are
confined to a
diagonal band,
Rudiment of Matrix

some k ≤ n.
Band Matrix
A sparse
matrix whose non-
zero entries are
confined to a
diagonal band,
Rudiment of Matrix

some k ≤ n.
Band Matrix
A sparse Matlab:
matrix whose non- Issymmetric,
zero entries are inv,
confined to a A’,
diagonal band, A*A; A*.A;
Addition
Rudiment of Matrix algebra
Addition
Multiplicative
Addition
Multiplicative
Properties
Addition
Multiplicative
What about complexity?
Properties
Properties of Matrix Transpose
Properties of Matrix Transpose
Matrix Inverse
Properties of Matrix Inverse

Viewing a Matrix – 4 Ways
v
Vector times Vector – 2 Ways
Mv
Matrix times Vector – 2 Ways
vM
Vector times Matrix – 2 Ways

MM
P
Matrix times Matrix – 4 Ways
Practical Patterns
The Five Matrix Factorizations

By Kenji Hiranabe with the kindest help of Prof. Gilbert Strang 24
Viewing a Matrix – 4 Ways
¿ ¿ ¿
1 matrix 6 numbers 2 column vectors 3 row vectors

with 3 numbers with 2 numbers
[ ] [ ] [ ]
𝑎11 𝑎12 ¿ ¿ − 𝒂∗
1 −
𝐴= 𝑎21 𝑎 22 = 𝒂𝟏 𝒂𝟐 = − 𝒂∗ −
2
𝑎31 𝑎 32 ¿ ¿ ∗
− 𝒂3 −
Here, column vectors are in bold as , row vectors are with * as

And transposed vectors/matrices are with T on the shoulders as

v Vector times Vector – 2 Ways
v1 ¿ ¿ Dot product (number) v2 ¿ ¿ Rank 1 Matrix
Dot product () is expressed as in matrix is a matrix . If neither are

language and yields a number. the result is a rank 1 matrix.
[ ] [ ][ ]
𝑥1 1 𝑥1
[1 2 3 ] 𝑥 2 = 2 ∙ 𝑥2 = 𝑥1 +2 𝑥2 +3 𝑥 3
𝑥3 3 𝑥3

Mv Matrix times Vector – 2 Ways
Mv1 ¿ Mv2 ¿ +¿
The row vectors of A are multiplied by a vector The product is a linear combination of the column
and become the three dot-product elements of . vectors of A.
At first, you learn (Mv1). But when you get used to viewing it as (Mv2),
you can understand as a linear combination of the columns of A. Those
products fill the column space of A denoted as .
The solution space of is the nullspace of A denoted as

vM Vector times Matrix – 2 Ways
vM1 ¿
A row vector is multiplied by the

two column vectors of A
and become the two dot-product
elements of .
vM2 ¿ +¿ +¿
The product is a linear

combination of the row
vectors of A.

MM
Matrix times Matrix – 4 Ways
MM
1
¿ MM
2
¿ ¿
Every element becomes a dot product of row vector and are linear combinations of columns of A.
and column vector.
MM
3
MM
4 ¿ +¿
¿ ¿
Multiplication is broken down to a sum of rank 1 matrices.
The produced rows are linear combinations of rows.

Four Ways to Multiply AB
¿ ¿
(Rows of A ) ・ (Columns of B ) A times (Columns of B )
¿
+¿
¿
(Rows of A ) times B Sum of (Columns of A ) (Rows of B )
SENT TO Prof. Strang (1/2) 6/18, 2022

Linear Equations

not be unique
Linear Equations

not be unique
Can also talk about non-square systems where A is mn, b is m1, and
x is n1
• Overdetermined if m>n:
“more equations than unknowns”
• Underdetermined if n>m:
“more unknowns than equations”
Can look for best solution using least squares
Singularity and nonsingularity
A is singular if some row is linear

combination of other rows.
Singular systems can be
underdetermined:
or inconsistent:
Example
Geometric Interpretation
Usually not a good idea to compute x=A-1b

Inefficient
Prone to roundoff error
Vector Norm
Magnitude, modulus, or absolute value for scalars
generalizes to norm for vectors
P-norm
For integer and vector

Vector Norm
Magnitude, modulus, or absolute value for scalars
generalizes to norm for vectors
P-norm
For integer and vector
Example
Properties Vector Norm
Example
Matrix Norm
Matrix norm corresponding to given vector norm defined by
Norm of matrix measures maximum stretching matrix does to

any vector in given vector norm
Matrix Norm
Matrix norm corresponding to given vector norm defined by
Norm of matrix measures maximum stretching matrix does to

any vector in given vector norm
Example, Script
Properties of Matrix Norm
Condition Number
Example, Script
Properties of Condition Number
Error Bounds
Condition number yields error bound for a computed solution to a linear system.
Error Bounds
Condition number yields error bound for a computed solution to a linear system.
Prove it
Solving Linear Equations
Caveats
1. Conditioning
2. Relative error ( See the book of Teath for more discussion)
3. ~10^2 slides are devoted to learning the matrix algebra. Learning of condition
number and relative error is explored. BUT!!!
The objective is still not described
We do not know how to solve linear equations  . Therefore,
Caveats
1. Conditioning
2. Relative error ( See the book of Teath for more discussion)
3. ~10^2 slides are devoted to learning the matrix algebra. Learning of condition
number and relative error are also settled. BUT!!!
The objective is still not described 
We do not know how to solve linear equations  . Therefore,
Q:What can be done?
Fundamental operations:
1. Replace one equation using linear combination of other equations
2. Interchange two equations
3. Re-label two variables
Q:What can be done?
Fundamental operations:
1. Replace one equation using linear combination of other equations
2. Interchange two equations
3. Re-label two variables
• Combine to reduce to trivial system
 Simplest variant only uses #1 operations,
but get better stability by adding
 #2 (partial pivoting) or #2 and #3 (full pivoting)
In parallel universe
Gauss-Jordan Elimination
 For each row i:

 Multiply row i by
 For each other row j:
 Add times row i to row j
 At the end, left part of matrix is identity, answer in right part
 Can solve any number of R.H.S. simultaneously
• Solve:
• Only care about numbers – form “tableau” or “augmented matrix”:

• Given:
• Goal: reduce this to trivial system
and read off answer from right column

• Basic operation 1: replace any row by

linear combination with any other row
• Here, replace row1 with 1/2 * row1 + 0 * row2
• Replace row2 with row2 – 4 * row1
• Negate row2
• Replace row1 with row1 – 3/2 * row2
• Read off solution: x1 = 2, x2 = 1

• Replace row1 with row1 – 3/2 * row2
• Read off solution: x1 = 2, x2 = 1

Operation count (Complexity)
 For one R.H.S., how many operations?

 For each of n rows:
 Do n times:
 For each of n+1 columns:
 One add, one multiply
 Total = n3+n2 multiplies, same # of adds

 Asymptotic behavior: when n is large, dominated by n3
Faster
Algorithms
• Our goal is an algorithm that does this in 1/3 n3 operations,
• Before we see that, let’s look at a few special cases that are even
faster
Tridiagonal Systems
• Common special case:
• Only main diagonal + 1 above and 1 below

Solving Tridiagonal Systems
• When solving using Gauss-Jordan:

• Constant # of multiplies/adds in each row
• Each row only affects 2 others
Running Time
• 2n loops, 4 multiply/adds per loop

(assuming correct bookkeeping)
Running Time

• This running time has a fundamentally different
dependence on n: linear instead of cubic
Running Time

• This running time has a fundamentally different
dependence on n: linear instead of cubic
• Can say that tridiagonal algorithm is O(n) while
Gauss-Jordan is O(n3)
Triangular Systems
• Another special case: A is lower-triangular

Triangular Systems
• Solve by forward substitution
Triangular Systems
Triangular Systems
Triangular Systems
•Both of these special cases (Upper or

Lower) can be solved in O(n ) time.
2
Triangular Systems
•Both of these special cases (Upper or

Lower) can be solved in O(n ) time.
2
•Q: How to factorize a matrix to

Triangular (Upper \times Lower) form?
What type of transformation
makes the solution
unchanged?
What type of transformation of the linear system leaves solution unchanged?
Claim We can premultiply (from left) both sides of linear

system by any nonsingular matrix without affecting
solution

solution
𝑀𝐴𝑥=𝑀𝑏

solution
𝑀𝐴𝑥=𝑀𝑏
Example Permutation matrix
• One in each row.

• One in each row.
• Premultiplying both sides of system by permutation matrix, reorders row, but

solutions remain unchanged.
What type of linear system is easy to solve?
• If one equation in system involves only one component of solution (i.e., only one entry
in that row of matrix is nonzero), then that component can be computed by division.
• If another equation in system involves only one additional solution component, then
by substituting one known component into it, we can solve for another component.
• If this pattern continues, with only one new solution component per equation, then all
components of solution can be computed in succession.
• If this pattern continues, with only one new solution component per equation, then all
components of solution can be computed in succession.
System with this

property called
triangular
Forward- and Back-Substitution
Code: Forward and Backward Sunstitution
%MATLAB program: forward-substitution for a lower triangular
linear system
function x=forwardSubstitution_1a(L,b,n)
% Solving a lower triangular system by forward-substitution
% Input matrix L is an n by n lower triangular matrix
% Input vector b is n by 1
% Input scalar n specifies the dimensions of the arrays
% Output vector x is the solution to the linear system
%Lx=b
% K. Ming Leung, 01/25/03
x=zeros(n,1);
for j=1:n
if (L(j,j)==0) error('Matrix is singular!'); end;
x(j)=b(j)/L(j,j);
b(j+1:n)=b(j+1:n)-L(j+1:n,j)*x(j);
end
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%
%%% L=[1 0 0; 4 1 0; 4 0.5 1]; b=[3 6 10]';
%%% x=forwardSubstitution_1a(U,b,n)
Code: Forward and Backward Sunstitution
function x=backSubstitution_1a(U,b,n)
% Solving an upper triangular system by back-substitution
% Input matrix U is an n by n upper triangular matrix
% Input vector b is n by 1
% Input scalar n specifies the dimensions of the arrays
% Output vector x is the solution to the linear system
%Ux=b
x=zeros(n,1);
for j=n:-1:1
if (U(j,j)==0) error('Matrix is singular!'); end;
x(j)=b(j)/U(j,j);
b(1:j-1)=b(1:j-1)-U(1:j-1,j)*x(j);
end
Transforming general linear system into triangular form
• You need to replace selected nonzero entries of matrix by zeros.
• This can be done by using the linear combination of rows.

• Let the vector be

• Find such
• Let the vector be

• Find such M
Transforming general linear system into
triangular form
Find such that
Find such that
=
Find such that
Find such that
=
Also, find out and

Gaussian Elimination using Upper/Lower triangular matrix
%b=[2 8 10]';
%A=[2 4 -2 ; 4 9 -3; -2 -3 7]; n=3;
A=[1 2 2 ; 4 4 2; 4 6 4]; b=[3 6 10]'; n=3;
m1=eye(n); MM=[];
Aold=A;
for k=1:2
m11=eye(n);
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
m1(k+1:n,k)=-Aold(k+1:n,k)/Aold(k,k);
m11(k+1:n,k)=m1(k+1:n,k);
m1A=m11*Aold;
m1b=m11*b;
M={Aold, m1, m11, m1A, m1b};
Aold=m1A;
b=m1b;
MM=[MM;M];
end
>> MM(end,4); %Uppertraingular amtrix
>> MM(end,5); % Transformed b
LU decomposition and solving linear equations
function [L,A]=LU_factor_using_Gaussian_1a(A,n)
% LU factorization of an n by n matrix A
LU decomposition
% using Gauss elimination without pivoting
% LU_factor.m
% A is factored as A = L*U
% Output:
% L is lower triangular with the main diagonal part = 1s.
% U is upper triangular and is stored in the original mtx A
% and must be zeroed out to get U
%A=[1 2 2 ; 4 4 2; 4 6 4]; n=3;
L=eye(n);
for k=1:n
if (A(k,k) == 0)
Error('Pivoting is needed!'); end
L(k+1:n,k)=A(k+1:n,k)/A(k,k);
for j=k+1:n
A(j,:)=A(j,:)-L(j,k)*A(k,:);
end
end
%%% [L U]=lu(A);
clear all
%A=[1 1 1; 3 1 -3; 1 -2 -5];
A=[2 4 -2; 4 9 -3; -2 -3 7];
b=[2 8 10]';
[m n]=size(A);
%%%%%%%%% Medthod 1 from the following
code%
[L1 U1]=LU_factor_using_Gaussian_1a(A,n);
Y2=forwardSubstitution_1a(L1,b,n);
x2=backSubstitution_1a(U1,Y2,n);
function [L,A]=LU_factor_using_Gaussian_1a(A,n)
% LU factorization of an n by n matrix A
% using Gauss elimination without pivoting
clear all % LU_factor.m
%A=[1 1 1; 3 1 -3; 1 -2 -5]; % A is factored as A = L*U
A=[2 4 -2; 4 9 -3; -2 -3 7]; % Output:
b=[2 8 10]'; % L is lower triangular with the main diagonal part = 1s.
[m n]=size(A); % U is upper triangular and is stored in the original mtx A
%%%%%%%%% Medthod 1 from the following % and must be zeroed out to get U
code% % K. Ming Leung, 01/26/03
[L1 U1]=LU_factor_using_Gaussian_1a(A,n); %A=[1 2 2 ; 4 4 2; 4 6 4]; n=3;
Y2=forwardSubstitution_1a(L1,b,n); L=eye(n);
x2=backSubstitution_1a(U1,Y2,n); for k=1:n
if (A(k,k) == 0)
Error('Pivoting is needed!'); end
L(k+1:n,k)=A(k+1:n,k)/A(k,k);
for j=k+1:n
A(j,:)=A(j,:)-L(j,k)*A(k,:);
end
end
%%% [L U]=lu(A);
(Baked from Matlab fucntion)
clear all
%A=[1 1 1; 3 1 -3; 1 -2 -5];
A=[2 4 -2; 4 9 -3; -2 -3 7];
b=[2 8 10]';
[m n]=size(A);
%%%%%% using Matlab function %%%%%%

[L U P]=lu(A);
%L(2,2)=10^-1;
%L(3,3)=10^-1;
%%%Method 1 for finding solution%%%
Y = L\(P*b);
x = U\Y;
%%%%%%%%%%%%%%%%%%%%%%
%%% Method b for finding solution %%%
Y1=forwardSubstitution_1a(L,P*b,n);
%%% [L,~]=LU_factor_using_Gaussian_1a(A,n)
x1=backSubstitution_1a(U,Y1,n);
Symmetric Positive Definite matrix
Symmetric Positive Definite matrix
Symmetric Positive Definite matrix : Cholesky factorization

Module1 A Whole

Uploaded by

Copyright:

Available Formats

You might also like

Module1 A Whole

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Module1 A Whole

Uploaded by

Copyright:

Available Formats

Module -1 a,b (prerequisite)

Is set of linear Equations

Is set of linear Equations

In matrix form it should be

Is set of linear Equations

In matrix form it should be

Is set of linear Equations

Is set of linear Equations

Is set of linear Equations

Solution may or may not exist, and may or may

We will be back later

Idempotent Nilpotent Orthogonal Involutory Matrix

Idempotent Nilpotent Orthogonal Involutory Matrix Sparse Matrix

𝑨𝑨𝑻 = 𝑰 𝑨𝟐 = 𝑰 Most of the

Idempotent Nilpotent Orthogonal Involutory Matrix Sparse Matrix

𝑨𝑨𝑻 = 𝑰 𝑨𝟐 = 𝑰 Most of the

Idempotent Nilpotent Orthogonal Involutory Matrix Sparse Matrix

𝑨𝑨𝑻 = 𝑰 𝑨𝟐 = 𝑰 Most of the

What about complexity?

Properties of Matrix Inverse

Vector times Matrix – 2 Ways

The Five Matrix Factorizations

1 matrix 6 numbers 2 column vectors 3 row vectors

Here, column vectors are in bold as , row vectors are with * as

By Kenji Hiranabe with the kindest help of Prof. Gilbert Strang 25

v1 ¿ ¿ Dot product (number) v2 ¿ ¿ Rank 1 Matrix

Dot product () is expressed as in matrix is a matrix . If neither are

By Kenji Hiranabe with the kindest help of Prof. Gilbert Strang 26

By Kenji Hiranabe with the kindest help of Prof. Gilbert Strang 27

A row vector is multiplied by the

The product is a linear

By Kenji Hiranabe with the kindest help of Prof. Gilbert Strang 28

The produced rows are linear combinations of rows.

By Kenji Hiranabe with the kindest help of Prof. Gilbert Strang 29

(Rows of A ) ・ (Columns of B ) A times (Columns of B )

(Rows of A ) times B Sum of (Columns of A ) (Rows of B )

SENT TO Prof. Strang (1/2) 6/18, 2022

By Kenji Hiranabe with the kindest help of Prof. Gilbert Strang 30

Solution may or may not exist, and may or may

Solution may or may not exist, and may or may

A is singular if some row is linear

Usually not a good idea to compute x=A-1b

For integer and vector

For integer and vector

Matrix norm corresponding to given vector norm defined by

Norm of matrix measures maximum stretching matrix does to

Matrix norm corresponding to given vector norm defined by

Norm of matrix measures maximum stretching matrix does to

 For each row i:

• Only care about numbers – form “tableau” or “augmented matrix”:

• Goal: reduce this to trivial system

and read off answer from right column

• Basic operation 1: replace any row by

• Replace row2 with row2 – 4 * row1

• Replace row1 with row1 – 3/2 * row2

• Read off solution: x1 = 2, x2 = 1

• Replace row1 with row1 – 3/2 * row2