Lecture 4.

LU decomposition (part II)

LU decompostion (pivoting)

LU decomposition without pivoting:

for k=1:n

A(k,k:n)=A(k,k:n)-A(k,1:k-1)A(1:k-1,k:n)

A(k+1:n,k)=(A(k+1:n,k)-A(k+1:n,1:k-1)A(1:k-1,k))/A(k,k)

end

In the k-th sub-step, A(k,k) is used as a denominator.

1) If Alk kl 0 , then the LU decomposition cannot continue, even if A is invertible.

Example: A 1
2) If Alk k is very small then the division by A(k,k) will make L and U

contains large error in the solution due to round-o error of digital computers.

Example: Solve

ii 侧 们
floating point arithmetic
Assume the computation is on a computer with 3 correct digits. The correct solution

is
001 ke
xirue
1t 999
truncation
1 999900 o 3 digits

LU Decomposition

iii li iii 104 1

11004

ggo
keeiIig9
 So the computed LU Decomposition is

i
1
Is
Forward substitution

li 1 1 y2 2 104 9998 9990

Backward substitution
1 x 1
1104
0
999011
别 fgg x f o

So the computed solution is

Xcomputed
1 1
which has a relative error 1in 2 norm

11Xcomputed Xtrue112 1119 l 999 112

70.7
11Xtrue 112 11 egg
112

To overcome these, we introduce a technique called “pivoting”.

Partial Pivoting:

• Permute the equations so that the pivot is as large as possible, i.e. we nd a

permutation matrix P s.t. the “pivot” in LU decomposition of PA

PA=LU

is as large as possible.

• We say P Rnxn is a permutation matrix if there is only exactly a “1” in each row
 and each column and all other entries are 0.

For example:

we have i.e.it
1 1 11 1 permutesfgt.gg

10

徽關
ie.it mute
1
we
have1 per
州
• To avoid a small A(k,k) at sub-step k, we choose the max entry in A(k:n,k) in

magnitude as “pivot”.

In the original LU:column k

11111111111 少
少 111111

0W k
l s __ _y y

Original LU is the same as

column
k
7 111111

Y
y

1111111111
必
Δ
LU with partial pivoting: (adding row switch before modify the row)
column
k

少
1
y
130row switching

ix

The modi cation of A(k,k+1:n) has to be done after row switching:

P I
for k l n

A kin kl A kin kl A kin I k 1 A 1 kt k

ik argmax 1Alj k 1
kj n

A k l nl A Jk I in

P k l n c sPCJk.li n

A k ktl n A k ktl n A k l k 1 All kt ktl n

Alktl in k A ktl n k A k k1

We will get PA LU Then

PAx̅ P5 L LU x̅ Plo LT Plo and ux̅ g

Consequently the forward substitution will solve LJ p5
 and the backward substitution will solve Unt g

Example 1: on a 3 digit
I I
computer
1104 1
1

LU decomposition with partial pivoting

k 1 1
1104
1 1 i ii
P 1
k 2

ii lie in yeisdigiy 104 0.999

d9999 0.999

u
1 ˇ y
S P L
0 0.999

Forward substitution Solve LT

品 p1 l
11 1 11 1
y 2 keep 3 digits

y2 1 10 4y 1 0.0002 0.9998 0.999

Back substitution Ux̅

lii.gg1 11 1 999
 0.999 0.999 2 1
X 2 2 1 1

Xcomputed
1i
Recall X true 9991

Xtrue Xcomputed 111品 1

relative error
Xtruelk 111 eee 1h

4g 0.1

Example: Find pivoted LU decomposition of

Here we ignore truncation

A 主 ㄅㄣ
别 or round off errors

Solution k 1 P 18 ǒgb

1 剖 営
k 2

焦 㦬 发 主

f
2 131

1
1
 k 3

告
到 iii 1 3ㄨ ㄨ 7

Therefore

1 准 彭唯 善 善
P A L U
• In partially pivoting, the strict lower triangular part of L has an absolute value

smaller than 1, because the numerator is always smaller than the denominator (i.e.

the pivot).

• There are other pivoting strategies, e.g., fully pivoting, where permutation on both

equations and unknowns are imposed.

column
kith Full pivoting choose pivot from

回 A1k k by switching b0ᵗʰ

of gy qnqgdymns
丽 he
P Q are both permutations

• Matrix decomposition/factorization is the main theme of matrix computation. We

will learn many other matrix factorizations such as QR, eigenvalue decomposition,
 singular value decomposition.

• There are other decompositions that can be used to solve Ax=b. For example, we

can use QR decomposition to solve Ax=b.

Exploiting Structure of A

Both memory and computation can be saved by exploiting the structure of A in

solving Ax=b.

• Symmetric Positive De nite (SPD) matrices

De nition: A matrix AER is symmetric positive de nite (SPD) if

1) A AT A is symmetric and
2) TA x̅ so for any x̅ ER and x̅ to A is positive definite
If the “>0” in 2) is replaced by “>=0”, then A is called symmetric positive semi-

de nite (SPSD).
Xm ㄨ
Facts: a) A CC for some C is SPSD. In particular, if
CER is
T
ER
nonsingular, then. A CCT is SPD.

b) If A is SPD, then A is nonsingular.

c) If A is SPD (SPSD respectively), then Ali iz i is SPD

sPSD respectivelyI
for iz i In particular all diagonal entries

a ii of A are positive non negative respectively if A is

SPD ISPSD respectively

Examples of SPSD/SPD matrices:

(a) The Hessian of a convex function is SPSD. In particular, the Hessian of a strictly

convex function is SPD.

(b) The covariance matrix is SPSD

(c)
A
1 i
0

proof A AT is obvious
For any x̅ 0

x̅ A x̅
i I 1
x

2ㄨ2
2x i 2X X2
2
x
2 x t x 仁 0

d)

0 1 D discrete Laplacian matrix

Afiiiii.idn. l
A AT obvious
ER and x̅ 0
2 For any x̅ 1劐
nai x.x

州 iii 劚
2 Xi Xixitl Xin Xi
2 xi 2 xixiti
xitxn for all x̅
x it xi 0
1㓷 to
Propertities of SPD/SPSD

(a) A is SPD There exists a nonsingular nxn matrix C

such that A CCT

proof only
AT CC
T
J CC
T
A

For any x̅ Ō TA x̅ x̅'CCT x̅ ix Tcix̅

11CTx̅1122 0

since x̅ 0 c is non singular we have CT is

nonsingular and Nullcc TI 0
Hence CT x̅ 0 and 11CTill i 0

Altogether TA x̅ 0
for any x̅ ō

(b) A is SPD A is symmetric

The eigenvalues of A are all positive
A is SPSD A is symmetric
The eigenvalues of A are all non negative
(c) If A is SPD/SPSD, then all principal sub matrix of A are SPD/SPSD. In particular,

A is SPD/SPSD, then all diagonals of A are positive/non-negative.

De nition: (principal submatrix of A ER ): Let s 1 2 n

Qij its jes is called a principal sub matrix of A

(d) If A is SPD, then A is non-singular.
Cholesky decomposition:

Let A be an SPD matrix. Then there exists a decomposition

A=LL

Where L ER
ㄨ is a lower triangular matrix with positive diagonals (not

necessarily unit lower triangular).

Algorithm for Cholesky decomposition:

We set

1 0 ER

lnilnzi iln.mil nn

Compute L Rnxn Column by column.

Sub-step 1: Partition L as

1
lni lnz ln n t ln n Hi 品 品 n.in

entries need to be computed

Then
1 1 0
412in
Dj
L
L LT
Liz n 2in 8 L 2 J
n 2in

1
L 2in 1 L 1 1
 We partition A accordingly

A
1AA 2in 11

Since A L LT
2
1 1 block L 1 11 A CI 11 LCI I All I

2.11 block L 12 11 1 L 1 1 A 2 n 1

L 2 n D A 2in DILLI D
Remark L 1 1 0 due to the fact A is SPD
Step k:

前 entries already computed

L lkt.int
entries to be computed

iininnnnnin.in
lnn
lniknln k

0 0
点 咒只 L kk O
44
L k 1in I k 1 L ktlin kl 4kt1in kt lin

T
LL

f
L k l ik_i L k 1 k 1 4 kk
Llktl n 1 k 1 L k 1 k1 L ktlin k L kk

Compare it with A Alk k 少

Alktlin k
 2
T
12 21 block Ak k L k l k 1 L k l i k 11 L k k

L_l8R
S L k k Alk.IR k 1 1 LCk 1 k 1

L k k1 0 A is spp will guarantee this

number is positive

3 2 block
T
A1ktI n k L ktl in I k 1 1L k 1 k D L1ktI n k L k k1
ㄒ
Llktl in k1 A ktl n kl hlktlin.li k 1 LCk I k 1 Lkk

Repeat until n-sub steps.

So, the full Cholesky decomposition algorithm is

for k=1:n

L(k,k)=(A(k,k)-L(k,1:k-1)(L(k,1:k-1))T )

L(k+1:n,k)=(A(k+1:n,k)-L(k+1:n,1:k-1)L(k+1:n,1:k-1)T )/L(k,k)

end
Alk kl
Remark: Note that
LCk.kare only
Alktlin.IR
used for L ktlin kl at sub-step k. To save
Ack
memory, k Alktlin.IR can be overwritten by 災
点n k
Thus, a memory e cient version is

for k=1:n

A(k,k)=(A(k,k)-A(k,1:k-1)(A(k,1:k-1))T)

A(k+1:n,k)=(A(k+1:n,k)-A(k+1:n,1:k-1)A(k+1:n,1:k-1)T)/A(k,k)

end

I
 for k l in

A kk Akk A k I k 11 A k 1 k 1J
主
A k k1 A k k1
ㄒ
Alktl n k A k 1 n k A ktl n I k 1 A1kt lin 1 k 1
A ktl n k A ktl n k A kk

end
Remark: (1) We operate/store only the lower triangular part. So the memory of

Cholesky is 1 2 3 n 竺
which is about half of standard LU.

(2) The computational cost is n lower order terms , which is about

half of the standard LU decomposition.

(3) A is SPD implies that A(k,k)-L(k,1:k-1)(L(k,1:k-1)T) is positive. Therefore,

pivoting is not necessary.

Graphical illustration:
column k 1

t ㄔㄨㄥ
__

主
ii

Example: Find the Cholesky decomposition of the matrix

UTD
 到
sont
A

下
籪

2 1 不合啦
信 喜

d.o 信

Therefore
f 吉 信
0 2 02 7

f
and A LLT
吉 信 0

Tridiagonal matrices:

De nition: is a tridiagonal matrix if

AER Qij 0
if I i J171
i.e.,

叫 iii iiiii
The LU decomposition without pivoting applied to A:

unchanged

A
信 if changed

òǒiǒiǒiiy
We see that

(a) Only 2(n-1) entries are modi ed. So, the LU decomposition for tridiagonal matrix

A can be simpli ed to:

k l in
for
Akk A kk A k k1 A k1 k

A kt1 k A k1 k A k k

end
Obviously, the computational cost is 意 3iiii.sn 3 0 n

(b) The resulting matrix is still tridiagonal, so that

iii iii
(L and U are also bi-diagonal.)

In forward substitution we solve LT 5

1 州
䵻
1
i.in
 1 b yz bz
b

lziy Ynlk.ktYk.it
bn Yn bn lk.k.ly k1

The computational cost is

2 2 2 n 1 O n
which is much smaller than 0 n in the general case

In backward substitution we solve ux̅ g

品
1
un. i
㦦

The kth equation

Ukkxkt Uk ktIXktl Yk
Xk Yk Uk ktIXkti Ukk k n n 1 1

Computational cost
3 2 3n 2 O n

which is very small compared to 0 n in general case

Consequently, the forward substitution (solving Ly=b) and backward substitution

(solving Ux=y) can be done in O(n) as well.

2 311 3 8n 7
3121in
(c) Altogether, solving Ax=b with tridiagonal A can be done in O(n) computations. As

a comparison, general dense A needs 0m (In particular 部) computations.

We can extend it to banded matrices

aij 0 if l i_j l s p

y ii
x x
bandwidth p