Cholesky Decomposition

Martin Licht

UC San Diego

Winter Quarter 2021

Cholesky Decomposition

The class of symmetric positive definite matrices appears in

many applications.

We will see that for those matrices, the LU decomposition can

always be performed without pivoting.

Moreover, the LU decomposition of symmetric positive definete

matrices leads to the Cholesky decomposition.
Symmetric Positive Definite Matrices

LU Decomposition of Symmetric Positive Definite Matrices

Specialized Algorithms
Symmetric Positive Definite Matrices

Let A ∈ Rn×n be a matrix. We call A symmetric if

A = At .

We call A positive definite if

∀x ∈ Rn \ {0} : hx, Axi > 0.

A matrix that is both symmetric and positive definite is called

symmetric positive definite.
Symmetric Positive Definite Matrices

Suppose that M ∈ Rn×n is invertible. Let A = Mt M. Then A is

symmetric because

At = (Mt M)t = Mt M = A

Also, A is positive definite: for any x ∈ Rn with x 6= 0 we can

write y := Mx and observe y 6= 0. Then

hx, Axi = hx, Mt Mxi = hMx, Mxi = hy, yi > 0.

It follows that A is symmetric positive definite.

Symmetric Positive Definite Matrices

Example

     
1 5 5 1 0
 1 ,  10 , 1 5 1
1 0.17 0 1 5
Symmetric Positive Definite Matrices

The previous example is already exhaustive for the class of

symmetric positive definite matrices: all spd matrices A can be
written in the form A = Mt M for some invertible matrix M.

In fact, we can achieve that M is upper triangular.

Cholesky Decomposition

Example

    
1 1 1 1 1
1 1   1 1  = 1 2 1 
1 1 1 1 2
    
2 2 3 1 4 6 2
3 2   2 2 = 6 13 5
1 2 1 1 2 5 6
√  √ −1

2 q 2 √ 0  
 −1 q2 q 2 −1 0
√ 3 
3

3  = −1
 2 q2

 2 − 2
2 −1
0 − 32 √2 √2 0 −1 2
3 3
Symmetric Positive Definite Matrices

LU Decomposition of Symmetric Positive Definite Matrices

Specialized Algorithms
Cholesky Decomposition

If A ∈ Rn×n is invertible and R ∈ Rn×n is upper triangular with

A = Rt R, then we call R a Cholesky factor of A.

We call A = Rt R the Cholesky decomposition.

Note that A invertible implies R invertible.

Cholesky Decomposition

We will show the Cholesky decomposition exists precisely for

the class of symmetric positive definite matrices.

The strategy will be showing that symmetric positive definite

matrices allow for an LU decomposition without pivoting, and
then we modify that LU decomposition to get a Cholesky
decomposition.
Cholesky Decomposition

Lemma
Suppose that A is symmetric positive definite. Then A satisfies
the principle minor criterion.

Proof.
Let A be symmetric positive definite n × n matrix. We want to
show that all principle minors Ak , 1 ≤ k ≤ n are invertible.
Suppose that there exists a principle minor Ak ∈ Rk×k with
1 ≤ k ≤ n that is not invertible. So there exists v ∈ Rk with v 6= 0
v ∈ Rn be the extension of v to an n vector. We
but Ak v = 0. Let e
v 6= 0. We get
have e

0 < he
v, Ae
vi = hv, Ak vi = hv, 0i = 0.

This is a contradiction. So Ak must have been invertible in the

first place.
Cholesky Decomposition

Corollary
The LU decomposition without pivoting exists for symmetric
definite positive matrices.
Cholesky Decomposition

Lemma
The LU decomposition of a symmetric positive definite matrix
has U with positive diagonal entries.

Proof.
Let A be s.p.d. and let L, U ∈ Rn×n be the matrices from its LU
decomposition. We want to show that Uii > 0 for all 1 ≤ i ≤ n.
Suppose that instead there exists 1 ≤ i ≤ n with Uii ≤ 0. Then
Uii = 0 cannot hold because U is invertible, so Uii < 0 must hold.
Let v ∈ Rn be the solution to Lt v = ei , and then let w := Uv. We
obtain v by backward substitution and find that vi = 1 and that
vj = 0 for all j > i. Hence wi = uii vi = uii .

0 < hv, Avi = hv, LUvi = hLt v, Uvi = hei , wi = wi = uii < 0

This is a contradiction. Hence uii > 0 must hold.

Cholesky Decomposition

Let A = LU be the LU decomposition of a symmetric positive

definite matrix A ∈ Rn×n .

We now define the factors in the Cholesky decomposition.

√ Let
D∈R n×n be the diagonal matrix with Dii = Uii . We define new
n × n matrices

R := D−1 U, S := LD.

We get that

A = LU = LDD−1 U = SR.

The upper triangular matrix R will be the Cholesky factor in our

Cholesky decomposition. To verify that, it only remains need to
show that St = R.
Cholesky Decomposition

To see that, we first use the symmetry of A to find

SR = A = At = Rt St .

Since R and S are invertible, we thus get

St = R−t SR.

If we can show that R−t S = I, then we get St = R.

Cholesky Decomposition

Since R is upper triangular, Rt is lower triangular.

The inverse of a lower triangular matrix is again lower triangular.

Hence R−t is lower triangular.

The diagonal entries of R−t are the reciprocals of the diagonal

entries of Rt .

Furthermore, we recall that Rt and S have the same diagonal

entries.

Hence the diagonal entries of R−t are the reciprocals of the

diagonal entries of S.

We conclude that R−t S has only 1’s on the diagonal.

Cholesky Decomposition

Let us abbreviate T := R−t S.

So far we have T = R−t S is a unit lower triangular matrix.

Suppose that T has any non-zero entry below the diagonal, say,
in row i and column j. Let i be so small that all entries to the left
of ti,j are zero.

Then the entry (i, j) of TR will be ti,j rj,i 6= 0, which contradicts

St = TR being upper triangular. Hence T is also not only a unit
lower triangular matrix but also a diagonal matrix with only 1’s
on the diagonal. In other words, T is the unit matrix.

So R−t S = I must hold.

Putting this together, we finally get St = R.

Cholesky Decomposition

Theorem
Every symmetric positive definite matrix has a Cholesky
decomposition.

But we have also established before that every matrix which

has a Cholesky decomposition is symmetric positive definite.

We conclude the following theorem.

Theorem
A matrix has a Cholesky decomposition if and only if it is
symmetric positive definite.
Cholesky Decomposition

The above arguments also show the following procedure for

computing the Cholesky decomposition of a symmetric
positive definite matrix:
1: CholeskyDecomposition
2: Compute the factor U in the LU Decomposition
3: for m = 1, 2, . . . , n do
4: for k = m, m + 1, . . . , n do
√
5: rm,k = um,k / um,m
6: end for
7: end for
Specialized Algorithms
Consider the matrix A ∈ R3×3 given by
 
4 2 0
2 10 −3 .
0 −3 5

Its LU decomposition can be calculated by

     
4 2 0 4 2 0 4 2 0
2 10 −3 1/2 9 −3 1/2 9 −3 ,
0 −3 5 0 −3 5 0 −1/3 4

that is,
    
1 0 0 4 2 0 4 2 0
1/2 1 0 0 9 −3 = 2 10 −3
0 −1/3 1 0 0 4 0 −3 5
Specialized Algorithms

... that is,

    
1 0 0 4 2 0 4 2 0
1/2 1 0 0 9 −3 = 2 10 −3
0 −1/3 1 0 0 4 0 −3 5

We extract the diagonal,

  1  
1 0 0 2 2 4 2 0
1  0 9 −3
· · · = 1/2 1 0  3   3
0 −1/3 1 2 1 0 0 4
2
Specialized Algorithms

We extract the diagonal,

  1  
1 0 0 2 2 4 2 0
1  0 9 −3
· · · = 1/2 1 0  3   3
0 −1/3 1 2 1 0 0 4
2

Bringing the diagonal entries of triangular matrices to matching

values, we get
    
2 0 0 2 1 0 4 2 0
 1 3 0 0 3 −1 = 2 10 −3
0 −1 2 0 0 2 0 −3 5

Note that we have divided the i-th row of U by the square root of
its i-th diagonal entry, and we have multiplied the i-th column of
L by the corresponding entry.
Symmetric Positive Definite Matrices

LU Decomposition of Symmetric Positive Definite Matrices

Specialized Algorithms
Specialized Algorithms

The Cholesky decomposition can be calculated from the LU

decomposition with only minimal effort.

However, a different algorithm seems to more common in

practice, as we will discuss in the next subsection.
Specialized Algorithms

Let A be a positive definite matrix, and let A = Rt R be its

Cholesky decomposition. Then R is an upper triangular matrix
with positive diagonal entries.
 
r1,1 r1,2 r1,3 . . . r1,n

 r2,2 r2,3 . . . r2,n 
R=
 r3,3 . . . r3,n .
 . . .. 
 . . 
rn,n
Specialized Algorithms

From A = Rt R,
  
r1,1 r1,1 r1,2 r1,3 . . . r1,n
r1,2 r2,2  r2,2 r2,3 . . . r2,n 
  
A = r1,3 r2,3 r3,3 r3,3 . . . r3,n 
.
 

 .. .. .. ..  .. .. 
 . . . .  . . 
r1,n r2,n r3,n . . . rn,n rn,n

The formula for the matrix-matrix multiplication leads to several

identities that relate the entries of A to the entries of R.
Specialized Algorithms

From A = Rt R we get the following identities for the entries of R.

Diagonal entries: for 1 ≤ i ≤ n we have

i
X i−1
X
2
aii = rk,i rk,i = rk,i + rii2 .
k=1 k=1

Since the entries of R are positive,

v
u
u i−1
X
rii = aii −
t 2 .
rk,i
k=1

This formula for rii only depends on entries of R above rii in the
i-th column of R.
Specialized Algorithms

Superdiagonal entries: for 1 ≤ j < i we have

j j−1
X X
aij = rk,i rk,j = rk,i rk,j + rj,i rj,j
k=1 k=1

Since rj,j is non-zero,

 
j−1
X
−1 
rj,i = rj,j aij − rk,i rk,j 
k=1

This formula for rj,i depends only on entries of R that are in a

previous column or row of R.
Specialized Algorithms
This suggests that we can calculate R entry by entry from the
given matrix A. We may use the following calculation:
I Calculate row 1 of R
I Calculate row 2 of R
I ...
I Calculate row n of R
In a pseudocode, this reads as follows:

1: CholeskyDecomposition
2: for j = 1, n do
q2, . . . ,P
j−1 2
3: rj,j = aj,j − k=1 rk,j
4: for i = j + 1,  j + 2,P. . . , n do 
−1 j−1
5: rj,i = rj,j ai,j − k=1 rk,i rk,j
6: end for
7: end for
Specialized Algorithms

Alternatively, we may use the following calculation:

I Calculate column 1 of R
I Calculate column 2 of R
I ...
I Calculate column n of R
In a pseudocode, this reads as follows:

1: CholeskyDecomposition
2: for i = 1, 2, . . . , n do
3: for j = 1, 2,. . . , i − 1 do 
−1 Pj−1
4: rj,i = rj,j ai,j − k=1 rk,i rk,j
5: end for q
ri,i = ai,i − i−1 2
P
6: k=1 rk,i
7: end for