Lecture 5: Eigenvalues of Hermitians Matrices

This lecture takes a closer look at Hermitian matrices and at their eigenvalues. After a few
generalities about Hermitian matrices, we prove a minimax and maximin characterization of
their eigenvalues, known as Courant–Fischer theorem. We then derive some consequences
of this characterization, such as Weyl theorem for the sum of two Hermitian matrices, an
interlacing theorem for the sum of two Hermitian matrices, and an interlacing theorem for
principal submatrices of Hermitian matrices.

1 Basic properties of Hermitian matrices

We recall that a matrix A ∈ Mn is called Hermitian if A∗ = A and skew-Hermitian if A∗ = −A,
and we note that A is Hermitian if and only if iA is skew-Hermitian. We have observed
earlier that the diagonal entries of a Hermitian matrix are real. This can also be viewed as a
particular case of the following result.
Proposition 1. Given A ∈ Mn ,
[A is Hermitian] ⇐⇒ [hAx, xi = x∗ Ax ∈ R for all x ∈ Cn ].

Proof. ⇒ If A is Hermitian, then, for any x ∈ CN ,

hAx, xi = hx, A∗ xi = hx, Axi = hAx, xi,
so that hAx, xi ∈ R.
⇐ Suppose that hAx, xi ∈ R for all x ∈ Cn . For any u, v ∈ CN , we have
hA(u + v), u + vi = hAu, ui + hAv, vi +hAu, vi + hAv, ui, so that hAu, vi + hAv, ui ∈ R.
| {z } | {z } | {z }
∈R ∈R ∈R
Taking u = ej and v = ek yields
ak,j + aj,k ∈ R, thus Im(ak,j ) = −Im(aj,k ),
then taking u = iej and v = ek yields
iak,j − iaj,k ∈ R, thus Re(ak,j ) = Re(aj,k ).
Altogether, this gives ak,j = aj,k for all j, k ∈ [1 : n], i.e., A = A∗ .

Proposition 2. Any matrix A ∈ Mn can be uniquely written in the form

A = H + S, where H ∈ Mn is Hermitian and S ∈ Mn is skew-Hermitian,
A = H1 + iH2 , where H1 , H2 ∈ Mn are both Hermitian.

Proof. If A = H + S with H Hermitian and S skew-Hermitian, then A∗ = H ∗ + S ∗ = H + S.

By adding and subtracting these two relations, we derive H = (A + A∗ )/2 and S = (A − A∗ )/2,
hence H and S are uniquely determined. Moreover, with H and S given above, it is readily
verified that H is Hermitian, that S is skew-Hermitian, and that A = H + S. For the second
statement, we use the fact that H2 is Hermitian if and only if S := iH2 is skew-Hermitian.

1
2 Variational characterizations of eigenvalues
We now recall that, according to the spectral theorem, if A ∈ Mn is Hermitian, there exists
a unitary matrix U ∈ Mn and a real diagonal matrix D such that A = U DU ∗ . The diagonal
entries of D are the eigenvalues of A, which we sort as

λ↑1 (A) ≤ λ↑2 (A) ≤ · · · ≤ λ↑n (A).

We utilize this notation for the rest of the lecture, although we may sometimes just write λ↑j
instead of λ↑j (A) when the context is clear. Note also that the columns u1 , . . . , un of U form an
orthonormal system and that Auj = λ↑j (A)uj for all j ∈ [1 : n]. We start by observing that

(1) λ↑1 (A)kxk22 ≤ hAx, xi ≤ λ↑n (A)kxk22 ,

with the leftmost inequality becoming an equality if x = u1 and the rightmost inequality
becoming an equality if x = un . The argument underlying the observation (1) will reappear
several times (sometimes without explanation), so we spell it out here. It is based on the
expansion of x ∈ Cn on the orthonormal basis (u1 , . . . , un ), i.e.,
n
X n
X
x= cj uj with c2j = kxk22 .
j=1 j=1

We now simply write

↑ Pn ↑

2 2
n n n n n  ≥ λ1 j=1 cj = λ1 kxk2 ,

cj λ↑j uj , λ↑j c2j =
X X X X X
(2) hAx, xi=h cj Auj , cj uj i=h cj uj i=
j=1 j=1 j=1 j=1 j=1 ≤ λ↑ Pn c2 = λ↑ kxk2 .

n j=1 j n 2

The inequalities (1) (with the cases of equality) can also be expressed as λ↑1 = minkxk2 =1 hAx, xi
and λ↑n = maxkxk2 =1 hAx, xi, which is known as Rayleigh–Ritz theorem. It is a particular case
of Courant–Fischer theorem stated below.

Theorem 3. For A ∈ Mn and k ∈ [1 : n],

(3) λ↑k (A) = min max hAx, xi = max min hAx, xi.
dim(V )=k x∈V dim(V )=n−k+1 x∈V
kxk2 =1 kxk2 =1

Remark. This can also be stated with dim(V ) ≥ k and dim(V ) ≥ n − k + 1, respectively, or
(following the textbook) as

λ↑k (A) = min max hAx, xi = max min hAx, xi.

w1 ,...,wn−k ∈Cn x⊥w1 ,...,wn−k w1 ,...,wk ∈Cn x⊥w1 ,...,wk
kxk2 =1 kxk2 =1

2
Proof of Theorem 3. We only prove the first equality — the second is left as an exercise. To
begin with, we notice that, with U := span[u1 , . . . , uk ], we have

min max hAx, xi ≤ max hAx, xi ≤ λ↑k (A),

dim(V )=k x∈V x∈U
kxk2 =1 kxk2 =1

where the last inequality follows from an argument similar to (2). For the inequality in the
other direction, we remark that our objective is to show that for any k-dimensional linear
subspace V of Cn , there is x ∈ V with kxk2 = 1 and hAx, xi ≥ λ↑k (A). Considering the subspace
W := span[uk , . . . , un ] of dimension n − k + 1, we have

dim(V ∩ W ) = dim(V ) + dim(W ) − dim(V ∪ W ) ≥ k + n − k + 1 − n = 1.

Hence we may pick x ∈ V ∪ W with kxk2 = 1. The inequality hAx, xi ≥ λ↑k (A) follows from an
argument similar to (2).

We continue with some applications of Courant–Fischer theorem, starting with Weyl theorem.

Theorem 4. Let A, B ∈ Mn be Hermitian matrices. For k ∈ [1 : n],

λ↑k (A) + λ↑1 (B) ≤ λ↑k (A + B) ≤ λ↑k (A) + λ↑n (B).

Proof. We use Courant–Fischer theorem and inequality (1) to write

   
λ↑k (A + B) = min max h(A + B)x, xi = min max hAx, xi + hBx, xi
dim(V )=k x∈V dim(V )=k x∈V | {z }
kxk2 =1 kxk2 =1
≤λ↑n (B)
 
≤ min max hAx, xi + λ↑n (B) = λ↑k (A) + λ↑n (B).
dim(V )=k x∈V
kxk2 =1

This establishes the rightmost inequality. We actually use this result to prove the leftmost
inequality by replacing A with A + B and B with −B, namely

λ↑k (A) = λ↑k (A + B + (−B)) ≤ λ↑k (A + B) + λ↑n (−B) = λ↑k (A + B) − λ↑1 (B).

A rearrangement gives the desired result.

Corollary 5. For Hermitian matrices A, B ∈ Mn , if all the eigenvalues of B are nonnegative

(i.e., hBx, xi ≥ 0 for all x ∈ Cn , or in other words B is positive semidefinite), then,

λ↑k (A) ≤ λ↑k (A + B) for all k ∈ [1 : n].

Weyl theorem turns out to be the particular case k = 1 of Lidskii’s theorem stated below
with the sequences of eigenvalues arranged in nonincreasing order rather than nondecreasing
order.

3
Theorem 6. For Hermitian matrices A, B ∈ Mn , if 1 ≤ j1 < . . . < jk ≤ n, then
k k k
λ↓j` (A λ↓j` (A) λ↓` (B).
X X X
+ B) ≤ +
`=1 `=1 `=1

Proof. Replacing B by B − λ↓k+1 (B)I, we may assume that λ↓k+1 (B) = 0. By the spectral
theorem, there exists a unitary matrix U ∈ Mn such that
B = U diag λ↓1 (B), . . . , λ↓k (B), λ↓k+1 (B), . . . , λ↓n (B) U ∗ .
 
| {z } | {z }
≥0 ≤0

Let us introduce the Hermitian matrices

B + := U diag λ↓1 (B), . . . , λ↓k (B), 0, . . . , 0 U ∗ , B − := U diag 0, . . . , 0, −λ↓k+1 (B), . . . , −λ↓n (B) U ∗ .
   

They have only nonnegative eigenvalues and satisfy B = B + − B − . According to Corollary 5,

λ↓j (A + B + ) ≥ λ↓j (A) and λ↓j (A + B + ) = λ↓j (A + B + B − ) ≥ λ↓j (A + B).
It follows that
k k n
λ↓j` (A + B) − λ↓j` (A) ≤ λ↓j` (A + B + ) − λ↓j` (A) ≤ λ↓j (A + B + ) − λ↓j (A)
X
X
X

`=1 `=1 j=1

k
λ↓` (B).
X
= tr (A + B + ) − tr (A) = tr (B + ) =
`=1

This is just a rearrangement of the desired result.

3 Interlacing theorems
The two results of this section locate the eigenvalues of a matrix derived from a matrix A
relatively to the eigenvalues of A. They are both consequences of Courant–Fischer theorem.
Theorem 7. Let A ∈ Mn be a Hermitian matrix and As be an s × s principal submatrix of A,
s ∈ [1 : n]. Then, for k ∈ [1 : s],
λ↑k (A) ≤ λ↑k (As ) ≤ λ↑k+n−s (A).
Remark. The terminology of interlacing property is particularly suitable in the case of an
(n − 1) × (n − 1) principal submatrix A,
e since we then have

λ↑1 (A) ≤ λ↑1 (A)

e ≤ λ↑ (A) ≤ λ↑ (A)
2 2
e ≤ λ↑ (A) ≤ · · · ≤ λ↑ (A) ≤ λ↑ (A)
3 n−1 n−1
e ≤ λ↑ (A).
n

As for the particular case s = 1, it gives

λ↑1 (A) ≤ aj,j ≤ λ↑n (A) for all j ∈ [1 : n],
which is also a consequence of (1) with x = ej .

4
Proof of Theorem 7. Suppose that the rows and columns of A kept in As are indexed by a
set S of size s. For a vector x ∈ Cs , we denote by x e ∈ Cn the vector whose entries on S
equal those of x and whose entries outside S equal zero. For a linear subspace V of Cs , we
x, x ∈ V }, which is a subspace of Cn with dimension equal the dimension of V .
define Ve := {e
Given k ∈ [1 : s], Courant–Fischer theorem implies that, for all linear subspace V of Cs with
dim(V ) = k,
max hAs x, xi = max hAe ei = max hAe
x, x ei ≥ λ↑k (A).
x, x
x∈V x∈V e∈Ve
x
kxk2 =1 kxk2 =1 ke
xk2 =1

Taking the minimum over all k-dimensional subspaces V gives λ↑k (As ) ≥ λ↑k (A). Similarly, for
all linear subspace V of Cs with dim(V ) = s − k + 1 = n − (k + n − s) + 1,
min hAs x, xi = min hAe ei = min hAe
x, x ei ≤ λ↑k+n−s (A).
x, x
x∈V x∈V e∈Ve
x
kxk2 =1 kxk2 =1 ke
xk2 =1

Taking the maximum over all (s−k+1)-dimensional subspaces V gives λ↑k (As ) ≤ λ↑k+n−s (A).

Theorem 8. Let A, B ∈ Mn be Hermitian matrices with rank(B) = r. Then, for k ∈ [1 : n−2r],

λ↑k (A) ≤ λ↑k+r (A + B) ≤ λ↑k+2r (A).

Before turning to the proof, we observe that the n×n Hermitian matrices of rank r are exactly
the matrices of the form
X r
(4) B= µj vj vj∗ , µ1 , . . . , µr ∈ R \ {0}, (v1 , . . . , vr ) orthonormal system.
j=1

Indeed, by the spectral theorem, the n × n Hermitian matrices of rank r are exactly the
matrices of the form
B = V diag µ1 , . . . , µr , 0, . . . , 0 V ∗ , µ1 , . . . , µr ∈ R \ {0}, V ∗ V = I,
 
(5)
which is just another way of writing (4).

Proof of Theorem 8. Let k ∈ [1 : n − r] and let B ∈ Mn be as in (4). We use Courant–Fischer

theorem to derive
λ↑k (A) = max min hAx, xi ≤ max min hAx, xi
w1 ,...,wk ∈Cn x⊥w1 ,...,wk w1 ,...,wk ∈Cn x⊥w1 ,...,wk ,v1 ,...,vr
kxk2 =1 kxk2 =1

≤ max min h(A + B)x, xi ≤ max min h(A + B)x, xi

w1 ,...,wk ∈Cn x⊥w1 ,...,wk ,v1 ,...,vr w1 ,...,wk+r ∈Cn x⊥w1 ,...,wk+r
kxk2 =1 kxk2 =1

= λ↑k+r (A + B).
This establishes the rightmost inequality. This inequality can also be used to establish the
leftmost inequality. Indeed, for k ∈ [1 : k − 2r], we have k + r ∈ [1 : k − r], and it follows that
λ↑k+r (A + B) ≤ λ↑k+r+r (A + B + (−B)) = λ↑k+2r (A).

5
 4 Exercises

Ex.1: Prove the second inequality in (3).

Ex.2: Verify in details the assertions made in (4) and (5).

Ex.3: Exercise 1 p. 174

Ex.4: Exercise 3 p. 174

Ex.5: Exercise 6 p. 174

Ex.6: Exercise 11 p. 175

Ex.7: Exercise 13 p. 175

Ex.8: Exercise 4 p. 181

Ex.9: Exercise 2 p. 198

Ex.10: Exercise 5 p. 199

Ex.11: Exercise 6 p. 199

Ex.12: Exercise 7 p. 199

Ex.13: Exercise 14 p. 200

Ex.14: Exercise 17 p. 200