Wedderburn Decomposition and Its Applications

to Matrix Factorizations

Moody T. Chu
Department of Mathematics
North Carolina State University
Raleigh, North Carolina 27695-8205

Joint work with R. E. Funderlic

Outline
Rank Reduction Formula
Wedderburn Process
Inverse Wedderburn Process
Applications

Rank Reduction Formula

An obvious result: (Wedderburn 34)
A Rmn, x Rn, y Rm, y T Ax 6= 0
+
T

A
B := A Axy
T
y Ax

rank(B) = rank(A) 1
The converse is true! (Egervary, 60; Householder 75)
u Rm , v Rn
+
B := A 1uv T
+
rank(B) = rank(A) 1
m
x Rn, y Rm 3 u = Ax, v = AT y, = y T Ax

 Block form: (Cline & Funderlic, 79)

U Rmk , R Rkk , V Rnk
+
B = A U R1V T
+
rank(B) = rank(A) rank(U R1V T )
m
X Rnk , Y Rmk
3 U = AX, V = AT Y, R = Y T AX

Repeated Application
Ak 6= 0 = xk Rn, yk Rm 3 k := ykT Ak xk 6= 0.
A sequence of matrices with decreasing ranks:
Ak+1 := Ak k1Ak xk ykT Ak
rank(A) = = A+1 = 0.
Wedderburn decomposition:
A =

k=1

k1Ak xk ykT Ak

:= 1T
:= diag{1, . . . , }.
:= [1, . . . , ], := [1, . . . , ]
k := Ak xk Rm
k := ATk yk Rn.
Different {x1, . . . , x } and {y1, . . . , y } = Different decomposition.
5

An Oblique Projection
A bilinear form on Rn Rm:
< x, y >:= y T Ax.
This is NOT an inner product.
Rewrite the Wedderburn formula:

< z, y >

x ,
Bz = A z
<
x,
y
>

< x, w > T

T
T

y A.
w B = w
< x, y >
Two projectors:
PA,x,y
P 0A,x,y

xy T A
:= I T
y Ax
Axy T
:= I T .
y Ax

Action of B in terms of A:
B = APA,x,y = P 0A,x,y A.
6

First Step in Wedderburn Process

Define:

u1 := x1
v1 := y1

< x2 , v 1 >
u1
u2 := Px2 = x2
< u1 , v 1 >
< u1 , y 2 >
T 0 T
v1
v2 := (y2 P ) = y2
< u1 , v 1 >
Results:
Au2
v2T A
< u2 , v 1 >
2

=
=
=
=

A2x2 R(A2)
y2T A2 R(AT2 )
< u1, v2 >= 0
y2T A2x2 =< u2, v2 > .

Wedderburn Process
rank(A) =
+
{x1, . . . x } Rn, {y1, . . . y } Rm
+
ykT Ak xk 6= 0

<xk ,vi >

uk := xk Pk1
i=1 <ui ,vi > ui
+
<ui ,yk >
vk := yk Pk1
i=1 <ui ,vi > vi
The process is well defined.
Bypassing the intermediate Ak :
Auk = Ak xk ,
vkT A = ykT Ak ,
k = ykT Ak xk =< uk , vk >
Biconjugacy:
< uk , vj >=< uj , vk >= 0.
8

Remarks
Wedderburn process Gram-Schmidt process:
xi Rn, yj Rm Vectors in a single space.
Bilinear form y T Ax Standard inner product.
ui, vj biconjugacy Orthogonality.
Wedderburn decomposition:
VkT AUk = k ,
T
A = AU 1
V A.
k := diag{1, . . . , k }.
The (1,2)-inverse AI :
T
AI = U 1
V .
T
A nonsingular = A1 = Un1
n Vn .
An SVD analogue, but Un and Vn not necessarily orthogonal.

Matrix From of Wedderburn Process

Rewrite
(x)

X k = U k Rk ,
(y)
Y k = V k Rk .

(x)
Rk

1
0

<x1 ,v1 >

<u1 ,v1 >

1
...

...
...
. . . ..
1
0

..

<xj ,vj1 >

<uj1 ,vj1 >

1
...

0
0

(x)

Note that Rk depends implicitly on Vk .

10

<xj ,v1 >

<u1 ,v1 >

. . . ..
1
0 1

Inverse Wedderburn Process

Think of the Wedderburn process as a function acting on
Rnk Rmk by
f (X, Y ) := (U, V ).
What is f 1(U, V )?

U Rnk , V Rmk
+
V T AU = Rkk diagonal
+
X := U Rx, Y := V Ry
+
Rx , R y
arbitrary upper triangular with unit diagonal
f
Same U and V

11

Recover the QR Decomposition

Suppose A = QR.
Identify QT AR1 = I = U = R1, V = Q.
Define
X := R1Rx
Y := QRy .
Rx, Ry arbitrary upper triangular with unit diagonal
entries.
Explicit dependence on Q and R (NOT good!)
Define
X := I
Y := A.
The Wedderburn process produces a matrix U which
is upper triangular and a matrix V whose columns are
mutually orthogonal. In fact, V T V = . Furthermore, AU = V .
12

Recover the LR Decomposition

Suppose A = LR.
Identify L1AR1 = I = U = R1, V = LT .
The choice
X := R1Rx
Y := LT Ry
for the Wedderburn process will produce the (implicit)
LR decomposition of A.
Define
X := I
Y := I.
The Wedderburn process produces A = VnT Un1
which is the unique LDMT decomposition of A.

13

Relation to SVD
Freedom in selecting the vectors xi and yj .
?
Division by k = ykT Axk =
Instability.

Desirable at each stage to choose xk and yk so as to

Maximize k = ykT Ak xk =< uk , vk >
Subject to xTk xk = 1, ykT yk = 1.
The Wedderburn process by meeting this requirement is
precisely the SVD of A.

14

 Lagrange multipliers =
Necessary condition:
ATk yk = k xk
Ak xk = k yk .
Maximal value = k = kAk k2 = Largest singular
value of Ak = The kth singular value k of A.
yk , xk = the kth left and right singular vectors.
The Wedderburn process leaves the singular vectors invariant.
Can X and Y be chosen before the SV D is known?
(N0?!)

15

Other Applications
Suppose A is symmetric and take X = Y . Then U = V .
The Wedderburn process gives the canonical form of A
with respect to congruence.
With specially selected of X and Y , the Wedderburn process includes the Lanczos algorithm as a special case.
We can explicitly describe X and Y . (Funderlic, November 9, 1993.)
Wedderburn (34) = Lanczos (50) = Hestenes and
Stiefel (52).
Welcome to the Lanczos Conference (December 12-17,
1993).

16

Occurrence in BFGS (and DFP)

Minimize f : Rn R.
Solve g(x) := f (x) = 0.
Jacobian G(x) of g(x) (Hessian of f (x)) is often unavailable or very expensive to compute.
Develop cheap and reasonable approximations to either G(x) or its inverse.
Prefer to preserve symmetry and positive definiteness.

17

 BFGS One successful Hessian update:

ycycT HcscsTc Hc
H+ := Hc + T T
.
y c sc
sc H c sc
Secant equation:
H + sc = y c
sc := x+ xc
yc := g(x+) g(xc)
Wedderburns foumula shows up:
H+scsTc H+
HcscsTc Hc
H+ T
= Hc T
.
sc H + sc
sc H c sc
What is BFGS doing?
A necessary condition:
H + P H+ = H c P Hc .
Tear and rebuild:
HcscsTc Hc H+scsTc H+
H+ = H c T
+ T
.
s
H
s
s
H
s
c {z c c } |
c {z + c }
|
pruning
grafting
18