Professional Documents
Culture Documents
Deflated Decomposition of Solutions of Nearly Singular Systems
Deflated Decomposition of Solutions of Nearly Singular Systems
Society for Industrial and Applied Mathematics is collaborating with JSTOR to digitize, preserve and extend
access to SIAM Journal on Numerical Analysis.
http://www.jstor.org
DEFLATED DECOMPOSITIONOF
SOLUTIONS OF NEARLY SINGULAR SYSTEMS*
TONY F. CHANt
Abstract.Whensolvingthe linearsystemAx = b,whereA maybe nearlysingularand b is not consistent
with A, one is often interestedin computinga deflatedsolution,i.e., a uniquesolutionto a nearbysingular
but consistentsystem A,xd = b.Keller [14], [15] has considereddeflatedsolutionswith A, corresponding
to settinga smallpivot of the LU-factorizationof A to zero. Stewart[22] proposedan iterativealgorithm
for computinga deflatedsolutionwithA, correspondingto settingthe smallestsingularvalueof A to zero.
Keller'sapproachexplicitlyuses submatricesof the LU-factorswhereasStewart'sapproachis implicitin
that it only involvessolvingsystemswith A. In this paper,we generalizethe conceptof a deflatedsolution
to that of a deflateddecomposition,
whichexpressesthe solution x in termsof xd and the null vectors of
As, We treat such decompositionsin a uniformframeworkthat includesthe approachesof Keller and
Stewartand introducesome new deflatedsolutionsbased on the LU-factorization.Moreover,we prove
some resultsthat relatethe differentkindsof deflatedsolutions.In particular,we provethat the difference
betweenone of the LU-baseddeflatedsolutionsand the SVD-baseddeflatedsolutiontends to zero as A
tendsto exactlysingular.In addition,we presentnoniterativeimplicitalgorithmsforcomputingthe LU-based
decompositions.Numericalresultsverifyingthe accuracyand stabilityof the algorithmsare presented.
Ax = b,
x = Xd + 79U,
where xd is purgedof u, for example by requiring that u Txd= 0. Of course, one would
like to computexd accurately.However,if one takes the obviousapproachand solves
DEFLATED
DECOMPOSITION
OF NEARLY
SINGULAR
SYSTEMS
739
the solution x of (1) in terms of two parts: a part in the null space u of A, and a
deflatedpartpurgedof u. In ? 2, we shallstudythe questionof existenceanduniqueness
of deflatedsolutions and deflateddecompositionsin a general setting.
Dependingon how A, and b are definedin termsof A and b, there may be many
differentdeflatedsolutions and decompositions.One possibilityis to define A, to be
the nearestsingularmatrixto A in the Frobeniusnormand definexd to be the minimal
length least squaressolution to the system A5xd = b. This correspondsto setting the
smallest singularvalue of A to zero in its singularvalue decomposition(SVD) to
obtain A, and taking b to be the orthogonalprojectionof b onto the range space of
A, To avoidcomputingthe SVD of A explicitly,whichis usuallymuchmore expensive
than solving linear systems [2], [11], Stewart [22] gave an implicit algorithmfor
computingxd which only requiresthe ability to solve linear systems with A. In ? 3,
we present this algorithmand show how it can be extended to compute the deflated
decomposition of x. Since the singular vectors required in his algorithm may be
inaccurate,Stewartused a formof iterativeimprovementto refinethe deflatedsolution.
We shall show in ? 6 that if the inverseiterationis arrangedso that these approximate
singularvectorssatisfya simplerelationship,thenno iterativeimprovementis necessary.
In ? 4, we define a class of deflated solutions based on a LU-factorizationof A
with a small pivot, with A, obtainedby changingsome elements of A by amountsof
approximatelythe same size as the smallest pivot in the LU factors. These matrices
A, have the propertythat their left or rightnull vectorscan be determinedaccurately
(to machine precision) by only one back-substitution.In ? 6, we present implicit
iterative algorithmssimilarto Stewart'sfor computingthese deflated solutions. We
analysethe convergenceand stabilityof these algorithmsand show that no iterative
improvementis necessary.
In ? 5, we provesomeresultsthatrelatethe variousdeflatedsolutions.In particular,
we show that the differencebetween one of the LU-based deflatedsolutionsand the
deflatedsolutionbased on the singularvectorstends to zero as A tends to singularity,
whereasfor the other LU-based deflatedsolutionsthis differencetends to something
proportionalto v[Tb where v, is the left null vector of As. These resultsare verified
by numericalexperimentsin ? 7.
We have made no attempt to survey all related work in this area although we
would like to mention the work of Peters and Wilkinson[18]. Throughoutthis paper,
uppercase Latin letters denote matrices,lower case Latin letters denote vectors and
lower case Greek letters denote scalars.We shall use the notation 1111to denote the
Euclideannorm,(u) k to denote the kth componentof the vector u andP"with 11u I= 1
to denote the orthogonalprojector - uuT.
2. Existence and uniqueness. Deflated solutionsof (1) are solutionsto a nearby
singularbut consistentsystemderivedfrom (1). All the deflatedsolutionsthat we are
going to define are solutionsto systems of the form
SAxd = Rb,
(3)
Asxd
(4)
NXd = Xd,
740
TONY F. CHAN
x = xd +(vs
b/K)uY (vsAxd/l)uw.
Proof. To show that x given by (5) satisfies Ax= b, note that Ax=
Axd+(vTb)y-(vTAxd)w=SAxd+
(I-R)b = Rb+ (I-R)b = b.
We call (5) a deflateddecompositionof x. When A is nonsingular,(5) can be
interpretedas a decompositionof the uniquesolution x of (1) into a part spannedby
approximatenull vectorsof A (whichcould have largemagnitude)and a deflatedpart
whose magnitude remains bounded. When A is singular, it appears that (5) is not
DEFLATED
DECOMPOSITION
OF NEARLY
SINGULAR
SYSTEMS
741
Xd
and p =
v[Tb/K.
AuSV= arsv
and
(7)
A TVSV
= (usv.
x=x5U+(vTLb/lf)usv
wherexsvis the uniquesolutionof thefollowingsystem:
(9)
Pv,Axsv = Pvsvb
and
(1 O)
Pusvxsv = Xsv'
742
TONY F. CHAN
4. Deflation using LU-factorizations.We assume that we can compute a LUfactorizationof A (of size n by n) with a small pivot in the kth position of the form
0 Ui U1 Ru
1 0
ufT
e
0
QRAQc=LU=
0
0
V2
LRL
L2J
U2]
L1
vT
(12)
ATvu =aek,
(14)
and
a = 1/11ATekII.
(15)
9. a= 0(E).
choice of w satisfiesvTw= 1. However, this is not the only way to define As. Instead
of the orthogonalprojectorPv, one can use an obliqueprojector.
DEFINITION 10. For any vector u with (u)j $ 0, where 1 j? n, define EiU=
I- ue[/(u)j.
The matrixE'"is singularand has simple left and right null vectors.
LEMMA
(ED
=0.
DEFLATED
DECOMPOSITION
OF NEARLY
SINGULAR
SYSTEMS
743
Since vl, has norm one, one can alwaysfind an index j such that (vl,)7-1=0(1),
independent of E. Now one can define S by choosing v, = vlU and w = ej/(vl,)j, or
equivalentlychoosing S = (Ei)9T. Note that this choice of w also satisfiesVTW= 1.
We now have two possible As's.
12. Ae-(E{
DEFINITION
) TA, A,
PvuA.
Aup= y
(17)
Aue =
Pej.
The following lemma shows that vlu,ue and up are related and that /8 and y are
0(?).
LEMMA
(18)
15.
Ae,(Ei{,)TA
= (E,i )TAEue=AE
=(E
)TAPu =A-(a1(vju)j)eje
(19)
744
TONY F. CHAN
operator is used. For example, Xepecorrespondsto using S = (Ei, )T, R = P,,1 and
k
N=E
N=EUe'
XSRN
is well defined.
18. Thecorresponding
systems(3) and (4) definingthedeflatedsolutions
XSRN have uniquesolutions.
THEOREM
The corresponding deflated decompositions are given in Table 4.1 and follow
directly from Theorem 5.
TABLE 4.1
Deflated decompositions of the solution of Ax = b.
R
Xeep
Xpee+(VT ub/(V1u)j3)Ue
Xpep-
Xepe + (VT
b/))up
E
+[(14
b-
a(Xeep)k)/1(V1u)jP]Ue
xepp
Xppe
(a(xepp)k/(Vlu)j)Ue
+(Lu
b/ y)up
+ (Vlb/y)up
P.
(a(Xpepk/
y)Up
+(VU
(Vlu)j
b/
10) Ue
Xp
+ [(V
ba
Some of the deflated solutions based on the LU-factorization have been defined
by Keller [14], [15], although he did not use the implicit formulation that we use here.
Specifically, he considered the case where the small pivot occurs at the (n, n)th position
of U (i.e., k = n). In that case, the unnormalized vlucan be shown to be (Liv1, -1)T*
If we choose j to be n also, then the unnormalized ue is (Uju1,-l)T*
Using these
null vectors, he explicitly derived the xeee-deflated decomposition in [15]. In [14], he
(a)
Xeee = Xpee
(f)
Xepe = Xppe,
(b)
Puexeee = Xeep,
(g)
PuXe,,e = Xepp,
(c)
E k Xeep = Xeee,
(h) E kXepp =
(d)
PupXpee= Xpep,
(i)
(e)
E UPpe-pe
xpep= (X EXe,j=E
J)
Xepe,
PupXppe =Xppp,
k x
=-x ppeup ppp-,
DEFLATED
DECOMPOSITION
OF NEARLY
SINGULAR
745
SYSTEMS
Proof. These equalities can most easily be proven by showing, using (18) and
(19), that the differencebetween the left- and right-handsides, say d, satisfiesthe
system(3) and (4) with homogeneousright-handsides. Uniquenessthen impliesd = 0.
For example, let d = Ek
eXeep - Xeee.Then it can be shown that d satisfies Aed=0 and
Eked = d. The uniqueness result of Theorem 1 implies that d = 0 which proves (c).
We thus see that there are two familiesof deflatedsolutions,namely (a)-(e) and
(f)-(j), dependingon the choice of the R-operator.
Next we will provethat,if ? and arare both small,then vtuis a good approximation
to Vs,v, Ue is a good approximationto u,v and that up is an even better approximation
to u,v.We shall use the notation o(81, 82) to mean some quantity(scalar,vector or
matrix)the norm of which is 0(max {81, 62}).
LEMMA 20. (a) IIvlu- v,v
(b)
(c)
IIup-usvI
O(E, 02),
0 2),
I=O(E,
IUe-Usv
O(E2
0f
).
yATvluu')up
Mlup = 0.
)usv-M2usv
0.
T= M2 + 02
-yaekU P
USVUT
becausealthough, is O(E), ATejis 0(1) in general. The proof for (a) is similar.
Next we use Lemma 20 to prove the main result of this section.
THEOREM
(b)||xI-sPXsvl_0(0.2,
(b)
|cO(S,E
E)(IVT
b/la + ci| xeepi
||),
2)(IVTb/.+2I
)(|vsv bl cr| + C2||xppp||I),
where cl and c2 are positive constants independent of E and O.. If A is singular, then
where At is the pseudo-inverse of A and q=
xPPP=xSVand xeep-xsv = (vTb)Atq,
vvV-
ejl ( Vvv)p
Proof. We shall start with the nonsingularcase and prove (a) first. The strategy
is to take the xsv-decomposition(8) and orthogonalizethe two parts with respect to
is a unique decompositionof x, by Theorem 6 the
ue. Since the xeep-decomposition
partsthat are orthogonalto ue must be equal to xeep.Thus, we have
Xsv = [xs
USV [USV-
Us)ue] + (Ue
Usv)ue
746
TONY
F. CHAN
-(UTX
U
)Ue]u
e Usv
= (USV+ 0(E,
UeXSV
of ))TX =
O(E, C-)xSV,
and
(21)
USV(U
TUsv)Ue
= Ue+
O(E,?r
2)-(1
0(&,?Oc2))Ue
O( cr2)
? =
O(a),
AUi+i = vi.
ui+1= ai+1/1Iui+1II.
ATIi31+j=ui+,.
II.
1
vj+1= vi+l/ VIi+l
DEFLATED
DECOMPOSITION
OF NEARLY
SINGULAR
SYSTEMS
747
The following theorem states the conditions under which Algorithm II.A will
converge to the desired solution XSRN.
THEOREM 22. Assume that the assumptionsof Theorem1 are satisfiedso that the
systemdefiningXSRN has a uniquesolution.FurtherassumethatN has a one-dimensional
null space Un and thatN2 = N. DefineK - 1- A-'AS, M NKN and # -vAu,. Then
thefollowing statementsare truefor AlgorithmIIA:
(a) All iteratesx satisfyNx = x.
(b) If x converges,then it will convergeto XSRN if , 0O.
< 1.
(c) The iteratex convergesif JIMII
(d) If M = 0, then we can obtainx directlyfrom x = NA-'Rb.
Proof. Since we always start with an x such that Nx = x, Step (3) of Algorithm
IIA guaranteesthat all iteratessatisfythe same constraintsince N2 = N. The iteration
can be written as the followinglinear stationaryiteration:
x = (I-NA-1As)x + NA-'Rb.
(23)
+ NA-1Rb = Mx + NA-1Rb,
x 't=N(I-A-1As)Nx
Table of K, M and .
M
N=E k
N = P. ,
Pu".
Us,,UT
N.A.
uee T/(Ue)k
(E ,)k
Psu
l/u.
.)e
pe k|
Tu)(Up
)k
OO
(U)k
( Up )k
748
TONY
F. CHAN
x = xd+(vI(b-Axd)/Ia)usv.
Xppp, because
only two approximatenull vectors instead of the three needed for the others, and
second, Step (1) of AlgorithmIIA can be simplifiedto S (b -Ax) since S =R.
We now say a few wordsabout the efficiencyof the algorithms.We firstnote that
the P-operatorstake one inner productto applywhereasthe E-operatorstake none.
The noniterative version of Algorithm IIA costs one LU-factorizationof A, one
back-substitutionand the cost of obtainingthe null vectors. It also requiresstorage
for the (2 or 3) approximatenull vectors and, in the case of (25), storing a copy of
A. The iterative version also requires storing a copy of A. Since the factorization
usuallyrequiresmuch more time (storage)than the back-substitutions(solution),the
work (storage)involvedis usuallynot muchmore thn the normalfactor-solveprocess
for (1). If there are more than one right-handsides, the cost of the extra back-solves
for the null vectors can also be amortizedover the total computingtime. For the
LU-based deflations,this extra cost is alwaystwo back-solves,regardlessof whether
A is nearlysingularor not. On the other hand,the cost and convergenceof the inverse
DEFLATED
DECOMPOSITION
OF NEARLY
SINGULAR
SYSTEMS
749
iteration for computing the singularvectors of A are much more sensitive to the
singularityof A (two backsolvesper iteration).Moreover,if the singularvectors are
not accurate, then the computed x,, will not be the minimumlength least squares
solution to Ax = b (althoughit will still be a deflatedsolution in our interpretation),
andthus,in view of the resultsof Theorem21, it is no morespecialthanthe LU-deflated
solutions. Furthermore,an extra copy of A must be stored to obtain the deflated
decomposition.Therefore, in applicationswhere A may occasionallynot be nearly
singularor when it is not knowna prioriwhetherA is nearlysingularor not, we argue
thatthe LU-deflationsare to be preferredbecausethey are noniterative.Suchsituations
arise, for example, in applyingcontinuationmethods to solving nonlinearequations
with Jacobianmatricesthat may become singular[3], [13], [16]. If it is known that A
is very nearly singular,then the SVD-deflationsare probablyto be preferredbecause
they do not depend on the assumptionthat the LU-factorizationof A has a small
pivot and because the inverse iterationwill converge very quicklyin that case. Both
of these implicit deflation techniquesare to be preferred to the explicit deflation
techniquesif the data structuresfor storingthe LU-factorizationof A are complicated,
for example, in band solvers and sparse solvers. An extra advantageof an implicit
algorithmis its modularity-it is independentof how the factorizationand solve is
done and requiresminimummodificationsto the conventionalfactor-solveprocedure.
The most effective approachmay be in the form of a hybridalgorithmwhich uses a
LU-based algorithmas a default and switches to a SVD-based algorithmwhen the
condition
? = 0(C)
is not satisfied.
T)
1) (I-2vv
T)
1 to 10-8.
Note that the smallest singularvalue of A1 and A2 is equal to cr.For A1, the
smallestsingularvalue has multiplicity2 when a = 1. The dimensionn of A is chosen
to be 20.
We will only be concerned with computing xsv, Xeee,
Xeep, Xppe,
750
TONY F. CHAN
TABLE 7.1
e as a function of a = 10-1.
Matrix A1
I
0
1
2
3
4
5
6
7
8
0.1504303E+01
0.1130482E + 00
0.8422814E-01
-0.7856242E-01
-0.8268356E-03
0.9228393E - 02
-0.1059374E-04
-0.7590279E-06
-0.2421439E - 06
computed cr
1
1
1
19
20
20
20
20
20
0.1000000E+01**
0.1000000E + 00
0.1000000E - 01
0.9999483E-03
0.9996640E-04
0.9993521E-05
0.1034953E-05
0.7292485E-07
0.2352214E-07
Matrix A2
I
0
1
2
3
4
5
6
7
8
0.2773407E+00
0.9002221E+00
0.5978710E+00
0.6968676E-01
0.7055007E-02
0.7061549E-03
0.7033348E-04
0.6996095E - 05
0.6780028E-06
20
20
20
20
20
20
20
20
20
computed
0.2233835E-01*
0.7668735E-01*
0.1000000E-01
0.9999985E - 03
0.9999866E-04
0.9996851E-05
0.9955706E-06
0.9902853E-07
0.9597004E-08
position, especiallywhen e is not small. In fact, for the case C = 10-3 for A1, the nth
pivot is not small at all. Note also that, when cris well-isolated,the computed ar'sare
rather accurateand have low absoluteerrors. However, when the smallest singular
value is not well isolated, the inverseiterationis not successfulat all. This is especially
true for A2 because its lowest eigenvaluesare ratherclose to each other.
The next set of tests check the accuracyand stabilityof AlgorithmIIA. For each
choice of S, the right-handsides b are generated,as discussedabove, so that VTb= 1,
and the deflatedsolutions Xeee, Xppe, xsv and xge are computed.The iterativeversionof
AlgorithmIIA was implementedbut one iterationalwaysprovedto be enoughfor all
the deflated solutions, so the results given here are computed with the noniterative
version. The relative errorsare displayedin Figs. 7.1 and 7.2. As expected, xge loses
accuracyas A becomes more singularwhereas the other deflated solutions remain
accurate to within roughly an order of magnitudeof the machine round-off level.
Moreover,one can draw a distinctcorrelationbetween the less accurateLU-deflated
solutionswith relativelylarge values of E (e.g., or= 10-5 for A1). These tests indicate
that the LU-deflatedsolutionsand the noniterativeversionof AlgorithmIIA together
form a ratherrobust procedurefor computingdeflatedsolutions.
The next set of tests is to verify the results of Lemma 20 and Theorem 21. A
large numberof matricesof the form A1 are generatedand the three null vectors usv,
ue and up are computed. In Fig. 7.3, 11
usv- Ue1 and 11
usv-up ll are plotted against 0r. It
DEFLATED
DECOMPOSITION
OF NEARLY
SINGULAR
751
SYSTEMS
Rl
1.OE+OO00
1.0OE-01
1I.E-02
F
1.OE-03
LU I.DE_04
0.000
2.000
m X(EEE)
0 X(PPE)
G X(GE)
4.000
6.000
8.000
+ X(SV)
FiG. 7.1. Relativeerrorsof computedXSRN VS. - =
for
10- A1f.
R2
1.lOE+00
l.OE-01
1. OE-02
I:Y 1.0E-03
Lu 1.OE-04
I.OE-05
LU .OE-07
1 OE-08
1.OE-09 ..I1.I
0.000
2.000
4.000
6.000
m X(EEE)
0 X(PPE)
G X(GE)
+ X(SV)
FIG. 7.2. Relativeerrorsof computedx RN VS. o= I0-' for A,.
8.000
752
TONY
F. CHAN
1.DE+00
1.OE-O1
'
1.OE-02
1.OE-03
Lii
uJ
ui
M1.ODE-05
1il.OE-O91
.ODE-05
0
E
P
-0UeI
6.000
4.000
2.000
0.000
8.000
P111(E) - PHI(SV)
P111(P) - PHI(SV)
LEASTSQUARES
FIT F0R m
FIT F0R 0
LEASTSQUARES
us-upIIvs. a
and
X(EEP)
X(SV) , R1
0~~~~~~~~~~~~~~~~~
1.OE-08 .
l.OE-08
1.ODE-O9B
0.000
2.000
0
O
PH
4.000
6.000
= .0001
P=.001
P=.01
P =.1
I, (P= vTb).
8.000
DEFLATED
DECOMPOSITION
OF NEARLY
X(PPP)
SINGULAR
753
SYSTEMS
X(SV), Rl
1. 0E+00
1.o E-01
1. 0E-02
1.OE-03
Li
1..OE-04
La.
1. OE-05
Lu
LL
C31.OE-06
1.OE-07
1. OE-08
1.0OE-09
2.000
0.000
0
a*
P =.0001
P=.001
P = .01
P = .1
FIG. 7.5.
6.000
?1.000
IIXs.-xvII VS.C=10I,
8.000
(p=vb).
is seen that upis indeed generallycloser to usvthan ue is for the same value of a. The
straightlines shownare the best least squaresfit to the two sets of data{log (difference),
log (o-)} by straightlines. FromLemma20 the exact slopes of the straightlines should
be 1 for Iusv-u, and 2 for usv- upJl.The slopes of the least squaresfit are 1.028
and 1.537 respectively.
Next, right-handsides are generated with values of VTLbvaryingfrom 10-4 to
10-1 for the matrices Al and all the deflated solutions computed with the same
-
Xeep 11and
the differencesdo vary linearlywith vTLbwhen cris small. The results of Lemma20
and Theorem 21 are thus verified.
8. Conclusions. We have offereda rathercomplete analysisof deflatedsolutions
and deflated decompositionsof solutions to nearly singularlinear systems.We have
provided a uniform frameworkthrough which it is possible to relate the various
approachesused in the literatureas well as some new approachesproposedhere. We
have analysed both theoretical questions of existence and uniquenessand practical
questionsof stable computationalalgorithms.The use of implicitalgorithmsresultsin
a modularapproachwhich only accesses the matrixas a linear solver. Therefore,the
algorithmsproposedhere shouldbe easilyextensibleto linearsolversother thandense
Gaussianelimination,for example,sparsedirect solvers,conjugategradientmethods
and multi-gridmethods. Extensionsto higher dimensionalnull spaces should also be
straightforward.
754
TONY
F. CHAN
ACM,Trans.Math.
[2] T. F. CHAN, An improvedalgorithmfor computingthesingularvaluedecomposition,
Software, 8 (1982), pp. 72-83.
, Deflationtechniquesand block-elimination
algorithmsfor solving borderedsingularsystems,
[3]
[4]
Tech. Rep. 226, Computer Science Dept., Yale Univ., New Haven, CT, 1982; SIAM J. Sci. Stat.
Comp., 5 (1984), pp. 121-134.
, On the existenceand computationof LU-factorizationswith small pivots,Tech. Rep. 227.
Computer Science Dept., Yale Univ., New Haven, 1982; Math. Comp., 42 (1984), pp. 535-547.
, Newton-likepseudo-arclength
methodsfor computingsimpleturningpoints,Tech. Rep. 233,
[5]
Computer Science Dept., Yale Univ., New Haven, CT, 1982; SIAM J. Sci. Stat. Comp., 5 (1984),
pp. 135-148.
[6] G. DAHLQUISTAND A. BJORCK,NumericalMethods,Prentice-Hall,EnglewoodCliffs,NJ, 1974.
[7] J. J. DONGARRA, J. R. BUNCH, C. B. MOLER AND G. W. STEWART,LINPACK Users' Guide,
Society for Industrial and Applied Mathematics, Philadelphia, 1979.
[8] R. E. FUNDERLIC AND J. B. MANKIN, Solutionof homogeneoussystemsof linearequationsarising
from compartmental models, SIAM J. Sci. Stat. Comp., 2 (1981), pp. 375-383.
[9] C. B. GARCIA AND W. I. ZANGWILL,PathwaystoSolutions,FixedPointsandEquilibria,
Prentice-Hall,
Englewood Cliffs, NJ, 1981.
[10] P. E. GILL, W. MURRAY AND M. WRIGHT, PracticalOptimization,
AcademicPress,New York, 1981.
[11] G. H. GOLUB
AND
[23] J. H.
WILKINSON,