An Introduction to the Conjugate Gradient Method Without the Agonizing Pain

Edition 1 1 4 Jonathan Richard Shewchuk August 4, 1994

School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213

Abstract
The Conjugate Gradient Method is the most prominent iterative method for solving sparse systems of linear equations. Unfortunately, many textbook treatments of the topic are written with neither illustrations nor intuition, and their victims can be found to this day babbling senselessly in the corners of dusty libraries. For this reason, a deep, geometric understanding of the method has been reserved for the elite brilliant few who have painstakingly decoded the mumblings of their forebears. Nevertheless, the Conjugate Gradient Method is a composite of simple, elegant ideas that almost anyone can understand. Of course, a reader as intelligent as yourself will learn them almost effortlessly. The idea of quadratic forms is introduced and used to derive the methods of Steepest Descent, Conjugate Directions, and Conjugate Gradients. Eigenvectors are explained and used to examine the convergence of the Jacobi Method, Steepest Descent, and Conjugate Gradients. Other topics include preconditioning and the nonlinear Conjugate Gradient Method. I have taken pains to make this article easy to read. Sixty-six illustrations are provided. Dense prose is avoided. Concepts are explained in several different ways. Most equations are coupled with an intuitive interpretation.

Supported in part by the Natural Sciences and Engineering Research Council of Canada under a 1967 Science and Engineering Scholarship and by the National Science Foundation under Grant ASC-9318163. The views and conclusions contained in this document are those of the author and should not be interpreted as representing the ofcial policies, either express or implied, of NSERC, NSF, or the U.S. Government.

Keywords: conjugate gradient method, preconditioning, convergence analysis, agonizing pain

Contents 1. Introduction 2. Notation 3. The Quadratic Form 4. The Method of Steepest Descent 5. Thinking with Eigenvectors and Eigenvalues 5.1. Eigen do it if I try 5.2. Jacobi iterations 5.3. A Concrete Example 6. Convergence Analysis of Steepest Descent 6.1. Instant Results 6.2. General Convergence 7. The Method of Conjugate Directions 7.1. Conjugacy 7.2. Gram-Schmidt Conjugation 7.3. Optimality of the Error Term 8. The Method of Conjugate Gradients 9. Convergence Analysis of Conjugate Gradients 9.1. Picking Perfect Polynomials 9.2. Chebyshev Polynomials 10. Complexity 11. Starting and Stopping 11.1. Starting 11.2. Stopping 12. Preconditioning 13. Conjugate Gradients on the Normal Equations 14. The Nonlinear Conjugate Gradient Method 14.1. Outline of the Nonlinear Conjugate Gradient Method 14.2. General Line Search 14.3. Preconditioning A Notes B Canned Algorithms B1. Steepest Descent B2. Conjugate Gradients B3. Preconditioned Conjugate Gradients i 1 1 2 6 9 9 11 12 13 13 17 21 21 25 26 30 32 33 35 37 38 38 38 39 41 42 42 43 47 48 49 49 50 51

B4. Nonlinear Conjugate Gradients with Newton-Raphson and Fletcher-Reeves B5. Preconditioned Nonlinear Conjugate Gradients with Secant and Polak-Ribi` ere C Ugly Proofs  Minimizes the Quadratic Form C1. The Solution to C2. A Symmetric Matrix Has  Orthogonal Eigenvectors. C3. Optimality of Chebyshev Polynomials D Homework

52 53 54 54 54 55 56

ii

About this Article An electronic copy of this article is available by anonymous FTP to WARP.CS.CMU.EDU (IP address 128.2.209.103) under the lename quake-papers/painless-conjugate-gradient.ps. A PostScript le containing full-page copies of the gures herein, suitable for transparencies, is available as quake-papers/painless-conjugate-gradient-pics.ps. Most of the illustrations were created using Mathematica.

 c 1994 by Jonathan Richard Shewchuk. This article may be freely duplicated and distributed so long as

no consideration is received in return, and this copyright notice remains intact. This guide was created to help students learn Conjugate Gradient Methods as easily as possible. Please mail me (jrs@cs.cmu.edu) comments, corrections, and any intuitions I might have missed; some of these will be incorporated into a second edition. I am particularly interested in hearing about use of this guide for classroom teaching. For those who wish to learn more about iterative methods, I recommend William L. Briggs A Multigrid Tutorial [2], one of the best-written mathematical books I have read. Special thanks to Omar Ghattas, who taught me much of what I know about numerical methods, and provided me with extensive comments on the rst draft of this article. Thanks also to James Epperson, David OHallaron, James Stichnoth, Nick Trefethen, and Daniel Tunkelang for their comments. To help you skip chapters, heres a dependence graph of the sections:
1 Introduction 10 Complexity

2 Notation

13 Normal Equations

3 Quadratic Form

4 Steepest Descent

5 Eigenvectors

7 Conjugate Directions

6 SD Convergence

8 Conjugate Gradients

9 CG Convergence

11 Stop & Start

12 Preconditioning

14 Nonlinear CG

This article is dedicated to every mathematician who uses gures as abundantly as I have herein. iii

iv

1. Introduction When I decided to learn the Conjugate Gradient Method (henceforth, CG), I read four different descriptions, which I shall politely not identify. I understood none of them. Most of them simply wrote down the method, then proved its properties without any intuitive explanation or hint of how anybody might have invented CG in the rst place. This article was born of my frustration, with the wish that future students of CG will learn a rich and elegant algorithm, rather than a confusing mass of equations. CG is the most popular iterative method for solving large systems of linear equations. CG is effective for systems of the form  (1) where is an unknown vector, is a known vector, and is a known, square, symmetric, positive-denite (or positive-indenite) matrix. (Dont worry if youve forgotten what positive-denite means; we shall review it.) These systems arise in many important settings, such as nite difference and nite element methods for solving partial differential equations, structural analysis, circuit analysis, and math homework. Iterative methods like CG are suited for use with sparse matrices. If is dense, your best course of action is probably to factor and solve the equation by backsubstitution. The time spent factoring a dense is roughly equivalent to the time spent solving the system iteratively; and once is factored, the system can be backsolved quickly for multiple values of . Compare this dense matrix with a sparse matrix of larger size that lls the same amount of memory. The triangular factors of a sparse usually have many more nonzero elements than itself. Factoring may be impossible due to limited memory, and will be time-consuming as well; even the backsolving step may be slower than iterative solution. On the other hand, most iterative methods are memory-efcient and run quickly with sparse matrices. I assume that you have taken a rst course in linear algebra, and that you have a solid understanding of matrix multiplication and linear independence, although you probably dont remember what those eigenthingies were all about. From this foundation, I shall build the edice of CG as clearly as I can.

2. Notation Let us begin with a few denitions and notes on notation. With a few exceptions, I shall use capital letters to denote matrices, lower case letters to denote vectors, and Greek letters to denote scalars. is an  matrix, and and are vectors that is,  1 matrices. Written out fully, Equation 1 is

 !

11 21

12 22

. . .

#1

#2

"" 1# %'&&&  1 %'&&&  1 & 2 & 2# 2 . . . .. . . ! .. . ( . ( ! . "" #$# # # (

%'&& &&

0)21 , and represents the scalar sum 3 #465 4 1 4 . Note that The inner product of two vectors is written 1 0)21 718)2 . If and 1 are orthogonal, then 0)21 0. In general, expressions that reduce to 1  1 matrices, 0)21 and 9)@ , are treated as scalar values. such as

Jonathan Richard Shewchuk

2

3
-4 -2

A
2

2
4 6

A
6

-2

CB
2

-4

-6

Figure 1: Sample two-dimensional linear system. The solution lies at the intersection of the lines. A matrix

is positive-denite if, for every nonzero vector ,

) ED

(2)

This may mean little to you, but dont feel bad; its not a very intuitive idea, and its hard to imagine how a matrix that is positive-denite might look differently from one that isnt. We will get a feeling for what positive-deniteness is about when we see how it affects the shape of quadratic forms. Finally, dont forget the important basic identities

F HGIP)QRGS)2T)

and

F HGUIWV 1 RGXV 1 V

1.

3. The Quadratic Form A quadratic form is simply a scalar, quadratic function of a vector with the form

where is a matrix, and are vectors, and d is a scalar constant. I shall show shortly that if  . Y `I and positive-denite, F is minimized by the solution to Throughout this paper, I will demonstrate ideas with the simple sample problem

Ia Y F`

1 2

) bBc )

Aed

(3) is symmetric

Rgf rs

3 2 2 6

hi

pgf B

2 8

hqi

(4)

The system is illustrated in Figure 1. In general, the solution lies at the intersection point B 1. For this problem, the solution is 7ut 2 B 2v ) . The of  hyperplanes, each having dimension  w I Y Y `I is illustratedi in Figure 3. corresponding quadratic form F appears in Figure 2. A contour plot of F

The Quadratic Form

150 100 4 2 50 0
2

I Y F`

0 6 -2 -4 -2 -6 -4 0 4

2
1

Figure 2: Graph of a quadratic form x`y . The minimum point of this surface is the solution to .

-4 -2 2 4 6

-2

-4

-6

Figure 3: Contours of the quadratic form. Each ellipsoidal curve has constant x`y6 .

Jonathan Richard Shewchuk

2

-4 -2 -2 2 4 6

-4

-6

-8

Figure 4: Gradient xy6 of the quadratic form. For every , the gradient points in the direction of steepest
increase of

x`y6 , and is orthogonal to the contour lines. Y wI Because is positive-denite, the surface dened by F
The gradient of a quadratic form is dened to be

is shaped like a paraboloid bowl. (Ill have more

to say about this in a moment.)

 " Y F `I % && Y `I & Y F wIa " F & (5) ! Y... wI ( F I Y ` The gradient is a vector eld that, for a given point , points in the direction of greatest increase of F . Figure 4 illustrates the gradient vectors for Equation 3 with the constants given in Equation 4. At the bottom Y `I Y wI equal to zero. of the paraboloid bowl, the gradient is zero. One can minimize F by setting F
1 2

With a little bit of tedious math, one can apply Equation 5 to Equation 3, and derive

If

Y F wI
is symmetric, this equation reduces to

1 2

) A

1 XBc 2

(6)

Y F wIRdBc

(7)

Setting the gradient to zero, we obtain Equation 1, the linear system we wish to solve. Therefore, the 7e is a critical point of Y F wI . If is positive-denite as well as symmetric, then this solution to

The Quadratic Form

(a) (b)

I Y F`
2

(c)

I Y F`
2

(d)

I Y F`
2

I Y F`
2

Figure 5: (a) Quadratic form for a positive-denite matrix. (b) For a negative-denite matrix. (c) For a
singular (and positive-indenite) matrix. A line that runs through the bottom of the valley is the set of solutions. (d) For an indenite matrix. Because the solution is a saddle point, Steepest Descent and CG will not work. In three dimensions or higher, a singular matrix can also have a saddle.

solution is a minimum of F , so can be solved by nding an that minimizes F . (If is not 1 T) symmetric, then Equation 6 hints that CG will nd a solution to the system 2 F A qIgh . Note that T ) q I 1 is symmetric.) A 2F Why do symmetric positive-denite matrices have this nice property? Consider the relationship between  V 1 . From Equation 3 one can show (Appendix C1) at some arbitrary point i and at the solution point that if is symmetric (be it positive-denite or not),

Y wI

fr

Y `I

If

paraboloid lies, but do not affect the paraboloids shape.

Y wI The fact that F is a paraboloid is our best intuition of what it means for a matrix to be positive-denite. If is not positive-denite, there are several other possibilities. could be negative-denite the result of negating a positive-denite matrix (see Figure 2, but hold it upside-down). might be singular, in which Y case no solution is unique; the set of solutions is a line or hyperplane having a uniform value for . If is none of the above, then is a saddle point, and techniques like Steepest Descent and CG will likely fail. Figure 5 demonstrates the possibilities. The values of and d determine where the minimum point of the

is positive-denite as well, then by Inequality 2, the latter term is positive for all il Y is a global minimum of .

I Y F wI Y Fji a A

1 Fji 2

B`I ) Fji BkwI

(8) . It follows that

Why go to the trouble of converting the linear system into a tougher-looking problem? The methods under study Steepest Descent and CG were developed and are intuitively understood in terms of minimization problems like Figure 2, not in terms of intersecting hyperplanes such as Figure 1.

6 Jonathan Richard Shewchuk 4. The Method of Steepest Descent In the method of Steepest Descent, we start at an arbitrary point 0n and slide down to the bottom of the 2m 2m "" until we are satised that we are close enough to the paraboloid. We take a series of steps 1 n i 2n i solution . When we take a step, we choose the direction in which decreases most quickly, which is the direction B Y F m 4 IoB m 4 . Y m I opposite F 4 n . According to Equation 7, this direction is n n

2m

4n Allow me to introduce a few denitions, which you should memorize. The error p 4 n is a  s t B m m 4 4 vector that indicates how far we are from the solution. The residual r n indicates how far we n m u  B m 4 4 are from the correct value of . It is easy to see that r n as p n , and you should vB Yof F the m 4 residual I , and you m think being the error transformed by into the same space as . More importantly, r 4 n n should also think of the residual as the direction of steepest descent. For nonlinear problems, discussed in Section 14, only the latter denition applies. So remember, whenever you read residual, think direction of steepest descent.
Suppose we start at 0 n 2 2v . Our rst step, along the direction of steepest descent, will fall i somewhere on the solid line in Figure 6(a). In other words, we will choose a point

m q m Be

2m wtxB

B )

m R m m 1n 0 n Aey r 0 n Y

(9)

The question is, how big a step should we take?

A line search is a procedure that chooses y to minimize along a line. Figure 6(b) illustrates this task: we are restricted to choosing a point on the intersection of the vertical plane and the paraboloid. Figure 6(c) is the parabola dened by the intersection of these surfaces. What is the value of y at the base of the parabola? From basic calculus, y minimizes when the directional derivative z F 1n is equal to zero. By the Y m IT Y F m I ) z m Y F m I ) r m . Setting this expression z|{ chain rule, z F 1n to zero, we nd that y 1n Y 1 nI 1n 0n m 2 m z|{ so that r 0n and z|F { 1n are orthogonal (see Figure 6(d)). should be chosen There is an intuitive reason why we should expect these vectors to be orthogonal at the minimum. Figure 7 shows the gradient vectors at various points along the search line. The slope of the parabola (Figure 6(c)) at any point is equal to the magnitude of the projection of the gradient onto the line (Figure 7, Y Y dotted arrows). These projections represent the rate of increase of as one traverses the search line. is minimized where the projection is zero where the gradient is orthogonal to the search line. To determine y , note that

Y 2m I

m IaCB r m , and we have Y F@ 1n 1n r )m 1 n r m 0 n F sB m 1n I ) r m 0n F oB F m 0n Aey r m 0n I}I ) r m 0n F sB m 0n I ) r m 0n B y F r m 0n I ) r m 0n F sB m 0n I ) r m 0n r )m 0 n r m 0 n y

0 0 0 0

m I) m y F ) r 0n r 0n m m I y r 0n F r 0n r )m 0 n r m 0n r )m 0n r m 0n

The Method of Steepest Descent

2

7 (b)

(a)

4 2 -4

-2 -2 -4

p m 0n

4 6

2.5 0
2

150 100 50 0 -2.5 0

I Y Fw

-2.5 -5

2.5 5
1

m I Y F m4 n Aey r 4 n
140 120 100 80 60 40 20

-6

(c)

(d)

4 2 -4 -2 -2

m
-4

0.2 0.4 0.6

-6

Figure 6: The method of Steepest Descent. (a) Starting at ~j 2  2 , take a step in the direction of steepest
descent of . (b) Find the point on the intersection of these two surfaces that minimizes . (c) This parabola is the intersection of surfaces. The bottommost point is our target. (d) The gradient at the bottommost point is orthogonal to the gradient of the previous step.

-2 -1 1 2 3 -1

-2

-3

Figure 7: The gradient x is shown at several locations along the search line (solid arrows). Each gradients
projection onto the line is also shown (dotted arrows). The gradient vectors represent the direction of steepest increase of , and the projections represent the rate of increase as one traverses the search line. On the search line, is minimized where the gradient is orthogonal to the search line.

Jonathan Richard Shewchuk

2

-4 -2 2 4 6

-2

-4

-6

Figure 8: Here, the method of Steepest Descent starts at Putting it all together, the method of Steepest Descent is:

~j 2  2

and converges at 2

~ W 2

r m 4 n Bc@m 4 n i ) m m 4 r ) m 4 n r 4 m n y n r m4 r 4n i @m 46 7@m 4 n m 4 r n Aey n 1n

(10) (11)

m4 n

(12)

The example is run until it converges in Figure 8. Note the zigzag path, which appears because each gradient is orthogonal to the previous gradient. The algorithm, as written above, requires two matrix-vector multiplications per iteration. The computational cost of Steepest Descent is dominated by matrix-vector products; fortunately, one can be eliminated. B and adding , we have By premultiplying both sides of Equation 12 by

r m 64 1n r m 4 n B y m 4 n r m 4 n (13) m Although Equation 10 is still needed to compute r 0 n , Equation 13 can be used for every iteration thereafter. The product r , which occurs in both Equations 11 and 13, need only be computed once. The disadvantage of using this recurrence is that the sequence dened by Equation 13 is generated without any feedback from m m the value of 4 n , so that accumulation of oating point roundoff error may cause 4 n to converge to some point near . This effect can be avoided by periodically using Equation 10 to recompute the correct residual.
Before analyzing the convergence of Steepest Descent, I must digress to ensure that you have a solid understanding of eigenvectors.

Thinking with Eigenvectors and Eigenvalues 5. Thinking with Eigenvectors and Eigenvalues

After my one course in linear algebra, I knew eigenvectors and eigenvalues like the back of my head. If your instructor was anything like mine, you recall solving problems involving eigendoohickeys, but you never really understood them. Unfortunately, without an intuitive grasp of them, CG wont make sense either. If youre already eigentalented, feel free to skip this section. Eigenvectors are used primarily as an analysis tool; Steepest Descent and CG do not calculate the value of any eigenvectors as part of the algorithm1 .

5.1. Eigen do it if I try An eigenvector of a matrix is a nonzero vector that does not rotate when is applied to it (except perhaps to point in precisely the opposite direction). may change length or reverse its direction, but it G 0 . The value is wont turn sideways. In other words, there is some scalar constant such that G an eigenvalue of . For any constant y , the vector y is also an eigenvector with eigenvalue , because G F I G . In other words, if you scale an eigenvector, its still an eigenvector. y y y Why should you care? Iterative methods often depend on applying to a vector over and over 4 again. 4 G G When is repeatedly applied to an eigenvector , one of two things can happen. If 1, then 4 D 1, then G will grow to innity (Figure 10). Each will vanish as approaches innity (Figure 9). If G time is applied, the vector grows or shrinks according to the value of .

B2 v v B3 v

Bv
Figure 9:
to zero.

is an eigenvector of

with a corresponding eigenvalue of

 0 5. As

increases,

converges

Bv

B2 v

B3 v

Figure 10: Here, has a corresponding eigenvalue of 2. As increases,  diverges to innity.

1 However, there are practical applications for eigenvectors. The eigenvectors of the stiffness matrix associated with a discretized structure of uniform density represent the natural modes of vibration of the structure being studied. For instance, the eigenvectors of the stiffness matrix associated with a one-dimensional uniformly-spaced mesh are sine waves, and to express vibrations as a linear combination of these eigenvectors is equivalent to performing a discrete Fourier transform.

Jonathan Richard Shewchuk is symmetric (and often if it is not), then there exists a set of  linearly independent eigenvectors G of , denoted 1 2 "" # . This set is not unique, because each eigenvector can be scaled by an arbitrary i i i nonzero constant. Each eigenvector has a corresponding eigenvalue, denoted 1 2 "" # . These are i i i uniquely dened for a given matrix. The eigenvalues may or may not be equal to each other; for instance, the eigenvalues of the identity matrix are all one, and every nonzero vector is an eigenvector of .
10 What if is applied to a vector that is not an eigenvector? A very important skill in understanding linear algebra the skill this section is written to teach is to think of a vector as a sum of other vectors 4 forms a basis for # (because a whose behavior is understood. Consider that the set of eigenvectors G symmetric has  eigenvectors that are linearly independent). Any  -dimensional vector can be expressed as a linear combination of these eigenvectors, and because matrix-vector multiplication is distributive, one G can examine the effect of on each eigenvector separately. In Figure 11, a vector is illustrated as a sum of two eigenvectors 1 and 2 . Applying to is G equivalent the 4 eigenvectors, and summing the result. On repeated application, we G 4 RG 4 to applying G 4 4 to G 4 have . If the magnitudes of all the eigenvalues are smaller than one, will 1A 2 1 1A 2 2 G converge to zero, because the eigenvectors that compose converge to zero when is repeatedly applied. If one of the eigenvalues has magnitude greater than one, will diverge to innity. This is why numerical analysts attach importance to the spectral radius of a matrix:

G If

4 4 is an eigenvalue of G . i  G I If we want to converge to zero quickly, F should be less than one, and preferably as small as possible.
max

G Ia F

v 2 x

B3 x Bx B2 x

(solid arrow) can be expressed as a linear combination of eigenvectors (dashed arrows), whose associated eigenvalues are 1 0 7 and 2 2. The effect of repeatedly applying to is best understood by examining the effect of on each eigenvector. When is repeatedly applied, one eigenvector converges to zero while the other diverges; hence, also diverges.

Figure 11: The vector

e

Its important to mention that there is a family of nonsymmetric matrices that do not have a full set of independent eigenvectors. These matrices are known as defective, a name that betrays the well-deserved hostility they have earned from frustrated linear algebraists. The details are too complicated to describe in this article, but the behavior of defective matrices can be analyzed in terms of generalized eigenvectors and G 4 converges generalized eigenvalues. The rule that to zero if and only if all the generalized eigenvalues have magnitude smaller than one still holds, but is harder to prove.

G ) G ) )@G is positive (for nonzero By the denition of positive-denite, the

Heres a useful fact: the eigenvalues of a positive-denite matrix are all positive. This fact can be proven from the denition of eigenvalue:

). Hence, must be positive also.

Thinking with Eigenvectors and Eigenvalues

5.2. Jacobi iterations

11

Of course, a procedure that always converges to zero isnt going to help you attract friends. Consider a more Qu . The matrix is split into two parts: , whose useful procedure: the Jacobi Method for solving diagonal elements are identical to those of , and whose off-diagonal elements are zero; and , whose  . diagonal elements are zero, and whose off-diagonal elements are identical to those of . Thus, A We derive the Jacobi Method:

Given a starting vector 0 n , this formula generates a sequence of vectors. Our hope is that each successive vector will be closer to the solution than the last. is called a stationary point of Equation 15, because if 2m 4 7 , then 2m 4 will also equal . 1n n Now, this derivation may seem quite arbitrary to you, and youre right. We could have formed any number of identities for instead of Equation 14. In fact, simply by splitting differently that is, by choosing a different and we could have derived the Gauss-Seidel method, or the method of G Successive Over-Relaxation (SOR). Our hope is that we have chosen a splitting for which has a small spectral radius. Here, I chose the Jacobi splitting arbitrarily for simplicity.

Because is diagonal, it is easy to invert. This identity can be converted into an iterative method by forming the recurrence 2m 4 GS@m 4 (15) n A 1n

B B V 1A V 1 A G GCB V 1 where Ae i i

V 1

(14)

Suppose we start with some arbitrary vector to the result. What does each iteration do?

n . For each iteration, we apply

to this vector, then add

outcome!

G m 4 I G F n Ae m p 4 A G A n G p m4n e A  A G p m4 (by Equation 14) A n i p m 4 1n G p m 4 n (16) 2m Each iteration does not affect the correct part of 4 n (because is a stationary point); but each iteration GI 1, then the error term p m 4 will does affect the error term. It is apparent from Equation 16 that if F n 2m converge to zero as approaches innity. Hence, the initial vector 0 n has no effect on the inevitable @m

Again, apply the principle of thinking of a vector as a sum of other, well-understood vectors. Express m m each iterate 4 n as the sum of the exact solution and the error term p 4 n . Then, Equation 15 becomes

m 4 1n

Of course, the choice of 0 n does affect the number of iterations required to converge to within GI a given tolerance. However, its effect is less important than that of the spectral radius F , which G determines the speed of convergence. Suppose that is the eigenvector of with the largest eigenvalue GI ). If the initial error p m , expressed as a linear combination of eigenvectors, includes a (so that F 0n component in the direction of " , this component will be the slowest to converge.

12

Jonathan Richard Shewchuk

2
4

-4 -2 2 7 -2 2 4 6

-4

-6

are directed along the axes of the paraboloid dened by the quadratic form . Each eigenvector is labeled with its associated eigenvalue. Each eigenvalue is proportional to the steepness of the corresponding slope.

Figure 12: The eigenvectors of

x`y

is not generally symmetric (even if is), and may even be defective. However, the rate of convergence GUI of the Jacobi Method depends largely on F , which depends on . Unfortunately, Jacobi does not converge for every , or even for every positive-denite .

5.3. A Concrete Example To demonstrate these ideas, I shall solve the system specied by Equation 4. First, we need a method of nding eigenvalues and eigenvectors. By denition, for any eigenvector with eigenvalue ,

Eigenvectors are nonzero, so 0

0 9 B q I F9 0 BcqI

must be singular. Then, det F9 0

The determinant of 0 is called the characteristic polynomial. It is a degree  polynomial in whose roots are the set of eigenvalues. The characteristic polynomial of (from Equation 4) is

F B 7I F B 2I i 7, and the eigenvalues are 7 and 2. To nd the eigenvector associated with B 2 f B q I g f 4 B 2 1 h 1h 0 F0 2 4 1 B 2 2 0

14

B 3 B 2 B f B 2 B 6 h 2 9 A det

Convergence Analysis of Steepest Descent 13 t x t B ) ) Any solution to this equation is an eigenvector; say, 1 2v . By the same method, we nd that 2 1v i i is an eigenvector corresponding to the eigenvalue 2. In Figure 12, we see that these eigenvectors coincide with the axes of the familiar ellipsoid, and that a larger eigenvalue corresponds to a steeper slope. (Negative Y eigenvalues indicate that decreases along the axis, as in Figures 5(b) and 5(d).)
Now, lets see the Jacobi Method in action. Using the constants specied by Equation 4, we have

m 64 1n G tj

Bf

1 3

f B0 1
3

m4 f h h nA B 2 m f 2 4 n A B 34 3 0 h 3 h
0
1 6

0 2 2 0

1 3

0
1 6

f B2 8 h h )

The eigenvectors of are 2 1v with eigenvalue 2 3, and 2 1v with eigenvalue 2 3. i i These are graphed in Figure 13(a); note that they do not coincide with the eigenvectors of , and are not related to the axes of the paraboloid. Figure 13(b) shows the convergence of the Jacobi method. The mysterious path the algorithm takes can be understood by watching the eigenvector components of each successive error term (Figures 13(c), (d), and (e)). Figure 13(f) plots the eigenvector components as arrowheads. These are converging normally at the rate dened by their eigenvalues, as in Figure 11. I hope that this section has convinced you that eigenvectors are useful tools, and not just bizarre torture devices inicted on you by your professors for the pleasure of watching you suffer (although the latter is a nice fringe benet).

txB

6. Convergence Analysis of Steepest Descent 6.1. Instant Results To understand the convergence of Steepest Descent, lets rst consider the case where p 4 n is an eigenvector m uB p m 4 CB 9p m 4 is also an eigenvector. Equation 12 gives with eigenvalue 0 . Then, the residual r 4 n n n

p m 46 1n

r )m 4 n r m 4 n m 4 m 4 p n A r )m 4 r m 4 r n n m n ) m r 4r 4 p m 4 n A r )mn 4 r mn 4 F B 9p m 4 n I n n 0

Figure 14 demonstrates why it takes only one step to converge to the exact solution. The point 4 n lies on one of the axes of the ellipsoid, and so the residual points directly to the center of the ellipsoid. Choosing m V y 4 n 1 gives us instant convergence. For a more general analysis, we must express p 4 n as a linear combination of eigenvectors, and we shall furthermore require these eigenvectors to be orthonormal. It is proven in Appendix C2 that if is

14

4 2

Jonathan Richard Shewchuk

2

(a)

4 2

(b)

-4 -2 2 4 6 -2

-4 -2 2 4 6

0 47

B 0 47

-2

-4

-4

-6

-6

(c)

(d)

-4 -2 2 4 6

-4

-2

p m 0n
-4

p m 1n

4 6

-2

-4

-6

-6

(e)

(f)

-4 -2 2 4 6 -2

-4 -2 2 4 6 -2

p m 2n

-4

-4

-6

-6

Figure 13: Convergence of the Jacobi Method. (a) The eigenvectors of are shown with their corresponding eigenvalues. Unlike the eigenvectors of , these eigenvectors are not the axes of the paraboloid. (b) The Jacobi Method starts at 2 2 and converges at 2 2 . (c, d, e) The error vectors 0 , 1 , 2 (solid arrows) and their eigenvector components (dashed arrows). (f) Arrowheads represent the eigenvector components of the rst four error vectors. Each eigenvector component of the error is converging to zero at the expected rate based on its eigenvalue.

~' W

~ 

Convergence Analysis of Steepest Descent

2

15

-4 -2 2 4 6 -2

-4

-6

Figure 14: Steepest Descent converges to the exact solution on the rst iteration if the error term is an
eigenvector.

symmetric, there exists a set of  orthogonal eigenvectors of . As we can scale eigenvectors arbitrarily, let us choose so that each eigenvector is of unit length. This choice gives us the useful property that

1 i 0

 l i 

(17)

Express the error term as a linear combination of eigenvectors

# (18) p m 4 n 5 " i 1 m where is the length of each component of p 4 n . From Equations 17 and 18 we have the following identities: (19) r m 4 n B p m 4 n CB i p m 4 2 p )m 4 p m 4 2 (20) n n n i p ) m 4 n p m 4 n F " ) I F $ I 2 (21) i r m 4 2 r )m 4 r m 4 2 2 (22) n n n i 2 3 (23) r )m 4 n r m 4 n

16

Jonathan Richard Shewchuk

2

-4 -2 2 4 6

-2

-4

Figure 15: Steepest Descent converges to the exact solution on the rst iteration if the eigenvalues are all
equal.

Equation 19 shows that r 4 n too can be expressed as a sum of eigenvector components, and the length of B these components are 8 . Equations 20 and 22 are just Pythagoras Law. Now we can proceed with the analysis. Equation 12 gives

p m 46 1n m

We saw in the last example that, if p 4 n has only one eigenvector component, then convergence is m V 1. Now lets examine the case where p m 4 is arbitrary, but all the achieved in one step by choosing y 4 n n eigenvectors have a common eigenvalue . Equation 24 becomes

r )m 4 n r m 4 n m 4 m 4 p n A r )m 4 r m 4 r n n n 3 2 2 m 4 m 4 p n A 3 2 3 r n

(24)

p m 64 1n

2 2 p m 4 n A 3 33 2 F B 9p m 4 n I 0

Figure 15 demonstrates why, once again, there is instant convergence. Because all the eigenvalues are equal, the ellipsoid is spherical; hence, no matter what point we start at, the residual must point to the center m V 1. of the sphere. As before, choose y 4 n

Convergence Analysis of Steepest Descent

2

17

-6 -4 -2 2 -2

Figure 16: The energy norm of these two vectors is equal. However, if there are several unequal, nonzero eigenvalues, then no choice of y 4 n will eliminate all the eigenvector components, and our choice becomes a sort of V compromise. In fact, the fraction in Equation 24 is best thought of as a weighted average of the values of 1 . The weights 2 ensure that longer components m the shorter components of p m 4 n of p 4 n are given precedence. As a result, on any given iteration, some of might actually increase in length (though never for long). For this reason, the methods of Steepest Descent and Conjugate Gradients are called roughers. By contrast, the Jacobi Method is a smoother, because every eigenvector component is reduced on every iteration. Steepest Descent and Conjugate Gradients are not smoothers, although they are often erroneously identied as such in the mathematical literature.

6.2. General Convergence To the convergence of Steepest Descent in the general case, we shall dene the energy norm p Wbound Fp ) p I 1 2 (see Figure 16). This norm is easier to work with than the Euclidean norm, and is in m W is equivalent to some sense a more natural norm; examination of Equation 8 shows that minimizing p 4 n

Jonathan Richard Shewchuk I m 4 F n . With this norm, we have p m 46 2 p ) m 46 p m 46 1n 1n Fp ) m 4 1n m 4 r )m 4 I Fp m 4 m m I (by Equation 12) Aty n n Aey 4 n r 4 n n n p ) m 4 p m 4 2 m 4 r )m 4 p m 4 m 4 r )m 4 r m 4 2 n n A y n n n Aey n n n (by symmetry of ) )m m r )m 4 n r m 4 n 2 )m 4 m 4 p m4 2 2 r ) 4 n r 4 m n B r )m 4 r m 4 n A r m 4 r 4 n n ng Aw r )m 4 r m 4 n" r n r n n m I n ) m 4 2 4 p m4 2 B Fr ) n r nm n r m4n r 4n )m m I p m4 2 1 B ) Fr m 4 n I r 4 n) 2 m I n Fr m 4 n r 4 n Fp m 4 n p 4 n F 3 2 2 I 2 p m4 2 1B (by Identities 21, 22, 23) n F 3 2 3 I F 3 2 I B 2 2I 2 F 3 p m4 2 @ 2 2 1 (25) I n 3 2 i F 3 F 3 2 I The analysis depends on nding an upper bound for . To demonstrate how the weights and eigenvalues 2. Assume that . The spectral condition number affect convergence, I shall derive a result for  1 2 of is dened to be 1 2 1. The slope of p m 4 n (relative to the coordinate system dened by the . We have eigenvectors), which depends on the starting point, is denoted 2 1 I 2 2 2 2 2 F 1 2 1B 1 A 2 IF2 I 2 2 2 3 2 3 F 1 1 A 2 2 1 1 A 2 2 1 B F 2 AI 2 I 2 I (26) F A 2 F 3 A 2 The value of , which determines the rate of convergence of Steepest Descent, is graphed as a function m of and in Figure 17. The graph conrms my two examples. If p 0 n is an eigenvector, then the slope is zero (or innite); we see from the graph that is zero, so convergence is instant. If the eigenvalues are equal, then the condition number is one; again, we see that is zero.
18 Y minimizing Figure 18 illustrates examples from near each of the four corners of Figure 17. These quadratic forms are graphed in the coordinate system dened by their eigenvectors. Figures 18(a) and 18(b) are examples with a large condition number. Steepest Descent can converge quickly if a fortunate starting point is chosen (Figure 18(a)), but is usually at its worst when is large (Figure 18(b)). The latter gure gives us our best Y `I intuition for why a large condition number can be bad: F forms a trough, and Steepest Descent bounces back and forth between the sides of the trough while making little progress along its length. In Figures 18(c) and 18(d), the condition number is small, so the quadratic form is nearly spherical, and convergence is quick regardless of the starting point. Holding constant (because is xed), a little basic calculus reveals that Equation 26 is maximized v . In Figure 17, one can see a faint ridge dened by this line. Figure 19 plots worst-case when . An starting points for our sample matrix . These starting points fall on the lines dened by 2 1

Convergence Analysis of Steepest Descent

19

0.8 0.6 100 80 0.4 0.2 60 0 20 40 20 5 10 10 15

Figure 17: Convergence of Steepest Descent as a function of

number of .

c

). Convergence is fast when

or

(the slope of ) and (the condition are small. For a xed matrix, convergence is worst when

2
6 4 2 -4 -2 -2 -4 6 4 2 -4 -2 -2 -4 2 2

(a)

2
6 4 2 4

(b)

-4

-2 -2 -4

(c)

(d)

6 4 2 4

-4

-2 -2 -4

Figure 18: These four examples represent points near the corresponding four corners of the graph in Figure 17. (a) Large , small . (b) An example of poor convergence. and are both large. (c) Small and . (d) Small , large .

20

Jonathan Richard Shewchuk

2

4
0

n
2

-4 -2 2 4 6 -2

-4

-6

Figure 19: Solid lines represent the starting points that give the worst convergence for Steepest Descent.
Dashed lines represent steps toward convergence. If the rst iteration starts from a worst-case point, so do all succeeding iterations. Each step taken intersects the paraboloid axes (gray arrows) at precisely a 45 angle. Here, 3 5.

upper bound for

(corresponding to the worst-case starting points) is found by setting

2:

5A 4A B 2 4 A 3 5 AB 2 I 4 A 3 F 1I 2 F B A 1 2 1 A 1
5

4 2

4 3

(27)

Inequality 27 is plotted in Figure 20. The more ill-conditioned the matrix (that is, the larger its condition number ), the slower the convergence of Steepest Descent. In Section 9.2, it is proven that Equation 27 is D 2, if the condition number of a symmetric, positive-denite matrix is dened to be also valid for 

0o 4 # i

the ratio of the largest to smallest eigenvalue. The convergence results for Steepest Descent are

B 1 4 m W p m4 W v n A 1 p 0n i and

(28)

The Method of Conjugate Directions

21

0.8

0.6

0.4

0.2

20

40

60

80

100

Figure 20: Convergence of Steepest Descent (per iteration) worsens as the condition number of the matrix
increases.

m 4 IB Y F wI Y F@ I Y F m n IB Y F w 0n

p )m 4 n p m 4 n p ) m 0n p m 0n 4 B 1 2 A 1
1 2 1 2

(by Equation 8)

7. The Method of Conjugate Directions 7.1. Conjugacy Steepest Descent often nds itself taking steps in the same direction as earlier steps (see Figure 8). Wouldnt it be better if, every time we took a step, we got it right the rst time? Heres an idea: lets pick a set of m m m orthogonal search directions 0 n 1 n "" # V 1 n . In each search direction, well take exactly one step, i i i and that step will be just the right length to line up evenly with . After  steps, well be done. Figure 21 illustrates this idea, using the coordinate axes as search directions. The rst (horizontal) step m leads to the correct 1 -coordinate; the second (vertical) step will hit home. Notice that p 1n is orthogonal to m 0n . In general, for each step we choose a point

2m 46 R 2m 4 m m (29) n Aey 4 n 4 n 1n m m m To nd the value of y 4 n , use the fact that p 4 1 n should be orthogonal to 4 n , so that we need never step in

22

Jonathan Richard Shewchuk

2

-4 -2 2 4 6

-2

@m

m 0n
-4

p m 1n m

-6

Figure 21: The Method of Orthogonal Directions. Unfortunately, this method only works if you already know
the answer.

the direction of

m4 n

again. Using this condition, we have

) m 4 n p m 46 1n ) m 4 n Fp m 4 n Ay m 4 n m 4 n I m y 4n

0 0

) B )m 4 n p m4n

(by Equation 29)

m4 m4n n

(30)

Unfortunately, we havent accomplished anything, because we cant compute y m and if we knew p 4 n , the problem would already be solved.

m4 n

without knowing p

m4

n;

m The solution is to make the search directions -orthogonal instead of orthogonal. Two vectors 4 n and m n are -orthogonal, or conjugate, if )m 4 n m n 0 Figure 22(a) shows what -orthogonal vectors look like. Imagine if this article were printed on bubble
gum, and you grabbed Figure 22(a) by the ends and stretched it until the ellipses appeared circular. The vectors would then appear orthogonal, as in Figure 22(b). Our new requirement is that p 46 1n be -orthogonal to 4 n (see Figure 23(a)). Not coincidentally, this m orthogonality condition is equivalent to nding the minimum point along the search direction 4 n , as in

The Method of Conjugate Directions

2

23
2

-4 -2 2 4 -2

-4 -2 2 4 -2

-4

-4

(a) Figure 22: These pairs of vectors are -orthogonal

(b)

W because these pairs of vectors are orthogonal.

Steepest Descent. To see this, set the directional derivative to zero:

Y F 2m 46 I 1n y Y F m 46 I ) m 46 1n 1n B r )m y 46 m 4 n 1n ) m m 6 4 4 n p 1n ) m B )m 4 n p m 4 n m4n 4n )m 4 n r m 4 n )m 4 n m 4 n

0 0 0 0

Following the derivation of Equation 30, here is the expression for y -orthogonal:

m4 n

when the search directions are

m y 4n

(31) (32)

Unlike Equation 30, we can calculate this expression. Note that if the search vector were the residual, this formula would be identical to the formula used by Steepest Descent.

24

Jonathan Richard Shewchuk

2

-4

m -2 p m 1n 1n m 0n -2 m 0n
-4

2 4 6

-4 -2 -2 p m 0n -4 2 4 6

-6

-6

(a)

(b)

Figure 23: The method of Conjugate Directions converges in steps. (a) The rst step is taken along some direction 0 . The minimum point 1 is chosen by the constraint that 1 must be -orthogonal to 0 . (b) The initial error 0 can be expressed as a sum of -orthogonal components (gray arrows). Each step of Conjugate Directions eliminates one of these components.

To prove that this procedure really does compute combination of search directions; namely,

in

steps, express the error term as a linear

The values of can be found by a mathematical trick. Because the search directions are -orthogonal, it is possible to eliminate all the values but one from Expression 33 by premultiplying the expression by ) m n :

# V 1 m m p 0n 5 n 0

(33)

)m n )m n

p m 0n p m 0n m n

m )m m n n n m )m m (by -orthogonality of vectors) n n n ) m p m 0n n m ) m n n ) m n Fp m 0n A 3 46 5 V 01 y m 4 n m 4 n I (by -orthogonality of m ) m n n ) m m n p n (By Equation 29). )m n m n m B m

vectors) (34)

4 n . This fact gives us a new way to look at the error By Equations 31 and 34, we nd that y 4 n term. As the following equation shows, the process of building up component by component can also be

The Method of Conjugate Directions viewed as a process of cutting down the error term component by component (see Figure 23(b)).

25

p m4n

4 V m p 0n A 5 V # 1 m 5 0 n # V 1 m 54 n

m m y n n 0 4V m B 1 m m n 5 n n 0 m n
(35)

After  iterations, every component is cut away, and p 7.2. Gram-Schmidt Conjugation

m #n

0; the proof is complete.

All that is needed now is a set of -orthogonal search directions to generate them, called a conjugate Gram-Schmidt process.

m 4 n . Fortunately, there is a simple way

Suppose we have a set of  linearly independent vectors 0 1 "" # V 1 . The coordinate axes will i i i m 4 do in a pinch, although more intelligent choices are possible. To construct 4 n , take and subtract out any components that are not -orthogonal to the previous vectors (see Figure 24). In other words, set m 0n 0, and for D 0, set 4

where the

)m 4 n m n
0

are dened for

V 1 4 m m 4 4 n A 5 n i 0 )m m n n m n i D

(36)

. To nd their values, use the same trick used to nd

1 )4 m n A 5 4 0) ) m 4 n A 4 m n B ) )4 m m n m n n

4V

(by -orthogonality of vectors) (37)

The difculty with using Gram-Schmidt conjugation in the method of Conjugate Directions is that all I the old search vectors must be kept in memory to construct each new one, and furthermore F 3 operations

u 0 u1

(0)

d (0)
u+ u*

d (1)
0

Figure 24: Gram-Schmidt conjugation of two vectors. Begin with two linearly independent vectors
1 . Set 0 to 0 , and . 1

k 0. The vector 1 is composed of two components: 0 , which is -orthogonal (or conjugate) , which is parallel to 0 . After conjugation, only the -orthogonal portion remains, and k

and

26

Jonathan Richard Shewchuk

2

-4 -2 2 4 6

-2

-4

-6

Figure 25: The method of Conjugate Directions using the axial unit vectors, also known as Gaussian
elimination.

are required to generate the full set. In fact, if the search vectors are constructed by conjugation of the axial unit vectors, Conjugate Directions becomes equivalent to performing Gaussian elimination (see Figure 25). As a result, the method of Conjugate Directions enjoyed little use until the discovery of CG which is a method of Conjugate Directions cured these disadvantages. An important key to understanding the method of Conjugate Directions (and also CG) is to notice that Figure 25 is just a stretched copy of Figure 21! Remember that when one is performing the method of Conjugate Directions (including CG), one is simultaneously performing the method of Orthogonal Directions in a stretched (scaled) space.

7.3. Optimality of the Error Term Conjugate Directions has an interesting property: it nds at every step the best solution within the bounds 4 of where its been allowed to explore. Where has it been allowed to explore? Let be the -dimensional m m m m m subspace span 0 n 1 n "" 4 V 1 n ; the value p 4 n is chosen from p 0n A 4 . What do I mean by best W m m i i i 4 4 p n (see solution? I mean that Conjugate Directions chooses the value from n A that m minimizes p m 0W Figure 26). In fact, some authors derive CG by trying to minimize p 4 n within p 0 n A 4 . In the same way that the error term can be expressed as a linear combination of search directions

The Method of Conjugate Directions

27

0 d

(1)

(0)

e (0)

e e (1) (2)

Figure 26: In this gure, the shaded area is 08X

span 0 2 0 1 . The ellipsoid is a contour on which the energy norm is constant. After two steps, Conjugate Directions nds 2 , the point on 0 2 that minimizes .

|}

(Equation 35), its energy norm can be expressed as a summation.

p m 4 W n

# V 1 # V 1 m 54 54 n # V 1 2 m )m 54 n n

m )m m n n n m n

(by Equation 35)

(by -orthogonality of vectors).

Each term in this summation is associated with a search direction that has not yet been traversed. Any other m m 4 vector p chosen from p 0n A must have these same terms in its expansion, which proves that p 4 n must have the minimum energy norm. Having proven the optimality with equations, lets turn to the intuition. Perhaps the best way to visualize the workings of Conjugate Directions is by comparing the space we are working in with a stretched space, as in Figure 22. Figures 27(a) and 27(c) demonstrate the behavior of Conjugate Directions in 2 and 3 ; lines that appear perpendicular in these illustrations are orthogonal. On the other hand, Figures 27(b) and 27(d) show the same drawings in spaces that are stretched (along the eigenvector axes) so that the ellipsoidal contour lines become spherical. Lines that appear perpendicular in these illustrations are -orthogonal. In Figure 27(a), the Method of Conjugate Directions begins at 0 n , takes a step in the direction of 0 n , @m m m and stops at the point 1 n , where the error vector p 1 n is -orthogonal to 0 n . Why should we expect m this to be the minimum point on 0 n A 1 ? The answer is found in Figure 27(b): in this stretched space, m The error vector p 1 n is a radius of the p m 1n appears perpendicular to m 0n because they are -orthogonal. W m m concentric circles that represent contours on which p is constant, so 0 n A 1 must be tangent at 1n W 2m @m @m m to the circle that 1 n lies on. Hence, 1 n is the point on 0 n A 1 that minimizes p 1n . This is not surprising; we have already seen in Section 7.1 that -conjugacy of the search direction and W Y the error term is equivalent to minimizing (and therefore ) along the search direction. However, p Wp along m after Conjugate Directions takes a second step, minimizing a second search direction 1 n , why W m should we expect that p will still be minimized in the direction of 0 n ? After taking steps, why will m Y F m 4 I be minimized over all 4 of 0 n A ? n

28

Jonathan Richard Shewchuk

2m m
0

nA @m
1

m m 2m m 0n n m 1n
0

nA

m 0n

r m 1n p m 1n m 1n

r m 1n p m 1n

(a)

m m m 0n

(b)
0

nA

n @m nA 2m
2

m
0 1

nA

@m

n @m
0

1n r m 1n m 1n 2m

m 0n

m r 1n m m 1n @ 2n

nA

(d) (c) Figure 27: Optimality of the Method of Conjugate Directions. (a) A two-dimensional problem. Lines that
appear perpendicular are orthogonal. (b) The same problem in a stretched space. Lines that appear perpendicular are -orthogonal. (c) A three-dimensional problem. Two concentric ellipsoids are shown; is at the center of both. The line 0 1 is tangent to the outer ellipsoid at 2 is 1 . The plane 0 tangent to the inner ellipsoid at 2 . (d) Stretched view of the three-dimensional problem.

9 f 0

9 f

The Method of Conjugate Directions 29 m m and 1 n appear perpendicular because they are -orthogonal. It is clear that 1 n m m must point to the solution , because 0n is tangent at 1 n to a circle whose center is . However, a three-dimensional example is more revealing. Figures 27(c) and 27(d) each show two concentric ellipsoids. m m The point 1 n lies on the outer ellipsoid, and 2n lies on the inner ellipsoid. Look carefully at these gures: m m the plane 0 n A 2 slices through the larger ellipsoid, and is tangent to the smaller ellipsoid at 2 n . The

m In Figure 27(b), 0 n

point

is at the center of the ellipsoids, underneath the plane.

Looking at Figure 27(c), we can rephrase our 2m question. Suppose you and I are standing at 1n , and want to walk to the point that minimizes p on 0 n A 2 ; but we are constrained to walk along the search m m m direction 1 n . If 1n points to the minimum point, we will succeed. Is there any reason to expect that 1 n will point the right way? Figure 27(d) supplies an answer. Because 1 n is -orthogonal to 0n , they are perpendicular in this m diagram. Now, suppose you were staring down at the plane 0 n A 2 as if it were a sheet of paper; the m sight youd see would be identical to Figure 27(b). The point 2n would be at the center of the paper, m m m and the point would lie underneath the paper, directly under the point 2 n . Because 0n and 1 n are m 2m 2m 2m perpendicular, 1 n points directly to 2 n , which is the point in 0 n A 2 closest to . The plane 0n A 2 m m is tangent to the sphere on which 2 n lies. If you took a third step, it would be straight down from 2 n to , in a direction -orthogonal to . 2 Another way to understand what is happening in Figure 27(d) is to imagine yourself standing at the m 4 solution point , pulling a string connected to a bead that is constrained to lie in 0n A . Each time the expanding subspace is enlarged by a dimension, the bead becomes free to move a little closer to you. Now if you stretch the space so it looks like Figure 27(c), you have the Method of Conjugate Directions. Another important property of Conjugate Directions is visible in these illustrations. We have seen that, m m 4 at each step, the hyperplane 0n A is tangent to the ellipsoid on which 4 n lies. Recall from Section 4 m that the residual at any point is orthogonal to the ellipsoidal surface at that point. It follows that r 4 n is B )m 4 orthogonal to as well. To show this fact mathematically, premultiply Equation 35 by 4 n :

@m

B )m 4 n )m 4

p m n m nr n

Because the search directions are constructed from the vectors, the subspace spanned by 0 "" 4 V 1 m i i 4 is , and the residual r 4 n is orthogonal to these previous vectors as well (see Figure 28). This is proven m by taking the inner product of Equation 36 and r n :

We could have derived this identity by another tack. Recall that once we take a step in a search direction, we need never step in that direction again; the error term is evermore -orthogonal to all the old search m m u B directions. Because r 4 n p 4 n , the residual is evermore orthogonal to all the old search directions.

V B # 1 m 54 n 0 o i

)m 4 m n n

(38) (39)

(by -orthogonality of -vectors).

)m 4 n r m n
0

)4 r m n A )4 r m n i

4 )m n r m n 5 0
1

4 V

(40) (by Equation 39) (41)

There is one more identity we will use later. From Equation 40 (and Figure 28),

) m 4 n r m 4 n )4 r m 4 n

(42)

30

Jonathan Richard Shewchuk

d
2

(2)

r(2)

(2)

(1)

d (0) g

u1

u 0

Figure 28: Because the search directions 0 1 are constructed from the vectors 0 1 , they span the same subspace 2 (the gray-colored plane). The error term 2 is -orthogonal to 2 , the residual 2 is orthogonal to 2 , and a new search direction 2 is constructed (from 2 ) to be -orthogonal to 2 . The endpoints of 2 and 2 lie on a plane parallel to 2 , because 2 is constructed from 2 by Gram-Schmidt conjugation.

To conclude this section, I note that as with the method of Steepest Descent, the number of matrix-vector products per iteration can be reduced to one by using a recurrence to nd the residual:

r m 64 1n

B B  r m4n

p m 64 1n Fp m 4 n Ay m 4 n m 4 n I B m4 m4 y n n

(43)

8. The Method of Conjugate Gradients It may seem odd that an article about CG doesnt describe CG until page 30, but all the machinery is now in place. In fact, CG is simply the method of Conjugate Directions where the search directions are constructed r m 4 ). by conjugation of the residuals (that is, by setting 4 n This choice makes sense for many reasons. First, the residuals worked for Steepest Descent, so why not for Conjugate Directions? Second, the residual has the nice property that its orthogonal to the previous search directions (Equation 39), so its guaranteed always to produce a new, linearly independent search direction unless the residual is zero, in which case the problem is already solved. As we shall see, there is an even better reason to choose the residual. Lets consider the implications of this choice. Because the search vectors are built from the residuals, the m m m 4 subspace span r 0 n r 1 n "" r 4 V 1 n is equal to . As each residual is orthogonal to the previous search i i i directions, it is also orthogonal to the previous residuals (see Figure 29); Equation 41 becomes

l (44)  i m Interestingly, Equation 43 shows that each new residual r 4 n is just a linear combination of the previous m m residual and 4 V 1 n . Recalling that 4 V 1np 4 , this fact implies that each new subspace 4 1 is formed 4 4 from the union of the previous subspace and the subspace . Hence, 4 4 span m 0n i m 0n i 2 m 0n i "" i 4 V 1 m 0n span r m r m 2 r m "" V 1 r m 0n 0n i 0n i 0n i i r
0

)m 4 r m n n

The Method of Conjugate Gradients

31

r(2)

d d

(2)

e (2) r(1)

(1)

(0)

r(0)

Figure 29: In the method of Conjugate Gradients, each new residual is orthogonal to all the previous
residuals and search directions; and each new search direction is constructed (from the residual) to be -orthogonal to all the previous residuals and search directions. The endpoints of 2 and 2 lie on a plane parallel to 2 (the shaded subspace). In CG, 2 is a linear combination of 2 and 1 .

This subspace is called a Krylov subspace, a subspace created by repeatedly applying a matrix to a m vector. It has a pleasing property: because 4 is included in 46 1 , the fact that the next residual r 46 1n m 46 1 (Equation 39) implies that r 46 1n is -orthogonal to 4 . Gram-Schmidt conjugation is orthogonal to m m becomes easy, because r 46 1 n is already -orthogonal to all of the previous search directions except 4 n ! Recall from Equation 37 that the Gram-Schmidt constants are 4 m 43, simplify this expression. Taking the inner product of r 4 n and Equation

B r )m 4 m $ ) m m ; n n n n

let us

r )m 4 n r m 1n m )m m y nr 4n n r )m 4 n m n 4

As if by magic, most of the terms have disappeared. It is no longer necessary to store old search vectors of new search vectors. This major advance is what makes CG as important to ensure the -orthogonality an algorithm as it is, because both the space complexity and time complexity per iteration are reduced from m 4F 2 I to4 4 F I , where is the number of nonzero entries of . Henceforth, I shall use the abbreviation V 1. Simplifying further: n

r ) m 4 n r m n B y m n r )m 4 n m n r ) m 4 n r m n B r )m 4 n r m 1 n m4 1 )m B {x r 1 4 n r )m 4 n i m 4 r nr ni { 0 i x 1 {6 z 6 }x z"6 i 0i
1 1 1 1

i A 1i otherwise A 1i D A 1

(By Equation 44.)

(By Equation 37.)

m4 n

)m 4 r m 4 n mn ) m 4 V 1n r 4 V 1n r )m 4 n r m 4 n r )m 4 V 1n r m 4 V 1n r

(by Equation 32) (by Equation 42)

32

Jonathan Richard Shewchuk

2

-4 -2 2 4 6

-2

-4

-6

Figure 30: The method of Conjugate Gradients. Lets put it all together into one piece now. The method of Conjugate Gradients is:

m 0n r m 0n Bc@m 0n i ) m m 4 r ) m 4 n r 4 m n (by Equations 32 and 42) y n m4 4 i n n m m m 46 R m 4 n Aey 4 n 4 n i 1n r m 46 1n r m 4 n B y m 4 n m 4 n i ) m m 46 r m 46 ) 1n r m 46 1n 1n r m4n r 4n i m 46 1n r m 46 1n A m 46 1n m 4 n

(45) (46)

(47) (48) (49)

The performance of CG on our sample problem is demonstrated in Figure 30. The name Conjugate Gradients is a bit of a misnomer, because the gradients are not conjugate, and the conjugate directions are not all gradients. Conjugated Gradients would be more accurate.

9. Convergence Analysis of Conjugate Gradients CG is complete after  iterations, so why should we care about convergence analysis? In practice, accumulated oating point roundoff error causes the residual to gradually lose accuracy, and cancellation

Convergence Analysis of Conjugate Gradients 33 error causes the search vectors to lose -orthogonality. The former problem could be dealt with as it was for Steepest Descent, but the latter problem is not easily curable. Because of this loss of conjugacy, the mathematical community discarded CG during the 1960s, and interest only resurged when evidence for its effectiveness as an iterative procedure was published in the seventies.
Times have changed, and so has our outlook. Today, convergence analysis is important because CG is commonly used for problems so large it is not feasible to run even  iterations. Convergence analysis is seen less as a ward against oating point error, and more as a proof that CG is useful for problems that are out of the reach of any exact algorithm. The rst iteration of CG is identical to the rst iteration of Steepest Descent, so without changes, Section 6.1 describes the conditions under which CG converges on the rst iteration.

9.1. Picking Perfect Polynomials

4 , where A n 4 4 span r m 0n i r m 0n i 2 r m 0n i "" i V 14 r m 0n span p m 2 p m 3 p m "" p m 0n i 0n i 0n i i 0n Krylov subspaces such as this have another pleasing property. For a xed , the error term has the form 4 m 4 p n A 5 p m 0n 1 m m The coefcients are related to the values y 4 n and 4 n , but the precise relationship is not important here. W m 4 What is important is the proof in Section 7.3 that CG chooses the coefcients that minimize p n . 4 I The expression in parentheses above can be expressed as a polynomial. Let F be a polynomial of degree . 4 can take either a scalar or a matrix as its argument, and will evaluate to the same; for instance, I 2 2 1, then F I 2 2 . This exible notation comes in handy for eigenvectors, for if 2 F A A 4 I 4 I 2 F . (Note that , 2 2 , and so on.) which you should convince yourself that F
We have seen that at each step of CG, the value p

m4

is chosen from p

Now we can express the error term as

if we require that 1. CG chooses this polynomial when it chooses the coefcients. Lets m m examine the effect of applying this polynomial to p 0 n . As in the analysis of Steepest Descent, express p 0 n as a linear combination of orthogonal unit eigenvectors

4 F 0IX

p m4n

4 F qI p m n i
0

# p m 0n 5 "" i 1

and we nd that

p m4 2 n

p m4n p m4 n

4 F 8 I 4 I F8 8" 2t 4 I 2 F8 v 8

34

Jonathan Richard Shewchuk

F I

(a)

F I

(b)

1 0.75 0.5 0.25 -0.25 -0.5 -0.75 -1 2 7

1 0.75 0.5 0.25 -0.25 -0.5 -0.75 -1 2 7

F I

(c)

F I

(d)

1 0.75 0.5 0.25 -0.25 -0.5 -0.75 -1 2 7

1 0.75 0.5 0.25 -0.25 -0.5 -0.75 -1 2 7

Figure 31: The convergence of CG after iterations depends on how close a polynomial
be to zero on each eigenvalue, given the constraint that

1.

of degree can

CG nds the polynomial that minimizes this expression, but convergence is only as good as the convergence I be the set of eigenvalues of , we have of the worst eigenvector. Letting F

p m4 2 n

m min max

min max

Figure 31 illustrates, for several values of , the that minimizes this expression for our sample problem I 1, and that with eigenvalues 2 and 7. There is only one polynomial of degree zero that satises 0 F 0 Id 1, graphed in Figure 31(a). The optimal polynomial of degree one is F I 1 B 2 9, is 0 F 1 IT 5 9 and 7 graphed in Figure 31(b). Note that 1 F 2 5 9, and so the energy norm of the error 1 term after one iteration of CG is no greater than 5 9 its initial value. Figure 31(c) shows that, after two iterations, Equation 50 evaluates to zero. This is because a polynomial of degree two can be t to three Is 1, F 2 I 0, and F 7 I 0). In general, a polynomial of degree  can t  1 points, points ( 2 F 0 A 2 2 and thereby accommodate  separate eigenvalues.

 n t 4 F I v  m n t 4 F I v 4 F IpB

2 8 2 m 2 p 0n
2

(50)

The foregoing discussing reinforces our understanding that CG yields the exact result after  iterations; and furthermore proves that CG is quicker if there are duplicated eigenvalues. Given innite oating point precision, the number of iterations required to compute an exact solution is at most the number of distinct 2m eigenvalues. (There is one other possibility for early termination: 0n may already be -orthogonal to m some of the eigenvectors of . If eigenvectors are missing from the expansion of 0n , their eigenvalues may be omitted from consideration in Equation 50. Be forewarned, however, that these eigenvectors may be reintroduced by oating point roundoff error.)

Analysis of Conjugate Gradients  F I  Convergence F I

2 5

35

2 1.5 1 0.5

0.5 1

2 1.5 1 0.5

0.5 1

-1 -0.5 -0.5 -1 -1.5 -2

10

F I
0.5 1

-1 -0.5 -0.5 -1 -1.5 -2

49

F I

2 1.5 1 0.5 -1 -0.5 -0.5 -1 -1.5 -2

2 1.5 1 0.5

0.5 1

-1 -0.5 -0.5 -1 -1.5 -2

Figure 32: Chebyshev polynomials of degree 2, 5, 10, and 49. We also nd that CG converges more quickly when eigenvalues are clustered together (Figure 31(d)) 4 than when they are irregularly distributed between 0 # and 0o , because it is easier for CG to choose a polynomial that makes Equation 50 small. If we know something about the characteristics of the eigenvalues of , it is sometimes possible to suggest a polynomial that leads to a proof of a fast convergence. For the remainder of this analysis, however, 4 I shall assume the most general case: the eigenvalues are evenly distributed between 0 # and s , the number of distinct eigenvalues is large, and oating point roundoff occurs.

9.2. Chebyshev Polynomials A useful approach is to minimize Equation 50 over the range # 0o v rather than at a nite number of i points. The polynomials that accomplish this are based on Chebyshev polynomials. The Chebyshev polynomial of degree is

t 4

 4 F I

1 2

(If this expression doesnt look like a polynomial to you, try working it out for equal to 1 or 2.) Several Chebyshev polynomials are illustrated in Figure 32. The Chebyshev polynomials have the property that B txB 1 1v , and furthermore that 44 F II 1 (in fact, they oscillate between 1 and 1) on the domain I i speaking, 4 F F is maximum on the domain l txB 1 i 1v among all such polynomials. Loosely increases as quickly as possible outside the boxes in the illustration.

 
F A

2 B 1I 4

B A F

2 B 1I 4





36

F I
1

Jonathan Richard Shewchuk

0.75 0.5 0.25

1 -0.25 -0.5 -0.75 -1

that minimizes Equation 50 for 2 and 7 in the general case. This curve is a scaled version of the Chebyshev polynomial of degree 2. The energy norm of the error term after two iterations is no greater than 0.183 times its initial value. Compare with Figure 31(c), where it is known that there are only two eigenvalues.
2

Figure 33: The polynomial

 "!

$#&%

It is shown in Appendix C3 that Equation 50 is minimized by choosing

p m4 n

4 F I

This polynomial has the oscillating properties of Chebyshev polynomials on the domain 0 4 # 0o (see Figure 33). The denominator enforces our requirement that 0 1. The numerator has a maximum 4 value of one on the interval between 0 # and , so from Equation 50 we have

0 4  1 A 2 ! B 1 Aw

 4 0o AB 44 # V p m n 0o V #  4 AB p m n W
1 0

 4 ( '0( )&'01 )& 1  V '  ' V  '0 )&)&11 V  ''  4  0 ' 4 F I 0o

2

1 1

B A

1 1

4% V 1 ( p m 0n W

(51)

The second addend inside the square brackets converges to zero as grows, so it is more common to express the convergence of CG with the weaker inequality

p m4 W n

B 1 4 m W 2 A 1 p 0n

(52)

Complexity

37

0.8

0.6

0.4

0.2

20

40

60

80

100

Figure 34: Convergence of Conjugate Gradients (per iteration) as a function of condition number. Compare
with Figure 20.

1 in Equation 51, we obtain The rst step of CG is identical to a step of Steepest Descent. Setting Equation 28, the convergence result for Steepest Descent. This is just the linear polynomial case illustrated in Figure 31(b). Figure 34 charts the convergence per iteration of CG, ignoring the lost factor of 2. In practice, CG usually converges faster than Equation 52 would suggest, because of good eigenvalue distributions or good starting points. Comparing Equations 52 and 28, it is clear that the convergence of CG is much quicker than that of Steepest Descent (see Figure 35). However, it is not necessarily true that every iteration of CG enjoys faster convergence; for example, the rst iteration of CG is an iteration of Steepest Descent. The factor of 2 in Equation 52 allows CG a little slack for these poor iterations.

10. Complexity The dominating operations during an iteration of either Steepest Descent or CG are matrix-vector products. I operations, where is the number of non-zero In general, matrix-vector multiplication requires F is sparse and entries in the matrix. For many problems, including those listed in the introduction, I F . to perform enough iterations to reduce the norm of the error by a factor of ; that is, p m 4 Suppose p we . Equation m wish 28 can be used to show that the maximum number of iterations required to n 0n

achieve this bound using Steepest Descent is

43

1 1 ln 2 1 2

$6
i

whereas Equation 52 suggests that the maximum number of iterations CG requires is

73

ln

$6

38

40 35 30 25 20 15 10 5 0

Jonathan Richard Shewchuk

200

400

600

800

1000

Figure 35: Number of iterations of Steepest Descent required to match one iteration of CG. I conclude that Steepest Descent has a time complexity of F I . Both algorithms have a space complexity of F

I Ff I , whereas CG has a time complexity of

.

Finite difference and nite element approximations of second-order elliptic boundary value problems I posed on -dimensional domains often have F 2 z . Thus, Steepest Descent has a time complexity of F 2 I for I two-dimensional problems, versus F 3 2 I for CG; I and Steepest Descent has a time complexity of F 5 3 for three-dimensional problems, versus F 4 3 for CG. 11. Starting and Stopping In the preceding presentation of the Steepest Descent and Conjugate Gradient algorithms, several details have been omitted; particularly, how to choose a starting point, and when to stop.

11.1. Starting Theres not much to say about starting. If you have a rough estimate of the value of , use it as the starting m m 0; either Steepest Descent or CG will eventually converge when used to solve value 0 n . If not, set 0 n linear systems. Nonlinear minimization (coming up in Section 14) is trickier, though, because there may be several local minima, and the choice of starting point will determine which minimum the procedure converges to, or whether it will converge at all.

11.2. Stopping When Steepest Descent or CG reaches the minimum point, the residual becomes zero, and if Equation 11 or 48 is evaluated an iteration later, a division by zero will result. It seems, then, that one must stop

Preconditioning 39 immediately when the residual is zero. To complicate things, though, accumulated roundoff error in the recursive formulation of the residual (Equation 47) may yield a false zero residual; this problem could be resolved by restarting with Equation 45.
Usually, however, one wishes to stop before convergence is complete. Because the error term is not available, it is customary to stop when the norm of the residual falls below a specied value; often, this m m 4 value is some small fraction of the initial residual ( r n r 0n ). See Appendix B for sample code.

82

12. Preconditioning Preconditioning is a technique for improving the condition number of a matrix. Suppose that R is a symmetric, positive-denite matrix that approximates , but is easier to invert. We can solve indirectly by solving V 1 V 1 (53) are better clustered than those of , we can iteratively V 1 is not generally solve Equation 53 more quickly than the original problem. The catch is that symmetric nor denite, even if and are.

V 1 qI If @F

@9

BA

@F I , or if the eigenvalues of V 1

We can circumvent this difculty, because for every symmetric, positive-denite there is a (not u ) necessarily unique) matrix that has the property that an can be obtained, for V 1 and V 1 . V) (Such have the same eigenvalues. This instance, by Cholesky factorization.) The matrices V 1 with eigenvalue , then ) is an eigenvector of V 1 V) is true because if is an eigenvector of with eigenvalue :

F V 1 V) I F ) I F ) V) I V 1 )
can be transformed into the problem

V 1 9 )

can be found by Steepest Descent or CG. The process of using CG to solve this system is called the Transformed Preconditioned Conjugate Gradient Method:

) V 1 V) V 1 i i V V ) 1 which we solve rst for , then for . Because is symmetric and positive-denite,

The system



$C

m 0n r m 0n V 1 sB V 1 V) 2m 0n i r )m 4 n r m 4 n m4 y n ) m 4 V 1 V) m 4 i n n 2m 46 2m 4 m4 m4 n Aey n n i 1n r m 46 1n r m 4 n B y m 4 n V 1 V) m 4 n i m ) m 4 r m 46 ) 1n r m 46 1n 1n r m4n r 4n i m 46 1n r m 46 1n A m 46 1n m 4 n

C C

$C C

C C C C

40

Jonathan Richard Shewchuk

2

-4 -2 2 4 6 -2

-4

-6

-8

Figure 36: Contour lines of the quadratic form of the diagonally preconditioned sample problem. This method has the undesirable characteristic that must be computed. However, a few careful m V 1 r m 4 and m 4 ) m 4 , and using the identities variable substitutions can eliminate . Setting r 4 n n n n m 4 )2 m 4 and V) V 1 V 1 , we derive the Untransformed Preconditioned Conjugate Gradient n n Method:

r m 0n oB2m 0n i m 0n V 1 r m 0n i ) V m m 4 r m 4 n ) 1m r 4 n y n m4n 4n i m m m 46 R m 4 n Aey 4 n 4 n i 1n r m 4 1n r m 4 n B y m 4 n m 4 n i ) V m m 4 r m 46 ) 1n 1 r m 46 1n 1n r m 4 n V 1r 4 n i m 46 1n V 1 r m 46 1n A m 46 1n m 4 n V 1 is needed. By the same means, it is possible The matrix does not appear in these equations; only to derive a Preconditioned Steepest Descent Method that does not use . V 1 , and occais determined by the condition number of The effectiveness of a preconditioner

sionally by its clustering of eigenvalues. The problem remains of nding a preconditioner that approximates well enough to improve convergence enough to make up for the cost of computing the product V 1 r m 4 n V 1; it is only necessary to be able to once per iteration. (It is not necessary to explicitly compute or V 1 to a vector.) Within this constraint, there is a surprisingly rich supply compute the effect of applying of possibilities, and I can only scratch the surface here.

9 9

Conjugate Gradients on the Normal Equations 41 Intuitively, preconditioning is an attempt to stretch the quadratic form to make it appear more spherical, R ; for this preconditioner, so that the eigenvalues are close to each other. A perfect preconditioner is V 1 has a condition number of one, and the quadratic form is perfectly spherical, solution takes only Q so one iteration. Unfortunately, the preconditioning step is solving the system , so this isnt a useful preconditioner at all.

The simplest preconditioner is a diagonal matrix whose diagonal entries are identical to those of . The process of applying this preconditioner, known as diagonal preconditioning or Jacobi preconditioning, is equivalent to scaling the quadratic form along the coordinate axes. (By comparison, the perfect precondi scales the quadratic form along its eigenvector axes.) A diagonal matrix is trivial to invert, tioner but is often only a mediocre preconditioner. The contour lines of our sample problem are shown, after diagonal preconditioning, in Figure 36. Comparing with Figure 3, it is clear that some improvement has occurred. The condition number has improved from 3 5 to roughly 2 8. Of course, this improvement is much more benecial for systems where  2.

ED

A more elaborate preconditioner is incomplete Cholesky preconditioning. Cholesky factorization is a ) , where is a lower triangular matrix. Incomplete technique for factoring a matrix into the form Cholesky factorization is a variant in which little or no ll is allowed; is approximated by the product ) , where might be restricted to have the same pattern of nonzero elements as ; other elements of are ) as a preconditioner, the solution to ) is computed by backsubstitution thrown away. To use ) is never explicitly computed). Unfortunately, incomplete Cholesky preconditioning is (the inverse of not always stable.

FC FC

FC

CF F C F C F C

FGF

F C F C IH

Many preconditioners, some quite sophisticated, have been developed. Whatever your choice, it is generally accepted that for large-scale applications, CG should nearly always be used with a preconditioner.

13. Conjugate Gradients on the Normal Equations CG can be used to solve systems where solution to the least squares problem

is not symmetric, not positive-denite, and even not square. A min

bBc 2

(54)

can be found by setting the derivative of Expression 54 to zero:

) R )

(55)

)@ is symmetric and positive (for any , 0)@)@  2 0). If  is not underconstrained, ) is nonsingular, and methods like Steepest Descent and CG can be used to solve Equation 55. The then T @ ) only nuisance in doing so is that the condition number of is the square of that of , so convergence is
signicantly slower. An important technical point is that the matrix never formed explicitly, because it is less sparse ) is multiplied by by rst nding is ) . Also, numerical than . Instead, the product , and then

If is square and nonsingular, the solution to Equation 55 is the solution to . If is not square, q is overconstrained that is, has more linearly independent equations than and variables then  there may or may not be a solution to , but it is always possible to nd a value of that minimizes Expression 54, the sum of the squares of the errors of each linear equation.



T)@

42 stability is improved if the value with itself.

Richard Shewchuk (in Equation ) ) Jonathan 46) is computed by taking the inner product of

14. The Nonlinear Conjugate Gradient Method CG can be used not only to nd the minimum point of a quadratic form, but to minimize any continuous Y wI for which the gradient Y can be computed. Applications include a variety of optimization function F problems, such as engineering design, neural net training, and nonlinear regression.

14.1. Outline of the Nonlinear Conjugate Gradient Method To derive nonlinear CG, there are three changes to the linear algorithm: the recursive formula for the residual cannot be used, it becomes more complicated to compute the step size y , and there are several different choices for .

F 4 n . The search In nonlinear CG, the residual is always set to the negation of the gradient; r 4 n directions are computed by Gram-Schmidt conjugation of the residuals as with linear CG. Performing a line search along this search direction is much more difcult than in the linear case, and a variety of procedures m Y m m m I can be used. As with the linear CG, a value of y 4 n that minimizes F 4 n Ay 4 n 4 n is found by ensuring that the gradient is orthogonal to the search direction. We can use any algorithm that nds the zeros of the t Y 2m m m I) m expression F 4 n Aey 4 n 4 n v 4 n .
In linear CG, there are several equivalent expressions for the value of . In nonlinear CG, these different best choice. Two choices are the expressions are no longer equivalent; researchers are still investigating the Fletcher-Reeves formula, which we used in linear CG for its ease of computation, and the Polak-Ribi` ere formula: )m m 46 )m m 46 B m 4 I

m CB Y m I

QPIR

m 46

r 4 ) 1n r m 1n 1n r m4n r 4n i

m 46

r 64 1n Fr ) 1m n r n 1n r m4n r 4n

Fortunately, convergence of the Polak-Ribi` max 0 . ere method can be guaranteed by choosing i the past search Using this value is equivalent to restarting CG if 0. To restart CG is to forget descent. directions, and start CG anew in the direction of steepest

The Fletcher-Reeves method converges if the starting point is sufciently close to the desired minimum, whereas the Polak-Ribi` ere method can, in rare cases, cycle innitely without converging. However, PolakRibi` ere often converges much more quickly.

Here is an outline of the nonlinear CG method:

m 0n r m 0n CB Y F m 0n I i m m m I Y @m Find y 4 n that minimizes F 4 n Aty 4 n 4 n i m4 m4 m 46 R m 4 n Aey n n i 1n m C B Y F m 46 I 6 4 r 1n 1n i )m m ) m 4 6 4 r r r 46 1n Fr m 46 1n B r m 4 n I m 46 m 1n 1n 6 4 or max 0 )m 4 r m 4 )m 4 r m 4 1n 1n i i r r n n n n

The Nonlinear Conjugate Gradient Method

m 46 1n r m 64 1n A

m 46 m 4 1n n

43

Nonlinear CG comes with few of the convergence guarantees of linear CG. The less similar is to a quadratic function, the more quickly the search directions lose conjugacy. (It will become clear shortly that the term conjugacy still has some meaning in nonlinear CG.) Another problem is that a general function Y may have many local minima. CG is not guaranteed to converge to the global minimum, and may not Y even nd a local minimum if has no lower bound. Figure 37 illustrates nonlinear CG. Figure 37(a) is a function with many local minima. Figure 37(b) demonstrates the convergence of nonlinear CG with the Fletcher-Reeves formula. In this example, CG is not nearly as effective as in the linear case; this function is deceptively difcult to minimize. Figure 37(c) shows a cross-section of the surface, corresponding to the rst line search in Figure 37(b). Notice that there are several minima; the line search nds a value of y corresponding to a nearby minimum. Figure 37(d) shows the superior convergence of Polak-Ribi` ere CG. Because CG can only generate  conjugate vectors in an  -dimensional space, it makes sense to restart CG every  iterations, especially if  is small. Figure 38 shows the effect when nonlinear CG is restarted every second iteration. (For this particular example, both the Fletcher-Reeves method and the Polak-Ribi` ere method appear the same.)

14.2. General Line Search Depending on the value of , it might be possible to use a fast algorithm to nd the zeros of For Y is polynomial in , then an efcient algorithm for polynomial zero-nding can be .used. instance, if y However, we will only consider general-purpose algorithms. Two iterative methods for zero-nding are the Newton-Raphson method and the Secant method. Both Y methods require that be twice continuously differentiable. Newton-Raphson also requires that it be I Y possible to compute the second derivative of F Aey with respect to y . The Newton-Raphson method relies on the Taylor series approximation

Y )

I Y F Aey I Y F e A y y

UT UT

where

Y F `I

I Y Fw A y e Y F wI Aey t Y F `I v )

WV y ` ` `

2 f 2 Y I I Y F Aty 5 0 A y 2 y 2 F e A y h 5 { { t Y F wI v ) y 2 ) Y F `I A 2 ) Y `I Aey F

YX

(56)
0

(57)

is the Hessian matrix

 " " Y F wI " " . ! .. ""

2 1 2 2 2

1 1

" " " "

2 1 2 2

` ` `

2 2

" "" " " "

2 1 2 2

` ` `

%'&& && & (

"
2

"" "$"
.
2

..

. . .

44

Jonathan Richard Shewchuk

(a)
6

(b)

4 500 250 6 0 4 -250 2 0 0 -2 -4 -2 6 -4 -2

I Y F`

n
2 4 6

2
(c)

-2

Y F m4 m I n Aey 4 n
600

(d)

400

200

@m

n
2 4 6

-0.04

-0.02

0.02

0.04

-4

-2

-200

-2

Figure 37: Convergence of the nonlinear Conjugate Gradient Method. (a) A complicated function with many local minima and maxima. (b) Convergence path of Fletcher-Reeves CG. Unlike linear CG, convergence does not occur in two steps. (c) Cross-section of the surface corresponding to the rst line search. (d) Convergence path of Polak-Ribi` ere CG.

The Nonlinear Conjugate Gradient Method

45

@m

n
4 6
1

-4

-2

-2

Figure 38: Nonlinear CG can be more effective with periodic restarts.

1 0.75 0.5 0.25

-1

-0.5 -0.25 -0.5 -0.75 -1

0.5

1.5

Figure 39: The Newton-Raphson method for minimizing a one-dimensional function (solid curve). Starting from the point , calculate the rst and second derivatives, and use them to construct a quadratic approximation to the function (dashed curve). A new point is chosen at the base of the parabola. This procedure is iterated until convergence is reached.

46

CB Y ) Y) y I Y The truncated Taylor series approximates F Ay with a parabola; we step to the bottom of the parabola Y Y is (see Figure 39). In fact, if is a quadratic form, then this parabolic approximation is exact, because Y just the familiar matrix . In general, the search directions are conjugate if they are -orthogonal. The Y Y varies varies with . The more quickly meaning of conjugate keeps changing, though, because @ m with , the more quickly the search directions lose conjugacy. On the other hand, the closer 4 n is to the Y
solution, the less varies from iteration to iteration. The closer the starting point is to the solution, the more similar the convergence of nonlinear CG is to that of linear CG. To perform an exact line search of a non-quadratic function, repeated steps must be taken along the line Y ) is zero; hence, one CG iteration may include many Newton-Raphson iterations. The values of until Y ) and )Y must be evaluated at each step. These evaluations may be inexpensive if )Y can be Y must be evaluated repeatedly, the algorithm is prohibitively analytically simplied, but if the full matrix slow. For some applications, it is possible to circumvent this problem by performing an approximate line Y search that uses only the diagonal elements of . Of course, there are functions for which it is not possible Y to evaluate at all. To perform an exact line search without computing , the Secant method approximates the second I 0 and , where is Y derivative of F Ay by evaluating the rst derivative at the distinct points y y an arbitrary small nonzero number:

I Y The function F Aey

Jonathan Richard Shewchuk is approximately minimized by setting Expression 57 to zero, giving

cb
0

t z Y F I v 5 Bt z Y F Iv 5 A e y A e y { { z|{ z|{ t Y F A I v ) Bt Y F wI v ) i which becomes a better approximation to the second derivative as y and I 2 Y F A y 2 y

dT

fe b

approach zero. If we substitute Expression 58 for the third term of the Taylor expansion (Equation 56), we have

(58)

Minimize

with a parabola, but instead of choosing the parabola by nding the rst and second derivative at a point, it nds the rst derivative at two different points (see Figure 40). Typically, we will choose an arbitrary on the rst Secant method iteration; on subsequent iterations we will choose A to be the value of from the previous Secant method iteration. In other words, if we let y 4 denote the value of y calculated during Secant iteration , CB 4 . then 46 1 y

uB t Y t Y I F ) wI v )B F A v t Y F `I v ) y Y Like Newton-Raphson, the Secant method also approximates F

I Y F Aey

I qt Y F `I v ) y Y F A e y A y

BT

bhg t Y F A b I v ) b

B7t Y F wI v )

pi

by setting its derivative to zero:

qb

(59)

br t

r ut

sr "t

I A7y

Both the Newton-Raphson and Secant methods should be terminated when is reasonably close to the exact solution. Demanding too little precision could cause a failure of convergence, but demanding too ne precision makes the computation unnecessarily slow and gains nothing, because conjugacy will break

The Nonlinear Conjugate Gradient Method

1

47

0.5

-1

-0.5 -1

-1.5

Figure 40: The Secant method for minimizing a one-dimensional function (solid curve). Starting from the
point , calculate the rst derivatives at two different points (here, 0 and 2), and use them to construct a quadratic approximation to the function (dashed curve). Notice that both curves have the same 0, and also at 2. As in the Newton-Raphson method, a new point is chosen at the base slope at of the parabola, and the procedure is iterated until convergence.

vb

vb

down quickly anyway if F varies much with . Therefore, a quick but inexact line search is often the better policy (for instance, use only a xed number of Newton-Raphson or Secant method iterations). Unfortunately, inexact line search may lead to the construction of a search direction that is not a descent ) direction. A common solution is to test for this eventuality (is r nonpositive?), and restart CG if necessary r. by setting A bigger problem with both methods is that they cannot distinguish minima from maxima. The result of nonlinear CG generally depends strongly on the starting point, and if CG with the Newton-Raphson or Secant method starts near a local maximum, it is likely to converge to that point. Each method has its own advantages. The Newton-Raphson method has a better convergence rate, and is ) Y can be calculated (or well approximated) quickly (i.e., in F I time). The Secant to be preferred if Y method only requires rst derivatives of , but its success may depend on a good choice of the parameter . Y It is easy to derive a variety of other methods as well. For instance, by sampling at three different points, I Y it is possible to generate a parabola that approximates F Ay without the need to calculate even a rst Y derivative of .

Y `I

14.3. Preconditioning Nonlinear CG can be preconditioned by choosing a preconditioner that approximates and has the V 1 r is easy to compute. In linear CG, the preconditioner attempts property that to transform the quadratic form so that it is similar to a sphere; a nonlinear CG preconditioner performs this transformation for a region m near 4 n . Even when it is too expensive to compute the full Hessian

Y , it is often reasonable to compute its diagonal

48

Jonathan Richard Shewchuk

2

m
2

n
4 6

-4

-2

-2

Figure 41: The preconditioned nonlinear Conjugate Gradient Method, using the Polak-Ribi` ere formula and
a diagonal preconditioner. The space has been stretched to show the improvement in circularity of the contour lines around the minimum.

for use as a preconditioner. However, be forewarned that if is sufciently far from a local minimum, the diagonal elements of the Hessian may not all be positive. A preconditioner should be positive-denite, so ) nonpositive diagonal elements cannot be allowed. A conservative solution is to not precondition (set when the Hessian cannot be guaranteed to be positive-denite. Figure 41 demonstrates the convergence of diagonally preconditioned nonlinear CG, with the Polak-Ribi` ere formula, on the same function illustrated Y at the solution point to precondition every in Figure 37. Here, I have cheated by using the diagonal of iteration.

A Notes Conjugate Direction methods were probably rst presented by Schmidt [14] in 1908, and were independently reinvented by Fox, Huskey, and Wilkinson [7] in 1948. In the early fties, the method of Conjugate Gradients was discovered independently by Hestenes [10] and Stiefel [15]; shortly thereafter, they jointly published what is considered the seminal reference on CG [11]. Convergence bounds for CG in terms of Chebyshev polynomials were developed by Kaniel [12]. A more thorough analysis of CG convergence is provided by van der Sluis and van der Vorst [16]. CG was popularized as an iterative method for large, sparse matrices by Reid [13] in 1971. CG was generalized to nonlinear problems in 1964 by Fletcher and Reeves [6], based on work by Davidon [4] and Fletcher and Powell [5]. Convergence of nonlinear CG with inexact line searches was analyzed by Daniel [3]. The choice of for nonlinear CG is discussed by Gilbert and Nocedal [8]. A history and extensive annotated bibliography of CG to the mid-seventies is provided by Golub and OLeary [9]. Most research since that time has focused on nonsymmetric systems. A survey of iterative methods for the solution of linear systems is offered by Barrett et al. [1].

Canned Algorithms
B Canned Algorithms

49

The code given in this section represents efcient implementations of the algorithms discussed in this article.

B1. Steepest Descent Given the inputs S 1:

, , a starting value

, a maximum number of iterations 0

, and an error tolerance

x w ryw w

While

Bc r) r

and D 2 0 do s r y Aey r If is divisible by 50 r sBk else r ) gr B y gr r A 1

r m 4 This algorithm r m . terminates when the maximum number of iterations o n 0n

w w w w w w w

has been exceeded, or when

The fast recursive formula for the residual is usually used, but once every 50 iterations, the exact residual is recalculated to remove accumulated oating point error. Of course, the number 50 is arbitrary; for large  ,  might be appropriate. If the tolerance is large, the residual need not be corrected at all (in practice, this correction is rarely used). If the tolerance is close to the limits of the oating point precision of the machine, a test should be added after is evaluated to check if 2 0 , and if this test holds true, the exact reevaluated. This prevents the procedure from terminating early residual should also be recomputed and due to oating point roundoff error.

50 B2. Conjugate Gradients Given the inputs S 1:

Jonathan Richard Shewchuk

, , a starting value
0

, a maximum number of iterations

, and an error tolerance

While

x w rw oB wgr ) # wur r w # Po Y D 2 # w @ y w w z Aey Bc ryw ryw r B y w r# ) r # Yz wg d@ w iw r feY A gh xu w A

0

and

do

If is divisible by 50 else

See the comments at the end of Section B1.

Canned Algorithms B3. Preconditioned Conjugate Gradients

Given the inputs , , a starting value , a (perhaps implicitly dened) preconditioner number of iterations o , and an error tolerance 1:

51

While

The statement be in the form of a matrix.

k wl9

V 1r

x w rw oB wj9 V ) r # wur w # Po Y D 2 # w @ y w w z Aey Bc ryw ryw V r B y k wj9 r w w r# ) k # Yz g w d@ iw k AeYg h xu w A

0
1 0

, a maximum

and

do

If is divisible by 50 else

implies that one should apply the preconditioner, which may not actually

See also the comments at the end of Section B1.

52 Jonathan Richard Shewchuk B4. Nonlinear Conjugate Gradients with Newton-Raphson and Fletcher-Reeves Given a function , a starting value , a maximum number of CG iterations Po , a CG error tolerance  1, a maximum number of Newton-Raphson iterations o , and a Newton-Raphson error tolerance 1:

25

While

r m 4 This algorithm r m . terminates when the maximum number of iterations o has been exceeded, or when n 0n ; the iterations are terminated when each update falls Each Newton-Raphson iteration adds to y y below a given tolerance ( y ), or when the number of iterations exceeds s . A fast inexact line Y `I with its search can be accomplished by using a small o and/or by approximating the Hessian F

xw w rw B Y F wI wgr ) # wur r w # D Po # Y 2 ) wu w z m t ` m B r y w z ` m n m n z z ww At y A D 5 s y z I rw B Y F ` w r# ) r # Yz wg w d@ eYg h i ww r A d  A r ) i ww r g xwu

0 0
0

and

do

Do

while

and

If

1 or

diagonal.

r ) whenever a search direction is computed that is not a Nonlinear CG is restarted (by setting descent direction. It is also restarted once every  iterations, to improve convergence for small  .
The calculation of y may result in a divide-by-zero error. This may occur because the starting point 0n Y is not sufciently close to the desired minimum, or because is not twice continuously differentiable. In the former case, the solution is to choose a better starting point or a more sophisticated line search. In the latter case, CG might not be the most appropriate minimization algorithm.

hw

2m

Canned Algorithms B5. Preconditioned Nonlinear Conjugate Gradients with Secant and Polak-Ribi` ere

53

Given a function , a starting value , a maximum number of CG iterations Po , a CG error tolerance 1, a Secant method step parameter 0, a maximum number of Secant method iterations o , and a Secant method error tolerance 1:

Calculate a preconditioner
1

While

r m 4 This algorithm r m . terminates when the maximum number of iterations o has been exceeded, or when n 0n Each Secant method iteration adds y to ; the iterations are terminated when each update y falls below a given tolerance ( y ), or when the number of iterations exceeds o . A fast inexact line search can be accomplished by using a small o . The parameter 0 determines the value of in Equation 59 for the rst step of each Secant method minimization. Unfortunately, it may be necessary to adjust this parameter

xw w ryw B Y F `I T Y F `I 9 k wn9k V r iw ) # Yow r w # Y D o # 2 ww ) z w Bqb t Y b I ) ypr q Ystw F A v p t Y `I ) w F v y w y @u y V uwAt vx y u w prq stw p 5 A D w o y z I ryw B Y F w w # Y w z 4 z w r ) k T Y F `I 9 V k wj9 ) r k w r # og w @ V ' h w d A eYg h  k z ww zw k A xw A

0 0
0

and

do

Do

while

and

Calculate a preconditioner
1

If

1 or

else

m V V j j n . Care must The Polak-Ribi` ere parameter is } j j j j}The be taken that the preconditioner is always positive-denite. preconditioner is not necessarily in the V

54 to achieve convergence.

Jonathan Richard Shewchuk

form of a matrix.

) whenever the Polak-Ribi` ere parameter is negative. It is Nonlinear CG is restarted (by setting also restarted once every  iterations, to improve convergence for small  . Nonlinear CG presents several choices: Preconditioned or not, Newton-Raphson method or Secant method or another method, Fletcher-Reeves or Polak-Ribi` ere. It should be possible to construct any of the variations from the versions presented above. (Polak-Ribi` ere is almost always to be preferred.)

 eYg h ' h 9 w r

u{ }| u{ | u{ }|
1 1 1

"{ ~ ~ "{
1 1 1

C Ugly Proofs C1. The Solution to



Minimizes the Quadratic Form

Suppose is symmetric. Let be a point that satises (Equation 3), and let p be an error term. Then

and minimizes the quadratic form

Y F p I A

I ) IBc ) A p F A p )  p )  1 p ) A A 2 ) dBc ) ) AtdA p Y F `I 1 p ) p A

1 F 2 1 2 1 2

(by Equation 3) F A p I Aed p Bc ) dBt ) p e (by symmetry of A d 1 ) Bc ) p A 2p p

If

is positive-denite, then the latter term is positive for all pl 0; therefore

minimizes .

C2. A Symmetric Matrix Has  Orthogonal Eigenvectors. Any matrix has at least one eigenvector. To see this, note 9 B is a polynomial in , and that det F  I has therefore has at least one zero (possibly complex); call it . The matrix determinant zero R B and is therefore singular, so there be a nonzero vector for which F 0. The vector is must an eigenvector, because . Any symmetric matrix has a full complement of  orthogonal eigenvectors. To prove this, I will demonstrate for the case of a 4  4 matrix and let you make the obvious generalization to matrices of arbitrary size. It has already been proven that has at least one eigenvector with eigenvalue . Let , which 4 has unit length. Choose three arbitrary vectors 2 3 4 so that the are mutually 1 i i orthogonal and all of unit length (such vectors can be found by Gram-Schmidt orthogonalization). Let t v . Because the 4 are orthonormal, ) , and so )c V 1 . Note also that for 1 2 3 4

uB

 q Q
1,
1 1 1 1 2 3 4

Ugly Proofs 0. Thus,

55

where is a 3 3 symmetric matrix. must have some eigenvalue with eigenvalue . Let be a vector of length 4 whose rst element is zero, and whose remaining elements are the elements of . Clearly,

0 0 0

2 3 4

1 2 3 4

1 2 3 4

In other words, 1 0 0 0 0, so

The more general statement that a symmetric matrix has orthogonal eigenvectors is proven by induction. For the example above, assume any 3 3 matrix (such as ) has 3 orthogonal eigenvectors; call them 1 , , and they are orthogonal because 2 , and 3 . Then 1, 2 , and 3 are eigenvectors of has orthonormal columns, and therefore maps orthogonal vectors to orthogonal vectors. Thus, has 4 orthogonal eigenvectors.

0
0 0 0
1

0 0 0

and

, and thus is an eigenvector of . Furthermore, 1 are orthogonal. Thus, has at least two orthogonal eigenvectors!

C3. Optimality of Chebyshev Polynomials Chebyshev polynomials are optimal for minimization of expressions like Expression 50 because they increase in magnitude more quickly outside the range 1 1 than any other polynomial that is restricted to have magnitude no greater than one inside the range 1 1 . The Chebyshev polynomial of degree , 1 2

can also be expressed on the region

From this expression (and from Figure 32), one can deduce that the Chebyshev polynomials have the property 1 1 1 and oscillate rapidly between 1 and 1: cos

" " &} }o } " }  &} W Q w B

2

1 1 as

cos cos

0 1

56 Jonathan Richard Shewchuk Notice that the zeros of must fall between the 1 extrema of in the range see the ve zeros of 5 in Figure 32.

oscillates in the range that 0 1.

of degree such that 0 1 and The proof relies on a cute trick. There is no polynomial is better than on the range . To prove this, suppose that there is such a polynomial; then, 1 on the range . It follows that the polynomial has a zero at 0, and also has zeros on the range . Hence, is a polynomial of degree with at least 1 zeros, which is impossible. By contradiction, I conclude that the Chebyshev polynomial of degree optimizes Expression 50.

I r p & f f & p u p p && f uu p p p s & &rp&f p u rp& pu s & & rrp r f p u I p& pu " s
Similarly, the function
2 1

"

1 1 . For an example,

on the domain

also satises the requirement

&

D Homework For the following questions, assume (unless otherwise stated) that you are using exact arithmetic; there is no oating point roundoff error. 1. (Easy.) Prove that the preconditioned Conjugate Gradient method is not affected if the preconditioning matrix is scaled. In other words, for any nonzero constant , show that the sequence of steps taken using the preconditioner is identical to the steps taken using the preconditioner 0 1 .

2. (Hard, but interesting.) Suppose you wish to solve for a symmetric, positive-denite matrix . Unfortunately, the trauma of your linear algebra course has caused you to repress all memories of the Conjugate Gradient algorithm. Seeing you in distress, the Good Eigenfairy materializes and grants you a list of the distinct eigenvalues (but not the eigenvectors) of . However, you do not know how many times each eigenvalue is repeated.

Clever person that you are, you mumbled the following algorithm in your sleep this morning: Choose an arbitrary starting point For 0 to 1

No eigenvalue is used twice; on termination, the list is empty.

j
1 1

Remove an arbitrary eigenvalue from the list and call it

(a) Show that upon termination of this algorithm, is the solution to . Hint: Read Section 6.1 carefully. Graph the convergence of a two-dimensional example; draw the eigenvector axes, and try selecting each eigenvalue rst. (It is most important to intuitively explain what the algorithm is doing; if you can, also give equations that prove it.)

"

Homework 57 (b) Although this routine nds the exact solution after iterations, you would like each intermediate iterate to be as close to the solution as possible. Give a crude rule of thumb for how you should choose an eigenvalue from the list on each iteration. (In other words, in what order should the eigenvalues be used?)

(c) What could go terribly wrong with this algorithm if oating point roundoff occurs? (d) Give a rule of thumb for how you should choose an eigenvalue from the list on each iteration to prevent oating point roundoff error from escalating. Hint: The answer is not the same as that of question (b).

58 References

Jonathan Richard Shewchuk

[1] Richard Barrett, Michael Berry, Tony Chan, James Demmel, June Donato, Jack Dongarra, Victor Eijkhout, Roldan Pozo, Charles Romine, and Henk van der Vorst, Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, SIAM, Philadelphia, Pennsylvania, 1993. [2] William L. Briggs, A Multigrid Tutorial, SIAM, Philadelphia, Pennsylvania, 1987. [3] James W. Daniel, Convergence of the Conjugate Gradient Method with Computationally Convenient Modications, Numerische Mathematik 10 (1967), 125131. [4] W. C. Davidon, Variable Metric Method for Minimization, Tech. Report ANL-5990, Argonne National Laboratory, Argonne, Illinois, 1959. [5] R. Fletcher and M. J. D. Powell, A Rapidly Convergent Descent Method for Minimization, Computer Journal 6 (1963), 163168. [6] R. Fletcher and C. M. Reeves, Function Minimization by Conjugate Gradients, Computer Journal 7 (1964), 149154. [7] L. Fox, H. D. Huskey, and J. H. Wilkinson, Notes on the Solution of Algebraic Linear Simultaneous Equations, Quarterly Journal of Mechanics and Applied Mathematics 1 (1948), 149173. [8] Jean Charles Gilbert and Jorge Nocedal, Global Convergence Properties of Conjugate Gradient Methods for Optimization, SIAM Journal on Optimization 2 (1992), no. 1, 2142. [9] Gene H. Golub and Dianne P. OLeary, Some History of the Conjugate Gradient and Lanczos Algorithms: 19481976, SIAM Review 31 (1989), no. 1, 50102. [10] Magnus R. Hestenes, Iterative Methods for Solving Linear Equations, Journal of Optimization Theory and Applications 11 (1973), no. 4, 323334, Originally published in 1951 as NAML Report No. 52-9, National Bureau of Standards, Washington, D.C. [11] Magnus R. Hestenes and Eduard Stiefel, Methods of Conjugate Gradients for Solving Linear Systems, Journal of Research of the National Bureau of Standards 49 (1952), 409436. [12] Shmuel Kaniel, Estimates for Some Computational Techniques in Linear Algebra, Mathematics of Computation 20 (1966), 369378. [13] John K. Reid, On the Method of Conjugate Gradients for the Solution of Large Sparse Systems of Linear Equations, Large Sparse Sets of Linear Equations (London and New York) (John K. Reid, ed.), Academic Press, London and New York, 1971, pp. 231254. [14] E. Schmidt, Title unknown, Rendiconti del Circolo Matematico di Palermo 25 (1908), 5377. [15] Eduard Stiefel, Uber einige Methoden der Relaxationsrechnung, Zeitschrift f ur Angewandte Mathematik und Physik 3 (1952), no. 1, 133. [16] A. van der Sluis and H. A. van der Vorst, The Rate of Convergence of Conjugate Gradients, Numerische Mathematik 48 (1986), no. 5, 543560.