CS321 NumericalAnalysis

Lecture5 SystemofLinearEquations System of Linear Equations

ProfessorJunZhang Professor Jun Zhang DepartmentofComputerScience UniversityofKentucky y y Lexington,KY405060046 October20,2010

SystemofLinearEquations
a11 x1 + a12 x 2 + L + a1n x n = b1 a 21 x1 + a 22 x 2 + L + a 2 n x n = b2 L L L a n1 x1 + a n 2 x 2 + L + a nn x n = bn
whereaij arecoefficients,xi areunknowns,andbi arerighthandsides. Writteninacompactformis
n

a ij x j
j =1

= bi , i = 1, K , n

Thesystemcanalsobewritteninamatrixform Ax = b wherethematrixis a11 a12 L a1n a a 22 L a 2 n 21 A= M O M M a n1 a n 2 L a nn

and

x = [ x1 , x 2 , K , x n ]T , b = [b1 , b2 , K bn ]T

KarlFriedrichGauss(April30,1777 February23,1855) GermanMathematicianandScientist G M th ti i d S i ti t

GaussianElimination
LinearsystemsaresolvedbyGaussianelimination,whichinvolvesrepeated procedureofmultiplyingarowbyanumberandaddingittoanotherrow toeliminateacertainvariable Foraparticularstep,thisamountsto a a ij a ij ik a kj a kk a ik bi bi bk a kk

( k j n)

Afterthisstep,thevariablexk,iseliminatedinthe(k + 1)th andinthelater equations TheGaussianeliminationmodifiesamatrixintoanuppertriangularform suchthataij = 0 foralli > j.Thesolutionofanuppertriangularsystemis pp g y theneasilyobtainedbyabacksubstitutionprocedure
4

IllustrationofGaussianElimination Illustration of Gaussian Elimination

BackSubstitution
Theobtaineduppertriangularsystemis a11 x1 + a12 x 2 + a13 x 3 + L + a1n x n = b1

a 22 x 2 + a 23 x 3 + L + a 2 n x n = b2 a 33 x 3 + L + a 3 n x n = b3 M M M a n 1, n 1 x n 1 + a n 1, n x n = bn 1 a nn x n = bn
Wecancompute

xn =

bn a nn

fromthelastequationandsubstituteitsvalueinotherequationsandrepeat theprocess n 1 xi = bi a ij x j a ii j = i +1 fori = n 1, n 2,, 1

7

ConditionNumberandError
Aquantityusedtomeasurethequalityofamatrixiscalledcondition number,definedas ( A) = A A 1 TheconditionnumbermeasuresthetransferoferrorfromthematrixA to therighthandsidevector If hasalargeconditionnumber,smallerror the right hand side vector b.IfA has a large condition number small error inA mayyieldlargeerrorinthesolutionx = A-1b.Suchamatrixiscalledill conditioned Theerrore isdefinedasthedifferencebetweenacomputedsolutionand theexactsolution ~ e = x x Sincetheexactsolutionisgenerallyunknown,wemeasuretheresidual ~ r = b A x asanindicatorofthesizeoftheerror
9

SmallPivot
x1 + x 2 = 1
x1 + x 2 = 2
forsomesmall.AfterthestepofGaussianelimination x1 + x 2 = 1

(1 1 )x 2 = 2 1
x2 = x1 = 2 1/ 1 1/ 1 x2

Wehave

Forverysmall,thecomputerresultwillbex2 = 1 andx1 = 0.Thecorrect resultsare 1 x1 = 1 1 1 2 x2 = 1 1

10

ScaledPartialPivoting g
Weneedtochooseanelementwhichislargerelativetootherelementsof thesamerowasthepivot LetL = (l1, l2,, ln) beanindexarrayofintegers.Wefirstcomputean arrayofscalingfactorasS = (s1, s2,, sn) where

s i = max a ij
1 j n

(1 i n)

Thefirstrow ischosensuchthattheratio The first row i is chosen such that the ratio |ai,1 |/si is the greatest Suppose isthegreatest.Suppose thisindexisl1,thenappropriatemultipliersofequationl1 aresubtracted fromtheotherequationstoeliminatex1 fromtheotherequations SupposeinitiallyL = (l1, l2,, ln) = (1, 2,, n), ifourfirstchoiceislj,we willinterchangelj andl1 intheindexset,notactuallyinterchangethefirst andthelj rows,toavoidmovingdataaroundthememory , g y
11

Example p
StraightforwardGaussianeliminationdoesnotworkwell(notrobust) x1 + x 2 = 1

x1 + x 2 = 2
ThescalefactorwillbecomputedasS = {1,1}.Inthefirststep,thescale factorratioarray{,1} So the 2 rowisthepivotingrow factor ratio array { 1}.Sothe2nd row is the pivoting row Aftereliminatingx1 fromthe1st equation,wehave (1 ) x 2 = 1 2

x1 + x 2 = 2
Itfollowsthat

1 2 x2 = 1 1 x1 = 2 x 2 1

Wecomputedcorrectresultsbyusingscaledpartialpivotingstrategy
12

GaussianEliminationwithPartialPivoting

13

LongOperationCount g p
Wecountthenumberofmultiplicationsanddivisions,ignoresummations andsubtractionsinscaledGaussianelimination and subtractions in scaled Gaussian elimination The1st step,findingapivotingcostsn divisions Additionaln operationsareneededtomultiplyafactortothepivotingrow foreachofthen 1 eliminations.Thecostisn(n 1) operations.Thetotal costofthisstepisn2 operations Thecomputationisrepeatedontheremaining(n 1) equations.Thetotal costofGaussianeliminationwithscaledpartialpivotingis

n 2 + ( n 1) 2 + L + 4 2 + 3 2 + 2 2 = n( n + 1)( 2n + 1) n3 1 6 3 Backsubstitutioncostsn(n 1)/2 operations

14

TridiagonalandBandedSystems
Bandedsystemhasacoefficientmatrixsuchthataij = 0 if |i j| w.Fora tridiagonal system,w = 2 d 1 c1 x1 b1 a d x b c2 1 2 2 2 a2 d 3 c3 x 3 b3 = M M M M M a n 1 d n 1 c n 1 x n 1 bn 1 a n 1 d n x n bn Generaleliminationprocedure di di
i i

( )c )b b b (
ai 1 d i 1 ai 1 d i 1

i 1

i 1

Thearrayci isnotmodified.Noadditionalnonzeroiscreated y Matrixcanbestoredinthreevectorarrays

15

TridiagonalSystems
Thebacksubstitutionisstraightforward b xn = n dn

bi c i x i +1 xi = di

( i = n 1, K ,1)

Nopivotingisperformed,otherwisetheprocedurewillbequitedifferent duetothefillin(thearrayc willbemodified)

Diagonaldominance: AmatrixA = (aij)nn isdiagonallydominantif
aii >
j =1, j i n

ij

(1 i n)

Foradiagonallydominanttridiagonalsystem,nopivotingisneeded,i.e.,no For a diagonally dominant tridiagonal system no pivoting is needed i e no divisionbyzerowillhappen WewanttoshowGaussianeliminationpreservesdiagonaldominance,i.e., We want to show Gaussian elimination preserves diagonal dominance i e

d i > a i 1 + c i
16

TridiagonalSystems
Thenewcoefficientmatrixhas0elementsattheai splaces.Thenew diagonalelementsaredeterminedrecursivelyas

d1 = d1 a i 1 d i = d i ci 1 (2 i n) d i 1

Weassumethat

di >| ai1 | + | ci |
Wewanttoshowthat

| d |>| ci |
i

Weuseinductiontoprovetheinequality Itisobviouslytruefori =1,as

d1 = d1
17

TridiagonalSystems
Ifweassumethat

| d |>| ci1 |
i 1

WeproveforindexI,as

a i 1 | d i |=| d i ci 1 | d i 1 | ci 1 | | d i | | ai 1 | | d i 1 |

>| ai 1 | + | ci | | ai 1 |=| ci |
Itfollowsthatthenewdiagonalentrieswillnotbezero,theGaussian eliminationprocedurecanbecarriedoutwithoutanyproblem elimination procedure can be carried out without any problem
18

Exampleofapentadiagonal matrix.Itisnearlytridiagonal.The matrixwithonlynonzeroentriesonthemaindiagonal,and thefirsttwobi diagonalsaboveandbelowit the first two bidiagonals above and below it

19

20

LUFactorization1
Asweshowedbefore,ann*nsystemoflinearequationscanbewrittenina matrixformas

Ax= Ax b
wherethecoefficientmatrixA hastheform

a11 a A = 21 M an1

a12 L a1n a22 L a2 n M O M an 2 L a nn

x istheunknownvectorandbistherighthandsideknownvector

Wealsoassume isoffullrank,andmostentriesof arenotzero We also assume A is of full rank and most entries of A are not zero
21

LUFactorization2
Therearetwospecialformsofmatrices.Oneiscalled(unit)lower triangular

0 1 l 21 1 L= M M ln1 ln 2

L 0 L 0 O M L 1

Theotherisuppertriangular

u11 u12 0 u 22 U = M M 0 0

L u1n L u2 n O M L u nn

Wewanttofindapairof and matrices,suchthat We want to find a pair of L and U matrices such that

A= LU

22

Example
Takeasystemoflinearequations

6 2 12 8 3 13 6 4

4 x1 16 6 10 x2 26 = 9 3 x3 19 1 18 x4 34 2

TheGaussianeliminationprocessfinallyyieldsanuppertriangularsystem

ThiscouldbeachievedbymultiplyingtheoriginalsystemwithamatrixM , suchthat such that

6 2 2 4 x1 16 0 4 2 2 x 6 2 = 0 0 2 5 x3 9 0 0 0 3 x4 3

MAx= Mb

23

Example
WewantthematrixM tobespecial,sothatMA isuppertriangular

6 2 2 4 0 4 2 2 =U MA = 0 0 2 5 0 0 0 3
Thequestionis:canwefindsuchamatrixM?Lookatthefirststepofthe Gaussianelimination

6 2 0 4 0 12 0 2

4 x1 16 2 2 x2 6 = 8 1 x3 27 3 14 x4 18 2

Thisstepcanbeachievedbymultiplyingtheoriginalsystemwithalower This step can be achieved by multiplying the original system with a lower triangularmatrix 1 1 24

M Ax = M b

Example
HerethelowertriangularmatrixM1is

1 0 0 0 2 1 0 0 M1 = 1 0 1 0 2 1 0 0 1
Thismatrixisnonsingular,becauseitislowertriangularwithamain diagonalcontainingall1s. Thenonzeroelementsinthefirstcolumnsarethenegativesofthe multipliersinthepositionswhere0swerecreatedintheoriginalmatrix multipliers in the positions where 0s were created in the original matrix

M1 Ax = M1b
25

Example
IfwecontinuetothesecondstepoftheGaussianelimination,wehave

6 2 2 4 x1 16 0 4 2 2 x 6 2 = 0 0 2 5 x3 9 0 0 4 13 x4 21

Thiscanbeachievedwiththemultiplicationofanotherlowertriangular matrix

1 0 0 1 M 2 = 0 3 1 0 2

0 0 0 0 1 0 0 1

Thus,wehave Thus we have

M2M1 Ax = M2M1b
26

Example
Thelaststepis

6 2 2 4 x1 16 0 4 2 2 x 6 2 = 0 0 2 5 x3 9 0 0 0 3 x4 3
withanotherlowertriangularmatrix

And,wehave

1 0 M3 = 0 0

0 0 1 0 0 1 0 2

0 0 0 1

M3M2M1 Ax = M3M2M1b
27

Example
Ifwedefine

M = M3M2M1
ThenM islowertriangularandMA isuppertriangular Since

M3M2M1 A = U
A = M 1U = M 11M 2 1M 31U = LU

theinverseofalowertriangularisagainalowertriangularmatrix.The productoftwolowertriangularmatricesisalowertriangularmatrix product of two lower triangular matrices is a lower triangular matrix

ThisshowsthatwehavetransformedamatrixA intotheproductofalower triangularmatrixandanuppertriangularmatrix ThisprocessiscalledLU factorization orLUdecomposition

28

Example
Thelowertriangularmatrixis

L=M M M
0 1 2 1 1 L= 3 2 1 1 2

1 1

1 2

1 3

whichinthecurrentexampleis

0 1 2 1 1 LU = 3 2 1 1 2

And,wehave

0 0 0 0 1 0 2 1 4 6 10 =A 9 3 1 18 2
29

0 0 6 2 2 4 6 2 0 0 0 4 2 2 12 8 = 1 0 0 0 2 5 3 13 0 0 0 3 6 4 2 1

IllustrationofLUFactorization ll f

30

Example
Theoriginalsystem

Ax= b
canbefactoredinto

LUx= b
Thesolutionprocesscanbeseparatedintotwophases.First,forward h l b d h f d eliminationwithanintermediatevariablez,as

Lz = b

Second,backsubstitutionstep

Ux = z

TheadvantageofLU factorization orLUdecompositionisthatseverallinear systemswiththesamecoefficientmatrixbutdifferentrighthandside vectorscanbesolvedmoreefficiently.Weonlyneeddothefactorization y y once.TheforwardeliminationandbacksubstitutionstepsareO(n2) operations

31

ComputetheInverse
Thesystemoflinearequations canbesymbolicallysolvedas

Ax= b 1 x=A b

ButtheinverseofA isseldomcomputedexplicitly.Thisisbecause performingGaussianeliminationislessexpensive. IftheexplicitinverseofamatrixA isneeded,itcanbeobtainedbyusingLU factorizationofA,notethat

AX = I
(n)

Wecancomputeaseriesoflinearsystemsas

And

LU [ x , x , L , x ] = [ I , I
(1) ( 2) (1) 1 (1) ( 2)

( 2)

,L, I

(n)

A = [ x , x ,L, x ]
(n)
32

SingularValueDecomposition(SVD)
Theeigenvalues andeigenvectorsofamatrixA,are

Ax = x

I.e.,theapplicationofA onthevectorofx isequivalenttoascaling.Here isaneigenvalue andx istheassociatedeigenvector ThesingularvaluesofamatrixA arethenonnegativesquarerootsofthe h l l f h f h eigenvalues of AT A BytheSpectralTheoremforMatrices,canbediagonalized byan By the Spectral Theorem for Matrices AT A can be diagonalized by an orthogonalmatrix,Q,as T 1

A A = QQ

where(orthogonality property) T T

Q Q = Q Q = I,

i.e., Q = Q
T

Thediagonalmatrixcontainstheeigenvalues ofonitsdiagonal AT A
33

SingularValueDecompositionII
Wecanseethat

A AQ = Q
T

SothecolumnsofQ areeigenvectorsof AT A

AT A Ifisaneigenvalue ofandx isacorrespondingeigenvector,then

AT Ax = x
And

|| A ||2 = ( A )T ( A ) = xT AT A = xT x = || x ||2 Ax Ax Ax Ax

Itfollowsthattheeigenvalue isrealandnonnegative. SinceQ isanorthogonalmatrix,itscolumnsformanorthonormal base. Theyareuniteigenvectorsof AT A Ifvj isthejth columnofQ,wehave

A T Av j = j v j

34

GeneH.Golub (February29,1932 November16,2007) AmericanMathematicianandComputerScientist A i M th ti i dC t S i ti t

35

IterativeSolutionMethods
Obtainanapproximatesolutiontoalinearsystemiteratively

a11 x1 + a12 x 2 + a13 x 3 + L + a1n x n = b1 a 21 x1 + a 22 x 2 + a 23 x 3 + L + a 2 n x n = b2 L L L a n1 x1 + a n 2 x 2 + a n 3 x 3 + L + a nn x nn = bn

Createasingleequationforeachvariable

b1 ( a12 x 2 + a13 x 3 + L+ a1n x n ) a11 b (a 21 x1 + a 23 x 3 + L + a 2 n x n ) x2 = 2 a 22 L L b ( a n1 x1 + a n 2 x 2 + a n 3 x 3 + L) x nn = n a nn x1 =

Weassignanarbitraryvaluetoeachofthevariablesintherighthandsideto startiteration
36

Jacobi Iteration JacobiIteration

Werepeatedlyuseeachsetofcompoundvaluesasthenextiterates,and hopetheprocesswillconvergetothetruesolution.(Maystopearlyif necessary.)ThisiterationprocedureiscalledJacobiiterationmethod. Convergenceisnotguaranteedforgeneralmatrices
( x 1k ) ( b1 (a 12 x (2k 1) + a 13 x 3k 1) + L + a1n x (nk 1) ) = a 11 ( ( b 2 (a 21x1k 1) + a 23 x 3k 1) + L + a 2 n x (nk 1) ) = a 22

(k) 2

L
( ( b n (a n1x1k 1) + a n 2 x (2k 1) + a n 3 x 3k 1) + L) = a nn

(k) n

37

CarlGustavJacobi(December10,1804 February18,1851) PrussianMathematician P i M th ti i

38

GaussSeidelIteration
Iftheiterationprocessconverges,itmaybefasterifweusethenew valuesassoonastheyareavailable.ThisgivestheGauss Seidel values as soon as they are available. This gives the GaussSeidel iterationmethod

(k) 1

( b1 (a 12 x (2k 1) + a 13 x 3k 1) + L + a 1n x (nk 1) ) = a 11 ( ( b 2 (a 21x1k ) + a 23 x 3k 1) + L + a 2 n x (nk 1) ) = a 22

x (2k )
L

L
( ( b n (a n1x1k ) + a n 2 x (2k ) + a n 3 x 3k ) + L) = a nn

(k) n

Ingeneral,GaussSeidelisfasterthanJacobi,butthelaterismore attractivetorunonparallelcomputers.Noconvergenceisguaranteed attractive to run on parallel computers No convergence is guaranteed

39

PhilippLudwigvonSeidel(October23,1821 August13,1896) GermanMathematician G M th ti i

SeidelwasastudentofJacobi

40

SORMethod
Around1950,DavidYoungatHarvardUniversityfoundthatiterationcan beacceleratedbyusingaweightedaverageofthesuccessiveapproximate b l db h d f h solutions.Givensomeparameter0 < < 2,weperform
~ x i( k ) = x i( k ) + (1 ) x i( k 1) i = 1,2, K , n

Thisstepsymbolizesthebeginningofmoderniterativemethods Withanarbitraryinitialguess,convergenceofthesebasiciterative methodsisguaranteedif 1.)Thematrixisstrictlydiagonallydominant; 2.)Thematrixissymmetricpositivedefinite,i.e.,A = AT andxTAx > 0

41

DavidM.Young,Jr.(19232008) AmericanMathematicianandScientist A i M th ti i d S i ti t

42

SplittingCoefficientMatrix p g
Thesystemoflinearequationsis

WecansplitthecoefficientmatrixA as

Ax A =b

A=M N

Suchthat isnonsingularand 1 iseasytocompute.Sowehave Such that M is nonsingular and M1 is easy to compute So we have

(M N ) x = b
and

Mx = Nx + b

So

x = M ( Nx + b) = M Nx + M b

Wecanusethisrelationshipforaniterationscheme

x ( k ) = M 1 N ( k 1) + M 1b Nx
43

IterationMatrix
ThematrixM1N iscalledtheiterationmatrix Fortheiterationprocesstobeinexpensivetocarryout,M mustbeeasyto invert Since Wehave We have

x ( k ) = M 1 Nx ( k 1) + M 1b N =M A

x ( k ) = M 1 ( M A) x ( k 1) + M 1b = ( I M 1 A) x ( k 1) + M 1b
Ifx(k) convergestothesolutionx,then g ,

x = ( I M 1 A) x + M 1b
Itfollowsthat

Ax A =b
44

Convergence g
We define the error vector at each iteration step as

e
It follows that

(k )

=x

(k )

e ( k ) = x ( k ) x = ( I M 1 A) x ( k 1) x + M 1b = ( I M 1 A) x ( k 1) ( I M 1 A) x = ( I M 1 A)( x ( k 1) x) = ( I M 1 A)e ( k 1)

If we want the error to become smaller and smaller, we want the iteration matrix

I M A

to be small, in some sense

45

IterationMatrix
Sowewantthematrix

M 1 A I M A
1

tobeclosetotheidentitymatrix.ItistosaythatwewantM isclosetoA Iffact,ifalleigenvelues oftheiterationmatrix If fact if all eigenvelues of the iteration matrix issmallerthanone,thentheiteration

e ( k ) = ( I M 1 A)e ( k 1)
convergestozeroandtheoriginaliterationconverges.Thisrequires

( I M 1 A) < 1
I.e.,thespectralradiusoftheiterationmatrixissmallerthanone guaranteestheconvergenceoftheinducediterationmethod
46

StationaryIterativeMethods y
FortheJacobiiteration

A = D (L + U )

FortheGaussSeidelMethod

A = ( L + D) U

FortheSORmethod,theiterationis

( D L) x ( k ) = [U + (1 ) D]x ( k 1) + b
Theseiterativemethodsarecalledstationarymethods,orclassiciterative methods.Theyconvergeslowly Moderniterativemethods,suchasmultigrid methodandKrylov subspace methodsconvergemuchmorerapidly.Thosemethodswillbestudiedin CS623
47