“S008 3002862 3 Jin Ho Kwak SuNncryYo HONG Linear Algebra BirkHAUSER BosTon ¢ Basel * BERLINxe 164 “93 1994 Jin Ho Kwak ‘Sungpyo Hong Departament of Mathematics Pohang University of Science and Technology Pohang, The Republic of Korea Library of Congress Cataloging-in-Publication Data Kwak, Sin Ho, 1948 Linear Algebra Sin Ho Kwak, Sungpyo Hong. pom. Tnelndes index. ISBN 0-8176-3999.3 (alk. paper) - ISBN 3-7643-3999-5 (alk. apes) 1, Algebras, Linear, I. Hong, Sungpyo, 1948- TL Tite QA184,K94. 1997 susan onsen oe Pred esi es peer 61997 BidbioerBoson sinner IB (Copysight is not claimed for works of U.S. Goverament employees. Allrighisteserved. Nopartof this publication may be reproduced, stored ina retrieval system, ortrinsmitted,inany form orby any mneans, electronic, mechanical, photocopying, recording, ‘or otherwise, without prior permission of the copyright owner, Pecmitsion to photocopy for internal or personal nse of specific clien's is granted by [Birkhiuser Boston for libraries and ther users registered with che Copyright Clearance ‘Center (CCC), provided thatthe base fee of $6.00 per copy, plas $0.20 per page is pad disectly to CCC, 222 Rosewood Drive, Danvers, MA 01923, US.A. Special requests should be addressed directly to Bickhusor Boston, 675 Massachusetts Aveane, Cembridge, MA (2139, USA, ISBN 05176-35993, ISBN 3-7660-9998-3 “Typssting by the aur a LAK Printed and bound by Hamiton Prag, Reasoelaer, NY Printed inthe US.A. 987654321Preface Linear algebra is one of the most important subjects in the study of science and enginsering becanse of its widespread applications in social or natural science, computer science, physics, or economies. As one of the most useful courses in undergraduate mathematics, it hes provided essential tools for industrial scientists. The basic concepts of linear algebra are vector spaces, linear transformations, matrices and determinants, and they serve as an abstract language for stating ideas and solving problems. ‘This book is based on the loctures delivered several years in a sophomore: level linear algebre course designed for science and engineering students. The primary purpose of this book is to give a careful presentation of the besic concepts of linear aigebra as a coherent part of mathematics, and to illustrate its power and usefulness through applications to other disciplines. We have tried to emphesize the computational skilis along with the mathematical abstractions, which heve also an integrity and beauty of their own. The book includes a variety of Interesting applications with meny examples not only to halp students understand new concepts but also to practice wide applications of the subject to such areas as differential equations, statistics, geometry, and physics, Some of those applications may not be centzal to the mathematical development and may be omitted or selected in a syllabus at the discretion of the instructor. Most basic concepts and introductory motivations begin with examples in Euclidean spece or solving» system of Tinear equations, and are gradually exemined from different points of views to derive general principles. For those students who have completed a year of calculus, linear algebra, may be the first course in which the subject is developed in an abstract way, and we often find that many students struggle with the abstraction and miss the applications. Our experience is that, to understand the material, students should practice with many problems, which are sometimes omitted because of a Inck of time. To encourage the students to do repeated practice, vvi Preface we placed in the middle of the text not only many examples but also some carefully selected problems, with answers or helpful hints. We have tried to make this book as easily accessible and clear es possible, but certainly there may be some awkward expressions in several ways. Any criticism or comment from the readers will be appreciated ‘We are very grateful to many colleagues in Korea, especially to the faculty members in the mathematics department at Pohang University of Science and Technology (POSTECH), who helped us over the years with various aspects of this book. For their valuable suggestions and comments, we would like to thank the students at POSTECH, who have used photocopied versions of the text over the past several years. We would also like to acknowledge the invaluable essistance we have received from the teaching assistents who have checked and added some answers or hints for the problems and exercises in this book. Our thanks also go to Mrs. Kathleen Roush who made this book much more legible with her grammatical corrections in the final manuscript. Our thanks finally go to the editing staff of Bizkhiiuser for gladly accepting ‘our book for publication. Jin Ho Kwak Sungpyo Hong E-mail: jinkwak@postech.ac.kr sungpyo@postech.ac kr April 1997, in Pohang, Kerea “Linear algebra is the mathematics of our modern technological world of complex multivartable systems and computers” ~ Alan Tucker ~ “We (Halmos and Kaplansky) share a love of linear algebra. I think it is our conviction that we'll never understand infinite-dimensional operators properly until we have a decent mastery of finite matrices. And we share a philosophy about linear algebra: we think basis.free, we write basis-free, but ‘when the chips are down we close the office door and compute with matrices like fury” ~ Irving Kaplensky —Contents Preface v 4 Linear Equations and Matrices 1 AL Introduction. 6.0.00... e eee cece eee eee 1 1.2 Gaussian elimination . . sees 4 13° Matrices... 0.2... seeeee 2 1.4 Products of matrices. . - w 1.5. Block matrices... . vee 22 16 Inverse matrices... . . “4 17 Elementary maitices . . Lene 27 18 LDU factorization . - . 38 1.9 Application: Linear models... 3B 1.10 Bxercises . 45 2 Determinants 49 2.1 Basie properties of determinant . 49 2.2 Existence and uniqueness... . oA 2.3 Cofactor expansion . . . a) 24 Cramer’stule.......... 65 2.5 Application: Area and Volume 68 2.6 Exercises . 2.7 3 Vector Spaces 7% 3.1 Vector spaces and . 75 3.2 Bases... +. 81 3.3 Dimensions ............ 8B 34 Row and columa spaces 6... ay 35 Rank and nullity ........ 00. CONTENTS 3.0 Bases for subspaces... . bene 104 3.7 Invertibility . - . cee eee eee ee MO 3.8 Application: Interpolation |... | soe UB 3.9. Applicatio: eee eee eee ee US BD Exercises oo. ose ee eee eee eee uz Linear Transformations 121 4.1 Introduction... . . IETS a7 ee 132 eee 185 4.2 Invertible linear transformations»... .. 4.3. Application: Computer graphics. . 4.4. Matrices of linear transformations . 4.5. Veotor spaces of linear transformations... . 4.8 Change of bases.....--.. 4.7 Similarity... 143 146 4.8 Dual spaces... . sees 152 49 Exercises»... ne 156 Inner Product Spaces 161 5.1 Inner products, oe 161 5.2 The lengths and angles of vectors - - 164 5.3 Matrix representations of inner products... 0.4... 4. 167 5.4 Orthogonal projections... .. . sees aw 5.5 The Gram-Schmidt orthogonalization .. . . ur 5.6 Orthogonal matrices and transformations 181 5.7. Relations of fundamental subspaces . 185 5.8 Least square solutions... . . eee eee eae 187 5.8 Application: Polynomial approximations... ... ss... + 192 5.10 Orthogonal projection matrices : 5.1 Bxercises 0. ee eee eee Rigonvectors and Eigenvalues 209 6.1 Introduction . : 6.2 Diagonalization of matrices... 6.3 Application: Difference equations . . - 26 221 6.4 Application: Differential equations I 226 68 Application: Diferential equations II cee 230 6.6 Exponential matrices wees 235 6.7 Application: Differential equations I... ........ 6. 240 8.8 Diagonalization of linear transformations. ,.......... 243CONTENTS: x G9 Exercises 2. ee BME 7 Complex Vector Spaces 251 7.1 Introduction vce cette eee e cece ees s 281 7.2 Hermitian and unitary matrices... es 259 73 Unitetily diogonalizable matrices... 4-045 2 0+ s « 268 74 Normal matrices 6.0 ee 268 7.5 The spectral theorem... 0... verve es OTL 78 Byercieess errr coo. 216 8 Quadratic Forms 81 Introduction... 0... eens 8.2 Diagonalizetion of a quedratic form... . 83 Congruence relation... .. 84 Extrema of quadratic forms . 85 Application: Quadratic optimization . 8.6 Definite forms . 8.7 Bilinear forms . 88 Exercises. 9 Jordan Canonical Forms B17 91 Introduction... eee ee - IT 9.2 Generalized eigenvectors... 6. ee eee eee BT 9.3 Computation of e+... 2... ~ 333 9.4 Cayley-Hamilion theorem .. . vs B37 9.5 Exercices - 340 Selected Answers and Hints 343 Index 365Linear AlgebraChapter 1 Linear Equations and Matrices 1.1 Introduction One of the central motivations for linear algebra is solving systems of linear equations. We thus begin with the problem of finding the solutions of system of m linear equations in unknowns of the foilowing form: ane) + ent + + aint, = by nz +t + + amin = by ant, + on Vans : 2 he ok Amit + Omg +o) + Oma = Oey where 21, 22, ..., #y ave the vuknowne and ay;'s and by’s denote constant (real or complex) numbers, ‘A soquence of numbers (61, 82; -++, $2) 18 called a solution of the system if m = 81, 29 = 5p, ..., Sn = Sx Satisfy each equation in the system simultaneously. When by = 2 = +++ = by = 0, we say that the system is homogeneous. The central topie of this chapter is to examine whether or not @ given system has a solution, and to find a solution if it has one. For tnstance, any homogeneous system always hes at least one solution 2; = 2 Zp = 0, called the trivial solution. A natural question is whether such a homogeneous system has a nontrivial solution, If so, we would like to have a systematic method of finding all the solutions. A system of linear equations is said to be consistent if it has at least one solution, and inconsistent if 12 (CHAPTER 1. LINEAR EQUATIONS AND MATRICES it has 20 solution. ‘The following exemple gives us an idea bow to answer the above questions, Example 1.1 When m =n = 2, the cyatem reduces to two equations in two unknowns # and y: ar + by = a age + by = Geometrically, each equation in the system represents a straight line when we interpret and y 28 coordinates in the y-piane. Therefore, a point P = (x,y) is a colution if and only if the point P lies on both lines. ‘Hence there are three possible types of solution set: (J) the empty set if the lines are parallel, (2) only one point if they intersect, (3) a straight line: i.e., infinitely many solutions, if they coincide, ‘The following examples and diagrams illustrate the three types: Case (1) Case (2) Case (3) (25 (a sy = 2 dy = y y a = e € ‘To decide whether the given system has ¢ solution and to find a general method of solving the system when it has a solution, we repeat here a well- ‘mown elementary method of elimination and substitution. Suppose first that the system consists of only one equation ax + by = ¢. ‘Then the aystem has ether infinitely many solutions (i. pointe on the straight line « = —4y + £ or y = —2 + § depending oa whether a % 0 or 40) or no solutions when a = b= 0 and c# 0.LL. INTRODUCTION 3 ‘We now assure that she system has two equations representing two lines in the plane. Then clearly the two lines are parallel with the same slopes if and only if az = Aa, and by = Mh for some A 7 0, or aibe ~ azbi = 0. Furthermore, the two lines either coincide (infinitely many solutions) or are distinct and parallel (no solutions) ecoording to whether oy = Ae: holds or not. Suppose now that the lines are not parallel, or ayby — agby # 0. In thi case, the two lines cross at a point, and hence there is exactly one solution: For instance, if the system is homogeneous, then the Hines cose at the origin, 80 (0,0) is the only solution. For a nonhomogeneous system, we may find the solution as follows: Express z in terms of y ftora the first equation, and then substitute it into the second equation (i.e., eliminate the varinble from the second equation) to get @- 2h =a- 2a oy ay Since arby — agh #0, this can be solved as 102 — 2201 ayb2 — aaby” which is in turn substitnted Sn one of the equations to find w and give a ‘complete solution of the system@™Ih detail, the process can be summarized as follows: (1) Without loss of generality, we may assume a; 7 0 since otherwise we can interchange the two equations, Thea the variable = can be eliminated from the second eqnation by adding - y times the first equation to the second, to get {“ + by = 4 a2 (a- Bay = @- Ba (2) Since a;b2—a2b # 0, y can de found by multiplying the second equation ‘an a ne ber ——“! . to get by a nonzero number <> pr bo wet a ae + by y = Seca ayby — anh;4 CHAPTER 1. LINEAR EQUATIONS AND MATRICES (8) Now, « is solved by substituting the volue of y into the firet equation, and we obtain the solution to the problem: = bashes aaxba — aby y 102 ~ 290 ayby — agby” Note that the condition ary — axby 0 is necessary for the system to have only one solution. a In this example, we have changed the original system of equations into a simpler one sing certain operations, from which we can get the solution of the given system. That is, if (7,4) satisfies the original system of equations, then and y must satisfy the above simpler system in (3), and vice versa, It is suggested thet the readers examine a system of three equations in thres unknowns, cach equation representing a plane in the 3-dimensional space R°, and consider the various possible cases in a similar way. Problem 1.1 For a system of three equations in three unknowns out + ony + one = hh one + omy + oz = by out + ony + cas = bss describe all the possible types of the solution set in RS. 1.2 Gaussian elimination As we have seen in Example 1.1, a basic idea for solving a system of linear equations is to change the given system into a simpler system, keeping the solutions unchanged; the example showed how to chenge @ general system to a simpler one. In fact, the main operations used in Example 1.1 are the following three operations, called elementary operations: (1) multiply a nonzero constant throughout an equation, (2) interchange two equations, (8) chenge an equation by adding a constant multiple of another oquation.1.2, GAUSSIAN ELIMINATION: 5 After applying a finite sequence of these elementary operations to the given system, one can obtain a simpler system from which the solution ean be derived directly. ‘Note also that each of the three elementary operations hes its inverse operation which is also an elementary operation: (1)' divide the equation with the same nonzero constant, (2)' interchange two equations again, (3)' change the equation by subtracting the same constant multiple of the same equation. By applying these inverse operations in reverse order to the simpler system, one can recover the original system. This meens that a solution of the original system must also be a solution of the simpler one, and vice versa, ‘These arguments can be formalized in mathematical language. Observe that in performing any of these basic operations, anly the coefficients of the variables are involved in the calculations and the variables 21, ..., tm and the equel eign “=” ore simply repeated. Thus, keeping the order of the variables and *=" in mind, we just extract the coeflicients only from the equations in the given system and make a rectangular atray of sumbers: a Gg 1 ain bt fax ag an Be Ort me °** mn bom ‘This matrix isWatled the augmented matrix for the systern. ‘The term ‘mairiz means just any rectangular array of numbers, and the numbers in this array are called the entries of the matrix, To exploin the above operations in terms of matrices, we first introduce some terminology even though in the following sections we shall study matrices in more dete!l. Within « matrix, the horizontal and vertical subarrays ay fou og =~ ein bi) ond | inj are called the i-th row (matrix) and the j-th column (matrix) of the aug- mented matrix, respectively. Note that the entries in the j-th column are6 CHAPTER 1. LINEAR EQUATIONS AND MATRICES just the coefficients of j-th variable a, s0 there is ¢ correspondence betwoon columns of the matrix and variables of the systesa. Since each row of the augmented metrix contains all the information of the corresponding equation of the system, we may deal with this augmented matrix instead of hendling the whole system of Jinear equations. ‘The elementary operations to a system of linear equations are rephrased as the elementary row operations for the sugmented matrix, as follows: (1) nmilliply @ nonzero constant throughout arom, (2) interchange two rows, (8) chonge a row by adding 2 constant multiple of another row. ‘The inverse operations are (2)! divide the row by the same constant, (2)! interchange two rows again, (3)' change the row by subtracting the same constant multiple of the other rom. Definition 1:1 Two augmented matrices (or systems of linear equations) are said to be row-equivalent if one can be transformed to the'other by a finite sequence of elementary row operations. Uf a matrix B can be obtained from a matrix A in this way, then we ‘can obviously recover A from B by applying the inverse elementary row operations in reverse order. Note again that an elementary row operation does not alter the solution of the system, and we can formalize the above argument in the following theorem: ‘Theorem 1.1 If two systerns of linear equations are row-equivalent, then they have the same set of eolutions, ‘The general procedure for finding the solutions will be illustrated in the following exaraple: Example 1.2 Solve the system of linear equations: yrds = 2 et Wy + 2 3 Be + dy + 62 = -1.1.2. GAUSSIAN ELIMINATION 7 Sclution: We could work with the augmented matrix alone. However, to compare the operations on systems of linear equations with those on the augmented matrix, we work on the system and the augmented matrix in parallel. Note thai the associated sugmanted matrix of the system is 024 2 122 3). 346-1 (1) Since the coefficient of z in the first equation is zero while that in the second equation is not zero, we interchange these two equations: z+ dy + 3 1 3 ty + 4 = 2 0 2]. Bn + 4y + 62 = -1 8 =a (2) Add ~3 times the first equation to the third equation: 3 122 3 2 o 24 2]. 0-20 dy +b de = %y = -10 ~0 { z+ ty + ‘Tho coefficient 1 of the first unknown « in the firet equation (row) is called the pivot in this first elimination step. ‘Now the second and the third equations involve cnly the two unknowns y end z, Leave the first equation (row) alone, and the same elimination procedure can be applied to the second end the third equations (rows): The pivot for this step is the coefficient 2 of y in the seeond equation (row}. ‘To eliminate y from the last equation, (3) Add 1 times the second equation (row) to the third equation (row): t+ yt ee = 3 122 3 My + 42 = 2 024 2]. 4 = -8 004 -8 ‘The elimination process done so far to obtain this result is called a for- ward ellmination: ‘.c., elimination of « from the last two equations {rows) ‘and then elimination of y from the last equation (row). Now the pivots of the second and third rows are 2 and 4, respectively. ‘To make these entries 1,8 CHAPTER 1. LINEAR EQUATIONS AND MATRICES (4) Divide each row by the pivot of the row: w+ yb 2 3 122 3 1 o12 1]. -2 oo1-2 vit z ‘The resulting matrix on the right side is called a row-echelon form of the matrix, and the 1's at the leftmost entries in each row are called the leading V's. ‘The process so far is called a Gaussian elimination. ‘We now want to eliminate numbers above the leading 1 (5) Add —2 times the third row to the second and the first rows, a+ yy = 7 120 7 y = 5 o10 5 z= -2 001-2 (6) Add —2 times the second row to the first row: = -3 100 -3 y = 5 o10 3s]. z= -2 001-2 ‘This matrix is called the reduced row-echelon form, The procedure to get this reduced row-echelon form from a row-echelon form is called the back substitution. The whole process to obtain the reduced row-echelon form is called a Gauss-Jordan elimination. ‘Notice that the corresponding system to this reduced row-echelon form 4s row-equivalent to the otiginal one and is essentially a solved form: i.e., the'solution is z= ~3, y= 5, z= —2, 5 nad Jn gereral, 9 matrix of row-echolon form satisfies the following prop- erties. (1) The first nonzero entry of each row is 1, called a leading 1. (2) A row containing only 0's should come after all rows with some nonzero entries. (8) ‘The leading 1's appear from left to the right in successive rows. That is, the leading 1 in the lower row oceurs farther to the right then the leading 1 in the higher row. ‘Moreover, the matrix of the reduced row-echelon form satisfies1.2, GAUSSIAN ELIMINATION 9 (4) Eech column thet contains a leading 1 has geros everywhere else, in addition to the above three properties. Note that on augmented matrix has only one reduced row-echelon form while it may have many row-echelon forms. In any case, the number of nonzero rows containing leading 1's is equal to the number of columns con- taining leading 1's. The variables in the system corresponding to columns: with the leading 1’s in a row-achelon form are called the basic variables. In general, the reduced row-echelon form U' may have columns that do not con- tain leading 1's. The vaziables in the system corresponding to the columns without leeding 1's are called free variables. Thus the sum of the number of basic variables and that of free variables is precisely the total number of variables. For example, the first two matrices below are in reduced row-echelon form, and the lest two just ia row-echelon form. 1126 O10). oo13 100 01506 1232 o210[,;90011],/o14 5], 000 ooo0e 0017 Notice that in en augmented matrix [A b], the last column b does not correspond to any variable. Hence, if we consider the four matrices above as augmented matrices for some systems, then the systems corresponding +o the first and the last two augmented matrices have only basic variables ‘but no free variables. In the system corresponding to the second augmented matcix, the second and the forth variables, xz and a, are basic, and the firet and the third variables, x1 and :tg, are free variables, ‘These ideas will be used in later chapters. In summary, by applying a finite sequence of elementary row operations, the augmented matrix for e system of linear equations can be changed to its reduced row-echelon form which is row-equivalent to the original one. From the reduced row-echelon form, we can decide whether the system has a solution, and find the solution of the given system if it; has one. Example 1.3 Solve the following system of linear equations by Ganss- Jordan elimination. fateck 3 20; + Gr, — Bag + dm = 18 zp + a3 + 3 = 10.10 CHAPTER 1. LINEAR EQUATIONS AND MATRICES Solution: The augmented matrix for the system is 13-20 3 26-24 8]. @1 13 00 The Gaussian elimination begins with: (1) Adding —2 times the first row to the second produces [bE eta} (2) Note that the coefficient of za in the second equation is zero and that in the third equation is not. Thus, interchanging the second and the third Tows produces . 13-20 3 01 1310 00 242 (3) The pivot in the third row is 2. ‘Thus, dividing the third row by 2 produces a row-echelon form | ‘This is a row-echelon form, and we now continue the back-substitution: (4) Adding —1 times the third row to the second, and 2 times the third row to the first produces (S) Finally, adding 3 times the second row to the first produces the reduced row-echelon form: How mre BSc 20 13 1 12 cor x con ere Hoo oak cor cre Hoe1.2. GAUSSIAN ELIMINATION Wn ‘The corresponding system of equations is a + tm + % 3 + 24 Since #1, 2, and 3 correspond to the columns containing leading 1's, they aro the basic variables, and ay is the free variable. Thus by solving this system for the basic variables in terms of the free vatiable 24, we have the system of equations in e solved form: a eA za By sssigning an arbitrary value ¢ to the free variable 24, the solutions con be written as 3 - em 4- % 6 Bes, (ei, a2, 2, 24) = (8-4, 4-8, 6-24, 1), for auy t ER, where R denotes the set of real numbers. a Remark: Consider » homogeneous system ang) + ope: + ob ante = 0 anc + ont. + + ante = 0 GmiZ1 + Gm2te + + Gmntn = 0, swith the number of unknowns gceater than the number of equations: that is, m < m, Since the number of basic variables cannot exceed the number of rows, a free variable always exists as in Example 1.3, 90 by assigning an arbitrary value to each free variable we can always find infinitely many: nontrivial solutions. Problem 1.2 Suppose that the augmented matrix for e system of linear equations ‘has been reduced to the reduced row-echelon form below by elementary row operae tiors. Solve the aystems: 100 5 1004-1 @lo10 2}, @lor02 6 ooo 4 0013812 CHAPTER 1. LINEAR EQUATIONS AND MATRICES We note that if a row-echelon form of an augmented matrix has a row of the type [0.0 --- 0] with b #0, then it represents an equation of the form 021 + O02 + +++ + Ou, = 5 with 5 x 0, In this case, the system ‘has no solution. If b = 0, then it has a row containing only 0's that can be neglected. Hence, when we deal with a row-echelon form, we may assume ‘that the zero rows are deleted. Note also that, as in Example 1.3, if there exists at least one free variable in the row-echelon form, then the system ‘has infinitely many solutions. On the other hand, if the eystam hus no free variable, the system has a unique solution. “To study systems of linear equations in terms of metrices systematically, ‘we will develop some general theories of matrices in the following sections. Problem 1.3 Solve the following systems of equations by Gaussian elimination. ‘What ate the pivots? -2 by + (hee ty ee de + by + ar wt ety “Su — We + oy + 2 O) Gy = ite + By — Be — whe ty 2 Q4 42 - My + 32 Se = By = 6 3 1 1 L Problem J.4 Determine the condition on by s0 that the following system has a s0- Tntion. c+ ky + 6 e+ Sy ~ 2 (4 2 = by = 2% 4m - y + & Boy + Ae = be fat yt oe bre bb by bs. 1.3. Matrices Rectangular arrays of real numbers arise in many real-world problems. His- ‘torically, it was the English mathematician A. Cayley who first introduced the word “matrix” in the year 1858. Tho meoning of the word is “that within which something originates,” and he used matrices simply as a source for rows and colurns to form squares. In this section we are interested only in very basic properties of such matrices. Definition 1.2 Aum by n (written mxn) matrix is a rectangular array of numbers arranged into m (horizontal) rows and n (vertical) columns. The13. MATRICES 13 size of a matrix is specified by the number m of rows and the number n of columns. In general, a matrix is written in the following form: laijhmxns in Ang 0 Br or just A = {oy} if tho size of the matrix is clear from the context. The number ai is called the (i,3)-entry of the matrix A, and can be also written as a3 = [A]. ‘An mx1 matrix is called 2 column (matrix) or sometimes a column, vector, and a 1xn matrix is called a row (matrix), or a row vector. ‘These special cases are important, as we will see throughout the book. We -vill generally use capital letters like A, B, C for matrices and small boldface letters like x, y, 2 for columns or row vectors. Definition 1.3 Let A = (ay] be an mxn matrix. The transpose of A is ‘the nxm matrix, denoted by AT, whose j-th column is taken from the j-th row af A: That is, AT = [bg] with by = ay 12 For example, if A [: i gamar-[2 ‘ Tn particular, the transpose of a column vector is a row vector and vice versa, For example, for an n x | cohimn vector au 7 fn its transpose x" = [ay Z2 +++ tn] is a row vector. Definition 1.4 Let A= [ey] be an m xn matrix. (A) A iscalled a square matrix of order n if m =n. In the following, we aamume that A is a square matrix of order n.uw CHAPTER 1. LINEAR EQUATIONS AND MATRICES (2) The entries a1, a22, ..., dan are called the diagonal entries of A. (8) A is called a diagonal matrix if all the entries except for the diagonal entries are zero. (4) Ais calied an upper (lower) triangular matrix if all the entries below (above, respectively) the diagonal are zero. ‘The following matrices U and L are the general forms of the upper tri- angular and lower triangular matriecs, respectively: 2 013 + An my O ve O 0 am + am on om + 0 v=| . bs : 0 0 Oo ny Og + on Note that @ matrix which is both upper end Jower triangular must be a diagonal matrix, and the transpose of an upper (lower) triangular matrix is lower (upper, respectively) triangular matrix. Definition 1,5 Two matrices A= [a,j] and B = [fj] are said to be equal, written A = B, if they have the same size and corresponding entries are equal: te, aig = yg for all {and j. ‘This definition allows us to write matrix equations. A simple exemple is {AT)T = A by definition. Let Mmsa(R) denote the set of all m xn matrices with entries of real numbers. Among the elements of Mmxn(R), We can define two operations, called scalar multiplication and the sum of matrices, as follows; Scalar multiplication: Given an m xn matrix A= |ayj] and a scalar k (which is simply a real number), the scelar multiplication kA of k and A is defined to be the matrix kA = [kaij): é¢., in an expanded fors: [2 . "| [2 + kai, Qa Orn J Lom => kan Sum of matrices: If A= [ay] and B ~ [by] are two matrices of the same size, then the sum A+ B is defined to be the matrix A+B = (aij + by)13, MATRICES 15 fe., in an expanded form: an ain) Pin os bm] f onthn : pyaye =} oG ent + rm | Lb ++ Bram | a + Bat Note that matrices of different sizes cannot be added. Tt is quite clear that A+ A= 2A, and A+ (A+A) = (A+A)+A = 8A. Thos, inductively wo define nA = (n—1)A+A for any positive integer n. IF B is any matrix, then —B is by definition the multiplication (—1)B. Moreover, if A and B are two matrices of the same size, then the difference A~ B is by definition the sum A+(—1)B = A+(—B). A matrix whose entries are all zero is celled fa zero matrix, denoted by the symbol 0 (or Omxn When we emphasize the number of rows and columns). Clearly, matrix addition has the same properties as the addition of real numbers, ‘The zeal numbers in the context here are tredisionally called scalars even though “numbers” is a perfectly good name and “scalar” sounds more technical. The following theorem lists some basic rules of these opera tions. aint bin Gon + ‘Theorem 1.2 Suppose that the sizes of A, B and C are the same. Then the following rules of mairis arithmetic are vatid: (1) (A+B) +C=A+(B+C), (written as A+B +C) (Assoclativity), (2) At0=044=4, (3) A+(-A) =(-A)+ 4 =0, () AtB=B+A, {Commutativity), (8) K(A+B) =kALAB, (8) (EE E)A=kA+EA, (7) (KEJA = (EA). Proof: We prove only (5) and the remaining are left for exercises. For any 6a, [A+ Bly = AIA + Bly = b(lAly + (Bly) = [Aly + Bly. * Consequently, k(A+ B) =kA+ kB. o16 CHAPTER 1. LINEAR EQUATIONS AND MATRICES Definition 1.6 A square matrix A is eaid to be symmetric if AT = A, or skew-symmetric if A? = —A. For example, the matrices A and B below o 12 » B=/|-1 03 -2 30 aze symmetric and skew-symmetric, respectively. Notice here that all the agonal entries of a skew-symmetric matrix must be zero, since ay = ~a By a direct computation, one can easily verify the following rules of the ‘transpose of matrices: A= oan awe wee ‘Theorem 1.3 Let A and B be mxn matrices. Then (kA) = RAT, ond (A+ BY = AT + BP. Problem 1.5 Prove the remaining parts of Theorem 1.2. Problem 1.6 Find a matrix B such that A4-B7 = (A—B)?, where 2-30 A=| 4-13 -1 of Protlem 1.7 Bind a, b, ¢ and d such that a b)ljfe 87, [246 aso [oa] ost] + [Sta 7S 1.4 Products of matrices ‘We introduced the operations sum and scalar multiplication of matrices in Section 1.3. In this section, we introduce the product of matrices. Unlike the sum of two matrices, the product of matrices is ¢ little bit more complicated, in the sense that it is defined for two matrices of different sizes or for square matrices of the same order. We define the product of matrices in three steps:14. PRODUCTS OF MATRICES a (1) For a1 x2 row matrix a = [ay --- o,] end an nx L column matrix x = [1 «+ eal”, the product ax is a 1x1 matrix (i.e. just « number) defined by the rule n = 2 ax=laz az ++ as} | fer tae + on = [Sooe E mt Tn Note that the number of columns of the first matrix must be equal to the numnber of rows of the second matrix to have entrywise multiplications of the entries. (2) Bor on rm xn matrix au a ain a ay a + Gin ay A=]. me ; Gl Oma 7 Ome an where a;’s denote the row vectors, and for ann x 1 column matrix % = {xi «++ eal”, the product Ax is by definition an m x 1 matrix defined by the rule: ax ag Axa]. |= , am or in an expanded form a a2 am |] fo aye, + argt2 ++ + Ointe an am + my | | ta 42) + arava +++ Oonte Gm, tmz > Oma | | tn may + Oya22 “b= Connitin which is just an m x 1 column matrix of the form [by bz --- mls ‘Therefore, for systom of 7m: linear equations in n unknowns, by writing the n unknowns as an n> 1 column matrix x and the coefficients as an m xn matrix A the system may be expressed as a matrix equation Ax = b. Notice that this Looks just like the usual linear equation in one variable: az ~ b.18 CHAPTER 1, LINEAR EQUATIONS AND MATRICES {8) Product of matrices: Let A be on mxn matrix ond B an nxr matrix, The product AB is defined to be an m x r matrix whose columns are the products of A and the columns of B in corresponding order. ‘Thus if A is mxn and B is nxr, then B has r columns and each column of B isan nx1 matrix. If we denote them by b!, ..., b',or B= [b! --- br], then AB = [ Abe ab? abr | arb! ayb? ay be ag! agb? a,b? amb! amb? - abt which is an m xr matrix. Thoreforo, the (j,j)-entry [AB]y of AB is the i-th entry of the j-th column matrix aybf sbi avi =| amb! + 15 fy it is the product of é-th row and i.e, for i= 1, j-th column of mand j (ABs = ab? = Saintes. = Example 1.4 Consider the matrices 23 1 20 [i 31 8 [3 -1 o}- ‘The columns of AB are the product of and each column of B: 23)f1] _ [2a43-5]_ [az ao][5] ~ [4a14+0-5]7| 47° 23)][ 2] _ [2-2438-]_[1 4 0]|-1} ~ [4240-2] 7] 8]° (23s) - 884]-(3] A14. PRODUCTS OF MATRICES 19 Therefore, AB is 23 i 20 wid 40 5 -1 0 430 Since A is a 2x2 metsix and B is a 2x3 matrix, the product AB is a 2x3 matrix. If we concentrate, for example, on the (2,1)-entry of AB, we single cout the second row from A and the frst column from B, and then we multiply corresponding entries together and add them up, ie, 4-140-5=4. OG Note that the product AB of A and B is not defined if the number of columns of A and the number of rows of B are not equal. Remark: In step (2), we could have defined for a 1x n row matrix 4 and annxr matric B using the same rule defined in step (1). And then in step (8) an appropriate modification produces the same definition of the product of matrices. We suggest the readers verify this (see Example 1.6), The identity matrix of order n, denoted by Jn (or J if the order is clear from the context ), is a diagonal matrix whose diagonal entries aze all 1, i.e, 1 0-0 hel? t Z 0 o on By a direct computation, one can easily see that Alp nxn matrix A, ‘Many, but not all, of the rules of arithmetie for real or complex numbers also hold for matrices with the operations of scalar multiplication, the sum and the product of matrices. The matrix Omxn Plays the role of the number 0, and J, that of the number 1 in the set of real numbers. ‘The rule that does not hold for matrices in general is the commutativity AB = BA of the product, while the commutativity of the matrix cum A+ B= B+ A does hold in general. The following example illustrates the noncommutativity of the product of matrices. Ind for any 1 0 Example 1.5 Let A [20 CHAPTER 1. LINEAR EQUATIONS AND MATRICES "Thus the matrices A and B in this example satisfy AB ¢ BA. a ‘The following theorem lists some rules of ordinary arithmetic that do hold for matrix operations. ‘Theorem 1.4 Let A, B, C be arbitrary matrices for which the matrix op- erations below are defined, and let k be an arbitrary scalar. Then (1) A(BC) = (AB)C, (written os ABC) (Associativity), (2) AB +0) = AB + AC, end (A+ B)C = AC + BC, (Distributivity), (3) 1A Al, (4) K(BC) = (KB)C = B(KC), (8) (AB)T = BTAT. Proof: Each equality can be shown by direct calculations of each entry of both sides of the equalities. We illustrate this by proving (1) only, and leave the others to the readers. Assume that A = [ay] is an mxn matrix, B = [bye] is an mxp matrix, and C’ = [eq] is a pr matrix. We now compute the (1, f)-entry of each side of the equation. Note that BC is an nxr matrix whose (i, j)-entry is, {BCliy = Dhan bincag Thus . ao ae [ABO gy = Lo eulBChs = DO ein 7 bureag = OY ainbercag- i mit me Similarly, AB is an mxp matrix with the (i, j)-entry [AB]y = Dear Gindjs and 2 pon ae KAB)Cly = SAB)acsy = D2 eiubprcry = LY oanbrcry- el jot si ‘This clearly shows that [A(BC)],y = [(AB)C}, for all i, j, and consequently A(BC) = (ABYC as desired. a Problem 1.8 Prove ot disprove: If A is not a zero matrix end AB = AC, then Bae. Problem 1.9 Show that any triangular matrix A satisfying AAT = A"A Is a disg- onal matrix.14, PRODUCTS OF MATRICES aL Problem 1.10 For a square maitix A, show thet (1) AAT and A+ AP are symmetric, (2) A- AT is skew-symmetic, and (8) Accan be expressed as the sum of eymmetric part B— }(A+AT) and ckew- aymmetrie part C = #(A—AT}, s0 that A= B+C. ‘Av an application of our resuilte on matrix operations, we shall prove the following important theorem: ‘Theorem 1.5 Any system of linear equations has either no solution, exactly one solution, or infinitely many solutions. Proof We have already seen that a system of linear equations may be written as Ax =b, which may have no solution or exactly one solution. Now sssume that the system Ax = b of linear equations has more than one solution and let x; and x2 be two different solutions so that Ax1 = b and Ax, = b. Let xp = %1—X2 #0. Since Ax is just » particular ease of a matrix product, Theorem 1.4 gives vs Alx1 + kxg) = Axi + kAxo = b+ k(Ax — Axa) = b, for any real number &. This says that x1 + kX9 is also a solution of Ax = b for any k. Since there are infinitely many choices for k, Ax = b has infinitely many solutions. a Problem 1.11 For which values of a does exch of the following systems have no solution, exectly one colution, or infinitely many solutions? a+ Gy ae = ar + 2~ . son + Qe2 CHAPTER 1. LINEAR EQUATIONS AND MATRICES 1.5 Block matrices Jn this section we introduce some techniques thst will often be very helpful in manipulating matrices. A submatrix of a matrix A is a matrix obtained from A by deleting certain rows and/or columns of A. Using a system of horizontal and vertical lines, we can partition a matrix A into submatrices, called blocks, of A as follows: Consider a matrix on a2 aig | a4 A=| on om aza| aus | , 51 432 09 | Ose divided up into four blocks by the dotted lines shown. Now, if we write aaef™ 22 2], a e[o], u [zs azz zy m= | ase f= [om a on J, 4=[ oa J, then A can be written as 4n Au 4 [ An An | called a block matrix. ‘The product of matrices partitioned into blocks also follows the matrix product formula, as if the Ay were numbers: An Aw By By]. A An Aga 3 | Bu Bus |* AB = | 4n8n+4nBn Anbu + AnBa AnBy + AnBy An By + AnBu |" provided that the number of colurnns in Ay, is equal to the number of rows in Bij. ‘This will be true only if the columns of A are partitioned in the same way as the rows of B. It is not hard to see that the matrix product by blocks is correct. Sup- pose, for example, that we have 9 2x3 matrix A and partition it as ai aye | aya Ay Ay oa sal-[ " |. 3, on | as * A1.5. BLOCK MATRICES 23 and a 3x2 matrix B which we partition as by ba a-[f m2] -[2:]: far bse ” ‘Then the entries of C= [eij] = AB aze = (aarbry + oiabay) + aisles - ‘The quantity ajrbi; + oxzbay is simply the (i)-entry of Ay: Puy if i < 2, and ‘the (éj)-ontry of Aa Bu ifi = 3. Similarly, aigba, is the (¢ f)-entry of Ai2Ba) Afi $9, and of AgpBa: if i= 3. Thus AB can be written as apa [Ox |_| AnBn+4nBa G9 | ~ | AB + AmBa In particular, if an m x n matrix A is partitioned into blocks of column vectors: ie. A= [al a? -.. at], where each block a/ is the j-th column, then the product Ax with x= {2; --- p|” is the sum of the block matrices (or column vectors) with coeficients 2;'s: malt mat +--+ 298", ea where xyai = ayfeay an5 +++ Gnj)”. Example 1.6 Let A be an m xn matrix partitioned into the row vectors A, 89; «1, My a8 its blocks, and let B be ann xr matrix so that thelr product AB is well-defined. By considering the matrix B as a block, the product 4B can be written aL aB abt ayb? ++. ayb™ mB bt ach? -- “agb” ape| laa [ Ppa an AmB amb! agb? +++ amb” where bi, b?, ---, bY denote the columns of B. Hence, the row vectors of AB are the products of the row vectors of A and B24 CHAPTER 1. LINEAR EQUATIONS AND MATRICES Problem 1.12 Compute AB using block multiplication, where 1 ol 2 12}1 0 yool2 A Sato 1), axle Be OO) =I 3-2/1 1.6 Inverse matrices ‘As wo saw in Section 1.4, a system of linear equations can be written as Ax = b in matrix form. This form resembles one of the simplest linear equation in one variable ex = b whose solution is simply z = ob when 20. Thus it is tempting to write the solution of the system as x= Ab. However, in the case of matrices we first have to have precise meaning of 4-1, To discuss this we begin with the following definition. Definition 1.7 For an m xn matrix A, ann xm matrix B is called a left inverse of A if BA = Iq, and ann x m matrix C is called a right inverse of A if AC = In. Example 1.7 From a direct calculation for two matrices 1-3 4e[b oT] oo -1 5], 207 wa 9 ‘Thus, the matrix B is a right inverse but not a left inverse of A, while A is a left inverse but not « right inverse of B. Since (AB) = BTA? and J? = J, a matrix A has a right inverse if and only if A? has a left inverse. a 6 2 -4 wehave AB=Ih,andBA=| 9 -2 6|¢h. However, if A is square matrix and has a left inverse, then we prove later (‘Theorem 1.8) that it has also a right inverse, and vice versa, Moreover, the following Jemma shows that the left inverses and the right inverses of a square matrix ere all equal. (This is not true for nonsquere matrices, of course). Lemma 1.6 If ann xn square matriz A has a left inverse B and a right inverse C, then B and C are equal, ie., B= C.1.6. INVERSE MATRICES 25 Proof: A direct calculation shows that B=BI= BAC) =(BA)O=IC=0. Now any two left inverses must be both equal to 6 right inverse C, and hence to each other, and any two right inverses must be both equal to a left inverse |B, and hence to each other. So there exist only one left and only one right inverse for a square matrix A ifit is known that A has both left and right inverses. Furthermore, the left and right invetees are equal. 0 ‘This theorem says that if a matrix A has both « right inverse and o left inverse, then they must be the seme. However, we shall see in Chapter 3 ‘that any mxn matrix A with m #n cannot have both a right inverse and left inverse: that is, a nonsquare matrix may have only a left inverse or only a right inverse. In this ease, the matrix may have mony loft iaverses or many right inverses. 10 Example 1.8 A nonsquare matrix A= | 0 1 | can have more than one oo left inverse. In fact, for any 2, y GIR, one can easily check that the matrix, ao [5 2G ]eeit mowers o oly Definition 1.8 An nx n square matrix A is said to be invertible (or nonsingular) i there exists a square matrix B of the same size such that ‘Such a matrix B is called the inverse of A, and is denoted by A“. A matrix Ais said to be singular if it is not invertible. Note that Leuuna 1.6 shows that if a square matrix A hes both left and ight inverses, then it must be unique. That is 7 we call B “the” inverse of A. Por instance, consider a 2x2 matrix A = | © tp] tari, ed ‘then it is easy to verify that . i got 4 -b)_| ad-te ad—be ad—te|~e a ai a26 CHAPTER 1. LINEAR EQUATIONS AND MATRICES since AA“! = Jy = AIA. (Check this product of matrices for practice!) Note that any zero matrix is singular. Problem 1.13 Let. A be on invertible matrix and & any nonzero scalar. Show that (Q) Ais invertible and (A“1}? = Ay (2) the matrix k is invertible and (KA)-! = 2A~>, (3) A? is invertible and (AT}“? = (ATH, ‘Theorem 1.7 The product of invertible matrices is also invertible, whose inverse is the product of the individual inverses in reverse onder: (AB)? = BAT, Proof: Suppose that A and B are invertible matrices of the same size, Then (AB\(B147) = A(BB A = AIA? = AA} = T, and similarly (B-14-1)(4B) = I. Thus AB has the inverse BLA“, o ‘We have written the inverse of A as “A to the power —1", so we can give the meaning of A for any integer #: Let A be a square matrix. Define A? = I, Then, for any positive integer k, we define the power At of A industively os Abs A(AN), ‘Moreover, if A is invertible, then the negetive intoger power is defined as Ae (Am#)* for k > 0, It is easy to check that with these rules we have A! = 4A? whenever the right hand side is defined. (If A is not invertible, A*-1) js defined but Al is not.) Problem 1.14 Prove: (2) If A has a zero row, 50 does AB. (2) If B hes « eto column, s0 does AB. (8) Any matrix with a zero row or a zero eolumna cannot be invertible, Problem 1.15 Let A be an invertible matrix, Is it true that (46)? = (A7)* for any integer A? Justify your enswer,1.7, ELEMENTARY MATRICES on 1.7 Elementary matrices ‘Wo now return to the system of linear equations Ax = b. If A has a right inverse B such that AB = Im, then x — Bb is a solution of the system since Ax = A(Bb) ~ (AB)b =b. In particular, if A is an invertible square matrix, then it has only one inverse A-! by Lemma 1.6, and x = A7'b is the only solution of the system. In this section, we discuss how to compute A~! when A is invertible, Recall that Gaussian elimination is @ process ix which the augmented matrix is transformed into its row-echelon form by a finite munber of cle mentary row operations. In the following, we will show that each elementary sow operation can be expressed os a nonsingular matrix, called an elementary matriz, and hence the process of Gaussian elimination is simply multiplying a finite sequence of corresponding elementary matrices to the augmented matrix. Definition 1.9 A matrix F obtained from the identity matrix , by exe: cuting only one elementary row operation is called an elementary matrix. For example, the following matrices are three elementary matrices cor- responding to each type of the three elementary row operations. a [ 5 3 | multiply the second row of Ip by —5; 1000 ooo1 5 10 0 1g]: letchange the second and the fourth rows of ej oi1900 103 (3) | 0 1 0 |: add 3 times the third row to the first row of Ia. oo. 1 is an interesting fact that, if F is an elementary mmatsix obtained by executing a certain elementary row operation on the identity matrix, then for eny mx n matrix A, the product EA is exactly the matriz that is obtained when the same elementary row operation in E is enccuted on A. ‘The following example illustrates this argument, (Note thet AH is not what wwe want, For this, see Problem 1.17).28 CHAPTER 1, LINEAR EQUATIONS AND MATRICES Example 1.9 For simplicity, we work on a 3x 1 columa matrix b. Suppose that we want to do the operation “adding (~2) x the first row to the second row" on suatrix b. Then, we execute this operation on the identity matrix J first to get an elementary matrix E: 1 E=|-2 0 ‘Multiplying the elementary matrix Eto b on the left produces the desired one Hee result: 100)]fm by Eb=|-210]| hy By — Dy oo1]| bs bs Similarly, the operation “interchanging the first and third rows" on the matrix b can be achieved by multiplying a permutation matrix P, which is an elemantary matrix obtained from J5 by interchanging two rows, to b on * by bs nl=|n]. by by Recall that each elementary row operation has an inverse operation, which is also an elementary operation, that brings the matrix back to the original one. Thus, suppoce that 2 denotes an elementary matrix corre- sponding to an elementary row operation, and let E! be the elementary matrix corresponding to its “inverse” elementary row operation in B. Then, oo1 Pb=/010 100 a (1) if B multiplies a row by ¢ # 0, then B’ multiplies the same row by 2; (2) if B interchanges two rows, then £” interchanges them agein; (3) if E adds s multiple of one row to another, then BY subtracts it: back from the same row. ‘Thus, for any m xn matrix A, EXEA = A, and BYE EE’. That is, every elementary matric is invertible so that E-! = Ei, which is also an elementary matris. For instance, if1.7, ELEMENTARY MATRICES 29 100 o Balde 0 0 |, then 001 1 10 010 Eyl=|0 Ie 100 ao oo1 Definition 1.10 A perrautation matrix is a square matrix obtained from the identity matrix by permuting the rows. Problenn 1.16 Prove: (1) A permutation mairix is the product ofa finite mumber of elementary matrices each of which is corresponding to the “row-interchanging” elementary row ‘operation. (2) Any permutation matrix P is invertible and Pl = PT, (3) The product of any two permutation matrices is a permutation matrix, (4) The transpose of permutation matrix is also a permutation matrix, Problem £.17 Deine the elementary column operations for & matrix by just replacing “row” by “column” in the definition of the elementary row operations. Show thet if A [s an m xm matrix and if E is an elementary matrix obtained by executing an clementary column operation on J,, then AF is exactly the matrix that is obtained from A when the same column operation is executed on A. ‘The next theorem establishes some fundamental relationships between nxn square matrices end systems of 7 linear equations inn unimowns. ‘Theorem 1.8 Let A be an nx n matrés. The following are emivalent: (1) A has 4 left inverse; (2) Ax=0 has only the érivial solution x (8) A is row-equivalent to In (4) A isa product of elementary metrices; (5) A is invertible; (6) A has @ right inverse. Proof (1) = (2): Let xbeasolution of the homogencous system Ax = 0, and let B be a left inverse of A. ‘Then x= Ink = (BA)x = BAx = BO =0.30 CHAPTER 1. LINBAR EQUATIONS AND MATRICES (2) + (8) : Suppose that the homogencous system Ax = 0 hes only the frivial solution x = 0; a = ty eo tm = 0. ‘This means that the augmented matrix [4 0] of the system Ax = 0 is reduced to the system [J,, 0] by Gauss-Jordan elimination. Hence, 4 is row-equivalent tol. (3) + (4) : Assume A is row-equivalent to Jn, s0 that A can be reduced to Jn by a finite sequence of elementary row operations, Thus, we can find elementary matrices By, Ba,...,E, sich that Exc EBA = In Since £1, Ba, ..., Hy are invertible, by multiplying both sides of this equation. oon the left successively by Ey7,...,B7',Ey', we obtain As By 1B) pM, = BPEy! Ey, ‘which expresses A as the product of elementary matrices. (4) = (5) is trivial, because any elementary matrix is invertible. Tn fact, AY = Bye ERs. (5) = (1) and (5) = (6) are trivial. (8) + (5) : If B is a right inverse of A, then A is a left inverse of B and we can apply (i) = (2) =» (3) + (4) = (6) to B and conclude that B is invertible, with A as its unique Inverse. That is, B is the inverse of A and 20 A is invertible. a This theorem shows that a squere matrix is invertible if it has a one-side inverse. Tn particular, if a square metrix A is invertible, then x = AW*b is a unique solution to the system Ax = b. Problem 1.18 Find the inverse of the product 1 00 100 100 0 10 oro};-ai10 o-e1f[-o1 oot7, ELEMENTARY MATRICES 31 ‘As an application of the preceding theorem, we give a practical method for finding the inverse A~? of an invertible nxn matrix A. If A is invertible, there are elementary matrices Fi, Hp, ..., Ey such that By: Ey E\A= Ip Hence, AC} = By BaBy = Bye By E ln, It follows that the sequence of row operations that reduces an inueriible ma- trie A to Jn will resolve I, to A~. In other words, let {A | I] be the aug- mented matrix with the columns of A on the left half, the columns of J on the right half. A Gaussian elimination, applied to both sides, by some elementary row operstions reduces the atigmented matrix [A | I] to {U | Kj, where U is a row-echelon form of A. Next, the back substitution process by another series of elementary row operations reduces [U | K] to {J | A~!]= [AL] + [Be FA] Be BT] = (0 | K) Fe AU | Pee = [2 | A“), where Bz-+- represents a Gaussian elimination and F---F, represents the back substitution. The following example illustrates the computation of fan inverse matrix. Example 1.10 Find the inverse of We epply Gauss-Jordan elimination to 123,100 lain = |23 5 [010 102 [00 1) (-1)row1+rows (-2)row 1+ row 2 0 1 0 12 3] 21 = |o -1 -1 | -2 0-2 -1 | -1 . 0 0 | (-tyrow 2 1 0 8 1 2. 2 8] 10 = fo 1 1] 2-1 0-2-1} 40 (Q)row 2 + row 332 CHAPTER 1. LINEAR EQUATIONS AND MATRICES 128 /i 00 + jo1i1|2-1 0]. oor} 3-24 ‘This is [U | K] obtained by Gaussian elimination, Now continue the back substitution to reduce [U | K] to [1 | A] 3 1 1 1238/1 00 (-Ljrow 8+ row 2 wiky = |o1 2-10 ik [3 resi (-8hrow 3+ row 1 fone | TSS HEL (ayrow 2-4 row 2 oo1| 3-2 1 100) -6 4-1 = |o10]-1 1-1 = [la] oo1| 842 1 Thus, we get 6 4-41 Avs) -1 1-1]. 3-2 1 (Tao reader should verify thet AA! =I = A-?A,) o Note that if A is not invertible, then, at some step in Gaussian elimina- tion, a zero row will show up on the left side in [U | X]. For example, the 16 4 16 4 matrix A=| 2 4 =1 | isrowequivalent to | -8 -9 | which iso 12 5 oo 0 noninvertible matrix, Problem 1.19 Write A~? ss a product of elementary matrices for A in Exemple 1.10, of A by using Gaussian climination. Praslem 1.00 Find the inverse of each of the following matric ei? a) ko00 1200 1kooO [3 i 3). rad ofOyot ko] Ae 1248 oo lk18, LDU FACTORIZATION 33 a 0 Problem 1.21 When is a disgonal matrix D = . nonsingular, and 0 dy what is D1? From Theorem 1.8, a square matrix A is nonsingular if and only if Ax = 0 has only the trivial solution. That is, a equare matrix A is singular if and only if Ax = 0 has a nontrivial solution, say xp. Now, for any column vector b= (bo; +++ bal’, if is « solution of Ax = b for a singular matrix A, then 80 18 kxo + x1 for any h: Alkxo +1) = h(Ax0) + Ax; = ROE b= This argument strengthens Theorem 1.5 as follows when A is a square matrix: ‘Theorem 1.9 If A is an invertible nxn matrés, then for any column vector b=l[b + bal’, the system Ax = b has exactly one sohution x = A™b. If A is not invertible, then ihe systern has either no solution or infinitely many solutions according to whether or not the system is inconsistent. 2 Problem 1.22 Write the system of lineer equations a+ by + & = 10 Qe = Wye Be = 1 dz — Sy + br = 4 in matrix form Ax = b and solve it by finding A~"b. 1.8 LDU factorization Recall that « basic method of solving a linear system Ax = b is by Gauss- Jordan elimination. For a fixed matrix A, if we want to solve more than ome system Ax = b for verious values of b, then the same Gaussian elimination on A has to be repeated over and over again, However, this repetition may ‘be avoided by expressing Gaussian elimination as an invertible matrix which i a product of elementary matrices. ‘We first assume that no permutations of rows are necessary throughgut ‘the whole process of Gaussian elimination on [A b]. Then the forward elim- inetion is just to multiply finitely many elementary matrices Ey, ..., Et to the augmented matrix [4 bl; that is, [Ee E1A Ey--- Eb] = [Ue],34 CHAPTER 1. LINEAR EQUATIONS AND MATRICES where each F; is a lower triangular elementary matrix whose diagonal entries are all 1's and [UJ ¢) is the augmenied matrix of the system obtained after forward elimination on Ax = b (Note that U need not be an upper triangular matrix if A is not a square matrix). Therefore, if we set L = (Es Exy-t = (Ey. --Ej!, then A= LU and c= Ux= Fy BA = Bye Eb = Lb. Note that L is a lower triangular matrix whose diagonal entries are all 1’s (see Problem 1.24). Now, for any column matrix b, the system Ax = LUx = b can be solved in two steps: first compute c = L~'b which is » forward climination, and then solve Ux = by the back substitution, This means that, to solve the é-systems Ax = b; fori =1, ..., & we. first find the matrices £ and U such that A = LU by performing forward Glimination on A, and then compute ¢; = Lb; for # = 1,...,€ ‘The solutions of Ax = by are now those of Ux = € Example 1.11 Consider the system of linear equations 2110][n 1 dx=| 4ta1]|a]=/-2]= -2211]|a 7 ‘The elementary matrices for Gaussian elimination of A are easily found tobe 100 100 100 Fy=}-210/],H=/01 0], and B=]01 oF, oo. 101 031 so that EsE.EA= oO 0-4 4 Note that U is the matrix obtained from A after forward elimination, and A=W with 1 00 2 104, 2 1 10 O -1 -2 1/=u. L= Ej) 53" By’ = -1 +31 which is a lower triangular matrix with 1’s on the diagonal. Now, the system 0 a = le=b: ) 24 + @ 2 ~a — Be 4+ ey 71.8, LDU FACTORIZATION 35 resolves to € = (1, —4,—4) and the system 2m + m2 + a5 Ux=e: ~~ My tm deg + Ag resolves to -14¢] [a 1 24 3b 2 3 a= e{7]} rt flap t 0 1 fort ©. Ibis suggested that the readers find the solutions for vatious values of b. o Problem. 1.24 Determine an LU decoraposition of the matrix i -1 0 A=[-1 2-1], O-1 2 and then find solutions of Ax = b for (2) (11 fF ond (2) b= po — 17. Problem 1.24 Let, A, B be two lower triangular matrices. Prove that, (@) their product is also a lower triangtat matrix, (2) if A is invertible, then its inverse is also a lower triangular matrix; (3) if the diagonal entries are all 1's, then the same holds for their product and their tnverses. Note tha: the same holds for upper trlanguler matrices, and for the product of more ‘then two matrices. Now suppose that A is a nonsingular square matrix with A = LU in which no row interchanges were necessary. Then the pivots on the diagonal of U are all nonzero, and the diogonel of Z aro all 1's, Thus, by dividing cach i-th row of U by the nonzero pivot d;, the matrix U is factorized into a diagonal matrix D whose diagonals are just the pivots d:, d2, ..., dy and a new upper triangular matrix, denoted again by U, whose diagonals ate all V's 60 that A= LDU. For example, ar a 0 we O 1 fd; sfdy é 0 d& 0 oo4 t/dy ~u fl |i 0 i th ufdna dy Owe mi fo wed36 CHAPTER 1. LINEAR EQUATIONS AND MATRICES ‘This decomposition of A is called the LDU factorization of A. Note that, in this factorization, UT is just a row-echelon form of A (with leading 1’s on the diagonal) after Gaussian elimination and before back substitution. In Example 1.11, we found a factorization of A as 211 0-1-2]. 0 0-4 ‘This can be further factored as A = LDU by taking 211 2 0 o}fi 1p 12 0-1 -2/=/0 -1 0 o1 2 DU. 1 00 A=| 2 10 1-31 0 0-4 0 o-4}{[o 0 1 Suppose now that during forward climinatioa row interchanges are nec- essary. In this case, we can first do all the row interchanges before doing any other type of elementary row operations, since the interchange of rows can be done at eny time, before or after the other operations, with the same effect ‘on the solution. Those “row-interehanging” elementary matrices altogether form a permutation matrix P so that no more row interchanges are aeeded during Ganssian elimination of PA. So PA has an LDU factorization. o12 0 1 0 |. For Gaussian 100 elimination, it is clearly necessary to interchange the first row with the thind Example 1,12 Consider a square matrix A oo1 row, that is, we need to multiply the permutation matrix P = [: 1 ‘| to A so that ~eiEH Hl Of course, if we choose a different permutation P’, then the LDU fac- torization of PA may be different from that of PA, even if there is an- other permutation matrix P" that changes P’A to PA. However, if we fix a permutation matrix P when it is necessary, the uniqueness of the LDU factorization of A can be proved. 0 1 =. con Heo cHo woe 1 o18. LDU FACTORIZATION ar ‘Theorem 1.10 For an invertible matris A, the LDU factorization of A is unique up to permutation: that is, for a fixed P the expression PA = LDU is unique. Proof: Suppose that A = [:D:U; = LoDo, where the L's are lower triangular, the U's are upper triangular, all with 1’s on the diagonal, and ‘the D’s are diagonal matrices with no zeros on the diagonal. We need to show Jy = In, Di = Da, and Uy =U. Noie that the inverse of a lower (upper) triangular matrix is also a lower (upper) triangular matrix. And the inverse of a diagonal matrix is also diagonal. Therefore, by multiplying ([1.D)~! = Dy7L;1 on the left and U;! on the right, our equation Ly D,U = LeD,Uy becomes alg? = DUT UaDs. ‘The left side is an upper triengular motrix, while the right cide is a lower triangular matrix. Hence, both sides must be diagonal. However, since the diagonal entries of the upper triangular matrix Uy? are all 1’s, it must be the identity matrix (see Problem 1.24). Thus UiU;! = I, i. Ur = Uo. Similarly, LyL2 = D, Dz? implies that Ly = Lz and Dy = D3. a In particular, if A is symmetric (i.¢,, A= AT), and if it can be factored into A= LDU without row interchanges, then we have LDU = A= AP = (LDU =U7DTLT = UTDIF, and thus, by the uniqueness of factorizations, we have U = L7 and A = EDT. 2-21 0 Pron 125d te ti, a =| =} 2 +]. o-1 2 ‘What is the solution to Ax = b for b = [10 -1]7 ? Problem 1.26 For all possible perrautation matrices P, find the LDU factorizatjon 123 242 laid of PA for A=