1

Preliminaries

1.1 Matrices

Definition 1.1. A matrix is an array of numbers arranged in rows

and columns.
Examples of matrices
Matrix A has two rows and four columns; matrix B has three " #
2 −1 5 12
rows and one column. We denote the matrices by A2×4 and B3×1 A=
4 0 −3 −1
respectively.  
A general matrix A of size m × n will be denoted by Am×n = [ aij ], 3
 
B= 5 
where aij represents the number in the ith row and jth column. In
−9
the example, b21 = 5 and a14 = 12.
A matrix having one row is called a row vector and a matrix
having one column is called a column vector. In the example above,
matrix B is a column vector. We shall use lowercase letters for vec-
tors and uppercase letters for matrices.
A matrix A is a square matrix if the number of rows and the
number of columns are equal. We denote a square matrix of size
n × n by An .
The diagonal of a square matrix An consists of the numbers
a11 , a22 , . . . , ann . Identity
 matrix
 I3 .
A square matrix of size n × n consisting of only 1s in the diago- 1 0 0
 
nal and 0s elsewhere is called an identity matrix of size n, denoted 0 1 0
0 0 1
by In .
Two matrices A and B are conformant if their dimensions are the
same. Two conformant matrices A and B are equal if corresponding
entries are equal, that is, aij = bij for each row i and column j.
Example 1.1. Find the values of p and q that satisfy
" # " #
2p + q 12
=
2q + 5 −3p

Solution: Equality of matrices require solving the system of equa-

tions (
2p + q = 12
2q + 5 = −3p
Solving the system of equations by elimination or substitution
yields
p = 29, q = −46
6 luciano m. medrano jr.

1.1.1 Matrix Operations

Addition and subtraction are defined only for conformant matrices.
If Am×n = aij and Bm×n = bij , then their sum and difference are

A + B = [ aij + bij ] (1.1)

A − B = [ aij − bij ] (1.2)

If a matrix is multiplied by a real number (called scalar), each ele-

ment of the matrix is multiplied by that scalar.

c · A = [c · aij ] (1.3)

Let Am× p = [ aij ] and B p×n = [bij ] be two matrices. The product AB
is a matrix Cm×n = [cij ] defined by

p
cij = ∑ aik bkj cij = ai1 b1j + ai2 b2j + ai3 b3j + · · · +
k =1 aip b pj

The condition for matrix multiplication is that the number of

columns of the first matrix equals the number of rows of the sec-
ond matrix. Note that the product Am× p B p×n is defined while
B p×n Am× p in not defined.

Example 1.2.
 
2 1 " #
 5 −4
  3 7 0
 
 0 −2 −2 −4 6
−3 −1

Solution: The dimension of the first matrix is 4 × 2 and the di-

mension of the second matrix is 2 × 3, therefore, the product is
 
? ? c13
? c13 = 2(0) + 1(6) = 6
 ? ? 
C 4×3 =  c32 = 0(7) + (−2)(−4) = 8
? c32 ? 
? ? ?

1.1.2 Determinant
Definition 1.2 (Determinant). Let A = [ aij ] be a square matrix of
size 2. The determinant of A, denoted |A|, is a real number defined
as  
a 
 11 a12 
|A| =   = a11 a22 − a12 a21 (1.4)
 a21 a22 

Definition 1.3 (Determinant). Let A = [ aij ] be a square matrix of

size n. Let Mij be the (n − 1) × (n − 1) submatrix of A obtained
by deleting the ith row and jth column. The determinant of Mij is
called the minor of aij . The cofactor Aij of aij a real number defined
as
Aij = (−1)i+ j |Mij | (1.5)
 numerical solutions for ce problems 7

Definition 1.4. The determinant of a square matrix A is given by

n
|A| = ∑ aij Aij any row i (1.6)
j =1
n
|A| = ∑ aij Aij any column j (1.7)
i =1

Eqn. (1.6) is called the row expansion of |A| and Eqn. (1.7) is
called the column expansion of |A|.
Example 1.3. Compute the determinant of the matrix
 
2 1 1
 
3 2 3
1 4 9
by expanding about
a. row i = 2

b. column j = 3

1.2 Taylor Series

Definition 1.5. Let a function f ( x ) be infinitely differentiable at x = a.

The function is said to have a Taylor Series representation
∞
f (k) ( a ) f ′ ( a) f ′′ ( a)
f (x) = ∑ ( x − a)k (1.8) f ( x ) = f ( a) + ( x − a) + ( x − a )2
k! 1! 2!
k =0
f ′′′ ( a) f (4) ( a )
where f (k) ( a) represents the value of the kth derivative at x = a. + ( x − a )3 + ( x − a )4 + · · ·
3! 4!
That is, f (1) ( a) = f ′ ( a), f (2) ( a) = f ′′ ( a), and so on. The notation
f (0) ( a) represents the value of the zero-th derivative at x = a, or
simply f ( a).
Table 1.1: Derivatives of ln x at x = 1
Example 1.4. Express the Taylor series representation of f ( x ) = ln x
k f ( k ) ( x ) f ( k ) (1)
about x = 1.
0 ln x 0
Solution: We take the first few derivatives of ln x and evaluate 1 1
1
x
each at x = 1. The results are summarized in Table 1.1.
2 − x12 -1
Therefore, we can write ln x as a Taylor series 2
3 x3
2
f (1) f ′ (1) f ′′ (1)
ln x = + ( x − 1) + ( x − 1)2 4 − x64 -6
0! 1! 2! 24
5 24
f ′′′ (1) f (4) ( 1 ) x5
+ ( x − 1)3 + ( x − 1)4 + · · ·
3! 4!
0 1 −1 2
= + ( x − 1) + ( x − 1)2 + ( x − 1)3
1 1 2 6
−6 4 24 5
+ ( x − 1) + ( x − 1) + · · ·
24 120
1 1 1
= ( x − 1) − ( x − 1)2 + ( x − 1)3
1 2 3
1 1
− ( x − 1) + ( x − 1)5 + · · ·
4
4 5
which can be expressed in the compact form
∞
1
ln x = ∑ (−1)k+1 n (x − 1)k
k =1
 numerical solutions for ce problems 7

Definition 1.4. The determinant of a square matrix A is given by

n
|A| = ∑ aij Aij any row i (1.6)
j =1
n
|A| = ∑ aij Aij any column j (1.7)
i =1

Eqn. (1.6) is called the row expansion of |A| and Eqn. (1.7) is
called the column expansion of |A|.
Example 1.3. Compute the determinant of the matrix
 
2 1 1
 
3 2 3
1 4 9
by expanding about
a. row i = 2

b. column j = 3

1.2 Taylor Series

Definition 1.5. Let a function f ( x ) be infinitely differentiable at x = a.

The function is said to have a Taylor Series representation
∞
f (k) ( a ) f ′ ( a) f ′′ ( a)
f (x) = ∑ ( x − a)k (1.8) f ( x ) = f ( a) + ( x − a) + ( x − a )2
k! 1! 2!
k =0
f ′′′ ( a) f (4) ( a )
where f (k) ( a) represents the value of the kth derivative at x = a. + ( x − a )3 + ( x − a )4 + · · ·
3! 4!
That is, f (1) ( a) = f ′ ( a), f (2) ( a) = f ′′ ( a), and so on. The notation
f (0) ( a) represents the value of the zero-th derivative at x = a, or
simply f ( a).
Table 1.1: Derivatives of ln x at x = 1
Example 1.4. Express the Taylor series representation of f ( x ) = ln x
k f ( k ) ( x ) f ( k ) (1)
about x = 1.
0 ln x 0
Solution: We take the first few derivatives of ln x and evaluate 1 1
1
x
each at x = 1. The results are summarized in Table 1.1.
2 − x12 -1
Therefore, we can write ln x as a Taylor series 2
3 x3
2
f (1) f ′ (1) f ′′ (1)
ln x = + ( x − 1) + ( x − 1)2 4 − x64 -6
0! 1! 2! 24
5 24
f ′′′ (1) f (4) ( 1 ) x5
+ ( x − 1)3 + ( x − 1)4 + · · ·
3! 4!
0 1 −1 2
= + ( x − 1) + ( x − 1)2 + ( x − 1)3
1 1 2 6
−6 4 24 5
+ ( x − 1) + ( x − 1) + · · ·
24 120
1 1 1
= ( x − 1) − ( x − 1)2 + ( x − 1)3
1 2 3
1 1
− ( x − 1) + ( x − 1)5 + · · ·
4
4 5
which can be expressed in the compact form
∞
1
ln x = ∑ (−1)k+1 n (x − 1)k
k =1
8 luciano m. medrano jr.

The series representation of f ( x ) = ln x is valid when | x − 1| < 1,

or
0<x<2
Outside this range, the series will have an infinite sum and is said
to diverge.
If we consider only the terms of Eqn. (1.8) up to ( x − a)n , the
Taylor series is truncated and the resulting expression is called the
Taylor polynomial of degree n, denoted Tn ( x ), that can be used to
approximate f ( x ).
n
f (k) ( a )
Tn ( x ) = ∑ ( x − a)k (1.9)
k =0
k!

Example 1.5. Use T5 to approximate ln 0.8.

Solution: We have

ln x ≈ ( x − 1) − 12 ( x − 1)2 + 13 ( x − 1)3 − 14 ( x − 1)4 + 15 ( x − 1)5

so that

0.22 0.23 0.24 0.25

ln 0.8 ≈ −0.2 − − − − = −0.223 131
2 3 4 5
The value of ln 0.8 is -0.223 143 55, so that the approximation is
‘good’ to four decimal places.
1

The graphs of different Taylor polynomials for ln x are shown in

Fig. 1.1. For values of x less than 1, the value of T4 ( x ) is closer to 0.5 1 1.5 ln x 2
T1
ln x than the other three Taylor polynomials. −1
T2
T3
−2
T4

Figure 1.1: Taylor approximations for ln x

2
System of Linear Equations

Many solutions to engineering and applied science problems re-

quire formulation of a mathematical model involving the variables
of interest. The simplest of these models is a system of linear alge-
braic equations, whose form is shown below.


 a11 x1 + a12 x2 + · · · + a1n xn = b1


 a21 x1 + a22 x2 + · · · + a2n xn = b2
.. (2.1)

 .



am1 x1 + am2 x2 + · · · + amn xn = bm

The system may be represented as the matrix equation

Ax = b

where
  
  
a11 a12 ··· a1n x1 b1
     
 a21 a22 ··· a2n   x2   b2 
A=
 .. .. .. .. 
, x= 
 ..  , b= 
 .. 
 . . . .   .   . 
am1 am2 ··· amn xn bm

We note that the general system above

• has m equations;

• has n unknowns (x1 , x2 , . . . , xn );

• the a’s are coefficients of the unknowns;

• the b’s are constants.

The system of linear equations can have a unique solution, no

solution or multiple solutions.
In this course, we shall consider only the case where the number
of equations equals the number of unknowns. Equivalently, we will
explore the case of a square coefficient matrix A. The solution can
be obtained by a direct method or by an iterative method. Larger
sytems are usually solved with the aid of a software.
10 luciano m. medrano jr.

2.1 Direct Methods

2.1.1 Gauss Elimination

The Gauss elimination method is an elementary method that re-
duces the system of an equivalent upper triangular system, which
can be solved by back substitution. This is illustrated by an exam-
ple.

Example 2.1. Solve the system

2x + y + z = 10 (1)
3x + 2y + 3z = 18 (2)
x + 4y + 9z = 16 (3)

Let us eliminate x first. Using equations (1) and (2), we obtain a

new equation
y + 3z = 6 (4)
Using equations (1) and (3), we obtain another equation

−7y − 17z = −22 (5)

We eliminate y using equations (4) and (5) where we get

z=5 (6)

We use equation (6) to solve for y in equation (4) and the results
to solve for x in equation (1). We find that the solution is

x = 7, y = −9, z = 5

Since we used equations (1), (4) and (6), we also have the system

2x + y + z = 10
y + 3z = 6
z=5

with the same solution. In matrix form, we write

    
2 1 1 x 10
    
0 1 3  y  =  6 
0 0 1 z 5

Elementary Row Operations

We define three row operations that mimic the elimination
method. Let ri be the ith row of a matrix.

row operation description

kri → ri the row vector kri is the new row i of
the matrix
ri ↔ r j the ith and jth rows are interchanged.
kri + r j → r j the row vector kri is added to the jth
row.
 numerical solutions for ce problems 11

Applying any row operation on a matrix yields a new matrix.

We say that the new matrix is “row equivalent” to the original
matrix.
We demonstrate the Gauss elimination using the three elemen-
tary row operations.

Example 2.2. Solve the system

2x + y + z = 10
3x + 2y + 3z = 18
x + 4y + 9z = 16

We first form the matrix M by augmenting A and b

  
2 1 1  10
  
M = 3 2 3  18

1 4 9  16

then we apply the row operations as many times necessary to ob-

tain an upper triangular matrix. To ‘eliminate x’ of r2 and r3 , we
perform
 
−3 12 r1 + r2 → r2
 
− 12 r1 + r3 → r3

The resulting matrix is row equivalent to M.

 
2 1 1 10
 
M≡ 1 3
0 2 2 3
0 72 17
2 11

We use r2 to ‘eliminate y’ of r3 . The resulting matrix is also row

equivalent to M.
 
2 1 1 10 2x + y + z = 10
 
M≡ 1 3
0 2 2 3  1 3
=3
 2y + 2z
0 0 2 −10 −2z = −10
The upper triangular system shown on the right is solved by back
substitution.

If the row operations 2r2 → r2 and 2r3 → r3 are performed

before eliminating y, we get an intermediate matrix
 
2 1 1 10
 
0 1 3 6
0 7 17 22

that is also row equivalent to M.

2.1.2 Gauss-Jordan Method

The drawback of the Gauss elimination method is the use to back
substitution to complete the solution of the system. The Gauss-
Jordan method, a modification of the Gauss elimination, eliminates
12 luciano m. medrano jr.

the back substitution by eliminating an unknown in all equations

except one (called the pivot equation). We illustrate this method.
Example 2.3. Solve the system
2x + y + z = 10
3x + 2y + 3z = 18
x + 4y + 9z = 16
by the Gauss-Jordan method.
In this method, we eliminate (a) the first unknown in all equa-
tions except the first, (b) the second unknown in all equations ex-
cept the second, and so forth by elementary row operations. Before
eliminating an unknown, we will force the coefficients of (a) the
first unknown in the first equation, (b) the second unknown in
the second equation, and so on, to unity using the row operation
kri → ri .
   
2 1 1 10 1 12 21 5
   
3 2 3 18 1 r1 → r1 3 2 3 18
 2  
1 4 9 16 1 4 9 16
 
1 12 12 5
−3r1 + r2 → r2 
0 1 3

 2 2 3
−1r1 + r3 → r3
0 72 172 11
At this stage, x is eliminated in all equations except the first. Now
we force a coefficient 1 on the variable y in equation 2, followed by
the elimination of y in equations 1 and 3.
   
1 12 12 5 1 12 12 5
   
0 1 3 3  
 2 2  2r2 → r2 0 1 3 6 
0 72 17 2 11 0 72 17
2 11
 
1 0 − 1 2
− 12 r2 + r1 → r2 
0 1 3

7  6 

− 2 r2 + r3 → r3
0 0 −2 −10
The last step is the elimination of z in equations 1 and 2.
   
1 0 −1 2 1 0 −1 2
  1  
0 1 3 6   
  − 2 r3 → r3 0 1 3 6
0 0 −2 −10 0 0 1 5
 
1 0 0 7
1r3 + r1 → r1  
0 1 0 −9
 
−3r3 + r2 → r2
0 0 1 5
The final augmented matrix is equivalent to the system
x=7
y = −9
z=5
 numerical solutions for ce problems 7

Definition 1.4. The determinant of a square matrix A is given by

n
|A| = ∑ aij Aij any row i (1.6)
j =1
n
|A| = ∑ aij Aij any column j (1.7)
i =1

Eqn. (1.6) is called the row expansion of |A| and Eqn. (1.7) is
called the column expansion of |A|.

Example 1.3. Compute the determinant of the matrix

 
2 1 1
 
3 2 3
1 4 9

by expanding about

a. row i = 2

b. column j = 3

1.1.3 Matrix Inverse

In this section, we learn two methods of finding a matrix B such
that BA = In where A is a square matrix and |A| ̸= 0. The matrix
B having this property is called the inverse of the matrix A, and is
denoted by A−1 .
Elementary Row Operations
Let A be a square matrix such that |A| ̸= 0. The inverse A−1 can
be obtained by following these steps:

1. Compose the augmented matrix M = [A | In ].

2. Perform the elementary row operations as many times as

necessary to obtain a matrix [In | C] that is row equivalent
to M. See the Gauss-Jordan method on page 13 for an
example.

The matrix C above is the inverse of A, and we write A−1 = C.

Example 1.4. Use elementary row operations to find the inverse of

the matrix " #
0 2
−7 3

Adjoint Matrix
Let A be a square matrix. The adjoint of A, denoted adj(A) is
defined as
adj(A) = [ A ji ] (1.8)

where Aij is the cofactor of aij . It should be noted that Aij is the
cofactor of the element in the ith row and jth column of A, but
8 luciano m. medrano jr.

its position on the adjoint matrix adj(A) is in the jth row and ith
column.
An important property of the adjoint matrix is given below.

A · adj(A) = adj(A)A = |A|In

Furthermore, if |A| ̸= 0, the inverse of A is

1
A −1 = adj(A) (1.9)
|A|
Example 1.5. Show that the adjoint of the matrix
 
10 0 0 0
−1 11 0 0
 
Q= 
 2 −1 10 0
0 3 −1 8

is  
880 0 0 0
 80 800 0 0 
 
Q −1 = 
−168 80 880 0 
−51 −290 110 1100

1.2 Taylor Series

Definition 1.5. Let a function f ( x ) be infinitely differentiable at x = a.

The function is said to have a Taylor Series representation
∞
f (k) ( a ) f ′ ( a) f ′′ ( a)
f (x) = ∑ ( x − a)k (1.10) f ( x ) = f ( a) + ( x − a) + ( x − a )2
k =0
k! 1! 2!
f ′′′ ( a) f (4) ( a )
where f (k) ( a) represents the value of the kth derivative at x = a. + ( x − a )3 + ( x − a )4 + · · ·
3! 4!
That is, f (1) ( a) = f ′ ( a), f (2) ( a) = f ′′ ( a), and so on. The notation
f (0) ( a) represents the value of the zero-th derivative at x = a, or
simply f ( a).
Table 1.1: Derivatives of ln x at x = 1
Example 1.6. Express the Taylor series representation of f ( x ) = ln x
k f ( k ) ( x ) f ( k ) (1)
about x = 1.
0 ln x 0
Solution: We take the first few derivatives of ln x and evaluate 1 1
1
x
each at x = 1. The results are summarized in Table 1.1.
2 − x12 -1
Therefore, we can write ln x as a Taylor series 2
3 x3
2
f (1) f ′ (1) f ′′ (1) 4 − x64 -6
ln x = + ( x − 1) + ( x − 1)2
0! 1! 2! 5 24
24
x5
f ′′′ (1) f (4) ( 1 )
+ ( x − 1)3 + ( x − 1)4 + · · ·
3! 4!
0 1 −1 2
= + ( x − 1) + ( x − 1)2 + ( x − 1)3
1 1 2 6
−6 24
+ ( x − 1)4 + ( x − 1)5 + · · ·
24 120
1 1 1
= ( x − 1) − ( x − 1)2 + ( x − 1)3
1 2 3
1 1
− ( x − 1) + ( x − 1)5 + · · ·
4
4 5
16 luciano m. medrano jr.

2.2.1 Jacobi Method

Convergence of the Jacobi method, also called the method of simul-
taneous displacements, relies on a special class of matrices known as
diagonally dominant. A square matrix An is diagonally dominant if
in each row, the absolute value of the diagonal entry is greater than
the sum of the absolute values of all the off-diagonal entries, that is,
n
| aii | > ∑ |aij |
j =1
j ̸ =i

The linear system Ax = b has a unique solution xs , and the

sequence of vectors generated by the Jacobi iteration, Eqn.(2.3),
converges to xs regardless of the choice of the initial vector x(0) .
Example 2.4. Use the Jacobi method to approximate the solution of
the linear system


 −v +11w − x +3y = 25

 3w − x +8y = 15

 10v − w + 2x =6


2v −w +10x −y = −11
Our matrices D and P are
   
10 0 0 0 0 1 −2 0
 0 11 0 0  1 0 1 −3
   
D=  P= 
0 0 10 0  −2 1 0 1 
0 0 0 8 0 −3 1 0
The inverse of the matrix D is
 
1
0 0 0
 10 
0 1
0 0
−1  11 
D = 
0 1
 0 10 0

1
0 0 0 8

Taking the initial vector

 
0
0
 
x (0) =0= 
0
0
we find  
3
 5 
 25 
 11 
x (1) = D −1 x (0) =  11 
− 
 10 
15
8
and
      
1 3
0 0 0 0 1 −2 0 6
 10    5   
0 1
0 0  −3  11   25 
  25   
(2)  11   1 0 1 
x =     11  +  
0 1
 0 10 0 
   −2 1 0 1    
 − 10  −11

1 15
0 0 0 8 0 −3 1 0 8 15
 numerical solutions for ce problems 17

 
1.0473
 
 1.7159 
 
= 
−0.8052
  Table 2.1: Successive approximations by
0.8852 the Jacobi method
k v w x y
0 0 0 0 0
Additional iterates are generated in a similar manner and are pre- 1 0.6000 2.2727 -1.1000 1.8750
sented in Table 2.1. 2 1.0473 1.7159 -0.8052 0.8852
3 0.9326 2.0533 -1.0493 1.1309
Note that
(7) (7) 4 1.0152 1.9537 -0.9681 0.9738
e = x − x(6) = 0.0163 5 0.9890 2.0114 -1.0103 1.0214
6 1.0032 1.9922 -0.9945 0.9944
so that x(7) is an acceptable solution to the system at ε = 0.02. 7 0.9981 2.0023 -1.0020 1.0036

The percent relative error of the kth approximate, defined as

(k)
e

x(k)

is often used as a stopping criterion. For instance, if the percent

relative error should not exceed 0.001,
(7)
e
= 0.0062
x (7)

so that additional iterations are needed.

2.2.2 Gauss-Seidel Method

Based on Eqn. (2.3), every component of x(k+1) is calculated en-

tirely from x(k) of the previous iteration. In other words, to access
x(k+1) , the kth iteration has to be completed so that x(k) is entirely
available. Performance of Jacobi iteration can be improved if the
most updated components of a vector are utilized, as soon as they
are available, to compute the subsequent components of the same
vector. This is the reasoning behind the Gauss-Seidel method. Gen-
erally the vector x(k) by this method is expected to be a better esti-
mate than the estimate by the Jacobi method.
The coefficent matrix A is expressed as Q − U, where U is an
upper triangular matrix. The matrix equation can then be rewritten
as
Qx(k+1) = Ux(k) + b

which can be solved for x(k+1) .

x(k+1) = Q−1 (Px(k) + b) (2.4)

Example 2.5. Use the Gauss-Seidel method to approximate the

solution of the linear system


 10v −w +2x =6

 −v +11w − x +3y = 25

 2v −w +10x −y = −11


3w − x +8y = 15