A P P E N D I X D

Review of Vectors
and Matrices

D.1 VECTORS
D.1.1 Definition of a Vector
Let p1, p2, Á , pn be any n real numbers and P an ordered set of these real numbers—
that is,
P = 1p1, p2, Á , pn2

Then P is an n-vector (or simply a vector). The ith component of P is given by pi. For
example, P = 11, 22 is a two-dimensional vector.

D.1.2 Addition (Subtraction) of Vectors

Consider the n-vectors

P = 1p1, p2, Á , pn2

Q = 1q1, q2, Á , qn2
R = 1r1, r2, Á , rn2

For R = P ; Q, component i is computed as ri = pi ; qi. In general, given the vectors

P, Q, and S,

P + Q = Q + P 1Commutative law2
1P ; Q2 ; S = P ; 1Q ; S2 1Associative law2
P + 1-P2 = 0 1zero or null vector2

CD-145
CD-146 Appendix D Review of Vectors and Matrices

D.1.3 Multiplication of Vectors by Scalars

Given a vector P and a scalar (constant) quantity u, the new vector
Q = uP = 1up1, up2, Á , upn2
is the scalar product of P and u. In general, given the vectors P and S and the scalars u
and g,
u1P + S2 = uP + uS 1Distributive law2
u1gP2 = 1ug2P 1Associative law2

D.1.4 Linearly Independent Vectors

The vectors P1, P2, Á , Pn are linearly independent if, and only if
n

a ujPj = 0 Q uj = 0, j = 1, 2, Á , n
j=1
If
n

a ujPj = 0, for some uj Z 0

j=1

then the vectors are linearly dependent. For example, the vectors
P1 = 11, 22, P2 = 12, 42
are linearly dependent because for u1 = 2 and u2 = -1,
u1P1 + u2P2 = 0

D.2 MATRICES
D.2.1 Definition of a Matrix
A matrix is a rectangular array of elements. The element aij of the matrix A occupies
the ith row and jth column of the array. A matrix with m rows and n columns is said to
be of size (or order) m * n. For example, the following matrix is of size 14 * 32.

a11 a12 a13

¥ = 7aij 7 4 * 3
a a22 a23
A = § 21
a31 a32 a33
a41 a42 a43
D.2.2 Types of Matrices
1. A square matrix has m = n.
2. An identity matrix is a square matrix in which all the main diagonal elements
equal 1 and all the off-diagonal elements equal zero. For example, a 13 * 32
identity matrix is given by
1 0 0
I3 = £ 0 1 0 ≥
0 0 1
 D.2 Matrices CD-147

3. A row vector is a matrix with one row and n columns.

4. A column vector is a matrix with m rows and one column.
5. The matrix AT is the transpose of A if the element aij in A is equal to element aji
in AT for all i and j. For example,
1 4
5 ≥ Q AT = a b
1 2 3
A = £2
4 5 6
3 6

6. A matrix B = 0 is a zero matrix if every element of B is zero.

7. Two matrices A = 7aij 7 and B = 7bij 7 are equal if, and only if, they have the same
size and aij = bij for all i and j.

D.2.3 Matrix Arithmetic Operations

In matrices only addition (subtraction) and multiplication are defined. Division,
though not defined, is replaced by inversion (see Section D.2.6).

Addition (Subtraction) of Matrices. Two matrices A = 7aij 7 and B = 7bij 7 can be

added together if they are of the same size 1m * n2. The sum D = A + B is obtained
by adding the corresponding elements. Thus,
7dij 7 m * n = 7aij + bij 7 m * n
If the matrices A, B, and C have the same size, then
A + B = B + A 1Commutative law2
A ; 1B ; C2 = 1A ; B2 ; C 1Associative law2
1A ; B2T = AT ; BT

Product of Matrices. The product D = AB of two matrices, A = 7aij 7 and B = 7bij 7, is

defined if, and only if, the number of columns of A equals the number of rows of B. If A
is of size 1m * r2 and B is of size 1r * n2, then D must be of size m * n, where m and n
are arbitrary positive integer values. In this case, the elements of D are computed as
r
dij = a aikbkj, for all i and j
k=1

For example, given

A = a b, B = a b
1 3 5 7 9
2 4 6 8 0
we have
11 * 5 + 3 * 6211 * 7 + 3 * 8211 * 9 + 3 * 02
D = a ba b = a b
1 3 5 7 9
2 4 6 8 0 12 * 5 + 4 * 6212 * 7 + 4 * 8212 * 9 + 4 * 02

= a b
23 31 9
34 46 18

In general, AB Z BA even if BA is defined.

CD-148 Appendix D Review of Vectors and Matrices

Matrix multiplication follows these general properties:

I mA = AI n = A, I m and I n are identity matrices

1AB2C = A1BC2
C1A ; B2 = CA ; CB
1A ; B2C = AC ; BC
a1AB2 = 1aA2B = A1aB2, a is a scalar

Multiplication of Partitioned Matrices. Let A be an 1m * r2 matrix and B an 1r * n2-

matrix. Assume that A and B are partitioned as follows:

B11 B12
A = a b, B = £ B21 B22 ≥
A 11 A 12 A 13
A 21 A 22 A 23
B32 B32

The partitioning assumes that the number of columns of A ij is equal to the number of
rows of Bij for all i and j. Then

A * B = a b
A 11B11 + A 12B21 + A 13B31 A 11B12 + A 12B22 + A 13B32
A 21B11 + A 22B21 + A 23B31 A 21B12 + A 22B22 + A 23B32

For example,

112142 + 12 32a b
1 4 + 2 + 24
12 3 4 30
8
£1 0 5≥ £1≥ = § ¥ = § ¥ = £ 44 ≥
a b142 + a ba b a b + a b
1 0 5 1 4 40
25 6 8 61
2 5 0 8 8 53

D.2.4 Determinant of a Square Matrix

Consider the n-square matrix
a11 a12 Á a1n
a a22 Á a2n
A = § 21 ¥
o o o o
an1 an2 Á ann

Next, define the product

Pj1j2 Ájn = a1j1a2j2 Á anjn

such that each column and each row of A is represented exactly once among the sub-
scripts of j1, j2, Á , and jn. Next, define

H j1j2 Ájn = e
1, j1 j2 Á jn even permutation
0, j1 j2 Á jn odd permutation
 D.2 Matrices CD-149

Let r represents the summation over all n! permutations, then the determinant of
A, det A or ƒ A ƒ , is computed as

a H j1j2 ÁjnPj1j2 Ájn

r

As an illustration, consider
a11 a12 a13
A = £ a21 a22 a23 ≥
a31 a32 a33
Then
ƒ A ƒ = a111a22 a33 - a23 a322 - a121a21 a33 - a31 a232 + a131a21 a32 - a22 a312
The properties of a determinant are:

1. The value of the determinant is zero if every element of a row or a column is zero.
2. ƒ A ƒ = ƒ AT ƒ .
3. If B is obtained from A by interchanging any two rows or any two columns, then
ƒBƒ = - ƒAƒ.
4. If two rows (or two columns) of A are multiples of one another, then ƒ A ƒ = 0.
5. The value of ƒ A ƒ remains the same if scalar a times a column (row) vector is
added to another column (row) vector.
6. If every element of a column or a row of a determinant is multiplied by a scalar
a, the value of the determinant is multiplied by a.
7. If A and B are two n-square matrices, then
ƒ AB ƒ = ƒ A ƒ ƒ B ƒ

Definition of the Minor of a Determinant. The minor Mij of the element aij in the
determinant ƒ A ƒ is obtained from the matrix A by striking out the ith row and jth
column of B. For example, for
a11 a12 a13
A = £ a21 a22 a23 ≥
a31 a32 a33

M11 = ` ` , M22 = ` 11 `,Á

a22 a23 a a13
a32 a33 a31 a33

Definition of the Adjoint Matrix. Let A ij = 1-12i + jMij be defined as the cofactor of
the element aij of the square matrix B. Then, the adjoint matrix of A is the transpose of
7A ij 7, and is defined as:
A 11 A 21 Á A n1

adj A = 7A ij 7 T
A A 22 Á A n2
= § 12 ¥
o o o o
A 1n A 2n Á A nn
CD-150 Appendix D Review of Vectors and Matrices

For example, if
1 2 3
A = £2 3 2≥
3 3 4

then, A 11 = 1-12213 * 4 - 2 * 32 = 6, A 12 = 1-12312 * 4 - 3 * 22 = -2, Á , or

6 1 -5
adj A = £ -2 -5 4≥
-3 3 -1

D.2.5 Nonsingular Matrix

A matrix is of a rank r if the largest square array in the matrix having a non-zero de-
terminant is of size r. A square matrix with a non-zero determinant is called a full-rank
or nonsingular matrix. For example, consider

1 2 3
A = £2 3 4≥
3 5 7

A is a singular matrix because

ƒ A ƒ = 1 * 121 - 202 - 2 * 114 - 122 + 3 * 110 - 92 = 0
But A has a rank r = 2 because

a b = -1 Z 0
1 2
2 3

D.2.6 Inverse of a Nonsingular Matrix

If B and C are two n-square matrices such that BC = CB = I, then B is called the in-
verse of C and C the inverse of B. The common notation for the inverse is B-1 and C -1.

Theorem. If BC = I and B is nonsingular, then C = B-1, which means that the inverse
is unique.

Proof. By assumption,
BC = I

then
B-1BC = B-1I

or
IC = B-1
 D.2 Matrices CD-151

or
C = B-1

Two important results can be proved for nonsingular matrices.

1. If A and B are nonsingular n-squre matrices, then 1AB2-1 = B-1A-1

2. If A is nonsingular, then AB = AC implies that B = C.

Matrix inversion is used to solve n linearly independent equations. Consider

a11 a12 Á a1n x1 b1
a a22 Á a2n x2 b
§ 21 ¥ § ¥ = § 2¥
o o o o o o
an1 an2 Á ann xn bn

where xi represents the unknowns and aij and bi are constants. These n equations can
be written in matrix form as

AX = b
Because the equations are independent, A must be nonsingular. Thus

A-1AX = A-1b
or
X = A-1b

D.2.7 Methods of Computing the Inverse of a Matrix1

Adjoint Matrix Method. Given A, a nonsingular matrix of size n,
A 11 A 21 Á A n1
1 1 A A 22 Á A n2
A-1 = adj A = § 12 ¥
ƒAƒ ƒAƒ o o o o
A 1n A 2n Á A nn

For example, for

1 2 3
A = £2 3 2≥
3 3 4
6 1 -5
adj A = £ -2 -5 4 ≥, ƒ A ƒ = -7
-3 3 -1

1
TORA’s inverse module is based on LU decomposition method. See Press and Associates (1986)
CD-152 Appendix D Review of Vectors and Matrices

Hence
6 1 -5 - 67 - 17 5
7
1
A-1 = £ -2 -5 4 ≥ = £ 72 5
7 - 47 ≥
-7 3
-3 3 -1 7 - 37 1
7

Row Operations (Gauss-Jordan) Method. Consider the partitioned matrix 1A
I2,

where A is nonsingular. Premultiplying by A-1, we obtain
1A-1A
A-1I2 = 1I
A-12

Thus, applying a specific sequence of row transformations, A is changed to I and I is

changed to A-1. To illustrate the procedure, consider the system of equations:

1 2 3 x1 3
£2 3 2 ≥ £ x2 ≥ = £ 4 ≥
3 3 4 x3 5

The solution of X and the inverse of the basis matrix can be obtained directly by
considering
A-11A
I
b2 = 1I
A-1
A-1b2

The following iterations detail the transformation operation:

Iteration 0
1 2 3 1 0 0 3
£2 3 230 1 0 3 4≥
3 3 4 0 0 1 5

Iteration 1
1 2 3 1 0 0 3
£0 -1 -4 3 -2 1 0 3 -2 ≥
0 -3 -5 -3 0 1 -4

Iteration 2
1 0 -5 -3 2 0 -1
£0 1 43 2 -1 0 3 2≥
0 0 7 3 -3 1 2

Iteration 3
6
1 0 0 -7 - 17 5
7
3
7
£0 1 0 3 72 5
7 - 47 3 67 ≥
0 0 1 3 2
7 - 37 1
7 7
 D.2 Matrices CD-153

This gives x1 = 37, x2 = 67, and x3 = 27. The inverse of A is given by the right-hand-side
matrix, which is the same as obtained by the method of adjoint matrix.

Product Form of the Inverse. Suppose that two nonsingular matrices, B and Bnext, dif-
fer exactly in one column. Further, assume that B-1 is given. Then the inverse Bnext
-1
can
be computed using the formula
Bnext
-1
= EB-1

The matrix E is computed in the following manner. If the column vector Pj in B is re-
placed with the column vector Pr to produce Bnext, then E is constructed as an m-iden-
tity matrix with its rth column replaced by

-1B-1Pj21
-1B-1Pj22

, 1B-1Pj2r Z 0
1 o
¶ ∂
1B-1Pj2r
j =
+1 ; rth place
o
-1B-1Pj2m

If 1B-1Pj2r = 0, then Bnext

-1
does not exist.
The validity of the formula Bnext
-1
is proved as follows. Define F as an m-identity
matrix whose rth column is replaced by B-1Pj—that is,

F = 1e1, er - 1, B-1Pj, er + 1, Á , em2

Because Bnext differs from B only in that its rth column is replaced with Pj, then
Bnext = BF
Thus,
Bnext
-1
= 1BF2-1 = F -1B-1

The formula follows by setting E = F -1.

The product form can be used to invert any nonsingular matrix, B, in the follow-
ing manner. Start with B0 = I = B0-1. Next, construct B1 as an identity matrix, except
that the first column is replaced with the first column in B. Then

B1-1 = E1B0-1 = E1I = E1

In general, if we construct Bi as an identity matrix with its first i columns replaced with
the first i columns of B, then
Bi-1 = EiBi-1
- 1 = E iE i - 1Bi - 2 =
-1 Á = EiE i - 1 Á E1

This means that for the original matrix B,

B-1 = EnEn - 1 Á E1
CD-154 Appendix D Review of Vectors and Matrices

The following example illustrates the application of the product form of the in-
verse. Consider
2 1 0
B = £0 2 0≥
4 0 1
Iteration 0
1 0 0
B0 = B0-1 = £0 1 0≥
0 0 1
Iteration 1
2 0 0
B1 = £ 0 1 0≥
4 0 1
2 ;r = 1
B0-1P1 = P1 = £ 0 ≥
4
+ 12 0 0
E1 = £ - 02 1 0≥
- 42 0 1
1
2 0 0
B1-1 = £ 0 1 0≥
-2 0 1

Iteration 2
2 1 0
B2 = £ 0 2 0≥ = B
4 0 1
1 1
2 0 0 1 2
B1-1P2 = £ 0 1 0≥ £2≥ = £ 2≥ ; r = 2
-2 0 1 0 -2
1
2
1 - 0
2 1 - 14 0
E2 = § 0 + 12 0¥ = £0 1 0≥
2
0 - -22 1 0 1 1

1 - 14 0 1
2 0 0 1
2 - 14 0
B -1
= B2-1 = E2B1-1 = £0 1 0 ≥£ 0 1 0≥ = £ 0 1 0≥
2 2
0 1 1 -2 0 1 -2 1 1
 D.2 Matrices CD-155

Partitioned Matrix Method. Suppose that the two n-nonsingular matrices A and B be
partitioned as shown as follows:

A 11 A 12
1p * p2 1p * q2
A = § ¥ , A 11 nonsingular
A 21 A 22
1q * p2 1q * q2
B11 B12
1p * p2 1p * q2
B = § ¥
B21 B22
1q * p2 1q * q2

If B is the inverse of A, then from AB = I n, we have

A 11B11 + A 12B21 = I p
A 11B12 + B12A 22 = 0

Also, from BA = I n, we get

B21A 11 + B22A 21 = 0
B21A 12 + B22A 22 = I q

Because A 11 is nonsingular, A 11
-1
exists. Solving for B11, B12, B21, and B22, we get

B11 = A 11
-1
+ 1A 11
-1
A 122D -11A 21 A 11
-1
2
B12 = -1A 11 A 122D
-1 -1

B21 = -D -11A 21 A 11
-1
2
B22 = D -1

where
D = A 22 - A 211A 11
-1
A 122

To illustrate the use of these formulas, partition the matrix

12 3
A = £2 3 2≥
33 4

such that

A 11 = 112, A 12 = 12, 32, A 21 = a b, A 22 = a b

2 3 2
3 3 4
CD-156 Appendix D Review of Vectors and Matrices

In this case, A 11
-1
= 1 and

D = a b - a b11212, 32 = a b
3 2 2 -1 -4
3 4 3 -3 -5
5
- 47
D -1 = - 17 a b = a 73 1b
-5 4
3 -1 -7 7

Thus,

B11 = A - 67 B , B12 = A - 17 5
7 B
2 5
- 47
B12 = a 73 b, B22 = a 7
1b
7 - 37 7

which directly give B = A-1

D.2.8 Matrix Manipulations Using Excel

Excel provides facilities for automatically performing the following matrix manipulations:

1. Transpose.
2. Multiplication.
3. Inverse of a nonsingular matrix.
4. Determinant value of a nonsingular matrix.

Figure D.1 provides illustrative examples (file excelMatManip.xls). In Example 1

(Transpose), A is a 2 * 3 matrix whose elements are entered in the range A4:C5.
Transpose(A), or AT, appears in the user-specified range E4:F6. The steps for obtaining
the output in the selected range are:

1. Enter the formula = TRANSPOSE1A4 : C52 in cell E4.

2. Select (highlight) the output cells E4:F6.
3. Press F2.
4. Press CTRL + SHIFT + ENTER.

In Example 2, the elements of the input matrices A and B are entered in the re-
spective ranges A10:C13 and A16:A18. The output matrix is in the (user-selected)
range E10:E13. Next enter the formula = MMULT(A10:C13,A16:A18) in cell E10
and follow steps 2 through 4 exactly as in Example 1 (replacing E4:F6 with E10:E13).
Notice that MMULT(A16:A18,A10:C13) is undefined. In Example 3, the inverse of
the matrix in the range A22:C24 is assigned to the range E22:G24 by entering the
formula = MINVERSE1A22 : C242 in cell E22, then following steps 2, 3, and 4 as in Ex-
ample 1. Finally, in Example 4, the determinant of the matrix in the range A28:C30 is ob-
tained by entering the formula = MDETERM1A28 : C302 in the user-selected cell E28.
 D.3 Quadratic Forms CD-157

FIGURE D.1
Matrix manipulations using Excel (file excelMatManip.xls)

D.3 QUADRATIC FORMS

Given
X = 1x1, x2, Á , xn2T
and
a11 a12 Á a1n
a a22 Á a2n
A = § 21 ¥
o o o o
an1 an2 Á ann
the function
n n
Q1X2 = X TAX = a a aijxixj
i=1 j=1
CD-158 Appendix D Review of Vectors and Matrices

is called a quadratic form. The matrix A can always be assumed symmetric because
1a ij + a ji2
each element of every pair of coefficients aij and aij 1i Z j2 can be replaced by 2
without changing the value of Q(X).
As an illustration, the quadratic form

1 0 1 x1
Q1X2 = 1x1, x2, x32 £ 2 7 6 ≥ £ x2 ≥
3 0 2 x3

is the same as

1 1 2 x1
Q1X2 = 1x1, x2, x32 £ 1 7 3 ≥ £ x2 ≥
2 3 2 x3

Note that A is symmetric in the second case. We will assume henceforth that A is al-
ways symmetric.
The quadratic form is said to be

1. Positive-definite if Q1X2 7 0 for all X Z 0.

2. Positive-semidefinite if Q1X2 Ú 0 for all X, and there exists X Z 0 such that
Q1X2 = 0.
3. Negative-definite if -Q1X2 is positive-definite.
4. Negative-semidefinite if -Q1X2 is positive-semidefinite.
5. Indefinite in all other cases.

It can be proved that the necessary and sufficient conditions for the realization of
the preceding cases are

1. Q(X) is positive-definite (-semidefinite) if the values of the principal minor de-

terminants of A are positive (nonnegative).† In this case, A is said to be positive
definite (semidefinite).
2. Q(X) is negative-definite if the value of the kth principal minor determinant of A
has the sign of 1-12k, k = 1, 2, Á , n. In this case, A is called negative-definite.
3. Q(X) is negative-semidefinite if the kth principal minor determinant of A either
is zero or has the sign of 1-12k, k = 1, 2, Á , n.

†
The kth principal minor determinant of A n * n is defined by

a11 a12 Á a1k

4 a21 a22 Á a2k 4
, k = 1, 2, Á , n
o o o o
ak1 ak2 Á akk
 Problems CD-159

D.4 CONVEX AND CONCAVE FUNCTIONS

A function f(X) is said to be strictly convex if, for any two distinct points X 1 and X 2,
f1lX 1 + 11 - l2X 22 6 lf1X 12 + 11 - l2f1X 22
where 0 6 l 6 1. Conversely, a function f(X) is strictly concave if -f1X2 is strictly
convex.
A special case of the convex (concave) function is the quadratic form (see
Section D.3)
f1X2 = CX + X T AX
where C is a constant vector and A is a symmetric matrix. It can be proved that f(X) is
strictly convex if A is positive-definite and f(X) is strictly concave if A is negative definite.

PROBLEMS
1. Show that the following vectors are linearly dependent.
1 -2 1
(a) £ -2 ≥ £ 4 ≥ £ -2 ≥
3 -2 -1
2 4
-3 -6
(b) § ¥ § ¥
4 8
5 10
2. Given
1 4 9 7 -1 2
A = £2 5 -8 ≥, B = £ 9 4 8≥
3 7 2 3 6 10
find
(a) A + 7B
(b) 2A - 3B
(c) 1A + 7B2T
3. In Problem 2, show that AB Z BA
4. Consider the partitioned matrices

1 5 7
2 3 -4 5
2 -6 9
A = § ¥, B = £1 2 6 7≥
3 7 2
3 1 0 9
4 9 1

Find AB using partitioned matrix manipulation.

5. In Problem 2, find A-1 and B-1 using the following:
(a) Adjoint matrix method
(b) Row operations method
(c) Product form of the inverse
(d) Partitioned matrix method
CD-160 Appendix D Review of Vectors and Matrices

6. Consider
5
2 1 2 4 - 58 - 38
1 1
B = £0 2 1 ≥, B = £ 2
-1
4 - 14 ≥
1 1
4 0 5 -1 2 2

Suppose that the third vector P3 is replaced with the V3 = P1 + 2P2. This means that the
resulting matrix is singular. Show how the product form of the inverse discovers the singu-
larity of the matrix.
7. Use the product form of the inverse to verify whether each of the following equations
has a unique solution, no solution, or an infinity of solutions.
(a) x1 + 2x2 = 3
x1 + 4x2 = 2
(b) x1 + 2x2 = 5
-x1 - 2x2 = -5
(c) x1 + 7x2 + x3 = 5
4x1 + x2 + 3x3 = 8
x1 + 3x2 - 2x3 = 3
8. Verify the formulas given in Section B.2.7 for obtaining the inverse of a partitioned matrix.
9. Find the inverse of

A = a b, B nonsingular
1 G
H B

10. Show that the following quadratic form is negative definite.

Q1x1, x22 = 6x1 + 3x2 - 4x1x2 - 2x21 - 3x22 -

27
4

11. Show that the following quadratic form is positive definite.

Q1x1, x2, x32 = 2x21 + 2x22 + 3x23 + 2x1x2 + 2x2x3

12. Show that the function f1x2 = ex is strictly convex over all real values of x.
13. Show that the quadratic function

f1x1, x2, x32 = 5x12 + 5x2 2 + 4x3 2 + 4x1x2 + 2x2x3

is strictly convex.
14. In Problem 13, show that -f1x1, x2, x32 is strictly concave.

SELECTED REFERENCES
Hadley, G., Matrix Algebra, Addison-Wesley, Reading, MA, 1961.
Hohn, F., Elementary Matrix Algebra, 2nd ed., Macmillan, New York, 1964.
Press, W., B. Flannery, B. Teukolsky, and W. Vetterling, Numerical Recipes: The Art of Scientific
Computing, Cambridge University Press, Cambridge, England, 1986.