Vectors, matrices, and gradients

Rupam Mahmood
Jan 11, 2024
Vectors and Matrices notations used in the book
Vector notations
● A vector is an ordered collection of numbers

● [1 4 3] is a row vector, which is three-dimensional, in short, a 3-vector

● is a column vector, written alternatively as [1 4 3]T, where T is transpose

● When a vector is represented symbolically, a lowercase bold letter will be

used and will denote a column vector
■ For example, x = [1 4 3]T; other examples, a, b, z

● Symbolically, a row vector will be written in terms of a column vector

■ For example, yT is row vector as y must be a column vector

● Symbolic scalars are lowercase non-bold letters, such as, a, b, or x

Matrix notations
● A matrix is a rectangular array of numbers arranged in rows and columns

● is a matrix with 3 rows and 2 columns or a 3 by 2 matrix

● When a matrix is represented symbolically, an uppercase bold letter will be

used, for example, the symbol A can denote the above matrix

● We can also write A 𝜖 M(3, 2), where M is the space of all 3 by 2 matrices

● Vectors can be seen as special matrices, M(m, 1) being a column vector, and
M(1, n) being a row vector
Notations for elements of vectors and matrices
● For a matrix B, the element at row i and column k is denoted by bi,k or bik

○ Here the elements are of matching letter but lowercase non-bold because they are scalars

● For a vector x, the kth element is denoted by xk

● When necessity arises, we will denote parts of matrices/vectors using the

operator [ ], which is a bit more complicated

○ This operator [ ] inspects inside a vector or matrix object

○ For example, bik = [B]i,k is an element of B, and xk = [x]k is an element of x

○ It also allows us to denote parts of a matrix, for example, [B]i,: denotes the ith row of B
Dot product of two vectors
● A dot product of two vectors x and y with matching dimensions n is given by

● For example, for x = [1 4 3]T and y = [7 -6 0]T, the dot product is

1·7 + 4·(-6) + 3·0 = 7 - 24 = -17

● In this course, we will denote such dot products as xTy, which is equivalent to
the following matrix multiplication

● Note that xT is a row vector

Example of matrix-vector multiplication
Calculate the following multiplication:

The following is the formula we typically use:

An alternate rule for matrix-vector multiplication
Rules for matrix-vector multiplication
Vectors and Matrices: examples
1. If x and y are n-dimensional column vectors, what is xTy?
a. A matrix
b. A vector
c. A scalar

2. What is xyT?
a. A non-square matrix
b. A vector
c. A square matrix
d. A scalar

3. If A is an n-by-n matrix, what is Ax?

a. A matrix
b. A scalar
c. A row vector
d. A column vector

4. What is xTAy?
a. A matrix
b. A scalar
c. A vector
Example of matrix-vector multiplication
Prove the following:

Here, A is in M(m, n), x is in M(n, 1), and b is in M(m, 1)

Gradients
Definition:

● Described as the gradient of f(x) wrt x

● Here, f must be a scalar-valued function that takes vector inputs

● So, gradient is an operation on a scalar wrt a vector

● The gradient operator produces another vector of the same dimension as x

● Each elements are partial derivatives

Gradients

Rules for derivatives Rules for gradients

f(x) = bx; df/dx = b f(x) = bTx = xTb; ∇f(x) = b

f(x) = x2; df/dx = 2x f(x) = xTx; ∇f(x) = 2x

f(x) = bx2; df/dx = 2bx f(x) = xTBx; ∇f(x) = 2Bx

B is symmetric
Gradients: example 1

If f(x) = (b - Ax)T(b - Ax), show that ∇f(x) = -2AT(b - Ax).

Proof: ∇f(x) = ∇(bTb -2bTAx + xTATAx)

= ∇(bTb) -2∇(bTAx) + ∇(xTATAx)

= 0 - 2 ATb + 2ATAx

= -2AT(b - Ax).
Gradients: example 2

If f(x) = ½ xTAx + xTb + c, show that ∇f(x) = Ax + b.

A is symmetric
Vectors, Matrices, and derivatives: resources
● Vectors and matrices (eclass)

● derivatives (eclass)

● The Matrix Cookbook