Vectors and Vector Spaces

Prof. Dr. Hani Mahdi

Computer and System Engineering Department
Spring 2020

1
 Vectors and Vector Spaces
• Matrices and their Applications 2nd
Time
• Types of Matrices (Shape)
• Applications
n
• Vectors in R
• Spanning Sets & Linear (In)Dependent
• The Four Spaces of a Matrix
• Rank of a Matrix and SLEs
• Basis & Dimension
• Vector Spaces
• Subspaces of Vector Space
First Proposed Fall 2019
2
Actual Topics will be stated in the Last Slide
 Networks

What if all the sources are Zeros?

3
 Matrices
• Definition: A matrix is a rectangular array of
numbers or symbolic elements
• Generally as in Databases, the rows of a
matrix will represent individuals cases
(people, items, objects,...) and columns will
represent attributes or characteristics
• The Dimension of a matrix is its number of
rows and columns, often denoted as
m x n (m rows by n columns)

4
 The Applications of Matrices
• Solving System of Linear Equations
• Equilibrium of Dynamic System (Eigenvectors)
• The Google Pagerank Algorithm
• Image Processing (Kernel)
• Computer Graphics
• Networks and Graph Theory
• Adjacency Matrices
• Machine Learning
• …
https://www.youtube.com/watch?v=rowWM-MijXU
5
 Matrix Representation
• Full Form
A=

• Abbreviated Forms
A = [aij] i=1, 2, …, n ; j= 1, 2, …, m
A j i=1, 2, …, n ; j= 1, 2, …, m
i
Am×n

6
 Types of Matrices (Shape)
Name Description Example
Rectangular matrix No. of rows is not
equal to no. of
columns (m <> n)
Square matrix No. of rows is equal to
no. of columns (m = n)
Row matrix A matrix with only 1
Row Vector row (m = 1)
Column matrix A matrix with only 1
Column Vector column (n = 1)
Vector

7
Special Types of Matrices (Square)
Name Description Example
Diagonal matrix Non-zero element in
principal diagonal and zero
in all other positions
Scalar matrix Diagonal matrix in which all
the elements on principal
diagonal and same
Identity matrix Diagonal matrix having
each diagonal element
equal to one (1)
Symmetric A matrix in which all the
matrix elements satisfy
(aij= aji and i ≠ j.)
Zero (Null) A matrix with all zero entries
matrix
8
 Applications of Vector Spaces
Many Applications for Example
• Spam Detecting
• Text Similarity
• Searching Engines
• ….

9
 The Vectors
• 2-D, 3-D, and n-D Vectors
• Vector Operations and
the Zero Vector 0
• Linear Combinations and
the Questions
• Spanning Sets and
Reconsidering the Null Space
• Linear Dependence and Independence
10
 2
Vectors in R

• Vectors in the plane (R2) is represented geometrically by a

directed line segment whose initial point is the origin and
whose terminal point is the point x = (x1, y1).
x1, y1: the components of x
y
• u = (u1, u2), v = (v1, v2), c is a scalar Terminal point
Equal: u = v iff u1 = v1& u2 = v2 (x1, y1)
Scalar multiplication: x
cv = c(v1, v2) = (cv1, cv2)
x
Initial point

11
 3
Vectors in R

12
 Row Vector & Column Vector
• Represent a vector u = (u1, u2, u3, …, un) in Rn
as a 1×n row matrix (row vector)

or

an n×1 column matrix (column vector)

13
 Vector Operations
• Let u, v, and w be vectors in the plane, and
let c and d be scalars.
Vector Addition
1. u + v is a vector in the plane
(closure under addition)
2. u + v = v + u
(commutative property)
3. (u + v) + w = v + (u + w)
(associative property)
4. u + 0 = u
5. u + (−u) = 0
Scalar Multiplication
6. cu is a vector in the plane
(closure under scalar multi.)
7. c(u + v) = cu + cv
(left distributive property)
8. (c + d)u = cu + du
(right distributive property)
9. c(du) = (cd)u
10. 1(u) = u
14
 Linear Combinations

• Example

15
 The Difficult Questions
If we have the vectors u1, u2, u3, u4 in C6
What vectors were you able to create?
• Could you could create a specific vector with a
suitable choice of four scalars?
• Could you create any possible vector from C6
by choosing the proper scalars?

16
 SLE as a Linear Combination


• We can now interpret the problem of solving

the system of equations as
determining values for the scalar multiples
that make the vector equation true.

17
 Another Example


• Row-reducing the augmented matrix leads to

the conclusion that the system is consistent
and has a free variable, hence infinitely many
solutions.
So for example, the two solutions

18
 Span of a Set of Vectors

• Example for a basic span)

Consider the set of 5 vectors, S, from C4

<S> is the infinite set of vectors formed from all

possible linear combinations of the elements

19
 Spanning Sets of Null Spaces
• Example
– Find a set of vectors, S, so that the null space of the matrix A
below is the span of S, that is, <S>=N(A)


With D = {1, 2, 4} and F = {3, 5}

20
 The Main Question
Which vectors are elements of the set, and
which are not?
Example


Just one of the Solutions

21
 Example
1 0 −1
• = 2 , 1 , 0
3 2 1

• The vectors in S2 lie in a plane

1 0 −1 1 0 −1
• 2 1 0 → 0 1 2
3 2 1 0 0 0
1
• = −2
1

22
Linearly Dependent and Independent
• S1 = {(1, 2, 3), (0, 1, 2), (−2, 0, 1)}
the vectors in S1 do not lie in a common plane
• S2 = {(1, 2, 3), (0, 1, 2), (−1, 0, 1)}
the vectorsz
in S2 lie in a common
z
plane

y y

x x

23
 Vectors’ Representations
• From the previous 2 Slides, we have 2
representations
1 0 −1
• = 2 , 1 , 0
3 2 1
• = {(1, 2, 3), (0, 1, 2), (−1, 0, 1)}
• Also we have the representation
• ={1 2 3 , 0 1 2 , −1 0 1 ,}

24
 Dependent and Independent
• We say that the vectors set
are Linearly Dependent if there exist scalars
such that not all of them are 0s
and
Otherwise, we say that the vectors are
Linearly Independent

25
 Test for Linearly Dependent
• Example for n = 4 vectors from C5,

Non Trivial
Solution!

S is Linearly
 α  Dependent.

26
 Test for Linearly (in)Dependent
• Example for Linearly Independent

S is Linearly
   xxxxxxxx.

27
 The Four Spaces of a Matrix
• Bases and Dimension of a Vector Space
• Row Bases and Column Bases
• Rank of a matrix r(A)
row-rank column-rank
• Nullity of a Matrix
• The Four Spaces of a Matrix
N(A) R(A) C(A) L(A)=N(AT)
• Orthogonal/Orthonormal Vectors
28
 Rethink about Dependency

•  SLE 

• REF   α1=0, α2=0, α3=0 The Null Space

 S is Linearly Independent
What is the Dimension of the considered Space?

29
 Vector basis
• A set of vectors (v1, ..., vk) is said to be
a basis for a vector space W if
(1) (v1, ..., vk) are linearly independent
(2) (v1, ..., vk) span W
• Standard bases:
R2 R3 Rn

30
 Dimension of a Space
• It often provides a concise, finite description
of an infinite vector space.
• The Standard Basis for R3
The set S = {(1, 0, 0), (0, 1, 0), (0, 0, 1)} is a basis for R 3.
 3 Dimensions Dim(3)
• A Nonstandard Basis for R2
The set S = {v1(1, 1), v2(1, −1)} is a basis for R2.
 2D Dim(2)
• The Dimension is the least number of the
vectors that represents the vector space
31
 Rethink about a previous Example
Slide 37 from first Part SLE

What is the Dimension of the Null Space?

What is (are) the Dimension (s) of the Space(s)?

What are the Dimensions of Column and Row Spaces? 32

The Column Bases and Dimension

• All The Columns corresponding to the

Non-Zero entries in the REF
2 1 3
• These are C(A) = 1 , 0 , 4
2 2 1
• What is the Dimension of C(A)?

33
 The Row Bases and Dimension

• All The Rows with Non-Zero entries in the REF

• These are
R(A)={[1 0 2 0 1], [0 1 -3 0 4], [0 0 0 1 2]}
• What is the Dimension of R(A)?
• But what the Difference between R(A) and
C(A)?
34
 Rethink again about Dependency

α

What is the Dimension of the Space?

What is the Dimension of the Null Space?

What is the Dimension of S? 35

 Spaces of a Matrix
• Null Space of a Matrix

• Column Space of a Matrix

• Row Space of a Matrix

• Left Null Space

36
 Rank of a Matrix
• “row-rank of a matrix” counts the max.
number of linearly independent rows.
• “column-rank of a matrix” counts the max.
number of linearly independent columns.
• Given any matrix,
its row-rank and column-rank are equal.
• The “rank of a matrix r(A) ”means either the
row-rank or column-rank
It is Dim(R(A)) or Dim(C(A))
37
 Nullity of a Matrix

Example Rank and nullity of a matrix

Each index in D creates a single basis vector, so the

column space of A will have dimension 4

Rank-Nullity Theorem: Rank + Nullity = n

38
 Dimensions of Four Spaces
• Suppose that A is an (m x n) matrix, and B is a
row-equivalent matrix in reduced row-echelon
form with r nonzero rows. Then

39
 Orthogonal/Orthonormal Vectors
• A set of vectors u1, u2, …, un is Orthogonal if
= , =0 for ≠
• set of vectors u1, u2, …, un is Orthonormal if
= , =0 for ≠
= , =1 for =1, …,n
• Example

40
 Vectors’ Venn Diagram
Vectors
Set of Vectors
Linear Independent Vectors
Orthogonal Vectors
Orthonormal Vectors

41
 The Row and Column Bases

2 1 3
• These are C(A) = 1 , 0 , 4
2 2 1
• What is the Dimension of C(A)?
R(A)={[1 0 2 0 1], [0 1 -3 0 4], [0 0 0 1 2]}

• Are R(A) and C(A) Orthogonal/Orthonormal vectors?

• What we can DO?

42
 The Orthogonal Column Basis

1 0 0
• These are C(A) = 0 , 1 , 0
0 0 1
• Is C(A) an Orthogonal/Orthonormal vector set?

43
 Rethink about Free Variables
• Slide 37 (SLE)

• Consider
What is the REF, if
Column 1 and 3
are exchanged?
44
 Vector Spaces and Subspaces
• Recycling Vector Space Definitions
set V along with 2 operations ‘♣♣’ and ‘♦
♦’
• Vector Spaces Examples and
NOT Vector Spaces Examples
• Spanning Set of a Vector Space
• Standard (Natural) Spanning Set
• Subspaces of Vector Spaces and
Examples of Subspaces
• Minimal Spanning Set
45
 Vector Spaces

A vector space consists of four entities:

a set of vectors, a set of scalars, and two operations
V：nonempty set
c：scalar
+ (u, v) = u + v：vector addition
• (c, u) = cu： scalar multiplication
(V , +, •) is called a vector space

46
 Vector Space Properties
Vector Addition
1. u + v is in V.
(closure under addition)
2. u + v = v + u
(commutative property)
3. (u + v) + w = v + (u + w)
(associative property)
4. V has a zero vector 0 s.t. for
every u in V, u + 0 = u
(additive identity)
5. For every u in V, there is a vector in V denoted by −u
s.t. u + (−u) = 0. (additive inverse)
Scalar Multiplication
6. cu is in V.
(closure under scalar multi.)
7. c(u + v) = cu + cv
(left distributive property)
8. (c + d)u = cu + du
(right distributive property)
9. c(du) = (cd)u (associative prop.)
10. 1(u) = u (scalar property)
47
 Recycling Definitions
• 2-D, 3-D, … , n-D (n-dimensional vectors)
• Take the properties as a definition of the
vector space
• No physical meaning (no arrows any more!)
• V is a set of objects (the “vectors”), with Two
Operations
“Vector Addition” and “Scalar Multiplication”

48
 The Definition
(Real) Vector Space ( V, ♣ ; R )
A vector space (over R) consists of a set V along with 2 operations ‘♣’ and ‘♦’ s.t.
(1) For the vector addition ♣ :
∀ v, w, u ∈ V
a) v ♣ w ∈ V ( Closure )
b) v ♣ w = w ♣ v ( Commutativity )
c) ( v ♣ w ) ♣ u = v ♣ ( w ♣ u ) ( Associativity )
d) ∃ 0 ∈ V s.t. v ♣ 0 = v ( Zero element )
e) ∃ −v ∈ V s.t. v ♣ (−v ) = 0 ( Inverse ) Inverse
(2) For the scalar multiplication ♦ :
∀ v , w ∈ V and a, b ∈ R, [ R is the real number field (R,+,×)
f) a ♦ v ∈ V ( Closure )
g) ( a + b ) ♦ v = ( a ♦ v ) ♣ (b ♦ v ) ( Distributivity )
h) a ♦ ( v ♣ w ) = ( a ♦ v ) ♣ ( a ♦ w )
i) ( a × b ) ♦ v = a ♦ ( b ♦ v ) ( Associativity )
j) 1 ♦ v = v

 ♣ is always written as + so that one writes v + w instead of v ♣ w

 × and ♦ are often omitted so that one writes a b v instead of ( a × b ) ♦ v
 Simple Verification
Vector Addition - Scalar Multiplication
1. u + v is in V.
2. V has a zero vector
s.t. for every u in V, u + 0 = u
3. c u is in V.

Or Combined
c1 u +c2 v is in V
c1 =0, c2 =0 c1 =1, c2 =-1, v =u
Inverse
50
 Examples of Vector Spaces (1/3)
(1) Matrix space:
V = M m×n (the set of all m ×n matrices with real values)
Simply V = Mmn For M22
0 0
=
0 0
 u11 u12   v11 v12   u11 + v11 u12 + v12 
+ = vector addition
u 
u22   v21 
v22   u21 + v21 u22 + v22 
 21
 u11 u12  cu11 cu12 
c  =  scalar multiplication
u u
 21 22   21 cu cu22 
51
 Examples of Vector Spaces (2/3)
(2) n-tuple space: Rn
= (0, 0, … , 0)

(u1 , u2 ,⋯ un ) + (v1 , v2 , ⋯ v2 ) = (u1 + v1 , u2 + v2 , ⋯ un + vn ) vector addition

α (u1 , u2 , ⋯ un ) = (α u1 , α u2 , ⋯α un ) scalar multiplication

52
 Examples of Vector Spaces (3/3)
(3) n-th degree polynomial space: V = { Pn ( x )}
(the set of all real polynomials of degree n or less)
p( x ) + q( x ) = (a0 + b0 ) + (a1 + b1 ) x + ⋯ + (an + bn ) x n
cp( x ) = ca0 + ca1 x + ⋯ + can x n

(4) The set of continuous functions:

53
NOT Vector Spaces Examples (1/2)
The set of second-degree polynomials is NOT a
vector space
• The set of all 2nd-degree polynomials is not a
vector space because it is not closed under
addition. For example,
p ( x) = x 2  (the 1st-degree poly.)
2  p ( x) + q( x) = x + 1
q ( x) = − x + x + 1

54
NOT Vector Spaces Examples (2/2)
• The set of Integers is NOT a vector space
Why?

• The set of Positive Real Numbers is NOT a

vector space
Why?

Others Examples for the two operations ♣ ♦!

55
 Application Example
Determine whether the following set of vectors in
2×2 matrix space is L.I. or L.D.
2 1 3 0 1 0 
S =   ,  ,  
0 1 2 1 2 0 
v1 v2 v3
Solution:
c1v1+c2v2+c3v3 = 0

2 1  3 0 1 0  0 0 
c1   + c2   + c3   = 
 0 1  2 1   2 0   0 0 

56
Application Example (Continued)
 2c1+3c2+ c3 = 0
c1 =0
2c2+2c3 = 0
c1 + c2 =0
 2 3 1 0 1 0 0 0
1 0 Gauss- Jordan Elimination 0
 0 0 1 0 0
 → 
0 2 2 0 0 0 1 0
   
1 1 0 0 0 0 0 0

 c1 = c2 = c3= 0 (This system has only the trivial solution.)

 S is linearly independent.
57
 Spanning Set of a Vector Space
• Does , , , span ?

• The question is that

Does a Linear combination span the vector
space?

!" +! +! +!$ =

58
 Spanning Set of a Vector Space
• Does {(t2+1), (t2+t), ( t+1)} span P2?
(the set of all real polynomials of degree n and less)
• The question a Linear combination span the vector
space
r1 (t2+1)+ r2 (t2+t)+ r3 ( t+1) = a t2+ b t + c
r1 + r2 =a
r2 + r3 = b
r1 + r3 = c

Homework: Show that this vector space spans P2.

59
 Standard (Natural) Spanning Set
1 0
• For R2 it is ,
0 1
A natural (typical) element in R2 is
1 0
So that = +
0 1
• The same idea for R3

The typical element is

1 0 0
The Natural spanning set will be 0 , 1 , 0
0 0 1
60
 Extend the idea for Vector Space
• A natural (typical) element M22 is
So that the natural spanning set is
1 0 0 1 0 0 0 0
, , ,
0 0 0 0 1 0 0 1
• A natural (typical) element M22 is

Homework: Show the natural element of P2.

61
 3 Independent Vectors in 3-D
• If we have 2 base vector in 2-D then the
standard base vectors will be {(1, 0), (0, 1)}
• Similarly the standard vectors for column
space
(Slides 32 42)

will be as shown in Slide 43

62
 Subspaces of Vector Spaces

• Subspace:
(V ,+,•) : a vector space
W ≠ φ  : a nonempty subset

W ⊆V
(W ,+,•) ：a vector space (under the operations of addition and
scalar multiplication defined in V)
 W is a subspace of V
 Trivial subspace:
Every vector space V has at least two subspaces.

(1) Zero vector space {0} is a subspace of V.

(2) V is a subspace of V.
 Venn Diagram of Subspace

V= %& , %' , … , %( ,

64
 Venn Diagram of Subspace

)
V= * |), *, + ∈ .
+

)
W= * |) + * − + =
+

65
 Two Approaches for Prove
• First
– It has zero vector
– It is closure under addition
– It is closure under scaler multiplication
• Second
– It can be represented as a linear summation of
base vectors of the origin space

66
Subspaces in R2

67
 Example
• Show that the line y=2x is a subspace of R2
and Find its dimension
0
•  for c=0
2 0
" +
"
• 2 + 2 = =
" 2 "+2 2
" / "
• s 2 = 2/ =
" " 2
– Second Method
1 0
• =c +2c
2 0 1

68
NOT Subspaces in R2

69
 Application Example
Finding the dimension of a subspace
Let W be the subspace of all symmetric matrices in
M2××2. What is the dimension of W?
  a b  
W =   a , b, c ∈ R 
  b c  
a b  1 0 0 1 0 0
∵  = a  +b  +c 
 b c   0 0   1 0   0 1 
1 0 0 1 0 0 
 S =   ,  ,   spans W and S is L.I.
0 0 1 0 0 1 
 S is a basis for W  dim(W ) = #(S) = 3
70
 Minimal Spanning Set

• W= | + − =0 =
+

1 0
• S= 0 , 1
1 1

71
 Column Space Again: Example

Minimal Subset


A Subspace of 3-D Space

72
End of Second Part of Linear Algebra
Spring 2020

73