Professional Documents
Culture Documents
Basics For Statistical Machine Learning
Basics For Statistical Machine Learning
M. Keller
Basics for Statistical Machine Learning
Motivation
Linear Algebra Basics Linear Algebra
Vectors
Matrices
Determinant
Inverses
Mikaela Keller Diagonalization
1 / 22
Statistical ML
M. Keller
Motivation
Linear Algebra
Vectors
Matrices
Motivation Determinant
Inverses
Diagonalization
2 / 22
Statistical ML
M. Keller
Motivation
Linear Algebra
Vectors
Matrices
Motivation Determinant
Inverses
Diagonalization
3 / 22
Motivation Statistical ML
Motivation
Linear Algebra
Vectors
Matrices
Determinant
Inverses
Diagonalization
Motivation
Linear Algebra
2 Vectors
Matrices
Determinant
Inverses
0
Diagonalization
−2
−2 0 2
Motivation
100
2 (i) Linear Algebra
90
Vectors
80 Matrices
70 0 Determinant
Inverses
60 Diagonalization
50
−2
40
1 2 3 4 5 6 −2 0 2
M. Keller
Motivation
Linear Algebra
I Most of the problems described previously end up Vectors
Matrices
reformulated into: Determinant
Inverses
I curves or surfaces to be discovered, Diagonalization
7 / 22
Statistical ML
M. Keller
Motivation
Linear Algebra
Motivation Vectors
Matrices
Determinant
Inverses
Linear Algebra Basics Diagonalization
Vectors
Matrices
Determinant
Inverses
Matrix Diagonalization
8 / 22
Vectors Statistical ML
M. Keller
Motivation
x = ... , xT = (x1 , . . . , xm ) .
xm x1
ym
9 / 22
Vectors Statistical ML
M. Keller
Motivation
Linear Algebra
Vectors
I “x and y are orthogonal (x ⊥ y)” ⇔ xT y = 0. Matrices
Determinant
Inverses
I The norm (length) of x: Diagonalization
√
kxk = xT x
.
I The distance between 2 vectors x and y is defined as
d(x, y) = kx − yk:
10 / 22
Matrices Statistical ML
M. Keller
Motivation
n Equations with m unknows x1 , . . . , xm : Linear Algebra
Vectors
Matrices
a11 x1 +
... + a1m xm = b1 Determinant
Inverses
.. ⇔ Diagonalization
.
an1 x1 + . . . +anm xm = bn
a11 . . . a1m x1 b1
.. . . .. .. = .. ⇔
. . . . .
an1 . . . anm xm bn
11 / 22
Matrices Statistical ML
M. Keller
Motivation
n Equations with m unknows x1 , . . . , xm : Linear Algebra
Vectors
Matrices
a11 x1 +
... + a1m xm = b1 Determinant
Inverses
.. ⇔ Diagonalization
.
an1 x1 + . . . +anm xm = bn
(a11 ,
. . . , a1m ) x1 b1
.. .. .. ⇔
. =
. .
(an1 , . . . , anm ) xm bn
Ax = b.
11 / 22
Matrices Statistical ML
M. Keller
Motivation
n Equations with m unknows x1 , . . . , xm : Linear Algebra
Vectors
Matrices
a11 x1 +
... + a1m xm = b1 Determinant
Inverses
.. ⇔ Diagonalization
.
an1 x1 + . . . +anm xm = bn
(a11 ,
. . . , a1m ) x1 b1
.. .. .. ⇔
. =
. .
(an1 , . . . , anm ) xm bn
Ax = b.
11 / 22
Matrices Statistical ML
Motivation
Linear Algebra
2-D Example Vectors
Matrices
Determinant
Inverses
2x1 − x2 =0 Diagonalization
x1 + 3x2 =2
12 / 22
Matrices Statistical ML
Motivation
Linear Algebra
2-D Example Vectors
Matrices
Determinant
Inverses
2x1 − x2 =0 Diagonalization
x1 + 3x2 =2
(1,2)
(0,0)
12 / 22
Matrices Statistical ML
Motivation
Linear Algebra
2-D Example Vectors
Matrices
Determinant
Inverses
2x1 − x2 =0 Diagonalization
x1 + 3x2 =2
12 / 22
Matrices Statistical ML
Motivation
Linear Algebra
2-D Example Vectors
Matrices
Determinant
Inverses
2x1 − x2 =0 Diagonalization
x1 + 3x2 =2
(−1,1)
(2,0)
12 / 22
Matrices Statistical ML
Motivation
Linear Algebra
2-D Example Vectors
Matrices
Determinant
Inverses
2x1 − x2 =0 Diagonalization
x1 + 3x2 =2
12 / 22
Matrices Statistical ML
M. Keller
n Equations with m unknows x1 , . . . , xm :
Motivation
Linear Algebra
Ax = b ⇔ Vectors
Matrices
Determinant
a11 . . . a1m x1 b1 Inverses
.. .. .. .. = .. ⇔ Diagonalization
. . . . .
an1 . . . anm xm bn
13 / 22
Matrices Statistical ML
M. Keller
n Equations with m unknows x1 , . . . , xm :
Motivation
Linear Algebra
Ax = b ⇔ Vectors
Matrices
Determinant
a11 . . . a1m x1 b1 Inverses
.. . . .. .. = .. Diagonalization
⇔
. . . . .
an1 . . . anm xm bn
a11 a1m b1
.. .. .. .
x1 . + . . . + xm . = .
an1 anm bn
A real valued matrix An×m is also seen as a linear transfor-
mation:
A : Rm −→ Rn
x −→ Ax
.
13 / 22
Matrices Statistical ML
Motivation
⇔ x1 + x2 = .
x1 +3x2 = 2 1 3 2
14 / 22
Matrices Statistical ML
Motivation
⇔ x1 + x2 = .
x1 +3x2 = 2 1 3 2
14 / 22
Matrices Statistical ML
Motivation
⇔ x1 + x2 = .
x1 +3x2 = 2 1 3 2
14 / 22
Matrices Statistical ML
Motivation
⇔ x1 + x2 = .
x1 +3x2 = 2 1 3 2
14 / 22
Matrices Statistical ML
Motivation
Linear Algebra
2-D Example Vectors
Matrices
Determinant
Inverses
Diagonalization
2x1 −2x2 = 0 2 −2 0
⇔ x1 + x2 = .
x1 −x2 = 2 1 −1 2
15 / 22
Determinant Statistical ML
M. Keller
Recursive Definition: Let A be a square matrix (m × m),
Motivation
a11 . . . a1m Linear Algebra
X m
det(A) = ... .. .. Vectors
= (−1)1+j a1j det(M1j ),
. .
Matrices
Determinant
am1 . . . amm j=1 Inverses
Diagonalization
Example:
a11 a12 a13
det(A) = a21 a22 a23
a31 a32 a33
a a23 a a23 a a
= a11 22 + a12 21 + a13 21 22
a32 a33 a31 a33 a31 a32
= a11 (a22 a33 −a32 a23 )+a12 (a21 a33 −a31 a23 )+a13 (a21 a32 −a31 a22 )
16 / 22
Inverses Statistical ML
M. Keller
Motivation
Linear Algebra
I Definition: A square matrix Am×m is called non-singular Vectors
Matrices
or invertible if there exists a matrix Bm×m such that: Determinant
Inverses
Diagonalization
1 ... 0
AB = Im = ... . . . ... = BA.
0 ... 1
17 / 22
Determinants and Inverses Statistical ML
Motivation
Linear Algebra
2-D Example Vectors
Matrices
Determinant
Inverses
2 −1 Diagonalization
|det(A)| = |
| = |2 · 3 − 1 · (−1)|
1 3
a .2
a .1
18 / 22
Determinants and Inverses Statistical ML
Motivation
Linear Algebra
2-D Example Vectors
Matrices
Determinant
Inverses
2 −1 Diagonalization
|det(A)| = |
| = OP.OQ. sin(θ2 − θ1 ).
1 3
θ2 P
θ1
18 / 22
Matrices Statistical ML
M. Keller
19 / 22
Matrix Diagonalization Statistical ML
M. Keller
M. Keller
Motivation
Linear Algebra
Vectors
Matrices
Determinant
I The Singular Value Decomposition is a generalization of Inverses
Diagonalization
21 / 22
Acknowledgement Statistical ML
M. Keller
Motivation
Linear Algebra
Vectors
Matrices
Determinant
Inverses
I Sources of inspiration: Diagonalization
22 / 22