Machine Learning and Neural Computing

Yi Sun

School of Physics, Engineering & Computer Science

University of Hertfordshire

March, 01, 2024

Yi Sun Machine Learning and Neural Computing

Learning Outcomes

At the end of this session, you should know

▶ the basic idea behind principal component analysis (PCA).
▶ how to run PCA to analyse a given dataset using Python in
Jupyter Notebook.
▶ how to explain the output of the principal component analysis.

Yi Sun Machine Learning and Neural Computing

How do we spend our time in this session?

▶ The first part

▶ General information (7 minutes)
▶ Data visualisation (10 minutes)
▶ Two examples of PCA (15 minutes)
▶ Quiz (5 minutes)
▶ A little bit of maths (13 minutes)
▶ Interval: 5 minutes
▶ The second part
▶ Principal Component Analysis - Definition (15 minutes)
▶ One more example of PCA (5 minutes)
▶ Quiz (10 minutes)
▶ How to run PCA in a Jupyter Notebook (15 minutes)
▶ Kernel PCA (3 minutes)

Yi Sun Machine Learning and Neural Computing

Machine Learning

▶ Machine Learning:
is the scientific study of algorithms and statistical
models that computer systems use to perform a specific task
without using explicit instructions, relying on patterns
and inference instead [4].
▶ Approaches
▶ Supervised learning
▶ Unsupervised learning: identify natural clusters in the data.
▶ Reinforcement learning

Yi Sun Machine Learning and Neural Computing

Coursework: Data Classification

Coursework - Data Classification will be released on 08/03/2024

on Canvas.
▶ It counts for 30% of the overall assessment for this module.
▶ Hand-in date: 12pm 10/April/2024 via Canvas.
▶ Analyse a given dataset using:
▶ Principal Component Analysis, an unsupervised learning
algorithm,
▶ Support Vector Classifier, a supervised learning algorithm,
▶ Programming language: Python in Jupyter Notebook.
▶ Two submissions: report (.pdf) and Jupyter notebook.

Yi Sun Machine Learning and Neural Computing

Yi’s drop-in sessions

▶ on campus: Friday, 3pm-4:30pm in LC265.

from Week 32 (w/c 26/02/2024) to Week 35 (w/c
18/03/2024)
▶ online: Monday 12pm-1pm via Zoom, and links are given on
the Zoom page on the module site.
from Week 33 (w/c 04/03/2024) to Week 35 (w/c
18/03/2024), plus 08/04/2024.

Yi Sun Machine Learning and Neural Computing

Example

Yi Sun Machine Learning and Neural Computing

Example

Yi Sun Machine Learning and Neural Computing

Data Visualisation

▶ Data visualisation is the presentation of data in a pictorial or

graphical format.
▶ Data visualisation is a visual display of measured quantities by
means of the combined use of a coordination system, points,
lines, shapes, digits, letters quantified by visual attributes.
▶ Data visualisation can help us understand trends, grasp
difficult concepts or identify new patterns.

Yi Sun Machine Learning and Neural Computing

Visualising Multivariate Data

▶ Many datasets involve a larger number of variables, making

direct visualisation more difficult.
▶ There are various statistical plots

Yi Sun Machine Learning and Neural Computing

Example on the Sparrow Dataset

▶ After a severe storm, about half of the 49 female sparrows

died. Researchers wanted to know whether they could find
any support for Charles Darwin’s theory of natural selection.
▶ This dataset includes five body measurements from those 49
birds. The five measurements are total length, alar extent,
length of beak and head, length of humerus and length of keel
of sternum.
This example is taken from Multivariate Statistical Methods: A
primer by B. Manly.

Yi Sun Machine Learning and Neural Computing

Example on the Sparrow Dataset

Yi Sun Machine Learning and Neural Computing

Scatterplot Matrices

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - Goals of PCA

▶ Extract the most important information from the data;

▶ Compress the size of the data set by keeping only this
important information;
▶ Visualise the structure of the data.

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - the First Example
(1)

The data set contains 3 classes of 50 instances each:

▶ Attribute information
▶ sepal length in cm
▶ sepal width in cm
▶ petal length in cm
▶ petal width in cm
▶ Class information
▶ Iris Setosa
▶ Iris Versicolour
▶ Iris Virginica
https://archive.ics.uci.edu/ml/datasets/iris
Yi Sun Machine Learning and Neural Computing
Principal Component Analysis (PCA) - the First Example
(2)

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - the Second
Example (1)

 
−2 2

 −4 −4 

Consider a small data set X = (x1 , x2 ) = 
 4 4 

 2 −2 
0 0

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - the Second
Example (2)

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - the Second
Example (3)

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - Computation (1)

Yi Sun Machine Learning and Neural Computing

A Vector in Space

Vector can be shown as an arrow:

▶ starting point is the origin;
▶ ending point is the value of the vector.

Yi Sun Machine Learning and Neural Computing

Data - Matrices

▶ Rectangular array of numbers

▶ 2D array
▶ Number of Rows × Number of Columns

The figure is obtained from

https://www.wikihow.com/Understand-the-Basics-of-Matrices

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - Computation (2)

▶ Define the sample mean for each dimension

n
1X i
x̄ = x
n
i=1

▶ Define the standard deviation and variance for each dimension

sP
n i 2
i=1 (x − x̄)
σ=
n−1

var (x) = σ 2

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - Computation (3)

▶ Covariance is a measure between 2 dimensions.

Pn
(x i − x̄)(y i − ȳ )
cov (x, y ) = i=1
n−1
▶ The covariance matrix

σ12
 
cov (x1 , x2 ) · · · cov (x1 , xd )
 cov (x2 , x1 ) σ22 · · · cov (x2 , xd ) 
Σ=  ........................................


cov (xd , x1 ) cov (xd , x2 ) · · · σd2

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - Computation (4)

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - Computation (5)

▶ Each eigenvector ui has a corresponding eigenvalue (λi ).

▶ The number of eigenvectors is equal to the number of original
variables in the dataset.
▶ All the eigenvectors of a square matrix are perpendicular,
that is at right angles to each other.

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - the Second
Example (4)

▶ x̄1 = 0 and x̄2 = 0

 
10 6
▶ Σ=
6 10
▶ λ1 = 16; uT √1 √1
1 = [ 2 2]
▶ λ2 = 4; uT √1 √1
2 = [− 2 2 ]

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - Definitions

▶ Eigenvalues are ordered: λ1 ≥ λ2 ≥ . . . λd .

▶ The first Principal Component of the data is the direction
with largest variance (where the values of the individual
points varies the most) amongst the data.
▶ The second principal component is always orthogonal to the
first component.
▶ Up to d Principal Components can be found by this method.
▶ Each principal component is obtained as linear combinations
of the original variables.

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - In Practice

▶ Data pre-processing: for example, normalise the data so that

they have zero means and unit standard deviations.
▶ Calculate the covariance matrix (For example: use numpy.cov
in Python ).
▶ Compute the eigenvectors and eigenvalues of the covariance
matrix (For example: use linalg.eig from Python library
numpy.linalg).
▶ Select principal components by ordering eigenvalues.
▶ Derive the new dataset (projections in the PCA space) -
project the normalised the data onto principal components
projected data =
normalised data × selected principal compoents .

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - Information
Conservation in PCs

▶ Total variation in original data is

σ12 + σ22 + · · · + σd2 = trace(Σ)
The trace of a square matrix Σ is defined to be the sum of
the elements on the main diagonal of Σ.
▶ Total variation of PCs is
P
λi
▶ According to property of eigenvalues of Σ
X
λi = trace(Σ)

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - Measuring
Information in PCs
▶ Pλi is the amount of information contained in i th PC
λj
(λ1 +λP
2 +...+λk )
▶ λj is proportion information in first k PCs
▶ Want this large but also want k small - feature reduction
▶ Scree plot

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - the Second
Example (5)
Example - calculating the projection of the first data point [−2, 2].
λ1 = 16, uT √1 √1 T √1 √1
1 = [ 2 2 ]; λ2 = 4, u2 = [− 2 2 ]

1 1
projection on PC1 = x11 ∗ u11 + x12 ∗ u12 = −2 × √ + 2 × √ = 0.
2 2
1 1 4
projection on PC2 = x11 ∗u21 +x12 ∗u22 = −2×(− √ )+2× √ = √ .
2 2 2

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - the Third Example
(1)

Revisit the Sparrow dataset.

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - the Third Example
(2)

 
1.0000 0.7350 0.6618 0.6453 0.6051

 1.0000 0.6737 0.7685 0.5290 

Σ=
 1.0000 0.7632 0.5263 

 1.0000 0.6066 
1.0000
Here Σ in fact is a correlation matrix, since data have zero-mean
and unit-variance.
▶ trace(Σ) = 5.0000;
▶ λ1 = 3.6160, λ2 = 0.5315, λ3 = 0.3864, λ4 = 0.3016, λ5 =
0.1645;
▶
P
λi = 5.0000

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - the Third Example
(3)

Table: Eigenvectors and eigenvalues

u1 u2 u3 u4 u5
x1 0.4518 -0.0507 -0.6905 0.4204 0.3739
x2 0.4617 0.2996 -0.3405 -0.5479 -0.5301
x3 0.4505 0.3246 0.4545 0.6063 -0.3428
x4 0.4707 0.1847 0.4109 -0.3883 0.6517
x5 0.3977 -0.8765 0.1785 -0.0689 -0.1924
λi 3.6160 0.5315 0.3864 0.3016 0.1645

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - the Third Example
(4)

The first two principal components have captured

82.95%(= 3.6160+0.5315
5 ) of the total variation among the data.
2

Non−survivor
1.5 Survivor

0.5

0
PC2

−0.5

−1

−1.5

−2

−2.5

−3
−5 −4 −3 −2 −1 0 1 2 3 4 5
PC1

Yi Sun Machine Learning and Neural Computing

Question 1

Yi Sun Machine Learning and Neural Computing

Question 2

Given a dataset with 5 features, the variance of each feature is

shown as [3, 5, 1, 4.5, 2].
After doing a Principal Component Analysis, the first two largest
eigenvalues are λ1 = 5.1 and λ2 = 4.5.
What percentage of the total variance has been captured by the
first and the second principal components?

Yi Sun Machine Learning and Neural Computing

PCA in Python - sklearn.decomposition.PCA

▶ The fit function and the transform function can be used

separately.
▶ Reduced PCA features are stored in projections.
▶ https:
//scikit-learn.org/stable/modules/generated/
sklearn.preprocessing.\StandardScaler.html
▶ https://scikit-learn.org/stable/modules/
generated/sklearn.decomposition.\PCA.html

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA) - Summary

▶ Transforming original variables into new ones which are

uncorrelated and account for decreasing proportions of
variance in the data.
▶ Transformation is a rotation of original points.
▶ New variables are linear combinations of original ones. For
example:
PC 1 = ax1 + bx2 + cx3 + dx4 + · · ·
where a, b, c and d are elements of the first eigenvector u1 ,
that is u1 = [a, b, c, d, · · · ]; x1 , x2 , x3 and x4 are the original
(normalised) variables in the dataset.
▶ May reveal group structure, outliers, and so on.
▶ Can assess the importance of original variables by checking
the coefficients of principal components (loadings).

Yi Sun Machine Learning and Neural Computing

Principal Component Analysis (PCA)

▶ Getting the old data back

normalised data =
projected data × selected principal compoents T .

Yi Sun Machine Learning and Neural Computing

Kernel Principal Component Analysis

▶ Perform PCA in the new feature space;

▶ Use kernel methods to simplify the computation.

Yi Sun Machine Learning and Neural Computing

Kernel Principal Component Analysis

Kernel will be introduced in our next lecture.

Yi Sun Machine Learning and Neural Computing
Reference

[1] Smith, L.I. (2002) A tutorial on Principal Components Analysis

It can be downloaded from
http://www.cs.otago.ac.nz/cosc453/student_tutorials/
\principal_components.pdf

[2] Kernel Principal Component Analysis and its Applications in

Face Recognition and Active Shape Models, Quan Wang, 2014.

[3]StatQuest: PCA main ideas in only 5 minutes:

https://www.youtube.com/watch?v=HMOI_lkzW08

[4] https://en.wikipedia.org/wiki/Machine_learning

Yi Sun Machine Learning and Neural Computing