Professional Documents
Culture Documents
Introd L
Introd L
Deep Learning
aka or related to
Deep Neural Networks
Deep Structural Learning
Deep Belief Networks
etc,
DL is providing breakthrough results in speech
recognition and image classification …
From this Hinton et al 2012 paper:
http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/38131.pdf
go here: http://yann.lecun.com/exdb/mnist/
From here:
http://people.idsia.ch/~juergen/cvpr2012.pdf
So, 1. what exactly is deep learning ?
but these algorithms are not good at learning the weights for
networks with more hidden layers
W1
W2
-2.5 f(x)
W3
1.4
-0.06
2.7
-8.6
-2.5 f(x)
0.002 x = -0.06×2.7 + 2.5×8.6 + 1.4×0.002 = 21.34
1.4
A dataset
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Training the neural network
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Training data
Fields class Initialise with random weights
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Training data
Fields class Present a training pattern
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 1.4
etc …
2.7
1.9
Training data
Fields class Feed it through to get output
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 1.4
etc …
2.7 0.8
1.9
Training data
Fields class Compare with target output
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 1.4
etc …
2.7 0.8
0
1.9 error 0.8
Training data
Fields class Adjust weights based on error
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 1.4
etc …
2.7 0.8
0
1.9 error 0.8
Training data
Fields class Present a training pattern
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 6.4
etc …
2.8
1.7
Training data
Fields class Feed it through to get output
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 6.4
etc …
2.8 0.9
1.7
Training data
Fields class Compare with target output
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 6.4
etc …
2.8 0.9
1
1.7 error -0.1
Training data
Fields class Adjust weights based on error
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 6.4
etc …
2.8 0.9
1
1.7 error -0.1
Training data
Fields class And so on ….
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0 6.4
etc …
2.8 0.9
1
1.7 error -0.1
1
strong +ve weight
low/zero weight
63
What does this unit detect?
1 5 10 15 20 25 …
…
1
strong +ve weight
low/zero weight
63
What does this unit detect?
1 5 10 15 20 25 …
…
1
strong +ve weight
low/zero weight
63
What does this unit detect?
1 5 10 15 20 25 …
…
1
strong +ve weight
low/zero weight
63
What does this unit detect?
1 5 10 15 20 25 …
…
1
strong +ve weight
low/zero weight
63
What features might you expect a good NN
to learn, when trained with data like this?
vertical lines
63
Horizontal lines
63
Small circles
63
Small circles
detect lines in
Specific positions
etc …
etc …
( horizontal line,
“RHS vertical lune” v
“upper loop”, etc…
successive layers can learn higher-level features …
detect lines in
Specific positions
etc …
etc …
( horizontal line,
“RHS vertical lune” v
“upper loop”, etc…