Lecture 5 PDF

Lecture 5:
Convolutional Neural Networks
Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 1 April 13, 2021

Administrative
Assignment 1 due Friday April 16, 11:59pm

- Important: tag your solutions with the corresponding hw
question in gradescope!
Assignment 2 will also be released April 16th

Administrative
Project proposal due Monday Apr 19, 11:59pm
This Friday’s discussion section will discuss how to design a

project

Last time: Neural Networks
Linear score function:
2-layer Neural Network
x W1 h W2 s
3072 100 10

f
Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 -5 April 13, 2021

“local gradient”

“Upstream
gradient”

“Downstream
gradients”
f
“Upstream
gradient”

“Downstream
gradients”
f
“Upstream
gradient”

“Downstream
gradients”
f
“Upstream
gradient”
So far: backprop with scalars
What about vector-valued functions?

Recap: Vector derivatives
Scalar to Scalar
Regular derivative:
If x changes by a
small amount, how
much will y change?

Scalar to Scalar Vector to Scalar
Regular derivative: Derivative is Gradient:
If x changes by a For each element of x,

small amount, how if it changes by a small
much will y change? amount then how much
will y change?

Scalar to Scalar Vector to Scalar Vector to Vector
Regular derivative: Derivative is Gradient: Derivative is Jacobian:
If x changes by a For each element of x, For each element of x, if it

small amount, how if it changes by a small changes by a small amount
much will y change? amount then how much then how much will each
will y change? element of y change?

Backprop with Vectors
Loss L still a scalar!
Dx
Dz
f
Dy
Dx
Dz
f
Dy
“Upstream gradient”
Dx
Dz
f
Dy Dz
For each element of z, how
much does it influence L?
Dx
Dz
“Downstream
gradients”
f
Dy Dz
Dx “local
gradients”
Dz
“Downstream
gradients”
f
Dy Dz
Dx “local
gradients”
[Dx x Dz] Dz
“Downstream
gradients”
f
[Dy x Dz]
Dy Dz
Jacobian
matrices “Upstream gradient”
Dx “local
gradients”
Dx [Dx x Dz] Dz
“Downstream
gradients”
Matrix-vector
multiply
f
[Dy x Dz]
Dy Dz
Jacobian
Dy For each element of z, how
Gradients of variables wrt loss have same dims as the original variable

Dx
Dx Dz
f
Dy Dz
Dy For each element of z, how
4D input x: 4D output z:
[ 1 ] [ 1 ]
[ -2 ] f(x) = max(0,x) [ 0 ]
[ 3 ] (elementwise) [ 3 ]
[ -1 ] [ 0 ]

[ 1 ] [ 1 ]
[ -2 ] f(x) = max(0,x) [ 0 ]
[ -1 ] [ 0 ]
4D dL/dz:
[ 4 ]
[ -1 ] Upstream
[ 5 ] gradient
[ 9 ]

[ 1 ] [ 1 ]
[ -2 ] f(x) = max(0,x) [ 0 ]
[ -1 ] [ 0 ]
Jacobian dz/dx 4D dL/dz:

[1000] [ 4 ]
[0000] [ -1 ] Upstream
[0010] [ 5 ] gradient
[0000] [ 9 ]

[ 1 ] [ 1 ]
[ -2 ] f(x) = max(0,x) [ 0 ]
[ -1 ] [ 0 ]
[dz/dx] [dL/dz] 4D dL/dz:

[1000][4 ] [ 4 ]
[ 0 0 0 0 ] [ -1 ] [ -1 ] Upstream
[0010][5 ] [ 5 ] gradient
[0000][9 ] [ 9 ]

[ 1 ] [ 1 ]
[ -2 ] f(x) = max(0,x) [ 0 ]
[ -1 ] [ 0 ]
4D dL/dx: [dz/dx] [dL/dz] 4D dL/dz:

[4] [1000][4 ] [ 4 ]
[0] [ 0 0 0 0 ] [ -1 ] [ -1 ] Upstream
[5] [0010][5 ] [ 5 ] gradient
[0] [0000][9 ] [ 9 ]

[ 1 ] [ 1 ]
[ -2 ] f(x) = max(0,x) [ 0 ]
Jacobian is sparse:
off-diagonal entries
always zero! Never [ -1 ] [ 0 ]
explicitly form
Jacobian -- instead
use implicit 4D dL/dx: [dz/dx] [dL/dz] 4D dL/dz:
multiplication [4] [1000][4 ] [ 4 ]
[0] [ 0 0 0 0 ] [ -1 ] [ -1 ] Upstream
[5] [0010][5 ] [ 5 ] gradient
[0] [0000][9 ] [ 9 ]

[ 1 ] [ 1 ]
[ -2 ] f(x) = max(0,x) [ 0 ]
Jacobian is sparse:
off-diagonal entries
always zero! Never [ -1 ] [ 0 ]
explicitly form
Jacobian -- instead
use implicit 4D dL/dx: [dz/dx] [dL/dz] 4D dL/dz:
multiplication [4] [ 4 ]
[0] z [ -1 ] Upstream
[5] [ 5 ] gradient
[0] [ 9 ]

Backprop with Matrices (or Tensors) Loss L still a scalar!
dL/dx always has the

[Dx×Mx]
same shape as x!
[Dz×Mz]
Matrix-vector
multiply
f
[Dy×My]
Jacobian
matrices


[Dx×Mx]
same shape as x!
[Dx×Mx]
[Dz×Mz]
“Downstream
gradients”
Matrix-vector
multiply
f
[Dy×My] [Dz×Mz]
Jacobian
[Dy×My] For each element of z, how


[Dx×Mx] “local same shape as x!
gradients”
[Dx×Mx]
[Dz×Mz]
“Downstream
Matrix-vector
gradients” multiply
[Dy×My] [Dz×Mz]
Jacobian
For each element of y, how much
does it influence each element of z? much does it influence L?

[Dx×Mx] “local same shape as x!
gradients”
[Dx×Mx] [(Dx×Mx)×(Dz×Mz)] [Dz×Mz]
“Downstream
Matrix-vector
gradients” multiply [(Dy×My)×(Dz×Mz)]
[Dy×My] [Dz×Mz]
Jacobian
For each element of y, how much
does it influence each element of z? much does it influence L?

Backprop with Matrices y: [N×M]
[13 9 -2 -6 ]
x: [N×D] Matrix Multiply [ 5 2 17 1 ]
[ 2 1 -3 ]
[ -3 4 2 ] dL/dy: [N×M]
w: [D×M] [ 2 3 -3 9 ]
[ 3 2 1 -1] [ -8 1 4 6 ]
[ 2 1 3 2]
[ 3 2 1 -2]
Also see derivation in the course notes:

http://cs231n.stanford.edu/handouts/linear-backprop.pdf

[13 9 -2 -6 ]
[ 2 1 -3 ]
[ -3 4 2 ] dL/dy: [N×M]
w: [D×M] [ 2 3 -3 9 ]
[ 3 2 1 -1] Jacobians: [ -8 1 4 6 ]
[ 2 1 3 2] dy/dx: [(N×D)×(N×M)]
[ 3 2 1 -2] dy/dw: [(D×M)×(N×M)]
For a neural net we may have
N=64, D=M=4096
Each Jacobian takes ~256 GB of
memory! Must work with them implicitly!

[13 9 -2 -6 ]
[ 2 1 -3 ]
[ -3 4 2 ] dL/dy: [N×M]
w: [D×M] [ 2 3 -3 9 ]
[ 3 2 1 -1] Q: What parts of y [ -8 1 4 6 ]
[ 2 1 3 2] are affected by one
[ 3 2 1 -2] element of x?

[13 9 -2 -6 ]
[ 2 1 -3 ]
[ -3 4 2 ] dL/dy: [N×M]
w: [D×M] [ 2 3 -3 9 ]
[ 3 2 1 -1] Q: What parts of y [ -8 1 4 6 ]
[ 2 1 3 2] are affected by one
[ 3 2 1 -2] element of x?
A: affects the
whole row

[13 9 -2 -6 ]
[ 2 1 -3 ]
[ -3 4 2 ] dL/dy: [N×M]
w: [D×M] [ 2 3 -3 9 ]
[ 3 2 1 -1] Q: What parts of y Q: How much [ -8 1 4 6 ]
[ 2 1 3 2] are affected by one does
[ 3 2 1 -2] element of x? affect ?
A: affects the
whole row

[13 9 -2 -6 ]
[ 2 1 -3 ]
[ -3 4 2 ] dL/dy: [N×M]
w: [D×M] [ 2 3 -3 9 ]
A: affects the A:
whole row

[13 9 -2 -6 ]
[ 2 1 -3 ]
[ -3 4 2 ] dL/dy: [N×M]
w: [D×M] [ 2 3 -3 9 ]
A: affects the A:
[N×D] [N×M] [M×D] whole row

[13 9 -2 -6 ]
[ 2 1 -3 ]
[ -3 4 2 ] dL/dy: [N×M]
w: [D×M] [ 2 3 -3 9 ]
[ 3 2 1 -1] [ -8 1 4 6 ]
[ 2 1 3 2] By similar logic:
[ 3 2 1 -2]
[N×D] [N×M] [M×D] [D×M] [D×N] [N×M] These formulas are

easy to remember: they
are the only way to
make shapes match up!

Wrapping up: Neural Networks
Linear score function:
2-layer Neural Network
x W1 h W2 s
3072 100 10

Next: Convolutional Neural Networks
Illustration of LeCun et al. 1998 from CS231n 2017 Lecture 1

A bit of history...
The Mark I Perceptron machine was the first
implementation of the perceptron algorithm.
The machine was connected to a camera that used

20×20 cadmium sulfide photocells to produce a 400-pixel
image.
recognized
letters of the alphabet
update rule:
Frank Rosenblatt, ~1957: Perceptron

This image by Rocky Acosta is licensed under CC-BY 3.0

A bit of history...
These figures are reproduced from Widrow 1960, Stanford Electronics Laboratories Technical
Widrow and Hoff, ~1960: Adaline/Madaline Report with permission from Stanford University Special Collections.

A bit of history...
recognizable math
Illustration of Rumelhart et al., 1986 by Lane McIntosh,

copyright CS231n 2017
Rumelhart et al., 1986: First time back-propagation became popular

A bit of history...
[Hinton and Salakhutdinov 2006]
Reinvigorated research in
Deep Learning
Illustration of Hinton and Salakhutdinov 2006 by Lane

McIntosh, copyright CS231n 2017

First strong results
Acoustic Modeling using Deep Belief Networks
Abdel-rahman Mohamed, George Dahl, Geoffrey Hinton, 2010
Context-Dependent Pre-trained Deep Neural Networks
for Large Vocabulary Speech Recognition
George Dahl, Dong Yu, Li Deng, Alex Acero, 2012
Imagenet classification with deep convolutional

Illustration of Dahl et al. 2012 by Lane McIntosh, copyright
neural networks CS231n 2017
Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton, 2012
Figures copyright Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, 2012. Reproduced with permission.

A bit of history:
Hubel & Wiesel,

1959
RECEPTIVE FIELDS OF SINGLE
NEURONES IN
THE CAT'S STRIATE CORTEX
1962
RECEPTIVE FIELDS, BINOCULAR
INTERACTION
AND FUNCTIONAL ARCHITECTURE IN
THE CAT'S VISUAL CORTEX
Cat image by CNX OpenStax is licensed
1968... under CC BY 4.0; changes made

A bit of history Human brain
Topographical mapping in the cortex:

nearby cells in cortex represent
nearby regions in the visual field
Visual
cortex
Retinotopy images courtesy of Jesse Gomez in the

Stanford Vision & Perception Neuroscience Lab.

Hierarchical organization
Illustration of hierarchical organization in early visual

pathways by Lane McIntosh, copyright CS231n 2017

A bit of history:
Neocognitron
[Fukushima 1980]
“sandwich” architecture (SCSCSC…)

simple cells: modifiable parameters
complex cells: perform pooling

A bit of history:
Gradient-based learning applied to
document recognition
[LeCun, Bottou, Bengio, Haffner 1998]
LeNet-5

A bit of history:
ImageNet Classification with Deep
[Krizhevsky, Sutskever, Hinton, 2012]
Figure copyright Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, 2012. Reproduced with permission.
“AlexNet”

Fast-forward to today: ConvNets are everywhere
Classification Retrieval
Figures copyright Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, 2012. Reproduced with permission.

Detection Segmentation
Figures copyright Clement Farabet, 2012.

Figures copyright Shaoqing Ren, Kaiming He, Ross Girschick, Jian Sun, 2015. Reproduced with permission.
Reproduced with permission. [Farabet et al., 2012]
[Faster R-CNN: Ren, He, Girshick, Sun 2015]

This image by GBPublic_PR is

licensed under CC-BY 2.0
NVIDIA Tesla line

(these are the GPUs on rye01.stanford.edu)
Note that for embedded systems a typical setup

Photo by Lane McIntosh. Copyright CS231n 2017.
would involve NVIDIA Tegras, with integrated
self-driving cars GPU and ARM-based CPU cores.

[Taigman et al. 2014] Activations of inception-v3 architecture [Szegedy et al. 2015] to image of Emma McIntosh,
used with permission. Figure and architecture not from Taigman et al. 2014.
Illustration by Lane McIntosh,

Figures copyright Simonyan et al., 2014. photos of Katie Cumnock used
[Simonyan et al. 2014] Reproduced with permission. with permission.

Images are examples of pose estimation, not actually from Toshev & Szegedy 2014. Copyright Lane McIntosh.
[Toshev, Szegedy 2014]
[Guo et al. 2014] Figures copyright Xiaoxiao Guo, Satinder Singh, Honglak Lee, Richard Lewis,
and Xiaoshi Wang, 2014. Reproduced with permission.

[Levy et al. 2016] Figure copyright Levy et al. 2016.

Reproduced with permission.
Photos by Lane McIntosh.

[Sermanet et al. 2011] Copyright CS231n 2017.
From left to right: public domain by NASA, usage permitted by

ESA/Hubble, public domain by NASA, and public domain. [Ciresan et al.]
[Dieleman et al. 2014]

This image by Christin Khan is in the public domain Photo and figure by Lane McIntosh; not actual
and originally came from the U.S. NOAA. example from Mnih and Hinton, 2010 paper.
Whale recognition, Kaggle Challenge Mnih and Hinton, 2010

No errors Minor errors Somewhat related
Image
Captioning
[Vinyals et al., 2015]
[Karpathy and Fei-Fei,
2015]
A white teddy bear sitting in A man in a baseball A woman is holding a cat

the grass uniform throwing a ball in her hand
All images are CC0 Public domain:

https://pixabay.com/en/luggage-antique-cat-1643010/
https://pixabay.com/en/teddy-plush-bears-cute-teddy-bear-1623436/
https://pixabay.com/en/surf-wave-summer-sport-litoral-1668716/
https://pixabay.com/en/woman-female-model-portrait-adult-983967/
https://pixabay.com/en/handstand-lake-meditation-496008/
A man riding a wave on A cat sitting on a A woman standing on a https://pixabay.com/en/baseball-player-shortstop-infield-1045263/
top of a surfboard suitcase on the floor beach holding a surfboard Captions generated by Justin Johnson using Neuraltalk2

Original image is CC0 public domain
Starry Night and Tree Roots by Van Gogh are in the public domain
Bokeh image is in the public domain Gatys et al, “Image Style Transfer using Convolutional Neural Networks”, CVPR 2016
Figures copyright Justin Johnson, 2015. Reproduced with permission. Generated using the Inceptionism approach
Stylized images copyright Justin Johnson, 2017; Gatys et al, “Controlling Perceptual Factors in Neural Style Transfer”, CVPR 2017
from a blog post by Google Research.
reproduced with permission


Recap: Fully Connected Layer
32x32x3 image -> stretch to 3072 x 1
input activation
1 1
10 x 3072
3072 10
weights

Fully Connected Layer
32x32x3 image -> stretch to 3072 x 1
input activation
1 1
10 x 3072
3072 10
weights
1 number:
the result of taking a dot product
between a row of W and the input
(a 3072-dimensional dot product)

Convolution Layer
32x32x3 image -> preserve spatial structure
32 height
32 width
3 depth

Convolution Layer
32x32x3 image
5x5x3 filter
32
Convolve the filter with the image

i.e. “slide over the image spatially,
computing dot products”
32
3

Convolution Layer Filters always extend the full
depth of the input volume
32x32x3 image
5x5x3 filter
32
Convolve the filter with the image

i.e. “slide over the image spatially,
computing dot products”
32
3

Convolution Layer
32x32x3 image
5x5x3 filter
32
1 number:
the result of taking a dot product between the
filter and a small 5x5x3 chunk of the image
32 (i.e. 5*5*3 = 75-dimensional dot product + bias)
3

Convolution Layer
32
32
3

Convolution Layer
32
32
3

Convolution Layer
32
32
3

Convolution Layer
32
32
3

Convolution Layer
activation map
32x32x3 image
5x5x3 filter
32
28
convolve (slide) over all

spatial locations
32 28
3 1

consider a second, green filter
Convolution Layer
32x32x3 image activation maps
5x5x3 filter
32
28

spatial locations
32 28
3 1

For example, if we had 6 5x5 filters, we’ll get 6 separate activation maps:
activation maps
32
28
Convolution Layer
32 28
3 6
We stack these up to get a “new image” of size 28x28x6!

Preview: ConvNet is a sequence of Convolution Layers, interspersed with
activation functions
32 28
CONV,
ReLU
e.g. 6
5x5x3
32 filters 28
3 6

Preview: ConvNet is a sequence of Convolution Layers, interspersed with
activation functions
32 28 24
CONV, ….
CONV, CONV,
ReLU ReLU ReLU
e.g. 6 e.g. 10
5x5x3 5x5x6
32 filters 28 24
filters
3 6 10

Visualization of VGG-16 by Lane McIntosh. VGG-16
Preview [Zeiler and Fergus 2013] architecture from [Simonyan and Zisserman 2014].

Preview

one filter =>
one activation map example 5x5 filters
(32 total)
We call the layer convolutional

because it is related to convolution
of two signals:
elementwise multiplication and sum of

a filter and the signal (image)
Figure copyright Andrej Karpathy.

preview:

A closer look at spatial dimensions:
activation map
32x32x3 image
5x5x3 filter
32
28

spatial locations
32 28
3 1

7
7x7 input (spatially)
assume 3x3 filter

7
assume 3x3 filter

7
assume 3x3 filter

7
assume 3x3 filter

7
assume 3x3 filter
=> 5x5 output

7

7
assume 3x3 filter
applied with stride 2

7
assume 3x3 filter

7
assume 3x3 filter
=> 3x3 output!
7

7
assume 3x3 filter
applied with stride 3?

7
assume 3x3 filter
applied with stride 3?
7 doesn’t fit!
cannot apply 3x3 filter on
7x7 input with stride 3.

N
Output size:
(N - F) / stride + 1
F
N e.g. N = 7, F = 3:
F stride 1 => (7 - 3)/1 + 1 = 5
stride 2 => (7 - 3)/2 + 1 = 3
stride 3 => (7 - 3)/3 + 1 = 2.33 :\

In practice: Common to zero pad the border
0 0 0 0 0 0
e.g. input 7x7
0 3x3 filter, applied with stride 1
0 pad with 1 pixel border => what is the output?
0
(recall:)
(N - F) / stride + 1

0 0 0 0 0 0
e.g. input 7x7
0
7x7 output!
0
(recall:)
(N + 2P - F) / stride + 1

0 0 0 0 0 0
e.g. input 7x7
0
7x7 output!
0 in general, common to see CONV layers with
stride 1, filters of size FxF, and zero-padding with
(F-1)/2. (will preserve size spatially)
e.g. F = 3 => zero pad with 1
F = 5 => zero pad with 2
F = 7 => zero pad with 3

Remember back to…
E.g. 32x32 input convolved repeatedly with 5x5 filters shrinks volumes spatially!
(32 -> 28 -> 24 ...). Shrinking too fast is not good, doesn’t work well.
32 28 24
….
CONV, CONV, CONV,
ReLU ReLU ReLU
e.g. 6 e.g. 10
5x5x3 5x5x6
32 filters 28 filters 24
3 6 10
Examples time:
Input volume: 32x32x3

10 5x5 filters with stride 1, pad 2
Output volume size: ?
Examples time:

Output volume size:

(32+2*2-5)/1+1 = 32 spatially, so
32x32x10
Examples time:

Number of parameters in this layer?
Examples time:

Number of parameters in this layer?

each filter has 5*5*3 + 1 = 76 params (+1 for bias)
=> 76*10 = 760
Convolution layer: summary
Let’s assume input is W1 x H1 x C
Conv layer needs 4 hyperparameters:
- Number of filters K
- The filter size F
- The stride S
- The zero padding P
This will produce an output of W2 x H2 x K
where:
- W2 = (W1 - F + 2P)/S + 1
- H2 = (H1 - F + 2P)/S + 1
Number of parameters: F2CK and K biases
Convolution layer: summary Common settings:
K = (powers of 2, e.g. 32, 64, 128, 512)

Let’s assume input is W1 x H1 x C - F = 3, S = 1, P = 1
Conv layer needs 4 hyperparameters: - F = 5, S = 1, P = 2
- Number of filters K - F = 5, S = 2, P = ? (whatever fits)
- The filter size F - F = 1, S = 1, P = 0
- The stride S
This will produce an output of W2 x H2 x K
where:
- W2 = (W1 - F + 2P)/S + 1
- H2 = (H1 - F + 2P)/S + 1
Number of parameters: F2CK and K biases
(btw, 1x1 convolution layers make perfect sense)
1x1 CONV
56 with 32 filters
56
(each filter has size
1x1x64, and performs a
64-dimensional dot
56 product)
56
64 32
(btw, 1x1 convolution layers make perfect sense)
1x1 CONV
56 with 32 filters
56
(each filter has size
1x1x64, and performs a
64-dimensional dot
56 product)
56
64 32
Example: CONV
layer in PyTorch

- The filter size F
- The stride S
PyTorch is licensed under BSD 3-clause.
Example: CONV
layer in Keras

- The filter size F
- The stride S
Keras is licensed under the MIT license.
The brain/neuron view of CONV Layer
32x32x3 image
5x5x3 filter
32
1 number:
32 the result of taking a dot product between
the filter and this part of the image
3
(i.e. 5*5*3 = 75-dimensional dot product)
32x32x3 image
5x5x3 filter
32
It’s just a neuron with local

connectivity...
1 number:
32 the result of taking a dot product between
the filter and this part of the image
3
(i.e. 5*5*3 = 75-dimensional dot product)
Receptive field
32
28 An activation map is a 28x28 sheet of neuron

outputs:
1. Each is connected to a small region in the input
2. All of them share parameters
32 “5x5 filter” -> “5x5 receptive field for each neuron”

28
3
32
E.g. with 5 filters,

28 CONV layer consists of
neurons arranged in a 3D grid
(28x28x5)
There will be 5 different

32 28 neurons all looking at the same
3 region in the input volume
5
Reminder: Fully Connected Layer
Each neuron
32x32x3 image -> stretch to 3072 x 1 looks at the full
input volume
input activation
1 1
10 x 3072
3072 10
weights
1 number:
the result of taking a dot product
between a row of W and the input
(a 3072-dimensional dot product)
two more layers to go: POOL/FC
Pooling layer
- makes the representations smaller and more manageable
- operates over each activation map independently:
MAX POOLING
Single depth slice

1 1 2 4
x max pool with 2x2 filters
5 6 7 8 and stride 2 6 8
3 2 1 0 3 4
1 2 3 4
y
Pooling layer: summary
Let’s assume input is W1 x H1 x C
- The spatial extent F
- The stride S
This will produce an output of W2 x H2 x C where:

- W2 = (W1 - F )/S + 1
- H2 = (H1 - F)/S + 1
Number of parameters: 0
Fully Connected Layer (FC layer)
- Contains neurons that connect to the entire input volume, as in ordinary Neural
Networks
[ConvNetJS demo: training on CIFAR-10]
http://cs.stanford.edu/people/karpathy/convnetjs/demo/cifar10.html
Summary
- ConvNets stack CONV,POOL,FC layers

- Trend towards smaller filters and deeper architectures
- Trend towards getting rid of POOL/FC layers (just CONV)
- Historically architectures looked like
[(CONV-RELU)*N-POOL?]*M-(FC-RELU)*K,SOFTMAX
where N is usually up to ~5, M is large, 0 <= K <= 2.
- but recent advances such as ResNet/GoogLeNet
have challenged this paradigm

Lecture 5 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lecture 5 PDF

Uploaded by

Copyright:

Available Formats

Lecture 5:

Convolutional Neural Networks

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 1 April 13, 2021

Assignment 1 due Friday April 16, 11:59pm

Assignment 2 will also be released April 16th

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 2 April 13, 2021

Project proposal due Monday Apr 19, 11:59pm

This Friday’s discussion section will discuss how to design a

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 3 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 4 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 -5 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 -6 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 -7 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 -8 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 -9 April 13, 2021

What about vector-valued functions?

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 11 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 12 April 13, 2021

Regular derivative: Derivative is Gradient:

If x changes by a For each element of x,

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 13 April 13, 2021

Regular derivative: Derivative is Gradient: Derivative is Jacobian:

If x changes by a For each element of x, For each element of x, if it

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 14 April 13, 2021

Loss L still a scalar!

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 24 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 25 April 13, 2021

Jacobian dz/dx 4D dL/dz:

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 26 April 13, 2021

[dz/dx] [dL/dz] 4D dL/dz:

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 27 April 13, 2021

4D dL/dx: [dz/dx] [dL/dz] 4D dL/dz:

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 28 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 29 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 30 April 13, 2021

dL/dx always has the

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 31 April 13, 2021

dL/dx always has the

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 32 April 13, 2021

dL/dx always has the

dL/dx always has the

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 34 April 13, 2021

Also see derivation in the course notes:

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 35 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 36 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 37 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 38 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 39 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 40 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 41 April 13, 2021

[N×D] [N×M] [M×D] [D×M] [D×N] [N×M] These formulas are

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 42 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 43 April 13, 2021

Illustration of LeCun et al. 1998 from CS231n 2017 Lecture 1

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 44 April 13, 2021

The machine was connected to a camera that used

Frank Rosenblatt, ~1957: Perceptron

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 45 April 13, 2021

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 46 April 13, 2021

Illustration of Rumelhart et al., 1986 by Lane McIntosh,

Rumelhart et al., 1986: First time back-propagation became popular

Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 5 - 47 April 13, 2021

[Hinton and Salakhutdinov 2006]