Multilayer and convolutional neural network examples

7.6 Quiz Question 5

The first hidden layer has 10 neurons, each with 10 inputs (so 11 weights), therefore the
layer has 10*11=110 weights.

The second hidden layer also has 10 neurons, each with 10 inputs from the previous layer
(so 11 weights), therefore the second hidden layer also has 10*11 = 110 weights.

The output layer has 5 neurons, each with 10 inputs (11 weights), therefore the layer has
5*11 = 55 weights.

Total number of weights = 110 + 110 + 55 = 275

7.6 Quiz Question 6

The input volume is 5x5x1, and the first layer is a convolutional layer with 5 filters, with a
2x2x1 receptive field. First we'll calculate the number of weights in the convolutional layer.

Each filter in the convolutional layer 'sees' a 2x2x1 portion of the input volume. So each
filter has 2x2x1=4 inputs. The number of weights is the number of inputs plus 1, so 5 in this
case. There are 5 filters, so 5x5=25 weights for the convolutional layer.

Now we need to know the size of the output volume of the convolutional layer. A 2x2
square can be placed inside a 5x5 square in 4x4=16 different positions. There are 5 filters, so
the volume has depth 5. The volume has size 4x4x5.

The second (and final) layer is fully connected and has 5 neurons in it. Each neuron in this
layer connects to every output of the previous layer, so each neuron has 4x4x5 inputs. The
number of weights for the layer is (4x4x5 + 1)*5=405.

Add the two layers: 25+405 = 430 weights in total.

A larger example
Suppose we have the following CNN architecture:

Input volume: 15x15x1 (width x height x depth)

First convolutional layer: 5 filters, with receptive field 3x3x1
Second convolutional layer: 5 filters, with receptive field 2x2x5
Output layer: fully connected with 10 neurons.

How many weights are there in each layer? We’ll go through the layers in order.
For the first convolutional layer, each filter has 3x3x1 = 9 inputs, so 10 weights. The layer
has 10*5 = 50 weights.

The output volume of the first convolutional layer is 13x13x5. The depth of 5 comes from
the number of filters – each 13x13 plane in the output volume is the output of one filter.
The width and height are both 13 because there are 13x13 positions where a 3x3x1 volume
can be placed within the 15x15x1 input (by sliding horizontally and vertically, as in the
diagrams in Unit 7.5.2).

For the second convolutional layer, each filter has 2x2x5 = 20 inputs, so 21 weights. The
layer as a whole has 21*5 = 105 weights.

The output volume of the second convolutional layer is 12x12x5. The depth of 5 comes
from the number of filters in the second convolutional layer. The width and height are
derived from the number of ways of placing a 2x2x5 volume within the 13x13x5 input for
the layer.

Finally, there is the output layer which is a fully connected layer with 10 neurons. Each
neuron has 12x12x5 inputs, so 12x12x5+1 = 721 weights. There are 10 neurons, so 7210
weights in the layer.

Total number of weights for the network: 50 + 105 + 7210 = 7365