Deep Learning Assignment Name:-Aryan Gupta

IT Section - 1, 6th Semester

Roll No- 𝑈𝐸218015

Q1) Develop and Explain a Deep Feed forward network.

Ans:- A deep feedforward neural network, also known as a deep neural network (DNN),
is a type of artificial neural network (ANN) that consists of multiple hidden layers
between the input and output layers. Each hidden layer contains a set of neurons
(nodes) that perform transformations on the input data. The term "deep" refers to the
depth of the network, which indicates the number of hidden layers it contains.

To Develop a neural network it contain few steps

Step 1:- Defining the kind of problem to solve which is whether the problem is of
regression or classification based on which data will be taken with inputs and labels
where for

classification y ∈ {0,1} { y is the predicted value using the trained model}

𝑛
regression y∈𝑅

Step 2 :- Using the appropriate model for the task like based on the number of inputs
how much parameters are there definition the pre activation layer and activation
function which one to use

Pre Activation layer ➖

h1=w2*x2+w1*x1+b

{ preactivation function for 1 layer from inputs layer w1 , w2 are weights and b in bias
for layer 1}

Activation Function ➖
Many activation function choice is there sigmoid , tanh , Relu function mostly used is
Relu for its properties of zero centered , no gradient vanishing problem etc but here lets
use sigmoid function
Sigmoid function ➖ −(𝑤2*𝑥2+𝑤1*𝑥1+𝑏)
y= 1/1 + 𝑒

We use this activation function in each layer just Output layer use different function
depend upon if it is a classification problem it will use a softmax function which will give
a probability distribution . this y1 is just a example of it
𝑤1*𝑥1
𝑒
𝑦1 = 𝑘 𝑤𝑘*𝑥𝑘
∑𝑒
𝑖=1

Step 3:- Choosing the parameter like w1, w2,w3 and b bias are the parameters which are
learned by the network now each layer have its own weights and bias and learn it correct
it or update it if wrong this is basically store the knowledge of model

Step 4:- Choosing learning algorithm to be used which is gradient decent to learn the
parameter and update them based on the loss function or cost function

Update rule of Gradient decent ➖

ω𝑡+1 = ω𝑡 − η∆ω
Same is for b bias and we compute gradient for all points and in each epoch update the
value of parameters and check loss function

Back propagation after one iteration of feedforward we compute the weights and bias ,
once this is done it calculate loss function and if loss is not zero it basically ask which
weight got the problem and compute gradients and then update that one accordingly to
reduce error

Step 5:- Loss function is important as it tell are the updates right or wrong or have we
reach to global minimum
𝑚
1 2
Regression:- 𝑚𝑖𝑛 𝐿(𝑤, 𝑏) = 𝑚
∑ (𝑓𝑖(𝑥) − 𝑦) { mean squared error }
𝑖=1
𝑛
Classification 𝑚𝑖𝑛 𝐿(𝑦𝑖) =− ∑ 𝑙𝑜𝑔(𝑦𝑖) { cross entropy }
𝑖=1

This steps will develop a feed forward neural network

Q2) Write an example function for Convolution operation and explain it in detail.

Ans:-A convolution operation operates on all the pixel values within its kernel's
receptive field, producing a single value by essentially multiplying the kernel weights
with the pixel values elementwise and adding a bias term to the result. This reduces the
dimensions of the input matrix as well.
This is a function wrote in python to do convolution operation
● Image.shape and kernel.shape define the width and height of the kernel and
image like kernel of 3 × 3 and image of 512 × 225
● Now we calculate the padding height and width as so we don’t loss dimension in
output matrix image here we divide the dimension with 2 to get padding
● Padded_image is new image with the padding
● Output is set to all zero of same dimension as the original image we used
● Now , we do a Nested loop i is image height and j is image width and computing
the value by multiply the value padded image of i+kernel_height ,j+kernel_width
and kernel , adding the value which is stored in output
● return the output image

Q 3 ) Describe how to find the gradient in a Recurrent Neural Network (RNN).

Ans:-

Lets take this Recurrent Neural Network and see how we can find the gradient in this
one

𝑇
Loss function = 𝐿(θ) = ∑ 𝐿𝑡(θ)
𝑡=1
We can calculate this two partial derivative
 𝑇 ∂𝐿𝑡(θ) 𝑇 ∂𝐿𝑡(θ)
∂𝐿(θ) ∂𝐿(θ)
∂𝑉
= ∑ ∂𝑉𝑡
, ∂𝑊
= ∑ ∂𝑊𝑡
𝑡=1 𝑡=1
As we know
𝑆 = σ(𝑈𝑥1 + 𝑊𝑠𝑡−1 + 𝑏)
𝑦𝑖 = σ(𝑉𝑠𝑡 + 𝐶)
Now in RNN we use something as Backpropagation in time

As 𝑆4 = σ(𝑊𝑠3 + 𝑏 )

∂𝑠4
We cannot simply do ∂𝑊
as 𝑠3 is also a function of W

To solve this calculate explicit and implicit partial derivatives here

+
∂ 𝑠4
Explicit :-
∂𝑊

Implicit:- Adding all indirect paths from 𝑠4 to W

From above we say that ,

∂𝑠4 4 ∂ 𝑠4
+
∂ 𝑠𝑘
∂𝑊
= ∑ ∂𝑠 ∂𝑊
𝑘
𝑘=1

Generic ➖
∂𝐿𝑡(θ) ∂𝐿𝑡(θ) 4 ∂ 𝑠4
+
∂ 𝑠𝑘
∂𝑊
= ∂𝑠𝑡
∑ ∂𝑠𝑘 ∂𝑊
𝑘=1

Hence this how we compute the gradient in recurrent neural network using this form for
all s values