Gradient Descent

Deep Learning
By
T.K. Damodharan
Vice President, RBS
Reg.No: PC2013003013008

Under the guidance of

Dr V.Rajasekar,
Associate Professor,
Department of Computer Science & Engineering,
SRM Institute of Science and Technology-Vadapalani Campus.
 Gradient Descent 

 Gradient descent is by far the most

popular optimization strategy used in 
machine learning and deep learning at the
moment.
 It is used when training data models, can be
combined with every algorithm and is easy to
understand and implement.
 Everyone working with machine learning
should understand its concept.
Gradient Descent 
 Gradient Descent is an optimization algorithm
for finding a local minimum of a differentiable
function.
 Gradient descent is simply used to find the
values of a function's parameters (coefficients)
that minimize a cost function as far as possible.
 It's based on a convex function and tweaks its
parameters iteratively to minimize a given
function to its local minimum.
What is a Gradient

"A gradient measures how much the output of a

function changes if you change the inputs a little
bit." — Lex Fridman (MIT)
A gradient simply measures the change in all
weights with regard to the change in error.
You can also think of a gradient as the slope of a
function. The higher the gradient, the steeper the
slope and the faster a model can learn.
But if the slope is zero, the model stops learning.
In mathematical terms, a gradient is a partial
derivative with respect to its inputs.
What is a Gradient
What is a Gradient

Imagine a blindfolded man who wants to climb to

the top of a hill with the fewest steps along the way
as possible.
He might start climbing the hill by taking really
big steps in the steepest direction, which he can do
as long as he is not close to the top.
As he comes closer to the top, however, his steps
will get smaller and smaller to avoid overshooting
it.
This process can be described mathematically
using the gradient.
What is a Gradient

Imagine the image below illustrates our hill from a

top-down view and the red arrows are the steps of
our climber.
Think of a gradient in this context as a vector that
contains the direction of the steepest step the
blindfolded man can take and also how long
that step should be.
What is a Gradient
What is a Gradient

Note that the gradient ranging from X0 to X1 is

much longer than the one reaching from X3 to X4.
This is because the steepness/slope of the hill,
which determines the length of the vector, is less.
This perfectly represents the example of the hill
because the hill is getting less steep the higher it's
climbed.
Therefore a reduced gradient goes along with a
reduced slope and a reduced step size for the hill
climber.
How Gradient Descent Works

Instead of climbing up a hill, think of gradient

descent as hiking down to the bottom of a valley.
This is a better analogy because it is a
minimization algorithm that minimizes a given
function.

Equation :b is the next position of our climber,

while a represents his current position.
The minus sign refers to the min part of GD.
The gamma in the middle is a waiting factor and
Gradient Descent
More details and Types of Gradient Descent
https://builtin.com/data-science/gradient-descent

Step by step Video guide:

https://youtu.be/sDv4f4s2SB8
Linear Models
A strong high-bias assumption is linear separability:
 in 2 dimensions, can separate classes by a line

 in higher dimensions, need hyperplanes

A linear model is a model that assumes the data is linearly

separable
Linear models
A strong high-bias assumption is linear separability:
 in 2 dimensions, can separate classes by a line

 in higher dimensions, need hyperplanes

A linear model is a model that assumes the data is linearly

separable
 Linear Regression
DATASET

inputs outputs
x1 = 1 y1 = 1
x2 = 3 y2 = 2.2

w x3 = 2 y3 = 2
 1  x4 = 1.5 y4 = 1.9
x5 = 4 y5 = 3.1

Linear regression assumes that the expected value of

the output given an input, E[y|x], is linear.
Simplest case: Out(x) = wx for some unknown w.
Given the data, we can estimate w.
Copyright © 2001, 2003, Andrew W. Moore Neural Networks: Slide 14
Linear models
A linear model in n-dimensional space (i.e. n
features) is define by n+1 weights:

In two dimensions, a line:

0 =w1 f1 + w2 f2 + b (where b = -a)

In three dimensions, a plane:

0 =w1 f1 + w2 f2 + w3 f3 + b
In m-dimensions, a hyperplane
m
0 =b + å wj fj
j=1
Which line will it find?
Which line will it find?

Only guaranteed to find some

line that separates the data
Linear models
Perceptron algorithm is one example of a linear
classifier

Many, many other algorithms that learn a line (i.e. a

setting of a linear combination of weights)

Goals:
 Explore a number of linear training algorithms

 Understand why these algorithms work

Linear models in general
1. pick a model
0 =b + å
m
wj fj
j=1

These are the parameters we want to learn

2. pick a criteria to optimize (aka objective function)

 Some notation: indicator function

ìï 1 if x =True üï
1[ x ] =í ý
î 0 if x =False
ï ï
þ

Convenient notation for turning T/F answers into numbers/counts:

drinks _ to _ bring _ for _ class = å 1[ x >=21]

xÎ class
 Some notation: dot-product
Sometimes it is convenient to use vector notation

We represent an example f1, f2, …, fm as a single vector, x

Similarly, we can represent the weight vector w1, w2, …, wm as a single

vector, w

The dot-product between two vectors a and b is defined as:

m
a ×b =å a j b j
j=1
Linear models
1. pick a model
0 =b + å
n
wj fj
j=1

These are the parameters we want to learn

2. pick a criteria to optimize (aka objective function)

n

å 1[ y (w ×x + b) £0]
i i
i=1

What does this equation say?

Convex functions
Convex functions look something like:

One definition: The line segment between any

two points on the function is above the function
Finding the minimum

You’re blindfolded, but you can see out of the bottom of the
blindfold to the ground right by your feet. I drop you off
somewhere and tell you that you’re in a convex shaped valley
and escape is at the bottom/minimum. How do you get out?
Finding the minimum

loss

How do we do this for a function?

 One approach: gradient descent

Partial derivatives give us the

slope (i.e. direction to move)
in that dimension
loss

w
 One approach: gradient descent

Partial derivatives give us the

slope (i.e. direction to move) in
that dimension
loss

Approach:
 pick a starting point (w)
 repeat: w
 pick a dimension
 move a small amount in that
dimension towards decreasing loss
(using the derivative)
 One approach: gradient descent

Partial derivatives give us the

slope (i.e. direction to move) in
that dimension

Approach:
 pick a starting point (w)
 repeat:
 pick a dimension
 move a small amount in that
dimension towards decreasing loss
(using the derivative)
Gradient descent

 pick a starting point (w)

 repeat until loss doesn’t decrease in all dimensions:
 pick a dimension
 move a small amount in that dimension towards decreasing loss
(using the derivative)
d
w j =w j - h loss(w)
dw j

What does this do?

Gradient descent

 pick a starting point (w)

 repeat until loss doesn’t decrease in all dimensions:
 pick a dimension
 move a small amount in that dimension towards decreasing loss
(using the derivative)
d
w j =w j - h loss(w)
dw j

learning rate (how much we want to move in the error

direction, often this will change over time)
Some maths
n
d d
dw j
loss = å
dw j i=1
exp(- yi (w ×xi + b))

n
d
=å exp(- yi (w ×xi + b)) - yi (w ×xi + b)
i=1 dw j
n
=å - yi xij exp(- yi (w ×xi + b))
i=1
Gradient descent

 pick a starting point (w)

 repeat until loss doesn’t decrease in all dimensions:
 pick a dimension
 move a small amount in that dimension towards decreasing loss
(using the derivative)

n
w j =w j + h å yi xij exp(- yi (w ×xi + b))
i=1

What is this doing?

Exponential update rule
n
w j =w j + hå yi xij exp(- yi (w ×xi + b))
i=1

for each example xi:

w j =w j + h yi xij exp(- yi (w ×xi + b))

Summary
Gradient descent minimization algorithm
 require that our loss function is convex
 make small updates towards lower losses
Gradient descent

 pick a starting point (w)

 repeat until loss doesn’t decrease in all dimensions:
 pick a dimension
 move a small amount in that dimension towards decreasing loss
(using the derivative)
d
wi =wi - h (loss(w) + regularizer(w, b))
dwi

n
w j =w j + h å yi xij exp(- yi (w ×xi + b)) - hl w j
i=1
The update
w j =w j + h yi xij exp(- yi (w ×xi + b)) - hl w j

learning rate direction to regularization

update
constant: how far from wrong

What effect does the regularizer have?

 The update
w j =w j + h yi xij exp(- yi (w ×xi + b)) - hlw j

learning rate direction to regularization

update
constant: how far from wrong

If wj is positive, reduces wj moves wj towards 0

If wj is negative, increases wj