Professional Documents
Culture Documents
Neuron Model: Sigmoidal Function
Neuron Model: Sigmoidal Function
• Sigmoidal Function
Increasing a
-10 -8 -6 -4 -2 2 4 6 8 10
• Back-propagation algorithm
Function signals
Forward Step
Error signals
Backward Step
3
Notation
4
Weight Update Rule
Update rule is based on the gradient descent method
take a step in the direction yielding the maximum
decrease of E
6
Definition of the Local
Gradient of neuron j
Local Gradient
We obtain
because
7
Update Rule
• We obtain
because
8
Compute local gradient of
neuron j
9
Error ej of output neuron
• Case 1: j output neuron
Then
10
Local gradient of hidden
neuron
• Case 2: j hidden neuron
11
Use the Chain Rule
from
We obtain
12
Local Gradient of hidden
neuron j
Hence
w1j e1 Signal-flow
δ ϕ’(v1) graph of
δ ϕ’(vj) back-propag
wkj 1
ek ation error
j
δ ϕ’(vk)
wm j signals to
k neuron j
em
δ ϕ’(vm)
m
13
Delta Rule
• Delta rule Δwji = ηδj yi
IF j output node
IF j hidden node
14
Local Gradient of neurons
a>0
if j hidden node
If j output node
15