Soft Computing

Neural Networks
k (3)
]É{Ç V
] V{tÉ}|tÇ f{|
f
g
Merchant Marine College
Shanghai Maritime University
Backpropagation
Function Approximation Example
Nominal Response
Parameter Variations
Performance Index
Chain Rule
Gradient Calculation
Steepest Descent
Jacobian Matrix
Backpropagation (Sensitivities)
Initialization (Last Layer)
Summary
Example: Function Approximation
Network
Initial Conditions
Forward Propagation
Transfer Function Derivatives
Backpropagation
Weight Update
Choice of Architecture
Choice of Network Architecture
Convergence
Generalization
 Variations
on
Backpropagation
 Variations
• Heuristic Modifications
– Momentum
– Variable Learning Rate
• Standard Numerical Optimization
– Conjugate Gradient
– Newton’s Method (Levenberg-Marquardt)
( g q )
Performance Surface Example
Squared Error vs. w11,1 and w21
Squared Error vs. w11,1 and b11
Squared Error vs. b11 and b21
Convergence Example
Learning Rate Too Large
Momentum
Momentum Backpropagation
 Variable Learning Rate (VLBP)
• If the squared error (over the entire training set)
increases by more than some set percentage ζ after
a weight update, then the weight update is discarded,
the learning rate is multiplied by some factor (1 >ρ>
0), and the momentum coefficient γ is set to zero.
• If the squared
q error decreases after a weight g update,
p ,
then the weight update is accepted and the learning
rate is multiplied by some factor η>1. If γ has been
previously
i l sett tto zero, it is
i resett to
t its
it original
i i l value.
l
• If the squared error increases by less than ζ, then the
weight update is accepted
accepted, but the learning rate and
the momentum coefficient are unchanged.
Example
Conjugate Gradient
Interval Location
Interval Reduction
Golden Section Search
Conjugate
j g Gradient BP ((CGBP))
Newton’s Method
Matrix Form
Hessian
Gauss-Newton Method
Levenberg-Marquardt
Adjustment of µk
Application to Multilayer Network
Jacobian Matrix
Computing the Jacobian
Marquardt Sensitivity
Computing the Sensitivities
 LMBP
• Present all inputs to the network and compute the
corresponding network outputs and the errors.
errors Compute the
sum of squared errors over all inputs.
• Compute the Jacobian matrix.
matrix Calculate the sensitivities
with the backpropagation algorithm, after initializing.
Augment
g the individual matrices into the Marquardt
q
sensitivities. Compute the elements of the Jacobian matrix.
• Solve to obtain the change in the weights.
• Recompute the sum of squared errors with the new
weights. If this new sum of squares is smaller than that
computed in step 1, then divide µk by υ, update the weights
and go back to step 1. If the sum of squares is not reduced,
then multiply µk by υ and go back to step 3.
3
Example LMBP Step
LMBP Trajectory
Questions ?