INDIAN INSTITUTE OF TECHNOLOGY ROORKEE

The Backpropagation Algorithm

 Multilayer Perceptrons
Three-Layer Network

2
Pattern Classification
Two-Layer XOR Network

3
Function Approximation
Function Approximation Network

4
Function Approximation
Nominal Response of Network

5
Function Approximation
Effect of Parameter Changes on Network Response

6
The Backpropagation Algorithm
Three-Layer Network, Abbreviated Notation

7
The Backpropagation Algorithm
Performance Index

Approximate Performance Index

Sensitivity

8
The Backpropagation Algorithm
Forward Propagation

Backward Propagation

9
The Backpropagation Algorithm

Backward Propagation

Weight Update (Approximate Steepest Descent)

10
Associative Learning

•An association is any link between a system’s input and output such
that when a pattern A is presented to the system it will respond with
pattern B

•When two patterns are linked by an association, the input pattern is

often referred to as the stimulus. Likewise, the output pattern is
referred to as the response
•Unconditioned stimulus
•Conditioned stimulus

Simple Associative Network

11
Banana Associator

•An association is any link between a system’s input and output such
that when a pattern A is presented to the system it will respond with
pattern B

12
Unsupervised Hebb Rule

The increase in the weight between a neuron’s input and output is in

proportion to their product:

In vector form:

At each iteration, the output is calculated in response to the input P,

and then the weights W are updated with the Hebb rule.

13
Unsupervised Hebb Rule

Hebb Rule with Decay:

the decay rate, is a positive constant less than one

Response of the Hebb Rule, With and Without Decay

14
Simple Recognition Network

A neuron that has a vector input, referred as an instar

Instar Rule

15
Simple Recognition Network

Example

16
 Simple Recall Network

The outstar network, shown in Figure,

has a scalar input and a vector output.
It can perform pattern recall by
associating a stimulus with a vector
response.

Outstar Rule:

If we set the decay rate  equal to the learning rate 

17
Simple Recognition Network

Example

18
References

Laurene Fausett, “Fundamentals of Neural Networks: Architectures,

Algorithms and Applications” PEARSON

David E. Goldberg, “Genetic Algorithms in Search, Optimization,

and Machine Learning”

19