Professional Documents
Culture Documents
03-Back Propagation Network
03-Back Propagation Network
www.myreaders.info
Learning
feed-forward
back-propagation
hidden
output
and
layers,
learning
network,
error
by
example,
computation
calculation.
multi-layer
in
input,
Back-propagation
6 hours)
Slides
03-11
function;
Simple
12-16
17-32
33
Back-Propagation Network
What is BPN ?
present
errors
for
the
units
of
the
hidden
layer
are
determined
by
Back-propagation
is
systematic
method
of
training
multi-layer
SC - NN - BPN Background
is
difficult
task
when
there
the
propagated
SC - NN - BPN Background
error
signal
back-propagated
for
each
from
the
such
target
outputs
to
output
the
pattern
inputs
in
is
then
order
to
SC - NN - BPN Background
1.1 Learning :
AND function
W1
AND
X1
0
0
1
1
Input I1
A
C
W2
Input I2
Output O
satisfied.
w 10 + w 2 0 <
w1 0 + w2 1 < ,
w11 + w2 0 <
w1 1 + w2 1 >
if both weights are set to 1 and the threshold is set to 1.5, then (1)
(0) + (1)(0) < 1.5 assign 0 ,
(1)(1) + (1)(0) < 1.5 assign 0 ,
is
"how
the network
can
SC - NN - BPN Background
Example 1
AND Problem
B
W1
AND
X1
0
0
1
1
Input I1
A
C
W2
Output O
Input I
sum of its two inputs and comparing this value with a threshold .
if the net input (net) is greater than the threshold, then the output
is 1, else it is 0.
mathematically, the computation performed by the output unit is
net = w1 I1 + w2 I2
if net >
then O = 1,
otherwise O = 0.
Example 2
Marital status and occupation
applying the
AND
operator
"class 1".
if
they
are
both
inputs are 1.
07
SC - NN - BPN Background
perceptron
consists of a set of
input
the net
input
net =
wi Ii
and
a threshold .
If the net input is greater than the threshold , then the output unit is
turned on , otherwise it is turned off.
To address the learning question, Rosenblatt solved two problems.
first, defined a cost function which measured error.
second, defined a procedure or a rule which reduced that error by
the
is explained
SC - NN - BPN Background
If
the
input
Ii
Wi
is
decreased by 1.
Case 2 :
If output
unit
SC - NN BPN Background
op
tp
dp
SC - NN - BPN Background
X2
+
+
+
+
++
++
++
D
X1
X1
1
X2
1
Y
1
0
0
1
0
0
0
11
X1
X2
1
0
1
0
X1
C
B
Learning By Example
II1
OI1
th
th
input neuron to j
hidden neuron to j
O
11
th
H1
11
th
IO1
W21
IH1
II2
V21
OI2
Input Layer
i - nodes
Wm1
IHm
OI
OO2
IH2
Vl1
II
IO2
H2
Vij
Hm
Hidden Layer
m- nodes
IOn
Wij
OOn
Output Layer
n - nodes
The table below indicates an 'nset' of input and out put data.
It shows inputs and the corresponding n output data.
Table : 'nset' of input and output data
No
Input
Ouput
I1
I2
....
O1
O2
....
On
0.3
0.4
....
0.8
0.1
0.56
....
0.82
2
:
nset
In this section, over a three layer network the computation in the input,
hidden
and
output
layers
are
explained
while
the
step-by-step
= I
x1
x1
th
Vij
th
input neuron
to
hidden layer.
- The input to the hidden neuron is the weighted sum of the outputs
of the input
=[V]
mx1
13
mx
x1
th
th
1
OHp =
where
Note :
(1+e
- (IHP HP)
)
th
OHp
IHp
HP
th
hidden neuron,
neuron;
input
that is always held at -1 and the non-zero threshold becomes the connecting
weight value as shown in Fig. below.
II1
II2
II3
II
1
2
3
OI1
OI2
V2p
I3
1
O
(1+e
- (IHP HP
3p
p Vp
I
HP
IO = -1 O
V1
OIO = -1
above equation.
The input to the output neuron is the weighted sum of the outputs of
the hidden neurons. Accordingly, Ioq the input to the q
th
output neuron
where (q =1, 2, 3 . . , n)
= [ W]
nx1
14
nxm
mx1
th
OOq =
where
Note :
(1+e
- (IOq Oq)
)
th
OOq
IOq
Oq
th
output neuron,
output neuron,
th
and
neuron;
Here
th
extra O
again
the
threshold
may
be
tackled
by
considering
OH1
IH2
W1q
IH3
IHm
OH2
w2q
H3
OHm
Wmq
3q
(1+e
- (IOq Oq)
OOq
m
Oq
HO = -1 O
HO =
a table indicating
an
'nset'
training)
Consider any r
th
in the table- 'nset' of input and output data" for the purpose of
training, calculate output O .
The error norm in output for the r
where E
output neuron is
= (1/2) e
th
r = (1/2) (T O)
2
th
neuron
r=1
This error function is for one training pattern. If we use the same
technique for all the training pattern, we get
E (V, W) =
nset
r=1
E (V, W, I)
that
objective
is
stated
is
an
optimization
problem
solving,
where
the
nset
, t
nset
and
[W]
mxn
over the
'nset'
SC - NN - BPN Algorithm
3. Back-Propagation Algorithm
The benefits of hidden layer neurons have been explained. The hidden layer
allows ANN
to
mapping. The
hierarchical
develop
complex
representation
and
just
the
linearly
separable ones.
The step-by-step algorithm for the training of Back-propagation network
is presented in next few slides. The network is the same , illustrated before,
has a three layer. The input layer is with nodes, the hidden layer with m
nodes
and
the
output
layer
with n
nodes.
An
example
for training a
BPN with five training set have been shown for better understanding.
17
SC - NN - BPN Algorithm
and
SC - NN - BPN Algorithm
Step 1 :
Normalize the I/P and O/P with respect to their maximum values.
For each training pair, assume that in normalized form there are
inputs given by
and
x1
n outputs given by
OO
nx1
Step 2 :
Assume
between
19
that
the number of
1 < m < 21
neurons
in
the
hidden
layers
lie
SC - NN - BPN Algorithm
Step 3 :
Let
represents the
[ V ]
weights
of synapses
connecting input
[W]
[
= [ random weights ]
V]
= [ random weights ]
0
= [
W]
= [0]
can
be
SC - NN - BPN Algorithm
Step 4 :
For training data, we need to present one set of inputs and outputs.
Present the pattern as inputs to the input layer I
then by using linear activation function, the output of the input layer
may be evaluated as
O
= IIx1
x1
Step 5 :
= [ V]
mx1
mx
x1
Step 6 :
Let
the hidden
layer
units,
evaluate
sigmoidal function as
1
O
(1+e
- (IHi)
mx1
21
the
output
using
the
SC - NN - BPN Algorithm
Step 7 :
= [ W]
nx1
nxm
mx1
Step 8 :
Let
the
output
layer
units,
function as
1
O
(1+e
- (IOj)
SC - NN - BPN Algorithm
Step 9 :
E =
(Tj -
Ooj )
th
training set as
Step 10 :
Find a term d as
d =
nx1
23
SC - NN - BPN Algorithm
Step 11 :
Find [ Y ] matrix as
d
1xn
OH
mx1
[Y] =
mxn
Step 12 :
Find
[ W]
mxn
t +1
[ W]
mxn
[Y]
mxn
Step 13 :
Find
e
1
= [W] d mx
mxn nx1
ei
d* =
(OHi) (1 OHi )
mx1 mx1
Find
[ X ] matrix
[X]
x1
as
I
d* =
1xm
d* 1 x
x1
24
1xm
SC - NN - BPN Algorithm
Step 14 :
Find
[
[ V]
1xm
t +1
V]
[X]
=
1xm
1xm
Step 15 :
Find
[V]
[W]
t +1
t +1
[V ]
[W ]
Step 16 :
t +1
+ [
V]
+ [
W]
t +1
error rate =
nset
Step 17 :
SC - NN - BPN Algorithm
Problem :
Consider a
S. No.
Input
I1
Output
O
I2
0.4
-0.7
0.1
0.3
-0.5
0.05
0.6
0.1
0.3
0.2
0.4
0.25
0.1
-0.2
0.12
In this problem,
- there are two inputs and one output.
- the values lie between -1 and +1 i.e., no need to normalize the values.
- assume two neurons in the hidden layers.
- the NN architecture is shown in the Fig. below.
0.4
0.1
0.2
-0.2
0.4
TO = 0.1
-0.5
-0.7
0.2
Input
layer
Hidden
layer
Output
layer
The
solution
to
problem
are
stated step-by-step in
the subsequent
slides.
26
SC - NN - BPN Algorithm
Step 1 :
II =
x1
OI =
x1
-0.7
2x1
0.1
0.4
-0.2
0.2
-0.5
2x2
Step 3 :
Find
= [ V]
0.1
2 x1
as
-0.2
0.18
-0.4
0.2
0.2
-0.7
=
0.02
SC - NN - BPN Algorithm
Step 4 :
1
(1+e
- (0.18)
)
0.5448
1
O
(1+e
- (0.02)
=
)
0.505
SC - NN - BPN Algorithm
Step 5 :
0.5448
[ W]
( 0.2 - 0.5 )
= - 0.14354
0.505
Step 6 :
1
(1+e
- (0.14354)
= 0.4642
Step 7 :
Error = (TO OO1 )
table first training set o/p
29
= (0.1 0.4642)
= 0.13264
from step 6
SC - NN - BPN Algorithm
Step 8 :
0.5448
[Y] = O
(d ) =
( 0.09058)
0.505
=
from values at step 4
Step 9 :
[
0.0457
W]
W]
[Y]
assume
=0.6
0.02958
0.02742
Step 10 :
= [W]
d =
-0.5
( 0.09058) =
0.018116
0.04529
SC - NN - BPN Algorithm
Step 11 :
d* =
at step 4
0.01132
at step 8
Step 12 :
[X] = O
( d* ) =
-0.7
( 0.00449
0.001796
0.004528
0.003143
0.007924
Step 13 :
V]
0.01132)
V]
[X] =
0.001077
0.002716
0.001885
0.004754
SC - NN - BPN Algorithm
Step 14 :
[V]
0.4
-0.2
0.2
0.001077
0.002716
0.001885
0.004754
[W]
0.0989
0.04027
0.1981
0.19524
0.02958
0.2
-0.5
0.02742
0.17042
0.52742
Step 15 :
[ V ]
and next training set is taken and the error will then get adjusted.
Step 16 :
Iterations are carried out till we get the error less than the tolerance.
Step 17 :
Once
the weights
inferencing
32
are adjusted
new objects .
the
network
is
ready
for
SC - NN - BPN References
4. References : Textbooks
1. "Neural
2. "Soft Computing and Intelligent Systems Design - Theory, Tools and Applications",
by Fakhreddine karray and Clarence de Silva (2004), Addison Wesley, chapter 5,
page 249-293.
33