Homework Problem

Training a Neural Network

Whether our neural network is a simple Perceptron, or a much

more complicated multilayer network with special activation
functions, we need to develop a systematic procedure for
determining appropriate connection weights.
The general procedure is to have the network learn the
appropriate weights from a representative set of training data.
In all but the simplest cases, however, direct computation of the
weights is intractable.
Instead, we usually start off with random initial weights and adjust
them in small steps until the required outputs are produced.
Trained networks are expected to generalize, i.e. deal
appropriately with input data they were not trained on.
Perceptron Learning
Convergence of Perceptron Learning
 w new
= wol d
+ p1 = 1.0 + 1 = 2.0
1 1
– 0.8 2 1.2
 The following general learning rule is adopted in neural network studies
The weight increases in proportion to the product of input x
and learning signal r
w    r  x x1=[1 0 -2 -1]';

where η is called learning rate w1=[-1 2 0 .5]';

  net1=w1'*x1

A perceptron using bipolar binary activation function is net1 = -1.5000

shown in Fig. 1. x1=[ 1 0 -2 -1]t and x2=[ -1 2 0 -1]t . The w2=w1+0.1*(1- sign(net1))*x1

target for x1 is 1 and for x2 is -1. The initial weight vector w2 = -0.8000

is w=[ -1 2 0 0.5]t Apply perceptron learning rule to 2.0000

find the weight w3 after one epoch. η = 0.1. -0.4000

0.3000
Fig. 1 x2=[-1 2 0 -1]';
w1
net2=w2'*x2
w2 net2 = 4.5000

w3=w2+0.1*(-1-sign(net2))*x2

w3 w3 = 0.6000

1.6000

w4 -0.4000

0.5000
 Problem: Perceptron Learning Rule
Learning rate = 0.1

X1=[1 –2 0 –1]1, X2=[0 1.5 -.5 -1]1, X3=[-1 1 .5 -1]1

d1 = - 1 d2 = -1 d3 = 1

w1=[1 -1 0 .5]; x1 w1
Net1=net1=[1 -1 0 .5]*[1 -2 0 -1]'=2.5
x2
w2=w1+.1*(-1-1)*x1 w2

w2 =[ 0.8000 -0.6000 0 0.7000] w3 o

x3
net2=w2*x2'= -1.6000 w4
w3=w2 x4
net3=w2*x3'=-2.1
w4=w2+.1*(1--1)*x3=[0.6000 -0.4000 0.1000 0.5000]
net4=w4*x1'=0.9
w5=w4+.1*(-1-1)*x1=[0.4000 0.0000 0.1000 0.7000];
net6=w5*x3'=-1.05
What is the capability of the following network? We added one more
layer with one neuron.

-1
x1
1
3
1
-1 1
x2
1 1
1
-4 -2.5
x3=1

This is a two layer network. This has one hidden layer.

Each neuron in the first layer divides the plane into two half- planes. The
half-planes where the neurons’ response s are +1 have been marked with
arrows pointing toward the positive response. The fourth neuron responds
+1 if and only if the first3 neurons’ output =1
x 1 -3=0
x2
3

x 1 -x 2 >0

3-x 1 >0
2

x 1 +x 2 -4>0

x 1 -x 2 =0 x 1 +x 2 -4=0

1 2 3
x1
 Activation functions of a neuron
Step function Sign function Sigmoid function Linear function

Y Y Y Y
+1 +1 +1 +1

0 X 0 X 0 X 0 X
-1 -1 -1 -1

1, if X  0  1, if X  0 1
Y step   Y sign   Y sigmoid  Y linear  X
0, if X  0  1, if X  0 1  e X

2
f ( net )   1 bipolar continuous
1  e ( net )
 1, net  0
f ( net )  sgn( net )   bipolar binary
  1, net  0
1
f ( net )  unipolar continuous
1  e ( net )
 1, net  0
f ( net )  sgn( net )   unipolar binary
0 , net  0