SUPERVISED LEARNING NETWORK

Perceptron Networks
 The basic networks in supervised learning
 Perceptron networks better than Hebb rule
 The perceptron network consist of three units
– Sensory unit (input unit)
– Associator unit (hidden unit)
– Response unit (output unit)
 Perceptron Networks
• Input unit is connected to hidden units with fixed weights -
1,0,-1 assigned at random
• binary activation function is used in input and hidden unit
• Output unit (1,0,-1) activation. The binary step with fixed
threshold ϴis used as activation.
• Output of perceptron is
y  f ( yin )

1ifyin   
 
f ( yin )  0if    yin   
 1 yin   
 
 Perceptron Networks
• Weight updation between hidden and output unit
• Checks out for error between hidden and calculated output
layer
• Error=target-calculated
• weights are adjusted in case of error
wi (new)  wi (old )  txi
b(new)  b(old )  t
• α is the learning rate, ‘t’ is the target which is -1 or 1.
• No error-no weight change-training is stopped
Single classification perceptron
network
 Perceptron Training Algorithm for
Single Output Classes
Step 0: initialize weights, bias, learning rate( 0<  <=1)
Step 1: perform step 2-6 until final stopping condition is false
Step 2: perform steps 3-5 for each bipolar or binary training pair indicated by s:t
Step 3: input layer is applied with identity activation fn:
xi=si
Step 4: calculate output response of each input j=1 to m
first, net input is calculated

activation are applied over the net input to calculate the output response.
 Perceptron Training Algorithm for
Single Output Classes
Step 5: Make adjustment in weight and bias j=1 to m and i=1 to n

Step 6: Test for the stopping condition. If there no change in

weight then stop the training process else start again from step2.
Flow chart for perceptron Network
with single output
 Problem
Implement AND function using perceptron
networks for bipolar inputs and target.
The perceptron network, which uses perceptron
learning rule, is used to train the AND function.
The network architecture is as shown in Figure.
The input patterns are presented to the network
one by one. When all the four input patterns are
presented, then one epoch is said to be
completed. The initial weights and threshold are
set to zero
W1 = W2. = b = 0 and ϴ= 0. The learning rate is set equal to=1.
• Calculate the net input

• The output y is computed by activation function

over the net input calculated:

ϴ = 0 hence, when yin =0, then y=0

Check whether t=y . Here t=1 and y=0 so t≠y,
hence weight updation take place
The weight W1 = 1, W2 = l, b = 1 are the final weights after first
input pattern is presented.
The same process is repeated for all the input patterns.
The process can be stopped when all the targets become equal
to the calculated output or when a separating line is obtained
using the final weights for separating the positive responses
from negative responses.
 Adaptive Linear Neuron (Adaline)
• A network with a single linear unit is called an ADALINE (ADAptive
LINear Neuron)
• Input-output relationship is linear
• It uses bipolar activation for its input signals and its target output
• Weights between the input and output are adjustable and has only
one output unit
• Trained using Delta rule (Least mean square) or (Widrow-Hoff rule)
 Delta Rule
• Delta rule for Single output unit
– Minimize the error over all training patterns.
– Done by reducing the error for each pattern one at a time
• Delta rule for adjusting the weight for ith pattern is (i=1to n)
wi   (t  yin) xi
• Delta rule in case of several output units for adjusting the
weight from ith input unit to jth output unit

wij   (t  yinj ) xi
 Adaline Model
x0=1
1

b
x1 w1
X1
yin= xiwi
f(yin)
w2 
x2
X2 wn

yin

xn
Xn e=t-yin
Output error t
Adaptive
generator
algorithm
 Adaline Training Algorithm
Step 0: Weights and bias are set to some random values other
than zero. Learning rate parameter α

Step 1: Perform Steps 2-6 when stopping condition is false.

Step 2: Perform steps 3-5 for each bipolar training pair s:t

Step 3: Set activations for input units i=1 to n xi=si

Step 4: Calculate the net input to the output unit

n
yin  b   xiwi
i 1
 Adaline Training Algorithm
Step 5: Update the weights and bias for i=1 to n

wi (new)  wi (old )   (t  yin) xi

b(new)  b(old )   (t  yin)

• Step 6: If highest weight change that occurred during training

is smaller than a specified tolerance then stop the training
else continue. (Stopping condition)
 Start
Stop

Y
Initialize weights and
bias and α
If Ei=Es

Input the specified

tolerance error Es

Calculate error
Ei=Σ(t-yin)2

For
each wi (new)  wi (old )   (t  yin ) xi
s:t b(new)  b(old )   (t  yin )

Activate input units Calculate net input

Xi=si Yin=b+Σxi wi
• Implement OR function with bipolar inputs
and target using Adaline network.
• The initial weights are taken to be W1 = W2 =
b = 0.1 and learning rate = 0.1. For the first input sample, X1
= 1, X2 = 1, t = 1, we calculate the net input as
 BACK-Propagation Network(BPN)
• Multilayer feed-forward network with continuous differentiable
activation function.
• Weight updation (gradient-method)
• Neural network is to train the net to achieve a balance between
the net's ability to respond (memorization) and its ability to give
reasonable responses to the input that similar but not identical
to the one that is used in training (generalization)
The training of the BPN is done in three stages
 The feed-forward of the input training pattern
 Calculation and back-propagation of the error
 Updation of weights.
 BACK-Propagation Network(BPN)
• Multilayer feed-forward network
(input, output and hidden layer)
• The neuron presents in the
hidden and output layer have
biases, which are the connection
from the unit whose activation is
always 1
• Bias term also acts as a weights
• Back-propagation phase of
learning, signals are sent in the
reverse direction.
• Output (binary or bipolar)
 BACK-Propagation Network(BPN)
Training Algorithm
• Step 0: Initialize the weight and learning rate (small random
values )
• Step 1: Perform step 2-9 when stopping condition is false.
• Step 2: Perform steps 3-8 for each training pair.
 BACK-Propagation Network(BPN)
Feed-forward phase (Phase I)
 BACK-Propagation Network(BPN)
Back-propagation of error (Phase-II)
Step 6: Each output unit receives a target pattern corresponding to the input
training pattern and computes the error correction term

Error correction term, update the change in weights and bias.

Step 7: Each hidden unit sum its delta inputs from the output units.

The term δinj gets multiplied with the derivative of f(zinj) to calculate the error
term.
 BACK-Propagation Network(BPN)
Weight and bias updation (Phase-III)
 Problem
Initial weight

[V11,V21,VO1]= [0.6,-0.1,0.3]

[V12,V22,VO2]= [-0.3,0.4,0.5]

[W1,W2,W0]= [0.4,0.1,-0.2]

Learning rate α = 0.25

 Radial Base Function(RBF) Network
 The radial basis function (RBF) is a
classification and functional
approximation neural network
developed by M.J.D. Powell.

 The network uses the most

common nonlinearities such as
sigmoidal and Gaussian kernel
functions.

 The Gaussian functions are also

used in regularization networks.

 The Gaussian function is generally

defined as
Radial Base Function(RBF) Network
 Radial Base Function(RBF) Network
Training Algorithm
Step 0: set the weights to small values
Step 1: perform step 2-8 when the stopping condition is false
Step 2: perform step 3-7 for each input
Step 3: Each input unit receives the input signals and transmits
to the next hidden layer unit.
Step 4: calculate the radial basis function
Step 5: select the centres for the radial base function . The
centre are selected from the set of inputs vector
 Radial Base Function(RBF) Network
Step 6: calculate the output from the hidden unit

where x^ji; is the center of the RBF unit for input variables; σithe width
of ith RBF unit; xji the jth variable of input pattern.
Step 7:calculate the output of the neural network

Where k is the number of hidden layer nodes (RBF function);ynet the

output value of mth node in output layer for the nth incoming pattern
Wim weight between ith RBF unit and mth output node; wo the
biasing term at nth output node.
Step 8: calculate the error and test for the stopping condition. The
stopping condition may be number of epochs or to certain extent
weight change.
  RBF learning process

K-means ci

A w
xp Basis Linear
Functions Regression
ci

K-Nearest i
Neighbor