Chapter-1

Introduction to Neural Network

11/4/2023 1
 Outlines

 Concept, biological neural network, evolution of artificial neural network,

McCulloch-Pitts neuron models
 Learning (Supervised & Unsupervised) and activation function
 Models of ANN-Feed forward network and feedback network,
 Learning Rules- Hebbian, Delta, Perceptron Learning and Widrow-Hoff, winner
take all.

11/4/2023 2
 Basic Concept

– Neural networks are information processing systems, which are constructed and
implemented to model the human brain.
– The main objective of the neural network research is to develop a computational
device for modeling the brain to perform various computational tasks at a faster rate
than the traditional systems.
– Artificial neural perform various tasks such as optimization function, approximation,
vector quantization, and data clustering.

11/4/2023 3
 Basic Concept Cont’d

– These tasks are very difficult for traditional computers, which are in algorithmic and
computational tasks and precise arithmetic operations.
– Therefore, for implementing artificial neural networks, high speed digital computers
are used, which makes the simulation of neural processes feasible.

11/4/2023 4
 Artificial Neural Network

– ANNs are computational models which mimic the brain's learning processes.
– Resembles the characteristic of biological neural network.
– Posses large number of interconnected processing elements called nodes or units of
neurons.
– Neuron is connected to other by a connection link. Each connection link is
associated with weight which has information about the input signal.

11/4/2023 5
 Artificial Neural Network Cont’d

– ANN processing elements are called as neurons or artificial neurons, since they
have the capability to model networks of original neurons as found in brain.
– Internal state of neuron is called activation or activity level of neuron, which is
the function of the inputs the neurons receives.
– Neuron can send only one signal at a time.

11/4/2023 6
 Artificial Neural Network Cont’d

Architecture of a simple ANN

 Here, X1 and X2 are input neurons, Y is the output neuron and 𝑤1 and 𝑤2 weighted
interconnection links.
 Hence, the net input is calculated as:
𝑦𝑖𝑛 = 𝑥1 𝑤1 + 𝑥2 𝑤2

11/4/2023 7
 Artificial Neural Network Cont’d

 Where 𝑥1 and 𝑥2 are the activations of the input neurons X1 and X2 ,i.e. the output
of input signals. The output 𝑦 of the output neuron Y can be obtained by applying
activations over the net input, i.e. the function of the net input :
𝑦 = 𝑓(𝑦𝑖𝑛 )
Output = Function(net input calculated)
– The function to be applied over the net input is called activation function.
– Weight involved in ANN is equal to the slope of linear straight line (y=mx).

11/4/2023 8
 Biological Neural Network

– Consists of the following main parts

• Soma or cell body-where cell nucleus is located
• Dendrites-where the nerve is connected to the cell body
• Axon-which carries the impulses of the neuron

11/4/2023 9
 Biological Neural Network Cont’d

– Electric impulse is passed between synapse and dendrites. Axon split into strands and
strands terminates into small bulb like organs called as synapse.
– It is a chemical process which results in increase /decrease in the electric potential
inside the body of the receiving cell.
– If the electric potential reaches a threshold value, receiving cell fires & pulse / action
potential of fixed strength and duration is send through the axon to synaptic junction
of the cell. After that, cell has to wait for a period called refractory period before it
can fire again.

11/4/2023 10
 Mathematical model of artificial neuron

 The above figure shows mathematical representation of the chemical processing taking place
in artificial neuron.
 In this model net input is calculated by:
𝑦𝑖𝑛 = 𝑥1 𝑤1 + 𝑥2 𝑤2 + ⋯… … .. + 𝑥𝑛 𝑤𝑛
𝑛

= 𝑥𝑖 𝑤𝑖
𝑖=1
11/4/2023 11
 Terminology Relation Between Biological And Artificial Neuron

Biological Neuron Artificial Neuron

Cell Neuron
Dendrites Weights or interconnections
Soma Net input
Axon Output

11/4/2023 12
 Comparison between Biological neuron and Artificial Neuron

Comparison between Brain and Computer

Brain Computer
– Biological – Integrated circuit
– Composed of a huge number of neurons – Composed of CPU
– Parallel processing – Sequential processing
– Fault tolerant – Not fault tolerant
– Fast but not precise computation – Fast and precise computation
– Learnable or trainable – Not learnable or trainable
– Adaptable to environment – Not adaptable to environment

11/4/2023 13
 Evolution of Neural Networks

Year Neural network Designer Description

1943 McCulloch and Pitts neuron McCulloch and Pitts Arrangement of neurons is
combination of logic gate.
Unique feature is threshold

1949 Hebb network Hebb If two neurons are active, then

their connection strengths should
be increased.

1958, 1959, 1962, 1988, Perceptron Frank Rosenblatt, Block, Minsky Weights are adjusted to reduce
1960 Adaline and Papert Widrow and Hoff the difference between the net
input to the output unit and the
desired output

11/4/2023 14
 Evolution of Neural Networks Cont’d

1972 Kohonen self-organizing Kohonen Inputs are clustered to obtain a

feature map fired output neuron.

1982, 1984, 1985, Hopfield network John Hopfield and Tank Based on fixed weights.
1986, 1987 Can act as associative memory
nets

1986 Back propagation Rumelhart, Hinton and i) Multilayered

network Williams ii) Error propagated backward
from output to the hidden
units

11/4/2023 15
 Evolution of Neural Networks Cont’d

1988 Counter propagation Grossberg Similar to kohonen

network network

1987-1990 Adaptive resonance Carpenter and Grossberg Designed for binary and
Theory(ART) analog inputs.

1988 Radial basis function Broomhead and Lowe Resemble back

network propagation network ,
but activation function
used is Gaussian function

1988 Neo cognitron Fukushima For character recogniton.

11/4/2023 16
 McCulloch-Pitts neuron

– The 1st computational model of a neuron was proposed by Warren MuCulloch

(neuroscientist) and Walter Pitts (logician) in 1943 and it is usually called as M-P
neuron.

– M-P neurons are connected by directed weighted paths.

– Activation of M-P neurons is binary (i.e) at any time step the neuron may fire or
may not fire.
– Weights associated with communication links may be excitatory(weights are
positive)/inhibitory(weights are negative).

11/4/2023 17
 McCulloch-Pitts neuron Cont’d

– Threshold plays major role here. There is a fixed threshold for each neuron and if
the net input to the neuron is greater than the threshold then the neuron fires.
– They are widely used in logic functions.

McCulloch-Pitts neuron model

11/4/2023 18
 McCulloch-Pitts neuron Cont’d

– A simple M-P neuron is shown in the figure. It is excitatory with weight (w>0) /
inhibitory with weight –p (p<0).

– In the Fig., inputs from x1 to xn possess excitatory weighted connection and xn+1 to
xn+m has inhibitory weighted interconnections.
– Since the firing of neuron is based on threshold, activation function is defined as

11/4/2023 19
 McCulloch-Pitts neuron Cont’d

– For inhibition to be absolute, the threshold with the activation function should
satisfy the following condition:
θ >nw –p
– Output will fire if it receives “k” or more excitatory inputs but no inhibitory inputs
where
kw≥θ>(k-1) w
– The M-P neuron has no particular training algorithm.
– An analysis is performed to determine the weights and the threshold.
– It is used as a building block where any function or phenomenon is modeled based
on a logic function.

11/4/2023 20
 Learning

– Learning in ANN is classified into three categories as:

i) Supervised learning
ii) Unsupervised learning
Iii) Reinforcement learning

11/4/2023 21
 Supervised Learning
– Learning with the help of a teacher.
– Example : learning process of a small child.
• Child doesn’t know read/write.
• Their each & every action is supervised by a teacher
– In ANN, each input vector requires a corresponding target vector, which represents
the desired output.

Supervised Learning
11/4/2023 22
 Supervised Learning Cont’d

– The input vector along with target vector is called training pair.
– The input vector results in output vector.
– The actual output vector is compared with desired output vector.
– If there is a difference means an error signal is generated by the network, it is used
for adjustment of weights until actual output matches desired output.

11/4/2023 23
 Unsupervised Learning

– Learning is performed without the help of a teacher. Example: tadpole – learn to

swim by itself.
– In ANN, during training process, network receives input patterns and organize it
to form clusters.

Unsupervised Learning

11/4/2023 24
 Unsupervised Learning Cont’d

– From the Fig. it is observed that no feedback is applied from environment to

inform what output should be or whether they are correct.
– The network itself discover patterns, regularities, features/ categories from the
input data and relations for the input data over the output.
– Exact clusters are formed by discovering similarities & dissimilarities so called
as self – organizing.

11/4/2023 25
 Reinforcement learning

– Learning based on critic information is called reinforcement learning & the

feedback sent is called reinforcement signal.
– The network receives some feedback from the environment. Feedback is only
evaluative.

11/4/2023 26
 Reinforcement learning Cont’d

– The external reinforcement signals are processed in the critic signal generator, and
the obtained critic signals are sent to the ANN for adjustment of weights properly to
get critic feedback in future.

11/4/2023 27
 Activation Functions

– To make work more efficient and for exact output, some force or activation is given.
– Like that, activation function is applied over the net input to calculate the output of an
ANN.
– An Activation Function decides whether a neuron should be activated or not.
– Information processing of processing element has two major parts: input and output.
– An integration function (f) is associated with input of processing element.

11/4/2023 28
 Activation Functions Cont’d

• Several activation functions are there.

1. Identity function:
• it is a linear function which is defined as:
f(x) =x for all x
• The output is same as the input.
2. Binary step function
• it is defined as:

where θ represents threshold value.

It is used in single layer nets to convert the net input to an output that is binary ( 0 or 1).

11/4/2023 29
 Activation Functions Cont’d

3. Bipolar step function:

It is defined as:

where θ represents threshold value.

used in single layer nets to convert the net input to an output that is bipolar (+1 or
-1).
4. Sigmoid function
used in Back propagation nets.

11/4/2023 30
 Activation Functions Cont’d

Two types:
a) Binary sigmoid function
logistic sigmoid function or unipolar sigmoid function.
- it is defined as:

where λ – steepness parameter

The range of sigmoid function is 0 to 1.

11/4/2023 31
 Activation Functions Cont’d

b) Bipolar sigmoid function

– Where λ- steepness parameter and the sigmoid range is between -1 and +1.
– It is closely related to hyperbolic tangent function, which is written as

11/4/2023 32
 Activation Functions Cont’d

5. Ramp function

– The graphical representation of all these function is given in the upcoming Figure

11/4/2023 33
11/4/2023 34
 Basic models of ANN

– Models are based on three entities

• The model’s synaptic interconnections.

• The training or learning rules adopted for updating and adjusting the connection
weights.
• Their activation functions
– The arrangement of neurons to form layers and the connection pattern formed within
and between layers is called the network architecture.

11/4/2023 35
 Feed Forward Networks

– If no neuron in the output layer is an input to a node in the same layer / proceeding
layer – feed forward network.

– Since the input signals are fed into the input layer, then, after being processed, they
are forwarded to the next layer.
– Feed forward networks can be: Single layer Feed- Forward Network or Multi-layer
Feed- Forward Network

11/4/2023 36
 Feed Forward Networks Cont’d

 Single layer Feed- Forward Network

– Layer is formed by taking processing elements and combining it with other
processing elements.

– Inputs are connected to the processing nodes with various weights, resulting in series
of outputs one per node.

11/4/2023 37
 Feed Forward Networks Cont’d

 Multi layer Feed- Forward Network

– Formed by the interconnection of several layers.
– Input layer receives input and buffers input signal.
– Output layer generated output.
– Layer between input and output is called hidden layer.
– Hidden layer is internal to the network.
– More the hidden layer, more is the complexity of network, but efficient output
is produced.

11/4/2023 38
 Feed Forward Networks Cont’d

Multi layer Feed- Forward Network

11/4/2023 39
 Feedback Network

– If outputs are directed back as input to the processing elements in the same
layer/proceeding layer is feedback network.
– When the neural network has some kind of internal recurrence, meaning that the
signals are fed back to a neuron or layer that has already received and processed that
signal, the network is of the type feedback.

11/4/2023 40
 Feedback Network Cont’d

Hidden Feedback
Input Layer

Output

Feedback
Feedback Network

11/4/2023 41
 Important Terminologies

 Weight
– The weight contain information about the input signal.
– It is used by the net to solve the problem.
– It is represented in terms of matrix & called as connection matrix.
– If weight matrix W contains all the elements of an ANN, then the set of all W
matrices will determine the set of all possible information processing configuration.

11/4/2023 42
 Important Terminologies Cont’d

• Bias
– Bias has an impact in calculating net input.
– Bias is included by adding x0 to the input vector x.
– The net output is calculated by

11/4/2023 43
 Important Terminologies Cont’d

– The bias is of two types

– Positive bias
» Increase the net input
– Negative bias
» Decrease the net input

11/4/2023 44
 Important Terminologies Cont’d

 Threshold
– It is a set value based upon which the final output is calculated.
– Calculated net input and threshold is compared to get the network output.
– The activation function of threshold is defined as
• where θ is the fixed threshold value

11/4/2023 45
 Important Terminologies Cont’d

• Learning rate
– Denoted by α.
– Control the amount of weight adjustment at each step of training.
– The learning rate range from 0 to 1.
– Determine the rate of learning at each step
• Momentum Factor
– Convergence is made faster if a momentum factor is added to the weight updating process.
– Done in back propagation network.
• Vigilance parameter
– Denoted by ρ.
– Used in Adaptive Resonance Theory (ART) network.
– Used to control the degree of similarity.
– Ranges from 0.7 to 1 to perform useful work in controlling the number of clusters.

11/4/2023 46
 Learning Rules

 Hebbian learning rule

– Donald Hebb stated in 1949 that “ In brain, the learning is performed by the change
in the synaptic gap”.
– When an axon of cell A is near enough to excite cell B, and repeatedly or
permanently takes place in firing it, some growth process or metabolic change
takes place in one or both the cells such than A’s efficiency, as one of the cells
firing B, is increased.

11/4/2023 47
 Learning Rules Cont’d

– According to Hebb rule, the weight vector is found to increase

proportionately to the product of the input and the learning signal.
– In Hebb learning, two interconnected neurons are ‘on’ simultaneously.
– The weight update in Hebb rule is given by
wi(new) = wi (old)+ xi y.
– It is suited more for bipolar data. If binary data is used, the weight
updation formula cannot distinguish two conditions namely
• A training pair in which an input unit is “on” and the target value is
“off”
• A training pair in which both the input unit and the target value is
“off”.

11/4/2023 48
 Learning Rules Cont’d

 Perceptron Learning Rule

– The learning signal is the difference between the desired and actual response of
the neuron. The perceptron learning rule is explained as:

– Consider a finite “n” number of input training vectors, with their associated
(desired) values x(n) and t(n), where “n” N. The target is either +1 or -1. The
output “y” is obtained on the basis of the net input calculated and activation
function being applied over the net input.

11/4/2023 49
 Learning Rules Cont’d

1 𝑖𝑓𝑦𝑖𝑛 > 𝜃
𝑦 = 𝑓(𝑦𝑖𝑛 ) 0 𝑖𝑓 − 𝜃 ≤ 𝑦𝑖𝑛 ≤ 𝜃
−1 𝑖𝑓𝑦𝑖𝑛 < 𝜃

– The weight updation in case of perceptron leaning is shown.

If 𝑦 ≠ 𝑡, then
𝑤 𝑛𝑒𝑤 = 𝑤 𝑜𝑙𝑑 + 𝑎𝑡𝑥 𝑎 − 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝑟𝑎𝑡𝑒

Else we have
𝑤 𝑛𝑒𝑤 = 𝑤 𝑜𝑙𝑑

11/4/2023 50
 Learning Rules Cont’d

– The weights can be initialized at any values in this method.

– The perceptron rule convergence theorem states that “if there is a weight vector W, such
that 𝑓(𝑥(𝑛)𝑊) = 𝑡(𝑛), for all 𝑛, then for any starting vector 𝑤1 , the perceptron
learning rule will converge to a weight vector that gives the correct response for all
training patterns, and this learning takes place within a finite number of steps provided
that the solution exists.

11/4/2023 51
 Learning Rules Cont’d

 Delta (Widrow-Hoff) Learning Rule

– Developed by Widrow and Hoff, it is one of the most common learning rules.

– It updates the connection weights with the difference between the target and the
output value. It is the least mean square learning algorithm falling under the
category of the supervised learning algorithm.

11/4/2023 52
 Learning Rules Cont’d

– The motive of the delta learning rule is to minimize the error between the output and
the target vector.
– The delta rule for adjusting the weights of the ith pattern (𝑖 = 1 𝑡𝑜 𝑛) is :
∆𝑤 = 𝛼 𝑡 − 𝑦𝑖𝑛 𝑥𝑖 ,
𝑤ℎ𝑒𝑟𝑒 𝛼 𝑖𝑠 𝑡ℎ𝑒 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝑟𝑎𝑡𝑒, 𝑥𝑖 = 𝑖𝑛𝑝𝑢𝑡 𝑣𝑎𝑙𝑢𝑒𝑠 𝑎𝑛𝑑 𝑦𝑖𝑛 = 𝑜𝑢𝑡𝑝𝑢𝑡, 𝑡
= 𝑡𝑎𝑟𝑔𝑒𝑡 𝑣𝑎𝑙𝑢𝑒

– Least mean square error = (t- yin)2, Adaptive linear neuron converges when the least
mean square error is reached.

11/4/2023 53
 Learning Rules Cont’d

 Winner-take-all learning rule

– Winner-take-all networks are based on the competitive learning rule and will use the
strategy where it chooses the neuron with the greatest total inputs as a winner.

– The connections between the output neurons show the competition between them and
one of them would be ‘ON’ which means it would be the winner and others would be
‘OFF’

11/4/2023 54
 Summary

– In this chapter we have discussed the basics of an ANN in comparison with biological
neuron and the evolution.

– Concepts of supervised learning, unsupervised learning and reinforcement learning are

explained.

– Various activation functions and different models of ANN (feed forward and feedback
network) are clarified.

11/4/2023 55