Introduction

Artificial Neural Networks (ANN)

Biological nervous system
Parallel distributed processing
interconnected neurons
ability to learn
acquire knowledge
use it
neurons: processing elements/nodes
perform information processing
1
stability-plasticity in NN
 Comparison of Brains and Traditional
Computers

200 billion neurons, 32 1 billion bytes RAM but trillions

trillion synapses of bytes on disk
Element size: 10-6 m Element size: 10-9 m
Energy use: 25W Energy watt: 30-90W (CPU)
Processing speed: 100 Hz Processing speed: 109 Hz
Parallel, Distributed Serial, Centralized
Fault Tolerant Generally not Fault Tolerant
Learns: Yes Learns: Some
Intelligent/Conscious: Intelligent/Conscious:
Usually Generally No
 History
1943- McCulloch and Pitts - Math of neural
processes
Simplified model of a biological neuron
1949 – Hebb - Hebbian Learning
1958 – Rosenblatt - The “Perceptron”
1969 - Minsky and Papert- Perceptron deficiency
Perceptron limitations

Rebirth in the mid 1980's

Nonlinear Neuron Functions
Back-propagation training


3

Biological Neuron
 Neurons in the Brain
Brain is composed of neurons
neuron receives input from other neurons
from
its synapses
Axons connect to dendrites via synapses

Inputs are approximately summed

When the input exceeds a threshold the

neuron
sends an electrical spike that travels
from the body, down the axon, to the next
neuron(s)
From Neurons to ANS
 From Neurons to ANS
Neurons are the nodes/units/processing element in
ANN

• Summation functions
• Threshold activation function
McCulloch-Pitts Neuron Model
Architecture
The connected weight may be excitatory or inhibitory.
Excitatory +ive weights
inhibitory  -ive weights
McCulloch-Pitts Neuron Model

Threshold is 2
McCulloch-Pitts Neuron Model

Threshold is 1
McCulloch-Pitts Neuron Model
McCulloch-Pitts Neuron Model
Types of learning
Supervised Learning

§ Compare the calculated output with expected response

to determine the error
minimize the error by adjusting error
§ Training continues until the expected output
§ Teacher is assumed to be in the learning process

Unsupervised Learning
§ no teacher
§ expected output not known
§ system learns by itself
§ more complex
§ difficult to implement
§ self learning networks
Reinforcement Learning

Teacher is present
But no expected output
Indicates only the computed output correct or not
Reward for correct output and penalty for wrong output
Hebbian Learning

From the book “Organization of behavior”

Learning rule- from synaptic connections between nerve cell

When an axon of cell A is near enough to excite a cell B and

repeatedly or persistently takes part in firing it, some growth
process or metabolic change takes place in one or both cells such
that A's efficiency, as one of the cells firing B, is increased.

Two rules:
Hebbian Learning Contd…

Δb=y

Two cases:
1. If the product of input and output is +ive, increase in weight
2. If the product of input and output is -ive, decrease in weight
Hebbian Learning Contd…

The net input is,

This is the relation

Hebbian Learning Contd…
Hebbian Learning Contd…
Hebbian Learning Contd…
Perceptron Learning

- refer the previous slide

Competitive Learning
§ when input is present all the neurons compete
§ winning neuron undergoes weight adjustment
§

Stochastic Learning
§ weights are adjusted in probabilistic fashion
§
 The Perceptron
By Resenblatt in 1959
Initial proposal of connectionist networks
Computational model of the retina of the eye

Processing Elements in NN
An ANN consists of perceptrons. Each of the perceptrons receives
inputs, processes inputs and delivers a single output.
 Perceptron
Summation Unit Thresholding Unit
x0=1
x1 w1
w0
w2
x2  o
.. n
I= wi xi
.
1 if I> θ
wn i=0 O=f(I)=
0 otherwise

xn
xi = input
wi =weight
O= Output
26
The Perceptron: Threshold
Activation Function
• Threshold activation function


1 if I>θ
O=f(I)=
0 otherwise
Linear Activation functions
• Output is scaled sum of inputs

N
y   wn xn
n 1
Nonlinear Activation
Functions
• Sigmoid Neuron unit function

1
O  f(I)  I
1 e
Perceptron Contd...
Training: supervised learning
Computed output compared against target output
Weights are adjusted to minimize the error

T=Target Output
O= Perceptron Output

Learning algorithm for training the perceptron:

1. If the output is correct no adjustment in weight

Wij(k+1) = Wij(k)
2. If the output is 1 and should be 0
Wij(k+1) = Wij(k) –αxi
3. If the output is 0 and should be 1
Wij(k+1) = Wij(k) +αxi
Perceptron and Linearly Separable
Problem
Perceptron can handle task which are linearly separable

Example: If the two different sets in 2D are linearly separable

Class1 Class2
 Exclusive Or (XOR) Problem
1 0

Input: 0,0 Output: 0

Input: 0,1 Output: 1
Input: 1,0 Output: 1
Input: 1,1 Output: 0

0
1

XOR Problem: Not Linearly Separable!

We could however construct multiple layers of perceptrons to get around this problem.
Single layer Perceptron

Output Unit Response

Single layer Perceptron Cntd…
Weight and Bias update
Perceptron for AND function with bipolar
inputs and targets
Solution

Initialize w1=w2=b=0, α=1, θ=0

For input pair (1,1), x1=1 and x2=1

Perceptron for AND function with bipolar
inputs and targets
Perceptron for AND function with bipolar
inputs and targets
Perceptron for AND function with bipolar
inputs and targets
Types of NN
§Single layer feed forward network
§Multi layer feed forward network
§Recurrent network

Single layer feed forward network

Multi layer feed forward network
Recurrent network
- atleast one layer with feedback loop
 Summary of Applications
Function approximation
Pattern recognition
Signal processing
Modeling
Control
Machine learning


42