Intro to Neural Networks

Niranjan Panda

29.10.09
 Before we start…

Information processing technology

inspired by studies of brain and the
nervous system.
 What is a Neural Network?

"...a computing system made up of a number of

simple, highly interconnected processing
elements, which process information by their
dynamic state response to external inputs. ”
 Biological Neurons

Inputs Outputs

Connection between cells

 Contd…

The brain is made up of neurons which have



A cell body (soma)

Dendrites (inputs)

An axon (outputs)

Synapses (connection between cells)

 There are around 1011 neurons, 1014 synapses

are present in human body and they are connected
in massively parallel.
 What is a Artificial Neural Network?

An ANN is a network of many very simple

processors ( “ units “ ), each possibly having a ( small
amount of) local memory. The units are connected by
unidirectional communication channels (“connections"),
which carry numeric ( as opposed to symbolic ) data.
The units operate only on their local data and on the
inputs they receive via the connections.
 HISTORICAL
1943 --- McCulloch and Pitts (start of the modern era of
neural networks).  Logical calculus of neural networks. A
network consists of sufficient number of neurons (using a
simple model) and properly set synaptic connections can
compute any computable function.

1949 --- Hebb's book "The organization of behavior".  An

explicit statement of a physiological learning rule for
synaptic modification was presented for the first time. 

Hebb proposes that the connectivity of the brain is

continually changing as an organism learns differing
functional tasks, and that neural assemblies are created by
such changes. 
 HISTORICAL
1958 --- Rosenblatt introduced Perceptron A novel
method of supervised learning. Perceptron
convergence theorem.Least mean-square (LMS)
algorithm

1969 --- Minsky and Papert showed limits on

perceptron computation. Minsky and Papert showed
that there are fundamental limits on what single-layer
perceptrons can compute.

1982 --- Hopfield's networks Hopfield showed how to

use "Ising spin glass" type of model to store
information in dynamically stable networks.
 HISTORICAL
1982 --- Kohonen's self-organizing maps (SOM)
Kohonen's self-organizing maps is capable of
reproducing important aspects of the structure of
biological neural nets: Data representation using
topographic maps (which are common in the nervous
systems). SOM also has a wide range of applications.

1985 --- Ackley, Hinton, and Sejnowski, developed

Boltzmann machine, which was the first successful
realization of a multilayer neural network.

1986 --- Rumelhart, Hinton, and Williams developed

the back-propagation algorithm --- the most popular
learning algorithm for the training of multilayer
perceptrons.
 Why Neural Networks?
Adaptive learning: An ability to learn how to do
tasks based on the data given for training or initial
experience.
Self-Organization: An ANN can create its own
organization or representation of the information it
receives during learning time.
Real Time Operation: ANN computations may be
carried out in parallel, and special hardware
devices are being designed and manufactured
which take advantage of this capability.
Fault Tolerance via Redundant Information Coding:
Partial destruction of a network leads to the
corresponding degradation of performance.
However, some network capabilities may be
retained even with major network damage.
 Artificial Neural Network

• ANNs incorporate the two fundamental components

of biological neural networks:

1. Neurons (nodes)
The basic computational unit.
2. Synapses (weights)
Connection links characterized
by certain weight known as synaptic weight.
Neuron Synapse

vs. vs.

Node weight
 Application area
Several areas are there where Neural Network works:

Aerospace Manufacturing
Automotive Medical
Banking Applications
Defense Oil and Gas
Electronics Robotics
Entertainment Speech Processing
Finance & Telecommunication
Insurance Transportation
 Example Applications
NETtalk (Sejnowski and Rosenberg, 1987)
 Maps character strings into phonemes for

learning speech from text.

Neurogammon (Tesauro and Sejnowski, 1989)
 Backgammon learning program

Speech recognition (Waibel, 1989)

 Converts sound to text

Character recognition (Le Cun et al., 1989)

Face Recognition (Mitchell)
ALVINN (Pomerleau, 1988)
 A Simple Artificial Neuron

Bias
b
x1 wi1
Activation
 Local
Field
function


Output
Input
signal
xj wij v
 () y

Summing
  function

xm wim
Synaptic
weights
 Contd..
The neuron is the basic information processing unit of a
NN. It consists of:
1 A set of synapses or connecting links, each link

characterized by a weight known as synaptic weight:

Wi1 , Wi2 , …,Wij ,…, Wim
2 An adder function (linear combiner)
which computes the weighted sum m
of the inputs: u   wijxj
j 1
3 Activation function (squashing function) φ for limiting
the amplitude of the output of the neuron.

y   (u  b)
 Contd..
Bias b ( weight is w0 but with a fixed input of x0 =1 )
has the effect of applying an affine transformation to u
i.e. it serves the purpose of increasing or decreasing
the net input of activation function depending on
whether it is +ve or –ve.
v=u+b
v is the induced field of the neuron
v

u
m
u   wijxj
j 1
 Contd..
Bias is an external parameter of the neuron.
Can be modeled by adding an extra input.
m

x0 = +1
w0 v w
j 0
ij xj

w0  b
x1 w1 Activation
Local function
Field


Input Output
signal x2 w2
v  () y
Summing
  function

xm wm Synaptic
weights
 Activation Functions
Threshold Function:

φ(v) = 1 , v ≥ 0
=0, v<0 1

φ(v) output

0 v input
 Contd..
Signum Function:

φ(v) = +1 , v > θ
= -1 , v ≤ θ 1

φ(v) output

0 θ
v input

-1
 Contd..
Piecewise Linear Function:

φ(v) = 0 , v < -0.5

= v+0.5 , -0.5< v < 0.5
= 1 , v > 0.5 1

φ(v) output

-0.5 0 0.5 I input

 Contd..
Sigmoid Function:
1
φ
(v) = 1 + exp ( -αv ) where α is the slope parameter

= 0.5 , v = 0
=1 ,v=…
= 0 , v = -…
 Contd..
Hyperbolic Tangent Function:

φ(v) = tan hv

φ(v) 1, v infinite
-1 , v -infinite
 Neural Network Architectures
ANN structure can be represented using a digraph
i.e. a graph G(V,E) with V no of vertices and E no of
edges ( Each edge assigned with a orientation ).
Neural Networks are classified into many types
according to their learning mechanisms.
However in general we consider following 3
fundamental types of networks:
> Single Layer Feedforward Network
> Multilayer feedforward Network
> Recurrent Networks
 Single Layer Feedforward Network
The nodes on the left are in the so-called input layer.
The input layer neurons are to only pass and distribute the
inputs and perform no computation. Thus, the only true layer of
neurons is the one on the right.
Each of the inputs x1,x2,…xN is connected to every artificial
neuron in the output layer through the connection weight.
Since every value of outputs y1,y2,…yN is
calculated from the same set of
input values, each
Input Output
output is aried based layer layer
on the connection weights.

Although the presented network is fully connected, the true biological neural
network may not have all possible connections - the weight value of zero can be
 Multilayer Feedforward Network
To achieve higher level of computational capabilities, a more
complex structure of neural network is required. Figure shows
the multilayer neural network which distinguishes itself from the
single-layer network by having one or more hidden layers. In
this multilayer structure, the input nodes pass the information to
the units in the first hidden layer, then the outputs from the first
hidden layer are passed to the next layer, and so on.
Multilayer network can be also viewed as cascading of groups
of single-layer networks.
The level of complexity
in computing can be Input Output
Seen by the fact that layer layer
many single-layer
networks are combined
into this multilayer network.
The designer of an artificial Hidden Layer
neural network should consider how many hidden layers are
 Recurrent Networks
In this type of networks there exists at least 1 feedback loop.
Output layer is again feed the outputs to the input layer
( arbitrary neurons ) as inputs of the same network.
There could be neurons with self feedback links.
 Learning Algorithm
There are a lot of learning algorithm – classified as
supervised learning and unsupervised Learning.
Supervised Learning – uses a set of inputs for which
the appropriate (desired) output are
known.Computed output and correct output are
compared to determine error.
Unsupervised Learning – only input stimuli are shown
to the network. The network is self-Organizing. The
system learns of it’s own by discovering and adapting
to structural features in input pattern.
 2 Main Types of ANN

Supervised Unsupervised

e.g: e.g:
 Adaline
 Perceptron Competitive learning networks
 MLP - SOM
 RBF - ART families
 Fuzzy ARTMAP - neocognition
 etc. - etc.
Supervised Network

Teacher

error +

-
ANN
Unsupervised ANN

Teacher

ANN
 How does an ANN learn
I
weights
O Connected by links-each link
N
U has a numerical weight
P
neurons T Weight
U
P
 basic means of long-term
T memory in ANNs
U
S  Express the strength
T
I Learns through repeated
S
G I
adjustments of these
N G weights
N
A A
L L
Input Middle Output S
S
layer layer Layer
 Learning Process of ANN
Learn from experience Compute
 Learning algorithms output
 Recognize pattern of
activities
Involves 3 tasks Adjust No Is
Desired
 Compute outputs Weight Output
achieved
 Compare outputs with
desired targets
yes
 Adjust the weights and
repeat the process Stop
 Learning Algorithm
Classification
NN Learning Algorithm

Supervised Learning Unsupervised Learning Reinforced Learning

( Error Based ) ( Outpt Based )

Hebbian Competitive
Error Correction Stochastic
Gradient descent

Least Backpropagation
Mean Square