Course Outline

The course is divided in two parts: theory and

practicum
1. Theory covers basic and advanced topics in
neural networks theory and application to
supervised and unsupervised learning
2. Practicum deals with Matlab programming
and application of ANN learning algorithms
in Matlab

Neural Networks NN 1 5
 COURSE OUTLINE
• Mix lectures
– Assignments,
– Presentation from students and
– Discussions
• A number of Lab sessions
– practical aspect of the ANN
– Mainly Matlab will be used
– The purpose is to put the learned into practice,
– deepen your understanding, and
– prepare you for future research
• Both Midterm and final exam will be based on
written paper
Neural Networks NN 1 6
 Suggested Reading
• Class notes
• Presentations
• Handouts
• Fundamental of Neural Network,
– Laurence Fausett, Prentice Hall, new edition,
• Neural Networks: a Comprehensive
Foundation,
– by Simon Haykin, 2nd edition, Prentice Hall,
1999

Neural Networks NN 1 7
 Major Examinations
• Mid Term Exam:
– Midterm – No.1
• As per University schedule
– Midterm – No.2
• As per university schedule
– Final Exam:
• According to the university schedule

Neural Networks NN 1 9
 Course Information

• Course information, plan, slides and links to

on-line material are available at
• ------------

Neural Networks NN 1 10
 Course Evaluation
Evaluation is based on the following:
– theory (weight 0.6): Quizes, Mid and final
exam at the end of the course consisting of
questions about theory part. (Dates to be
announced)
– Matlab part (weight 0.1): Matlab programming
assignments/project to be done
– Research Paper/ANN application (weight 0.3):
– (Available during the course at group)

Neural Networks NN 1 11
 Course evaluation
• Term paper:
– Each student will be required to complete a term paper on
approved topic (individually or in group (not more than 2)).
– Students will be required to chose any neural network model
(studied during the course) and apply it to real world problem.
– A research paper (IEEE format) will be prepared and submitted
at the completion of the term paper.
– Deadline for the term paper will be fixed during the lectures.
• Marks Distribution:
– Mid Term 25%
– Final Project (matlab) 30%
– Lab work and exercises 10%
– Final Exam 35%
-----------------------------------------------------------------------------------
– Total 100%

Neural Networks NN 1 12
PREFACE
HUMAN BRAIN

NN 1
 PREFACE PREFACE
Mouse BRAIN MOUSE BRAIN

Neural Networks NN 1 14
 Human and Artificial Neurons
(FUNCTIONALITIES)

• There are hundreds different types of

neurons, each with its characteristic function,
shape and location, but the main features of a
neuron of any type are its cell body, called
soma, synapses, dendrites and the axon.
• Information is transferred from one neuron to
another at specialised junctions, called
synapses. The dendrites act as the input units
of external signals to the neuron and the axon
acts as the output unit.
Neural Networks NN 1 15
 Human and Artificial Neurons
(FUNCTIONALITIES)
• Much is still unknown
– How brain train itself/process information
– Theories abound
DENDRITES

CELL BODY

AXON
CELL BODY
SYNAPSE
AXON

Neural Networks NN 1 16
 Human and Artificial Neurons
(FUNCTIONALITIES)
• Dendrites input
• Cell Body
– soma
• Axon output
– Electrical pulses
• Synapses
– junction
• Biochemical activities
• Brain's functionality
– Thought, emotion and cognition

Neural Networks NN 1 17
 Human and Artificial Neurons
(FUNCTIONALITIES)
• Neurons receive inputs
– synapses
• Outputs are sent
– axons
• Membrane potential
– Exceeds a threshold,
– the neuron can propagate an all-or-nothing
action potential down its axon
• Many neurons can be active simultaneously
– state vector.

Neural Networks NN 1 18
 Human and Artificial Neurons
(Investigating similarities)

• The human brain is the most sophisticated device.

• One of the most important features of the brain is the
ability to learn from previous experience.
• Thus humans have a great success in dealing with
unforeseen situations utilising the knowledge gained
from previously experienced similar situations.
• This is not true in systems based on computer
technology as they rely entirely on human pre-
defined instructions, spelling out each step for all
tasks. Thus bugs, in these instructions may cause all
sorts of unexpected results.

Neural Networks NN 1 19
 BRAIN vs CPTR
• Brain
– Operating by 100-mV nerve
impulses lasting nearly a
millisecond. Neuron takes 4
ms to complete a firing cycle.
– It is robust and fault tolerant.
Nerve cells in the brain die
every day without affecting
its performance significantly.
– It accepts fuzzy, noisy, poorly
conditioned inputs and
produces an approximate
output.
– It is highly parallel due to
massive inter-connectivity
between neurons.
– Connection with other
elements via synapse.
Neural Networks
• Computer
– Technology Operating on 5-V signal
levels switching at nanosecond intervals.
Computers with 33 MHz takes about 40 ns
to execute a single instruction. Super
computers take about 3 ns to complete
a single instruction.
– The destruction of even a single
transistor may cause complete
loss of functionality.
– It can only handle precise data
fed in properly.
– The conventional computer are
totally sequential. (Few
connections between its basic
elements)
– Links/weights
NN 1 20
 ANN vs CPTR
• Algorithmic approach
• Like human brain
• A cognitive approach:
• Interconnected
neurons – the way the problem is to solved must
be known and stated in small
• Learn by examples unambiguous instructions.
• Un-programmable – These instructions are then converted
• Careful selection of to a high level language program and
then into machine code that the
examples computer can understand.
• Ops unpredictable • Predictable; if anything goes wrong
is due to a software or hardware
fault.

Neural Networks NN 1 21
 SEQUENCE
• Comparison between human brain and
neuron
• MATLAB session
• BASIC MODEL
• WORKING MECHANISM

Neural Networks NN 1 22
 ANN vs CPTR
• ANNs and conventional algorithmic computers
– Not in competition
– Complement each other.
– Few Tasks algorithmic approach like arithmetic
operations suited for computers
– Few Tasks that are more suited to ANN.
– Large number of tasks
• Require systems that use a combination
• A conventional computer is used to supervise the neural
network
• Neural networks do not perform miracles. But if
used sensibly they can produce some amazing
results.
Neural Networks NN 1 23
 From Human neurons to artificial
neurons (AN)

• We conduct AN by deducing essential

feature and interconnections of human
neurons
• Prog a cptr to simulate them
• As Knowledge is incomplete and
computation power is limited
– We reach to Gross idealization of real
neurons
Neural Networks NN 1 24
 Human and Artificial Neurons
(FUNCTIONALITIES)
• Much is still unknown
– How brain train itself/process information
– Theories abound
DENDRITES

CELL BODY

AXON
CELL BODY
SYNAPSE
AXON

Neural Networks NN 1 25
 An Artificial Neuron Model

Neural Networks NN 1 26
 What are artificial neural
networks?

• Artificial neural networks

(ANNs) are software or
hardware systems designed
to simulate the operation of
a simple biological nervous
system.
 A Typical ANN Structure
• ANNs are collections of
interconnected entities
named “neurones”.
• Similarly to the biological
model, each neurone has
many inputs (the dendrites)
but a single output (the
axon).
• Some inputs have excitatory
effect on the axon while
others have inhibitory effect.
• The activity of the ANN is the
combined effect of the
operation of the constituent
neurones of the ANN.
 A Simple Neurone Model
(The Perceptron)

• A perceptron has
analogue inputs but
binary output.

• Each input has an

associated weight.
out  f (net )
• Positive weights n
correspond to net   w i  x i
excitatory inputs and i 1
negative weights to
1 if net  threshold
inhibitory inputs. f 
0 if net  threshold
 Other Neuron Models

• The activation function can

take a number of forms
depending on the type of
application.

• These mathematical
alternatives approximate
with different degrees of
accuracy the biological
neurone activity.
 Neurones with Radial Basis
Functions

• The output signal is

maximum when the
input vector X is
identical to a
reference vector W.
• The output
decreases fast as n
the vector X moves   xi  w i 
2

away from W.
out  e i 1
 What can ANNs do?

• Similarly to biological brains,

neural networks have the
capacity to learn, memorize
and create relationships
amongst data.
• Their functionality depends on
the neurone weights and on
the network architecture.
 DEFINITION REVISITED
• A mathematical model composed of a large number
of processing elements organised into layers.
• Single Artificial neuron or processing elements
Input Output
Neuron

• Artificial neuron with n input and output

x1 w1
y'
w2 A
x2
F
wn
xn n

y= 
i=1
xiwi

A.F = Activation Function

Neural Networks NN 1 33
 DEFINITION REVISITED

• ANN are used to predict and learn from a given

set of data.
• More robust at data analysis than statistical
methods
– Handle small variations of parameters and noise.
• Receives data in binary (0s, 1s) or bipolar +1/-1
• The threshold value determine the final output
• If the summation < threshold, -1 is the output
• If the summation > threshold, +1 is the output
Neural Networks NN 1 34
 BASIC MODEL OF ANN

Neural Networks NN 1 35
Neural Networks NN 1 36
 WORKING MECHANISM
• These inputs go through a vector W of
Weights of N dimension. Processed by the
Summation node, "a" is generated
• a = bias + x1 * w1 + x2 * w2 + …. + xn * wn

• “a" is then processed through an AF

– Compares the value of "a" to a predefined
Threshold
• If a < the threshold, the model will not fire.
• If a > the threshold, the model will fire, produce output.
• Note: fire means neuron is active
Neural Networks NN 1 37
 Dimensions of a Neural Network

• network architectures
• types of neurons
• learning algorithms (learning strategy)
• Applications (problem solving cap)

Neural Networks NN 1 38