Inspiration from Neurobiology

Human Biological Neuron

1
2/2/2022
 Model Of A Neuron
Wa
X1

Wb Y
X2  f()

Wc
X3

Input units Connection Summing

computation
weights function

(dendrite) (synapse) (axon)

(soma)
2/2/2022
 Artificial Neural Network
Synapse Nukleus


w1

x1

y
Axon
x2 w2 Activation Function:
yin = x1w1 + x2w2 (y-in) = 1 if y-in >= 
and (y-in) = 0

Dendrite
-A neuron receives input, determines the strength or the weight of the input, calculates the total
weighted input, and compares the total weighted with a value (threshold)

-The value is in the range of 0 and 1

- If the total weighted input greater than or equal the threshold value, the neuron will produce the output,
and if the total weighted input less than the threshold value, no output will beproduced

2/2/2022
 History
• 1943 McCulloch-Pitts neurons
• 1949 Hebb‟s law
• 1958 Perceptron (Rosenblatt)
• 1960 Adaline, better learning rule (Widrow,
Huff)
• 1969 Limitations (Minsky, Papert)
• 1972 Kohonen nets, associative memory

2/2/2022
 • 1977 Brain State in a Box (Anderson)
• 1982 Hopfield net, constraint satisfaction
• 1985 ART (Carpenter, Grossfield)
• 1986 Backpropagation (Rumelhart, Hinton,
McClelland)
• 1988 Neocognitron, character recognition
(Fukushima)

2/2/2022
 Characterization
• Architecture
– a pattern of connections between neurons
• Single Layer Feedforward
• Multilayer Feedforward
• Recurrent
• Strategy / Learning Algorithm
– a method of determining the connection weights
• Supervised
• Unsupervised
• Reinforcement
• Activation Function
– Function to compute output signal from input signal

2/2/2022
 Single Layer Feedforward NN

x1 w11

w12 ym

w21

yn
x2

w22
output layer
Input layer

ADALINE, AM, Hopfield, LVQ, Perceptron, SOFM

2/2/2022
 Multilayer Neural Network
z1

 
x1 V11
w11
w12
V1n
w12 y1

x2 z2

   y2


 zn

xm Vmn  
Input layer Output layer
Hidden layer
CCN, GRNN, MADALINE, MLFF with BP, Neocognitron, RBF, R2C3E
2/2/2022
 Recurrent NN
Input Outputs

Hidden nodes

ART, BAM, BSB, Boltzman Machine, Cauchy Machine, Hopfield,

RNN
2/2/2022
 Strategy / Learning Algorithm
Supervised Learning

• Learning is performed by presenting pattern with target

• During learning, produced output is compared with the desired output
– The difference between both output is used to modify learning
weights according to the learning algorithm
• Recognizing hand-written digits, pattern recognition and etc.
• Neural Network models: perceptron, feed-forward, radial basis function,
support vector machine.

2/2/2022
 Unsupervised Learning

• Targets are not provided

• Appropriate for clustering task
– Find similar groups of documents in the web, content
addressable memory, clustering.
• Neural Network models: Kohonen, self organizing maps,
Hopfield networks.

2/2/2022
 Where can neural network systems help…
• when we can't formulate an algorithmic
solution.
• when we can get lots of examples of the
behavior we require.
„learning from experience‟
• when we need to pick out the structure
from existing data.

2/2/2022
 Who is interested?...
• Electrical Engineers – signal processing,
control theory
• Computer Engineers – robotics
• Computer Scientists – artificial
intelligence, pattern recognition
• Mathematicians – modelling tool when
explicit relationships are unknown

2/2/2022
 Problem Domains
• Storing and recalling patterns
• Classifying patterns
• Mapping inputs onto outputs
• Grouping similar patterns
• Finding solutions to constrained
optimization problems

2/2/2022
 Coronary
Disease

Classification

STOP

01 10
Neural
11
Net 11 10 00 00 Input patterns
00 11

Input layer

Output layer

00 01 10 11
Sorted
00 10 11 pa.tterns

00 11 34
2/2/2022
 Clustering

00 11
10
11 10

00 11

00
01

35
2/2/2022
 ANN Applications

Medical Applications

Information
Searching & retrieval
Chemistry

Education
2/2/2022 Business & Management
 Applications of ANNs

• Signal processing
• Pattern recognition, e.g. handwritten
characters or face identification.
• Diagnosis or mapping symptoms to a
medical case.
• Speech recognition
• Human Emotion Detection
• Educational Loan Forecasting

2/2/2022
 Abdominal Pain Prediction

Intensity Duration
Male Age Temp WBC Pain Pain
adjustable
1 20 37 10 1 1
weights

AppendicitisDiverticulitis Ulcer Pain Cholecystitis Obstruction Pancreatitis

Duodenal Non-specific Small Bowel
Perforated

2/2/2022
 Voice Recognition

2/2/2022
 Educational Loan Forecasting System

2/2/2022
 Advantages Of NN
NON-LINEARITY
It can model non-linear systems

INPUT-OUTPUT MAPPING
It can derive a relationship between a set of input & output
responses

ADAPTIVITY
The ability to learn allows the network to adapt to changes in
the surrounding environment

EVIDENTIAL RESPONSE
It can provide a confidence level to a given solution

2/2/2022
 Advantages Of NN
CONTEXTUAL INFORMATION
Knowledge is presented by the structure of the network.
Every neuron in the network is potentially affected by the
global activity of all other neurons in the network.
Consequently, contextual information is dealt with naturally in
the network.

FAULT TOLERANCE
Distributed nature of the NN gives it fault tolerant capabilities

NEUROBIOLOGY ANALOGY
Models the architecture of the brain

2/2/2022
The machine learning framework
y = f(x)
output prediction Image
function feature

• Training: given a training set of labeled examples {(x1,y1), …,

(xN,yN)}, estimate the prediction function f by minimizing the
prediction error on the training set
• Testing: apply f to a never before seen test example x and output
the predicted value y = f(x)

2/2/2022
The machine learning framework
• Apply a prediction function to a feature representation of the
image to get the desired output:

f( ) = “apple”
f( ) = “tomato”
f( ) = “cow”
2/2/2022
 Steps
Training Training
Labels
Training
Images
Image Learned
Training
Features model

Testing

Image Learned
Prediction
Features model
2/2/2022
Test Image
Features
• Raw pixels

• Histograms

•…

2/2/2022
Classifiers: Nearest neighbor

Training
Training Test
examples
examples example
from class 2
from class 1

f(x) = label of the training example nearest to x

• All we need is a distance function for our inputs

• No training required!

2/2/2022
Classifiers: Linear

• Find a linear function to separate the classes:

f(x) = sgn(w  x + b)
2/2/2022
Many classifiers to choose from
• SVM Which is the best one?
• Neural networks
• Naïve Bayes
• Bayesian network
• Logistic regression
• Randomized Forests
• Boosted Decision Trees
• K-nearest neighbor
• RBMs
• Etc.

2/2/2022
 Recognition task and supervision
• Images in the training set must be annotated with the “correct
answer” that the model is expected to produce

Contains a motorbike

2/2/2022
 Unsupervised “Weakly” supervised Fully supervised

Definition depends on task

2/2/2022
 Generalization

Training set (labels known) Test set (labels

unknown)

• How well does a learned model generalize from the

data it was trained on to a new test set?
2/2/2022
Activation Function

An Activation Function decides whether a neuron should be activated

or not. This means that it will decide whether the neuron’s input to the
network is important or not in the process of prediction using simpler
mathematical operations.
It’s just a thing function that you use to get the output of node. It is also
known as Transfer Function.

2/2/2022
Why we use Activation functions with Neural
Networks?
It is used to determine the output of neural network like yes or no. It
maps the resulting values in between 0 to 1 or -1 to 1 etc. (depending
upon the function).
The Activation Functions can be basically divided into 2 types-
1. Linear Activation Function
2. Non-linear Activation Functions

2/2/2022
Linear or Identity Activation Function

As you can see the function is a line or linear. Therefore, the output of
the functions will not be confined between any range.

2/2/2022
Cont…
Equation : f(x) = x
Range : (-infinity to infinity)
Note:
It doesn’t help with the complexity or various parameters of usual
data that is fed to the neural networks.

2/2/2022
Other Linear functions
• Binary Step Function
Binary step function is a threshold-based activation function which
means after a certain threshold neuron is activated and below the said
threshold neuron is deactivated.

2/2/2022
Bipolar step function: if the value of Y is above a certain value known
as the threshold, the output is +1and if it's less than the threshold then
the output is -1. It has bipolar outputs (+1 to -1).

2/2/2022
Non-linear Activation Function

• The Nonlinear Activation Functions are the most used activation

functions. Nonlinearity helps to makes the graph look something like
this

2/2/2022
It makes it easy for the model to generalize or adapt with variety of data
and to differentiate between the output.
The main terminologies needed to understand for nonlinear functions
are:
• Derivative or Differential: Change in y-axis w.r.t. change in x-axis.It
is also known as slope.
• Monotonic function: A function which is either entirely non-
increasing or non-decreasing.
The Nonlinear Activation Functions are mainly divided on the basis of
their range or curves.

2/2/2022
 Sigmoid or Logistic Activation Function

The Sigmoid Function curve looks like a S-shape.

The main reason why we use sigmoid function is because it exists
between (0 to 1). Therefore, it is especially used for models where we
have to predict the probability as an output. Since probability of
anything exists only between the range of 0 and 1, sigmoid is the right
choice.

2/2/2022
Tanh or hyperbolic tangent Activation Function

The range of the tanh function is from (-1 to 1). tanh is also sigmoidal (s
- shaped).
The advantage is that the negative inputs will be mapped strongly
negative and the zero inputs will be mapped near zero in the tanh graph.

2/2/2022