Artificial Neural Network

Objectives
• To understand :
▫ the similarities between biological neuron and
artificial neuron;
▫ how artificial neural network learns;
▫ the difference between machine learning and
deep learning;
▫ The applications of machine learning and deep
learning.
Human Intelligence vs
Artificial Intelligence
Human Intelligence
Artificial Intelligence
AI, ML and DL
ANN Applications
Biological neurons
 Biological neural network
Synapse

Synapse Dendrites
Axon
Axon

Soma Soma
Dendrites
Synapse
Artificial Neural Network (ANN)
The neuron as a simple computing
element
The neuron uses a transfer function to
produce an output
Logical AND operator with ANN…
• Epoch 1, Input 1 :
• X=(x1*w1)+(x2*w2)
• =(0*0.3)+(0*-0.1)
• =0
Threshold value for step function, ϴ=0.2) to
compute actual output (Y)
Y = 0 because X < 0.2
E=Yd – Y
= 0-0=0
…Logical AND operator with ANN…
• Epoch 1, Input 2:
• X=(x1*w1)+(x2*w2)
• =(0*0.3)+(1*-0.1)
• =0-0.1=-0.1
Threshold value for step function, ϴ=0.2) to
compute actual output (Y)
Y = 0 because X < 0.2
E=Yd – y
= 0-0=0
…Logical AND operator with ANN…
• Epoch 1, Input 3 :
• X=(x1*w1)+(x2*w2)
• =(1*0.3)+(0*-0.1)
• =0.3-0=0.3

Threshold value for step function, ϴ=0.2) to compute

actual output (Y)
Y = 1 because X > 0.2
E=Yd – y
= 0-1=-1
Adjust weight for input that is positive : w1 = 0.3-0.1=0.2
(if error is negative, reduce the positive weight)
…Logical AND operator with ANN
• Epoch 1, Input 4 :
• X=(x1*w1)+(x2*w2)
• =(1*0.2)+(1*-0.1)
• =0.2-0.1=0.1

Threshold value for step function, ϴ=0.2) to compute

actual output (Y)
Y = 0 because X < 0.2
E=Yd – y
= 1-0=1
Adjust weight by learning rate : w1=0.2+0.1=0.3
w2=-0.1+0.1=0
(if error is positive, increase the positive weight)
…Logical AND operator with ANN
• The training process continues with the same
data but with different weights since there are
some errors occur in the first epoch.
Exercise
Solution : epoch 1
• Sample training using sigmoid activation
function
-0.06

W1

W2
-2.5
f(x)
W3

1.4
-0.06

2.7

-8.6
-2.5
f(x)
0.002
x = -0.06×2.7 + 2.5×8.6 + 1.4×0.002 = 21.34

1.4
Training the neural network
Fields class
1.4 2.7 1.9 0 Initialise with random weights
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Training data
Fields class Present a training pattern
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
1.4

2.7

1.9
Training data
Fields class Feed it through to get output by multiplying
1.4 2.7 1.9 0 with random weight
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
1.4

2.7

1.9
Training data
Fields class Compare with target output
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
1.4

2.7
1
1.9
Training data
Fields class Adjust weights based on error
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
1.4

2.7
1
1.9
Training data
Fields class And so on ….
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
6.4

2.8 0.9
1
1.7

Repeat this thousands, maybe millions of times – each time

taking a random training instance, and making slight
weight adjustments
Algorithms for weight adjustment are designed to make
changes that will reduce the error
 How does ANN learn?
• By making small adjustments to the
weights to reduce the difference
between the actual and desired
output
 What is the purpose of the training
process?
• To save the final weights of each input so that
they can be used in the testing process for
performance evaluation.
Training Neural Network using
Matlab
Lab 3: Objective : To understand how Artificial Neural Network (ANN) learns
and the effect of the number of hidden nodes (and/or other parameters) to
the ANN performance.

 Open Matlab, and type nprtool in the Command Window.

 Read and understand the information provided and click Next.
 Click Load Example Data Set.
 Select a data set from the Window on the left and click Import.
 Click Next on the Select Data window.
 Click Next on the Validation and Test Data window.
 Type the number of hidden neurons in the Network Architecture window and click
Next.
 Click Train in the Train Network window. You may click Retrain a few times or
until the value of %E for Testing in the Results window is 0. Otherwise, click Back
and change the number of hidden neurons and repeat the process.
 Click Next in the Evaluate Network window.
 Click Next in the Deploy Solution window.
 Click Advanced Script in the Save Results window and save the file. Then, click
Finish.
 In the workspace window, save the input and target files.
FEATURES AND CLASSIFIERS FOR
OBJECT RECOGNITION
FEATURES CLASSIFIERS

Color ANN

Texture SVM

shape KNN
DEMO
 43

Sample classification results

Feature Classifier Accuracy

HOG SVM 0.65

RGB colour SVM 0.85
moment

RGB colour KNN 0.94

moment
 Advantages and Disadvantages of
ANN
• Advantage :
▫ A neural network can perform tasks in which a
linear program cannot perform.
• Disadvantage :
▫ The neural network requires training to operate
 What influence the performance
of an ANN?
• An ANN’s performance is influenced by :
▫ Size of training data
▫ Architecture of the network
▫ Complexity of the problem
 How to determine the size of a
training set?
• Refer to Widrow’s rule of thumb : N = nw/e
where N is the number of training data, nw is the
total number of weights (nodes) and e is the
network error permitted for the testing phase.
• Eg. If nw is 10, and e is 10%, then N=10/0.1 = 100
 What is data massaging?

• An ANN works best when all its inputs and

outputs vary within the range between 0 and 1
• All data must be massaged before can be used in
an ANN model
 How do we massage the data?

• Data can be divided into 3 main types :

▫ Continuous
▫ Discrete
▫ categorical
 Continuous data

• Continuous data vary between two pre-set values :

minimum and maximum and can be massaged to
the range between 0 and 1 as follows:
▫ Massaged_value=
(actual-value – minimum_value)/(maximum-value
– minimum_value)
 Discrete data

• Discrete data such as the number of bedrooms and

bathrooms can be massaged as follows:
No of bedrooms Massaged value
1 0
2 0.25
3 0.5
4 0.75
 Categorical data
• Eg gender and marital status can be massaged by
using 1 of N coding.
• Eg. Marital status can be either single, married,
divorced or widowed can be massaged as follows:
single married divorces widowed
0 0 0 1
0 0 1 0
0 1 0 0
1 0 0 0
Convolutional Neural Network
https://vision.cs.hacettepe.edu.tr/siu2016-tutorial/part-i-history-and-motivations.pdf
https://vision.cs.hacettepe.edu.tr/siu2016-tutorial/part-i-history-and-motivations.pdf
https://vision.cs.hacettepe.edu.tr/siu2016-tutorial/part-i-history-and-motivations.pdf
https://vision.cs.hacettepe.edu.tr/siu2016-tutorial/part-i-history-and-motivations.pdf
https://vision.cs.hacettepe.edu.tr/siu2016-tutorial/part-i-history-and-motivations.pdf
https://www.slideshare.net/roelofp/deep-learning-a-birdseye-view
STATE-OF-THE-ART OF DEEP LEARNING

Easy access to massive data

Increased computing power

Pre-trained models
WHY IS DEEP LEARNING POPULAR?

Results achieved are

substantially better
Large Scale Visual Recognition Challenge (ILSVRC2017)
1 mil images, 1000 categories, classification error : 0.027
CONVOLUTIONAL NEURAL NETWORK
(CNN)
 CONVOLUTIONAL LAYER
•Line up the feature and the
image patch.

•Multiply each image pixel by

the corresponding feature
pixel.

•Add them up.

•Divide by the total number of

pixels in the feature.
CONVOLUTIONAL LAYER
 MAX-POOLING LAYER

•Pick a window size (usually 2 or 3).

•Pick a stride (usually 2 or 3).
•Walk your window across your filtered
images.
•From each window, take the maximum
value.
SIMILAR PATTERNS BUT IN A
SMALLER SIZE
POOLING LAYER
SAMPLE DATA NOT SUITABLE
FOR CNN
RULE OF THUMB OF USING CNN

If your data is just as useful after swapping

any of your columns with each other, then
you cannot use CNN.
Factors that affect the performance of CNN

Convolution - Number and size of features

Pooling - Window size and stride

Fully connected - Number of neurons

Architecture
•How many of each type of layer
•In what order
Training Convolutional Neural
Network using Matlab
SAMPLE APPLICATION:
DIGIT RECOGNITION
 TRAIN CNN FROM SCRATCH
DATA PREPARATION
SAMPLE DIGITS IN DIGITDATASET
DIGIT RECOGNITION PERFORMANCE
(PARAMETER TUNING)

Conv layer stride epoch accuracy Training+Testing time (s)

5,20 2 15 0.9812 57.644

5,20 3 15 0.9848 31.606

9,40 3 15 0.9908 47.471

DIGIT RECOGNITION PERFORMANCE
(ADDITIONAL LAYERS)

Conv layer stride Epoch, learning rate accuracy Training time/s

5,20 3
15, 0.0001 0.9144 36.708
3,20 3
9,40 3 15, 0.0001 0.9192, 50.612

3,20 2 30, 0.0001 0.9492 100.417

9,40 3
15, 0.001 0.9964 45.115
5,20 2
PRE-TRAINED NETWORK MODEL
 25x1 Layer array with layers of Alexnet
1 'data' Image Input 227x227x3 images with 'zerocenter' normalization
2 'conv1' Convolution 96 11x11x3 convolutions with stride [4 4] and padding [0 0]
3 'relu1' ReLU ReLU
4 'norm1' Cross Channel Normalization cross channel normalization with 5 channels per element
5 'pool1' Max Pooling 3x3 max pooling with stride [2 2] and padding [0 0]
6 'conv2' Convolution 256 5x5x48 convolutions with stride [1 1] and padding [2 2]
7 'relu2' ReLU ReLU
8 'norm2' Cross Channel Normalization cross channel normalization with 5 channels per element
9 'pool2' Max Pooling 3x3 max pooling with stride [2 2] and padding [0 0]
10 'conv3' Convolution 384 3x3x256 convolutions with stride [1 1] and padding [1 1]
11 'relu3' ReLU ReLU
12 'conv4' Convolution 384 3x3x192 convolutions with stride [1 1] and padding [1 1]
13 'relu4' ReLU ReLU
14 'conv5' Convolution 256 3x3x192 convolutions with stride [1 1] and padding [1 1]
15 'relu5' ReLU ReLU
16 'pool5' Max Pooling 3x3 max pooling with stride [2 2] and padding [0 0]
17 'fc6' Fully Connected 4096 fully connected layer
18 'relu6' ReLU ReLU
19 'drop6' Dropout 50% dropout
20 'fc7' Fully Connected 4096 fully connected layer
21 'relu7' ReLU ReLU
22 'drop7' Dropout 50% dropout
23 'fc8' Fully Connected 1000 fully connected layer
24 'prob' Softmax softmax
25 'output' Classification Output crossentropyex with 'tench', 'goldfish', and 998 other classes
 IMAGENET DATASET

•15 million images

•22000 categories
SAMPLE IMAGES FROM IMAGENET
5 categories, 15 images each category
Accuracy: 1
Training and testing time: 45.772s
VISUALIZE THE FEATURES
EXTRACTED AT EACH
CONVOLUTIONAL LAYER
A glimpse into the future

https://www.youtube.com/watch?v=vvimBPJ3XGQ
Review questions
1. What is ANN?
2. What is the difference between machine
learning and deep learning?
3. How does an ANN learn?
4. What type of application is suitable to be
developed using ANN ?