241-320 Design Architecture and Engineering for Intelligent System Suntorn Witosurapot

Contact Address: Phone: 074 287369 or Email: wsuntorn@coe.psu.ac.th

January 2011

Lecture 17: Introduction to Neural Networks

What Are Artificial Neural Networks?

An extremely simplified model of the brain Essentially a function approximator
Transforms inputs into outputs to the best of its ability

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

What Are Artificial Neural Networks?

Composed of many neurons that co-operate to perform the desired function

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

What Are They Used For?

Classification
Pattern recognition, feature extraction, image matching

Noise Reduction
Recognize patterns in the inputs and produce noiseless outputs

Prediction
Extrapolation based on historical data

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

What Are They Used For? (cont.)

Examples:
Predicting time-series in financial world ( ) Fraud detection (- ) Object Separation () etc

Very useful in Data Mining imitate a humans ability to learn from experience better results are the hope
(Adequately designed and trained NN can capture varied patterns)

Drawback models tend to be difficult to understand (Is it necessary to understand?)

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

What Are They Used For? (cont.)

Real Estate Appraiser is not applying some set formula, but balancing her experience and knowledge of sale prices of similar housesher knowledge about housing prices is not static...fine tuning her calculation to fit the latest data
241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

 Loan Prospector ( )

A Neural Network is like a black box that knows how to process inputs to create a useful output. The calculation is quite complex and difficult to understand, yet the results are often useful
241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

Neural Net Attractions

Ability to learn
NNs figure out how to perform their function on their own

Determine their function based only upon sample inputs

Ability to generalize
i.e. produce reasonable outputs for inputs it has not been taught how to deal with

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

Neural Net Limitations

Neural Nets are good for prediction and estimation when: Inputs are well understood Output is well understood Adequate examples are trained the neural net Neural Nets are only as good as the training set used to generate it. The resulting model is static and must be updated with more recent examples and retraining for it to stay relevant
241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

10

Feed-Forward Neural Net Examples

One-way flow through the network from inputs to outputs

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

11

Feed-Forward Neural Net Examples (cont.)

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

12

A Biological neuron
Dendrits cell and nucleus Dendrits: (Input) Getting other activations Axon: (Output ) forward the activation (from 1mm up to 1m long) Synapse: transfer of activation: to other cells, e.g. Dendrits of other neurons a cell has about 1.000 to 10.000 connections to other cells Cell Nucleus: (processing) evaluation of activation

Axon (Neurit) Synapsis

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

13

Natural vs. Artificial Neuron

w1i w2i

...

wji

net i w ji o j
acti f act (neti , i )
Dendrits cell and nucleus
net w act fact
j

oi f out (acti )

oi
: input from the network : weight of a connection : activation : activation function : bias/threshold : output function (mostly ID) : output 14

Axon (Neurit) Synapsis

241-320 Design Architecture & Engineering for Intelligent System

fout o

Introduction to Neural Networks

Abstraction
Dendrits: weighted (real number) connections Axon: output: real number Synapse: --(identity: output is directly forwarded) Cell nucleus: unit contains simple functions input = (many) real numbers processing = activation function output = real number (~activation)
241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

15

Artificial Neuron

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

16

Loan Prospector
( )

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

17

NN can have multiple output neurons

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

18

How does Neural Net work?

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

19

Common transfer functions

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

20

Where Do The Weights Come From?

The weights in a neural network are the most important factor in determining its function Training - process of setting the best weights on the edges connecting all the units in the network Use the training set to calculate weights such that NN output is as close as possible to the desired output for as many of the examples in the training set as possible

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

21

Where Do The Weights Come From? (cont.)

learning examples

Network

evaluation network error

changing
network parameters

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

22

Where Do The Weights Come From? (cont.)

There are two main types of training Supervised Training
Supplies the neural network with inputs and the desired outputs Response of the network to the inputs is measured Adjust weights such that differences between desired and actual outputs are minimized

Unsupervised Training
Only supplies inputs

The neural network adjusts its own weights so that similar inputs should generate the similar outputs The network identifies the patterns and differences in the inputs without any external assistance
241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

23

Example: Voice Recognition

Task: Learn to discriminate between two different voices saying Hello Data
Sources
Steve Simpson David Raubenheimer

Format
Frequency distribution (60 bins) Analogy: cochlea
241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

24

Network architecture
Feed forward network
60 input (one for each frequency bin) 6 hidden 2 output (0-1 for Steve, 1-0 for David)

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

25

Presenting the data

Steve

David

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

26

Presenting the data (untrained network)

Steve
0.43
0.26

David
0.73 0.55

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

27

Calculate error
Steve
0.43 0 0.26 1 = 0.43 = 0.74

David
0.73 1 0.55 0 = 0.27 = 0.55

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

28

Backprop error and adjust weights

Steve
0.43 0 0.26 1 = 0.43 = 0.74 1.17 David 0.73 1 0.55 0 = 0.27 = 0.55 0.82
241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

29

 Repeat process (sweep) for all training pairs

Present data Calculate error Backpropagate error Adjust weights

Repeat process multiple times

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

30

Presenting the data (trained network)

Steve

0.01 0.99

David

0.99 0.01

241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

31

Results Voice Recognition

Performance of trained network Discrimination accuracy between known Hellos
100%

Discrimination accuracy between new Hellos

100%

Summary: Network has learnt to generalise from original data Networks with different weight settings can have same functionality Trained networks concentrate on lower frequencies Network is robust against non-functioning nodes
241-320 Design Architecture & Engineering for Intelligent System

Introduction to Neural Networks

32