Nanjing University of Science & Technology

Pattern Recognition:
Statistical and Neural

Lonnie C. Ludeman

Lecture 24

Nov 2, 2005
1
 Lecture 24 Topics

1. Review and Motivation for Link Structure

2. Present the Functional Link Artificial Neural
Network.
3. Simple Example- design using ANN and FLANN
4. Performance for Neural Network Designs
5. Radial Basis Function Neural Networks
6. Problems, Advantages, Disadvantages, and
promise of Artificial Neural Network Design

2
 Generalized Linear Discriminant Functions
w1
g1(x) x
w2
g2(x) x
… wj
x
gj(x) + g(x)
x
…

wM
gM(x) x
Review 1 3
Patterns are linearly separated in the 3-dim space

Separating plane

4
Review 2
 Example: Decision rule using one nonlinear
discriminant function g(x)

Given the following g(x) and decision rule

Illustrate the decision regions R1 and R2

where we respectively classify as C1 and C2
for the decision rule above
Review 3 5
 Solution:

R1 decide C1
R2 decide C2

every where else

every where else

Review 4 6
 Find a generalized linear discriminant
function that separates the classes
Solution:
d(x) = w1f1(x)+ w2f2(x)+ w3f3(x)
+ w4f4(x) +w5f5(x) + w6f6(x)
T
= w f (x)

in the f space (linear)

Review 5 7
 where

in the original pattern space: (nonlinear)

Review 6 8
Decision Boundary in original pattern space
x2
from C1
2
from C2

1 2 3 4 x1
-1
Boundary
-2 d(x) = 0

Review 7 9
 Potential Function Approach – Motivated
by electromagnetic theory + from C1
- from C2

Sample space
Review 8 10
 Plot of Samples from the two classes

Review 9 11
Given Samples x from two classes C1 and C2

S1 C1
S2 C2

Define Total Potential Function

K(x) = ∑ K(x, xk) - ∑ K(x, xk)
x k C S1 xk C S2
Potential
Function
Decision Boundary
K(x) = 0
Review 10 12
 Algorithm converged in 1.75 passes through
the data to give final discriminant function as

Review 11 13
Functional Link Neural Network

14
Quadratic Functional Link

15
Fourier Series Functional Link

16
Principal Component Functional Link

fk(x), k=1 to N are chosen as

the eigen vectors of the
sample covariance matrix

17
Example: Comparison of Neural Net and
functional link neural net
Given two pattern classes C1 and C2 with the
following four patterns and their desired outputs

18
(a)Design an Artificial Neural Network to
classify the two patterns given
(b) Design a Functional Link Artificial
Neural Network to classify the
patterns given.
(c) Compare the Neural Net and
Functional Link Neural Net designs

19
(a) Solution: Artificial Neural Net Design

Select the following structure

20
After training using the training set and the
backpropagation algorithm the design
becomes
Values determined
by neural net

21
(b) Solution: Functional Link
Artificial Neural Net Design

22
A neural net was trained using the functional
link output patterns as new pattern samples

The resulting weights and structure are

23
(c) Comparison Artificial Neural Net (ANN)
and Functional Link Artificial Neural Net
(FLANN} Designs

FLANN has simpler structure than the

ANN with only one neural element and
Link.

Fewer iterations and computations in the

training algorithm for FLANN.

FLANN design may be more sensitive to

errors in patterns. 24
Determining Performance of Neural Net Design

on Training Set

25
 Determine Performance for
Design using Training Set

Classify each member of the training

set using the neural network design.

Test Design
on Testing Set

Classify each member of the testing

set using the neural network design.

26
 Could use
(a) Performance Measure ETOT
(b) The Confusion Matrix
(c) Probability of Error
(d) Bayes Risk

27
(a) Local and global errors- Used in
Neural Net Design procedure
Local Measure

Global Measure

28
(b) Confusion Matrix- Example

Correct Incorrect
Classification Classification
29
(c) Probability of Error- Example
Estimates of Probabilities of being Correct

Estimate of Total Probability of Error

30
(d) Bayes Risk Estimate

31
Radial Basis Function (RBF)
Artificial Neural Network

Functional
Link

32
Functional Form of RBF ANN

where

Examples of Nonlinearities

33
 Design Using RBF ANN
Let F(x1, x2, … , xn) represent the function we
wish to approximate.

For pattern classification F(x) represents the

class assignment or desired output (target value)
for each pattern vector x a member of the training set
Define the performance measure E by

E
We wish to Minimize E by selecting
M, a ,b1, b2, ... , bM and z1, z2, ... zM
34
Finding the Best Approximation using RBF ANN

Usually broken into two parts

(1st ) Find the number M of prototypes and
the prototypes { zj : j=1, 2, ... , M} by using a
clustering algorithm(Presented in Chapter 6)
on the training samples
(2nd ) With these fixed M and { zj: j=1,2, ... , M}
find the a ,b1, b2, ... , bM that minimize E.

Notes:
You can use any minimization procedure you wish.
Training does not use the Backpropagation Algorithm
35
Problems Using Neural Network Designs
Failure to converge
Selection of insufficient structure
Max iterations too small
Lockup occurs
Limit cycles

Good performance on training set –

poor performance on testing set
Training set not representative of variation
Too strict of a tolerance - “grandmothering”

36
Advantages of Neural Network Designs

Can obtain a design for very complicated

problems.

Parallel structure using identical elements

allows hardware or software implementation

Structure of Neural Network Design

similar for all problems.

37
Other problems that can be solved using
Neural Network Designs

System Identification
Functional Approximation
Control Systems

Any problem that can be placed in the

format of a clearly defined desired output
for different given input vectors.
38
Famous Quotation
“Neural network designs are the second
best way to solve all problems”

39
Famous Quotation
“Neural network designs are the second
best way to solve all problems”

??????????
??????????

40
Famous Quotation
“Neural network designs are the second
best way to solve all problems”

??????????
??????????
The promise is that a Neural Network can
be used to solve all problems; however,
with the caveat that there is always a better
way to solve a specific problem.
41
So what is the best way to
solve a given problem ???

42
So what is the best way to
solve a given problem ???

43
So what is the best way to
solve a given problem ???

?
A design that uses and
understands the structure
of the data !!!
44
 Summary Lecture 24

1. Reviewed and Motivated Link Structure

2. Presented the Functional Link Artificial Neural
Network.
3. Presented Simple Example with designs using
ANN and FLANN
4. Described Performance Measures for Neural
Network Designs
5. Presented Radial Basis Function Neural
Networks

45
6. Discussed Problems, Advantages,
Disadvantages, and the Promise of
Artificial Neural Network Design

46
End of Lecture 24

47