Soft Computing

Neural Networks
k (2)
]É{Ç V
] V{tÉ}|tÇ f{|
f
g
Merchant Marine College
Shanghai Maritime University
 Perceptron
Learning Rule
 Learning Rules
• Supervised Learning
– Network is provided with a set of examples of proper
network behavior (inputs/targets)
{ p1,t1 } ,{p2,t2} … {pQ,tQ}
• Reinforcement Learning
– Network is only provided with a grade, or score, which
indicates network performance.
• Unsupervised Learning
– Only network inputs are available to the learning
algorithm. Network learns to categorize (cluster) the
inputs.
 Perceptron rule
• Description of objects in terms of attributes
• Encoding of attributes in binary (or numeric)
representation, as a feature vector xi
• Linear classifier

• Perceptron algorithm: update on mistake: if

f(xi)≠yi
f(

2007-4-9
4
 Discriminant Function

• Formally (for binary inputs)

• More generally:

• Maps a pattern to its class or label

 Linear Discriminant

• IIntuitively,
t iti l each h attribute
tt ib t which
hi h iis presentt contributes
t ib t
with a weight that may be positive or negative.
• A pos
positive
t e weight
e g t indicates
d cates tthat
at tthe
e att
attribute
bute p
provides
o des
evidence for the mushroom being edible, a negative
weight provides evidence for the opposite hypothesis.
• The magnitude of the weight determines the influence of
this attribute relative to other attributes.
• The bias specifies the (negative of the) threshold for
making
ki th the fifinall d
decision
i i bbased d on ththe accumulative
l ti
evidence.
 Geometry of Linear Classifiers

• Decision boundary is a
hyperplane (line in 2d).
• Input space is divided into
two half-spaces.
• Weight vector is normal
vector.
• Distance from origin is
controlled by b (and || w ||).

2007-4-9
7
 Functional Margin
• Functional margin of a data point xi with
respect to a hyperplane (w
(w, b)

• For ||w|| = 1 this is also called the geometric

margin,
g , since it corresponds
p to the distance
of xi from the decision surface
• The sign indicates whether a data point ends
up on the
th correctt (positive)
( iti ) or iincorrectt
(negative) side of the hyperplane.
 Learning Linear Discriminant
• Goal: Adjust weights and bias in way
that maximizes the classification
y
accuracy.
• Problem has been restricted to a pre-
specified hypothesis class (linear
discriminants)
• Learning procedure = parameter fitting
 Perceptron Algorithm (1)
• Initialize the weights & bias
• Cycle through the training data
– Test whether current example p is correctly
y
classified
– If not,, perform
p an update
p step
p by
y adjusting
j g
w,b. Learning from mistakes.
• Until all training data are correctly
classified
Perceptron Algorithm (2)
 Sanity
y Check ...
• How does the value of f change on a training
example (xi, yi) after an update has been
performed?
• We denote by the margin before the
update and by the margin after the
update.
update
• On can write

• Th
The signed
i d margin
i off the
th example l xi will
ill
increase by at least 1 after an update step
using this example
example. Why is this good?
 Linearly Separable

• A set of labeled points {(x1, y1), . . . , (xn, yn)} is called

linearly separable, if there exists w, b such that

– This means there is a hyperplane such that all positive

training points will be in one half-space and all negative ones
in the other.

• What happens, if the training data are not linearly

separable?
– The perceptron termination condition can never be fulfilled
fulfilled.
So perceptron learning will not converge!
Novikoff’s Theorem
γ-Separating Line
Novikoff’s Theorem,, Proof (1)
( )
Novikoff’s Theorem, Proof (2)
 Novikoff’s Theorem, Proof (3)

2007-4-9
Perceptron Architecture
Single-Neuron Perceptron
Decision Boundary
Example - OR
OR Solution
 Multiple-Neuron Perceptron

Each neuron will have its own decision boundary.

iw p + bi = 0
T

A single neuron can classify input vectors

into two categories.
categories
A multi-neuron perceptron can classify
input vectors into 2S categories.
Learning Rule Test Problem
Starting Point
Tentative Learning Rule
Second Input Vector
Third Input Vector
Unified Learning Rule
Multiple-Neuron Perceptrons
Apple/Banana Example
Second Iteration
Check
 Perceptron Rule Capability

The perceptron rule will always

converge to weights which accomplish
the desired classification,
classification assuming that
such weights
g exist.
Perceptron Limitations
S
Supervised
i dH Hebbian
bbi L Learning
i
 Hebb’s Postulate
• “When an axon of cell A is near enough to excite
a cell B and repeatedly
repeatedl or persistently
persistentl takes part in
firing it, some growth process or metabolic change
takes place in one or both cells such that A A’ss
efficiency, as one of the cells firing B, is
increased.”
D. O. Hebb, 1949
Linear Associator
Hebb Rule
Batch Operation
Performance Analysis
Example
Pseudoinverse Rule - (1)
Pseudoinverse Rule - (2)
Relationship to the Hebb Rule
Example
Autoassociative Memory
Tests
Variations of Hebbian Learning
Widrow-Hoff Learning
(LMS Algorithm)
ADALINE Network
Two-Input ADALINE
Mean Square Error
Error Analysis
Stationary Point
Approximate Steepest Descent
Approximate Gradient Calculation
LMS Algorithm
Multiple-Neuron Case
Analysis of Convergence
Conditions for Stability
Steady State Response
Example
Iteration One
Iteration Two
Iteration Three
Adaptive Filtering
Example: Noise Cancellation
Noise Cancellation Adaptive Filter
Correlation Matrix
Signals
Stationary Point
Performance Index
LMS Response
Echo Cancellation
Questions ?