Chapter 6: Counterpropagation Network

I. Combine competitive network and Grossbergs outstar

structure
II.
Simulating functional mapping, i.e.,
( ) y = x
, where

: a continue function , e.g., cos

III.
Backward-mapping, i.e.,
1
( )

x = y
6-1
Forward-mapping CPN
6.1. CPN Building Blocks
Three major components
1. Instar: hidden node with input weights
2. Competitive layer: hidden layer composed of instars
3. Outstar: a structure
6.1.2 The Instar
Input:

t
net I w
. Assume
1 I w
6-2
Output:
( ) y ay b net + &
, a, b > 0 ------ (A)
Solve the above equation and assume y(0) = 0
( ) ( )(1 ) (1 )
at eq at
b
y t net e y e
a


( )
eq
b
y net
a

If input removed at t

, i.e., net = 0
From
( ), y ay b net y ay + & &

Solve it ( )
eq at
y t y e

Learning of instar -- learn to respond maximally to a particular

input vector
c d + w w I & , c, d > 0 ------- (6.8)
6-3
In the absence of I

c w w &

Called forgetfulness
To avoid forgetfulness, an alternative learning rule:
( ) ( ) c d U net + w w I &
------ (6.10)
where
1 0
( )
0 0
net
U net
net

'

When net
0
,
,
Let . Then, ( )
( 1) ( ) ( ) ( ( ))
c d c t d t
t
c t d t
t t t t

+ +

+ + +
w
w w I w w I
w I w
w w w w I w
&

Learn a cluster of patterns

Set initial weight vector to Average
some member of the cluster
6-4
Learning step:
1. Select an input vector
i
I
at random
2.
Calculate
( ) t w
(I
i

( ) t w
)
3.
Update
( 1) ( ) ( ) t t t + + w w w
4. Repeat steps 1-3 for all input vectors in the cluster
5. Repeat 1-4 several times

Reduce the value of

as training proceeds
6.1.3. Competitive Layer
The hidden layer is a competitive layer formed by instars
6-5
An on-center off-surround system can be used to implement
the competition among a group of instars
The unit activation
[
( ) ( ) ]
( ( ) )
i i i i i
k
i k
k i
net
x Ax B x f x net
x f x

+ +
1
+
1
]
&
----- (6.13)
where A, B: positive constant,
( ) f
: transfer function
Rearrange
[
( ) ] ( )
i i i i i k k
k k
x Ax B f x net x f x net
1
+ + +
1
]
&
----- (6.13)
Sum over i,
( ) ( )
k k
k k
x Ax B x f x net
1
+ +
1
]
&
,
i
i
x x
Let
k k
x xX
, where
k
X
: reflectance variable
( ) ( )
k k
k k
x Ax B x f xX net
1
+ +
1
]
&
----- (6.14)
6-6
From
,
i i i i i ii i
x xX x xX xX xX x xX +
& &
& & & &
----- (A)
Substitute (6.13)
[
( ) ] ( )
i i i i i k k
k k
x Ax B f x net x f x net
1
+ + +
1
]

&
and (6.14)
( ) ( )
k k
k k
x Ax B x f xX net
1
+ +
1
]

&
into
(A)
[
( ) ] ( )
{ ( ) ( ) }
i i i i i i i k k
k k
k k i
k k
xX x xX Ax B f x net x f xX net
Ax B x f xX net X
1
+ + +
1
]
1
+ +
1
]


&
& &
---
(B)
(
i i
AxX Ax
) [ ]
( ) ( )
i i i k k
k k
B f x net x f xX net
1
+ +
1
]

( ) ( ) ( )
( ) ( )

i k i k
k k
i i i k i k
k k
i k i k
k k
B x X f xX B x X net
Bf x Bnet x net BX f xX
BX net xX net

+
+



6-7
( ) ( )
(Let ( ) ( ) )
( ) ( )

i i k i i k
k k
i i i k k
k
i i k
k
Bf xX BX f xX Bnet BX net
f g
BxX g xX BX xX g xX
Bnet BX net

+

[ ]
( ) ( )

i k i k i
k
i k i i
k i
BxX X g xX g xX Bnet
BX net BX net

[ ]
( ) ( )
(1 )
i k i k
k
i i i k
k i
BxX X g xX g xX
B X net BX net

[ ]
( ) ( )
(1 )
i i k i k
k
i i i k
k i
xX BxX X g xX g xX
B X net BX net

&
----- (6.15)
where
1
( ) ( ) g w w f w

or
( ) ( ) f g
Let
( ) (linear) ( ) 1 f g
6-8
[ ]
( ) ( ) 0
(1 )
stablizes (i.e., 0) at

at equilibrium
i k i k
k
i i i i k i i k
k i k
i
i i i
k
k
eq
i
i i
k
k
BxX X g xX g xX
xX B X net BX net Bnet BX net
net
X X X
net
net
x xX x
net

&
&
Q
If the input pattern is removed (i.e.,
0
i
net
),
(6.14) =>
( ) ( )
k
k
x Ax B x f xX +

&
(
( ) ( ) ( ( ) )
k k k
f xX f x x f Q Q
( )
k k
k k
f xX x x

)
2
( ) ( ) Ax B x x B A x x +
If , then 0 decreasing
If , increases until , which
is then stored permanently on the units.
B A x x
B A x x B A
<
>
&
Called short-term memory

Example:
( ) (linear function) f w w
6-9
Example:
2
( ) f w w
(fast-than-linear output function)

1 1 2
( ) ( ) g w w f w w w w

Q
( ) ( ) [ ] g xX g xX xX xX x X X
i i i
k k k

i. if , [ ( ) ( )] is an
excitatory term for


k i k
X X X g xX g xX
i
k
x
i
>
6-10
Otherwise, an inhibitory term for x
i

The network tends to enhance the activity of the unit with
the largest value of
X
i
ii. After the input pattern is removed ,
(6.13) =>
2 2 2 2
( ) ( )
( )
i i i i k
k
i i i k i k i
k k
x Ax Bf x x f x
Ax Bx x x Bx A x x

+
+ +
&

With appropriate and , 0 for only
the unit with the largest
i
i
A B x
x
> &
6-11
iii. ( ) can be used to implement a
winner-take-all network when (n > 1)
n
f w w

Example: sigmoid function (contrast enhancement)
Quenching threshold (QT): units whose net inputs are above
the QT will have their activities enhanced
6.1.4. Outstar
-- is composed of a single hidden unit and all output units
6-12
Classical-conditioning
Hebbian rule
6-13

Initially, the conditioned stimulus (CS) is assumed to be
unable to elicit a response from any of the units to which
it is connected
An unconditioned stimulus (UCS) can cause an
unconditioned response (UCR)
If the CS is present while the UCS is causing the UCR , then
the strength of the connection from the CS unit to the UCR
6-14
unit will also be increased
Later, the CS will be able to cause a conditioned response
(CR), even if the UCS is absent
During the training period, the winner of the competition
on the hidden layer turns on, providing a single CS to the
output units. The UCS is supplied by the Y portion of the
input layer.
After training is complete, the appearance of the CS will
cause the CR value to appear at the output units even
though the UCS values will be zero
Recognize an input pattern through a winner-take-all
competition. Once a winner has been declared, that
6-15
unit becomes the CS for an outstar.
The outstar associates some value or identity with the
input pattern
The output values of the outstar
i, During training phase

y ay by cnet
i i i i

+ + &

where , , 0, : training input
: the output of the outstar node
a b c y
i
y i
i
>

Only a single hidden unit c (winner) due to competition

has a nonzero output at any given time.

Net input to any output neuron i reduces to a single
term
( 1 the output of the winner) w z z
ic

. ., i e net z w w y ay by cw
i ic ic i i i ic

+ + &
ii, After training --

Training input absent y
i
----- (6.17)
i
y ay cw
i i

+ &
During training, the evolution of weights
( ) ( )
i i i
w dw ey z U z + &

(similar to instar)
Only the winner
( 1, ( ) 1) z U z
involves learning
6-16
----- (6.19)
i i i
w dw ey + &
At equilibrium, 0
i
w &
,
for identical output of the
members of a cluster
: for average output of the
members of a cluster
:
eq
i i
eq
i i
e
w y
d
e
w y
d

< >

After training, (6.17)
At eguilibrium
Want , ,
From (6.19), ( 1) ( ) ( ( ))
(
i i i
eq
i i
eq
i i
eq eq
i i i
i i i i
i
ce
y ay y
d
c e
y y
a
d
y y a c d e
y y w
w t w t ey dw t
w t

+ < >

< >

< >

< >
+ +

&
Q
) ( ( ))
i i
y w t +

Summary of training the CPN
Two learning algorithms for
(a) Competitive layer (input hidden layers)
(b) Output layer (hidden output layers)
6-17
(a) Competitive layer training
1. Select an input vector I
2.
Normalize
( 1) x x
3. Apply to the competitive layer
4. Determine the winner c
5.
Calculate
( )
c
x w
for the winner
6. Update the winner weight vector

( 1) ( ) ( ) t t
c c c
+ + w w x w
6-18
7. Repeat steps 1- 6, until all input vectors have been
processed
8. Repeat step 7 until all input vectors have been
classified properly
We may remove the hidden nodes, which havent
been used
(b) Output layer training
1.
Apply a normalized input vector
k
x
, and its
corresponding output vector
k
y
, to the X and
Y portions of the CPN, respectively.
2. Determine the winner c
3. Update its associated weights
( 1) ( ) ( ( )) w t w t y w t
ic ic i ic
+ +
,
1, ..., i m
4. Repeat steps 1- 3, until all vectors of all classes
map to satisfactory outputs
Forward processing (production phase)
1. 2
, 1
i
i
i
I
x
I
n

x
2. Apply the input vector to X portion of the input layer
Apply zero vector to Y portion of the input layer
6-19
3. X portion units are distributed to the competitive layer
4. The winner unit has an output of 1
All other units have outputs of 0
5. Excites the corresponding outstar
The output of the outstar units is the value of the weights
on the connection from the winner unit.
6.2.4. Complex CPN
The Full CPN:
Forward mapping
Reverse mapping

'

x y
y x
6-20
During training, both x and y are applied to the input units

input ( , ) results in an output ( )
After training,
input ( , ) results in an output

'

x y x
y x
Let r: the weight vector from the portion of x inputs to any
hidden unit
s: the weight vector from the portion of y inputs to any
hidden unit
For hidden unit i,
i
net + r x s y
Its output
1 if max{ }
0 otherwise
k
net net
i
k
z
i

'

During training
( ) ( ) ,
i x i i y i
r x r s y s
& &
Note that only the winner is allowed to learn for a given
input vector
For output layer,
, units have weight vector
units have weight vector
i
i

y w
x v
The learning laws:
, ( ) ( )
ij y i ij ij x i ij
w v y w x v & &
Note that only the winner is allowed to learn
6-21