Professional Documents
Culture Documents
04 Associative Memory
04 Associative Memory
04 Associative Memory
.in
rs
de
ea
yr
Associative Memory : Soft Computing Course Lecture 21 – 24, notes, slides
.m
w
w
www.myreaders.info/ , RC Chakraborty, e-mail rcchak@gmail.com , Aug. 10, 2010
,w
ty
http://www.myreaders.info/html/soft_computing.html
or
ab
www.myreaders.info
kr
ha
C
C
R
Associative Memory
Soft Computing
yr
.m
w
w
,w
Soft Computing
ty
or
ab
Topics
kr
ha
C
5. References 42
02
fo
.in
rs
de
ea
yr
Associative Memory
.m
w
w
,w
ty
or
ea
1. Associative Memory
yr
.m
w
w
An associative memory is a content-addressable structure that maps a
,w
ty
or
04
fo
.in
rs
de
SC - AM description
ea
1.1 Description of Associative Memory
yr
.m
w
w
An associative memory is a content-addressable structure that allows,
,w
ty
or
Blaise Pascal
Marco Polo
Neil Armstrong
Sigmund Freud
ea
[Continued from previous slide]
yr
.m
w ■ Associative memory is a system that associates two patterns (X, Y)
w
,w
ty
Auto-associative memory
Consider, y[1], y[2], y[3], . . . . . y[M], be the number of stored
pattern vectors and let y(m) be the components of these vectors,
representing features extracted from the patterns. The auto-associative
memory will output a pattern vector y(m) when inputting a noisy or
incomplete version of y(m).
Hetero-associative memory
Here the memory function is more general. Consider, we have a
number of key-response pairs {c(1), y(1)}, {c(2), y(2)}, . . . . . . ,
{c(M), y(M)}. The hetero-associative memory will output a pattern
vector y(m) if a noisy or incomplete verson of the c(m) is given.
ea
1.2 Working of Associative Memory
yr
.m
w
w
• Example
,w
ty
or
ab
a stored pattern.
Fig below illustrates the working of an associated memory.
Input Recalled
Pattern Pattern The association is represented
by the symbol
∆ 7 ÷ Γ
ea
1.3 Associative Memory - Classes
yr
.m
w
w
As stated before, there are two classes of associative memory:
,w
ty
or
auto-associative and
ab
kr
hetero-associative memory.
ha
C
C
R
Examples
08
fo
.in
rs
de
SC - AM description
ea
1.4 Related Terms
yr
.m
w
w
Here explained : Encoding or memorization, Retrieval or recollection,
,w
ty
or
• Encoding or memorization
C
R
p
W= α Σ Wk where
k=1
α is the proportionality or normalizing constant.
09
fo
.in
rs
de
SC - AM description
ea
• Retrieval or recollection
yr
.m
w
w
After memorization, the process of retrieving a stored pattern, given
,w
ty
or
+1 if input j ≥ θ j
Yj =
-1 other wise
10
fo
.in
rs
de
SC - AM description
ea
• Errors and noise
yr
.m
w
w
The input pattern may contain errors and noise, or may be an
,w
ty
or
11
fo
.in
rs
de
SC - AM description
ea
• Performance Measures
yr
.m
w
w
The memory capacity and content-addressability are the measures
,w
ty
or
12
fo
.in
rs
de
SC - AM models
ea
2. Associative Memory Models
yr
.m
w
w
An associative memory is a system which stores mappings of specific input
,w
ty
or
13
fo
.in
rs
de
SC - AM models
ea
2.1 Associative Memory Models
yr
.m
w
w
The simplest and among the first studied associative memory models
,w
ty
or
possesses a very low memory capacity and therefore not much used.
ea
2.2 Network Architectures of AM Models
yr
.m
w
w
The neural associative memory models follow different neural network
,w
ty
or
15
fo
.in
rs
de
SC - AM models
ea
• Linear Associator Model (two layers)
yr
.m
w
w
It is a feed-forward type network where the output is produced in a
,w
ty
or
of processing units, one serving as the input layer while the other as
ha
C
C
the output layer. The inputs are directly connected to the outputs,
R
via a series of weights. The links carrying weights connect every input
to every output. The sum of the products of the weights and the
inputs is calculated in each neuron node. The network architecture
of the linear associator is as shown below.
w11
x1 y1
w21
w12
w22
x2 y2
Xn wnm
Ym
− all n input units are connected to all m output units via connection
ea
[Continued from previous slide]
yr
.m
w − Encoding : The process of constructing the connection weight matrix
w
,w
ty
matrix Wk for an associated pattern pair (Xk, Yk) are computed as:
ha
C
th
(xi)k is the i component of pattern Xk for i = 1, 2, ..., m, and
(yj)k is the j th component of pattern Yk for j = 1, 2, ..., n.
ea
• Auto-associative Memory Model - Hopfield model (single layer)
yr
.m
w
w
Auto-associative memory means patterns rather than associated
,w
ty
or
V1 V2 V3 V4
outputs V
unit is connected to every other unit in the network but not to itself.
− connection weight between or from neuron j to i is given by a
number wij. The collection of all such numbers are represented
by the weight matrix W which is square and symmetric, ie, w ij =wji
for i, j = 1, 2, . . . . . , m.
− each unit has an external input I which leads to a modification
in the computation of the net input to the units as
m
input j = Σ xi w i j + I j for j = 1, 2, . . ., m.
i=1
− each unit acts as both input and output unit. Like linear associator,
ea
[Continued from previous slide]
yr
.m
w Construction of weight matrix W is accomplished by
w
− Weight Matrix :
,w
p
ty
k=1
kr
ea
• Bidirectional Associative Memory (two-layer)
yr
.m
w
w
Kosko (1988) extended the Hopfield model, which is single layer,
,w
ty
or
w11
x1 y1
w21
w12
w22
x2 y2
Xn wnm
Ym
20
fo
.in
rs
de
SC - AM – auto correlator
ea
3. Auto-associative Memory (auto-correlators)
yr
.m
w
w
,w
In the previous section, the structure of the Hopfield model has been
ty
or
Working of an auto-correlator :
21
fo
.in
rs
de
SC - AM – auto correlator
ea
• How to Store Patterns : Example
yr
.m
w
w
Consider the three bipolar patterns A1 , A2, A3 to be stored as
,w
ty
or
an auto-correlator.
ab
kr
A1 = (-1, 1 , -1 , 1 )
ha
C
C
A2 = ( 1, 1 ,
R
1 , -1 )
A3 = (-1, -1 , -1 , 1 )
T 1 -1 1 -1
-1 1 -1 1
[A1 ] 4x1 [A1 ] 1x4 =
1 -1 1 -1
-1 1 -1 1
i
T 1 1 1 -1
1 1 1 -1
[A2 ] 4x1 [A2 ] 1x4 =
1 1 1 -1
i -1 -1 -1 1
T 1 1 1 -1
1 1 1 -1
[A3 ] 4x1 [A3 ] 1x4 =
1 1 1 -1
i -1 -1 -1 1
ea
• Retrieve a Pattern from the Stored Patterns (ref. previous slide)
yr
.m
w
w
The previous slide shows the connection matrix T of the three
,w
ty
or
i
kr
ha
T 3 1 3 -3
3
C
1 3 1 -1
Σ
C
3 1 3 -3
i=1
j -3 -1 -3 3
i= 1 2 3 4 α β
α = ∑ ai t i , j=1 1x3 + 1x1 + 1x3 + -1x-3 = 10 1
α = ∑ ai t i , j=2 1x1 + 1x3 + 1x1 + -1x-1 = 6 1
α = ∑ ai t i , j=3 1x3 + 1x1 + 1x3 + -1x-3 = 10 1
α = ∑ ai t i , j=4 1x-3 + 1x-1 + 1x-3 + -1x3 = -1 -1
new old
Therefore aj = ƒ (ai t i j , aj ) for ∀j = 1 , 2 , . . . , p is ƒ (α , β )
new
a1
= ƒ (10 , 1)
new
a2 = ƒ (6 , 1)
new
a3 = ƒ (10 , 1)
new
a4 = ƒ (-1 , -1)
23
fo
.in
rs
de
SC - AM – auto correlator
ea
• Recognition of Noisy Patterns (ref. previous slide)
yr
.m
w
w
Consider a vector A' = ( 1, 1 , 1 , 1 ) which is a noisy presentation
,w
ty
or
The HDs of A' from each of the stored patterns A1 , A2, A3 are
i= 1 2 3 4 α β
α = ∑ ai t i , j=1 1x3 + 1x1 + 1x3 + 1x-3 = 4 1
α = ∑ ai t i , j=2 1x1 + 1x3 + 1x1 + 1x-1 = 4 1
α = ∑ ai t i , j=3 1x3 + 1x1 + 1x3 + 1x-3 = 4 1
α = ∑ ai t i , j=4 1x-3 + 1x-1 + 1x-3 + 1x3 = -4 -1
new old
Therefore aj = ƒ (ai t i j , aj ) for ∀j = 1 , 2 , . . . , p is ƒ (α , β )
new
a1
= ƒ (4 , 1)
new
a2 = ƒ (4 , 1)
new
a3 = ƒ (4 , 1)
new
a4 = ƒ (-4 , -1)
ea
4. Bidirectional Hetero-associative Memory
yr
.m
w
w
,w
Definition : If the associated pattern pairs (X, Y) are different and if the
model recalls a pattern Y given a pattern X or vice-versa, then it is
termed as hetero-associative memory.
25
fo
.in
rs
de
SC - Bidirectional hetero AM
ea
4.1 Bidirectional Associative Memory (BAM) Operations
yr
.m
w
w
BAM is a two-layer nonlinear neural network.
,w
ty
or
Denote one layer as field A with elements Ai and the other layer
ab
kr
26
fo
.in
rs
de
SC - Bidirectional hetero AM
ea
• Retrieve the Nearest of a Pattern Pair, given any pair
yr
.m
w (ref : previous slide)
w
,w
ty
Example
or
ab
Retrieve the nearest of (Ai , Bi) pattern pair, given any pair (α , β ) .
kr
ha
C
C
gi = , fi < 0
-1 (bipolar)
previous g i , fi = 0
Kosko has proved that this process will converge for any
correlation matrix M.
27
fo
.in
rs
de
SC - Bidirectional hetero AM
ea
• Addition and Deletion of Pattern Pairs
yr
.m
w
w
Given a set of pattern pairs (Xi , Yi) , for i = 1 , 2, . . . , n and a set
,w
ty
or
of correlation matrix M :
ab
kr
− an existing pair (Xj , Yj) can be deleted from the memory model.
R
T T T T
Mnew = X1 Y1 + X1 Y1 + . . . . + Xn Yn + X' Y'
28
fo
.in
rs
de
SC - Bidirectional hetero AM
ea
4.2 Energy Function for BAM
yr
.m
w
w
Note : A system that changes with time is a dynamic system. There are two types
,w
ty
or
the system e.g., weights, data flows. The Energy function (or Lyapunov function)
is a bounded function of the system state that decreases with time and the
system solution is the minimum energy.
Let a pair (A , B) defines the state of a BAM.
− to store a pattern, the value of the energy function for that pattern
We wish to retrieve the nearest of (Ai , Bi) pair, when any (α , β ) pair
is presented as initial condition to BAM. The neurons change
their states until a bidirectional stable state (Af , Bf) is reached. Kosko
has shown that such stable state is reached for any matrix M when it
corresponds to local minimum of the energy function. Each cycle of
decoding lowers the energy E if the energy function for any point
T
(α , β ) is given by E = α M β
T
If the energy E = Ai M Bi evaluated using coordinates of the pair
(Ai , Bi) does not constitute a local minimum, then the point cannot
be recalled, even though one starts with α = Ai. Thus Kosko's encoding
method does not ensure that the stored pairs are at a local minimum.
29
fo
.in
rs
de
SC - Bidirectional hetero AM
ea
• Example : Kosko's BAM for Retrieval of Associated Pair
yr
.m
w
w
The working of Kosko's BAM for retrieval of associated pair.
,w
ty
or
A1 = ( 1 0 0 0 0 1 ) B1 = ( 1 1 0 0 0 )
A2 = ( 0 1 1 0 0 0 ) B2 = ( 1 0 1 0 0 )
A3 = ( 0 0 1 0 1 1 ) B3 = ( 0 1 1 1 0 )
1 1 -3 -1 1
1 -3 1 -1 1
T T T -1 -1 3 1 -1
M = X1 Y1 + X2 Y2 + X3 Y3 =
-1 -1 -1 1 3
-3 1 1 3 1
-1 3 -1 1 -1
Suppose we start with α = X3, and we hope to retrieve the associated pair
Y3 . The calculations for the retrieval of Y3 yield :
α M = ( -1 -1 1 -1 1 1 ) ( M ) = ( -6 6 6 6 -6 )
Φ (α M) = β' = ( -1 1 1 1 -1 )
β' M T = ( -5 -5 5 -3 7 5 )
Φ (β' M T) = ( -1 -1 1 -1 1 1 ) = α'
α' M = ( -1 -1 1 -1 1 1 ) M = ( -6 6 6 6 -6 )
Φ (α' M) = β" = ( -1 1 1 1 1 -1 )
= β'
ea
• Example : Incorrect Recall by Kosko's BAM
yr
.m
w
w
The Working of incorrect recall by Kosko's BAM.
,w
ty
or
A1 = ( 1 0 0 1 1 1 0 0 0 ) B1 = ( 1 1 1 0 0 0 0 1 0 )
A2 = ( 0 1 1 1 0 0 1 1 1 ) B2 = ( 1 0 0 0 0 0 0 0 1 )
A3 = ( 1 0 1 0 1 1 0 1 1 ) B3 = ( 0 1 0 1 0 0 1 0 1 )
X1 = ( 1 -1 -1 1 1 1 -1 -1 -1 ) Y1 = ( 1 1 1 -1 -1 -1 -1 1 -1 )
X2 = ( -1 1 1 1 -1 -1 1 1 1 ) Y2 = ( 1 -1 -1 -1 -1 -1 -1 -1 1 )
X3 = ( 1 -1 1 -1 1 1 -1 1 1 ) Y3 = ( -1 1 -1 1 -1 -1 1 0 1 )
-1 3 1 1 -1 -1 1 1 -1
1 -3 -1 -1 1 1 -1 -1 1
-1 -1 -3 1 -1 -1 1 -3 3
3 -1 1 -3 -1 -1 -3 1 -1
= -1 3 1 1 -1 -1 1 1 -1
-1 3 1 1 -1 -1 1 1 -1
1 -3 -1 -1 1 1 -1 -1 1
-1 -1 -3 1 -1 -1 1 -3 3
-1 -1 -3 1 -1 -1 1 -3 3
ea
[Continued from previous slide]
yr
.m
w
w
Let the pair (X2 , Y2 ) be recalled.
,w
ty
X2 = ( -1 1 1 1 -1 -1 1 1 1 ) Y2 = ( 1 -1 -1 -1 -1 -1 -1 -1 1 )
or
ab
kr
ha
Φ (α M) = ( 1 -1 -1 -1 1 1 -1 -1 1 ) = β'
β' M T = ( -11 11 5 5 -11 -11 11 5 5 )
T
Φ (β' M ) = ( -1 1 1 1 -1 -1 1 1 1 ) = α'
α' M = ( 5 -19 -13 -5 1 1 -5 -13 13 )
Φ (α' M) = ( 1 -1 -1 -1 1 1 -1 -1 1 ) = β"
= β'
αF = α' = ( -1 1 1 1 -1 -1 1 1 1 ) = X2
βF = β' = ( 1 -1 -1 -1 1 1 -1 -1 1 ) ≠ Y2
But β' is not Y2 . Thus the vector pair (X2 , Y2) is not recalled correctly
by Kosko's decoding process.
32
fo
.in
rs
de
SC - Bidirectional hetero AM
ea
[Continued from previous slide]
yr
.m
w Check with Energy Function : Compute the energy functions
w
,w
ty
T
or
T
C
Y2' = ( 1 -1 -1 -1 1 -1 -1 -1 1 )
33
fo
.in
rs
de
SC - Bidirectional hetero AM
ea
4.3 Multiple Training Encoding Strategy
yr
.m
w
w
Note : (Ref. example in previous section). Kosko extended the unidirectional
,w
ty
or
T
kr
retrieve the nearest pair given any pair (α , β ), with the help of recall
R
equations. However, Kosko's encoding method does not ensure that the stored
pairs are at local minimum and hence, results in incorrect recall.
The new value of the energy function E evaluated at (Ai , Bi) then becomes
T T T
E' (Ai , Bi) = – Ai M Bi – (q – 1) Ai Xi Yi Bi
ea
• Example : Multiple Training Encoding Strategy
yr
.m
w
w
,w
A1 = ( 1 0 0 1 1 1 0 0 0 ) B1 = ( 1 1 1 0 0 0 0 1 0 )
A2 = ( 0 1 1 1 0 0 1 1 1 ) B2 = ( 1 0 0 0 0 0 0 0 1 )
A3 = ( 1 0 1 0 1 1 0 1 1 ) B3 = ( 0 1 0 1 0 0 1 0 1 )
X2 = ( -1 1 1 1 -1 -1 1 1 1 ) Y2 = ( 1 -1 -1 -1 -1 -1 -1 -1 1 )
T
Choose q=2, so that P = X2 Y2 , the augmented correlation matrix M
T T T
becomes M = X1 Y1 + 2 X2 Y2 + X3 Y3
4 2 2 0 0 2 2 -2
2 -4 -2 -2 0 0 -2 -2 2
0 -2 -4 0 -2 -2 0 -4 4
= 4 -2 0 -4 -2 -2 -4 0 0
-2 4 2 2 0 0 2 2 -2
-2 4 2 2 0 0 2 2 -2
2 -4 -2 -2 0 0 -2 -2 2
0 -2 -4 0 -2 -2 0 -4 4
0 -2 -4 0 -2 -2 0 -4 4
ea
[Continued from previous slide]
yr
.m
w Now give α = X2, and see that the corresponding pattern pair β = Y2
w
,w
ty
Φ (α M) = ( 1 -1 -1 -1 -1 -1 -1 -1 1 ) = β'
R
Φ (β' M T) = ( -1 1 1 1 -1 -1 1 1 1 ) = α'
α' M = ( 14 -28 -22 -14 -8 -8 -14 -22 23 )
Φ (α' M) = ( 1 -1 -1 -1 1 1 -1 -1 1 ) = β"
αF = α' = ( -1 1 1 1 -1 -1 1 1 1 ) = X2
βF = β' = ( 1 -1 -1 -1 1 1 -1 -1 1 ) = Y2
ea
[Continued from previous slide]
yr
.m
w Note : The previous slide showed that the pattern pair (X2 , Y2 ) is correctly
w
,w
ty
T T T
ha
M = X1 Y1 + 2 X2 Y2 + X3 Y3
C
C
R
but then the same matrix M can not recall correctly the other
pattern pair (X1 , Y1 ) as shown below.
X1 = ( 1 -1 -1 1 1 1 -1 -1 -1 ) Y1 = ( 1 1 1 -1 -1 -1 -1 1 -1 )
Φ (α M) = ( -1 1 1 1 1 1 1 1 -1 ) = β'
β' M T = ( 16 -16 -18 -18 16 16 -16 -18 -18 )
Φ (β' M T) = ( 1 -1 -1 -1 1 1 -1 -1 -1 ) = α'
α' M = ( -14 28 22 14 8 8 14 22 -22 )
Φ (α' M) = ( -1 1 1 1 1 1 1 1 -1 ) = β"
αF = α' = ( 1 -1 -1 -1 1 1 -1 -1 -1 ) = X1
βF = β' = ( -1 1 1 1 1 1 1 1 -1 ) ≠ Y1
Thus, the pattern pair (X1 , Y1 ) is not correctly recalled, using augmented
correlation matrix M.
ea
[Continued from previous slide]
yr
.m
w The previous slide shows that pattern pair (X1 Y1 ) cannot be recalled
w ,
,w
ty
under the same augmentation matrix M that is able to recall (X2 , Y2).
or
ab
kr
ha
Now observe in the next slide that all three pairs can be correctly recalled.
38
fo
.in
rs
de
SC - Bidirectional hetero AM
ea
[ Continued from previous slide ]
yr
.m
w Recall of pattern pair (X1 , Y1 )
w
,w
X1 = ( 1 -1 -1 1 1 1 -1 -1 -1 ) Y1 = ( 1 1 1 -1 -1 -1 -1 1 -1 )
ty
or
ab
αM = ( 3 33 31 -3 -5 -5 -3 31 -31 )
C
R
Φ (α M) = ( 1 1 1 -1 -1 -1 -1 1 -1 ) = β'
(β' M T ) = ( 13 -13 -19 23 13 13 -13 -19 -19 )
Φ (β' M T ) = ( 1 -1 -1 1 1 1 -1 -1 -1 ) = α'
α' M = ( 3 33 31 -3 -5 -5 -3 31 -31 )
Φ (α' M) = ( 1 1 1 -1 -1 -1 -1 1 -1 ) = β"
αF = α' = ( 1 -1 -1 1 1 1 -1 -1 -1 ) = X1
βF = β' = ( 1 1 1 -1 -1 -1 -1 1 -1 ) = Y1
αF = α' = ( -1 1 1 1 -1 -1 1 1 1 ) = X2
βF = β' = ( 1 -1 -1 -1 -1 -1 -1 -1 1 ) = Y2
αF = α' = ( 1 -1 1 -1 1 1 -1 1 1 ) = X3
βF = β' = ( -1 1 -1 1 -1 -1 1 0 1 ) = Y3
ea
[Continued from previous slide]
yr
.m
w Thus, the multiple training encoding strategy ensures the correct recall
w
,w
ty
correlation matrix, for the correct recall of all training pairs, is written as
ha
C
C
N T
R
40
fo
.in
rs
de
SC - Bidirectional hetero AM
ea
• Algorithm (for the Multiple training encoding strategy)
yr
.m
w
w
To summarize the complete process of multiple training encoding an
,w
ty
or
¯i , Y
¯i , q
C
R
For i ← 1 to N
¯i ) ⊗ ( X
M ← M ⊕ [ qi ∗ Transpose ( X ¯i ) end
(symbols ⊕ matrix addition, ⊗ matrix multiplication, and
∗ scalar multiplication)
Step 3 Read input bipolar pattern Ā
end
41
fo
.in
rs
de
SC – AM References
ea
5. References : Textbooks
yr
.m
w
w
1. "Neural Network, Fuzzy Logic, and Genetic Algorithms - Synthesis and
,w
ty
42