Unifying the Hopfield and Hamming binary associative memories
P Houselander and J T Taylor
Department of Electronic & Electrical Engineering, University College London, UK
Abstract
In this paper we shall introduce a mathematical
technique to enable calculation of the capacity
(which is defined) of an associative memory. We
shall use the technique to investigate the
performance (in t e r m of capacity) of the original
Hopfield network and some novel derivatives.
Through a logical sequence of structural
improvements, we show that a natural conclusion
to the extension of the Hopfield nemork is the
Hamming network. This network exhibits the
theoretical maximum capacity obtainable from
an associative memory structure.
The Hopfield Network
The Hopfield network has been the subject of
much research in the field of artificial neural
networks (ANN) since its proposition in 1982
[Hop82]. It operates as an auto-associative
memory and is therefore able to recall stored
patterns (exemplars) when a facsimile of the
pattern .is presented as an input. The network
can be represented as a 2 layer network consisting
of N nodes in each layer (where N is the number
of inputs) and this structure can be seen in Fig. 1
[Was89]. The nodes in layer 2 sum the weighted
U determine its usefulness [Hou90a]:
Layer 1 Layer 2
Fig. 1 Hopfield network
outputs from the nodes in layer 1 and compare
the total input to a threshold. If the input exceeds
the threshold then the output of the node will go
high (+1) otherwise the output will go low (-l),
assuming the use of bipolar signalling. Unlike the
majority of other ANN architectures, the
CH2868-8/90/oooO-Ol88$1.00
Hopfield network does not go through a training
phase in the accepted sense. The network is in
essence programmed with the exemplar patterns
that we wish to be stored and the mining scheme
employed is an example of the application of the
Hebbian rule [Heb48]. To operate the network,
an input (probe) is applied to the network (Xi)
such that the outputs of layer 1 (Oj) at time zero
take on the probe value and the outputs of layer 2
assume a value commensurate with the output of
layer 1 and the connection weights Wjk. The
probe is then removed and the outputs allowed to
change (iterate) until they stabilise at a particular
binary level. This is the output of the network and
would ideally be the stored pattern which most
resembled the input pattern presented (i.e.
smallest Hamming distance). The storage
prescriptions used (when using the Hebbian rule)
to set the value of the interconnections weights
along with the update equations for the nodes in
layers 1,2 maybe expressed as follows.
if j o k Wjk =Z,(Smj.S,) (la)
ifj=k w , k = O (1b)
Oj(nT) = X i ; n = O (W
Oj(nT) = 0,; n > 0 (2b)
0, = f(xj(Ojewjk) = f(x) (2c)
Where w j k is the connection between the layer 1
and layer 2, S,, ’corresponds to “Bit a” in
memory vector m, Xi is the input to the network
(probe), oj and 0, are the outputs of layer 1,2
respectively, n is the number of iterations from
initialisation, T is the time between successive
iterations and f(x) = 1; x > 0; f(x) = -1; x <= 0.
To assess the capabilities of an associative
memory such as the Hopfield network we have
defined two fundamental characteristics which
(1) Capacity, C: the maximum number of stable
binary vectors that can be stored (on average).
This assumes that a particular storage prescription
has been applied to a sufficiently large arbitrarily
chosen set of memories using a specified network
architecture. This is an extension of the
definition implied in [Hop82]].
(2) Associativity, A: the capability (on average)
of a particular network architecture to take an
Q 1990 E E E
arbitrarily chosen erroneous probe and correct the
bits in error within the probe to reproduce exactly
the nearest exemplar. This assumes that the
network was trained using a specified storage
prescription and that the Hamming distance (H)
of the probe to one stable stored memory
(exemplar) is known and also less than the
Hamming distance to any other stored memory.
We can use these definitions to compare the
performance of modified Hopfield networks but
due to space constraints this paper will only
consider the effect on the capacity C. A thorough
examination of the capabilities of an associative
memory would require the. assertion of the
networks associativity. We can however gain a
useful indication of the performance of a
structurally modified network by considering the
capacity alone, if the training strategy is
u n a1tered.
To aid the analysis of the Hopfield network we
can modify the structure shown in Fig. 1 to
include an extra layer with each node in this extra
layer corresponding to a particular stored
memory. This modified structure can be seen in
Fig. 2 and is functionally identical to the one
depicted in Fig. 1. When a probe is applied to the
network, the output of layer 2 will be the
correlation of each exemplar with the probe, with
the highest output (in terms of magnitude)
I I
Layer 1 Layer 2
Fig. 2 Modified Hopfield network
corresponding to the memory which is closest (in
the Hamming sense) to the probe. The limited
capacity of the Hebbian trained Hopfield network
is due to errors caused by the effective correlation
noise of the probe with the memories other than
the one it is closest too. To determine the
capacity (C) we must first find an expression for
189
the average correlation for an arbitrary probe
applied to an arbitrary memory vector. This is
calculated as follows:
Ac = 2.z[((N- 1) Com x).((N-l)-2.~)]/2~(3)
where x=Oto (N-1)/2 - 1
= [(N-l).((N-1) Com (N-1)/2) (4)
using Stirlings approximation
(N-l)! 2 sqr(2. .(N-l)).e-(N-l).(N-l)(N-l)
=> Ac %sqr(2.(N-l)/~) ( 5 )
Where Ac is the average correlation, N Com x is
the number of ways of choosing x bits from a
vector of N bits. The memory capacity can now
be directly calculated. An error can only occur
when the summation of the average correlation of
M-1 memories can equal or exceed the
correlation of the probe with the required
exemplar thus:
(N- 1) = (M- 1).Ac (6)
=> M = int(N-1)/Ac + 1) + 1 (7)
Where M represents the minimum number of
memories that will cause an error and the
function int returns the integer part of the
parameter. Since C is defined as the maximum
number of memories that can occur before an
error occurs then:
C=M-1 (8)
This shows that as the size of the network
increases, the increase in available capacity
follows a square root characteristic.
Equation 7,8 assumes that the probability of the
M-1 memories aligning themselves is greater than
unity. i.e.:
N.M/(2(M-l))>= 1 (9)
If N is greater than 42 however then the
probability of the M-1 memories aligning
themselves is less than unity and the capacity C
will be higher than the figured calculated from
(8). The revised figure for C is C' and is
calculated thus:
M' = M + 2.r (10)
C'=M'- 1 (11)
where M', C' are the revised values of M and C
respectively. The increase in capacity is
incremented in steps of 2 to allow the
cancellation of an aligned non-stored memory to
one which is non-aligned (i.e. (M-l).Ac + r.Ac -
r.Ac = (M-1).Ac >= (N-1)). The value r is defined
as, the value of r that satisfies the following
condition:
N.M'.&(M'-I) Com XI >= 2(M'-1) (12)
where the range of the index x is 0 to r. Equations
7,8,9 are the special case of 10,11,12 respectively
when r = 0 (N < 43).
Improving the capacity of the Hopfield
network
To improve the capacity we can modify the basic
structure in several ways. 'The first is to remove
the stipulation that the principle diagonal in the
nodal connection matrix should be equal to zero
and the effect of this is to remove the direct
connections between the input and layer 2 in Fig.
2. The capacity is increased [Oh881 and this
increase is formalised below :
for an error to occur
(N-1) + M = (M-l).Ac (13)
as M must be an integer
M = int(((N-l)+Ac)/(Ac-1)) + 1 (14)
as before
M'=M+2.r, C'=M'- 1
where r is derived from condition 12. This gives
an increase in capacity of approximately Ac/(Ac-
1>.
One feature of the Hopfield network is the
automatic storage of each exemplars complement.
Of course, the effective storage of the
complement is generally unwanted and so it
would be useful to introduce a modification
which prevented its storage and this will be the
second modification to the original network. A
simple method for achieving this is to add a non-
linear transfer function to the nodes in layer 2 of
Fig. 2 which will allow positive inputs through
unchanged but prevents negative excursions (i.e.
f(x) = x; x > 0; f(x) = 0; x C= 0; i.e. the response
of an "ideal diode"). This of course does limit
the error correcting capability of the network to
Layer 1
Fig. 3 Modified Hopfield network using subtracter
N/2 away. In practice however, if a random
probe is applied to a memory set then the
correlation of the probe to half of the memories
will be positive (i.e. less than a Hamming
distance of N/2) and equal to an average of Ac
and so the modification should not cause a
problem in terms of the networks error correcting
capability. As on average one half of the outputs
of layer 1 will be negative, the capacity is
approximately doubled as shown below:
for an error to occur
(N-1) + 1 = (A~-l).(M-l)/2 (15)
as M must be an integer
M = int((1 + 2.N/(Ac-1)) + 1 (16)
as before M' = M + 2.r, C' = M' - 1
Where the value r is defined as, the value of r that
satisfies the following condition:
N.M'.xx((M'-l)/2 Com x) >= 2"M'-*)D) (17)
Where x = 0 to r. With this modification, the
output of layer 1 will be limited to positive
outputs only. We can increase the relative
differmce in the amplitude of the node in layer 2
which represents the required exemplar and the
amplitude of the other nodes. This can be done
by subtracting from each output of layer 1 the
avcrage of the remaining outputs of layer 1 (by
using the average of the outputs, the amount
subtracted is never greater than any individual
output). This new structure (which is the third
modification) can be seen in Fig. 3 and the
connections between layer 1 and layer 2 can be
described thus:
Ok = f(oj- b.zOk(0,)) = f(x) (1 8)
where f(x) = x; x > 0; f(x) = 0; x<= 0 and b = I/
(M-1). We can reduce the connections between
layer1 and layer2 by applying the following
simple modification shown below [Hou90b]:
Layer 2 Layer 3
190
This modification is shown in Fig. 4 and it should
be noted that the circuit shown is functionally
identical to the structure shown in Fig. 3. The
capacity for the subnactor circuit is derived
below:
output of the node in layer 1 which represents the
exemplar Oj = (N-1) + 1 (20)
output of each node in layer 1 which does not
represent the exemplar Oj = (Ac-1) (21)
for an error to occur (using equation 19,20,21)
3.N/2 - (Ac-1) - (Ac-l).(M-l)/d= 9 (22)
=> M = int(3.(2.N/(Ac-i)-l)) (23)
:is before M‘ = M + 2.r, C’ = M‘ - 1
Layer 2 Layer 3
Fig 4 Improved subtracter circuit
where r is defined as the value that condition 20.
Although the modifications suggested in this
paper offer a significant improvement in capacity,
the modified structures wii! not give an
associativity of 100% which is independent of the
Hamming distance of the Probe from its
corresponding exemplar. We can of course add
further subtracter circuits and each additional
circuit will give an improved performance at the
expense of extra circuitry.
The Hamming network
As each subtracter circuit is identical, we can
simulate each additional circuit by feeding the
output uf layer 2 to the input of layer 1 (and
removing the input to the network after the nodes
in layer 1 have been prjmed). This also removes
the necessity for the connections between layer 1
and layer 3 (Fig. 2). Layer 1,2 can then be
allowed to “iterate” until only one output remains
positive. This structure will then exhibit a
capacity and associativity that is equal to the
theoretical maximum that can be obtained and is
equivalent to the Hamming network (modified to
reduce the iteration layer connections)
1.91
Conclusions
In this paper we have introduced a technique to
enable the calculation of the capacity of the
Hopfield network and its enhanced derivatives.
We have suggested a logical sequence of
improvements to the original network and
presented appropriate formulae for the capacity of
each structure. These improvements (although
slightly conmved) lead us to a structure whose
characteristics in terms of both capacity and
associativity is equal to the theoretical maximum.
This structure is the Hamming network (modified
to reduce the iteration layer connections).
Although the sequence of improvements
suggested do not in any way constitute a proof,
they do suggest that the Hamming network is the
logical conclusion to the optimisation (in
performance) of the original Hopfield network.
References
[Heb49] Hebb, D., “Organization of behaviour”,
New York science editions, 1949.
[Hop821 Hopfield J.J., “Neural networks and
physical system with emergent collective
computational abilities”, Roc. Natl. Acad. Sci.
79, pp 2554-2558, 1982.
[Hou90a] Houselander, P.., Taylor, J.T.
“Measuring the error correcting capability of the
Hopfield network”, IEE Elec. Letts., Vol 26, No
2, 1990.
[Hou90b] Houselander, P., Taylor, J.T.,
“Improving the efficiency of the Hamming
associative memory”, under preparation.
[Lip871 Lippmann R., “An introduction to
computing with Neural Nets”, IEEE ASSP
Magazine, April 1987.
[Oh881 Oh S., Yoon T., Kim J.C., “Associative-
memory model based on neural networks:
modification of Hopfield model”, Optics letters,
V0123, NO 1, pp 74-76,1988.
[Was891 Wasserman P., “Neural computing
theory and practice”,Von Nostrand Reinhold,
1989.