See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/2976332

A Modular T-Mode Design Approach for Analog Neural Network Hardware

Implementations

Article  in  IEEE Journal of Solid-State Circuits · June 1992

DOI: 10.1109/4.133157 · Source: IEEE Xplore

CITATIONS READS
33 67

4 authors, including:

Bernabé Linares-Barranco E. Sanchez-Sinencio

Spanish National Research Council Texas A&M University
242 PUBLICATIONS   6,629 CITATIONS    484 PUBLICATIONS   12,713 CITATIONS   

SEE PROFILE SEE PROFILE

Jose L. Huertas
Spanish National Research Council
346 PUBLICATIONS   4,717 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Understanding the effects on ionizing radiation on deep submicron electronic devices View project

EU H2020 NEURONN: Two-dimensional oscillatory neural networks for energy efficient neuromorphic computing View project

All content following this page was uploaded by E. Sanchez-Sinencio on 04 September 2013.

The user has requested enhancement of the downloaded file.

 IEEE JOURNAL OF SOLID-STATE CIRCUITS. VOL. 27, NO. 5, MAY 1992
701

A Modular T-Mode Design Approach for Analog

Neural Network Hardware Implementations
Bernab6 Linares-Barranco, Edgar S6nchez-Sinencio, Senior Member, IEEE,
Angel Rodriguez-Vtizquez, Member, IEEE, and
JOS6 L. Huertas, Member, IEEE

Abstract–A modular transconductance-mode (T-mode) de- terns the outputs always have a digital nature, being there-
sign approach is presented for analog hardware implementa-
fore more immune to imprecise components.
tions of neural networks. This design approach is used to build
Previous neural network analog VLSI implementations
a modular bidirectional associative memory (BAM) network.
We will show that the size of the whole system can be increased have been specific for particular neural network algo-
by interconnecting more modular chips together. Also, we will rithms. However, if there were a modular hardware im-
show that by changing the interconnection strategy different plementation able to be reconfigured to realize different
neural network systems can be implemented, such as a Hopfield neural network systems, it could be integrated with a con-
network, a winner-take-all network, a simplified ART1 net-
ventional digital control system and generate very low-
work, or a constrained optimization network. Experimentally
measured results from CMOS 2-pm double-metal, double- cost, very high-efficient, and versatile real-time neural
polysilicon prototypes (MOSIS) are presented. processors. The work we present in this paper belongs in
this category [20] -[23], and we explore this in connection
to the use of transconductance-mode (T-mode) analog cir-
I. INTRODUCTION
cuit techniques which have been demonstrated to be very
ANY neural network algorithms have been pro-
M posed and studied in the computer science related
well suited for high-speed
plication contexts [24].
analog processing in other ap-

literature [ 1]-[ 14]. Most of these algorithms have been We will present a very simple yet powerful fully ana-
implemented in a software environment. However, it is log, continuous-time, T-mode circuit design technique
obvious that for many applications where real-time pro- that can be used to implement most of the neural network
cessing is necessary and/or the size of the complete com- systems proposed so far in the literature. This implemen-
puting system needs to be reduced, some type of special- tation technique is modular in the sense that the size of
purpose hardware implementation needs to be devised. In the system can be increased by interconnecting more chips
particular, analog circuits’ capability for intrinsic high- together. No special interface or interchip communication
speed operation with moderate area and power consump- hardware is needed for this. The convergence time of the
tion [15] makes these techniques worthy to be explored in system is not degraded when increasing its size. In the
connection to neural networks. test prototypes we will present in this paper we use three
In general, hardware circuit implementations of neural different types of modular chips—one for the synapses,
network systems can be made with low-precision com- one for the neurons, and one for the external inputs. We
ponents. This property is enhanced in neural systems that did this mainly for test purposes, but these chips cart be
include adaptive learning or self-organization [16], be- reduced to just one single modular chip. Also, the speed
cause as the system learns to perform a certain function it of the system can be drastically increased by sacrificing
implicitly compensates for imperfections and nonideali- the modular property and integrating the complete system
ties present in the physical components of which the whole in one single nonmodular chip.
system is made. However, there is a category of circuits, The experimental results we will present in this paper
often referred to as being also neural networks, for which for constrained optimization circuits correspond to hard-
the precision of the components is of high importance. ware realizations built with modular components that were
These circuits are known as nonlinear programming cir- originally designed to implement a bidirectional associa-
cuits or constrained optimization circuits [ 17]- [20]. The tive memory (BAM) network. This means that we will use
outputs of these circuits have, in general, an analog na- low-precision components to assemble an optimization
ture, while for the other more conventional neural sys- circuit. Therefore, as we will see, the results generated
will be of moderate precision. However, it will serve to
Manuscrlpt received October 8, 1991; revised January 3, 1992. illustrate the underlying argument of this paper, which is
B. Linares-Barranco, A. Rodrfguez-Vfizquez, and J, L. Huertas are with the great versatility of the proposed implementation tech-
the Centro National de Microelectr6mca (CNM), 41012 Sevilla, Spain. nique.
E. Sinchez-Sinencio is with the Department of Electrical Engineering,
Texas A&M University, College Station, TX 77843.
In the next section we will present the analog T-mode
IEEE Log Number 9107226. circuit design technique to be used in our further imple-

00 18-9200/92$03 .00 @ 1992 IEEE

702 IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 27, NO. 5, MAY 1992

Fig. 1. Implementation of neuron interconnections using transconductance devices.

mentations. Then we will go directly to the experimental In (1) the output of a neuron yj = ~(.xl ) can be repre-
results and show the high potential of this technique by sented physically by a voltage signal, the synaptic con-
giving examples of a 5 x 5 BAM, a 9 x 9 BAM, a five- nection by a transconductance amplifier of input voltage
neuron Hopfield network, a five-neuron winner-take-all yj and output current ~ji Y~, and the external inputs 1, by
network, a 5 X 5 simplified ART 1 network, and a mod- current variables. The lossy term – ax, can be imple-
erate precision three-variable three-constraint quadratic mented by using a resistor of value R = 1 /tx, and the
constrained optimization network. The prototypes were operation Cii can be realized by a capacitor. All this
fabricated in a standard 2-pm double-metal, double-poly- would produce a T-mode circuit representation as is shown
silicon CMOS process (through and thanks to MOSIS). in Fig. 1, where the function ~ (. ) is implemented using a
Elsewhere [21], [25] we will demonstrate that this nonlinear voltage-to-voltage amplifier.
T-mode analog circuit design technique of neural network Assuming the network is stable and it converges to a
systems “fits like a glove” for making learning or self- stable steady state, consider for each neuron the associa-
organizing Hebbian type systems with little extra cost. tion of the linear resistor R and the nonlinear voltage am-
We will also show how to add an on-chip analog memory plifier~(. ) (see Fig. 2(a)). If .liO is the steady-state current
for each synapse. entering this association and yiO is the steady-state output
voltage, then
II. THE T-MODE NEURAL CIRCUIT DESIGN TECHNIQUE
Yio = f(RJm) @ ~,o = ;f-’(Y,o) (3)
Most of the neural network algorithms available in the
literature have a short-term memory (STM) whose contin-
which can be visualized as a nonlinear resistor with a
uous-time versionl operation can be described by the fol-
driving point characteristic g(”) such that
lowing set of nonlinear first-order differential equations:
N
J;. = g(ylo) = +.f-’(Yio) (4)
CX, = ‘~Xi + ~~1 W]~f(XJ) + l,, i=l, ”.. ,N

as is shown in Fig. 2(b).

(1) By generalizing the concept of Fig. 2 to the nonsteady -
where x, is the activity of neuron i, Wj( is the weight of state case as well, the circuit implementation of Fig. 1 is
the synaptic interconnection from neuron j to neuron i, Zi modified into the one shown in Fig. 3. Obviously the dy-
is the external input to neuron i, a and C are positive namics of the system of Fig. 3 is no longer described ex-
constants, and ~(o) is a nonlinear, monotonically increas- actly by the set of equations in (1). The new system is
ing function with maximum and minimum saturation val- now described by
N
ues.
In some cases the system of equations in (1) is gener- Cyi = ‘g(y, ) + j~l W,iyj + l,, i=l, ”””, N.

alized by enriching its dynamics as in the Cohen–Gross-

berg description [27], or by sophisticating the synaptic (5)
interconnectivity like in high-order neural networks [28], However, once the steady state is reached, both de-
or by adding constraint variables like in constrained op- scriptions are equivalent. In Appendix A we will give a
timization networks [17] -[20]. stability proof for the system of equations in (5) as a par-
The system of equations in (1) can be directly imple- ticular case of a stability proof for quadratic optimization
mented with analog hardware by using transconductance systems. In Appendix B we show that for an equivalent
amplifiers as the synaptic interconnections. A transcon- initial state both descriptions will produce the same equiv-
ductance amplifier provides an output current iO propor- alent final state.
tional to its input voltage vi: In the next section we will use the T-mode circuit de-
sign technique of Fig. 3 to build a set of modular chips
iO = grrf ‘i (2)
intended to implement an arbitrary-size continuous BAM
where g~ is the transconductance gain of the amplifier. network [5]. Afterwards we will show how to use these
modular chips to assemble other neural network systems
‘Grossberg provides a method [26] to map a discrete-time description of
such as a Hopfield network [ 1]-[4], a winner-take-all net-
a neural system into a continuous-time one, and vice-versa. Therefore, the
neural network algorithms reported with discrete-time dynamics can also work [1 1], [29], and a constrained optimization network
be represented by (1). [18] -[20].
LINARES-BARRANCO et al,: MODULAR T-MODE DESIGN FOR ANALOG NEURAL NETWORK
703

Fig. 2. T-mode simplified neuron implementation.

Ii

Y,
0

=
/m)
Yl Y2 YN c
I==
Fig. 3. T-mode simplified implementation forneural networks.

III. EXPERIMENTAL RESULTS - %D-

A set of modular chips was designed and fabricated in
a 2-pm double-metal, double-polysilicon, 10-V ( V~~ =
i-5 V, V~~ = –5 V) CMOS process (MOSIS), and used
V2, +
to assemble several neural network systems [30]. 1

The current sources for the implementation of the ex-

ternal inputs Ii of Fig. 3 were realized using the transcon-
ductance amplifiers of Fig. 4, which had a size of 20 x %

20 ~m2 each. The value of V~i~, is the same for all current Fig. 4. Circuit implementation of transconductance amphtier,
sources Ii. Depending on the sign of Ii it was either V,i =
V~~ and V2i = V~~, or Vll = V~~ and Vi2 = V~~.
The neuron is composed of a nonlinear resistor in par-
allel with a linear resistor and a capacitor. The circuit im-
plementation of the nonlinear resistor is depicted in Fig.
5. If E - s yi = E + transistors Ml and i142 are OFF and
.ll = O. The only resistor in parallel with the integrating
capacitor C of Fig. 3 is the parallel connection of all out-
put impedances of the synaptic multipliers with outputs to (a) (b)
this node. The value of this linear resistor is not critical Fig. 5. (a) Nonlinear resistor circuit implementation. (b) Transfer curve.
for proper operation, which allows us to rely on parasitic
elements for its physical implementation. If yi < E- then
hD
Ml is ON and M2 OFF, and Ji is negative and large. If y, 1 I

> E + then Ml is OFF and M2 ON, and J, is positive and

large. Therefore, the circuit of Fig. 5 with the parallel
connection of the output impedances of the synaptic mul-
tipliers with outputs to node yi has driving point charac-
teristics similar to the ones of Fig. 2(b).
For the synaptic transconductors -a very simple circuit
based on Gilbert’s multiplying cell [31] was used, as is
shown in Fig. 6. The size of the cell was 50 X 40 pmz.
The top PMOS current mirror was intentionally unbal- -L
anced for offset compensation, sacrificing linearity. All %s

synaptic multipliers share the same Vbi~~ = – 3.77-v volt- Fig. 6. Actual schematic of transconductance multipliers.
age, as well as all GNDfOP = – 1.00-V and GNDbOttO~ =
– 2 .00-V connections. Fig. 7 shows the input output ‘V. A high degree of nonlinearity can be observed, espe-
characteristics of the parallel connection of five synaptic cially around W = – 2.0 V. However, as we will see, this
multipliers loaded with a 20-kfl resistor and for W = will not affect the correct operation of the complete neural
–2.8, –2.6, –2.2, –2.0, –1.8, –1.6, –1.4, and –1.2 network systems.
704 IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 27, NO. 5, MAY 1992

W=.2.8V
W..2.6V
.— W=-2,4V

W=.2.2V
W=-2.OV

W=-1.8V

W=-1.6V
w=-] .4V
W=-1.2V

%t=9@3.GnV/div Ho~300.OmV/div

Fig. 7. Measurement of the dc transfer curves of five multipliers in parallel

for Vbrd, = –3.77 V.

+
Y2

+
Y3

+
YN

layer2
Fig, 8. T-mode circuit implementation of BAM algorithm.

A. BAikl Networks Using the circuit design technique represented in Fig.

A BAM network is a two-layer neural network in which 3, a circuit realization of a 5 X 5 BAM network would be
all neurons in one layer are connected to all neurons in as shown in Fig. 8. Three different chips, one for the syn-
the other layer, and there are no connections between neu- aptic matrix, one for the neurons, and one for the external
rons in the same layer [5]. The weight of the synapse that inputs, were fabricated in a 2-~m, double-metal, double-
goes from neuron i in layer 1 ( yi ) to neuron j in layer 2 polysilicon CMOS process (MOSIS). Up to three patterns
(Y; ) is the same that the one that goes from neuron j in (with correct retrieval) could be stored in this 5 X 5 BAM.
layer 2 ( yj ) to neuron z’in layer 1 ( yi ), and is denoted ~ji. We programmed the patterns shown in Fig. 9. The nor-
 LINARES-BARRANCO et al,: MODULAR T-MODE DESIGN FOR ANALOG NEURAL NETWORK
705

Y’1 Y’2 Y’3Y’4 y’s Y’1 Y’2 Y’3Y’4Y’5 Inputs Chip

Y’ 1 y’~ Y’3 Y’4 Y’s

m~~

‘~ ‘~ c~
yl)’2y3y4y5 YIY2Y3)’4)’5

Fig. 9. Three patterns to be stored in the 5 X 5 BAM.

. T Neurons Chip

[’ Fig. 11. Illustration of modular capability of T-mode circuit BAM imple-

mentation.

Ii’ connected, the BAM will settle to one of the stored pat-
t terns, depending on the hamming distances between the
stored patterns and the input.
TOPTrace=300.OMV/div Bottom Traee=4.OQVldiv Tbnebase*2Wdiv
Exploiting the modular capability of the T-mode ap-
Fig. 10. Convergence to pattern A when the input is pattern A in 5 x 5
BAM. Top traces are neuron outputs; bottom trace is initial conditions trig-
proach, we assembled several of the chips in Fig. 8, as is
gering signaL shown in Fig. 11, and built a 9 x 9 BANI.3 The patterns
shown in Fig. 12 were loaded with the normalized syn-
realized synaptic matrix for these patterns is aptic matrix
.
–3 –1 –1 –1 3 –1 1 –1 –1
r –3 1 1 1 17
1 3 –1 –1–1–1133

I
1–3 1 1 1
–1 1–311 1–111
–31111 (6)
1 –1 -1 3 –1 3 –3 –1 –1
11 1–31
11 1 –3 1–3 3 1 1 (8)
1 1 –3 1 –3
1–1–131 3 –3 –1 –1
and to program these weights the following voltages were
1 –1 –1 3 –1 3 –3 –1 –3
used (see Figs, 6 and 7):
1 3–1–1–1–1131
w = –2.2 v, forwti= +1
–1–3111 1 –1 –3 –3
w= –1.2V, for WO = –3. (7)
using the following weight voltages (see Figs, 6 and 7):
The nonlinear resistors are biased with E+ = –0.5 V,
E- = — 1.5 V, and the input current sources with Vtri~S= W = –2.8 V, for Wti = +3
– 2.50 V. Fig. 10 shows the convergence to pattern A
w = –2.2 v, forwti= +1
when the input pattern is A. For this some switches were
added to the neurons in order to set the initial conditions W= –1.8V, forwti= –1
and visualize the transient response. The BAM network
also converged correctly to patterns B and C when the w= –1.2V, for Wti = –3. (9)
input patterns were B and C, respectively. In a continu-
This larger network is more sensitive to systematic offset
ous-time BAM the way to verify what minimum of the
in the synaptic multipliers. The value of Vhi,S in Fig. 6 for
energy surface has been reached is by disconnecting the
all the synaptic chips needed to be readjusted in order to
external inputs after the steady state has been reached. We
minimize this offset and obtain a correct retrieval of the
did this for the case of Fig. 10 (as well as when the inputs
stored patterns of Fig. 12.4 Fig. 13 shows how the neu-
where patterns B ~nd C) and the system kept the same
final state. When the external inputs are not patterns A, 3The voltage of one of the ten neurons in each layer in Fig. 11 was
B, or C the BAM might reach in some cases a stable state connected to y,0 = Y;O = GND~,,P = -1.0 V and the external inputs for
these neurons were set to 1,0 = I lo = O, so that they would not have any
slightly different from the stored patterns.2 However, once
effect on the rest of the network.
this steady state is reached and the external inputs are dis- 4Note that here we are compensating a global offset which is similar for
each synapse. Due to the nature of neural systems we do not anticipate any
2This discrepancy depends on the ratio between the values of the external misbehavior due to random offsets in the synapses, as long as the mean
current sources and the curreut levels of the synaptic multipliers. random offset remains zero.
706 [EEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 27, NO. 5, MAY 1992

Inputs Chip

B!aa!Ei
‘a ‘a CR Fig. 12, Patterns to be stored in the 9 X 9 BAM.

,8,1,
Neurons Chip

Fig. 14. Hopfield network built with T-mode BAM’s modular chip com-
ponents.

using the weight voltages (see Figs. 6 and 7)

W = –2.8 V, forwv= +1

w = –2.0 v, for Wti = O

1----’ I w= –1.2V, forwti = –1. (11)

Fig. 15 shows the stable patterns observed for each input

Top Trace=500.OmV/div Bottom Trace=5.CKIV/div Thnebase.5@3n.#div
pattern configuration. The stable pattern is always either
Fig. 13. Convergence to pattern A with input pattern A in 9 x 9 BAM. “10101” or “01010” depending on the hamming dis-
Top traces are layer 1 neuron outputs; bottom trace is initial conditions
triggering signal.
tance of the input patterns to the stored pattern.
A winner-take-all network can be considered as a spe-
cial case of a Hopfield network. It is a fully intercon-
rons of one layer converge to pattern A when the input is nected network in which all self-connections are excita-
pattern A. Note that the convergence time of this network tory and all interconnections between different neurons are
is of the same order as the one shown in Fig.- 10, being a inhibitory. However, now excitato~ connections have to
new system four times larger. This is because the settling remain excitatory always, and so do inhibitory connec-
time of this T-mode approach is independent of the size tions. This means that the synaptic multipliers can no
of the system. However, the time response does depend longer be four-quadrant ones; here they have to be two-
on the g~’s of the synapses, which depend on Vbi,,. quadrants multipliers. This is accomplished by making
GND,OP in Fig. 6 have the same value as E- = – 1.5 V
B. Hop~eld and Winner-Take-All Networks in Fig. 5. The values for the synaptic matrix could be
In order to demonstrate the versatility of the proposed
T-mode technique we also assembled some other neural 0.5 –1 –1 –1 –1
network algorithms. For example, a Hopfield network is
–1 0.5 –1 –1 –1
a fully interconnected neural network without self-con-
nections [1]-[4]. Interconnecting the modular chips of –1 –1 0.5 –1 –1 (12)
Fig. 8, as shown in Fig. 14, a five-neuron Hopfield net-
–1 –1 –1 0.5 –1
work can be obtained. Hopfield networks have a very poor
pattern capacity ( = 0.15 x N for more than one pattern, –1 –1 –1 –1 –0.5
where N is the number of neurons [1]). Therefore, for five
using the following weight voltages (see Figs. 6 and 7):
neurons we can only successfully store just one pattern.
Note that now each interconnection between neurons is w = –2.’4 v, forwti = +1/2
made of two transconductance multipliers in parallel. In
w= –1.2V, forwti = –1. (13)
order to store the pattern ‘‘ 10101, ” the following nor-
malized matrix needs to be programmed: Fig. 16 shows the transient response for neurons 1 and 4
when the input to the circuit is the pattern “10010.” As
o –1 1 –1 1 can be seen, only one of the two neurons with high inputs
is declared the winner.
–1 o –1 1 –1

1 –1 0 –1 1 (lo) C. Simpli$ed ART1 Network

–1 1 –1 o –1 A very simplified view of Carpenter and Grossberg’s

ART1 network is as a BAM network [8] in which the ex-
[ 1 –1 1 –1 o1 ternal inputs to one of the Iayep have been substituted by
LINARES-BARRANCO et al.: MODULAR T-MODE DESIGN FOR ANALOG NEURAL NETWORK 707

Input kible Pauem

11111
(o) WC& 10101 (21)
(1) 10101 (21)
(2) 00010 01010 (lo)
(3) 00011 10101 (21)
(4) 00100 10101 (21)
(5) 00101 10101 (21)
(6) 00110 10101 (21)
(7) 00111 10101 (21)
(8) O1ooo 01010 (10)
(9) 01001 10101 (21)
(lo) 01010 01010 (lo)
(11) 01011 01010 (lo)
(12) 01100 10101 (21)
(13) 01101 10101 (21) Fig. 17. Topology forsimplified ARTl network using the BAM modular
(14) 01110 01010 (lo) chips.
(15) 01111 10101 (21)
(16) 100CKl 10101 (21)
(17) 1000I 10101 (21)
(18) 10010 10101 (21)
(19) 10011 10101 (21)
(20) 10100 10101 (21)
(21) 10101 10101 (21)
(22) 10110 10101 (21)
(23)
(24)
(25)
(26)
10111
11(X)O
llml
11010
10101
10101
10101
01010
(21)
(21)
(21)
(lo)
B: m
X] X2X3X4X5
(27) 11011 10101 (21)
(28) Ill(Xl 10101 (21)
(29) 11101 10101 (21) c: KIB
(30) 11110 10101 (21)
(31) 11111 10101 (21) X,X2X3X4X5

D: ~
Fig. 5. Measured stable states for Hopfield circuit.
X,X2X3X4X5

E: ~

‘---”-”T77
Fig. 18. Five patterns stored in the simplified ART 1 network.

The winner-take-all section was biased as indicated in (12)

and (13). The output pattern configuration is represented
in Fig. 19, where we can see that the network converges
to the stored pattern (or patterns) with the minimum ham-
I + I
ming distance to the input pattern.
l--r---+---”--+ Note that the ART1 network achieves a maximum stor-

L’.— I J
age capacity of five patterns,
network the capacity was three, and for the Hopfield net-
while for the 5 x 5 BAM

work the capacity was one, all of them having 25 pro-

TopTrace=31XLOmV/div BottomTrace=300mV/div Timebase=100#4iv grammable synapses.

Fig. 16. Winner-take-all T-mode circuit with input (10010): the two traces
correspond to neurons 1 and 4. The integrating capacitance of each neuron
is 10 nF. Biases are: ‘bias = ‘3.77 v> GNDb.tt.m = –2.00 V, GND,oP = D. Quadratic Constrained Optimization Network
E= -1.50 V,and E+ = –0.5V.
The general problem of constrained optimization can be
formulated as minimizing a given cost function of vari-
a winner-take-all interconnection matrix. This is illus- ables Vl, v2, o . “ , Vq,
trated in Fig. 17. The five patterns shown in Fig. 18 were
programmed with the normalized’ synaptic matrix 4’(Z4,q, “ “ “ , Vq). (16)

1 –1 1 –1 1 subject top constraints,

1
1 –1 –1 1 1

1 –1 1 –1 –1 (14)

1 –1 –1 1 –1
. . .
[ 1 –1 1 –1 –1
.&(vl> Vz> “ “ “ , Vq) = o
(17)
using the following weight voltages (see Figs. 6 and 7):
where q and p are two independent positive integers.
W = –2.8 V, forwv = +1
Mathematically, the problem can be solved using the
w= –1.2V, forwti = –1. (15) Lagrange multiplier method [32] by defining the La-
708 IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 27, NO. 5, MAY 1992

SwbJe Pattern g(kl)

—
f,(v)
Layer 1
W

Wlolo
Ooo1o
Layer 2
O1ooo
01OOO
01100
Oallo
B
B
BC
D
x:

!&3g( L2)
.
t

Om&l Olmo B —
00100 c + f2(v)
Ooo1o D
A t
;%!! BDE ‘?i----l
Ocolo D
Ooo1o D
Imoo A . ...
10010 AD
10100 AC
Irmo AD
A
i%% B
mlcm c
01010 BD
10100 AC
Woo
CX3100 : (a)
10100 AC
10100 AC
Ocml E
ml +Kkj)=f j(v)
Ooo1o E
A
k% E
10100 AC
11111 10ooo A
11111 1000O A

Fig. 19. Stable patterns obtained for the simplified ART 1 network, I
(b)
Fig. 20. (a) General constrained optimization circuit. (b) Nonlinear resis-
tor curve.
grange function

P
L(v17 V2, “ - “ , Vq, ~1, h2j . “ . , hp) = @ + j~, hj$ and

(18)

‘[::1[:”1
h
where Xj are called the Lagrange multipliers. The solution
...
is obtained by solving
[ &

[1
E,
— . . . ~ f). (21)
J(7) ~ 0, Aj ~ 0, Aj./j(@) = O (19) EP

where the unknowns are v,k and hj. The circuit of Fig. 20 A T-mode circuit that implements (19) when *(@) and
[17], [18] solves this system of equations, assuming it ~(@) are defined by (20) is shown in Fig, 21.
converges to a stable steady state. In Appendix A we show The linear programming circuit [4] is a particular case
that this circuit is completely stable under certain condi- of the quadratic programming circuit for which Gti = O.
tions. The solution of a general optimization circuit has an
The quadratic constrained optimization problem is a analog nature. It does not saturate to a maximum or min-
particular case of the general problem described by (16) imum value as happens in the BAM, Hopfield, winner-
and (17), such that take-all, and ART1 networks. This fact implies that higher
precision components need to be used. However, for il-

H
VI lustration purposes we will use the same modular BAM
chips that we have used so far. This means that precision
@(a) =[A1””” Aq] ““” + ;[V* “ “ “ Vq]
in the solution will be sacrificed to a certain degree.
Let us implement the following three-variable, three-
constraint quadratic optimization problem. Minimize

1
G,,.”. GIq V1 @ = 2V1V3 – 2V2V3 + v! (22)
. . . . . . . . . . . . (20)
subject to the constraints
Gql - “ “ Gqq Vq
1[ V1 2 0, V2 5 ;, V3 > 0. (23)
LINARES-BARRANCO et a[, : MODULAR T-MODE DESIGN FOR ANALOG NEURAL NETWORK
709

i,

z
i,
+1

+
VI

+
V2

+
‘q

Fig. 21. T-mode implementation of constrained quadratic optimization problem

The exact solution to this problem is and Xl, h2, h3 have a solution within this linear range. In
the steady state the circuit satisfies
‘VI = o, V* = ;, V3=;. (24)

The corresponding normalized matrices and vectors of

ZZ~+;G5~+;G~7~+B~X~= o
(20) and (21) are
B;T> ~T. (28)

r
[1
002 100] However, if in the steady state any of vi or Xi is beyond
the linear range of the multipliers the solution is not valid

[1
G= O 0 –2 ,B= o–lo,
and the problem needs to be resealed. This can be done
2 –2 2 o 01

o 0
2=0, s= 0.5. (25) and in order to keep the problem unchanged (see (28)) we
also need to define a new ~‘ such that
Ho [1 0
In the actual circuit we will have E! = ~Ei . (30)

@ = ;G~~v; + G13Z4V3 For our case, a factor ~ = 1/4 produced values of v; and
A; that were within the linear range A 500 mV of the mul-
f = Bllvl >0 tipliers.
The circuit configuration assembled with the modular
h=BZZVZ–E2Z0
BAM chips is depicted in Fig. 22. The interchip buffers
~ = B33V3 >0 (26) were used to eliminate the bidirectional nature of the syn-
aptic multipliers. The nonlinear resistors were biased us-
with
ing E+ = O V and E- = –5 V, so that they would im-
G33 = 2g0, Gil = 2g0, G23 = ‘2g0 plement the characteristics of an ideal diode. The
measured steady-state response of this circuit was
l?ll = gO, BZ2 = –gO, B33 = gO
vi = 90 mV, Z& = 180 mV, v: = 125 mV
Et 1 go
—.—
*E2=— — (27) ~j = –3OO mV,
Bxz 2 2 X; = –250 mV, hi = 20 mV (31)

where gO is a scaling transconductance. For ~bi~~ = –3.77

which corresponds to the normalized problem solution
V, and if W = –1.2, –2.8 V, according to Fig. 7, it
would be gO = 30 pmhos. Since each multiplier in Fig.
VI = 0.36, V2 = 0.72, V~ = 0.50
21 has a linear range of approximately +500 mV (see
Fig. 7) we have to make sure that the variables VI, V2, V3 Al = 1.20, X2 = 1.00, h3 = 0.08 (32)
710 IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 27, NO. 5, MAY 1992

● At least one (and maybe more) solution to the prob-

lem exists. Consequently, the cost function is
bounded from below within the region over which
the constraints are satisfied.
● The functions @(” ) and ~( c) are continuous, and all
their first and second derivatives exist and are con-
tinuous.

F’J-oo~ The equations for the network are

Fig. 22. Interconnection topology for optimization circuit. ajj ~=l. ... q
Cvi’–g– h- >
1 j=l ‘a?)j’
while the exact solution should have been
~j = g(.fj(z)), j=l, ”””, p. (34)
v, = 0.00, v~ = 0.50, V3 = 0.50
Since g(.), @(” ), and J (” ) are continuous, (34) can be
A, = 1.00, X2 = 1.00, As = 0.00. (33) written as

The discrepancy is due to the low-precision components i(t) = Z(@(t)) (35)

used. When using high-precision analog circuit design
techniques, like switched-capacitor circuits, very precise where ~(”) is a continuous function from (R~ to (R9. Con-
CMOS optimization networks can be achieved [20]. sider the scalar function E(Z): (R9 + (R,

IV. CONCLUSIONS
We have proposed, developed, and demonstrated a
compact, modular, versatile, cheap, and powerful circuit
design technique for the implementation of continuous- taking time derivatives yields
time analog neural networks. This technique is based on
the use of transconductance synaptic multipliers and
neural nonlinear resistors. We have used this approach to
design a set of modular chips intended for the implemen-
tation of arbitra~-size BAM networks.
These BAM’s were successfully tested. Afterwards, we (37)

used the same modular chips to assemble other neural net-

work algorithms, such as a five-neuron Hopfield network, Therefore, dE/dt s O. This implies that E(t) is strictly
a five-neuron winner-take-all network, a 5 x 5 ART 1 net- decreasing unless tii = O for all i = 1, “ . “ , q, which
work, and a moderate precision three-variable, three-con- corresponds to the steady state. This means that E(Z) is
straint optimization network. a Lyapunov jivz.cti<n of the system, which together with
This circuit design technique has been extended by in- the continuity of h (”) ensures that the system is com-
cluding an on-chip Hebbian learning rule for each synapse pletely stable, i.e., any trajectory d ( o) eventually con-
[21], [30] as well as an on-chip analog dynamic memory verges to some equilibrium point @ * in (R9 depending on
for the weight storage of each synapse. The correspond- the initial state 70 [19].
ing results will be published elsewhere [25]. Corollary: The fully interconnected T-mode neural
In the experimental results given in this paper, abso- network (see Fig. 3) is a particular case of the T-mode
lutely no care was taken to minimize the response times. constrained quadratic optimization network, and is there-
However, since this is an analog approach, such response fore (via the previous theorem) also completely stable.
times are given by the time constants of the g~’s and ca- The circuit of Fig. 3 can be viewed as the following
pacitors involved, and can be minimized to those values constrained quadratic optimization network:
characteristic of other analog circuits.
In summary, this paper demonstrates the high potential

H
Y1
and versatility of the proposed T-mode circuit design
technique for the analog hardware implementation of Y(i$)=[zl”””zq] ““” + ;[y, “ “ “ yq]
neural networks on standard low-cost CMOS processes.
l_.yqJ
APPENDIX A

Theorem: The circuit of Fig. 20 is completely stable in

the sense that it will never oscillate or display other exotic
modes of operations [19], assuming the following condi-
tions are satisfied:
LINARES-BARRANCO et al.; MODULAR T-MODE DESIGN FOR ANALOG NEURAL NETWORK 711

Fig. 23. T-mode neural network as a particular case of constrained optimization circuit.

subject to the constraints therefore the circuit of Fig. 23 is equivalent to the one in
Fig. 3.
.fl: Y1 ~ –E”

A: Y1 = +E+ APPENDIX B
Theorem: Given the same equivalent initial conditions
A: Y2 ~ –E-
for the neural network described by (1) and the neural
f4: Y2 ~ +-E+ network described by (5), they will arrive at the same
equivalent final state if the weight matrix of the fully in-
. . .
terconnected network is invertible.
Prooj This will happen if there exists a set of func-
L,-l: Yq ~ –E-
tions
f2q: Y, = +~+. (39) i,j=l, . . ..N
hu(x,, “ “ “ ,xN), (41)
These constraints equations in matrix form are
such that the following perturbative approximation can be
1 0“””0 o made:
N
1 0“””0 o
Yi = f(%) + j~, ~jhij(y)
.fl YI
0 1“””0 o
$2 Y2
0 1“””0 o
... . . . (42)
. . . . . . . .

[ f% :1 Yq
0 0“””0 1 and hij ( 2 ) does not diverge once the steady state is
reached. If (42) are satisfied and if there is a set of tra-
0 0“””0 1
jectories xi(t) that solves the description of (1) for a given
–E-
initial condition, then there is also a set of trajectories

1
yi (t) that solves the description of (5) for the same equiv-
+E+ alent initial condition, and both sets of trajectories arrive
— (40) to the same equivalent steady state. If we can show that
. . .
the functions hti (” ) do exist then the theorem is proved.
+E+ Taking the time derivative in (42) and neglecting the
[
terms in iji, results in
\
The circuit of Fig. 21 for this particular case is shown N
in Fig. 23. Note that the circuits comprised by broken (43)
Yi = .f’(xi)~i + j~, YjhU(Z).
lines behave like the nonlinear resistors of Fig. 5, and
112 IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 21, NO. 5, MAY 1992

gramming circuit, ” IEEE Trans. Circuits Syst., vol. CAS-33, pp.

Also, by (42)
533-541, May 1986.
[5] B. Kosko, “Adaptive bidirectional associative memories, ” APPL
Opt., vol. 26, pp. 4947-4960, Dec. 1987.
,f-’(yj) = x, + P(~i)j$, ~jhj(2)
[6] B. Kosko, “Bidirectional associative memories, ” IEEE Trans. SY~f.,
Man, Cybern., vol. 18, pp. 49-60, Jan./Feb. 1988.
P(%) =f’-’(fw)))). (44) [7] B. Kosko, Neural Networks and Fuzzy Systems. Englewood Cliffs,
NJ: Prentice-Hall, 1992.
Substituting the time derivative of (1) in (43), this result [8] G. A. Carpenter and S. Grossberg, “A massively parallel architecture
for a self-organizing neural patten recognition machine, ” Comput.
together w;th (44) in (5) yields
Vision, Graphics, Image Processing, vol. 37, pp. 541115, 1987.
N [9] T. Kohonen, “The ‘neural’ phonetic typewriter, ” Computer, vol. 21,
pp. 11-22, Mar. 1988.
C[~’(X;) – 1].ii + ~[~(XJ – 1] ZI .ij/ZJ (Y)
[10] T. Kohonen, Selj-Organization and Associative Memory, 2nd ed.
New York: Springer Verlag, 1988.
NN [11] R. P. Lippmann, “An introduction to computing with neural nets, ”
IEEE ASSP Msg., Apr. 1987.
[12] J. A. Anderson, “A simple neural network generating interactive
memory, “ Math. Biosci., vol. 14, pp. 197-220, 1972.
NN
[13] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning rep-
(45) resentations by back-propagating errors, ” Nature, vol. 323, pp. 533–
536, 1986.
[14] G. E. Hinton and R. J. Sejnovski, ‘ ‘Learning and relearning in Boltz-
which, in matrix form, can be expressed as mann machines, ‘‘ in Parallel Distribution Processing, vol. 1, D. E.
Rumelhart and J. L. McClelland, Eds. Cambridge, MA: M. IT.
C[f ’ – I]D+T + O![p – l]~H*T Press, 1986, ch. 7.
[15] E. A. Vittoz, “Future of analog in the VLSI environment, ” in Proc.
+ HW[.f’]D~~+ WH$T= O (46) IEEE Int. Symp. Circuits Syst. (L$CAS90) (New Orleans), 1990, pp.
1372-1375.
where His the matrix of elements hij ( Z ), W is the one of [16] C. Mead, ‘ ‘Neuromorphic electronic systems, ” Proc. IEEE, vol. 78,
no. 10, pp. 1629-1636, Oct. 1990.
elements Wti, and [~’ – l]D, [p – l]D, and [~’]D are [17] L. O. Chua and G. N. Liu, “Nonlinear programming without com-
diagonal matrices of diagonal elements ~ ‘(xi ) – 1, P (xi) pupation, ” IEEE Trans. Circuits Syst., vol. CAS-31, pp. 182-188,
- 1, and f ‘(xi), respectively. Since (46) has to hold for Feb. 1984.
[18] M. P. Kennedy and L. 0. Chua, “Unifying the tank and Hopfield
any ~‘, it must be
linear programming circuit of Chua and Lin, ” [EEE Trans. Circuits
Syst., vol. CAS-34, pp. 210-214, Feb. 1987.
C[f ‘ – l]D +.a[p – l]DH + HW[f ‘]D + WH = O. [19] M. P. Kennedy and L. O. Chua, “Neural networks for nonlinear pro-
cramming, ” IEEE Trans. Circuits Syst., vol. 35, pp. 554-562, May
(47) 1988.
[20] A. Rodrfguez-V4zquez, R. Domitrguez-Castro, A. Rueda, J. L.
The solution of this matrix equation provides the func-
Huertas, and E. Sanchez-Sinencio, “Nonlinear switched-capacitor
tions /tti, and therefore, the result of (42). If this solution ‘neural’ networks for optimization problems, ” IEEE Trans. Circuifs
exists and does not diverge, the theorem is proved. Sysf., vol. 37, pp. 384-398, Mar. 1990.
[21] B. Linares-Barranco, E. Sanchez-S inencio, A. Rodrfguez-Vtizquez,
In the steady state f‘ (xi) = O and p (xi) – 1 = O.
and J. L. Huertas, “VLSI implementation of a transconductance mode
Therefore, in the steady state we have continuous BAM with on chip learning and dynamic analog mem-
ory, ” in Proc. IEEE Int. Symp. Circuits Syst (ISCAS91) (Singapore),
H = –CW-l[l]D. (48) 1991, pp. 1283-1286.
[22] M. Robinson, “Analog CMOS neural Hamming networks, ” Mas-
This solution exists and is bounded if the weight matrix ter’s thesis, Texas A&M Univ., College Station, Dec. 1991.
[23] H. Yoneda, ‘ ‘VLSI implementation of neocognitron, ” Master’s the-
is invertible. Since the final state is bounded as well as
sis, Texas A&M Univ., College Station, Dec. 1991.
the ‘initial state and both (1) and (5) are well behaved, the [24] B. Linares-Barranco, A. Rodrfguez-Vizquez, E. Sanchez-S inencio,
transient response will also be bounded. Furthermore, if and J. L. Huertas, “CMOS OTA-C high-frequency sinusoidal oscil-
lators, ” IEEE J. Solid-State Circuits, vol. 26, no. 2, pp. 160-165,
there is a solution for (1) with trajectories Xi(t), there is
Feb. 1991.
also a solution for (5) with trajectories yi (t) which can be [25] B. Linares-Barranco, E. SAnchez-Sinencio, A. Rodrfguez-V~zquez,
computed through (47) and (42), and both sets of trajec- and J. L. Huertas, “A CMOS learning BAM with on-chip analog
memory, ” to be submitted to IEEE Trans. Neural Networks.
tories have the same equivalent initial and final states de-
[26] S. Grossberg, ‘ ‘Non~inear neural networks: Principles, mechanisms,
fined by and architectures, ” Neural Networks, vol. 1, no. 1, pp. 17-61, 1988.
[27] M. A. Cohen and S. Grossberg, “Absolute stability of global pattern
Yi = f (xi),
~=1,...,~, (49) formation and parallel memory storage by competitive neural net-
works, ” IEEE Trans. Syst., Man, Cybern., vol. 13, pp. 815-826,
1983.
REFERENCES [28] C. L. Giles and T. Maxwell, ‘ ‘Learning, invariance, and general-
ization in high-order neural networks, ” Appl. Opt., vol. 26, no. 23,
[1] J. J. Hopfield, “Neural networks and physical systems with emergent Dec. 1987.
collective computational abilities, ‘‘ Proc. Nat. Acad. Sci. USA, vol. [29] C. Mead, Analog VLSI and Neural Systems. Reading, MA: Addi-
79, pp. 2554-2558, Apr. 1982. son-Wesley, 1989.
[2] J. J. Hopfield, “Neurons with graded response have collective com- [30] B. Linares-Barranco, “Analog neural network VLSI implementa-
putational properties like those of two-state neurons, ” Proc. Nat, tions, ” Ph.D. dissertation, Te~as A&M Univ., College Sta~ion, .Dec.
Acad. Sci. USA, vol. 81, pp. 3088-3092, May 1984. 1991.
[3] J. J. Hopfield and D. W. Tank, “Neural computation of decisions in [31] B. Gilbert, “A precise four-quadsant multiplier with subnanosecond
optimization problems, ” Biol. Cybern., vol. 52, pp. 141-152, 1985. response, ” IEEE J. Solid-State Circuits, vol. SC-3, no. 6, Dec. 1968.
[4] D. W. Tank and J. J. Hopfield, <‘Simple ‘neural’ optimization net- [32] D. A. Wismer and R. Chattergy, Introduction to Nonlinear Optimi-
works: An A/D converter, signal decision circuit, and a linear pro- zation. Amsterdam: Elsevier, 1978.
LINARES-BARRANCO et al,: MODULAR T-MODE DESIGN FOR ANALOG NEURAL NETWORK 713

Bernabd Linares-Barranco was born in Gran- ON NEURAL NETWORKS Special Issues (1991-1993). He was a Committee
ada, Spain, on November 26, 1962. He did his Member of the Scientific Committee of the Sixth, Eighth, and Ninth
elementary studies in Germany until 1975. Here- European Conferenceson Circuit Theory and Design (ECCTD), aCom-
ceived the B. SC. degree in electronic physics in mittee Member of the Technical Program of the IEEE International Sym-
June 1986, and the M. S. degree in microelectron- posiums on Circuits and Systems (ISCAS)in 1983, 1987, 1989, and 1990,
ics in September 1987, both from the University and Chairman of the IEEE CAS Technical Committee on Neural Systems
of Seville, Sevilla, Spain. He received his first and Applications (1990-1991). Heiscurrently amember of the IEEECAS
~h. D. degree in high-frequency OTA-C oscillator Board of Governors, Liason Representative, Region 9, IEEE/CAS, and an
design in June 1990 from the University of Se- IEEE Fellow Member.
ville, Spain, and the second Ph.D. degree in an-
alog neural network design in December 1991
from Texas A& MUniversit~, College Station. - Angel Rodriguez-Vazquez (M’80) received the
Since September 1991 he has been a Senior Researcher at the Analog
Licenciado en l%ica degree in 1977, and the Doc-
Design Department of the Microelectronics National Center (CNM), Sev-
tor en Ciencias Ffsicas degree in 1983, both from
illa, Spain. His research interests are in the area of nonlinear analog and
the University of Seville, Sevilla, Spain.
neural network microelectronic design.
Since 1978 he has been with the Departamento
de Electr6nica y Electromagnetism at the Uni-
versidad de Sevilla, where he is currently em-
ployed as an Associate Professor. His research in-
terest lies in the fields of analog/digital integrated
circuit design, analog integrated neural and non-
Edgar Sanchez-Sinencio (S’72-M’74-SM’83) linear networks, and modeling of analog inte-
received the M. S. E. E. degree from Stanford Uni- grated circuits.
versity, Stanford, CA, and the Ph. D. degree from
the University of Illinois at Champaign-Urbana in
1970 and 1973, respectively.
Currently, he is with Texas A&M University,
College Station, as a Professor. He is the co-au- JOS6 L. Huertas (M’74) received the Licenciado
thor of Switched-Capacitor Circuits (Van Nos- en Ffsica degree in 1969, and the Doctor en Cien-
trand-Reinhold, 1984), and co-editor of Artijcial cias Ffsicas degree in 1973, both from the Uni-
Neural Networks: Paradigms, Applications, and versity of Seville, Sevilla, Spain.
Hardware Implementation (IEEE Press, 1992). From 1970 to 1971 he was with the Philips In-
His irtterests are in the area of solid-state circuits, including CMOS neural ternational Institute, Eindhoven, The Nether-
network implementations, and computer-aided circuit design. lands, as a Postgraduate Student. Since 1971 he
Dr. Sanchez-Sinencio was the General Chairman of the 1983 26th Mid- has been with the Departamento de Electr6nica y
west Symposium on Circuits and Systems. He has been Associate Editor Electromagnetism at the Universidad de Sevilla,
for IEEE Circuits and Systems Magazine (1982- 1984), for IEEE Circuits where he is currently employed as a Professor. His
and Device Magazine (1985–1988), for the IEEE TRANSACTIONS ON CIR- research interest lies in the fields of multivahred
CUITS AND SYSTEMS (1985– 1987), and for the IEEE TRANSACTIONS ON logic, sequential machines, analog circuit design, and nonlinear network
NEURAL NETWORKS (1990- ). He is Guest Editor for IEEE TRANSACTIONS analysis and synthesis.

View publication stats