Neural networks

Author(s): Bikas K. Chakrabarti

Source: Current Science , 25 January 1995, Vol. 68, No. 2, a special issue: Neuroscience–a
multidisciplinary approach (25 January 1995), pp. 153-155
Published by: Current Science Association

Stable URL: https://www.jstor.org/stable/24096496

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms

is collaborating with JSTOR to digitize, preserve and extend access to Current Science

This content downloaded from

52.66.29.202 on Tue, 02 Jul 2024 17:27:46 +00:00
All use subject to https://about.jstor.org/terms
NEUROSCIENCE - A MULTIDISCIPLINARY APPROACH

Neural networks

Bikas K. Chakrabarti

Saha Institute of Nuclear Physics, 1/AF Bidhannagar, Calcutta 700 064, India

Brief review and state of the art

The attempts by the neuroscientists, computer
scientists and physicists to capture the computa
tional architecture and processes of the human It brain
is now more than a hundred years that the basic
and their implementation in silicon devices are now blocks (or atoms), called neurons, of the neural
building
giving encouraging results. The studies of model network (in the human brain) were identified and
neural networks and their properties are now at a discovered by Cajal in 1888 (see e.g. reference 1). Fifty
very exciting stage. We start with a brief intro years back (in 1943), McCulloch and Pitts2, inspired by
duction of the developments made in neural network the knowledge gathered already in neurobiology,
modelling. The current research activities in this proposed a model for the (nonlinear) dynamics of the
field in India (to the best of author's knowledge) are (state of) activity of a neuron interacting (via synaptic
then discussed and some suggestions regarding the connections Wy) with other neurons in the network.
future course of action indicated.
They introduced a simplistic binary-state (active and
passive) threshold neuron model. If the state of the ith
neuron is denoted as «,(/) at (discrete) time t (where
The overwhelming superiority of human brain over
digital computers in almost every task (e.g. faster i = 1, 2, ..., N, N being the total number of neurons in
the network) then the dynamical law for the change of
recognition of objects, faces, processing visual
state for the model neuron is
information, etc.) motivates studies on the design and
performance of artificial intelligence (AI) models of the
brain, to capture the essential processes of neural m{t+ 1) = ©(A,(0 -4), i = 1,2,..., N, (1)
computation. The prospect of implementing the basic
architecture of the human brain (consisting of about 10u where the net input on the ith neuron
neurons working in parallel and each operating in a
typical millisecond time scale) in high-speed electronic A,(t) = X, Wy rij (0 (j * ij =1,2,
computers (presently employing about 1010 constituent
transistors operating typically in nanosecond time scale,The threshold function ®(x) =1 for x > 0 and @(
with sequential operation program) is proving to be a for x < 0. fa denotes the threshold for the neuron i
new reality. What are the possible tasks an artificial that it is in an active state («/ = 1) if the net input (A
neural network can do? Expectations from such simple it exceeds <j>i and remains inactive («/ = 0) othe
networks are enormous: McCulloch and Pitts2 suggested that, with a sui
choice for the synaptic strengths Wy, such network
i) Such networks should evolve in a flexible way - to be
able to 'learn' and accommodate themselves to new carry out any imaginable computation. With this sta
the investigations of artificial neural networks3"18.
environments and need not be programmed.
Around 1960, Rosenblatt and coworkers3 studi
ii) Should be able to process informations which are
specific network architecture, called perceptrons, w
fuzzy, probabilistic, noisy or even inconsistent.
iii) Should be fault-tolerant (robust) and compact layered structure and feed-forward connections
appropriate for a particular type of computationa
(efficient).
(similar to processing of sensory information or
Neural networks have emerged as an alternative perception). A severe criticism on the limitations of the
paradigm for computation. Instead of the processing (one-layer) perceptron model, by Minsky and Papert4,
being sequential, as in the usual von Neumann archite left this model abandoned (virtually dead) for* almost
cture of computers, the instructions can be distributed twenty years. In a very significant development in the
amongst the different computing units of a neural last decade, the group led by Rumelhart et al.5 redis
network. Their potential applications lie mainly in covered independently the merits of an (optimization)
computer science and engineering, in addition to their algorithm called back propagation (first found by
use as a paradigm for modelling in neuroscience and in Werbos6 in 1974) to find the Wy's in a multilayer
sensory and cognitive psychology. They will also openperceptron. Most of the successful implementations
an avenue to new investigations, by simulating processes today of the neural network architecture for recognizing
that cannot be studied in live brain.
complicated patterns and quasichaotic time series

CURRENT SCIENCE, VOL. 68, NO. 2, 25 JANUARY 1995 153

This content downloaded from

52.66.29.202 on Tue, 02 Jul 2024 17:27:46 +00:00
All use subject to https://about.jstor.org/terms
 SPECIAL ISSUE ON

employ this back propagation algorithm and its networks. These led to quite rigorou
extensions. Some examples are: theories13'14 regarding the network (loading
(i) NETtalk (Sejnowski and Rosenberg7), having a f°r associative memory- Extensions of such n
three-layer feed-forward network (with 80 'hidden' learn,ng' stonng and recalling tempor
units), could 'learn' (like a child) to pronounce written patter"s (rePla<;ing Points by "mi
text. This, of course, is a compromise (solution) to the als° been made\ The, ablllty ,of such
much harder (reverse) problem of speech recognition. ftWOrks t0 assoc,ate 1(3caI minima w,th t
The network was trained to associate a phonetic value patterns has been u,tlhzed by
with a certain group of letters provided by an example t0u SUgfSt a neural-network
set of 1024 English words. After completion of the L0catmg the (defnerate> sol«tlons °f
'training' (typically 50 cycles or iterations), it could Salesman(typeof combinatorial optimiza
accurately pronounce 95% of the words from the the C0St funCtl0n' *be *' f
training set and about 80% of the words from 'unseen' synaptl,C connections This brilliant .dea (of
texts parallel functions of the neurons in the network to
z;;\ Tk. t e l°°k Por tbe associative solution of such NP-hard
(il) The primary structure of a protein consists of a ,, . ,
linear sequence of amino acids, chosen from 20 Priems) has, so far not led to any reasonable success
possibilities, like a text with an alphabet of 20 letters. (comPared t0 that obta!ned usin8 other non-network
The secondary structure determines the local aPPr°ac es).
,. ., j.-/ Hardware implementations of the Hopfield model m
stereochemical form of the amino acid chain (e.g. a ., , . .. , , , . , ...
, „ , „ , . , . the analog VLSI circuits have been achieved, and chips
helix, ß sheet, ß turn or random coil). In an obvious ... ... . c ,, . ,
ii 7 „ .,77. no- i ■ j o l 7 j Wlth (N =) 256 fully connected units or neurons, using
parallel to NETtalk, Sejnowski and Rosenberg' and , . „;,2. . . . < ... , ,
t, , . ,8 j i . • . . ., i.i about 130,000 (= 2N ) resistors to simulate IT/,-s, have
Bohr et al. developed independently neural networks , , i6 TL ,
.... , e . c , ... j-.j been made . These connections (Wu s) are 'jdifficult to
which learned from a set of samples and then predicted v 1
achieve in silicon technology (where units or the
the secondary structure of the central sequence. The best
neurons are simpler to construct, using nonlinear
reported accuracy is 62% for the training set and about
„0/ e t , „ r amplifiers), which is reverse in optical technology.
53% for unseen sequences. One recognizes, therefore, .. . , . ]7 . , , , . b.
,, r. , . , ° . . Mead and coworkers have developed an electronic
the ability of such networks to ext
fhp Ohl IITA/ at cnr»n npru/Arl'c ta pvtrcir-t fhp riinotmnn I 1

model of the retina

relations that have so far escaped human intelligence! . ...
..... TU .... . , , . animals: with 48 x 48 nearest-neighbour connected
(111) There are now multilayer networks (using , „ , . . ....
...... .. . c ... c , photosensors. The chip is reported to be able to perform
essentially back propagation) for navigation of a car (so , , . .... - ■
..... i j j • . .. , , , the basic tasks of image processing for discontinuity and
that the car could drive on a road through a wooded . , . . 0 v . . .. ... ..
... .... ... , . motion detection, in a manner quite similar to biological
area), signal prediction and time series (e.g. sunspot . n °
activity series) prediction with impressive reported
success (over non-network and even human attempts).

There has been another root of development of the Criticis

McCulloch and Pitts2 type of network9"18. In 1955,
Cragg and Temperley9 identified the McCulloch-Pitts There h
(two-state) neurons as (Ising) spins or magnetic and philosop
moments. Attempts were also made by Little1* and artificia
others (in 1970s) to implement Hebb's11 ideas about intellige
learning in the brain, to construct the synaptic conne- Can such
étions W0; IT,,'s take positive (excitory) and negative what i
(inhibitory) values determined by the patterns to be conscio
learned. However, the coupled dynamical equations incomple
were still not amenable to statistical physics. Hopfield in sucb p
1982 made a 'clever step backward' and assumed
Wij = Wji (which is not correct biologically) in a model Curr
network and introduced the 'energy function' or the
'hamiltonian' for the network. Memories were then Presently
identified as metastable minima (acting as local attractor science,
fixed points in the dynamics) in the'energy landscape'. of IITs, i
The energy function or the hamiltonian being very network r
similar to that of spin glasses (in statistical physics), a the comp
remarkable understanding developed and most of the IIS
recently acquired ideas about the free-energy landscape Kharagp
of spin glasses were translated to those of such for several
154 CURRENT SCIENCE, VOL. 68, NO. 2, 25 JANUARY 1995

This content downloaded from

52.66.29.202 on Tue, 02 Jul 2024 17:27:46 +00:00
All use subject to https://about.jstor.org/terms
NEUROSCIENCE - A MULTIDISCIPLINARY APPROACH

being investigated. A piece of such work investigated by

strength and also perhaps the weakness (in the absence
one of these groups (see e.g. reference 20) uses fuzzy
of regular dialogue across the disciplines)!
sets for replacing the binary-state neurons in a multi
layer perceptron network, for the classification of fuzzy
Suggestions
(linguistic) patterns. A couple of physics groups, e.g., in
IISc (Bangalore) and in SINP (Calcutta), are involved in
the (statistical) physics of such networks21'22. TheFor
DST may consider sponsoring the following:
example, an extended associative memory model(i)for Multidisciplinary interaction meetings/workshops
processing EEG oscillations was utilized to process the interested investigators, about once a year.
involving
epilepsy-generating 'kindling' in the laboratory (ii) Funds for procuring books, journals for the investi
animals21. The predicting capacity of a perceptron
gator(3groups. (In most cases the university/institute
layer; using back propagation) was investigated22 for cannot procure them, as there are no precise
libraries
quasichaotic (time series) sunspot data. Recent investi
groups of faculty engaged in such research.)
gations of extending the Hopfield-Tank type of network
(iii) Funds for research fellows/associates, where
for finding the optimal solutions of (randomly diluted)
necessary, and a minimal grant for procuring necessary
lattice version of the Travelling Salesman's
capital instruments (e.g. dedicated workstations in
problem22 have indicated encouraging physics possibilities.
groups).
Physics groups in various universities (JNU, NEHU,
etc.) are also showing keen interest in neural-network
1. Shaw, G. L. and Palm, G. (eds.), Brain Theory, Reprint Volume,
research. One of such groups (in JNU) is World studying
Scientific, Singapore, 1988.
the time-depiendent properties of neural networks
2. McCulloch, W. S. and Pitts, W., Bull. Math. Biophys., 1943, 5,
115.
with continuous spin variables. A few defence
3. Rosenblatt, F., Principles of Neurodynamics, Spartan, New
laboratories (e.g. DRDO, Hyderabad) are also doing York, 1962.
research in pattern recognition aspects of neural 4. Minsky, M. and Papert, S., Perceptrons: An Introduction to
networks. Computational Geometry, MIT Press, Cambridge, 1969.
5. Rumelhart, D. E., Hinton, G. E. and Williams, R. J., Nature,
This report on the Indian Work is certainly
1986, 323, 533.
incomplete, as much of the work being done in various
6. Werbos, P., PhD Thesis, Harvard University, 1974.
places has started recently (reports not yet published)
7. Sejnowski, T. J. and Rosenberg, C. R., Complex Systems, 1987,
and there is no regular interaction yet among the various1, 145.
groups in the community. 8. Bohr, H. et al., FEBS Lett., 1988, 241, 223.
9. Cragg, B. G. and Temperley, H. N. V., Brain, 1955, 78 II, 304.
10. Little, W. A., Math. Biosci., 1974, 19, 101.
11. Hebb, D. O., The Organization of Behavior, Wiley, New York, 1949.
Future perspective 12. Hopfield, J. J., Proc. Natl. Acad. Sei. USA, 1982, 79, 2554.
13. Amit, D. J., Gutfreund, H. and Sompolinsky, H., Ann. Phys.,
As discussed earlier, the interest in the functioning of
1987, 173, 30.
14. Gardner, E., J. Phys. A, 1988, 21, 257.
the human brain exists not only for understanding man's
15. Hopfield, J. J. and Tank, D. W., Science, 1986, 233, 625.
superb ability to survive, but also this acquired
16. Graf, H. P. et al., AIP Conf. Proc., 1987, 151, 182.
knowledge will have profound implications in17.the Mead, C., Analog VLSI and Neural Systems, Addison-Wesley,
development of (the next generation) computer. It will Reading, 1989.
also have potential applications in neurobiology (in 18.
theHerz, J., Krough, A. and Palmer, R. G., Introduction to the
sense of simulating live processes) and in medical Theory of Neural Computation, Santa Fe Institute Studies in
Sciences of Complexity, Addison-Wesley, Redwood City, 1991.
treatments.
19. Penrose, R., Wolfson Lecture - Oxford 1991, in Proc. CERN
This multidisciplinary science of neural networks isof Computing (ed. Verkerk, C.) 1991.
School
20. Pal,
presently at the threshold of development23. Its scope is S. K. and Mitra, S., IEEE Transactions on Neural
Networks, 1992, 3, 683.
thus enormous. The subject is truly multidisciplinary in
21. Dasgupta, C., Physica A, 1992, 186, 49.
nature: it involves neurobiology, computer science,
22. Chakrabarti, B. K. and Dasgupta, P., Physica A, 1992, 186, 33.
(statistical) physics, mathematics, etc. This multidisci
23. Scientific American (Special issue on Mind & Brain), September
plinary nature is inherent to the subject. Here lies its
1992.

CURRENT SCIENCE, VOL. 68, NO. 2, 25 JANUARY 1995 155

This content downloaded from

52.66.29.202 on Tue, 02 Jul 2024 17:27:46 +00:00
All use subject to https://about.jstor.org/terms