Second UKSIM European Symposium on Computer Modeling and Simulation

Glial Reservoir Computing

David Reid Liverpool Hope University reidd@hope.ac.uk Abstract
In trying to mimic biological functions of the brain, Artificial Neural Network (ANN) research has, out of computational necessity, made a number of assumptions. Firstly, it is assumed that the complexity of biological processes can be usefully replicated artificially by abstracting a relatively few key or essential characteristics from the biological system. Secondly, it is often assumed that a single entity, the neuron, is solely responsible for biological cognitive processing or computation. Thirdly, it is also often assumed that this processing is entirely dependant on microscopic factors within the neuron. Recent research using Spiking Neural Networks (SNNs) has addressed the first assumption, highlighting that emphasizing alternative biological functionality may afford massive computational gain. In an attempt to address the last two assumptions, the authors propose that the glial network may be acting as a feature extraction network in a way that is similar to the function of a reservoir computer.

Mark Barrett-Baxendale Liverpool Hope University barretm@hope.ac.uk

However, despite their practicality, Reservoir Computers currently have no biological basis. Recent research in neurobiology strongly implies that the neuron is aided in cognitive processing by glial cells. It is now well established that glial cells intercommunicate with neurons as well as with each other [4], [5], [6]. A particular type of glial cell, the astrocyte, is known to respond to neuronal activity through the elevation of intracellular Ca2+ levels. This causes the release of neurotransmitters which in turn adapt the transmission of signals across the neural synapse, hence forming a feedback mechanism (which could be positive or negative depending on the type of neurotransmitter released). The elevation of calcium levels within the astrocyte also propagates as a Ca2+ wave across the astrocyte network. It is probable that the neural and glial networks co-operate and feedback to each other in a far more sophisticated way than has previously been thought. The authors propose that both neuronal weight change (synaptic plasticity) and mesoscopic/macroscopic functionality of the brain is performed in whole or in part by the glial network. We further propose that the glial network provides a similar functionality to the preprocessing network in reservoir computers and thus forms a biological analogy for these types of system. This paper will demonstrate how a network of glial cells could be acting as a medium in which input patterns are encoded and learnt by the underlying neural network and how the neural network could be moderated by the glial network itself. The paper combines emerging thinking about the active role of glia in information processing with recent work performed in artificial neural network research (in particular, reservoir computing).

1. Introduction
A number of researchers have recently been looking at several new types of Recurrent Neural Network (RNN) that utilize their powerful temporal processing capability by using a randomly initialized, and static, network as a filter. This new type of neural network preprocesses inputs in the static network and transforms the inputs into a pattern that is more easily discriminated. Liquid State Machines, Echo State Machines and Backpropagation/Decorrelation Networks all have the above property and can be generally classed as Reservoir Computers [1], [2]. Reservoir computers have been shown to be very powerful pattern classifiers of temporal information; in speech recognition tasks for example they are superior to the current best systems using Hidden Markov Model classifiers [3].

978-0-7695-3325-4/08 $25.00 2008 IEEE DOI 10.1109/EMS.2008.74

81

2. Glial Reservoirs
2.1. The Glial Network
Recent research in neurobiology has uncovered the intriguing fact that the neuron is not the only entity involved in cognitive processes. [4], [5], [6]. The glial cells outnumber neurons by a factor of around 10:1, yet because they do not signal electrically as neurons do, it has been assumed that they are merely the glue or support framework for the neuron. Neurons can have long axon fibres (hence forming long-range connections) while typical glial cells such as astrocytes are limited to connections of a few tens of m. Glial cells appear to connect to their immediate neighbourhood in an indiscriminate manner but neurons can connect to very specific targets. Whereas neurons transmit signals quickly (~100ms-1), astrocytes do so much more slowly (intracellular- ~2x10-5 ms-1; extracellular- ~2x10-7 ms-1). The main constituent of astrocytic signaling is the propagation of Ca2+ waves. Ca2+ waves are dependant on communication to other astrocytes and neurons by using both a network of connections consisting of intracellular gap junctions (via intercellular stores of Ca2+ and IP3) and also by extracellular means (using the neurotransmitters ATP and Glutamate). There is also growing evidence that glial cells such as astrocytes communicate not only with neurons but also with other astrocytes independently of the neural network. This has led to the assumption that glial cells play only a minor part in the communication process of the brain. However, it has been shown that glial cells do signal to each other through intracellular and extracellular Ca2+. Thus simulations of neural networks dating from Hebb in the 1940s [7] up to the current spiked neural networks assume that the neuron is the only entity capable of computation in the brain and that neurons alone are responsible in providing the feedback mechanism in learning to alter synaptic plasticity and dendritic weights in the learning process. The authors propose that the brains glial network works in harmony with the neural network in a far more sophisticated way than has previously been thought.

and as a result delineates patterns presented to the input space far more effectively. This transformation is explicitly computed and takes into account the temporal nature of the network and stimulus. There are many different types of node in the literature, for example; linear nodes, threshold gates and Fermi neurons have all been used. The type of reservoir that is closest to the system the authors propose is that using spiking neurons with a dynamic synaptic model, which has been shown to have good performance compared to other node types. The proposed reservoir is similar to the Liquid State Machine (LSM) as described by Maas. However, rather than using spiking neurons the reservoir nodes, the reservoir will instead use elements that mimic the behaviour of glial cells. Maass [5] states that LSMs need to satisfy two characteristics. Firstly, that the reservoir is capable of point-wise separation and secondly, that the reservoir is capable of universal approximation. The first characteristic is usually satisfied by using dynamic synapses in order to separate the input patterns into a higher dimensionality, and the second characteristic is usually satisfied by implementing a simple linear regression function in order to distinguish between those patterns in a higher dimension. The authors glial based system uses extra cellular Ca2+ wave propagation and interference patterns to satisfy the first characteristic of point wise separations, and uses the underlying spiking neural network as a more flexible classification system than a simple linear discriminator to satisfy the second, universal approximation characteristic. The authors believe that this type of system has greater flexibility and computational power and is more biologically appropriate. In general, all reservoir computers have the two reservoir characteristics described above and consist of three parts: an input section, a reservoir, and a classification section (figure 1). In most systems the input section and the reservoir consist of the node types described above and the classification section is a simple linear descriminator.

2.2. Reservoir Computing

Reservoir computers can be considered to preprocess input by applying a transformation from the input space to the feature space. This feature space is at a higher level of dimensionality than the input space

82

Input Reservoir Separate

2.3. Glial Reservoir

Glial reservoirs form a sophisticated feedback system to the neural network. They not only delineate the input patterns for processing by the neural network but also provide a mechanism through which distant neurons can communicate. This glial mediated communication between neurons may, in time, lead to long term changes in the neuron structure. These changes could be the development of new dendrite connections or changes in the synaptic functionality between neurons (for example alteration of weights, threshold levels, delay time, or refractory periods). The propagation of Ca2+ is the main agent for this feedback system. The authors propose that there are two basic ways in which this can be done; by Ca2+ diffusion into the extra-cellular medium, and by propagation of wave fronts in that medium. 2.3.1 Diffusion model. With this model the calcium diffuses to the adjacent extra cellular medium. Either a partial differential equation such as used in the heat equation or in Ficks first or second laws of diffusion can be used or a more complex model using continuous time stochastic processes as used in Brownian motion can be used to model particle movement. In its simplest form the standard diffusion equation can be used to calculate the Ca2+ flux from one cell to another, for example the flux J is proportional to the gradient in concentration of Ca2+:

Classify Classifier Figure 1. Reservoir system 2.2.1 Training. Most training for the classifier is assumed to use linear regression (non-linear techniques have so far not been considered). Training can be offline or on-line. With off-line training, in which a batch of inputs is presented to the system, an attempt is made to find a weight matrix that minimizes the distance between all of the inputs read into the classifier; the reservoir states, inputs, bias etc, and the classifiers desired outputs. On-line training generally uses some type of Least Mean Squares learning algorithm for each input presented sequentially to the system. 2.2.2 Static versus Dynamic Reservoirs. The original reservoirs had fixed weights and acted as static filters. Recently more interest has been given to dynamic reservoirs that change their weights to optimize a given task. This idea is used to evolve a set of optimal reservoirs by Evolino from a pool of randomly chosen reservoir alternatives. Steil and Jaeger have examined ways of dynamically adapting the reservoirs node characteristics. This can be done in many ways; for example altering the learning parameters, or changing the gain and threshold of a spiking neuron. 2.2.3 Structured Reservoirs. Originally, reservoirs where uniform structures. However Haeussler has recently shown that performance can be significantly increased if specialist sub-reservoirs are included in a global reservoir [9]. These sub-reservoirs perform specific delineation of input patterns for a specific task. Such structures have an analogous function in biological systems, where specialist areas of the brain perform specific tasks.

J = D

dn dx

where D denotes the diffusion coefficient of the material, and

dn is the concentration gradient of Ca2+ between dx

adjacent cells. 2.3.2 Wave propagation model. The wave propagation model is more biologically relevant than the simple diffusion model presented above. Glial networks generate Ca2+ waves spreading across relatively large areas of the brain. Moreover, these waves interact to produce complex interference patterns. It is these interference patterns and the Ca2+ levels encoded within them that form the basis of this model. The wave model, unlike the diffusion model, recognizes that Ca2+ stores are not unlimited and must be used sparingly. It has recently been shown that in real biological systems that a rapidly oscillating calcium wave spread quickly but is much more localized than a calcium wave that is slowly

83

oscillating. This is probably due to the fact that the amplitude of the calcium wave to begin with is influence by the initial concentration of calcium released. The larger the release the larger the amplitude of the wave and the longer it will last. In corollary the smaller the release of calcium the more quickly a new release can be made. Oscillations will be made more quickly, and with a smaller amplitude and produce waves with less longevity. This differing level of wave instigation has the added advantage of further increasing the capability of the system to distinguish between, and consequently classify, input patterns.

the synaptic gap, and the ability for neurotransmitters to cross this gap. If raised Ca2+ either reduces the size of this gap or increases the ability of the synapse to send or receive neurotransmitters across this gap then the curve used to model the PSP changes, this corresponds to synaptic plasticity, or weight changes, in the neural network system. In practice we alter the equation below to simulate the effect of raised Ca2+ levels. The ratio between Sk and Si determine the initial steepness of the curve, the curves decay and the overall height of the curve.

3. Proposed System
The system the authors propose uses the interactions between glia, synapse and neuron to provide macroscopic function. The interference patterns of raised levels of Ca2+ waves modulate transmission properties of the surrounding synapses, which in turn has an effect on the functionality of the neurons they are connected to. This is illustrated in Figure 2.
P A3 A1 A2

(Sk ) =

S Sk 1 exp k S Sk k

where Sk and Si are synaptic time constants. Normal Ca2+ level spike delay Increased Ca2+ level

spike
N3 N2

delay Figure 3. Variation of post-synaptic 2+ potential with Ca level

N1

Figure 2. Interaction of neuron and astrocyte networks In figure 2, at time t1, neurons N1, N2, N3 are quiescent, as are astrocytes A1, A2, A3. At time t2 (t2>t1), neurons N1 and N3 are responding to an input stimulus, in turn stimulating astrocytes A1 and A2. A1 and A2 emit Ca2+ waves which propagate through surrounding astrocytes. At time t3 (t3>t2), points of constructive interference (P) will cause hot spots of astrocytic activity (e.g. A2) stimulating nearby neurons (e.g. N2). In this model the authors consider raised Ca2+ levels to have 3 major effects on synaptic functionality. Firstly, it has the potential to vary the post synaptic potential (PSP). For example the model of the PSP of an excitatory or inhibitory synapse may, after raising Ca2+ levels, have a much steeper initial climb and a much shallower tail. Moreover, the height and area under the curve will increase. In biological systems this makes sense, the function the authors are modeling is

Secondly, new synapses may be formed if sufficient dendritic/axonal activity occurs between neurons. This direction (and mitality) of dendritic development may be coordinated by the direction and strength of the Ca2+ waves. Moreover, the interference between waves tells the system about the direction of the waves, the location of the initial wave and its immediate history. The neuron itself is directly influenced by the synapses it is attached to, and therefore indirectly affected by the Ca2+ wave. However, it may also be directly affected by raised Ca2+ levels in that the refractory period of the neuron may be decreased if the Ca2+ levels around the neuron are raised. Again this makes biological sense; if a neuron becomes exhausted then the more stores of Ca2+ it has to replenish itself the shorter its refractory periods will be (figure 4).

84

Normal Ca2+ level

(a)

Raised Ca2+ level

(b) Refractory Periods rel abs.

Figure 4. Variation of absolute refractory 2+ period with Ca level Thus the authors propose the feedback model shown in figure 5 is at work at the macroscopic level in a neural network. ATP/Glutamate
Glial cell

Figure 6. Glial wave simulations. (a), 2+ dynamic Ca wave interference, (b), total 2+ Ca movement within a system The following sections describe possible implementations that the authors believe will more efficiently model the glial reservoir described in this paper. There is an elegant synergy between the particular implementations based on parallelism, presenting the intriguing possibility of ultimately combining technological solutions to maximum benefit.

Spike Train

Ca

2+

Neuron Spike

Ca2+ Synapse

4.1. Graphical Processing Unit (GPU)

It has long been recognized that the computational capability of the GPU has been outstripping that of the CPU by an increasingly large factor, principally because GPUs do not have to cater for legacy code, complex flow and cache control as CPUs do and can dedicate more real estate to mathematical and data processing functions, parallel processing and fast IO. Recently, GPU manufacturers have produced Clike languages (e.g. CUDA and Brook+) to exploit the power of GPUs for general computation, and have developed GPUs for general purpose computing (e.g. Nvidia Tesla). These systems are called stream computers, which are massively parallel thread- based systems that have single/double precision mathematic functions built in. So far very little work has been done in simulating neural networks on these devices. The model as described in this paper can be parallelized in a number of ways. The synaptic function, the neuronal function, the glial cells, the spikes and Ca2+ waves are all able to be parallelized. Currently the authors are implementing CUDA code to

Figure 5. Glial-neuron feedback The glial cell responds to neuronal activity through the elevation of intracellular Ca2+ levels. This causes the release of neurotransmitters which in turn adapt the transmission of signals across the neural synapse, hence forming a feedback mechanism (which could be positive or negative depending on the type of neurotransmitter released).

4. Implementation
Currently the components of the system described are implemented in software using C++ and OpenGL. The two screenshots in figure 6 show dynamic Ca2+ wave interference (figure 6a) and total Ca2+ movement within a system. (figure 6b). These form the basis of Ca2+ calculation for the underlying spiking neural network.

85

simulate Ca2+ wave activation (creating a reservoir) in order to implement spiking neural network interaction/categorization of this reservoir.

4.2. Field Programmable Gate Array (FPGA)

Field Programmable Gate Arrays are programmable devices consisting of a number of logic blocks. The fine grained parallelism possible on these devices has led them to be used increasingly as High Performance Computers. Although operating at fairly modest clock speeds they are capable of a massive amount of parallelism, and are capable of both data and instruction parallelism. C- like high level programming languages have allowed software engineers to exploit these devices further. FPGAs have been used recently to implement spiking neural networks. To the authors knowledge, no one has yet implemented a glial network or simulated Ca2+ waves using these devices.

identified, using state of the art hardware, and these implementations have in common that they are highly parallel. The system proposed in this paper combines a number of techniques that have hitherto been unexplored such as: the use of a reservoir separate from the neural network itself and inspired by the glial network; simulation of neural networks on GPUs; simulation of glial cells using digital or analogue electronic circuits. The authors are investigating a number of areas where this system could be applied. The main application is in a speech/voice recognition system called Chorus. This system is intended to mimic the human auditory system, providing the main filtering mechanism from ambient noise in order to identify individual voices/commands from a group of speakers.

6. References
[1] Benjamin Schrauwen, David Verstraeten and Jan Van
Campenhout An overview of reservoir computing: theory, applications and implementations Proceedings of the 15th European Symposium on Artificial Neural Networks (2007) [2] David Verstraeten, Benjamin Schrauwen, Michiel D`Haene and Dirk Stroobandt An experimental unification of reservoir computing methods Neural Networks (2007) [3] Benjamin Schrauwen, et. al., Compact hardware Liquid State Machines on FPGA for real-time speech recognition Neural Networks (2008)

4.3. Field programmable analogue device (FPAD)

The advantages of implementing neural networks using analogue electronic circuits have been widely recognised and include, for example, dramatically increased speed of operation over equivalent digital hardware implementations. Work has been done on the development of analogue electronic neurons, but so far this approach has not been taken in the simulation of glial cells. The authors believe that there may be a significant computational advantage to an analogue electronic implementation, not just for the neural network but for the glial reservoir too. This has been given new impetus with the emergence of field programmable devices such as field programmable analogue arrays (FPAA), which are commercially available from manufacturers such as Anadigm. However, the state of commercially available technology is such that it would not currently be practical to implement a reservoir with more than a few nodes, and so work in this area is currently restricted to SPICE simulations.

[4] V. Alvarez-Maubecin, F. Garcia-Hernandez, J.T.

Williams, E.J. Van Bockstaele, Functional coupling between neurons and glia. J Neurosci 2000, 20:4091-4098.

[5] Rouach N, Glowinski J, Giaume C: Activity-dependent neuronal control of gap-junctional communication in astrocytes. J Cell Biol 2000, 149:1513-1526. [6] Araque A, Parpura V, Sanzgiri RP, Haydon PG: Tripartite synapses: glia, the unacknowledged partner. Trends Neurosci 1999, 22:208-215 [7] D.O. Hebb.(1949), The organization of behavior, New
York: Wiley

5. Conclusion
This paper has shown how the glial network has biological relevance to reservoir computing and proposes a proof-of-concept system. In particular, the paper has described how growing evidence for the central role of the glial network in pre-processing stimuli opens up the possibility of a step change in the computational power afforded by neural networks. A number of implementation approaches have been

[8] W. Maass, T. Natschlager, et al. (2002). Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations. Neural Computation 14: 2531-2560 [9] S. Haeusler, W. Maass. A statistical analysis of information-processing properties of lamina-specific cortical microcircuit models. Cerebral Cortex 17(1):149162, 2006.

86