Professional Documents
Culture Documents
Theoretical Analysis of Cross-Correlation of Time-Series Signals Computed by A Time-Delayed Hebbian Associative Learning Neural Network
Theoretical Analysis of Cross-Correlation of Time-Series Signals Computed by A Time-Delayed Hebbian Associative Learning Neural Network
Department of Biological Sciences, University of North Texas, Denton, Texas 76203, USA
Abstract: A theoretical proof of the computational function performed by a time-delayed neural network implementing a
Hebbian associative learning-rule is shown to compute the equivalent of cross-correlation of time-series functions, show-
ing the relationship between correlation coefficients and connection-weights. The values of the computed correlation coef-
ficients can be retrieved from the connection-weights.
Keywords: Time-delayed neural networks, cross-correlation function, Hebbian learning rule, associative learning, time-series
signal processing.
Mathematically, it states that if the input and output of a Let x(t) and y(t) denotes the input and output signals
neuron are activated simultaneously, then the connection- at time t , respectively, and w(t, ) denotes the connection-
weight for the inputs are changed. To rephrase in the current weight between them with a lag-time, , then the modified
neural network terminology, it states that the weight con- time-delayed Hebbian learning-rule is given by:
necting two neural elements will change if and only if both
neural elements are activated at the same time; otherwise the if x(t ) 0 and y(t) 0
connection-weight remains the same. then w(t, ) 0
Let x(t) and y(t) denote the real-valued activation else w(t, ) = 0
functions of input and output of a neuron, respectively, and if
w(t) denotes the real-valued connection-weight between where x(t ) denotes the input signal delayed by the lag-
x(t) and y(t) , and w(t) represents the weight change time, , and w(t, ) denotes the change in connection-
between successive time-steps, then the Hebbian associative weight (or the weight-change). Thus, a continuous-time
learning-rule is give by: time-delayed Hebbian learning-rule is given by extending
Eq. 1:
if x(t) 0 and y(t) 0
w(t, ) = x(t )y(t) (3)
then w(t) 0
For hardware implementation, we use discrete lag-times
else w(t) = 0 ( = kt ) in integral, k , multiples of t to delay the input
where non-zero value of the activation functions represents signal by multiple delay-tap lines. Therefore, the time-
activation and zero value represents non-activation. More delayed Hebbian associative learning-rule at the k -th delay-
concisely, a Hebbian learning-rule that satisfies the above line is given by:
condition is expressed mathematically as follows: w k (t,kt) = x(t kt)y k (t) (4)
w(t) = x(t)y(t) (1) where k is an integer constant, w k (t,kt) is the change
The relationship between the input and output with re- in the k -th connection-weight between the time-delayed
spect to the associated connection-weight is shown in Fig. input, x(t kt) , and the k -th output, y k (t) (see Fig. 1).
(1), and the output is given:
A single time-series signal is used as the input to the
y(t) = w(t)x(t) (2) network. This time-delayed input is cascaded into multiple
branches as inputs to successive neurons to provide the in-
MODIFIED TIME-DELAYED HEBBIAN ASSOCIA-
TIVE LEARNING-RULE puts for the modified Hebbian learning-rule (Eq. 4) to update
the corresponding connection-weights. The network would
A time-delayed neural network architecture is used to produce as many outputs as there are discrete time-delays.
process the time-varying input signal in this neural network. The k -th output of the network in Fig. (1) is established by:
This time-delayed network is similar but different from the
hybrid network introduced earlier by Tam [11]. The initial y k (t) = w k (t,kt)x(t kt) (5)
input is delayed successively by a time-delay element in each Alternatively, each of the delay-tap lines in Fig. (1) can
input stage of the network (see Fig. 1). Thus, the time-delay be considered as feeding into a pseudo-neuron as the first
produces the modified Hebbian learning-rule such that the (pseudo) layer of the network in Fig. (2). This first layer can
connection-weight will change only if the time-delayed input be considered as a pseudo-layer for the network because it
and current output are activated rather than if the current does not perform extra computation, except for conceptuali-
input and current output are activated simultaneously. In zation of the equivalent neural network architecture.
other words, the output is associated with the previous input
x(t) x0(t)
rather than the current input. x(t) y0(t)
x(t) y0(t)
x(t) x(t-t) x1(t)
w0(t, 0) t y1(t)
x(t-t) y1(t)
t x2(t)
x(t-2t) y2(t)
x(t-2t) t
t y2(t)
w2(t, 2t) x(t-3t) x3(t) y3(t)
t
x(t-3t) y3(t) t
w3(t, 3t)
. .
. . .
. . .
. x(t-kt)
t
x(t-kt) yk(t) xk(t) yk(t)
wk(t, kt) t
Fig. (1). Architecture of the time-delayed neural network showing Fig. (2). Diagram showing how the time-delayed inputs are cas-
the relationships between the time-delayed input, x(t kt) , con- caded into forming a layer of pseudo-input neurons. This network
nection-weights, wk (t, kt) and their output, yk (t) . architecture is equivalent to the diagram shown in Fig. (1).
Cross-Correlation of Time-Series Signals The Open Cybernetics and Systemics Journal, 2007, Volume 1 3
The output of the k -th time-delayed pseudo-input neuron It will be shown below that the cross-correlation coeffi-
(in the first pseudo-layer) can be expressed in terms of the cients are computed by the weight-sum of the time-delayed
initial input signal by: inputs by the output neuron at the k -th connection-weight
after successive iterative training.
x k (t) = x(t kt) (6)
COMPUTATION OF ADAPTIVE TIME-DELAYED
The main reason why we represent the network in this
CONNECTION-WEIGHTS
equivalent architectural form is that now the layer of time-
delayed inputs is a parallel layer rather than a cascaded se- When the network is trained with n iterations of the dis-
quential input layer. In other words, it transforms the single crete time step, t , the resulting connection-weight is given
sequential time-series input into parallel inputs by the delay- by:
lines, which allows for simultaneous parallel processing
rather than sequential processing. This represents the spatio-
w k (nt,kt) = w k (0,kt) +
temporal transformation of the input signal explicitly by the n
w ( jt,kt)
(9)
alternate network architecture, although they are equivalent k
implicitly. j= 0
Such a network would have a single sequential input, for t = nt and = kt . The continuous-time time-
x(t) , branched into (k + 1) parallel lines by k discrete delayed Hebbian learning-rule of Eq. (3) can be re-expressed
delays. It will also have k outputs, y k (t) . The k -th output in terms of the discrete-time step (for t = jt ) as:
of the network in Fig. (2) is given by:
w k ( jt,kt) = x( jt kt)y( jt) (10)
y k (t) = w k (t,kt)x k (t) (7)
The resulting connection-weights after iterating n dis-
These outputs can be further merged into a single output, crete time steps becomes:
y(t) , to form a network produces a single output signal w k (nt,kt) = w k (0,kt) +
rather than multiple outputs (see Fig. 3). This results in the
n
x(t) x0(t)
x(t) y0(t) The corresponding cross-correlation function for a dis-
crete time step, t , and lag time, = kt , is given by:
x(t-t) x1(t)
t y1(t)
1 k
t
x(t-2t) x2(t)
y2(t) y(t)
rxy (kt) = lim lim
n t 0 nt
x( jt,kt)y( jt)t (13)
j= 0
x(t-3t) x3(t)
t y3(t) RELATIONSHIP BETWEEN CROSS-CORRELATION
. .
FUNCTION AND TIME-DELAYED HEBBIAN CON-
. . NECTION-WEIGHTS
. .
x(t-kt)
xk(t) Substituting Eq. 11 into Eq. 13, the relationship between
t yk(t) the cross-correlation function and the time-delayed Hebbian
Fig. (3). Diagram showing the architecture of the single-input and connection-weights is revealed:
single-output network that computes the weight-sum of time- 1
delayed input signal. rxy (kt) = lim lim
n t 0
[w k (nt,kt) w k (0,kt)] (14)
nt
Thus, this network architecture will provide a single in-
put and a single output to process the time-series signal using This equation proves that theoretically the cross-
a pseudo-input layer. This design satisfies the main objective correlation function is essentially computed by cumulating
of creating a neural network that correlates two time-series the connection-weight after iterations of n learning time
signals, x(t) and y(t) , using a set of time-delayed Hebbian steps using the time-delayed Hebbian learning-rule intro-
duced in this paper. This relationship shows that the cross-
associative learning-rules.
correlation function is computed simply by the difference
4 The Open Cybernetics and Systemics Journal, 2007, Volume 1 David Tam
between the initial and final connection-weights of the the processing results are retrieved from the connection-
TDNN. weights rather than obtained from the output of the network.
Once trained, the network can still predict the output, y(t) ,
The correlation coefficient, rxy (kt) , at lag-time
like other neural networks by computing the weighed-sum of
= kt can be retrieved directly from the k -th connec- the time-delayed input signal at the k -th connection-weight.
tion-weight of the network. This provides a theoretical
closed-form solution of the relationship between the connec- Although this is linear network collapseable into an
tion-weights and the correlation coefficients of a cross- equivalent single-input, single-output network, it can per-
correlation function. form multiple (parallelizable) computations of all the corre-
lation coefficients simultaneously, which are retrievable
This also shows, in contrast with most other neural net- from each of the connection-weights. Thus, this analysis
works, the computational result of the network is retrieved bridges the conceptual framework between traditional engi-
from the connection-weights rather from the output of the neering technology in cross-correlation and the novel tech-
network. Furthermore, in contrast with most other neural nology implemented by a time-delayed neural network.
networks, the network performs linear computation rather
than nonlinear computation, since cross-correlation is essen- Furthermore, since the cross-correlation function is com-
tially a linear operation. Because of the linearity, there are puted using a neural network, such computation can be im-
multiple equivalent networks that can represent the same plemented in hardware to process signals in real-time.
computation, as already shown in the above analysis. Hardware implementation of this network can provide high-
speed processing of time-series signals when cross-
TRAINING OF THE NETWORK correlation computation is required. Demonstration of the
The neural network shown above illustrates how the net- applications of this time-delayed Hebbian network for proc-
work can self-organize to compute the cross-correlation essing time-series signals, such as auditory signals for
function by adapting its connection-weights after n itera- sound-localization and frequency-tone discrimination will be
tions of time-steps. The remaining question is: How does the given in a subsequent paper in further details.
network “know” what the output, y(t) , should be? The an- REFERENCES
swer lies in how the network is trained. During the training
[1] E. Oja, “Principal components, minor components, and linear neu-
phase, the network is detached from producing its own out- ral networks”, Neural Networks, vol. 5, pp. 927-936, Nov-Dec
put. The time-series signals to be cross-correlated, x(t) and 1992.
y(t) , are fed into the input and output of the network, re- [2] E. Oja, “A simplified neuron model as a principal components
analyzer”, J. Math. Biol., vol. 15, pp. 267-273, Nov 1982.
spectively, so that the connection-weights can be formed [3] Y. Chauvin, “Principal component analysis by gradient descent on
internally during training. a constrained linear Hebbian cell”, Proc. IJCNN, Washington, DC,
vol. I, pp. 373-380, June 21-24, 1989.
After training, in the retrieval phase, the time-series sig- [4] A. Krogh, and J. Hertz, “Hebbian learning of principal compo-
nal y(t) is detached from the network so that the network nents”, In: Parallel Processing in Neural Systems and Computers
(R. Eckmiller, G. Hartmann and G. Hauske, eds.) Elsevier, Am-
can produce its own output, y(t) , by computing the sterdam, pp. 183-186, 1990.
weighted sum of the time-delayed input based on the adapted [5] S. Kung, and K. Diamantras. “A neural network learning algo-
rithms for adaptive principal component extraction (APEX)”, Proc.
connection-weights. The correlation coefficients are re- ICASSP-90, Albuquerque, NM, pp. 861-864, April 1990.
trieved from the connection-weights directly at the k -th [6] R. Linsker, “Self-organization in a perceptual network”, Computer,
connection-weight after training. vol. 21, pp. 105-117, Feb 1988.
[7] E. Oja, H. Ogawa, and J. Wangviwattana, “Learning in non-linear
DISCUSSIONS constrained Hebbian networks”, In: Artificial Neural Networks. (T.
Kohonen, K. Mäkisara, O. Simula, and J. Kangas, eds.) North-
Although it can be shown by other investigators [1-8] Holland, Amsterdam, pp. 385-390, 1991.
that a time-delayed Hebbian network can compute mathe- [8] J. A. Sirat, “A fast neural algorithm for principal component analy-
sis and singular value decomposition”, Int. J. Neural Sys., vol. 2,
matical functions such as PCA, this paper illustrated that a pp. 147-155, 1991.
time-delayed Hebbian network can compute cross- [9] D. C. Tam, “Computation of cross-correlation function by a time-
correlation function too. A theoretical analysis is given to delayed neural network”, In: Intelligent Engineering Systems
prove that the connection-weights developed after training is through Artificial Neural Networks. (Cihan H. Dagli, Laura I.
Burke, Benito R. Fernández, Joydeep Ghosh, eds.), American Soci-
equivalent to the computation of the coefficients of a cross- ety of Mechanical Engineers Press, New York, NY, vol. 3, pp. 51-
correlation function. The TDNN processes the time-series 55, 1993.
signals, x(t) and y(t) , in such a way that the connection- [10] D. O. Hebb, The organization of behavior. New York: Wiley,
1949.
weights developed after training would produce the correla- [11] D. C. Tam, “A hybrid time-shifted neural network for analyzing
tion coefficients of the cross-correlation between x(t) and biological neuronal spike trains”, In: Progress in Neural Networks,
y(t) . (O. Omidvar, ed.) Ablex Publishing Corporation: Norwood, New
Jersey, vol. 2, pp.129-146, 1994.
The main difference between the computation achieved
by this network and other traditional neural networks is that
Received: July 10, 2007 Revised: July 20, 2007 Accepted: July 30, 2007