Sound Source Localization Using Deep Residual Networks
*Nelson Yalta (Waseda University), Kazuhiro Nakadai (Honda Research Institute Japan Co., Ltd), and Tetsuya Ogata
(Waseda University)
1. Introduction
Machine and robots has become part of humans
every-day life. Thus for natural interactions with
humans, robots should have auditory functions [1] which
can be used for Sound Source Localization (SSL), where
not only is necessary to recognize the sound locations
but also detect sound events. However, environmental
factors such as background noise, moving sound sources,
and room reverberations change dynamically in the real
world [2], and the number of microphones and the
acoustic properties of robots body complicates the
SSL.
Conventional Methods to improve the performance of
robots SSL uses subspace-based methods like Multiple
Signal Classification (MUSIC) [3], however, the
performance of these methods depends on the number of
microphone used for the task, and it is reduced relatively
according to the signal-to-noise ratio.
Recently, Deep Neural Networks (DNN) and Deep
Convolutional Neural Networks (DCNN) approaches has
shown a performance than even exceed humans
capability on different tasks. In this paper, we refine a
DCNN structure using deep residual learning for SSL
tasks.
2. Sound Source Localization
2.1. MUSIC
Multiple Signal Classification (MUSIC) [3], a method
for SSL, where the localization process used
representations of the energy of the signals and the time
difference on the arrival of the signal. These
representations are called steering vector and can be
obtained, in the frequency-domain, using measurements
or physical models. Implementation for MUSIC are
based on Standard Eigen-Value Decomposition (SEVD-
MUSIC), and even when SEVD-MUSIC, because its low
computation cost, is easy to be implemented on robots
and can provides easy detectable reliable peaks[4], it
works when the assumption of, the target sound sources
has sufficiently high power compared to noise, is
satisfied. Generalized Eigen-Value Decomposition
(GEVD-MUSIC) solve the mislocalization on high-
powered noise sources problem, by determining the
correlation matrix in terms of noise. Because in most of
robot SSL cases, the sounds and the noises are cross
correlated, make possible to GEVD can be adopted to the
definition of the MUSIC spatial spectrum . With this all
noises are suppressed, making GEVD-MUSIC a robust
method for noises[4].
2.2. Task
MUSIC presents as a robust method for SSL on
relatively noisy environments, however, the performance
depends on the number of microphones that the array has
and it can drastically drop when the signal-to-noise rate
lows.
3. Implementation
3.1. Deep Residual Network
Recently, research evidence that the network depth is
important on challenging tasks. However, the problem of
vanishing/exploding gradients presents and obstacle for
implementing this very deep networks. With the network
depth increasing, the accuracy training saturates,
presenting this as a degradation problem. Deep Residual
Networks (DRN) are introduced on [5], in order to
address the degradation. The residual learning can be
denoted as
F(x) := H(x) x, (1)
where H(x) becomes the desire underlying mapping.
The residual learning let the layers fit a residual mapping
F(x), instead of hoping each few stacked layers directly
fit a desired underlying map. Then, the original mapping
is recast into F(x) + x, this formulation can be
implemented by feedforward neural network using
shortcut connections (Fig.1). A shortcut is presented as an
identity mapping, which skip one or more layers, and
because it does not present computation complexity
neither need extra parameter can be trained end-to-end
using a common library. As a concern about the training
time, Deeper Bottleneck are introduced too[5]. Here a
stack of 3 layers are used for each residual function F, so
in order to reduce and restore the dimensions, a 1x1 3x3
and 1x1 kernel convolutional layers are used.
3.2. Audio Feature Extraction
The input data was prepared using Short-Time Fourier
Transform (STFT), and a label with the angle was settled
as target. For input, we transformed from a normalized N
channel audio with 16KHz of sampling rate, into STFT
features, extracting a frame with length of 400 samples
(25ms) and a hop of 160 samples (10ms). From the STFT
Fig. 1 Residual Learning
we only used the power information, which after remove
the imaginary part, was normalized on the W frequency
bin dimension, on a range between 0.1 and 0.9. Since one
STFT frame from the audio has corrupted information
due to the noise, we stacked H frames, so the input of the
network has a dimension of NxWxH.
3.3. Proposed Model
We used two DRN implemented with 1 residual block
(ResNet1) and 2 residual blocks (ResNet2), and trained
with supervised learning. After the input two 1x1 kernel
convolutional layers with TanH activation are used, and
then they are connected with to the residual block. Each
residual block contained 3 Bottleneck, making 9
convolutional layers for a residual block. Then, an
empirical kernel sized of 6 convolutional layers is used.
The empirical kernels size was set to 45x4 dims. With this
configuration, the ResNet1 has implemented 17
convolutional layers.
After each convolutional layer follows a Batch
Normalization layer, and the ELU activation was used for
the rest of all the networks. Nevertheless, a TanH
activation was also evaluated on the first residual block of
the ResNet2. A Max-Pooling layer was used on the
middle of the network and another one at the end of the
convolutional layers. Finally, after the second max-
pooling layer 2 fully connected layers was implemented.
For evaluating the performance of the residual learning,
we also trained a plain network with the same number of
layers as ResNet1, and at a same number of iterations.
4. Evaluation
4.1. Audio Dataset
For training and test we use the Acoustic Society of
Japan-Japanese Newspaper Article Sentence (ASJ-JNAS)
corpora, which includes utterances from 216 speakers for
training and 42 speakers for test, in Japanese language
and with different length. For preparing the training
dataset, was used the impulse responses from HEARBO
robot equipped with an array microphone of 16 channels,
obtained from a room with 200ms of reverberation, and
4x7 meters of dimension, every each 5dg. A clean single
channel utterance was convoluted with the multiple
channel impulse response of a random angle, and then
white noise was added to each channel, on range between
clean data and -30 dB of signal-to-noise ratio (SNR). The
noise differs from each channels in order to try to
simulate a real environmental noise.
4.2. Networks Training Process
For our experiments, from the audio data using STFT
we got an input of 16 audio channels x 257 freq bins x 20
frames. We trained 4 different networks using ADAM
solver and SoftMax with Cross Entropy as loss function.
The initial alpha for the solver was set to 1e-4, and the
target label was from 0-355 degrees. However, the output
of the network was set to 361 dimensions, where the 0 to
Fig. 2 Results
359 outputs activate with it respective angle label, and the
360 output activates when the input data has no-audio
information. Doing this, the network not only can locate
the sound source angle, but also detect if there is a sound
source.
4.3. Results
The results of our experiments are showed in Fig. 2,
where the mean block accuracy of the median angle was
evaluated with a tolerance of 0 degrees. For the
evaluation, 50 audio files (25 from females, and 25 from
males) was used and tested every 5 degrees from 0 degree
to 355 degrees. A similar white noise was added at all
channels in a SNR range from the Clean data up to
-35dB, so the noise evaluated is different to the used on
the training process. We used HARK for evaluating
SEVD-MUSIC. From each audio, we obtained the
predicted median angle, and this was compared with the
true angle.
5. Conclusion
In this paper, we proposed a SSL method using a DRN.
We showed that not only a DCNN can perform a better
block accuracy than SEVD-MUSIC on noisy
environments, but also we showed that a DRN has better
performance that a DCNN plain network with the same
number of layers. The performance of the networks does
not drop even when a different noise is used, from which
the networks has been trained.
ACKNOWLEDGEMENTS
The work has been supported by MEXT Grant-in-Aid for
Scientific Research (A) 15H01710.
REFERENCES
[1] K. Nakadai, T. Lourens, G. O. Hiroshi, and H. Kitano, Active audition for
 humanoid, Proc. Natl. Conf. Artif. Intell., pp. 832839, 2000.

[2] K. Nakadai, H. Nakajima, Y. Hasegawa, and H. Tsujino, Sound source

separation of moving speakers for robot audition, ICASSP, IEEE Int. Conf.

Acoust. Speech Signal Process. - Proc., pp. 36853688, 2009.

[3] R. Schmidt, Multiple emitter location and signal parameter estimation,

IEEE Trans. Antennas Propag., vol. 34, no. 3, pp. 276280, 1986.

[4] K. Nakamura, K. Nakadai, F. Asano, Y. Hasegawa, and H. Tsujino,

Intelligent Sound Source Localization for Dynamic Environments, pp. 664

669, 2009.

[5] K. He, X. Zhang, S. Ren, and J. Sun, Deep Residual Learning for Image

Recognition, Arxiv.Org, vol. 7, no. 3, pp. 171180, 2015.