See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/51574320

Partially Supervised Speaker Clustering

Article  in  IEEE Transactions on Pattern Analysis and Machine Intelligence · August 2011

DOI: 10.1109/TPAMI.2011.174 · Source: PubMed

CITATIONS READS
48 2,024

4 authors, including:

Hao Tang Stephen M Chu

Central South University IBM
101 PUBLICATIONS   2,034 CITATIONS    81 PUBLICATIONS   1,976 CITATIONS   

SEE PROFILE SEE PROFILE

Mark Hasegawa-Johnson
University of Illinois, Urbana-Champaign
431 PUBLICATIONS   8,792 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Acoustic Landmark and Automatic Speech Recognition View project

Collaborative Patient Portals View project

All content following this page was uploaded by Stephen M Chu on 14 April 2015.

The user has requested enhancement of the downloaded file.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX 1

Partially Supervised Speaker Clustering

Hao Tang, Member, IEEE, Stephen Chu, Member, IEEE, Mark Hasegawa-Johnson, Senior
Member, IEEE, and Thomas Huang, Life Fellow, IEEE

Abstract—Content-based multimedia indexing, retrieval and processing as well as multimedia databases demand the structuring
of the media content (image, audio, video, text, etc.), one significant goal being to associate the identity of the content to the
individual segments of the signals. In this paper, we specifically address the problem of speaker clustering, the task of assigning
every speech utterance in an audio stream to its speaker. We offer a complete treatment to the idea of partially supervised
speaker clustering, which refers to the use of our prior knowledge of speakers in general to assist the unsupervised speaker
clustering process. By means of an independent training data set, we encode the prior knowledge at the various stages of
the speaker clustering pipeline via 1) learning a speaker-discriminative acoustic feature transformation, 2) learning a universal
speaker prior model, and 3) learning a discriminative speaker subspace, or equivalently, a speaker-discriminative distance metric.
We study the directional scattering property of the Gaussian mixture model (GMM) mean supervector representation of
utterances in the high-dimensional space, and advocate exploiting this property by using the cosine distance metric instead of
the Euclidean distance metric for speaker clustering in the GMM mean supervector space. We propose to perform discriminant
analysis based on the cosine distance metric, which leads to a novel distance metric learning algorithm – linear spherical
discriminant analysis (LSDA). We show that the proposed LSDA formulation can be systematically solved within the elegant
graph embedding general dimensionality reduction framework.
Our speaker clustering experiments on the GALE database clearly indicate that 1) our speaker clustering methods based on the
GMM mean supervector representation and vector-based distance metrics outperform traditional speaker clustering methods
based on the “bag of acoustic features” representation and statistical model based distance metrics, 2) our advocated use of
the cosine distance metric yields consistent increases in the speaker clustering performance as compared to the commonly
used Euclidean distance metric, 3) our partially supervised speaker clustering concept and strategies significantly improve the
speaker clustering performance over the baselines, and 4) our proposed LSDA algorithm further leads to the state-of-the-art
speaker clustering performance.

Index Terms—Speaker clustering, partial supervision, distance metric learning.

1 I NTRODUCTION segmentation and clustering, or “who spoke when”,

C ONTENT- BASED multimedia indexing, retrieval
and processing as well as multimedia databases
are active fields of research in the information era
[1]. In many situations it is highly demanded that we
structure the media content (image, audio, video, text,
etc.) so that the identity of the content (face, voice,
keywords, etc.) can be associated with the individual
segments of the data. Often, clustering multimedia
data is a first step to multimedia content analysis
as well as multimedia database construction, mining,
search, and visualization [2].
In this paper, the problem of speaker clustering [3],
[4], [5], [6], [7], [8] is specifically addressed. Speaker
clustering aims to assign every speech utterance in an
audio stream to its respective speaker, and is an es-
sential part of a task known as speaker diarization [9],
[10], [11], [12], [13], [14]. Also referred to as speaker
• H. Tang, M. Hasegawa-Johnson, and T. Huang are with the Depart-
ment of Electrical and Computer Engineering, University of Illinois at
Urbana-Champaign, Urbana, IL, 61821 USA.
E-mail: {haotang2,jhasegaw,t-huang1}@uiuc.edu.
• S. Chu is with the Human Language Technologies Group at the IBM
T. J. Watson Research Center, Yorktown Heights, NY, 10598 USA.
E-mail: schu@us.ibm.com.
speaker diarization is the process of partitioning an
input audio stream into temporal regions of speech
signal energy contributed from the same speakers. A
typical speaker diarization system consists of three
stages. The first is the speech detection stage, where
we find the portions of speech in the audio stream.
The second is the segmentation stage, where we find
the locations in the audio stream likely to be change
points between speakers. At this stage, we often over-
segment the audio stream, resulting in only one single
speaker in each segment. The last is the clustering
stage, where we associate the segments from the same
speakers together. Figure 1 illustrates the process of
speaker diarization. In this paper, we mainly focus
on the clustering stage, not only because the clus-
tering stage is the most important part of speaker
diarization, but also most techniques developed for
the clustering stage can be readily applied to the
segmentation stage (for example, with the help of a
sliding window of fixed or variable length).
Unlike speaker recognition (i.e. identification and
verification), where we have training data for the
speakers and thus training can be done in a super-
vised fashion, speaker clustering is usually performed
in a completely unsupervised manner. The output of
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX
Fig. 1. The process of speaker diarization. A typical
speaker diarization system consists of a speech de-
tection stage, a segmentation stage, and a clustering
stage.
speaker clustering is a unique arbitrary code for each
speaker (e.g., spk1, spk2, etc.) rather than his or her
real identity (e.g., Tom, Mary, etc.). An interesting
question is: Can we do speaker clustering in a some-
how supervised manner? That is, can we make use of
all available prior information that may be helpful for
speaker clustering?
Our answer to this question is positive. It is worth
noting that a few researchers in the field of speaker di-
arization have already tried to incorporate some prior
knowledge into their methods and indeed gained
noticeable improvements in the performance. For ex-
ample, the use of a universal background model
(UBM) for adapted Gaussian mixture model (GMM)
based clustering was attempted in [9] and [10], and
the GMM mean supervector as the utterance rep-
resentation was recently adopted in [13] and [14].
However, none of the previous work addresses every
facet of the problem. In this paper, we offer a complete
treatment to the conceptually new idea of partially
supervised speaker clustering, which refers to the use
of our prior knowledge of speakers in general to
assist the unsupervised speaker clustering process.
By means of an independent training data set, we
acquire prior knowledge about speakers in general by
1) learning a speaker-discriminative acoustic feature
transformation, 2) learning a universal speaker prior
model (i.e. a UBM) which is then adapted to the
individual utterances to form the GMM mean su-
pervector representation, whose directional scattering
properties we study and exploit, and 3) learning a
discriminative speaker subspace, or equivalently, a
speaker-discriminative distance metric.
Figure 2 is a general speaker clustering pipeline.
Basically, there are four critical elements in any
speaker clustering algorithm and it is these elements
that make a difference. We incorporate our prior
knowledge of speakers into the various stages of
this pipeline through an independent training data
set. First, at the feature extraction stage, we learn
a speaker-discriminative acoustic feature transforma-
tion based on linear discriminant analysis (LDA)
[15]. Second, at the utterance representation stage,
we adopt the maximum a posteriori (MAP) adapted
GMM mean supervector representation [16] based on
2
Fig. 2. The general speaker clustering pipeline. There
are four essential elements in any speaker clustering
algorithm. This paper transfers knowledge from an in-
dependent training set in order to improve every stage
in the speaker clustering pipeline.
a UBM [17], which can be considered as a universal
speaker prior model. Third, at the distance metric
stage, we learn a speaker-discriminative distance met-
ric through a novel algorithm – linear spherical dis-
criminant analysis (LSDA). Note that at the clustering
stage, conventional clustering techniques such as k-
means [18] and hierarchical clustering [19] can be
naturally employed.
The contribution of our paper is at least three-fold.
First, we propose methods that allow the transfer
of learning from an independent training set to the
unsupervised clustering problem, and show that these
strategies significantly improve the speaker clustering
performance over the baselines. Our methods out-
perform traditional speaker clustering methods based
on the “bag of acoustic features” representation and
statistical model based distance metrics. Second and
in particular, we study the directional scattering prop-
erty of the GMM mean supervector representation of
utterances in the high-dimensional space, and advo-
cate exploiting this property by using the cosine dis-
tance metric instead of the Euclidean distance metric
for GMM mean supervector speaker clustering. Last
but not least, we propose to perform discriminant
analysis based on the cosine distance metric, which
leads to a novel distance metric learning algorithm
– linear spherical discriminant analysis (LSDA). We
show that the proposed LSDA formulation can be
systematically solved within the elegant graph em-
bedding [20] general dimensionality reduction frame-
work. We demonstrate that the LSDA algorithm leads
to the state-of-the-art speaker clustering performance.
This paper is organized as follows. In Sections
2, 3, 4, and 5, the four stages of the speaker clus-
tering pipeline, namely feature extraction, utterance
representation, distance metric, and clustering, are
described. In each section, we first review the cur-
rent state-of-the-art approaches, and then present the
strategies that incorporate our partially supervised
speaker clustering concept into the corresponding
stage of the speaker clustering pipeline. In Section
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX
6, we describe our experiment setup and protocol,
introduce the performance evaluation metrics, and
present the experiment results as well as provide a
discussion of the results. Finally, we conclude the
paper in Section 7.
2 F EATURE E XTRACTION
2.1 Acoustic Features
The first stage of the speaker clustering pipeline is
feature extraction. Feature extraction is the process of
identifying the most important cues from the mea-
sured data while removing unimportant ones for a
specific task or purpose based on domain knowledge.
For speaker clustering, the most widely used features
are the short-time spectrum envelope based acoustic
features such as the mel-frequency cepstral coeffi-
cients (MFCC) and perceptual linear predictive (PLP)
coefficients [21]. Although MFCC and PLP were not
originally designed for representing the information
relevant to distinguishing among different speakers,
and in fact, their primary use is in speech recognition,
they work reasonably well for speaker clustering in
practice. The higher-order MFCCs (e.g., 13-19) are
known to correspond to the source characteristics in
the source-filter model of speech production [22], and
thus convey speaker information. In order to account
for the temporal dynamics of the spectrum, the basic
MFCC or PLP features are usually augmented by
their first-order derivatives (i.e. the delta coefficients)
and second-order derivatives (i.e. the acceleration
coefficients). Higher-order derivatives may be used
too, although that is rarely seen. These derivatives
incorporate the time-evolving properties of the speech
signal and are expected to increase the robustness of
the acoustic features.
2.2 Speaker-discriminative Acoustic Feature
Transformation
The use of first and second order derivatives of
the basic acoustic features introduces the character-
ization of the temporal dynamics of the spectrum.
However, such characterization is completely unsu-
pervised, and thus lacks the potential to discrimi-
nate between speakers. Using an independent train-
ing data set, we can simultaneously characterize the
temporal dynamics of the spectrum and maximize
the discriminative power of the augmented acoustic
features based on a discriminative learning frame-
work. Specifically, we first compute 13 PLP features
for every speech frame with cepstral mean subtraction
(CMS) and cepstral variance normalization (CVN) to
compensate for the inter-section and inter-channel
variability [23]. Then, instead of augmenting the basic
PLP features by their first and second order deriva-
tives, we augment them by the basic PLP features of
the neighboring frames spanning a window centered
3
on the current frame. More precisely, the PLP features
of the current frame, those of the KL (e.g., 4) frames
to the left and those of the KR (e.g., 4) frames to
the right are concatenated to form a high-dimensional
feature vector, referred to as the context-expanded
feature vector. In the context-expanded feature vec-
tor space, we learn a speaker-discriminative acoustic
feature transformation by LDA based on the known
speaker labels of the independent training data set.
The context-expanded feature vectors can then be
projected onto a low-dimensional (e.g., 40) speaker-
discriminative feature subspace, which is expected to
provide optimal speaker separability. In this way we
transfer knowledge about the speakers in one corpus
to improve clustering of the speakers in a different
corpus.
In the experiment section, we specifically compare
the proposed LDA transformed acoustic features with
the acoustic features traditionally augmented with the
first and second order derivatives and show that the
LDA transformed acoustic features outperform the
traditional acoustic features on speaker clustering un-
der the same clustering conditions. This validates that
the proposed speaker-discriminative acoustic feature
transformation strategy can provide a better frontend
to speaker clustering as compared to traditional ones.
3 U TTERANCE R EPRESENTATION
3.1 “Bag of Acoustic Features” Representation
The second stage of the speaker clustering pipeline is
utterance representation. Utterance representation, as
its name suggests, is the task of compactly encoding
the acoustic features of an utterance. In the literature
on speaker clustering, the mainstream utterance rep-
resentation is the so-called “bag of acoustic features”
representation where the acoustic feature vectors are
described by a time-independent statistical model
such as a Gaussian or GMM. The rationale behind
this representation is that in speaker clustering the lin-
guistic content of the speech signal is considered to be
irrelevant and normally disregarded. Thus, temporal
independence between inter-frame acoustic features is
assumed.
Most often, due to its unimodal nature a single
Gaussian is far from being sufficient to model the
probability distribution of the acoustic features of
an utterance, and a GMM is preferred. The theo-
retical property that a GMM can approximate any
continuous probability density function (PDF) arbi-
trarily closely given a sufficient number of Gaussian
components makes the GMM a popular choice for
parametric PDF estimators.
The acoustic features of an utterance are modeled
by an m-component GMM, defined as a weighted sum
of m component Gaussian densities
m
X
p(x|λ) = wi N (x|µi , Σi ) (1)
i=1
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX 4

where x is a d-dimensional random vector, wi is the E step, the posterior probability of a training vector
ith mixture weight, and N (x|µi , Σi ) is a multivariate falling into every Gaussian component is computed
Gaussian PDF
w0i N (xt |µ0i , Σ0i )
1 T −1
p(i|xt ) = Pm , i = 1, 2, · · · , m
j=1 w0j N (xt |µ0j , Σ0j )
1
N (x|µi , Σi ) = d/2 1/2
e− 2 (x−µi ) Σi (x−µi ) (2)
(2π) |Σi | (3)
Note that Equation 3 is the probability that we
with mean vector µi and covariance matrix Σi . wi can
re-assign the training vector xt to the ith Gaussian
be interpreted as the a priori probability that an obser-
component of the UBM λ0 = {w0i , µ0i , Σ0i }m
i=1 . Based
vation of x comes from the source governed by the ith
on these posterior probabilities, we compute the suf-
Gaussian distribution. Thus it satisfies the properties
ficient statistics of the training data
0 ≤ wi ≤ 1 and m
P
i=1 i = 1. A GMM is completely
w
specified by its parameters λ = {wi , µi , Σi }m
i=1 and the
T
X 1 X
T
estimation of the PDF reduces to finding the proper ni = p(i|xt ), Ei = p(i|xt )xt (4)
ni t=1
values of λ. t=1
A central problem of the GMM is how to es- T
timate the model parameters λ. This problem can 1 X
Ei2 = p(i|xt )x2t (5)
be practically solved by maximum likelihood esti- ni t=1
mation (MLE) techniques such as the expectation-
In the M step, the sufficient statistics of the training
maximization (EM) algorithm [24]. However, it is
data are combined with the prior model sufficient
widely known that MLE easily over-fits with insuffi-
statistics by interpolation. The new model parameters
cient training data. The number of free parameters of a
are obtained as follows
GMM, p, depends on the feature dimension d and the
number of Gaussian components m. More precisely, wi′ = [αi ni /T + (1 − αi )wi ]δ (6)
p = md2 /2+3md/2+m−1, which grows linearly in m
but quadratically in d. In order to alleviate this “curse µ′i = βi Ei + (1 − βi )µi (7)
of dimensionality” [25], diagonal covariance matrices
σi′2 = γi Ei2 + (1 − γi )(σi2 + µ2i ) − µ′2
i (8)
are often used in the Gaussian components. In this
case, p = 2md + m − 1, which grows linearly in both where δ is a scaling factor computed over all new
m and d. mixture weights to ensure that they sum to unity.
The interpolation coefficients in Equations 6-8 are data
dependent and automatically determined for every
3.2 GMM Mean Supervector Representation Gaussian component using the empirical formula νi =
A relatively new utterance representation that has ni /(ni + rν ) where ν ∈ {α, β, γ} and rν is a fixed
emerged in the speaker recognition area is the GMM relevance factor for ν. This empirical formula offers
mean supervector representation, which is obtained a smart mechanism to control the balance between
by concatenating the mean vectors of the Gaussian the new and old sufficient statistics. Figure 3 demon-
components of a GMM trained on the acoustic fea- strates the basic idea of MAP adaptation for a GMM.
tures of a particular utterance [26].
3.2.2 GMM Mean Supervectors
3.2.1 UBM and MAP Adaptation The GMM mean supervector representation of an
When an utterance is short, the number of acoustic utterance is obtained by first MAP adapting the UBM
feature vectors available for training a GMM is small. to the acoustic features of the utterance and then
To avoid over-fitting, we can first train a single GMM concatenating the component mean vectors of the
on an independent training data set, leading to a target GMM to form a long column vector. Figure 4
well-trained GMM known as the UBM in the speaker
recognition literature [27]. Since the amount of data
used to train the UBM is normally large, and the data
is fairly evenly distributed across many speakers, the
UBM is believed to provide a good representation of
speakers in general. Therefore, it can be considered
as a universal speaker prior model in which we en-
code the common characteristics of different speakers.
Given a specific utterance, we can then derive a target
GMM by adapting the UBM to the acoustic features
of the utterance. This is done by MAP adaptation [28]. Fig. 3. The basic idea of MAP adaptation. MAP adap-
MAP adaptation starts with a prior model (i.e. the tation starts with a prior model and iteratively performs
UBM), and iteratively performs EM estimation. In the regularized EM estimation.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX
Independent
Input speech
training speech
Feature Extraction Feature Extraction
Acoustic features Acoustic features
MAP Adaption Model Training
(Mean Only) (GMM)
GMM mean
supervector
UBM
Fig. 4. The generation of a GMM mean supervector. A
GMM mean supervector is obtained by MAP adapting
only the component means of a UBM.
gives a block diagram that shows how a GMM mean
supervector is generated.
Once a target GMM is obtained for an utterance, its
component means are stacked to form a GMM mean
supervector
s = [µ1 T µ2 T ... µm T ]T
It is numerically beneficial to subtract the mean su-
pervector of the UBM from a GMM mean supervector,
namely,
s′ = s − s0 (10)
where s0 is the mean supervector of the UBM. Without
causing any ambiguity, we call s′ (instead of s) a
GMM mean supervector. A complete set of GMM
mean supervectors forms a high-dimensional space
called the GMM mean supervector space.
A supervector created by stacking component
means, as shown in Equation 9, represents local first-
order differences between the UBM and adapted
GMM: shifts in local modes and regional centers
of mass. Concatenating variances to the supervector
would allow us to also represent local second-order
adaptation, e.g., changes in the local compactness of
the GMM. The scattering patterns of second-order
information in supervector space are very different
from those of first-order information, however, so we
choose to include only first-order information.
3.2.3 Property of GMM Mean Supervectors
The GMM mean supervector is an effective utterance
representation that has been applied to speaker recog-
nition. However, it has come to our attention that
the use of the GMM mean supervector representation
for speaker clustering is still rare. The GMM mean
supervector representation allows us to represent an
utterance as a single data point in a high-dimensional
5
space, where conventional clustering techniques such
as k-means and the hierarchical clustering can be
naturally applied.
Figure 5 visualizes the GMM mean supervectors of
many utterances from five different speakers using
2D scatter plots of their two principal components.
In each plot, the different speakers are shown in
different colors. For each speaker, there are about 150
utterances, denoted by small dots. As one can see,
the data points belonging to the same speaker tend to
cluster together. Thus, the Euclidean distance metric
is a reasonable choice for speaker clustering in the
GMM mean supervector space. However, one can also
observe that the data points show very strong direc-
tional scattering patterns. The directions of the data
points seem to be more informative and indicative
than their magnitudes. This observation motivated us
to favor the cosine distance metric over the Euclidean
distance metric for speaker clustering in the GMM
mean supervector space.
A reasonable explanation as to why the GMM
mean supervectors show strong directional scattering
patterns is that when we perform mean-only MAP
adaptation, only a subset of the UBM component
means is adjusted, and the particular subset that is ad-
justed seems to be rather speaker-dependent. Hence,
the speaker-specific information is encoded in those
(9)
component means which are adapted. Therefore, the
utterances from the same speaker tend to yield a
cluster of GMM mean supervectors that scatter in a
particular direction in the GMM mean supervector
space.
As presented later in the experiment section, our
experiment results on all speaker clustering tasks
clearly demonstrate that the cosine distance metric
consistently outperforms the Euclidean distance met-
ric when using the GMM mean supervector as the
utterance representation. This strongly supports our
discovery of the directional scattering property of the
GMM mean supervectors and forms the foundation
of our original motivation to perform discriminant
analysis in the cosine distance metric space.
4 D ISTANCE M ETRIC
The third stage of the speaker clustering pipeline is
distance metric. The distance metrics that can be used
for speaker clustering are closely related to the par-
ticular choice of utterance representations. Two popu-
lar categories of distance metrics, namely likelihood-
based distance metrics and vector-based distance met-
rics, are widely used for the two corresponding utter-
ance representations, respectively.
4.1 Likelihood-based Distance Metrics
For the “bag of acoustic features” utterance repre-
sentation, the distance metric should represent some
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX
Fig. 5. The property of GMM mean supervectors. The
data points show strong directional scattering patterns.
measure of the distance between two statistical mod-
els. A famous likelihood-based distance metric ex-
tensively used for speaker clustering is the Bayesian
information criterion (BIC) [29]. For a given utterance,
the BIC value indicates how well a model fits the
utterance and is given by
λ metric for speaker clustering in the GMM mean su-
BIC(Mi ) = log L(Xi |Mi ) − ki log(ni ) (11)
2
where L(Xi |Mi ) is the likelihood of the acoustic fea-
tures Xi given the model Mi , λ is a design parameter,
ki is the number of free parameters in Mi , and ni is the
number of feature vectors in Xi . The distance between
two utterances Xi and Xj is given by the ∆BIC value.
If we assume Mi and Mj are both Gaussian, then
∆BIC is given by
∆BIC(Xi , Xj ) = n log Σ − ni log Σi − nj log Σj − λP
(12)
where n = ni + nj , Σi and Σj are the covariance ma-
trices of Xi and Xj , respectively, Σ is the covariance
matrix of the aggregate of Xi and Xj , and P is a
penalty term given by
1 1
P = (d + d(d + 1)) log n (13)
2 2
with d being the dimension of the acoustic feature
vectors.
Other likelihood-based distance metrics include the
generalized likelihood ratio (GLR), Gish distance,
Kullback-Leibler divergence (KLD), divergence shape
distance (DSD), Gaussian divergence (GD), cross BIC
(XBIC), cross log likelihood ratio (XLLR), and so forth
[30]. All these metrics have been proposed for the
“bag of acoustic features” utterance representation.
4.2 Vector-based Distance Metrics
For the GMM mean supervector utterance represen-
tation, since an utterance can be represented as a
6
single data point in a high-dimensional vector space,
the most often used distance metric is the Euclidean
distance metric
d(x, y) = (x − y)T (x − y) (14)
As we discussed earlier, in the GMM mean super-
vector space, the data points belonging to the same
speaker tend to cluster together. Thus, the Euclidean
distance metric is a reasonable choice for speaker clus-
tering in the GMM mean supervector space. However,
we observe that the data points show very strong di-
rectional scattering patterns. The directions of the data
points seem to be more informative and indicative
than their magnitudes. This observation motivated
us to advocate the use of the cosine distance metric
instead of the Euclidean distance metric for speaker
clustering in the GMM mean supervector space. The
cosine distance metric is a measure of the angle
between two vectors in the space and is irrelevant to
the norms of the vectors. It is defined as
xT y
d(x, y) = 1 − √ p (15)
xt x yT y
Our experiments show that the cosine distance met-
ric consistently outperforms the Euclidean distance
pervector space.
4.3 Distance Metric Learning versus Linear Sub-
space Learning
Although the Euclidean and cosine distance metrics
can be directly used, they are optimal only if the
data points are uniformly distributed in the entire
space. In a high-dimensional space, most often the
data points lie in or near a low-dimensional manifold,
or preferably a linear subspace, of the original space.
In this case, it is extremely advantageous if we can
learn an optimal distance metric for the data.
We define a generalized Euclidean distance metric
between two data points x and y as
d(x, y) = (x − y)T A(x − y) (16)
where the positive definite matrix A is aimed to
compensate for the non-uniform data distribution. If
A coincides with the covariance matrix of the data,
this generalized Euclidean distance metric reduces to
the Mahalanobis distance [31]. Equation 16 can be re-
written as
1 1 1 1
d(x, y) = (A 2 x − A 2 y)T (A 2 x − A 2 y)
= (W x − W y)T (W x − W y) (17)
That is, the generalized Euclidean distance metric
between x and y can be re-organized as the Euclidean
distance metric between two linearly transformed
1
data points W x and W y where W = A 2 .
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX
Similarly, we define a generalized cosine distance
metric between x and y as
xT Ay
d(x, y) = 1− √ √
xT Ax yT Ay
(A1/2 x)T (A1/2 y)
= 1− √ √ hypersphere
(A1/2 x)T (A1/2 x) (A1/2 y)T (A1/2 y)
(W T x)T (W T y)
= 1− √ √ (18)
(W T x)T (W T x) (W T y)T (W T y)
Likewise, the generalized cosine distance metric
between x and y is the cosine distance metric between
two linearly transformed data points W x and W y
1 y1
where W = A 2 . In this sense, it is clear that learning
an optimal distance metric is equivalent to learning
an optimal linear transformation of the original high-
dimensional space. There exist various linear sub-
space learning methods that can fit into this context.
4.4 Distance Metric Learning in Euclidean Space
Linear subspace learning can be classified into two
distinct categories: unsupervised learning and su-
pervised learning. For unsupervised linear subspace
learning, principal component analysis (PCA) [15]
may be the most early developed and prevailing
technique, and when applied to speech or speaker
recognition, is known as the eigenvoice approach [32].
Other more recent unsupervised learning techniques
include the locality preserving projection (LPP) [33],
neighborhood preserving embedding (NPE) [34], etc.
All these techniques may be applied to speaker clus-
tering. However, we are most interested in supervised
learning since the goal of speaker clustering is re-
lated to classification. It is natural that we prefer a
learning technique that is discriminative rather than
generative. The most famous technique for supervised
linear subspace learning is Fisher’s LDA. LDA has
been applied to speaker clustering, and the resulting
technique is termed the fishervoice approach [35].
The term “fishervoice” is analogous to “fisherface”
in the face recognition literature, where the fisherface
approach refers to the face recognition method based
on LDA while the eigenface approach refers to the
face recognition method based on PCA.
4.5 Distance Metric Learning in Cosine Space
Most existing linear subspace learning techniques
(e.g., PCA and LDA) are implicitly based on the
Euclidean distance metric. As we mentioned ear-
lier, due to the directional scattering property of the
GMM mean supervectors, we favor the cosine dis-
tance metric over the Euclidean distance metric for
speaker clustering in the GMM mean supervector
space. Therefore, we propose to perform discriminant
analysis in the cosine distance metric space.
7
xd
SDA h-dimensional
x2
yh
x1
d-dimensional LSD
A
linear space
y2
h-dimensional
linear space
Fig. 6. The schematic illustration of SDA and LSDA.
Under two mild conditions, the nonlinear projection can
be linearized and thus SDA reduces to LSDA.
4.5.1 Linear Spherical Discriminant Analysis
We coined the phrase “spherical discriminant anal-
ysis” (SDA) to denote discriminant analysis in the
cosine distance metric space. We define a projection
from a d-dimensional linear space to an h-dimensional
hypersphere where h < d
WTx
y= (19)
kW T xk
We note that such a projection is nonlinear. How-
ever, under two mild conditions, this projection can
be linearized. One condition is that the objective
function for learning the projection only involves the
cosine distance metric. The other condition is that only
the cosine distance metric is used in the projected
space. In this case, the norm of the projected vector
y has impact on neither the objective function nor
distance computation in the projected space. Thus,
the denominator term of Equation 19 can be safely
dropped, leading to a linear projection y = W T x,
which is called “linear spherical discriminant analy-
sis” (LSDA). Figure 6 illustrates the basic ideas of SDA
and LSDA.
Formally speaking, the goal of LSDA is to seek a
linear projection W such that the average within-class
cosine similarity of the projected data is maximized
while the average between-class cosine similarity of
the projected data is minimized. Assuming that there
are c classes, the average within-class cosine similarity
is defined to be the average of the class-dependent
average cosine similarities between the projected data
vectors. It can be written in terms of the unknown
projection matrix W and original data points x
c
1X
SW = Si (20)
c i=1
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX 8

1 X yjT yk
Si = p √ same vertex set but have different similarity matrices.
|Di ||Di | yjT yj Ty
yk
yj ,yk ∈Di
k
The graph similarity preserving criterion is given by
1 X xT T
j W W xk
X (i) (p)
= p √ (21) W = arg min kf (xi , W ) − f (xj , W )k2 (sij − sij )
|Di ||Di | xT
j
W W T xj xT
k
W W T xk W
xj ,xk ∈Di i6=j
(25)
where |Di | denotes the number of data points in the where f (x, W ) is a general projection with parameters
ith class. Similarly, the average between-class cosine W . Note that the above objective function integrates
similarity is defined to be the average of the average the two aforementioned graphs through the subtrac-
cosine similarities between any two pairs of classes. tion of the similarities in the penalty graph from the
It can be likewise written in terms of W and x similarities in the intrinsic graph. One can easily see
c X c that minimizing this objective function ensures that if
1 X
SB = Smn (m 6= n) (22) the data points xi and xj are close in the sense of the
c(c − 1) m=1 n=1 similarities in S (i) and S (p) then their projections in
the low-dimensional space are close, too.
1 X yjT yk
In Equation 25, if we use a spherical projection
Smn = p √ of the form of Equation 19, we obtain the following
|Dm ||Dn | yjT yj Ty
yk k
yj ∈Dm criterion
yk ∈Dn 2
X W T xi W T xj

1 xT T
j W W xk (i) (p)
X
= p √ (23) W = arg min kW T xi k − kW T xj k (sij − sij )

|Dm ||Dn | xT W W T xj xT W W T xk W
j k i6=j
xj ∈Dm
xk ∈Dn (26)
Although there is no closed-form solution to the
where |Dm | and |Dn | denote the number of data
optimization problem of Equation 26, as shown in
points in the mth and nth classes, respectively.
[37], this problem can be solved using a steepest
The LSDA criterion is to maximize SW while min-
descent algorithm, with the gradient derived as
imizing SB , which can be written in the trace differ-
ence form P n f W T x xT f W T x xT W T (xi xTj +xj xTi ) o (i) (p)
G = 2 i6=j ij f 3 f i i + ij f 3 f j j − fi fj (sij −sij )
i j j i
W = arg max(SW − SB ) (24) (27)
W p q
where fi = T T
xi W W xi , fj = T T
xj W W xj , and
Note that there are various forms of the criterion T T
that may be adopted. We choose the trace difference fij = xi W W xj . If we expand the L2 norm term of
form, which is similar to the work of Ma et al. [36]. Equation 26, by some simple manipulations we obtain
However, we systematically solve our LSDA formu- 2
W T xi W T xj T T
 
lation in an elegant general dimensionality reduction − = 2 1 − xi W W xj
kW T xi k kW T xj k kW T xi k kW T xj k
framework known as graph embedding [20], [37]. (28)
Thus, the criterion in Equation 26 is equivalent to
4.5.2 Graph Embedding Solution to LSDA the following criterion
Graph embedding is a general framework for dimen- X xi T W W T xj (i) (p)
sionality reduction, where, an undirected weighted W = arg max q (sij −sij )
W
p
xTWWTx xTWWTx
graph, G = {X, S}, with vertex set X and similarity i6=j i i j j

matrix S, is used to characterize certain statistical or (29)

geometrical properties of a data set. A vertex xi in By comparing Equation 29 to Equations 20–24, we
X represents a data point in the high-dimensional conclude that the graph embedding criterion of Equa-
space. An entry sij in S, denoted as the weight of tion 26 is equivalent to the LSDA criterion of Equation
the edge connecting xi and xj , represents the sim- 24 if the entries of the similarity matrices S (i) and S (p)
ilarity between the two corresponding data points. are set to proper values, as follows
The purpose of graph embedding is to represent each (i) 1
vertex of the graph as a low dimensional vector that sjk ← if xj , xk ∈ Di , i = 1, ..., c
c|Di ||Di |
preserves the similarities in S. (p) 1
Graph embedding unifies most dimensionality re- sjk ← if xj ∈ Dm , xk ∈ Dn
c(c − 1)|Dm ||Dn |
duction algorithms into a general framework. For a m, n = 1, ..., c, m 6= n (30)
specific dimensionality reduction algorithm, we often
use two graphs: the intrinsic graph {X, S (i) }, which That is, by assigning appropriate values to the
characterizes the data properties that the algorithm weights of the intrinsic and penalty graphs, our LSDA
aims to preserve, and the penalty graph {X, S (p) }, formulation can be systematically solved within the
which characterizes the data properties that the al- elegant graph embedding general dimensionality re-
gorithm aims to avoid. These two graphs share the duction framework.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX 9

5 C LUSTERING TABLE 1
Experiment settings.
The last stage of the speaker clustering pipeline is
clustering. We are interested in conventional cluster- Test set Indep. training set
ing techniques which can be applied to the GMM #speaker 630 498
#utterance 19024 18327
mean supervector space. We mainly focus on two #utt/spk (ave.) 30 ∼ 40 30 ∼ 40
traditional classes of algorithms. One is “flat” clus- utt duration (ave.) 3 ∼ 4s 3 ∼ 4s
tering – clustering by partitioning the data space.
The other is hierarchical clustering, where we try
not to construct a partition but a nested hierarchy of Our experiments are based on a test set of 630
partitions. In most of real-world applications, one of speakers and 19024 utterances extracted from the
these two classes of algorithms is employed. GALE database. In order to implement our partially
The representative flat clustering algorithm is k- supervised speaker clustering strategies at the various
means whose objective is to partition the data space stages of the speaker clustering pipeline, we employ
in such a way that the total intra-cluster variance an independent training set, which was also extracted
is minimized. It iterates between a cluster assigning from the GALE database. Note that the test set and
step and a mean updating step until convergence. the independent training set were chosen in such a
Spherical k-means [38] is an extension of k-means that way that speakers in the independent training set
is based on the cosine distance metric. do not exist in the test set. Table 1 lists the detailed
The representative hierarchical clustering algorithm experiment settings.
is agglomerative clustering [15] whose objective is to To guarantee a statistically significant performance
obtain a complete hierarchy of clusters in the form comparison, we carry out our experiments as follows.
of a dendrogram. The algorithm adopts a bottom-up 1. A case is an experiment associated with a specific
strategy. First, it starts with each data point being a number of test speakers, namely 2, 5, 10, 20, 50,
cluster. Then, it checks which clusters are the closest and 100, respectively;
and merges them into a new cluster. As the algorithm 2. For each case, this number of speakers are drawn
proceeds, it always merges the two closest clusters randomly from the test set, and all the utterances
until there is only one single cluster left. from the selected speakers are used in the exper-
A remarkable question related to agglomerative iment;
clustering is how to determine which clusters are the 3. For each case, 10 independent trials are run, each
closest. There exist several methods that measure the of which involves a random draw of the test
distance between two clusters, for instance, the single speakers;
linkage, complete linkage, average linkage, “ward” 4. For each case, the mean and standard error of the
linkage, and so on [19], [39]. We empirically discover clustering results over the 10 independent trials
that the “ward” linkage yields the best performance are reported.
for speaker clustering. The “ward” linkage is a func-
tion that specifies the distance between two clusters
X and Y by the increase in the error sum of squares 6.2 Performance Evaluation Metrics
(ESS) after the merging of X and Y (Z = X ∪ Y ):
We report our experiment results based on two perfor-
1 X X mance evaluation metrics, namely the clustering accu-
x̄ = x, ESS(X) = |x − x̄|2
|X| racy and the normalized mutual information (NMI)
x∈X x∈X
d(X, Y ) = ESS(Z) − [ESS(X) + ESS(Y )] (31) [41]. These two metrics are standard for evaluating
(general) data clustering results [42]. The clustering
accuracy is given by
6 E XPERIMENTS
N
1 X
6.1 Experiment Setup and Protocol r= [ci = li ] (32)
N i=1
We conduct extensive speaker clustering experiments
on the GALE Mandarin database [40]. The GALE where N denotes the number of test utterances, ci is
database contains about 1900 hours of broadcast news the cluster label of the ith utterance returned by the
speech data collected from various Mandarin TV pro- algorithm, li is the true cluster label, and [v] is an
grams at various times. The waveforms were sampled indication function which returns 1 if v is true and 0
at 16 KHz and quantized at 16 bits per sample, otherwise.
and were automatically over-segmented into short The NMI is another popular, information-
utterances using the BIC criterion, with each utterance theoretically interpreted metric given by
being as pure as possible, namely, each utterance
being from a single speaker. A random sample of the I(C, L)
r= (33)
results were further verified by human listeners. [H(C) + H(L)]/2
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX
where I(C, L) is the mutual information
T transformed acoustic features.
XX \ |ci lj |
I(C, L) = |ci lj | log
i j
|ci ||lj |
and H(C) and H(L) are the entropy
X |ci | |ci | X |li |
|li |
H(C) = − log , H(L) = − log
N N N N
i i
(35)
T
In the above formulas, |ci |, |lj | and |ci lj | areTthe
number of utterances from speaker ci , lj , and ci lj ,
respectively.
The above two metrics are used for utterance-based
evaluations. We extend them to frame-based evalua-
tions by simply replacing the number of utterances in
the above formulas with the corresponding number of
frames. This allows us to investigate how the duration
of an utterance affects the clustering performance.
6.3 Experiment Results and Discussions
Our main experiment results are presented in Tables
2–5. Specifically, we conduct speaker clustering 1) in
the GMM mean supervector space with the Euclidean
and cosine distance metrics; 2) in the PCA subspace
with the Euclidean distance metric(i.e. the eigenvoice
approach); 3) in the LPP subspace with the Euclidean
distance metric; 4) in the NPE subspace with the
Euclidean distance metric; 5) in the LDA subspace
with the Euclidean distance metric (i.e. the fishervoice
approach); 6) in the kernel-LDA nonlinear manifold
with the Euclidean distance metric; 7) in the LSDA
subspace with the cosine distance metric. In each
experiment, we utilize both k-means (or spherical k-
means) and agglomerative clustering. In order to com-
pare our methods to the traditional “bag of acoustic
features” methods, we employ the ”Gaussian+BIC”
method as the baseline. The experiment results are
presented in four forms – utterance-based cluster-
ing accuracies (Table 2), utterance-based NMIs (Table
3), frame-based clustering accuracies (Table 4), and
frame-based NMIs (Table 5). For each case, we present
the mean of the results over 10 trials√ as well as the
standard error of the mean, se = s/ n, in parentheses,
where s is the standard deviation of the results and
n the number of trials (10). In all tables, Orig stands
for the original GMM mean supervector space, k for
k-means, and h for hierarchical clustering.
Additionally, we compare the proposed LDA trans-
formed acoustic features with the acoustic features
traditionally augmented with the first and second
order derivatives. Specifically, 13 basic PLP features
augmented by their first and second order derivatives
form a 39-dimensional traditional acoustic feature
vector. Table 6 gives a comparison of the results (clus-
tering accuracies) of both kinds of acoustic features on
speaker clustering under the same clustering condi-
tions. In this table, “traditional” stands for traditional
10
acoustic features, and “LDA” for the proposed LDA
(34) Finally, to further demonstrate the statistical signifi-
cance of our performance improvements, we perform
a paired t-test (at the default 5% significance level) be-
tween our proposed method and the ”Gaussian+BIC”
method for every case. The p-values of all tests turn
out to be so close to zero that Matlab rounds most of
them to zero. This clearly indicates that the results of
our method are significantly different from the results
of the ”Gaussian+BIC” method.
Next, we perform a step-by-step verification and
discussion of the improvements that our partially
supervised speaker clustering strategies have made to
the experiment results, as follows:
In Tables 2–5, our experiment results show that our
speaker clustering methods based on the GMM mean
supervector representation and vector-based distance
metrics significantly outperform traditional speaker
clustering methods based on the “bag of acoustic fea-
tures” representation and statistical model based dis-
tance metric such as the BIC (Rows: Orig vs. Baseline).
It is worth mentioning that in the “Gaussian+BIC”
method, if we utilize agglomerative clustering, the
computational load can become prohibitive as the
number of speakers gets larger. This is because at
each iteration, along with a new cluster being formed
by merging the two closest ones, a new statistical
model representing the new cluster has to be re-
trained, and the distance between the new model
and any other model updated. On the contrary, ag-
glomerative clustering can be done very efficiently
in the GMM mean supervector space by using the
“ward” linkage. In addition, in the GMM mean su-
pervector space, although speaker clustering based
on the Euclidean distance metric achieves reasonably
good results, the cosine distance metric consistently
outperforms the Euclidean distance metric (Rows:
COS/Orig vs. EU/Orig), thanks to the directional
scattering property of the GMM mean supervectors,
which is discussed in Section 3.2.3.
In Table 6, our experiment results show that the
LDA transformed acoustic features consistently out-
perform the traditional acoustic features by 1%-3%.
The proposed speaker-discriminative acoustic feature
transformation implements one of our partially super-
vised speaker clustering strategies, which can provide
a better frontend to speaker clustering as compared to
traditional ones.
Due to the difficulty of handling high-dimensional
data, and in order to alleviate the “curse of di-
mensionality”, linear subspace learning methods are
used to derive various subspaces in which the final
speaker clustering is performed. From the experiment
results in Tables 2–5, we see that the unsupervised
methods (i.e. PCA/eigenface approach, LPP, NPE)
more or less improve the performance, but significant
improvements of the performance are achieved by
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX 11

TABLE 2
Performance comparison of speaker clustering based on utterance-based clustering accuracies.

2 spk 5 spk 10 spk 20 spk 50 spk 100 spk

×100% ∼ 60 utt ∼ 150 utt ∼ 300 utt ∼ 600 utt ∼ 1500 utt ∼ 3000 utt
k h k h k h k h k h k h
Orig 94.7 96.0 81.6 85.0 77.3 82.6 70.5 78.1 58.4 69.4 47.2 57.7
(0.28) (0.25) (0.84) (0.81) (1.01) (0.98) (1.07) (1.02) (1.31) (1.24) (1.47) (1.45)
EU PCA 96.6 96.2 84.8 85.5 81.3 82.9 78.5 79.3 69.7 69.9 59.4 58.5
(0.26) (0.26) (0.77) (0.74) (0.82) (0.84) (1.02) (1.00) (1.26) (1.28) (1.43) (1.45)
LPP 98.3 98.1 92.5 92.8 87.9 88.6 85.6 84.7 77.4 77.0 70.1 69.8
(0.19) (0.19) (0.39) (0.38) (0.69) (0.70) (0.75) (0.76) (1.09) (1.08) (1.27) (1.28)
NPE 97.4 97.0 84.3 85.0 83.2 83.9 78.7 79.3 70.4 69.8 58.1 58.3
(0.20) (0.18) (0.78) (0.74) (0.81) (0.82) (1.01) (1.00) (1.25) (1.23) (1.39) (1.45)
LDA 98.3 98.4 94.1 94.0 89.9 90.8 87.2 86.6 79.5 79.6 73.1 72.3
(0.20) (0.18) (0.32) (0.32) (0.49) (0.50) (0.83) (0.84) (1.01) (0.97) (1.19) (1.19)
KLDA 98.2 98.1 93.0 92.8 87.3 87.5 84.6 83.9 75.2 75.3 70.4 70.1
(0.18) (0.19) (0.38) (0.37) (0.58) (0.57) (0.79) (0.82) (1.05) (1.04) (1.26) (1.24)
Orig 99.0 99.1 88.3 90.7 84.1 86.5 80.6 82.2 74.7 77.7 66.4 69.3
COS
(0.13) (0.13) (0.51) (0.51) (0.73) (0.72) (0.99) (0.98) (1.07) (1.09) (1.30) (1.26)
LSDA 99.2 99.1 97.8 98.0 95.0 94.7 90.3 90.0 84.3 85.9 77.9 79.4
(0.12) (0.12) (0.19) (0.20) (0.31) (0.32) (0.51) (0.50) (0.77) (0.77) (1.01) (1.00)
Baseline (Gaussian+BIC) 82.5 83.8 71.6 72.0 58.3 60.5 53.1 52.7 43.2 44.1 35.0 37.4
(0.84) (0.85) (1.19) (1.21) (1.44) (1.42) (1.59) (1.47) (1.72) (1.69) (1.87) (1.91)

TABLE 3
Performance comparison of speaker clustering based on utterance-based NMIs.

2 spk 5 spk 10 spk 20 spk 50 spk 100 spk

×100% ∼ 60 utt ∼ 150 utt ∼ 300 utt ∼ 600 utt ∼ 1500 utt ∼ 3000 utt
k h k h k h k h k h k h
Orig 91.9 93.7 79.0 82.7 74.4 80.3 67.7 75.3 55.9 66.4 44.3 55.3
(0.41) (0.37) (0.96) (0.97) (1.11) (1.05) (1.16) (1.09) (1.27) (1.46) (1.45) (1.36)
EU PCA 93.9 93.5 82.7 83.0 78.9 79.9 76.2 76.7 66.9 67.6 56.9 56.3
(0.37) (0.37) (0.79) (0.79) (0.99) (0.96) (1.08) (1.05) (1.31) (1.32) (1.45) (1.47)
LPP 96.1 95.4 89.9 89.9 85.1 86.2 83.1 82.0 74.8 74.0 67.2 67.3
(0.25) (0.25) (0.61) (0.61) (0.83) (0.84) (0.77) (0.78) (1.09) (1.09) (1.32) (1.27)
NPE 95.3 94.3 81.6 82.3 81.1 81.3 76.0 77.1 67.8 67.0 55.7 55.9
(0.26) (0.25) (0.80) (0.79) (0.90) (0.88) (1.06) (1.03) (1.24) (1.35) (1.43) (1.46)
LDA 96.2 95.8 92.0 91.5 87.5 88.2 85.0 84.0 77.4 77.6 71.1 70.2
(0.25) (0.25) (0.41) (0.41) (0.64) (0.63) (0.83) (0.83) (1.04) (1.09) (1.27) (1.28)
KLDA 96.0 95.6 91.8 91.7 86.0 86.2 83.3 82.9 74.1 74.0 66.9 67.2
(0.25) (0.26) (0.40) (0.43) (0.63) (0.63) (0.90) (0.88) (1.11) (1.09) (1.30) (1.33)
Orig 96.6 96.8 86.0 88.0 81.4 83.6 78.2 79.2 72.1 75.2 63.8 67.1
COS
(0.25) (0.25) (0.60) (0.57) (0.78) (0.79) (1.02) (1.01) (1.12) (1.11) (1.32) (1.27)
LSDA 97.2 97.0 95.3 95.3 92.2 92.6 87.3 87.9 82.3 83.2 75.3 77.4
(0.16) (0.16) (0.22) (0.23) (0.38) (0.38) (0.60) (0.61) (0.85) (0.80) (1.07) (1.08)
Baseline (Gaussian+BIC) 79.9 81.5 69.3 69.5 56.2 58.4 50.9 50.4 41.0 41.5 32.2 35.2
(0.91) (0.90) (1.26) (1.28) (1.46) (1.45) (1.62) (1.58) (1.77) (1.81) (1.93) (1.93)

supervised methods (i.e. LDA/fishervoice approach,
LSDA). Notably, our proposed LSDA algorithm leads
to the state-of-the-art speaker clustering performance.
This clearly indicate that LSDA benefits significantly
from the directional scattering property of the data in
the GMM mean supervector space.
Owing to the non-linearity of the data, a kernel
discriminant analysis method may be preferable over
LDA. In Tables 2–5, we present the results obtained
via kernel LDA [43] where the affinities used in LSDA
are used as the Gram matrix in kernel LDA. Our
experiment results show that kernel LDA is com-
parable to LDA when applied to smaller numbers
of test speakers (2,5,10). When the number of test
speakers gets larger, kernel LDA performs slightly
worse than LDA (20,50,100). The reason behind this
observation may be explained as follows: kernel LDA
is based on an implicit nonlinear mapping from the
original data space to a much higher dimensional
feature space where linear projections are found. As
the number of clusters increases, linear separation
in the higher dimensional feature space increasingly
comes at the cost of very small margins. In order to
avoid small margins, the kernel LDA algorithm may
choose suboptimal clustering strategies.
Also noted is that the frame-based performance
is better than the utterance-based performance. The
rationale behind this is that longer utterances tend to
be more often correctly classified than shorter utter-
ances, which is reasonable because longer utterances
can provide more speaker-discriminative information
than shorter ones.
In every experiment, we employ two clustering
algorithms, namely k-means and agglomerative clus-
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX 12

TABLE 4
Performance comparison of speaker clustering based on frame-based clustering accuracies.

2 spk 5 spk 10 spk 20 spk 50 spk 100 spk

×100% ∼ 60 utt ∼ 150 utt ∼ 300 utt ∼ 600 utt ∼ 1500 utt ∼ 3000 utt
k h k h k h k h k h k h
Orig 95.9 97.2 83.4 87.7 80.6 85.8 74.4 82.5 63.3 74.7 53.5 64.4
(0.24) (0.22) (0.77) (0.75) (0.91) (0.78) (1.01) (0.99) (1.19) (1.17) (1.38) (1.31)
EU PCA 97.4 97.7 86.2 86.6 84.4 85.7 81.9 83.1 74.5 74.8 66.1 67.2
(0.22) (0.23) (0.70) (0.72) (0.73) (0.74) (0.97) (0.98) (1.19) (1.20) (1.38) (1.30)
LPP 99.1 99.0 94.7 94.8 90.5 90.6 88.7 88.9 82.6 82.3 76.7 76.9
(0.13) (0.13) (0.31) (0.31) (0.56) (0.56) (0.60) (0.58) (0.97) (0.89) (1.19) (1.20)
NPE 98.8 98.5 87.1 87.7 86.1 86.7 82.5 83.6 74.9 74.7 64.5 64.7
(0.16) (0.19) (0.70) (0.68) (0.74) (0.77) (0.93) (0.96) (1.20) (1.20) (1.36) (1.32)
LDA 99.2 99.4 96.2 96.0 92.5 92.6 91.0 90.8 84.4 84.3 80.2 81.7
(0.13) (0.12) (0.26) (0.26) (0.38) (0.40) (0.63) (0.64) (0.89) (0.87) (1.06) (1.13)
KLDA 98.6 98.4 94.8 94.6 90.1 90.4 88.0 87.6 80.3 80.1 76.7 77.4
(0.19) (0.19) (0.35) (0.34) (0.50) (0.54) (0.64) (0.63) (1.04) (1.02) (1.26) (1.20)
Orig 99.4 99.5 89.9 92.5 87.3 88.1 85.1 86.5 79.3 81.2 72.3 74.9
COS
(0.13) (0.13) (0.45) (0.42) (0.73) (0.70) (0.89) (0.86) (1.08) (0.99) (1.21) (1.20)
LSDA 99.7 99.5 98.9 99.2 97.1 96.8 92.5 92.1 87.0 88.2 82.6 83.2
(0.10) (0.09) (0.17) (0.15) (0.22) (0.25) (0.44) (0.44) (0.68) (0.71) (0.93) (0.96)
Baseline (Gaussian+BIC) 84.7 85.1 72.9 73.4 61.6 63.2 57.7 56.9 48.6 49.4 42.7 44.1
(0.77) (0.74) (1.17) (1.13) (1.43) (1.38) (1.46) (1.40) (1.63) (1.57) (1.83) (1.92)

TABLE 5
Performance comparison of speaker clustering based on frame-based NMIs.

2 spk 5 spk 10 spk 20 spk 50 spk 100 spk

×100% ∼ 60 utt ∼ 150 utt ∼ 300 utt ∼ 600 utt ∼ 1500 utt ∼ 3000 utt
k h k h k h k h k h k h
Orig 93.4 95.0 81.2 84.9 78.4 80.1 71.5 80.2 60.7 72.4 51.0 61.6
(0.36) (0.31) (0.95) (0.90) (1.00) (0.93) (1.08) (1.01) (1.27) (1.38) (1.39) (1.36)
EU PCA 95.0 95.6 83.5 84.5 81.9 82.7 78.9 80.1 72.2 72.1 63.4 65.1
(0.31) (0.30) (0.77) (0.76) (0.87) (0.88) (0.98) (1.02) (1.24) (1.35) (1.45) (1.40)
LPP 96.8 96.8 92.4 92.3 87.6 88.2 86.2 86.9 80.1 80.1 74.0 74.6
(0.24) (0.26) (0.47) (0.52) (0.74) (0.67) (0.78) (0.79) (1.04) (0.98) (1.16) (1.22)
NPE 96.2 95.7 84.7 85.0 83.5 84.6 80.2 80.8 72.0 72.5 61.8 62.0
(0.25) (0.29) (0.78) (0.76) (0.84) (0.83) (0.91) (1.01) (1.26) (1.23) (1.42) (1.45)
LDA 97.0 96.5 93.6 93.2 89.5 90.4 88.7 88.0 81.6 81.7 77.2 79.2
(0.22) (0.23) (0.37) (0.37) (0.59) (0.57) (0.72) (0.76) (1.04) (0.99) (1.13) (1.16)
KLDA 96.6 96.1 93.0 92.8 87.1 88.0 85.2 84.9 77.1 77.1 74.3 75.4
(0.25) (0.25) (0.38) (0.38) (0.58) (0.63) (0.86) (0.87) (1.06) (1.09) (1.29) (1.23)
Orig 97.1 97.0 87.8 89.9 84.7 85.8 82.2 84.4 76.8 78.7 69.5 72.6
COS
(0.22) (0.21) (0.61) (0.58) (0.68) (0.68) (0.93) (0.95) (1.06) (1.06) (1.30) (1.25)
LSDA 97.4 97.3 96.1 96.5 94.9 94.5 89.9 89.4 84.5 86.2 79.8 81.0
(0.16) (0.15) (0.22) (0.22) (0.36) (0.37) (0.59) (0.62) (0.83) (0.82) (1.07) (1.10)
Baseline (Gaussian+BIC) 82.5 82.8 70.2 71.3 59.4 60.7 55.4 54.1 46.3 47.0 40.2 41.7
(0.85) (0.82) (1.23) (1.23) (1.39) (1.46) (1.52) (1.54) (1.71) (1.64) (1.78) (1.85)

tering. Although there are pros and cons in each
algorithm, we observe that in the same subspace, the
speaker clustering performance of the two algorithms
is comparable. However, k-means is sensitive to ini-
tialization, which means the results across multiple
runs may not be identical. Thus, we need to restart
the k-means algorithm many times (e.g., 50) with a
different initialization at each time, and record the best
result. Therefore, k-means is normally much slower
than agglomerative clustering. On the other hand,
agglomerative clustering with the “ward” linkage
method runs very fast. For the case of 100 speakers
(about 3000 utterances), it takes our Matlab program
less than one minute to complete the job on a Linux
machine with a mainstream configuration.
An issue that is not addressed in this paper is the
determination of the number of speakers. Automat-
ically finding the number of clusters in a dataset in
a completely unsupervised manner is still an open
research problem. Many speaker diarization systems
deal with this problem through hierarchical clustering
using a BIC-based stopping criterion [11]. A similar
method could have been used to determine the num-
ber of speakers automatically in our paper. However,
the exact number of speakers is not accurately com-
puted by this simple method. In general, clustering
results may vary dramatically for different numbers
of speakers determined. In order to eliminate the
influence of the number of speakers and single out
the extent to which the proposed partially supervised
strategies may improve the speaker clustering perfor-
mance, we assume that the number of test speakers
is known a priori, and defer the investigation of this
issue to our future work.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX 13

TABLE 6
Performance comparison of the proposed LDA transformed acoustic features with the traditional acoustic
features based on clustering accuracies.

2 spk 5 spk 10 spk 20 spk 50 spk 100 spk

×100% ∼ 60 utt ∼ 150 utt ∼ 300 utt ∼ 600 utt ∼ 1500 utt ∼ 3000 utt
k h k h k h k h k h k h
Traditional 93.5 94.7 80.1 83.7 76.2 81.3 69.0 76.6 57.5 68.4 46.0 56.6
EU
(0.28) (0.28) (0.85) (0.85) (1.07) (1.03) (1.10) (1.04) (1.29) (1.27) (1.55) (1.47)
LDA 94.7 96.0 81.6 85.0 77.3 82.6 70.5 78.1 58.4 69.4 47.2 57.7
(0.29) (0.25) (0.82) (0.79) (1.03) (0.99) (1.05) (1.04) (1.31) (1.27) (1.51) (1.41)
Traditional 97.4 97.7 87.3 89.4 82.9 85.4 79.6 81.2 73.5 76.4 64.9 67.8
COS
(0.16) (0.16) (0.56) (0.58) (0.77) (0.80) (1.02) (1.02) (1.10) (1.10) (1.34) (1.30)
LDA 99.0 99.1 88.3 90.7 84.1 86.5 80.6 82.2 74.7 77.7 66.4 69.3
(0.13) (0.12) (0.49) (0.51) (0.71) (0.72) (1.00) (1.00) (1.06) (1.09) (1.25) (1.28)

7 C ONCLUSIONS T. J. Watson Research Center for their continuous

In this paper, we propose the conceptually new idea
of partially supervised speaker clustering and of-
fer a complete treatment of the speaker clustering
problem. By means of an independent training data
set, our strategies are to encode the prior knowl-
edge of speakers in general at the various stages
of the speaker clustering pipeline via 1) learning
a speaker-discriminative acoustic feature transforma-
tion, 2) learning a universal speaker prior model,
and 3) learning a discriminative speaker subspace, or
equivalently, a speaker-discriminative distance metric.
We discover the directional scattering property of the
GMM mean supervector representation of utterances
and advocate the use of the cosine distance metric
instead of the Euclidean distance metric. We propose
to perform discriminant analysis based on the cosine
distance metric, leading to a novel distance metric
learning algorithm – LSDA. We show that the pro-
posed LSDA formulation can be systematically solved
within the elegant graph embedding framework. Our
speaker clustering experiments indicate that 1) our
speaker clustering methods based on the GMM mean
supervector representation and vector-based distance
metrics outperform traditional methods based on the
“bag of acoustic features” representation and statisti-
cal model based distance metrics, 2) our advocated
use of the cosine distance metric yields consistent
increases in the speaker clustering performance as
compared to the commonly used Euclidean distance
metric, thanks to the directional scattering property
of the GMM mean supervectors discovered, 3) our
partially supervised speaker clustering concept and
strategies significantly improve the speaker clustering
performance over the baselines, and 4) our proposed
LSDA algorithm further leads to the state-of-the-art
speaker clustering performance.
ACKNOWLEDGMENTS
This work was supported in part by the DARPA
contract HR0011-06-2-0001 and in part by the NSF
Grant 08-03219. The authors would like to thank Dr.
Lidia Mangu and Dr. Michael Picheny at the IBM
encouragement and support.
R EFERENCES
[1] M. Lew, N. Sebe, C. Djeraba, R. Jain. Content-based multimedia
information retrieval: State of the art and challenges. ACM
Trans. on Multimedia Computing, Communications, and Ap-
plications, 2(1):1-19, 2006.
[2] Y. Gong, W. Xu. Machine Learning for Multimedia Content
Analysis. Springer, 2007.
[3] H. Jin, F. Kubala, and R. Schwartz, “Automatic speaker cluster-
ing,” Proc. DARPA Speech Recognition Workshop’97.
[4] S. Chen, and P. Gopalakrishnan, “Clustering via the Bayesian
information criterion with applications in speech recognition,”
Proc. ICASSP’98.
[5] D. Reynolds, E. Singer, B. Carson, G. O’Leary, J. McLaughlin,
and M. Zissman, “Blind clustering of speech utterances based
on speaker and language characteristics,” Proc. ICSLP’98.
[6] A. Solomonoff, A. Mielke, M. Schmidt, H. Gish. Clustering
speakers by their voices. Proc. ICASSP, 2:757-760, 1998.
[7] R. Faltlhauser, and G. Ruske, “Robust speaker clustering in
eigenspace,” Proc. ASRU’01.
[8] W. Tsai, S. Cheng, and H. Wang, “Speaker clustering of speech
utterances using a voice characteristic reference space,” Proc.
ICSLP’04.
[9] M. Ben, M. Betser, F. Bimbot, and G. Gravier. Speaker diariza-
tion using bottom-up clustering based on a parameter-derived
distance between adapted GMMs. Proc. ICSLP, 2004.
[10] Barras, C., Zhu, X., Meignier, S. and Gauvain, J.L., “Multistage
speaker diarization of broadcast news,” IEEE Trans. ASLP. vol.
14 no. 5, pp. 1505-1512, Sept. 2006.
[11] S. Tranter, D. Reynolds. An Overview of Automatic Speaker
Diarization Systems. IEEE T-ASLP, 14(5):1557-565, 2006.
[12] C. Wooters and M. Huijbregts, “The ICSI RT07s Speaker
Diarization System,” Lecture Notes in Computer Science, 2007.
[13] P. Kenny. Bayesian analysis of speaker diarization with eigen-
voice priors. CRIM, 2008.
[14] D. Reynolds, P. Kenny, F. Castaldo. A study of new approaches
to speaker diarization. Interspeech, 2009.
[15] R. Duda, P. Hart, D. Stork. Pattern Classification (2nd ed.).
Wiley Interscience, 2000.
[16] W. Campbell,D. Sturim, D. Reynolds. Support vector machines
using GMM supervectors for speaker verification. Signal Pro-
cessing Letters 13(5):308-311, 2006.
[17] D. Reynolds, T. Quatieri, R. Dunn. Speaker Verification using
Adapted Gaussian Mixture Models. Digital Signal Processing,
10:19-41, 2000.
[18] J. MacQueen. Some Methods for classification and Analysis of
Multivariate Observations. Proc. 5th Berkeley Symposium on
Math. Stat. and Prob. 1:281-297, 1967.
[19] A. Jain, R. Dubes. Algorithms for clustering data. Prentice-
Hall, Inc., 1988.
[20] S. Yan, D. Xu, B. Zhang, H. Zhang, Q. Yang, S. Lin. Graph
Embedding and Extensions: A General Framework for Dimen-
sionality Reduction. IEEE T-PAMI 29(1):40-51, 2007.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. X, XXX 20XX
[21] H. Hermansky. Perceptual linear predictive (PLP) analysis of
speech. J. Acoust. Soc. Amer., 87:1738-1752, 1990.
[22] G. Fant. Acoustic Theory of Speech Production. Mouton De
Gruyter, 1970.
[23] P. Jain, H. Hermansky. Improved mean and variance normal-
ization for robust speech recognition. Proc. ICASSP, 2001.
[24] A. Dempster, N. Laird, D. Rubin. Maximum likelihood from
incomplete data via the EM algorithm. J. Roy. Stat. Soc. B,
39(1):1-38, 1977.
[25] C. Bishop. Pattern Recognition and Machine Learning.
Springer, 2006.
[26] P. Kenny, G. Boulianne, and P. Dumouchel. Eigenvoice mod-
eling with sparse training data. IEEE Trans. Speech and Audio
Processing, 13(3):345–354, 2005.
[27] Douglas A. Reynolds and Richard C. Rose. Robust Text-
Independent Speaker Identification Using Gaussian Mixture
Speaker Models. IEEE Trans. Speech and Audio Processing,
3(1):72–83, 1995.
[28] J-L. Gauvain, C.-H. Lee. Maximum a posteriori estimation for
multivariate Gaussian mixture observations of Markov chains.
IEEE Trans. Speech and Audio Processing, 2:291–298, 2004.
[29] G. Schwarz. Estimating the dimension of a model. Annals of
Statistics 6(2):461-464, 1978.
[30] X. Miro. Robust Speaker Diarization for Meetings. PhD thesis,
Universitat Politecnica de Catalunya, 2006.
[31] P. Mahalanobis. On the generalised distance in statistics. Na-
tional Institute of Sciences of India 2(1):49-55, 1936.
[32] R. Kuhn, P. Nguyen, J. Junqua, L. Goldwasser, N. Niedziel-
ski, S. Fincke, K. Field, M. Contolini. Eigenvoices for speaker
adaptation. Proc. ICSLP, 1771-1774, 1998.
[33] X. He, P. Niyogi. Locality Preserving Projections. NIPS 2003.
[34] X. He, D. Cai, S. Yan, H. Zhang. Neighborhood Preserving
Embedding. Proc. ICCV, 2005.
[35] S. Chu, H. Tang, T. Huang. Fishervoice and Semi-supervised
Speaker Clustering. Proc. ICASSP, 2009.
[36] Y. Ma, S. Lao, E. Takikawa, M. Kawade. Discriminant Analysis
in Correlation Similarity Measure Space. Proc. ICML, 227:577-
84, 2007.
[37] Y. Fu, S. Yan, T. Huang. Correlation Metric for Generalized
Feature Extraction. IEEE T-PAMI 30(12):2229-235, 2008.
[38] I. Dhillon, D. Modha. Concept decompositions for large sparse
text data using clustering. Machine Learning, 42(1):143-75, 2001.
[39] A. Jain. Data Clustering: 50 Years Beyond K-Means. Technical
Report TR-CSE-09-11. Under review by the Pattern Recognition
Letters.
[40] S. Chu, H. Kuo, L. Mangu, Y. Liu, Y. Qin, Q. Shi. Recent
advances in the IBM GALE mandarin transcription system.
Proc. ICASSP, 2008.
[41] P. Viola, W. Wells III. Alignment by maximization of mutual
information. Int. J. Comput. Vision, 24(2):137-154, 1997.
[42] J. Moore, E.-H. Han, D. Boley, M. Gini, R. Gross, K. Hastings,
G. Karypis, V. Kumar, B. Mobasher. Web Page Categorization
and Feature Selection Using Association Rule and Principal
Component Clustering. Workshop on Information Technologies
and Systems, 1997.
[43] S. Mika, G. Ratsch, J. Weston, B. Scholkopf, K. Mullers. Fisher
discriminant analysis with kernels. IEEE Neural Networks for
Signal Processing Workshop, pp. 41-48, 1999.
Hao Tang received the Ph.D. degree in elec-
trical engineering from the University of Illi-
nois at Urbana-Champaign in 2010, the M.S.
degree in electrical engineering from Rut-
gers University in 2005, the M.E. and B.E.
degrees, both in electrical engineering, from
the University of Science and Technology of
China, in 2003 and 1998, respectively. Since
2010, he has been with HP Labs. His broad
research interests include statistical pattern
recognition, machine learning, computer vi-
sion, speech and multimedia signal processing.
View publication stats
14
Stephen Mingyu Chu was born in Bei-
jing, China, in 1970. He studied Physics
at Peking University before coming to the
United States, where he continued his ed-
ucation and received the M.S. degree and
the Ph.D. degree in Electrical Engineering
from the University of Illinois at Urbana-
Champaign, in 1999 and 2003, respectively.
Since May, 2003, he has been with the
speech group at the IBM T. J. Watson Re-
search Center. His research interests include
speech recognition, multimodal signal processing, graphical models,
and machine learning.
Mark Hasegawa-Johnson (M’97-SM’04) re-
ceived the S.B., S.M., and Ph.D. degrees in
electrical engineering and computer science
from MIT in 1989, 1989, and 1996 respec-
tively. From 1986-1990 he was a speech cod-
ing engineer, first at Motorola Labs, Schaum-
burg, IL, then at Fujitsu Laboratories Lim-
ited, Kawasaki, Japan. From 1996-9 he was
a post-doctoral fellow at the Speech Pro-
cessing and Auditory Physiology Lab, UCLA,
funded by the NIH, and by the 19th annual
F.V. Hunt Post-Doctoral Fellowship of the Acoustical Society of Amer-
ica. Since 1999, Prof. Hasegawa-Johnson has been on the faculty of
the University of Illinois. He is currently an Associate Professor in the
Department of Electrical and Computer Engineering, with Affiliate
appointments in Speech and Hearing Sciences, Computer Science,
and Linguistics, and with a research appointment at the Beckman
Institute for Advanced Science and Technology.
Thomas Huang (S’61-M’63-SM’76-F’79-
LF’01) received the B.S. degree in electrical
engineering from National Taiwan University,
Taipei, Taiwan, China, and the M.S. and
Sc.D. degrees in electrical engineering from
the Massachusetts Institute of Technology
(MIT), Cambridge. He was at the faculty of
the Department of Electrical Engineering at
MIT from 1963 to 1973, and at the faculty
of the School of Electrical Engineering and
Director of its Laboratory for Information
and Signal Processing at Purdue University from 1973 to 1980.
In 1980, he joined the University of Illinois at Urbana-Champaign,
where he is now William L. Everitt Distinguished Professor of
Electrical and Computer Engineering, Research Professor at the
Coordinated Science Laboratory, Head of the Image Formation
and Processing Group at the Beckman Institute for Advanced
Science and Technology, and Co-chair of the Institute’s major
research theme: human-computer intelligent interaction. Dr. Huang’s
professional interests lie in the broad area of information technology,
especially the transmission and processing of multidimensional
signals. He is a Member of the National Academy of Engineering,
a Foreign Member of the Chinese Academies of Engineering and
Science, and a Fellow of the International Association of Pattern
Recognition, IEEE, and the Optical Society of America.