IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 21, NO.
11, NOVEMBER 2013 2451
Unsupervised Spoken Language Understanding for a
Multi-Domain Dialog System
Donghyeon Lee, Minwoo Jeong, Kyungduk Kim, Seonghan Ryu, and Gary Geunbae Lee, Member, IEEE
Abstract—This paper proposes an unsupervised spoken lan-
guage understanding (SLU) framework for a multi-domain dialog
system. Our unsupervised SLU framework applies a non-para-
metric Bayesian approach to dialog acts, intents and slot entities,
which are the components of a semantic frame. The proposed
approach reduces the human effort necessary to obtain a se-
mantically annotated corpus for dialog system development. In
this study, we analyze clustering results using various evaluation
metrics for four dialog corpora. We also introduce a multi-domain
dialog system that uses the unsupervised SLU framework. We
argue that our unsupervised approach can help overcome the
annotation acquisition bottleneck in developing dialog systems.
To verify this claim, we report a dialog system evaluation, in
which our method achieves competitive results in comparison
with a system that uses a manually annotated corpus. In addition,
we conducted several experiments to explore the effect of our
approach on reducing development costs. The results show that
our approach be helpful for the rapid development of a prototype
system and reducing the overall development costs.
Index Terms—Dialog system, spoken language understanding,
unsupervised learning.
I. INTRODUCTION
M UCH research has been conducted to develop spoken
dialog systems (SDSs) that provide a natural and effec-
tive interface between humans and machines [1]–[5]. In most
cases, SDSs have been implemented for a specific domain, e.g.,
weather [6], travel [7] and vehicle navigation [8]–[10]. In SDSs,
a spoken language understanding (SLU) module is one of the
key components that aims to fill domain-specific semantic frame
slots from input utterances.
Recently, multi-domain SDSs have been employed to support
a wide range of tasks [11]–[16]. Multi-domain SDSs generally
utilize a distributed architecture and consider extensibility and
scalability. Thus, such architectures contain several different
domain-specific SLU modules. The domain spotter module
Manuscript received October 24, 2012; revised February 27, 2013, May 19,
2013, and July 09, 2013; accepted August 26, 2013. Date of publication Au-
gust 29, 2013; date of current version October 15, 2013. This work was sup-
ported by the Industrial Strategic technology development program, 10035252,
development of dialog-based spontaneous speech interface technology on mo-
bile platform, funded by the Ministry of Knowledge Economy (MKE, Korea).
A preliminary version of this paper appeared in ICASSP 2012, Kyoto, Japan.
The associate editor coordinating the review of this manuscript and approving
it for publication was Prof. Renato De Mori.
D. Lee, K. Kim, S. Ryu, and G. G. Lee are with Department of Computer
Science and Engineering, Pohang University of Science and Technology
(POSTECH), Pohang 790-784, Korea (e-mail: semko@postech.ac.kr;
getta@postech.ac.kr; ryush@postech.ac.kr; gblee@postech.ac.kr).
M. Jeong is with Microsoft, Redmond, WA 98052 USA (e-mail: minwoo.
jeong@microsoft.com).
Digital Object Identifier 10.1109/TASL.2013.2280212
1558-7916 © 2013 IEEE
identifies the target domain of the input utterance; the utter-
ance is then delivered to the appropriate domain-specific SLU
module. To develop a multi-domain SDS with a data-driven
approach, the domains of interest must first be defined. Next,
a corpus for each domain must be collected. The collected
corpora are then used to train the domain spotter module. The
corpus for a specific domain is used to train the corresponding
domain-specific module.
A semantically annotated corpus is required to automatically
train the statistical SLU module. The process that generates the
annotated corpus from the raw corpus can be divided into two
sub processes: the design step and the labeling step. In the de-
sign step, a linguistic expert creates guidelines for the anno-
tator. These guidelines specify information regarding the slots
and values contained in semantic frames. The design step might
not be difficult if the system specification is clearly and restric-
tively defined. Otherwise, the design step is not easy, even for
experts, when the system is developed from an unlabeled dialog
corpus. In the labeling step, the annotator assigns semantic tags
to elements in the raw corpus using the guidelines. The labeling
step is time-consuming and labor-intensive. For these reasons,
developing a multi-domain SLU module requires considerable
effort and expense. Therefore, this process can be a major bar-
rier that inhibits the rapid development of multi-domain SDSs.
This paper aims to resolve this issue by applying a fully
unsupervised approach to the SLU problem. Instead of ap-
plying a classical unsupervised approach, such as the k-means
algorithm, in which the number of clusters must be fixed in
advance, we propose using a non-parametric Bayesian ap-
proach that can flexibly infer efficient cluster numbers during
the learning process. The use of this non-parametric Bayesian
approach considerably reduces the human efforts required for
the design and labeling steps. We employ an unsupervised SLU
framework in conjunction with each domain corpus to develop
a multi-domain SDS. During this process, the domain spotter
model is also trained.
Another contribution of this paper is that we use an SDS to
evaluate the unsupervised SLU framework. To our knowledge,
this is the first attempt to do so. For evaluating the unsuper-
vised SLU framework, comparing it with human annotation is
a simple but limited method. Differences between human an-
notations and the results of the unsupervised SLU framework
may not often be a significant problem because human annota-
tions are not necessarily the best result. Additionally, the funda-
mental objective is to use the unsupervised SLU framework in
an SDS. Therefore, the experiments we conduct are intended to
determine the effects of the unsupervised SLU approach in the
context of its end-use application.
2452 IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 21, NO. 11, NOVEMBER 2013
The organization of this paper is as follows. In Section II,
we introduce traditional SLU techniques and some previous at-
tempts to reduce the human effort necessary to train the module.
In Section III, we describe our proposed unsupervised SLU
framework for multi-domain SDSs. In Section IV, we explain
the structure of a dialog system that uses the unsupervised SLU
framework. In Section V, we present our experimental results.
Finally, we state our conclusions and describe possibilities for
future work in Section VI.
II. RELATED WORK
Much research regarding SLU modules has been conducted.
Initially, many studies used a rule-based approach that depends
on handcrafted grammar [17]–[19]. This rule-based approach
often achieves good performance in spite of not requiring a sig-
nificant amount of training data. Moreover, this method fea-
tures outstandingly good manageability and has been applied to
a number of commercial systems. However, a problem remains:
this rule-based approach requires high-level linguistic skills and
manual labor to write the grammar, which incurs high costs
when the approach must be applied to a new domain or a new
language. This strategy is also not robust against speech recog-
nition errors. Contrastingly, many data-driven approaches have
utilized statistical models that are automatically learned from
labeled training data [20]–[22]. Collecting labeled training data
is less expensive than writing a handcrafted grammar. Hence,
data-driven approaches can be considered portable because they
can more easily be applied to new domain or a new language.
Additionally, data-driven approaches tend to be relatively ro-
bust because they generate models in a various forms from data.
However, a problem remains: data-driven methods require a sig-
nificant amount of data to obtain good performance and over-
come the intrinsic limit of data sparseness. For these reasons,
many studies have proposed hybrid approaches that take advan-
tage of both types of methods [23]–[25].
In an SLU task, a semantic frame can be divided into two in-
dicative parts [26]: (1) the meaning of input utterance and (2)
the named entity. Generally, each part is treated separately as a
classification problem and a sequence labeling problem, respec-
tively, by assuming a specific domain context. Jeong et al. [27]
introduced triangular chain conditional random fields, which
can be applied to joint modeling, to solve the SLU problem.
This approach can improve the performance of the SLU module;
however, this strategy still incurs high costs and requires much
effort to develop the system because of the need for labeled
training data.
Many researchers have focused on reducing the human ef-
fort that is required to develop SLU modules. To resolve this
problem, Tur et al. [28] applied active learning and semi-super-
vised learning. The authors built a statistical model by applying
a supervised learning method to a small amount of labeled data
from a call classification task. Next, the authors applied the
trained model to a large amount of unlabeled data. Among those
data, the authors selected the data with low confidence scores
and then performed human labeling for these data. Wu et al. [29]
also introduced an SLU approach based on a two-stage classifi-
cation that can apply weakly supervised training. In their study,
the authors could significantly reduce the human effort required
to train the model. However, the proposed method still requires
a design step and some labeled data.
Recently, an unsupervised topic modeling approach has been
used in tasks associated with SLU modules. This approach is
based on latent Dirichlet allocation (LDA) [30]. Unsupervised
modeling of dialog acts has been applied to online conversation
and social networking applications [31], [32]. For the named en-
tity recognition for news articles and search queries [33], [34], a
technique that uses a topic model has also been introduced. Ce-
likyilmaz et al. [35] proposed a generative Bayesian model that
can support joint learning for a domain, dialog act and named
entity, in conjunction with all of the labeled and unlabeled ut-
terances. Furthermore, the authors utilized external knowledge,
such as web search logs, to reinforce their model. The unsuper-
vised approach introduced above eliminates the cost incurred
during the labeling step; however, the cost of the design step
still remains because the number of clusters must be fixed. Thus,
human effort is required to analyze the data and obtain the op-
timal number of clusters.
More recently, the hierarchical Dirichlet Process (HDP) [36],
[37], which is a non-parametric Bayesian approach, has been
used to resolve the clustering problem. Unlike parametric ap-
proaches, this method allows the model to learn an appropriate
cluster number. Crook et al. [38] employed a non-parametric ap-
proach to conduct an unsupervised classification of dialog acts
in a travel planning dialog corpus. Furthermore, Higashinaka et
al. [39] applied an HDP hidden Markov model (HDP-HMM) to
a dialog corpus. HDP-HMM is capable of reflecting the sequen-
tial structure of dialog. The authors focused only on dialog acts
and considered only one dialog corpus.
Our work is also based on a non-parametric approach. How-
ever, our work differs from the previous works in several as-
pects. We focus on the entire task performed by an SLU module
and consider dialog corpora collected from multiple domains.
In addition, the previous works evaluated the performance in
terms of the clustering results. In contrast, we also conduct ex-
periments to determine the effects of an unsupervised model in
the context of an end-to-end dialog system.
III. UNSUPERVISED SLU FRAMEWORK
A. Problem Definition
An SLU module aims at extracting semantic frames from user
utterances. The semantic frames, which consist of slot-value
pairs, are used by the dialog manager. In an SLU task, we divide
a semantic frame into three sub-parts: the dialog act (DA), the
intent and the slot entity (SE) [26]. The DA and intent indicate
the meaning of the input utterance at the discourse level.
Whereas the DA is a domain-independent and surface-level
concept, the intent is a domain-dependent and function-level
concept. The slot entity denotes a domain-specific semantic
constituent at the word or phrase level. Fig. 1 shows an example
of a semantic frame for a user utterance.
The supervised approach considers determining DAs and in-
tents as classification problems and determining SEs as a se-
quence labeling problem. Given the input utterance, the DA and
intent are assigned to one of the predefined classes. The SE is as-
signed to one of the classes, which include the class ‘none’ for
LEE et al.: UNSUPERVISED SPOKEN LANGUAGE UNDERSTANDING FOR A MULTI-DOMAIN DIALOG SYSTEM
Fig. 1. A semantic frame example for an input utterance.
the words that compose the input utterance. The classes used
for the DA are a collection of surface-level speech acts such as
yn_question, wh_question and request, and they are domain-in-
dependently defined. In contrast, the classes used for the in-
tent and the SE are defined depending on the domain. For ex-
ample, in the building guidance domain, the intent classes in-
clude search_loc, search_phone and guide_loc. The SE classes
include ROOM_NAME, ROOM_TYPE and ROOM_NUMBER.
To develop a multi-domain SDS, we have collected a raw
dialog corpus from the target domains. The raw dialog corpus
consists of dialogs that contain a sequence of user utterances and
system acts. In the building guidance domain, the system acts
include GREET, INFORM, GUIDE and SAY. In the supervised
approach, the process for training the SLU and DM models is as
follows. First, classes for the DAs, intents and SEs are defined
for each target domain. Second, for each user utterance included
in the raw dialog corpus, all of the labels related to the DAs,
intents and SEs must be annotated prior to further processing.
Especially for multi-domain SDSs, we must additionally train
the domain spotter model. In this case, annotation is not required
because the dialog corpus is gathered when the specific domain
is specified.
In this paper, our goal is to apply a fully unsupervised SLU
approach to analyze dialog corpus that is gathered from mul-
tiple domains. We collect the information regarding the intents,
DAs and SEs that should be used for a semantic frame using a
clustering method. In this approach, we can omit two processes:
(1) the process that defines classes for DAs, intents and SEs and
(2) the process for human labeling. In the proposed approach,
cluster IDs are assigned to DAs, intents and SEs rather than
classes. Although cluster IDs are not stored in a human-read-
able format, this might not be a problem for SDS development
because the system can still determine whether two objectives
are identical. We can assign a meaningful label to each cluster
after analyzing the results if required.
B. Overall Procedure
In this paper, we propose an unsupervised SLU framework
that uses a cascade approach. The framework is divided into
three clustering steps. These steps are performed in the fol-
lowing order: (1) dialog act clustering, (2) slot entity clustering
and (3) intent clustering. The result of each step is required for
the subsequent step.
In the dialog act clustering step, utterance clustering is con-
ducted and a word source is assigned to the words that belong to
an utterance. The input for the dialog act clustering algorithm is
an augmented corpus that is a combination of the dialog corpora
from each domain. During the clustering process, we obtain a
word distribution for each domain. The word distribution can
2453
Fig. 2. The overall procedure for our unsupervised SLU framework.
Fig. 3. A graphical representation of the dialog act clustering model.
be used for the domain spotter model. Unlike dialog act clus-
tering, SE and intent clustering are conducted separately for the
dialog corpus of each domain. Prior to conducting slot entity
clustering, the SE candidates must be generated by using word
sources that have been assigned during the dialog act clustering
step. Clustering is performed by applying a virtual context doc-
ument concept for each candidate. In the final step, intent clus-
tering, utterance clustering is performed by using words, SEs
and system acts as features. In each clustering step, little human
effort is necessary. The overall procedure of our proposed unsu-
pervised SLU framework is illustrated in Fig. 2. The following
sections explain the details of the model for each clustering step.
C. Dialog Act Clustering
The model used for dialog act clustering is a combined form
of the HDP-HMM and the content model. The content model,
which was proposed by Barzilay and Lee [40] for summariza-
tion tasks, utilizes the HMM model for topic transition with
each topic generating message. By considering dialog act to
be a hidden state, we assume that each user utterance is gener-
ated from a dialog act. Based on the content model, a non-para-
metric approach and a Bayesian extension for dialog corpus are
applied. A graphical representation of the dialog act clustering
model is presented in Fig. 3.
Our model is applied to multiple domain-specific dialog cor-
pora. Each domain-specific dialog corpus consists of dialogs
for a single domain. Each dialog is a sequence of dialog
acts , and each act generates a sentence. The hidden variables
are sampled from the posterior distributions using the following
equations.
2454 IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 21, NO. 11, NOVEMBER 2013

The notation is as follows: is the number of words in

utterance of dialog ; is the vocabulary size,
is the number of bigram transitions; is the number
of source-word pair occurrences; is the number
of source-word pair occurrences in a specific domain ; and
is the number of act-word pair occurrences. Fig. 4. The words , entities and sources for an example sentence.
Our model uses a Dirichlet process (DP) to define an a priori
distribution for transition matrices in the infinite state spaces.
The state transitions are generated by , whose prior with symmetric hyperparameter . In a background
is . The transition prior is generated by with model , the word distribution is defined as follows:
strength parameter , which presents the inverse-variance of
the DP, and base distribution , which presents the mean of the
DP. The base distribution is generated using with
hyperparameter . GEM is the GEM distribution, which is also
known as the stick-breaking construction. The hyperparameter
controls the number of states, and controls the sparsity of If the word source is the domain words , then a
the transition matrix. For the transition matrices , the distri- word is generated using with a prior generated
bution is defined as follows. using with symmetric hyperparameter . The domain
model can be used for the domain spotter, which is a key
issue in multi-domain SDSs. For a specific domain , the
word distribution of the domain model can be defined as
Each sentence is represented by a bag of words that is rep- follows:
resented using the plate, respectively. We assume that each
word is generated from one of three sources: words in the cur-
rent dialog act , general words and domain
words . Thus, a new hidden variable determines the
source of each word. This variable is initially assigned a value
of zero and is drawn from with a parameter gen-
erated by with a parameter . In a specific domain , D. Slot Entity Clustering
the distribution for the source of each word is defined as
Slot entity clustering is separately applied to the dialog corpus
follows:
for each specific domain. First, the SE candidates must be ex-
tracted. Next, clustering is performed for the SE candidates.
1) Slot Entity Candidate Generation: To generate SE
candidates, we use the source of each word. The sources are
determined by the hidden variable in the dialog act clus-
tering model. Primarily, we generate temporary SE candidates
by extracting consecutive words, which are drawn from the
domain words. However, segmentation is required for certain
temporary SE candidates, as shown in Fig. 4. That figure
includes the words, human-annotated entities and word sources
General words that frequently occur in many sentences and for an example sentence. A word source is 2 when a corre-
domain words that frequently occur only in a specific domain sponding word is generated from the domain words. Therefore,
are not helpful for identifying a dialog act. By applying different the last four words in the example sentence are considered a
emission distributions, depending on the word source, sentence SE candidate. According to human annotation, the candidate
clustering can be conducted in a more tailored manner for each should be divided into two candidates. To determine the final
dialog act. SE candidates, we apply a segmentation algorithm using the
If the word source is the words in the current dialog act point mutual information (PMI)-based stickiness function
, then the word is generated using with a proposed by Li et al. [41].
prior generated using with symmetric hyperparam- The goal of segmentation is to divide a temporary SE can-
eter . For a specific dialog act , the word distribution of the didate into consecutive segments that each contain one or
dialog act model is defined as follows: more words, where . When the optimal segmen-
tation is obtained, each segment is added to the final SE candi-
date list. To identify the optimal segmentation out of all possible
segmentations, the following objective function is used:

If the word source is the general words , then the

word is generated using with a prior generated using
LEE et al.: UNSUPERVISED SPOKEN LANGUAGE UNDERSTANDING FOR A MULTI-DOMAIN DIALOG SYSTEM
Fig. 5. A graphical representation of our model for slot entity clustering.
To measure the stickiness of a segment, we use PMI, which
is a measurement of word collocation. The stickiness function,
, is defined by mapping the generalized PMI to
the range [0, 1]. Given the segment , the equa-
tions are as follows, where Pr(.) is the probability of the word
sequence and is calculated for a given corpus:
By using the method mentioned above, SE candidates can be
extracted without using external knowledge. When extracting
the SE candidates, the performance can be improved by incor-
porating a general domain SE recognizer or domain dictionary
that has been built in advance.
2) Slot Entity Candidate Clustering: For classification and
latent semantic association of SEs, topic models, such as LDA,
can be used by applying a virtual context document concept.
Our method for slot entity clustering is inspired by such ap-
proaches. The goal of slot entity clustering is to group the SE
candidates into SE types. For this purpose, we use a non-para-
metric Bayesian approach. Fig. 5 shows a graphical represen-
tation of our model for slot entity clustering. We assume that
each SE candidate has SE type and that each SE type gener-
ates a virtual context document, which is represented by a bag
of context features using plate. The SE type is generated
using , whose prior is . The prior is the distribution
of the SE types of all the SE candidates and is generated using
a DP with hyperparameter and base distribution . The base
distribution is generated using a GEM distribution with hyper-
parameter . The context feature is generated using , of entities in utterance of dialog ,
whose prior is . The prior is a distribution of the context
features in the SE types and is generated using with
symmetric hyperparameter . By generating the virtual con-
text documents from all of the SE candidates and then learning
the model, we can cluster the SE candidates.
Context words and collocations are included in the virtual
context document for each SE candidate. Context words and
collocations were used as features for slot classification by Wu
et al. [29]. If the words in a sentence
are extracted to an SE candidate , the features that compose the
virtual context document for the SE candidate are presented in
Fig. 6.
2455
Fig. 6. The features of a virtual context document for a slot entity candidate.
E. Intent Clustering
The objective of this process is to determine clustering ut-
terances that correspond to an identical intent. The intent clus-
tering model has a form similar to that of the dialog act clus-
tering model, which is also used for clustering utterances. DAs
are domain-independent. However, intents denote domain-de-
pendent meanings of utterances. Because of this dependence,
intent clustering is applied to the dialog corpus of a specific do-
main. Additionally, a DA is a surface-level concept; thus, it uses
only words as features. However, an intent is a functional-level
concept; thus, it uses SEs and system acts in addition to words as
features. According to the study of Jeong et al. [27], a cascade or
joint approach that combines intent classification and SE recog-
nition for SLU can perform better than non-hybrid methods.
In addition, intents are utilized as significant information when
predicting the system act in the DM. Thus, SEs and system acts
are significantly correlated with intents and assumed to be useful
features for intent clustering.
A graphical representation of the intent clustering model is
shown in Fig. 7. The transition matrices can be defined in the
same manner as for the dialog act clustering model. Each dialog
is a sequence of intents , and each intent generates an utter-
ance and system acts . The hidden variables are sampled from
the posterior distributions given by the following equations:
We define additional notation as follows: is the number
is the number of unique
SEs, is the number of unique system acts and is the
number of intent-system act pair occurrences.
Each utterance is represented by a bag of words and entities
that shown using the , and plates, respectively. The
entity word is replaced with the entity class name. The entities
are generated using with a prior that is generated
using Dir with symmetric hyperparameter . As for the
specific intent , the entity distribution can be defined as
follows:
2456 IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 21, NO. 11, NOVEMBER 2013
Fig. 7. A graphical representation of the intent clustering model.
The system acts are generated using , with a prior
that is generated using Dir with symmetric hyperpa-
rameter . For specific intent , the system act distribution
can be defined as follows.
In intent clustering, unlike in dialog act clustering, each word
is generated from one of two sources: the general words or
the words in the current intent. A new hidden variable de-
termines the source of each word and is drawn from
(the Bernoulli distribution) with parameter , which is gener-
ated using (the beta distribution) with parameter .
The word distribution for a specific intent and the word dis-
tribution for a general word in the background model can be
applied in the same manner as in the dialog act clustering model.
F. Inference
To perform inference in our models, we use Gibbs sampling
[42], which is a Markov chain Monte Carlo algorithm. The
hidden variables are sampled from the posterior distributions.
In our models, several hyperparameters must be provided. The
simplest way to choose hyperparameters is a grid search. How-
ever, a grid search is time-consuming and expensive. Therefore,
instead of using a grid search, we use Bayesian priors for all
of the hyperparameters. In the Bayesian approach, we can
automatically select the hyperparameters by treating them as
additional hidden variables. We sample the hyperparameters
using Gamma(0.1, 0.1) priors after each iteration of Gibbs
sampling [37].
IV. MULTI-DOMAIN SDS USING UNSUPERVISED
SLU FRAMEWORK
A. An Example-Based Dialog Management Framework
Our multi-domain SDS is implemented using an example-
based dialog management (EBDM) framework [14], [43]. The
EBDM framework, which is a data-driven approach, is inspired
by example-based machine translation [44]. The EBDM frame-
work is a simple and powerful method for rapidly developing
SDSs for multi-domain dialog processing. In addition to these
features, the use of an unsupervised SLU module strengthens the
Fig. 8. Overall strategy of the EBDM framework.
EBDM framework’s advantages, such as supporting rapid pro-
totyping in various applications and incurring reasonably low
costs during development.
With a traditional EBDM framework that uses a supervised
approach, the dialog corpora are collected at each domain and
then, semantic tags (e.g., DA, intent and SE) are manually as-
signed to user utterance. With the proposed method, however,
semantic tags are automatically assigned by applying an unsu-
pervised SLU module to a dialog corpus. Then, a dialog ex-
ample database (DEDB) is automatically built using the anno-
tated dialog corpus.
Fig. 8 presents the overall strategy of the EBDM framework
for SDSs. The EBDM framework employs a statistical SLU
approach that performs modeling by using conditional random
fields (CRFs) [45] to extract semantic frames from user utter-
ances. To train the SLU model, the EBDM framework uses user
utterances and the semantic tags attached to the user utterances
in a dialog corpus that has been automatically annotated using
an unsupervised SLU module.
The dialog manager performs three steps to determine a
system act: query generation, example search, and example
selection. First, an SLU module result is generated by using
a semantic frame and discourse history for the current utter-
ance (this step is the query generation step). Next, the dialog
manager searches the DEDB for a dialog example with a
semantically similar form (this step is the example search step).
If no matched item is found, the search is conducted again after
the search condition is relaxed (this is referred to as a relaxation
strategy). Finally, the best dialog example is selected according
to the score of the searched examples. The score is determined
based on utterance similarity and discourse history similarity
(this step is referred to as the example selection).
LEE et al.: UNSUPERVISED SPOKEN LANGUAGE UNDERSTANDING FOR A MULTI-DOMAIN DIALOG SYSTEM
The content database contains several contents that are
related to the domain. To determine the content that the user
wants, SEs are used. The discourse history stores and maintains
previous discourse information. The natural language gener-
ation (NLG) module is a semantic representation of a system
act. The NLG module generates the system utterance. To do so,
the NLG module utilizes a system template that is predefined
for each system act.
When the unsupervised SLU framework is applied, the
intents, DAs and SEs of the semantic frames are filled with
cluster IDs. The cluster IDs are not in a human-readable format;
however, this format does not create any problems for the EBDM
framework. Therefore, the human effort traditionally required in
the design and annotation steps of the SLU module and DM in the
EBDM framework can be eliminated, because of the application
of the unsupervised SLU framework. Regardless, during the
process of accessing knowledge source such as content DBs,
DB fields cannot be matched successfully using only unsuper-
vised tags of slot entities. Therefore, a human must link the
unsupervised tags and DB fields. However, this problem is not a
problem only for unsupervised approach. Human intervention is
also required in the supervised approach because the DB field
related to a given slot entity label must be specified in advance.
B. Domain Spotter
Our multi-domain SDS employs a distributed architecture
that can be used to easily augment and modify the domain.
The distributed structure includes a domain spotter and sev-
eral domain-specific modules. The domain spotter module is
responsible for determining the domain of an input utterance
and for delivering the input utterance to the target domain-spe-
cific module. In this study, we employ a hybrid method, which
was used by Lee et al. [14], that combines keyword-based and
feature-based approaches for the domain spotter module. This
method uses linguistic (words), semantic (dialog acts) and key-
word features (n-best keyword and n-best class), as is done
for maximum entropy classifiers [46]. Lee et al. used term fre-
quency and inverse document frequency [47] to ex-
tract keyword features. However, we utilized a domain model
for dialog act clustering. Our method is related to LDA-style
topic identification [30].
C. Agenda Graph
The EBDM framework can use an agenda graph as prior
knowledge [48]. In this approach, an agenda graph is used to
address the robustness problem for practical applications. An
agenda graph provides a simple method of encoding domain-
specific dialog control to complete the task. An agenda graph is
composed of nodes that denote subtasks and edges that connect
two nodes. Each node has a precondition that is satisfied before
the subtask is completed. The precondition is defined as the in-
tent of the current utterance and the accumulated SEs from the
beginning of the dialog until the current utterance.
A clustering technique based on k-means has been proposed
to reduce the human effort required to construct an agenda graph
[49]. However, this technique requires one to fix the number
of nodes included in an agenda graph prior to clustering be-
2457
TABLE I
CORPORA STATISTICS
cause k-means is a type of parametric approach. To resolve this
problem, we propose an enhanced method that automatically
generates an agenda graph by modifying the intent clustering
method. We assume that the sub-task generates words, entities
and a system act similar to that of intent clustering. Addition-
ally, we consider the entities accumulated from the beginning
of the dialog until the current utterance, which is dissimilar to
what is done for the intent clustering. An agenda graph can be
automatically generated after clustering in this manner. Upon
clustering completion, the state denoting the sub-task is consid-
ered to be the node and the transition probability between states
is considered to be the edge weight.
V. RESULTS AND DISCUSSION
In this section, we discuss in detail the experiments that we
conducted by applying the proposed models to dialog corpora.
Our experimental results consist of two parts. First, we com-
pare the automatically labeled corpora, which were generated
using unsupervised methods, with the human-labeled corpora
by performing a clustering evaluation. Next, we present a di-
alog system evaluation to demonstrate the effectiveness of our
models in an end-use application.
A. Dialog Corpora
Korean-language dialog corpora from various domains
were collected to construct a dialog system. We used the
Wizard-of-Oz method [50] to construct the corpus. Given
domain-specific tasks (e.g., “Find the president’s room for a
visit” in the building guidance domain), human-human dialogs
were collected by ten individuals who were acquainted with the
dialog system. In such an environment, unrealistic dialogs can
sometimes be included. To create a refined corpus for training,
we filtered out erroneous dialogs by considering utterance pat-
terns, vocabulary and dialog flow. We collected goal-oriented
corpora in the following four domains: car navigation, weather
information, TV program guidance and building guidance.
After collecting the raw dialog corpora, we defined the DA,
intent and SE classes for each corpus through human analyses.
Next, we manually assigned the DA and intent classes to the ut-
terances and the SE classes to the words in all of the utterances
in the raw corpora. Statistics describing the manually labeled
corpora are presented in Table I. When using our unsupervised
approach, only the raw corpora were required to generate auto-
matically labeled corpora. The manually labeled corpora were
used for reference in our experiments.
B. Clustering Evaluation
1) Methods and Measures: We performed clustering eval-
uation by measuring the distance between the clustering result
2458 IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 21, NO. 11, NOVEMBER 2013
TABLE II
RESULTS OF THE DIALOG ACT CLUSTERING FOR VARIOUS MODELS;
MEAN FOR EACH CRITERION
Significantly better than HDP .
Significantly better than HDP-HMM .
Significantly better than HDP-HMM-A .
and the gold standard. Here, we regarded the manually labeled
corpora as the gold standard. For the clustering evaluation, we
computed the following measures: purity (Pur), the Rand index
(RI), the F-measure (F-M) and the V-measure (V-M) [51], [52].
For automated quantitative analyses, we obtained the clus-
tering result after 1000 iterations and computed the measures
described above over 100 trials for each method. These mea-
sures range between 0 and 1, and a higher value indicates
better clustering quality. In addition, we analyzed the clustering
quality by examining the confusion matrix, which can be used
to visualize how much a cluster differs from the gold standard.
2) Dialog Act Clustering: To verify the effects of our dialog
act clustering method (HDP-HMM-A), which was introduced
in Section III-C, we compared it with other methods, such as
HDP-HMM, HDP and LDA-HMM-A. The HDP-HMM method
uses only bags-of-words as features for the dialog act clustering,
and HDP is a method that ignores the state transition probability
in HDP-HMM. Finally, LDA-HMM-A is a method that applies
a parametric approach to our approach by specifying the cluster
number in advance. In this case, for the cluster number, we use
the number as defined by human annotation.
For the dialog act clustering, all of the dialog corpora from the
four domains were used. The clustering performance of various
methods is presented in Table II. This table includes the various
measures previously mentioned and the number of clusters (#C).
Each resulting value is obtained by calculating the average of
the values for 100 attempts.
The experimental results indicate that the transition proba-
bility contributes to enhanced performance. Considering the di-
alog structure yields an even better effect. Our method demon-
strated improvements in the dialog act clustering performance
because it alleviated the tendencies of the utterances being clus-
tered together at the same domains by using the additional vari-
ances for the word source. The LDA-HMM-A method exhibited
worse performance than HDP-HMM-A because the number of
DAs defined by human annotation was not the optimal number
of clusters. In most applications in which parametric approaches
are used, it might be necessary to conduct several experiments
that use different number of clusters to determine the optimal
number of clusters. In our non-parametric approach, however,
the model infers the optimal number of clusters.
To construct the confusion matrix, we selected the dialog act
clustering result that was closest to the average value of the
number of clusters calculated from 100 attempts (Fig. 9). The
cluster ID is determined during the learning process. The figure
indicates the cluster IDs and corresponding human labels. A sig-
nificant difference between human annotation and dialog act
Fig. 9. The confusion matrix for the clustering result of our dialog act clus-
tering model.
TABLE III
RESULTS OF THE SLOT ENTITY CLUSTERING FOR THE HDP AND LDA
MODELS USING THE GENERATED SLOT ENTITY CANDIDATES;
MEAN FOR EACH CRITERION
Significantly better than HDP .
clustering result can be observed from the confusion matrix.
However, for ‘request’ and ‘wh_question’ which are the most
frequent dialog acts in the human-annotated corpora, the clus-
tering was successfully performed.
3) Slot Entity Clustering: We also evaluated the SE
candidate generation and clustering method (introduced in
Section III-D). First, the SE candidate generation used the word
source from the dialog act clustering model. We used the result
that applied the same criteria as were used for the confusion
matrix in dialog act clustering, as mentioned above. The perfor-
mance of the SE candidate generation yielded an F-measure of
0.667 (precision and recall ). The reason why
the precision is low is that domain-specific expressions that are
not SEs were extracted along with SE candidates. To improve
the SE detection performance, we filtered out erroneous SE
candidates using hand-written rules. The rules were defined by
specifying POS tag sequences that cannot be SE candidates.
For example, an SE candidate that ends with a post-position
or verb was filtered out because such POS tag sequences are
unnatural for SEs in the Korean language. When filtering was
applied, the F-measure increased to 0.849 (
and ).
Second, we performed SE candidate clustering evaluation
for each domain (Table III). We constructed the reference by
matching the SE candidates with the human labels. If the candi-
date was not an SE, the answer was set to the “none” class. Our
slot entity clustering method (HDP) uses a virtual context docu-
ment as a feature without using the transition probability. For the
comparison method, we used a parametric approach (LDA) that
LEE et al.: UNSUPERVISED SPOKEN LANGUAGE UNDERSTANDING FOR A MULTI-DOMAIN DIALOG SYSTEM
Fig. 10. A confusion matrix for the clustering result of our slot entity clustering
model for the building guidance domain.
TABLE IV
RESULTS OF THE SLOT ENTITY CLUSTERING FOR THE HDP AND
LDA MODELS USING THE CORRECT SLOT ENTITY CANDIDATES;
MEAN FOR EACH CRITERION
Significantly better than HDP .
has the same structure. The performance was improved because
the cluster number suggested by the human annotation was used.
For our method, the confusion matrix for the slot entity clus-
tering result in the building guidance domain is illustrated in
Fig. 10. The clustering result selection criteria are the same as
for the dialog act clustering. Some different SEs that have sim-
ilar context words are observed in the same cluster.
In addition, we evaluated the slot entity clustering method
by extracting the correct SE candidates from the human labeled
corpus (Table IV). When the correct SE candidates were given,
a better performance was obtained. Although we focused on a
fully unsupervised approach in these experiments, we believe
that the use of external resource in the SE candidate genera-
tion step resulted in performance superior to that of slot entity
clustering.
4) Intent Clustering: In Section III-E, we introduced the
various features used for our intent clustering method. To deter-
mine the effects of each feature, we employed various models
in our experiments. Among them, we used HDP, which em-
ploys only bags-of-words as a feature, and HDP-HMM, which
considers the state transition probability as a baseline. We also
separately applied the model extension of HDP-HMM. Briefly,
we measured the respective performances in three difference
cases: HDP-HMM-W, in which the word source is applied;
HDP-HMM-N, in which the SE distribution is applied; and
HDP-HMM-S, in which the system act distribution is applied.
Moreover, we also measured the performance in two additional
cases: HDP-HMM-A (in which our intent clustering methods
were used to apply all of the features) and LDA-HMM-A
2459
TABLE V
RESULTS OF THE INTENT CLUSTERING FOR VARIOUS MODELS; MEAN FOR
EACH CRITERION . (a) Car navigation. (b) Weather information.
(c) TV program guide. (d) Building guidance
Significantly better than HDP .
Significantly better than HDP-HMM .
Significantly better than HDP-HMM-A .
(the same as HDP-HMM-A except that a specified number of
clusters was used).
We also evaluated the intent clustering for each domain
(Table V). Different tendencies were observed for different
domains. Whereas the use of word source negatively impacted
the performance for all domains except for the weather infor-
mation domain, the use of the SE distribution enhanced the
clustering performance for all domains except for the weather
information domain. Use of the system act distribution yielded
a significant improvement in performance for every domain.
The experimental results indicate that the proposed method is
effective for intent clustering. The confusion matrix for the
intent clustering result of our method applied to the building
guidance domain is shown in Fig. 11. The clustering result
selection criteria are the same as were used when dialog act
clustering was conducted. Some utterances that were labeled
with different classes were assigned to the same clusters. Such
instances occurred when those utterances appeared in similar
2460 IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 21, NO. 11, NOVEMBER 2013
Fig. 11. A confusion matrix for the clustering result of our intent clustering
model for the building guidance domain.
dialog contexts (e.g., bye and thank in the building guidance
corpus). In contrast, some utterances that were labeled with the
same class were assigned to different clusters. Such instances
occurred because of differences in the dialog context and
differences in the SEs contained in utterances (e.g., search_loc
and guide_loc in the building guidance corpus).
C. Dialog System Evaluation
In the previous evaluation, we measured the clustering per-
formance using the manually annotated labels as the target clus-
tering. For two reasons, this strategy is not the optimal method
to evaluate the clustering results. First, we cannot guarantee that
human annotation is the optimal answer. Second, the ultimate
goal is to use the automatically labeled dialog corpus in a di-
alog system. Thus, for an enhanced evaluation, evaluating per-
formance using the dialog system is necessary.
1) Methods and Measures: In our experiments, we used the
EBDM model [14] that was introduced in Section IV. We eval-
uated the two dialog systems using the agenda graphs for each
corpus. The dialog systems were trained on the human anno-
tated corpus (HC) or the automatically annotated corpus (AC).
To construct models using the AC, we selected one result among
the multiple clustering results using the criteria that were used
to build the confusion matrix in the experiment in Section V-B.
The agenda graphs were generated in a handcrafted manner for
the HC and automatically generated using a non-parametric ap-
proach for the AC, as explained in Section IV-C.
Even in the AC, a small amount of human intervention was
required for the content DB manager and natural language gen-
eration module of the dialog system. We performed two evalua-
tions: a simulated user evaluation and a real user evaluation. The
evaluations were conducted for each domain. An experiment for
multi-domain was performed only using only real user evalua-
tion because our dialog simulator does not consider multi-do-
main environments. Additionally, we conducted experiments on
the domain spotter used for our multi-domain SDS.
To measure the quality of the dialog system, we computed the
average turn length (ATL) and the task completion rate (TCR)
for the dialogs. In addition, we defined an average score function
(SCORE), which represents a combined measure of the ATL
and the TCR. SCORE is similar to the reward score commonly
used in reinforcement learning-based dialog systems [2], [53].
SCORE is calculated as follows. For each dialog, the system
TABLE VI
RESULTS OF THE DIALOG SYSTEMS FOR THE SIMULATED USER
EVALUATION; THE ATL, THE TCR AND SCORE ARE AVERAGED
OVER 1000 DIALOGS
starts with 30 points, and the points are decremented in each
dialog turn. If the task of a dialog fails, SCORE is 30 points.
In the simulated user evaluation, the ATL, the TCR and
SCORE were automatically calculated by the simulator. In the
real user evaluation, the participants completed questionnaires
about successful turn and task completion. Using the results of
questionnaires, the ATL, the TCR and SCORE were calculated.
Additionally, we computed the successful turn rate (STR).
2) Simulated User Evaluation: We used a dialog sim-
ulator that is composed of user-intention, user-surface and
ASR channel simulators [54]. The intention simulator was
implemented by using the conditional random field model [45]
to consider dialog sequences. This simulator generated the
next user intention given the current discourse context. After
selecting the user intention, the surface simulator was used to
generate the user utterance to express the selected intention
using intention-specific part-of-speech tag sequences and dic-
tionaries. Then, the ASR channel was used to automatically
transform the raw utterance into a noisy utterance according to
a specified word error rate (WER).
To automatically measure the success of the generated
dialogs, the final states of task completion were defined. The
models for the simulator were trained on the manually an-
notated dialog corpus. For each dialog system, we used 1000
simulated dialogs and the 5 best recognition hypotheses to mea-
sure the TCR, the ATL, and SCORE under WER conditions
ranging from 0% to 50%.
Table VI shows the performance of the two dialog system
for each domain when WER is zero. In addition, Fig. 12 shows
SCORE of the two dialog systems under various WER condi-
tions. In most cases, the HC exhibits better performance than
the AC, especially for SCORE. In spite of this result, the dif-
ferences between the two systems are not considerable except
for the weather information domain. As the value of WER in-
creases, the differences tend to be smaller. In the case of TCR,
the two dialog systems have similar performance. Even in some
noisy environments, AC performed better than HC. Thus, we
conclude that the performance difference between the two sys-
tems is not significant for the limited tasks investigated, partic-
ularly in goal-oriented dialog systems with noisy environments.
3) Real User Evaluation: To compare the performance of di-
alog systems in the real world, ten undergraduate students were
employed in a real-user evaluation. To prevent the participants
from knowing whether the system was trained on the HC or
the AC, the participants tested two dialog systems in a random
LEE et al.: UNSUPERVISED SPOKEN LANGUAGE UNDERSTANDING FOR A MULTI-DOMAIN DIALOG SYSTEM
Fig. 12. SCORE of the dialog systems from simulated user evaluation; SCORE
is averaged over 1000 dialogs under various WER conditions.
TABLE VII
RESULTS OF THE DIALOG SYSTEMS FOR THE REAL USER EVALUATION; THE
ATL, THE TCR AND SCORE ARE AVERAGED OVER 50 DIALOGS
order. In the evaluation, we considered only text inputs. There-
fore, the WER was 0%.
We provided the participants with five pre-defined tasks for
each domain and three pre-defined tasks for multi-domain. The
participants used the dialog system to accomplish the tasks. The
participants judged the task success for each dialog and the turn
success for each utterance. Finally, we measured the ATL, the
TCR, STR and SCORE for each dialog system for the resulting
dialogs. Similar to the simulated-user evaluation, the system
with the HC and the system with the AC performed. The re-
sults are shown in Table VII. Although these results may not be
significant because of the small test set sizes, HC demonstrated
better performance compared with AC in all cases. However, the
performance of AC is acceptable. We believe that our approach
will be applicable for developing various domain systems for
real users.
4) Domain Identification Evaluation: To evaluate the do-
main spotter introduced in Section IV-B, we confirmed if it cor-
rectly predicted the target domain for each utterance. For the
evaluation, we used linguistic, semantic and keyword features
for a maximum entropy classifier, similar to the work of Lee
et al. [14]. To extract the keyword features, we considered two
different cases: using (1) and (2) the domain model
2461
TABLE VIII
RESULTS OF THEDOMAIN SPOTTER; THE ACCURACY IS CALCULATED
USING FIVE-FOLD CROSS VALIDATION
TABLE IX
RESULTS OF THE COLLECTION STEP FOR THREE EXPERIMENTAL CONDITIONS
trained during our dialog act clustering. For each respective
method, a five-fold cross validation was used.
The experimental results are presented in Table VIII. The re-
sults indicate that extracting keyword features using our domain
model yields an increased accuracy compared with
. The results suggest that the domain model that can
be obtained using the proposed unsupervised SLU framework
to develop multi-domain SDS can be helpfully utilized for the
domain spotter. The domain spotter using our domain model
was applied to multi-domain SDS, which was used in the user
evaluation of the dialog system. Using our domain model, the
proposed method can be more effective than the baseline be-
cause it enables the domain spotter to separate the words that
are unrelated to domain keywords by using word sources.
D. Development Costs
Several processes are necessary in developing a data-driven
dialog system. Among these processes, the collection step, de-
sign step and labeling step require more human effort and time
than any other processes. We conducted several experiments re-
garding the ‘TV program guide’ domain to explore the effect of
our unsupervised approach on reducing development costs.
First, to collect utterances in the ‘TV program guide’ domain,
we used the WOZ method and established the following three
experimental conditions:
• WOZ_NON: A situation where no dialog system exists.
• WOZ_SDS: A situation where a dialog system exists.
• WOZ_CON: A situation where a dialog system exists, and
the user understands a function to be handled by the system
Ten participants were employed for each experimental con-
dition. We collected 50 utterances for each participant. For each
experimental condition, we measured the elapsed time required
for a total of 500 utterances and the percentage of valid utter-
ances among the 500 utterances (Table IX). A valid utterance
means an utterance that can be handled by the dialog system
and can be used in the training step.
When the results of WOZ_NON and WOZ_SDS were com-
pared, the elapsed time was reduced by 55 minutes and 46 sec-
onds, as the dialog system supports system response generation.
When the results of WOZ_SDS and WOZ_CON were com-
pared, the elapsed time was reduced by 57 minutes and 43 sec-
onds, and the percentage of valid utterances was increased from
2462 IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 21, NO. 11, NOVEMBER 2013
TABLE X
RESULTS OF THE DESIGN AND ANNOTATION STEP FOR
FOUR EXPERIMENTAL CONDITIONS
76.2% to 99.8%. The reason for this result is that valid utter-
ances were rapidly generated by the participants who under-
stood the functions of the dialog system.
The experimental result shows that the rapid development of
the prototype system and the definition of the function range that
could be handled by the system are important for the efficiency
of the collection step. Usually, an initial raw dialog corpus is
collected without implementation of the dialog system. To de-
velop the prototype system using the corpus, the design step and
the labeling step should be performed on the corpus.
To measure the costs required for the design step and the la-
beling step, the following four experimental conditions were es-
tablished:
• HUMAN: two steps are performed by a human under the
condition that only raw dialog corpus is provided.
• HUMAN_K: two steps are performed by a human under
the condition that the raw dialog corpus and knowledge
database are provided.
• AUTOMATIC: two steps are replaced with clustering re-
sults (Our approach)
• SEMI_AUTO: two steps are performed by a human
under the condition that the dialog corpus, knowl-
edge database and clustering results are provided
Our approach human intervention
For each experimental condition, five participants with ex-
perience in developing a dialog system performed the design
step and the labeling step. We measured the average elapsed
time and gain rate (Table X). The gain rate measures the relative
improvement level and was calculated as follows: Gain rate
, where and represent the elapsed time
under the HUMAN and the corresponding experimental con-
dition, respectively. The initial raw dialog corpus used for the
experiment is the ‘TV Program guide’ domain introduced in
Table I, which includes 138 dialogs and 1272 utterances.
With the HUMAN regarded as the baseline, the elapsed
time was reduced by 11.2% when the knowledge database was
provided, which means that the information in the knowledge
database is useful for the design step and the labeling step.
The AUTOMATIC is applied by our approach introduced in
this paper, and the elapsed time shown in the table represents
the running time of the program. In the AUTOMATIC, the
development of prototype system is significantly faster than
under any other condition, but it has a disadvantage in that
it shows relatively low performance (Table VI and Fig. 12),
and the labeled corpus is not in a human readable format. In
the SEMI_AUTO, the elapsed time was reduced by 48.0%
compared with the HUMAN. In addition, in contrast to the AU-
TOMATIC, because the labeled corpus from the SEMI-AUTO
is in a human readable format, it would be useful in providing
a guideline for an additional data collection step.
The experimental results show that our approach would be
helpful for the rapid development of a prototype system and
determining the function range that could be handled by the
system. In addition, when observing the preceding experimental
results that led to effective data collection, it could be contribute
to reducing the overall development costs.
E. Discussion
1) Design and System Performance: The dialog act, intent
and slot entity clustering results generated using our unsuper-
vised SLU framework might not be satisfactory when compared
with human annotation. According to the clustering evaluations,
the values of the F-measure and the V-measure ranged from 0.3
to 0.6. However, our objective is to generate clustering results
as an alternative to human annotation in dialog system devel-
opment. The ultimate objective of an SDS is to perform system
acts that comply with the desired intention of the user. To ef-
ficiently support this objective, a semantic frame is used in the
SLU module and the DM. However, achieving an optimal de-
sign for a semantic frame to fully meet the ultimate objective of
the SDS is difficult. When determining a system act, we use var-
ious features and consider the semantic frame and the discourse
history of the current utterance. Although the DA, intent and SE
were designed to be different, the system was able to determine
an identical system act for an identical utterance. Therefore, the
dialog system did not yield a significant difference in system
performance during the evaluation, despite the differences be-
tween the clustering results and the human annotation.
2) Joint Model: Our SLU framework uses a cascade ap-
proach. In this approach, dialog acts, slot entities and intents
are clustered sequentially. During the clustering process, each
is clustered using different model, respectively. Therefore, our
method is constrained to consider only one-way dependency as
specified in advance. To overcome this constraint, a joint ap-
proach can be useful for a unified learning. Jeong et al. [27] used
a joint model in an SLU framework with a supervised approach.
Many studies about unsupervised grammar induction [55] also
consider unified learning. By considering these methods, we
will apply a joint approach, in which dialog acts, slot entities
and intent are clustered using one model.
VI. CONCLUSIONS AND FUTURE WORK
In this paper, we presented an unsupervised SLU framework
that uses a non-parametric Bayesian approach. Our unsuper-
vised SLU framework was used to perform dialog act, intent
and slot entity clustering for dialog corpora. DAs, intents and
SEs are components of semantic frames. The primary advan-
tage of our method is the considerable reduction of the human
effort required for the design and labeling steps yielded by the
application of a non-parametric Bayesian approach. We also ap-
plied our unsupervised SLU framework to a dialog system. In
our experiments, we not only conducted clustering evaluation
but also performed a dialog system evaluation using four di-
alog corpora. Moreover, we evaluated the domain spotter for a
multi-domain SDS. The experimental results indicate that our
proposed method is applicable and useful for the rapid develop-
ment of multi-domain SDSs.
LEE et al.: UNSUPERVISED SPOKEN LANGUAGE UNDERSTANDING FOR A MULTI-DOMAIN DIALOG SYSTEM
Several aspects of our approach require further research. We
focused on a fully unsupervised method. However, the clus-
tering quality can be improved by applying partial supervision
and domain knowledge. We plan to develop a method that com-
bines our unsupervised model with heuristic rules. We also plan
to perform a real user evaluation that uses many more partici-
pants and verify our framework using a more complicated di-
alog system. In addition, we must implement a dialog system
development toolkit that provides an unsupervised method for
faster development and more efficient management of dialog
systems.
REFERENCES
[1] D. Bohus and A. I. Rudnicky, “The RavenClaw dialog management
framework: Architecture and systems,” Comput. Speech Lang., vol. 23,
no. 3, pp. 332–361, 2009.
[2] J. D. Williams and S. Young, “Partially observable Markov decision
processes for spoken dialog systems,” Comput. Speech Lang., vol. 21,
no. 2, pp. 393–422, 2007.
[3] L. F. Hurtado, D. Griol, E. Sanchis, and E. Segarra, “A stochastic
approach to dialog management,” in Proc. IEEE Workshop Autom.
Speech Recogn. Understand., 2005, pp. 226–231.
[4] C. Lee, S. Jung, K. Kim, D. Lee, and G. G. Lee, “Recent approaches to
dialog management for spoken dialog systems,” J. Comput. Sci. Eng.,
vol. 4, no. 1, pp. 1–22, 2010.
[5] S. Jung, C. Lee, and G. G. Lee, “Using utterance and semantic level
confidence for interactive spoken dialog clarification,” J. Comput. Sci.
Eng., vol. 2, no. 1, pp. 1–25, 2008.
[6] V. Zue, S. Seneff, J. R. Glass, J. Polifroni, C. Pao, T. J. Hazen, and L.
Hetherington, “JUPlTER: A telephone-based conversational interface
for weather information,” IEEE Trans. Speech Audio Process., vol. 8,
no. 1, pp. 85–96, Jan. 2000.
[7] M. Walker, J. Aberdeen, J. Boland, E. Bratt, J. Garofolo, L. Hirschman,
A. Le, S. Lee, S. Narayanan, and K. Papineni, “DARPA communi-
cator dialog travel planning systems: The June 2000 data collection,”
in Proc. Eur. Conf. Speech Commun. Technol., 2001, pp. 1371–1374.
[8] O. Lemon, K. Georgila, J. Henderson, and M. Stuttle, “An ISU dialogue
system exhibiting reinforcement learning of dialogue policies: Generic
slot-filling in the TALK in-car system,” in Proc. Eur. Chap. Assoc.
Comput. Linguist., 2006, pp. 119–122.
[9] W. Minker, U. Haiber, P. Heisterkamp, and S. Scheible, “The SENECA
spoken language dialogue system,” Speech Commun., vol. 43, no. 1,
pp. 89–102, 2004.
[10] F. Weng, S. Varges, B. Raghunathan, F. Ratiu, H. Pon-Barry, B.
Lathrop, Q. Zhang, H. Bratt, T. Scheideck, and K. Xu, “CHAT: A
conversational helper for automotive tasks,” in Proc. Int. Conf. Spoken
Language Processing, 2006, pp. 1061–1064.
[11] J. Allen, D. Byron, M. Dzikovska, G. Ferguson, L. Galescu, and A.
Stent, “An architecture for a generic dialogue shell,” Natural Language
Engineering, vol. 6, no. 3&4, pp. 213–228, 2000.
[12] K. Komatani, N. Kanda, M. Nakano, K. Nakadai, H. Tsujino, T. Ogata,
and H. G. Okuno, “Multi-domain spoken dialogue system with
extensibility and robustness against speech recognition errors,” in
Proc. 7th SIGDIAL Workshop on Discourse and Dialogue, 2009, pp.
9–17.
[13] O. Lemon, A. Gruenstein, A. Battle, and S. Peters, “Multi-tasking and
collaborative activities in dialogue systems,” in Proc. 3rd SIGDIAL
Workshop on Discourse and Dialogue, 2002, pp. 113–124.
[14] C. Lee, S. Jung, S. Kim, and G. G. Lee, “Example-based dialog mod-
eling for practical multi-domain dialog system,” Speech Communica-
tion, vol. 51, no. 5, pp. 466–484, 2009.
[15] B. Lin, H. Wang, and L. Lee, “A distributed architecture for coopera-
tive spoken dialogue agents with coherent dialogue state and history,”
in Proc. IEEE Workshop on Automatic Speech Recognition and Under-
standing Workshop, 1999, p. 4.
[16] I. O’Neill, P. Hanna, X. Liu, D. Greer, and M. McTear, “Implementing
advanced spoken dialogue management in Java,” Science of Computer
Programming, vol. 54, no. 1, pp. 99–124, 2005.
[17] J. Dowding, J. M. Gawron, D. Appelt, J. Bear, L. Cherny, R. Moore, and
D. Moran, “Gemini: A natural language system for spoken-language
understanding,” in Proc. Association for Computational Linguistics,
1993, pp. 54–61.
2463
[18] S. Seneff, “TINA: A natural language system for spoken language ap-
plications,” Computational Linguistics, vol. 18, no. 1, pp. 61–86, 1992.
[19] Y. Y. Wang, “A robust parser for spoken language understanding,” in
Proc. Eurospeech, 1999, pp. 2055–2058.
[20] N. Gupta, G. Tur, D. Hakkani-Tur, S. Bangalore, G. Riccardi, and M.
Gilbert, “The AT&T spoken language understanding system,” IEEE
Trans. Audio, Speech, and Language Processing, vol. 14, no. 1, pp.
213–222, 2006.
[21] Y. He and S. Young, “Semantic processing using the hidden vector state
model,” Computer Speech & Language, vol. 19, no. 1, pp. 85–106,
2005.
[22] W. Minker, S. Bennacef, and J. L. Gauvain, “A stochastic case frame
approach for natural language understanding,” in Proc. Int. Conf.
Spoken Lang. Process., 1996, pp. 1013–1016.
[23] M. Rochery, R. Schapire, M. Rahim, N. Gupta, G. Riccardi, S. Ban-
galore, H. Alshawi, and S. Douglas, “Combining prior knowledge and
boosting for call classification in spoken language dialogue,” in Proc.
Int. Conf. Acoust,, Speech, Signal Process., 2002, pp. I-29–I-32.
[24] Y. Y. Wang, A. Acero, M. Mahajan, and J. Lee, “Combining statistical
and knowledge-based spoken language understanding in conditional
models,” in Proc. Comput. Linguist., Annu. Meeting Assoc. Comput.
Linguist., 2006, pp. 882–889.
[25] C. Wutiwiwatchai and S. Furui, “Combination of finite state automata
and neural network for spoken language understanding,” in Proc. Eu-
rospeech, 2003, pp. 2761–2764.
[26] A. Stolcke, K. Ries, N. Coccaro, E. Shriberg, R. Bates, D. Jurafsky,
P. Taylor, R. Martin, C. V. Ess-Dykema, and M. Meteer, “Dialogue
act modeling for automatic tagging and recognition of conversational
speech,” Comput. Linguist., vol. 26, no. 3, pp. 339–373, 2000.
[27] M. Jeong and G. G. Lee, “Triangular-chain conditional random
fields,” IEEE Trans. Audio, Speech, Lang. Process., vol. 16, no. 7, pp.
1287–1302, Sep. 2008.
[28] G. Tur, D. Hakkani-Tür, and R. E. Schapire, “Combining active and
semi-supervised learning for spoken language understanding,” Speech
Commun., vol. 45, no. 2, pp. 171–186, 2005.
[29] W. L. Wu, R. Z. Lu, J. Y. Duan, H. Liu, F. Gao, and Y. Q. Chen,
“Spoken language understanding using weakly supervised learning,”
Comput. Speech Lang., vol. 24, no. 2, pp. 358–382, 2010.
[30] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,”
J. Mach. Learn. Res., vol. 3, pp. 993–1022, 2003.
[31] S. Joty, G. Carenini, and C. Y. Lin, “Unsupervised modeling of dialog
acts in asynchronous conversations,” in Proc. Int. Joint Conf. Artif.
Intell., 2011, pp. 1807–1813.
[32] A. Ritter, C. Cherry, and B. Dolan, “Unsupervised modeling of twitter
conversations,” in Proc. North Amer. Chap. Assoc. for Comput. Lin-
guist.: Human Lang. Technol., 2010, pp. 172–180.
[33] J. Guo, G. Xu, X. Cheng, and H. Li, “Named entity recognition in
query,” in Proc. Assoc. Comput. Machinery’s Special Interest Group
Inf. Retrieval, 2009, pp. 267–274.
[34] D. Newman, C. Chemudugunta, and P. Smyth, “Statistical entity-topic
models,” in Proc. 12th Assoc. Comput. Machinery’s Special Interest
Group Knowl. Discov. Data Mining Int. Conf. Knowl. Discov. Data
Mining, 2006, pp. 680–686.
[35] A. Celikyilmaz and D. Hakkani-Tur, “A joint model for discovery of
aspects in utterances,” in Proc. Assoc. Comput. Linguist., 2012, pp.
330–338.
[36] E. B. Fox, E. B. Sudderth, M. I. Jordan, and A. S. Willsky, “An
HDP-HMM for systems with state persistence,” in Proc. Int. Conf.
Mach. Learn., 2008, pp. 312–319.
[37] Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei, “Hierarchical
dirichlet processes,” J. Amer. Statist. Assoc., vol. 101, no. 476, pp.
1566–1581, 2006.
[38] N. Crook, R. Granell, and S. Pulman, “Unsupervised classification of
dialogue acts using a Dirichlet process mixture model,” in Proc. Assoc.
Comput. Linguist., 2009, pp. 341–348.
[39] R. Higashinaka, N. Kawamae, K. Sadamitsu, Y. Minami, T. Meguro,
K. Dohsaka, and H. Inagaki, “Unsupervised clustering of utterances
using non-parametric Bayesian methods,” in Proc. Interspeech, 2011,
pp. 2081–2084.
[40] R. Barzilay and L. Lee, “Catching the drift: Probabilistic content
models, with applications to generation and summarization,” in Proc.
North Amer. Chap. Assoc. Comput. Linguist.: Human Lang. Technol.,
2004, pp. 113–120.
[41] C. Li, J. Weng, Q. He, Y. Yao, A. Datta, A. Sun, and B. S. Lee, “Twiner:
Named entity recognition in targeted twitter stream,” in Proc. Assoc.
Comput. Machinery’s Special Interest Group Inf. Retrieval, 2012, pp.
721–730.
2464 IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 21, NO. 11, NOVEMBER 2013
[42] S. Geman and D. Geman, “Stochastic relaxation, Gibbs distributions,
and the Bayesian restoration of images,” IEEE Trans. Pattern Anal.
Mach. Intell., vol. PAMI-6, no. 6, pp. 721–741, Nov. 1984.
[43] H. Murao, N. Kawaguchi, S. Matsubara, Y. Yamaguchi, and Y. In-
agaki, “Example-based spoken dialogue system using WOZ system
log,” in Proc. 4th Annu. Meeting Spec. Interest Group Discourse Dial.,
2003, pp. 140–148.
[44] M. Nagao, “A framework of a mechanical translation between Japanese
and English by analogy principle,” in Proc. Int. NATO Symp. Artif.
Human Intell., 1984, pp. 173–180.
[45] J. D. Lafferty, A. McCallum, and F. C. N. Pereira, “Conditional
random fields: Probabilistic models for segmenting and labeling
sequence data,” in Proc. Int. Conf. Mach. Learn., 2001, pp. 282–289.
[46] A. Ratnaparkhi, “Maximum entropy models for natural language ambi-
guity resolution,” Ph.D. dissertation, Univ. of Pennsylvania, Philadel-
phia, PA, USA, 1998.
[47] G. Salton and C. S. Yang, “On the specification of term values in auto-
matic indexing,” J. Document., vol. 49, no. 4, pp. 351–372, 1973.
[48] C. Lee, S. Jung, K. Kim, and G. G. Lee, “Hybrid approach to robust di-
alog management using agenda and dialog examples,” Comput. Speech
Lang., vol. 24, no. 4, pp. 609–631, 2010.
[49] C. Lee, S. Jung, K. Kim, and G. G. Lee, “Automatic agenda graph
construction from human-human dialogs using clustering method,”
in Proc. North Amer. Chap. Assoc. Comput. Linguist.: Human Lang.
Technol., 2009, pp. 89–92.
[50] V. Rieser and O. Lemon, “Learning effective multimodal dialogue
strategies from Wizard-of-Oz data: Bootstrapping and evaluation,” in
Proc. Assoc. Comput. Linguist., 2008, pp. 638–646.
[51] N. X. Vinh, J. Epps, and J. Bailey, “Information theoretic measures for
clusterings comparison: Is a correction for chance necessary?,” in Proc.
Int. Conf. Mach. Learn., 2009, pp. 1073–1080.
[52] A. Rosenberg and J. Hirschberg, “V-measure: A conditional entropy-
based external cluster evaluation measure,” in Proc. Empir. Meth. Nat-
ural Lang. Process. Comput. Nat. Lang. Learn., 2007, pp. 410–420.
[53] E. Levin, R. Pieraccini, and W. Eckert, “A stochastic model of human-
machine interaction for learning dialog strategies,” IEEE Trans. Speech
Audio Process., vol. 8, no. 1, pp. 11–23, Jan. 2000.
[54] S. Jung, C. Lee, K. Kim, M. Jeong, and G. G. Lee, “Data-driven
user simulation for automated evaluation of spoken dialog systems,”
Comput. Speech Lang., vol. 23, no. 4, pp. 479–509, 2009.
[55] E. Ponvert, J. Baldridge, and K. Erk, “Simple unsupervised grammar
induction from raw text with cascaded finite state models,” in Proc.
Proc. 49th Annu. Meeting Assoc. Comput. Linguist.: Human Lang.
Technol., 2011, pp. 1077–1086.
Donghyeon Lee is a Ph.D. student at the Depart-
ment of Computer Science and Engineering at
POSTECH, Pohang, South Korea. He received his
B.S. degree at the Department of Computer Science
and Engineering at Sung Kyun Kwan University,
Suwon, South Korea. His research interests include
unsupervised spoken language understanding and
spoken dialog system.
Minwoo Jeong is an applied scientist at Microsoft
Corporation. He received his B.S. degree at the De-
partment of Computer Engineering from Chonbuk
National University, Jeonju, South Korea. He re-
ceived his M.S./Ph.D. degrees at the Department of
Computer Science and Engineering from POSTECH,
Pohang, South Korea. His research interests include
spoken language understanding and spoken dialog
system.
Kyungduk Kim is a Ph.D. student at the Department
of Computer Science and Engineering at POSTECH,
Pohang, South Korea. He received his B.S./M.S. de-
grees at the Department of Computer Science and En-
gineering from POSTECH. His research interests in-
clude multi-modal dialog system and robust dialog
management.
Seonghan Ryu is a Ph.D. student at the Department
of Computer Science and Engineering at POSTECH,
Pohang, South Korea. He received his B.S. degree
at the Department of Computer Science and Engi-
neering at Dongguk University, Seoul, South Korea.
His research interests include multi-domain spoken
dialog system, robust dialog management, and ex-
ploiting web resources.
Gary Geunbae Lee received his B.S. and M.S.
degrees in Computer Engineering from Seoul Na-
tional University in 1984 and 1986 respectively. He
received the Ph.D. degree in Computer Science from
UCLA in 1991 and was a research scientist in UCLA
from 1991 to 1991. He has been a professor at the
CSE department, POSTECH in Korea since 1991.
He is a director of the Intelligent Software laboratory
which focuses on human language technology
research including natural language processing,
speech recognition/synthesis, and speech translation.
He authored more than 100 papers in international journals and conferences,
and has served as a technical committee member and reviewer for several
international conferences such as ACL, COLING, IJCAI, ACM SIGIR, AIRS,
ACM IUI, Interspeech-ICSLP/EUROSPEECH, EMNLP and IJCNLP. He
is currently leading several national and industry projects for robust spoken
dialog systems, computer assisted language learning, and expressive TTS.