Issues in Text-to-Speech Synthesis
Marian Macchi
Bellcore
mjm@bellcore.com
Abstract
The ultimate goal of text-to-speech synthesis is to convert
ordinary orthographic text into an acoustic signal that is
indistinguishable from human speech. Originally,
synthesis systems were architected around a system of
rules and models that were based on research on human
language and speech production and perception
processes. The quality of speech produced by such
systems is inherently limited by the quality of the rules
and the models. Given that our knowledge of human
speech processes is still incomplete, the quality of text-to-
speech is far from natural-sounding. Hence, today’s
interest in high quality speech for applications, in
combination with advances in computer resource, has
caused the focus to shift from rules and model-based
methods to corpus-based methods that presumably bypass
rules and models. For example, many systems now rely
on large word pronunciation dictionaries instead of
letter-to-phoneme rules and large prerecorded sound
inventories instead of rules predicting the acoustic
correlates of phonemes. Because of the need to analyze
large amounts of data, this approach relies on automated
techniques such as those used in automatic speech
recognition.
1. Introduction
The ultimate goal of text-to-speech synthesis is to
convert ordinary orthographic text into an acoustic signal
that is indistinguishable from human speech. The
conversion process, illustrated in Figure 1, is considered
to have two parts, because the two parts involve different
types of knowledge and processes. The front end handles
problems in text analysis and higher level linguistic
features; it interprets orthographic text and outputs a
phonetic transcription that specifies the phonemes and an
intonation for the text. The back end handles problems in
phonetics, acoustics and signal processing; it converts the
phonetic transcription into a speech waveform containing
appropriate values for acoustic parameters such as pitch,
amplitude, duration, and spectral characteristics.
For both parts the conversion process is usually
performed through one of two approaches. One approach
is to rely on rules formulated either by experts in natural
language, linguistics, or acoustics. Another approach is to
avoid rules and instead include large lists from machine-
readable dictionaries or to formulate rules by automatic
methods from statistical analysis of a large corpus of
transcribed speech. The trend today is to use the corpus-
based approach ([1-6]). The dependency on large
amounts of data processing necessitates the use of
automated techniques such as those used in automatic
speech recognition. Crucial issues for this approach are
the coverage of the corpus and how a system deals with
cases outside the corpus.
2. Front-end of text-to-speech
The goal of the front-end of a synthesizer is to
convert text into phonetic transcription. Because text is an
impoverished representation of a speaker’s verbal
intentions, the front-end must attempt to regularize the
text and resolve ambiguities by inferring the speaker’s
intention from the text.
2.1. Text normalization.
As illustrated in Table 1, text contains many
occurrences of strings that are not pronounceable as such.
Some of these strings can be interpreted in a context-free
fashion and can be handled by a simple table lookup. For
example, the abbreviation “Mr.” can be replaced with the
word “Mister”. Number strings are more complicated
and cannot be handled by a dictionary per se. Even
simple number cases require an algorithm that assigns
interpretation to the entire number.
In natural language, acronyms are sometimes spelled
out, as is “NAACP”, and are sometimes pronounced as
words, as is “VISA”. A synthesizer must have some
mechanism for determining whether such uppercase text
should be pronounced or spelled out. Some synthesizers
use a dictionary listing the most common acronyms and
then simply spell out all other occurrences of upper case
text. Another approach is to have algorithms based on
constraints of letter sequences in actual words. For
example, a word consisting of a long sequence of
consonants or of vowels, such as “JKLHP” and
“OAEIE,” is not pronounceable as an English word.
Consequently, such sequences should be spelled out. An
advantage of the algorithmic approach is that it can be
 O rth o g ra p h y T h e c h ild re n re a d to D r. S m ith .
could collect the set of N-grams for “St.,” consisting of
T e x t N o rm a liz a tio n
[re a d ] d o c to r
all translations of “St.” plus the adjacent words or word
] [V p re s ] [ categories. For each N-gram, we would tabulate the
F ro n t E n d W o rd P ro n u n c ia tio n
rid
+
d a k tR
number of “St.”s representing “Saint” and the number
In to n a tio n
representing “Street,” as illustrated in Table 2 using
bigrams. Then, one simple approach is to use the ratio of
+ + + +
P h o n e tic % D x C Ild rx n rid tx d a k tR s m IT % occurrences of the different pronunciations as a measure
T ra n s c rip tio n
A rtic u la to ry M o d e l
of the disambiguation power of the context. So for each
Back End
S y n th e s is -b y -R u le N-gram, we would compute the ratio of the two
or
C o n c a te n a tiv e s y n th e s is pronunciations and store them in a table. For synthesis,
we would choose the highest-valued ratio in the table for
A c o u s tic the matching context.
7DEOH$EEUHYLDWLRQGLVDPELJXDWLRQXVLQJELJUDPV

)LJXUH7H[WWRVSHHFKOHYHOV Bigram Word #occurrences Ratio

St. John = Street 1 1/177=.005

applied to material that contains only uppercase text, as is St. John = Saint 177 177/1=177
the case with many databases.
John St. = Street 37 37/10=3.70
Context-dependent text normalization of homographs John St. = Saint 10 10/37=.270
can be resolved in different ways. One approach is to
rely on restrictions of the particular application of text-to- Disambiguation Table
speech. For example, if an application is designed to St. John > Saint 177.0
.
speak only names, then the abbreviation “St.” becomes .
John St. > Street 3.7
7DEOH7H[WQRUPDOL]DWLRQH[DPSOHV
This same methodology can be used with word
Context-free categories, like part of speech or other labels. A
Mr. Mister disadvantage of using words is the coverage of the
22 twenty-two corpus: for example, the name “Aloysius” might not
$N N dollars
NAACP N. A. A. C. P. appear in a corpus. Another disadvantage of the word
VISA visa category approach is that the categories may be too
Johnd@xyz.com John D. at XYZ dot com
Context-dependent coarse.
St. saint street The same basic approach can be used to resolve
Dr. doctor drive
Jesus Jesus /hAzus/ semantic ambiguities like that of “bass”, which has two
1998 nineteen ninety-eight pronunciations, corresponding to different meanings, one
one thousand nine hundred….
one nine nine eight referring to “bass” as a type of fish and one referring to
read /rid/ (Vpres) /rEd/ (Vpast) “bass” in music. In this way, “bass” preceded by the word
convict /kxnvikt/ (Verb) /kanvikt/ (Noun)
bass /bAs/ (music) /bas/ (fish) “sea”, would refer to the fish. However, non-local context
(for example, words occurring anywhere in the same
unambiguously “Saint”. However, most applications sentence or paragraph) may be needed to disambiguate
require a more general solution. One possibility is to such semantic ambiguities. For example, in a text
attempt to resolve the ambiguity through a list of nearby containing the word “opera”, “bass” would most likely
words. For example, the abbreviation “St.” followed by refer to the musical term. Given that text is an
“John”, “Joseph”, etc. would be “Saint.” Another impoverished representation of a speaker’s intention,
possibility is to use word categories such as proper noun many such cases cannot be resolved with high probability
or first name, and specify that “St.” followed by that even by a human reading text; however, a synthesizer
category would be “Saint.” The list of disambiguating must choose some pronunciation, and the choice can be
cases can be prepared either via the intuition of the based on general relative frequency of occurrence of one
researcher or automatically, through statistical analysis of pronunciation versus another.
a pre-labeled corpus. Thus, in both context-free and context-dependent
Typically, statistical analysis is based on the normalization, table lookup translations can be derived
distribution of occurrences of the abbreviation plus one or either by intuition or by statistical analysis of a large
two preceding or following words. For example, given a corpus. A problem with intuition-based lists is that
corpus of text in which the abbreviation “St.” has been preparation of the list is labor intensive and subject to
translated by a human into either “Saint” or “Street”, we errors of omission and subjectivity of the preparer. A
problem with the automatic, statistical preparation method
is that unless the corpus on which the analysis is based is
extremely large, the coverage of the corpus material will
suffer from errors of omission. Further, automatic
methods require that the corpus be pre-labeled by a
human with the correct expansions, a process which is
itself labor intensive and error-prone.
Part-of-speech homonyms (“read” /rEd/, rhyming
with “bread”, as past tense of the verb; “read” /rid/,
rhyming with “seed”) are also context-dependent
ambiguities. They can be resolved, in general, by a part
of speech tagger. Today, most taggers are based on
methodology akin to that discussed above, using statistics
of short, 2- or 3-word sequences or part-of-speech label
sequences that occur in a corpus that has been prelabeled
with part of speech for each word. Syntactic parsers are
not used today, due to their computational demands. The
utility of this approach depends in part on the granularity
of the part-of-speech labels as well as on the size of the
corpus.
2.2. Word pronunciation
Given the words, a synthesizer converts orthography
for each word into phonemes and stress. To perform this
conversion, synthesizers typically use a combination of a
dictionary and rules, as shown in Figure 2.
The first process in word pronunciation is usually
dictionary lookup; words found in the dictionary need not
be processed further. In the original conception of a word
pronunciation component, the dictionary contains
pronunciations for only those words that are exceptions to
the rules. For example, in English the word ‘two’ is
pronounced /tu/, which is unpredictable from any English
rules, so that it would definitely appear in the dictionary.
However, what words constitute exceptions can be only
defined in terms of the rules. Compared to many other
languages, the correspondence between orthography and
phonemes in English is particularly complex, and
therefore, it is a challenge to formulate a comprehensive
set of rules.
Consequently, one current approach to word
pronunciation is to greatly expand the dictionary to cover
a large number of cases and to depend on rules only for
backup. In this way, the dictionary would implicitly
build-in all the rules of word pronunciation. Given the
availability of machine-readable dictionaries, it might
seem sufficient simply to use a dictionary and dispense
with any rules. However, many applications for speech
synthesis involve items that are not covered by readily-
available dictionaries, for example, inflected words (e.g.,
“sings” versus “sing”) and proper names. Solving the
problem of pronouncing out-of-vocabulary words
involves either compiling one’s own dictionary or
developing other strategies, like traditional rules or
analogy systems. A typical partial solution is to prepare a
Dictionary
Beethoven > /be’ to vxn/
two > /tu/
read, Vpres > /rid/
Ethnic Classification Rules
Letourneaux > “french”
Fujisaki > “japanese”
Morphological Rules
hopeful > hope + ful
hoped > hope + ed
baseline > base + line
Letter-to-Phoneme Rules
o : _{C}e$ > /o/ (hope)
o > /a/ (hop)
u, “japanese” > /u/ (fuji)
u > /yu/ (fugitive)
Syllabification & Lexical Stress
/sInTxsYzR/ > /sI’n Tx sY` zR/
Pronunciation: phonemes & stress
)LJXUH:RUGSURQXQFLDWLRQFRPSRQHQW
dictionary covering the most frequently occurring words
and to use rules for the other cases.
Morphological decomposition is often used in
synthesis systems because some letter-to-phoneme rules
can be stated more simply if the rules refer to morphemes.
For example, in English, final “e” in many words is silent,
while the medial letter “e” is often pronounced. In
compound words in which a non-final morpheme ends in
“e”, the “e” is also silent. So, for example, in the word
“Vaseline”, the medial “e” is pronounced as a schwa,
while in the word “baseline”, which is composed of two
morphemes (“base” + “line”), the “e” in “base” is silent.
If morphological rules decompose compounds, the same
silent “e” pronunciation rule can be used to refer to both
word and morpheme-final silent “e.” Also, inflectional
affixes (“-s”, “-ed”, “-ing”) are typically split off for
letter-to-phoneme rules, because inflectional affixes do
not affect the pronunciation of the base word. However,
morphological decomposition is not trivial, since all
occurrences of these letter sequences do not function as
inflectional affixes (“Ahmed”, “bring”), and many
possible decompositions are not, in fact, correct
decompositions (for example, “Vaseline” is not
decomposed into “vase” + “line”).
Some synthesizers tag words and names as to ethnic
identity, assuming that certain letter sequences are
pronounced differently depending on the ethnic origin of
the word. Finally, letter-to-phoneme rules map letter
sequences into sequences of phonemes and stress.
Because many applications for text-to-speech involve
pronouncing the names of people and places, and because
no complete pronouncing dictionaries for names exist,
much attention has been directed toward name
pronunciation. This subproblem is particularly
challenging in the United States, where the population is
ethnically diverse and therefore has many names from
different ethnic origins. Name pronunciation is one area
where text-to-speech may have the potential of
performing better than a single person. That is, since no
single person can know how to pronounce all names
correctly, it might be possible for a text-to-speech system
to cull knowledge about name pronunciation and perform
better, on average, than a person. Although synthesizers
might make more serious pronunciation errors than a
person would make, some synthesizers actually perform
as well or better than educated humans at pronouncing
names with low frequency of occurrence.
7DEOH$FFHQWSUHGLFWLRQIDFWRUV
Part of Speech
Content word: Which book did John leave?
Preposition: Which car did John leave in?
Particle: Which paragraph did John leave in?
Lexical accent
Normal case
large number
John Smith
long street
Walnut Drive
Word compounds with deaccented last word
telephone number
school building
heart attack
Walnut Street
Given versus new information
Lexical match
After John brought me an apple, I ate the apple.
Morphology
John brought me an apple, but I don’t like apples.
John likes to swim, but I hate swimming.
Compounds
After John felt pains near his heart, he took better care
of his heart.
After John had a heart attack, he took better care of his
heart.
Real world knowledge
John brought me an apple, but I don’t like fruit.
After I met Bill Clinton, I had lunch with the President.
2.3 Intonation
In natural speech, the same phonemic sequence can
occur with a variety of different intonational patterns:
accent, melody, and phrasing. A speaker’s selection of an
intonation is not random; the choice corresponds to
different syntactic, semantic and discourse-level factors.
However, many of these factors are not explicitly
indicated orthographically in text; a synthesizer, like a
human who reads aloud from a text, must infer them.
Accent. In natural speech, certain words are more
prominent or emphasized more than other words.
Consequently, a synthesizer must assign accent to some
words in sentences. Many factors affect accent
placement, as illustrated in Table 3.
Accent is in part correlated with part of speech. An
oft-cited default accent rule is to place the primary accent
on the last content word of a sentence. Function words
like auxiliary verbs and prepositions tend to be
deaccented. However, words that can be used as
prepositions are accented when they serve as verb
particles. Determining the particle/preposition distinction
is often handled via part-of-speech tagging or by other
general text normalization procedures, but the resolution
is often difficult, as suggested by the cases in Table 3.
Another factor that affects accent is noun phrase
structure. Some noun phrases are accented on the last
word (e.g., “Avenue,” as in “Fifth Avenue”) and some are
accented on earlier words (e.g., “Fifth Street”). One
approach to predicting accent is to list word compounds
whose accent does not fall on the last word. Another
approach is to formulate more lexically based or semantic
rules. For example, in proper names of streets, most
street words are accented (“Avenue”, “Drive”, “Lane”,
“Road”), but “Street” is always deaccented and causes
accent to shift to the previous word.
However, accent is also determined by more global
semantic properties of the text. In the absence of any
special information, speakers often accent the last word in
a phrase. However, speakers do not accent previously
given information in many situations (in Table 3, given
words are shown in italics, and accented words in the last
phrase are underlined.). So for example, in first example,
the second occurrence of “apple” is deaccented.
Identifying what information is given is clearly a problem
in natural language processing. However, synthesizers
adopt heuristics for deaccenting. For example, one
approach to identifying given information is to rely on
lexical match. However, as other examples in Table 3
show, correct resolution requires identification of
morphology, compounding, and perhaps most difficult,
real-world knowledge.
Phrasing. In natural speech, speakers often break up
sentences into several phrases, which can be articulated
with pausing and which serve as the domain of certain
types of accent placement, timing and melody rules.
Many factors play a role in determining intonational
phrases (Table 4). Sometimes punctuation can serve as a
guide, but in other cases, punctuation can be misleading,
and in other cases, no punctuation is given when phrase
breaks should be inserted. One possibility would be to
determine syntactic constituency via a parser, and to use
the syntactic phrases in combination with other factors,
such as phrase length, to decide on intonational phrases.
One approach to predicting intonational phrases uses
statistical analysis of a corpus that has been pre-labeled
for phrase boundaries, using part of speech and other
automatically derivable word categories.
Melody. Melody refers to the pattern of tones with
which a phrase, sentence, or paragraph is spoken. Much
early work on melody did not include any independent
level of representation for melody. That is, synthesizers
simply mapped from a sentence type to a physical pitch

7DEOH6RPHIDFWRUVLQSKUDVLQJ
Punctuation
201-234-5678
(A + B) * C
We saw Bill, and Mary saw Bob.
Syntactic parse
We saw Bill and Mary leave.
We saw Bill and Mary saw Bob.
time function. Since then, a great deal has been learned
about the syntax of tones. However, much less is known
about predicting the relationship between particular
melodic patterns and syntactic, semantic, and discourse-
level factors. Synthesizers typically have a relatively
small taxonomy of melody patterns, and in synthesis,
assign utterances one of the patterns. For example,
deaccented words are not assigned tones, the so-called
‘yes-no’ questions can end in a rising tone pattern, while
‘wh-‘ questions do not; lists can exhibit particular
downstepping melodies. The conversion of the pattern of
tones to a physical pitch time function is usually
considered a problem for the back-end of the synthesizer.
Timing. The timing or duration of speech is not
usually considered a feature of the front-end of a
synthesizer per se. Instead, timing is usually considered a
problem related to the back-end of a synthesizer, under
the assumption that the durational effects to be modeled
are a function of other factors that are part of the phonetic
transcription. Factors that are often considered to
determine timing include phonemic identity, syllable
structure, phrasing, accent, melody, speaking rate, and
intonational structures like the word or the metrical foot.
3. Back-end of text-to-speech
The goal of the back end of a speech synthesizer is to
convert a given phonetic transcription into an acoustic
signal. The back end addresses the problem that the same
phoneme can have many different acoustic realizations,
depending on many factors, for example, the identity of
nearby phonemes, stress, emphasis, and position in a
phrase. Further, the back end must realize the intonation
specified in the phonetic transcription as acoustic
parameters such as pitch, duration, and amplitude.
3.1 Approaches
Synthesis systems traditionally adopt one of three
general approaches to converting a phonetic transcription
into an acoustic signal: articulatory synthesis, synthesis-
by-rule, synthesis-by-concatenation.
Articulatory synthesis is based on a model of the
human vocal tract, and has articulatory parameters
corresponding to organs such as the lips, tongue, velum,
and vocal folds. The ultimate research solution for speech
synthesis is probably based on this approach. Articulatory
synthesis requires two types of rules: linguistic rules
specifying the relation between phonetic transcription and
articulatory activity and physical rules specifying the
relation between articulatory activity and acoustic signal.
Typically, the linguistic rules are formulated phoneme by
phoneme. The linguistic rules specify, for example, when
the articulators move to produce a phoneme, where they
move, how fast they move, how they are coordinated with
movements of other articulators. Because a phoneme need
not have movements specified for all articulators,
different articulators can show movements for different
phonemes at the same point in time. In this way,
articulatory synthesis can provide elegant solutions to
some difficult problems, particularly for the phonetics of
consonant clusters and syllable structure. The physical
rules in articulatory synthesis specify how particular
vocal tract configurations map onto an acoustic signal.
Both linguistic and physical rules, as well as the
articulatory model itself, crucially affect the quality of the
synthesized speech. However, current knowledge about
all the details of both sets of rules and about the model is
incomplete, and consequently the speech produced by
articulatory synthesis is not completely natural-sounding.
For these reasons, articulatory synthesizers today are
research laboratory tools or prototypes and are not used in
commercial applications.
Synthesis-by-rule is based on a model of the
perceptually significant acoustic parameters underlying
speech, such as formants, bandwidths, nasal zeros,
aspiration, and voicing. This approach also involves
several sets of rules. Linguistic rules specify so-called
formant “target” positions for each phoneme in the
utterance and the timing of those target positions. Other
rules smooth the target sequence and convert the resulting
time functions into an acoustic waveform. Synthesis-by-
rule has provided much insight into speech production
and perception. However, for text-to-speech systems, the
speech quality produced by synthesis-by-rule suffers from
the fact that current knowledge about the rules and of the
acoustic model itself is incomplete.
3.2 Concatenative Synthesis
The original rationale for synthesis-by-concatenation
was to overcome the inadequacies of rule-based systems
by eliminating the need for some, or perhaps even all, of
the rules and models. The way this is done is by storing
an inventory of sound units that are not models at all, but
rather are fragments of human speech, usually consisting
of a sequence of several phonemes. The units can then be
concatenated to form a new utterance. So in the
concatenative approach, the inventory itself incorporates
some of the acoustic variation that must be produced by
rules in other systems.
 Most high quality speech synthesizers today are
concatenative synthesizers. In a concatenative system, a
person records speech containing a large set of basic
sound units, usually corresponding to a relatively short
sequence of phonemes. The units are excised from the
speech, and in most systems, the units are processed with
some type of speech coding method, and the resulting
templates are stored in an inventory. For synthesis of a
new utterance, given a phonetic transcription, the system
uses rules to select the appropriate units, extracts them
from the inventory and concatenates them.
Typically, a synthesis-by-concatenation system
lib e lib e l
1 - s y ll 2 -s y ll
F 0
S p e c tru m
W a v e
l a y b l a y b
)LJXUH  'XUDWLRQ DQG SLWFK IRU OD\E LQ ´OLEHµ DQG
´OLEHOµVSRNHQLQLVRODWLRQZLWKIDOOLQJWRQHSDWWHUQ
contains only one token of each unit. Since pitch,
duration, and amplitude for a short phoneme sequence
vary widely with different intonational features (Figure 3)
the system must modify the suprasegmental acoustic
features of the units, like the pitch, duration and
amplitude, so that the concatenated sequence of units
represents the appropriate intonation. Usually, a set of
rules modifies the templates by adjusting these parameters
and may also smoothe across the boundary at the splicing
point.
The quality of the speech produced by concatenative
synthesis is determined by the choice of the basic
concatenative unit, the rules for concatenating and
modifying the units, and the speech coding method.
Speech coding methods. The reason that
concatenative synthesizers need speech coding methods is
to allow the synthesizer to change the pitch and duration
of the templates without changing the phonemic identity
of the unit. Coding methods are signal processing
techniques that attempt to separate the source signal
(vocal fold vibration or noise frication) from acoustic
parameters representing the vocal tract and so allow
independent control over the source from the vocal tract.
The earliest speech coding method used in
concatenative synthesis was LPC (linear predictive
coding). The LPC model decomposes the waveform into
acoustic parameters representing formants, pitch, and
amplitude over short 5- or 10-msec intervals. Because
LPC has a very simplified model of the source (glottal
fold vibration), the speech it produces is somewhat
robotic-sounding. For this reason, newer systems are
pursuing alternative coding methods that provide a more
enriched representation. Some of these methods are based
on LPC (e.g., MPLPC, or multipulse LPC, RELP, or
residual-excited LPC) but have more elaborate
representation of glottal fold vibration. An extremely
simple coding method, PSOLA (pitch-synchronous
overlap and add) uses raw waveforms in which the pitch
period onsets have been labeled. In PSOLA, a template is
shortened by eliminating pitch periods; a template is
lengthened by repeating pitch periods; pitch is raised by
overlapping the native pitch periods more closely together
in time; pitch is lowered by spreading out the intervals.
However, with any of these techniques, when the
synthesizer imposes large changes to the native pitch and
l duration of the inventory templates, as is required in
synthesis, the resulting speech quality is often degraded.
Consequently, a new approach to synthesis-by-
concatenation attempts to avoid using any speech coding
at all by using only natural speech. To avoid having to
change pitch and duration for a template, it is necessary to
have multiple variants of each sound unit, representing all
the possible pitch and duration values with which that
unit could occur in natural speech. While this approach
might avoid the speech quality distortions associated with
speech coding methods, because of the large range of
pitch and duration variation that exists in a language, this
pure concatenation method requires huge sound
inventories. Further, since manual labeling of such a large
inventory is prohibitive if not impossible, it is necessary
to devise automatic methods to label the speech; today
tools borrowed from automatic speech recognition such
as HMM (Hidden Markov Modeling) are being used to
that end.
Basic sound units. There is obviously an interaction
between the number and size of the sound unit inventory
and the number and complexity of the rules that are
needed. Traditionally, the set of units has been selected
according to linguistic criteria. Figure 4 lists a variety of
different kinds of units that might be considered for
concatenative synthesis. Some of the units have been
proposed based on knowledge about the kinds of phonetic
factors that cause acoustic variation in phonemes, for
example, allophones. Some are based on the pragmatics
of an application, for example phrases and sentences,
which are typically used in voice response systems.
 In general, the longer the unit, the more information
it contains, and therefore fewer or less complex rules are
needed. To the extent that the rules are not completely
understood, the higher quality the speech. Most
concatenation systems are based on relatively small units:
allophones, diphones, triphones, or demisyllables.
Recently, through the availability of large amounts of
computer memory, longer units are not out of the
question, and it is possible to envision systems that blend
voice response with text-to-speech and systems that have
huge sound inventories.
Figure 5 illustrates one issue associated with relying
on large inventories: namely, the coverage of the corpus.
Figure 5 shows, for different kinds of units, the number of
other names in the language cannot be synthesized from
an inventory based on 50K names. Synthesizing the next
100 names after the 50K names will require 83 more
sound units. Consequently, if coverage is inadequate, the
synthesizer needs to adopt a strategy for synthesizing
names that require missing units. One possibility is to fall
back on subunits of existing units; however, this method
will succeed only if subunit concatenation rules are good.
a
^b_a
b_a

units needed to synthesize subsets of the most frequent

50K names in the United States. As the length of the unit b_a_s
b_a_^s d_a ^d_a

Unit Length Unit # Units # Rules Quality

(English)
Short Many Low
Phoneme/Allophone 40/65
Diphone <40 2 -65 2 )LJXUH  $XWRPDWLF VHOHFWLRQ RI SKRQHPHVL]HG XQLWV
Triphone <40 3 -65 3 XVLQJVWDWLVWLFDOFOXVWHULQJ
Demisyllable 2K
Syllable 11K One goal of a sound unit inventory is to represent all
VC*V
the perceptually significant acoustic variations that occur
2-syllable <11K 2
Word 100K-1.5M in natural speech. Therefore, one approach is to use
Phrase statistical clustering techniques in combination with some
Sentence
8

Long Few High physical, phonetic distance measures to automatically

select a set of units given a large database that has been
phonemically labeled. Typically, the distance measures
)LJXUH7UDGLWLRQDOVHJPHQWDOXQLWVIRUFRQFDWHQDWLYH are those that have frequently been used in automatic
V\QWKHVLV speech recognition. As shown in Figure 6, the basic
increases, the number of units increases, and, more method involves computing the distance between all units
importantly, the slope of the coverage line increases. with the same phonetic labels, and using hierarchical
That is, if the basic sound unit is the diphone or clustering techniques to minimize the distance among
demisyllable, only a few thousand units are needed to units within the same cluster and maximize the distance
between clusters. The inventory units are represented by
the final leaves of the tree. Figure 6 shows a leaf
50000 2-Syllable Slope
.83
VC*V representing the phoneme /a/ preceded by /d/, for
Triphones
40000 Syllables example. This technique can be used to select units on the
Demisyllables
Diphones basis of segmental environment alone or on the basis of
30000
# Units

both segmental features and suprasegmental features like

20000
10000
0 1
1 10K 20K 30K
Top N Surnam es (rank)
)LJXUH&RUSXVFRYHUDJHIRUGLIIHUHQWVRXQGXQLWV
synthesize all 50K names, and the low coverage slope
means that their coverage is good; few additional units
are needed to cover the rest of the language. However, if
the basic sound unit is two-syllable unit, the slope of the
coverage line is very steep; the probability is high that
.23
pitch and duration. In addition to the issue of coverage, a
.15 challenge for this kind of approach is the perceptual
.11
reality of the physical distance measure that forms the
.02
.003
basis of the clustering.
40K 50K 1.5 M In another approach, there are no predefined sound
units or templates per se. Rather the sound inventory
consists of a large corpus of prerecorded speech, which
has been labeled in terms of phonemes and sometimes
prosodic features. In synthesis, units matching the
requirements of the new utterance are extracted from the
corpus on a case-by-case basis, using various criteria, as
illustrated in Figure 7. In this approach, the concatenation
point between two units is also determined on a case-by-
case basis, using some objective measure of the distance
 New utterance to synthesize
Turn right on Walnut Drive
turn right off South
right on Woodland
Units in Walnut Creek
Scott Drive
)LJXUH  8QLW VHOHFWLRQ DQG FRQFDWHQDWLRQ ZLWK QRQ
XQLIRUPXQLWV&DQGLGDWHXQLWVDUHXQGHUOLQHGVSOLFLQJ
SRLQWVDUHVKRZQE\DUURZV
between two units at the splicing point. The idea is to
make the splice at the point at which the discontinuity
between two adjacent units is minimized. In order to
measure the discontinuity, the distance is computed
between frames in the two units, based typically on
physical parameters like spectrum and possibly amplitude
and pitch.
Rules in concatenative synthesis. A great deal of
research has been done over the years on duration and
pitch rules. For duration, most recent work has focussed
on predicting durations for individual phonemes, using
large speech corpora that have been segmented and
annotated with labels for various phonetic facts that are
potentially predictive of duration. Statistical techniques
such as multiple regression and neural networks have
been applied to the corpora to produce quantitative
models for use in synthesis. Similar approaches have been
adopted for predicting pitch.
However, since these rules are still not completely
understood, one approach is to entertain the possibility of
minimizing or even eliminating the need for such
modifications by designing a larger inventory that
contains multiple versions of each unit. Another approach
is to by replace some or all duration rules with inventory
templates for duration and pitch, in analogous fashion to
templates for the sound inventory. For example, pitch
time functions for particular melody patterns would be
stored in an inventory. For synthesis, rules would select
appropriate pitch templates and align them with the
concatenated sound units. A similar approach can be used
for duration rules using time warps that have been derived
from methods like DTW (Dynamic Time Warping).
Different voices. One method for providing a new
voice is to produce an entirely new sound inventory based
on the new speaker. However, because of the amount of
work involved, another approach is to apply signal
processing techniques to the templates or the output of a
synthesizer to transform the acoustic parameters for the
original speaker into those that are appropriate for a
different speaker.
4. Monolog versus Dialog
Most research on text-to-speech to date has involved
situations in which the synthesizer speaks a monolog –
reading books, newspaper articles, email. However, many
applications of synthesis in the future are likely to involve
dialog between a human speaker and a computer. For
example, a system might give directions in response to
questions from a traveler, as shown in Table 5. The table
compares accents for monolog and dialog, showing that
in both cases, it is inappropriate to accent the second
occurrence of “Walnut Drive.” Thus, a computer system
should consider the output of the speech recognizer as
part of the context for the text to be synthesized.
7DEOH'HDFFHQWLQJLQPRQRORJDQGGLDORJ
Text: Monolog
After you get to Walnut Drive, turn right on Walnut Drive.
* turn right on Walnut Drive
Dialog
Q: I’m coming up on Walnut Drive. What do I do?
A: Turn right on Walnut Drive.
*Turn right on Walnut Drive.
5. Conclusion
Today, text-to-speech for unrestricted text is still far
from entirely natural. Although there are a few
subproblems such as name pronunciation where text-to-
speech may perform better than humans, in general, text-
to-speech systems still only inadequately approach human
speech and language competence. Improvements to
speech quality have come most recently from the
incorporation of rules based on analysis of large amounts
of data or from storing large tables, dictionaries, and
sound inventories. Future improvements will depend on
improvements from natural language understanding and
more natural-sounding speech quality.
6. References
[1] J. Allen, S. Hunnicut, and D. H. Klatt, From Text to Speech:
the MITalk System, Cambridge University Press, Cambridge,
UK, 1987.
[2] C. Bailly and C. Benoit (eds.), Talking Machines, North-
Holland, Amsterdam, 1992.
[3] Y. Sagisaka, N. Campbell, and N. Higuchi (eds.), Computing
Prosody, Springer-Verlag, New York, 1997.
[4] D. B. Roe and J. G. Wilpon (eds.), Voice Communication
between Humans and Machines, National Academy Press,
Washington, D.C., 1994.
[5] J. P. H. van Santen, R. W. Sproat, J. P. Olive, and J.
Hirschberg (eds.), Progress in Speech Synthesis, Springer-
Verlag, New York, 1997.
[6] V. J. van Heuven and L. C. W. Pols (eds.), Analysis and
Synthesis of Speech, Mouton de Gruyter, Berlin, 1993.