Play Audio File

Digital Synthesisof Natural and Unnatural Sounds

MAX V. MATHEWS

Murray Hill, NJ 07974, USA

The sampling theorem guarantees theoretically that any sound which can be heard
by the human ear can be synthesized from digital samples. Today, computers have
actually produced a great variety of interesting and powerful timbres. These range
from close imitations of musical instruments to sounds never heard before. An analysis-
synthesis technique has been developed which involves analyzing natural sounds and
synthesizing approximations to these sounds using simplified models. This work has
led not only to rich synthesized sounds, but also to a fundamental understanding of
the important factors in natural sounds. Unnatural sounds include pitch and rhythm
paradoxes in which the apparent pitch or tempo of a sound can simultaneously increase
or decrease; timbres with nonharmonic overtones which preserve.some perceptions of
classic harmony while rejecting others; and sounds in which the timbre depends on
unusual ways in which the spectrum of the sound changes during a note. Examples of
various synthesized sounds are included.

The sampling theorem says that theoretically any
sound that can be heard by the human ear can be syn-
thesized from computer-generated samples of a sound
waveform. The question is, how far have we come
toward this theoretical goal? The original sounds made
in the late 1950s and early 1960s were far from the
ultimate possibilities. Since that time, great progress
has been made by many people,
One of the classic early achievements was the tech-
nique which Jean-Claude Risset developed for syn-
thesizing and understanding normal instrumental
sounds. The first sound sample consists of ten tones
from Risset. Five of the tones are recordings of atrumpet
and five of the tones are synthesized on the computer,
These are randomly intermixed. The correct order of
the sounds is TCTCCTCTTC (C--computer, T--
trumpet ). The normal person, in listening to these
sounds, can get about 50% correct; in other words, the
person is guessing. A good trumpet player can get about
70% correct,
What did Risset find out is necessary to make good
trumpet tones? Some of the things that were well known
DIGITAL
AUDIO
had to be synthesized correctly, including the right
average spectrum of the sound, the right attack time,
and the right decay time, but these things in themselves
are not sufficient. The real key that he found is that
when the tone gets louder, the proportion of high-fie-
quency energy must increase. At low amplitude the
first overtone or the first harmonic is the strongest, and
the rest of the harmonics fall off fairly rapidly; at loud
sounds the highest energy is in one of the higher har-
monics. This particular pattern of increase in the high-
frequency energy as the tone builds up is the key to
making a good trumpet sound.
How did Risset discover this? Risset used a two-
pronged attack on the problem. He did a frequency
analysis of sounds he recorded from a real trumpet.
He found out lots of important information from the
analysis, and he found out much unimportant infor-
mation. How did he separate the two? He used synthesis
to separate them. He synthesized the sounds on the
tape, putting in some of the informationfrom the anal-
ysis. When the synthesis sounded good to his ear, he
knew he had the important information about what
239
MATHEWS
constitutes the trumpet sound. This is an example of
what is called the analysis-synthesis technique for
studying timbres. It is indeed a powerful technique,
If you hear a passage with a sequence of sounds put
together as Risset knew how to make them, you would
instantly be able to tell the difference between the syn-
thesized trumpet and a real trumpet. Risset, at the time
he made these sounds, did not know what we would
today call the prosodic features of playing the trumpet,
In other words, he did not know how one note is blended
into the next note by a real trumpet player,
Dexter Morrill at Colgate University studied this
question extensively and has made some very nice con-
nected passages. Morrill is a trumpet player himself,
The second sample on the tape is one of his passages
synthesized on the computer,
There are many prosodic features in this example,
the most prominent of which are the pitch glides that
connect one note to the other,
The third sample is a flute sound, also by Risset.
Fig. 1 shows the diagram of the computer program
which produced this sound. The program is best de-
scribed by an interconnected set of unit generators.
One generator is an oscillator that makes the waveform,
It has two controls, an amplitude control and a frequency
control. It has a waveshape that can be specified by a
stored function in the computer memory. The stored
function that Risset used contained one to six harmonics,
one harmonic for the high pitched notes and more har-
monics for the lower pitched notes. Another oscillator
imposes an attack and decay function on the waveform
oscillator. It determines how the sound builds up and
decays away. This oscillator is used in a degenerate
way; it produces only one cycle for each note.
Another oscillator imposes a sinusoidal fluctuation
on the amplitude. The rate of that sinusoidal fluctuation
is 5 Hz, yet another oscillator imposes a slight random
quaver on the amplitude at a fairly high rate of 60 Hz,
but a low amplitude of about 1% of the average am-
plitudeof thenote.
o.o, 60 5 ,,Ofo
R_
HARMONICS
Fig. 1. Risset flute.
240
The complete details of this instrument are not im-
portant here. The point is that we have a language for
describing complex sounds so that once one person has
found out how to make a sound, this person can write
down and publish the instrument used plus the score.
Other people can then learn from this experience.
To recapitulate our understanding of instrumental
sounds, we have the analysis-synthesis technique, we
have instruments which involve unit generators to de-
scribe the structure of what we know, and we have
scores which say what quantitative parameters have to
be put into these instruments. This knowledge has
formed the basis of much subsequent research by many
people. In the last 20 years we have learned more about
instrumental sounds, primarily through these tech-
niques, than was known in the entire period before
then.
After considering instrumental sounds, let us turn
to some unnatural sounds. A class of the most interesting
unnatural sounds are called the paradoxical sounds.
The pitch paradox is one of these. The fourth sample
contains a pitch paradox first discovered by Roger
Shepard. It consists of 12 tones played over and over
again. The characteristics of these tones are such that
they sound like a continually ascending sequence of
pitches. How is this done? It is actually done in a very
simple way. Fig. 2 shows the spectrum of a few of the
tones in the series. Not all the harmonics are present.
The overtones are all one octave apart; none of the
other overtones are in the sound. The computer can
easily synthesize this. A normal instrument cannot.
The paradox takes advantage of the fact that tones
that are an octave apart are hard to tell apart by listeners.
People tend to make octave mistakes. The amplitude
of the tones is arranged so that the low-frequency tones
and the high-frequency tones are very faint, and, indeed,
the lowest frequency tone is so faint that you cannot
hear it. When the pitch is moved up an octave, the
highest frequency tone is so faint that you cannot hear
it.
As the pitch goes up, the amplitude of the low-fre-
quency tones increases and the amplitude of the high-
frequency tones decreases. The bell-shaped curve that
with pitch. When the pitch has moved up exactly one
octave, then the highest frequency tone can be elimi-
nated and a low-frequency tone can be added; you cannot
specifiesthe envelope of the spectrumdoes not change
AMPLITUDE
I10 220 440 880 1760 3,520
LOG FREQUENCY
Fig. 2. Discrete pitch paradox after R. N. Shepard.
DIGITAL
AUDIO
 APPLICATIONS

hear these changes. Thus we have a cyclic phenomenon
which sounds like a sequence of tones that are contin-
ually going up in pitch,
Risset generalized Shepard's technique into a con-
tinuous glissando. The fifth sample is a glissando where
the pitch is going down. It is a stereo example, The
sound from the two loudspeakers is slightly out of phase
with respect to the downward sweep in pitch, and so
you will hear a beating at about 2 Hz between the two
loudspeakers. This adds both to the beauty of the tone
and to the illusion,
Risset was able to identify two different components
of the pitch. One is associated with the fundamental
frequency of the tone; the other is associated with the
peak of the spectrum where the energy is highest. He
makes sounds, for example, where the pitch goes up
and at the same time the point of the highest energy
goes down. In this way he makes a sound which both
descends and ascends in pitch at the same time. He
describes it as a sound which goes down the scale, but
goes up in brightness. It is the sixth sample,
Another paradox can be created with tempo or rhythm,
Ken Knowlton invented a sound which apparently gets
faster and faster, but in reality is periodic so that every
20 seconds it repeats. How is this sound made? The
sound starts out as a regular series of pulses, about five
pulses per second. As time goes on, these pulses get
faster, but in addition every other pulse gets smaller.
Eventually, when the time between two successive
pulses equals half the original time, every other pulse
has been reduced to zero in amplitude so that the sound
is as it was at the beginning. The seventh sample dem-
onstrates thiseffect,
This illusion is not quite as convincing as the pitch
illusion, but it nevertheless does work.
A last paradox is demonstrated in sample eight. It
is the up/down fast/slow paradox, again by Risset, where
he puts all these effects together and produces a sound
which goes down the scale and up in brightness, slows
better understanding of the nature of violin resonances.
The work was carried out by analyzing sounds from a
real violin sound and then by synthesizing sounds and
tuning the electrical resonances to achieve a good sound
quality.
Fig. 3 shows the amplitude-frequency response of
three different sets of resonances which have been tried
with the electronic violin. The three curves differ pri-
marily in the Q of the resonators, the curves being low
Q (zero), medium Q, and high Q. The medium Q re-
sonators produced the best tone, the low Q resonators
produced harsh tones, and the high Q resonators pro-
duced a nasal or pinched effect. The ninth sample dem-
onstrates a scale played on the medium Q resonators
which give the best sound.
It is difficult to judge a violin by a scale, so the tenth
sample on the disk is a two-violin duet, one violin
being a normal acoustic violin and the other an electronic
violin. Most people have great difficulty in determining
which is which. The violin which begins the duet is
the electronic violin. A good violinist can tell the elec-
tronic violin from the acoustic one, but it is clear that
the two instruments are close enough together in timbre
so that they can be effectively played together. The
electronic violin can play the repertoire of the normal
violin.
The signal from the violin string and the pickup need
4
o
-4
d8
200
i 500
i IOI
O0 2000
I 5000
i
,_
co_
4

down in tempo, but gets faster. With the right adjust- o x AA^ A AA,I_AA_A_!
ments of the stereo loudspeakers, the sound will also
goaroundtheroom.
It is unnecessary to say that these sounds are unique
-4
-o
-r2
dB-,6
vv vv,w vvv,v,vv,
tO the computer; no other instrument could produce -20
them. The spatial effects that are used in the eighth -24
-28

sound sample depended very heavily on techniques that -_2

I [ I I I
were developed by John Chowning who used rever- 2oo 50o HZ_ooo 20oo 5000
beration, Doppler frequency, and loudness to control cb_
the apparent location of sounds.
Next we will deviate from the purely digital part of
this talk to mention some violin sounds. These sounds o _ ,,, , l_lllllJJl_lJ
have used the same techniques for their development -4
that we have used for the digital sounds, and they also -6
dB-12

show the possibility of making natural and unnatural -.6

soundswitha violin. -20
The violin is an electronic violin where the body of -24
-28

the violin has been removed and, instead, the vibrations -32 z_o _o '
_OOO zc_ '
50O0

of the strings are picked up by contact microphones Hz

and processed through a set of parallel resonances. The {=_
sound comes out of a loudspeaker. This work led to Fig. 3. Violin formant. (a) Harsh. (b) Best. (c) Pinched.
DIGITAL AUDIO 241
MATHEWS
not.be processed by a series of violinlike resonances.
Instead, we can substitute something very different,
We can go back to what Risset learned about brass
sounds and arrange a voltage-controlled low-pass filter
plus an amplitude detector, so that when the violin
string is played louder, the resulting sound will have
more high-frequency energy. We would expect that
this might have a brasslike sound. The 1lthsoundsam-
ple shows that it does.
Another instrument which is important to synthesize
with the computer, and also very difficult for the eom-
puter to synthesize, is the human voice. Much effort
has gone into synthesizing a good speaking voice. One
can also impose melodic pitch patterns on the speech
and produce a song with the same synthesis equipment.
Joseph Olive at Bell Labs wrote a short opera. The
characters in the opera are a woman scientist, who
made a speaking computer, and the computer itself,
The computer falls in love with its creator, and they
sing the duet which is reproduced in sample 12. Alas,
shortly after this moment of passion the scientist became
alarmed at their incestuous relationship and took the
computer apart. Alas, operas have to have sad endings,
Charles Dodge took techniques developed by Joe
THE AUTHOR
Max V. Mathews was born in Columbus, Nebraska,
on 1926 November 13. He received the following de-
grees in the field of electrical engineering: B.S. in
1950 from the California Institute of Technology; M.S.
in 1952, and Sc.D. in 1954 from the Massachusetts
Institute of Technology.
He joined the staff of Bell Telephone Laboratories
in 1955 and is currently director of the Acoustical and
Behavioral Research Center. This laboratory carries
out research in speech communication, visual eom-
munication, human memory and learning, programmed
instruction, analysis of subjective opinions, and phys-
ical acoustics.
Mr. Mathews' personal research is concerned with
sound and music synthesis with digital computers and
with the application of computers to areas in which
man-machine interactions are critical. He developed a
program (Music V) for the direct digital synthesis of
sounds and, more recently, a program (Groove) for
the computer control of a sound synthesizer. Music V
is now widely used in music departments in the United
States, Europe, and Japan. His past research included
242
Olive and applied them in his unique way to distort
speech for musically expressive purposes. The kinds
of distortion that he found interesting are illustrated in
sample 13. Dodge demonstrates strong and interesting
pitch and time distortions in the speech that can be
made.
Many listeners find the quality of these speech sounds
expressive and interesting, but not very beautiful. If
one is not trying to say words and communicate infor-
mation, one can make very beautiful speechlike sounds.
The 14th sample was prepared by Risset. I will not
discuss how it was made, but simply say that I find it
gorgeous. These samples show that the computer can
make both natural and unnatural speech sounds.
I have demonstrated computer-synthesized examples
of sounds, some of which are very normal and some
very abnormal. In sample 15, I end with an excerpt
from a piece that Risset composed, entitled Songe.
When one initially hears the piece, the timbres seem
so beautiful that one feels they must have been played
on a very old instrument. Upon closer listening, one
realizes that the sounds could only be produced by a
computer and no other instrument could possibly have
made them.
development of computer methods for speech pro-
cessing, studies of human speech production, studies
of auditory masking, and the invention of techniques
for computer drawing of typography. He is currently
investigating the effect of resonances on sound quality.
Resonances are essential parts of human speech and
of violins. Both better speech transmission and better
violins may be achieved by a better understanding in
this area.
Mr. Mathews is a member of the Audio Engineering
Society and Sigma Xi. He is a fellow of the Acoustical
Society of America and the Institute of Electrical and
ElectronicsEngineers.In 1973he receivedthe IEEE
David Sarnoff Gold Medal Award.
From 1974 to 1980 he was the scientific advisor to
the Institut de Recherche et Coordination Acoustique/
Musique (IRCAM) in Paris, France. In 1975 April he
was elected a member of the National Academy of
' Sciences, andin 1979 March he was elected a member
of the National Academy of Engineering. In 1982 May
he was elected fellow of the American Academy of
Arts & Sciences.
DIGITAL
AUDIO