Musical Performance: A Synthesis-by-Rule Approach

Author(s): Johan Sundberg, Anders Askenfelt and Lars Frydén

Source: Computer Music Journal , Spring, 1983, Vol. 7, No. 1 (Spring, 1983), pp. 37-43
Published by: The MIT Press

Stable URL: https://www.jstor.org/stable/3679917

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms

The MIT Press is collaborating with JSTOR to digitize, preserve and extend access to Computer
Music Journal

This content downloaded from

193.140.185.164 on Fri, 22 Dec 2023 09:02:30 +00:00
All use subject to https://about.jstor.org/terms
Johan Sundberg Musical Performance:
Department of Speech Communication
and Music Acoustics
Royal Institute of Technology (KTH) A Synthesis-by-Rule
S-100 44, Stockholm, Sweden
Approach
Anders Askenfelt
Department of Speech Communication
and Music Acoustics
Royal Institute of Technology (KTH)
S-100 44, Stockholm, Sweden

Lars Fryden
Conservatory of the Swedish Radio
Edsberg, Sweden

Introduction If players violate one or more of these rules, they

run the risk of being classified as poor musicians.
The musical act of converting a string of note signs These rules seem to be scientifically interesting.
into a sequence of tones is by no means a simple An important task of scientific research is to elimi-
one-to-one translation. If this task is left to a com- nate mystery originating from lack of knowledge.
puter, the result is generally very poor from a musi-Some of the mystery surrounding music can be
cal point of view. eliminated if some of the discrepancies typically
The complexity of musical performance has been observed between the notated and the played ver-
recognized for a long time. Seashore (1938) mea- sion of a piece of music can be explained by means
sured the actual durations of the tones in several of a set of "pronunciation" rules. Moreover, the
musical performances of the same song. Among rules would mirror certain aspects of the auditory
other things, he found that in none of the singers and cognitive processing of the musical sounds in-
was there "a slightest approach to an even time for volved in listening. Thus, pronunciation rules could
a measure." Also, he found that musicians tend to reveal part of the intra- and extramusical basis of
show a modest degree of reproducibility as regards musical communication. From the point of view of
the performance of a given piece of music. Later in- practical application, the rules would be useful in
vestigations reporting on measurements of timing digital synthesis of music and in musical pedagogy.
in performed music have confirmed this observa-
tion (Bengtsson and Gabrielsson 1977).
These findings may seem almost trivial to any- Method
body acquainted with music listening. We all know
that a piece of music can be played in a number of To explore the rules that a musician normally must
different ways, all of which are musically excellent. obey in playing a piece of music, we can choose
And yet, the musician's freedom is by no means either measurements or synthesis-by-rule. The rea-
unlimited; there is a large class of performances son why we have chosen the synthesis-by-rule
that are musically unacceptable. This implies the ex- strategy is that we assume the rules are numerous
istence of rules that state how a string of notes nor- and that they interfere with one another. Also, the
mally should be converted into a sequence of sounds. pedagogical and artistic experience of one of the au-
thors (Fryd6n) had generated a number of hypothe-
Computer Music Journal, Vol. 7, No. 1, ses as regards such rules.
Spring 1983, 0148-9267/83/010037-07 $04.00/0, As in a previous study of musical performance
C 1983 Massachusetts Institute of Technology. focusing on singing (Sundberg 1978), a computer-

Sundberg, Askenfelt and Fryden 37

This content downloaded from

193.140.185.164 on Fri, 22 Dec 2023 09:02:30 +00:00
All use subject to https://about.jstor.org/terms
 Fig. 1. Schematic illustra-
tion of a typical relation-
ship between fundamental
frequency and overall am-
plitude in musical
instruments.

controlled vowel synthesizer was used (MUSSE; see

Larsson 1977). MUSSE can generate one part only.
The complexity of the synthesis was reduced to an
absolute minimum, with the only variables being a,
fundamental frequency and amplitude. An advan-
tage of this simplicity was that a minimum of syn-
thesis parameters required control by rule. The Cd
basic assumption was that any acoustic variableLo
that can be varied in musical performance is varied
by the performer for the purpose of musical expres-
sion. The timbre resembled that of a woodwind in-
Log fundamental frequency
strument. The pitch frequency changed in steps in
accordance with the equal-tempered scale, and
there was a very slight vibrato. The amplitude
could be changed in steps of 1/4 db. The duration should be noted that therefore no corrections are
was controlled with a time unit corresponding tomade with regard to the durations of adjacent notes.
8-12 msec, depending on the tempo, which, ac-Thus, the effect of rule 1 is that the entire "time-
cording to findings by van Noorden (1975) is accu- table of tone departures" is perturbed.
rate enough for such an experiment.
The computer programs used for controlling the
synthesizer were (1) a notation program (Askenfelt Rule 2: Duration and Note Value
and Elenius 1977) by means of which the melody
can be written in ordinary notation on the com- Rule 2 deals with notes of various values. If a piece
puter screen and (2) a text-to-speech program writ- of music is played with exactly the durations that
ten by Carlson and Granstrim (1975). The inputare theoretically specified in the notation, the re-
information is the melody as written in ordinarysult is rigid, as in Sound Example 2a. In Sound Ex-
music notation. This information is then translated ample 2b and 2c, we have introduced a rule that
into "vowel sounds" possessing duration, pitch, fre- increases the contrasts between durations slightly.
quency, and amplitude. The rules are triggered by In short, the rule works as something like a "per-
specific sequences of durations and/or pitches, that verse tax system," where the tax rate is high for the
is, by information contained in the input notation. poorest and zero for the wealthiest. If we exaggerate
this rule, the result is as in Sound Example 2b. The
mechanical character obtained with nominal note
Rules durations is replaced by lack of rhythmic balance.
A more moderate quantity of the rule gives the
Rule 1: Amplitude and Pitch version in Sound Example 2c.

In almost every musical instrument, the amplitude

increases somewhat with the fundamental fre- Rule 3: Duration and Pitch Increase
quency, as is illustrated schematically in Fig. 1. If
this is not modeled in a performance, the sound Rule 3 simply states that the duration of each note
gives a peculiar, lifeless impression. Sound Exam-be decreased by 3% as soon as the following note
ples la, lb, and 1 c demonstrate amplitude increases has a higher pitch. In sequences of rising pitches,
of 0, 8, and 4 db per octave pitch-frequency rise. this rule has the effect of increasing the tempo
(See Soundsheet Examples at the back of this issue.) somewhat. Sound Examples 3a, 3b, and 3c demon-
Rules 2-4 affect the duration of the individual strate the effect. In the first version (3a), the rule is
note, either by lengthening or by shortening it. It not applied; in the second (3b) it is exaggerated; and

38 Computer Music Journal

This content downloaded from

193.140.185.164 on Fri, 22 Dec 2023 09:02:30 +00:00
All use subject to https://about.jstor.org/terms
Fig. 2. Schematic illustra- completed during a tenth initial value, which is tude; DR = duration; A =
tion of the accents used in of this same duration. The reached at the end of the accent; and V = peak of
the rule system: the ampli- increased amplitude value note. The figure also illus- accent.

tude increase is inversely is maintained for 40 msec, trates the two categories of
proportional to the dura- after which the amplitude notes that receive the ac-
tion of the note and is gradually decreases to its cents. A = constant ampli-

J. J J
A+A

0 Time

1
A1 DR2

1
A DR3

in the third (3c) the 3% perturbation

two tones. This is present,
is the effect of rule 5,inwhich de-
accordance with our formulation of rule 3.
creases the amplitude of the tone to zero during the
last 30 or 40 msec of its duration, depending on the
width of the leap; the fourth being the boundary.
Rule 4: Duration and Leaps The effect of this rule is illustrated in Sound Exam-
ples 5a, 5b, and 5c, where the first version is played
Our next rule, rule 4, has the effect of increasing without this rule (5a), the second version is played
the duration of all tones that terminate a melodic with the rule exaggerated (5b), and the third version
leap. The quantity is 3% for tones terminating (Sc) is played with rule 5 as described above.
a leap of a minor or major third, 6% for leaps
wider than fifths, and 5% for intermediate inter-
vals. Sound Examples 4a, 4b, and 4c are played Rule 6: Accents and Note-Value Contrasts
without the rule (4a), with the rule exaggerated (4b),
and then with the rule quantity of rule 4 (4c) Rule 6 marks contrasts in note value by accents,
respectively. that is, small and very rapid increase-decrease ges-
tures in the amplitude. Rule 6 adds an amplitude
increment proportional to the duration of the tone.
Rule 5: Pauses and Leaps The details of the accent used are illustrated sche-
matically in Fig. 2. The rule adds such an accent to
In instrumental music, particularly that played on two types of notes, as is indicated in the figure. One
bowed instruments, wide melodic leaps are often is a short note surrounded by longer notes. The other
performed with a very short pause just between the case is a note terminating a specific pattern of changes

Sundberg, Askenfelt and Fryd n 39

This content downloaded from

193.140.185.164 on Fri, 22 Dec 2023 09:02:30 +00:00
All use subject to https://about.jstor.org/terms
 Fig. 3. Schematic illustra-
tion of rule 7, by which
amplitude contrasts be-
tween adjacent tones are
decreased. A = amplitude.

in durations: a decrease followed by an increase.

This rule has a clear effect, particularly on the short
notes after a dotted note. Sound Examples 6a, 6b,
and 6c illustrate this by presenting a melody first
without the rule (6a), then with the rule exagger-
ated (6b), and then with rule 6 (6c).
A2

< -.8(A, - A,)

Rule 7: Amplitude Continuity

If tones of differing amplitudes follow each other,

T A,
the melodic continuity may be disturbed. Rule 7
states that the last amplitude reading of a tone be
corrected by a quantity corresponding to 80% of Time
the decibel difference between it and the following
note (Fig. 3). The effect is demonstrated in Sound
Examples 7a and 7b, where a piece is first played rules. Therefore, it is the summed effect of all rules
without the rule (7a), and then with rule 7 as just that will improve the performance; hence, they
described (7b). should be tested together.
In Sound Examples 1-7, the effect of one rule at A listening tape was prepared, in which different
a time has been demonstrated. Sound Example 8 melodies were presented in pairs of performances.
presents two different performances of a piece of In each pair, one version was nominal, with no
music. In example 8a, none of the rules are used, changes in durations or amplitudes, while in the
and in example 8b, all seven rules are applied. other, all rules were used. The musical excerpts are
listed in the Appendix. Each of the excerpts was
chosen so as to demonstrate clearly the effect of at
Evaluation least one of our 7 rules. Thirteen different examples
were recorded on the tape. The tape was presented
An opinion commonly encountered is that random via headphones to nine musically experienced
variations, as opposed to variations effected accord- judges who were asked to tell which performance
ing to rules, occur in musical performance. An in- of each pair they preferred. The results are shown
formal experiment was made in which all of our in Table 1.
rule-controlled deviations from nominal tone dura- It can be seen from Table 1 that most examples
tions were multiplied by a random number between were liked better by more than half of the jury
0 and 1. Thus, there was no system in the devia- when played in accordance with the rule system.
tions from the nominal tone durations. As expected, Still, the scores are low in some cases. One cause of
the results demonstrated that randomized devia- some low scores seemed to be an artifact, though.
tions often work quite well, but in cases of "bad In tuning the pauses occurring between tones sepa-
luck" an embarrassing lack of rhythmic balance oc-rated by a leap, we listened on loudspeakers. Thus,
curs. This demonstrates the axiom that in an ideal the reverberation time of the listening room was
performance all that can be perceived serves a mu-certainly influencing our work. In the listening
sical purpose. test, on the other hand, headphones were used,
The rules presented here do not appear to improve eliminating room reverberation. The effect of this
the musicality of a performance to any appreciable was that the pauses sounded too long, and this hurt
extent if they are applied only one at a time. Accord-
the musical acceptability of some examples that
ing to our hypothesis, however, the effect of an in-contained many leaps, such as the Nursery tune
dividual rule is dependent on the effects of other and Geistliches Lied. The examples chosen in order

40 Computer Music Journal

This content downloaded from

193.140.185.164 on Fri, 22 Dec 2023 09:02:30 +00:00
All use subject to https://about.jstor.org/terms
Table 1. Result of listening experiment in which Discussion
nine musical experts compared the musical quality
of performances Our general experience during work with the rules
was that the magnitude of the effects (e.g., the mag-
Rule Example Preference nitude of the durations that are added to notes ter-
minating leaps) is very critical. If the magnitude is
1 Handel: Sonata 9 too great, the effect is almost embarrassing, as the
Mozart: Alla Turca 6
reader might have observed in listening to the exag-
2 Schubert: Sonata 4 gerated versions of the sound examples. We found
Tegndr: Nursery tune 5 that the optimum magnitude yielded an effect that
was barely noticeable: it was scarcely possible to
3 Jularbo: Dance music 5
Mozart: Alla turca 6 tell what physical reality the rule affected. It was
impossible to "diagnose" the rule. A typical exam-
4 Bach: Bourree 8 ple is the sharpening of the note-value contrasts. If
Bach: Courante 8
the magnitude is too great, it is easy to hear that
5 Bach: Geistliches Lied 5 the short notes are too short; and when the magni-
Bach: Gavotte 8 tude is correct, the diagnosis is nearly impossible
to make, yet the rule has achieved a clear effect,
6 Traditional: Happy Birthday 5
Bellman: Hvila vid denna kiilla 3 namely elimination of the lifeless, mechanical
character of the performance. It is probably reveal-
7 Schubert: Fruehlingstraum 7 ing that these effects are not marked explicitly in
Bach: Sarabande 8 the musical score.
Note: The rule number indicates which rule which w
larly highlighted by the example. The preference numb
Rules forpreferred
cate how many of the experts Different Musicalthe
Styles rule-contr
performance. The examples are specified in the Appen

There are certainly more rules for musical perfor-

mance than the seven presented in this paper. Pre-
to demonstrate clearly the
sumably, some of sharpening of
the as yet missing rules take intodu
contrasts and the marking of aspects
account the harmonic certain notes
of the music, as men-
cents (rules 2 and 6) resulted tioned. All present rulesinoperate
close towin-
with a time 50%
ence or even lower. Future research efforts should dow of two or three notes, and there must be rules
be directed at investigating the reason for these low
operating on longer strings of notes. Also, there
scores. would be different rule systems for different musi-
In examining the results of the listening cal styles. The rules discussed here have been
experi-
ment (Table 1), it may be observed tested that mainly
excerpts on examples selected from eighteenth-
from music of the baroque era are and ratednineteenth-century
higher than compositions. Their effects
excerpts from more recent music. The on other
possibility
repertoires remain to be tested. If the rules
cannot be excluded that the reason for this
reflect pureis conventions
that developed within music,
our rules fit baroque music better than then the rules' generality
more recent can be expected to be re-
music. We believe, however, that the reason
stricted. If, onisthe
that
other hand, the rules reflect ex-
our examples from the classical, post-baroque tramusical reference
peri- systems, such as listeners'
ods are more dependent on the harmonic associations with physical movement (e.g., a scale
progres-
sions underlying the melody; such examples
upward would be areassociated with climbing uphill
likely to suffer more from the factby many as
that, listeners),
yet, we then the rules would possess a
have introduced no rules dealing with the
greater harmonic
generality. This question is left open for fu-
context. ture investigation based on a more complete sys-
tem of performance rules.

Sundberg, Askenfelt and Frydin 41

This content downloaded from

193.140.185.164 on Fri, 22 Dec 2023 09:02:30 +00:00
All use subject to https://about.jstor.org/terms
 Another important point: we have not tried to tions for some of the rules. Lacking any kind of
model the multitude of choices that are available to data at this point, we are free to speculate. For in-
professional musicians and that allow them to play stance, the increase in tempo during sequences of
the same piece in many different ways, all equally rising intervals may have a psychological origin; a
acceptable from a musical point of view. The rules rising interval may possess an "activating" connota-
do, however, leave room for some musical liberties. tion that the player should stress by increasing the
One would be to allow for variations in the magni- tempo. Also, pitch rises are often combined with an
tude of the rules' effects. Another possibility would increased tempo in excited speech. Again, these
be to mark in the score the performer's interpreta- last-mentioned attempts to interpret the rules are
tion of the music. We could then formulate rules no more than speculation. The point is that our 7
that operated on these marks. (Musicians often rules probably have a foundation of some kind that
adopt a similar practice.) may be independent of music. We believe that fur-
ther research on the foundations of performance
rules will be interesting and rewarding.
Violating the Rules

We do not believe that our performance rules must Conclusions

always be obeyed in a good performance. On the
contrary, such a rule-controlled performance might From this research, we conclude the following.
be boring in the long run. We believe that musi- First, it is possible to improve the musical accept-
cians can and should violate one or more of the per-ability of a performance by applying a limited set of
formance rules in order to communicate something pronunciation rules. Second, these rules can be dis-
in particular. Evidently, the perceptual effects of covered by means of an analysis-by-synthesis ap-
such violations of a rule system depend on the rule proach. Third, such an approach enables us to
system being well known by both players and formulate new hypotheses as to how our present
audience.
set of rules should be extended, thus contributing
According to the results of the listening test to knowledge about and scientific understanding of
(Table 1), our rules seem to contribute to the musi-music.
cal acceptability of performance of a melody. An in-
teresting question is then what the origin of these
rules might be. We believe that some of the rules References
are derived from the human voice. The reduction of
amplitude steps at tone boundaries and the general Askenfelt, A., and K. Elenius. 1977. "Editor and Search
growth of amplitude with fundamental frequency Programs for Music." Speech Transmission Labora-
may be examples of this derivation. A possible in- tory-Quarterly Progress and Status Report 4/1977.
terpretation of these rules would be that they serve Stockholm: KTH.
the purpose of making the performance slightly Carlson, R., and B. Granstr6m. 1975. "A Phonetically
more similar to singing. Oriented Programming Language for Rule Description
Other rules are likely to have a psychoacoustic of Speech." In Speech Communication, Vol. II, ed. G.
origin. The principle of lengthening the note that Fant. Stockholm: Almqvist and Wiksell, pp. 245-253.
Bengtsson, I., and A. Gabrielsson. 1977. "Rhythm Re-
terminates a leap might reflect certain effects stud-
search in Uppsala." In Music Room Acoustics. Publica-
ied by van Noorden (1975); a melody may split into
tion 17. Stockholm: Royal Swedish Academy of Music,
two quasi-simultaneous melodies (fission) if wide pp. 19-56.
leaps occur in rapid tempo. This fission might be Larsson, B. 1977. "Music and Singing Synthesis Equip-
avoided if the tone terminating the leap were ment (MUSSE)." Speech Transmission Laboratory-
lengthened. Quarterly Progress and Status Report 1 / 1977:38-40.
There may also be purely psychological founda- Stockholm: KTH.

42 Computer Music Journal

This content downloaded from

193.140.185.164 on Fri, 22 Dec 2023 09:02:30 +00:00
All use subject to https://about.jstor.org/terms
Seashore, C. 1938. Psychology of Music. New York: Bourrie from Suite in C major for
McGraw-Hill. cello solo, BMV 1009
Sundberg, J. 1978. "Synthesis of Singing." Swedish Jour- Sarabande from Suite in C minor for
nal of Musicology 60(1): 107-112. cello solo, BMV 1011
van Noorden, L. P. A. S. 1975. "Temporal Coherence in Dir, dir, Jehova ... (Geistliches Lied),
the Perception of Tone Sequences." Druk vam BMV 452
Voorschooten (Diss.).
C. M. Bellman: Hvila vid denna kiilla, Fredman
Epistlar, no. 82
G. F. Handel: Sonata in E major for violin and
Appendix: Origin of the Melody Excerpts Used continuo, op. 1:15
in the Evaluation Test K. Jularbo: Livet i Finnskogarna (Dance music)
W. A. Mozart: Alla Turca from Sonate ffir
Parentheses indicate abbreviated titles used in Klavier in A major, K 331
Table 1. F. Schubert: Fruehlingstraum from Winterreise,
op. 89:11, D 911
J. S. Bach: Gavotte from Partita in E major for Sonata for Violin and Piano, op.
violin solo, BMV 1006 posth. 137:1, D 384
Courante from Suite in D major for A. Tegndr: Eckorn satt i granen (Nursery tune)
cello solo, BMV 1012 Traditional: Happy Birthday

Sundberg, Askenfelt and Fryden 43

This content downloaded from

193.140.185.164 on Fri, 22 Dec 2023 09:02:30 +00:00
All use subject to https://about.jstor.org/terms