Professional Documents
Culture Documents
2003 Shtyrov Grammar Processing Outside The Focus of Attention
2003 Shtyrov Grammar Processing Outside The Focus of Attention
discussions, stats, and author profiles for this publication at: http://www.researchgate.net/publication/8931460
CITATIONS
DOWNLOADS
VIEWS
65
99
4 AUTHORS, INCLUDING:
Yury Shtyrov
Friedemann Pulvermller
Aarhus University
SEE PROFILE
SEE PROFILE
Risto J Ilmoniemi
Aalto University
304 PUBLICATIONS 15,734 CITATIONS
SEE PROFILE
Abstract
& To address the cerebral processing of grammar, we used
whole-head high-density magnetoencephalography to record
the brains magnetic fields elicited by grammatically correct
and incorrect auditory stimuli in the absence of directed
attention to the stimulation. The stimuli were minimal short
phrases of the Finnish language differing only in one single
phoneme (word-final inflectional affix), which rendered them
as either grammatical or ungrammatical. Acoustic and lexical
differences were controlled for by using an orthogonal design
in which the phonemes effect on grammaticality was inverted.
We found that occasional syntactically incorrect stimuli elicited
INTRODUCTION
For over a century, the processing of language in the
brain has remained one of the most intriguing issues in
science. Although its many aspects have been tackled by
numerous studies, little is still known of how our central
nervous system goes about processing grammar. Grammar is one of the main intrinsic properties of the human
language. The very presence of a grammatical system
distinguishes our ability of verbal communication from
all signaling systems used by animals. Earlier investigations of the neurophysiology of syntactic processing
(e.g., Hagoort, Brown, & Groothusen, 1993; Osterhout
& Holcomb, 1992; Neville, Nicol, Barss, Forster, & Garrett, 1991) have led to important insights into this issue.
Three major syntax-related phenomena have been
described on the basis of the brains electric responses
to language stimuli. The first one is the so-called early
left anterior negativity (ELAN) occurring, with a latency
of 125 msec, in response to function words1 whose
placement in a phrase violates sentence structure rules
(Neville et al., 1991). Secondly, grammatically related
frontal negativities with longer latencies (over 250 msec)
have been found, for example, for different morphological or syntactic errors (Gunter, Friederici, & Schriefers,
1
2000; Mu
nte, Schiltz, & Kutas, 1998). Finally, a late
positive shift (often termed P600), reaching its maximum at 600 msec and maximal at centro-parietal
recording sites, has been considered the most robust
neurophysiological response to ungrammatical sentences because it could be observed in many linguistic
experiments (for review, see Osterhout, McLaughlin, &
Bersick, 1997; Osterhout & Hagoort, 1999).
When studying the processing of grammar in the brain,
it seems important to control for a number of issues. For
instance, in such studies, subjects are typically asked to
attend to presented sentences (e.g., Friederici, Wang,
Herrmann, Maess, & Oertel, 2000; Osterhout & Swinney,
1993; Neville et al., 1991); often, the task is to judge the
sentence grammaticality. In such cases, as attention is
required, one cannot be sure to what extent the registered responses are influenced by brain correlates of
attention rather than by the language-related activity as
such. Attention-related phenomena are known to modulate a variety of the brains evoked responses in a
substantial span of time after stimulus onset and to
involve a number of brain structures including those
close to, or overlapping with, the core language areas
(see, e.g., Yamasaki, LaBar, & McCarthy, 2002; Yantis
et al., 2002; Escera, Alho, Winkler, & Naatanen, 1998;
Tiitinen, May, & Naatanen, 1997; Woods, Alho, & Algazi,
1993; Woods, Knight, & Scabini, 1993; Alho, 1992; Picton
& Hillyard, 1974). It also appears possible that subjects
pay more attention to incorrect sentences as they try to
Journal of Cognitive Neuroscience 15:8, pp. 11951206
make some sense of them or that they process grammatical and ungrammatical items using different strategies.
These different strategies and attention variation may
find their reflection in the event-related measures, overlapping with true syntax-related activity. Therefore, limiting the attention-related effects seems to be necessary
at this stage for further investigation of the neural
processing of syntax.
A methodologically important aspect in such experiments is stimulus selection. Grammatically correct and
incorrect phrases would inevitably differ in many other
features: e.g., sound onset times and durations in
acoustic stimulation, as well as visual geometry and
overall luminance in visual tasks, would change when
words or affixes are added, removed, or relocated to
modulate grammaticality. Differences even in basic
physical features may lead to differential brain activation
(Korth & Nguyen, 1997; Naatanen & Picton, 1987) that
could in principle overlap with or be misinterpreted as
language-related effects. For example, it was shown that
word-elicited responses are highly dependent on the
word length (Osterhout, Bersick, & McKonnon, 1997;
Assadollahi & Pulvermu
ller, 2003), suggesting the necessity to control for such factors. Furthermore, the
physical stimulus parameters are of crucial importance
for obtaining ERP effects of syntactic processes (Gunter,
Friederici, & Hahne, 1999). It seems most logical to us
to apply an experimental design in which physical
differences between the grammatical and ungrammatical items are controlled for by presenting the same
physical contrasts to the subjects both in ungrammatical
and grammatical contexts; this sort of an orthogonal
design would help to disentangle purely syntactic effects
from those related to the physical stimulus features.
So, in sum, it appears crucial to limit the influence of
attention and different processing strategies on the
brain response to grammatical and ungrammatical
strings as well as to apply maximal control over stimulus
features using a counterbalanced stimulus design. Controlling for these issues simultaneously in a single study
can be a challenging task, which, as to our knowledge,
has not been realized so far. The present study represents an attempt to achieve this goal.
Here, we addressed brain activity related to grammatical processing in the absence of directed attention
towards stimuli and without an active task which would
possibly invite subjects to invent a strategy for dealing
with unusual stimuli. We used a counterbalanced stimulus design, which allowed us to control for effects of
the physical and lexical parameters of the stimuli. To
gain high temporal resolution necessary in such an
experiment, we opted for multichannel magnetoencephalographic (MEG) recording of brain responses,
which can also provide information about the spatial
location of the registered activity.
We utilized the so-called mismatch negativity (MMN),
a unique indicator of automatic cerebral processing of
1196
RESULTS
Magnetic MMN responses were elicited in all conditions.
The overall analysis of single equivalent current dipole
Volume 15, Number 8
Figure 1. Spectrograms of acoustic Finnish-language stimuli used in the four experimental conditions. The standard and deviant stimuli in each
pair are indiscriminable up to the divergence point (marked with white arrowheads) at their end. The standarddeviant contrasts are identical
across all experimental conditions. Grammatically incorrect phrases are marked with an asterisk (*). See Table 1 for more details.
Deviant
Standard
Condition 1a
Condition 1b
Condition 2a
Condition 2b
Ma
tuon
*Ma
tuot
*Sa
tuon
Sa
tuot
(I bring)
(*I bring)
(*you bring)
(you bring)
*Ma
tuot
Ma
tuon
Sa
tuot
*Sa
tuon
(*I bring)
(I bring)
(you bring)
(*you bring)
English translations are given in parentheses, ungrammatical phrases are marked with (*). The standard and deviant stimuli in each pair are
indiscriminable up to the divergence point at their end. The standarddeviant contrasts are identical across all experimental conditions. Critical
contrasts are in bold.
Shtyrov et al.
1197
Figure 2. ECD models of MMNm activity in the left hemisphere (grand average). Both ungrammatical deviant stimuli produced stronger MMNm
source than the corresponding correct phrases regardless of the acoustic, phonetic, phonological, or lexical properties of the stimuli. Red and blue
contour lines indicate positive and negative values of the normal component of the magnetic field on the helmet-shaped MEG-array surface,
respectively. The arrows show the dipole location and are proportional in relative size to the response magnitude. The view is from the left side of
the head.
Figure 3. Waveforms
(magnetic field gradient) of
MMNm responses in the left
hemisphere (grand average).
Both ungrammatical deviant
stimuli produced stronger
MMNm than the corresponding
correct phrases regardless of
the acoustic, phonetic,
phonological, or lexical
properties of the stimuli.
Temporal gradiometer with the
most prominent responses
(channel 0242) was used for
producing this diagram.
1198
Figure 5. Grand-average L1
MCEs of MMNm in the left
hemisphere. The MCE solutions
are projected unto
triangularized gray matter
surface. Both ungrammatical
deviant stimuli produced
stronger MMNm source than
the correct phrases regardless
of the acoustic, phonetic,
phonological, or lexical
properties of the stimuli.
Shtyrov et al.
1199
Figure 6. Waveforms
(magnetic field gradient)
of auditory responses to
standard stimuli in the left
hemisphere (grand
average). Although the
MMNm responses were
affected by grammaticality,
there was no significant
effect on the standard
responses (cf. Figure 3).
Temporal gradiometer
with the most prominent
responses (channel 0242)
was used for producing
this diagram.
DISCUSSION
In the present study, mismatch responses were elicited
by grammatically correct or incorrect pronounverb
phrases. The choice of the single final phoneme in all
phrases rendered the whole structure as syntactically
correct or as ungrammatical. To exclude confounding
effects, the phonetic and lexical contrasts were counterbalanced over the experimental conditions using a fully
orthogonal stimulus design. The subjects were instructed
not to attend to the stimuli; they did not perform any
stimulus-related language task.
We found that MMNm responses to the syntactically
incorrect phrases were always larger than those to the
correct phrases ending in the same word regardless of the
direction of the acoustic contrast or of the preceding
pronoun. That is, neither the choice of the final phoneme
nor that of the pronoun in the sentence altered this effect,
which was present in both conditions in which the
deviant stimuli were grammatically inconsistent.
The present data are consistent with earlier results on
the cerebral processing of syntactic violations (see Introduction), which suggested negative-going components of brain responses at time delays of 125 msec or
more following syntactic errors. The present modification of the MMN may be related to this early negativity.
There are, however, important differences between the
present study and the earlier ones. Importantly, we were
able to find syntactically related effects in the absence of
directed attention towards the stimuli as subjects were
engaged in a different task and were instructed to ignore
the stimulation. The violations in this study modulated
the MMN response, which is considered to reflect a preattentive automatic level of auditory processing in the
cerebral cortex (Naatanen & Alho, 1995; Naatanen &
Escera, 2000; Tiitinen et al., 1997). This suggests that the
1200
METHODS
Subjects
Twelve healthy right-handed (handedness assessed
according to Oldfield, 1971; no left-handed family members) native Finnish speakers (age 2129, 7 women)
Shtyrov et al.
1201
with normal hearing and no record of neurological diseases were presented with four sets of spoken
native-language stimuli in four separate experimental
conditions.
Stimuli
The four short-phrase stimuli (see Figure 1 and Table 1)
were prepared with requirements that the acoustic,
phonetic, and phonological difference between the
standard and the deviant stimuli is identical in each
condition and that the stimuli themselves are as similar
acoustically as possible. These Finnish stimuli were:
(1a) Ma
tuon (I bring, syntactically correct),
(1b) *Ma
tuot (*I bring, syntactically incorrect: the
verb tuot is in the second person inflection form, while
the pronoun ma
, I, requires the first person); here
and throughout we use linguistic convention of marking
ungrammatical phrases with an asterisk (*),
tuon (*you bring, syntactically incorrect:
(2a) *Sa
the verb tuon is in the first person form, while the
pronoun sa
, you, requires the second person);
(2b) Sa
tuot (you bring, syntactically correct). 2
Therefore, it is the single final phoneme, inflectional affix [n] or [t], which may render the whole phrase
as syntactically correct or as ungrammatical. Remarkably,
for ma-stimuli (1a and 1b), the final [n] means that
the phrase is grammatical, while for the sa-stimuli
(2a and 2b) the combination is exactly the opposite:
[t] is needed in the end for the whole phrase to be
correct. This orthogonal design is implemented in
order to control for purely acoustic, phonetic, and phonological effects on mismatch responses as well as
for putative lexical and semantic differences between
the words.
As seen in Table 1 presenting the complete stimulus
design, the stimuli starting with the same pronoun were
always presented in the same experimental condition,
one of them serving as deviant and the other one as
standard stimulus. Thus, the standarddeviant acoustic
phonetic contrast, the critical variable determining the
MMN (Naatanen & Alho, 1997), was the same in all
conditions. However, in each pair of conditions (1a, b
and 2a, b) the responses were elicited either by correct
or incorrect deviant forms. Furthermore, while the
acoustic contrasts were identical in the corresponding
conditions (tuon/tuot in 1a2a and tuot/tuon in 1b2b),
the syntactic contrast (determined by the pronoun) was
the opposite: stimuli ending in tuon were correct in
conditions 1a and 1b, whereas tuot-ending phrases were
correct in Conditions 2a and 2b. This fully orthogonal
design, therefore, allowed us to control not only for
possible acoustic/phonetic effects of the verb contrast,
but even for an unlikely late differential effect of the two
preceding pronouns.
1202
For stimulus preparation, we recorded multiple repetitions of each word uttered by a female native speaker
of Finnish. With great care we selected a combination of
recordings, whose vowels matched in their fundamental
frequency (F0) and whose overall length and maximal
sound energy were identical both for sa
and ma
and for
tuon and tuot. We then combined the four words in the
four abovementioned phrases with a 50-msec gap between the pronoun and the verb; all phrases were
766 msec in duration. Thus, the two stimuli in each
condition were perceptually identical up to a time point
(referred to as divergence point) shortly before their
end (see Figure 1). In order to determine this divergence point, a gating experiment was performed, in
which initial fragments of increasing duration of both
tuon and tuot were presented to the subjects who were
asked whether they could perceive any difference between the two. In average, they could identify the
difference only when the first 336 msec (SEM 9 msec)
of the words were presented (which equals to 636 msec
after the onset of the complete phrase). This perceived
divergence point is therefore considered as the onset
of the perceptual standard/deviant contrast and a zero
time reference in the current study. Importantly, it is
the same in all pairs of stimuli due to their design.
All stimuli were normalized to have the same peak
sound energy. For the analysis and production of the
stimuli, we used the Cool Edit 96 program (Syntrillium
Software, AZ).
Acoustic Stimulation
All four experimental conditions (Table 1) were performed with every subject, their order being counterbalanced across the subject group. The stimuli were
binaurally presented at 50 dB above the individual
hearing threshold (determined using the experimental
stimuli) via headphones connected to an STIM setup
(Neuroscan Labs, VA). The interstimulus (stimulus
onset asynchrony, SOA) interval was 1450 msec. In each
condition, the deviant stimulus was presented with a
probability of 16.7% among the repetitive standard
stimuli: A pseudorandom sequence of stimuli was created so that there were always at least two standard
stimuli between any two deviants. During the stimulation, the subjects were seated in a magnetically shielded
chamber and instructed to watch a silent video film
of their own choice and to pay no attention to the
auditory stimulation.
Magnetoencephalographic Recording
The evoked magnetic field was recorded (passband
0.03200 Hz, sampling rate 600 Hz) with 204 planar
gradiometer channels of a whole-head Neuromag Vectorview MEG system (Elekta Neuromag, Helsinki) during
the auditory stimulation (Ahonen et al., 1993). The
Volume 15, Number 8
1203
Acknowledgments
We thank Olaf Hauk, Vadim Nikulin, Maritta Maltio-Laine, Simo
Monto, Tuomas Murdoch, Christopher Bailey, Jussi Nurminen,
Johanna Salonen, Anthea Hills, Seppo Kahko
nen, and William
Marslen-Wilson for their contribution at different stages of this
work. We would also like to thank three anonymous referees
for their helpful comments and constructive critique.
Reprint requests should be sent to Dr. Yury Shtyrov, Medical
Research Council, Cognition and Brain Sciences Unit, 15
Chaucer Road, CB2 2EF Cambridge, UK, or via e-mail:
yury.shtyrov@mrc-cbu.cam.ac.uk.
Notes
1. Function words are grammatical words, such as articles,
conjunctions, auxiliary verbs, and so on and so forth, which
have no concrete meaning by themselves and are utilized for
grammatical purposes.
2. These spoken-language stimuli represent the most usual
colloquial pronunciation used in the Helsinki area; therefore,
we only used Finnish subjects who had lived in the Helsinki
area for a minimum of 6 years before the experiment.
REFERENCES
Ahonen, A. I., Hamalainen, M., Kajola, M., Knuutila, J., Laine, P.,
Lounasmaa, O., Parkkonen, L., Simola, J., & Tesche, C.
(1993). 122-channel SQUID instrument for investigating the
magnetic signals from the human brain. Physica Scripta,
T49, 198205.
Alho, K. (1992). Selective attention in auditory processing
as reflected by event-related brain potentials.
Psychophysiology, 29, 247263.
Alho, K. (1995). Cerebral generators of mismatch negativity
(MMN) and its magnetic counterpart (MMNm) elicited by
sound changes. Ear and Hearing, 16, 3851.
Alho, K., Connolly, J. F., Cheour, M., Lehtokoski, A.,
Huotilainen, M., Virtanen, J., Aulanko, R., & Ilmoniemi,
R. J. (1998). Hemispheric lateralization in preattentive
processing of speech sounds. Neuroscience Letters, 258,
912.
Assadollahi, R., & Pulvermu
ller, F. (2003). Early influences of
word length and frequency: a group study using MEG.
Neuroreport, 14, 11831187.
Bentin, S., McCarthy, G., & Wood, C. C. (1985). Event-related
potentials, lexical decision and semantic priming.
Electroencephalography and Clinical Neurophysiology, 60,
343355.
Berg, P., Kakigi, R., Scherg, M., Dobel, C., & Zobel, E. (1999).
Source modelling of the EEG and MEG oddball response in a
subject with a large P300. Electroencephalography and
Clinical Neurophysiology Supplement, 49, 189193.
Binder, J. R., Rao, S. M., Hammeke, T. A., Frost, J. A.,
Bandettini, P. A., Jesmanowicz, A., & Hyde, J. S. (1995).
Lateralized human brain language systems demonstrated by
task subtraction functional magnetic resonance imaging.
Archives of Neurology, 52, 593601.
Donchin, E. (1981). Surprise! ... Surprise? Psychophysiology,
18, 493513.
Escera, C., Alho, K., Winkler, I., & Naata
nen, R. (1998). Neural
mechanisms of involuntary attention to acoustic novelty
and change. Journal of Cognitive Neuroscience, 10,
590604.
Ferreira, F., & Henderson, J. M. (1990). Use of verb information
1204
1205
1206