Professional Documents
Culture Documents
μου κάνει
μου κάνει
μου κάνει
net/publication/43532482
CITATIONS READS
56 2,457
2 authors:
Some of the authors of this publication are also working on these related projects:
Effectiveness of early storybook reading for babies with and without a hearing loss. View project
All content following this page was uploaded by Marleen F Westerveld on 10 July 2017.
Marleen F. Westerveld
Gail T. Gillon
College of Education
University of Canterbury
Contact Details
College of Education
University of Canterbury
Email: marleen.westerveld@canterbury.ac.nz
Citation
Westerveld, M. F., & Gillon, G. T. (2010). Profiling oral narrative ability in young school-aged
10.3109/17549500903194125
1
Abstract
This study aimed to determine if oral narrative comprehension and production measures derived
in a fictional story retelling task could be used to create a profile of strengths and weaknesses in
oral narrative ability (PONA) in young school-aged children. The story retelling task was field-
tested with 169 typically developing children, aged between 5;0 and 7;6 years. Children listened
twice to an unfamiliar story while looking at the pictures in a book. Comprehension questions
were asked after the first exposure. Following the second exposure, children were asked to retell
the story without the use of the pictures. Story retellings were analysed on measures of
semantics, morphosyntax, verbal productivity, and narrative quality. Results indicated sensitivity
for age on measures of comprehension, narrative quality, semantics, and verbal productivity, but
not for morphosyntactic measures. Factor analysis indicated that oral narrative performance
comprised three factors, explaining more than 80% of the variance. Two clinical case examples
are presented, which show the potential of the PONA to reveal different patterns of strengths and
weaknesses across the oral narrative measures. Although early evidence suggests the potential
usefulness of the PONA, further research is now needed to test the validity, reliability and
Key words: oral narrative skills, language sampling analysis, school-age children
2
Profiling Oral Narrative Ability in Young School-aged Children
Introduction
In the last decade, researchers and clinicians have become increasingly interested in finding ways
in which children’s oral narrative abilities can be described in order to assist clinicians in making
clinical judgments about their students’ oral narrative performance (e.g., Bliss, McCabe, &
Miranda, 1998; Justice et al., 2006; Petersen, Gillam, & Gillam, 2008; Strong, 1998). This
interest stems from research indicating that the ability to understand and produce oral narratives
(e.g., Feagans & Appelbaum, 1986). Clinical assessment tools can be divided into standardised
tests and criterion-referenced procedures. Although standardised tests of oral narrative ability
will determine if a child functions significantly below expectations compared to his or her peers
(e.g, Bishop, 2004; Gillam & Pearson, 2004), in general these tests do not yield the detail in oral
criterion-referenced procedures provide the clinician with detail about a child’s oral narrative
performance (e.g., Applebee, 1978; Justice et al., 2006; Petersen et al., 2008). Ideally, both
comprehension and production of oral narratives should be addressed (Boudreau, 2008) and
Merritt, & Purcell, 1995). The current study, therefore, examined if a story retelling task that was
conversation and personal narratives, see Westerveld, Gillon, & Miller, 2004, for a copy of the
protocol) would yield descriptive data in all these aspects of oral narrative performance to
provide the clinician with a profile of a child’s strengths and weaknesses in oral narrative ability.
3
The importance of oral narrative skills for academic achievement
Research supports the predictive and concurrent relationship between aspects of oral narrative
ability and academic performance (e.g., Feagans & Appelbaum, 1986). Moreover, studies of oral
narrative performance in children with language and/or reading disabilities have revealed
weaknesses in these children’s oral narrative ability when compared to chronologically age-
matched and/or reading-age matched peers (Feagans & Short, 1984; Snyder & Downey, 1991;
Westerveld, Gillon, & Moran, 2008). It is therefore not surprising, that general consensus exists
that oral narrative analysis should play a pivotal role in the speech-language pathologist’s
assessment and intervention practices (Boudreau, 2008; Johnston, 2008). Apart from
recommends the use of spontaneous language sampling, not only to support standardised test
results but also to provide the clinician with the linguistic detail needed for intervention and
monitoring purposes (Gillon, Moriarty, & Schwarz, 2006; Miller, 1996). Although a plethora of
guidelines have been published on how to elicit and analyse oral narrative samples (see Hughes,
McGillivray, & Schmidek, 1997), local field-test data are urgently needed to allow the clinician
Oral narrative ability can be assessed in two modalities (comprehension and production) and
analysed at two levels (macrostructure and microstructure levels). From a clinical perspective, a
child needs adequate skills in all of these aspects to meet the ever increasing higher-level
discourse demands of the school curriculum. Children are exposed to narratives on a daily basis;
They are required to listen to and understand the teacher’s narrative explanations, are expected to
4
share personal stories and re/tell fictional stories, and are exposed to written narratives that
increase in complexity as children progress through the reading levels (Milosky, 1987).
Theoretical perspectives on the links between oral narrative ability and reading comprehension
confirm the need for evaluation of oral narrative ability across domains and modalities (Graesser,
Singer, & Trabasso, 1994; Kintsch, 1988). It is thus proposed that analysis of oral narrative
ability may be used to obtain important information about a child’s spoken language abilities that
are deemed important to academic participation and reading comprehension (see also Cain,
Oakhill, & Bryant, 2004). More specifically, analysis of oral narrative comprehension ability will
help determine the child’s ability to understand text-level language (e.g., Gough & Tunmer,
1986). Analysis of oral narrative production ability at microstructure level will potentially
provide an indication of the child’s strengths and weaknesses in the spoken language domains of
semantics, syntax, and morphology, whereas macrostructure analysis of oral narrative production
ability will provide insight into the child’s ability to apply narrative structure knowledge and
produce a coherent narrative that contains all the important story grammar elements (e.g., see
Procedures used for assessing oral narrative comprehension include asking the child to act out
action units contained in a narrative (Feagans & Short, 1984), asking the child comprehension
questions following exposure to a story (and prior to retelling that story) (Norbury & Bishop,
2002; Westerveld et al., 2004), or asking comprehension questions immediately after a child has
retold a story (Liles, 1987; Snyder & Downey, 1991; Wagner, Sahlen, & Nettelbladt, 1999).
Although some researchers support the use of story retell to assess a child’s narrative
understanding (e.g., Skarakis-Doyle & Dempsey, 2008), others argue that comprehension and
5
production of oral narratives may be distinct skills (Wagner et al., 1999). Even though both
methods rely to some extent on the child’s productive language abilities, it can be argued that
answering questions may yield more specific information about the child’s receptive abilities
than a story retelling task. A retelling task is more complex, as it not only involves the memory
processes underlying successful comprehension but also entails a semantic and morphosyntactic
Merritt and Liles (1989) suggested using comprehension questions following a story
retelling to identify if factors other than expressive spoken language skills influenced the child’s
oral narrative production performance. The advantage of asking comprehension questions prior
to the story retelling task is, however, that a more immediate measure of oral narrative
comprehension is obtained (i.e., which has not been affected by a child’s difficulty to store
information in short-term memory and/or which has not been influenced by the child’s attempt to
retell the story). Furthermore, previous research has shown high correlations between story
comprehension and the ability to tell the story afterwards both in children with language
impairment and in children with typically developing skills (Norbury and Bishop, 2002).
Both story generation and story retelling tasks are used in the research literature to elicit oral
narrative production skills. Examples of generation tasks include telling a story about a single
picture (Justice et al., 2006; Ripich & Griffith, 1988), completing a story started by the examiner
(Merritt & Liles, 1989), and creating a story about something that is not real (Roth & Spekman,
1986). Story retelling tasks can be described as either spontaneous (e.g., retelling a familiar story
or a TV programme) or more directed. Directed tasks include story productions after movie
viewings (Gummersall & Strong, 1999; Liles, 1985) and story retelling, with or without pictures,
6
after listening to a story (with or without pictures) (Ripich & Griffith, 1988; Schneider, 1996).
Merritt and Liles (1989) found clear advantages to using a story retelling task rather than a story
generation task for assessing oral narrative production abilities. Results indicated that retold
narratives were generally longer and contained more story grammar components and complete
episode structures than story generations. In addition, generated stories were often confusing to
When asking a child to retell a story, both the quality and the length of the resulting oral
narrative will depend on the elicitation tasks and the sampling conditions (Schneider, 1996).
Factors that may influence task performance include the child’s perception of the examiner’s
familiarity with the model story (Masterson & Kamhi, 1991), the number of exposures to the
model story (Gummersall & Strong, 1999), the contextual support for the story (e.g., single
pictures, picture books, see Masterson & Kamhi, 1991), and the linguistic complexity and length
of the model story (e.g., Holloway, 1986). After careful evaluation of the literature, Westerveld
and Gillon (1999/2000) concluded that optimal sampling conditions for eliciting fictional oral
narratives in school-age children (with and without spoken language disorders) involved:
• A listener who is unfamiliar with the story the child is retelling (Liles, 1985; Masterson &
Kamhi, 1991). For example, Masterson and Kamhi demonstrated that children retell
• Two exposures to the model story (Gummersall & Strong, 1999). The results from
Gummersall and Strong’s study indicated that a more informative oral narrative language
sample (containing more words and more complex syntactic structures) may be obtained
if a child listens to a model story twice, before being asked to retell the story.
7
• A retell of the story without the support of picture-sequences (Masterson & Kamhi,
1991). Masterson and Kamhi found that children’s story retelling samples that were
collected without the support of picture books contained more compound sentences,
higher lexical accuracy, and better fluency than those obtained in a picture supported
• Stimulus materials that contain levels of linguistic complexity that are similar to, or more
advanced than the child’s own (Griffith, Ripich, & Dastoli, 1986; Holloway, 1986).
Consideration needs to be given to the type of measures that are included when investigating a
child’s oral narrative performance. Some recent research studies investigating clinical
applications of scoring systems have focused on relatively complex measures; one example is the
Index of Narrative Microstructure (INMIS) by Justice et al. (2006) that is an aggregated score of
Complexity (Petersen et al., 2008) that assigns scores to a range of categories related to oral
narrative performance, including macro- and microstructural aspects. For the current profile of
narrative ability it was decided to use transparent, frequently used microstructure measures of
oral narrative performance that are known to correlate significantly with advancing age and to be
sensitive to disordered language performance (e.g., Botting, 2002; Boudreau & Hedberg, 1999;
Fey, Catts, Proctor-Williams, Tomblin, & Zhang, 2004; Watkins, Kelly, Harbers, & Hollis,
1995). These measures include mean length of utterance, number of different words and
grammatical accuracy. However, consistent with previous studies, to capture the quality of the
oral narrative, a total score will be assigned that covers a range of characteristics such as story
8
grammar elements, theme, and coherence (e.g., Fey et al., 2004; Miller, Heilmann, & Nockerts,
2006).
The current study aimed to determine if a Profile of Oral Narrative Ability (PONA), based on
measures derived from a story retelling task, has clinical utility with young school-aged children.
The story retelling task was field-tested with 169 typically developing children, aged between
5;0 and 7;6. Children were asked comprehension questions following the first exposure to the
story, and story retelling samples were analysed at macrostructure and microstructure levels.
Method
Participants
The 169 children in this study were participants in the New Zealand database project (see
Westerveld et al., 2004). All story retelling transcripts from children who were aged between 5;0
and 7;6 were included in the current study. The children were randomly selected from
mainstream primary schools in major urban areas in New Zealand. Children with diagnosed
disorders such as autism, Down syndrome, and cerebral palsy were excluded prior to the random
selection. The schools reflected a range of socio-economic areas. Only children who spoke
English as their first language and who had no history of sensory deficits or neurological
disorder were included in the study. There were slightly more girls than boys (ranging from 53%
to 56 % per age-group). The ethnicity of the groups resembled the largest cultural groups in New
Zealand (Ministry of Education, 1998) and comprised the following subcategories: New Zealand
European (61.5%), Maori (20.7%), Pacific Island (4%), Asian (3%), and Other (9.8%). Ethnicity
was based on the cultural groups the parents had identified on their child’s school-enrolment
form. Details on the age and gender distribution of the participants are shown in Table I.
9
Insert Table I about here
Procedures
Elicitation procedures. All testing was conducted by certified speech pathologists working for
Group Special Education (New Zealand Ministry of Education). These clinicians were trained by
the first author on the elicitation procedures. Each child was seen individually in the child’s
school setting and was administered the full Westerveld & Gillon Language Sampling Protocol
(LSP). The original protocol included elicitation of spoken language in three conditions:
conversation, personal narratives, and story retelling. The protocol has been included in
http://www.education.canterbury.ac.nz/documents/gillon/
languageprotocol.pdf. In addition, a New Zealand speech and language screening test (Gillon &
Schwarz, 1998) was administered. Any child who performed very poorly on the receptive
language screening task (i.e., who could not follow basic instructions) was excluded from the
database study. Children’s language samples were excluded from the database for reasons such
as poor taping quality or not engaging in the task (i.e., unwilling to retell the story).
The data presented in the current study relate to the story retelling condition of the
language sampling protocol. In this context, the child was required to listen twice to an audio-
recording of an unfamiliar story (while looking at the pictures in a story book). The story was an
English translation of Ko au na Galo [Ana gets lost] (Swan, 1992). The story is about a Pacific
Islands girl who gets lost in the city, while looking for her mum and dad. It is a 10-page reader of
the type typically used in Year 1 and Year 2 New Zealand classrooms, with coloured pictures
and Tokelauan text. The original translation of Ko au na Galo was adapted slightly to add a little
further length and complexity to the story. A transcript of the story is presented in Appendix A.
10
Following the first exposure to the story, the child was asked eight questions about the
story to evaluate story comprehension. To ensure all children had access to the same information
prior to the children listening to the story for a second time, children were provided with the
correct information after answering the questions if they did not respond to the question, or if
their answers were clearly incorrect. This method was used to make sure that if children
generated weak stories, they were likely to be the result of the children’s difficulty in applying
story structure knowledge when retelling the story, rather than the result of the children’s failure
to remember and/or pay attention to important elements of the story. Following the second
exposure to the story, the child was asked to retell the story without the use of pictures.
Transcription procedures. All language samples were tape-recorded. Samples were transcribed
by speech pathology students who were trained in the standard Systematic Analysis of Language
Transcripts – New Zealand version (SALT-NZ) transcription conventions (Miller, Gillon, &
Westerveld, 2008). Utterance segmentation was based on communication units (CU), using
Loban's (1976) rules. Defined by Loban, a CU consists of ‘‘each independent clause with its
modifiers and associated independent clauses, or of an answer to a question lacking only the
repetition of the question elements to satisfy the criterion of independent predication’’ (p. 9).
Only complete and intelligible (C&I) utterances were used for analysis; Interrupted and
Measures
Oral narrative comprehension (ONC). Each child was asked eight questions following the first
exposure to the story. ONC was calculated as a percent questions correct score. The questions
11
Microstructure. Only quantitative measures of oral narrative ability that are relatively easy to
compute by clinicians, that have been shown to distinguish between children with spoken
language impairment and children with typical development, and that are likely to change with
age or ability were included: number of different words (NDW, see Miller, 1996; Watkins et al.,
1995), total number of utterances (UTT), mean length of utterance in morphemes (MLU-M, see
Brown, 1973), and grammatical accuracy (percent grammatically correct utterances: GA, see Fey
et al., 2004).
Macrostructure. The full story retelling transcripts were used and analysed at macrostructure
level. To evaluate the child’s ability to apply story structure knowledge when retelling a story,
the story retellings were scored on a story quality rubric. The rubric was adapted from Jones and
Lodholz (1999) and assessed inclusion of six text structure elements: introduction, main
holistic coherence. It also investigated whether the child included the theme of the story. Theme
is defined as ‘‘the overall coherent topic of the text and its essential points’’ (Westby, 2005, p.
162). For more details or to obtain a copy of the rubric used in the current study, please contact
the first author. The child was awarded points for each characteristic: 5 points if the child
proficiently included the characteristic, 3 points if the skill was emerging, and 1 point if the child
provided minimal or no information. The rubric includes specific scoring examples to promote
easy and reliable scoring by other examiners. The scores were totalled to yield an Oral Narrative
Quality (ONQ) score. As a result, the minimum score was 8 and the maximum score was 40.
Reliability
12
Accuracy of transcription and coding of the original stories was reported in detail in Westerveld
et al. (2004). For the present study, all samples were re-checked by the first author and any
For the macrostructure analysis, 20% of the transcripts were randomly selected and
scored by an independent examiner who was trained in scoring the stories on the ONQ rubric.
Reliability of the total scores on the ONQ rubric, using Cronbach’s alpha analysis, was 0.96. The
total scores (of all individual stories) awarded by the two examiners differed by 4 points or less.
For individual characteristic scores, the scores never differed more than 2 points (i.e., one
category).
Results
Descriptive Statistics
Table II reports the means and standard deviations by age group on the six measures of oral
narrative ability: oral narrative comprehension (ONC), oral narrative quality (ONQ), mean
utterances (UTT) and grammatical accuracy (GA). As expected, there was a general linear
increase in performance with age for all the measures, except for MLU-M which was the same
To determine if the oral narrative measures were sensitive to age, separate linear
regression analyses were conducted with Age as the independent variable. The following linear
regressions were significant (p ≤ .001): ONC (r = .25), ONQ (r = .35), NDW (r = .39), UTT (r =
.36). The regression was significant at p ≤ .05 for GA (r = .16) and not significant for MLU-M (r
= .14). It was noted, however, that there were 2 outliers (scoring below 3 standard deviations)
13
when the linear regression was conducted for GA. When excluding those outliers, the effect of
Correlational Analyses
To determine internal consistency of the measures, correlation coefficients between the oral
narrative measures were calculated and are reported in Table III. This shows that the GA
measure was not correlated to any of the other oral narrative measures. In contrast, all other
measures (ONC, ONQ, MLU-M, NDW, and UTT) showed significant correlations, except for
Factor Analysis
A principal component factor analysis, using Varimax rotation with Kaiser normalisation was
conducted, entering all six oral narrative measures. At first, two factors were identified using
eigenvalue >1 as the cutoff value (explaining 49.2% and 16.7% of the variance respectively).
Because the third factor demonstrated an eigenvalue of .997 and explained an additional 16.6%
of the variance, it was decided to retain the third factor. The first factor reflected a content
component and consisted of the variables NDW, UTT, ONQ and to a lesser extent ONC. The
second factor reflected a grammatical ability component and comprised GA and to a lesser extent
ONC and MLU-M. The third factor reflected a grammatical complexity component and
consisted of MLU-M and to a lesser extent UTT (see Table IV). This factor structure did not
match the hypothesised macrostructure and microstructure components of oral narrative ability,
14
which would have resulted in loadings of NDW, UTT, GA, and MLU-M on one factor
(microstructure) and ONQ and perhaps ONC on the second factor (macrostructure).
Distribution Statistics
To determine the shape of the distribution of the oral narrative performance scores derived in the
story retelling task, several analyses were conducted. First, the 5th, 10th, 25th, 50th, 75th, 90th, and
95th percentiles were calculated, based on weighted averages (see Table V). As shown in Table
V, median scores for all oral narrative measures were similar to mean scores, except for the GA
measure in the 5- and 7-year-old age groups. Next, the distribution of the scores was
investigated, using skewness and kurtosis statistics. Normal distributions have skewness (degree
of asymmetry of the distribution around its mean) and kurtosis (the peakedness or flatness of a
distribution) statistics close to 0. Values of two standard errors of skewness or more (in this case
> 0.38) are considered skewed to a significant degree (Tabachnick & Fidell, 1996 ). As
illustrated in Table IV, about half of the oral narrative measures showed a skewed distribution,
with the GA measure showing a very high skewness statistic in every age group. This indicates
that scores for all age-groups on this measure clustered at the high end of the scale. The results
also showed significantly peaked (positive) and flat (negative) distributions, with around half of
the values greater than two standard errors of kurtosis (i.e., > 0.75).
Case examples
15
To provide further insights into how the PONA may potentially assist speech pathologists in
clinical practice, two case examples of children with delayed development are presented: James
James, aged 6 years 3 months, was referred to the University clinic because of concerns
about his expressive language skills. James has a twin brother, was born at 32 weeks, and
showed some delay in developmental milestones. James and his brother started school on their
fifth birthday. Although James had received speech-language therapy intervention in the past, he
was not receiving intervention at the time of the study. As part of the routine assessment battery,
the following tests were administered: Clinical Evaluation of Language Fundamentals, Preschool
– 2nd Edition (Wiig, Secord, & Semel, 2004): Core Language standard score (SS) 63, Receptive
language SS 72, Expressive language SS 66; Peabody Picture Vocabulary Test – 3rd Edition
(Dunn & Dunn, 1997): SS 86. Preschool and Primary Inventory of Phonological Awareness
(PIPA) (Dodd, Crosbie, McIntosh, Teitzel, & Ozanne, 2000): rhyme awareness SS 3 and
story retelling condition, using the Westerveld & Gillon LSP (see also Westerveld et al., 2004).
When comparing 50 complete and intelligible utterances derived in the personal narrative
context to the database of New Zealand language samples integrated into SALT (Miller et al.,
2008), the standard measures profile report indicated below average performance on measures of
semantics (NDW) and morphosyntax (high number of omitted words and word errors). Table V
shows James’ performance on the PONA, using the story retelling task of Ana gets lost, and the
grammatical ability. More specifically, he provided a sufficient number of utterances to retell the
16
story (UTT), showed satisfactory performance on the oral narrative quality (ONQ) measure, and
scored in the 25th percentile for NDW. However, he demonstrated significant difficulties on
measures of comprehension (ONC), and grammar (MLU-M and GA). Further inspection of the
ONQ score shows a weakness in the coherence characteristic and particular difficulty with the
Shelley, aged 6 years 8 months, was referred for speech pathology evaluation as a potential
candidate for a research project, because of teacher concerns about her language and literacy
skills. As part of the routine assessment battery, the following tests were administered: Test Of
Nonverbal Intelligence (TONI; Brown, Sherbenou, & Johnsen, 1997): SS 93; Clinical Evaluation
of Language Fundamentals – 3rd Edition (Semel, Wiig, & Secord, 1995): Total Language SS 65,
Receptive language SS 65, Expressive language SS 69; PPVT-III: SS 83. Neale Analysis of
Reading Ability – 3rd Edition (NARA; Neale, 1999): Reading accuracy percentile rank (PR)
score 16, reading comprehension PR 10. Spontaneous language was elicited in a personal
narrative and a story retelling condition, using the Westerveld & Gillon LSP. When comparing
50 complete and intelligible utterances derived in the personal narrative context to the SALT-NZ
database, the standard measures profile report indicated significant difficulties in areas of
semantics (NDW) and morpho-syntax (MLU-M and grammatical accuracy). Table VII shows
Shelley’s performance on the PONA and the transcript is included in Appendix C. As shown in
Table VII, Shelley’s performance was below the level expected for her age on all measures of
17
oral narrative performance, but consistent with standardised language testing. Further analysis of
the ONQ score shows immature to emerging performance across all characteristics.
Discussion
The current study aimed to determine if the profile of oral narrative ability (PONA) based on the
results of a story retelling task has clinical utility with young school-aged children. The story
retelling task comprised a comprehension and a retelling component. Children’s retelling skills
were analysed at macrostructure (oral narrative quality) and microstructure measures (semantic
and morphosyntactic ability as well as story length). The task was field-tested with a
representative sample of 169 typically developing children, aged between 5;0 and 7;6 years.
Descriptive analyses
First, it was examined if the oral narrative measures derived from the story retelling task were
sensitive to age. Regression analyses revealed sensitivity for age for measures of comprehension
(ONC), narrative quality (ONQ), semantic diversity (NDW), and number of utterances (UTT). In
and grammatical accuracy (GA). Closer inspection of the MLU-M measure (see Table II) reveals
an increase in MLU-M from 5 to 6 years, and no change between 6 and 7 years. This seems a
reflection of the MLU-M measure, rather than the task, as similar trends have been observed in
previous studies. For example, Justice et al. (2006) analysed performance on a story generation
task and noticed a linear increase in MLU from 5 to 7 years and then a decrease in performance
18
on the MLU-M measure from 7 to 8 years. In contrast, Leadholm and Miller (1992) observed
very similar performances when comparing 5- to 6-year-olds on the MLU-M measure (based on
100 utterances derived in a narrative context), followed by an increase in MLU from 6 to 7 years.
When inspecting the GA measure in the current study, a linear upward trend is seen from 5 to 7
years of age, but no significant differences were found between the different age groups. Other
researchers who have used this measure also found little variation with age for typically
developing children (Fey et al., 2004). Despite the finding that the MLU-M and GA measures
did not show a significant increase with age, results from previous studies demonstrating these
measures’ sensitivity to language ability support the importance of including these measures in
the PONA (Fey et al., 2004; Liles et al., 1995; Scott & Windsor, 2000). However, further
research investigating the performance of groups of children with language impairment on the
Descriptive statistics revealed that performance on the oral narrative measures was not
always normally distributed. Although the median scores were generally close to the mean
scores, ONC and GA in particular showed scores that were clustered onto the high end of the
scale. The obvious explanation concerns ceiling effects for ONC and GA, with 10 to 25 percent
of the children in all age groups responding correctly to all comprehension questions and
each age group was not unexpected, as previous research has shown a generally large variability
in performance on oral narrative tasks (e.g., Justice et al., 2006; Liles et al., 1995). The debate on
whether language sampling analysis can be standardised is likely to continue (e.g., Miller,
Heilmann, Nockerts, Andriacchi, & Iglesias, 2006), until, as Justice and her colleagues argued, a
19
theoretical explanation can be found for these “distributional irregularities” (Justice et al., 2006,
p. 187). Although this inherent variability in oral narrative performance affects its potential for
norm-referencing and indicates that the results should be used with caution, it should not rule out
its use in clinical practice. It is thus proposed that the PONA may be used by clinicians as part of
their language assessment battery to determine a child’s relative performance compared to his or
her peers.
Analysis of the correlational data showed consistently strong correlations between ONC,
NDW, ONQ, and UTT. MLU-M only showed significant correlations with ONQ and NDW,
whereas GA was not significantly correlated to any of the other measures. The factor analysis
yielded three dimensions, a content factor (containing NDW, UTT, ONQ and to a lesser extent
ONC), a grammatical ability component (comprising GA and to a lesser extent ONC and MLU-
M), and a grammatical complexity component (consisting of MLU-M and to a lesser extent
UTT). These findings do not seem to confirm the hypothesised distinction between
microstructure and macrostructure measures and appear to contradict results from previous
research in this area (Liles et al., 1995). Liles et al. conducted a factor analysis of oral narrative
measures derived in story retelling conditions, which showed two distinct categories, referred to
as “Content Organization” (macro-level) and “Linguistic Structure” (more local level) (p. 421).
The measures under investigation in the Liles et al. study included: 1) Content factor: total
number of episodes, ratio of possible episodes, ratio of complete cohesive ties, and 2) Linguistic
structure factor: mean number of words per subordinate clause, mean number of subordinate
clauses per utterance, and ratio of grammatical utterances. Interestingly, the mean number of
words per main clause measure did not load on the matrix at all. There are several possible
explanations for the differences in findings between the current study and the Liles et al. (1995)
20
study. First, different oral narrative measures were used in the studies, which accounted for only
41.3% of the variance in Liles et al. study, compared to over 82% in the current study. For
example, Liles et al. did not include a NDW measure. Based on results from the current study, it
seems reasonable to expect that NDW would have loaded on to the Content Organisation factor
in the Liles et al. study. In addition, Liles et al. did not include a comprehension measure, which
could have accounted for additional variance. Second, the current study uses a much larger
Another finding from the factor analysis relates to the separate loadings of the GA and
the MLU-M measure, indicating these measures tap different constructs. Results from Justice et
al. (2006) also showed that microstructure measures of oral narrative performance did not
represent a single construct. When analysing oral narrative production measures derived from a
story generation task (n = 250, age 5-12 years), factor analysis revealed two factors, accounting
for a combined 76.3% of the variance. One factor (labelled productivity) contained measures of
conjunctions, and number of complex sentences. The second factor (complexity) contained
measures of MLU and proportion of complex sentences. The authors admitted, however, that the
factor loadings did not match the hypothesised productivity and complexity loadings. In
summary, results from the current study confirm that oral narrative performance taps multiple
underlying factors (Justice et al., 2006; Liles et al., 1995) and highlight the importance of
analysing oral narratives on a range of measures. Future research should consider a wider range
of oral narrative measures to further investigate the dimensionality of oral narrative performance.
Case studies
21
The two case studies, James and Shelley, showed the potential application of the PONA for
assessment and intervention practices. The percentile scores for the oral narrative measures (see
Table V) can be used to create a profile of a child’s strength and weaknesses in oral narrative
performance across these measures. More interestingly, although James and Shelley showed
similar performance on a broad spectrum standardised language test, their performance on the
PONA differed. As shown in Tables VI and VII, James’ strengths were in measures tapping
semantic ability, including number of different words, oral narrative quality and the number of
utterances used to retell the story. In contrast, Shelley demonstrated difficulties in all aspects of
oral narrative ability, indicating both semantic and grammatical problems. Further inspection of
the children’s performance on the story quality rubric as well as analysis of the types of
grammatical errors produced during story retelling will add further detail to the results obtained
with the PONA. As a result, and consistent with evidence-based practice, these data can thus
inform the programming of intervention goals to specifically address areas of weakness in oral
narrative performance.
One limitation of the current study is the limited sensitivity of the ONC and GA measures.
However, this may be due to the task that was used: a 10-page picture book with a relatively
short story. For typically developing 5- to 7-year-old children this task may not have been
sufficiently challenging, resulting in ceiling effects on both of these measures. Despite the
skewed distribution for both these measures, however, the mean scores were close to the median
scores and the percentile scores showed a linear increase. Furthermore, the case examples
highlighted the difficulties on these two measures demonstrated by the two children with
language impairment. In addition, results from our research involving children with reading
22
disability showed performance on this oral narrative task to differentiate between poor readers
and their typically developing peers on the ONC and the ONQ measures (Westerveld et al.,
2008) and on the MLU-M and the GA measures (Westerveld, Gillon, & Miller, 2006). Further
research with larger groups of children is needed to determine if the PONA can reliably
differentiate children with language impairment from their typically developing peers.
The PONA should be used with caution with children from different cultural and/or
linguistic backgrounds. The results from a pilot study investigating the performance of 6-year-
old children from Maori, Pacific Islands, and New Zealand European backgrounds revealed no
differences in performance between the cultural groups on a range of oral narrative measures
(Westerveld & Gillon, 2001). It is not clear, however, if these findings can be extended to
younger children or to children for whom English is their second language. Future research
should consider cultural-linguistic styles that may influence performance on the PONA to help
distinguish disordered performance from cultural differences in oral narrative performance (see
It is not clear if the PONA can be used to demonstrate progress in oral narrative ability
following intervention. Because the story is short, a learning effect may be evident, and further
study of the story retelling task to determine test-retest validity is clearly needed. The PONA
should not be used as a screening measure until further research into this application is
conducted. Rather, until further research is undertaken, its purpose is to provide clinicians with
local field test data that can be used as an addition to their battery of language assessments.
Conclusion
The PONA provides the clinician with an overview of a young school-aged child’s strengths and
23
developing peers. However, clinicians should be careful not to use the story retelling sample as
the basis for a full microstructure analysis. It is well established that analysis of at least 50
semantic, syntactic, and morphological skills (Miller, 1996). Therefore, the PONA shows
potential to be used as one aspect of the clinician’s assessment battery to describe children’s
spoken language performance. This profile of oral narrative ability will then contribute to the
clinical evaluation of a child’s spoken language ability and will potentially provide detailed
Acknowledgments
The authors would like to thank the children and the children’s parents and class teachers for
their cooperation in the database project. Thanks are also extended to the speech pathologists
who assisted in the data collection process. The support of the New Zealand Ministry of
Education, the New Zealand Speech-Language Therapists’ Association and the Don Bevan
Travel Scholarship was greatly appreciated. Finally, this project would not have been possible
without the assistance from Professor Jon Miller and Ann Nockerts from SALT Software, LLC.
References
Applebee, A. (1978). The child's concept of story. Chicago: University of Chicago Press.
Bliss, L. S., McCabe, A., & Miranda, A. E. (1998). Narrative assessment profile: Discourse
Botting, N. (2002). Narrative as a tool for the assessment of linguistic and pragmatic
24
Boudreau, D. (2008). Narrative abilities: Advances in research and implications for clinical
Boudreau, D. M., & Hedberg, N. L. (1999). A comparison of early literacy skills in children with
specific language impairment and their typically developing peers. American Journal of
Brown, L., Sherbenou, R., & Johnsen, S. (1997). Test of Nonverbal Intelligence: A Language-
Brown, R. (1973). A first language: The early stages. Cambridge, MA: Harvard University
Press.
Cain, K., Oakhill, J., & Bryant, P. (2004). Children's reading comprehension ability: Concurrent
Dodd, B., Crosbie, S., McIntosh, B., Teitzel, T., & Ozanne, A. (2000). Preschool and Primary
Dunn, L. M., & Dunn, L. M. (1997). Peabody Picture Vocabulary Test - III. Circle Pines, MN:
Feagans, L., & Appelbaum, M. I. (1986). Validation of language subtypes in learning disabled
Feagans, L., & Short, E. J. (1984). Developmental differences in the comprehension and
25
Fey, M. E., Catts, H. W., Proctor-Williams, K., Tomblin, J. B., & Zhang, X. (2004). Oral and
written story composition skills of children with language impairment. Journal of Speech,
Gillam, R. B., & Pearson, N. A. (2004). Test of Narrative Language. Austin, TX: Pro-ed.
Gillon, G., Moriarty, B., & Schwarz, I. (2006). An international literature review of best
practices in speech and language therapy; Assessment and intervention practices for
children with speech and language impairment. Update January 2006. Christchurch,
Gillon, G., & Schwarz, I. (1998). Effective provision and resourcing of speech and language
services for Special Education 2000; Resourcing speech and language needs in Special
Education. Database and best practice validation. New Zealand: Ministry of Education.
Gough, P. B., & Tunmer, W. E. (1986). Decoding, reading, and reading disability. Remedial and
Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text
Griffith, P. L., Ripich, D. N., & Dastoli, S. L. (1986). Story structure, cohesion, and propositions
narrative context. Language, Speech, and Hearing Services in Schools, 30, 152-164.
Gutierrez-Clellen, V. F., Peña, E., & Quinn, R. (1995). Accommodating cultural differences in
26
Holloway, K. F. C. (1986). The effects of basal readers on oral language structures: A
Hughes, D., McGillivray, L., & Schmidek, M. (1997). Guide to narrative language: Procedures
Johnston, J. R. (2008). Narratives: Twenty-five years later. Topics in Language Disorders, 28(2),
93-98.
Jones, M. A., & Lodholz, C. (1999). Story Grammar Rubric. Madison Metropolitan School
District and the Language Analysis Lab, Waisman Center, University of Wisconsin,
Madison: Unpublished document created by the SALT working group, Jon Miller Chair.
Justice, L. M., Bowles, R. P., Kaderavek, J. N., Ukrainetz, T. A., Eisenberg, S. L., & Gillam, R.
B. (2006). The index of narrative microstructure: A clinical tool for analyzing school-age
15(2), 177-191.
with and without language disorder. Journal of Speech and Hearing Research, 30, 185-
196.
Liles, B. Z., Duffy, R. J., Merritt, D. D., & Purcell, S. L. (1995). Measurement of narrative
discourse ability in children with language disorders. Journal of Speech and Hearing
27
Loban, W. (1976). Language development: Kindergarten through grade twelve. Urbana, IL:
Masterson, J. J., & Kamhi, A. G. (1991). The effects of sampling conditions on sentence
Merritt, D. D., & Liles, B. Z. (1989). Narrative analysis: Clinical applications of story generation
and story retelling. Journal of Speech and Hearing Disorders, 54, 429-438.
Miller, J., Gillon, G., & Westerveld, M. (2008). SALT NZ: Systematic Analysis of Language
Miller, J. F. (1996). Progress in assessing, describing, and defining child language disorder. In K.
N. Cole, P. S. Dale & D. J. Thal (Eds.), Assessment of communication and language (6th
Miller, J. F., Heilmann, J., & Nockerts, A. (2006). Oral language and reading in bilingual
Miller, J. F., Heilmann, J., Nockerts, A., Andriacchi, & Iglesias. (2006). Can language sample
Florida.
Milosky, L. M. (1987). Narratives in the classroom. Seminars in Speech and Language, 8(4), 329
- 343.
Neale, M. D. (1999). Neale Analysis of Reading Ability (3rd ed.). Melbourne, VIC: Australian
28
Norbury, C. F., & Bishop, D. V. (2002). Inferential processing and story recall in children with
Petersen, D. R., Gillam, S. L., & Gillam, R. R. (2008). Emerging procedures in narrative
115-130.
Ripich, D. N., & Griffith, P. L. (1988). Narrative abilities of children with learning disabilities
Roth, F. P., & Spekman, N. J. (1986). Narrative discourse: Spontaneously generated stories of
Schneider, P. (1996). Effects of pictures versus orally presented stories on story retellings by
86-96.
Scott, C. M., & Windsor, J. (2000). General language performance measures in spoken and
written narrative and expository discourse of school-age children with language learning
Semel, E. M., Wiig, E. H., & Secord, W. (1995). CELF: Clinical Evaluation of Language
Fundamentals (3rd ed.). San Antonio, TX: Psychological Corporation, Harcourt Brace
Jovanovich.
29
Skarakis-Doyle, E., & Dempsey, L. (2008). Assessing story comprehension in preschool
Snyder, L. S., & Downey, D. M. (1991). The language-reading relationship in normal and
Strong, C. J. (1998). The Strong Narrative Assessment Profile. Eau Claire, WI: Thinking
Publications.
Swan, E. (1992). Ko au na galo (Ana gets lost). Wellington, New Zealand: Learning Media,
Ministry of Education.
Tabachnick, B. G., & Fidell, L. S. (1996 ). Using multivariate statistics (3rd ed.). NY: Harper
Collins.
Wagner, C. R., Sahlen, B., & Nettelbladt, U. (1999). What's the story? Narration and
Watkins, R. V., Kelly, D. J., Harbers, H. M., & Hollis, W. (1995). Measuring children's lexical
diversity: Differentiating typical and impaired language learners. Journal of Speech and
& A. G. Kamhi (Eds.), Language and reading disabilities (2nd ed., pp. 157-232). Boston,
Westerveld, M., & Gillon, G. (1999/2000). Narrative language sampling in young school-age
30
Westerveld, M., & Gillon, G. (2001). Oral language sampling in 6-year-old New Zealand
Westerveld, M. F., Gillon, G. T., & Miller, J. F. (2004). Spoken language samples of New
Westerveld, M. F., Gillon, G. T., & Miller, J. F. (2006). Morpho-syntactic language measures
Westerveld, M. F., Gillon, G. T., & Moran, C. (2008). A longitudinal investigation of oral
Wiig, E. H., Secord, W., & Semel, E.M. (2004). Clinical Evaluation of Language Fundamentals
31
Table I. Age and gender distribution of the participants
32
Table II. Group performance on oral narrative measures
Production
Macrostructure ONQ 21.5 5.9 24.5 6.6 28.2 5.9
Note: ONC = Oral narrative comprehension, maximum score is 8; ONQ = oral narrative quality,
range of scores is 8 – 40; MLU-M = Mean length of utterance in morphemes; NDW = number of
33
Table III. Correlations between the oral narrative measures
length of utterance in morphemes; NDW = Number of different words; UTT = number of total
34
Table IV. Rotated factor loadings
Note: ONC = Oral narrative comprehension; NDW = Number of different words; UTT = number
of total utterances; ONQ = Oral narrative quality; GA = Grammatical accuracy; MLU-M = Mean
35
Table V. Distributional performance and statistics by age on the oral narrative measures
Age Group M Mdn SD 5% 10% 25% 50% 75% 90% 95% Skewness Kurtosis
5-year-olds
(n = 39)
NDW 40 39 13.3 18 22 32 38 50 62 63 0.03 - 0.81
UTT 10.6 10 4 5 6 8 10 14 15 16 0.43 - 0.61
ONQ 21.5 22 5.8 14 16 18 22 25 30.4 32 0.43 - 0.57
ONC 6 6 1.5 3 4 5 6 7 8 8 - 0.66 0.012
GA 86 90 14.9 56 61 85 90 100 100 100 - 1.3 1.01
MLU-M 6.9 7.1 0.2 5.2 5.4 6 7.1 7.9 8.5 8.5 0.007 - 0.9
6-year-olds
(n = 81)
NDW 48.5 48 16 25 29 36 48 62 69 77 0.25 -0.95
UTT 13 13 5 6 7 9 12 17 20 22 0.29 - 0.9
ONQ 24.5 24 6.6 12 16 20 24 28 34 36 0.036 - 0.45
ONC 6.4 7 1.4 4 4 6 7 7 8 8 -0.82 0.11
GA 87.9 88.9 11 66.7 77.8 81.8 88.9 95.2 100 100 -1.59 4.8
MLU-M 7.3 7.3 0.9 5.8 6.1 6.7 7.3 7.9 8.4 8.5 - .184 -0.53
7-year-olds
(n = 49)
NDW 59 63 14.4 35 37 50 63 69 73 76 -0.71 0.29
UTT 16 17 4.6 8 9 13 17 19 21 21.6 -0.28 -0.66
ONQ 28.2 28 5.9 18 20 24 28 32 35 38 -0.17 -0.32
ONC 7 7 1.2 5 5 6 7 8 8 8 -0.97 0.025
GA 91.8 94.4 9.8 73.3 77.9 88.2 94.4 100 100 100 -1.36 1.73
MLU-M 7.3 7.2 0.9 5.9 6.1 6.8 7.2 7.8 8.4 8.5 0.41 0.9
Note: NDW = number of different words; UTT = total number of utterances; ONQ = oral narrative quality, range of scores is 8 – 40;
ONC = Oral narrative comprehension, maximum score is 8; GA = grammatical accuracy in percent grammatically correct utterances;
MLU-M = Mean length of utterance in morphemes.
36
Table VI. James’ performance on the PONA
narrative quality, range of scores is 8 – 40; ONC = Oral narrative comprehension, maximum
37
Table VII. Shelley’s performance on the PONA
narrative quality, range of scores is 8 – 40; ONC = Oral narrative comprehension, maximum
38
Appendix A
One Saturday morning, Ana's mum and dad went fishing on the beach.
Ana had been sick all week, so she had to stay at home with her big brother Tom.
So when Tom fell asleep, she decided to go looking for her mum and dad.
They thanked the policeman for finding Ana and bringing her home safely.
39
Appendix B
Comprehension questions
40
Appendix C
C = child; E = examiner. Mazes (reformulations, filled pauses) are put in brackets. [NGA] = not
grammatically accurate. * indicates unfinished word. > indicates unfinished sentence. < >
ONQ score: 24
Resolution: 5; Conclusion: 5.
41
Story retelling by Shelley, age 6;8 years
ONQ score: 18
Resolution: 3; Conclusion: 3.
42