Researching student comprehension of

different kinds of image-language relations

in State, National and International
standardized reading test.
Dreamwalker
 The Consensus - image/text relations and literacy
In order to understand the role of print in the digital age, it is
essential to have a solid grasp of the growing integrative
relationship of print and graphics" (p.22)
Dresang, E. (1999). Radical Change: Books for youth in a digital age. New York: Wilson.!

The changing nature of society underlines the imperative for

increased metatextual awareness and visual literacy, including an
understanding of pictureword combinations.

Russell, G. (2000). Print-based and visual discourses in schools: implications for pedagogy. Discourse:
studies in the cultural politics of education, 21(2), 205-217.!

... no text is an image. No text has the exact set of meaning-

affordances as any image. No image or visual representation means
in all and only the same ways that some text can mean. It is this
essential incommensurablility that enables genuine new meanings to
be made from the combinations of modalities.
Lemke, J. (2002). Travels in hypermodality. Visual Communication, 1(3), 299-325.
...many contemporary texts make use of image and of writing at
the same time, using both to carry meaning in specific ways. In
that context, a theory of reading which relates to the graphic
material of 'letters' alone is no longer able to explain how we
derive meaning from texts.
Kress, G. (2003). Literacy in the New Media Age. London: Routledge.!

Serial cognitive processing of linear print text no longer

adequately characterizes contemporary reading and writing, which
now involve parallel processing of multimodal text-image
information sources.

Luke, C. (2003). Pedagogy, connectivity, multimodality and interdisciplinarity. Reading Research
Quarterly, 38(10), 356-385.!

it is the visual/verbal interface that is at the heart of literacy

learning and development for both computer-users and those
without access to computers (p.63)
Andrews, R. (2004). Where next in research on ICT and literacies. Literacy Learning: The Middle Years,
!
12(1), 58-67.
National English Curriculum: Initial Advice
paper (http://www.ncb.org.au/home_page.html)

The subject of English has historically been largely about the reading and
writing of printed texts. More recently there has been debate about the
growing significance of visual and non-print communication such as
speaking and listening, combinations of visual information with language,
and the new digital developments. In considering the tasks that young
learners face in school, in their further education and training, and in
workplaces, the argument has been that subject English should expand its
scope to include more focus on these non-print forms.

Clearly these forms of communication are expanding in and out of formal

education, and so they have an important place in a national English
curriculum. (National Curriculum Board, 2008, p.8 )
 Australian National Curriculum: English
http://www.australiancurriculum.edu.au/English/
English / Year 3 / Literature / Creating literature
Create imaginative texts based on characters, settings and events from
students own and other cultures using visual features, for example
perspective, distance and angle.
English / Year 4 / Language / Expressing and developing ideas
Explore the effect of choices when framing an image, placement of elements
in the image, and salience on composition of still and moving images in a
range of types of texts.

English / Year 5 / Literacy / Creating texts

Plan, draft and publish imaginative, informative and persuasive print and
multimodal texts, choosing text structures, language features, images and
sound appropriate to purpose and audience.

English / Year 6 / Literacy / Creating texts

Plan, draft and publish imaginative, informative and persuasive texts,
choosing and experimenting with text structures, language features, images
and digital resources appropriate to purpose and audience.
http://www.australiancurriculum.edu.au/English/

English / Year 7 / Literature / Responding to literature

Compare the ways that language and images are used to create character,
and to influence emotions and opinions in different types of texts.

English / Year 8 / Literacy / Interpreting, analysing, evaluating

Explore and explain the ways authors combine different modes and
media in creating texts, and the impact of these choices on the viewer/
listener.

English / Year 9 / Literacy / Interpreting, analysing, evaluating

Explore and explain the combinations of language and visual choices that
authors make to present information, opinions and perspectives in different
texts
How well prepared are we What does research
to implement a national tell us about school
curriculum which demands students
attention to the significance comprehension of
of image/language relations image/language
in English literature and relations?
literacy?

How does
NAPLAN address
image/language
relations?

How does PISA deal

with image-language
relations?

How do reading
Why is theorizing tests deal with
image/language image-language
interaction crucial? relations online?
 Results of an ARC study1 indicating the
relative difficulty of different types of image/
language relations in the NSW Basic Skills
Test (Reading Comprehension).
Chan, E., & Unsworth, L. (2011). Image-language interaction in online reading
environments: Challenges for students reading comprehension. Australian
Educational Researcher, 38(2), 181-202.
Daly, A., & Unsworth, L. (2011). Analysis and comprehension of multimodal texts.
Australian Journal of Language and Literacy, 34(1), 61-80.
Unsworth, L. and Chan, E. (2009) Bridging multimodal literacies and national
assessment programs in literacy. Australian Journal of Language and Literacy, Vol.
32 No 3, pp 245-257.
Unsworth, L., & Chan, E. (2008). Assessing integrative reading of images and text in
group reading comprehension tests. Curriculum Perspectives, 28(3), 71-76.
1An Australian Research Council (ARC) Linkage Grant in conjunction with the Educational Measurement and Schools
Assessment Directorate of the New South Wales (NSW) Department of Education and Training (DET), 2006-2008,
entitled New dimensions of group literacy tests for schools: Multimodal reading comprehension in conventional and
computer-based format.
New South Wales Basic Skills Test
 Investigating image-language
relations in State-wide reading tests

ANALYSIS OF
VISUAL-VERBAL RELATIONS IN
TEST STIMULUS MATERIALS
 Intermodal relations: a representational view

homospatiality
partial
ELABORATION equivalence
concurrence complete
exposition
exemplification
EXPANSION
augmentation
EXTENSION intra-process
distribution
inter-process
divergence
complementarity
time
ENHANCEMENT place
locution
PROJECTION result
idea manner
 Intermodal relations: targeted questions

homospatiality
partial
ELABORATION equivalence
concurrence complete
exposition
exemplification
EXPANSION
augmentation
EXTENSION intra-process
distribution
inter-process
divergence
complementarity
time
ENHANCEMENT place
locution
PROJECTION result
idea manner
Intermodal relations: included targeted questions

homospatiality
partial
ELABORATION equivalence
concurrence complete
exposition
exemplification
EXPANSION
augmentation
EXTENSION intra-process
distribution
inter-process
divergence
complementarity
time
ENHANCEMENT place
locution
PROJECTION result
idea manner
 Collapsing types of image/language
relations
CONCURRENCE

Equivalence
Elaboration

Exposition

COMPLEMENTARITY
Augmentation
Extension

Distribution
Two types of intermodal relationships were examined:

1. Concurrence, that is, where there is ideational

equivalence between image and text in
participant-process-phenomenon configuration,
in a relationship where one mode elaborates on
the meaning of the other by further specifying
or describing it while no new element is
introduced by the text or image.

2. Complementarity, where the meanings in the

image/s and text are different but complementary;
that is, different meaning is conveyed through
different modes.
Concurrence: Elaboration: Equivalence
Concurrence: Elaboration: Exposition
Complementarity: Extension: Augmentation
2007 BST P.9 BIG DOG
 distribution

Gill, T. (2002). Visual and Verbal Playmates: An exploration of visual and verbal modalities in childrens picture books. Unpublished
B.A.(Honours), University of Sydney.
 distribution

scrunched up
wrapping paper
 distribution

That Hannah unwrapped the parcel is not specified, but is rather presupposed by the
revelation of what the parcel contained. The presence of scrunched up wrapping
paper on the bed beside Hannah implies that she has just unwrapped a present the
toy gorilla. Thus the image relates to the verbal text by implying that a process
presupposed by the verbal text did indeed occur. [Interestingly, the red colour of the
wrapping paper serves to increase its salience.]

Gill, T. (2002). Visual and Verbal Playmates: An exploration of visual and verbal modalities in childrens picture books. Unpublished
B.A.(Honours), University of Sydney.
 Increasing order of difficulty of image/
language relations
DIFFICULTY
augmentation

distribution

exposition

equivalence
 Findings: image-text relations
in print-based materials
Item difficulty & image-text relations - analysis of variance

I-T RELATION Mean Std. Error 95% Confidence Interval

Lower Bound Upper Bound
augmentation 1.170 .362 .445 1.896
distribution .578 .256 .065 1.091
exposition -.142 .296 -.734 .450
equivalence: partial -.742 .274 -1.290 -.194
equivalence: complete -.967 .274 -1.515 -.418

increasing item difficulty

Dependent Variable: Logit (); mean differences significant at 0.05 level

 Ann Daly Doctoral Study
The Item Analysis function on NSW DET School Measurement,
Assessment and Reporting Toolkit [SMART] was used to order test items
on the 2005 BST according to the percentage of students who achieved
each skill, that is, from easiest to most difficult.

The table below, shows the number of Year 3 and Year 5 items that
assessed image-language interaction out of the total number of items
located in quartiles of difficulty.
 Examples of image-language
augmentation as the most
challenging category for student
reading comprehension.
2005 BST 5
2007 BST 3
2007 BST 5
Tobwabba1 Only 44%
of students
were able to
correctly
answer
question 28.

LOW ACHIEVING READERS
Interviewer: And why have
you chosen the light shape at
the bottom right hand side?
Tianii: Because I wasnt
quite sure and I saw the
closest, like the arrow was
pointing to that light shape and
then there was a fish right next
to it.
Interviewer: Okay then.
Alright. What about question
28?
Claire: How I got that one, I
thought like because of
coming from a corner and the
other sheets were like near it
and it kind of looked like it was
I just chose that one because
it looked more like a net.
Only 44% of students
were able to correctly HIGH ACHIEVING READERS
answer question 28.
Interviewer: Okay, thank you.
For 28, youve looked back
and coloured in the bubble
pointing to the dark shape.
Where did you look?
Theresa: First I didnt know
what it was so I went back to
the paragraph and looked at
what it said were the nets and
the traps.
Tobwabba 2
Interviewer: And I forgot to
ask you about the top picture.
Whats that showing?
Sean: It shows all different
shapes at the back, triangles,
squares, trapeziums, and then
theres a few brown bits, which
they said in the writing is traps.
Only 44% of students were able to correctly answer question 28.

What is new is the assignment

of the ideational meaning of fish
traps and nets to their
symbolization as abstract dark
areas in the painting. This
new information is constructed
jointly by the image and the
language in two steps:
(1) the relating verb shown
identifies the dark areas with
the various fish traps and
nets;
(2) the exophoric reference the
establishes the visual referent
of the verbal participant, the
dark areas.
 Only 37% of
Complementarity: Extension: Augmentation
the New South
2007 BST P.9 BIG DOG Wales State
cohort of year
three children
in government
schools were
able to answer
this question
correctly.
 There is no direct image-
Complementarity: Extension: Augmentation language relation that visually
identifies Jens older brother
2007 BST P.9 BIG
as DOG
the narrator. There are
two self references by the first
person narrator but no visual
representation.

I felt a bit nervous myself.

Only 46% of
the New South
Wales State
cohort of year
five children in
government
schools who
did the test
was able to
answer this
question
correctly.
 46% to answered
correctly.

Low achieving reader: High achieving reader:

Interviewer: How many
characters are in this text?
Francine: Two.
Interviewer: Who are they?
Francine: The kid and
grandpa.
Interviewer: How did you
get your answer?
Francine: Looking at the
picture.
Interviewer: How did you
Puddles get your answer?
Helen: Well from the
speech a lot, cause you can
see that theres three
people speaking, and
yeahthe pictures show
that theres two of them, so
that can mislead you a little
bit, from both.
46% to answered
correctly.
NAPLAN
 How does
NAPLAN address
image/language
relations?

The reality of limited attention to reading

images in text comprehension in NAPLAN
compared with previous NSW Basic Skills
Tests (BST)
Proportion of test items involving images
in the BST 1998-2000
Proportion of test items involving images
in the 2005 and 2007 BST
Proportion of test items involving image in
the 2008 NAPLAN
Proportion of test items involving image in
the 2009 NAPLAN
Proportion of test items involving image in
the 2010 NAPLAN
 Marginalization of multimodal reading in
NAPLAN

Over the 4 tests for years 3, 5, 7 and 9, there were 6

questions that required attention to images out of a total of
164 questions.

Image/text relations in the 2010 NAPLAN

 Question 4 [The pictures of the
skulls are used to show -(key =
the shape of the beak)] targets
the hyponymic relationship
between the feature the shape of
the beak (stated in the answer
key), and the relations of
equivalence which extend across
image and language.

Q 4 does not target the information

relating to the function of beak shape,
which is the hypertheme of the stimulus
text. Only a low-level reading of image-
text relations is required. An integrative
reading of meaning across modes which
targets the central meaning of the text i.e.
relationship between beak form and
function is not probed by the item.
 Also in Year 5

Q 20 The reader first needs to read the main text to

locate lehr and understand it is a long tunnel with
rollers whose function is cooling. Then match cooled
with the caption cooling in the colour diagram (&
possibly match rollers and long tunnel with their visual
equivalents.) Then match the element in the colour
diagram with the black and white equivalent in Q 20.
Q23 Most likely reading steps seem to be:
1. Verbally label Sheet A and Sheet B with the
quality "thick/thin"
2. Read main text to locate "thicker"/"thinner"
3. Note the synonymic link between "speed" in
the question option with "fast" in the main text.

The image/text relation that needs to be

processed here does not actually relate to
the reading of the stimulus text. They could
have said simply "A glass company want to
make thinner glass sheets. What should it
do to achieve this?"
The question targets the relations between the childs
activity, depicted in the image, and her interpretation of
what she views on TV verbalized in a speech bubble.

The words augment the image of the girl watching TV

additional participants and processes not depicted in
image are represented in the projected speech
[projection; simultaneously selects augmentation].
Not a multiple choice question. The question targets the
cohesive relation between pull the plug on a
metaphorical level and the plug attached to the power
point depicted in the picture quite literally.
The diagram is clearly background.
The relationship to the main text is
enhancement: manner (enhancing
sank in the first sentence.)

Caption 4 provides additional information

to the image the image shows
separation of bow and stern; the caption
provides a sequence of resulting states
The bow plunged to bottom, nose first.
Then the stern sank [distribution: intra-
process]
 Potential of NAPLAN for
including image-language
relations
The very limited attention to images and image/text relations
was not due the lack of opportunity in relation to the
affordances of the stimulus texts included in the tests.

The following texts (one narrative and one informational) are

typical of several texts for which no questions at all were
asked about images or image/text relations.
 Cover some
Use a pencil to
divide the frogs
of these with
The need body into sections glue.
for
intermodal Contrast the
reading orange lentils with
the green split
peas.
a. How do you know what is
meant by sections in
step 2?
b. How do you know which
sections to cover with
glue?
c. What does contrast
mean in step 3?
d. Which picture puts you
closest to the frog?
Additional Image Information

The age of the children

Wearing school uniforms
Children happy the positive
Boys interacting with girls
It was probably summer
Image/text relations identify
children
Recycling symbol on box
PISA
As in earlier PISA surveys, the
assessment instruments in PISA
2009 were developed around units.
A unit consists of stimulus material,
including texts, diagrams, tables
and/or graphs, followed by
questions on various aspects of the
text, diagram, table or graph, with
the questions constructed so that
tasks students had to undertake
were as close as possible to those
they might come across in the real
world. (p.22)
The reading literacy framework
developed for PISA 2009 covers
both print and digital literacy. (p.6)
Texts cover the range of materials that are
read. Digital texts are a subset of written
texts. For the purposes of PISA 2009,
digital text is synonymous with hypertext
(a text or texts that are displayed on a
computer or other electronic device with
hyperlinks that allow the reader to
immediately access text that are located in
a different area.). They are texts
composed predominantly of language
rendered in a graphic form. Non-verbal
graphic elements (including illustrations,
photographs, icons and animations)
constitute part of a digital text in PISA,
however oral language (such as audio
recordings) are not included in this
definition of text. (p.7)
 Reading Image/Language
Relations Online

In the third year of the ARC project1, 32 year six students (17
boys and 15 girls) were followed up for a study of online
reading comprehension. The participants included high,
medium and low performers on the 2005 NSW BST from
four metropolitan Sydney schools.

1Unsworth, L. (2006-2008) An Australian Research Council (ARC) Linkage Grant in conjunction with the Educational
Measurement and Schools Assessment Directorate of the New South Wales (NSW) Department of Education and
Training (DET), 2006-2008, entitled New dimensions of group literacy tests for schools: Multimodal reading
comprehension in conventional and computer-based format.
 Study Outline

Each student worked individually with a researcher to read

online a selection of web-pages from sites within Australian
Museum Education online (2003), responding to orally-
presented questions relating to each of the web-pages. The
students answered a total of twenty questions which
required the negotiation of a range of image-text relations,
including those described above for the BST study,
however, in addition to the interaction of image and
language were challenges presented by how the material is
organized on the internet, such as the need to co-ordinate
information from images with text segments hyperlinked
through roll overs or pop up links.
 Results
Detailed Results:
Chan, E. (2009). Image-language interaction in online reading environments: Challenges for
students reading comprehension. Paper presented at the 44th RELC International Seminar
The Impact of Technology on Language Learning and Teaching: What, How and Why.
Singapore, 20 22 April 2009.

In Brief:
The relative difficulty of the different types of image/text
relations found in the BST study (augmentation> distribution>
exposition> equivalence) seemed to be reflected also in the
students negotiation of online texts.
Equivalence of meaning in image
and text

91%
How many legs do
Correct
hatchlings have?
Complementarity augmentation
of meaning in image and text
Which element of nature is represented by the Serpent shown in the painting?
Complementarity augmentation
of meaning in image and text
Which element of nature is represented by the Serpent shown in the painting?

13%
correct
The challenges of the integrative reading of images and
language in multimodal texts are clearly important in reading
both traditional paper media texts and perhaps even more so
in online texts, especially as further issues specific to online
formats, such as the non-simultaneous, sequential display of
a text window followed by an animated image, for example,
add complexity to the integration of information required for
coherent meaning-making from multimodal texts.
Why is theorizing
image/language
interaction crucial?
 A pedagogy of multiliteracies
The orientation of our work in this unit aligns with the agenda for a pedagogy of multiliteracies as

proposed in the Harvard Educational Review by the New London Group (1996). This group

[Courtney Cazden (Harvard, USA); Bill Cope (University of Technology Sydney);

Norman Fairclough (Lancaster University, UK); Jim Gee (Clark University,USA);

Mary Kalantzis (James Cook University of North Queensland);

Gunther Kress (University of London, UK); Allan Luke (University of Queensland);

Carmen Luke (University of Queensland); Sarah Michaels (Clark University, USA);

Martin Nakata (James Cook University of North Queensland)] advise that the multiliteracies

for the future will need to be built on

... an educationally accessible functional grammar; that is, a

metalanguage that describes meaning in various realms.

These include the textual and the visual, as well as the

multi-modal relations between the different meaning-making
processes that are now so critical in media texts and texts of
electronic multimedia.

New London Group (1996). A pedagogy of multiliteracies: designing social futures. Harvard Educational Review,
66(1), 60-91.
 Conclusions: Implications for curriculum,
pedagogy and assessment
reading comprehension tasks involving image/language relations in the NSW BST are
very typical of routine curriculum area reading
different types of image/text relations in hard copy and online texts differ in the degree of
difficulty they pose for students
negotiating some of these image/text relations are among the most difficult reading
comprehension tasks
we need an account of the ways in which images of various kinds interact with language in
different kinds of texts to construct the interpretive possibilities to which readers respond
When governments and education authorities use large scale group reading
comprehension tests as key indicators of students literacy standards,
effectiveness of teaching and of school resource needs, such tests must include
the negotiation of image/text relations
This is essential to understand the nature of reading comprehension, to provide a
basis for pedagogy to ensure students are being taught how to most effectively
interpret multimodal texts, and to inform the assessment of reading
comprehension so that what is being assessed is addressing the fundamental
competencies needed to negotiate the actual texts students need to read and
understand.