B9 MODES, MEANING AND ACTION

Multimodal discourse analysis, the analysis of how multiple modes of

communication interact when we communicate, tends to be divided into two
broad approaches, one which focuses on ‘texts’ (like magazines and web pages)
and the other which focuses on ‘real time’ interactions. One important concept
that is common to both of these approaches is the idea that different modes have
different affordances and constraints. We introduced the idea of affordances and
constraints in the last section when we said that different ‘cultural tools’ make
some actions easier and other actions more difficult.

When we say that different modes have different affordances and constraints, we
mean both that they have different sets of ‘meaning potential’ and that they
allow us to take different kinds of actions. For example, in written text and
spoken language we must present information in a sequential way governed by
the logic of time. Thus, an author or speaker can manipulate the order and speed
at which information is given out, perhaps withholding certain facts until later in
the text or conversation for strategic purposes. Images, on the other hand, are
governed by the logic of space. The producer of the image presents all of the
elements in the image all at once and has limited control over the order in which
viewers look at those elements. Similarly, images allow for the communication of
very fine gradations of meaning when it comes to things like shape and color –
the exact shade of pink in someone’s cheeks, for example – whereas language
forces us to represent things in terms of types – the word ‘pink’ for example,
cannot represent an exact color, but only a range of colors within a particular
class.
The fact that different modes make different kinds of meanings more possible
and others less possible is one of the reasons why people strategically mix
different modes when they are communicating, so that the constraints of one
mode are balanced out by the affordances of others. While there are some things
that ‘just cannot be expressed in words,’ it might be possible to express them
with a carefully timed facial expression or a carefully placed image.

Communicative functions of modes

In Section A4 we introduced Halliday’s idea that language has three basic

functions: it is used to represent our experience of the world; it is used to
communicate something about the relationship between us and the people with
whom we are communicating; and it is used to organize ideas, representations
and other kinds of information in ways that people can make sense of.
WeTcMalled these three functions the ideational function, the interpersonal
function and the textual function. Although these three functions were originally
conceived of as a model for understanding language, Kress and van Leeuwen
insist that they provide a useful starting point for studying all modes. In their
book Reading images: The grammar of visual design, for example, they explore
how images also

89
fulfill these three functions, but do so in a rather different way than language
(see Section D9).

Ideational function

As noted in Section A4, the ideational function of language is accomplished

through the linking together of participants (typically nouns) with processes
(typically verbs), creating what Gee (2011) calls ‘whos doing whats’. In images,
on the other hand, participants are generally portrayed as figures, and the
processes that join them together are portrayed visually.

Images can be narrative, representing figures engaged in actions or events,

classificatory, representing figures in ways in which they are related to one
another in terms of similarities and differences or as representatives of ‘types’,
or analytical, representing figures in ways in which parts are related to wholes.

In narrative images, action processes are usually represented by what Kress and
van Leeuwen call vectors, compositional elements that indicate the directionality
of an action. In figure B9.1, for example, the arm of the boxer on the left
extending rightward towards the head of the other boxer portrays the process of
‘hitting’. There are also other processes portrayed. For example, the upward
gazes of the figures in the background create vectors connecting the spectators
with the fighters.

TM

Figure B9.1 Warriors (photo credit Claudio Gennari)

90
Like this image, many images actually represent multiple processes
simultaneously. Figure B9.2, for example, also involves action processes, the face
on the left (representing a library user) joined to the different kinds of resources
he or she ‘consumes, uses, evaluates, creates, combines, and shares’. This image,
however, is more abstract, and so the vectors are represented as labeled arrows
rather than visual representations of these actions. At the same time, the image
also contains classificatory relationships – the objects portrayed under the
headings ‘Information Literacy’, ‘Media Literacy’ and ‘Digital Literacy’
representing distinct classes of things ‐‐ and analytical relationships ‐‐ the
smaller faces in the lower right corner, for example, portrayed as parts of a
larger social network.

Figure B9.2 Using information, media and digital literacy (credit Karin Dalziel)

Interpersonal function

Another important function of any mode is to create and maintain some

kiTndMof relationship between the producer of the message and its recipient. As
we said in Section A4, in language these relationships are usually created
through the language’s system of modality, as well as through the use of different
‘social languages’ or ‘registers’.

91
In images, viewers are placed into relationships with the figures in the image,
and, by extension, the producers of the image, through devices like perspective
and gaze. The image of the child in figure B9.3 illustrates both of these devices.
The camera angle positions the viewer above the child rather than on the same
level, creating the perspective of an adult, and the child’s direct gaze into the
camera creates a sense of intimacy with the viewer, though the expression on the
child’s face does denote some degree of uncertainty. Another important device
for expressing the relationship between the viewer and the figures in an image is
how close or far away they appear. Long shots tend to create a more impersonal
relationship, whereas close‐ups tend to create a feeling of psychological
closeness along with physical closeness.

Figure B9.3 Child (photo credit Denis Mihailov)

‘Modality’ in images is partially realized by how ‘realistic’ the image seems to the
viewer. Photographs, for example, generally attest more strongly to the ‘truth’ of
a representation than drawings or paintings. However, this is not always the
case. Scientific diagrams and sketches, for example, are often regarded as having
even more ‘authority’ than photographs, and black and white images like those
often found in newspapers are often regarded as more ‘realistic’ than highly
saturated color images in magazine advertisements.
TM
Textual function

As we said above, while texts are organized in a linear fashion based on

sequentiality, images are organized spatially. Figures in an image, for example,
can be placed in the center or periphery of the image, on the top or the bottom,
the left or the right, and in the foreground or in the background. Although

92
producers of images have much less control than producers of written texts over
how viewers ‘read’ the image, they can create pathways for the viewer’s gaze by,
for example, placing different figures in different places within the frame and
making some more prominent and others less prominent.

One obvious way to do this is by creating a distinction between foreground and

background, the figures which seem closer to the viewer generally commanding
more prominence. Another way is to place one or more figures in the center of
the image and others on the margins. Many images make use of the
center/margin distinction to present one figure or piece of information as the
center or ‘nucleus’ of the image and the marginal figures as somehow dependent
upon or subservient to the central figure (Kress and van Leeuwen, 2006).

Two other important distinctions in the composition of images, according to

Kress and van Leeuwen (2006) are the distinction between the left side and the
right side of the image, and the distinction between the upper part and the lower
part. Taking as their starting point, Halliday’s idea that in language, ‘given’
information (information that the reader or hearer is already familiar with)
tends to appear at the beginning of clauses, and new information tends to
appear closer to the end of clauses, they posit that, similarly, the left side of an
image is more likely to contain ‘given’ information and the right side to contain
‘new’ information. This is based on the assumption that people tend to ‘read’
images in the same way they read texts, starting at the left and moving towards
the right.
This, of course, may be different for people from speech communities that are
accustomed to reading text from right to left or from top to bottom.

The distinction between the upper part of an image and the lower part is related
to the strong metaphorical connotations of ‘up’ and ‘down’ in many cultures
(Lakoff and Johnson, 1980). According to Kress and van Leeuwen, the top part of
the image is often used for more ‘ideal’, generalized or abstract information, and
the bottom for ‘real’, specific and concrete information. They give as an example
advertisements in which the upper section usually shows ‘the “promise of the
product”, the status of glamour it can bestow on its users’ and the lower section
tends to provide factual information such as where the product can be obtained
(2006: 186).

Both of these principles can be seen in figure B9.4. In order to make sense of the
‘narrative’ of HIV transmission that the text tells, one must begin at the far left of
the advertisement and move to the far right. The figures of the man and the
woman on the left of the image constitute ‘given’ information, while the virus on
the right of the image constitutes the ‘new’ information. There is also a clear
demarcation between the upper half of the text and the lower half. While the
upper half does not portray the positive ‘promise’ of a particular product aTs M
many advertisements do, it does represent a kind of idealized hypothetical
situation which the viewer is invited to imagine. Rather than a ‘promise’,
however, it is something more akin to a ‘threat’. And the lower half of the image,
rather than giving information about where the product portrayed in the upper

93
half can be obtained, it gives information on how this hypothetical situation can

94
be avoided, along with specific information about such things as the name of the
organization that produced the ad and the condom company that sponsored it.

Figure B9. 4 AIDS prevention advertisement (Abrasco, Brazil)

This text also illustrates how images and words often work together. The words
‘Joy Stick’, ‘Play Station’ and ‘Game Over’ tell the viewer how the images are to be
interpreted, and the slogan in the lower half of the text (‘You only have one life:
use a condom’), explains the image of the condom above it. Finally, this text
shows how multimodality can be effective in getting viewers to make
connections between different ‘Discourses’. While the images belong to the
‘Discourse of biomedicine’, the words invite the viewer to interpret these images
within the framework of the ‘Discourse of video games’.

 Find additional examples online

Multimodality in Interaction

Modes in face‐to‐face interaction such as gaze and gesture also fulfill these three
functions. The mode of gaze, for example, has an obvious interpersonal function,
creating a relationship between the gazer and whomever or whatever is the
object of the gaze. It also carries ideational meaning, conveying that the gazer is
looking at, watching or paying attention to something. Finally, gaze is oftenTaMn
important textual resource, helping people to manage things like turn‐taking in
conversations.

While the ‘inter‐modal’ relationships (the ways multiple modes work together)
in static texts like the advertisement analyzed above can be complicated, they
can be even more complicated in dynamic interactions. One of the problems
with

95
analyzing real time, face‐to‐face interactions is that participants have so
many modes available to them to make meaning. There are what Norris
(2005) calls ‘embodied’ modes such as gaze, gesture, posture, head
movement, proxemics (the distance one maintains from his or her
interlocutor), spoken language, and prosody (features of stress and
intonation in a person’s voice). And there are also ‘disembodied’ modes
like written texts, images, signs, clothing, the layout of furniture and the
architectural arrangement of rooms and other spaces in which the
interaction takes place. All of these different modes organize meaning
differently. Some, like spoken language and gaze tend to operate
sequentially, while others like gesture and prosody tend to operate
globally, often helping to create the context in which other modes like
spoken language are to be interpreted (see Section B6). Not all of these
modes are of equal importance to participants at any given moment in
the interaction. In fact, different modes are likely to take on different
degrees of importance at different times. How then is the analyst to
determine which modes to focus on in a multimodal analysis?

Another problem with analyzing multimodality in face‐to‐face

interactions is that the spatial boundaries of interactions are not always
as clear as the spatial boundaries of texts. While the frame of an image
clearly marks what should be considered as belonging to the image and
what should be considered external to it, a conversation in a coffee shop
is not so clearly bounded. In analyzing such an interaction, how much of
the surrounding modes should be taken into account? Should the
analyst consider, for example, the signs and posters on the walls, the
conversations occurring at other tables, the ambient music playing over
the p.a. system and the sounds of milk being steamed? What about the
smell and taste of the coffee?

Norris (2005) solves these two problems by adopting the practice of

mediated discourse analysis (see Sections A8 and B8) and taking action
as her unit of analysis. Thus, in determining which modes to focus on,
the analyst begins by asking what actions participants are engaged in
and then attempts to determine which modes are being used to
accomplish these actions.

As we said in Section A8, actions are always made up of smaller actions

and themselves contribute to making up larger actions. Norris divides
actions into three types: lowerlevel actions, the smallest pragmatic
meaning units of communicative modes (including things like gestures,
postural shifts, gaze shifts, and tone units), higherlevel actions (such as
‘having a cup of coffee’), and frozen actions (previously performed
actions that are instantiated in material modes—a half eaten plate of
food, for example, or an unmade bed).
One of the goals of multimodal interaction analysis, then, is to
understand how participants in interaction work cooperatively to
weave together lower‐levTeMl actions like gestures, glances, and
head and body movements into higher‐level actions, and, in doing, so
help to create and reinforce social practices, social relationships and
social identities (see Section C9).

 Look deeper into this topic online