8.

1 Background to multimodal discourse analysis

Much of the work in multimodal discourse analysis draws from Halliday’s ( 1978 , 1989a ) social
semiotic approach to language, a view that considers language as one among a number of
semiotic resources (such as gesture, images and music) that people use to communicate, or
make meaning, with each other. Language, in this view, cannot be considered in isolation from
meaning but needs to be considered within the sociocultural context in which it occurs.
Multimodal discourse analysis, thus, aims to describe the socially situated semiotic resources
that we draw on for communication.

Multimodal discourse analysis considers how texts draw on modes of communication such as
pictures, film, video, images and sound in combination with words to make meaning. It has
examined print genres as well as genres such as web pages, film and television programmes. It
considers how multimodal texts are designed and how semiotic tools such as colour, framing,
focus and positioning of elements contribute to the making of meaning in these texts.

Halliday (2009a) describes three types of social meanings, or functions that are drawn on
simultaneously in the use of language. These are ideational (what the text is about),
interpersonal (relations between participants) and textual meanings (how the message is
organized).

8.2 Examples of multimodal discourse analysis

Kress ( 2010 ) in Multimodality: A social semiotic approach to contemporary communication

provides a social-semiotic theory of multimodality. Among other things, a social semiotic theory
of multimodality asks:
-What meaning is being made in a text?
-How is meaning being made in the text?

-What resources have been drawn on to make the meaning in the text?

-In what social environment is the meaning being made?

-Whose interest and agency is at work in the making of the meaning?

In the case of the cover of Time , the image of Michelle Obama is foregrounded and carries the
major informational load in the text. It aims to attract readers to buy the magazine and read the
article on Michelle Obama within it pages. The text is written within the early days of Barrack
Obama’s time as president of the United States and in its image, and text, describes a Michelle
Obama who is both casual and in charge. The image that is chosen for the cover and its
presentation conveys exactly this message. It is an orderly yet inviting image that parallels the
way she is described in the text inside the pages of the magazine. The authors of the text (Gibbs
and Scherer 2009 ) clearly have an interest in presenting this image to their readers and give
agency to their subject by presenting, with sympathy, her personal views on life in the White
House, as the First Lady, and the way she conducts her self in this role.

8.3 Genre, speech acts and multimodality

Van Leeuwen ( 2005a , 2005b ) discusses speech acts and genre in relation to multimodality,
using these two notions to capture the ‘how’ (vs. the ‘what’) of multimodal communication. A
key point he draws from speech act theory is how a speech act is both an illocutionary act
(what the speech act is aiming to do) and a perlocutionary act (the effect it has on the thoughts
and actions of people). An advertisement, thus, may aim to persuade a person to buy a
particular product (the illocutionary act). If the person is convinced by the advertise ment and
buys the product, this is the effect, or perlocutionary force , of the advertisement. Texts, thus,
draw on a range of modalities to create a perlocutionary effect. It is not just the words of the
advertisement that persuade a person to buy the product. It is through the use of linguistic and
visual resources in combination with other non-linguistic and contextual factors that this occurs
(van Leeuwen 2005a ).