A Social Semiotic Analysis of the

Multimodality of an Advertisement^

Dan Xu
Nanjing Normal University, Nanjing, China

Abstract; For decades, multimodality, an intrinsic attribute of

discourse, has long been neglected in discourse analysis. Adopting the
theoretical framework of visual communication grammar by Kress and
van Leeuwen, which grew out of Halliday's systemic functional
grammar, this paper makes a detailed social semiotic analysis of the
three-fold metafunctions of the visual and verbal modes of one of the No
Smoking Day print advertisements. The analysis shows that these two
semiotic modes complement and supplement each other to add to the
denotation and connotation of the discourse.
Keywords; social semiotics; multimodality; advertisements

1. Introduction
For a long time, "verbal mode" has been the research focus of linguists.
Discourse analysis used to be mainly concerned with the spoken and written
forms of discourse, for instance, literary works, newspaper articles,
advertisement slogans, etc. However, no verbal mode of discourse can exist
alone. Instead, many types of texts are often produced with illustrations and
different typefaces; newspaper articles are likely to be accompanied by eye-
catching photos relating to the news ; in print advertisements, images always take
a more salient position with special size and color. While the nonverbal parts of
such discourses have been regarded as ancillary and are treated as the research
subject of specialists in other fields, image research is the business of specialists
in art design, and the effect of music should be studied by musicians. It seems
that everyone has his/her specialty in a special discipline; but in practice, it's

φ A Project funded by the Priority Academic Program Development of Jiangsu Higher

Education Institutions (The project number is 20110101).
• 154 ·

Brought to you by | New York University Bobst Library Technical Services

Authenticated
Download Date | 2/29/16 1:58 PM
Chinese Semiotic Studies Dan Xu

hard to find a single mode which exists alone in a discourse. Every mode plays a
part in meaning making. Kress & van Leeuwen (2001) point out that semiotic
resources should be integrated, and that a multimodal approach should be
adopted.
Adopting the multimodal approach, this thesis is going to analyze one of the
No Smoking Day print advertisements in the image library of No Smoking Day
organization's official website : www. nosmokingday. org. uk. To make the
campaign more effective, the No Smoking Day organization will invite special art
designers to design eye-catching advertisements.
This thesis attempts to answer the following questions :
• How do the No Smoking Day advertisements persuade smokers?
• In what way can the advertisements convey the intended meaning?
• What is the relationship between visual images and verbal texts as well as
the layout?
• How do the semiotic modes interact in the No Smoking Day
advertisements?

2. Theoretical Framework of Multimodal Analysis

2.1 Social Semiotics

The term semiotics has been defined as "the science of the life of signs in
society" (Saussure, 1974). "Social semiotics has developed out of an intensive
critical reading of earlier work from a particular stand point, rejecting some parts,
incorporating, reordering or transforming other parts into a theory which aims to
be coherent and powerful in its own right" (Hodge & Kress, 1988, p. 13).
Social semiotics is strongly inspired by the Paris School Semiotics of the
1960s and 1970s. It applies linguistic ideas from Saussure and other linguists to
other fields of communication. And the key impetus for the development of social
semiotics is Μ. A. K. Halliday's Systemic Functional Grammar. The
development of social semiotics is also influenced by critical discourse analysis.
Thus the theory of social semiotics has been outlined by a number of scholars
such as Kress & van Leeuwen, O'Toole, O'Halloran, Baldry & Thibault, Lemke,
etc.
Hodge & Kress think social semiotics emphasizes social action, context and
use (1988, p. 5 ) . It has two premises, "the first is the social dimension in
understanding language structure and process...[the second is] no single code
can be successfully studied or fully understood in isolation" (1988, p. vii).
Van Leeuwen writes in his Introducing Social Semiotics (2005, p. x i ) :

• 155 ·

Brought to you by | New York University Bobst Library Technical Services

Authenticated
Download Date | 2/29/16 1:58 PM
Chinese Semiotic Studies

Just as in linguistics, the focus changed from the "sentence" to the

"text" and its "context", and from "grammar" to "discourse", so in
social semiotics, the focus changed from "sign" to the way people use
semiotic " resources" both to produce communicative artefacts and
events, and to interpret them—which is also a form of semiotic
production—in the context of special social situations and practices.

Rather than constructing separate accounts of the various semiotic modes—

the " semiotics of image", the " semiotics of music", etc., social semiotics
compares and contrasts semiotic modes, exploring what they have in common as
well as how they differ, and investigating how they can be integrated in
multimodal discourses and events.
2.2 Key Concepts of Multimodality
Multimodality has been developing fast since the 1990s. The pioneering work
is in the application of systemic functional linguistics to architecture, sculpture,
and visual semiotics. Gradually, a number of scholars such as Lemke and
O'Halloran also adopt the systemic functional linguistic approach to study other
semiotic resources such as mathematics, science, hypertext, etc. Here are
some key concepts in multimodality.
"Discourse" is a very important concept in social semiotics. It is different
from that in the linguistic field. Kress & van Leeuwen (2001, p. 2 0 ) point out :

Discourses are socially situated forms of knowledge about (aspects

of reality). This includes knowledge of events constituting that reality
(who is involved, what takes place, where and when it takes place,
and so o n ) as well as a set of related evaluations, purposes,
interpretations and legitimations.

Another concept is "mode". "Modes are semiotic resources which allow the
simultaneous realization of discourses and types of (inter)action" (ibid, p. 2 0 ) ,
such as the visual mode, the verbal mode, the musical mode, etc. A multimodal
discourse usually combines several modes.
"Multimodality" is defined as "the use of several semiotic modes in the
design of semiotic product or event, together with a particular way in which these
modes are combined" (ibid, p. 2 0 ) .
"Modality" in multimodal discourse refers to how people use semiotic
resources to create truth. It's a very complicated concept and has different
modality markers in different semiotic resources. It does not simply equal to
modal verbs or auxiliary verbs, i. e. can, may, and must.
• 156 ·

Brought to you by | New York University Bobst Library Technical Services

Authenticated
Download Date | 2/29/16 1:58 PM
Chinese Semiotic Studies Dan Xu

2.3 Visual Social Semiotics

Kress & van Leeuwen have developed a framework of social semiotic
analysis of visual communication. They assign to images the three metafunctions
of representational, interpersonal and compositional meanings.
The representation of images is realized by narrative representation and
conceptual representation. There are two participants in every semiotic act, the
interactive participants (viewer, reader, image designer), and the represented
participants (things depicted in the image). The narrative pattern is about
unfolding actions and events, process of change, and transitory spatial
arrangements. Vectors can represent actional process, and eyelines reflect
reactional process. By using vectors and eyelines, images can be represented
"narratively", that is what we say in verbal texts of "doing" and "happening"
(Kress & van Leeuwen, 2006, p. 73). The hallmark of a narrative visual
proposition is the presence of a vector. Narrative structures always have one,
while conceptual structures never do (Kress & van Leeuwen, 2006). The
conceptual image does not involve actions and reactions of the represented
participants; it reveals more generalized, almost stable, timeless essence, in
terms of class, structure and meaning.
As far as the interpersonal feature is concerned, it is expressed by facial
expressions, perspectives and social distances. And the relationship between the
represented participants and interactive participants is taken into consideration.
The designer uses the image to do something to the reader. If the represented
image looks at the reader directly, it forms an image act of "demand". If the
represented participants address the reader indirectly, this kind of image is a kind
of "offer" (Kress & van Leeuwen, 2006). Social distance reveals the level of
involvement and power relations between the image designer and the reader.
Perspective means the selection of angles, which reveals the subjective
attitude towards represented participants. At a high angle, the viewer is superior
to the represented participants, at a low angle, the viewer is at an inferior status,
and at the eye level, it means equality. High and low are vertical angles, and
they are closely related to power, horizontal angles of frontal and oblique reveal
the relationships between involvement and detachment.
Modality refers to the reliability of messages. It relates to both issues of
representation and social interaction. Kress& van Leeuwen have outlined the
following visual standards for visual modality: color saturation, color
differentiation, color modulation, contextualization, representation, depth,
illumination and brightness. On the basis of the theoretical work of Jürgen
Harbermas and Bourdieu, they adopt Bernstein's term "coding orientation" to
refer to the different " sets of principles which inform the way in which texts are
• 157 ·

Brought to you by | New York University Bobst Library Technical Services

Authenticated
Download Date | 2/29/16 1:58 PM
Chinese Semiotic Studies

coded by specific social groups, or within specific institutional context" (2006.

p. 165). They distinguish four coding orientations; technological, sensory,
abstract, and naturalistic.
Finally, the analysis of compositional features relates to the information
value, salience and visual frame. Information value means that different
placements of elements have specific information values. For example, "left and
right" reveals "given and new", "top and bottom" corresponds with "ideal and
real", and "centre and margin" is linked with "important and subservient".
Salience can help the represented participants by attracting the viewer's
attention. Salience can be achieved with the help of such factors as foreground
and background contrast, relative size, contrasts in color or differences in
sharpness, etc. The third key element is framing. Frames can create a sense of
detachment between the picture and the reader. The stronger the framing of an
element is, the more it is represented as a separate unit of information. If there is
no frame, then the image covers the whole area of the page or a double spread
and it can invite the reader into the picture.
Table 1 The metafunctions of verbal mode and visual mode
Semiotic Modes Functions Subsystems
Ideational Transitivity, modulation, polarity

Language Interpersonal Mood, modality

Textual Theme, information distribution, cohesion

Representational Narrative representation , conceptual representation

Image Interactive Image act, social distance, perspective, modality

Compositional Information value, salience, framing

2.4 Hasan's Generic Structure Potential of Advertisements

Generic Structure Potential (or GSP for short) expresses "the total range of
optional and obligatory elements" (Halliday & Hasan, 1989, p. 64). Hasan
(1996, pp. 41—42) proposes a generic structure for advertisements "Capture "
Focus " Justification". The caret sign indicates sequence. "Capture" is to attract
attention, to be " realized in the written mode through the management of visual
layout, the typeface patterns and/or the presence of pictures". "Focus signals
out which is being advertised". "Justification" that indicates the content of the
advertisement is convincing well defends the " Focus".
Hasan proposes in this GSP that visual images have certain functions. She
mentions that Capture and Focus can be realized visually in the form of pictures.
• 158 ·

Brought to you by | New York University Bobst Library Technical Services

Authenticated
Download Date | 2/29/16 1:58 PM
Chinese Semiotic Studies Dan Xu

This GSP provides only a general picture of the multimodal nature of the
advertisements, for Hasan does not specify all the elements that could possibly
appear in the structure of advertisements and she doesn't clarify the interaction
between different semiotic modes in this GSP.

2.5 Cheong Yin Yuen's Generic Structure Potential of Advertisements

Based on Hasan's GSP, Cheong Yin Yuen (2004) from National University
of Singapore has explored a modal best expressing the interaction between
multimodalities.
Lead * (Display) * Emblem * (Announcement) * (Enhancer) * (Tag) ' (Call-
and-Visit Information)
The round brackets " ( ) " indicate optionality of enclosed elements, so
Display, Announcement, Enhancer, Tag, and Call-and-Visit Information are
optional.

Table 2 Proposed generic structure of print advertisement (Cheong, 2004, p. 165)

Visual components Lead: Locus of Attention (LoA) ; LoA

Complement to the Locus of Attention ( Comp. )
Display: Explicit, Implicit, Congruent, Incongruent
Emblem
Linguistic components Announcement; Primary Secondary
Enhancer
Emblem
Tag
Call-and-Visit Information

This table shows the visual and linguistic components in advertisements.

They are combined by their interaction to create ideational, interpersonal and
textual meanings.

3. Discussion

We will have a detailed explanation of the function of these elements and

how they interact in the following parts.

• 159 ·

Brought to you by | New York University Bobst Library Technical Services

Authenticated
Download Date | 2/29/16 1:58 PM
Chinese Semiotic Studies

Figure 1 No Smoking Day Advertisement

According to Cheong, Lead is interpersonally most salient through choices in

size, position and color. It has the greatest meaning potential. Figure 1, the
middle picture which takes much of the space of the advertisement is the Lead. It
draws people's attention first. Lead consists of Locus of Attention (LoA) and the
Complement to the Locus of Attention (Comp. LoA). LoA is the element which
is placed in the foreground in the Lead, and has an outstanding size, color, etc.
To be exact, it is LoA that captures readers' attention. In Figure 1, the
young lady and the gentleman are most salient in the Lead. They look at each
other seriously. Judging from their distance, readers may have a clue that they
have intimate relations. This close distance can enable them to hug or whisper.
Readers will guess what happened to them and what would happen next. It
reveals the ideational meaning and interpersonal meaning and serves as a
"spring board" for the further development of the central idea. The idea is
reinforced in the linguistic text : "Your Big Chance to KISS IT GOODBYE". Here
it constitutes an announcement. In O'Toole's ( 1 9 9 4 ) terms, the Announcement
has Relative Prominence in scale and color, font and size. The typeface of
"Your big chance to" is designed artistically, while "KISS IT GOODBYE" is
designed in a very formal style, all bold capital letters. On one hand, it becomes
• 160 ·

Brought to you by | New York University Bobst Library Technical Services

Authenticated
Download Date | 2/29/16 1:58 PM
Chinese Semiotic Studies Dan Xu

next salient to the LoA, on the other, it conveys ideational and interpersonal
meanings. It's not a romantic kiss goodbye; it's formal and serious. It will also
lead the reader back to LoA, thus reminds the reader that it's a farewell scene.
The Complement to the LoA means the component accompanying LoA,
which is less salient. It serves a subordinate role in the Lead. In Figure 1, there
are two complements to LoA. The first is the image of extinguishing a cigarette in
an ashtray. It is overlapped at the shoulder part of the gentleman. Compared
with LoA, this image is reduced in size. It serves as a link between LoA and
Announcement. The kiss goodbye scene has something to do with the
gentleman's extinguishing cigarettes. In the close distance, it's easy for the lady
to smell the smoke from the gentleman's clothes. And some imagined lines of the
lady are triggered: " Honey, you've promised to give up smoking many times,
and I've already given you many more chances. This time, I'm serious. If you
don't give it up, then w e ' re definitely over." What*s more, it reveals the
rhetorical device of pun in the Announcement. If you don't take this opportunity to
kiss cigarettes goodbye, then your chance of kissing your beloved goodbye is
coming.
The other Comp. LoA is the background of the Lead. It is a curtain in a soft
focus, without being illuminated. This background serves to contrast with the
well-illuminated LoA, and brings it into the reader's focus.
There is an interplay between the Lead and the Announcement. The Lead of
Figure 1 serves as a reference to the Announcement. It compresses linguistic
meaning. Thus the combination of them endows greater persuasive power, by
"mutually reinforcing and enhancing" (Cheong, 2004, p. 168).
As for Display, it refers to how the Lead is presented in the advertisement;
explicitly, implicitly, congruently, or incongruently (Cheong, 2004). In Figure 1,
we have implicit display of the LoA and explicit display of Comp. LoA 1. The
congruent display is to present the thing advertised in a symbolic way, while the
incongruent is just the opposite.
The Emblem may be realized by the logo of the thing advertised and its
linguistic realization is in the form of the brand name of the product/service
(Cheong, 2004, p. 171). The emblem makes use of the text, image, color and
typography to convey an ideational and interpersonal meaning. It can be located
anywhere in the advertisement. The Enhancer is usually a paragraph of linguistic
explanation of the thing advertised. It aims to persuade the reader of some
convincing qualities of the product. As Cheong has mentioned, Enhancer is an
optional element; therefore, there is no Enhancer in Figure 1. Tag is something
not included in the Enhancer. In Figure 1, it is in the smallest size : for help and
advice on stopping smoking and information on local sen/ices. Call-and-visit
Information is used for further contact. All the No Smoking Day advertisements
• 161 ·

Brought to you by | New York University Bobst Library Technical Services

Authenticated
Download Date | 2/29/16 1:58 PM
Chinese Semiotic Studies

have it.
The reading path of Figure 1 is :
Lead—Announcement—Emblem—Call-and-Visit Information—Tag
This reading path is decided by the salience of the components.
We have discussed the interaction of different components in Cheong Yin
Yuen's GSP. Elements in a multimodal discourse don't do addition. It's not that
one plus one equals two, but that one plus one is greater than two.
Lead carries a lot of interpersonal and ideational meaning. It is linked to the
Announcement. Announcement and Lead work bidirectionally. Lead can enrich
ideational meaning of the Announcement by meaning compression.
Announcement acts as stabilizer to straitjacket meaning. The visual and verbal
components not only elaborate each other but also extend and enhance each
other. That's the essence of multimodality.
Only by multimodal analysis can we completely understand the meaning of
multimodal discourse.
There are many printed and TV advertisements which make a conscious
effort to restrict or do without language altogether or which more subtly and
deceptively use the language as the apparent primary source of meaning,
whereas from the advertiser's standpoint, the most important things are made
possibly more surreptitiously and covertly, by other modalities such as clothes
(Barthes, 1960 ) , music and sound ( v a n Leeuwen, 1999 ) , gesture
(Birdwhistell, 1952, 1972 [ 1 9 6 1 ] ; Kendon, 1981; McNeil, 1992), movement
(Birdwhistell, 1952, 1972 [ 1 9 6 1 ] ; Kendon, 1981), posture (Scheflen, 1972,
1973) and spatial relations (Hall, 1972 [ 1 9 6 3 ] ) (cited from Baldry & Thibault.
2006, p. 2 0 ) .
The multimodal nature of the No Smoking Day advertisements enacts us to
rethink the discourse analysis we have made previously.
Quoting Barthes, Cook (1992, p. 216) states that advertisements represent
a "restless" discourse type. He explains (ibid,p. 2 1 7 ) :

The conventions of advertisements change fast, driven by an

internal dynamic, by changes in society, and by changes in the
discourse types on which they are parasitic or in which they are
embedded... they are... constantly transmuting and re-combining, so
that at present any lasting characterization is impossible.
Synchronically, there are too many exceptions. Diachronically, the rules
are in a flux.

So the GSP for advertisement is, in Cheong's own terms, "chameleon like,
• 162 ·

Brought to you by | New York University Bobst Library Technical Services

Authenticated
Download Date | 2/29/16 1:58 PM
Chinese Semiotic Studies Dan Xu

slippery to define and ever-evolving". Although Cheong has detailed the structure
in the print advertisements, this structure does not intend the nature of the
meaning-making process in the No Smoking Day advertisement.
Analyzing metafunctions is a better way to approach meaning-making
process in the multimodal advertisements. As we've discussed, the three-fold
metafunctions in the visual mode and verbal mode are approximately the same,
although they are realized by different subtypes.
Do these metafunctions of different modes juxtapose and function only in
their own modes or do they interact?
The experiential function of verbal modes expresses the meaning of doing,
happening, feeling, etc. and the participants in these processes. In the visual
mode, we also have narrative process and conceptual process. The Lead and
the Announcement in the advertisement "KISS IT GOODBYE" use reaction
process and material process. The Announcement influences the reader's
interpretation of the reactional process and the advertiser's conveyance of
meaning. " It provides a context for viewers to adapt/pursue the preferred thread
of meaning intended by the advertisers" (Cheong, 2004, p. 179). The Lead
enriches the experiential meaning carried in this Announcement. It is a visual
exemplification of the statement " KISS IT GOODBYE". If we replace the
Announcement with something else, say the names of the two characters in the
Lead: Humphrey Bogart and Lauren Bacall, it will look like a movie poster.
Actually, Bogart and Bacall made several movies together in the 1930s and
1940s and they were married to each other. People who are familiar with these
movies will instantly recognize this image. And Bogart is a heavy smoker in
reality, he died of a smoking-related illness at the age of 57. So the experiential
meaning of the Lead and the Announcement is bidirectionally invested.
In the interpersonal meaning, we have a speech act and an image act. The
Announcement is a request. On reading this request, the reader has no idea of
who kisses whom goodbye. The gaze of Bogart and Bacall in the Lead together
with their stance and distance suggest the relationship between them and they
are talking something seriously, but without the hint of the Announcement, the
reader cannot interpret the exact meaning the advertiser wants to convey.
Hence, the interpersonal meaning in the Lead and the Announcement mutually
expand.
Textually, it is a kind of cross-mode cohesion between the Lead and the
Announcement. They both refer to the same scene. Moreover, the typography of
"KISS IT GOODBYE" is in bold capital letters; and it has more weight than other
verbal components. As we have mentioned, the Lead also takes the salient
position. The textual and compositional function of the visual and verbal modes
coordinate to enhance the meaning of this advertisement.
• 163 ·

Brought to you by | New York University Bobst Library Technical Services

Authenticated
Download Date | 2/29/16 1:58 PM
Chinese Semiotic Studies

4. Conclusion
The investigation of how visual and verbal semiotic resources interact shows
that they do not simply add to each other, and they elaborate each other, extend
the meanings of each other, and enhance the effect of integration. All these
interact to make the No Smoking Day advertisement more powerful persuasions
and they are well received by people.
The theory of multimodality is still developing. It includes the knowledge of a
variety of disciplines, for example, fashion, arts, computer technology,
architecture, etc. Interdisciplinary knowledge is necessary for researchers to carry
out multimodal discourse analysis. Because of limited knowledge about art, it's
hard for me to write the part on color, and it is not so objective and neutral.
Moreover, multimodal discourse analysis relies a lot on computer-based
technology. So does this one. If I can do some further image process by a
computer-based skill, like when analyzing the interactions of verbal and visual
part, using computer technology to obliterate the other resources and then
contrast them. That will be a more effective illustration.

References
Baldry, A. & P. J. Thibault ( 2 0 0 6 ) . Multimodal Transcription and Text
Analysis. London; Oakviile.
Cheong, Υ. Y. ( 2004 ). The Construal of Ideational Meaning in Print
Advertisements. In O'Halloran, K. ( E d s . ) , Multimodal Discourse Analysis.
London & New York; Continuum, pp. 163—195.
Cook, G. (1992). The Discouse of Advertising. London : Routledge.
Halliday, Μ. A. K. (1976). Systems and Functions in Language (G. Kress,
E d . ) . London : Oxford University Press.
Halliday, Μ. A. K. (1978). Language as Social Semiotic. London: Arnold.
Halliday, Μ. A. K. & R. Hasan ( 1 9 8 9 ) . Language, Context and Text (2nd
e d . ) . Victoria; Deakin University.
Hasan, R. (1996). What's Going On : A Dynamic View of Context in Language.
In C. Cloran, D. Butt and G. Williams ( E d s . ) , Ways of Saying·. Ways of
Meaning. London; Cassel, pp. 3 7 — 5 0 .
Hodge, R. & G. Kress (1988). Social Semiotics. Oxford; Polity Press.
Kress, G. & T. van Leeuwen (2001). Multimodal Discourse—The Grammar of
Visual Design. London; Arnold.
Kress, G. & T. van Leeuwen ( 2 0 0 6 ) . Reading Images—The Grammar of
Visual Design. London : Routledge.
Jewitt, C. & G. Kress ( E d s . ) . (2003). Multimodal Literacy. New York; Peter
Lang.
• 164 ·

Brought to you by | New York University Bobst Library Technical Services

Authenticated
Download Date | 2/29/16 1:58 PM
Chinese Semiotic Studies Dan Xu

Martin, J. R. ( 1992 ) . English Text—System and Structure. Amsterdam;

Benjamins.
O'Toole, M. ( 1 9 9 4 ) . The Language of Displayed Art. London; Leicester
University Press.
Saussure, F. de. (1974). Course in General Linguistics ( J . Culler, Ed. W.
Baskin, Trans.). London : Fontana.
Thibault, P. J. ( 1 9 9 7 ) . Re-reading Saussure·. The Dynamics of Signs in Social
Life. London and New York : Routeledge.
van Leeuwen, T. (2005). Introducing Social Semiotics. London; Routledge.

165 ·

Brought to you by | New York University Bobst Library Technical Services

Authenticated
Download Date | 2/29/16 1:58 PM