Boria Et Al (2020) Translation and Multimodality

TRANSLATION AND
MULTIMODALITY
Translation and Multimodality: Beyond Words is one of the first books to explore how
translation needs to be redefined and reconfigured in contexts where multiple
modes of communication, such as writing, images, gesture, and music, occur
simultaneously. Bringing together world-leading experts in translation theory and
multimodality, each chapter explores important interconnections among these
related, yet distinct, disciplines.
As communication becomes ever more multimodal, the need to consider
translation in multimodal contexts is increasingly vital. The various forms of
meaning-making that have become prominent in the twenty-first century are
already destabilising certain time-honoured translation-theoretic paradigms, causing
old definitions and assumptions to appear inadequate. This ground- breaking
volume explores these important issues in relation to multimodal translation with
examples from literature, dance, music, TV, film, and the visual arts.
Encouraging a greater convergence between these two significant disciplines,
this text is essential for advanced students and researchers in Translation Studies,
Linguistics, and Communication Studies.
Monica Boria is Affiliated Lecturer in the Department of Italian at the University

of Cambridge and a translator.
Ángeles Carreres is Senior Language Teaching Officer in the Department of

Spanish and Portuguese at the University of Cambridge.
María Noriega-Sánchez is Senior Language Teaching Officer in Spanish at the

University of Cambridge, and a Fellow in Modern Languages at Sidney Sussex College.
Marcus Tomalin is Fellow at Trinity Hall, Cambridge, and a Senior Research

Associate in the Cambridge Machine Intelligence Laboratory.
TRANSLATION AND
MULTIMODALITY
Beyond Words
Edited by
Monica Boria, Ángeles Carreres,
María Noriega-Sánchez, and
Marcus Tomalin
First published 2020
by Routledge
2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN
and by Routledge
52 Vanderbilt Avenue, New York, NY 10017
Routledge is an imprint of the Taylor & Francis Group, an informa business
© 2020 selection and editorial matter, Monica Boria, Ángeles Carreres,
María Noriega-Sánchez, and Marcus Tomalin; individual chapters, the
contributors.
The right of Monica Boria, Ángeles Carreres, María Noriega-Sánchez, and
Marcus Tomalin to be identified as the authors of the editorial material, and
of the authors for their individual chapters, has been asserted in accordance
with sections 77 and 78 of the Copyright, Designs and Patents Act 1988.
All rights reserved. No part of this book may be reprinted or reproduced or
utilised in any form or by any electronic, mechanical, or other means, now
known or hereafter invented, including photocopying and recording, or in
any information storage or retrieval system, without permission in writing
from the publishers.
Trademark notice: Product or corporate names may be trademarks or
registered trademarks, and are used only for identification and explanation
without intent to infringe.
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library
Library of Congress Cataloging-in-Publication Data
A catalog record has been requested for this book
ISBN: 978-1-138-32442-8 (hbk)
ISBN: 978-1-138-32443-5 (pbk)
ISBN: 978-0-429-34155-7 (ebk)
Typeset in Bembo
by Apex CoVantage, LLC
Visit www.routledge.com/9781138324435
This book is dedicated to our good friend and former colleague,
Nicole Robertson, without whom it would never have existed.
CONTENTS
List of figures ix
List of contributors x
Acknowledgementsxii
Introduction1
Monica Boria and Marcus Tomalin
1 Transposing meaning: translation in a multimodal semiotic

landscape24
Gunther Kress
2 A theoretical framework for a multimodal conception of

translation49
Klaus Kaindl
3 Meaning-(re)making in a world of untranslated signs:

towards a research agenda on multimodality, culture,
and translation 71
Elisabetta Adami and Sara Ramos Pinto
4 From the “cinema of attractions” to danmu: a multimodal-

theory analysis of changing subtitling aesthetics across
media cultures 94
Luis Pérez-González
viii Contents
5 Translating “I”: Dante, literariness, and the inherent

multimodality of language 117
Matthew Reynolds
6 The multimodal dimensions of literature in translation 134

Marcus Tomalin
7 Translations between music and dance: analysing the

choreomusical gestural interplay in twentieth-and twenty-
first-century dance works 158
Helen Julia Minors
8 Writing drawingly: a case study of multimodal translation

between drawing and writing 179
Tamarin Norwood
Beyond words: concluding remarks 198

Ángeles Carreres and María Noriega-Sánchez
Index 204
FIGURES
0.1 The squiggle flourish 4

0.2 Tramonto 15
1.1 The translation game 25
1.2 Drawing by a three-year-old: “this is a car” 34
1.3 Plant cell 37
1.4a Medical student touches patient 40
1.4b Surgeon touches patient 40
1.5 “Fixing” the ephemeral: documenting meaning in a
game of “chase” 43
3.1 Bleeding nose used in anime to signify sexual arousal 73
3.2–3.4 Amy, Penny, and Bernadette go wedding dress shopping 81
4.1 Screenshot of video material featuring danmu107
6.1 Le Dromadaire 143
6.2 New image 149
6.3 New text 149
7.1 Mapping music-dance interdependency read through the
translation of languages, senses, and cultures 164
7.2 Bars 79–82 167
7.3a Bar 1 167
7.3b Bar 94 167
CONTRIBUTORS
Elisabetta Adami is University Academic Fellow in Multimodal Communication

at the University of Leeds. She specialises in social semiotic multimodal analysis,
with a recent focus on inter-, trans-, and cross-cultural contexts. Her publications
include works on sign-making practices in face-to-face interaction, and in digital
environments. She is editor of Visual Communication.
Monica Boria is Affiliated Lecturer in the Department of Italian at the University

of Cambridge and a translator. Her research interests are in contemporary Italian
cultural studies and humour studies. With Linda Risso she co-edited Laboratorio di
nuova ricerca. Investigating Gender, Translation & Culture in Italian Studies (Troubador,
2007). She has translated the first Italian edition of Dorothy Edwards’s short stories
Ammutinamento e altri racconti (ETS, 2019).
Ángeles Carreres is Senior Language Teaching Officer in the Department of Span-

ish and Portuguese at the University of Cambridge. She is the author of Cruzando
límites: la retórica de la traducción en Jacques Derrida (Peter Lang, 2005) and, with María
Noriega-Sánchez and Carme Calduch, of Mundos en palabras: Learning Advanced
Spanish Through Translation (Routledge, 2018).
Klaus Kaindl is Professor of Translation Studies at the University of Vienna, Austria.

His research interests range from the translation of multimodal/multimedial texts
(e.g., opera, comics, popular music), to translation theory, translation sociology, and
fictional representations of translators and interpreters.
Gunther Kress was Professor of Semiotics and Education at the UCL Institute
of Education, University of London. He was interested in communication and
Contributors xi
meaning (-making) in contemporary environments, and was instrumental in develop-

ing a social semiotic theory of meaning-making and (multimodal) communication.
Helen Julia Minors is Associate Professor of Music and Course Leader of degrees
in Music and Creative Music Technology at Kingston University, London. She was
co-investigator for the AHRC project “Translating Music,” and is current co-lead
of the award-winning project “Taking Race Live.”
María Noriega-Sánchez is Senior Language Teaching Officer in Spanish at the

University of Cambridge and a Fellow in Modern Languages at Sidney Sussex
College. She is particularly interested in translation pedagogy and has recently co-
authored with Ángeles Carreres and Carme Calduch the book Mundos en palabras:
Learning Advanced Spanish through Translation (Routledge, 2018).
Tamarin Norwood is an artist and writer. She was recently artist/writer- in-
residence at Spike Island Bristol and was part of Hubbub, the Wellcome Collec-
tion’s inaugural interdisciplinary residency. She gained her doctorate at the Ruskin
School of Art, University of Oxford, in 2018.
Luis Pérez-González is Professor of Translation Studies and Co-director of the

Centre for Translation and Intercultural Studies at the University of Manchester. He
is author of Audiovisual Translation:Theories, Methods and Issues (Routledge 2014) and
editor of The Routledge Handbook of Audiovisual Translation (2018).
Sara Ramos Pinto is Lecturer in Translation Studies at the University of Leeds. She
has an enduring interest in the translation of dialects, but her overall interest lies
in audiovisual and theatre translation, and the challenges that multimodal products
bring to translation theory and practice. Recently, she has approached these issues
from the perspective of experimental reception studies.
Matthew Reynolds is Professor of English and Comparative Criticism at Oxford,

where he chairs the Oxford Comparative Criticism and Translation Research Cen-
tre (OCCT). He is the author of Translation: A Very Short Introduction (2016), Like-
nesses:Translation, Illustration, Interpretation (2013), and The Poetry of Translation: From
Chaucer & Petrarch to Homer & Logue (2011).
Marcus Tomalin is a Fellow at Trinity Hall, Cambridge, and Senior Research Associate
in the Cambridge Machine Intelligence Laboratory. He is interested in the relation-
ships between natural language, mathematics, and philosophy, and his many publications
include Linguistics and the Formal Sciences (CUP, 2006), Romanticism and Linguistic Theory:
William Hazlitt, Language, and Literature (Palgrave, 2009), “And he knew our language”:
Missionary Linguistics on the Pacific Northwest Coast (John Benjamins, 2011), and The
French Language and British Literature, 1756–1830 (Routledge, 2016).
ACKNOWLEDGEMENTS
This book emerged from the workshops and panel discussions organised by the
research group Cambridge Conversations in Translation (CCiT) that was based in the
Centre for Research in the Arts, Social Sciences, and Humanities (CRASSH) at
the University of Cambridge from 2015–2018. We are especially grateful to the
support and encouragement we received from Simon Goldhill and Steve Conner
(successive Directors of CRASSH), Tim Lewens and Jan-Melissa Schramm (succes-
sive Deputy Directors of CRASSH), our Faculty Advisors Tim Crane, Ruth Davis,
Robin Kirkpatrick, James Montgomery, and Rowan Williams, as well as our external
referees Duncan Large and Michael Schmidt. The excellent logistical and technical
support we received from Esther Lamb, Glenn Jobson, Imke van Heerden, and Una
Yeung was essential in ensuring the smooth running of the CCiT events. We would
also like to thank the many speakers who participated in the discussions and work-
shops we organised: Adriana X. Jacobs, Theodor Dunkelgrün, Simone Kotva, Tony
Street, Nicholas King, David Charlston, Danielle Sands, Duncan Large, Georgina
Collins, Hephzibah Israel, Paul Russell, Angel Gurría-Quintana, Orri Tomasson,
Rosa Van Hensbergen, Manuela Perteghella, Hannah Conway, María Mencía, Peter
Robinson, Olivia McCannon, Viviane Carvalho da Annunciação, Jennifer Harris,
Jeremy Munday, Delia Chiaro, Paul Howard, Caroline Summers, Pauline Henry-
Tierney, Jen Calleja, Olga Castro, Daniel Hahn, Gillian Lathey, Maria Nikolajeva,
Anthea Bell, Lucile Desblache, Andrew Jones, Judi Palmer, Rachel Godsill, Helen
Julia Minors, Lucy Taylor, Graeme Ritchie, Carol O’Sullivan, Catherine Boyle, Cris-
tina Marinetti, Carole-Anne Upton, Alfredo Modenessi, Adrià de Gispert, Andrew
Rothwell, Francesca Billiani, Federico Federici, Rory Finnin, Shady Hekmat Nasser,
Youssef Taha, Manuel Portela, Arnaud Regnauld, and Gabriel Tremblay-Gaudette.
Many of the conversations that took place during and after these events refined our
thinking about numerous issues that are considered at greater length in this book.
Acknowledgements xiii
The ensuing chapters were first presented at a conference organised by CCiT

that took place on 5–6 July 2018. We gratefully acknowledge the funding for this
event that we received from CRASSH, the Institute of Modern Languages Research
(IMLR), and the Cross-Language Dynamics: Reshaping Community (CLDRC)
Open World Research Initiative Project (the Translingual Strand).We also benefited
greatly from the administrative expertise of Oliver Wright at CRASSH. After the
conference, the authors were able to revise their chapters for this volume in the
light of the resulting discussions. We thank them for the care with which they
prepared those modified versions. We would also like to express our gratitude to
Routledge for supporting this venture with such conviction from the outset. Louisa
Semlyen, and our editorial assistants Hannah Rowe and Eleni Steck, provided us
with effective guidance throughout the process; and the comments our proposed
monograph received from several anonymous reviewers were invaluable. We would
also like to thank the artist Gurpran Rau for kindly granting us permission to use
one of her works as the image on the cover of this book.
The editors and publishers would like to thank the following copyright holders
for permission to reproduce the following material in this book:
Raoul Dufy, 1877–1953 and Guillaume Apollinaire, 1880–1918, “Le Droma-

daire” in Le Bestiaire ou cortège d’Orphée (Paris: Deplanche, Éditeur d’Art,
1911); woodcut on Holland Zonen wove paper; image: 205 x 195 mm
(8 1/16 x 7 11/16 in.); sheet: 325 x 250 mm (12 13/16 x 9 13/16 in.); The
Fine Arts Museums of San Francisco, gift of the Reva and David Logan
Foundation, 1998.40.38.10; image courtesy of the Fine Arts Museums of San
Francisco.
“Il tramonto” in L’Allegria by Giuseppe Ungaretti (Milan: Mondadori,
1942). © 2015 Mondadori Libri Spa.
While we have done our very best to identify and correct any lingering slips or
blunders, we have surely missed one or two. If so, then nostra culpa.
Gunther Kress passed away, unexpectedly, while this book was in the final stages
of preparation. We would like to express our gratitude for the encouragement and
support he gave to this project from the very beginning, and we hope that the
chapter he contributed is a fitting memorial.
Monica Boria
Ángeles Carreres
María Noriega-Sánchez
Marcus Tomalin
INTRODUCTION
Monica Boria and Marcus Tomalin
1. Translation in a multimodal world

The current Wikipedia entry for translation begins with a confidently succinct defi-
nition: “[t]ranslation is the communication of the meaning of a source-language
text by means of an equivalent target-language text.”1 That all sounds simple
enough, but the simplicity conceals many lurking complexities. What does “mean-
ing” mean in this context? What is a “text” exactly? And when is one text “equiva-
lent” to another? Indeed, the difficulties and uncertainties concerning translation
as a practice tend to proliferate the more one thinks about it, and especially when
the source text contains nuanced intricacies of form and meaning. Here is the first
stanza of François Villon’s famous “Ballade des dames du temps jadis,” which was
one of the poems in his collection Le Testament (1461):
Dictes moy ou, n’en quel paÿs,

Est Flora, la belle Romaine,
Archipiadés, ne Thaÿs,
Qui fut sa cousine germaine,
Echo parlant quand bruyt on maine
Dessus riviere ou sur estan,
Qui beaulté ot trop plus qu’umaine.
Mais ou sont les neiges d’anten?
(Villon 1994 [1461], 74)
If these lines of medieval French function as a source-language text, then their

meaning can be translated into a target-language text, such as this English version
by the Victorian poet and artist Dante Gabriel Rossetti:
Tell me now in what hidden way is

Lady Flora the lovely Roman?
2 Monica Boria and Marcus Tomalin
Where’s Hipparchia, and where is Thais,

Neither of them the fairer woman?
Where is Echo, beheld of no man,
Only heard on river and mere –
She whose beauty was more than human? –
But where are the snows of yester-year?
(Rossetti 1870, 177)
It is immediately apparent that “meaning” (at least in the rudimentary sense of

semantic content) is not the only factor being taken into consideration here.
Rossetti’s translation reveals an attentiveness to the rhyme scheme of his source
material, for example. Following the conventions of Villon’s forme fixe ballade, Ros-
setti ensures that “way is” rhymes with “Thais,” “Roman” just about rhymes with
“woman,” “no man” with “human,” while “mere” rhymes with the final syllable of
the compound “yester-year.” Should these structural characteristics be considered
part of the “meaning” of the original, or can we separate the form from the content
without compunction? If semantic equivalence between the source and target texts
is the primary goal of the translation, then attempts to preserve the rhyme scheme
are likely to thwart that purpose (e.g., a “way” is hardly a “pays,” and “the fairer
woman” has very little to do with “sa cousine germaine”). So maybe a prose transla-
tion would have been more accurate (at least in some sense[s])? And what about the
broader cultural context of the stanza? Is it an innocent act to produce an English
translation of these lines that reads as if it originated in an English-speaking culture
rather than a French-speaking one? Or should that kind of approach be viewed
negatively, as perpetrating cultural hegemony, assimilation, and, possibly, subordina-
tion? Perhaps the translation should seek instead to convey the foreignness of its
source material more overtly? After all, Gallic allusions to figures from Classical
history and myth have distinctive cultural resonances that are not retained when
expressed in English.
And then there is the bewilderingly vast panoply of specific, yet intricately
interconnected, socio-cultural perspectives. Take gender-related considerations,
for example. Villon’s stanza about celebrated female figures refers particularly to
Flora (a Roman courtesan) and Thaÿs (another courtesan, but a Greek one this
time), as well as “Archipiadéa,” who is probably Alcibiades, the male general from
fifth-century BC Athens who was sometimes mistaken for a woman in medi-
eval texts because his beauty had been praised by Boethius in his De consolatione
philosophiae (Boethius 2001 [524], 66). All of these references are subsequently
translated by Rossetti, a male translator working in the late nineteenth century
when traditional notions of gender as a socio-political category were starting to
be re-evaluated (e.g., the National Society for Woman’s Suffrage was formed in
1867). So, how should a translator respond, if at all, to those aspects of the source
text that are inherently gendered? Rossetti, it should be noted, has replaced Villon’s
“Archipiadéa” with “Hipparchia.” Since the latter was undoubtedly a woman, the
wife of the Crates of Thebes, she does not complicate the catalogue by blurring
Introduction 3
binary gender boundaries and creating ambiguities. Should Rossetti’s handling

of this be viewed positively, as the correcting of a regretful error in the source
text, or negatively as a disguising of an intriguing complication? And the various
issues summarised briefly here represent just a tiny sample of the many possible
evaluative and interpretative options that confront any translator.2 Indeed, in 1987
Eliot Weinberger and Octavio Paz showcased the remarkable range and variety of
translation by taking a four-line poem by the eighth-century Chinese poet Wang
Wei and offering nineteen different versions of the text, most in English (ranging
from literal to freer renderings), but some in French, Spanish, German, and mod-
ern Mandarin (Weinberger and Paz 2016 [1987]).
The fact that so many theoretical and practical considerations intersect, inter-
twine, and (not infrequently) conflict virtually guarantees that no single target
text will content everyone, which is precisely why Peter Robinson has described
translation as “the art of the impossible” (2010, title page). And a fascination
with the impossibility of the task helps explain why topics such as those summa-
rised earlier, and countless more, have received extensive attention in translation
studies over many decades now. Yet, in recent years there has been a growing
unease about the boundaries that determine the scope of what has traditionally
been classified as translation. For instance, when Vaslav Nijinsky choreographed
L’Après-midi d’un faune for Sergei Diaghilev’s Ballets Russes company in 1912,
he was inspired both by Stéphane Mallarmé’s 1876 poem of the same name and
by Claude Debussy’s symphonic work Prélude à l’après-midi d’un faune that had
premiered in 1894. This cluster of related artistic creations prompts uneasy ques-
tions. Can Debussy’s musical representation of Mallarmé’s text be called a “trans-
lation”? Does it communicate the “meaning” of its source material, even though
a shift from writing to music has occurred? Is it acceptable to refer to the music
as a “text” and, if so, does that include a performance/recording of the piece, or
only the written score – or all of these? And how should Nijinsky’s ballet be clas-
sified, given that it involves a further transference of meaning from writing and
music to dance? There is currently no consensus about which sorts of meaning
(re)communications can legitimately be classified as instances of translation. And
it is particularly unclear how these questions should be answered in situations
where the forms of communication are unambiguously multimodal (e.g., they
involve more than merely writing).3
These theoretical difficulties may have been introduced here via a discussion
of a ballet from 1912, but the need for a focused consideration of translation in
multimodal contexts is becoming increasingly urgent in the modern world as com-
munication involving words, images, movement, gesture, music, and so on occurs
with ever greater frequency. The first decades of the twenty-first century have wit-
nessed an unparalleled proliferation of interconnected social, economic, cultural, and
technological changes that have already begun to transform the nature of human
communication in discernible ways. Many of these developments are closely linked
to the joint processes of globalisation and technological innovation. The increased
flow of goods, people, ideas, and money across national boundaries has profoundly
altered the composition of the prevailing social and semiotic environments. Cor-
porations (for instance) have increasingly perceived the need to adapt their prod-
ucts, branding, and advertising/marketing strategies to the cultural specificities
of local markets. Consequently, globalisation dynamics have intertwined with
localisation practices, using both verbal and nonverbal resources, to target differ-
ent regional audiences/consumers – and we encounter these strategies daily as we
interact with websites, packaging, billboards, computer games, interactive adverts,
and the like. In addition, the ubiquity of the internet, and the rapid burgeoning
of social media, mobile phones, and other digital technologies, have engendered a
cultural moment in which texts, images, and sounds regularly combine to convey
complex messages. In their different ways, emojis, internet memes, the automatic
captioning of live-streamed online shows, and e-literature are all illustrative exam-
ples of this broad trend.
Nonetheless, as the discussion of Nijinsky’s ballet has already indicated,
none of these things is especially new in kind, though the incidence of them
is undoubtedly increasing. Multimodal objects are easily identified in any cul-
tures from virtually any historical period, whether it is Chinese logograms in
the seventh century BC, or twelfth-century illuminated manuscripts, or illus-
trated children’s books, such as Lewis Carroll’s Alice’s Adventures in Wonderland
(1865). Indeed, certain cultural objects from earlier periods consciously drew
attention to the difficulties that arise when meaningful communications in
one mode have to be re-expressed in a different mode. In Laurence Sterne’s
extraordinary novel The Life and Opinions of Tristram Shandy, Gentleman (1759–
67), the character Uncle Toby and his faithful manservant, Corporal Trim, have
a long conversation in which they discuss bachelorhood and celibacy. At one
point, having stated that “[w]hilst a man is free –,” Trim gives “a flourish with
his stick thus –”:
FIGURE 0.1 The squiggle flourish

Introduction 5
As the eponymous narrator remarks, “[a] thousand of my father’s most subtle syl-
logisms could not have said more for celibacy” (Sterne 1767, 17). The exchange here
playfully alludes to the multimodal nature of the conversation. The physical flourishing
of the stick in the air is a meaning-bearing gesture, and the communicative difficulty
arises when that meaning has to be represented in a non-gestural mode (i.e., writing).
Sterne is wittily highlighting some of the difficulties that occur when novelists seek to
record realistic conversations in detail, since face-to-face human interactions have never
only involved sequences of words. Corporal Trim’s gesture with his stick is a profoundly
meaningful one in the context of the discourse, yet it needs to be seen in order to be
interpreted; and the graphical squiggle that approximates the movement of the flourish
is little more than a rough translation of the actual gesture. This is the kind of com-
municative exchange that Norwood considers in her chapter for this book (Chapter 8),
when she reflects upon the process of writing “drawingly.”
Multimodal literary examples such as this merely reflect the commonality of such
interactions in real life, and the emerging field of multimodality studies has sought
to provide analytical frameworks that facilitate the study of such phenomena. In par-
ticular, the influential work of Gunther Kress, Jeff Bezemer, Carey Jewitt, Theo van
Leeuwen, and others has outlined a social semiotic approach for understanding how
distinct modes such as speech, writing, gesture, image, and sound function as semiotic
resources in order to facilitate the representation and communication of meanings. In
recent years, multimodality has been studied extensively in relation to domains such
as psychology, advertising, social media, storytelling, discourse analysis, literary criti-
cism, and gaming (to name just a few; see Jewitt 2014). Curiously, though, despite its
increasing prevalence, minimal attention has so far been paid specifically to the impact
that multimodal communication is having, and/or is likely to have, upon the theory
and practice of translation. One exception to this is audiovisual translation, which
has been discussed fairly extensively since the late 1990s, with a particular focus on
applications in the media and entertainment sector (e.g., film subtitling; see Pérez-
González 2014a). Indeed, some translation theorists/scholars occasionally give the
impression that multimodality only refers to audiovisual phenomena.
Nonetheless, as the prior discussion has emphasised, audiovisual translation con-
stitutes merely a single manifestation of a much broader cultural shift, one in which
nonverbal modes play far more than simply a minor contextualising role. The neglect
of other multimodal environments in relation to translation is unfortunate. This is
partly because the greater prevalence of these forms of communication and meaning-
making has already begun to destabilise certain time-honoured translation-theoretic
paradigms, causing old definitions and assumptions to seem fusty and inadequate. For
instance, the word translation is currently deployed to denote meaning transfer across
different modes and also within the same mode. Some influential figures, such as Kress,
have argued that these distinct transferences deserve a more fine-grained analytical
vocabulary. Transduction could be used to refer to the moving of meaning from one
mode to another (e.g., a painting based on a poem), for example, while transforma-
tion could denote the transfer of meaning within the same mode (e.g., an English
version of a French poem) (Kress 2010, 124–31). At present, though, there is no

agreement about such terminological details, and the absence of a stable and broadly
accepted critical vocabulary frequently undermines discussions involving theorists
who approach multimodality from different disciplinary backgrounds. Jargon-related
anxieties of this kind do not generally arise in the traditional translation paradigms,
which (as we have seen) usually concern the conversion of a source text (frequently
a literary one) into a target text. A translation theorist might examine in detail, for
example, how Rossetti Englished Villon’s poem – and, as suggested earlier, such an
analysis might focus on domesticating/foreignising tendencies, and/or problems
posed by humour and wordplay, and/or the relationship between the respective sys-
tems of metre and rhyme, the importance of the cultural context, and so on. Since the
1980s there has been a much greater focus on various cultural and sociological aspects
of translation, and issues such as power, ideology, culture, race, and gender have been
extensively considered in texts such as Lawrence Venuti’s The Translator’s Invisibility:
A History of Translation (1995), Gideon Toury’s Descriptive Translation Studies and Beyond
(1995), Luise von Flotow’s Translation and Gender: Translating in the ‘Era of Feminism’
(1997), and Susan Bassnett and André Lefevere’s Constructing Cultures: Essays on Liter-
ary Translation (1998), to quote but a few key works.
More recently still, pioneering scholars such as Martin Heusser, Willard Bohn,
Marta Arnaldi, Helen Julia Minors, and Lucile Desblache have begun to explore
different kinds of text-to-music, text-to-dance, text-to-image, music-to-text,
dance-to-text, and image-to-text as instances of meaning transference (Bohn 2001;
Heusser et al. 2005; Minors 2013; Arnaldi 2016). Although some of these analyses
draw upon well-established traditions of ekphrasis and iconology that can be traced
back through the centuries (e.g., to illuminated medieval manuscripts; or, more
recently, to Gérard Genette’s notion of visual paratexts), others probe in new and
provocative ways the very limits of those activities we may wish to classify as “trans-
lation.” Although this is still an inchoate interdisciplinary field, it has already started
to inspire ground-breaking analytical approaches that richly deserve more probing
scrutiny. Mary Snell-Hornby’s identification of four different genres of multimodal
texts, for instance, still raises at least as many questions as it answers (Snell-Hornby
2009, 44). Other, more recent studies have applied cognitive theories to describe
how the process of translating illustrated texts differs from that involved in translat-
ing a “monomodal” text (e.g., Ketola 2016). However, the outcomes of such studies
have proved inconclusive at best, primarily exposing the urgent need for further
research. This book addresses such issues directly, making them the central focus of
the discussion throughout.
In order to provide some disciplinary context for the main chapters of this
book, it is important to summarise some of the dominant trends both in translation
studies and in multimodality studies, since, if these two domains are to interact in a
more effective manner in the future, then some mutual comprehension and famili-
arity is necessary.We will begin with an overview of the development of translation
studies, focusing in particular on recent trends, before offering a summary of the
emergence of multimodality as an academic discipline.
Introduction 7
2. Trends in translation studies

Since its emergence in Europe during the 1970s, the field of translation studies has
grown exponentially and internationally, and its development as an academic disci-
pline has been charted numerous times (e.g., Baker 2001; Munday 2010; Lambert
2013; Bassnett 2014). Therefore, only a brief outline of its history will be summa-
rised here, to give readers unfamiliar with translation studies some understanding of
how and why this field has developed, to supply key references for further reading
about the different schools and approaches, and to intrigue those who wish to delve
deeper. The main focus will fall on some of the central controversies and on the
most influential research traditions that have prepared the ground for contemporary
multimodal approaches.
Theorising about translation has a long and distinguished history. In the West, it
is usually traced back to Cicero and Horace (first century BC). St Jerome (fourth
century) and Martin Luther (sixteenth century) are regarded as pivotal figures, and
the German Romantics, especially Schleiermacher, later exerted a powerful influ-
ence (Munday 2010, 421–423). However, a formal academic discipline devoted
to the study of translation only became institutionalised in the twentieth century.
Its development was prompted by several factors, including momentous historical
events (e.g., the reliance on translation at the Nürnberg trials), Cold War advances
in computer technology (e.g., machine translation), and the emergence of lin-
guistics as a promising new “scientific” field of academic enquiry. In an effort to
map linguistic transfer, the focus was initially placed on written language and on
notions of equivalence between source and target texts. Roman Jakobson’s early
taxonomy of translation as interlingual, intralingual, and intersemiotic (1959) was
highly influential (see Kress’s and Kaindl’s chapters in this book for further discus-
sion), but other linguists, such as George Mounin in France and Eugene Nida in
the United States, were also pioneers of the field. Mounin is widely regarded as the
founder of traductologie, the discipline devoted to the study of translation. He was
taught by the French structuralist linguist André Martinet, and his work Les belles
infidèles (1955) is widely regarded as a founding manifesto of the discipline. Nida, in
turn, drew upon Chomsky’s theory of generative grammar, and introduced a dis-
tinction between “formal” and “dynamic” equivalence in translation. The former
refers to a rendering that prioritises the form and linguistic structure of the source
text, whereas the latter attempts to produce a target text that achieves an equiva-
lent effect on the intended audience (Nida 1964). As these examples indicate, the
importance of linguistics as a shaping force of the nascent discipline cannot be
underestimated. Indeed, until the late 1960s translation studies was generally seen
“as a branch of applied linguistics” (Baker 2001, 279), even though it had become
clear over the years that notions of linguistic equivalence were ill-suited to an activ-
ity that involved much more than language (in a narrow sense).
Consequently, translation studies became established in the 1970s, and its aca-
demic scope opened up to other fields, such as literary theory and compara-
tive literature. The so-called Leuven group of international scholars argued that
translation should be recognised as a field in its own right. Their excitement at

the increasing relevance of translation worldwide was matched by their frustra-
tion at its marginality in academia. James Holmes introduced the term “transla-
tion studies” in 1972, and at a 1976 conference in Leuven a manifesto was drawn
up to announce the remit of the new field which, in the words of another par-
ticipant, “concerns itself with the problems raised by the production and descrip-
tion of translations” (Lefevere 1978, 234). The new discipline aimed to study
translation – the activity and its products – in all its forms, to understand its
underlying “norms,” and possibly establish some general principles, which would
inform practical applications, for example in translator training, translation criti-
cism, and the like (Holmes 1988 [1972]).4 It was seen as being an inclusive field
with a natural tendency towards interdisciplinarity, which involved not only
scholars from different subjects but also professional translators. The hope was
to foster a research culture where theory and practice would speak to each other
and to the outside world.
The success of this new academic discipline in the following two decades was
largely due to the shift from linguistic notions of “equivalence” and “transference,” or
ideas of “ faithfulness” to the source text, towards an analysis of the socio-historical
context in which translation takes place, the political and ethical issues that impinge
on it, and the role (and status) of the translator (Berman and Porter 2014, 3). Several
key scholars contributed to these developments, both from a theoretical perspective
and by taking practical measures to establish the field within academia and beyond.
Susan Bassnett’s volume Translation Studies, published in 1980 (and now in its fourth
edition), provided a fascinating and accessible overview of the important role of
translation in Western literary and cultural history (Bassnett 2002). In outlining the
various theories that informed its practice, Bassnett brilliantly demonstrated the
rightful claim of the discipline to its place as an autonomous field of study. She has
since been a strong advocate of the need for theory and practice to be in constant
dialogue, and for a shift away from narrowly language-based approaches towards
broader culture-based translation strategies. As Bassnett and Lefevere put it, “trans-
lation, like all (re)writings is never innocent. There is always a context in which the
translation takes place, always a history from which a text emerges and into which
a text is transposed” (Bassnett and Lefevere 1990, 11). Important contributions
to the so-called “cultural turn” in translation studies in the 1980s came from the
Manipulation Group, with Itamar Even-Zohar, Gideon Toury, José Lambert, and
Theo Hermans. These scholars came from very different linguistic and academic
backgrounds, and their theories embedded the study of translation in its socio-
cultural context, presenting it as a phenomenon that influences the literary system
it enters. Hermans has summed up some of their central ideas concisely:
a view of literature as a complex and dynamic system; a conviction that there

should be a continuous interplay between theoretical models and practical
case studies; an approach to literary translation which is descriptive, target-
oriented, functional and systemic; an interest in the norms and constraints that
Introduction 9
govern the production and reception of translation, in the relation between

translation and other types of text processing, and in the place and role of
translations both within a given literature and in the interaction between
literatures.
(Hermans 1985, 10–11)
Prompted by similar interests, Mary Snell-Hornby theorised an integrated approach

involving linguistics and literary studies, dismantling notions of equivalence as illu-
sory and misplaced, and advocating the importance of the translator’s cultural com-
petence alongside their linguistic skills (Snell-Hornby 1988).
As this suggests, translation studies was partly the outcome of those winds of
change that blew through universities in Europe and North America from the late
1960s onwards, and this partly explains why it has so frequently sought to “chal-
lenge orthodoxies” (Bassnett 2014, 25). The restructuring of higher education pro-
vision that followed those years of protests undoubtedly offered opportunities for
new subjects to emerge, especially within the humanities and the newly established
social sciences. The process was, of course, different in different countries, and
despite the international make-up and vocation of translation studies, the extent of
the influence exerted by post-structuralism, feminism, anthropology, postcolonial
approaches to literature, and semiotics, varied considerably. The fact that transla-
tion, along with modern languages, had traditionally occupied an ancillary position
within the Anglo-American context placed translation studies at the margins of
academia. It had to fight against ingrained prejudice to assert itself, and this lent the
discipline an anti-establishment air that set it apart from its European counterparts
(Munday 2010; Bassnett 2014, 26). Lawrence Venuti (1995, 17) was one of the
earliest and most vocal scholars and translators to expose Anglo-American ethno-
centrism and its conservative attitudes towards translation. In discussing “domesti-
cating” practices that favour readability, obscure the inherent alterity of texts, and
undermine the role of the translator, he exposed the insular, imperialistic culture
of the United States, where translations (to date) barely reach 5% of publications.
In the UK the situation is not dissimilar, with translations totalling just over 3% of
publications, though recent statistics suggest that percentage is starting to increase
(Anderson 2019). Foreign literature in translation is often segregated from English
literature, frequently confined to different shelves in bookstores, unlike other coun-
tries (such as France or Italy) where Balzac, Calvino, and Dickens can usually be
found in the same sections. Unsurprisingly perhaps, in France one in nine books is
a translation, and the ratio is even higher in Italy, especially in children’s literature,
where translations represent over 40% of the total titles for young and young-adult
readers (ISTAT 2018; Armitstead 2019). European (and bilingual Canadian) culture
has traditionally been much more open towards translation, within the publishing
industry and in society at large, with a long tradition of institutional investment in
translators’ training and research (Lambert 2013, 9).
Due to its hybrid nature as both an applied and a speculative subject, as well as its
interconnections with other disciplines and schools of thought, translation studies
has been able to respond more effectively to the challenges and opportunities of
globalisation than many other disciplines – and it remains a vital academic domain.
The vast number of available journals, national and international symposia, associa-
tions, federations (e.g., CETRA, IATIS, EST, ATISA), research centres, masters and
doctoral programmes, and so on, all testify to the success of this “interdiscipline”
(Snell-Hornby 2006) that has explored its subject matter from such a wide range of
theoretical perspectives. From postcolonial contexts to gender and sexuality studies,
translating and translations have been scrutinised extensively to reveal the power
imbalances, the economic and political inequalities, the gender hierarchies, and/
or heteronormative biases that have influenced cultural and linguistic representa-
tions. As the study of translation has shifted from the search for an ideal linguistic
equivalence to a pervasive focus on difference, literary writing and translation have
increasingly been caught in a complex tangle of power asymmetries. From differ-
ences based on language (global/local; national/regional) to those pertaining to
culture (dominant/marginal; Eastern/Western), including issues of gender politics,
the role of writers and translators in the construction of meaning and identity has
attracted much scholarly attention. To take just two prominent examples, seminal
work in feminist translation was pioneered by the Canadian scholars and translators
Luise von Flotow and Sherry Simon (von Flotow 1991, 1997; Simon 1996), while
research into the strategies of postcolonial translation provided the focus for Tejas-
wini Niranjana, Harish Trivedi, and Susan Bassnett (Niranjana 1992; Trivedi and
Bassnett 1999). Studies such as these have teased out the implications of hegem-
onic discourses on the creation and circulation of literature, and have broadened
the scope of the debate to include wider linguistic contexts and other disciplinary
angles.
The expansion of media and cultural studies since the late 1990s afforded
translation studies unprecedented opportunities, and, as mentioned earlier, audio-
visual translation (AVT) has emerged as the primary focus for multimodal studies
of translation (Díaz Cintas 2009, 3). The field has increasingly attracted research-
ers from a large spectrum of academic backgrounds and countries (Dror Abend-
David 2014). This poses the challenge of engaging with subject-specific jargon
and research methods, not to mention different academic traditions, and such
complexities are usually a sign of genuine interdisciplinarity. Audiovisual transla-
tion’s increased prominence as a conceptual and aesthetic framework is testified
by recent innovative studies such as Carol O’Sullivan and Jean-François Cornu’s
The Translation of Films, 1900–1950 (2019), which argues that translation has made
a crucial contribution not only to the circulation of films but also to the art of
cinema more generally. O’Sullivan and Cornu make their case by exploring the
complexities of editing for international distribution, the aesthetic considerations
of intertitles and subtitles, issues of censorship, and many other factors. In addi-
tion to these research areas, recent developments in digital technologies and the
emergence of user participation, crowdsourcing, fansubbing, and other forms of
amateur translation are providing audiovisual translation, and translation studies as
a whole, with new sets of ethical and methodological questions (see the chapters
Introduction 11
by Luis Pérez-González, and Elisabetta Adami and Sara Ramos Pinto in this
book). Undoubtedly, the choice of English as the discipline’s lingua franca has been
one of the contributing factors to its international reach and success. However,
from the 1990s onwards, this field has widened its horizons beyond European and
American perspectives. An increasing number of edited collections, anthologies,
and readers have started to explore more diverse traditions of translation theory
and practice, enriching the discipline with new visions and idioms, notably from
East Asia, and perhaps exposing its own hegemonic bias (Bermann and Porter
2014, 6). Some scholars have even mooted a “post-translation studies” agenda, to
move beyond disciplinary boundaries towards wider transdisciplinary discourses
on the translational nature of societies and individual identities (Arduini and
Nergaard 2011, 8).
Despite these developments, translation studies’ original disciplinary fields –
linguistic and literary studies – have continued to influence translation-
based
research, and, in different ways, the chapters in this volume by Matthew Reynolds
and Marcus Tomalin form part of that time-honoured tradition.Yet it is undeniable
that in our increasingly intercultural, multilingual, and transnational world, transla-
tion as a discipline has truly come of age. The so-called “translation turn” of the
late 1990s saw other associated disciplines focus their attention on the theory and
practice of translation, exploring its methods, and acknowledging its importance
not only as a subject in its own right but also as the new paradigm that shapes our
contemporary world. And that world is undoubtedly a multimodal one.
3. The rise of multimodality

In some respects, the development of multimodality studies as a free-standing dis-
cipline is a similar story. Over the past twenty years or so, it emerged from a cluster
of interrelated interdisciplinary research domains which recognised that represen-
tation and communication involve far more than merely linguistic exchanges. As
mentioned earlier, interest in this topic has arisen partly in response to the per-
vasiveness of globalisation and the burgeoning of digital media and technological
innovation (e.g., Facebook, Twitter). It has already become a platitude to state that,
in the modern world, communication frequently involves more than one mode.
Internet memes combine writing and images, while vlog posts frequently involve
speech, text, moving images, and music, all of which are coordinated to constitute
and convey the meaning of the whole. In response to this, the term multimodality has
been used to cover a wide range of approaches associated with the arts, the humani-
ties, the social sciences, as well as engineering and artificial intelligence, and several
different theoretical frameworks have been proposed. For instance, the centrality of
multimodal communication was explored extensively during the 1990s by those
seeking to create more effective human-computer interfaces, while in a parallel tra-
dition that drew upon Michael Halliday’s Systemic-Functional Grammar, theorists
like Anthony Baldry and Paul J. Thibault were outlining multimodal approaches
to the analysis of texts (Bunt, Beun, and Borghuis 1998; Baldry and Thibault 2006).
Having considered the various research traditions, Carey Jewitt, Jeff Bezemer, and
Kay O’Halloran recently identified three core premises that characterise multi-
modal research in general (Jewitt, Bezemer, and O’Halloran 2016, 3):
1 Meaning is made with different semiotic resources, each offering distinct

potentialities and limitations.
2 Meaning making involves the production of multimodal wholes.
3 If we want to study meaning, we need to attend to all semiotic resources being
used to make a complete whole.
Unsurprisingly, these premises come with several caveats, some of which are due to
different terminological traditions and conventions. Not all multimodality theorists
focus on “meaning making” overtly, for instance. Some claim that they are primarily
interested in “multimodal discourse” or “multimodal communication,” while oth-
ers deploy phrases such as “semiotic resource” rather than the term “mode.” While
these variations reveal contrasting methodological and ideological allegiances, they
can sometimes unintentionally distract from the underlying shared interest in forms
of communication that involve more than merely language. Nonetheless, the field
has acquired sufficient coherence that specialist journals have started to appear (e.g.,
Multimodal Communication was established in 2011) as well as devoted research series
(e.g., the Routledge Series in Multimodality Studies), and these provide prominent
fora for cutting-edge research into multimodality.
It is worth considering one particularly influential theory as a specific case
study here – namely, Multimodal Social Semiotics (MSS). Although the origins of
MSS can be traced back to the 1980s, it has already exerted a significant influence
upon numerous academic fields. In essence, it is a branch of semiotics that seeks to
understand how people communicate in specific social settings. Like several other
theories of multimodality, it has deep roots in the work of Michael Halliday, and
especially his monographs Language as Social Semiotic (1978) and An Introduction to
Functional Grammar (1985). Halliday emphasised repeatedly that language should be
interpreted “within a sociocultural context, in which the culture itself is interpreted
in semiotic terms” (Halliday 1978, 2). Therefore, semiosis is not something that
occurs merely in minds; it arises from social practices in a community. To facilitate
his analysis, Halliday identified three related variables: field, tenor, and mode. Field
refers to the subject matter or content of what is being discussed, thereby direct-
ing the focus towards what is happening, when it is happening, and to whom it
is happening. By contrast, tenor denotes the social relation that exists between the
participants in the interaction (e.g., father/daughter) and their purposes. It can
therefore influence the communicative choices made by the interlocutors (e.g., the
selection of a specific stylistic register). Finally, the term mode is used to refer to the
channel of communication being used (e.g., speech, writing). These three variables
combine to characterise any kind of situation in which meaning is communicated.
Obviously, if more than one mode is being used simultaneously, then the interac-
tion is (by definition) multimodal.
Introduction 13
The Hallidayan framework that underlies MSS was sketched out in Rob-
ert Hodge and Gunther Kress’s Social Semiotics (1988), and it has been gradu-
ally developed and refined in many subsequent publications, such as Kress and
Theo van Leeuwen’s Reading Images (1996) and Multimodal Discourse (2001), van
Leeuwen’s Introducing Social Semiotics (2005), Jewitt’s The Routledge Handbook of
Multimodal Analysis (2009; 2nd ed. 2013), Kress’s Multimodality (2010), Bezemer
and Kress’s Multimodality, Learning, and Communication (2015), and Jewitt, Beze-
mer, and O’Halloran’s Introducing Multimodality (2016), to name just a few. The
fundamental analytical framework that MSS advocates necessarily draws upon
aspects of older semiotic theories, but it reconfigures them anew. For instance,
its insistent emphasis on the making and sharing of meaning enables it to be
distinguished in various ways from most varieties of semiotics that trace their
origins back to the work of Charles Sanders Pierce and/or Ferdinand de Saus-
sure. MSS does not make use of Pierce’s tripartite classification of signs into
the sub-g roups iconic, indexical, and symbolic, for example, and unlike Saussurian
semiology, it foregrounds the sign-maker’s interest in the making of signs and
meaning (using modes as semiotic resources), which leads to the striking conclu-
sion that the relationship between form and meaning is never entirely arbitrary
(Kress 2010, Chapter 4). As Kress and others have emphasised insistently, one of
the underlying convictions that has guided MSS research from the very begin-
ning is that multimodal interaction has always been the “normal state of human
communication” – but, arguably, that claim is truer of the modern world than
it has ever been (Kress and van Leeuwen 2001, 21). The different modes can
be viewed as “semiotic resources which allow the simultaneous realisation of
discourses and types of (inter)action,” and they have different affordances – that is,
particular potentialities and constraints that impact the making of signs in rep-
resentations (Kress and van Leeuwen 2001, 21). This is perhaps best explained
by means of an example. A still image, such as a nineteenth-century oil paint-
ing, uses the logic of space (rather than that of time), and its affordances include
things like the materials of the surface the image is created on (e.g., a canvas), the
kind of paint deployed (e.g., oils), the manner in which the paint is applied (e.g.,
impasto), and so on. These specific affordances are not shared by other modes
such as music or speech. Each set of modal affordances is determined by the ways
in which meaning is created with particular semiotic resources, and the latter are
inevitably characterised by their material, cultural, social, and historical develop-
ment. In many respects, there is nothing especially new in all of this. It had long
been recognised that (say) different art-forms functioned in different ways, and
the problematical word ekphrasis (from the Greek roots for “out” and “to speak”)
is still frequently used to refer to situations in which a physical object is brought
vividly to life for a reader/listener by being described in extensive detail using
either speech or writing. However, MSS offers a robust investigative framework
that facilitates the analysis of such phenomena. In particular, it offers a way of
conceptualising ensembles of modes – that is, representations that use more than
one mode in a purposeful manner, to create collective and interrelated meanings.
Therefore, ensembles are said to be “orchestrated” by at least one meaning-maker

(Kress 2010, 157–58), and an attentive consideration of the orchestration pro-
cesses emphasises the extent to which monomodal communication is the excep-
tion rather than the norm. Recognising this fact profoundly destabilises many of
our most familiar logocentric presuppositions about communication (generally)
and about translation (specifically).
Inevitably, one consequence of a social semiotic theory of multimodality is that
language is necessarily displaced from a position of centrality in the analytical
framework – just as it has been in many different sub-branches of translation
studies. More specifically, it loses its privileged status as the primary agent of
meaning-making, and it becomes merely one of many possible ways of creating
and communicating meanings. Indeed, for Kress, language does not even merit
being classified as a “mode” in its own right, since it is an abstraction that dis-
guises the very real differences that distinguish speech and writing as semiotic
resources. As he puts it,
collapsing speech and writing with their entirely different materiality into one
category, thereby joining and blurring over the distinct logics of time and
space, of sequence and simultaneity, exposes the implausibility of a mode
called “language.” It is difficult to see what principles of coherence might
serve to unify all these features. So I take speech and writing to be distinct
modes.
(Kress 2010, 86)
This observation raises several important issues. When it is recognised, for

instance, that various modes (of which speech and writing are but two) can all
be used as semiotic resources to make and convey meanings, then it is necessary
to reflect upon the vocabulary conventionally deployed to refer to different ways
of carrying meaning from one mode (or genre, or ideological complex, and so
on) to another. Traditionally, of course, the term translation has been reserved for
this purpose: a poem originally written in (say) French is translated into (say)
English. In the Western tradition (at least), the conventional terminologies used
to denote this process often imply some kind of moving or carrying across of
the meaning. The etymology of translation reveals a compounding of two Latin
roots: “trans” (i.e., across, over) + “latus” (i.e., borne, carried) – and the same
is true of words such as traduire (French; from the Latin “trans” + “duco” [i.e.,
to lead]) and übersetzen (German: “über” [i.e., over] + “setzen” [i.e., to put]; so
with the compound sense of “to pass or cross over”). Although this vocabulary
generally refers to situations in which the meaning is moved from one mode to
the same mode (i.e., from written French into written English, as in the Villon-
Rossetti example discussed earlier), translators do, in practice, routinely deal
with linguistic and non-linguistic elements (e.g., lexical and sentential mean-
ing, metre, rhyme scheme, page layout, font type). Be that as it may, the non-
linguistic elements become even more apparent when the meaning-material is
Introduction 15
moved across modal boundaries. For example, compare Giuseppe Ungaretti’s

1916 poem “Tramonto” (“Sunset”) with a digital image inspired by the text that
was created in 2017:5
Tramonto
Il carnato del cielo
sveglia oasi
al nomade d’amore
FIGURE 0.2 Tramonto
It is not possible (other than metaphorically) to analyse the image in terms of

its lexis, syntax, morphology, and so on (though some images may contain some
words and/or numbers), since the affordances of writing and image are so different.
Nonetheless, we might still wish to classify this carrying over of meaning as con-
stituting an instance of “translation,” even though it involves more than one mode.
Certainly, depending on our “reading” of the image, some elements seem to be
quite closely associated with the desert scene described by the text: a sunset, palm
trees – perhaps even a nomad in a tent?
As mentioned earlier, Kress has suggested that the high-level term translation is
too vague to be helpful in cases such as these and should be divided into distinct
sub-types. Therefore, the digital image inspired by “Tramonto” could be classified
as a transduction of the poem-text, since the meaning-material has been moved from
one mode to another – i.e., from writing to image (Kress 2010, 124–31). By con-
trast, Rossetti’s English version of Villon’s poem would constitute a transformation,
since the meaning-material has been moved from one mode into the same mode.
In cases like the latter, a change of culture is involved (i.e., a French cultural context
is replaced by an English cultural context), even though the mode remains the same,
and this suggests that terms such as “intracultural” and “intercultural” are also perti-
nent (Davies 2012). There are, of course, many obvious differences between writ-
ten French and English (e.g., syntax, vocabulary, alphabet, diacritics) but, in the MSS
framework, the mode remains constant nonetheless. Although most multimodality
theorists recognise the importance of being more precise about these distinctions,
they do not always agree about the preferred terminology. Klaus Kaindl favours the
terms “intermodal” and “intramodal” translation respectively, though the nature
of the distinction is essentially the same (Kaindl 2013, 261–62). It is important to
note, though, that the words used to sub-classify translation always reveal something
about the underlying metaphors and conceptual models that guide our thinking
about the essential activity. As mentioned previously, terms that use the Latinate
“trans” root all imply movement of one kind or another, and caution is required to
ensure that this does not restrict the way we think about the underlying processes
involved. Other languages offer different ways of understanding what we do when
we translate. In Finnish, for example, the verb kääntää means “to translate,” but its
root meaning is “to turn.” So, rather than being carried across a barrier, or from one
place to another, translations in Finnish turn something over, to reveal another side
(Chesterman 1997, 60–2). While there is still some kind of movement here, it does
not involve transportation or relocation. This topic is discussed in greater detail by
Kress in his chapter in this book (pp. 30–2).
Distinguishing these different sub-types of translation, and reflecting upon how
meaning is communicated, compels us to look at familiar processes and classifica-
tions afresh. For example, if speech and writing are distinct modes (as Kress argues),
rather than being merely distinct manifestations of the same mode (i.e., language),
then the carrying, or turning over, of meaning from one mode to another should
not be handled in a privileged manner. To revisit an example considered earlier, if
Stéphane Mallarmé’s written poem L’après-midi d’un faune is read aloud in French,
then the meaning is moved, or turned, from the mode of writing to the mode of
speech, and therefore the very act of speaking the text must presumably be classi-
fied as an instance of intermodal translation (i.e., transduction). This would place
it in the same category as a performance of Debussy’s symphonic work Prélude à
l’après-midi d’un faune, since the latter is a musical re-telling of Mallarmé’s written
text. The fact that these two instances – the reading aloud of the poem and the per-
formance of a composition based on the poem – fall into the same basic category
should cause us to pause and reflect, since this classification destabilises most tradi-
tional analyses of translation. Despite the conspicuous differences between speech
and writing that Kress accurately identifies, it remains true that there are more
direct mappings between certain affordances associated with these two modes (e.g.,
the written and spoken forms of the same noun; the written and spoken forms of
the same sentence) than exist between the affordances of writing and music (e.g.,
is there a direct correspondence of any kind between a specific noun or sentence
in the written text and particular chords, phrases, modulations, cadences in the
music?). Seemingly, some modes facilitate the intermodal communication of mean-
ing more effectively than others, which in turn implies that it should be possible to
elaborate an ontology of the various modes that specifies the nature of the relation-
ships between them, as Tomalin suggests in his chapter for this book (pp. 138–42).
Despite all of this, translation theorists have only really begun to engage with
multimodality (whether MSS or other varieties) as an academic discipline.While, as
mentioned previously, there is a fairly extensive literature on the topic of audiovis-
ual translation, there have been very few attempts to probe the broader implications
of multimodality for translation studies as a heterogeneous whole. Klaus Kaindl is
one prominent figure who has recognised the need for such an engagement. He
has written at length about the multimodal aspects of translating comics (a topic
subsequently discussed by Federico Zanettin, Michał Borodo, and others), and he
also contributed an important chapter on multimodality and translation to the
Introduction 17
Routledge Handbook of Translation Studies (2013), which sketches some of the intel-
lectual background while also identifying significant research areas (Kaindl 2004,
2013; Zanettin 2008; Borodo 2015). Tellingly, he concludes that chapter by remark-
ing that “translation studies has to develop appropriate investigation instruments
for non-language modes” (Kaindl 2013, 266). Another chapter-length summary
of multimodal translation, this time by Elisabetta Adami, appeared in The Oxford
Handbook of Language and Society (2016), while, more recently, Sara Dicerto (2018)
has attempted to elaborate a framework for analysing translation from a multimodal
perspective, which places particular emphasis on the often unacknowledged role
that pragmatics plays in such communicative situations. The burgeoning interest in
these topics is also becoming increasingly manifest in academic journals. In 2013,
Carol O’Sullivan and Caterina Jeffcote edited a special edition of the Journal of Spe-
cialised Translation devoted to the topic of “Translating Multimodalities,” while a few
years later Catalina Jiménez Hurtado, Tiina Tuominen, and Anne Ketola edited a
volume of Linguistica Antverpiensia which explored “Methods for the Study of Mul-
timodality in Translation” (Jiménez Hurtado, Tuominen, and Ketola 2018). This
all suggests that Luis Pérez-González was quite correct when he remarked a few
years ago that “[m]ultimodality is bound to become even more central to transla-
tion scholarship in future years” (Pérez-González 2014b, 127). Several subsequent
studies have focused on the multimodal implications of translating advertisements,
illustrated technical texts, and the front covers of books (Pan 2015; Ketola 2016; Li,
Li, and Miao 2018). As these examples suggest, the convergence between transla-
tion studies ad multimodality studies is already underway, and this volume overtly
seeks to encourage and facilitate more extensive interactions.
4. An overview of this book

The previous sections have summarised some of the key developments in both
translation studies and multimodality studies that have characterised those respec-
tive disciplines during the last twenty years or so. They have also indicated some
of the ways in which these two important research fields have started to merge
and mingle. This book seeks to foster and facilitate such connections by offering
insights into cutting-edge research trends and by identifying specific topics that
merit more extensive consideration in the future. As one of the first book-length
studies of translation and multimodality, it provides an unprecedented opportunity
to examine some of the distinctive practical and theoretical challenges that confront
those who create and/or consume translations in multimodal environments.
In Chapter 1, Gunther Kress offers an authoritative view of translation from
the perspective of MSS. Starting with a game in which people produce written
descriptions of images that subsequently prompt the (re)creation of the original
images, he outlines a framework for thinking about translation as a “transpos-
ing” of meaning. The discussion seeks to offer a preliminary account of particular
site-specific mode-resources for making meaning and of the site-specific mode-
resources for transposing meaning. Similar themes are addressed by Klaus Kaindl
in his chapter, but from a slightly different analytical perspective – one in which
multimodality is viewed as a form of academic “rock ‘n’ roll.” After sketching in
some of the intellectual background that associates contemporary approaches to
multimodality with semiotic theories of “multimediality” from the 1970s, the focus
shifts to exploring the way in which modes are unavoidably intertwined with both
medium and genre. The core insight is that mode, medium, and genre are three
building blocks that form the basis for a translation-theoretical approach that serves
to overcome the language-centeredness of translation studies, and which enables
us to view translation as a modal, medial, and generic practice. As a specific case
study, Kaindl considers the reworking of Elvis Presley’s song Hound Dog (1956) in
the German Schlager popular-music genre, and the detailed analysis shows how the
German version is characterised by changes that were made at the generic, medial,
and modal levels.
Building on some of the topics raised by Kress and Kaindl, Elisabetta Adami
and Sara Ramos Pinto present an ambitious joint semiotic and translation research
programme that seeks to outline ways in which both multimodality studies and
translation studies can share their respective domains of expertise to address a
series of disciplinary gaps that can only be effectively filled through a transdisci-
plinary enterprise. In essence, they encourage us to develop a deeper understand-
ing of meaning-making practices in a changed, increasingly transnational, social
semiotic landscape. The kind of research they envisage requires methodological
developments and disciplinary integration to facilitate the exploration of such
things as the circulation of multimodal products, the practices of multimodal
text production and translation, and the practices of (re)interpretation and re-
signification. In a similar spirit, Luis Pérez-González’s chapter concentrates on the
evolving role of subtitling in the modern media landscape, specifically charting
how this form of translation was gradually freed from the constraints of the film
industry and came to serve more democratic forms of media “prosumption” asso-
ciated with digital culture (e.g., Chinese danmu). In particular, Pérez-González
traces the shift from an ontology of referentiality, based on the hegemonic nar-
rational regime, to the ontology of deconstruction that characterises the digital
media ecology. He goes on to argue that, where ordinary rhetors have gained
greater visibility and agency, their performance of citizenship involves the decon-
struction of representation by exposing the cultural and social make-up of mul-
timodal ensembles.
Adopting a markedly different approach, Matthew Reynolds situates his chapter
in the long tradition of literary translation by taking a specific work of literature
as his starting point – namely, Dante’s Divina Commedia (1320). In particular, he
focuses mainly on one short passage from Paradiso (canto 26, line 134) in which a
vertical (or slightly slanting) line is presented as the original name for God. This
small detail raises non-trivial problems for translators. Does the symbol indicate
the vowel “I” or the numeral “1,” or is it something else entirely – something non-
linguistic perhaps? The use of this vertical line also problematises the (multi)modal
character of the poem, since the image appears as part of the written poem, and
Introduction 19
therefore it is not clear how it should be pronounced. These translation-related

uncertainties prompt a probing consideration of the classificatory problems that
multimodal frameworks present. For instance, if we are unable to delineate a solid
boundary around what constitutes a mode such as “speech” or “writing,” then
we will inevitably struggle to define what modes actually are – and that in turn
potentially destabilises some of the analytical presuppositions frequently adopted
in multimodal studies. Similar concerns are examined in Marcus Tomalin’s chapter,
which concentrates on the translation difficulties posed by livres d’artistes – that is,
publications which purposefully combine texts and images to create multimodal
ensembles. The publication Le Bestiaire ou Cortège d’Orphée (1911) resulted from a
collaboration involving the poet Guillaume Apollinaire and the French Fauvist art-
ist Raoul Dufy, and Tomalin takes one “poem” from this text, “Le Dromadaire,” as a
starting point. By analysing both the poem-text and the associated woodcut image,
Tomalin introduces a (quasi-)mathematical formalism that specifies more precisely
the way in which multimodal ensembles are transformed during various processes
that can be referred to as instances of “translation.”
The final two chapters in the book relate some of these topics to other kinds of
multimodal practices. More specifically, Helen Julia Minors explores the roles and
processes of translation in relation to twentieth-and twenty-first-century music-
dance works. The particular examples considered include Debussy’s Prélude à
l’après-midi d’un faune and the Nijinsky ballet the music inspired, the controver-
sial Stravinsky-Nijinsky collaboration Le sacre du printemps, and Erik Satie’s La
Parade. The discussion of these works illustrates the significance of understanding
the transfer of “meaning” between these arts for those actively involved in the
creative process, as well as for those who subsequently interpret and analyse the
multimodal ensembles produced. The discussion of these issues raises numerous
questions regarding how meaning is produced within and between these tem-
poral gestural arts, and it is argued that the interplay of musical movement and
danced movement relies on various forms of translation, which require cognitive
mapping, gestural interpretation, and an awareness of somatic experience. By
contrast with this emphasis on music and dance, Tamarin Norwood concentrates
on how a novel method of writing can be derived from a particular technique
of “half-blind” drawing. In her experimental chapter, she shows how this process
of derivation can be understood as a form of multimodal translation in which
a drawing method (i.e., the original “text” being translated) is extracted from
its original mode and reconstituted in the mode of writing, thereby offering a
method of writing “drawingly.”
This brief summary hopefully gives some idea of the range and scope of the
issues this book addresses. In all cases, the discussions presented here are not the last
word on any of the topics considered – quite the opposite, in fact. It is hoped that
the core ideas and approaches outlined in the following chapters prompt others
to reflect upon the relationship between translation and multimodality more pro-
foundly and extensively. If that occurs, then this book will have more than served
its purpose.
Notes
1 https://en.wikipedia.org/wiki/Translation (last accessed on 2 September 2018).
2 The theoretical issues briefly alluded to here are all addressed in classic works gathered
together in collections such as Schulte and Biguenet 1992; Weissbort and Eysteinsson
2006; Venuti 2012.
3 Some of these issues are addressed further by Helen Julia Minors in her chapter for this
book (pp. 166–170), where she discusses L’Après-midi d’un faune specifically.
4 For a discussion of Holmes’s project and the way it influenced the development of the
discipline, see Malmkjær (2013).
5 The poem can also be translated as follows: “The flesh-coloured sky awakens oases in
the nomad of love.” The image was created by Monica Boria and Marcus Tomalin at the
Translation and Multimodality conference that took place at the Centre for Research in the
Arts, Social Sciences, and Humanities, at the University of Cambridge, on 26 May 2017.
References
Abend-David, Dror, ed. 2014. Media and Translation: An Interdisciplinary Approach. New York
and London: Bloomsbury.
Adami, Elisabetta. 2016. “Multimodality.” In The Oxford Handbook of Language and Society,
edited by Ofelia García, Nelson Flores and Massimiliano Spotti, 451–72. Oxford: Oxford
University Press.
Anderson, Porter. 2019. “Nielson Reports Translated Literature in the UK Grew 5.5 Per-
cent in 2018.” Publishing Perspectives, March 6. Accessed April 20, 2019. https://
publishingperspectives.com/2019/03/nielsen-reports-t ranslated-l iterature-i n-u k-
grows-5-percent-in-2018-booker/.
Arduini, Stefano, and Siri Nergaard. 2011. “Translation: A New Paradigm.” Translation:
A Transdisciplinary Journal, Inaugural Issue: 8–16.
Armitstead, Claire. 2019. “Mirror Writing: Authors and Their Translators.” The Guardian
Review 6: 7–11.
Arnaldi, Marta. 2016. “Moving Wor(l)ds:Translation as Transformation in Poetry and Dance.”
Logios Journal 1: 35–46.
Baker, Mona. 2001. “Translation Studies.” In Routledge Encyclopedia of Translation Studies,
2nd ed., edited by Mona Baker and Kirsten Malmkjær, 277–80. London and New York:
Routledge.
Baldry, Anthony, and Paul J. Thibault. 2006. Multimodal Transcription and Text Analysis: A Mul-
timedia Toolkit and Coursebook. London: Equinox.
Bassnett, Susan. 2002. Translation Studies, 3rd ed. London and New York: Routledge.
———. 2014. Translation. London and New York: Routledge.
Bassnett, Susan, and André Lefevere. 1990. “Introduction: Proust’s Grandmother and the
Thousand and One Nights: The ‘Cultural Turn’ in Translation Studies.” In Translation,
History and Culture, edited by Susan Bassnett and André Lefevere, 1–13. London and New
York: Cassell.
———. 1998. Constructing Cultures: Essays on Literary Translation. Clevedon: Multilingual
Matters.
Bermann, Sandra, and Catherine Porter. 2014. “Introduction.” In A Companion to Trans-
lation Studies, edited by Sandra Bermann and Catherine Porter, 1–27. Chichester:
Wiley-Blackwell.
Bezemer, Jeff, and Gunther Kress. 2015. Multimodality, Learning, and Communication: A Social
Semiotic Frame. London and New York: Routledge.
Introduction 21
Boethius. 2001 [524]. Consolation of Philosophy. Translated by Joel C. Relihan. Indianapolis

and Cambridge: Hackett Publishing.
Bohn, Willard. 2001. Modern Visual Poetry. Newark: University of Delaware Press.
Borodo, Michał. 2015. “Multimodality,Translation, and Comics.” Perspectives: Studies in Trans-
lation Theory and Practice 23 (1): 22–41.
Bunt, Harry, Robbert-Jan Beun, and Tijn Borghuis, eds. 1998. Multimodal Human-Computer
Communication: Systems,Techniques, Experiments. Berlin and Heidelberg: Springer-Verlag.
Chesterman, Andrew. 1997. Memes of Translation: The Spread of Ideas in Translation Theory.
Amsterdam and Philadelphia: John Benjamins.
Davies, Eirlys E. 2012. “Translation and Intercultural Communication: Bridges and
Barriers.” In The Handbook of Intercultural Discourse and Communication, edited by
Christina Bratt Paulson, Scott F. Kiesling and Elizabeth S. Rangel, 367–88. Oxford:
Wiley-Blackwell.
Díaz Cintas, Jorge. 2009. New Trends in Audiovisual Translation. Bristol: Multilingual Matters.
Dicerto, Sara. 2018. Multimodal Pragmatics and Translation: A New Model for Source Text Analysis.
London: Palgrave Macmillan.
Halliday, Michael. 1978. Language as Social Semiotic: Social Interpretation of Language and Mean-
ing. Baltimore: University Park Press.
———. 1985. An Introduction to Functional Grammar. London: Edward Arnold.
Hermans,Theo. 1985. “Translation Studies and a New Paradigm.” In The Manipulation of Lit-
erature: Studies in Literary Translation, edited by Theo Hemans, 7–15. London and Sidney:
Croom Helm.
Heusser, Martin et al., eds. 2005. On Verbal/Visual Representation. Amsterdam and New York:
Rodopi.
Hodge, Robert, and Gunther Kress. 1988. Social Semiotics. Ithaca, NY: Cornell University
Press.
Holmes, James. 1988 [1972]. “The Name and Nature of Translation Studies.” Translated!
Papers on Literary Translation and Translation Studies, 67–80. Amsterdam: Rodopi.
ISTAT. 2018. Produzione e lettura di libri in Italia. Report, December. Accessed April 19, 2019.
www.istat.it/it/files//2018/12/Report-Editoria-Lettura.pdf.
Jakobson, Roman. 1971 [1959]. “On Linguistic Aspects of Translation.” In Selected Writings II:
Word and Language, 260–66. The Hague and Paris: Mouton.
Jewitt, Carey, ed. 2014. The Routledge Handbook of Multimodal Analysis, 2nd ed. London and
New York: Routledge.
Jewitt, Carey, Jeff Bezemer, and Kay O’Halloran. 2016. Introducing Multimodality. London and
Jiménez Hurtado, Catalina, Tiina Tuominen, and Anne Ketola, eds. 2018. “Methods for the
Study of Multimodality in Translation.” Linguistica Antverpiensia 17.
Kaindl, Klaus. 2004. “Multimodality in the Translation of Humour in Comics.” In Perspec-
tives on Multimodality, edited by Eija Ventola, Cassily Charles and Martin Kaltenbacher,
173–92. Amsterdam and Philadelphia: John Benjamins.
———. 2013. “Multimodality and Translation.” In The Routledge Handbook of Translation
Studies, edited by Carmen Millán and Francesca Bartrina, 257–70. London and New
York: Routledge.
Ketola, Anne. 2016. “Towards a Multimodally-oriented Theory of Translation: A Cognitive
Framework for the Translation of Illustrated Technical Texts.” Translation Studies 9 (1):
67–81.
Kress, Gunther. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication.
London and New York: Routledge.
Kress, Gunther, and Theo van Leeuwen. 1996. Reading Images:The Grammar of Visual Design.
———. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication.
London: Hodder Education.
Lambert, José. 2013. “The Institutionalization of the Discipline.” In The Routledge Handbook
of Translation Studies, edited by Carmen Millán and Francesca Bartrina, 7–27. London and
Lefevere, André. 1978. “Translation Studies. The Goal of the Discipline.” In Literature and
Translation: New Perspectives in Literary Studies with a Basic Bibliography of Books on Translation
Studies, edited by James Holmes, José Lambert and Raymond van den Broeck, 234–35.
Louvain: Acco.
Li, Long, Xi Li, and Jun Miao. 2018. “A Translated Volume and its Many Covers – A Multi-
modal Analysis of the Influence of Ideology.” Social Semiotics 29 (2): 261–78.
Malmkjær, Kirsten. 2013. “Where Are We? (From Holmes’s Map until Now).” In The Rout-
ledge Handbook of Translation Studies, edited by Carmen Millán and Francesca Bartrina,
31–44. London and New York: Routledge.
Minors, Helen Julia, ed. 2013. Music, Text, Translation. London, Delhi, New York and Sydney:
Bloomsbury.
Munday, Jeremy. 2010. “Translation Studies.” In Handbook of Translation Studies, edited by
Yves Gambier and Luc van Doorslaer, 419–28. Amsterdam: John Benjamins.
Nida, Eugene. 1964. Toward a Science of Translating:With Special Reference to Principles and Pro-
cedures Involved in Bible Translating. Leiden: E. J. Brill.
Niranjana, Tejaswini. 1992. Siting Translation: History, Post-structuralism, and the Colonial Con-
text. Berkeley: University of California Press.
O’Sullivan, Carol, and Jean-François Cornu, eds. 2019. The Translation of Films, 1900–1950.
Proceedings of the British Academy. Oxford: Oxford University Press.
O’Sullivan, Carol, and Caterina Jeffcote, eds. 2013. “Special Issue on Translating Multimo-
dalities.” Journal of Specialized Translation 20.
Pan, Li. 2015. “Multimodality and Contextualisation in Advertisement Translation: A Case
Study of Billboards in Hong Kong.” Journal of Specialized Translation 23: 205–22.
Pérez-González, Luis. 2014a. Audiovisual Translation:Theories, Methods, and Issues. London and
———. 2014b. “Multimodality in Translation and Interpreting Studies.” In A Companion to
Translation Studies, edited by Sandra Bermann and Catherine Porter, 119–31. Chichester:
Wiley–Blackwell.
Robinson, Peter. 2010. Poetry and Translation: The Art of the Impossible. Liverpool: Liverpool
University Press.
Rossetti, Dante Gabriel. 1870. Poems. London: Ellis.
Schulte, Rainer, and John Biguenet, eds. 1992. Theories of Translation: An Anthology of Essays
from Dryden to Derrida. Chicago: The University of Chicago Press.
Simon, Sherry. 1996. Gender in Translation: Cultural Identity and the Politics of Transmission.
Snell-Hornby, Mary. 1988. Translation Studies: An Integrated Approach. Amsterdam and Phila-
delphia: John Benjamins.
———. 2006. The Turns of Translation Studies: New Paradigms or Shifting Viewpoints? Amster-
dam and Philadelphia: John Benjamins.
———. 2009. “What’s in a Turn? Fits, Starts, and Writhings in Recent Translation Studies.”
Translation Studies 2 (1): 41–51.
Introduction 23
Sterne, Laurence. 1767. The Life and Opinions of Tristram Shandy, Gentleman,Vol. 9.York and
London: T. Becket and P. A. Dehondt.
Toury, Gideon. 1995. Descriptive Translation Studies and Beyond. Amsterdam and Philadelphia:
John Benjamins.
Trivedi, Harish, and Susan Bassnett, eds. 1999. Post-colonial Translation: Theory and Practice.
van Leeuwen, Theo. 2005. Introducing Social Semiotics. London and New York: Routledge.
Venuti, Lawrence. 1995. The Translator’s Invisibility: A History of Translation. London and New
York: Routledge.
———, ed. 2012. The Translation Studies Reader, 3rd ed. London and New York: Routledge.
Villon, François. 1994 [1461]. François Villon, Complete Poems, edited by Barbara N. Sargent-
Baur. Toronto: University of Toronto Press.
von Flotow, Luise. 1991. “Feminist Translation: Contexts, Practices and Theories.” TTR:Tra-
duction,Terminologie, Rédaction 4 (2): 69–84.
———. 1997. Translation and Gender: Translating in the ‘Era of Feminism’. Ottawa: University
of Ottawa Press.
Weinberger, Eliot, and Octavio Paz. 2016 [1987]. Nineteen Ways of Looking at Wang Wei (with
More Ways). New York: New Directions.
Weissbort, Daniel, and Ástráður Eysteinsson, eds. 2006. Translation:Theory and Practice: A His-
torical Reader. Oxford: Oxford University Press.
Zanettin, Federico, ed. 2008. Comics in Translation. Manchester: St Jerome.
1
TRANSPOSING MEANING
Translation in a multimodal
semiotic landscape
Gunther Kress
1. Preamble: the question of translation in the

contemporary multimodal semiotic landscape
In everyday interactions, banal or profound, actors in the contemporary social and
semiotic world draw on a wide range of material means for making meanings
(materially) evident – the modes of the semiotic landscape. In this chapter the aim
is to explore what issues arise in moving from the dominance of one means (“lan-
guage”) to a recognition of the potential equality of many means of making mean-
ing evident (“the modes of multimodality”) and to consider the effects of that
change on present conceptions of translation. Unlike, say, the chapter by Matthew
Reynolds in this volume, the present chapter is not written from within transla-
tion theory. Its focus is a sketch of a paradigm shift, in terms of the issue at its core,
moving from the centrality of “language” to a focus on “meaning”; and in terms
of disciplines, a move from a linguistic to a semiotic frame. In that, it is part of the
much larger changes affecting the global political, social, economic, and cultural
landscapes, in Europe as elsewhere, which are unmaking centuries of relative stabil-
ity. That brings the problem of “naming”: the names which had lent some stability
to that world no longer adequately describe the contemporary social and semiotic
landscape. Clarity in “naming” is required and, with that, the development of a ter-
minology coherent and apt for that change. Throughout the chapter, I introduce
requisite (semiotic) terms, tying them in to essential elements of the discussion.
2. A slightly odd take on translation: “Chinese Whispers”

with a twist
At a semi-formal meeting, about twenty people – in the main, craftspeople/artists
of various kinds, a few academics and others sprinkled in – are gently cajoled by
Transposing meaning 25
FIGURE 1.1 The translation game
the Master of Ceremonies (MC) of the evening to participate in a game of “Chi-

nese Whispers.” The MC has copied a range of different kinds of images – abstract,
figurative, contemporary, traditional, “western” or, indeed, from anywhere on the
globe. Members of the group are invited to select one of the images. Everyone has
received an A4 piece of paper and a pencil. The MC tells the group to “produce a
written description of ‘your’ image, in maximally thirty minutes” (see Figure 1.1).
When the time is up, the images that had been handed out are gathered by the
MC, together with the written descriptions. She then distributes these accounts
randomly, one for everyone there. Around the room are tables with boxes of col-
oured pencils and crayons, to produce the drawings that are to be made on the basis
of the written description each person has received. Everyone now tries to produce
the image that corresponds with the written description they have. After maybe
forty minutes of drawing, the MC calls a halt. She invites everyone to find the
original image which had been the basis of the description. There is considerable
interest in comparing the written descriptions with the original and with what has
just been drawn from what had been written.
Although everyone in the room is familiar with the game of “Chinese Whis-
pers,” there is, nevertheless, astonishment at just how badly our notions of “com-
munication” fit with a reality marked by difference of all kinds and at all points:
26 Gunther Kress
difference of personal “interest” of the participants, leading to difference in focus,

leading to difference in selecting aspects from a message; based on differences in the
“deeper” perspectives from which each of us selects elements to newly constitute
the message.
The game, as it is traditionally played, makes use of a shared meaning resource:
speech. The “twist,” here, lies in the fact that there is no shared meaning-resource –
no shared “code” – for moving from image to writing and back to image. The
building blocks of images are not words; and while there are relations between ele-
ments of the image, they are neither the elements nor the relations of the grammar
of writing: no Subject-Verb-Object relations nor, for instance, indications of time as
tense. The meanings at issue in the game – whether written or visual – do express
the meanings of a particular community, yet they are made evident by entirely dif-
ferent material/semiotic means: more or less adequately, in line with the distinct
affordances of writing and image. It raises the question: even though meanings of
one social group are dealt with, the absence of a shared code for writing and image
makes the activity, in many ways, quite similar to that of translation. (One difference
being the absence, here, as yet, of relatively well understood assumptions of various
kinds, which exist in language-to-language translation.)
The traditional version of Chinese Whispers relies on the fact that the meaning
resource of speech is shared by all players: it is used by each in relation to their dif-
fering interests and perspectives in the moment of playing. In the “twisted game”
there is no shared meaning resource for moving between speech and image, even
though there may be a broadly settled, general sense of shared meanings in that
community of players. The “twisted game” brings out a strongly unsettling sense
that writing and image each enable meanings to be made which cannot, however,
easily be “shifted” from the one mode to the other – nor, actually, made at all in
the other mode.
Each mode offers profoundly distinct potentials for documenting meaning.
Immediately the problem of “naming” emerges: in the move from one mode, writ-
ing, to a different mode, image, translation is not exactly what is the issue here. A term
such as transduction, indicating a shift of a specific kind, suggests itself.
The traditional version of the game makes clear that notions of “en-coding”
and “de-coding” barely work even when using the shared resource of speech; in
the “twisted” version it makes no sense. At a somewhat abstract level, writing
and image realise aspects of social functions; they do not, however, share anything
resembling a commensurate “code.”
What questions – beyond those we know from the traditional version – arise
from playing the “twisted version”? Any painter, sculptor, or composer will be
familiar with the request “Could you give me some idea what this is about?” At
times the response might be an exasperated “Have a good close look; do listen;
reflect for a while; it’s all there.” In schools, in most contemporary textbooks and
in most online resources, whether in Biology or Mathematics, in Geography or
Physics, the majority of the page or the screen is given over to image, rather than
to writing. Much the same – or even more so – is the case in an ever-increasing
number of social domains. How can we expect this to work when as a society we
have no real sense of the resources needed to reconstitute, to transpose a meaning
which appears as an image – a diagram of a magnetic field, let’s say – with the
resources of writing, of words in syntactic relations?
Writing is still the privileged mode in important social domains (in schools, for
instance, in the case of most exams; in legal domains; etc.).Yet the game does high-
light the shortcomings of an approach based on the existing commonsense: writing
does particular kinds of semiotic work; it does not and cannot do all of it. Despite
centuries-long certainties that “language” is fully capable of expressing all mean-
ing that needs to be expressed, whether as writing or as speech, each is a partial
means of making meaning. The same is true of image, as it is true of every means
of making meaning. It is that which makes the case for a social semiotic multimodal
approach.
In the everyday, most messages still occur in face-to-face interaction; yet more
and more frequently, messages come via the small screens of the devices we carry
with us everywhere. There, image and writing usually appear together; though
ever more frequently image is dominant or appears by itself. The majority of our
communicational life is beginning to resemble versions of Chinese Whispers, with
multiple twists.
The huge project looming to be addressed is to bring to light the characteristics
and the regularities of the wide range of meaning resources we now use every day,
in still very new ways. That is an urgent issue already in just about every area of
our lives. In relation to the interests of the group assembled that evening, the game
highlighted, yet again, how difficult – impossible even – it is to describe meanings
that inhere in “making” and in “things made”: that is, meaning beyond the presently
recognised, canonical means. There is a need to find ways other than the limiting,
only partially adequate, routes of writing or speech for making such meanings evi-
dent: giving attention and recognition to meaning-makers in all areas of social life, to
making, with all kinds of materials and to all things made.
In the game played at the meeting, the MC kept the instructions simple: “Docu-
ment the meanings made by your image as a written account, an account sufficient
to allow the meaning of the original in your written account to be reconstituted as
an image, by another member of this group.” Nothing could seem simpler, right?
Yet what consequences loom if we do not develop relatively reliable means for such
a simple task?
3. A brief sketch of a social semiotic theory and its tools

for documenting and transposing meaning
Social semiotics (Hodge and Kress 1988; van Leeuwen 2005; Kress 2010; Bezemer
and Kress 2016) offers a clear focus on meaning and meaning-making. It suggests
both a frame and tools to account for the constitution of meaning. In that frame, it
offers an account of translation seen as the transposition of meaning in the multimodal
semiotic landscape of the contemporary social world. The term transposition offers
28 Gunther Kress
a conceptual/theoretical frame capable of dealing with changes in the making of

meaning in all areas. “Position” – as an encompassing metaphor – is the crucial issue
in all instances of making meaning: whether position in social environments, of
semiotic resources, of personal “interest,” or of other significant factors. In what is
still called translation, “position” is the issue.
A rudimentary sketch of social semiotic theory is needed to make that case.
Here, a brief outline of that theory is followed by an account of sign-making as the
core of that theory. A further step suggests that translation/transposition – in the now
expanded sense of the transposition of all and any meaning – is an instance of commu-
nication and is accounted for in that overarching theory.
Social semiotic theory takes it as given that meaning arises in social actions
and interactions, communication being perhaps the most common of these. As
discussed in the Introduction, the theory draws on the work of the semiotician-
linguist Michael Halliday, and specifically on two assumptions: one (stated by him in
slightly varying forms), that “language is as it is because of the functions it serves in
society” (Halliday 1978, 18–19); the other, that language, as a full communicational
resource, has to satisfy three (social) functions. One deals with meanings about
“states of affairs in the world,” the ideational function; another deals with the mean-
ings of the social relations of the participants in interaction, the interpersonal function.
A third deals with the meanings produced by relations of elements within the text, as
well as the relation of a text with its environment, the textual function.
The first assumption asserts that the semiotic resources of a society are the out-
come of their shaping in social (inter-)actions. The second asserts that in order for
any semiotic resource to be a fully functioning means for human communication,
it has to deal with meanings which arise in each of these three distinct yet entirely
integrated social domains.
By substituting the phrase “semiotic resource” for “language,” and by bringing
human agency clearly into the frame, Halliday’s statement becomes an encompass-
ing social semiotic theory: “semiotic resources are as they are because of the func-
tions they are made to serve in society.” It provides a workable frame for a social
semiotic multimodal theory of meaning and meaning-making. It asserts, among
other things, that every community has a range of resources for making meanings evi-
dent: speech, gesture, gaze, writing, and others; that is, the modes of social semiotic
multimodal theory. Further, the characteristics of the modes (their elements and
relations) are the product of social semiotic action – of social and semiotic work, in
a specific community – with and on the material “stuff ” of each mode. Modes and
their constituent semiotic elements mirror both the interests of the makers and the
potentials and limitations of the materials chosen for making meaning: their affor-
dances (Gibson 2014). Examples of “material stuff ” are: movement of parts of the
body against the (relatively) still upper torso in the case of gesture; air, in the case
of speech; movement of the body in space, in the case of dance. Meanings are made
constantly by every member of a community; hence, all meanings are the meanings
of that specific community.
As a consequence, modes – together with their elements and the relations of

elements – are repositories, traces of the history of meanings shaped in one com-
munity. The distinctly different affordances of each mode are the outcomes of cease-
less social semiotic work with and against the potentials and the resistances of the
materials for making meaning. This work produces the semiotic/cultural resources
of a social group. In these, each mode is both general (it meets the requirements of
the three functions) and specific, given the different affordances of each mode for
making meanings materially evident. All meanings made materially evident in any
of the modes of a society bear a traceable affinity with others made evident in the
mode, as well as with the meanings made evident in each of the modes of the soci-
ety. “Affinities of meaning” has two aspects: a social-(semiotic) aspect and affinities
due to the shared materiality of a mode – a semiotic-(social) aspect.
This brief summary encompasses kinds of issues recognised in present theories
and practices of translation, and it points to continuities of these with issues arising
in the expansion of the field suggested here. It underpins the case of a paradigm
shift from language to meaning, as well as that of a disciplinary shift from linguistics
to semiotics.
Mode, defined as material shaped in the history of social and semiotic work, makes the
term and the uses of “language” problematic. Given that both the material (air, in
the case of speech; surfaces and traces on these in the case of writing) as well as the
histories of social semiotic work in each case are distinctly different, it is difficult
to see how speech and writing can reasonably be subsumed under the one label,
language. As a consequence, in the social semiotic multimodal approach outlined
here, the term language is replaced by the terms speech and writing, each naming a
distinct mode.
One of the benefits of the move from the use of the term of language to that
of mode is that whatever the semiotic resources of a community are – for instance,
whether they do or do not use an alphabetic or syllabic or character-based or any
other kind of “script” – all are included in the one set of semiotic principles. All
communities have developed semiotic modes requisite to social need and material
potentials. All have and use modes. Crucially, the theoretical notions of meaning-
making and of the transposition of meaning apply to all.
Abandoning the term “language” introduces a need for careful, apt naming. We
cannot, now, use a term that names a semiotic resource (e.g., “English, the lan-
guage”) to name (even if implicitly) a social-cultural-political entity: as for instance
in “The mode of gesture works differently in English than it does in French”
(where “English” names both a mythical place, Englishland, and a semiotic resource).
“English,” the semiotic resource, is shaped by the social practices of very different
social-cultural/political entities: in Nigeria, in Hong Kong, in England, in Scotland,
on the East Coast of the US, in Wales, etc. In short, the signifier “English” is now a
problematic homonym. Given the present lack of adequate names, circumlocutions
will have to do: for instance, “the mode of writing in Nigerian English culture,” “the
mode of gesture in Hong Kong English culture.”
30 Gunther Kress
The recognition of a wide range of modes, and an awareness of the changed

significance in communicational uses of modes beyond those of speech or writ-
ing, forces a rethinking of issues around meaning. On the one hand, each mode,
despite its differing affordances, provides means for realising meanings in each of
the three functions of social semiotic theory. That is a general and unifying aspect
of the theory. On the other hand, given the constant social work in a specific
community on materials chosen for semiotic use, each mode has distinct semi-
otic characteristics. The affordances of speech, for instance, are an effect jointly of
its temporal characteristics (the logic of time and sequence) interacting with the
material potentials of sound (such as pitch, energy, duration, pace), both shaped in
social semiotic work in particular environments. The characteristics/affordances
of (still) image are the result of its spatial logic interacting with the potentials of
mark-making on a surface, together with the histories of its semiotic work. The
mode of gesture combines the logics of spatiality and of temporality (given the
movement of hands, fingers, arms, against the stable upper body of the signing
person).
In the well-understood academic practice of transcription – from speech to writ-
ing in one community – there is a pretty well taken for granted assumption (not
expressed in this way) that the target mode (writing) provides the transcriptional
resource – letters usually (or, in some instances, lexical items/words) – by means of
which the elements of the source mode – speech sounds (or lexical entities of
speech) – can be newly constituted. Transcription provides a good example for
documenting the manifold problems inherent in this seemingly straightforward
practice: the alphabet being a notoriously bad resource for the task of newly con-
stituting meanings spoken as meanings written.
Transcription does, however, supply two general principles: one, that the resource
to be used in the transposition of meaning is a resource available in the target mode
(letters in this case; or lexical items in standard written form); and two, that the rela-
tion of the elements of the reconstituting/transpositioning mode to the elements of
the originating mode is that of aptness more or less, hence metaphoric. That is, the
“meanings” (the sonic quality) of the sounds of speech (as the signified) are made
materially evident in the resource of letters (as signifiers), treated as apt means of
achieving this. In other words, the relation between the signs of the source mode
(here speech) and the signs of the target mode (here writing) is never one of equiv-
alence, but always one of aptness, hence of metaphor.
The meanings produced in one community can be realised by any of its modes.
At one level they are the “same” meanings, even though realised in the affordances
of the different modes of a community. If there is a need for a meaning such as
“intensity,” it can be expressed (“be given material realisation,” “made materially
evident”) through the affordances of each of the different modes. In the mode of
gesture, “intensity” might be realised by pace and/or by the extent of the sweep of the
gesture. In the mode of speech, “intensity” might be realised through greater energy,
as loudness, or by lengthening a sound. In writing, “intensity” might be realised by bold-
ing, or as the thickness of a line in handwriting, or by capitalisation. In both the modes
of speech and of writing, lexis is available as a means of indicating intensity, via use of
words such as very, huge, and terrible.
Such issues are dealt with in present conceptions of translation.What differs here
is that different realisations of “one meaning” occur within one and the same com-
munity, and at times have to be expressed – made materially evident – in another
of that society’s modes and its affordances. In that “other mode,” social semiotic
work will have shaped the different material of that mode in forms apt to the users
of that mode. An example might be a “gestural” unit or “the shot” of a kineiconic
mode (Burn 2014). Hence “one meaning” has a distinctly different “appearance,” a
distinct material realisation in the one community.
This “different look” may have given rise to the notion of “different semiotics”
(and of intersemiotic translation) to deal with different materialities: semiotics of
film, gesture, image, music, dance, etc. The previous centrality of language – with the
assumption of its capacity as the means of making all meanings evident – may have
had a large effect on this: when language was seen as central and all-encompassing,
other means of making meaning might need to be dealt with by a different (semi-
otic) theory.
Across societies, specific social semiotic work will have shaped the modes of
speech, image, gesture, and writing in the distinct ways required by the source and the
target society. Given the different development of modes in specific communities,
the assumption of “same mode, same meanings” cannot be sustained.
The realisation that semiotic features are differently realised across societies and
languages (in the traditional sense) is taken for granted in practices of translation.
However, the previous example (and the theory underpinning it) combines that
with the recognition of a multiplicity of modes in each of the communities, each
with distinctly different affordances given the material differences of modes. The
principles are – in the main – of the same kind as those of translation across different
“languages.” What traditional approaches to translation had raised (e.g., Jakobson’s
1959 “intersemiotic translation”), but not made central, are the means for the rec-
ognition of modes and the conditions of the entirely usual transposition of meaning
involving all modes: whether in the same or across different societies or, not unu-
sually, both. Social semiotic theory assumes a homology of modes in the source
society matched by a homology of modes – distinct and different – in the target
society. The difficulty lies in the likely absence of such apt transpositional resources
by modes in the target society.
The term transposition, proposed in this chapter, draws attention to two core
issues. One is “position” in a social sense and the other is “position” in a semiotic
sense: that of the material semiotic resources involved. That is, signs are made in
specific environments, social and semiotic; they bear the traces of the environments
of their making. “Position” in the second sense refers to the materiality of the mode
in which meaning is made materially evident, in the signifier. That “position” has
epistemological and ontological effects (as pointed out later). Both the initial maker
of the sign and the person engaging with that sign in interaction are positioned
socially and semiotically, though most likely differently in one or both.
32 Gunther Kress
To restate this slightly: neither a sign nor a multimodal ensemble of signs “has”
a meaning. In their initial making in a specific environment and in the materiality
of specific modes, the meanings of the sign maker are made materially evident. In
engaging with the sign as a prompt, it is a meaning resource for the social semi-
otic actor who engages with the prompt. “Meaning” is not transposed: rather, the
potentials for making meaning occur in two positions. In each, signs are made: once
in initial making and once – as a new and different sign – in its “interpretation.”
What has been transposed is a potential for making meaning, from one position to
another. In each position, meaning is made according to the characteristics of the
environment and the affordances of the modes. Meaning is never transported in the
manner of a book that is parceled up and sent to me by Amazon.
In translation it is assumed that the source and the target community have modes
with the same (or at least similar) semiotic entities. They might both have modes
with lexical items, grammatical (morphological and syntactic) forms, and both have
larger textual organisations, genres maybe. One-to-one correspondence cannot be
assumed. The theory needs to work equally well for societies which use modes
based on the alphabetic “rendering” of speech and those which are not; for socie-
ties which have communicational resources such as so-called Australian Aboriginal
paintings; or be capable of dealing with the relation between a character-based
script and speech. Nearer to home would be the relation between means of docu-
menting mathematical meanings and their transposition into the affordances of
alphabetically rendered speech.
To recap at this point: Halliday’s amended dictum “semiotic resources are as they
are because of the functions they are made to serve in society” allows us to expand
the present frame of “translation.” It makes it possible to bring the field as tradition-
ally conceived together with the now essential task of dealing with transposition of
meaning of all kinds in any situation into one coherent field.
Modes realise meanings in all three functions. That is a unifying principle for
meanings and modes, whether across a community or beyond that across all social
organisations – groups, societies, ethnicities.What modes exist in a community, how
and to what extent they are (materially) developed, which are communicationally
foregrounded, differs from one social group to another. The theory and its func-
tional requirements are the same.
4. The partiality of the semiotic resource of modes

As a consequence of the differing materiality and the social semiotic histories
of shaping, and hence of the difference in the affordances of modes, in a social
semiotic account all modes, speech and writing included, are seen as partial
means for realising meanings. While all modes meet the requirements of the
three functions, they do so in terms of their different affordances. For exam-
ple, the social and semiotic feature of coherence will need to be realised in all
modes of a social group. If the mode is writing (in UK English, say), the entities
available to realise coherence include pronouns, (other) lexical items, syntactic

forms, punctuation, type of font, and so on. These entities are not available in
the mode of image. There, different semiotic features are available to realise
coherence: colour, for instance (as colour palette), or placement of elements
in spatial relations. In speech (in UK English), intonational features are used
as cohesive features to produce coherence. In arranging a group of people for a
photograph, placement in the group or proximity can be used as semiotic features –
signifiers – to produce signs of coherence.
These are instances of signifieds of the textual function being realised materi-
ally, in each case through different signifiers of cohesion and coherence in different
modes.While all modes realise meanings in each of the three functions, their differ-
ent affordances mean that signifieds (meanings) are made materially evident differ-
ently. This is where the notion of the partiality of modes arises: each mode meets
the functional requirements differently, as Tamarin Norwood’s discussion in her
chapter for this volume demonstrates. In each case, the difference matters. Through
complementarity of signs in ensembles of different modes, the meanings intended
are more likely to be achieved.
One corollary of partiality is that modes tend to occur in multimodal complexes:
whether to provide means for complementarity, or for additional as well as for parallel
or divergent meanings in a multimodal complex. Take (as an invented example) a
written element in a multimodal complex: “the trees stood out against the dark-
ening sky.” Here the inceptive aspect “darkening” of the mode of writing (in UK
English) might be accompanied by elements of the mode of moving image show-
ing both “developing darkness,” together with uses of colour, all complementing,
“underlining” the morphemic meaning of the “inceptive” aspect.
The potential of foregrounding a mode (or two or three) exists in any multi-
modal complex, as do different uses attributed to and met by each mode. These
are a matter of socially motivated design in specific instances, always including
intentions of both rhetor and designer of the complex. Those who engage with
the multimodal complex cannot, of course, know whether the rhetor and designer
(whether as one or as two persons) felt that they had fully achieved the meaning
they wanted to convey. That is true of all multimodal complexes, whether of a
(largely) musical composition, a painting, a film, etc. Was Kandinsky satisfied when
he had completed the painting entitled “Kossaks”? Complementation does not
guarantee that meaning has been fully represented. Comprehension is always par-
tial; interpretation is always a hypothesis.
In relation to the transposition of meaning, whether within a society or across
societies, given the different affordances of modes in different social environ-
ments, there is always a question whether and which of the potentials for creating
meanings of modes, of all or some in a modal complex need to be transposed; or
whether indeed in the process of transposition quite different modal complexes will
need to be designed and produced. Although, as noted earlier, meanings are never
transposed – the potentials for creating meaning are.
34 Gunther Kress
5. Sign-making: interest, criteriality, aptness

Sign-making is meaning-making; it is the core of semiosis. Signs are the constitu-
tive elements of modes and of modal complexes as texts. Sign-makers use available
material-semiotic resources, signifiers, from within specific modes, chosen as apt in
relation to the signifieds of the social group in which the signs are to be made.
Every sign (and therefore all multimodal sign-complexes) tells us something
about the sign-maker. How she or he saw the world in the moment of making the
sign is revealed in the choice of the signifier which was regarded as apt. The follow-
ing example makes the point. A three-year-old boy, sitting on his father’s lap, draws
a series of seven circles (see Figure 1.2).When he has finished, he says: “this is a car.”
The question arises as to how this is or could be “a car.” While drawing, the
child had said: “here’s a wheel, here’s another wheel, that’s a funny wheel . . . this is a
car.” The criterial feature of a wheel was that it was circular; hence circles were apt
signifiers for representing wheels. The criterial feature of “car” seemed to be that
it had many wheels; hence the arrangement of seven circles seemed an apt signifier
for representing “a car.”
To represent wheels by circles rests on analogy: wheels are like circles; “aptness” is
a feature of analogy. The outcome of the analogy is a metaphor: “a wheel is (like) a
circle.” Similarly with the representation of “a car”: “a car is something with many
wheels.” The car-sign, and its meanings, is the result of a sequence of two meta-
phors. For this sign-maker, the signifiers (the form) “circle” and “many wheels” are
apt (that is, their material form and their histories of use make them suitable) to be
the carriers of the signifieds of “wheel” and of “car.”
We might ask further: why and how could, for this three-year-old, a circle be
the signifier for a wheel, and how can wheels be the criterial feature for “car.”
The answer to the first question seems self-evident: both are round. “Roundness”
provides the criterial feature for the analogy. As far as the second question is con-
cerned, the answer might be that if we imagined the eye-level view of a three-year-
old, looking at the family car (in this case, a 1982 VW Golf, with its prominent,
FIGURE 1.2 Drawing by a three-year-old: “this is a car”

chunkily visible wheels, especially at the three-year-old observer’s height), we

might conclude that his position in the world, literally, physically, cognitively, and
affectively, might well lead him to see cars – as it did on this occasion – in that way.
His drawn sign represents his “position,” broadly speaking his “interest,” arising out
of his (physical, affective, cultural, social) position in the world at that moment, vis-
à-vis the object to be represented.
Generalising, we can say that interest shapes attention to a part of the world which
is in focus and acts as the motivation for (principles of) selection of that which is to be
represented (Kress 1997).
The point is that the interest (in the sense indicated) of the sign-maker/meaning-
maker determines, in the moment of its (material) realisation, what is taken as
criterial about an entity. That which is taken as criterial will be the signified and
becomes central in the making of the sign. The child’s drawing suggests and realises
a view of a part of the world which is, for him, historically, socially, and culturally
shaped. What the meaning-maker takes as criterial determines what (s)he will rep-
resent about that entity or phenomenon. The drawing is the result of the child’s
semiotic work in his engagement with a part of the world: it makes his distinct interest
evident in his selection of the material signifier.
Three points are crucial in this example. First, the relation between meaning/
signified and form/signifier is not arbitrary. It is motivated. That is, the form suggests
itself to the sign-maker because it is, for her or him at this point, the apt means to be
the carrier of the meaning. The signifier (in its material form), taken together with
the history of its prior uses (in ways known to the sign-maker: here, as a personal
history of two years of making constantly changing circular drawings) satisfies the
interest of the sign-maker in finding an apt material means of expressing the signi-
fied. Form and meaning exist in a relation of motivation. This sign, like all signs,
whoever the sign-maker, is constituted on the basis of a motivated relation of form
and meaning.
That process is no different in the transpositions of translation: the choice of sig-
nifier (whether sound, word, phrase, or larger textual entity) in the target mode is
made on the basis of the same principles. Interest, of course, is (among other things)
often professionally shaped. It is one means by which power enters into the trans-
position of meaning.
The second issue, closely linked to the first, is that the sign – at the moment
of its constitution – is based on that which is criterial to the sign-maker about that
which is to be represented. All signs represent that which is criterial about what is
to be realised, materially, to be represented. In that sense the sign is always both full
and partial: fully representing that which is criterial in the moment of sign-making
and partial in relation to the many other features which – from other perspec-
tives, on other occasions, and of course for other sign-makers – constitute the
semiotic entity being reconstituted in being transposed. In every sign the relation
between form and meaning is motivated by the interest of the sign-maker. The sign
is always both a “full” representation of the sign-maker’s interest and only ever a
partial account of some entity or phenomenon when seen from a different position,
36 Gunther Kress
such as professional or other considerations and demands, which shape attention

and selection.
The third point follows from the previous two: every sign is newly made. It is a
crucial assumption in social semiotics that all signs, with no exception, are newly
made in the moment of their making, in the manner just described. Signs are
never used; signs are newly made. The prompting sign is taken as the signifier, to be
interpreted. The new sign having been made internally, the signifier is transformed,
however minutely, bearing the traces of its recent as well as those of all other uses.
That is the case for all signs, made by any maker of signs, anywhere, however banal
or portentous the sign might seem to be.
The notion of “new making” – of novelty, creativity, “interpretation” – is bound
to be problematic for traditional notions of “accuracy,” “precision,” “veracity” in
mainstream practices and assumptions about translation. That is a fact to be recog-
nised and dealt with in a theory of translation as transposition. Among others, it is
a matter of what audience “holds the aces.”
The first of these three issues permits making hypotheses about the interest of the
sign-maker: the sign is the trace of the semiotic work of selecting an apt signifier
for the signified. The choice of signifier provides an insight into the sign-maker’s
principles underlying the sign-making. When the sign is made in response to some
prompt, it gives insight into the sign-maker’s semiotic work of selecting elements
from the prompt and of the semiotic work of making a sign that aptly constitutes –
for her or him, at this moment – an apt response.
The second issue, the partiality of the sign, provides, in the response, an indica-
tion of what was considered to be criterial by the sign-maker about the prompt and
what was not taken as relevant or criterial. The third point, the always new making
of the sign, moves away entirely from still existent mainstream notions of commu-
nication (the notion of the faithful copy, the accurate decoding), of translation as
an instance of communication in which the aim is producing faithful copies of the
original – accurate replicas, as near as possible.
6. Communication in the frame of social semiotic theory

In the sketch of the social semiotic theory presented so far, sign-making as action is
present, even if only implicitly, whereas interaction is not. Sign-making, however, is
always the action of a sign-maker embedded in the social and semiotic world. In the
absence of the social environment in which she or he is acting, there would be no
indication of what has prompted the sign-maker to make the sign, nor any indica-
tion for whom the sign has been or will be a prompt. The prompt is crucial to the
making of a new sign. It represents the social characteristics of the interaction with
an addressee, which shapes the prompt.
In communication, the prompt originates with one of the participants in the
interaction. It is a social and a semiotic phenomenon: as a message produced by one
of the participants for another and as a prompt to engage with. Interest provides the
recipient’s motivation for engagement and shapes attention. It leads to a framing
of aspects of the prompt and a consequent selection of elements in that frame for
interpretation according to principles brought by the interpreter. The engagement,
as with the car-sign, is with that which the interpreter regards as criterial. In that
process the participant makes her or his sign “internally,” leading to a reorganisation
of the interpreter’s “inner” resources.
In this theoretical frame, communication cannot be regarded as a matter of
decoding: it always results in the new making of a sign. Communication has hap-
pened when there has been interpretation resulting in the new making of a sign.
Communication is a process of the transposition of semiotic material. An example
from a science classroom in inner London – 13-to 14-year-old women – may help
make the point. The example comes from a series of Biology lessons: the topic was
“plant cells” (for details, see Kress et al. 2001, Kress 2010).
The class has had four lessons on cells. There has been much talk by the teacher;
images have been shown and drawn; an experiment has been conducted. In the
fourth of these lessons, the class is back in their homeroom. The teacher asks: “Can
anyone tell me something about plant cells?” One young woman raises her hand,
and on being nominated says, “a cell has a nucleus, Miss.” The teacher says “Good!
Susie, can you come up to the front and draw what you have just told us?” Susie
walks up to the whiteboard, and Figure 1.3 is (roughly) what she draws. She has
responded to the teacher’s request for a reconstitution of meaning, a transposition: most
immediately a transduction from the mode of speech to the mode of image.
The crucial point here is about mode, modal affordances, and the “drawing on” of
resources for making meaning across modes – the twinned processes of transduction in
transposition. The modes of writing or speech each prompt specific questions about “the
world”: here the relation of cell and nucleus: different to those posed by image. The
epistemological affordances of writing (and of speech, similarly) suggest questions such
as: “How is the relation between the cell and the nucleus characterised?” “Is it as a
possessive (‘a cell has a nucleus’) or as a spatial relation? (‘The nucleus is in the cell’)?”
The epistemological and ontological affordances of image differ from those of speech
or writing. They might prompt a question such as “Where in the cell is the nucleus
FIGURE 1.3 Plant cell

38 Gunther Kress
located?” “What size, relative to the cell, is the nucleus?” Each mode brings specific
epistemological and ontological commitments. It is not possible to draw a cell without
placing the nucleus somewhere within the cell wall. A teacher might not pay too much
attention to this absolutely unavoidable epistemological and ontological commit-
ment, though a student in the science classroom is entitled to regard the teacher’s
placement of the cell as criterial: an indication of certain knowledge.
The distinct epistemological and ontological potentials of different modes are rec-
ognised across different communities. The carpenter’s sketch, the architect’s three-
dimensional model, the police investigator’s drawn reconstruction, the researcher’s
diagrams and transcripts, all “fix” (Kress 2000) a specific “take” on the matter in
focus. Each exploits the distinct potential of modes to provide different insights into
the world framed and in focus. This makes epistemological/ontological questions
an unavoidable issue in the constitution of meaning, and therefore in every act of
transposition. If meaning comes into being as the result of the semiotic work of
interpreting, we might say that “knowledge” comes into being only when it has
been made materially evident. (It is necessary to distinguish my use of “interpret-
ing” here from the specific sense of interpretation in the translation context, that is,
as “impromptu oral paraphrase”). Knowledge in the spoken response “the cell has
a nucleus” is ontologically different from the knowledge displayed in the drawing.
That is a “take” beyond that of translation from one written text to another. It
tends to be resisted by teachers, who say “I’m interested in knowledge; I’m here
to teach Biology, not to worry about drawing or writing or whatever.” The cell
example shows that as learners make signs, they are required to transpose potentials
for meanings to be made in one (or more) modes or ensemble of modes, in one
environment to other modes or ensembles of modes in a different environment:
from laboratory to classroom, to examination room, with pen and paper as tools. It
matters how a teacher labels this practice. Each shift entails transposing the poten-
tials for meaning and reconstituting the meaning in terms of the affordances of the
modes used and the site in which it is done. That is always shaped by the designer’s
interest and agency, the distinct affordances of modes, each with epistemological
and/or ontological commitments. There is no possibility of a “perfect translation”
across modes. Transposition and reconstitution of meaning across modes produces
different “takes” on the world. To recapitulate: the transposition of the potentials
for creating meaning enables meaning to be reconstituted in the new site.
Meanings made in any one (social) site are constantly and frequently remade in
different sites, in different social and semiotic environments. The sociologist Basil
Bernstein used the sociological term re-contextualisation when, for example, a profes-
sional practice involved in the making of an object – a wooden table, let’s say – is
re-contextualised as part of the school curriculum subject woodwork (Bernstein
1996). Re-contextualisation is an entirely usual and unremarked, ubiquitous practice,
in all of social life, in all kinds of social activity.
In the classroom, where meaning is – or ought to be – the issue, the sociological
practice becomes the semiotic matter of transposition of meaning. When a profes-
sional practice is re-contextualised, its corresponding configuration of a multimodal
ensemble is reconstituted, and the resources for meaning-making are transposed. Not
only forms of speech or writing are changed, but many integral practices and objects
are changed, as are their inter-relations. In a not-too-distant past, when the assump-
tion had been that what mattered was to adjust, adapt, and translate “the language”
used in the originating site into “the language” of the target site, not all that much
seemed at issue. There was a question of students being competent in the linguistic
forms of the originating site and learning – gaining competence in – the usually dif-
ferent linguistic forms of the target site. This was understood to be a problem that
had to be dealt with; the solutions differed in line with degrees and kinds of power.
It pays to pause briefly over this example of the re-positioning of a practice and
the transpositioning and new constitution of meaning. In this instance – the pro-
fessional practice of carpentry becoming the school curriculum subject of wood-
work – the forms of speech in the originating site, the sounds, the words – the
syntactic forms let’s say – will differ in some ways from those in the target site.Yet
sounds remain sounds, broadly, even if not entirely the same; so do words, and so
do – by and large – grammar and syntax. That is, the traditional notion of transla-
tion worked on the basis that the constituting entities of the source and the target
language were broadly the same.
Translation – as the transpositioning of meaning resources and the reconstitu-
tion of meaning – also applies to larger textual entities, whether these are treated
as genres or as other larger text categories such as “the novel,” “the website,” “the
opera,” or are seen as quite different kinds of categories, particularly across different
societies. On this issue, see Kaindl’s chapter in this volume (Chapter 2).
7. The transposition of the “undocumented” and of

“tacit” meaning
A third-year medical student and a surgeon are standing at an operating table.
They are about to start operating on a small lump on the patient’s abdomen. As
the patient is lying flat on his back, the lump is not visible. The operating light
is focused on the patient’s navel. Before the surgeon makes the first incision, she
points with her left hand to where the (presently invisible) lump that they will
operate on is located and asks the student if he wants to have “a feel of that.” The
medical student replies “yeah,” lightly touching at three different points around the
focal area with a swab in his left hand. He then “feels” superficially with his right
hand. He holds his hand flat, putting gentle pressure on various points with the
tips of his fingers, covering an area of about three inches below the navel. He also
makes a sweeping movement between two pressure points, as shown in Figure 1.4a.
The surgeon then joins him in “feeling,” using her left hand. Her hand is slightly
tilted, and she creates more pressure with the tip of her fingers, pushing deeper into
the skin of the abdomen below the navel, as shown in Figure 1.4b.
The pressure points mark out and make visible the circumference of the lump.
This is then followed by a grasping action by the surgeon involving her middle
finger and her thumb, which lasts for a couple of seconds.
FIGURE 1.4a Medical student touches patient FIGURE 1.4b Surgeon touches patient
The surgeon’s action – her touch – is different from the student’s. The surgeon’s
touch is more specific, deeper, firmer, involving (the tip of) a flat, angled hand, as well
as a grasping action; the student’s touch is less focussed, more superficial, and involves
(the tip of) his flat hand only.
The different characteristics of their actions signify different engagement with
the “lump” – the world at issue here – and, in that, distinctly different interest and
knowing. The surgeon’s actions are designed to plan the incision: where to place the
first cut with her scalpel. Her actions are guided by prior experience. The student’s
actions are designed to feel the lump; he has had no prior experience to draw on.
The surgeon knows, broadly, what to expect; the student doesn’t. Thus, the actions
of surgeon and medical student point to different resources, to different means of
(embodied) knowing; they demonstrate different histories of prior experience and
knowing. Their actions, as signifiers, are apt to carry the meanings of each: the
signs which each makes are motivated. At the same time, the actions of both lead
to and constitute an extension of experience.With the making of their actions/signs,
their semiotic work – much as a child’s making marks on paper – they expand their
resources for making, transposing, and constituting meaning: in this case, resources
for “reading” lumps of a certain kind and the specifics of the lump of this patient.
In this action and interaction, in the demonstration of experience and inexperi-
ence, both have learned about the other sign-maker: the medical student has inter-
preted how the surgeon touches, and the surgeon has interpreted how the student
touches (Bezemer and Kress 2014).
Transposed to a social semiotic multimodal account, we can say that every sign
made here is new, is an “innovation”; its making is an act of “creativity.” If the
example of the car is an instance of the constitution of meaning, here in the example
of touch, we have an interaction in which each of the two participants interprets the
other’s actions-as-signs. It is an example of the transposition and reconstitution of
meaning by a very different route from those mentioned so far. It takes place in the
student’s interpretation and his tentative “simulation” of the actions/signs of the initi-
ating participant. By the end of this brief segment, the student’s touch has become
different. The signs made by each participant are interpreted by the other, on the
basis of the existing experiential/semiotic resources of each of the two participants.
The point to draw out is that the transposition and reconstitution of meaning –
of all sign-and meaning-making – is always based on the existing resources of
each participant. It does not depend on or apply merely to the canonical means
of documentation: of image, writing, speech. In this process the resources of each
participant change. The meaning constituted by one participant is reconstituted in
the interpretation by the other. In each case this happens on the basis of existing –
and constantly changing – resources, which shape engagement, attention, selection,
transposition, and reconstitution of meaning.
The effects of the interpretation of each other’s sign-making (the constitution
and reconstitution of meaning) have changed the “inner” resources of each partici-
pant. That change in resources will be evident in their subsequent actions and will
shape all future actions. For instance, in subsequent actions, certain features of their
42 Gunther Kress
actions might be highlighted: a foregrounding through slow motion, say, or atten-

tion to the precise direction of a movement. With every sign made, the sign-maker’s
knowing is transformed; this applies to student and surgeon. Neither could not learn
from touching the patient or from observing the actions of the other.
If translation is the process of making knowledge and understanding available and
accessible to someone else by means of semiotic resources which that person does
not yet possess (fully), then the process of demonstration here might well count as an
instance of translation – of the transposition of knowledge.We can say that the recon-
stitution of meaning is an act of translation; or, that translation is the reconstitution of
meaning for specific others; or that translation is the rhetorical act of the reconstitu-
tion of meaning for specific others, with specific (rhetorical) aims in mind.
In the case of touch – the case of the “lump” – the question of the formal docu-
mentation of the entities involved in the reconstitution of meaning did not arise.
For both participants the entities were those of the mode of touch: even though
not named nor formalised, beyond unspoken suggestions/indications about intensity,
angle, direction, density. The entities were observable but were not given other formal
realisation or names. Yet the idea of the reconstitution of meaning exists there as
much as in the reconstitution from one stretch of speech to another, or across modes.
If there was to be a need to include a segment on touch in a textbook or on an online
resource for beginning clinicians, then the question of the transcoding resource
would arise and the question as to what would be apt modes and apt metaphors.
As a last example here is an investigation of “tacit meaning”: it is another exam-
ple from schooling, an ephemeral event in “early years.” In a social semiotic per-
spective, a chasing game, played by say four-year-old boys, is an ephemeral event.
Whether it is going to be considered to be significant will rely on the semiotic work
done and its persuasiveness. The event is deliberate, significant to its participants,
and with significant social and emotional consequences inside this “early years”
setting and beyond. Whether it is worthy of attention as an instance of meaning-
making, such as drawing, writing, or making three-dimensional models are seen to
be, depends on whether semiotic means can be found to demonstrate its pedagogi-
cal significance. It is, in that sense, a question of translation: the documentation of
meaning and the transposition of that meaning into a pedagogical environment.
In preschool settings – and not only there – meanings made beyond “canonical
means” are often overlooked; or, as in this case, disregarded as trivial: “boys just run-
ning around.”
A major problem is methodological: how can we document, “fix” (Kress 2000)
the “goings on,” to provide useable, reliable documentation, a record, of a running
game, which will make it possible to analyse the meanings made and thereby widen-
ing the scope of what can get recognised as meaningful in young children’s meaning-
making (see also Bezemer and Mavers 2011; Cowan and Kress 2017). In playing this
game, children use a range of “ephemeral” modes, such as gesture, gaze, speech, features
of movement; that is, all semiotic modes which leave no material traces.Yet, as poten-
tial data, these can constitute a rich set of materials. The meanings made in the
game belong to, or in, the community in which the game exists. The multimodal
character of the game rules out attempts at a traditional written “transcript,” even
if that were an apt means of (re-)constituting its meaning. A written vignette is not
a reconstitution of meaning in the sense so far discussed. A vignette cannot depict
the children’s many rapid, changing movements and their complex use of the space:
a vignette is a commentary rather than a reconstitution of the meaning. The prob-
lem is to identify, to document the meanings, the signifieds, and to decide what
material means can serve aptly as signifiers.
Some of the material features here which are giving material realisation to social
signifieds are speed and control of speed, direction and control of direction, distance, and
bodily position. Socially, the game is about a hierarchy to be established via the social
signifieds; semiotically, the material signifiers are spatial and temporal in various ways.
In her study, the researcher Kate Cowan chose a map-like “genre” of layout as
an apt frame – the “ground” – for documenting the salient features of meaning of
a game of chase: making use of spatial-visual layout and colour to give approxima-
tions of the modes used by the children.
Figure 1.5 represents a 30-second extract of George and Billy’s game of chase,
offering a bird’s-eye view of part of the outdoor play area. Lines represent the
movement of each child; the arrows along the lines indicate the direction of their
running; the arrows are positioned at approximate one-second intervals to show
when the children were moving slowly (the arrows being closer together) or at
speed (the arrows being further apart). Whilst this can represent distance, direc-
tion, and speed, it less adequately represents sequentiality, and so multiple “fixings”
were produced for the different stages of the game. As stillness was of significance
as well as action, pauses were incorporated into the design using a circle with a
FIGURE 1.5 “Fixing” the ephemeral: documenting meaning in a game of “chase”

44 Gunther Kress
number denoting the time (in seconds) that the children were still. The children’s
talk was incorporated by locating speech bubbles at the relevant points along their
movement “path.” The notational resources used were carefully selected on the basis
of their aptness as means for documenting the most salient features of the episode.
The task undertaken by the researcher here is to construct an apt means of
documenting the social and semiotic event that had taken place. It is clear that
the tools for documenting the meanings – the signifiers – that had been made are
metaphors: for instance, distance covered as a means of indicating speed. We might
note as a general point that the resources which might serve for documenting the
reconstitution and transposition of meaning, in this instance, are material metaphors
of the signifieds which are at issue. That formulation, however, also applies to the
(various) commonly used practices of “transcription,” for instance, where sounds
of speech (treated as phonemes) are represented by letters of the alphabet (or various
often complex supplements to the alphabet, such as dots, dashes, and lines of vari-
ous kinds to indicate further aspects of spoken interaction). Or in other forms of
dealing with speech, where “words” (as signifiers) are taken as the basic units for
documenting the reconstituting of meaning.
The “documentary fixing” (Kress 2000) of ephemeral aspects of meanings
of the game did foreground and make visible certain features that had not been
immediately obvious from the original video recording. The shift of perspective,
as much as the act of mapping itself, draws attention to patterns and points of
interest, enabling scrutiny of movement and use of space as a crucial aspect of the
play. In this way, the process of notation supports certain noticings. The mapping
highlights how the rules of the play are communicated through subtle combina-
tion of movements, including decreasing speed, changing direction, and keeping
other participants at a distance through an outstretched arm gesture. The play
is successfully “wound up” by one participant before they discuss the reason for
stopping, which ensures avoiding being caught and surrendering. In this way, the
chase is paused and suspended, subtly communicating the message of “truce”
through multiple ephemeral modes.
In this example the reconstitution of meaning happens across two communities:
that of the children at play in the formal setting of “early years” schooling and that
of the researchers puzzling about the meanings of the children’s actions, seen as
semiotic work. It is an instance of “noticing,” documenting, and transposing meaning.
8. “Naming”
The world of meaning and of the making of meaning is in the process of a pro-
found change: a paradigm change from a focus on language to a focus on meaning;
a change from a focus on one to many means of making meanings evident; and a
shift in disciplines from linguistics to semiotics. If that is the case, then the terms which
named the former world cannot deal with the new conditions and practices. Many
of the disciplines, still in use, were developed to deal with problems in a social world
that was entirely differently conceived: “stability” might be a short-hand means to
refer to that; while “provisionality,” “fragmentation,” and “diversity” might be terms

that are apt for the present time. A central part of notion of stability was the role
of language, as one central guarantor of meaning and of safe exchanges of meaning.
For the issue at hand here – translation – the change away from “stability,” cou-
pled with a concurrent move from an assumption of one means to many means of
making meaning evident, is having far-reaching consequences wherever meaning is
the issue: in both theoretical and practical terms. The names that served the former
order entrenched its theoretical notions, propping up the edifice of stability with
great effect. The need is for names – a coherent terminology – which captures and
aptly expresses the theory and its elements, practices, and consequences.
Social semiotics, the theory proposed here to deal with “the issue at hand,” has
“process” in the form of ceaseless semiosis at its core. The three central issues in sign-
making – interest, criteriality, and the newly made motivated sign – are crucial in
naming and in doing away with the misleading view of the “transport of meaning.”
As the choice of an apt signifier for a signified is based on the interest of the sign-
maker, each sign that has been made leads to a transformation of the signifier that
was used. From that perspective there is no possibility of stable meanings. As the
signified is shaped by the always varying conditions of each prompt and of what is
seen as criterial at that point, there is no stability of the signified. The condition of
the always new making of the sign, both in the initial sign-maker’s production and
in the interpretation by the addressee, means that the idea of stability in semiosis is
unfounded and a problematic distraction.
However, as the existence of a plurality of modes available to each social group is
now a commonplace, it means that the practices and processes of changes in mean-
ing have become everyday issues: whether banal or profound, within and beyond
social groups. Apt naming has become essential and urgent.
Different criteria and principles play their part in naming. To start with the term
proposed in this chapter, transposition: it responds to two aspects of the contempo-
rary social and semiotic world. One, meaning is always made in social interaction, in
a social environment. There is a proliferation and diversity of social sites. “Position”
is crucial in sign-and meaning-making. Prompts (as the material means of making
meaning evident) are constructed initially in a specific social site and are interpreted
in a different site. These are social aspects of “position,” and the morpheme “trans-”
signifies a change in these aspect. Two, there is a recognition of the multiplicity of
modes – resources for making meaning evident – each with specific affordances
and with accompanying epistemological and/or ontological commitments. The
choice of modes represents a semiotic positioning, arrived at by the communicational
requirements both of the initial maker of a sign or sign-complex and the interpreta-
tive semiotic work of the person who engages with the prompt.
Meaning is made in each position, both by the initial maker (of the text as
message) and by the interpreter of the message. For both there is the sequence of
interest → attention to part of the world in focus → motivation for (principled) →
selection of that which is (either) to be → produced or interpreted. The difference
between the two positions is that the initial maker sets the frame and the contents
46 Gunther Kress
of the frame, whereas the interpreter selects from the prompt that which is to be
framed and what from within that frame is to be interpreted. That is, the initial
maker has a wider range of choice: she or he sets the agenda, so to speak.
Transposition names the fact of a change of “position” – both of the social and the
semiotic positions. It indicates that the site where meaning is made has been changed;
there has been a change in that. Transduction names the process of change of mode;
though transduction does not indicate which modes have been involved. Transduction
names a process in which ontological change takes place. That is, the source and
the target mode have entities of a different kind: whether in terms of the logics (of
temporality or spatiality) or of “substance”: e.g., the syntactic relation of two or more
elements versus the spatial, locative relation of two or more elements (e.g., centrality,
adjacency, marginality).
Transcription, by contrast, names the manner and the means of the relation of
two modes: the relation of speech (sound) and writing (script) in societies that use
an alphabetic script. The alphabet is not an ideal resource for the transposition of
meaning: the relation of phonemes (the theoretical generalised entities of speech)
and graphemes/letters is always metaphoric and sustained by the power of conven-
tion. The term transcription is used elsewhere, as in the mode of music, in a manner
analogous to the relations of speech and writing: i.e., the transcription of an orchestral
piece to a piece for solo instrument.
While transduction is used for transposition where two modes are involved (the
entities of the one are unlike those of the other), transformation names a process
of change within one mode and its elements. Transformation involves changes in
ordering, e.g., of deletion and addition. It also involves changes from one textual –
generic – entity to another; for instance, the change of a report to a narrative.
Several of these distinct processes may occur in multimodal complexes; where
one modal element may be transducted, another may be transformed, and yet oth-
ers remain, in terms of mode. The social environment may be changed from one
occasion to another, leading to transposition in addition to modal changes.
9. Conclusion
On the one side, interlinguistic, intralinguistic and intersemiotic translation

pointed out by Roman Jakobson (1971) are obviously describable on the
basis of a single translation process model. On the other side, all types of com-
munication in culture could be presented as the process of translation of texts
(or fragments) into other texts.
(Torop 2003, 271)
What was the case in 2003 is clearly much more the case now, at a time when mul-
timodality has become widely noticed and widely adopted, albeit in theoretically
pretty well unconstrained ways.
Some things are clearer now. As I have tried to imply, the notion of “interse-
miotic translation” finds no grounding in social semiotic theory: it assumes that all
meanings made in one society are dealt within the one theory. The fact that mean-
ings appear in many ways is dealt with by the semiotic category of mode: differently
made evident in image or gesture or writing, but made evident in the categories
of the one theory.
My account is the merest sketch: I have not, for instance, concerned myself
with larger semiotic (and social) entities such as “the novel,” “the film,” or “the
opera.” The problem of naming is not restricted to the domain of translation: it
affects all of the disciplines in the Humanities and Social Science, and no doubt
beyond. One problem is the disappearance of formerly reasonably firm bounda-
ries.When scholars from mass media studies discuss a notion such as “media” with
scholars from literary or social or cultural studies, the debates tend to be rehears-
als of “how we have always used that term.” I feel confident that the principles
sketched here would – in more expanded and detailed form – provide resolutions
to many of these questions.
I remain puzzled by how one might translate semiotic entities such as cartoons
produced over the last years by the English cartoonists Martin Rowson or Steve
Bell. The social conditions in which they are produced would be the first point
of investigations. One might find affinities with the cartoons, drawings, and etch-
ings of William Hogarth or the paintings of Hieronymus Bosch. That would hint
at certain regularities in the relation between the social and the semiotic. What
could be adequate transpositional resources? In a different vein, there is the ques-
tion of “What is and what is not translated? The libretto of the Mozart opera but
not the score; the poems of the Schubert song cycle but not the music?” There
are contemporary stagings, for example, of Wagner’s Ring as a feud in a capitalist
industrial family, while the libretto or the musical score are left untouched. Helen
Julia Minors reflects upon some of these issues at greater length in her chapter for
this volume (Chapter 7).
These, along with many other issues, will need to form the agenda of the next
two decades or more. The question “what is translation” has, if any answer, an
institutional one. The real issue looming is not technology but the future shape of
what we call “the social.”
Acknowledgements
I wish to thank the organisers of the CRASSH conference (Cambridge, July 2018)
for inviting me to speak and the reviewers of my draft paper for helpfully difficult
comments and questions. My colleagues Kate Cowan (who gave me permission
to use Figure 1.5) and Jeff Bezemer (who gave me permission to use Figures 1.4a
and 1.4b) have been constantly generous in allowing me to share aspects of their
grappling with their research. Last but not least, thanks to Prue Cooper, the MC
at the event I recount, for great kindness, generosity, and stimulation. I alone am
48 Gunther Kress
responsible for the awkwardness in attempting to express what is still not clearly
worked out and trying to be clearer in getting a grasp on this topic.
References
Bernstein, Basil. 1996. Pedagogy, Symbolic Control and Identity: Theory, Research, Critique. Lon-
don: Taylor & Francis.
Bezemer, Jeff, and Gunther R. Kress. 2014. “Touch: A Resource for Making Meaning.” Aus-
tralian Journal of Language and Literacy 37 (2): 77–85.
———. 2016. Multimodality, Learning and Communication: A Social Semiotic Frame. London and
Bezemer, Jeff, and Diane Mavers. 2011. “Multimodal Transcription as Academic Practice:
A Social Semiotic Perspective.” International Journal of Social Research Methodology 14 (3):
191–207.
Burn, Andrew. 2014. “The Kineikonic Mode: Towards a Multimodal Approach to Image-
Media.” In The Routledge Handbook of Multimodal Analysis, edited by Carey Jewitt, 375–85.
Cowan, Katharine, and Gunther R. Kress. 2017. “Documenting and Transferring Meaning in
the Multimodal World: Reconsidering ‘Transcription’.” In Remixing Multiliteracies: Theory
and Practice from New London to New Times, edited by Frank Serafini and Elisabeth Gee,
50–61. New York and London: Teachers College Press.
Gibson, James J. 2014. The Ecological Approach to Visual Perception. London and New York:
Routledge.
Halliday, M. A. K. 1978. Language as a Social Semiotic. London: Edward Arnold.
Hodge, Robert, and Gunther R. Kress.1988. Social Semiotics. Ithaka: Cornell University Press.
Jakobson, Roman. 1971[1959]. “On Linguistic Aspects of Translation.” In On Translation,
edited by Reuben A. Brower, 232–39. Cambridge, MA: Harvard University Press.
Kress, Gunther R. 1997. Before Writing: Rethinking the Paths to Literacy. London and New
York: Routledge.
———. 2000. “Text as the Punctuation of Semiosis: Pulling at Some of the Threads.” In
Intertextuality and the Media: From Genre to Everyday Life, edited by Ulrike H. Meinhof and
Jonathan Smith, 132–53. Manchester: Manchester University Press.
———. 2001. “You’ve just got to learn how to see: Curriculum subjects, young people, and
schooled engagement with the world.” Linguistics and Education 11 (4): 401–15.
———. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. Lon-
don and New York: Routledge.
Kress, Gunther R., Carey Jewitt, Jon Ogborn, and Charalampos Tsatsarelis. 2001. Multimodal
Teaching and Learning:The Rhetorics of the Science Classroom. London: Bloomsbury.
Torop, Peeter. 2003. “Intersemiosis and Intersemiotic Translation.” In Translation Translation,
edited by Susan Petrilli, 271–82. Amsterdam and New York: Rodopi.
van Leeuwen, Theo. 2005. Introducing Social Semiotics: An Introductory Text. London and New
York: Routledge.
2
A THEORETICAL FRAMEWORK FOR
A MULTIMODAL CONCEPTION OF
TRANSLATION
Klaus Kaindl
1. Introduction
Multimodality has become a widely used buzzword in many disciplines. As dis-
cussed in the Introduction, translation studies reacted to this trend – albeit hesi-
tantly, at first – not least due to the booming field of audiovisual translation, which
evolved from a relatively neglected subject area to a central research focus of the
discipline (see the chapter by Luis Pérez-González in this volume). However, this
may at times give the impression that multimodality serves merely as a cosmetic
means of concealing a continued focus on linguistic aspects in translation studies.
In this regard, translation studies are still a long way from a semiotisation, which,
according to Berressem (2004, 219), is a general trend in science.1
The aim of this chapter, firstly, is to situate multimodality in the semiotic research
tradition with regard to its novelty value. Secondly, it will outline the efforts of
translation studies to develop a multimodal understanding of text and, thus, also of
the discipline. Based on this, the added value of a multimodal approach to transla-
tion studies and its practical implications for the definition of translation will be
discussed. Finally, a multimodal approach to translation analysis will be presented
through the examination of a rock ‘n’ roll song by Elvis Presley.
2. Multimodality as academic rock ‘n’ roll

Multimodality appears to have replaced outdated semiotics and is often regarded
as a leap forward compared to old sign-theoretical approaches that emerged from
semiotics. In translation studies, too, the mode concept seems to have forced out
the semiotic sign concept. Since the semiotic dimension, which was introduced
into translation studies by Jakobson (1959), was to play a central role, and an aware-
ness of former theoretical and conceptual contexts of the scholarly debate is key to
50 Klaus Kaindl
meaningful progress, the consistencies between semiotics and multimodal discourse

as well as the historical development, which is largely neglected by Kress and van
Leeuwen (2001), will be outlined first.
Progress in science can be determined by various criteria: it can indicate an
increase in scientific knowledge, provide solutions to problems, or give us a new
view of a section of reality, e.g., an object of research.2 However, some theorists
are also sceptical about such a linear progression of scientific advancement. For
example, Thomas Kuhn, in his work The Structure of Scientific Revolutions (1970),
questions the view that scientific progress can be described as a cumulative gain in
knowledge.
To clarify the question whether the concept of multimodality has a novelty
value, and, if so, what it consists of, I will adopt a quite unorthodox scientific-
theoretical analogy and use the musical genre of rock ‘n’ roll as illustrative material
for a multimodal translation analysis in the further course of this chapter. Rock ‘n’
roll was, as will be described in more detail later, a genre that shook the conservative
1950s. It was perceived as something completely new and revolutionary. De facto,
however, it was not a new form of music, but merely a combination of different
musical styles – elements of “white” country music, at least in rockabilly style, for-
mal and melodic characteristics of “black” rhythm and blues as well as balladesque
elements. Therefore, the novelty did not consist of some creation ex nihilo, but of a
combination and re-evaluation of individual, already existing musical components.
The concept of multimodality presents a somewhat similar case. Since its intro-
duction by Kress and van Leeuwen (2001), multimodality has been regarded as a
new way of understanding the semiotic complexity of texts. However, the indi-
vidual components that make up multimodality are not entirely new. In a way,
Kress and van Leeuwen have created academic rock ‘n’ roll: they combined and
reassessed already existing knowledge, primarily from semiotic studies and interme-
diality research. The following brief outline of the history of multimodality is not
intended to diminish the merits of Kress and van Leeuwen, but to acknowledge the
previous achievements of the semiotic tradition and to clarify what the contribu-
tion of the multimodal approach ultimately consists of. In a sense, such a critical
appraisal follows a fundamental principle of semiotics, which, according to Julia
Kristeva (1968/1977, 38), lies in viewing one’s own research subject not as a static
truth, but rather as a never-ending research process that must be constantly reflected
upon and critically re-evaluated.
The cornerstones presented by Kress and van Leeuwen in their book Multimodal
Discourse (2001) – the social contextualisation and social use of modalities, modes
as a result of cultural processes, the functional entanglement of modes, the close
relationship between mode and medium – are all aspects that were already discussed
in semiotics in the 1970s, albeit under the heading of multimediality. This can be
traced back to Charles Sanders Peirce, who pointed out the limitations of sign
theory much earlier and suggested to place it in a medial context: “All my notions
are too narrow. Instead of ‘Signs’, ought I not say Medium?” (Peirce 1906, cited in
Walther 1997, 79).
A multimodal conception of translation 51
When Roland Posner (1976a, 39) noted that the theoretical prerequisites for
multimedial (in today’s sense of multimodal) communication were missing, a num-
ber of semioticians took it upon themselves to develop building blocks for such a
theory in the following years. Posner himself provided an outline of the key issues
for semiotic studies, which he saw primarily in developing sign systems, which are
not viewed as neutral, static units, but rather as a means of communication that
perform various functions (cf. 1976b, 2). Hence, Posner prefers to use the term
“Zeichenverhalten” (“behaviour of signs”) (1976a, 25) rather than “signs,” in order to
emphasise their dynamic and changeable nature. According to Posner, signs do not
only change synchronously – with regard to the purpose of communication and
the respective context in which they are used – but also diachronically through a
change of usage over time. Thus, if the multimodal approach explicitly distances
itself from a structuralist view of the sign concept (e.g., Leeuwen 2005, 47–48), it
only follows those trends in semiotic studies that already departed from a static code
concept and a representationalist conception of signs in the 1970s.3
For Posner, signs are also always bound by their “relationship to the medium.”
According to him, the associated sensory modalities and the communication chan-
nels used cannot and should not be examined separately. It is only through the
“mutual relations” (1976a, 25, my translation) between sign and medium that the
sign behaviour can be adequately analysed. The development of a corresponding
terminology for exploring the functions of signs in relation to their media remained
a desideratum for Posner: “Moreover, the relationship between code, channel and
modality is still unclear” (1976a, 39, my translation).
Ernest Hess-Lüttich, among others, took up Posner’s suggestions and designed
a research programme in 1978, which incorporated what would later become the
central components of Kress and van Leeuwen’s multimodal discourse. His aim was
to create a research framework that goes beyond semiotic analyses in various fields,
such as film studies, musicology, art and literature studies, sociology, and psychology,
and would enable the development of an interdisciplinary notation and analysis
system. In that framework, multimediality incorporates the medium as a textual
means in the broadest sense, social processes, sign systems, their technical distribu-
tion, and institutional systems for the realisation and organisation of medial prod-
ucts. Hess-Lüttich emphasises the “communicative-functional conditions of social
interaction” (1978, 38, my translation) as being the decisive point of reference for
the analysis of how the various semiotic resources and medial realisations are con-
nected. Similar to Kress and van Leeuwen, he views the functional entanglement
of semiotic resources and their embedding in social action contexts as the central
starting point for research into multimedial or multimodal communication.4
The fact that these functional relationships are not static but dynamic as well as
culturally specific, as Kress and van Leeuwen emphasise, was also recognised and
incorporated very early in semiotic studies. Kloepfer, for example, in his analy-
ses of text-image relationships in various medial contexts, shows that these are
“dynamic, polyfunctional and also serve to change the underlying codes” (1976, 42,
my translation). The same also applies to the role of culture in shaping the meaning
52 Klaus Kaindl
of semiotic resources, which van Leeuwen seems to miss in traditional semiotic

approaches to sign systems (cf. 2005, 160). Umberto Eco noted as early as 1972
that the meaning of a sign is not derived from its relationship to the object, but
is the result of perceptive and cognitive processes, which, in turn, are influenced
by culture. Eco illustrates this with the example of a zebra, which in our culture
is primarily identified by its stripes, whereas in another culture, where it is more
important to distinguish a zebra from a similarly striped hyena, it is identified by its
shape, which for us resembles that of a donkey (cf. 1972, 202).
The embedding of the mode concept into a social-communicative context, as
suggested by Kress and van Leeuwen, was thus already postulated and developed in
semiotics in the 1970s, but perhaps not consistently applied. Multimodality research
also shares its goals with semiotic studies, which, according to Jewitt (2014a, 17–20),
consist of describing semiotic resources and investigating their relations and the
impact of media. Nevertheless, multimodality offers a novel approach due to the
combination of concepts, not so much on account of the terminology,5 but because
of the differentiation of its modal components. The four dimensions of mode –
discourse, design, production, and distribution – sharpen our view of the different
dimensions of signs and modes, which enables us to grasp the functional interac-
tion between the semiotic resources and their medial realisations more precisely.
At the same time, consistent socio-cultural embedding brings into focus not only
the dynamics of how meaning is constructed, but also the fact that such changes do
not occur arbitrarily but according to socially established rules. Thus, a frequently
neglected factor in semiotics takes centre stage: power in Foucault’s discursive-
political sense as an integral component of signs or modes. In contrast to semiotic
approaches, the question of how mode and social identity are connected can also be
answered with the sociological inventory of multimodal discourse.
Therefore, the emphasis on the sociological, functional, and medial aspects of the
multimodal approach makes it so interesting for many disciplines, including transla-
tion studies. As previously discussed, these factors were already present in applied
semiotics in the 1970s, but translation studies was still far too strongly tied to the
linguistic paradigm at that time, which resulted in a merely rudimentary integra-
tion of semiotic considerations into the scholarly debate. While the text concept
remained static for a long time, particularly in translation studies (cf. Kaindl 2013,
259–60), the multimodal approach captures its functional dynamics. These dynam-
ics result from the cultural and sociological variables and the respective forms of
medial realisation. The fact that the shift in translation studies towards culturally
sensitive, sociologically motivated, and medially conscious research coincided with
the development of the multimodal approach may be one of the reasons why trans-
lation studies is increasingly incorporating it into its scholarly investigations.
The paths to scientific progress are manifold and not always straightforward.
The present case illustrates that progress does not necessarily happen cumulatively
by building on what already exists. Rather, Karl Popper’s view of progress applies
here, which – in reference to Darwin – occurs as a result of natural selection (cf.
1998, 273). Kress and van Leeuwen provide a modern take on a well-known
phenomenon by combining and re-evaluating previously researched aspects, which

shows that a scientific approach is ultimately more than the sum of its parts. As a
next step, I will critically examine the possible consequences of this holistic view
for translation research.
3. On the (bumpy) road to multimodal translation studies

Translation is – and this it probably shares with mode – a colourful phenomenon,
which appears to elude clear, universally valid definitions. Definitions, however,
are often considered as a prerequisite for scholarly research or, to put it another
way, a necessary evil. An evil, because definitions – at least of cultural and social
phenomena – are not neutral, innocent acts but attempts to “discipline” an object
of research, which is always guided by particular interests. In the case of translation,
these disciplinary measures are probably also one of the reasons why translation
studies struggle with multimodality. Its monomodal orientation can be explained
with the emergence of the discipline after World War II. The initial focus of
machine translation in the 1950s was on language systems and the question of how
to create the greatest possible invariance between word and sentence units of two
languages (cf. e.g., Bar-Hillel 1960). Roman Jakobson’s realisation from 1959 that
we communicate not only with language, but also use other communicative means
to make ourselves understood, went largely unnoticed at the time and only found
resonance in translation studies at a much later point. In particular, the integration
of audiovisual translation into translation studies, which began in the 1990s, has
made a significant contribution (e.g., Delabastita 1989; Gottlieb 1994).
As the summary in the Introduction indicates, translation studies initially
remained a purely language-centred discipline that opened up to text linguistics
in the 1970s and, thus, increasingly included pragmatic aspects in its research. The
sender and his/her intention, the recipient and his/her level of knowledge and
understanding, as well as text types and their conventions, now dominated the
discussion. Rigid demands for equivalence (e.g., Catford 1965; Kade 1968), which
aimed at producing the greatest likeness of a linguistic unit from one language to
another, were now bent to ensure that the effect of source and target text would
be as similar as possible on the recipients (e.g., Nida and Taber 1969; Reiß 1971).
However, the fact that in text linguistics and semiotically oriented linguistics, texts
were seen not only as linguistic but also as non-linguistic units was – as we will see
later – only hesitantly integrated in translation studies.6
For a long time, translation studies remained a discipline whose focus was on
the transfer of meaning from one language to another.7 When the so-called suc-
cess story of translation studies (Bassnett and Lefevere 1990, ix) began in the 1980s
and 1990s, this was primarily due to an expansion of research interests to include
the cultural and sociological dimensions of translation. Translation was no longer
merely a transfer of linguistic meaning, but a multi-layered transfer of culture, ide-
ology, power, gender, race, etc.; and the focus of translation research was no longer
on the source text as the main point of reference, but rather on its functioning in
54 Klaus Kaindl
the target culture. These new perspectives, be they the descriptive, culture-sensitive
approaches of Toury (1995) and Lefevere (1992), or the functional, goal-oriented
approaches of Vermeer (1986) and Holz-Mänttäri (1984), generated an awareness
for previously largely neglected areas such as audiovisual translation, the translation
of theatre plays, illustrated children’s books, and, at the turn of the millennium,
websites, video games, etc. For a long time, all these texts were excluded from the
translation studies research canon, as they could not be examined using source-text
focused equivalence criteria. Moreover, the monomodal view of texts did not allow
for a high proportion of non-linguistic text components or for medial realisation to
influence the composition of texts.
Nevertheless, the verbal fixation continues to have a firm grip on translation
studies to this day, and the expansion of translation theory and analysis tools to
other modes and media is taking place rather hesitantly and in an unfocused man-
ner. Therefore, building on existing approaches, the following section will present
basic principles that delve deeper into the multimodal reality of communication
and hence also translation.
3.1. Building blocks of a multimodal theory of translation

The elements of a multimodal communication theory, as defined by Kress and van
Leeuwen, represent an important basis for exploring multimodality in translation
studies and can be related to communication-sensitive and culture-sensitive transla-
tion theories. A theoretical approach in translation studies, which has numerous
relations to Kress and van Leeuwen, is Holz-Mänttäri’s translation theory (1984),
based on action theory. Similar to the role of human beings, “their social agency” pre-
senting “a criterial aspect” in multimodal theory (Kress and Jewitt 2003, 9), Holz-
Mänttäri regarded translation as an activity where different actors participate in
developing a text. Holz-Mänttäri emphasised – similar to Kress and van Leeuwen –
the design character of translation. Translation cannot be reduced to the transfer
of linguistic meaning, but it is designing texts across cultural barriers. However, the
aim of a translator as a text designer is not to understand the text himself/herself,
but to produce texts for the needs of somebody else (Holz-Mänttäri 1993, 303).
The design of texts across language and cultural barriers needs a specification for
production; this is negotiated in the interaction of different actants who act as a part
of a social complex of actions. Now, the source text is not the determining factor
anymore; it has been replaced by the function of the translation in the target culture,
which depends on the actual context of use as well as the expectations and the level
of knowledge of the target audience.
Translators are normally specialised in the transfer of verbal texts, so the mul-
timodal design of texts often requires them to work with other experts like pho-
tographers, composers, graphic designers, etc. Concerning the production of
design texts, Holz-Mänttäri explicitly referred to their multimodal character and
called them message conveyor compounds (“Botschaftsträger-im-Verbund” 1984,
76). The compound character of the different modes exactly corresponds to the
functional relation, as Kress and van Leeuwen pointed out. Thus, in a technical
manual, for example, the visual mode as well as the linguistic mode can take the
explanation of the operating steps.Whether the communicative function is fulfilled
by images or by linguistic explanations is, on the one hand, culture-specific, and on
the other hand, depends on the production context.
Kress and van Leeuwen as well as Holz-Mänttäri clearly pointed out that the
individual steps of text production – from discourse and design to production and
reception – determine which modes are used in which combination to achieve a
communicative aim. While Kress and van Leeuwen pointed out the characteris-
tics of multimodality in their theory with the transcultural aspect hardly playing a
role, Holz-Mänttäri above all investigated the steps of actions which are relevant
for producing multimodal texts across language and cultural barriers. Thus, both
theoretical approaches can additionally be related to each other for perceiving mul-
timodality in translation studies.
Translation studies, with its focus on transfer processes of modes across modal
and cultural boundaries, can help identify the specifics of modes and their functions;
and Kress’s chapter for this volume (Chapter 1) sketches a multimodal framework
for this kind of analysis. A transfer is only possible if there are enough similarities
between modes for them to be comparable and, thus, transferable, and if there is a
sufficient difference that makes the transfer necessary. Therefore, translation gives us
an insight into what modes have in common and what differentiates them.
The contribution of a theory of multimodal communication for translation stud-
ies is the specification of the modality notion, which Holz-Mänttäri mentioned,
but did not elaborate comprehensively. In addition, it can end the conceptual con-
fusion that prevails in this area of translation studies. The varied terminology that
has emerged since Katharina Reiß’s introduction of the subsidiary, then audio-
medial, and finally multimedial text has prevented a coordinated debate in transla-
tion studies based on a common definition.8
A multimodal understanding of translation requires that not only language and
its functions/functioning, but also other central modalities such as music and image
as well as the associated submodalities9 must be examined with regard to their
functions and cultural specificity – and, in their different ways, the chapters in this
volume by Elisabetta Adami and Sara Ramos Pinto (Chapter 3), Matthew Reyn-
olds (Chapter 5), Marcus Tomalin (Chapter 6), and Helen Julia Minors (Chapter 7)
all explore various complexities that arise from this process of examination. How-
ever, the majority of translation research dealing with nonverbal modalities regard
them as an obstacle to translation work, rather than an integrative component of
a functional text. Despite the fact that recent scholarly works incorporate multi-
modality as a basic principle, Jewitt’s (2014b) differentiation of various multimodal
approaches shows that the potential offered by multimodal theory has not been
fully exploited to date.
It is fair to say that the majority of translation research falls into the category of
social semiotic modality. According to Jewitt, the focus here is on different cultural
and social contexts in which multimodal meaning-making takes place. Numerous
56 Klaus Kaindl
works (e.g., Gambier 2006; de Pedro Ricoy 2012; Jiménez Hurtado and Gallego
2013; Borodo 2015; Cross 2016; Oittinen, Ketola, and Garavini 2017) adopt the
term mode merely as a substitute for the outdated sign concept and neglect the
other dimensions associated with mode as well as the social conditionality of modes,
which ultimately leads to a rather rudimentary, not to say deficient, reception of
multimodal communication theory in translation studies. Despite the fact that
Munday (2014) argued that translation analyses of advertising should be increas-
ingly examined from a multimodal point of view, theoretic scholarly work in this
area is still very scarce. Among the few works are Ketola (2016), who developed a
framework for the cognitive dimension of multimodal translation; Lee (2012), who
examined modes with regard to their technological interconnectedness; and Pérez-
González (2014), who argued for an integration of multimodality into translation
and interpreting studies beyond the semiotic nature of modes.
The other two approaches identified by Jewitt can hardly be found in translation
studies at all or only in homeopathic doses. Exceptions include Baldry and Tay-
lor (2002), who employed the second approach, multimodal discourse analysis, to
identify functional units for subtitling. The third approach, multimodal interaction
analysis, which would be particularly useful for interpreting situations, has hardly
received any attention either, with very few exceptions (e.g., Krystallidou 2016).
One of the characteristics of Kress and van Leeuwen’s multimodality concept
is that mode is seen not only with regard to its semiotic nature, but also as insepa-
rably connected to the medium. This dimension is particularly central to transla-
tion studies. Media are relevant for translation studies in two respects. On the one
hand, media, like modes, are not essentialistically predetermined entities but rather
culturally constructed mediation devices that influence both the selection and the
functioning of modes. On the other hand, transfers do not cross only mode bound-
aries but also media boundaries, and the associated processes still remain relatively
unexplored in translation studies.
Since McLuhan described media as “living vortices of power” (1950, vi), which
shape and change our society, a lot has happened in research. His view of media as
being “extensions of man,” as indicated by the subtitle of his book Understanding
Media (1964), would probably be inappropriate from today’s gender perspective,
and the equation of medium and technology also seems a little too narrow. How-
ever, the fact that media are more than just a means of conveying certain informa-
tion and that they also shape communication is still valid today and has increasingly
been put into focus by intermediality studies.10
Like mode, the term medium is used by different disciplines in various theoretical
and practical contexts and is, therefore, defined and understood in a variety of ways.
Its ubiquitous presence makes efforts for a uniformly binding definition seem futile.
What is required, however, is a coherent conceptual clarification for the respective
approach.
Kress and van Leeuwen define media as “the material resources used in the pro-
duction of semiotic products and events, including both the tools and the materials
used” (2001, 22). However, a focus on the technical-material dimension of media is
too narrow, as Elleström (2010a, 5) notes. Furthermore, he identifies “basic media”

and “qualifying aspects of media” as core facets of the media concept, which,
according to him, are central to a substantiated scholarly discourse. The former
defines the basic characteristics of the medium, and the latter includes the historic,
social, cultural, aesthetic, and communicative dimensions of a medium.
The media spectrum encompasses not only production and carrier media, such
as musical instruments, pens and pencils, cameras or books, computers, newspapers,
etc., but also art forms such as films, opera, theatre, etc. Media differ, according to
Elleström (2017, 668–69), not only in their material but also sensorial and spati-
otemporal qualities. It appears crucial to me that media not only have a material
and social aspect but also a semiotic and cultural one. The material dimension of a
medium, e.g., the page of a glossy brochure or of a trashy magazine, creates expec-
tations in the recipient that can decisively influence the constitution of meaning.
The spatiotemporal and thus also sensorial differences between the book medium
and the theatre medium influence the choice, arrangement, and perception of the
modes used.
Moreover, Elleström (2010b, 26) points out that a medium is also closely con-
nected to the genre.11 The medium of film consists of the genres comedy, thriller,
science fiction, etc. If we assume that every text is already multimodal (cf. Iedema
2003, 40; Gambier 2006, 6, among others), then this also means that genres per se
are multimodal.
Derrida (1980, 65) already noted that every text is associated with one or more
genres. Genres are made up of a number of conventions, which include topics,
characters, character constellations, content, narration, as well as the discursive and
formal use of modes. They determine what readers expect, how they approach and
interpret the text. Like modes and media, they have constructive characteristics and
are culture-specific, or as Elleström (2010b, 26) puts it: “A genre cannot be circum-
scribed as an abstract entity without considering how both ‘form’ and ‘content’ are
related to both aesthetic and social changes.”
Mode, medium, and genre are three building blocks that form the basis for a
translation-theoretical approach that serves to overcome the language-centredness
of translation studies and to understand translation as a modal, medial and generic
practice. It must be noted, however, that mode, medium and genre are part of a
closely interwoven conceptual network (see also Ryan and Thon 2014, 9–10).12
Their separation may have heuristic value on a systemic level, but in translation
practice, they have to be regarded as three parts of one whole. If we take the idea
of mode as semiotic resource seriously, this does not only mean that modes have
no pre-given meaning, but concretise their semantic potential only in a given com-
municative context. Consequently, we cannot understand and analyse modes in
isolation, but only in their interconnectedness with media and genre.
In order to illustrate these complex relationships, I would like to use the Ger-
man translation of the opera Carmen as an example.13 Shortly after its premiere in
Paris in 1875 at the Opéra Comique, Carmen was performed at the Vienna Court
Opera. The change of mediation context determined the adjustment of the genre
58 Klaus Kaindl
to which Carmen pertains: the opéra comique. This genre has specific modal char-
acteristics, such as the distribution of spoken and sung passages. The musical mode
is also characterised by specific compositional styles; similar to the sung mode, it
comprises a number of typical elements, which are expressed as “détaché,” “trés
léger,” or “très également et simplement,” and which can be traced back to the
roots of the opéra comique in spoken theatre. Carmen was explicitly composed for the
Opéra Comique, and thus the medial context, genre, and intermodal realisation were
all part of one whole. The decision to stage the German-language performance
at the Vienna Court Opera had a significant impact on the genre as well as the
modal realisation and intermodal relationships associated with it. The much larger
dimensions of the stage created new semiotic potential, which in turn affected the
use of extras, props, and stage decor as well as the verbal, vocal, and musical mode.
This required a modification of the original genre, as the intimate setting of the
opéra comique was not suitable for the Vienna Court Opera. To overcome this prob-
lem, the translator, Julius Hopp, modified his text to suit the conventions of the
Romantic opera, which was reflected in the musical realisation and the relationship
between spoken and sung modes. The orchestra was extended, the spoken passages
were deleted, and the singing style was adapted to the conventions of the Romantic
opera. This also led to dramatic changes in the realisation on stage: the choice of
costumes, the number of extras, and thus also the representation of the characters
had to be adapted according to the new genre and medial conditions. The medial
context, the genre-specific conventions, and the modal relationships influenced
each other and turned a realistic chamber play into a bourgeois Romantic opera.
3.2. A multimodal definition and taxonomy of translation

If the research focus of translation studies shifts from the linguistic aspects of a text
to texts in all their modal realisations, medial contexts, and generic forms, then
the scope of research has to be expanded to an understanding of translation as a
conventionalised cultural interaction in which a mediator transfers texts in terms
of mode, medium, and genre across semiotic and cultural barriers for a new target
audience.14 A text that serves as the basis for a translation consists of the combined
use of different modalities; these are based on distinct discourses, and they are pro-
duced, distributed, and received on the basis of a specific design.
Such a definition differs from language-centred approaches in two respects. On
the one hand, there is no fixed relation to the source text based on similarity, equal-
ity, or equivalence and, on the other hand, the role of language in translation does
not take centre stage. This also means that it is no longer necessary to rename
translation phenomena that deviate from these criteria. For example, translations
that differ from their source texts are frequently labelled differently, e.g., editing,
free rendering, rewriting, appropriation, or adaptation. The same applies to transla-
tions that transcend modal boundaries. Depending on the discipline, we can find
a variety of different labels: in translation studies we have transmutation in refer-
ence to Jakobson (1959), para-translation (Yuste Frías 2012), post-translation (Gentzler
2017) or, more generally, adaptation or version. Media research makes use of terms
such as medial transposition, transmedialisation, intermedial transfer; while in multimodal
communication theory we can find transmodal moment (Newfield 2014), transmodal
operation (Wyatt-Smith and Kimber 2009), or transmodal redesign (Mavers 2011).15
The term transduction introduced by Kress plays a central role in this regard.
Originally, it was used to describe how students create meaning across different
modes. It was linked to translation at a much later point (2010), and more recently
it has also been used in translation research (e.g., Poulsen 2017).
For the reasons mentioned already, we will not use the term proposed by Kress,16
as it would limit the understanding of translation applied in current translation-
theoretical approaches. There are, however, links to the characteristics associated
with transduction. For example, what Kress says about producers of multimodal
texts also applies to translators: they “stretch, change, adapt and modify all of the
elements used, all the time, and thereby change the whole set of representational
resources with its internal relations” (Kress 2000, 155). Similar to action-oriented,
sociological, and descriptive translation studies, Kress also emphasises the action
component and the social, cultural, and historical conditionality of transfers across
modal boundaries: “An adequate theory of semiosis will be founded on a recogni-
tion of the ‘interested action’ of socially located, culturally and historically formed
individuals, as the remakers, the transformers and the re-shapers of the representa-
tional resources available to them” (2000, 155).
Newfield notes from a multimodal viewpoint that
The concept of the transmodal moment focuses attention on the relational

aspect of the transmodal chain, on the way in which a modal shift impacts on
meaning and on the way in which the links are connected or discontinuous
with one another.
(2014, 103)
The aspect of relationality, which is changed by transfer, also provides a parallel

to translation research. In translation, a multimodal whole is also transferred into
another temporal, social, and cultural context. Assuming that the meaning of modes
always depends on the specific context of use, such a transfer also means change,
since the recipients and with them the social and cultural parameters change. Thus,
every translation is inevitably subject to change. Fixed, rigid relational specifica-
tions, such as the relationship between source and target text, are just as obsolete as
the idea that modes could pass through the process of translation without undergo-
ing change.
Even though translation studies are still occupied with the linguistic dimension
as its prototypical core area, texts and their transfers should no longer primarily be
examined based on language aspects, but through the lens of mode, medium, and
genre. These should also form the basis for a taxonomy of translation. One of the
first scholars to establish a classification of translation that went beyond language
was Roman Jakobson (1959) with the famous triad of intralingual, interlingual,
60 Klaus Kaindl
and intersemiotic translation. However, he remains committed to the linguistic

paradigm and only considers interlingual translation to be translation in the proper
sense, while intersemiotic translation, which he also calls transmutation (1959, 233),
represents a special case.
Jakobson’s classification may have been influential, but it proved problematic
due to its conceptual inconsistency, which has been repeatedly criticised. The term
intersemiotic translation is unfortunate inasmuch as language is also a semiotic sys-
tem and, thus, the translation between two language systems would also have to
be regarded as intersemiotic translation. Toury (1994, 1114) avoids this problem
by defining Jakobson’s interlingual translation as intrasemiotic translation, which can
be divided into intrasystemic (i.e., intralingual) and intersystemic translation. For
the latter, he explicitly only gives interlingual translation as an example, but other
modalities would also be conceivable here. Toury’s definition of intersemiotic
translation, which translates between different modes, is similar to Jakobson’s. In
principle, however, semioticity does not seem to offer itself as a suitable basis for
a taxonomy of translation, due to the all-encompassing variety of meanings of
the term. Therefore, it is preferable to differentiate among the categories mode,
medium, and genre. Thus, at the modal level, a distinction would have to be made
between intra-and intermodal translation, which can take place intra-or transcul-
turally. While intramodal translation involves translating within a modality (verbal-
verbal, image-image, etc.), intermodal translation exceeds modality limits. Examples
range from the translation of a verbal instruction manual into a pictorial representa-
tion to the translation of bible text into a comic.
At the medial level, intramedial transfers refer to processes where the medium
remains unchanged. However, this can also pose problems due to cultural differ-
ences. For example, US-American and European music videos differ on a visual
level due to different editing and image sequences. Intermedial translations refer
to a change of medium, e.g., from a novel to a screenplay and subsequently to a
film.
Finally, at the generic level, translation can also take place intra-and interge-
nerically. Similar to the medium, culture-specific problems can also arise in intra-
generic transfers, for example, when writing conventions differ in two languages:
in French, the alexandrine verse form is conventionally used for drama, whereas
German drama is not characterised by the use of verse metres. Or, in the case of the
translation of an Argentine tango into a Finnish tango, the two have different musi-
cal conventions. An example of an intergeneric translation would be the transfer
of a French opéra comique into a German Singspiel or – as will be shown later– of an
American rock ‘n’ roll song into a German Schlager.
It should be noted that mode, medium, and genre are inextricably linked and
that a change in one dimension fundamentally affects the functional whole, which
consists of all three parts. Thus, it is impossible to examine mode, medium, and
genre independently from each other in the context of translation. Neverthe-
less, the differentiation makes sense for translation studies in that all three com-
ponents can each pose their own translation problems. These range from modal
communicative-functional aspects through medial technical-social implications to

generic text-conventional issues.
4. Mode, medium, and genre in the translation of

rock ‘n’ roll
If translational action is understood as being semiotic and translation as a multi-
modal practice, then a number of questions and challenges arise for theoretical
reflection on and the practical application of translation:
• What are the (multi)modal culture-specific textual conventions?

• What meaning(s) is activated by which modes?
• How can multimodal translation units be identified?
• What does a multimodal translation-relevant text analysis look like?
• Which modes are changed in the translation process and how does this affect
the functional multimodal whole?
I would like to answer at least some of these questions with the example of the
song Hound Dog (written and composed by Jerry Leiber and Mike Stoller, and per-
formed by Elvis Presley) and its translation into German.17 The modal, medial, and
generic specifics and their functional interconnectedness serve as a starting point
for the multimodal translation analysis.18
As mentioned earlier, the development of the genre rock ‘n’ roll was shaped by
cultural and medial factors. The 1950s economic boom in the US also created new
financial opportunities for a previously economically weak social group, teenagers.
The political situation of the Cold War and the associated repressive climate in the
US increased the need for distraction and entertainment, and the gradual awaken-
ing of self-confidence among the African American population contributed to the
spread and popularisation of black forms of music, which were also adopted by
white music producers for economic reasons. Record production and radio stations
and their programmes became more diverse, and the technological development of
new carrier media – record, tape, and transistor radios – contributed to increased
proliferation.
As far as music and language are concerned, rock ‘n’ roll is a classic example
of the combination of black and white music. It is influenced by (white) country
music – at least in rockabilly style – as well as by (black) rhythm and blues. Rock ‘n’ roll
exhibits numerous formal and melodic characteristics of R&B, but we can also find
balladesque elements. As mentioned at the beginning of this article, the novelty lies
in the combination and reassessment of the individual components.19
At the same time, popular music in the German-speaking world was dominated
by the Schlager genre. The 1950s Schlager was light, popular music with a rather
conservative touch, which was mostly geared towards evoking blissful and har-
monious feelings, and its sentimental-romantic lyrics were free of sexual themes
or themes that might contradict accepted forms of relationship at the time. The
62 Klaus Kaindl
music, however, was quite flexible and absorbed different musical styles, which – as
we will also see in the following example – were adapted to culture-specific medial
mediation contexts.20
Hound Dog was originally written for the singer Big Mama Thornton as a rhythm
and blues number and intergenerically translated for Elvis Presley as a rock ‘n’ roll
song. It became a number-one hit in 1956:
You ain’t nothin’ but a hound dog,

Cryin’ all the time.
You ain’t nothin’ but a hound dog,
Cryin’ all the time.
Well, you ain’t never caught a rabbit,
And you ain’t no friend of mine.
When they said you was high classed,
Well, that was just a lie.
When they said you was high classed,
Well, that was just a lie.
You ain’t never caught a rabbit,
And you ain’t no friend of mine.
(Presley 2002/2003)
The story being told in the lyrics is about a lazy, useless person who is not much
better than a lazy dog. Considering the background of the original singer, the slang
contraction “you ain’t” could also be seen as a reference to the Afro-American ori-
gin of the song. In a sense, one of the essential characteristics of rock ‘n’ roll, namely
the combination of black and white music, is transferred to the verbal and vocal
mode here. Elvis’s vocal characteristics – the throaty, rough voice projected with
high pressure, the blurred notes that start as low tonal sounds and the “smearing,”
the intonational smearing of notes, which make up his unique style (cf. Middleton
1977/1992) – are in keeping with the rhythm and blues tradition. The pressure he
puts on his voice corresponds to the verbalised disappointment and anger, which
gives the song an overall aggressive tone.
The musical submodes of the song can be differentiated into harmony, melody,
and rhythm. All three factors are in keeping with the classic blues tradition. The
harmonic elements express the tension that permeates the song in the verbal and
vocal mode: this is achieved by contrasting the major harmonies, which form the
basis of the composition, with the “blue notes” in the vocals, in other words, with
minor thirds. Moreover, the breaking up of the major power chord of the bass as
well as the the I-IV-I-V-IV-I chord progression create a mix of major/minor col-
ouring of the song, which is typical of blues.
The rhythm is primarily determined by the dotted quarter notes of the bass, and
the blues style is further emphasised by the “claps.” The melody also exhibits clas-
sic blues characteristics. Blue notes create a minor coloration of the tune, which in
turn contrasts the major harmonies.
As far as the medial dimension is concerned, the sound design depends not
only on the arrangement but also on the sound technology used. The high-energy
sound is created with the help of a specific recording technique that amplifies the
sound electronically, which makes it powerful and loud at maximum sound levels.
Both the individual modes and the production media have a functional relationship
with each other, which gives the song its specific coloration and mood.
In the German version, changes were made at the generic, medial, and modal
levels:
Ja das ist der neue Rhythmus, [Yes, this is the new rhythm,]
Das ist Rock and Roll [This is rock ‘n’ roll]
Ja, da wo jeder mit muss, [Yes, everyone wants to do it]
Das ist Rock and Roll. [This is rock ‘n’ roll]
Ja der Rock ist der aller letzte Schrei. [Yes, rock is the latest craze.]
Jeder ist dabei. [Everyone is doing it.]
In Peru und in Chile, [In Peru and in Chile,]
In Paris und Madrid [In Paris and Madrid]
Tanzt man Rock ohne Pause, [Everyone is dancing rock without a break]
Und wir tanzen mit. [And we dance along.]
Rock and Roll morgens mittags abends [Rock ‘n’ roll in the morning, at
noon and in the evening]
Tanzen wir nur Rock and Roll. [We all dance the rock ‘n’ roll.]
(Presley n.d.)
The functional connection between the modes – language, music, and vocals – and
the medial production context differ from the multimodal make-up of the source
text. Both the multimodal and medial elements serve to adapt the rock ‘n’ roll num-
ber to the Schlager genre.
In the language mode, a socio-cultural change occurs in comparison to the
original. The foreign genre is explained verbally and presented as dance music,
which also finds reflection in the musical mode: the melody and rhythm of the
original, as well as its composition, are toned down with a view to presenting a
decent and clean dance style that is appropriate for young adults.
The vocal mode does not produce any pressure due to intonation, volume, and
articulation, and it sounds rather soft and instrumental; in other words, it is just a
tonal addition to the composition. The melody does not contain any “blue notes,”
which leaves us without the “dirty” rhythm and blues sound. Instead, the melody
is played in several voices and in thirds above the keynote. The harmony is largely
kept in major key, creating a serene sound, and the rhythm is based on that of the
big-band revues of the 1950s. The instrumental part is put in the foreground and
harmonises with the lyrics, which deal exclusively with the new dance rock ‘n’ roll.
This adaption is further reflected in the recording technique, which does not
use any electronic sound compression but reproduces the classic sound of Schlager
music. Although the same instruments – electric guitar, contrabass, and drums, with
64 Klaus Kaindl
the addition of the saxophone and trumpet – are used, the technical media create a
sound that is compatible with Schlager music.
Overall, the individual modes are in harmony with each other: as opposed to
creating modal tension, language, music, and vocals support each other in order
to present a “foreign” genre in a familiar and, above all, harmless guise, i.e., in the
form of Schlager music. To a certain extent, the functional connections have been
reversed in the translation. However, the translation represents a coherent multi-
modal whole, whose changes can only be explained by the socio-cultural back-
ground of the genre into which it was translated.
5. Conclusion
At present, the interdisciplinary interaction between the multimodal communica-
tion approach and translation studies still seems to be quite one-sided. While the
former is hardly aware of translation studies, the latter has recognised multimodality
as a relevant topic. This is evidenced, on the one hand, by entries in handbooks
and encyclopaedias (e.g., Kaindl 2013; Pérez-González 2014), and on the other
hand, by the large number of individual studies dealing with multimodality – albeit
mostly in a very selective manner. Conversely, when the topic of translation is
touched upon in multimodality research, a very outdated, static concept of transla-
tion frequently serves as the basis of investigation, and recent theoretical work in
translation studies is almost never used. This article hopefully has shed some light
on the common ground between the two disciplines: both deal with texts from a
medial, modal, and generic perspective; both regard transformation and relationality
as essential characteristics of their research subject; and both consider their subject
in a social and cultural context, rather than as an abstract and static entity. This
seems to be a good basis for increased interdisciplinary networking in the future.
Moreover, increased networking across all disciplines dealing with modal, medial,
and generic transfer processes would be desirable. In addition to translation studies
and multimodality studies, these include intermediality studies and adaptation stud-
ies. Translation studies has recently entered into a mutual dialogue with the latter
in particular. While the first edition (2006) of Linda Hutcheon’s standard work A
Theory of Adaptation still contains no mention of possible points of reference to
translation studies, the new edition (2013) is at least aware of it. A dialogue between
the two disciplines, as conducted by van Doorslaer and Raw (2016), in order to
work out the similarities but also differences between the two disciplines, could also
be an example of a future exchange between the two transfer disciplines.
A dose of multimodal awareness would undoubtedly be beneficial for translation
studies as well as translators. This means recognising other modes beyond language
as relevant for translation and developing corresponding transfer skills. This does
not mean that translators must be able to compose, draw, create graphics, and the
like. However, they must be able to transfer the functional whole of a multimodal
text in cooperation with other experts in accordance with the expectations and
requirements resulting from the context of use and reception in the target culture.
This requires not only classic translation skills, such as linguistic and cultural com-
petence, specialist competence, research competence, and transfer skills, but also
comprehensive multimodal awareness, which also includes media and genre aware-
ness. In concrete terms, this means that translators must be able to classify semiotic
resources, analyse their meaning and functioning, and understand their functional
links with other modes.
However, what Jäger called “transcriptive intelligence” (2002, 35), i.e., the innate
ability to communicate and transpose content across mode and media boundaries,
is not sufficient for professional translatorial work. A (professional) translator must
develop this – seemingly – innate ability into a conscious competence. Above all,
this requires knowledge of the culture-contrastive dimension of modes, media, and
genres.
Multimodal translation studies, and thus an extension of the research interests to
all modes and their transfers, does not mean relativising the linguistic dimension in
its relevance for translation, but rather placing it in a larger context and thus ulti-
mately delving deeper into the transcultural mechanisms of text composition and
design. To this end, it would certainly be necessary to clarify its terminology with
regard to modality and mediality, and to develop translation-relevant analysis instru-
ments for other modes. Linguistic theories as well as image-and music-theoretical
approaches are suitable for this purpose. Finally, translator training should also be
adapted to recognise multimodal competence as an integral part of translation
competence.
The required interdisciplinary competence builds on disciplinary competence,
which manifests in a stringent understanding of the subject matter and a clearly
defined research object. In this respect, translation studies are confronted with
the question of how “translation proper” and, thus, its subject matter, is actually
defined. Roman Jakobson’s answer – interlingual translation – is probably obso-
lete in the age of multimedia. In the twenty-first century, the answer can only be
that translation is a multimodal practice and translatorial action is a multimodal
semiotic act.
Notes
1 Although Roman Jakobson (1959) introduced the category of “intersemiotic transla-
tion,” strictly speaking, his classification was not semiotic but linguistic. Later, explicitly
semiotic definitions of translation (e.g., Gorlée 1994; Stecconi 2004) did not bring about
a semiotic reorientation in mainstream translation studies, either.
2 For a more detailed overview, see Bird (2007).
3 Similarly, Hess-Lüttich points to the “constructive gestalt” (1981, 324) of those signs
that generate meaning only in the context of concrete communication processes. In this
regard, he also sees the term “sign” as problematic. Decades later, Kress and van Leeuwen
solved this problem by introducing the term “semiotic resource.” Cf. also van Leeuwen
(2005, 95): “the term ‘resource’ is preferred, because it avoids the impression that ‘what
a sign stands for’ is something pre-g iven, and not affected by its use.”
4 For a more detailed examination of the communicative-sociological contextualisation of
signs, see Richter (1978).
66 Klaus Kaindl
5 “Mode” is an established term in various disciplines including linguistics, psychology,

narratology, medicine, and, of particular relevance to us, in interpreting studies, where
mode is used to describe the various forms of interpreting, e.g., simultaneous interpret-
ing, consecutive interpreting, remote interpreting, etc., which can be problematic with
regard to interdisciplinary work.
6 One of the first scholars to explicitly address multimodal texts in translation studies was
Katharina Reiß (1971).
7 For an overview of the positioning of translation studies within various linguistic subdis-
ciplines, see Kaindl (2004, 15–22).
8 For the various terms and the associated implications, see Kaindl (2013, 259–61).
9 The submodes of spoken language include volume, voice qualities, intonation, and speed;
the submodes of written language are typography and layout; the submodes of images are
colour, lines, and forms; and the submodes of music include rhythm, melody, harmony,
and dynamics. Stöckl (2004, 14) also differentiates between “peripheral modes,” which
result from the media realisation of the core modes.
10 For an overview of this research area and the different forms of intermediality, see
Rajewski (2002).
11 Bateman (2008) examines genres from a multimodal perspective.
12 The distinction between mode and medium is not always clear, which is also evidenced
by some inconsistencies in Kress and van Leeuwen’s classification. While, for example,
colour is initially referred to as “semiotic mode” (Kress and van Leeuwen 2002), van
Leeuwen later classifies it as “expression media” (2016, 109).
13 A detailed discussion of the translation with regard to genre, medium, and the effects on
the relationship among language, music, and production can be found in Kaindl (1995,
188–206).
14 This definition is an expansion on Prunc’s (2004) definition, which is more language-
centred, but very comprehensive in principle.
15 Newfield (2014) provides an overview of the nomenclature of intermodal transfer
phenomena.
16 It is also debatable whether the term is a good choice. Although the term is etymologically
related to translation, transduction is also a central concept in genomics research, where it
describes the process by which foreign DNA is transported into a cell via a virus or bacterium.
17 For a detailed examination of Elvis Presley songs from 1956–69 with a particular focus
on the verbal dimension, see Kaindl (2012); for the musical dimension of Hound Dog, see
also Dorkin (2009, 94–100).
18 In the following I will focus on the verbo-musical dimension. Of course, the visual
mode, Presley’s body movements, his way of dancing, also had an impact on the recep-
tion of the lyrics and the music.
19 For a detailed overview of the social development context, see Szatmary (1996), and for
musical characteristics, see Brown (1987).
20 For more details on the development and characteristics of Schlager music, see Port le roi (1998).
References
Primary sources
Presley, Elvis. 2002/2003. Elvis Forever. RCA/BMG.
Presley, Elvis. n.d. Elvis singt deutsch Folge 3. Bear Records.
Secondary sources
Baldry, Anthony, and Christopher Taylor. 2002. “Multimodal Corpus Authoring Sys-
tem: Multimodal Corpora, Subtitling and Phasal Analysis.” In Multimodal Resources and
Multimodal Systems Evaluation, edited by Mark Maybury and Jean-Claude Martin, 45–51.
Las Palmas: LREC.
Bar-Hillel, Yehoshua. 1960. “The Present Status of Automatic Translation of Languages.”
Advances in Computers 1: 91–163.
Bassnett, Susan, and André Lefevere. 1990. “Introduction: Proust’s Grandmother and the
Thousand and One Nights. A ‘Cultural Turn’ in Translation Studies.” In Translation, His-
tory and Culture, edited by Susan Bassnett and André Lefevere, i–xiii. London and New
York: Pinter.
Bateman, John. 2008. Multimodality and Genre: A Foundation for the Systematic Analysis of Mul-
timodal Documents. New York: Palgrave MacMillan.
Berressem, Hanjo. 2004. “Poststruktularismus.” In Grundbegriffe der Literaturtheorie, edited by
Ansgar Nünning, 218–21. Stuttgart: Metzler.
Bird, Alexander. 2007. “What Is Scientific Progress?” Nous 41 (1): 64–89.
Borodo, Michal. 2015. “Multimodality, Translation and Comics.” Perspectives 21 (1): 22–41.
Brown, Charles T. 1987. The Art of Rock and Roll, 2nd ed. Englewood Cliffs, NJ: Prentice
Hall.
Catford, John C. 1965. A Linguistic Theory of Translation. London: Oxford University Press.
Cross, Judith Leah. 2016. “Enigma: Aspects of Multimodal Inter- Semiotic Translation.”
Hermes 55: 91–103.
Delabastita, Dirk. 1989. “Translation and Mass Communication: Film and TV Translation as
Evidence of Cultural Dynamics.” Babel 35 (4): 193–218.
De Pedro Ricoy, Raquel. 2012. “Multimodality in Translation: Steps Towards Socially Useful
Research.” Multimodal Communication 1 (2): 181–203.
Derrida, Jacques.1980. “La loi du genre.” Glyph 7: 176–201.
Dorkin, Katharina. 2009. Die Übersetzung der musikalischen Dimension in der Popularmusik am
Beispiel der Lieder von Elvis Presley. Wien: Unpubl. MA-thesis.
Eco, Umberto. 1972. Einführung in die Semiotik. München: Fink.
Elleström, Lars. 2010a. “Introduction.” In Media Borders, Multimodality and Intermediality,
edited by Lars Elleström, 1–8. New York: Palgrave MacMillan.
———. 2010b. “The Modalities of Media: A Model for Understanding Intermedial Rela-
tions.” In Media Borders, Multimodality and Intermediality, edited by Lars Elleström, 11–48.
New York: Palgrave MacMillan.
———. 2017. “Transfer of Media Characteristics Among Dissimilar Media.” Palabra Clave
20 (3): 663–85.
Gambier, Yves. 2006. “Multimodality and Audiovisual Translation.” In Audiovisual Transla-
tion Scenarios: Proceedings of the Second MuTraConference in Copenhagen 1–5 May, edited
by Mary Carroll, Heidrun Gerzymisch-Arbogast and Sandra Nauert. Accessed May 05,
2018. www.euroconferences.info/proceedings/2006_Proceedings/2006_Gambier_
Yves.pdf.
Gentzler, Edwin. 2017. Translation and Rewriting in the Age of Post-Translation. London and
Gorlée, Dinda. 1994. Semiotics and the Problem of Translation:With Special Reference to the Semiot-
ics of Charles S. Peirce. Amsterdam and Atlanta: Rodopi.
Gottlieb, Henrik. 1994. “Diagonal Translation.” Perspectives 2 (1): 101–21.
Hess-Lüttich, Ernest W. B. 1978. “Semiotik der multimedialen Kommunikation. Eine Prob-
lemskizze.” In Angewandte Semiotik, edited by Tasso Borbé and Martin Krampe, 21–48.
Wien: H. Egermann.
———. 1981. Grundlagen der Dialoglinguistik. Berlin: Erich Schmidt.
Holz-Mänttäri, Justa. 1984. Translatorisches Handeln: Theorie und Methode. Helsinki: Suoma-
lainen Tiedeakatemia.
68 Klaus Kaindl
———. 1993. “Textdesign – verantwortlich und gehirngerecht.” In Traducere Navem: Fest-

schrift für Katharina Reiß zum 70. Geburtstag, edited by Justa Holz-Mänttäri and Christiane
Nord, 301–20. Tampere: Tampereen Yliopisto.
Hutcheon, Linda. 2006. A Theory of Adaptation. London and New York: Routledge.
Hutcheon, Linda, and Siobhan O’Flynn. 2013. A Theory of Adaptation, 2nd ed. London and
Iedema, Rick. 2003. “Multimodality, Resemiotization: Extending the Analysis of Discourse
as Multi-Semiotic Practice.” Visual Communication 2 (1): 29–57.
Jäger, Ludwig. 2002. “Transkriptivität: Zur medialen Logik der kulturellen Semantik.” In
Transkribieren: Medien/Lektüre, edited by Ludwig Jäger and Georg Stanitzek, 19–41.
München: Fink.
Jakobson, Roman. 1959. “On Linguistic Aspects of Translation.” In On Translation, edited by
Reuben A. Brower, 232–39. Cambridge, MA: Harvard University Press.
Jewitt, Carey. 2014a. “An Introduction to Multimodality.” In The Routledge Handbook of
Multimodal Analysis, 2nd ed., edited by Carey Jewitt, 15–30. London and New York:
Routledge.
———. 2014b. “Different Approaches to Multimodality.” In The Routledge Handbook of Mul-
timodal Analysis, 2nd ed., edited by Carey Jewitt, 31–43. London and New York: Routledge.
Jiménez Hurtado, Catalina, and Silvia Soler Gallego. 2013. “Multimodality, Translation,
Accessibility: A Corpus-Based Study of Audio Description.” Perspectives 21 (4): 577–94.
Kade, Otto. 1968. Zufall und Gesetzmäßigkeit in der Übersetzung: Beiheft zur Zeitschrift Fremd-
sprachen I. Leipzig:VEB Enzyklopädie.
Kaindl, Klaus. 1995. Die Oper als Textgestalt: Perspektiven einer interdisziplinären Übersetzungswis-
senschaft. Tübingen: Stauffenburg.
———. 2004. Übersetzungswissenschaft im interdisziplinären Dialog: Am Beispiel der Comicüber-
setzung. Tübingen: Stauffenburg.
———. 2012.“ ‘Little Linda hör zu’: Rock ‘n’ Roll Gender und Übersetzung.” In Translation-
swissenschaftliches Kolloquium II: Beiträge zur Übersetzungs-und Dolmetschwissenschaft (Köln/
Germersheim), edited by Barbara Ahrens, Silvia Hansen-Schirra, Monika Krein-Kühle and
Michael Schreiber, 307–25. Frankfurt am Main: Peter Lang.
———. 2013. “Multimodality and Translation.” In Routledge Handbook of Translation Stud-
ies, edited by Carmen Millán and Francesca Bartrina, 257–70. London and New York:
Routledge.
Ketola, Anne. 2016. “Towards a Multimodally Oriented Theory of Translation: A Cognitive
Framework for the Translation of Illustrated Technical Texts.” Translation Studies 9 (1):
67–81.
Kloepfer, Rolf. 1976. “Komplementarität von Sprache und Bild.” Sprache im technischen Zei-
talter 57: 42–56.
Kress, Gunther. 2000. “Design and Transformation. New Theories of Meaning.” In Multi-
literacies: Literacy Learning and the Design of Social Futures, edited by Bill Cope and Mary
Kalantzis, 153–61. Melbourne: Palgrave MacMillan.
Kress, Gunther, and Carey Jewitt. 2003. “Introduction.” In Multimodal Literacy, edited by
Gunther Kress and Carey Jewitt, 1–18. New York: Peter Lang.
Kress, Gunther, and Theo van Leeuwen. 2001. Multimodal Discourse: The Modes and Media of
Contemporary Communication. London: Hodder Arnold.
———. 2002. “Colour as a Semiotic Mode – Notes for a Grammar of Colour.” Visual Com-
munication 1 (3): 343–69.
Kristeva, Julia. 1968/1977. “Semiologie – kritische Wissenschaft und/oder Wissenschaftskri-

tik.” In Textsemiotik als Ideologiekritik, edited by Peter V. Zima, 35–51. Frankfurt am Main:
Suhrkamp.
Krystallidou, Demi. 2016. “Investigating the Interpreter’s Role(s): The A.R.T. Framework.”
Interpreting 18 (2): 172–97.
Kuhn, Thomas S. 1970. The Structure of Scientific Revolutions. Chicago: Chicago University
Press.
Lee,Tong-King. 2012. “Performing Multimodality: Literary Translation, Intersemioticity and
Technology.” Perspectives 21 (2): 241–56.
Lefevere, André. 1992. Translation, Rewriting and the Manipulation of Literary Fame. London
and New York: Routledge.
Mavers, Diane. 2011. Children’s Drawing and Writing: The Remarkable and the Unremarkable.
McLuhan, Marshall. 1950. “Foreword.” In Empire and Communications, edited by Harold A.
Innis, v–xii. Toronto: University of Toronto Press.
———. 1964. Understanding Media:The Extensions of Man. New York: McGraw-Hill.
Middleton, Richard. 1977/1992. “All Shook Up? Innovation and Continuity in Elvis Pres-
ley’s Vocal Style.” In The Elvis Reader: Texts and Sources of the King of Rock ‘n’ Roll, edited
by Kevin Quain, 3–12. New York: St. Martin’s Press.
Munday, Jeremy. 2014. “Advertising: Some Challenges to Translation Theory.” The Translator
10 (2): 199–219.
Newfield, Denise R. 2014. “Transformation, Transduction and the Transmodal Moment.” In
The Routledge Handbook of Multimodal Analysis, 2nd ed., edited by Carey Jewitt, 100–13.
Nida, Eugene A., and Charles R. Taber. 1969. The Theory and Practice of Translation. Leiden:
E. J. Brill.
Oittinen, Riitta, Anne Ketola, and Melissa Garavini. 2017. Translating Picturebooks:
Revoicing the Verbal, the Visual and the Aural for a Child Audience. London and New
York: Routledge.
Pérez-González, Luis. 2014. “Multimodality in Translation and Interpreting Studies: Theo-
retical and Methodological Perspectives.” In A Companion to Translation Studies, edited by
Sandra Bermann and Catherine Porter, 119–31. Chichester: Wiley-Blackwell.
Popper, Karl R. 1998. Objektive Erkenntnis: Ein evolutionärer Entwurf, 4.Verbesserte und ergän-
zte Auflage. Hamburg: Hoffmann und Campe.
Port le roi, André. 1998. Schlager lügen nicht: deutscher Schlager und Politik in ihrer Zeit. Essen:
Klartext.
Posner, Roland. 1976a. “Semiotische Paradoxien in der Sprachverwendung – Am Beispiel
von Sternes ‘Tristam Shandy’.” Sprache im technischen Zeitalter 57: 25–41.
———. 1976b. “Zum Thema.” Sprache im technischen Zeitalter 57: 2–4.
Poulsen, Søren Vigild. 2017. “The ‘Same’ Meaning Across Modes? Some Reflections on
Transduction as Translation.” In New Studies in Multimodality. Conceptual and Methodo-
logical Elaborations, edited by Ognyan Seizov and Janina Wildfeuer, 37–64. London:
Bloomsbury.
Prunc, Erich. 2004. “Zum Objektbereich der Translationswissenschaft.” In Und sie bewegt sich
doch…: Translationswissenschaft in Ost und West. Festschrift für Heidemarie Salevsky zum 60.
Geburtstag, edited by Ina Müller, 263–85. Frankfurt am Main: Lang.
Rajewski, Irina O. 2002. Intermedialität. Tübingen: A. Francke.
Reiss, Katharina. 1971. Möglichkeiten und grenzen der übersetzungskritik: Kategorien und kriterien
für eine sachgerechte beurteilung von übersetzungen. München: Hueber.
70 Klaus Kaindl
Richter, Helmut. 1978. “Zum kommunikationssoziologischen Inhalt des Medienbegriffs.”

In Stilforschung und Rhetorik, Patholinguistik und Sprecherziehung, edited by Ernest W. B.
Hess-Lüttich, Günter Peuser, and Helmut Geißner, 37–43. Stuttgart: Hochschulverlag.
Ryan, Marie-Laure, and Jean Noël Thon. 2014. “Storyworlds Across Media: Introduction.”
In Storyworlds Across Media: Toward a Media-Conscious Narratology, edited by Marie-Laure
Ryan and Jean Noël Thon, 1–21. Lincoln and London: University of Nebraska Press.
Stecconi, Ubaldo. 2004. “Interpretive Semiotics and Translation Theory: The Semiotic Con-
ditions to Translation.” Semiotica 150 (1–4): 471–89.
Stöckl, Hartmut. 2004. “In Between Modes: Language and Image in Printed Media.” In Per-
spectives on Multimodality, edited by Eija Ventola, Cassily Charles and Martin Kaltenbacher,
9–30. Amsterdam: Benjamins.
Szatmary, David P. 1996. A Time to Rock:A Social History of Rock and Roll. New York: Schirmer
Books.
Toury, Gideon. 1994. “A Cultural-Semiotic Perspective.” In Encyclopaedic Dictionary of Semi-
otics, Vol. 2 N-Z, edited by Thomas A. Sebeok, 1111–24. Berlin and New York: Mouton
de Gruyter.
———. 1995. Descriptive Translation Studies – and Beyond. Amsterdam and Philadelphia:
Benjamins.
van Doorslaer, Luc, and Laurence Raw. 2016. “Adaptation Studies and Translation Studies:
Very Interactive Yet Distinct.” In Border Crossings: Translation Studies and Other Disciplines,
edited by Yves Gambier and Luc van Doorslaer, 189–204. Amsterdam and Philadelphia:
John Benjamins.
———. 2016. “A Social Semiotic Theory of Synesthesia? A Discussion Paper.” Hermes 55:
105–19.
Vermeer, Hans J. 1986. “Übersetzen als Kultureller Transfer.” In Übersetzungswissenschaft –
eine Neuorientierung: Zur Integrierung von Theorie und Praxis, edited by Mary Snell-Hornby,
30–53. Tübingen: Francke.
Walther, Elisabeth. 1997. “The Sign as Medium, the Medium Relation as the Foundation
of the Sign.” In Semiotics of the Media: State of the Art, Projects and Perspectives, edited by
Winfried Nöth, 79–86. Berlin and New York: Mouton de Gruyter.
Wyatt-Smith, Claire, and Kay Kimber. 2009. “Working Multimodally: Challenges for Assess-
ment.” English Teaching Practice and Critique 8 (3): 70–90.
Yuste Frías, José. 2012. “Fun for All: Videojuegos y paratraducción.Video Game’s Translation
and Paratranslation.” In Blog de Yuste: On y sème à tout vent. Vigo: Blog de Investigación
T&P, 23/03/2012. Accessed May 5, 2018. www.joseyustefrias.com/index.php/blog/
item/fun-for-all-1.html.
3
MEANING-(RE)MAKING IN A
WORLD OF UNTRANSLATED SIGNS
Towards a research agenda on multimodality,
culture, and translation
Elisabetta Adami and Sara Ramos Pinto
1. Introduction
Multimodality has always been “the normal state of human communication” (Kress
2010, 1). From face-to-face interaction combining resources such as gesture, body
movement, and face expression, to disembodied representations combining image
with writing, multimodal texts (and the need to translate them) have always been
integral to (intercultural) communication. In recent years, technological develop-
ment has made this only the more manifest by facilitating multimodal text produc-
tion in a myriad of platforms.
This has led disciplines like translation studies (TS), traditionally focused on the
verbal, to turn to the recent field of multimodal studies (MS) for adequate meth-
odologies and analytical frameworks that can ground research on the translation
of multimodal texts such as films, websites, or comics (for instance, see the chapter
by Luis Pérez-González in this volume, Chapter 4). A new multimodal turn in TS
has adopted some of the analytical tools and concepts developed within the area
of multimodality, yet with little in-depth reflection on the principles upon which
those tools have been built or on what accepting those principles means for TS.
As a result, translation theories remain mostly focused on the verbal while limit-
ing the discussion of all other semiotic resources (integral to meaning-making in
multimodal texts) to a contextualising role.
The link between TS and MS is – or should be – bidirectional. Globalisation and
the consequent rising transnational circulation of goods, people, and cultural artefacts
has increasingly exposed us to visual, auditory, and material resources produced in
other areas of the world, while contributing to a multiplication and fragmentation of
communities with specific sign-making practices (New London Group 1996). This
questions the idea of homogenous nation-and language-bound communities sharing
the same cultural background for meaning interpretation.Yet MS has not yet tackled
72 Elisabetta Adami and Sara Ramos Pinto
the issue of “culture” in relation to resources other than language. Not only is there a
lack of studies on the use of semiotic resources with a cross-cultural comparative take
as well as in intercultural contexts, but also, while it is assumed that modes are culture-
specific resources (Kress 2010), the notion of culture remains under-defined and
under-explored (Adami 2017; Hawkins 2018). MS has yet to focus on the meaning-
making dynamics of transnational circulation, that is, on how multimodal resources
are (re)interpreted and (re)signified when they move across place, time, and social
groups, including “small and big cultures” (Holliday 1999).
Because of the changed and fast- changing social semiotic landscape, both
MS and TS need to start working on new assumptions and address a series of dis-
ciplinary gaps, which can be filled only through a joint transdisciplinary enterprise.
In this chapter we aim to sketch a joint semiotic and translation research agenda,
to develop understanding of meaning-making practices in a changed, increasingly
transnational, social semiotic landscape. We conceive this undertaking as a pre-
requisite to derive implications for TS and provide indications that can support
translation practice and training in today’s communicative landscape. We will start
by revising some of MS’s principles in order to identify the gaps in and most imme-
diate issues for social semiotic research.We will then discuss the implications for TS
of adopting a multimodal approach. In this effort, we will review some of the basic
concepts of TS and raise several questions we believe should be at the center of
the discipline. We will finish by proposing the sketch lines of a research agenda that
would allow MS and TS to address the issues previously brought forward.
2. Issues for a way forward in social semiotic research

As discussed in the Introduction, by moving away from structuralist semiotics
centred on codes, social semiotics (Hodge and Kress 1988; van Leeuwen 2005;
Kress 2010) elaborated the notion of multimodality and introduced the concept
of “mode,” following Halliday’s (1978) use of the term to distinguish between
speech and writing. Modes are conceived as “a socially and culturally shaped set of
resources for making meaning” (Bezemer and Kress 2008, 6), which have different
affordances deriving from their materiality and the ways they have been shaped
historically in different social groups to fulfil specific communicative needs.
Social semiotic research has undermined the assumption of the arbitrariness of
language (Kress 1993) vs. an assumed iconicity of nonverbal resources. Relevant
to our discussion is the understanding that resources like image, often referred to
as iconic, do not have a more “natural” relation (of resemblance) with the world
than language and do not make meaning universally. The meaning potentials of
any given semiotic resource result from the history of its past uses in given social
groups; hence, knowledge is required for its interpretation, or better, to formulate
hypotheses on the meaning expressed by a given use of the resource (given that,
in each specific instance, sign-makers have agency in associating a given form as
most apt to represent the meaning that they want to express). As two banal exam-
ples, the floppy-disk-shaped button in software interfaces has no more “natural” or
Meaning-(re)making 73
FIGURE 3.1 Bleeding nose used in anime to signify sexual arousal

Source: (Original anime: Karin)
“universal” resemblance to its signified than its equivalent written label “save.” To
understand the meaning of an anime character’s sudden bleeding nose (Figure 3.1),
viewers need to know the “conventions” for that specific use of image in anime,
used to signify “sexual arousal.” These conventions are neither arbitrary nor the
result of natural resemblance; in both cases, the relation between the signifier/form
and its signified/meaning is motivated (Kress and van Leeuwen 1996; Kress 2010),
and the motivation lies as much in the materiality as in the past uses of that specific
signifier in given social groups.1
Dismantling a presumed universally understood relation of resemblance with
the world for resources other than language brings two issues to the fore, namely
the need for a redefinition of “context” vs. “co-text” and that of shared/non-shared
“culture,” as semiotically conceived. Let us examine each in detail.
2.1 The “context” vs. “co-text” approach

The first issue concerns the question of conceiving of resources that come together
with speech and writing as either “context” or “co-text.” In this work, we use
“text” to define any multimodally composed meaningful whole (or multimodal
ensemble), rather than restricting it to writing. By “context” we mean the social
semiotic environment for the design, production, distribution/circulation of, and
engagement with so-defined texts (including participants in each of these processes,
and the social and semiotic resources available to make signs and meanings); by “co-
text” we mean signs (in any mode and their combinations) co-occurring with those
that are the momentary focus of attention in a text.
Linguistics and translation traditionally adopt the “context” approach and
consider what they normally term as “nonverbal”2 resources not as signs (forms
associated with meaning) but as a mere issue of cultural reference. In this approach
what is shown visually, for example, is (rather than represents) a certain phenomenon
in the world, which may be more or less culture specific or shared.
In turn, social semiotics treats all communicative manifestations as signs. This
view adopts a “co-text approach,” which considers all resources co-occurring with
writing or speech as signifying elements that make meaning on their own and
in relation to each other. Rather than cultural references, semiotic resources are
conceived as forms of expression (i.e., signifiers that are associated to meanings/
signifieds to make signs), fully capable of a range of meaning potentials, which have
accumulated through past uses among specific social groups. These socially shaped
(culture-specific) meaning potentials are not only denotational but also affective,
identity shaping, and constructing of register, tone and style, mood and modality,
as well as cohesion and coherence within the overall representation. Taking the
example of the anime bleeding nose (Figure 3.1), the way the bleeding is repre-
sented (as a burst or as a dropping, etc.) and its relation with the character’s facial
expression and body movement modulate the (sexual arousal) feeling expressed as
well as the character’s positioning towards it; its relation with the soundtrack and
other filmic effects (e.g., stills, flashes etc.) shapes the overall tone of the scene (e.g.,
as dramatic, romantic, or humorous).
Adopting either the context or the co-text approach involves crucial epistemo-
logical differences and consequences in terms of the decisions to be made in trans-
lation. Following the “context” approach, when considering, for example, visuals
to be circulated to different audiences, it is a matter of deciding whether and how
to fill the gap of shared/background knowledge needed to understand the cultural
reference, as when facing cultural references in language; it is in sum a matter of
knowledge about the world that is depicted in the visual (i.e., cultural knowledge),
rather than knowledge of how the visual shapes the world that it depicts (i.e., semiotic
knowledge).
When adopting the “co-text” approach, by considering all semiotic resources as
signs, questions open not only for the cultural knowledge needed to understand
the referential/denotational meaning (i.e., to grasp the cultural reference). They
involve the required semiotic knowledge, i.e., the knowledge needed to understand
how resources are used to make meaning at all levels. This means considering the
meaning potentials of specific gestures, facial expressions, proxemics, music, filmic
effects, camera angles, colour, clothing, and so on (as well as their combination)
to represent reality, shape affect, identity, politeness/distance, formality/informality,
and to modulate truth values, for example.
In sum, in considering meaning as multimodally constituted and orchestrated,
the co-text approach assumes complementarity of resources in a multimodal com-
plex; hence, in a representation, all resources are used following or innovating from
past uses for meaning-making in all its aspects. These interact with all others pre-
sent, requiring translators to make holistic multimodal choices as to what needs
to be translated, from which modal resource, into which other modal resource,
on the basis of an assessment of the target audience’s semiotic knowledge in all
modes, rather than solely of their knowledge of the cultural references of the source
text. From this derives a second issue, which is yet to be tackled in social semiotic
research, as to the mapping of semiotic knowledge across “cultures,” and hence to
defining culture in semiotic terms.
2.2 Semiotic knowledge and the issue of culture

If, to understand each other, meaning-makers need to share (some extent of)
semiotic knowledge, that is, knowledge of semiotic practices in all modes at
play (rather than only cultural references), then the issue of what is shared/non-
shared in any form of expression achieves primary significance. This is true in
all communicative events (as when assessing how an interlocutor will interpret a
certain gesture in face-to-f ace interaction), and becomes crucial when designing
or adapting multimodal texts for transnational circulation, since we can hypoth-
esise that as circulation widens, so the range of meaning possibilities increases
(Rymes 2014).
A main assumption to be questioned is that boundaries in shared uses of semi-
otic resources such as image or music match linguistic boundaries/communities;
this assumption is not grounded empirically, yet it is widely present both in studies
on intercultural communication and in localisation practices (e.g., when replac-
ing images on different language versions of a company website). Questioning (or
verifying) it involves considering the historically different dynamics of circulation
of resources other than (what has been normalised as) language. This has been
underexplored in social semiotic research, while disciplines such as musicology
or iconography have specialised only on individual modes; yet it is, we argue, of
primary importance for the examination of the relation between translation and
multimodality in today’s globalised world.
2.2.1 Language as an exception: National codification and

translation tradition
Nation-states have historically devoted extensive political, economic, and educa-
tional effort into codifying language use among their citizens,3 through a series
of homogenising forces, including dictionaries, grammars, and literacy enterprises
through the education system and national mass media. Ideologically shaped
notions of “national languages” (or standards) have been constructed through selec-
tive normativity on the mode of writing (imposed also onto the mode of speech,
particularly in prestige forms and formal contexts, as “correct” usage). National
codification and standardisation of linguistic form/meaning associations have been
strengthened also through cross-linguistic comparison, the production of bilingual
dictionaries, and foreign language education. By establishing systems of equivalence
between national languages, these mono-, bi-, and multilingual literacy enterprises
have taught us how to read, interpret, and use linguistic resources, thus constraining
their range of meaning-making possibilities.4
Other semiotic resources have not been subject to the same systematic extent
of national codification. Nation-states have not put the same effort into codifying
gestures, music, or filmic conventions as in codifying writing. The propagation of
hegemonic uses of different semiotic resources varies, so certain modes have gener-
ated conventions of use in certain practices because of their professionally oriented
centres of prestige (e.g., influential groups of film-makers, photographers, musicians,
graphic or fashion designers), while certain others, like gestures and facial expres-
sions, have developed more through face-to-face contact (and then circulated in
visual representations), possibly standardised as best/correct practices of behaviour
in certain contexts (e.g., “etiquette”).Yet, centres of prestige for sign-making prac-
tices in nonverbal modes have historically been more transnationally connected.
Descriptions of “prestige/authoritative” uses of specific semiotic resources from
authoritative practitioners and critics (e.g., manuals and treatises on graphic design,
photography, film, architecture, and music) have normally originated and circulated
from internationally influential centres/schools/traditions (consider, for “Western”
areas of the world, e.g., the reach of Christian iconography for image since pre-
nation-state times, the circulation of dress codes around Europe and through colo-
nisation, up to, e.g., Hollywood for films). More significantly, nonverbal modes did
not undergo national standardisation through literacy enterprises, and have been
therefore less influenced by national homogenising forces and systems of equiva-
lence in form-meaning associations.
Only writing (and speech), particularly in its ideological notion of “national lan-
guage,” has systematically been subject to a history of (intra-semiotic) translation.
Translation of language too, while mediating between different languages, has para-
doxically also contributed to strengthening the divide between national linguistic
communities, who received representations mediated into “their” language, rather
than being exposed to the language of “others” and forced to make (their own)
meaning out of it, as happened with visual representations or music, for example.
The exceptional history of codification and translation that language has under-
gone has ultimately contributed to determining different dynamics of circulation
and meaning-making of language than of other resources.
2.3 Meaning-making in a world of untranslated signs

If we adopt a semiotic perspective and conceive of anything in our environments
as a potential sign, we come to realise that we live in a communicative world of
mainly untranslated signs; not only is language an exception, because of its histori-
cal national codification processes, but so is translation, as a form of cultural and
semiotic mediation. With the exception of writing and speech, we normally make
meaning without anybody or anything mediating between us and the semiotic
environment. Rather than making signs in nonverbal resources more universally
understood, this makes interpretation of form/meaning association at the same
time more individualised and shared within social groupings that cross national
boundaries.
Hence the coincidence of semiotic knowledge in these modes with a nationally

defined linguistic knowledge is a problematic assumption (and ultimately reinforces
an essentialist view of culture, long-time dismissed in culture research). Further
enhanced by the transnational and fragmented/multiplied character of communi-
ties and “affinity spaces” (Gee 2005) in today’s globalised world, a given target audi-
ence identified for language translation cannot be assumed to be homogeneous in
its interpretation of other resources. With blurred boundaries between foreign and
local signs, intersectional variables may be more significant in driving interpreta-
tion, along with individuals’ personal trajectories, in terms of which communities
of practice/interest and affinity spaces specific individuals participate in, and of
which sign-making practices they have been more exposed to and have become
more familiar with, and hence have entered their semiotic (rather than linguistic-
only) repertoires. In the example of the anime nosebleed mentioned earlier, its
association with “sexual arousal” may be known to anime fans speaking different
languages and yet not known to a Japanese speaker who comes across that sign in
anime for the first time. In other terms, sharing (more or less) the “same” language
does not necessarily coincide with the sharing of semiotic knowledge for image
or music, for example, and the meaning the latter two construct in their relation
with language.
Adopting a co-text approach to meaning as multimodally constituted and con-
sidering issues of shared/non-shared semiotic knowledge across linguistic com-
munities (because of the different circulation dynamics of resources other than
language) has far-reaching implications both for MS and TS. On the one hand,
while what we label here a co-text approach is a long-established assumption in
MS, the issues of transnational circulation and national codification have not been
adequately reflected upon in the field, precisely because of its little concern so far
with translation as broadly defined; in this sense, a joint discussion and investigation
with TS (eminently concerned with issues of “culture”) can help social semiotic
research to fill the gap, as proposed in the research agenda in section 4. On the
other hand, adopting a co-text approach and considering shared/non-shared semi-
otic knowledge in all resources constitutes a crucial epistemological shift for TS,
with far-reaching implications not just for research, but also for translation practice
and training. In section 3 we discuss how adopting such an approach forces us to
broaden the limits of translation and revisit some of the most fundamental concepts
in TS.
3. Issues for a way forward in translation

To acknowledge the multimodal nature of communication means accepting that the
realms of translation and TS include more than words in context. It means accept-
ing that translation needs to consider all modes and the meanings they promote (on
their own and in relation to other modes) but also that all resources co-occurring
with writing/speech are signs in their own right that might present different chal-
lenges to (different) viewers. This constitutes a fundamental shift which implies a
different understanding of translation and leads us to revisit fundamental concepts

such as “text,” “source text” and “target text,” or “equivalence.” It also opens the
question on whether, as socioculturally shaped resources, modes other than writ-
ing and speech might need to be translated and how that could be achieved. The
discussion on the (non-) translation of nonverbal resources (which is addressed, in
different ways, by Marcus Tomalin (Chapter 6), Helen Julia Minors (Chapter 7), and
Tamarin Norwood (Chapter 8) in this volume) comes hand-in-hand with issues to
do with the aesthetics, authorship, prestige, and function of multimodal texts, which
are in fact some of the issues we would like to raise in this chapter.
Some of these questions have been asked by the first scholars in the field of Audio-
visual Translation (AVT), who tried to reflect on what it meant for TS to expand its
basic concepts and include products composed by speech/writing along with image,
sound, or movement (Gambier 2003). The discussion, however, has not matured much
beyond the initial questioning, and nonverbal resources have remained reductively
considered only as contextual elements (see section 2.1). Multidisciplinary approaches
have been suggested (Cattrysse 1992; Remael 2001; Gambier 2013; Pérez-González
2014), and MS is one of the areas that researchers have turned to on their quest for
new analytical tools/frameworks purposely developed to examine multimodal texts.
However, taking advantage of the analytical frameworks of a different area of study
demands reflection on the implications deriving from the assumptions underpinning
those frameworks. In the next section we will reflect on the implications that the
issues introduced in Section 2 have for TS.
3.1 Multimodality and the concept of translation

As a first crucial question, if meaning is achieved through intermodal relations
that are culturally specific, how can we assume that the target audience can access
such meaning when only one of those modes is translated? Setting any reserva-
tions around issues of “meaning transfer” (for which we may prefer Kress’s phrase
“transposing meaning” – see his chapter in this volume, Chapter 1) and “equiva-
lence” aside, it must be recognised, however, that practices translating only the ver-
bal (e.g., subtitling, dubbing, translation of comics) implicitly or explicitly assume
that (i) nonverbal resources are universal and easily interpreted without any further
mediation; and that, as a result, (ii) no new intermodal relations and meanings are
introduced when the verbal resources are replaced by verbal resources in another
language. These assumptions seem difficult to reconcile with the founding notion
that translation is about meaning in context, leading us to revisit concepts such as
“text,” now including multimodal products, as well as “source text” and “equiva-
lence,” given that all modes are included in what is considered for translation, and
equivalence is sought for verbal and nonverbal modes alike as well as the meanings
erected through intermodal relations.
Such logocentric assumptions do not seem to be true in all areas of translation or
to all translation practitioners. Those involved in fansubbing, for example, openly
recognise the role played by nonverbal resources in meaning creation along with
the difficulty users might have in interpreting them. The subtitling developed in
this context has challenged professional practice (deemed unable to account for the
complexity and multitude of levels of meaning put forward by the source text) and
boasts an innovative set of mediating/translation procedures of verbal and nonver-
bal resources alike. Another case in point, within professional practice, is localisa-
tion. In this context, a high-level manipulation of the source text is accepted and
strategies of mediation/translation are extended to nonverbal resources (alongside
verbal ones), including, at times, the replacement of source nonverbal resources by
target nonverbal resources, if deemed necessary in the context of the translation
brief. Even if only at a glance, the discussion of these two cases immediately reveals
a symptomatic difference between professional and non-professional practice with
regard to their understanding of translation. Non-professional translation results
from a widening of the concept of translation beyond the verbal. Professional prac-
tice either does not consider the potential need to translate meaning nonverbally
expressed or, when it does, it does not consider it within the boundaries of transla-
tion. Revealing in this sense is the fact that localisation is acknowledged not as a
form of translation (Esslink 2000) but as a broader process that includes the transla-
tion of the verbal mode and the “adaptation” of nonverbal modes. While replac-
ing one language with another is considered as a necessary change, mediating/
translating an image (whatever the procedure used), for example, is considered to
be a high-level intervention, which is referred to in the industry with words such
as “creative,” “free,” and “adaptation” (Jiménez-Crespo 2013). The development
of different terms certainly indexes the need to distinguish a type of translation in
which function and commercial purpose are the priorities, but it also expresses the
need for a different set of knowledge and skills than the ones translators tradition-
ally have, as well as a different set of tools and the extra associated costs. In this con-
text, translation remains defined within an understanding of equivalence limited to
the verbal mode and one in which other resources, given their assumed iconicity/
universality, are to be engaged with directly, in an unmediated way, or else their
“original” meaning would be lost. However, the consideration of resources other
than the verbal, the different possible levels of intervention according to transla-
tion brief, prestige, or symbolic capital associated to the author or source text are
something that translation theory should be able to account for within functional
approaches to translation. After all, the existence of different translations following
different strategies to cater to different audiences or fulfil different functions is not
a new phenomenon for translation studies. Yet, to extend such considerations to
resources other than the verbal, a new paradigm or conceptualisation of translation
is needed.
Further research on meaning-making practices is needed to allow the discus-
sion on translation and equivalence to broaden its scope and consider not only
the meanings expressed by all resources but also the possible unexpected results of
current translation practices. Target multimodal texts might be assuming a different
profile than the one expected due to the fact that, contrary to common assump-
tion, the newly translated verbal resources establish a new multimodal ensemble
in the target context when co-occurring with non-mediated nonverbal resources.

Another important aspect calling for further conceptual and empirical analysis is
the fact that recent technological developments have made manipulation of non-
verbal resources both easier than before and a common practice with a growing
number of text types. Replacing source images by target images is recurrently being
used in the localisation of animated films (as in the case of Doraemon, in which a
bento box was replaced by a pizza, Chaume 2016), without the corresponding
theoretical and empirical analysis to ground the mediation process.
3.2 Translation of meaning rather than elements

Another crucial question deriving from considering meaning as multimodally con-
stituted is then “what” to translate when a text needs to reach a different audi-
ence. Considering elements such as images, gestures, or sounds as socially situated
signs means acknowledging that (i) their meaning is positioned in time and space;
(ii) it is mediated by specific traditions of use within that specific (target) context
(which in turn shape audience’s expectations/interpretations); and (iii) part of what
is being expressed comes from the purposeful selection and organisation of specific
resources in the context of a given product. The reality instead is that target audi-
ences are left to make meaning without any type of mediation between them and
what can be very complex semiotic environments.
Different levels of meaning pose specific challenges to viewers from a different
context. Drawing on previous work (Kovačič 1995; Chesterman 2005, 2007; Gam-
bier 2013; Ramos Pinto 2018), we propose three levels of meaning, adapted from
the “representational,” “interpersonal,” and “textual” metafunctions used in social
semiotics. The first level is the most obvious one, i.e., what is being represented,
and the difficulties the audience might have in identifying a given gesture or object,
like a floppy disk in a 1980s advert, for example. The inability of a younger genera-
tion to identify floppy disks has even been used for humorous purposes, as seen in
a popular Internet meme with the sentence “I showed my 12 year old son an old
floppy disk . . . He said: ‘WOW . . . Cool . . .You 3D printed the save icon!’ ”5 Visual
and aural elements, for example, might often allow for a more immediate first-level
identification by appealing to the audience’s knowledge of the world; however, this
may not always be true, either because the sign is not part of the audience’s context
(like floppy disks for 12-year-olds), or because there is no natural resemblance with
the world (e.g., blue curved stripes used to signify [waves of] freshness in a tooth-
paste package).
The second level refers to the social meaning associated to a given object, sound,
or gesture and the difficulties an audience might face not because they cannot iden-
tify the referents in question but because they might not share the necessary socially
shaped semiotic knowledge to interpret them. In the television series The Big Bang
Theory (s11, ep22), the character Amy, after trying on several wedding dresses that,
following (Western) mainstream socio-cultural fashion trends, could be interpreted
as “modern/fashionable/sexy” (Figure 3.2), decides on a dress (Figure 3.3) that
FIGURES 3.2, 3.3, and 3.4 Amy, Penny, and Bernadette go wedding dress shopping
Source: (The Big Bang Theory, s11, ep22, 00:07:39–00:08:22)
causes surprise to her friends (Figure 3.4). The viewer is only able to understand
this reaction if he/she is able to interpret the chosen dress’s social meaning as
“old-fashioned/tacky/not-sexy.”
Notwithstanding the different affordances of subtitling and dubbing, the chal-
lenge for translation thus comes from the fact that, even when nonverbal resources
build on a shared resemblance with the world (identifying the dress as a wedding
dress would not be challenging to most viewers), presenting them as in the source
text without further mediation might not ensure cross-cultural transfer at the level
of social meaning: this type of wedding dress can be instead common in certain
social groups, who, if not accustomed to current Western fashion trends, may inter-
pret it as “beautiful” and find it challenging to understand Amy’s friends’ reaction.
Perhaps more importantly, due to the common belief in the universality of non-
verbal resources and the common practice of being left alone to interpret them,
the audience may not recognise the existent “cultural bump” (Leppihalme 1997)
and proceed interpreting on the basis of the social meaning promoted in the target
context. This is the case not only for laypersons but also for translators whose train-
ing does not traditionally include multimodal analysis as discussed in this chapter.
Translation practices such as professional subtitling seem to deny such potential for
confusion, but practices such as localisation and fansubbing together with recent
reception data (Chiaro 2014) seem to point otherwise.
The final level refers to the meaning resulting from the intermodal relations
established between the different modes for specific diegetic purposes. In the wed-
ding dress scene, the verbal mode with Amy saying “This is it. This is the one!”,
combined with the dress-as-sign, with its specific social meaning, and the friends’
facial expressions of surprise, produce a comedic moment (supported by a live
audience’s laughter) that reinforces specific diegetic meaning, namely Amy’s char-
acterisation as “geek” and hence peculiarly not associated with the social world of
the other characters (which viewers are required to recognise).6
The challenge thus comes from the need to ensure that the audience is able to
identify not only the elements in the image but also their social and diegetic mean-
ings. This discussion has not had the attention it deserves in professional practice;
even translation practices focused on accessibility such as Subtitling for the Deaf
and Hard of Hearing and audio description (which have long called attention
to the importance of nonverbal modes) often show a strong focus on describing

sounds or images to ensure identification, undervaluing considerations of the audi-
ence’s (in)ability to interpret the social or diegetic meaning even after identification
has been made possible.
The intricate relationship between the different modes brings an additional chal-
lenge related to the difficulty in distinguishing the contribution provided by each
mode and to have clearly defined translation units. Notwithstanding the doubts
raised by Stöckl (2004) regarding the independent existence of contributions, one
possible way forward might come from results in perceptual psychology, which
suggest that perception is selective, i.e., that “we attend to objects that bear salient
meaning for certain goals” (Gibson 1979, 48). Furthering our understanding of rel-
evance and narrative salience, expanding it to meaning produced multimodally (cf.
Section 4) could be helpful in this context, as well as integrating the contribution
from specific disciplines for different genres (like film studies and media studies).
3.3 Translation without nationally codified resources

Alongside the “what” to translate comes the issue of “how” to translate nonverbal
resources. In localisation practices, source nonverbal elements are often replaced
by other elements of the same mode more familiar to the target audience (e.g.,
images or colours used in packaging). Similar procedures of substitution involving
image manipulation are sometimes used in other genres such as comics (e.g., when
someone bangs Asterix in the head, he sees “birds” in the French edition and “stars”
in the Portuguese, Zanettin 2014) or children’s literature (e.g., Arabic editions of
Disney’s Cinderella often present female characters wearing a hijab, Zitawi 2014).
Non-professional practices such as fansubbing present other solutions by taking
advantage of headtitles and pop-up balloons for annotation of nonverbal meaning
or different colours for character or tone identification (Pérez-González 2014).
While localisation seems to assume that nonverbal resources are to be translated
by target resources of the same nature, the use of colours and verbal headtitles in
fansubbing suggests that the meaning expressed in one mode can be translated by
resources in a different mode. These practices, however, have been (and still are)
developed on the basis of intuition and most often not by translators or trained pro-
fessionals. To enable informed decisions, data are needed on the effects of existing
and new translation strategies onto meaning-making.
With the lack of national codification of nonverbal resources, translators cannot
draw on a system of equivalence to inform translation practice or a translation tra-
dition against which positioning their choices. Some genres have developed specific
conventions of use that audiences assimilate through exposure; besides the already
mentioned example of anime, comics developed a tradition on reading direction
and representation of sounds (Zanettin 2014), and the same applies to expectations
built up by music in horror films (Bordwell and Thompson 1979). Such convention
development does not follow the language boundaries in which the translation and
distribution industry is organised. The multimodal nature of texts seems thus to
bring additional layers of complexity to the already challenging concept of “target

audience”; the heterogeneous character of a product’s audience is multiplied, given
that this might differ in semiotic knowledge, making it more difficult to make
decisions, and leaving the translator at the crossroads of different traditions. The
lack of research on shared/non-shared semiotic knowledge will continue to leave
practitioners without a way forward or enough data to consider the implications
of their decisions.
3.4 Translation of nonverbal resources as a mediated activity

As with any decision-making process in translation, the (non-)translation of non-
verbal resources will also be mediated by contextual factors. As previously men-
tioned, the technological development of recent years has opened a number of
options ranging from image manipulation to pop-up balloons and dynamic subti-
tles, meaning that the nonverbal nature of certain resources no longer immediately
constrains the mediation processes or define the translation strategy (and its higher/
lower degree of manipulation or no manipulation). Other contextual factors need
to be considered in this respect. Given our tolerance to image manipulation in
advertising, for example, and the likely rejection of such strategy when translating
a film, it will be important to investigate the correlation between translation strate-
gies, translators’/viewers’ attitudes and issues such as genre, authorship, prestige (of
the director and film), function of the text, and audience expectations.
4. Conclusion: A joint research agenda

With seeing the world as a multimodally orchestrated meaningful reality comes the
realisation that actual translation is infinitesimal compared to the signs presented
to us or surrounding us that are not translated. If this has always been the case
without being perceived as an issue, then why do we need to further research on
“multimodality and translation”? Without wanting to frame the issue of “equiva-
lence” in terms of faithfulness, we believe that questions need to be asked regard-
ing the profile of the texts target audiences have access to, but also how this might
be indirectly reinforcing power imbalances. After all, due to their transnational
prestige and wide distribution, Western sign-making practices are possibly more
easily interpreted (and their underlying values and ideologies absorbed), as in the
case of The Big Bang Theory examined earlier. This might lead to standardisation of
meaning-making practices in favour of Western practices, while pushing divergent
practices (and the “other” behind them) to the periphery, which we have more dif-
ficulty accessing and interpreting.
We hope to have highlighted the importance of considering translation beyond
the verbal but, in doing so, we know how many unanswered questions we are left
with. In TS, an increasing number of works analyse translation and localisation of
multimodal products (O’Hagan and Mangiron 2013; Dicerto 2018), yet the cases
examined are still very specific and isolated, making it difficult to compose a wider
picture of generalizable significance for TS as a whole and day-to-day translation

practice. Useful theoretical discussions have so far made sense of the state-of-the-
art on the issue (O’Sullivan and Jeffcote 2013; Pérez-González 2014; Gambier and
Ramos Pinto 2016; Dicerto 2018), yet the lack of systematic investigations on
the subject makes it difficult to derive broader frames/approaches that enable us
to address the relation between multimodality and translation more holistically.
We strongly believe that it is no longer possible to move forward without a more
robust theoretical framework grounded on relevant empirical data yet to be col-
lated. In this concluding section we sketch the work needed in semiotic and transla-
tion research to further our understanding of the implications of multimodality for
translation, to ground empirically the decisions to be made in translation practice,
as well as to foresee their possible implications. We present the types of research
questions and the lines and areas of enquiry, along with the methodological and
theoretical integrations required to attempt to answer them.
4.1 Research questions

Faced with a semiotic landscape of transnationally circulating untranslated signs
(except language), the semiotician asks “how do people make meaning out of
this?” While issues of culture specificity in meaning-making have been so far dis-
regarded, social semiotics needs to start also asking “who shares (some extent of)
semiotic knowledge about the uses of specific resources, and for the meaning made
through their relations?”; this means asking “what is shared/non-shared culture in
a given semiotic landscape?”, “how do the ways in which different resources and
signs circulate influence the ways in which they are interpreted?”, and “how does
the participation in different affinity spaces/communities influence an individual’s
meaning-making of nonverbal resources?”
In conjunction with insights deriving from addressing the aforementioned-
questions, TS needs firstly to provide descriptions and explanations about past and
current translation practice (following a historical and cultural approach), asking
questions such as “how have multimodal products and nonverbal elements within
them been approached in translation?” and “how has that mediated the circulation
and reception of translated multimodal products as well as our understanding of
the other(s)?” Secondly, and possibly grounded empirically on findings from all
of these questions, TS needs to seek ways to provide support for informed deci-
sions in translation practice, thus asking questions such as “how do we identify the
elements in need of translation considering that meaning is multimodally con-
structed?”, “how do people interpret translated multimodal products?”, and “what
is the impact of specific translation strategies in cross-relation with factors such as
modes, media, genre, domain, audience, and purpose?”
We do not want to reduce the aims of social semiotics and TS research to a mere
provision of indications for translation practice and training. Providing answers to
these questions can further understanding into the social dynamics of meaning-
making and into the very nature of translation, while supporting better-informed
decisions in translation practice and kickstarting a reflection on how translation

training could develop to answer today’s challenges.
4.2 Lines and areas of research

To start addressing these questions, a first broad line of enquiry involves investi-
gating how shared/non-shared semiotic knowledge distributes across populations
and individuals participating in different semiotic spaces. A second line of enquiry
involves investigating how humans make meaning the first time they engage with
specific signs.
4.2.1 Distribution of semiotic knowledge: modes, media, genres,

and domains
The first line of enquiry requires examining sign-making and meaning-making
practices in (at least) three intertwined areas. One area involves investigation of spe-
cific semiotic modes to understand their historical development and the extent of
codification and dynamics of transnational circulation (which may be different for
different modes). A second area involves investigating meaning produced multimo-
dally in/across different media, as the way we make meaning may differ between a
printed advertisement and a social media meme due to media-specific expectations
regarding purpose, kinds of authorship, context, etc. A third area needs to exam-
ine sign-and meaning-making practices in specific genres and domains, with their
own specific communities, how these differ and/or distribute transnationally (the
resources of image have not undergone the same type of codification in, e.g., tech-
nical drawing, cartoons, and maps; some theatre traditions, like the Japanese one,
have had a different transnational reach and higher level of national codification
than others), as well as how they influence each other across genres and domains.
A better understanding of meaning-making practices and how these circulate
will assist translators in their own interpretation of the source text and evaluation of
what needs to be translated. It will also assist TS in mapping the development/cir-
culation of translated multimodal products and examination of translators’ choices.
The intertwined areas of investigation (mode, media, genres, and domains) on
shared/non-shared semiotic knowledge need to be (i) situated in and across time/
space and (ii) mapped onto and across linguistic communities, for multimodal texts
that also involve (and require translation of ) language. Investigation through time
(which would give way to historical social semiotic research) needs to include mul-
tiple timescales (Lemke 2000, 2009) and be considered in its inseparable relation
with space (as chronotopes, Bakhtin 1981). This involves a radical rethinking of
notions of boundaries, which may often not coincide with language/national bor-
ders. Especially in today’s globalised and technologically connected societies, space
needs to be less geographically (and linguistically) conceived and more socially
defined (following conceptions of space and place that stem from Lefebvre 1984;
for a review, Jaworski and Thurlow 2010). The mapping of semiotic knowledge
onto (more or less linguistically homogeneous) social groups can and needs to lead
to a re-definition of social group in a transnationally connected world (unevenly
distributed between hegemonic and minoritised cultural flows). Given that indi-
viduals participate in multiple and multilingual spaces/communities/groups (as in
the example of anime fans mentioned earlier), each with specific semiotic practices,
investigation should test the advantages of adopting notions centred on spaces (as
in Gee’s 2005 affinity spaces) rather than those centred on groups/communities,
possibly also contributing to a shift in perspective within TS from the problem-
atic notion of source/target audiences to that of source/target spaces (of shared
semiotic knowledge). A perspective on spaces where texts and semiotic practices
and knowledge circulate would avoid associations to people’s membership to com-
munities (which trigger epistemologically dangerous national/local labelling, e.g.,
the Japanese, the Germans, etc.), while focusing on practices (rather than people’s
belonging) and embracing more immediately the fluid character of their circulation
in, out, across, and through different spaces.
4.2.2 Principles of semiosis

Investigation of how modes have developed specific uses, how signs circulate
through different media and in specific genres and domains, how their meanings
change across time/space, and how shared/non-shared semiotic knowledge distrib-
utes across affinity spaces, social groups, and linguistic communities cannot be done
through fixed categorisations, which would risk producing artificial inventories/
taxonomies. These investigations need to be dynamically conceived; individuals’
semiotic repertoires change and vary constantly as they encounter new signs/rep-
resentations and engage in new interactions; in other terms, semiotic repertoires are
relational, adaptive, and flexible (see applications of Complexity Theory to language
development in Cameron and Larsen-Freeman 2007; Larsen-Freeman 2015).
Consequently, rather than compiling a set of formal tools of reference for, e.g.,
visual signs, a second line of enquiry needs to account for the dynamic complex-
ity of human semiosis; thus, the mapping of shared/non-shared semiotic knowl-
edge needs to be paired with research focused on understanding general/common
meaning-making principles and strategies. If shared semiotic knowledge enables
common ground of interpretation between sign-and meaning-maker (e.g., inter-
preting the floppy disk button as the “save” functionality), then in-depth under-
standing of common principles of semiosis can enable the formulation of informed
hypotheses on how a sign will be interpreted when it is engaged with for the first
time (as when encountering a new symbol for the button of a given software
functionality, or a new filmic effect), thus further supporting informed decisions
in translation practice. This can be done by empirically testing current theories of
semiosis, such as Kress’s (1993, 2010) motivated sign (cf. Section 1), as well as by
testing against the whole semiosis current theories on meaning-making developed
for language, such as Relevance Theory (Sperber and Wilson 1986) and Conceptual
Metaphor (Lakoff and Johnson 1980).
These two lines of enquiry, grounded in and across time/space, and situated in
and across different modes, media, and genres/domains, are the first necessary steps
for an empirically informed formulation of hypotheses on how a given resource
will be interpreted by members of a given audience (or by appearing in a given
affinity space), whether any mediation will be needed, and what kind of strategy
would be most apt.
4.3 Methodological developments

The research that we envisage demands methodological development and integra-
tion to encompass exploration of (i) multimodal products and how these circulate,
(ii) the practices of multimodal text production and translation, and the discourses
attached to them, as well as (iii) practices of (re)interpretation and re-signification.
Investigating the research questions in the lines and areas of enquiry sketched
here requires extensive descriptive studies on actual multimodal products. These
demand new tools of multimodal analysis capable not only of integrating the inves-
tigation of all modes in equal terms, as well as the intermodal relations and their
translation, but also of enabling quantitative analysis of larger datasets that can offer
generalizable results. Traditional social semiotic multimodal analysis has developed
tools for fine-grained investigation of small samples of multimodal texts. These
findings can hardly be combined with those from textual corpora analysis (which
takes advantage of quantitative data analysis), thus reinforcing even further the une-
qual treatment of nonverbal resources vs. language. Development of multimodal
corpus analysis requires implementation of software tools for data mining of non-
verbal resources and integration of extant technologies for visual recognition.
Alongside larger-scale examinations of multimodal products and the identi-
fication of regularities in sign-making and its translation, it will be essential to
investigate both ends of production and what is commonly termed as “reception”
in TS. We need methodological integration to examine the discourses and prac-
tices of text production and translation as well as the audiences’ interpretations and
the impact of translation strategies on such interpretations. By cross-relating the
regularities found in the analysis of multimodal products with contextual factors,
we will be able to examine the social, economic, and cultural factors promoting
specific multimodal sign-making and translation practices, as well as to map their
development, circulation, and reception. This will require the undertaking of large
sociological investigations that revisit Bourdieu’s (1986) methods and variables for
mapping “cultural capital” against the contemporary socio-semiotic landscape (for a
problematisation of the concept, Adami 2018), as well as extensive reception studies
focused on collecting data on meaning-making processes across different variables
such as genre, purpose, and type of audience. Data collection methods range from
interviews, surveys, and questionnaires to eye-tracking and memory recall protocols
(for an overview, Pérez-González 2014).
Three further methodological issues need to be addressed. Firstly, most
data-collection methods and means of academic dissemination are still heavily
logocentric, forcing both research participants and researchers to describe multi-

modally produced meaning almost exclusively through language (be it speech for
participants’ interviews in reception studies or writing for both questionnaires and
for academic publications). The exploration of meaning of nonverbal resources
requires us to develop ways to elicit participants’ feedback beyond verbal output
to avoid possible risks of forcing meaning into hardly apt linguistic categorisations
(which, moreover, require certain levels of literacy and articulation). Some meth-
odological attempts in this sense are being developed in social semiotics (as when
asking participants to draw rather than to tell their meanings, e.g., Kress 2010) and
in sensory research (e.g., asking participants to gesture or make shapes to express
what they taste).7 These methods need to be systematically developed and adopted
also in studies concerned with meaning of translated multimodal products. Parallel
to forms of nonverbal data collection, we need new forms of multimodal dissemi-
nation of findings; research in “multimodality and translation” could contribute
further to push the academic publishing system to promote (recognition of) forms
of dissemination such as visual and video essays, as well as online publishing that
exploits the hypertextual and multimedia affordances of the web (see the journal
Kairos, http://kairos.technorhetoric.net/).
Secondly, even if very much needed to fill a gap in multimodal research, large-
scale quantitative investigations are not enough to address the research questions
mentioned earlier. Big data analysis can incur the risk of overgeneralising differences
in meaning-making practices onto macro-categories, such as assumed “national
cultures” or “linguistic communities,” while screening out crucial micro-level vari-
abilities. To explore how variables such as individuals’ personal trajectories and
participation in multiple semiotic spaces influence their meaning-making of non-
verbal resources (and in their relation with language), there is a need for in-depth,
fine-grained qualitative research. This requires integration with ethnographic
approaches, which is already underway in social semiotic multimodal analysis (e.g.,
Dicks et al. 2011, although not focusing on “culture/semiotic knowledge”), and
could constitute a methodological innovation for TS, leading the way to an ethnog-
raphy of translation. Findings at a more qualitative level could inform the testing of
hypotheses at the quantitative level, while the latter results could be further probed
qualitatively, through iterative cycles of analysis.
Lastly, the type of multi-layered examination proposed here will only be pos-
sible if we embrace large-team (quantitative and qualitative) examinations involving
both professional and “lay” designers and producers of multimodal products, as well
as audiences, alongside researchers in social semiotics, TS, sociology, ethnography,
cognitive linguistics, and statistics, among others. Research could thus explore par-
ticipatory methods more extensively. Analogous to the role of citizen sociolinguists
advocated by Rymes (2014), both semiotic and translation research should con-
sider the value of today’s distributed knowledge and start conceiving of contribu-
tions outside of academia in the form of “citizen socio-semioticians” and “citizen
translation scholars.” We all reflect and meta-comment not only on our and other
people’s use of language, but also on images, clothing, font, music, architecture, and
all sign-making practices that we encounter, as well as on actual translations and

how we think these could better fulfill our needs. Given the breadth and width
of the phenomenon to be investigated and its research directions, knowledge and
resources need necessarily be drawn also from outside the confines of academia.
4.3 Theoretical integration

Such a broad and articulated research enterprise needs necessarily to cross disci-
plinary boundaries, not only between social semiotics and TS. We list here four
transdisciplinary directions:
• Integration is needed with linguistics in several respects. To the suggestions

mentioned throughout the chapter, we add the potential contribution of stud-
ies on relevance (in the tradition of Sperber and Wilson 1986, yet re-defined in
a co-text rather than context approach), narratology salience (Bortolussi and
Dixon 2003), and cognitive load and processing (Kalyuga 2012), as well as a
cultural mapping and comparative expansion of extant cognitive approaches to
multimodality (e.g., Forceville and Urios-Aparisi 2009).
• Knowledge on the development and circulation of specific semiotic resources
in different contexts can be drawn from specialised fields, such as visual com-
munication, media studies, graphic design, musicology, art history, and film
studies.
• Transdisciplinary crossing is required also with studies on culture in anthropol-
ogy, sociology, and intercultural communication, to test and re-define different
notions of culture against a co-text rather than context approach. In this, social
semiotics could use extant definitions of culture to verify variation in patterns
of semiosis, while studies on culture could benefit from a semiotic perspective
to redefine notions of culture (in a co-text rather than context approach). This
transdisciplinary undertaking could also facilitate reconceptualisations of target
audience/space in TS.
• Because of the multiple dimensions of human semiosis, which involves not
only social but also psychological, biological, sensory/perceptual, and material
aspects and variables, further transdisciplinary engagement is required with
other disciplines, such as psychology, sensory ethnography, material semiotics,
and cognitive neuroscience, each being equipped to investigate one specific
aspect, while these need to be considered in their complex intertwining.
The joint semiotic/translation research enterprise sketched here is only prelimi-

nary; it requires refinement and further articulation, possible only once investiga-
tion has started; yet, as is, it is already indicative of its wide, broad, and extensively
ambitious scope and reach. The whole enterprise is not exempt of risks; we would
like to conclude by mentioning a caveat, which has started to concern us while
discussing the possible implications of such an investigation. A mapping of shared/
non- shared semiotic knowledge onto social groups risks producing selective
descriptions, and hence prescriptions and regulations on uses, which may not only
constrain individuals’ agency in sign-making but also contribute to exacerbating
the divide between hegemonic and minoritised cultural practices (impacting on the
people associated with them). Analogously, description of multimodal translation
practices and formulation of possible (inter-and intra-modal) translation strategies
may risk producing prescriptions and promoting practices that favour a supposedly
easier-to-interpret domestication of nonverbal resources, thus risking to contribute
to separating communities and constraining their meaning-making possibilities as
well as their exposure to diversity in sign-making. Research in sociolinguistics and
applied linguistics (e.g., García 2009; García and Wei 2014; Canagarajah 2017) has
provided ample evidence of the negative effects of national codification, standardi-
sation, and prescription onto language use, pointing to their role in (re)producing
social inequalities. Drawing on the insights of linguistic research on these themes,
the investigative enterprise we are proposing needs to be entirely innovative and
thoughtful of the possible implications, and not merely reproduce or adapt methods
used for language in the past.
Notes
1 The history of the nosebleed to signify sexual arousal in anime is uncertain, and multiple
explanations have been hypothesised; these range from tracing the use back to single authors
(like manga artist Yasuji Tanioka in the 1970s), to unverified popular sayings in the Japanese
tradition used to deter youths from sexual situations, up to identifying the nosebleed as a
taboo-avoidance metaphor for male ejaculation (later extended also to female characters
in anime); for a discussion, see https://kotaku.com/sexual-arousal-doesnt-cause-bloody-
noses-says-medical-5953124 (Accessed 30 August 2018); an online search of ‘anime nose-
bleed’ (and their equivalent Japanese hanaji or 鼻血) generates dozens of different sources,
discussions, and hypotheses. The very range of these substantiates the social semiotic pos-
tulate on meaning-making/interpretation as an individual’s hypothesis on the motivation
underpinning the association of a form/signifier to a meaning/signified, on the basis of the
combined materiality and past uses of a given resource.
2 Social semiotics avoids the label “nonverbal” for resources other than speech and writing,
because of its implied subordination of these resources to language, and because the stem
“verbal” assumes that writing and speech can be considered as one mode (as the label
“language” supposes). We use it here as a shorthand way to label all resources that, unlike
speech and writing, have been considered para-or extra-linguistic and hence context, in
traditional linguistic research and TS.
3 This has applied also to its non-citizen residents (as shown in current controversies on
migrants’ language literacy) and to colonised peoples (e.g., Canagarajah 2017).
4 This is proved, through counter-evidence, by the fluidity of language use, meaning, and
perception among speakers of oral communities (see, for example, Goodchild and Weidl
2018).
5 Meme available from: https://me.me/i/i-showed-my-12-year-old-son-an-old-floppy-
3250837 (Accessed 22 November 2018).
6 We isolate one single sign and shot for exemplifying purposes; obviously, the more audi-
ences are exposed to intertextual relations throughout the series, the more they will con-
struct (shared) semiotic knowledge on the signs of “geeky-ness,” as happens in all exposure
of foreignising translations.
7 On sensory research, see work conducted by CenSes, The Centre for the Study of the
Senses: https://philosophy.sas.ac.uk/centres/censes.
References
Adami, Elisabetta. 2017. “Multimodality and Superdiversity: Evidence for a Research
Agenda.” Tilburg Papers in Culture Studies 177: 1–28.
———. 2018. “Shaping Public Spaces from Below: The Vernacular Semiotics of Leeds Kirk-
gate Market.” Social Semiotics 11 (October): 1–25. doi:10.1080/10350330.2018.1531515.
Bakhtin, Mikhail. 1981. The Dialogic Imagination: Four Essays. Austin: University of Texas
Press. doi:10.2307/2068977.
Bezemer, Jeff, and Gunther Kress. 2008. “Writing in Multimodal Texts: A Social Semiotic
Account of Designs for Learning.” Written Communication 25: 166–95.
Bordwell, David, and Kristin Thompson. 1979. Film Art: An Introduction. New York:
McGraw-Hill.
Bortolussi, Marisa, and Peter Dixon. 2003. Psychonarratology: Foundations for the Empirical
Study of Literary Response. Cambridge: Cambridge University Press.
Bourdieu, Pierre. 1986. Distinction: A Social Critique of the Judgement of Taste. London and New
York: Routledge.
Cameron, Lynne, and Diane Larsen-Freeman. 2007. Complex Systems and Applied Linguistics.
Oxford: Oxford University Press.
Canagarajah, Suresh. 2017. The Routledge Handbook of Migration and Language. London and
Cattrysse, Patrick. 1992. “Film (Adaptation) as Translation: Some Methodological Proposals.”
Target 4 (1): 53–70.
Chaume, Frederic. 2016. “Audiovisual Translation Trends: Growing Diversity, Choice, and
Enhanced Localization.” In Media Across Borders: Localizing TV, Film and Video Games,
edited by Andrea Esser, Miguel Angel Bernal-Merino, and Ian Robert Smith, 68–84.
Chesterman, Andrew. 2005. “Consilience in Translation Studies.” Revista Canaria de Estudios
Ingleses 51 (Novembre): 19–32.
———. 2007. “Bridge Concepts in Translation Sociology.” In Constructing a Sociology of
Translation, edited by Michaela Wolf and Alexandra Fukari, 171–83. Amsterdam and
Philadelphia: John Benjamins.
Chiaro, Delia. 2014. “The Eyes and Ears of the Beholder? Translation, Humour, and Percep-
tion.” In Media and Translation: An Interdisciplinary Approach, edited by Dror Abend-David,
197–220. London and New York: Bloomsbury.
Dicerto, Sara. 2018. Multimodal Pragmatics and Translation: A New Model for Source Text Analysis.
London: Palgrave Macmillan.
Dicks, Bella, Rosie Flewitt, Lesley Lancaster, and Kate Pahl. 2011. “Multimodality and Eth-
nography: Working at the Intersection.” Qualitative Research 11 (3): 227–37.
Esslink, Bert. 2000. A Practical Guide to Localisation. Amsterdam and New York: John Benjamins.
Forceville, Charles, and Eduardo Urios-Aparisi. 2009. Multimodal Metaphor: Journal of Prag-
matics,Vol. 42. Berlin: Mouton De Gruyter.
Gambier, Yves. 2003. “Screen Transadaptation: Assessment and Reception.” The Translator 9
(2): 171–89.
———. 2013. “Moves Towards Multimodality and Language Representation in Movies.” In
Reading and Intersemiosis in Multimedia: Proceedings of the 3rd ICOM Pavia, May 2006, edited
by Elena Montagna, 1–10. Pavia: Como.
Gambier, Yves, and Sara Ramos Pinto. 2016. “Special Issue on Audiovisual Translation: The-
oretical and Methodological Challenges.” Target 28 (2).
García, Ofelia. 2009. “Education, Multilingualism and Translanguaging in the 21st Century.”
In Multilingual Education for Social Justice: Globalising the Local, edited by Ajit Mohanty,
Minati Panda, Robert Phillipson, and Tove Skutnabb-Kangas, 140–58. New Dehli: Ori-
ent Blackswan.
García, Ofelia, and Li Wei. 2014. Translanguaging: Language, Bilingualism and Education. New
York: Palgrave Macmillan.
Gee, James Paul. 2005. “Semiotic Social Spaces and Affinity Spaces: From The Age of
Mythology to Today’s Schools.” In Beyond Communities of Practice: Language, Power and
Social Context, edited by David Barton and Karin Tusting, 214–32. Cambridge: Cam-
bridge University Press.
Gibson, James. 1979. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin.
Goodchild, Samantha, and Miriam Weidl. 2018. “Translanguaging Practices in the Casa-
mance, Senegal: Similar but Different: Two Case Studies.” In Making Signs,Translanguaging
Ethnographies: Exploring Urban, Rural, and Educational Spaces, edited by Ari Sherris and
Elisabetta Adami, 133–51. Clevedon: Multilingual Matters.
Halliday, Michael. 1978. Language as Social Semiotic: The Social Interpretation of Language and
Meaning. London: Arnold.
Hawkins, Margaret R. 2018. “Transmodalities and Transnational Encounters: Fostering Crit-
ical Cosmopolitan Relations.” Applied Linguistics 39 (1): 55–77.
Hodge, Robert, and Gunther Kress. 1988. Social Semiotics. Cambridge: Polity.
Holliday, Adrian. 1999. “Small Cultures.” Applied Linguistics 20 (2): 237–64.
Jaworski, Adam, and Crispin Thurlow. 2010. “Introducing Semiotic Landscape.” In Semiotic
Landscapes: Language, Image, Space, edited by Adam Jaworski and Crispin Thurlow, 1–41.
London: Continuum.
Jiménez-Crespo, Miguel. 2013. Translation and Web Localisation. London and New York:
Routledge.
Kalyuga, Slava. 2012. “Instructional Benefits of Spoken Words: A Review of Cognitive Load
Factors.” Educational Research Review 7 (2): 145–59.
Kovačič, Irena. 1995. “Reception of Subtitles: The Non-Existent Ideal Viewer.” Translatio
(FIT Newsletter) 14 (3–4): 376–83.
Kress, Gunther. 1993. “Against Arbitrariness:The Social Production of the Sign as a Founda-
tional Issue in Critical Discourse Analysis.” Discourse and Society 4 (2): 169–91.
Kress, Gunther, and Theo van Leeuwen. 1996. Reading Images:The Grammar of Visual Design.
Lakoff, George, and Mark Johnson. 1980. Metaphors We Live By. Chicago: University of Chi-
cago Press.
Larsen-Freeman, Diane. 2015. “Complexity Theory.” In Theories of Second Language Acqui-
sition, edited by Bill Van Patten and Jessica Williams, 2nd ed., 227–44. London and
Lefebvre, Henri. 1984. The Production of Space. Oxford: Blackwell.
Lemke, Jay. 2000. “Across the Scales of Time: Artifaces, Activities, and Meanings in Ecosocial
Systems.” Mind, Culture and Activity 7 (4): 273–90.
———. 2009. “Multimodality, Identity, and Time.” In The Routledge Handbook of Multimodal
Analysis, edited by Carey Jewitt, 140–50. London and New York: Routledge.
Leppihalme, Ritva. 1997. Culture Bumps: An Empirical Approach to the Translation of Allusions.
Clevedon: Multilingual Matters.
New London Group. 1996. “A Pedagogy of Multiliteracies: Designing Social Futures.” Har-
vard Educational Review 66: 60–92.
O’Hagan, Minako, and Carmen Mangiron. 2013. Game Localization: Translating for the Global
Digital Entertainment Industry. Amsterdam and New York: John Benjamins.
O’Sullivan, Carol, and Caterina Jeffcote. 2013. “Special Issue on Translating Multimodalities.”
Journal of Specialized Translation 20.
Pérez-González, Luis. 2014. “Multimodality in Translation and Interpreting Studies.” In A
Companion to Translation Studies, edited by Sandra Bermann and Catherine Porter, 119–31.
Chichester: Wiley-Blackwell.
Ramos Pinto, Sara. 2018. “Film, Dialects and Subtitles: An Analytical Framework for the
Study of Non-Standard Varieties in Subtitling.” The Translator 24 (1): 17–34.
Remael, Aline. 2001. “Some Thoughts on the Study of Multimodal and Multimedia Transla-
tion.” In (Multi)Media Translation: Concepts, Practices and Research, edited by Yves Gambier
and Henrik Gottlieb, 13–22. Amsterdam and New York: John Benjamins.
Rymes, Betsy. 2014. Communicating Beyond Language: Everyday Encounters with Diversity. Lon-
Sperber, Dan, and Deirdre Wilson. 1986. Relevance: Communication and Cognition. Oxford:
Blackwell.
spectives on Multimodality, edited by Eija Ventola, Cassily Charles, and Martin Kaltenbacher,
9–30. Amsterdam and New York: John Benjamins.
Zanettin, Federico. 2014. “Comics in Translation: An Overview.” In Comics in Translation,
edited by Federico Zanettin, 1–32. London and New York: Routledge.
Zitawi, Jehan Ibrahim. 2014. “Disney Comics in the Arab Culture(s). A Pragmatic Perspec-
tive.” In Comics in Translation, edited by Federico Zanettin, 270–306. London and New
York: Routledge.
4
FROM THE “CINEMA OF
ATTRACTIONS” TO DANMU
A multimodal-theory analysis of changing
subtitling aesthetics across media cultures
Luis Pérez-González
1. Introduction
Multimodal artefacts, including audiovisual texts, result from the interplay between
different types of co-operating signifying resources that have to be “described and
describable together” (Kress and Ogborn 1998, quoted in Iedema 2003, 39), not
least when accounting for the transformations such texts undergo through transla-
tion. But while terms like “multimodality” or “multimodal theory” feature ever
more frequently in translation studies research, the interface between both dis-
ciplinary domains remains woefully under-explored. As Christopher Taylor has
noted, multimodality theorists have “not as yet focused on questions of translation”
(2016, 222), while translation scholars have not yet begun to make use in a sys-
tematic and sustained fashion of the concepts and methods that multimodality has
to offer, despite the widely held perception that the latter provides a robust inter-
and transdisciplinary platform to support research across the humanities. As far as
audiovisual translation research is concerned, this divide has been exacerbated by
the entrenched prevalence of “monomodality” – understood as the dominance of
one signifying constituent, typically written language, over other types of meaning-
making resources or modes used in a text – in what have been traditionally per-
ceived as the most prestigious forms of cultural or artistic expression and “[t]he
specialised theoretical and critical disciplines which developed to speak of these
arts” (Kress and van Leeuwen 2001, 1). Until relatively recently, audiovisual texts
and their translations have not featured among such artistic manifestations or in the
discourses that the latter mobilise.
A robust challenge to the dominance of monomodality, however, has been
mounted by the scale and depth of recent changes to the medial environment
in which translations are produced and received. Over the last three decades,
technological advances triggered by the shift from the print to the screen and
From the “cinema of attractions” to danmu 95
then to digital culture have been instrumental for the emergence of new tex-
tualities where written language routinely interacts with still/moving images
and/or sound in new and productive ways – whether within physical artefacts
such as magazines or textbooks, or in the context of hypertextual assemblages.
The impact of these transformations in “materiality and its cognates, medial-
ity and technicity” (Littau 2016a, 82) on the semiotic make-up of digital texts
has not gone unnoticed by translation scholars. Their heightened awareness of
the materiality of communication – encompassing “all those phenomena and
conditions that contribute to the production of meaning, without being mean-
ing themselves” (Gumbrecht 2004, 8, quoted in Littau 2016a, 83) – has fostered
deeper engagement with the dialectic between media and semiotic resources,
and has placed the material history of translation under renewed scrutiny (Littau
2016b). As Elisabetta Adami and Sara Ramos Pinto emphasise in their chapter
for this volume (Chapter 3), these developments have occurred in different ways
in different cultures during a period of increasing globalisation. In this context,
calls for “an integrated approach to translation” that gives careful consideration
to “the physical support (stone, papyrus, CD-ROM), the means of transmission
(manuscript, printing, digital communication),” and “how translations are carried
through societies over time by particular groups” (Cronin 2003, 29) are congru-
ous with efforts undertaken in other disciplinary domains to develop conceptual
and methodological tools for the study of communicative practices in all their
semiotic complexity (Iedema 2003).
Against the backdrop of growing engagement with the study of multimodal
texts, this chapter sets out to chart the changing ontological contribution that sub-
titles have made to the multimodal fabric of audiovisual texts produced since the
silent film era to the present. A number of multimodal issues associated with sub-
titling have been reported in previous publications that do not explicitly associate
themselves with multimodality studies (see Pérez-González 2020 for a detailed
survey), yet a growing body of subtitling scholarship that subscribes explicitly to
multimodal theory has explored the disciplinary connections between multimodal-
ity and audiovisual translation in more depth (Remael 2001; Taylor 2003; Gambier
2006; Pérez-González 2007; Mubenga 2009; O’Sullivan 2011; Desilla 2012; Tay-
lor 2014; Gambier and Ramos Pinto 2016). This chapter offers a novel approach
to the study of the multimodal dimension of subtiling practices, examining the
organic enmeshment of subtitles within the overall semiotics of audiovisual content
across successive media cultures, moulded by evolving configurations of technol-
ogy, power, ontological status, and social practices. The chapter aims to yield an
in-depth understanding of the implications that the shift from the dominant nar-
rational regime of Western modernity to postmodern aesthetics emerging in the
context of digital culture has for multimodal textualities and their translation. This
study is therefore not based on the analysis of small-scale translation shifts in a
multimodal context. Instead, it engages with wider theoretical and programmatic
debates in translation, multimodal, and cultural studies to inform future reflective
practice at this disciplinary interface.
96 Luis Pérez-González
The chapter is structured as a critique of the changing role of subtitling in the

media landscape, charting the gradual emancipation of this modality of translation
from the dictates of the film industry as it comes to serve the more democratic
forms of spectatorial media “prosumption” associated with digital culture. Gunther
Kress and Theo van Leeuwen’s distinction between “recording” and “synthesising
technologies” provides the conceptual network required to trace the shift from an
“ontology of referentiality,” based on the hegemonic narrational regime, to the
“ontology of deconstruction” that lies at the heart of the digital media ecology,
shaped by variable degrees of convergence between industrial and amateur practices
(Kress and van Leeuwen 2006, 217). Developments in the digital arena, as instanti-
ated by Chinese danmu, are presented as contemporary instances of “representation-
as-design” driven by rhetorical choices (Kress and van Leeuwen 2006, 219) and
analysed in terms of the affordances of “semiotic software” (Djonov and van Leeu-
wen 2017) and the social semiotic practices that the latter enables. Ultimately, this
study shows how, where ordinary rhetors have gained greater visibility and agency,
their performance of citizenship involves the deconstruction of representation by
exposing the cultural and social make-up of multimodal ensembles.
2. Subtitling in the era of recording technologies

As is also the case with users of other multimodal texts (for instance, see Marcus
Tomalin’s discussion of the livre d’artiste in his chapter for this volume, Chapter 6),
mainstream film audiences must make sense of meaning distributed across vari-
ous semiotic systems. The range of signifying means deployed by film creators to
encode multiple strands of acoustic and visual meaning effectively constitute “a
single unified gestalt in perception” (Stöckl 2004, 16). Presented with these unified
semiotic entities, film viewers have to make connections across different types of
signifying resources in a routinised, subconscious manner, drawing on their accrued
spectatorial experience to decode conventionalised ensembles of sign types. The
demands originally placed on viewers have grown as film semiotics began broaden-
ing its repertoire of meaning-making resources as early as in the pre-sound film era.
The relentless drive to heighten the appeal of motion pictures as commodities led
silent film creators to attempt to convey mute diegetic speech to their audiences
in various ways, including, but not limited to, film editing techniques and the use
of live narrators, “speaking actors,” title cards, and intertitles (Dwyer 2005; Pérez-
González 2014a; O’Sullivan and Cornu 2018).
The advent of sound in the late 1920s further cemented the status of film as the
first modern audiovisual form of entertainment and paved the way for the develop-
ment of subtitling and dubbing in a bid to preserve and increase the commercial
flow of films across linguacultures. Subtitling is widely held to disrupt the multi-
modal configuration of the original artefact, as conceived by filmmakers, for the
ex post-facto incorporation of snippets of text to a finished film calls for additional
intermodal connections that viewers of the original ensemble were not required to
draw. While commercial subtitles delivering translations of the original dialogue do
not generally influence either the “pacing and rhythm” of the film or its “intellec-
tual and emotional content” (MacDougall 1998, 168), the need to process written
language superimposed on the image represents an additional cognitive effort that
may detract from the enjoyment of audiences in dubbing-dominated markets. The
impact of dubbing conventions on the co-deployment of semiotic resources in
films will not be pursued further in this chapter due to space constraints.
Although the partnership between film and subtitles spans approximately one
century, we are only beginning to gain reliable insights into the implications that
the superimposition of snippets of written text on images has for the processing of
subtitled films. The burgeoning body of eye-tracking research on the behaviour
of viewers’ gaze shows that the on-screen mobilisation of subtitles always draws
attention from viewers, even when audience members are able to understand the
dialogue they are presented with (d’Ydewalle and De Bruycker 2007). After read-
ing the subtitles (Jensema et al. 2000), gaze trajectories generally alternate between
the subtitles and the images they are inscribed on, as viewers adjust their process-
ing strategies according to the genre they are watching (Perego et al. 2010). Most
viewers spend more time reading the subtitles than looking at the images, although
the fixations – understood as the pauses between eye movements, during which
the viewer’s eyes remain static over an area of interest – on the latter are normally
longer (Perego et al. 2010; Kruger 2018). Effectively, once viewers have read the
subtitles in full, they tend to focus their attention on key areas of interest – primarily
faces, when these are present – for as long as possible. The evidence emerging from
eye-tracking research thus confirms that viewers of subtitled films have less time
to explore visual semiotic resources, although audiences in subtitling-dominated
audiovisual markets are able to watch subtitled programmes relatively effortlessly, as
they are regularly exposed to this specific multimodal configuration (d’Ydewalle
and De Bruycker 2007).
Crucially, this incipient body of evidence was not yet available during the 1930s
and 1940s, at the time when the industry was developing the set of subtitling prac-
tices that, as acknowledged by film scholars, have remained virtually unchanged
ever since (James 2001). The conventions in question seek to achieve full syn-
chrony between the duration of successive blocks of diegetic speech and the on-
screen display of the subtitle(s) conveying their meaning into the target language,
assuming standard (albeit adjustable) sets of reading conditions associated with dif-
ferent viewer profiles. Under these “spatio-temporal constraints” imposed by the
film medium, spoken dialogue must be substantially condensed to fit into compact,
tightly worded subtitles, which have been shown to have a detrimental impact
on the viewers’ perception of characterisation, humour, irony, and other culture-
specific aspects of subtitled film (Mason 2001; Remael 2003; Desilla 2012). Draw-
ing on Stöckl’s (2004) conceptualisation of multimodal resources, to be elaborated
on later, the process of semiotic transfer known as industrial subtitling therefore
revolves almost exclusively around two “medial variants” (written and spoken) of
the same “mode” (language). Except for those instances where the decoding of
nonverbal semiotic resources requires culture- specific knowledge (see featured
example 6.1 in Pérez-González 2014a, 188–90), it is generally assumed that view-

ers will manage to activate any connections across different types of semiotics that
the subtitled film requires, just as the original film audience members did.
Critical theory and film studies scholars with an interest in the genealogy of
subtitling practices, however, have taken the view that industrial subtitling practices
are not, strictly speaking, the necessary product of spatio-temporal constraints asso-
ciated with the film medium. Instead, the cult of synchrony underpinning those
subtitling conventions is argued to be consistent with other semiotic developments
in the production of motion pictures, not least the evolution of early presenta-
tional films – i.e., “the films of the cinema of attractions” where “theatrical display
dominates over narrative absorption” (Grochowski 2007, 56) – into representa-
tional ones, which seek to create a self-contained diegetic world driven by nar-
rative motivation allowing viewers to suspend their disbelief. In addition to the
deployment of sophisticated editing and montage practices, the construction of
this self-enclosed fictional world is heavily contingent on the use of synchronous
diegetic sound because it provides an allegedly “unmediated access to [the diegetic]
reality” (Minh-ha 2005, 129). Synchronous diegetic sound allows producers to shift
viewers’ attention away from the tools and relations of production – i.e., the spaces
between image, sound, and text that can be seen in the films of the cinema of
attractions – and minimises the potential for subjective spectatorial experiences.
So while sound synchrony would, on the face of it, aim to promote an aesthetic
of objectivity, the centrality of the diegetic in modern film production ultimately
signals the industry’s interest in spreading hegemonic commercial discourses that
work through linear narratives to control film reception.
Drawing on Gunning (1986) and Musser (2006), Grochowski explores the
financial and political drivers behind the expansion of the Hollywood studios,
gauging the extent of their impact on the consolidation of the narrational regime.
For Grochowski, the synchrony-driven switch from the presentational to the nar-
rational was the outcome of a “struggle between producers of motion content and
exhibitors (where the production of narrative films becomes merely a strategy, a
“tool” to use by the producer to control exhibition)” (2007, 56). Ultimately, sound
synchrony, and the shift from the presentational to the narrational paradigm that
it brought about, enabled the US film industry to impose a taste for homogenis-
ing, easier-to-export linear narratives, as part of a wider synchronicity-led initiative
to dominate the global film market, often to the detriment of incipient narrative
conventions emerging elsewhere in the world, that were soon replaced by their
imported American counterparts (see McDonald 2006; Chung 2007 for accounts
of how this acculturation process played out in Japan and Korea, respectively). Ira-
nian film and cultural studies scholar Hamid Naficy (2003) has also examined the
dynamics of film narrative colonisation and revealed the geopolitical import of this
industrial strategy. In Naficy’s words:
[A]t the heart of the US policy of technological transfer and development

aid for the Third World since the 1950s, was this notion of homogenisation
and synchronicity of the world within Western consumerist ideology. This is

a shift from the earlier policy of diachronicity, promoted by colonists, which
tended to keep the developed and the under-developed worlds apart. The
emerging form of post- industrial capitalism sought synchronicity in the
interest of creating global markets.
(2003, 193)
Hollywood’s imposition of the dominant narrational regime of Western moder-

nity as a means to exercise maximum control over the reception and commercial
performance of films required the enforcement of hierarchical power structures
between authorship and viewership, and the technologies available at the time
played an instrumental role in facilitating the expansion of this hierarchical regime
of signification. The mutually influencing relationship between the technologi-
cal and the social is aptly articulated by Stephen Heath, who notes that “[c]inema
does not exist in the technological and then become this or that practice in the
social; its history is a history of the technological and social together . . . in which
the ideological is there from the start” (1981, 6). Unsurprisingly, the period dur-
ing which the blueprint of American film narratives was drawn up featured the
rise and consolidation of “recording technologies,” one of the three historically
successive types of production technologies identified by Kress and van Leeuwen.
Aimed primarily at the eye and the ear, recording technologies “allow more or less
automated analogical representation” of what was presented as unmediated reality
(Kress and van Leeuwen 2006, 217), thus providing the ideal inscription tools to
enable the shift from diachronicity to synchronicity that underpins the narrational
regime. Recording technologies placing synchrony at the centre of the temporal/
spatial order favour homogenising narratives and uphold the authority of the crea-
tor within existing media power structures. Acting as a form of “suture” (Heath
1981), they foster an ontology of representation restricting the range of relations
that a film is constructed to establish with its viewers, ultimately endorsing an
aesthetic of objectivity based on referential expressions of reality (Silverman 1988).
Indeed, the emphasis of cinematography on synchronous diegetic sound as a way
to keep the machinery and production process behind filmmaking hidden is typi-
cal of the “ontologies of referentiality, a view of representation being founded on
direct, referential relations between the representations and the world” (Kress and
van Leeuwen 2006, 218).
In order to facilitate the implementation of this industrial/commercial strat-
egy, the centrality of synchrony had to be extended to the subtitling of US film
commodities. Minh-ha coins the term “suture subtitling” to foreground the
impact that the generalisation of the narrational film regime had on subtitling
and the processes of multimodal meaning transfer that it entails (Minh-ha 1992).
The interplay between power structures and recording technologies during the
formative years of subtitling allowed the film industry to “collapse, in subtitling,
the activities of reading, hearing, and seeing into one single activity, as if they
were all the same,” and hence “to naturalize a dominant, hierarchically unified
worldview” (Minh-ha 1992, quoted in Nornes 1999, 18). Under the aesthetic
of objectivity associated with recording technologies, subtitles have contributed
to upholding the ontology of referentiality, acting as gatekeepers between the
diegetic and the extra-diegetic. The role of subtitles in the era of recording tech-
nologies has been conventionally confined to delivering in the target language
approximate linguistic representations of the verbal meanings and/or communi-
cative intentions encoded in the source text. The emphasis has been thus placed
on the original message that filmmakers intend to convey through their char-
acters’ speech – the assumption being that translators should privilege the “pri-
mordial” meaning expressed in the film dialogue. Further, while the contribution
of visual and acoustic semiotic resources to the overall meaning of audiovisual
texts is ostensibly acknowledged by audiovisual translation scholars (Taylor 2003;
Pérez-González 2007, 2014a, 2014b, 2020; Desilla 2012), professional subtitlers
working for the film industry during the era of recording technologies have seen
the scope of their mediation, and hence the exercise of their professional discre-
tion, reduced to one single form of semiotic transfer – involving the re-encoding
of speech through written language.
3. Subtitling during the period of ontological transition

Although the hegemony of the narrational regime continues to place stringent
constraints on the multimodal configuration of commercial films, a growing body
of documentary and ethnographic cinematic practices is revealing the extent to
which “the sound track, like the image track, involves mediations, translations, and
representational practices that push the sound into the realm of ideology” (Naficy
2004, 141), thus framing suturing synchrony as a commercial choice, rather than the
only option available under the medial constraints of film. The formal organisation
of Trinh Minh-ha’s Surname Viêt Given Name Nam (1989), for example, “question[s]
the representational nature of film” by breaking down mainstream filmmaking
techniques and pointing to their flaws, therefore preventing the audience from
suspending their disbelief (Vietnamese Cinema Blog 2015). Just as Trinh’s filming
style tampers with the norms of cinematic realism (for example, by using lighting to
disrupt the mood of a scene, camera movements that wander away from the subject
and sound overlapping techniques), it also draws on
superimposed titles and subtitles extensively, graphically and critically. Their

large numbers and varied contents and layout give this film a truly calligraphic
accent. Throughout, subtitles consisting of the translation of the film’s dia-
logue and voice-over and of Vietnamese poetry and proverbs are displayed, as
is customary, in the lower third of the screen. However, on many occasions,
what the diegetic women say in Vietnamese or in heavily accented English is
superimposed in different layouts, as blocks of English text on various regions
of the film frame, including over the characters’ faces. These graphic titles,
or what Trinh calls “visualized speech,” act as traditional subtitles by aiding
spectator comprehension. . . [although] they also serve other graphic, critical,

and deconstructive functions.
(Naficy 2004, 145–46)
The ontology of referentiality is also critiqued in “accented films” shot by mem-

bers of exilic and diasporic communities since the 1960s (Naficy 2004). Drawing
on artisanal production techniques, accented creators adopt a performative stance
involving subjective film shots (Naficy 2004, 136) and other techniques that enact a
free indirect style in their films. Significantly, the deployment of counter-hegemonic
filmic discourses based on sound misalignment and de-centred compositional pat-
terns – that allows accented filmmakers to “comment upon cinema and reality,
instead of just recording, reporting on, or representing reality” (146) – calls for the
use of profuse on-screen titles to channel “the problematic of linguistic, cinematic,
and exilic translation and displacement” (146).
While the erosion of the boundary between the diegetic and the extra-diegetic
can be accelerated in documentary and accented films by displaying subtitles delib-
erately out of synchrony, among other multimodal strategies, commercial films
may also seek to undermine the narrational regime by exploring new forms of
interplay between the story and the audience. Pablo Romero-Fresco, for exam-
ple, has explored new subtitling approaches where, in contrast to what happens in
conventional films and drama series, the material dimension and the function of
on-screen titles are conceptualised prior to the production of the multimodal text
they will be embedded in. Creative subtitling, as defined by Romero Fresco, adopts
and extends certain features of ethnographic subtitling (MacDougall 1998, Chap-
ter 7), primarily its capacity to foster new narrative mechanisms through which
the authorial voice bypasses the diegetic characters to engage directly with the
audience (Romero-Fresco 2018). Creative subtitles – variously referred to in the
literature as “decotitles” (Kofoed 2011), “authorial titles” (Pérez-González 2012),
“impact captions” (Sasamoto 2014; O’Hagan and Sasamoto 2016), or “integrated
titles” (Fox 2016) – do not seek to deliver translations of the original spoken dia-
logue but rather to enhance aspects of characterisation or progress the narrative in
more involving or immersive ways (Dywer 2015).
Peter Kosminsky’s four-part drama The State (2017) – which dramatises the
experiences of four British men and women who leave their lives behind to
join Islamic State in 2015 – illustrates the innovative use of authorial titles in
mainstream media content. As the new recruits arrive in Syria and settle into
the organisation, men begin combat training and the women are introduced
to the society’s strict rules under the guidance of commanders and house lead-
ers, respectively. English-speaking characters going about their tasks make use of
Arabic sentences pertaining to various rituals and cultural conventions associated
with Islam, as a way to signal and reinforce their shared collective identity and
community affiliation. When such expressions are used, stylised titles – that look
“too aesthetic to be merely informative” (Crawley 2017) – display the transliter-
ated Arabic sentences and their English translation in varying areas of the frame.
For example, when Shakira Boothe – a British doctor and single mother who
joins the Islamic State in the hope of working in a state hospital – makes arrange-
ments to see her first patients, she is told she cannot work without a mahram, at
which point the title “mahram | male guardian” appears on screen. Another exam-
ple of commercial film subverting the hierarchical power structures between the
narrative and on-screen titles is Paolo Sorrentino’s Il Divo (2008), a biographical
drama about Giulio Andreotti, one of Italy’s most controversial Italian politicians,
who found himself at the centre of a conspiratorial shady network that eventually
led to his prosecution. Although the film sets out to re-create the reality of key
historical events, various semiotic resources are deployed from the performative
perspective of auteurship. This is particularly striking in the opening sequence,
that flashes a series of murder sequences in quick succession, complete with styl-
ised captions providing details of the victims and the circumstances surrounding
their death. Donald Clarke, for example, draws attention to the “artful captions
that float behind buildings and emerge from car doors – offering indigestibly
comprehensive descriptions of their [the victims’] role in the debasement of Ital-
ian politics” (Clarke 2009), while Mark Jenkins notes the resemblance between
the playfulness of Sorrentino’s captions and those used in Danny Boyle’s Slumdog
Millionaire (2008) (Jenkins 2009). As Millicent Marcus elaborates:
[t]he pretense to investigative seriousness is immediately undercut by the

playfulness of Sorrentino’s captions – the upside-down hanging corpse of
Roberto Calvi [one of the murder victims] is accompanied by a correspond-
ingly upside- down caption, which turns right- sideup as the photogram
rotates 180 degrees to its proper position. Similarly, the caption identifying
Michele Sindona, poisoned in his jail cell, is shown backwards until the cam-
era circles around the dying man so that we can read his name from left to
right. Throughout the film, captions dance about the screen in a kind of free-
floating semiotic abandon. Words, like images, have come untethered from
their “signifieds.” By transforming the convention of the caption – hitherto
understood as an indicator of its film’s unproblematic and unequivocal ref-
erentiality – into one more sign to be manipulated at will, Sorrentino asserts
his ambiguous relationship to the truth claims of the documentary genre, and
its offshoot in the films of cinema politico.
(Marcus 2010, 253; emphasis in the original)
The authorial titles deployed in The State and Il Divo enable the transgression of tra-
ditional narrative boundaries through “narrative metalepsis,” as conceptualised by
Genette (1983 [1980]).1 Once they are incorporated into the mise en scène, authorial
titles become an additional narrative plane intersecting with the unfolding diegetic
events, dramatising the blurring of the boundary between fiction and reality. The
creators of The State and Il Divo use authorial titles to transgress traditional nar-
rative conventions and exploit the “double-layeredness” of filmic communication
(Vanoye 1985) to intrude in the diegetic action. The vertical level of interpersonal
communication (between the filmmakers and the audience) is prioritised in places

over its horizontal counterpart (between the diegetic characters), as titles provide
viewers with information and knowledge about the fictional world that would
normally be confined to the diegetic characters. The combined use of cinema-
tography and metaleptic captions in Il Divo, for instance, disrupts classical hierar-
chies in film narratives, undermining the ontological status of fictional subjects. As
Ambrose Hogan argues, criminal allegations are not made against specific individu-
als through diegetic speech (Hogan 2009). Instead, they are communicated to the
viewer through the interplay between visual signifying means and performative
titles, some of which “are spat out of the mouth of a victim of poisoning, or swing
from under Blackfriars Bridge along with the cooling corpse of [Italian banker]
Roberto Calvi” (Hogan 2009).
The previous examples have shown how the deployment of titles can be used to
destabilise the ontology of referentiality for various purposes, even in the context
of media content produced for commercial release. Rather than simply contribut-
ing to the representation of a linear narrative or “hiding behind the pretense of
an unacknowledged spectator” (Gunning 1993, 5), authorial titles comment upon
the diegetic and extra-diegetic realities, promoting the same “parataxic” reading
experience as the primitive intertitles featuring in films of the cinema of attractions
(Grillo and Kawin 1981). Whether it is through the delivery of diegetic subtitles
out of synchrony or in unusual regions of the frame (as is the case with Trinh
Minh-ha’s “visualised speech” [Naficy 2001, 123]), or through the deployment of
authorial titles that challenge the privileged status of the diegetic narrative over
its viewers (creating new spectatorial experiences where creators interpellate the
audience directly), films and serialised dramas are presenting audiences with new
processing demands. Instead of coherent single gestalts, viewers are faced with mul-
timodal artefacts where the strands of meaning conveyed through the juxtaposition
(rather than integration) of different signifying means must be dissected and inter-
preted independently – thus heightening the potential for subjective interpretations
of the diegetic narrative.
4. Subtitling in the era of synthesising technologies

The advent of digitisation has brought about the emancipation of subtitling from
the media industries. In the digital culture, media content is increasingly shaped
by variable degrees of convergence between industrial and amateur practices,
with growing numbers of ordinary people becoming involved in its appropria-
tion, annotation (including subtitling), editing, and distribution (Pérez-González
2014a, 2017; Dwyer 2017a). Driven by the ubiquity of relevant technological
tools, amateur (often participatory) subtitling has quickly reconfigured individual
viewership as part of wider processes of social interaction around the production
and consumption of media content, turning the “spectatorial culture into par-
ticipatory culture” (Jenkins 2006, 41). Under this new postmodernist aesthetic,
where subtitled media content is often watched via social media platforms, “net
users [including amateur subtitlers] are increasingly disinterested in the metanar-

rative (content) and instead become obsessed with multiple small narratives (com-
ments)” (Xu 2016, 439). In this context, media content represents an opportunity
for the negotiation of intersubjectivity – understood as a shared and reciprocal
perception of the meaning created through joint activity and socially managed
interaction – among members of virtual, geographically dispersed communities
of interest. In selecting what is to be subtitled and steering decision-making pro-
cesses on the materiality and wording of their subtitles, these agencies subordinate
the authority of the original to the subtitles’ capacity to mobilise political or
playful affinity through the online comment facility that accompanies the video.
Participation, in short, takes over spectatorship.
Amateur subtitling is typically executed by virtual communities that capital-
ise on the affordances of networked communication to exploit their members’
skills sets or collective intelligence. Subtitlers involved in this form of “co-creative
labour” (Banks and Deuze 2009) draw on their linguistic proficiency – which
does not necessarily equate with translation competence – and activist orientation.
“Aesthetic activism” (Pérez-González 2014a), one of the two main strands covered
in the literature, is best illustrated by “fansubbing,” a term traditionally associated
with the prolific global subculture built around the subtitling of Japanese anime by
fans of this genre and, in more recent times, with a form of immaterial subtitling
labour undertaken by followers of commercial dramas, series, or musical genres
(Barra 2009). Fansubbing practices are normally fuelled by a desire to tamper with
commercially translated media content that fans-turned-translators regard as “cul-
turally odourless” (Iwabuchi 2002, 27). Fansubbing practices have been shown to
experiment with the formal dimension of subtitles (Dwyer 2012) and bestow a high
degree of visibility on translators, who maximise the deconstructive affordances of
digital technologies to act as intercultural brokers between the text producers and
users from inside the frame. On the other hand, the archival, annotation (through
subtitles), and recirculation of media content are also central to the activities of net-
works of politically engaged amateur subtitlers, for digital culture has facilitated the
rise and consolidation of networks of activist subtitlers seeking “to elaborate and
practise a moral order in tune with their own narratives of the world” (Baker 2006,
481). Pérez-González (2010, 2013, 2016) has shown how participatory networks of
activist subtitling resist the dynamics of the news media industry and challenge the
control that global corporations have traditionally exerted over the distribution and
consumption of the content they produce. These participatory communities take
on the role of self-appointed translation commissioners and selectively appropri-
ate the content they wish to subtitle and redistribute among strategically targeted
linguistic constituencies that otherwise would not have been able to access that
content through mainstream or commercial media circuits. In the same vein, Mona
Baker has explored how subtitlers embedded in various activist collectives con-
tribute to exploring prefigurative principles (including horizontality, solidarity, and
rejection of representational modes of practice), choosing to bring their mediation
into light, “rather than obscure it and lull the viewer into the illusion of a direct,
unmediated experience” of the subtitled content (Baker 2018, 460–61).
As was also the case with aesthetic activists, politically engaged subtitlers ulti-
mately work with their viewers towards the co-construction of affinity spaces for
the negotiation of their individual and collective identities. Unconstrained by com-
mercial and financial interests, amateur subtitling by ordinary citizens is contribut-
ing to foreground a postmodern aesthetic based on “the primacy of our mundane
day-to-day experiences over that of the rigid structures” favoured by the media
industry and the narratives that embody their cultural and economic hegemony
(Xu 2016, 439). This change in the role that subtitling plays within the multimodal
configuration of audiovisual texts is consistent with other manifestations of “citizen
media” practices, which encompass
[t]he physical artefacts, digital content, practices, performative interventions

and discursive formations of affective sociality produced by unaffiliated citi-
zens as they act in public space(s) to effect aesthetic or socio-political change
or express personal desires and aspirations, without the involvement of a third
party or benefactor. It also comprises the sets of values and agendas that
influence and drive the practices and discourses through which individuals
and collectivities position themselves within and in relation to society and
participate in the creation of diverse publics.
(Baker and Blaagaard 2016, 16)
As noted earlier, citizen subtitlers – a term adopted here to encompass ordinary

citizens-turned-subtitlers working with different agendas – subscribe to a postmodern
aesthetic of mundane playfulness or political engagement that often ignores the con-
straints of narrative motivation and the concomitant need for synchronous temporal
and spatial relationships between the original speech and its subtitled version. Indeed,
the status of the original is subverted by the comment culture that amateur subti-
tling practices seek to foster and stimulate. Positing spectatorial subjectivity through
parataxic reading practices is at odds with the ontology of referentiality underpin-
ning the production of classical films and therefore requires the use of “synthesizing
technologies” (Kress and van Leeuwen 2006, 217). Against the crisis of representation
that the shift from recording to synthesising technologies has prompted, digitisation
enables alternative forms of “representation-as-design” (Kress and van Leeuwen 2006,
217) that open up new possibilities for more democratic spectatorial engagement by
bringing the world of the story closer to the space of the audience. The ontology of
referentiality associated with the narrational regime is thus superseded by an ontol-
ogy of deconstruction under which ordinary citizens can watch and produce subti-
tles, and the diegetic world is no longer closed off to viewers, who can make use of
digital technologies to experiment with and expose the multimodal circuitry of the
multimodal text, “deconstructing the combinatorial possibilities and laying bare their
cultural/social sources” (Kress and van Leeuwen 2006, 219).
5. The multimodal implications of design and rhetoric in

Chinese danmu
In digital media habitats, where amateur rhetors have gained greater visibility and
influence, subtitling is emerging as a meaning-making practice that contributes
to the deconstruction of the original text by exposing the interstices around its
multimodal wiring. As reported in the literature surveyed in the previous section,
the ever-closer alignment between amateur subtitling and other manifestations of
citizen media entails the adoption of experimental practices that mobilise a range
of signifying means to style on-screen text in unconventional ways for maximum
effect. In terms of the ontological value that they attribute to the source text, citizen
subtitlers steer us away from the individual or subject position of their professional
counterparts, towards collective discursive spaces of translatorship involving com-
plex negotiations of mundane or ethical identity.
Technological developments – whether in the form of proprietary web-based
captioning tools such as Amara (Amara n.d.), that have been made available to
support volunteer initiatives as a way of accruing symbolic capital in the commu-
nity; freeware editing and subtitling applications that citizen subtitling groups have
chosen to integrate within their collective workflows; or social media platforms
providing the tools to develop a participatory comment culture around the con-
sumption of subtitled output – have been instrumental in delivering this ontologi-
cal shift. By enabling users to “select from a range of different semiotic resources”
as well as “incorporat[ing] and represent[ing] knowledge about what constitutes
effective use of these resources” in citizen subtitling, these tools qualify as “semiotic
software” (Djonov and van Leeuwen 2017, 567). As befits synthesising technolo-
gies, semiotic software has its own semiotic regime, insofar as the “rules for using
the semiotic resources available within a software product are externalised in and
tacitly imposed by its design” (Djonov and van Leeuwen 2017, 571). Significantly,
semiotic software is not only important for enabling multimodal affordances and
setting restrictions in terms of how various resources – including, but not limited to,
colour, font, texture, or animation – can be jointly deployed (van Leeuwen, Djonov,
and O’Halloran 2013), but also for being semiotic artefacts in their own right with
the capacity to influence social practices (Poulsen, Kvåle, and van Leeuwen 2018,
596). The dialectic between the affordances and constraints imposed by semiotic
software and the social practices that it enables will be explored in the remainder
of this chapter through the lens of Kress and van Leeuwen’s (2006) well-established
analytical notions of “design” and “rhetoric.” In the context of this study, the for-
mer designates the multimodal meaning-making potential of the technological
resources facilitating the work of ordinary people involved in participatory sub-
titling, whereas the latter will be used to foreground the scope and complexity of
the social relations that obtain around the production and consumption of subtitled
media content.
As the practices associated with successive fansubbing turns (Dwyer 2018) and
political subtitling (Pérez-González 2010, 2013, 2016; Baker 2018) are relatively
well documented in the literature, the remainder of this paper examines the inter-
play between semiotic technology (design) and social practices (rhetoric) – and the
ontology of deconstruction that they bring about – as instantiated in the context
of the under-explored Chinese participatory subculture known as danmu. Craig
Howard describes danmu – also known as “barrage commenting” (Li 2017; Dwyer
2017b) – as a video annotation system that was first launched in 2007 by a Japa-
nese ACG (anime, comic, game) video-sharing website (Howard 2012). The sys-
tem, which was introduced in China in 2010 by digital platforms like Bilibili.com,
involves the display of viewer-generated titles on the very screen where the media
content in question is being played. The titles – which, as elaborated further, can
deliver translations of the diegetic dialogue or, alternatively, viewers’ comments on
a range of issues more or less directly connected with the video they are inscribed
on – are therefore superimposed on the multimodal text, rather than in a separate
section, as is the case in YouTube. According to Jannis Androutsopoulos, danmu – a
term that designates both the platform allowing for the generation and sharing of
comments, as well as the comments themselves – constitutes a form of “participa-
tory spectacle” where media content is reduced to a simple background, with the
production and reading of comments taking over as the focal attraction (Androut-
sopoulos 2013). Indeed, the climatic parts of a series episode or film may bring
about so many danmu that comments may form a multi-layered “bullet curtain”
(which is how danmaku, the original term used in Japanese to refer to this practice,
translates into English) that may become thick enough to block the visuals (see
Figure 4.1).
FIGURE 4.1 Screenshot of video material featuring danmu

Source: (Bilibili.com)
In what follows, the analysis of the danmu aesthetic as a form of representation-

as-design will be informed primarily by Stöckl’s (2004) typology of multimodal
resources – a framework that, whilst not primarily developed to analyse audio-
visual texts, provides a productive and systematic set of analytical tools based on
Stöckl’s conceptualisation of semiotic resources as constituents of hierarchical
networks of choices.
As is also the case with mainstream and citizen subtitling, the danmu titling
culture mobilises a range of modes, i.e., “sign systems from which communicators
can pick their signs to realise their communicative intentions” (Stöckl 2004, 11).
Language and image remain as core modes – understood as those types of signify-
ing means that are “deeply entrenched in people’s popular perceptions of codes
and communication” (Stöckl 2004, 14) – in the danmu environment, although
the prominence of the user-generated titles signals very clearly (i) how traditional
relationships between authorship and viewership are being subverted; and (ii) the
extent to which “the inner layer (metanarrative) loses primacy to the surface layer
(nonnarrative)” (Xu 2016, 448) that viewers generate. The fact that danmu titles
seek primarily to negotiate a perceived sense of intersubjectivity among viewers,
rather than to enjoy the media content per se, entails a clear asymmetry between
the volume of information conveyed through each of the two sensory channels at
play in this site, with the written word (i.e., the medial realisation of the language
mode within the visual channel) taking priority over its auditory counterpart.
The inscription of danmu on top of the audiovisual text foregrounds the crisis of
the representational regime and, more specifically, the emancipation of titles from
the primordial narrative.
As is also the case in other forms of citizen subtitling, the Bilibili platform – i.e.,
the semiotic software driving the danmu culture – allows for a range of multimodal
enhancements through the activation of one or more title submodes, namely “the
building blocks of a mode’s grammar” (Stöckl 2004, 14). Font and colour choices, for
example, are among the most productive submodes that viewers can use to style writ-
ten text and to construct a visual identity for themselves amid danmu that have been
previously posted by other viewers. Interestingly, in this environment characterised
by the proliferation of superimposed comments flowing horizontally over the video,
from the right to the left of the screen, it is static written text that becomes relatively
salient and commands a higher degree of ontological authority. Static danmu – which
can only be mobilised by Bilibili premium users – feature at the top and bottom of
the frame (Wu et al. 2018) to deliver explanations on both general and specific aspects
of the video (Ma and Cao 2017) and to deliver translations of the diegetic speech by
way of traditional subtitles (Wu et al. 2018), respectively.
Capitalising on the affordances enabled by semiotic software, members of the
danmu subculture act as rhetors making cumulative decisions pertaining to the
deployment of semiotic resources among the network of submodal options avail-
able at any given point. The danmu conglomerate therefore emerges as a composite
artefact moulded by the decisions made by users during the design stage – as is also
the case with other forms of citizen subtitling. Ma and Cao, for example, report
that users playing specific roles (e.g., uploader) can stick to the same colour “when
commenting repeatedly throughout the video”; groups of posters involved in a
shared task, such as translating the lyrics of a diegetic song into different languages
and dialects through danmu may choose to exhibit a shared visual identity through
the adoption of the same colour-cum-font pattern; and individual users wishing to
interact as part of a self-contained group can signal their shared membership by
prefacing their danmu with strings of words (Ma and Cao 2017, 779). In these cases,
then, the selection of fonts and colours may seek to reflect what is being talked
about, or the relationship between the interactants, just as in other forms of citizen
subtitling. But the deconstructing role of synthesising technologies in the danmu
platforms also has some liberating effects: submodal choices are not bound by the
need to ensure optimum visibility and readability conditions, as in chaotic barrage
commenting environments “the consumption of the metanarrative [the diegetic
plot] is cast away in favour of the irrelevant and carnivalesque aspect” of users’
danmu (Xu 2016, 439).
From a temporal perspective, there is a difference between static, subtitle-like
danmu featuring at the bottom of the frame – which are delivered in synchrony
with diegetic speech – and non-translational danmu occupying the entire screen
and often blocking out the actual video. Users wishing to post danmu do so by
linking the display time for their contribution to the specific time code of the
video moment that it comments on. New danmu can be added all the time, but
previously logged ones will be replayed every time the video is watched, thus turn-
ing the interface into an ever-g rowing archive of user-generated contributions. As
Jinying Li explains, “the temporality structured by the danmaku interface leads to
a collective user experience of ‘virtual liveness’ ”: while all past danmu are shown
simultaneously, even though there might be important gaps between the times in
which they were inscribed, it is impossible for users to engage in live conversations.
Danmu posted in response to a previous comment may never be read by the latter’s
author, so interacting in this environment effectively amounts to debating “with
someone who spoke in the past but whose words always appear in the present” (Li
2017, 248).
In overstepping the boundaries between the diegetic and the extra-diegetic,
danmu exhibits many similarities to other forms of title-based mediation driven by
a deconstructive ontology examined in previous sections, including authorial titles,
understood as a transitional regime between recording and synthesising technolo-
gies, and citizen subtitling, as performed by communities galvanised around both
fandom or political affinity. But while the network of social relations that emerge
among participants in citizen subtitling has been theorised in some depth (Pérez-
González 2014a), the rhetorical dimension of danmu has not yet been explored. In
the remainder of this section, the scope of the discussion on the rhetorical con-
struction of danmu will be limited to a specific dimension, namely, the politics of
“mutual recognition” (Thrift 2009).
Amateur subtitling communities have been found to resemble other partici-
patory networks that operate in the citizen media culture insofar as they index
“self-referential properties” to their preferred “values, beliefs and practices” (Deuze

2006, 71). Enhancing the visibility of amateur prosumers, in their capacity as repre-
sentatives of a collective identity shared by subtitlers and their audiences, is there-
fore crucial to the proliferation of participatory subtitling. Essentially, the drive
to enforce an aesthetic or a politics of mutual recognition between subtitlers and
viewers empowers amateur translator networks to adopt an interventionist media-
tion approach, moulding media experiences to fit viewers’ expectations. By draw-
ing on the affordances of synthesising technologies and reflexively engaging in
the manipulation of media content that circulates in their environment, ordinary
people are able to establish new participatory sites for the expression of subjective
spectatorial experiences and promote, as is also the case with other citizens engaged
in forms of self-mediation that do not involve translation, “new forms of playful
citizenship, critical discourse and cosmopolitan solidarity” (Chouliaraki 2010, 227).
Danmu take the politics of recognition much further, as illustrated by the
abundant textual traces of viewer investment in and engagement with the video
playing in the background. Viewers have been shown to rely on danmu for dif-
ferent purposes (Wu et al. 2018). In some cases, they post comments serving the
same function as traditional subtitles, i.e., delivering Chinese versions of the orig-
inal diegetic speech – that, as noted earlier, will often end up featuring near the
bottom of the frame. The collaborative nature of the subtitling process under-
taken through danmu has been documented by Wu et al. (2018), who report on
groups of users collaboratively creating Chinese “subtitles” for different sections
of the same video by inscribing their danmu on the relevant fragments. Plotters,
however, use danmu to issue warnings about imminent plot twists, typically by
framing such climatic events in terms of the emotional reactions they are likely
to elicit. Examples include: “ ‘High-energy reaction ahead!!!’ (i.e., the following
plot is very exciting), ‘High-energy reaction ahead, please wear a helmet!!!’ (i.e.,
the following plot is very exciting and intense, please be prepared mentally), [and]
‘Tut tut, see the end from here’ (i.e., the end can be guessed through this plot)”
(Wu et al. 2018, 212). Enforcing community regulation norms and participating
in “parasocial interaction” – where users engage in conversation with imaginary
interlocutors or with diegetic characters – are other purposes that danmu may
serve. The great bulk of danmu inscribed on any video, however, contributes to
the materialisation of a site that enacts the aesthetic of mutual recognition, where
viewers become visible to each other when attempting to learn foreign languages,
sharing jokes or criticisms, and other more or less mundane experiences. As a
textually and formally disruptive contrivance moulded by the interplay between
design and rhetoric, danmu environments contribute to the formation of spaces
of expressivity and interventionist mediation enabled by the co-existence of
diegetic (translational) titles and extra-diegetic posts. In their search for mutual
recognition, viewers of media content in the digital culture build communities
of rhetors by exploring the affordances of the semiotic software and gauging the
potential and limitations of barrage commenting.
6. Conclusion
As the ontology of referentiality is replaced by one of deconstruction, synthe-
sising technologies are exposing the crisis of the “representational” role that
subtitles have played within traditional audiovisual ensembles. Although exper-
imental subtitling practices can also be observed in mainstream media content,
the proliferation of citizen subtitling practices and participatory environments
like danmu is foregrounding interventionist reconfigurations of multimodal tex-
tualities driven by aesthetic and/or political allegiances. Further, the dynamics
of subtitling in the digital culture are also drawing attention to the extent to
which semiotic software – a semiotic artefact in its own right – is influencing
the practices through which viewers choose to engage with texts and with fel-
low viewers.
The popularity of danmu platforms raises important questions for subtitling
studies scholars regarding the interplay between danmu that serve the same func-
tion as a translational subtitle and other more parasocial comments. Barrage titling,
and the obscuring of the multimodal text that they are inscribed on, problematises
the long-standing contention among industry players that subtitles represent a form
of intrusion in the audiovisual ensemble. Debates concerning the need for com-
mercial subtitles to prioritise clarity and legibility – not least by jettisoning verbal
references to acoustic and visual signifying resources that mainstream audiences
can access directly – should gauge whether, and to what extent, viewers’ capacity
to establish intermodal connections is being affected. Even if we accept that danmu
are ushering in “a form of postmodern aesthetic that favours eclectic, user-driven
experiences that are detached from the actual meaning or narrative of the prod-
ucts they are actually consuming” (Xu 2016, 447), (i) understanding the specific
impact of calligraphic overflow and spectatorial saturation, and (ii) examining how
a potential contagion of the barrage culture outside of China might affect the pro-
duction of the primordial audiovisual narratives are bound to emerge as important
disciplinary concerns.
The development of these new textualities which, as noted previously, enable
the collaborative production and critique of subtitles may open new avenues for
the expansions of translation crowdsourcing models that have not been exploited
to date. Equally important are the methodological challenges involved in the
study of community formation by tracing and connecting multiple overlaid lay-
ers of written text whose authors and identities are not yet retrievable. Beyond
these specific issues, the emergence and popularisation of danmu, so far restricted
to China, will provide translation scholars who are interested in the interna-
tionalisation of disciplinary discourses with an opportunity to theorise this new
regime of signification based on an “aesthetic of hyperflatness” (Xu 2016, 447)
and informed by East Asian artistic traditions that works not only towards “the
direct subversion of the metanarrative but the very nascency of the Chinese post-
modern consciousness” (449).
Note
1 See O’Sullivan (2011) and Pérez-González (2014a) for a more extended analysis of how
subtitling enables this form of metalepsis.
References
Amara. n.d. Amara Subtitling Software. https://amara.org/en/.
Androutsopoulos, Jannis. 2013. “Participatory Culture and Metalinguistic Discourse: Per-
forming and Negotiating German Dialects on Youtube.” In DISCOURSE 2.0 Language
and New Media, edited by Deborah Tannen and Anna Marie Trester, 47–73. Washington,
DC: Georgetown University Press.
Baker, Mona. 2006. Translation and Conflict: A Narrative Account. London and New York:
Routledge.
———. 2018. “Audiovisual Translation and Activism.” In The Routledge Handbook of Audi-
ovisual Translation, edited by Luis Pérez- González, 453–67. London and New York:
Routledge.
Baker, Mona, and Bolette Blaagaard. 2016.“Reconceptualizing Citizen Media: A Preliminary
Charting of a Complex Domain.” In Citizen Media and Public Spaces: Diverse Expressions of
Citizenship and Dissent, edited by Mona Baker and Bolette Blaagaard, 1–22. London and
Banks, John, and Mark Deuze. 2009. “Co-creative Labour.” International Journal of Cultural
Studies 12 (5): 419–31.
Barra, Luca. 2009. “The Mediation Is the Message: Italian Regionalization of US TV Series
as Cocreational Work.” International Journal of Cultural Studies 12 (5): 509–25.
Chouliaraki, Lilie. 2010. “Self-mediation: New Media and Citizenship.” Critical Discourse
Studies 7 (4): 227–32.
Chung, Sung-ill. 2007. “Four Variations on Korean Genre Film: Tears, Screams,Violence and
Laughter.” In Korean Cinema from Origins to Renaissance, edited by Mee-hyun Kim, 1–14.
Seoul: Communication Books.
Clarke, Donald. 2009. “Il Divo,” Irish Times, March 20. www.irishtimes.com/culture/
il-divo-1.726848.
Crawley, Peter. 2017. “The State Review: What Has Driven 1,000 UK People to Fight
with Isis?” The Irish Times, August 24. www.irishtimes.com/culture/tv-radio-web/
the-state-review-what-has-driven-1-000-uk-people-to-fight-with-isis-1.3196993.
Cronin, Michael. 2003. Translation and Globalization. London: Routledge.
Desilla, Louisa. 2012.“Implicatures in Film: Construal and Functions in Bridget Jones Roman-
tic Comedies.” Journal of Pragmatics 44: 30–53.
Deuze, Mark. 2006. “Participation, Remediation, Bricolage: Considering Principal Compo-
nents of a Digital Culture.” The Information Society 22: 63–75.
Djonov, Emilia, and Theo van Leeuwen. 2017. “The Power of Semiotic Software: A Critical
Multimodal Perspective.” In The Routledge Handbook of Critical Discourse Analysis, edited
by John Flowerdew and John E. Richardson, 566–81. London and New York: Routledge.
Dwyer, Tessa. 2005. “Universally Speaking: Lost in Translation and Polyglot Cinema.” Lin-
guistica Antverpiensia 4: 295–310.
———. 2012. “Fansub Dreaming on Viki: ‘Don’t Just Watch but Help When You Are Free’.”
In “Non-professional Translation,” special issue edited by Şebnem Susam-Saraeva and
Luis Pérez-González. The Translator 18 (2): 217–43.
———. 2015. “From Subtitles to SMS: Eye Tracking, Texting and Sherlock.” Refractory:
A Journal of Entertainment Media 25. Accessed June 20, 2018. http://refractory.unimelb.
edu.au/2015/02/07/dwyer/.
———. 2017a. Speaking in Subtitles: Revaluing Screen Translation. Edinburgh: Edinburgh Uni-
versity Press.
———. 2017b. “Hecklevision, Barrage Cinema and Bullet Screens: An Intercultural Analy-
sis.” Participations: Journal of Audience and Reception Studies 14 (2): 571–89. www.participations.
org/Volume%2014/Issue%202/29.pdf.
———. 2018. “Audiovisual Translation and Fandom.” In The Routledge Handbook of Audiovis-
ual Translation, edited by Luis Pérez-González, 436–52. London and New York: Routledge.
d’Ydewalle, Géry and Wim De Bruycker. 2007. “Eye Movements of Children and Adults
While Reading Television Subtitles.” European Psychologist 12: 196–205.
Fox,Wendy. 2016. “Integrated Titles – An Improved Viewing Experience? A Contrastive Eye
Tracking Study on Traditional Subtitles and Integrated Titles for Pablo Romero-Fresco’s
Joining the Dots.” In Eyetracking and Applied Linguistics, Open Access Book Series Translation
and Natural Language Processing, edited by Silvia Hansen-Schirra and Sambor Grucza,
5–30. Berlin: Language Science Press.
Gambier,Yves. 2006.“Multimodality and Subtitling.” In Proceedings of the Marie Curie Euroconfer-
ences MuTra: Audiovisual Translation Scenarios, edited by Mary Carroll, Heidrun Gerzymisch-
Arbogast and Sandra Nauert. Copenhagen May 1–5. Accessed September 15, 2013. www.
euroconferences.info/proceedings/2006_Proceedings/2006_proceedings.htm.
Gambier, Yves, and Sara Ramos Pinto. 2016. “Introduction.” Target: International Journal in
Translation Studies 28 (2): 185–91.
Genette, Gérard E. 1983 [1980]. Narrative Discourse: An Essay in Method. Translated by Jane E.
Lewin. Ithaca, NY: Cornell University Press.
Grillo, Virgil, and Bruce Kawin. 1981. “Reading at the Movies: Subtitles, Silence and the
Structure of the Brain.” Post Script: Essays in Film and the Humanities 1 (1): 25–32.
Grochowski, Thomas. 2007. “Horse Feathers, Monkey Business and Lion Taming: The
Marxes in Transition.” In A Century of the Marx Brothers, edited by Joseph Mills, 53–71.
Newcastle upon Tyne: Cambridge Scholars Publishing.
Gumbrecht, Hans Ulrich. 2004. Production of Presence:What Meaning Cannot Convey. Stanford:
Stanford University Press.
Gunning, Tom. 1986. “The Cinema of Attraction: Early Film, its Spectator and the Avant-
Garde.” Wide Angle 8 (3): 58–70.
———. 1993. “ ‘Now You See It, Now You Don’t’: The Temporality of the Cinema of
Attractions.” The Velvet Light Trap 32: 4–12.
Heath, Stephen. 1981. Questions of Cinema. Bloomington: Indiana University Press.
Hogan, Ambrose. 2009. “Il Divo.” Thinking Faith, April 1. www.thinkingfaith.org/articles/
film_20090401_1.htm.
Howard, Craig D. 2012. “Higher Order Thinking in Collaborative Video Annotations: Inves-
tigating Discourse Modeling and the Staggering of Participation.” Unpublished PhD
Thesis, Indiana University Bloomington. https://eric.ed.gov/?id=ED552082.
Iedema, Rick. 2003. “Multimodality, Resemiotization: Extending the Analysis of Discourse
as Multi-semiotic Practice.” Visual Communication 2 (1): 29–57.
Iwabuchi, Koichi. 2002. Recentering Globalization: Popular Culture and Japanese Transnationalism.
Durham, NC: Duke University Press.
James, Meg. 2001. “Language Barrier Slows Movie Releases.” Los Angeles Times, August 25.
Accessed June 20, 2018. http://articles.latimes.com/2001/aug/25/business/fi-38050Y.
Jenkins, Henry. 2006. Fans, Bloggers, and Gamers: Exploring Participatory Culture. New York:
NYU Press.
Jenkins, Mark. 2009. “Powerful Mystery Man of Italian Politics.” April 24. www.
npr.org/templates/stor y/stor y.php?stor yId=103290407&t=155423705174
2&t=1554763303526.
Jensema, Carl, Sameh E. Sharkawy, Ramalinga Sarma Danturthi, Robert Burch, and David
Hsu. 2000. “Eye Movement Patterns of Captioned Television Viewers.” American Annals
of the Deaf 145 (3): 275–85.
Kofoed, D. T. 2011. “Decotitles, the Animated Discourse of Fox’s Recent Anglophonic
Internationalism.” Reconstruction 11 (1): n.p.
Kress, Gunther, and Jon Ogborn. 1998. “Modes of Representation and Local Epistemologies:
The Presentation of Science in Education.” In Subjectivity in the School Curriculum, Work-
ing Paper 2, Institute of Education. London: University of London.
Contemporary Communication. London: Arnold.
———. 2006. Reading Images: The Grammar of Visual Design, 2nd ed. London and New York:
Routledge.
Kruger, Jan-Louis. 2018. “Eye Tracking in Audiovisual Translation Research.” In The Rout-
ledge Handbook of Audiovisual Translation, edited by Luis Pérez-González, 350–66. London
Li, Jinying. 2017. “The Interface Affect of a Contact Zone: Danmaku on Video-Streaming
Platforms.” Asiascape: Digital Asia 4: 233–56.
Littau, Karin. 2016a. “Translation and the Materialities of Communication.” Translation Stud-
ies 9 (1): 82–96.
———. 2016b. “Translation’s Histories and Digital Futures.” International Journal of Com-
munication 10: 907–28.
Ma, Xiaojuan, and Nan Cao. 2017. “Video-based Evanescent, Anonymous, Asynchronous
Social Interaction: Motivation and Adaption to Medium.” In Proceedings of the 2017 ACM
Conference on Computer Supported Cooperative Work and Social Computing, 770–82. https://
dl.acm.org/citation.cfm?id=2998256.
MacDougall, David. 1998. “Subtitling Ethnographic Films.” In Transcultural Cinema, Princeton,
edited by David MacDougall, 165–78. Princeton, NJ: Princeton University Press.
Marcus, Millicent. 2010. “The Ironist and the Auteur: Post-Realism in Paolo Sorrentino’s Il
Divo.” The Italianist 30 (2): 245–57.
Mason, Ian. 2001. “Coherence in Subtitling: The Negotiation of Face.” In La traducción en los
medios audiovisuales, edited by Frederic Chaume and Rosa Agost, 19–32. Castelló de la
Plana: Servei de Publicacions de la Universitat Jaume I.
McDonald, Keiko. 2006. Reading a Japanese Film: Cinema in Context. Honolulu: University
of Hawai’i Press.
Minh-ha, Trinh T. 1992. Framer Framed. London and New York: Routledge.
———. 2005. The Digital Film Event. London and New York: Routledge.
Mubenga, Kajingulu Somwe. 2009. “Towards a Multimodal Pragmatic Analysis of Film Dis-
course in Audiovisual Translation.” Meta 54 (3): 466–84.
Musser, Charles. 2006. “Rethinking Early Cinema: Cinema of Attractions and Narrativity.”
In The Cinema of Attractions: Reloaded, edited by Wanda Strauven, 389–416. Amsterdam:
Amsterdam University Press.
Naficy, Hamid. 2001. An Accented Cinema: Exilic and Diasporic Filmmaking. Princeton, NJ:
Princeton University Press.
———. 2003. “Theorizing ‘Third World’ Film Spectatorship: The Case of Iran and Iranian
Cinema.” In Rethinking Third Cinema, edited by Anthony R. Guneratne and Wimal Dis-
sanayake, 183–201. London: Routledge.
———. 2004. “Epistolarity and Textuality in Accented Films.” In Subtitles: On the Foreignness
of Film, edited by Atom Egoyan and Ian Balfour, 131–51. Cambridge, MA and London:
The MIT Press.
Nornes, Abé Mark. 1999. “For an Abusive Subtitling.” Film Quarterly 52 (3): 17–34.
O’Hagan, Minako, and Ryoko Sasamoto. 2016. “Crazy Japanese Subtitles? Shedding Light
on the Impact of Impact Captions with a Focus on Research Methodology.” In Eye-
tracking and Applied Linguistics, Open Access Book Series Translation and Natural Language
Processing, edited by Silvia Hansen-Schirra and Sambor Grucza, 31–58. Berlin: Language
Science Press.
O’Sullivan, Carol. 2011. Translating Popular Film. London: Palgrave Macmillan.
O’Sullivan, Carol, and Jean-François Cornu. 2018. “History of Audiovisual Translation.” In
The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-González, 15–30.
Perego, Elisa, Fabio del Missier, Marco Porta, and Mauro Mosconi. 2010. “The Cognitive
Effectiveness of Subtitle Processing.” Media Psychology 13: 243–72.
Pérez-González, Luis. 2007. “Intervention in New Amateur Subtitling Cultures: A Multi-
modal Account.” Linguistica Antverpiensia 6: 67–80.
———. 2010. “Ad-hocracies of Translation Activism in the Blogosphere: A Geneological
Case Study.” In Text and Context: Essays on Translation and Interpreting in Honour of Ian
Mason, edited by Mona Baker, Maeve Olohan and María Calzada Pérez, 259–87. Man-
chester: St. Jerome.
———. 2012. “Co-Creational Subtitling in the Digital Media:Transformative and Authorial
Practices.” International Journal of Cultural Studies 16 (1): 3–21.
———. 2013. “Amateur Subtitling as Immaterial Labour in Digital Media Culture: An
Emerging Paradigm of Civic Engagement.” Convergence: The International Journal of
Research into New Media Technologies 19 (2): 157–75.
———. 2014a. Audiovisual Translation: Theories, Methods, Issues. London and New York:
Routledge.
———. 2014b. “Multimodality in Translation and Interpreting Studies: Theoretical and
Methodological Perspectives.” In A Companion to Translation Studies, edited by Sandra
Bermann and Catherine Porter, 119–31. Chichester: Wiley-Blackwell.
———. 2016. “The Politics of Affect in Activist Amateur Subtitling: A Biopolitical Perspec-
tive.” In Citizen Media and Public Spaces: Diverse Expressions of Citizenship and Dissent,
edited by Mona Baker and Bolette Blaagaard, 118–35. London and New York: Routledge.
———. 2017. “Investigating Digitally Born Subtitling Agencies in the Context of Popular
Culture.” In Advances in Amateur Subtitling, edited by David Orrego Carmona and Yvonne
Lee, 15–36. Newcastle: Cambridge Scholars.
———. 2020. “Multimodality.” In The Routledge Encyclopedia of Translation Studies, 3rd ed.,
edited by Mona Baker and Gabriela Saldanha, n.p. London and New York: Routledge.
Poulsen, Søren Vigild, Gunhild Kvåle, and Theo van Leeuwen. 2018. “Introduction.” In
“Social Media as Semiotic Technology,” special issue edited by Søren Vigild Poulsen,
Gunhild Kvåle and Theo van Leeuwen. Social Semiotics 28 (5): 593–600.
Remael, Aline. 2001. “Some Thoughts on the Study of Multimodal and Multimedia Transla-
tion.” In (Multi) Media Translation: Concepts, Practices and Research, edited by Yves Gambier
and Henrik Gottlieb, 13–22. Amsterdam: John Benjamins.
———. 2003. “Mainstream Narrative Film Dialogue and Subtitling.” The Translator 9 (2):
225–47.
Romero-Fresco, Pablo. 2018. “Accessible Filmmaking: Translation and Accessibility from
Production.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-
González, 96–113. London and New York: Routledge.
Sasamoto, Ryoko. 2014. “Impact Caption as a Highlighting Device: Attempts at Viewer
Manipulation on TV.” Discourse, Context and Media 6: 1–10.
Silverman, Kaja. 1988. The Acoustic Mirror: The Female Voice in Psychoanalysis and Cinema.
Bloomington: Indiana University Press.
spectives on Multimodality, edited by Eija Ventola, Cassily Charles and Martin Kaltenbacher,
9–30. Amsterdam and Philadelphia: John Benjamins.
Taylor, Christopher. 2003. “Multimodal Transcription in the Analysis: Translation and Subti-
tling of Italian Films.” The Translator 9 (2): 191–205.
———. 2014. “Multimodality and Audiovisual Translation.” In Handbook of Translation Stud-
ies, Vol. 4, edited by Yves Gambier and Luc van Doorslaer, 98–104. Amsterdam: John
Benjamins.
———. 2016.“The Multimodal Approach in Audiovisual Translation.” Target 28 (2): 222–36.
Thrift, Nigel. 2009. “Non-representational Theory.” In The Dictionary of Human Geography,
5th ed., edited by Derek Gregory, Ron Johnston, Geraldine Pratt, Michael Watts and
Sarah Whatmore, 503–5. London: Wiley-Blackwell.
van Leeuwen, Theo, Emilia Djonov, and Kay L. O’Halloran. 2013. “ ‘David Byrne Really
Does Love PowerPoint’: Art as Research on Semiotics and Semiotic Technology.” Social
Semiotics 23 (3): 409–23.
Vanoye, Francis. 1985. “Conversations publiques.” Iris 3 (1): 99–188.
Vietnamese Cinema Blog. 2015. “Happy Mother’s Day from Surname Viêt Given Name Nam
by Surname Nam Given Name Nam.” Accessed June 20, 2018. www.vietnamesecinema-
blog.com/2015/05/10/happy-mothers-day-from-surname-viet-g iven-name-nam-by-
surname-nam-given-name-nam/.
Wu, Qunfang, Yisi Sang, Shan Zhang, and Yun Huang. 2018. “Danmaku vs. Forum Com-
ments: Understanding User Participation and Knowledge Sharing in Online Videos.” Pro-
ceedings of the 2018 ACM Conference on Supporting Groupwork, 209–18. Accessed April 20,
2019. https://dl.acm.org/citation.cfm?id=3148344.
Xu, Yizhou. 2016. “The Postmodern Aesthetic of Chinese Online Comment Cultures.”
Communication and the Public 1 (4): 436–51.
Filmography
Boyle, Danny. 2008. Slumdog Millionaire. IMDb entry. www.imdb.com/title/tt1010048/?
ref_=nv_sr_1?ref_=nv_sr_1.
Kosminsky, Peter. 2017. The State. IMDb entry. www.imdb.com/title/tt6068620/?ref_
=fn_al_tt_2.
Minh-ha, Trinh T. 1989. Surname Viêt Given Name Nam. IMDb entry. www.imdb.com/title/
tt0098414/.
Sorrentino, Paolo. 2008. Il Divo. IMDb entry. www.imdb.com/title/tt1023490/?ref_
=fn_al_tt_1.
5
TRANSLATING “I”
Dante, literariness, and the inherent
multimodality of language
Matthew Reynolds
1. Adam, language, and the idea of multimodality

In Dante’s Paradiso, the character named “Dante” (who is also the first-person nar-
rator) encounters the first human, Adam. With what may seem to be some lack of
tact, Dante addresses Adam thus: “O pomo” (“O apple”) (Alighieri 1975a, 383).Yet
Adam does not bridle at being reminded of what one might expect he would most
want to forget; in fact, he does not even appear to notice. The characters seem here
to inhabit a realm of smooth communication where potential misunderstandings
and faux pas evaporate. Like the other souls in Paradise, Adam is at ease with his
quondam sin, since now he is wholly in harmony with the divine plan.
In line with this paradisal way of being, Adam, who is represented by a hover-
ing light, has the power of reading (as we say) Dante’s mind. He can apprehend for
himself the questions that Dante wants to ask, since he sees them reflected in the
verace speglio (“true mirror”) of divine knowledge: how long did Adam spend in the
garden of Eden? What was the real reason for the Fall? And what was the language
or kind of speech (idiöma) that he used? However, in the phrasing of this last ques-
tion, which must have been conceived by Dante in the second person but is uttered
by Adam in the first person, there is a ripple that disrupts the perfect communica-
tion we have come to expect:
e l’idïoma ch’usai e che fei.

And the language (or “speech,” or “idiom”) that I used and that I made (or
“created”).
(Alighieri 1975a, 384)
The ripple is “e che fei.” Dante-the-character cannot have known that Adam cre-
ated the language spoken in Eden, so those words cannot have originated in his
118 Matthew Reynolds
thoughts. If there was something more in Dante’s mind than the question “what
language did you use?” it can only have been “and how was it created?” The reason
is that this was a live problem in medieval linguistics, and in his earlier book De
vulgari eloquentia, Dante had expressed a different opinion:
Dicimus certam formam locutionis a Deo cum anima prima concreatam

fuisse.
We assert that a certain (or “particular”) form of speech (or “language”) was
simultaneously created by God along with the first soul (or “person”).
(Alighieri 1996, 1. 6)
So, there is an energy of correction in Adam’s words: “that I made, that was made
by me.” We might read this as an element in the drama of the poem: Adam-the-
character puts Dante-the-character to rights. Or we might read it in terms of intel-
lectual biography: Dante-the-writer revises his earlier view. Either way, what is
presented as miraculous and perfect repetition in another mind and medium, the
representation of one person’s thought in another person’s speech, turns out to gen-
erate significant change. The thought cannot but become different as it emerges in
Adam’s words. In the act of uttering the question, he begins to answer it.
This moment points towards the sort of perception later articulated by Derrida,
that what might seem like mere repetition always involves some element of change:
there is always iteration within reiteration (Derrida 1982, 325). It also gives reason
for probing the definition of what we nowadays call “modes.” In Gunther Kress’s
words, “Mode is a socially shaped and culturally given semiotic resource for making
meaning. Image, writing, layout, music, gesture, speech, moving image, soundtrack and 3D
objects are examples of modes.” (Kress 2010, 79). But what about thought? Is thought
a mode? Thought is certainly “socially shaped,” but it does not seem quite right
to call it “culturally given.” The claim that writing or image are inextricable from
culture seems plausible, but it is possible to conceive of something that we might
want to call “thought” existing independently from anything we might define as
culture. Perhaps there are some kinds of thought (such as thinking in words) that are
culturally given and other kinds of thought (such as the perception of being sleepy)
that are not. So perhaps there are different modes of thought. Perhaps thought-
speech belongs to the same mode as vocal-speech and is something different from
thought that is not verbalised. Is the move from Dante-the-character’s thoughts to
Adam’s words then an instance of translation across modes, or reiteration within a
mode? We could try to argue that if the thoughts have been verbalised in Dante’s
mind, then it is more like reiteration within a mode, and if they have not, then it
is more like translation across modes, but the truth is that there is no right answer.
We cannot draw a solid boundary around what counts as “speech”; therefore, we
cannot give a clear definition of mode; and therefore, there is no secure distinction
between translating across two modes and reiterating within one mode.
Kress has recognised the blurriness in the definition of modes: “there is no
straightforward answer to questions such as: ‘Is font a mode; is layout a mode; is colour a
Translating “I” 119
mode?’ ” His way around this problem is to suggest that “what a community decides
to regard and use as mode is mode” (Kress 2010, 87). Certainly, observation of a com-
munity’s practices and assumptions can help in deciding whether it makes sense to call
something a mode. The fact that there are people whose job is graphic design gives
support to the idea that “layout” should count as a different mode from “writing.”
Equally, the fact that many people are likely to agree that speech is distinguishable
from gesture can support the claim that “gesture” and “speech” are different modes.
All the same, it is worth insisting that a community does not usually decide to regard
anything “as mode,” for “mode” is a technical term. Really, Kress and other theorists
of multimodality decide what to regard as mode, and the reason for chopping up
the landscape of meaning-making in the way they do lies in the explanatory power
of that act of conceptualisation. Is “handwriting” one mode, “typewriting” another,
“word processing” another, “writing online chat” another again? And what about
writing “with a biro” vs. “with a fountain pen” vs. “with a quill pen” vs. “with chalk”
vs. “with a marker on a whiteboard” vs. “with an electronic pen on a smartphone”?
We could argue that these are all different modes, all distinct resources for meaning-
making that are socially shaped and culturally given for different contexts and pur-
poses. Or we could decide that they all count as “writing,” which is one mode. The
choice will be determined by the kind of argument we want to make. There is also
the non-trivial problem of identifying “submodes,” which Marcus Tomalin considers
in his chapter for this volume (Chapter 6).
The reason why this clarification matters is that it affects what we take “mul-
timodality” to mean. Just like “multilingualism,” “multimodality” does not name
a multitude of entities called “modes” that are separate and countable. Rather, it
points to an ever-varying continuum of resources for meaning-making that can
be divided up in different ways. Social practice bunches some of these together so
that it can seem obvious they should be defined together as one mode (e.g., “writ-
ing”). The same is true of language (e.g., “Italian”). But if you look closer you
can always find reasons for separating out the strands and seeing them as different
modes (again, the same is true of languages, with their different dialects, idioms,
registers, and styles).
Back in Dante’s Commedia we find a picture of language, and of its place in the
ever-varying continuum of meaning-making, that illustrates these points. As Adam
continues his disquisition in Paradiso 26, he explains that the language he spoke in
the garden of Eden had completely disappeared even before the catastrophe of the
Tower of Babel (traditionally, of course, the moment at which language variety
began). The reason flows from the fact that he alone had created that language:
every product of human intelligence (“effetto … razïonabile”) is subject to change
because human pleasure or choice (“piacere”) continually refreshes itself just like
the sky (“rinovella/seguendo il cielo”). He gives an example:
Pria ch’i’scendessi a l’infernale ambascia,

I s’appellava in terra il sommo bene
onde vien la letizia che mi fascia;
e El si chiamò poi: e ciò convene,

ché l’uso d’i mortali è come fronda
in ramo, che sen va e altra vene.
(Alighieri 1975a, 384)
Before I went down to the infernal stifling-suffering,/ I was the name on

earth of the highest good/ whence comes the happiness that swathes me;/
and EL he was called later: and that’s the way it has to be/ because the work
of mortals is like a leaf/ on a branch: it goes and another comes.
Since Adam is right about language-change, any piece of language might be taken
to exemplify the processes of reiteration and alteration to which he refers: they are
all-pervasive. But the texture of this passage, just like Adam’s voicing of Dante’s
thoughts, seems designed to bring their variety and complexity into the light.
There is the way the two words for God are hooked into the phonetic and alpha-
betic weave of the lines, one finding an echo and reflection in the line before, and
the other in the line after, in a concatenation of repetition-with-difference:

. . .
e EL si chiamò poi: e ciò convene,
ché l’uso d’i mortali è come fronda
There is the word fronda which, with its accompanying image, has itself gone and
come, from its origin in Horace’s Ars Poetica, through many different appearances
with varying connotations throughout the Commedia; as when, 60-odd lines earlier
in this same canto, Dante-the-character has described all creatures in all of creation
as leaves or fronds that enfrond the garden of the eternal gardener, “le fronde onde
s’infronda tutto l’orto/de l’ortolano etterno,” and then described his own reaction
at being introduced to Adam as being like a leaf (fronda) that bends in the wind
before bouncing back (Alighieri 1975a, 382–83). And then there is the way this
passage too represents a change of mind on the part of Dante-the-writer, for in De
vulgari he had thought that Adam’s language had remained unchanged until the
moment of Babel; that it had been Hebrew, and that the name for God was “El”
(Alighieri 1996, 1. 4–6). In line with this account of continual change, what might
be called “foreign” languages are not marked off as distinct but rather brought
together into one flexible linguistic weave. As Gianfranco Contini, Zygmunt G.
Barański, and others have observed, Virgil’s Latin, the Latin of the church, Arnaut
Daniel’s Provençal, and many dialectal forms all blend with Dante’s mainly Floren-
tine Italian to create one variable tongue which continually shuffles registers and
genres (Contini 1970, 171; Barański 1986, 8).
Translation between what might be thought of as the separate modes of speech
and writing likewise figures more as a continuum than a leap. The poem focuses
on moments where movement morphs into sound, and sound into writing, so that
its own processes of finding words (both oral and alphabetic) are layered into an
imagined universe of people and objects that are continually themselves becom-
ing language. There are many instances that we could point to: the moments in
Paradiso when the shining lights of the blessed souls sing and dance themselves into
significant shapes: an eagle of justice, a river of life, or letters of the alphabet. The
episode in Purgatorio 10 where Dante views bas reliefs so brilliantly done that his
mind’s ear can hear the figures speak. Inferno 26, where the flame in which the soul
of Ulysses is punished flutters back and forth like a speaking tongue until at last it
flicks out words. Or Inferno 13, where Dante and Virgil are in a dark, knotty, thorny
wood. Dante reaches out his hand a little (“porsi la mano un poco avante”) and
gathers a little branch (“colsi un ramicel”). A voice leaps out: “Why do you snap
me?” (“Perché mi schiante?”). From the break come words and blood together,
as from a green stick that is burning at one end so that the other end groans and
whistles as the air blows out (“geme/e cigola per vento che va via”). Dante, startled,
drops the twig and becomes “as the man who fears” (“stetti come l’uom che teme”)
(Alighieri 1975a, 54).
We could draw on the theoretical work of Nelson Goodman, in Languages of Art,
and W. J. T. Mitchell, in Iconology, to gloss this episode. As they show, the distinction
between “linguistic” and “non-linguistic” signification is hazy and cross-hatched
with continuities. Signs cannot be separated into the neat categories of “icon,”
“index,” and “symbol”; rather, “the differences between sign-types are matters of
use, habit and convention” (Mitchell 1986, 56–69). So, in Inferno 13, something
that we might choose to call an “index” of pain – the flow of blood and breath
together – develops seamlessly through whistling and groaning into sounds and
letters which have “symbolic” meaning as words (“Perché mi schiante”), though
they also, in their own patterning, offer an “iconic” picture of the sound of the
stuttering branch: “ché ... chi.” Dante’s reaction is an “index” of something fright-
ening, but also an “icon” of fear, and one that can be readily evoked by “symbolic”
verbal signification. Bodies blur into body-language, and that language is a mix of
oral suggestion and visual patterning. A sequence of letters towards and across a
line-end draws a diagram of the casual, hesitant advance of Dante’s hand towards
the branch: “porsi la mano un poco avante/e colsi”; but the letters, of course, also
evoke sounds, and the feel of the sounds inhabiting a mouth with a moving tongue
and lips, and perhaps an accompanying gesture, most evokes the physicality of the
movement that is being described. Recent neuroscientific investigations of read-
ing have recognised the complexity of the relationship between sight and sound
in language. In reading, we can harvest meaning from silent words without having
to pronounce them mentally, but the implied sound always has the potential to
be activated, sometimes in harmony and sometimes in conflict with the spelling
(Dehaene 2010, 26). Dante’s text provides rich illustration of the multimodal grada-
tion which always traverses language when it has a written as well as an oral form:
from the visual to the pictorial to the patterned to the written; from writing to
imagined speech to uttered speech to noise.
2. Practices of translation and the multimodality of

language
As language is not one simple, distinct mode, so translation is not one simple, dis-
tinct activity. Rather, the inherent and complex multimodality of language gives rise
to a great variety of translation practices. At the macro level of professional train-
ing and employment, there is a broad distinction between the translation of speech,
usually called “interpreting” in English, and the translation of writing, usually just
called “translation.” This established and practical division of labour is complicated by
many transmodal practices. Subtitling (discussed in the previous chapter) is the most
prominent of these; others occur when simultaneous interpreters mug up expected
words, phrases, and lines of argument from written documents in preparation for their
oral performances, or when the speech of consecutive interpreters in legal contexts
is taken down to form a written record. On the other side of the broad distinc-
tion, translators of writing, especially in the literary sphere, often take account of the
implied sonorities of the texts they read and recreate, while translators of plays or
political speeches produce written texts that are designed for speech. Poetry can min-
gle writtenness and spokenness in complex ways, as in a recent performance of poems
by the German poet Ulrike Sandig, at which the poet spoke lines in both German
and English into a sound box, so that they recurred on a loop, with more language
being spoken over them, and Sandig’s translator Karen Leeder at times chiming in too,
in a vivid oral expansion and mutation of written parallel text.1
The field of written translation is also haunted by speech in the shape of metaphor.
Good translations are often described in terms of voice, as when Antoine Berman
praises successful translations of Sappho as “des poèmes rendus parlantes,” or when
John Dryden famously aimed to make Virgil “speak such English, as he wou’d himself
have spoken, if he had been born in England, and in the present Age” (Berman 1999,
84; Dryden 1987, 330–31). These images of voice imply a degree of what is often
called “freedom” on the part of the translator. They are generally contrasted with
translations that are described in terms specific to writing, and are taken to be more
“close”: translations that are “literal” (i.e., literally, “to do with letters”) or “word for
word” or “line for line” – formulations which suggest the measuring out of bits of
written text. In harmony with the widespread phonocentric assumptions traced by
Derrida, the “freedom” of a metaphorical appeal to “speech” in written translation
is typically presented as a higher kind of fidelity. By hearing the author’s “voice” and
imitating it in your translation – the idea is – you are reaching for a deeper sort of
meaning than can be gathered from attention to the words as they are printed, the
“mere words” as they were called by a Victorian translator, F. A. Paley (Aeschylus
1864, v.). An alternative contrast, between “letter” and “spirit,” enters into the same
pattern of metaphorical suggestion, as “spirit” is derived from the Latin for “breath.”
No doubt, “free” translations are sometimes easier to read out loud than “literal”
ones. All the same, it is important to see that in this wide field of kinds of transla-
tion there is no simple gradation from the more visual modality of the written to the
more oral modality of the spoken. The metaphors associated with speech and writing
do not line up with the presence of physical spokenness or writtenness in a given

translation practice, and the elements can combine in surprising ways. What we are
faced with here is not a contrast between attention to “writing” on the one hand and
“speech” on the other, but rather a multitude of varying practices of interpretation,
all of which are multimodal. For instance, when Browning translated the Agamemnon
in 1877, he aimed to produce a “transcript” of the Greek text from which all hear-
able “magniloquence and sonority” had been lost, leaving only “the very turn of each
phrase in as Greek a fashion as our English will bear.” To Ezra Pound the result was
“stilted unsayable jargon” (Reynolds 2013, 153).Yet to Tony Harrison, whose transla-
tion of the Oresteia became one of the most compellingly sonic theatre experiences of
the 1980s, Browning’s text seemed full of noises:“the seeds of my principal choices,” he
said, “were lurking there in Browning from the beginning” (Reynolds 2011, 53). The
extreme instance of this ambushing of the literal by the phonetic is the word-for-word
cum voice-for-voice translation of Catullus by Louis and Celia Zukofsky which, as
they explained, “follows the sound, rhythm, and syntax of his Latin – tries, as is said, to
breathe the ‘literal’ meaning with him,” as when they translate this line from poem 70:
in vento et rapida scribere oportet aqua
into this English:
in wind o wet rapid a scribble reported in water.

(Catullus 1969, n. p.)
We might see this as the most “written” form of translation, with letter after letter
being transcribed, or we might hear it as a primarily oral enterprise, the Latin being
pronounced and mimicked in English sounds, which are then recorded on paper.
In fact, both processes are running at once and inseparably. It is no less true that a
translator who aims for “speech” must also attend to visual structuration. Dryden
gave at least as much attention to the patterns of Virgil’s verse as to its imagined
intonations. And in practice, of course, he did not make Virgil speak at all, but
rather made him write such English as he would have written if he had been born in
England and in the present age.
As I have argued elsewhere, building on work by Derrida, Naoki Sakai, Robert
Young, and others, we need to take account of the ever-varying continuum of
language when we think about translation. The idea that translation transfers some-
thing called “meaning” between separate “languages” is a huge simplification – I call
it “Translation Rigidly Conceived” (Reynolds 2016, 18). This notion, or “repre-
sentation” of translation, as Sakai describes it, collaborates with the state-sponsored
processes of linguistic standardisation which lead people to believe that there is a
language called (say) “English,” which is both internally homogeneous and sepa-
rate from other languages (say “French”), and that these separate languages can be
lined up to reveal their parallel meanings, as in the image presented by a bilingual
dictionary (Reynolds 2016, 18–23; Reynolds 2019). In fact, the communicative
elements and practices that can count as English overlap and intermingle with those
that can count as French and other languages. What is too simply called translation
“between languages” is in fact translation “across language”: it involves using mate-
rials from a different point on language’s ever-varying continuum to create a new
text with new meanings. These new meanings, however, will be sufficiently similar
to the meanings of the first text to count as the same for some purposes: to enable
us to say that we have read Proust when in fact we have read a translation of Proust,
or to understand that we are passing through what we would call “customs” when
really it is “douane.” Translations are texts which, though obviously different from
their sources, are in some ways used as though they were the same. This is how we
can distinguish translations from, on the one hand, printed copies of a book (which
are the same for almost every purpose) and, on the other, adaptations and derived
or related works which, though they might be similar to the source, are not usually
used as though they were the same.
Attention to multimodality harmonises with this account and adds some com-
plications. It helps us to keep in mind that translators may be caught, not only
between the time-honoured binary of “form” and “content” but among many
meaning-creating aspects of the spoken and/or written text: tone of voice (actual
or imagined), pace, rhythm, layout, visual structures, patterns of sound, font, gesture,
timbre, handwriting, and so on. These considerations may seem more relevant to
literary translation – especially the translation of poetry – than to other kinds. Yet
the complexities that flourish openly in literature, as we have seen with Dante, are
latent in all language use. Any translator or interpreter will encounter some of the
challenges that literary translators confront in concentrated form.
3. The case of Dante’s “I”

Even though translation always operates with and among multimodality, a radically
multimodal sign can still cause a translational short circuit. Indeed, it may well be
because of translation’s pervasive multimodality that this occurs. When challenged
by a radically multimodal sign, complexities that normally muddle along happily in
the shadows are forced out into the light, where the established idea (or “represen-
tation”) of translation clashes sharply with the reality of meaning-making practices.
This is what happens with the “I” that Dante writes for Adam.
Here it is once more:

There is no authorial manuscript of the Commedia. However, the early witnesses

were authoritatively collated in 1966–7 by Giorgio Petrocchi, building on work
published by Edward Moore back in 1889, and they concur in asserting that what
Dante must have written is a vertical – or perhaps slightly slanting – line (Alighieri
1966–7, 1. 245; 4. 440). But what did it mean? Was it a letter or a numeral? If a letter,
which one, and why? Perhaps it wasn’t exactly a letter or a numeral at all?
Dante’s recent editor, Anna Maria Chiavacci Leonardi, lists some of the pos-
sibilities that were explored by early commentators: “I” might be a single-letter
noun, chosen for its extreme simplicity (“massima semplicità”), which is compa-
rable to that of the divine, or else a number corresponding to divine unity. She
prefers another option: that Dante was adopting one of the names of God listed by
St Jerome, that is “Ia,” understood by medieval lexicographers as a consonantal “I”
pronounced “Ia” but written with its single initial letter (Alighieri 2015, 731). This
idea had been exhaustively documented in an article by Gino Casagrande, who
pointed to the incorporation of “Ia” into “alleluia” as a supporting factor (“alleluia”
is much chanted in Paradiso) (Casagrande 1976). Robert Hollander, in a long note
in his edition, noted the further possibility – first suggested by Ernesto Trucchi in
1936 – that “I” might stand for “Iesus” (written forms approximating to the English
“I” and “J” are the same letter in Italian) (Hollander 2007).
The various interpretive possibilities left their marks on the manuscripts. The
vertical might be written in a form that is unequivocally a letter, as in the C14th
“Codice 076 (88)” in Cortona which gives a lowercase “i.” Or it might be rep-
resented by the letters that form the sound of the number 1 in Italian, “un,” as in
“Codice 304 (Riccardiano 1010)” in Florence. Another Florentine manuscript,
“Codice 321 (Riccardiano 1035),” which was made by Giovanni Boccaccio, has
“un” in the main text, with “I” added in the margin ( Alighieri n. d.). These altera-
tions reveal processes of reading and transmission in which accident and inter-
pretation mix. The variant “un” may have arisen because a scribe pondered the
conundrum and chose to clarify what he took to be its meaning, but it more likely
came about because the copy text was being read aloud so that scribes could write
from dictation: Giorgio Petrocchi observes that the Commedia was often repro-
duced via dictation because it was so popular (Alighieri 1966–7, 1. 94). The reader
simply uttered the easiest spoken equivalent to the mark, perhaps without even
realising that there were alternatives. Conversely, even if the mark were transcribed
in such a way as to become unquestionably the letter “I,” the pronunciation would
remain ambiguous: either the simple Italian /i/ (as in the English “if ”), or else
something more like /y/ (as in English “yes”) if the reader has the same preferences
as Casagrande, Hollander, and Leonardi.
These variants are not the only ones. Another possibility was that the line rep-
resented the letter “L,” which could stand for the word “El” because their pronun-
ciation was identical. As we have seen, back when he wrote De vulgari eloquentia,
Dante did think that Adam’s name for God was “El,” and he makes his character
Adam correct that earlier idea here in Paradiso. But this intellectual drama may
well not have struck copyists who were inclined to trust authorities and were per-
haps unnerved by inconsistency. All the same, someone who reads the vertical
line as “El,” and writes the letters “E” and “l” to make that clear, will then have to
change the second name for God which Adam mentions, which really is “El.” Many
manuscripts give “El” for the first name and “Eli” for the second. So much effort to
avoid the vertical line. But why?
Because these copyists and commentators want the vertical (or slightly slanting)
line to have a meaning. It is their job to write sense, or to clarify the sense. And so
they pull the radically multimodal mark into one system of signification or another:
numerals, or the alphabet in the form that was shared across Latin and Italian. And
the commentators adduced (and carry on adducing) reams of theological authori-
ties to justify their choice.
There is, however, one among them who has taken the force of the radical
uncertainty that lies beneath all the ambiguities we have explored. In 1794, the
Veronese scholar Giovanni Iacopo Dionisi published a learned compendium enti-
tled De’ Blandimenti funebri o sia delle acclamazioni sepolcrali cristiane (much of it in fact
consists of observations on the text of Dante). Chapter two is a list of acronyms and
abbreviations: for instance, “A. B. M.” stands for “anime benemerenti” (Dionisi 1794,
15). The entry for “I” lists “innocens. justus. ispes. Istesanus. in. imperante. Imperatore.
indictione. junius, o junias. incomparabili, unus (numero)”; it is also pedestalled by an
enormous note on Adam’s name for God. Dionisi ridicules both the readings “El”
and “Un,” pointing out that the whole weight of what Adam says lies in the fact
that the language he once spoke has been lost: it is a “ridicolosa finzione, che il
primo Padre abbia parlato secondo la moderna favella” (“ridiculous idea, that the
first father could have spoken a still-existing language”). Dionisi then elaborates on
the iconic suggestiveness of the vertical line: in pronunciation, it tends upwards, as
though towards the heavens (“all’insù, quasi al cielo”), while its visual form is the
most simple of all (“la più semplice di tutte”): it is therefore suited both to express
the love of the soul for heavenly things and the simplicity of the supreme being. He
adds that it also happens to be the initial letter of various appropriate words such
as “imperio (nome caro al Poeta)” (“empire (a term dear to the poet)”) (Dionisi
1794, 18).
Dionisi begins to explore the fact that Adam cannot be translating a word from
the Edenic language into Italian. Rather, Adam is quoting from that lost tongue.
This means we cannot know which writing system the line belongs to, and we
cannot know the system of sounds to which that writing system (whatever it is)
corresponds. If “I,” in the Edenic writing system, can be a numeral, then there is
no reason why it should correspond to the Arabic numeral “1.” It might just as eas-
ily be a million, or infinity, or indeed seventeen. And if “I,” in the Edenic writing
system, can be a letter, then it could be pronounced like “a,” “u,” “o,” “q,” “s,” “b,”
“r” – or in absolutely any way at all. We simply have no idea. Was there even writ-
ing in Eden? If not, then we have to imagine Adam speaking an unknown word
which Dante represents in the Italian writing system with “I.” But it might equally
be that Adam invented speech and writing together, so that as soon as he named
the animals, he also knew how to write those names: he will be able to visualise
the line as he speaks the word. In this case, the shape of the vertical line will be
an intrinsic part of the word that he utters. Right from the very beginning, lan-
guage will have been radically multimodal, layering speech, writing, picture – and
perhaps an accompanying gesture, too. Perhaps the significance of the sound-and-

line, for Adam, had something in common with the speculations of twelfth-century
Sufi mystics, such as Sahl al-Tustari, Muhasibi, and Niffari, about the vertical mark
which forms part of the Arabic writing system, the Alif. As Annemarie Schimmel
explains, the Alif was felt to be the primary and perfect signifier, from which all
other letter-shapes had been generated by distortion:
To know the alif meant, for the Sufis, to know the divine unity and unicity;
he who has remembered this simple letter need no longer remember any
other letter or word. In the alif, all of creation is comprehended.
(Schimmel 1977, 417)
The consensus of European and American scholarship is that Dante did not know
of these ideas. But since, as Schimmel says, “there is scarcely a popular poet in the
Muslim world, from Turkey to Indonesia, who has not elaborated this topic,” it
is possible he may have got wind of them (Schimmel 1977, 418). Even if he did
not, one can imagine these kinds of thoughts gathering around the vertical line,
as Dante’s mind dwelt on that amazing idea, the word for God, in the language of
Eden, which had been entirely lost even before the Tower of Babel. As Dionisi rec-
ognised, the single stroke of the pen seems to signify beyond the codes of known
verbal languages: it is simple and it points upwards – that is enough. Like the
pictorial lines described by the artist Deanna Petherbridge in The Primacy of Draw-
ing, it “asserts its abstract, directional and motile qualities” (Petherbridge 2010, 88).
I would not put it past Dante to have known that he was creating a sign that would
be hard for scribes to copy, so that their texts would participate in the processes
of linguistic change that Adam describes, each manuscript a leaf that goes so that
another may come.
4. Translating the “I”

As a process of re-writing, translation has continuities both with scribal transcrip-
tion and with commentary (especially when the commentary is done in another
language such as the Latin of many early expositions of Dante). It is often said that
“every translation is an interpretation.” However, as I have shown elsewhere, the
relation between translation and interpretation is complex (Reynolds 2011, 59–72).
Someone offering an interpretation of a literary text, for instance, aims to expound
and clarify its meanings, but a translator aims to create another literary text, which
will sustain interpretation in a similar way. A word or mark such as Dante’s “I” may
be complex to interpret yet simple to translate.
But can we really speak of “translating” the “I,” given that – as we have
discovered – it is not really “in” Italian, or any known language, in the first place?
Here, attention to multimodality can add nuance to our understanding of the com-
plexities of translation. If you think of translation in the naïve way that I call “Trans-
lation Rigidly Conceived,” i.e., as taking something called “meaning” out of one
language and re-inserting it into another, then the “I” cannot be translated. But
if (as I suggested earlier) you think of translation as the creation of a very similar
text using resources from a different area of the ever-varying continuum that is
language, then of course the “I” can be translated. In this, it is like other proper
nouns, to which Derrida drew attention in his essay, “Des tours de Babel” (Derrida
1985). Names like “Babel” or “Paris” do not belong “in” any language, in the way
that “oui” belongs to French and “yes” to English, so they cannot be “translated” on
the model of Translation Rigidly Conceived. But of course they can be translated
if you have a better conception of translation: you just write out the same sequence
of letters, in the new location, and perhaps pronounce it slightly differently. In fact,
Derrida’s point also holds for many words that are not proper nouns. Is “table”
English or French? Even “yes” and “oui” can pop up and be understood in wildly
different points of the continuum of language variation.
Yet Dante’s “I” is also different from cases like “Babel” or “Paris” because of its
flagrant multimodality. As we have seen, it partakes of the modes of “image” and
“gesture” as well as of “writing” and “speech.” But this does not put it out of lan-
guage, or beyond the reach of translation, any more than does its being a proper
noun. Rather, it is an extreme case – a dense microcosm – of the iconic suggestive-
ness that is always latent in language, and is often made explicit in larger-scale struc-
tural features like bullet points or verse form. In considering what to do with the
“I,” translators respond to a multitude of multimodal factors, just as they always do.
The obvious move is simply to write “I” again. This is what happens in the very
first translations, which were done in the Iberian peninsula. In 1427, Don Enrique
de Aragón, Señor de Villena, produced a crib translation in Castilian, which is tran-
scribed in the margin of a manuscript of the Italian text. Where Dante wrote “I,”
Villena puts “I” (Alighieri 1427). The same was true in Johannis de Serravalle’s
Latin translation of 1416–17 and Andreu Febrer’s translation into Catalan in 1429
(Serravalle n.d, 1416–17; Alighieri 1878, 553). However, the fragility of these acts of
accurate perception is indicated by the fact that Villena’s “I” has been dropped from
the scholarly edition of his work done by Pedro M. Cátedra: it seems that Cátedra
was unable to process the mark, just like many of Dante’s copyists and commenta-
tors (Villena 2000, 977).
Later translations rely on manuscripts or printed editions in which the “I”
has already been replaced by something more straightforwardly linguistic and/or
numerical. Balthazar Grangier’s French translation of 1597 gives “UN”; Lebrecht
Bachenschwanz’s German translation of 1769 gives “der Eine” (Alighieri 1597, 529,
1769, 194).When the enterprise of Dante translation at last got going in English, in
the late eighteenth century, the same picture obtained. The first complete English
version of the Commedia was by Henry Boyd, a Church of Ireland vicar living in
County Down, and was published in 1802. This contained what was also the first
Englishing of the lines from Paradiso which interest us:
By one mysterious Name the Lord of all

Was known, before I heard the awful call
That led me downward to th’accursed clime.
XXVII.
‘To Eli then the nations learn’d to pray: . . .
(Alighieri 1802, 3. 305–6)
Like Grangier and Bachenschwanz, Boyd must have been using an edition which
opted for the reading “un.” Perhaps – in his case – it was the one printed by
Giuseppe Comino in Padua in 1727: it gives “UN” – in capital letters – and “ELI” –
also in capitals, just like Boyd’s translation (Alighieri 1727, 426).
The second complete English Commedia – also the second attempt to English
our lines – was by Henry Francis Cary, the Church of England vicar, writer of verse
and scholar of languages whose translation was so much admired throughout the
nineteenth century that he was buried in Poets’ Corner in Westminster Abbey, with
“TRANSLATOR OF DANTE” engraved on the stone. Cary gives:
Ere I descended into hell’s abyss,

El* was the name on earth of the Chief Good,
Whose joy enfolds me: Eli then ’t was call’d.
(Alighieri 1819, 3. 239–40)
The note signalled by “*” alerts readers to the textual crux: “Some read Un, ‘One,’
instead of El,” but it then claims that the matter is decided by reference to the mention
of “El” in De vulgari eloquentia. Cary must have been using one of the rival editions to
Comino’s, perhaps that edited by Baldassare Lombardi (Rome 1791), which gives “El”
and “Eli,” just as he does, and points to De vulgari in a note (Alighieri 1830, 672–76).
It was possible to read a correct text at this time. Dionisi’s edition, which came
out in 1796, prints “I” (Alighieri 1796, 162).Yet it does not seem to have made its
way into the hands of English translators. Throughout the nineteenth century, “El”
dominates: in the German of Karl Ludwig Kannegiesser (1825) and Karl Streckfuss
(1854a); in the French of Eugène Aroux (1842); in the Spanish of D. Juan de la
Pezuela (1868) and in the English of Charles Cayley (1854b), John Dayman (1865),
Longfellow (1867), and Frederick Kneller Haselfoot Haselfoot (1899). A few years
earlier than Haselfoot’s translation, in 1889, the Oxford scholar Edward Moore had
published Contributions to the Textual Criticism of the Divina Commedia, which, among
much else, reasserts the reading preferred by Dionisi, though with a twist:
We have here a great number of variants, but all of them seem to be easily
accounted for through a misunderstanding of an original “I” or “J,” standing
for the Hebrew letter “Jod,” and representing either the “Jah” or “Jehovah”
of Ps.lxviii. 4, &c., or still more probably (since Adam says that his primitive
language, of which this would be presumably a fragment, was “tutta spenta,” l.
124), the letter “Jod,” regarded as the cabalistic symbol for God.
(Moore 1889, 488)
We can see the traces of this interpretation in a new note to the 1908 Everyman’s
Library re-issue of Cary’s translation: “for El . . . we should read J,” as well as in the
1915 version by another Oxford scholar, Charles Lancelot Shadwell, who gives
“Jah,” as does Laurence Binyon in 1943 (with the spelling Yah), and Barbara Reyn-
olds, completing the famous Dorothy L. Sayers’s translation in 1962.
But still, no “I.” For that, anglophone readers had to await Charles Singleton’s
prose translation of 1975, which took the authoritative text established by Giorgio
Petrocchi in 1966–7 as its source:
“Before I descended to the anguish of Hell the Supreme Good from whom
comes the joy that swathes me was named I on earth; and later He was called El.”
(Alighieri 1975b, 297)
At once, a new, unshiftable obstruction to translating Dante’s “I” into English becomes
evident. The sign that, in Dante’s text, is so ambiguously multimodal, at once writ-
ing, speech, gesture and picture, is in English, all too everyday and plain. In Dante,
as we have seen, the “I” echoes the shortened form of the Italian “io,” “i,” in “pria
ch’i’scendessi” in the previous line: this sounds as a thought-provoking half-echo, and/
or a lilt of verbal melody. In English, Adam so much more bluntly refers to himself
as “I” and in the same sentence says the name of God is “I.” The static is unbearable.
What to do? Allen Mandelbaum, in 1984, chose to place the two “I”s as far apart
as possible:
Before I was sent down to Hell’s torments,

on earth, the Highest Good – from which derives
the joy that now enfolds me – was called I;
and then He was called El.

(Alighieri 1984)
Robin Kirkpatrick, in 2007, adopts a contrasting tactic, dialing up the static:
Before I sank to Hell’s deep agonies,

the Highest Good – from which derives the joy
I’m swathed in here – was known on earth as ‘I’.
Then afterwards we called it El.
(Alighieri 2012, 446–47)
An extra “I”-as-first-person-pronoun is inserted, and placed emphatically at the

start of the line which finishes with “I”-as-the-name-of-God. Clive James, in 2013,
does the opposite:
Even before I went down to Hell’s vat

Of pain, the Good Supreme that radiates
The endless joy that swathes me now was named,

On Earth, in several ways, and that was fit
For human usage can no more be blamed. . .
(Alighieri 2013, 488)
He simply leaves out Dante’s “I,” and ducks the challenge. My own suggestion
would be to look for another ambiguously multimodal mark, one which stands
in a similar relation to English as “I” does to Italian, which might be a numeral as
well as a letter, and is not a word in itself. That mark is “O.” Of course, this would
bring in many quite different suggestions from Dante’s “I” – God as zero!? God
as an expression of surprise!? – but it is still a picture and implicitly a gesture, as
well as a letter and a numeral, and one whose form can be taken as a symbol of
perfection.
What we can see in all these offerings is the variable terrain across which trans-
lation moves, the ever-shifting continuum of language, which is also a continuum
of multimodality. “I” – like all elements of language – is inherently multimodal, in
English as well as in Italian. But the ratios of its multimodality are different. In Eng-
lish it is pulled further into the coded area of the language: it is more of an everyday
word. This does not make it “untranslatable,” for any translation of anything always
introduces some difference: the creation of difference is part of what translation is
for. Rather, Dante’s “I” is an especially visible instance of the multimodality that
permeates all language, and that therefore affects every act of translation.
Note
1 This event took place at Oxford Translation Day, hosted by the Oxford Comparative
Criticism and Translation research centre at St Anne’s College, 8 June 2018.
References
Aeschylus. 1864. Aeschylus,Translated into English Prose. Translated by F. A. Paley. Cambridge:
Deighton, Bell, and Co.
Alighieri, Dante. n.d. www.danteonline.it/italiano/codici_indice.htm.
———. 1427. Comedia Dantis Allegerii florentini in qua tracta de penis et punicionibus viciorum et
de meritus et premiis virtutum. Biblioteca Digital Hispánica, MSS/10186.
———. 1597. La comédie de Dante. Translated by M. B. Grangier. Vol. 3. Paris: Vve Drobet.
Gallica: ark:/12148/bpt6k8701064j.
———. 1727. La divina commedia già ridotta a miglior lezione dagli accademici della Crusca. Padua:
Giuseppe Comino.
———. 1769. Von dem Paradiese. Translated by L. Bachenschwanz. Leipzig: Privately Printed.
———. 1796. La divina commedia. Edited by G. J. Dionisi.Vol. 3: Paradiso. Parma: Nel Regal
Palazzo.
———. 1802. The Divina Commedia of Dante Alighieri. Translated by Henry Boyd. 3 Vols.
London: T. Cadell Jun and W. Davies.
———. 1819. The Vision; or Hell, Purgatory and Paradise, 2nd ed. Translated by Henry Francis
Cary. 3 Vols. London: Taylor and Hessey.
———. 1825. Die göttliche Komödie des Dante. Edited by Karl Ludwig Kannegiesser. 5 Vols in
2.Vienna: Fr Schade.
———. 1830. La divina commedia. With a Commentary by P. Baldassare Lombardi. Vol. 3.
Florence: Leonardo Ciardetti.
———. 1842. La divine comédie: enfer, purgatoire, paradis. Edited by Eugène Aroux.Vol. 2. Paris:
Blanc-Montanier.
———. 1854a. Göttliche Komödie. Translated by Karl Streckfuss (German, 1854). Dante Lab.
dantelab.dartmouth.edu.
———. 1854b. Divine Comedy. Translated by C. B. Cayley. Vol. 3: The Paradise. London:
Longman, Brown, Green, and Longmans.
———. 1865. The Divine Comedy of Dante Alighieri. Translated by John Dayman. London:
Longmans, Green.
———. 1867. ‘Longfellow.’ Digital Dante. https://digitaldante.columbia.edu.
———. 1868. La divina comedia de Dante Alighieri:Traducida al castellano. Translated by D. Juan
de la Pezuela. Vol. 3. Barcelona:Viuda de Luis Tasso.
———. 1878. La Comedia de Dant Allighier, Tralsada de rims vulgars toscans en rims vulgars cath-
lans. Translated by Andreu Febrer. Barcelona: D. Álvaro Verdaguer.
———. 1899. The Divina Commedia. Translated by Frederick. K. H. Haselfoot, 2nd ed. Lon-
don: Duckworth.
———. 1908. The Vision of Dante Alighieri. Translated by H. F. Cary. London: J. M. Dent.
———. 1915. The Paradise of Dante Alighieri. Translated by Charles Lancelot Shadwell. Lon-
don: Palgrave Macmillan.
———. 1943. Dante’s Paradiso. Translated by Laurence Binyon. London: Palgrave Macmillan.
———. 1962. The Comedy of Dante Alighieri the Florentine. Cantica III: Paradise <Il Paradiso>.
Translated by Dorothy L. Sayers and Barbara Reynolds. Harmondsworth: Penguin.
———. 1966–7. La commedia secondo l’antica vulgata. Edited by Giorgio Petrocchi. 4 Vols.
Verona: Mondadori.
———. 1975a. La divina commedia. Edited by Giorgio Petrocchi. Turin: Einaudi.
———. 1975b. The Divine Comedy. Translated by Charles Singleton.Vol. 3. London: Rout-
ledge & Kegan Paul.
———. 1984. The Divine Comedy of Dante Alighieri: Paradiso. Translated by Allen Mandel-
baum. The World of Dante. www.worldofdante.org.
———. 1996. De vulgari eloquentia. Edited by Steven Botterill. Cambridge: Cambridge Uni-
versity Press. www.danteonline.it/.
———. 2012. The Divine Comedy: Inferno, Purgatorio, Paradiso. Translated by Robin Kirkpat-
rick. London: Penguin.
———. 2013. The Divine Comedy. Translated by Clive James. London: Picador.
———. 2015. Commedia. Con il commento di Anna Maria Chiavacci Leonardi.Vol. 3: Para-
diso. Milan: Mondadori.
Barański, Zygmunt G. 1986. “ ‘Significar per Verba’: Notes on Dante and Plurilingualism.”
The Italianist. 6 (1): 5–18. DOI: 10.1179/ita.1986.6.1.5.
Berman, Antoine. 1999. La traduction et la lettre ou l’auberge du lointain. Paris: Seuil.
Casagrande, Gino. 1976. “ ‘I s’appellava in terra il sommo bene’ (Paradiso XXVI, 134).” Aevum
50 (3): 249–73.
Catullus. 1969. Catullus (Gai Valeri Catulli Veronensis Liber). Translated by Celia and Louis
Zukofsky. London: Cape Goliar Press.
Contini, Gianfranco. 1970. Varianti e altra linguistica. Turin: Einaudi.
Dehaene, Stanislas. 2010. Reading in the Brain: The New Science of How We Read. London:
Penguin.
Derrida, Jacques. 1982. Margins of Philosophy. Translated by Alan Bass. Brighton: Harvester.
———. 1985. “Des tours de Babel.” In Difference in Translation, edited by Joseph F. Graham,
209–48. Ithaca, NY: Cornell University Press.
Dionisi, Giovanni Iacopo. 1794. Dei blandimenti funebri o sia delle acclamazioni sepolcrali Cristi-
ane. Padova: Stamperia del Seminario.
Dryden, John. 1987. The Works of John Dryden. Edited by E. N. Hooker, H. T. Swedenberg, Jr.
et al. 20 Vols:Vol. 5. Berkeley and London: University of California Press.
Goodman, Nelson. 1976. Languages of Art, 2nd ed. Indianapolis and Cambridge: Hackett.
Hollander, Robert. 2007. “Robert Hollander (English, 200–2007).” Dartmouth Dante Project.
https://dante.dartmouth.edu.
Mitchell, W. J. T. 1986. Iconology: Image, Text, Ideology. London and Chicago: University of
Chicago Press.
Moore, Edward. 1889. Contributions to the Textual Criticism of the Divina Commedia. Cam-
bridge: Cambridge University Press.
Petherbridge, Deanna. 2010. The Primacy of Drawing: Histories and Theories of Practice. New
Haven and London:Yale University Press.
Reynolds, Matthew. 2011. The Poetry of Translation: From Chaucer & Petrarch to Homer & Logue.
Oxford: Oxford University Press.
———. 2013. Likenesses:Translation, Illustration, Interpretation. Oxford: Legenda.
———. 2016. Translation: A Very Short Introduction. Oxford: Oxford University Press.
———, ed. 2019. Prismatic Translation. Oxford: Legenda.
Schimmel, Annemarie. 1977. Mystical Dimensions of Islam. Chapel Hill: University of North
Carolina Press.
Serravalle, Johannis de. n. d. “Johannis de Serravalle (1416–17).” Dartmouth Dante Project.
https://dante.dartmouth.edu.
Villena, Enrique de. 2000. Obras Completas. Edited by Pedro M. Cátedra. Vol. 3. Madrid:
Turner.
6
THE MULTIMODAL DIMENSIONS
OF LITERATURE IN TRANSLATION
Marcus Tomalin
1. Introduction
In his 1828 essay “On the Causes of Popular Opinion,” William Hazlitt admitted
that “[t]ill I began to paint, or till I became acquainted with the author of The
Ancient Mariner, I could neither write nor speak” (Hazlitt 1828, 312). This is a
disconcerting remark. The disjunctive conjunction implies uncertainty about the
origins of his mature linguistic ability: either it was learning to paint that triggered
it, or else it was his encounter with the poet Samuel Taylor Coleridge (whose cel-
ebrated The Rime of the Ancyent Marinere had appeared in Lyrical Ballads in 1798) –
or maybe both occurrences were equally influential? His syntax is elusive. Despite
these tantalising ambiguities, though, Hazlitt’s statement suggests a close connection
among the visual arts, literature, and language. And he knew what he was talking
about. Having developed a practical interest in painting while in his teens, he had
sojourned in Paris from October 1802 to January 1803, where he examined and
copied various works by Old Masters, such as Titian and Poussin, that were on dis-
play in the Louvre.1 On his return to England he spent three years as an itinerant
portrait painter, before making a career for himself as a writer. Despite his mid-
life career change from paintbrush to pen, he never lost either his love for, or his
knowledge of, the visual arts; and in his celebrated essay “On the Pleasure of Paint-
ing” (1821), he reflected upon the differences between the two art forms. Having
stated bluntly that he never experienced “much pleasure in writing these Essays, or
in reading them afterwards,” his disenchantment with prose composition was placed
in stark contrast to his enjoyment of creating visual images: “[o]ne is never tired of
painting, because you have to set down not what you knew already, but what you
have just discovered” (Hazlitt 1821, 7).
Hazlitt’s fascination with the associations and disassociations between literary
composition and painting may have been distinctive, but it was far from unique.
Multimodal dimensions of literature in translation 135
Many other writers have perceived close connections among language, literature,
and the visual arts – and some were creatively involved in all of these domains.
Indeed, Hazlitt’s older contemporary,William Blake, provides yet another conspicu-
ous example. His illuminated books, which include Visions of the Daughters of Albion
(1793), Songs of Innocence and Experience (1974), Milton (1804–1810), and Jerusalem
(1804–1820), were printed using relief-etched copper plates that combined words
and images. His poetry is therefore inherently pictorial, his letter-forms calligraphic,
and his use of colour, line, layout, and design all contribute to the meaning of
each work as a whole (Bindman 2009). As a trained artist and printmaker, Blake
was aware of the great traditions of literature that combined linguistic forms with
visual design, and he drew inspiration from them. The antiquarian sub-cultures of
Romanticism made some of the older traditions of pictorial representation (e.g.,
illustrated manuscripts and Renaissance woodcuts) much more familiar, while
other kinds of contemporaneous images (e.g., illustrations in children’s books) were
becoming increasingly common. And the intertwining of image and text con-
tinued to flourish in the ensuing decades. To take just two examples from many:
Charles Dickens’ novels were illustrated by artists such as George Cruikshank,
Hablot Night Browne (a.k.a. Phiz), and John Leech, while the idiosyncratic English
writer Stevie Smith produced doodle-like sketches that accompanied many of her
printed poems from the 1930s onwards (Cohen 1980; May 2015). In their very
different ways, therefore, these examples constitute works of literature that mingle,
or juxtapose, sequences of words and images (in one form or another) to create
multimodal ensembles with perceived aesthetic value. Throughout this chapter, the
term “ensemble” will be used in the manner defined in the Introduction (p. 13);
that is, it will denote representations or communications that contain two or more
modes that have been brought together purposefully to produce a collective and
interrelated meaning. Further, the person, or people, responsible for creating the
ensemble “orchestrate” it by determining how the constituent parts are combined
(though this established jargon becomes problematical when musical examples are
considered, since such works can involve “orchestration” of a different kind). From
a literary critical perspective, multimodal ensembles are of considerable significance
in and of themselves, yet the focus in this chapter will fall primarily upon the dif-
ficulties that arise when such literary “texts” are translated. The cautionary quota-
tion marks usefully signal a difficulty that will need to be confronted in the ensuing
discussion. In general, I will use words like “text” sparingly (whenever possible),
since they can be interpreted as insinuating the dominance of linguistic elements
in literary works that combine more than one mode. Sensitivity to such matters is
essential since, as will be emphasised later in this chapter, the hierarchical orderings
of the modes in specific multimodal ensembles are rather more complex than is
often acknowledged, especially when translation is involved.
Accordingly, this chapter will outline a framework for approaching the diffi-
cult task of translating multimodal works that involve both texts and images, and
which can be classified as “literature.” Once again, the cautionary quotation marks
here are salutary, for several reasons. They remind us to avoid the dubious, but
136 Marcus Tomalin
time-honoured, convention of privileging writing over all other modes, but they
also recognise that, in the current age of experimental multimodal fiction, audio-
visual art installations, and the like, the denotation of “literature” has never been less
stable (Gibbons 2012). The importance of approaching these issues cautiously is
hopefully self-evident. While literary translation has arguably received a dispropor-
tionate amount of critical attention over the centuries, providing a central concern
for theorists ranging from St Jerome to John Dryden to Friedrich Schleiermacher
to Eugene Nida to André Lefevre to Jacques Derrida to Lawrence Venuti (Venuti
2000), comparatively little attention has been devoted to how translators handle
literary works that combine multiple modes. However, the recent emergence of
theories of multimodality has created an urgent need to reconsider which par-
ticular processes of transposition and meaning (re)constitution should be classi-
fied specifically as instances of translation.2 While this topic can undoubtedly be
considered in relation to a wide variety of communicative practices, ranging from
the prestigious to the seemingly banal, a focus on works of literature reveals distinc-
tive intricacies that merit especially careful scrutiny. This is partly because certain
authors have purposefully sought to destabilise culturally specific modal ontolo-
gies that were conventionally presupposed during a given epoch. Consequently, in
particular poems, novels, and plays, the boundaries separating image from writing,
writing from speech, speech from music, and so on, become faint to the point
of invisibility – and this has unavoidable consequences for those who undertake
the task of translating those multimodal works of literature. Translation in these
situations, therefore, has not infrequently probed the limits of communication and
meaning-making in playful ways, often with inventive brilliance.
Consequently, the remaining sections of this chapter will examine in detail some
of the challenges posed by the translating of multimodal “literature,” with a particu-
lar emphasis on the livre d’artiste. As Tong-King Lee has emphasised insightfully, this
undertaking involves determining the role(s) translation plays in the “interplay of
different semiotic plans of representation” (Lee 2012, 242). The specific framework
outlined here is not presented as being pristine and exhaustive in every respect, of
course – far from it. It is, instead, merely offered as a tentative invitation for oth-
ers to reflect at greater length upon the kinds of issues this chapter addresses in an
unavoidably rough and ready manner.
2. Multimodal literary translation

As Gunther Kress and others have stressed repeatedly in recent years, “commu-
nication has always been multimodal” (Kress et al. 2001, 2). Whether we are
considering letters, diaries, architectural plans, medical textbooks, spontaneous
conversations, discussions in sign language, comics, scientific treatises, atlases,
TV broadcasts, vlogs, or anything else, some combination of speech, writing,
images, gestures, music, and so on is usually involved. Since literature is merely an
imperfect umbrella term for a wide range of distinctive forms of communication
(e.g., novels, short stories, sermons, essays, sonnets, epic poems), then it follows
naturally that literature has always been multimodal too. Yet there are clearly
degrees of multimodality, and some literary works are certainly more multimodal
than others. Shakespeare’s Sonnet 1, for example, as printed in the 1609 Quarto,
can be subjected to various forms of traditional literary analysis that address issues
relating to its form and content:
From faire!t creatures we de!ire increa!e,

That thereby beauties Ro!e might neuer die,
But as the riper !hould by time decea!e,
His tender heire might beare his memory:
But thou contracted to thine owne bright eyes,
Feed’!t thy lights flame with !elfe !ub!tantiall fewell,
Making a famine where aboundance lies,
Thy !elfe thy foe, to thy !weet !elfe too cruell:
Thou that art now the worlds fre!h ornament,
And only herauld to the gaudy !pring,
Within thine owne bud burie!t thy content,
And tender chorle mak!t wa!t in niggarding:
Pitty the world, or el!e this glutton be,
To eate the worlds due, by the graue and thee.
It was printed using a specific type-font and a particular set of Early Modern
orthographic conventions (e.g., the spelling “fewell” in line 6). Structurally, the
text can be divided into three quatrains and a couplet, which follow the rhyme
scheme ABAB CDCD EFEF GG. Thematically, it contains allusions to the Eden of
Genesis, aesthetics, procreation, and the undesirability of wasteful self-consumption.
These properties and qualities can all be examined via a literary critical response
that focuses primarily upon the text as a piece of writing. Since writing is one of
the modes conventionally recognised in multimodal analyses, then a consideration
of such matters results from a predominantly monomodal reading. Of course, like
most written texts, the sonnet can be read aloud or recited from memory, and when
that happens the semiotic affordances of speech come into play (e.g., accent, intona-
tion contour, voice type). Therefore, even Sonnet 1, which appeared initially in a
silent printed form on the page, has the potential to become a multimodal entity.
This foregrounds the relationships that exist between certain modes, and this cru-
cial topic merits more attentive scrutiny than it has so far received. For instance,
most written texts can be spoken aloud, and therefore speech is (in some sense) a
latent mode since it is potentially available whenever we encounter a sequence of
written words.3 Yet the degree of latency varies depending on the modes involved.
In order to explore this in a preliminary way, later in this chapter I will define spe-
cific sub-types – namely, proximal and distal modes – and reflect upon how they
are involved in the process of translating multimodal ensembles.
So, like most written texts, Shakespeare’s Sonnet 1, in its printed 1609 Quarto
form, can become a multimodal entity if it is read aloud. But perhaps it was
138 Marcus Tomalin
already multimodal to begin with, even before the mode of speech becomes
involved? Any such claim necessarily presupposes some degree of certainty about
the particular modes that are recognised and deployed as such by a specific com-
munity. For instance, printed written texts which also involve illustrations reveal a
purposefully multimodal intention underlying their creation, and such examples
have proliferated dramatically in recent years, partly because writing conventions
have changed due to pervasive technological advances. An interactive work of
online e-literature such as María Mencía’s El Winnipeg: El barco de la esperanza
(2017) intentionally combines images (e.g., maps, photographs), writing (e.g.,
texts by Pablo Neruda, extracts from letters, moving words/characters), and so on.
It takes the form of a multimodal ensemble to tell the story of the SS Winnipeg,
the ship that arrived in Chile in 1939 with more than 2,000 Spanish passengers
aboard, all of them immigrants fleeing Francisco Franco’s dictatorial regime.4 The
meaning of the composition results from the interplay of these different mul-
timodal features. They combine, and can be combined, to enable the reader(s)
to (re)constitute a range of meanings. It is trivially apparent that a work of this
kind is inherently far more multimodal than the Quarto text of Shakespeare’s
Sonnet 1, but this is an important point nonetheless. As will be shown later, the
multimodal dimensionality of any given source ensemble can be altered by a
process of translation, with the result that the target entity has a different multi-
modal dimensionality compared to the source. This phenomenon has rarely been
explored at length in previous studies, but it will provide the primary focus for
this chapter.
Given this overarching thematic concern, it will prove useful to introduce a
simple notational system that indicates the multimodal dimensionality of any given
object (be it a work of literature or otherwise). For instance, if P is a specific mul-
timodal ensemble – say, an e-poem that involves writing, images, and music – then
P’s multimodal dimensionality can be indicated by a mode-quantifying superscript:
P 3. This notation merely indicates that P is an ensemble consisting of three distinct
modes. Therefore, if we have two multimodal ensembles, P3 and Q5, we can easily
see that the former has a lower multimodal dimensionality than the latter, and this
can help us compare and contrast these ensembles. The usefulness of this notation
will become more apparent later in this chapter when the discussion concentrates
more directly on the processes of literary translation.
3. Identity and/or difference?

It should be apparent, though, that the rudimentary notational system introduced
here presupposes (i) that the various modes are all distinct and well-defined (at least
in relation to specific social groups of users), and (ii) that there exists an ontology
which specifies the relations among the various modes. In practice, while there is
emerging agreement about the sorts of things modes are and the sorts of semiotic
purposes they serve (whether in isolation or as part of ensembles), there is usually
no consensus as to the specific set of modes that should feature in any given analysis
in relation to any given social group. As noted in the Introduction, while some
theorists treat language as a distinct mode, others (especially Kress) view that usage
as a misleading conflation of two distinct modes – namely, speech and writing. These
disagreements usually have their roots in the many cultural conventions that overtly
or covertly influence our thinking about the semiotic affordances of specific modes.
For instance, as mentioned earlier, speech and writing are often closely associated
with each other in the Western world. In languages such as English, French, Italian,
Spanish, and German (to mention just a few high-profile ones), it is generally pos-
sible to establish a one-to-one correspondence between their respective constitu-
ent units (e.g., a spoken word and a written word; a spoken phrase and a written
phrase), and therefore their respective frames share correspondences.5 Even though
the contractions and disfluencies of spontaneous speech complicate this a little
(e.g., does “gonna” consist of one lexical item or two?), it is still possible to think
of speech and writing as being proximal modes. That is, structural mappings from
one to the other are well-established by prevailing mainstream usage conventions
in relatively large social groups (e.g., those for whom, say, British English is a first
or second language). The same is true, to a different extent, for speech (or writing)
and gesture-based sign language.
From a theoretical perspective, it is intriguing that processes of monolingual
intermodal translation involving proximal modes have not commonly been clas-
sified as translations at all. That is why the reading aloud of a text is not generally
considered to be a translation of the source material, even though the meaning of
the written words has been (re)constituted using a different mode with different
semiotic affordances (e.g., loudness, stress, intonation contour, rhythm, pitch, dura-
tion). Indeed, (re)constitutions of meaning involving proximal modes are usually
only considered instances of translation proper if (i) they are language-related and
(ii) they involve a change of language system.6 For instance, if a printed British
English poem is read aloud in Metropolitan French, then, in common usage at
least, the spoken version would uncontroversially be classified as a translation of the
written source. Despite the added bilingual dimension, the modes involved in this
case are still proximal: there are broadly accepted ways of mapping written British
English to spoken Metropolitan French, even though various options would be
available to the translator at many points (e.g., <sorrow> could be translated into
spoken French as /tʁis.tɛs/ or /ʃa.ɡʁɛ/̃ , depending on the context).7 By contrast,
there are few established conventions for mapping units of speech to units of image.
There is no well-established correspondence between, say, the phoneme sequence
/aɪ laɪk ðɪs ˈpəʊɪm/ and the grammar of visual design (i.e., line, colour, shape,
texture, composition), excluding the visual design of written type-fonts, of course.
This is why if 100 people were asked to draw a picture to convey the meaning of
the IPA-notated utterance given earlier, they would produce 100 very different pic-
tures. Indeed, several studies of ekphrasis over the years have emphasised that images
do not have framing counterparts in spoken utterances that have been established
by long-standing conventions of usage (and vice versa), and this is why speech and
image can be classified as distal modes.8 Specifically, unlike proximal modes, the
140 Marcus Tomalin
structural mappings from one distal mode to the other are not well-established by
prevailing mainstream usage amongst particular social groups.
Curiously, as Helen Julia Minors notes in her chapter for this book (Chap-
ter 7), (re)constitutions of meaning involving distal modes (e.g., a written poem
reworked as a ballet) have not traditionally been classified as instances of transla-
tion either. The term has most often been used metaphorically, implying that
choreographing a ballet based on a poem is like the process of translation. This
situation is further complicated by the fact that an unavoidable relativism char-
acterises the use of modes. As Kress has put it, “[w]hat a community decides to
regard and use as mode is mode” (Kress 2010, 87). Consequently, a specific group,
within a particular culture, may start using X (where X could be certain colours
or sounds, or whatever) in such a way that it functions as a mode as far as the
members of that group are concerned, even if no other communities regard and
use it in the same manner. Therefore, the precise answer to the important ques-
tion “is X a mode?” will depend entirely upon the community being considered.
Sometimes the right answer will be “no,” and sometimes it will be “yes.” Clearly,
practices that emerge in the social activities of one group can, in some cases,
spread to other groups, until they become more widely adopted. The situation
is dynamic and fluid, never static and rigidly fixed. And, of course, the funda-
mental underlying assumption that all modes are discretely separable from each
other, and therefore can be unproblematically counted using positive integers,
is little more than a crude simplifying abstraction. For instance, as discussed in
more detail later, speech and music might appear to be obviously distinct modes,
but singing and speech are not always easily distinguished from each other (e.g.,
sprechgesang). Consequently, Derrida’s provocative methodological question is
especially pertinent for theories of multimodality: “[s]hould one save oneself by
abstraction or save oneself from abstraction?” (Derrida 2002, 42).
Recent multimodal theorising about fonts illustrates these categorisation problems
well. Most French readers would consider Poems 1 and 2, which are “both” by the
avant-garde French poet Guillaume Apollinaire, to have exactly the same meaning:
Poem 1: Poem 2:
Le Dromadaire Le Dromadaire
Avec ses quatre dromadaires Avec ses quatre dromadaires
Don Pedro d’Alfaroubeira Don Pedro d’Alfaroubeira
Courut le monde et l’admira. Courut le monde et l’admira.
Il fit ce que je voudrais faire Il fit ce que je voudrais faire
Si j’avais quatre dromadaires. Si j’avais quatre dromadaires.
These poems share the same spellings, the same punctuation conventions, the
same line breaks, the same capitalisation patterns, the same semantic connotations,
the same syntax, the same lexis, and so on; hence the conclusion that they are
effectively identical. The only differences are the style and size of their respective
fonts (i.e., Times New Roman, 11pt for Poem 1, and Calibri, 14pt for Poem 2).
Even though these features could easily form part of the bundle associated with the
mode referred to as writing (i.e., any printed written text will necessary use specific
font types and sizes), most French readers would place minimal importance on
such things, to the extent that they would essentially ignore them. By contrast, a
community of graphic designers might share both established regularities of usage
and assumptions regarding the meaning content of different fonts, and therefore,
for that specialist community, Times New Roman and Calibri might well convey
consistently distinct meanings.9 Indeed, some fonts have acquired relatively stable
semantic connotations that now extend beyond the confines of a small group of
professionals with particular knowledge and expertise. Fonts such as Helldorado, for
instance, share typological features with several others that were widely used in title
and credit sequences in Western films from the 1930s onwards. Films such as Destry
Rides Again (1939), Stagecoach (1939), Last Train from Gun Hill (1959), The Good,
the Bad and the Ugly (1966), and many many more, all used typographical conven-
tions derived from posters printed in the so-called Wild West from the mid-to late
nineteenth century. Therefore, the two following sentences have different (self-
referential) meanings, even for many readers who have no specialist knowledge
either of typographical conventions or of Westerns:
Sentence 1:
This sentence makes me think of Clint

Eastwood.
Sentence 2:
The usage-related conventions of fonts such as Helldorado have become so strongly

established that virtually any letter-forms with horizontals thicker than the verticals
now convey some kind of (vague) notion of Westerns, as a cultural genre. Con-
sequently, while Sentence 1 seems inexplicable to most people – why should that
particular sequence of words prompt thoughts about the actor and director Clint
Eastwood? – Sentence 2 makes much more sense: it suggests “Clint Eastwood”
because the text is written in a font similar to those used in many of Eastwood’s
most famous cowboy films. Recognising that certain letter- forms can convey
meanings in this way, Theo van Leuwen, Nina Nørgaard, Frank Serafini, Jennifer
Culson, and others have recently argued that font should be recognised as a mode in
its own right (van Leuwen 2006; Nørgaard 2009; Serafini and Culson 2012). And
a greater attentiveness to these issues has non-trivial consequences for the study of
literature. As Nørgaard in particular has pointed out, “there is a general tendency
in literary criticism to disregard the semiotic potential of typography in literature
by focusing monomodally on word-meaning only” (Nørgaard 2009, 141). This
142 Marcus Tomalin
pervasive disregard reflects the logocentric tendencies of certain literary critical

traditions, and it is an attitude that multimodal approaches inevitably challenge and
destabilise. Since different publishers have different house styles, we become inured
to small variations in the letter-forms in the literary texts we read, and we assume
that such changes have no significant impact on the meaning of the printed materi-
als. Consequently, if fonts are indeed one of the features bundled together to create
the mode known as writing, then they are generally deemed to be ignorable ones,
even though that stance is evidently a precarious one in certain cases.
Nonetheless, Poem 2 would not usually be classified as a translation (either a trans-
duction or a transformation, to use Kress’s terms) of Poem 1.10 The meaning-material
created by the writing has not been moved about or (re)constituted in any way. How-
ever, if font were to be recognised as a mode in its own right (for a given community, at
least), then a mode-related change has clearly taken place, and therefore some kind of
transposition of the meaning-making resources has occurred. If that were the case, then
Poem 2 would presumably constitute an intramodal translation (i.e., a transformation)
of Poem 1, even though this would stretch the traditional denotation of the term “trans-
lation” to its breaking point. Similar complications arise when fonts become difficult to
separate from the mode image. Ornate, decorative, calligraphic embellishments known
as “flourishes” produce written texts that make highly stylised use of colour, contour,
line, and composition (Hildebrandt 1995; Lupfer 2003).When considering texts of this
kind from a multimodal perspective, it is difficult to determine where writing ends
and image begins. Once again, when such entities are created, the precise ontology of
the modes often becomes opaque and/or indistinct. Nonetheless, these considerations
emphasise strongly the extent to which the multimodal dimensionality of any ensemble
(and especially a work of literature) is necessarily conditioned on the meaning-making
practices of a given community. Consequently, to extend further the formalism intro-
duced in the previous section, we can state that the entity P has a multimodal dimen-
sionality of 3 in relation to the specific community c1, and this can be notated as P c31 .
If the same object were considered in relation to a different community, c2, and if only
two of the three modes were regarded and used by the members of c2, then we can
capture this with the notation P c22 . In other words, the multimodal dimensionality of
an ensemble can only be determined in relation to a specific social group, and that may
often have an impact on the analysis adopted. It also provides a more rigorous frame-
work for exploring the kinds of communicational complexities that can arise when
members of c1 and c2 interact concerning multimodal ensembles such as P, for instance
via a process of translation. This topic will be considered at much greater length later
in the next sections.
4. Translating texts and/or images

By contrast with the complexities that characterise the relationship between
Poems 1 and 2, it is much less outlandish to assert that Poem 3 is a translation of
one of them (or both):
Poem 3:
The Dromedary
With his four dromedaries,
Don Pedro of Alfaroubeira
Went all around the world, admiring it.
He did the very thing I would have done
If I had four dromedaries.
Here the written French text of Poem 1 has been converted into a written English
text; therefore, intramodal translation has occurred in the familiar way. As men-
tioned already, this is the kind of monomodal meaning (re)constitution that we tra-
ditionally think of whenever translation is discussed. It is, in essence, the prototypical
kind of translation. Although they both use the mode of writing, the fact that two
languages are involved means that Poem 1 (say) and Poem 3 differ in terms of lexis,
syntax, metrical structure, and the like; therefore, they deploy different language
systems – so far, so platitudinous. But this conventional scenario becomes more
complex when the first printed edition of Apollinaire’s poem is considered. It was
published in a collection of short verses entitled Le Bestiaire ou Cortège d’Orphée
(1911), and it looked like this:
FIGURE 6.1 Le Dromadaire

144 Marcus Tomalin
The woodcut images that accompanied Apollinaire’s concise poems were all
created by the French Fauvist artist Raoul Dufy. The volume as a whole constitutes
a livre d’artiste, and it emerged from a collaboration involving the writer and the
visual artist. In this sense, it was not atypical of Apollinaire’s oeuvre. He had already
collaborated with the painter André Derain for his prose poem L’Enchateur pouris-
sant (1909), and his abiding interest in the interplay between writing and image has
been scrutinised attentively in recent years. In particular, Clive Scott has probed
Apollinaire’s fascination with handwriting, doodling, drawing, watercolours, alpha-
bets, scripts, and signatures, noting that “the calligraphic and the graphic live in easy
intercourse with one another” in his literary creations (Scott 2015, 21). As a typical
work of Apollinarian literature, therefore, “Le Dromadaire” is conspicuously and
intentionally multimodal. It combines writing and image (not to mention typeface
and the proximal mode of speech). And the placing of the image in relation to the
text is of considerable interest and importance. It appears neither on a separate page
nor at the foot of the same page, after the poem. Instead, it occurs between the title
and the first line of verse. As a result, in terms of layout, the image occupies the
same physical space as the poem text, and this emphasises the extent to which writ-
ing and image are combined in this particular work. Nonetheless, despite having a
multimodal dimensionality of 2 (at least), translations of the poem usually concen-
trate on the written text and simply provide a version of that in the target language,
implying that the two modes used in the original can be harmlessly disaggregated.
Consequently, if “Le Dromadaire” is translated into English as Poem 3, then it has
a lower multimodal dimensionality than its source material. Crudely, assuming that
S (for “Source”) denotes “Le Dromadaire” in Figure 6.1, T (for “Target”) denotes
Poem 3, while f1 and e1 denote the communities of (say) Metropolitan French and
British English readers, respectively, we can express the effect of the process of
translation, →, as follows:11
S 2f → Te11 (1)
1
As (1) indicates, this instance of intramodal translation (i.e., writing to writing)

has reduced the multimodal dimensionality of the original poem from 2 to 1: the
poem text has been disaggregated from its accompanying image, and the target only
(re)constitutes the meaning of the former. For several reasons – some ideological,
some practical – this has been the standard practice in literary translation for many
years. In his Selected Writings of Guillaume Apollinaire (1971), Roger Shattuck merely
presented an English translation of the text with no accompanying image; although
advances in modern printing technology have made it much easier now for images
to be incorporated into texts, there is still a tendency to separate them (Shattuck
1971). X. J. Kennedy’s 2011 translation of “Le Dromadaire” does includes Dufy’s
woodcut, but the poem text (in both French and English) appears on page 20, while
the image appears stranded by itself on page 21 (Kennedy 2011, 20–21). There may
have been logistical reasons for Shattuck’s exclusion of the woodcuts. The cost and
technical difficulty of including images in a printed text was much greater in the
1970s. Also, Le Bestiaire ou Cortège d’Orphée was still subject to copyright, which
may have delimited the available options somewhat. As for the format favoured by
Kennedy, it could be argued that such an approach is justified because the writing
and the image were produced by different people. While a response of this kind
makes rather naïve assumptions about the communal aspects of meaning-making,
it is worth pointing out that exactly the same approach was adopted for many
decades in relation to William Blake’s illustrated books (mentioned, briefly, earlier).
For instance, his works appeared in writing-only form even in prestigious editions
such as Geoffrey Keynes’s Blake: Complete Writings with Variant Readings (1972), even
though the same person had created both the texts and the images in the originals.
It was not until good scholarly editions of Blake’s prints became available in the
later 1970s that greater emphasis was placed on the relationship between his words
and pictures in literary critical studies. As David Erdman noted in the introduction
to his epochal The Illuminated Blake (1974), “Blake’s readers are becoming educated
spectators also” (Erdman 1974, 13). The process, however, was a gradual one.
It is crucial to acknowledge, though, that there are other processes of meaning
(re)constitution which could also be classified as forms of translation, and which
have different kinds of impact. For instance, consider the alternative approach to
the standard form of literary translation exemplified by Poem 3. A contrasting
“translation” would separate the woodcut from its accompanying text, and then
(re)constitute its meaning by creating a new visual counterpart that uses the semi-
otic affordances of the image mode in a different way. In this case, we would no
longer be dealing with prototypical linguistic translation, since the mode featured
in the target would be neither speech nor writing, and no specific language sys-
tem would be involved. Obviously, a “translation” of this kind would prioritise
image (as a mode), and a specific example is given below as Poem 4:12
146 Marcus Tomalin
The images in Figure 6.1 and Poem 4 both depict camels (specifically, drom-
edaries), but they do so in ways that convey different meanings (at least to
readers/viewers belonging to groups with certain shared image-related semi-
otic conventions). The fact that the original illustration is a black-and-white
woodcut unavoidably associates it with the long tradition of relief printing in
the Western world. That particular printmaking technique was deployed from
c.1400 onwards, and was developed and refined by Old Masters such as Albrecht
Dürer and Giovanni Battista Palumba. While single-leaf images were frequently
produced, the method was widely used to provide distinctive pictures in books
from the Renaissance onwards, and the technique had been revitalised by Paul
Gauguin in the 1890s in works such as his Noa Noa series (created in 1893–94;
published in 1901). In an 1897 essay, the painter and etcher Félix Bracquemond
had advised that harmony between typography and illustration is essential in
a livre d’artiste; therefore, it is better if black-and-white are maintained across
each page – hence the advantage of using woodcuts (Bracquemond 1897).
Influenced by these artistic trends, Apollinaire was convinced by 1909 that
“[i]ntimement liée à l’invention de l’imprimerie, la gravure sur bois est celle
dont le style se marie le plus heureusement à l’aspect d’un feuillet imprimé”
[“Being closely connected with the invention of printing, the style of the
woodcut is best suited to the appearance of a printed sheet”] (Apollinaire 1977,
1071). Dufy was knowingly alluding both to the prestigious woodcut tradition
and to these more recent aesthetic predilections, when he created the images
for Le Bestiaire ou Cortège d’Orphée – though his approach has more in com-
mon with Renaissance illustrations than contemporaneous experimental works
of French post-impressionism.13 His image of the dromedary has characteris-
tic features of design and composition. The two-dimensional anti-naturalist
single-humped camel is shown in a setting that suggests a desert oasis. Highly
stylised palm trees and other forms of vegetation are clearly visible (as they are
in many of the other woodcuts in Le Bestiaire). By contrast, the coloured image
in Poem 4 was created using a digital camera and subsequent post-processing.
This camel has a greater sense of three-dimensional modelling (e.g., the shad-
ows on the legs), and it appears against a textured background that could be a
rock face. There is no sign of an oasis. Unlike its counterpart in Figure 6.1, this
dromedary is wearing a saddle (or a blanket, at least) and a bridle, which sug-
gests domestication and human ownership.
While we might be quick (too quick?) to accept that Poem 3 is a translation of
“Le Dromadaire,” we would likely experience a much greater sense of disquiet if
Poem 4 were to be described as a translation of it as well; this disquiet endures, per-
haps, even if we distinguish between “linguistic translation” and “visual translation.”
This difference in response is of interest since, at one level of abstraction, the basic
processes involved in both cases are identical:
S 2f1 → Te11 (2)

Formalism (2) captures the fact that a bimodal ensemble has been disaggregated and
then translated monomodally to produce both Poem 3 and Poem 4. Acknowledging
this, our inclination to classify the two so differently – i.e., the former constitutes a trans-
lation proper; the latter does not – is intriguing, to say the least. Presumably this ortho-
dox stance has its roots in our (logocentric) cultural presuppositions about language and
meaning-making. Although two modes are combined in the original source material,
when Figure 6.1 is considered from a literary perspective, we are more likely to recognise
that we are primarily encountering a text-based poem. Therefore, some kind of priority
is given to the mode of writing, even though it constitutes only part of the multimodal
ensemble. Presumably, if Poem 4 had been titled “Picture 1” instead, then our expecta-
tions would have been quite different. This suggests that the readers/viewers/translators
of such literary works do not treat each mode in the ensemble as being of equal status.
A hierarchical ordering is perceived (whether consciously or subconsciously), and, as
a result, one or more modes can become prioritised. While this hierarchical ordering
might remain implicit, even imperceptible, when the ensemble is merely being read/
viewed by members of a given community, it often becomes glaringly apparent when
it is translated, since the perceived hierarchy partly determines the way in which the
meaning of the source is reconfigured as the associated target (i.e., the translation) is
created. Michel Foucault drew attention to this aspect of multimodal ensembles in the
early 1970s when he reflected upon the relationship between writing and image:
ou bien le texte est réglé par l’image (comme dans ces tableaux où sont
représentés un livre, une inscription, une lettre, le nom d’un personnage); ou
bien l’image est réglée par le texte (comme dans les livres où le dessin vient
achever ce que les mots sont chargés de représenter, comme s’il suivait seule-
ment un chemin plus court). Il est vrai que cette subordination ne demeure
stable que bien rarement car il arrive au texte du livre de n’être que le com-
mentaire de l’image et le parcours successif, par les mots, de ses formes simul-
tanées. Il arrive aussi au tableau d’être dominé par un texte dont il effectue,
plastiquement, toutes les significations.
[e]ither the text is ruled by the image (as in those paintings where a book, an
inscription, a letter, or the name of a person are represented); or else the image
is ruled by the text (as in books where a drawing completes, as if it were merely
taking a short cut, the message that words are charged to represent). True, the
subordination remains stable only very rarely. What happens to the text of the
book is that it becomes merely a commentary on the image, and the linear chan-
nel, through words, of its simultaneous forms; and what happens to the picture is
that it is dominated by a text, all of whose significations it figuratively illustrates.
(Foucault 1973, 39–40; Foucault 1983, 32–3)
With its emphasis on the shifting patterns of dominance and subservience, this
analysis indicates that multiple stages of transposition and meaning (re)constitution
are involved in the process of multimodal (literary) translation. First the author
148 Marcus Tomalin
constitutes a meaning (or meanings) by orchestrating an ensemble (the source),

then the translator/publisher transposes the meaning-making resources, provid-
ing an opportunity for the (re)constituting of the meaning(s) by orchestrating
a related entity (the target, which may or may not be multimodal) – and then
the readers/viewers of the translation (re)constitute the meaning of the target
(and/or the source) as they interpret it.14 Crucially, therefore, the translation of
multimodal ensembles is a process of transposition and meaning (re)constitution
that frequently reveals the modal hierarchy perceived by the translator/publisher.
Presented with multimodal source material, one mode may be prioritised, or fore-
grounded, and this decision may be influenced by an intricate mixture of literary
considerations (e.g., genre, style, form) and subjective personal convictions and
aptitudes (e.g., greater confidence working with writing rather than with images).
Obviously, in cases where an ensemble like Figure 6.1 is converted into a mono-
modal target text like Poem 3, it is clear that one mode has been prioritised over
the other, in a permanent way. The translator’s decision to create a target entity
that has (say) a lower modal dimensionality than its source insinuates a perceived
hierarchical ordering of the modes. However, to indicate the difference between
the processes that produced Poem 3 and Poem 4, the system of notation needs to
be modified to indicate the prioritised mode (and, of course, in some cases, more
than one mode might be prioritised). If w denotes the mode “writing” and if i
denotes the mode “image,” then the two translation processes concerned can use-
fully be formalised as follows:
w
S 2f1 → Te11 (3)
i
S 2f1 → Te11 (4)
Here the second superscript on T in (3) indicates that the dominant mode is writ-
ing, while in (4) it indicates that the dominant mode is image. This formalism
now shows how these processes differ, despite being similar, due to the translator/
publisher’s prioritising of one of the modes encountered in the multimodal source.
In practice, this prioritising may be entirely subjective, or it may arise overtly from
logistical constraints (e.g., copyright restrictions).Whatever the cause, the reduction
in the number of modes from 2 to 1 in (3) and (4) means that these processes can
both be broadly classified as instances of hypointermodal translation.
To explore these ideas further, Figures 6.2 and 6.3 (see next page) both offer
examples in which certain aspects of the multimodal source (i.e., Figure 6.1) have
been translated, while others have been left completely unmodified.
In the case of Figure 6.2, the poem text is entirely unchanged from the source,
but the picture of the dromedary has been converted into a completely different
target image. This has a significant impact on the meaning of the whole, since the
historical and aesthetic connotations of the woodcut image (discussed previously)
have been discarded in favour of an image that possesses colour, texture, and quasi-
three-dimensional shaping. Conversely, in Figure 6.3, the source image remains
FIGURE 6.2 New image FIGURE 6.3 New text
untouched, but the poem text has been translated from French into English. Once
again, in obvious ways, this causes the meaning of the target to differ from that of
the source. Assuming, once again, that Figure 6.2 is encountered by a community
of Metropolitan French readers, while Figure 6.3 is intended for British English
readers, the two translation processes exemplified here can be captured by the fol-
lowing respective formalisms:
i
S 2f1 → Te21 (5)
w
S 2f1 → Te21 (6)
In both of these cases, the multimodal dimensionality of the literary works remains
unchanged, but the perceived prioritising of one mode over the other can be
inferred by considering those components of the multimodal ensemble that the
translator chose to alter. Crucially, it does not follow from this that the priori-
tised mode (or modes) is necessarily perceived by the reader/viewer as being the
dominant or most important one in the resulting ensemble. For instance, a French
reproduction of Richard Pousette-Dart’s ink and watercolour painting Gothic Gar-
den (c.1948) might give the title as Jardin Gothique. In this case, the title would have
been prioritised by the translator, assuming that the reproduction of the painting
was left entirely unchanged, but most French-speaking viewers would probably still
consider the image to be the most important part of the ensemble. In examples (5)
150 Marcus Tomalin
and (6), both modes are retained in both translations, but in Figure 6.2 the image
has been changed by the translator’s activity, while the text in Figure 6.3 has been
rendered into a different form. In these cases too, the translator’s prioritising of
certain modes does not determine a hierarchical ordering on behalf of the reader/
viewer. As ever, those individuals are free to interpret the multimodal ensemble as
they wish, and it is certainly possible (indeed likely) that the relative dominance/
subservience of the modes will shift and change during the process of interpreta-
tion (as Foucault recognised).
5. A musical dimension
The examples considered so far have only included instances where (a subset of)
the modes involved in the source ensemble are also involved in the target. However,
other scenarios should be scrutinised, too. For instance, in 1919 Francis Poulenc
set six of the Bestiaire poems to music, and the first song in the sequence is “Le
dromadaire” (Poulenc 1920). In fact, there are two different versions of the song.
Initially, Poulenc scored it for flute, clarinet, bassoon, string quartet (i.e., two violins,
viola, and cello), and voice, but he subsequently published a version for voice and
piano only. The relationship between either of these versions and Apollinaire’s and
Dufy’s poem-picture merits careful consideration. In Poulenc’s settings, the writ-
ten text of the original has been disaggregated from its associated image (as usual);
a transposition has taken place, and the meaning-material of the former can now
be (re)constituted using the semiotic resources of the mode of music. Immediately,
though, some caution is required, since “music,” like “language,” is potentially an
implausible mode. If the term is used loosely, with reference to all forms of music –
everything from printed scores, to mp3 recordings, to live performances – then
arguably it conflates different materialities and logics that should rightly be kept
separate. Therefore, at the very least, it makes sense to distinguish between written
music and performed music – that is, music as manifest silently by means of a writ-
ten system of notation on a page (e.g., the score of a string quartet) versus music
as manifest in acoustic soundwaves (e.g., an actual performance of a string quartet
using instruments). This is similar to the aforementioned distinction between writ-
ing and speech, and therefore it goes without saying that close conventional associa-
tions of usage connect the semiotic affordances of the two modes of written and
performed music. The violinist in a string quartet produces the required acoustic
soundwaves by reading the notes on the page and interpreting them by means of
physical gestures (e.g., the placing of fingers on the fingerboard; movements of the
bow). To use the terminology introduced earlier, written music and performed
music are proximal modes, since, like speech and writing, they respectively make
available established one-to-one mappings between specific structural units (e.g., an
E-flat major chord, a crotchet note, a minim rest, a plagal cadence). Given this initial
distinction, we may also wish to subdivide both written and performed music into two
submodes – singing and accompaniment – since these are characterised by different
affordances. Singing usually involves linguistic elements that are made available by
the text underlay in the printed score (e.g., words, phrases, sentences), and, in spe-
cific contexts, it can mingle with the mode of speech (hence the existence of tech-
niques such as “sprechgesang” and “sprechstimme”). This suggests that singing and
speech are a proximal mode/submode pairing, since they have an especially close
relationship. By contrast, accompaniment usually involves language-less music (e.g.,
a piano part), which means that speech and accompaniment form a distal mode/
submode pairing.15
Given all of this, if we focus on a particular live performance of Poulenc’s “Le
dromadaire” (and, to simplify matters, specifically the original version, for voice and
chamber orchestra), then the song can be viewed as an audiolinguistic intermodal
translation (i.e., a transduction) of the source material. If we assume that the com-
munity encountering Poulenc’s song is the same French-speaking community that
read “Le Dromadaire” (Figure 6.1), and if p denotes the mode performed music, then
the nature of this translation process can be represented as follows:
p
S 2f1 → T f11 (7)
This example reveals a further limitation in the system of notation being developed
here. More precisely, (7) unhelpfully conceals the fact that the mode involved in T
is not involved in S at all. Once again, though, the system can be easily modified
to indicate when (some of) the modes or submodes in the source and the target
are the same and when they are different. Henceforth, the superscript n[x] will be
added to the symbol for the target, to indicate that x of the n target modes are also
encountered in the source. Using this notation, formulae (1) and (7) can be rewrit-
ten as:
w
S 2f1 → Te11[1] (8)
p
S 2f → T 1f [ 0 ] (9)
1 1
In short, (8) indicates that the source (Figure 6.1) and target (Poem 3) have the
mode of writing in common, while (9) shows that the prioritised mode in the
Poulenc song (i.e., performed music) is not involved at all in the Apollinaire-Dufy
original (though the submode of singing is proximal to speech, which is a latent
mode in Figure 6.1).16 The plus sign (+) associated with the mode p simply indi-
cates that this prioritised mode does not feature in S. As mentioned already, though,
it is often a misleading simplification to designate only one mode in an ensemble as
having been “prioritised,” since different modes may rise to prominence at differ-
ent times as the listener (re-)constitutes the meaning of the target, especially when
the ensemble has temporal duration (as in the case of a performed song). To con-
sider one specific instance, in Poulenc’s setting of Apollinaire’s poem text, there are
10 bars of music before the vocal part begins. Therefore, inevitably, the submode of
accompaniment is necessarily prominent for the duration of those bars, since no other
modes or submodes are involved. However, the hierarchy is potentially readjusted
152 Marcus Tomalin
when the two submodes of singing and the accompaniment occur together, starting
from bar 11.
Although the examples considered so far have only included instances where the
multimodal dimensionality of the target is less than, or equal to, that of the source,
other translation processes can produce target ensembles that have a higher multi-
modal dimensionality. Accordingly, if Figure 6.1 were displayed online in such a
way that an audio recording of the Poulenc song played whenever the website were
accessed, then the readers/viewers/listeners would encounter the French poem
text, the woodcut image, and the musical realisation of the poem text as a single
multimodal ensemble. In that case, the multimodal dimensionality of the target
would be greater than that of the source. While two of the modes involved would
be the same (i.e., writing and image), one would be entirely new (i.e., performed
music). This is captured in formula (10):
p+
S 2f1 → T f31[ 2 ] (10)
This process can be classified as one of hyperintermodal translation, since the mode
of performed music has been added to the existing multimodal ensemble. While
three modes are involved in this example, performed music has been prioritised
by the person who orchestrated the ensemble simply because it has been added to
the existing complex. As ever, though, readers/viewers/listeners may perceive very
different shifting patterns of modal prioritisation as they read, look at, and/or listen
to the webpage. Examples of this kind compel us to confront theoretical issues that
have received far less attention than they deserve. As Simon McKerrell and Lyndon
C. S.Way have recently reminded us “[m]ultimodal analysis and indeed social semi-
otic treatments of music have been theorized primarily upon the static and inter-
related modes of written text and image” (McKerrell and Way 2017, 8). Given this,
any processes that lead to the creation of multimodal ensembles that contain static
and dynamic elements ensure that the focus cannot fall exclusively on atemporal
configurations. This is just one of the reasons why the study of how translation
converts ensembles into ensembles merits serious and sustained attention.
6. Conclusion
In his thought-provoking book about translating Apollinaire, Clive Scott has sought
to shift the focus of translation studies away from a preoccupation with the inter-
pretation of the source text towards an exploration of what he calls “the phe-
nomenology of reading.” He urges us to view reading as “a psycho-physiological
experience of text, as an adventure of consciousness and perception in reading, and
writing that experience, that consciousness and perception, back into the transla-
tion of the [source text].” Crucially, as he points out, an undertaking of this kind
“necessitates the multiplication and extension of the linguistic, graphic, and picto-
rial resources of the translator” (Scott 2015, 18). Viewed in this way, translators are
free to present their experience on the page in any manner they see fit.Words need
not be constrained by conventions of syntax; the linear orderings of the source can
be reworked in tabular arrangements that convey simultaneity and semantic pro-
liferation; typographical forms can be deployed to suggest performative gestures;
photographic fragments can appear as marginal glosses, to supplement the written
text and further destabilise conventional linearities and orders. And the writings
of Apollinaire provide a rich corpus for developing a theoretical translation frame-
work of this kind – perhaps uniquely so.
While the enterprise briefly summarised here could certainly be applied to
translation in all its forms, Scott’s abiding concern with literary translation fre-
quently insinuates itself into his vocabulary. His theory deals with scenarios in
which translation involves “reading” a “text” and creating a new “text,” prompted
by, but not constrained by, the source – and the product of that act of creative
reading is expressed primarily by means of “writing,” even though “image” can
play a significant role, too. The overarching textual and linguistic emphasis of this
formulation may seem a little outmoded in this age of hypertext fiction and ani-
mated poetry, yet literature (broadly conceived) undoubtedly merits astute scrutiny
when multimodal translation is discussed. This is partly because, over the centuries,
numerous writers have sought to destabilise the relationships that exist between the
semiotic resources we have come to think of as modes. In his chapter for this book
(Chapter 5), for instance, Matthew Reynolds reflects upon Dante’s inclusion of a
vertical (or slightly slanting) line as one of the names for God in Paradiso (canto 26,
line 134). As Reynolds demonstrates, this line creates difficulties when considered
in its literary context. For metrical reasons, it must have some kind of (latent) pho-
netic form (e.g., it must be associated with at least one phoneme and one syllable),
yet because the grapheme “I” is not a conventional part of the linguistic systems
of either written or spoken Italian, it is difficult to know which specific form(s) it
should take. This complicates the relationship between the proximal modes writing
and speech. Indeed, the line seems to have more to do with image than anything
else, but how should images be uttered in Italian? And given all of these uncertain-
ties, how should the mystical line be translated?
While Dante’s name for God provides a particularly noteworthy instance, it is
merely one of many. The witty complexities caused by Corporal Trim’s flourishing
of his stick in Laurence Sterne’s Tristram Shandy were alluded to in the Introduction,
and, as discussed there, that little detail places an anomalous semiotic burden on
the modes involved. It is amusing enough that Trim’s physical gesture is described
as being as persuasive on the subject of celibacy as a thousand syllogisms, yet the
absurdity is further heightened by the need for the printed page in the novel to
represent the physical movement involved (Sterne 1767, 17). Once again, therefore,
translation becomes tricky. Do gesture-imitating squiggles mean the same thing
in all languages, or do they need to be re-expressed in different cultural contexts?
Finally, to give just one more example, James Joyce’s Finnegans Wake is a remorse-
lessly experimental novel that causes the modes of speech and writing to mingle
and merge extensively. The unconventional orthographies and pervasive neolo-
gisms enable Joyce to create a bespoke linguistic system that he characterises as
154 Marcus Tomalin
a “cellelleneteutoslavzendlatinsoundscript,” where “-soundscript” is at once “San-

skrit” but also simultaneously a compounding of “sound” + “script” (Joyce 1939,
217).17 In punning coinages such as this, orthography and phonology combine to
create homophonic possibilities that gesture towards the opportunity of concealed
meanings, some of which are revealed when seen, and others only when heard. It is
no surprise, therefore, that translating this particular novel has become one of West-
ern literature’s most arduous and forbidding undertakings (O’Neil 2013).
With full awareness of intriguing case studies such as these, this chapter has
attempted to initiate a conversation concerning some of the ways in which mul-
timodal works of literature can be translated. In particular, the quasi-mathematical
analytical formalism developed in the foregoing sections offers a more precise
framework to facilitate the exploration of the impact the translation process can
have on the multimodal dimensionality of ensembles. Nonetheless, it is important
to ensure that the denotation of the term “translation” does not become so absurdly
all-inclusive as to be effectively meaningless. Accordingly, as a preliminary step, this
chapter has advocated much greater specificity about the exact instances of transpo-
sitions and meaning (re)constitution being considered in a given case. For instance,
translation involving the proximal linguistic modes of speech and writing can be
classified as a different sub-type compared to forms involving (say) music and dance.
Greater clarity concerning the ontology of the various (sub)modes would help
clarify how different types of translation should be categorised. And the analysis
of such things must include nuanced evaluations of the perspectives, strategies, and
preferences adopted by the original author(s), the translator/publisher(s), and the
reader(s)/viewer(s)/hearer(s). Sometimes, as we have seen, the source ensembles are
disaggregated, and certain (sub)modes are discarded; sometimes parts of the ensem-
ble are left completely unmodified, while, on other occasions, new (sub)modes are
added to the target, thereby prompting a reconfiguring of the whole ensemble.
During this process, certain (sub)modes might be prioritised at specific points, and
the act of translation can reveal the nature of these prioritisings. Understanding and
elucidating these complexities is a daunting and delicate business, and, as ever, there
is extensive scope for a multiplicity of contrasting viewpoints and interpretations.
Nonetheless, as this chapter has sought to demonstrate, it is possible to formalise
and quantify certain aspects of the multimodal ensembles involved in these fascinat-
ing translation-based scenarios, and, by so doing, the strategies adopted by transla-
tors of such source material may become more readily apparent.
Notes
1 See Grayling 2000, chapter 4; Wu 2008, chapter 4.
2 Following Kress’s terminology introduced in the first chapter of this book, I will refer
to translation as involving both the “transposition” of the resources for meaning-making
(e.g., a shift from, say, the semiotic affordances of writing to those of dance) and the “(re)
constitution” of meaning (e.g., the meaning(s) created by the audience of the resulting
dance). However, the added parenthesis in my usage of the latter arise from unease con-
cerning the prefix “re-”. It is not clear to me when and how the meaning of the source
becomes sufficiently disassembled to require reconstituting. Also, it is not clear to me that

the meaning of the target is ever merely a recreation of the meaning of the source. Often
it is glaringly obvious that the two are quite distinct – sometimes intentionally so.
3 This closely related to what Matthew Reynolds, in his chapter for this book (Chapter 5),
refers to as “thought-speech” (p. 118) – that is, speech that is unspoken: the speech one
hears silently in one’s mind as one tacitly reads a written text.
4 María Mencía, “The Winnipeg:The Boat of Hope” (2017): http://winnipeg.mariamencia.
com/?lang=es
5 Obviously, this is not the case for all languages since many of them have no written form.
See Michael Cahill and Keren Rice, Developing Orthographies for Unwritten Languages
(Dallas: SIL International, 2014). As is hopefully apparent, I am using “frame” here in the
technical sense – that is, to mark and/or delimit a space, whether material or conceptual,
that determines a domain of meaning at a particular level. See Kress 2010, 149–154.
6 The denotation of “language system” is broader than that of “language.” For instance,
Geoffrey Chaucer’s The Canterbury Tales (1387–1400) has been “translated” into modern
English numerous times (most famously by Nevill Coghill in the 1970s), even though
Middle English and Modern English are, strictly, merely different versions of the same
language. They can, however, be classified as distinct language systems, since they have dif-
ferent pronouns, different morphological conventions, different syntactic rules, different
phoneme sets, and so on.
7 Following well-established conventions, angled brackets (i.e., < >) denote the ortho-
graphical form of a given word, while slant brackets (i.e., //) denote its phonological
form. The French words used in this example are <tristesse> and <chagrin>.
8 The wording here is purposefully alluding to the subtitle of Kress and van Leeuwen
2006: The Grammar of Visual Design. The symbols in slanted brackets give the Inter-
national Phonetic Alphabet transcription of the sentence “I like this poem.” For more
information about ekphrasis, see Heffernan 2004.
9 See the discussion in Kress 2010, 87–88.
10 Kress’s terminology was discussed in the Introduction, on p. 5.
11 Unlike Scott, I will use “S” and “T” (i.e., “Source” and “Target”), rather than “ST’ ” (i.e.,
“Source Text”) and “TT” (Target Text) in this chapter (see Scott 2015, 18). As men-
tioned earlier, since the focus here is strongly upon multimodal works of literature, I am
keen not to allow traditional terminology associated primarily with writing (i.e., “text”)
to dominate unnecessarily.
12 In his chapter in this volume (Chapter 1), Kress refers to this phenomenon as “fore-
grounding” (see pp. 32–33).
13 For a detailed discussion of Apollinaire’s and Dufy’s collaboration, see Read 2013, 17–30.
14 The term “translator/publisher” is used here because, in practice, the work of profes-
sional translators is often determined, and even constrained, in various ways by the pub-
lishers they are working with.
15 Obviously in certain kinds of choral music (especially a capella pieces) there may be a
sung accompaniment, and this may involve linguistic elements. However, such examples
can be reasonably classified as involving the submode of singing exclusively.
16 For further discussion of submodes, see Stöckl 2004.
17 The compound noun refers punningly to various members of the Indo-European lan-
guage family: Celtic, Hellene, Teuton, Slav, Zend (Persian), Latin, and Sanskrit.
References
Apollinaire, Guillaume. 1977. Oeuvres en prose complètes, Vol. 1. Edited by Michel Décaudin.
Gallimard: Paris.
Apollinaire, Guillaume, and Raoul Dufy. 1911. Le Bestiaire ou Cortège d’Orphée. Paris:
Deplanche.
156 Marcus Tomalin
Bindman, David. 2009. William Blake: The Complete Illuminated Books. London: Thames and
Hudson.
Bracquemond, Félix. 1897. Étude sur la gravure sur bois et la lithographie. Paris: Henri Beraldi.
Cahill, Michael, and Keren Rice. 2014. Developing Orthographies for Unwritten Languages. Dal-
las: SIL International.
Cohen, Jane R. 1980. Charles Dickens and his Original Illustrators. Ohio: Ohio State University
Press.
Derrida, Jacques. 2002. “Faith and Knowledge: The Two Sources of ‘Religion’ at the Limits
of Reason Alone.” In Acts of Religion, edited by Gil Anidjar, 40–101. London and New
York: Routledge.
Erdman, David V., ed. 1974. The Illuminated Blake: All of William Blake’s Illuminated Works with
a Plate-by-Plate Commentary. Garden City, NY: Anchor Press/Doubleday.
Foucault, Michel. 1973. Ceci n’est pas une pipe. Montpellier: Fata Morgana.
———. 1983. This Is Not A Pipe. Translated by James Darkness. Berkeley: University of
California Press.
Gibbons, Alison. 2012. “Multimodal Literature and Experimentation.” In Routledge Com-
panion to Experimental Literature, edited by Joe Bray, Alison Gibbons, and Brian McHale,
Grayling, A. C. 2000. The Quarrel of the Age: The Life and Times of William Hazlitt. London:
Weidenfeld and Nicholson.
Hazlitt,William. (1821) 1930.“On the Pleasure of Painting.” In The Complete Works of William
Hazlitt, Table-talk,Vol. 8, edited by Percival Presland Howe, 1–18. London and Toronto: J.
M. Dent and Sons.
———. (1828) 1933. “On the Causes of Popular Opinion.” In The Complete Works of William
Hazlitt, Uncollected Essays,Vol. 17, edited by Percival Presland Howe, 308–13. London and
Toronto: J. M. Dent and Sons.
Heffernan, James W. 2004. Museum of Words: The Poetics of Ekphrasis from Homer to Ashbery.
Chicago and London: The University of Chicago Press.
Hildebrandt, Bill. 1995. Calligraphic Flourishing: A new Approach to an Ancient Art. Boston:
David R. Godine.
Joyce, James. 1939. Finnegans Wake. London: Faber and Faber.
Kennedy, X. J., ed. and trans. 2011. Guillaume Apollinaire,The Bestiary, or Procession of Orpheus.
Baltimore: The Johns Hopkins University Press.
Kress, Gunther, Carey Jewitt, Jon Ogborn, and Charalampos Tsatsarelis. 2001. Multimodal
Teaching and Learning:The Rhetorics of the Science Classroom. London: Bloomsbury.
Lee,Tong-King. 2012. “Performing Multimodality: Literary Translation, Intersemioticity and
Technology.” Perspectives: Studies in Translatology 21 (2): 241–56.
Lupfer, Earl A. 2003. Ornate Pictorial Calligraphy: Instructions and Over 150 Examples. London:
Dover.
May, Will, ed. 2015. The Collected Poems and Drawings of Stevie Smith. London: Faber and
Faber.
McKerrell, Simon, and Lyndon C. S. Way. 2017. “Understanding Music as Multimodal Dis-
course.” In Music as Multimodal Discourse: Semiotics, Power, and Protest, edited by Lyndon
C. S. Way and Simon McKerrell, 1–20. New York and London: Bloomsbury.
Mencía, María. 2017. “El Winnipeg: El barco de la esperanza.” http://winnipeg.mariamen-
cia.com/?lang=es.
Nørgaard, Nina. 2009. “The Semiotics of Typography in Literary Texts: A Multimodal

Approach.” Orbis Litterarum 64 (2): 141–60.
O’Neil, Patrick. 2013. Impossible Joyce: Finnegans Wakes. Toronto, Buffalo and London: Uni-
versity of Toronto Press.
Poulenc, Francis. 1920. Le Bestiaire ou cortège d’Orphée, poëmes de Guillaume Apollinaire. Paris:
Max Eschig.
Read, Peter. 2013. “Dufy Draws Apollinaire: Illustration and Commemoration in the livre
d’artiste.” In The Art Book Tradition in Twentieth-Century Europe, edited by Kathryn Brown,
Scott, Clive. 2015. Translating Apollinaire. Exeter: University of Exeter Press.
Serafini, Frank, and Jennifer Culson. 2012. “Typography as a Semiotic Resource.” Journal of
Visual Literacy 31 (2): 1–16.
Shattuck, Roger, ed. and trans. 1971. Selected Writings of Guillaume Apollinaire. New York:
New Directions Books.
Sterne, Laurence. 1767. The Life and Opinions of Tristram Shandy, Gentleman,Vol. 9.York and
London: T. Becket and P. A. Dehondt.
spectives on Multimodality, edited by Eija Ventola, Cassily Charles, and Martin Kaltenbacher,
9–30. Amsterdam: John Benjamins.
van Leuwen, Theo. 2006. “Towards a Semiotics of Typography.” Information Design Journal 14
(2): 139–55.
Venuti, Lawrence, ed. 2000. The Translation Studies Reader, 3rd ed. London and New York:
Routledge.
Wu, Duncan. 2008. William Hazlitt:The First Modern Man. Oxford: Oxford University Press.
7
TRANSLATIONS BETWEEN
MUSIC AND DANCE
Analysing the choreomusical gestural
interplay in twentieth-and
twenty-first-century dance works
Helen Julia Minors
1. Translation and multimodality: languages, senses,

and cultures
Multimodal translation is a highly relevant topic today, in a society where we strive
to accept, to learn from, and to engage positively with diversity. The temporal arts
of music and dance enable us to share a somatic experience. Three key reasons for
the needs to consider translation in the context of the performing arts (especially
music and dance) are that:
1 The place of choreomusical studies within institutions is reducing and, as

Stephanie Jordan highlights in her recent chapter, it looks as though the
“drawbridge is all set to be raised,” further segregating the arts (Jordan, forth-
coming). A reassessment of how multimodal translation is a benefit to both
creative practice and critical reflection is much needed in order to revitalise
this situation.
2 There has been a cultural turn in the arts whereby we recognise that we expe-
rience and interpret that experience through metaphor. In other words, in
collaborating to co-create music-dance works, the artist experiences things in
the terms they understand: musicians apply musical understanding to dance
and vice versa (Minors 2012a, 2012b).
3 The interaction of different art forms, different cultures, different peoples, is a
necessary and important experience which draws out meaning “from dynamic
interactions” (Cook 2013, v). The dynamic nature of lived exchanges, I claim,
relies on a process of translation. The dialogue between, the exchange across,
and the transfer among the artists and their artistic media is a process whereby
each strives to understand the other. All interpretative and creative acts have a
level of translation (Minors 2013a).
Translations between music and dance 159
Significantly, analogies are created to language. Meaning is constructed (though not

only via language but also through the responsive acts of each artistic media). Bar-
riers are present which resist the formation of understanding, as for dancers who
do not have a musical background, or for musicians who do not have a movement
background, there is a context with which they are unfamiliar – hence the need
for translation. The different contexts highlight the different languages (metaphori-
cally speaking), per se, and so too the different ways of understanding. This issue is
of central concern to this chapter as artists and audiences are able to understand in
multiple ways, through their different experiential (translation) contexts. The arts
are understood through and by language, and that language is specific to the art form,
though equivalences exist: for example, dynamics exist in both music and dance,
but in the former we refer to volume changes, whereas in the latter we refer to
muscle intensity. As Jordan remarks, there is both a problem of language (2000) and
therefore a sense that one needs to learn an “alien vocabulary” in order to engage in
multimodal discussion (Jordan forthcoming). As most choreomusical scholars note,
the different notational systems represent one key barrier to sharing the process of
creation. Other dance scholars also remark on dance in such a way as to refer by
analogy to language: that dance is a “culturally codified and meaningful human
action” (Buckland 2007, 187).
Beyond verbal and textual language though, we also have a somatic experience
of live performance arts (akin to language issued via braille), and as such the arts are
understood through and by our senses: like spoken discourse, we hear the music (or
verbal language) and we see the dance (physical gesture). The audiovisual intercon-
nections are mapped cognitively in our total experience of the work as we read, hear,
and feel the gestures: those gestures are the moments that are understood as signifi-
cant in our personal interpretative translation of the work (Minors 2012b; Zbikowski
2012a). As Peter Dayan remarks in his analysis of the inter-art aesthetic of early
twentieth-century works: “There can be no direct translation and no unproblematic
collaboration” (Dayan 2011, 3). The problem referred to by Jordan (in relation to
language analogies) and Dayan (in relation to the transfer of sense across and between
the arts) is in fact the kernel of creative exchange, as Eisenstein observed in relation
to early film: “this relation is not one of similarity, but, as a rule, one of question and
answer, affirmation and negation, appearance and essence” (in Albright 2004, 93).
Jordan, in her recent work on choreomusicology, asserts a significant point which
reminds us that we should look not only to difference but also to the shared points:
Whilst we might still be able to trace the separate development of the two
media, these two sensory planes now meet to affect each other and to create
a new identity from their meeting. The old terms of congruence and non-
congruence, similarity and difference, now seem inadequate.
(Jordan forthcoming)
Bringing this back to the field of multimodal translation, there has been recogni-
tion that language and music offer particular forms of understanding: “The battle
160 Helen Julia Minors
between ‘logocentrism’ and ‘musicocentrism’ has raged for centuries, and is far
from over” (Apter and Herman 2016, 6). I would add here that the particular
context and perspective from which the artist comes offers their own battle,
which might in fact be “dancentrism,” where another way of experiencing and
communicating is dominant, other than music and verbal-textual language. As
Susan Bassnett claimed, translation exists within a “multilingual and multicultural
context” (2014, 1).
Drawing on intercultural arts research might offer new insights into consider-
ing how and where music and dance meet, how they function together, what they
produce, and how they produce what they do together as a single work, created
from the catenation of different elements. In considering these modes of expression
and communication, we ask how we are able to transfer sense across boundaries:
they may exist within and across art, language, and cultural domains. The act of
translation requires the translator to be creative. Venuti’s conclusion that “no act
of interpretation can be definitive” asserts the need for creativity, emphasising the
individuality of the translator (Venuti 1998, 46); in the context of choreomusical
studies, I highlight that the translator is not invisible, but rather poignantly visible
and audible. The arts are understood through and by culture. In drawing on transla-
tion here to reassess the interrelationship of music and dance, I, like Pamela Burnard
et al. (2018), use translation to interrogate my field in order to ask what dance-
music works are and how they express themselves. “[I]ntercultural translations are
co-constructed through collaboration” (Burnard et al. 2018, 232). The issue of
collaboration is central to multimodal creativity and so it is central to multimodal
translation. All music-dance works discussed here are co-created and all rely on an
exchange of understanding in collaboration and performance to achieve the result-
ing work.
The cultural context of any work and the performance of the creative act
inform meaning construction. A particular identifiable movement, such as an ara-
besque, has particular cultural references. Notably, the French name of this dance
movement means “in an Arabic style.” It is usually performed on point, with a
raised leg, at 45 or 90 degrees, and is known as à demi hauteur or à la hauteur. The
meaning of physical movement is reliant on its context. In music, an arabesque is
likewise reflective of Arabic styles: the highly embellished style of Arabic architec-
ture is reflected by highly embellished melodies. Whether meaning delivery has
been issued by verbal, sonic, or physical gestures, each is created within a cultural
context, or across cultural contexts. Similarly to Dayan, Gunther Kress, in discussing
multimodal communication, asserts that:
I assume that “translations” across modes within a culture are both possible
and hugely difficult [but] that translations across cultures, whether the same
mode ... or across different modes are also possible, though always achieved
with enormously difficult selection; at a considerable level of generality; and
inevitably with significant changes in meaning.
(Kress 2010, 10)
I identify therefore three key approaches to the translation of music and dance
as communicative active media: senses, languages, and cultures. The problematic
nature of artistic collaboration, language delivery, meaning transfer, and expressive
exchange is necessary to all forms of multimodal artistic collaboration and in fact
the reason why translation in this context deserves assessment. These three catego-
ries of meaning exchange were also identified as a result of the Arts and Humanities
Research Council’s funded research project “Translating Music.”1 The difference in
this chapter, however, is that we are considering the somatic experience of music-
dance works and the sonic-visual dimension of such multimodal works.
Kress defines mode as:
a socially shaped and culturally given semiotic resource for making mean-
ing. Image, writing, layout, music, gesture, speech, moving image, soundtrack and 3D
objects are examples of modes used in representation and communication.
Phenomena and objects which are the product of social work have meaning
in their cultural environments.
(Kress 2010, 79)
The list of modes here is interesting for this discussion, as dance crosses all of these
modes. It exists in and through gesture and image, with and between musics, on
screen, and as a multidimensional body, and it does this even if, as Matthew Reyn-
olds suggests in his chapter for this volume (pp. 118–19), it is not always clear
exactly where one mode ends and the next one begins. The capacity for dance to
function across all of these modes asserts its role as a medium (as an artistic medium
which functions within the collaborative whole, which expresses the design mani-
fest in corporeal medium) to be used in this translation process (Kress and van
Leeuwen 2001, 6). Likewise, music exists alongside speech and text, embedded
with gesture, and functions on screen. In response to Kress then, I assert dance and
music as modes that can be involved in translation: “What a community decides to
regard and use as mode is mode” (Kress 2010, 87). The three categories (sense, lan-
guage, culture) provide ways in which we might assess the translation process that
takes place between music and dance. The notion of translation therefore invites
us to explore the ways in which music and dance speak to each other, the ways
they adopt each other’s processes or contrast the other in a way that formulates an
active dialogue across the arts, within a specific performance space, for a specific
performance event.
In order to approach this interdisciplinary field, I draw then on a background
in three areas: (i) music and dance studies, often referred to as choreomusicology
(Minors 2006, 2009, 2012a, 2012b); (ii) music and translation studies (Minors 2013a,
2019); and (iii) intercultural arts research (Minors 2016). Drawing on these varied
fields, I attempt to tackle the concern raised by Kress that “ ‘translations’ across
modes” (2010, 10), across multiple cultures, might dilute the observations. Speaking
from within multiple fields (though this is not exhaustive) enables me to suggest
three categories to the translation of music and dance, which do exist between
similarity and difference, and to speak from the point of liveness – the lived experi-
ence of the work whether from the perspective of the creative collaborative artist
or from the point of felt, lived experience of interpreting the work in performance.
Choreomusical studies would benefit from exploring the concept of translation,
in order to further assess how meaning is transferred between these artistic media.
This approach looks at translation as a way to understand music-dance work. It
seeks also to understand the dialogue and interaction between music-dance and
to explore how considering the three categories (senses, languages, cultures) might
offer new insights for choreomusical studies. The chapter explores first translation
in the context of choreomusical studies, before using three case studies to illus-
trate how translation can offer new perspectives to interpreting music-dance works.
How might translation facilitate new understanding of the creative and receptive
dimensions of music-dance works? Can it reveal something of the creative choices
made by composers and choreographers working in collaboration?
2. New approaches to music-dance translation

For choreomusicology, the temporal nature of these movement arts, which are dis-
creet due to their audiovisual nature, requires us to theorise the audiovisual transfer,
but in order to do so acknowledgement of the problems of language must be raised
again. As Şebnem Susam-Saraeva (2008, 189) notes in her seminal study on music
and translation: “The mere mention of translation within the context of music opens
a huge can of worms for many researchers and practitioners.” The can is expanded
when we consider dance as well as music. Translation is of relevance to many musical
texts (Minors 2013a), and in asking how “music speaks” (Albright 2009), it is neces-
sary to ask who it speaks to, how it speaks, and what it says to dance and music in par-
ticular. Despite the difficulties in translating across media, the examples in this chapter
have a shared common practice in that I select balletic examples in which there is
a culture of music-dance works produced for the theatre. As such, the domains of
music and dance are shaped by their location within the genre of ballet, and more
specifically within a Western twentieth-century balletic practice.
Collaborative music-dance works are complex due to the fundamental differ-
ences between how we produce and experience these arts. There is a sensory divide,
but it is not an impenetrable wall. We assume difference due partly to the name we
give these arts, but some cultures have one word to embrace them both. Theorists
recognise the overlap of the many arts: Daniel Albright’s pivotal work on panaes-
thetics proposes that the arts are not separate and that in fact “All art is inscribed on
the body” (2014, 281). Moreover, in establishing The Principles of Art (1923), Robin
Collingwood claims that “the dance is the mother of all languages” (in Copeland
1983, 371). The reciprocal nature of these arts is highlighted when Albright notes
that “A dance is a response to music, and a making of its own music” (2014, 284).
In essence, Albright asserts the well-cited remark of Walter Pater that “all art aspires
toward the condition of music” (Herzog 1996, 122). Whatever the condition (cul-
tural context, performance location, or message), the arts can function in the terms
of the other, adopting processes and equivalences in order to exchange gestures.

Although the arts form an interdependent relationship and can transfer content,
style, and gestures, they cannot “be physically shifted, translated into the medium
of another” (Dayan 2011, 21). The body has however always moved and predates
language and formal semantic structures: in many ways the audiovisual cognitive
mapping we enact when experiencing multimodal works could be said to predate
and so underpin language, and therefore be central to all forms of translation studies.
It is necessary to question: how do the aural and visual dimensions react to one
another? Translation offers a way to look at not only an exchange of content but
also the ways in which artists respond to each other by the choices they make.What
choices are made by composers writing for dance, and particularly for ballet dur-
ing the twentieth century and beyond? What choices are made by choreographers
in setting dance to music (whether they chose pre-composed music or work in
real-time collaboration)? Indeed, how does an act of collaboration between music
and dance rely on a process of translation? Both music and dance have their own
vocabulary and syntax (metaphorically speaking) in the ways in which they put their
art together: it is a sense of musical metre, or a sense of dance metre (as one exam-
ple), which is different from verbal language. Henrietta Bannerman observes that
this language is “meta-kinetic rather than metalinguistic” (2014, 66). The difference
between textual language and music-dance language is grounded in the experience
of movement and non-semantic musical sound. This is not to say that music cannot
be referential in its meaning, but it cannot represent specific words or the nuances
of sentences. Fundamentally, much of the equivalence is drawn out in the experi-
ence of time, and therefore of meter, rhythm, phrasing, and tempo.
Reacting to the cultural turn in audiovisual translation (Apter and Herman
2016) and to the continued concern with cultural musicology, this chapter requires
this theoretical overview of translation in this particular music-dance context. The
concept of musical gesture (Minors 2012b) as an analytical approach to exploring
how music and dance form meaningful relationships is central. Gestural analysis
is proposed within the realm of conceptual mapping, where the integration and
blending of domains is essential. We are aware of the separate art forms, their con-
tent and structure, but we experience the work as a whole, during the moment. As
Massimiliano Locanto (2018, 38) remarked, in response to my previous work on
music-dance gesture (Minors 2012b), the many networks form from the different
conceptual domains, and the mapping of those domains form a new perspective of
the interplay and communication between music and dance.
Developing a model to start to explore translation across music and dance,
I expand a model I proposed elsewhere (Minors 2012b, 171), in which music and
dance spaces are shown in a Venn diagram, to illustrate the blended space of the
two domains. This model illustrates three categories through which the mapping
of music and dance can be explored in terms of translation (senses, languages, and
cultures) (see Figure 7.1). The mapping might be produced by the creative art-
ists during collaboration and by theorists and by spectators in receiving the work.
The translation perspective (be that an exploration of language, sense, or culture)
FIGURE 7.1
Mapping music-dance interdependency read through the translation of
languages, senses, and cultures
feeds how we read the mapping of music and dance. It informs the ways in which
we might interrogate those relations, ensuring that we go beyond the traditional
notions of similarity and difference (as outlined earlier). The experience is situated
within a particular context (era, venue, country). Notably though, if translation
enables the creative artists and theorists to interrogate the relationships of music and
dance as interdependent, then the model needs to recognise that there is a continual
reciprocal exchange and transfer of content during the creative process and during
the performance of the work, in which each art modifies the other. What we hear
is modified by what we see, and so, what we see changes how we hear the music.
In the field of intercultural research, similar questions are now being raised as
well: Burnard et al. asks how both music and musicking might “function as tech-
nologies of translation that allow us to understand and conceptualize practices of
diverse musical creativities” (2018, 233). This approach considers thus the partici-
pation of all involved in the works’ creation and reception. It extends to consider
then, beyond textual-verbal language, how music is translated and how much it can
translate; how dance is translated and how dance can translate.
3. Translation and the gestural arts of music and dance

Why is the notion of translation significant to music-dance works and to the grow-
ing field of choreomusicology in particular? Theorising translation in this context
aims to illustrate its significance as a new approach to choreomusicology and to

refine the fact that music translation is not only the translation of various musi-
cal texts (such as programme notes, lyrics, or transcriptions), but it also claims the
capacity for dance and music themselves to translate the content of other artistic
media. Through the following three case studies, I ask: can a process of translation
be read when choreographers work with pre-composed music? Or on the contrary,
how do composers and choreographers utilise a process of translation and transfer
when they collaborate in forming new music-dance works? Considering the crea-
tive process, we then turn to the resulting work to ask: how do we read those emer-
gent meanings between the audiovisual elements? A caveat is required: translation as
a term is in danger of becoming overused and might lose its usefulness unless it is
refined. The term in music overlaps with notions of meaning transfer, artistic adap-
tation, arrangement, covers, rewriting, transcriptions, and transformations (Minors
2013a). As a process, translation enables researchers to question the choices of the
creative collaborators, to chart the developments in the field, and to find ways to
analyse audiovisual works. As such, it is interrogative in nature. It requires the map-
ping of domains. But it does not exchange one note for a movement (or a word for
another): there are no claims for the exactitude of meaning or syntax.
What are the implications of translation for music-dance works? If a choreogra-
pher, for example, tries to translate the content of a musical score into dance, might
the result only mimic music through elemental equivalence (rhythmic mimicry,
for example)? Or, do those equivalences, which are read between the audio and
visual domains, produce a greater variety of interrelationships which reveal a state
of interdependence (namely a state of co-existence where each relies on the other,
but each does retain its own autonomy within the whole ephemeral experience)?
Notably, I focus on narrative balletic works to refine the scope of the exam-
ple. I utilise three case studies to illustrate the three approaches to the translation of
music and dance outlined earlier: (i) of a composer and choreographer collaborating
together, as well as an example of the same choreographer selecting a pre-composed
score, to illustrate the translation of sense across the audiovisual divide (sensory trans-
lation), which relates to Kress’s notion of “moving meaning” in translation (2010,
124); (ii) of the use of music visualisation, to unpack how the language translation of
music and dance can be understood as translating one another, as an intersemiotic
translation seeking language translation by analogy, ultimately seeking to visualise the
music, or to audioise the dance, which relates to the subtype of translation, transduc-
tion, denoted by Kress, whereby “meaning-material [is moved] from one mode to
another” (2010, 125); and (iii) of an (inter)cultural translation of a score interpreted
twice, a century apart, in order to show the re-appropriation of music and dance into
a new cultural context, whereby new meanings are imposed but some meanings are
also carried forward. As stated earlier, a cross-reference to Kress is possible, as the “re-
ordering of the elements” occurs “across cultures in the same mode” and so represents
a subtype of translation he labels “transformation” (2010, 129).
The examples I have chosen to draw on are specific to the need for further
research and discussion across music and dance: the initial two case studies are
chosen for their popularity in the field of choreomusical studies and the qual-
ity of the music and choreography. By taking examples that have been previously
researched and analysed, I aim to bring the new lens of translation to them to argue
that the process of interpretative translation is one that can facilitate better under-
standing and engagement across the disciplines. The final case study is chosen to
acknowledge that there are meanings beyond the collaboration which emerge to
engage with contemporaneous culture, politics, and society. The example illustrates
a re-interpretation and modernised adaptation which is grounded very much in
cultural history and not only lends itself well to the cultural turn in dance and
music research, but enables me to iterate the importance of considering translation
as an appropriate and engaging tool by and through which to explore music-dance
works. A deliberate choice has been made, therefore, to select case studies which are
already well trodden from a variety of perspectives, to add the concept of translation
to these seminal works and to offer a new perspective to choreomusical studies.
4. Sensory translation as an approach

As mentioned briefly in the Introduction, Vaslav Nijinsky was a lead dancer of
the famous Ballets Russes, led by Serge Diaghilev, and he was commissioned by
Diaghilev to create new choreography for two works: the first, Claude Debussy’s
already successful Prélude à l’après-midi d’un faune, for their 1912 season, and the sec-
ond work, Igor Stravinsky’s Le sacre du printemps, for their 1913 one. Debussy’s work
had been composed in 1894 in response to Stéphane Mallarmé’s symbolist poem
of the same name, while Stravinsky’s work was composed in response to his own
vision of a sacrificial rite in which he saw a young virgin danced herself to death.
Debussy’s choice is particular: the music, like the poem, is arguably symbolist,
offering 110 bars to respond to the 110 lines of the prose poem (Wenk 1976, 161);
it also offers the repeated melody once on a different tone centre of A (bar 23),
as the poem likewise refers to tuning to the A (line 37). There is direct music-
text correlation. The symbol of the panpipes is shown through the flute solo
(for example, at bars 94–99), and as two pipes are referred to (line 20); the music
also uses two unison flutes earlier in the piece (which has no significant musical
impact whatsoever, so seems certain to respond to the text, moving from divid-
ing the motive between them to playing together from the climax of the phrase,
bars 26–29). Likewise, the sexual chase is illustrated through the imitating horns
(known for the illustration in nineteenth-century opera of the hunt). Importantly
though is the sense of movement felt in distinctive ways through and between
music and dance. This sense of movement in the moment is integral to a sensory
translation, as this relies on a somatic experience and an experience of the ephem-
eral qualities of music.
I claim this example as one of sensory translation between music and dance
(and one could argue by extension to the ekphratic source of Debussy’s music, the
poem) because the shared response to the poem and the shared emergent mean-
ings are developed through different artistic processes, but the end sharing of sense
is coherent. The blend of the senses is significant. The poem places the idea of
dreams in opposition to reality, and the solo faun who lusts after the mythical
nymphs seems to be across both states of existence. The music conjures up this
variable state of existence through two key means: (i) the solo melody avoids all
forms of conventional tonal harmony, avoiding establishing the key and avoiding
any closed cadences – the music seems suspended around the tone centre (C#); and
(ii) the opening statement (flute solo, bars 1–4) is repeated many times, with five
statements in the first section (bars 1–29) – Section A1 uses modified material but
includes the opening modified statement twice before further developing material
(bars 31–36). A nocturne section follows. The reprise of the original statement
occurs at bars 79–82, before four further reprisal statements. The return of material
could be understood as memory.
Importantly though, the changes to rhythm, phrasing, and to the length of the
statements modifies the experience of time. Debussy creates a statement which is
languid and does not have regular beats, but pitches duple and triple patterns side by
side within a phrase, offering an improvisatory feel to the statement. The sustained
opening note is held and sustained during its third statement (bar 21) and follow-
ing. The reprise statement uses a new time signature, focusing on duple meter and
regular beat patterns (bars 79–82, see Figure 7.2). Reprise statement 3 is similar to
the opening but now in 4/4 rather than 9/8 time (bars 94–99) (see Figure 7.3a bar 1
and Figure 7.3b bar 94). This is significant as Debussy makes great efforts to distort
our experience of time, but to include constant musical motion. Nijinsky’s response
to both Mallarmé’s poem and Debussy’s music ensures to retain the uncertain state
FIGURE 7.2 Bars 79–82
FIGURE 7.3A Bar 1
FIGURE 7.3B Bar 94

of whether the faun (of the title) experiences the situation, or whether in fact he
dreams it. This deliberate doubtful state of existence is advanced in how Nijinsky
choreographs movement to Debussy’s score. Whereas Debussy uses constant rhyth-
mic movement with changeable duple-triple rhythms, Nijinsky offers a static form
of dance whereby the faun poses, lying on his side atop a mound, for the opening
statement. The physical stasis gives a sense of motionless to the music and in fact
enables a listener to draw out the similarity of the music, not in stasis, but in the
repeated aspects of the opening statement (the descent and rise to and from the
same note, the exact repetition of bar 1 immediately in bar 2) (Nijinsky 2013, time
code 00:00–02:07). The fact that the faun also holds a pipe and mimes playing it
unifies the connections between the audiovisual domains and tricks the listener into
imagining the faun playing this melody; as such the theme becomes associated with
the faun and its reprisal therefore with his changing states of being.
The equivalence of music and dance languages is considered in order to use
their difference to project a particular sensory experience of motionless and a
dream-like state. The audience is invited to “acknowledge the autonomy of each”
(Duerden 2007, 77) art form due to the different sensory mode of delivery, but
this does not result in difference only. The contrasting artistic features produce
a consistent emergent meaning through their mapping. The difference in pro-
cess and production are able to result in similarity of meaning. Let’s test this
further: the pedestrian walking movements of the nymphs, moving in a two-
dimensional-like side-on presentation, much like Egyptian hieroglyphics, adds
to the sense of stasis, notably at their presentation with the fifth statement of the
theme (bar 26–30; Nijinsky 2013, 02:53–03:30). The lead nymph slowly holds
out her scarf and places it on the floor, further adding to the concrete motionless-
ness via the object being relinquished. Notable too is Nijinsky’s response to the
repeated flute turn motive (bar 27) in the angular, stuttering movements of the
faun’s head. The minute physical movement accompanies a complex alternation
of the two flutes: his contrasting movements give a further sense of smoothness
to the musical line. Nonetheless, again the audiovisual difference comes together,
meeting in a blended space, to offer a sense of unity where the faun and nymphs
lay eyes on each other. The uneasy transfer between the audiovisual elements is
necessary to convey the dream-reality binary. That dance has a “refusal to coop-
erate” (Albright 2000, 7) with the music, and this is central to using the different
elemental features of these art forms to combine to issue a single narrative. Fur-
thermore, there is also an analogy between the artistic media of music and poetry:
dance offers unity of message at the first point where two flutes play in union.
The levels of complexity one can read when considering how dance might trans-
late the sensory experience of the music affords the fact that equivalence of unity
may be used, despite the difference of rhythmic movement. Only together do
music and dance formulate the resulting audiovisual image.
The notion of mutual implication proposed by Claudia Gorbman (1980, 189),
though rather old, stands fast in research which explores the transfer of sense and the
exchange of ideas across audiovisual media.Whereas traditional notions of translation
rely on words through speech and text, artistic translation relies in part on the sensory
experience, various ways of “speaking” or issuing content, and a variety of ways of
documenting that content (for example, the score, audio recording, dance notation,
video recording). As such, Michel Chion’s approach to film theory, noting that a
“transsensorial perception” is necessary (Chion 1990, 137), is highly pertinent to
this current context, too. The constant exchange between media enables the further
emphasis of the dream versus reality as the spectator is drawn to question from whose
perspective we are viewing this activity. Even the faun asks:“Aimai-je un rêve?” [“Was
it a dream I loved?”] (Mallarmé, in Austin 1970, 23, line 5). Further emphasis that
a sensory translation might offer more to the work is shown in Mallarmé’s own
response to Debussy’s music, when he claimed that the music “prolongs the emotion
of my poem, and sets its score more vividly than color” (Austin 1970, 13).
Nijinsky’s approach to working with a composer was of course different: in work-
ing with Igor Stravinsky on Le sacre du printemps, he worked with creating new met-
rical structures and a very different form of choreography again, which disregarded
conventional balletic gestures, preferring instead knocked knees, flat feet, angular
movements, and a distortion of the body (Nijinsky 1987). Stravinsky’s score was cre-
ated to his own vision of a sacrificial rite where a virgin would dance herself to
death (a tale often repeated by Stravinsky in interviews and documentaries). Nijinsky
seems to have felt the multiple characters of the scene, in which young girls, sages,
and senior figures compete in their role in the ritual. Responding to the friction in
the story, and to the many musical textural layers, we see for the first time a curtain
lift on the second statement of an extremely high bassoon solo (which is based on
Lithuanian folk music, Stravinsky 1947, rehearsal figure 12), not a dance soloist, duet,
or corps de ballet, but on a scene where there are distinct groups, with a circle of men
back left, an old woman alone and knock-kneed and bent over at the front, with a
circle of girls kneeling over at the centre and another group at the back right, with
a further group crouching at the front right (Stravinsky 2008, 02:38). Stravinsky’s
music is grounded in metrical structures, which use regular rhythms off set with
syncopation. Here the 2/4 metre offers four regular quaver beats in a bar, but the
accents disrupt the flow. Nijinsky, seeking no doubt to enforce the primal sense of
the story and to emphasise the primitive nature of this rhythmical accented music,
matches the arm movements of the first male group to the accents of the music: arms
rise and fall only on the accents. The feet all the while move with the four quavers as
though the accent were on the first beat of the bar. The choreographic approach, in
matching physical movement to musical movement, is in complete opposition to his
approach to choreographing Debussy’s music. The physical matching of the different
rhythmic layers of the score has the effect of intensifying how the accents are felt and
seen. Both seem stronger together than alone. Interestingly, “Stravinsky conceives of
language as normally not only expressive, but invasive, claiming the right to translate
into its own logic the functioning of all media, including music” (Dayan 2011, 125).
Stravinsky spoke of music-dance relations: what is striking is the importance for him
of the rhythmic correspondence, as he refers to the correspondence being “measured
to the musical unit” (Dayan 2011, 17). This level of invasiveness seems to impact on
the collaborative relationship when we look to the rhythmical structures used in

Nijinsky’s choreography.
Both of these examples illustrate some of the early seminal changes in twentieth-
century narrative ballet and in musical composition, as the old traditional notions of
form, metre, and harmony were being challenged. The sensory translation propa-
gates that challenge to their own disciplines while also superimposing them for the
spectator. The examples also illustrate how a choreographer, who is new to his role,
responds to and translates these composers’ revolutionary approaches to texture,
timbre, rhythm, and thematic development.
5. Language translation as analogy

Language translation is most keenly illustrated through examples of music visualisa-
tion, where the elemental features and syntax of music and dance are read by the
choreographer as analogous to language. The elemental features of music, such
as rhythm and pitch, are read as equivalences to dance rhythm and height (the
metaphorical equivalent of pitch). The approach to translate the content of one
art form similarly into another has caused some concern over mickey-mousing, a
state whereby the different domains aim to mimic one another. Music visualisa-
tion requires “analogical thinking,” whereby there is direct correlation between the
elements of each domain (Zbikowski 2018, 60). As Paul Hodgins identified when
he established the term choreomusical, “the degree of music visualization in dance
depends to a large extent upon both the choreographer’s own predilections and the
translatability into movement gesture of the musical topology involved” (1992, 13).
This process starts with identifying similarity in terms of those elemental features,
but it develops to embrace complexity as we map the domains to form structural
correlations offering more than the sum of their parts.
Carroll and Moore remarked that “the dance movement functions as a transla-
tion from one medium to another – from the musical movement impulse into
flesh and blood movement” (2008, 16). They claim that the most intriguing music-
dance relations are those whereby there is “embodied translation of the musical
motion impulse” (Carroll and Moore 2008, 17). An early proponent of music visu-
alisation, whereby music is the catalyst for dance, was Doris Humphrey. Her Air for a
G String (1934) shows that pedestrian movements (walking, steps, waves) move with
the same rhythmic motion as the music. The lower levels of the body correlate to
the bass line of the music while the hands and upper body correlate to the melody
(Humphrey 1934, 00:00–00:26). Humphrey was very aware of the audiovisual pro-
cessing of the audience and the role of translating the audiovisual experience. She
warned that “not only is the eye faster, but, in a contest with the ear, will invariably
take precedence” (Jordan 2012, 225). Her awareness, also relevant in much film
theory, draws out the fact that in mapping the audiovisual components, there is not
always equality. The speed with which we process these audiovisual arts is different;
the senses by which we digest them are also different, and our experience of “see-
ing” a production dominates the eye. Interestingly then, that many choreographers
have sought to work with this notion by visualising the music. Do we therefore
hear the music better when we see their dance?
How music impacts dance is so significant to Humphrey that an approach con-
sidering translation explores not only that there is an effect, but how the effect has
resulted from the creative process. The impact of music is so great on dance in this
context that it changes the way it is seen and understood. Rather than experience
the individual arts, the audience is now invited to “one whole experience, music
and dance inextricably combined” (Jordan 2012, 226).
As commented later in the twentieth century, the famous choreographer George
Balanchine remarked that “music must be seen” (Zbikowski 2012a, 219). But how
is it seen here? What exactly is seen, and how does it project an analogy? The ele-
mental features are considered as equivalents often as the nouns which label them
are the same between the arts. But fundamentally the importance resides first in the
co-existence of these as temporal and therefore rhythmic arts, able to move in time
and to shape, divide, and structure time. Humphrey responds to Bach’s Air on a G
String by illustrating the musical phrases in the movement lines she depicts, as well
as ensuring, as noted, the rhythmic structures are seen.
Ultimately the transfer of process, of music visualisation in dance, is not an end
in itself. It is one way to bring a new work to life, but it requires a challenge – that
instability in translation affords a benefit to such a creative situation. The chore-
ographer must add something and must preserve qualities of both the arts. Their
autonomy and their relationship is important, hence the notion of interdepend-
ence: the combination must be greater than the sum of its parts, but it must also
respect the distinctive features of each domain. The process of mapping the analogy
does not remove the significant agency of each domain.
The most current and notable example of music visualisation comes from the
choreography of Mark Morris. He often uses pre-composed music and works from
the musical score. In using Purcell’s Dido and Aeneas, he created a new work, which
incorporated a live soprano on stage, ensuring the possible gestural communica-
tion in the moment between musicians and dancers. His Dido’s Lament (1995)
illustrates how choreographers use music visualisation to translate the sonic musical
elements in movement terms and in particular to the structure (the passacaglia) and
to the lyrics (physical and sonic gestures metaphorically convey the meaning of the
libretto). It points also to the importance of differences that result in gestural con-
tent between music and dance, as the visualisation is not simply mimetic.
Morris’s musical understanding is particular among contemporary choreogra-
phers, and he is rare in also conducting his works. Significantly here though,“Morris
would refer in interview to his musical approach invariably in terms of translation”
(Jordan 2015, 77). It seems this approach is unique. It highlights the process of his
creativity; it gives voice to the translator (rather than making the translator invis-
ible). Morris has a particular role in creating his dances and then translating the
content onto bodies, into performances, and considering the interrelationships of
the arts as he acts as choreographer, dancer, and conductor. Moreover, many of his
dances were made on himself, using his own body to choreograph the movement
on and then to perform the work, including the premiere of Dido’s Lament.Various
processes of translation are enacted then when he interprets the musical structure,
the visual realisation, the embodied experience. His approach is embodied, but as
he draws from the music as a starting point, we must ask whether his “dance offer[s]
us new insights into music?” (Jordan 2018, 88). Certainly, ground bass (repeated bass
phrases) and canons (imitative melodic structures) are visualised in such a way that a
listener may become aware of musical repetitions which would otherwise become
part of the larger musical texture. The opening ground bass here (Morris n.d.,
02:17 onwards) sees the bass rhythm represented in the feet of the corps de ballet.
As the bass line descends, the dancers move upstage (towards the rear). As others
have noted (Jordan 2015), Morris used gestures which are analogous directly to
language in embedding sign language, notably for “memory” (04:06) and “breast”
(03:33). These symbolic gestures also assert and bear out clearly his approach to
interrogating the analogies of the temporal arts.
The combination of music and dance is not one of “marriage,” as Damsholt
(2002, 238) remarks in relation to our metaphorical understanding of music and
dance through binaries. It is less consensual in that, to borrow from Jordan, “music
infects the dance” (2012, 226) in this particular choreographic process. The journey
to seek analogies and to visualise music places importance on the sonic elements in
such a way that music can “impose ... its own characteristics on the other” artistic
media (Cook 1998, 103). There is a deliberate choice though with these illustrated
choreographers. Music as source text does not mean that dance, the target text, only
represents music. The reciprocal nature whereby the analogous experience of these
temporal arts is mapped means that both change our perspective of the other, as a
spectator. This identifies that any translation adapts the original source text and that
the target text, although containing content of the source text, is never identical,
and neither is that the aim in creative works. The interdependency of music-dance
works extends from this necessary reciprocal exchange (even when this exchange
draws out from language analogy).
If one were to perform from music a “scientific translation into bodily action”
as St Denis remarked in discussions of music visualisation, the focus must reside on
those elements where equivalences can be found, including “the rhythmic, melodic
and harmonic structure” (in Damsholt 2018, 22). This formal plotting of musical
gesture to danced gesture is not a strict translation and it privileges a limited num-
ber of elements (especially rhythm). It reminds us that the choreographer in these
instances has many choices to make. It is a process whereby some of the content
of one medium, selected by the choreographer, is transposed to another media. As
such, the meaning and style of the work changes – exactitude is not carried forward.
6. Intercultural translation as commentary

Cultural translation can be seen in many areas, especially in music, where styles and
genres migrate around the globe, modifying according to cultural location and era.
Music is rooted in diverse cultures across all races. As such, the choice to illustrate
cultural translation is selective but remains in line with the previous twentieth-
century balletic examples. As “culture filter[s] our experiences of the world” (Sturge
2011, 67), it is imperative to consider the cultural contexts of music-dance works.
There was a recent revival of Erik Satie’s La Parade (originally choreographed by
Leonide Massine, with a libretto by Jean Cocteau, designs by Pablo Picasso, staged
by Serge Diaghilev’s Ballet Russes in 1917 [Satie 2007]) produced by National
Dance Company Wales (Satie 2017) to celebrate the centenary of the Russian Rev-
olution. But, how can a music-dance work examine cultural translation? What is
done to the work to revive it? In the new version, P.A.R.A.D.E. (carefully re-spelt
in capitals with full stops between each letter), there is a clear cultural and temporal
transplantation of a French work, performed now in Wales, to commemorate the
Russian Revolution. As such, the work (the score and aspects of the narrative)
are resituated into a new location (different culture, era, modified narrative) with
new emergent meanings. The new version has new choreography and modified
scenario, for a new translation of the music a century after its original premiere.
The BBC broadcast of the work describes it as a “bold new take” on Satie’s score
(P.A.R.A.D.E, 00:10–00.30). The director, Marc Rees, utilises the conflict present
in the political and technological changes which have encompassed the globe in
the last century. This is notable as Satie’s La Parade was set to Cocteau’s play in
which economic business was subverted through the lens of the circus managers
and a marketing ploy to sell tickets to the next show. The original was a stark satiri-
cal commentary on globalisation and technological advancement with costume
and set designs by the cubist artist Picasso: amongst the designs are managers’ cos-
tumes constructed from skyscrapers and horns, as well as a knock-kneed American
girl and a Chinese Conjurer. The cultures represented were explored through ste-
reotypes and visual cultural symbols, but the musical score was not immune to such
subversion. The siren, lottery wheel, typewriter, and so on embed the sounds of the
city. As such, this score and narrative is pregnant for a revival. The new setting like-
wise has a political dimension, an economic dimension, and a technological dimen-
sion. The translation might move the work to a new narrative and new context, but
some of the context remains and is mapped to the new context.
The circus is reimaged as a factory. The technological advancement of skyscrap-
ers is now transplanted to a humanoid robot. The outside location of the managers
selling tickets for a circus is now an immersive event where the audience becomes
the protestors against a political speaker. Armed with a flag, on which a capital P.
was written on a red background, as an audience member I was encouraged to
shout against the political speaker and the robot, and to march from the outside into
the factory (the Millennium Concert Hall, Cardiff) to experience the performance
as though the workers had prepared something to display to the senior political
figure. The notion of outside and inside, us and them, is a stable feature across both
the 1917 and 2017 versions.
The technological ideas in Satie’s score, not only through the typewriters et al.
but also through the repeated ostinato and mechanistic rhythmic passages which
return throughout, and the angular accented melodic lines, were used in movement
by correlating repeated angular movements. In movement, however, the mechani-

sation is shown through a wall of cardboard boxes which is slowly dismantled (the
curtain rises to show the wall, 09:56). The impenetrable wall of translation referred
to earlier is made live here and is disrupted, broken down.
The first dancer to appear is folded into a box and slowly works the contorted
body out of the box, leaving feet contained and held, restricted. The collaborative
team was vast: National Dance Company Wales, led by director Marc Rees, were
joined by BBC National Orchestra of Wales, as well as Rubicon Dance and Dawns
i Bawb. Choreographer Marcos Morau was joined by graffiti artist Pure Evil, archi-
tectural designer Jenny Hall, and aerialist Kate Lawrence in order to develop the
introductory immersive event. Using segments from Satie’s introduction to Parade,
musical motives are diffused through speakers in Cardiff Bay, disrupted by static and
the sounds of the protesters (04:35–04:45). The performance boundaries are dis-
solved as audience members become participants. Though as noted earlier, the role
of translation can be invasive where one media imposes its characteristics on another.
Here this imposition is on the audience and local community, with tall high-r ise flats
surrounding the commotion hearing and seeing the activity from their balconies.
The ballet production is subversive – the robot is subverted to rebel against
the political leader, working with and for the workers – much as Cocteau’s origi-
nal scenario had integrated difference and diversity. Like the surreal nature of the
first performance, the choreographer utilises the objects of the everyday working
environment to impose on the dancer – tape is used not only to seal boxes but
also restricts the dancers, as they wrap it around themselves and others and across
the floor. The ostinato and rhythmic repetition is seen in the danced movements,
although they are deliberately not synchronised, to convey the rebellion taking
place (focusing on the cultural meaning of the work, avoiding music visualisation).
Finally, the robot appears at the end alongside the workers, but notably the workers,
having previously de-robed down to white underwear, now wear costumes which
they have made live, on-stage, during the performance, from the cardboard boxes
and tape: they become mechanised themselves.
To ask where dance and music meet, one also needs to ask how they come
together, and in coming together what they do. In other words, what does their
joint, collaborative activity produce and how do they produce it? Bannerman noted
that “from the point of view of structure, dance shares commonalities with lan-
guage and [that] like language it communicates according to cultural codes” (2014,
66). These cultural codes are shared here through the meta-themes of technol-
ogy, economics, and political change. Not only has 1917 been brought forward
to 2017, in recognition of the Russian Revolution, but the commentary, political
speech, and audience inclusion acted further as a commentary on UK politics, with
direct references made to the 2017 Conservative Party conference (with coughing
interrupting the political speech). The immediate moment was a distillation of a
number of eras, events, locations, cultures, and political responses. The mapping was
complex but generated from shared cultural experience. This shared experience
facilitated the translation both of the co-creators and the spectators.
7. Conclusion
This chapter aimed to explore how music and dance work together, using transla-
tion to begin to introduce how translation is relevant to the creative and receptive
dimension of these multimodal arts. In questioning how sense is transferred between
the arts of music and dance, the three case studies reveal emergent meanings (Cook
2001) in the interplay of these temporal arts read through three potential catego-
ries of translation: senses, languages, cultures. The range of examples is unified by the
choice of narrative ballets drawn from the twentieth-century onwards. The three
themes of translation in music-dance works, which are those relating to language
translation, sensory translation, and cultural translation, show that translation ena-
bles scholars, as well as creative artists, to interrogate what they do and how they do
it, but primarily it recognises a process of translation as a fundamental act within all
creative collaborative works.
When referring to narrative film music, Gorbman asserted that “The point is
that image, sound effects, dialogue, and music-track are absolutely inseparable dur-
ing the viewing experience, and they form a combinatoire of expression” (1980,
190). Translation is a bringing together, not a separation. The three case studies
show that the notion of interdependence is vital to understanding these works as
narrative ballets, but also to understanding their artistic gestures. Interdependence is
created as these arts and artists are in dialogue – and dialogue cannot exist without
translation of some form.
Collaboration and creative work rely on the fact that the notion of translation is
not precise. “Translation in art must never be secure” (Dayan 2011, 133). As such,
in asserting a new cultural location (cultural translation), or in mapping music and
dance which respond differently through different elements to the same stimu-
lus (sensory translation), or in mapping the dance visualisation of music (language
translation), the mutability of music and dance is co-creative and is further inter-
rogated in performance through the concept of translation which seeks not only
exactitude of message, but endeavours to look to the transference of process as well
as message.
Note
1 “Translating Music” (online) was a Network funded by the Arts and Humanities Research
Council, which I co-led with Lucile Desblache. Each of these three categories of trans-
lation (Sensory, Language, Cultural Translation) has been outlined in relation to opera,
music accessibility, and education (Desblache and Minors 2019).
References
Albright, Daniel. 2000. Untwisting the Serpent: Modernism in Music, Literature, and Other Arts.
Chicago: University of Chicago Press.
———, ed. 2004. Modernism and Music: An Anthology of Sources. Chicago: University of Chi-
cago Press.
———. 2009. Music Speaks: On the Language of Opera, Dance, and Song. Rochester: University
of Rochester.
———. 2014. Panaesthetics: On the Unity and Diversity of the Arts. New Haven: Yale Univer-
sity Press.
Apter, Ronnie, and Mark Herman. 2016. Translation for Singing. London: Bloomsbury.
Austin, William W., ed. 1970. Debussy, Prelude to “The Afternoon of a Faun”: An Authoritative
Score. London: W. W. Norton and Company.
Bannerman, Henrietta. 2014. “Is Dance a Language? Movement, Meaning and Communica-
tion.” Dance Research 32 (1): 65–80.
Bassnett, Susan. 2014. Translation Studies, 4th ed. London and New York: Routledge.
Buckland,Teresa. 2007. “In Search of Structural Geist: Dance as Regional and National Iden-
tity.” In Dance Structures: Perspectives on the Analysis of Human Movement, edited by Adrienne
L. Kaeppler and Elsie I. Dunn, 187–234. Budapest: Akadémiai Kiadó.
Burnard, Pamela,Valerie Ross, Laira Hassler, and Lis Murphy. 2018. “Translating Intercultural
Creativities in Community Music.” In The Oxford Handbook of Community Music, edited
by Brydie-Leigh Bartleet and Lee Higgins, 229–42. Oxford: Oxford University Press.
Carroll, Noel, and Margaret Moore. 2008. “Feeling Movement: Music and Dance.” Review
Internationale de Philosophie 4 (246): 413–35.
Chion, Michel. 1990. Audio-Vision: Sound on Screen. Translated by Claudia Gorbman. New
York: Columbia University Press.
Cook, Nicholas. 1998. Analysing Musical Multimedia. Oxford: Oxford University Press.
———. 2001. “Theorizing Musical Meaning.” Music Theory Spectrum 23 (2): 170–95.
———. 2013. “Foreword.” In The Psychology of Music in Multimedia, edited by Siu-Lan Tan,
Annabel J. Cohen, Scott D. Lipscombe, and Roger A. Kendall. Oxford: Oxford University
Press.
Copeland, Roger, and Cohen Marshall. 1983. What Is Dance? Oxford and New York: Oxford
University Press.
Damsholt, Inger. 2002. “The Marriage of Music and Dance.” In Of Another World: Dancing
Between Dream and Reality, edited by Monna Dithmer, 237–49. Copenhagen: Museum
Tusculanum Press.
———. 2018. “Identifying Choreomusical Research.” In Music-Dance: Sound and Motion in
Contemporary Discourse, edited by Patrizia Veroli and Gianfranco Vinay, 19–34. London
Dayan, Peter. 2011. Art as Music, Music as Poetry, Poetry as Art, From Whistler to Stravinsky and
Beyond. Farnham am Main: Ashgate.
Desblache, Lucile, and Helen Julia Minors. 2019. “Translating Music.” In Translating Cultures
Encyclopedia. London: Arts and Humanities Research Council.
Duerden, Rachel. 2007. “Dancing in the Imagined Space of Music.” Dance Research: The
Journal of the Society for Dance Research 25 (1): 73–83.
Gorbman, Claudia. 1980. “Narrative Film Music.” Yale French Studies 60: 183–203.
Herzog, Patricia. 1996. “The Condition to Which All Art Aspires: Reflections on Pater on
Music.” The British Journal of Aesthetics 36 (2): 122–34.
Hodgins, Paul. 1992. Relationships Between Score and Choreography in Twentieth-Century Dance:
Music, Movement and Metaphor. Lewiston: Edwin Mellen Press.
Humphrey, Doris. 1934. Air for a G String. Accessed December 10, 2018. www.youtube.
com/watch?v=NjwJyaSIRqY.
Jordan, Stephanie. 2000. Moving Music: Dialogues with Music in Twentieth-Century Ballet. Lon-
don: Dance Books.
———. 2012. “Mark Morris Marks Music, or: What Did He Make of Bach’s Italian Con-
certo?” In Bewegungen zwischen Hören und Sehen: Denkbewegungen über Bewegungskünste,
edited by Stephanie Schroedter, 219–36. Würzburg: Königshausen & Neumann.
———. 2015. Mark Morris: Musician-Choreographer. London: Dance Books.
———. 2018. “Acts of Transformation: Strategies for Choreographic Intervention in Mark
Morris’s Settings of Existing Music.” In Music-Dance: Sound and Motion in Contemporary
Discourse, edited by Patrizia Veroli and Gianfranco Vinay, 76–90. London and New York:
Routledge.
———. Forthcoming. “Choreomusicology and Dance Studies: From Beginning to End.”
Under Peer Review.
Contemporary Communication. Oxford: Oxford University Press.
Locanto, Massimilian. 2018. “Choreomusicology Beyond ‘Formalism’: A Gestural Analysis of
Variations for Orchestra (Stravinsky-Balanchine 1982).” In Music-Dance: Sound and Motion
in Contemporary Discourse, edited by Patrizia Veroli and Gianfranco Vinay, 35–56. London
Minors, Helen Julia. 2006. “Paul Dukas’ La Péri as Interpreted by Two Balletic Collabora-
tors.” The Opera Quarterly 22 (1): 117–35.
———. 2009. “La Péri, Poème Dansé (1911–12): A Problematic Creative – Collaborative
Journey.” Dance Research: Ballets Russes Special Volume Part 2 27 (2): 227–52.
———. 2012a. “Music and Movement in Dialogue: Exploring Gesture in Soundpainting.”
Les Cahiers de la Societe quebecoise de recherche en musique 13 (1–2): 87–96.
———. 2012b. “In Collaboration:Toward a Gestural Analysis of Music and Dance.” In Bewe-
gungen zwischen Hören und Sehen: Denkbewegungen über Bewegungskünste, edited by Steph-
anie Schroedter, 163–80. Würzburg: Königshausen & Neumann.
———, ed. 2013a. Music,Text and Translation. London: Bloomsbury.
———. 2013b. “Soundpainting: The Use of Space in Creating Dance-Music Pieces.” In
Sound, Music and the Moving-Thinking Body, edited by Marilyn Wyres and Lorenzo Glieca,
27–34. London: Cambridge Scholars Press.
———. 2016. “Mediating Cultures and Musics: An Intercultural Production of a Midsum-
mer Night’s Dream.” In Routledge Handbook of Intercultural Arts Theory, Research and Practice,
edited by Pamela Burnard, Elizabeth McKinley, and Kimberley Powell, 417–30. London
———. 2019 (in press). “Opera and Intercultural Musicology as a Mode of Translation.” In
Opera and Translation: Eastern and Western Perspectives, edited by Adriana Şerban and Kelly
Chan. Amsterdam: John Benjamins.
Morris, Mark. n.d. Dido’s Lament. Accessed December 10, 2018. www.youtube.com/
watch?v=ZrzC_KLI8KM.
———. 1995. Falling Down Stairs – Sarabande. Accessed December 10, 2018. www.youtube.
com/watch?v=kAX7WF6iMcU.
National Dance Company Wales. 2017. Programme ‘P.A.R.A.D.E.’ October 24–25. Wales
Millenium Centre.
Nijinsky, Vaslav. 1987. The Rite of Spring, reconstruction Millicent Hodson, Joffrey Ballet.
Accessed May 20, 2018. www.youtube.com/watch?v=jF1OQkHybEQ.
———. 2013. Prelude à l’aprés-midi d’un faune – reconstruction, Paris Opera Ballet. Accessed
June 24, 2018. www.youtube.com/watch?v=4qjvGIMeIhU.
Satie, Erik. 2007. Parade, Picasso et la Danse, Festival Europa Danse. Accessed December 12,
2018. www.youtube.com/watch?v=YejpJ4kMH_0.
———. 2017. P.A.R.A.D.E., Carolyn Finn (choreographer), BBC 4, aired December 2.
Accessed December 12, 2017. www.bbc.co.uk/programmes/b09f469j.
Stravinsky, Igor. 1947. Le sacre du printemps. London: Boosey and Hawkes.
———. 2008. Le sacre du printemps, Vaslav Nijinsky (choreography), Millicent Hodson
(reconstruction), Set/Costume (Nicholas Roerich), Marinsky Orchestra and Ballet,
Valery Gergiev (cond.) [DVD] BelAir BAC241.
Sturge, Kate. 2011. “Cultural Translation.” In Routledge Encyclopedia of Translation Studies,
edited by Mona Baker and Gabriela Saldanha, 67–70. London and New York: Routledge.
Susam-Saraeva, Şebnem, ed. 2008. “Special Issue: Translation and Music.” The Translator
14 (2).
Venuti, Lawrence. 1998. The Scandals of Translation. London and New York: Routledge.
Wenk, Arthur. 1976. Claude Debussy and the Poets. Berkeley: University of California Press.
———. 2012a. “Music and Movement: A View from Cognitive Musicology.” In Bewegun-
gen zwischen Hören und Sehen: Denkbewegungen über Bewegungskünste, edited by Stephanie
Schroedter, 151–62. Würzburg: Königshausen & Neumann.
———. 2012b. “Music, Dance, and Meaning in the Early Nineteenth Century.” Journal of
Musicological Research 31 (2–3): 147–65.
———. 2018. “Ways of Knowing: Social Dance, Music, and Grounded Cognition.” In Music-
Dance: Sound and Motion in Contemporary Discourse, edited by Patrizia Veroli and Gian-
franco Vinay, 55–75. London and New York: Routledge.
8
WRITING DRAWINGLY
A case study of multimodal translation
between drawing and writing
Tamarin Norwood
Introduction
This chapter offers an anecdotal account of a particular method of “half-blind”
drawing (part i), from which a novel method of writing is derived (in part ii). This
process of derivation can be understood as a form of multimodal translation in
which a drawing method (the original “text” being translated) is extracted from its
original mode of drawing and reconstituted in the mode of writing, producing a
method of writing “drawingly.”
The writing that results from this method is, for academic writing at least, for-
mally and structurally unusual. Vertical ticks appear in the body of the text where
digressions present themselves during the act of writing, and the possibilities of
these digressions are drawn out and explored further along the text, creating linked
lines of textual fragments that feel their way down and across the page. As they
feel their way, these lines closely probe the object of the writing so as to gradually
reveal its contours, in much the same way that the tip of the pencil probes the page
and the eye probes the object in drawing. As Jean-Luc Nancy observes, the object
of a drawing is sought and gradually found through the gradual formation of its
image on the page (Nancy 2013, 10). Through this process, he writes, one arrives
at an idea of the object that one had not formed and could not have formed before
the drawing began (10). This enquiry and gradual encounter is played out on the
surface of the page, the “formative force” of each exploratory venture leaving on
the page a visible record: a material concretion of the process that formed it (Nancy
2013, 12). I propose that writing produced drawingly can likewise be read as the
material residue of the search it has undertaken; a residue produced almost inciden-
tally to the process of writing but kept nevertheless, rather like the gestural line kept
on the page of Tristram Shandy after the flourishing movement of Corporal Trim’s
180 Tamarin Norwood
stick (Sterne 1978 [1762], 743).1 Here, process – in writing as in drawing – takes
precedence over product.
Can the same be said of the particular process of multimodal translation I have
employed? Has the translation process taken precedence over the text it has pro-
duced? Reflecting on his approach to translating Derrida’s What is a ‘Relevant’
Translation?, Lawrence Venuti chose to adhere “as closely as possible to his French,
trying to reproduce his syntax, lexicon, and typography by inventing comparable
textual effects – even when they threaten to twist English into strange new forms,”
while balancing this strangeness against the need to maintain “a level of intelligibil-
ity and readability” for the English-language reader (Venuti 2001, 173). Commit-
ting to a process of writing drawingly demands a similar balance between writerly
intelligibility and close adherence to drawing praxis – only the prospect of twisting
the writing into strange new forms presents itself more as an opportunity than as a
threat. Open to this opportunity, my approach to translating between drawing and
writing shares the myopic and closely probing perspective of the pencil tip that
Derrida associates with the movement of the blind man who, in order to move at
all, “must advance, advance or commit [himself], … run through space as if running
a risk” (Derrida 1993, 5). Once I had identified the character and features of the
half-blind drawing method, and established the methodological parameters they
implied for writing, what remained was to advance – to run the task of translation as
though running a risk – and thus, returning to Nancy, to arrive at something that
could not have been formed before the drawing (or the translation) had begun. In
this way the text in part (ii) is not only the residue of a myopic writing process but
of a myopic translation process too: a process of translating “drawingly.”
Through this process the form of the writing, adhering as closely as possible to
its origin in drawing, is indeed twisted into strange new forms. The experience of
writing the text, which I hope is mirrored by the reader’s experience of reading
it, was an effort of concentration not unlike my pencil’s close pursuit of the life
model’s contours as they were interrupted by the junctions of his fingers, knuck-
les, and forearm in the anecdotal description that follows. Every detail of these
interruptions needed following along their length until they gave way to the next
digression and the next, and only in this way did a picture of my object – in draw-
ing as in writing – gradually take shape on the page.
If the example of multimodal translation in the present chapter is a process of
translating “drawingly,” can we look further into the praxis of drawing for new ways
of analysing the translation process? As the writing in part (ii) myopically felt its
way across the surface of the page, questions of multimodal translation were not in
sight and were never explicitly addressed. But the reader interested in the subject of
translation will no doubt encounter junctions of my text that draw their attention
towards digressions of their own, and by following the path of such associations
that “arise in the mind unbidden,” they might arrive at a picture of translation that,
following Nancy, had not formed before their reading had begun.2
For instance, the body of the life model might be imagined as an original “text,”
with the pencil sketch its translation into drawing. Following the path of this
Writing drawingly 181
association, one might consider whether, by presenting his body in multiple poses,
the model is offering multiple readings of his body, and whether each view of his
body is a new reading of the same text or a new text altogether. One might follow
the path further still and ask whether the eye of the artist moves across the body
of the model as the translator’s eye moves across the body of the text; and further
still, to ask whether artist and translator similarly aspire to a perfect correspondence
of eye and hand, rendering their intervention perfectly imperceptible and their
translation, or their drawing, perfectly equivalent to the original. And given that
perfect equivalence is unattainable in drawing as in translation, one might probe
further still, to seek in examples of translation practice the crisis point John Berger
so often encounters when his drawings near their completion, when he “begins
to draw according to the demands, the needs, of the drawing” (Berger 2005, 8). If
at this point the drawing is “in some way true,” he goes on, “then these demands
will probably correspond to what one might still discover by actual searching. If
the drawing is basically false, they will accentuate its wrongness” (8). Can the same
be said of a translation?3 One could go on, creeping further and further along this
and other wandering paths that present themselves in the analogy between drawing
and translation.4
Finally, what emerges from the writing process is a depiction of this point of
contact as a myopic, wayfaring, and pathbreaking movement provoked above all by
a desire on the part of the person drawing to draw up, use up, or consummate their
object in the laying down of a graphite line. It is a movement depicted as slow, sen-
sitive, yielding, and almost caressing, but also duplicitous in the way it approaches its
object, seeming to caress it with the closeness of its touch only to capture it and, in
capturing and pinning it onto the page, to obliterate whatever had been so uniquely
ungraspable about it.5
If drawing is indeed an embodied impulse to caress and capture its object, tak-
ing in the risk of obliterating that object’s uniqueness in the process, can the same
be said of translation, or at least of the example of translation offered in the present
chapter? In translating it into a method of writing, have I approached my original
text – the method of drawing – with the softness of a yielding caress that conceals
the threat of obliteration and, if so, what is left of the original text? To these ques-
tions I would respond by returning to Derrida as he dramatises a personal account
of drawing: “it is as if, just as I was about to draw, I no longer saw the thing. For
immediately it flees, drops out of sight, and almost nothing of it remains. . . . It
blinds me while making me attend the pitiful spectacle” (Derrida 1993, 37). The
object’s flight at the very moment of its capture is, in the pitiful spectacle of draw-
ing and perhaps also of translation, what guarantees the life of the original even
after its contours have been captured (always imperfectly, if caressingly) onto the
page. Left unconsummated at the conclusion of the drawing process is the “ineluc-
table presence – the thingness of the thing” (Schwenger 2001, 102) that remains
untranslatable but is described by the search left behind on the page by the myopic
and committed advance of the tip of the pencil, the tip of the eye, the tip of the
translator’s task.
182 Tamarin Norwood
i
Rolled up on my desk lie several scuffed sheets of cartridge paper holding the
drawings and notes I made over a couple of autumn months four years ago. I had
joined an evening life drawing class because I never seemed to draw anymore.
I have not looked at the drawings much since, but I remember the class well.
Each session opened with a couple of very quick poses, then some five-or ten-
minute exercises focusing on light, tone, or line, and then work would begin on a
final half-hour pose. Before embarking on this final drawing, we were reminded of
how to mark up our pages to avoid making errors of measurement or composition
that might only surface when it was too late to correct them.We should hold out our
pencils against our view of the model, horizontally and then vertically, and use these
impromptu rulers to measure with our thumbs a set of relative heights and distances
between landmarks about his body: shoulder to knee, for instance, shoulder to toe,
and head to navel to hand. The task was to plot this network of points onto the page
and elaborate them into a composed and proportioned image that should resemble, in
the end, what each of us saw of the model from where we were standing.
The drawings I produced in this way always seemed to lack something, and it
took me a while to understand what. Certainly I was out of practice, and to look
at, the drawings were not all they could have been. But it was not a problem of
looking; it was a problem of touch. The drawings lacked what I realised I must have
come to the class to enjoy: the soft fur and scratch of the lead with the grain, the
time spent against the page, the tugging and coaxing and scraping of form – there
was none of this. Instead, before the encounter had even begun I had constructed
an invisible apparatus of sight-lines to articulate the gap between the model and my
eye, and between my eye and hand and page: an apparatus that hardened the net-
work of landmarked points and the features between them, extracted them, shifted
them through the air and delivered them onto the paper. The apparatus seemed
to get in the way, or seemed to thicken the air and make that get in the way, with
the effect that the model’s body and my drawing were held apart by the very lines
of sight that were meant to bridge them. The pencil darted about in the air like a
scalpel or a beak, pecking at the paper rather than burrowing into it, and without
those sensitive and spontaneous excavations, drawing with a pencil didn’t seem such
a compelling thing to do.
When I went back the following week, I drew in a different way. I looked only
at the model, and never once at the page, until the drawing was done. I continued
to work in this way, half-blind as I thought of it, for the rest of the drawing course.
It was engrossing to draw like this. It seemed to seal up the gap between the object
and the page and the eye, contracting all the imaginary apparatus into one point of
contact that sees, measures, and marks in a singular probing act which, best of all, is
always only the burrowing activity that engages me most about pencil work.
The way I worked was to plant my eye at some point on the skin of the model
and at the same time plant the tip of my pencil at some point on the empty page,
which from then on would refer to the corresponding point on the model’s skin.
Now paired against their surfaces, the eye would begin to root its way across the
skin as the pencil began to root its way across the page. The tip of the pencil
became the tip of the eye – a groping proboscis feeling its way about the soft form
of the object and seeing only the little dot of its surface presently under description
by the very end of the lead. As the eye moved, so did the pencil, as much as I could
manage, and in this way the eye navigated the page by means of the blind point of
the pencil, and drawing proceeded without my ever looking at the page nor ever
losing contact with its surface.
Some weeks I would try to make the drawing resemble the model, and to do
this I needed to be more strategic. Once paired against the page and the model’s
skin, pencil and eye had to move exactly in time with one another, because if either
temporarily slowed down or sped up, inconsistencies of scale tended to be intro-
duced. If they both moved too quickly, details might be missed that could not be
revisited for correction later on, because I would be unable to find the place again.
Every detail needed attending to at the very place and time it was first encountered.
Burrowing about the surface of the page, the tip of the pencil maintained contact
with the model’s body only by the contact it maintained with the paper, so if it
came detached from the page, the body and the drawing would drop out of sight
and could not be retrieved.
Sometimes I was able to adjust marks laid down very recently and very close by.
It turned out to be possible to retrace a route just taken provided the muscles of my
hand could remember the last few flexes of the fingers or the last adjustment of the
wrist, and were able to repeat this sequence of movements in reverse. The memory
of the hand offered a couple of inches of revision – a second or so – and this redress
could be put to use strategically. For instance, I might find the junction between the
model’s knuckle and fingers comprises five paths short enough that each finger can
be traced onto the page with each return trip to the knuckle brief enough and swiftly
enough that the muscle memory of my drawing hand can render it all quite well.
But it would be a different matter at more complex or multiple junctions of his body.
What would happen when the forearm intersected the collarbone, escalated into a
hand of its own and then needed returning to the collarbone to intersect it a little
further along, such that the positions and angles of the arm and the remaining length
of collarbone looked uninterrupted by my foray through the hand and back again?
The muscles of my drawing hand would fail to remember a procedure as complex as
this, and tides of error would be introduced. The form would end up flayed across the
page, elements pivoting through one another at every junction. I would try alternative
strategies: crawling the pencil along in a series of branching advances and retreats, or
choosing the routes that might be salvaged by muscle memory and attending to them
together, then accepting as inevitable that regions with sparse detail will disorient the
pairing of pencil and eye, creating regions and contours that are irregular or indistinct;
and that any route I choose might leave whole islands altogether unreachable and
corresponding gaps on the page.
I began to bring thread and masking tape with me to the classes, and later on blu-
tac and drawing pins, too. With these I would construct lines, regions, and points to
184 Tamarin Norwood
guide my hand as it moved blindly about the page. It meant I would spend the first
few minutes of each pose preparing the paper, estimating the centre point and mark-
ing it with a drawing pin, or taping thread onto the drawing board to bisect the page
horizontally and vertically and give me four quadrants into which I could divide
my composition. Sometimes I would measure the model with my thumb held out
against my pencil, as we had in the first class, and plot certain landmarks on the page
in blu-tac before I began to draw, so I could elaborate the reference points later on
by touch. The more intricate the apparatus, the more accurate a picture I was able to
draw, and with practice I learned to obtain a drawing that quite closely resembled its
object. The tip of my eye and the tip of my pencil were still paired as before, and they
would still move about their respective surfaces in time with one another, but now
my hand was obliged to the guides as well and had to get used to subtly stretching
time or stretching space by speeding or slowing its movement. The correspondence
of hand and eye had lost its closeness. It had become only one of a complex of cor-
respondences that included the skin of the model and its relation to the flatness of
the page, my memory of how I had divided up the page and its relation to my view
of the model, and how all of these related to the feel of thread or other apparatus
that would sometimes, unexpectedly or otherwise, cross my path. By this stage I had
noticed I was losing interest in assembling these ever-more complex guides. There
came a point, I felt, that I might as well just be looking at the page.
*
Throughout the life drawing course I was engrossed, I believe, in the experience
and sensation of drawing. But at the same time I was gripped by the detail of the
drawing process. Every little aspect of the process seemed to me a completely new
invention, and moreover seemed to bristle at every turn with obscure significance,
as though every aspect were a metaphor for something unknown, or at least not
visible from where I stood. Perhaps if I could stand somewhere else I would be able
to see. Perhaps this is why I was so anxious to store in my memory every peculiar-
ity of the process: the strategies, restrictions, and obligations, the precision of the
matching of pencil and eye and surface and skin, and the ranging of touch and
sight – what was missing and what was there in excess. Perhaps I wanted to store
them so that I could be among them again at another time, from another vantage
point. Whatever the reason, these reflections nagged and nagged at me until the
effort to remember them, and the anxiety that I might forget them, began to dis-
tract me from the drawing altogether.
When this happened, maybe two or three times in the course of a half-hour
drawing, I would stop work, scribble a note at the right-hand edge of the page, and
then resume drawing. Braver lines, reads one note: through space, | launching forth, | not
blind | in the dark. Coming across such notes now, a few years after they were written,
it is difficult for me to recall what I must have meant. In some of the notes, this dif-
ficulty is compounded by the handwriting, which is scrawling and ambiguous: was
it launching forth, or launching form? Not blind, or not island? In any case I remember
how these scrawls were put on the page. When I needed to write something down,
I would fix my eyes very keenly on the model’s skin and, without moving the
pencil from its position on the page, I would feel my way towards its tip with my
left hand, which until then would generally have been propped at the top edge of
the drawing board to keep it steady. The left thumbnail would press against the tip
of the pencil, oriented in the direction of its movement, and then, eyes still fixed in
place, the pencil could be lifted up from the thumbnail marker and away from the
page. From here it was easy enough to feel with my right hand the right edge of the
page, scribble the words, locate the thumbnail again, and go on with the drawing
in the direction indicated by its orientation. It took a greater effort of concentra-
tion to hold my eyes exactly still throughout the operation, and perhaps they did
wander a little without my noticing. Little gaps or interruptions in the line were no
doubt also introduced when I made these notes, but these were a small price to pay
if I could shed the distraction of trying to keep everything in mind.
But it would accumulate again, and more than once I stopped drawing altogether
and left the studio early so no more would come. On the walk home, drawings rolled
up under my arm, I would try to go through everything that had happened, and once
home, with the drawings laid out across the table, I would consider everything at a
pace I found acceptable: slowly and uninterrupted by the generation of even more.
*
Of course more comes anyway, even now that no drawing is going on. It seems
to me that all this work has yielded two kinds of forms: the visible forms marked
in pencil on the page and the invisible forms of the processes that had created
those pencil marks. Both kinds of form tend to germinate and become increas-
ingly proliferate the more time I spend with them. I can look at the marks on
the page and identify the body parts I have drawn, even if some are ambiguous or
sparsely described. But if I study patches of pencil marks close up, or squint and
take in whole compositions at once, I can also see resemblances of things I had
never knowingly drawn, like seeing particular pictures in clouds or blots of ink,
but more concerted somehow. If I look at it long enough, the scribbled joint of a
knuckle, for instance, resembles a copse of trees, then a root system or a junction of
roads, then a clump of eyes or mouths or parts of ears, and now foliage again. One
image shifts into another, sometimes swiftly and sometimes gradually and partially,
so that distinct pictures might coexist or temporarily borrow one another’s parts.
But the same is true of the other kind of forms – if form is really the right word
to use – the invisible impressions and ideas that have collected around the drawing
process. Although I cannot see their forms, they are palpable enough to grip hold
of in my imagination, and I can spend time with them. As I squint and scrutinise
them, I have found they bristle with resemblances in just the same way as the con-
figurations of pencil they left on the page. The moving tip of the pencil becomes
the tap of the seeing eye of a white cane, for instance, which taps a line or taps a
reservoir that leaks and spills, or it grazes the surface of the ground at the risk of
harming its ocular membrane, which becomes the scratched tip of the pencil again,
and so on. These invisible forms and their analogues move through and across one
another in my mind, coexisting, borrowing one another’s parts, reinforcing, chang-
ing, and tempering one another as I watch. As they continue to proliferate, the
186 Tamarin Norwood
bonds between them multiply and thicken and spread out, attaching to one another
at the edges and from inside. One idea splits into many more, each of these divert-
ing through paths of their own, some closer, some further away, some vanishing
only to reappear, strange and familiar, later on.
It seems to me, as I write, that these resemblances are to be taken seriously. If
I can follow their proliferation I might hope to approach whatever kernel they cir-
cle around. There is no directing the course of their growth, but there is something
familiar about the way they grow that leads me to the looming impression – a kind
of heavy inkling – that they are all describing the nature of the very tip of the pencil
and the particular kind of contact it makes as it moves about the page. Just this tiny
thing. A dot. I am surprised it has enough mass to pull me towards itself.
ii
A theologian loses his sight |. He learns to make his way | to work |. By sound, smell,
the movement of air on his skin | and the tap of the cane against the pavement he
learns to track his position on the ground with some precision. Meanwhile, his
mind is in the air, so to speak, picturing himself as a moving point on a memorised
| map of the route. As long as he can keep his movement along the ground and on
the map in time with one another, he knows where he is. It becomes difficult when
the two topologies suddenly cease to correspond. An obstacle on the pavement can
throw him off course completely. With the white cane, he can move around small
obstacles without losing his bearings, but with larger or more complex obstruc-
tions it is a different matter. A car parked across the pavement, or worse, one car
after another parked at unexpected angles, are structures that risk introducing tides
of error into his course as he tries to follow their perimeters back to the route he
is taking. Turned through all these angles, he might still know more or less where
he is on the memorised map, but not which way he is facing | (Hull 1990, 90–91).
Loses his sight. His story reads like a parable, although the lesson it teaches is
unclear.
He learns to make his way. I imagine he makes his way more than I, who can
see, make mine.
To work. The object(ive) of his drawing. His walk produces not the line of his
movement but his arrival at work.
I wonder if his account sounds the way it does – conversational, open to possi-
bility, feeling their way – because it is composed in the moment of speaking.
Then again, perhaps I hear these qualities in his account only because
I know he spoke it into a tape recorder.
I would like to know if he spoke to an imaginary listener as he recorded,
or just to the tape.
Memorised. I watched a film once of a child’s sight being restored. While he was
blind he would race around the house, every orienting marker – doorknobs,
lampstands, the plant pot on the windowsill – landing at his fingertips just as he
expected. These were his pivots, and he would ricochet from them and navi-
gate his home with reckless precision. The film showed him running down
the stairs and out of the door, excited to get to the hospital for the procedure.
Once home from the hospital, his restored sight seemed to blank out what
he knew about his surroundings. He stumbled and tripped, his turns were
miscalculated, his feet and fingertips weren’t where he expected them. He
had to learn to make his way around all over again.
Blind, bind
Many years ago I heard a woman on the radio telling the story of her
grandmother leaping and dancing arabesques through the fields as a
little girl, delighted that the proud day had come for her feet to be
bound. The irony is so concise I now find myself wondering if I had
misremembered, and it was fiction after all.
Was it Emily Prager’s short story “A Visit from the Footbinder” (Prager
1999)? It describes much the same thing happening, only it was set
in the China of 1260 and the child scampers around a palace, not
the countryside, while her elder female relatives shuffle and walk
with canes. Her story was criticised for perpetuating “the opposi-
tion between Western observer and Eastern ‘object,’ reinforcing a
readerly politics of domination” (Newman 2007, 6).
Exotics, erotics, “visual erotics allows the object of vision to remain
inscrutable” (Marks 2000, 184).
“The seeking of the caress constitutes its essence by the fact that
the caress does not know what it seeks. This ‘not knowing,’
this fundamental disorder, is the essential. It is like a game
with something slipping away, a game absolutely without pro-
ject or plan, not with what can become ours or us, but with
something other, always other, always inaccessible, and always
still to come” (Levinas 1989 [1947], 51).
Which way he is facing. There is a game children play. One puts on a blindfold
and is spun around and around by the other children, slowly to begin with
and then more quickly until she no longer knows which way she is facing.
As she begins to stumble, the other children flee and she runs after them |,
hands outstretched, following the sound of their voices until she has caught
each one – or else gives up, unties the blindfold, and restores herself to sight.
She knows that the knot is within her reach and she can untie it at any time
|, but she also knows that if she does it will bring the game to an end.
188 Tamarin Norwood
The other children flee and she runs after them. Imagine all the children leav-
ing traces of their movements on the ground, or a video camera set above
them and the aerial view of their movements faithfully transcribed onto
a page. The task of translating movement into mark would demand not
only copying the lines of each child’s path but drawing something about
their speed, the certainty with which they place their feet, the work of
their hands and eyes. It would be easy to identify in such a diagram which
child wore the blindfold and which were free.
She can untie it at any time. “Naturally his eyes would be able to see. But they
are blindfolded . . . not naturally but by the hand of the other, or by his own
hand, obeying a law that is not natural or physical since the knot behind
the head remains within a hand’s reach of the subject who could undo
it. . ., as if he chose it, at the risk of a fall |” (Derrida 1993, 13, emphasis
original).
Fall. I have wondered whether he means “fall” in the biblical sense too,
when things were first lost to words.
One of my half-blind drawings troubles me. It’s not the fact that the leg
looks mauled and out of joint that offends me (after all, the jaw looks
fractured and pulled away from the skull and I don’t mind this); it’s that
the leg looks wrong in the wrong way. The rest of the body is compelling,
precise, sensitively explored, and the many anomalies resulting from my
blindness on the page coax from the body a heightened sensitivity of its
own. She seems to be lost in thought. Then a leaden foot is planted, her
only contact with the ground, the pencil ragged now and nearly blunt. It
gets the spacing wrong, and the spacing looks all the more wrong because
the angles are fairly right. I know I drew the legs last. Perhaps I had lost
interest by then. Perhaps I knew by touch or sound that the lead had got
too thick and its lines would be bad for the body, but I wanted the body
complete, so I went through the rest without much care.
Then again, perhaps the trouble is not the drawing but the fact that I’m
looking at it. Perhaps the trouble is that I’m looking at it and wanting it
to look good.
The tip of the pencil is something like an eye moving along the surface of the page
|.With a focal length of nil, it watches the black deposit of graphite dragging | from
its tip and adhering to the page, seeming all the blacker because there is no light
to see by at such close range. There is always only this | black. Whatever it draws
| onto the page, however the speed and orientation of the line might change, the
little spot under the scrutiny of the pencil’s eye varies very little at all. The surface
of the eye collects scratches | and marks as it grazes | the page, of which some must
surely damage its sight | – sight so blunt it is nearly touch |, sensing nothing of the
ground behind it or ahead |. To see what lies ahead it must commit to drawing
forward into the dark, to running through space as if running a risk |.
An eye moving along the surface of the page. Derrida imagined what the pencil
might see, pressed so close to the page, and he found it deeply myopic. From
the “aperspective of the graphic act,” the stylus is blind both to the form that
precedes it (the thing it sets out to draw) and the form that follows it (the
drawing that will appear on the page) with the result that it sees nothing
but the immediate present of its “originary, pathbreaking moment” (Derrida
1993, 45).
The black deposit of graphite dragging. “Here |, to trace is to find, and in order to
find, to seek a form to come (or let it seek and find itself) – a form to come
that should or that can come through drawing” (Nancy 2013, 10).
Here. Here, on this page, too.
This. This, the black of this ink, too.
Whatever it draws. Or writes. Imagine that when you write there is always only
this black: the myopic tip of the pen, or the short-sighted eye of the I-beam
cursor, blinking (I . . . I . . . I. . .).
The surface of the eye collects scratches. I wonder if it hurts.
And if it hurts the page as it grazes the surface.
As it grazes. “Consider the cows, grazing as you pass by; they do not know
what is meant by yesterday or today, they leap about, they eat, rest, digest, leap
about again, and so on from morning until night and from day to day, fettered
to the moment and its pleasure or displeasure, and thus neither melancholy
nor bored. This is a hard sight for man to see; for he thinks himself better
than the animals because he is human, but he cannot help envying them
their happiness – what they have, a life neither bored nor painful, is precisely
what he wants, yet he cannot have it because he refuses to be like an animal”
(Nietzsche 1997 [1876], 60).
Some must surely damage its sight.
“Mit allen Augen sieht die Kreatur
das Offene. Nur unsre Augen sind
wie umgekehrt und ganz um sie gestellt
als Fallen, rings um ihren freien Ausgang.
Was draußen ist, wir wissens aus des Tiers
Antlitz allein; denn schon das frühe Kind
wenden wir um und zwingens, daß es rückwärts
Gestaltung sehe, nicht das Offne, das
im Tiergesicht so tief ist.”
(With wholly open eyes a creature sees/the Open. Our eyes alone are as if
turned about and wholly ranged about/as traps that shut its clear way out off.//
We sense what lies outside from animal/countenance alone; newborns even/we
turn about and force them on their backs/see artifice and not the Open, set/ deep
within the animal.) Rilke 1997 [1923], translation by Anton Viesel
190 Tamarin Norwood
Sight so blunt it is nearly touch. As he constructs an analogy between drawing

and blindness, Derrida writes less about the absence of sight than the pres-
ence of touch. The stylus is a staff of the blind that feels its way, providing a
tactile proxy for sight, and the fingertips grope about the page “as if a lidless
eye had opened at the tip of the fingers, as if one eye too many had grown
right next to the nail, a single eye, the eye of a Cyclops or one-eyed man”;
this lidless eye is “a miner’s lamp” whose light makes inscription possible.
Throughout his account of blindness, the conflation of movement, touch,
and sight is never out of reach (Derrida 1993, 3).
Sensing nothing of the ground behind it or ahead. “Proceeding on our way things
fall into and out of sight, as new vistas open up and others are closed off. . . .
Thus the knowledge we have of our surroundings is forged in the very course
of our moving along them” (Ingold 2007, 87–88).
Running through space as if running a risk. Derrida describes the blind men in the
drawings of Antoine Coypel. “Like all blind men, they must advance, advance or
commit themselves, that is, expose themselves, run through space as if running
a risk. They are apprehensive about space, they apprehend it with their groping,
wandering hands; they draw in this space in a way that is both cautious and bold;
they calculate, they count on the invisible. It would seem that most of these blind
men do not lose themselves in absolute wandering” (Derrida 1993, 5).
Of the many allegorical drawings of blind men | in Memoirs of the Blind, some move
forward alone, with no cane and with arms outstretched, fingers groping ahead | of
themselves. They are deeply closed into the dark, into the touch of the immediacy
of the place: a picture of exquisite closeness | and excess of sensation |. This must
be a kind of wandering, and it is portrayed as nothing good |. The blind men in
these pictures have in their minds’ eyes not a map of their surroundings but a hope
of sight restored. This is why these men are drawn: to show the miracle of vision,
coming into the light, the salvation of enlightenment |.
Allegorical drawings of blind men. There are many such drawings printed in
Memoirs of the Blind (Derrida 1993, 5).
Fingers groping ahead. Ahead. “To anticipate is to take the initiative, to be out
in front, to take (capere) in advance (ante). Different than precipitation, which
exposes the head | (prae-caput), the head first and ahead of the rest, anticipa-
tion would have to do with the hand. The theme of the drawings of the blind
is, before all else, the hand. For the hand ventures forth, it precipitates, rushes
ahead, certainly, but this time in place of the head, as if to precede, prepare,
and protect it. . . Anticipation guards against precipitation” (Derrida 1993, 4).
Exposes the head. Sometimes, wearing a scarf on cold days, it occurs to the blind
theologian that he could equally go about with his whole head covered, not
just the lower part of his face (Hull 1990, 53).
“And so with my forehead I ran against the wall thousands and thousands
of times, all day and all night long, and I was glad when I banged myself
bloody, for this was proof that the wall was beginning to harden” (Kafka
2007 [1931], 165).
Vertiginous somersault, perilous freefall, bang your head bloody, graze
your eyes.
A picture of exquisite closeness. For example, Antoine Coypel, Study of the Blind,
Louvre Museum. How wide open is his eye, how much surface it has.
And excess of sensation. “I suspect, nevertheless, that he was not very capable of
thought. To think is to forget a difference, to generalize, to abstract. In the
overly replete world of Funes there were nothing but details, almost contigu-
ous details” (Borges 1985, 104).
Portrayed as nothing good. The blind lead the blind into a pit: underground.
Luke 6:39.
The salvation of enlightenment. Laura U. Marks writes about touch and the occlu-
sion of sight and points out that lacking sight does not mean lacking insight.
She writes that the analogy between seeing and knowing depends upon a
conception of sight roundly questioned by the feminist and phenomenologi-
cal lines of thought that support her account (Marks 2000, 133). Because it
can perceive over distances, she writes, sight has come to be understood as the
sense most separate from the body, the most cerebral and the most suited to
objective observation, in contrast to non-visual senses consequently thought
of as natural, prediscursive, or not cultivated (132). And from there, the disem-
bodied, all-seeing eye of vision has become associated with the objectifying
gaze that propagates a separation between subject and object (162). There are
other ways to attend to this separation, Marks writes, if we can approach it by
means of insight brought about by the occlusion of sight (191–93).
No roots are put down when moss grows |, so patches can be lifted up and gath-
ered with the hands or by sliding a trowel just underneath the surface of the soil. It is
sometimes possible to pull it up in handfuls, like pulling hair, though this is a delicate
manoeuvre which must be done slowly so that you can monitor the many resistances
of the rhizomes it nests in the soil and vary the direction of your movement so you
tear as few as possible.You can hear the low sound of fibres breaking in the soil as you
tug. Once collected, the fragments can be held in the palm of the hand | and carried
to the garden |, where they can be arranged on ground prepared in advance |. Once
positioned, the surface must be watered deeply and pressed down by treading over it
with your feet, and if necessary, patches should be held in place with vertical metal pins |.
No roots are put down when moss grows. Moss is a rhizome.“Unlike trees or their
roots, the rhizome connects any point to any other point. . . . It has neither
beginning nor end, but always a middle (milieu) from which it grows and
192 Tamarin Norwood
which it overspills. [Its] lines, or ligaments, should not be confused with line-
ages of the arborescent type, which are merely localizable linkages between
points and positions” (Deleuze and Guattari 2004, 21).
The fragments can be held in the palm of the hand. Held in the hand, a tuft of acro-
carpous moss resembles a small rodent |, and can be stroked from nose to tail.
Resembles a small rodent. “A fragment, like a small work of art, has to be entirely
isolated from the surrounding world and be complete in itself like a hedge-
hog” (Schlegel 2002 [1798–1800], fragment 206).
Appraising the way the fragment was used by the Jena romantics, Philippe
Lacoue-Labarthe and Jean-Luc Nancy expand upon “what should be called
the logic of the hedgehog” (1988, 44). Fragment 206, they write, compares
the fragmentary work to a “small work of art,” having an individuality and
totality of its own, but being “neither directly nor absolutely the Work” (43).
Because it is a genre of plurality – that is, “to write the fragment is to write
fragments” – an individual fragment is both complete in itself and indicates
the completeness of the whole of which it is part (43–44).
Carried to the garden. You could leave moss where you find it on the forest floor
and still cultivate it. I wonder, then, whether it would still be the forest floor
or a garden.
Ground prepared in advance. To cultivate a moss garden from scratch, the area
must first be cleared of all vegetation and plant debris, the land smoothed to
encourage branching and rhizome attachment, and contoured if necessary
to eliminate any small depressions that might collect water. The surface soil
must be loosened slightly with a rake just before the moss is laid down, to
help it make contact with the ground.
The area cleared and smoothed, and lightly raked: I think of the presupposi-
tionless starting point Edmund Husserl described when he began to work
on phenomenology (1970, 263). Imagine lightly raking the surfaces of
certain phenomena and lightly raking the surface of the mind that might
brush against it.
Imagine these surfaces bristling warmly in a manner that is almost feline,
the way a cat presses the arch of its back against your hand to redouble
your caress as you stroke it.
Held in place with vertical metal pins. The “logical and organized garden,”
writes Jyrki Siukonen, is an enclosed place, familiar and cultivated – the
place before the Fall – and it stands in contrast to the unorganised, hostile,
even sublime forest with its limits unknown (1996, 2). “Attempts to under-
stand the forest with the logic of gardening,” he writes, “are doomed to
fail. Gardening is, especially in its domestic forms, something that can be
learned from books, while in the forest you need knowledge that can only
be learned by living in the forest” (2). In the forest “lies the root of poeti-
cal knowledge,” something which, like intuitive knowledge, resists being
clearly understood by logical thought (2). He describes one such effort as
an attempt to “clear away all obscurity and create an organised and reli-
able system of knowledge . . . by cutting off all ‘overgrown’ branches and
end[ing] up with a tree of knowledge pruned into exact mathematical
shape, like those seen in formal gardens” (3).
A film by Shauna Beharry, Seeing is Believing, shows a photograph of a piece of

fabric, shot | at very close range (Beharry 1991). The closeness means the shot is
often out of focus, and is very dark indeed when the lens makes contact with the
photograph. When the image is dark, it becomes grainy and even harder to see. |
Watching the video, most of all you are aware of surfaces: the surface of the lens, of
the photograph and of the fabric, of the screen upon which the video plays, even
the surface of your eyes. | Of watching the film, Laura U. Marks writes: “I realize
that the tape has been using my vision as though it were a sense of touch; I have
been brushing the (image of the) fabric with the skin of my eyes, rather than
looking at it” (2000, 127). Each surface is pressed so near to the next that there is
never enough distance to see the whole of any object, to understand its geography,
to understand the route being made across the terrain. You see it so closely that it
evades your vision, but neither is there anything to touch. |
Shot. Words also murder things (Schwenger 2001).
Grainy and even harder to see. Beharry is “using video to show the limits of
vision” (Marks 2000, 112).
“The caress of the eye over the skin is so utterly, so extraordinarily gentle, and
the sensation is so bizarre it has something of a rooster’s horrible crowing”
(Bataille 2001 [1963], 66).
But neither is there anything to touch. Marks writes about haptic visuality in the
context of diaspora film and video, where the haptic answers a longing to
return to the sensual experience of inhabiting an environment you have left
behind. Where you can no longer touch the skin of a loved one, she argues,
you can use the lens to engage with the touch of skin while engaging with
the absence and incommunicability of that touch. The touch of lens against
fabric is what makes the fabric inaccessible to sight: it is “so engaged in the
present that it cannot recede into cognition” (Marks 2000, 191). Meanwhile,
as much as haptic images “might attempt to touch the skin of the object, all
they can achieve is to become skinlike themselves” (192). In this way haptic
visuality affords a way of knowing that “implies a fundamental mourning
of the absent object or the absent body, where optical visuality attempts to
resuscitate it and make it whole” (191).
194 Tamarin Norwood
The films and videos she describes create the conditions for haptic visuality in
order to inspire “an acute awareness that the thing seen evades vision”: to
acknowledge its unknowability, and keep its unknowability intact by estab-
lishing a relationship not of mastery but of mutuality (191). In this “bodily
relationship between the viewer and the image,” “it is not proper to speak
of the object of a haptic look as to speak of a dynamic subjectivity between
looker and image” (164).
Our neighbour had always been blind. He knew the roads around us very
well. I believe the maps in his mind were drawn not from the air but from
the ground: a series of memorised cues that told him the story of his route
as he encountered them one by one. On the way home he would arrive
at his gate, walk half a dozen steps further to our gate next door, tap it
with his cane, and then turn back to his own gate and push it open. It was
a habitual gesture, as practised and perfunctory as turning a key in a lock,
but however briskly he seemed to execute the move, he would never omit
it. Our gate must have been the final orienting cue.
On a page of Tristram Shandy a fictional | Corporal waves his stick in a
flourishing gesture of freedom and spontaneity (Sterne 1978 [1762],
743). The gesture itself is not quite free nor exactly spontaneous; it
is an expression of these qualities, exerted upon the air to complete a
spoken utterance about how freely a man can live outside the confine-
ment of marriage.
Fictional. Ingold’s description of the line indulges the fiction of the
novel, imagining Corporal Trim’s gesture really to have preceded
that of the author who simply copies it: “Like any other gesture, the
Corporal’s flourish embodies a certain duration. The line to which
it gives rise is, therefore, intrinsically dynamic and temporal. When,
pen in hand, Sterne recreated the flourish on the page, his gesture
left an enduring trace that we can still read” (Ingold 2007, 72).
But Trim’s flourish is not like any other gesture. It is imaginary, and at
some point in the process of its being imagined it is drawn over with
the near-identical gesture of the one who imagined it. I wonder
which is more real, the imaginary line of the flourish or the visible
one on the page? I wonder which came first: whether there really
was an image of the Corporal’s gesture in Sterne’s mind first of all,
which he faithfully transcribed onto the page; and if so, whether it
was an image of the line he was to draw, or an image of the move-
ment he was to make.When he saw the line on the page, was it new
to him? Otherwise, perhaps the author slipped through the surface
of the fiction for a moment, so to speak, and embodied his char-
acter, and the pen embodied his stick. He might have propped his
manuscript upright on an easel, adjusted his grip on the pen, readied
himself, and then struck: a performance.
In any case, once the line was on the page: then, afterwards, that was
how the Corporal had moved his stick.
But which way around are we to imagine it? Are we to imagine
Trim standing on our side of the paper, facing away from us
towards the page; or facing towards us, in which case he is behind
the page, or within it, so that the page contains him?
What about the inhabitants of the empire depicted in Umberto
Eco’s imaginary map on a scale of one to one (Eco 1995)? If
such a map should be spread out to cover the entirety of the ter-
rain it depicts, are we to imagine the inhabitants of that terrain
walking about underneath the map and consult it by looking
up, or walking about upon the surface of the map, consulting
it by looking down? In the latter case, as Eco points out, they
would “in reality, inhabit the map” and not the terrain at all (98).
Imagine the impossibility of producing a translation on a scale
of one to one, its surface so similar to and so closely overlay-
ing the terrain of the original text that the two are all but
indistinguishable. A translation so perfect it need barely exist.
What would be left of the original text? Would it degenerate,
or at least be altered, by so close a translation, just as a ter-
rain closely covered by the surface of its map would gradu-
ally deteriorate for lack of sunlight and precipitation (Eco
1995, 98)? And if so, just as the map would cease to faith-
fully represent the terrain as its ecology of adapts to new
conditions, would the translation likewise begin to fail?
I imagine tapping a gate as you might tap a tree: drilling a hole into the surface
until the sap begins to bead and drip, and then inserting a spile to direct the
fluid down the line. Or tapping the skull to alleviate pressure, or tapping a
line to listen in. He taps the gate, extracts from it his route and takes it home.
Notes
1 Indeed, might all writing be read in this way, to varying degrees?
2 Peter Schwenger explores the generative potential of “pay[ing] attention to images that
flicker so briefly at the borders of reading that we are scarcely aware of them” and his
reminder that “associations arise in the mind unbidden, so that it is almost never possible
to read at the denotative level alone” (2012, 25, 30). His account of reading, here informed
by jazz improvisation as much as literary theory and cognitive science, has much in com-
mon with the tensioned myopia and digressive expansiveness of writing (and reading?)
drawingly.
3 Can the same be said, indeed, of analogy between drawing and translation, of which cer-
tain paths of association might be “in some way true” and others “basically false”?
196 Tamarin Norwood
4 Developing the possibility that certain lines of enquiry might be more “true” or more
“false” than others, Schwenger provides an insightful discussion of the “uses” and “pleas-
ures” of any reader’s “superfluous associations” in the reading of any text (2012, 27).
5 This characterization of drawing bears resemblance to the Biblical story of the origin
of language, in which Adam’s first act of naming transforms a thing into the object of
a speaking subject. Naming the animals of Eden “nullified them as beings on their own
account” (Hegel 1979, 221) and assimilates their being with the human conception of that
being. Schwenger outlines the tensioned relationship in literature between the desire to
capture and the risk that capture might “murder” its object (2001); alternative strategies
for keeping the unknowability of the object intact by establishing a relationship not of
mastery but of mutuality through the caressing look of “haptic visuality” in film and video
are considered by Marks (2000).
References
Bataille, Georges. 2001 [1963]. Story of the Eye. London: Penguin Classics.
Beharry, Shauna. 1991. Seeing is Believing. Video. Montréal, Québec: Groupe Intervention
Vidéo.
Berger, John. 2005. Berger on Drawing. Cork: Occasional Press.
Borges, Jorge Luis. 1985. “Funes the Memorius.” In Fictions. Translated by Anthony Ker-
rigan, 97–106. London: John Calder.
Deleuze, Gilles, and Guattari, Felix. 2004. A Thousand Plateaus: Capitalism and Schizophrenia.
Translated by Brian Massumi. London: Continuum.
Derrida, Jacques. 1993. Memoirs of the Blind. Translated by Pascale-Anne Brault and Michael
Nass. Chicago: University of Chicago Press.
Eco, Umberto. 1995. “On the Impossibility of Drawing a Map of the Empire on a Scale of
1 to 1.” In How to Travel with a Salmon and Other Essays. Translated by William Weaver,
95–106. London: Harcourt.
Hegel, Georg Wilhelm Friedrich. 1979 [1802]. System of Ethical Life and First Philosophy of
Spirit. Translated by H. S. Harris and Malcolm Knox. Albany, NY: State University of
New York Press.
Hull, John M. 1990. Touching the Rock: An Experience of Blindness. London: SPCK.
Husserl, Edmund. 1970. Logical Investigations. Translated by J. N. Findlay. New York: Humani-
ties Press.
Ingold, Tim. 2007. Lines: A Brief History. New York: Routledge.
Kafka, Franz. 2007 [1931] “The Burrow.” In Kafka’s Selected Stories, translated and edited by
Stanley Corngold. New York: W. W. Norton and Company.
Lacoue-Labarthe, Philippe, and Jean-Luc Nancy. 1988. The Literary Absolute. Translated by
Philip Barnard and Cheryl Lester. Albany, NY: State University of New York Press.
Levinas, Emmanuel. 1989 [1947]. “Time and the Other.” Translated by Richard A. Cohen.
In The Levinas Reader, edited by Séan Hand, 37–58. Oxford: Blackwell.
Marks, Laura U. 2000. The Skin of the Film. Durham, NC and London: Duke University
Press.
Nancy, Jean-Luc. 2013. The Pleasure in Drawing. Translated by Philip Armstrong. New York:
Fordham University Press.
Newman, Judie. 2007. “The Readerly Politics of Western Domination: Emily Prager’s ‘A
Visit from the Footbinder’.” Journal of the Short Story in English 48: 1–12.
Nietzsche, Friedrich. 1997 [1876]. Untimely Meditations, edited by D. Breazeale, translated by
R. J. Hollingdale. Cambridge: Cambridge University Press.
Prager, Emily. 1999. “A Visit from the Footbinder.” In A Visit from the Footbinder and Other
Stories. London:Vintage.
Rilke, Rainer Maria. 1997 [1923]. Duineser Elegien. Munich: Deutscher Taschenbuch Verlag.
Schlegel, Friedrich. 2002 [1798–1800]. “Athenaeum Fragments.” In Classic and Romantic Ger-
man Aesthetics, edited by J. M. Bernstein, 246–60. Cambridge: Cambridge University Press.
Schwenger, Peter. 2001.“Words and the Murder of the Thing.” Critical Inquiry 28 (1): 99–113.
———. 2012. At the Borders of Sleep: On Liminal Literature. Minneapolis, MN: University of
Minnesota Press.
Siukonen, Jyrki. 1996. On Artistic Knowledge: Notes for a Minor Platonic Exercise. Leeds: The
Centre for the Study of Sculpture.
Sterne, Lawrence. 1978 [1762]. The Life and Opinions of Tristram Shandy, Gentleman. Edited by
Melvyn and J. New. Gainesville, FL: University Press of Florida.
Venuti, Lawrence. 2001. “ ‘Introduction’ to Jacques Derrida, ‘What Is a ‘Relevant’ Transla-
tion?’ ” Critical Inquiry 27 (2): 169–73.
BEYOND WORDS
Concluding remarks
Ángeles Carreres and María Noriega-Sánchez
The chapters in this book have explored the interplay between translation and
multimodality from a range of viewpoints and disciplinary-theoretical paradigms.
The scope and nature of the questions they have asked, and the centrality of those
questions to both multimodality and translation studies, are in themselves testimony
to the need for further cross-disciplinary dialogue. It is not surprising that, in setting
out to map the complex connections between multimodality and translation, the
contributions to this volume have asked as many questions as they have answered.
In so doing, they have helped establish a set of coordinates that can serve as a
grounding framework for future research.
Placed under close scrutiny, some of the key metaphors that have traditionally
helped us conceptualise translation appear suspect when considered from a mul-
timodal perspective. In particular, the notion of meaning transfer – built into the
etymology of the word “translation” itself – that underpins much of Western trans-
lation theory, is perceived as inadequate in a social semiotics framework, as Kress
has emphasised in his chapter. Writing within literary studies, Reynolds reaches a
similar conclusion: notions of meaning transfer perpetuate the fiction of languages
as wholly discreet entities that can be mapped onto one another in seamless equiva-
lence. Metaphors of meaning re/trans-positioning, re-constitution, or re-creation
are felt to capture more accurately the processes at play in multimodal commu-
nication as well as in translation. Connected to this, the question presents itself as
to what sorts of meaning reconstitution/transposition can legitimately be called
translation without overstretching the term to the point where it would cease to be
useful, and it is no surprise that several contributors to this volume have focused on
this crucial issue (e.g., Kress, Kaindl, Tomalin, Minors).
As may be expected given the inchoate state of research in the area of transla-
tion and multimodality, discussions on and around terminology have featured in a
number of chapters (Kaindl, Kress, Tomalin). While the need for translation studies
Concluding remarks 199
to engage with multimodal communication is recognised throughout this volume,

there is no consensus as yet regarding the terms used to name the various kinds of
meaning reconstitution. We find two distinct takes on this question: one sees the
need for specific terminology to denote instances of meaning reconstitution tak-
ing place across modes (e.g., transduction, transposition), as opposed to within the
single mode of language (Kress); the other opts for retaining the term “translation”
for all cases of meaning reconstitution, and to distinguish between intra-and inter-
modal translation (Kaindl). While the use of a specific term has the advantage of
focusing attention on the multimodal element, it also risks entrenching an outdated,
language-centred view of translation which – as several chapters in this book argue –
it is time for us to move past. A more relevant notion of translation understands it as
a practice of cultural mediation in which texts are transposed in terms of a range of
variables: specifically, mode, medium, and genre, in Kaindl’s definition. Beyond the
terms we choose to employ, the need to open up the field of translation – and of
language – to an understanding of its multimodal dimension is one of the key tenets
of this volume, and one that should inform future research in the field.
Indeed, the centrality of language to human communication and, by extension,
to translation is one of the long-standing assumptions that contributors to this
volume have sought to destabilise. As is richly illustrated in the various chapters,
in today’s hyper-connected world we find ourselves unprecedentedly immersed in
processes of multimodal meaning-making.Yet this should not obscure the fact that
language itself is, and has always been, inherently multimodal (Reynolds), and that,
therefore, even “language-centred” translation practices need to engage with, and
account for, the multimodal dimension. In relation to this, it is important to bear
in mind that the notion of mode remains – for good reason – a functional, relative
one. What one community regards as a distinct mode may be subsumed within a
different mode in a different community. What functions as a mode in a certain
kind of exchange for a certain purpose may not do so when either the context and/
or the communicational purpose shifts (Kress, Reynolds, Tomalin).
This understanding of mode as a functional entity has served social semiotics
well. However, translation comes to complicate the picture – in Kress’s words:
Social semiotic theory assumes a homology of modes in the source society

matched by a homology of modes – distinct and different – in the target
society. The difficulty lies in the likely absence of such apt transpositional
resources by modes in the target society.
(p. 31, this volume)
This difficulty of mapping the modes of one culture onto another is also high-
lighted by Adami and Ramos Pinto in their chapter. In outlining a joint research
agenda for translation and multimodal studies, they point to the need to study
specific modes and meaning-making practices in order to understand their degree
of codification and how they can be shown to correspond or not with linguistic
and national boundaries. However, even as they advocate this agenda, they warn
200 Ángeles Carreres and María Noriega-Sánchez
against the dangers of over-codification. Mapping semiotic competence onto social

groups is not only an extremely complex endeavour but also one that risks bring-
ing about undesirable political outcomes (e.g., it may exacerbate certain social and
cultural hierarchies, or result in the constriction of meaning-making activity in a
given social setting).
Responding to this increasing realisation that multimodal analytical frameworks
are required, new research methods have been developed in recent years to under-
take the analysis of multimodal outputs, such as multimodal transcriptions and digi-
tal multimodal corpora (Pérez-González 2014b, 127). While this preliminary work
has already had a significant impact, further methodological advances are required.
As Adami and Ramos Pinto argue in their chapter for this volume, most research
methodologies (e.g., data-collection methods) and means of academic dissemina-
tion are still heavily logocentric. Academic research needs to develop new ways of
eliciting data and disseminating findings that go beyond verbal output and formats.
In this sense, Norwood’s chapter in this book offers a bold experiment in methodo-
logical multimodal translation. Applying a method taken from the field of drawing
to writing, Norwood’s work uses language to encourage us to move beyond the
merely linguistic, and hence enacts precisely the kind of procedural innovation
Adami and Ramos Pinto advocate. While elaborating a general research agenda
for translation and multimodal studies, Adami and Ramos Pinto also highlight the
need to strike a careful balance between large-scale quantitative investigations and
in-depth fine-grained qualitative research. Their suggestion of pursuing an integra-
tion with ethnographic approaches, which is already taking place in social semiotic
multimodal analysis, could certainly yield fruitful methodological innovations for
translation studies. In wider terms, research would be envisaged as a participatory
endeavour to include “citizen socio-semioticians” and “citizen translation scholars”
(Adami and Ramos Pinto, p. 88 in this volume).
As should be apparent from the discussions offered in several of the chapters in this
book, collaboration is often crucial to multimodal creativity and multimodal trans-
lation. Minors, in particular, emphasises this point in her analysis of works involving
music and dance. Collaboration, manifested in the convergence of industrial and
amateur subtitling practices, also constitutes a cornerstone of the performance of
citizenship, as discussed by Pérez-González in his chapter. In the products resulting
from fansubbing and other participatory media forms (e.g., the Chinese danmu), the
translators-commentators are anything but invisible; on the contrary, the desire to
make their aesthetic, ethical, or political agendas visible, thus countering established
narratives and discourses, clearly drives their interventions (Pérez-González). These
forms of multimodal experimentation open up new spaces – both aesthetic and
political – in which a range of meaning-making practices come together and inter-
act in intricate ways. The translation of multimodal ensembles such as these calls
for a revision of established translation practices and methodologies. The insights
gained from a deeper understanding of this phenomenon will, in turn, have far-
reaching consequences for the field of translator training. Translation didactics is
still mostly focused on linguistic transfer competences (Kaindl 2013, 266), but that
model is no longer adequate. Although professional translators cannot be expected

to double up as graphic designers, visual artists, dancers, software developers, and so
on, a more comprehensive approach to collaborative translation competence that
includes a multimodal awareness should be fostered.
As the foregoing summary suggests, translation studies’ gradual move away from
monomodal perspectives increases the need for more genuinely interdisciplinary
interactions. The chapters in this volume have analysed a range of specific instances
of multimodal translation, thus providing useful insights that can be applied to a
wider range of multimodal outputs, such as children’s books, comics, plays, operas,
video games, advertisements, animated poetry, spoken word performances, hyper-
textual fiction, dialogue interpreting, to name just a few.1 It is worth noting that
audiovisual translation, and in particular film subtitling, has already benefited greatly
from the application of multimodal theory (Taylor 2004, 2013, 2016; Gambier
2006; Pérez-González 2014a), and forms of assistive mediation, like audio descrip-
tion for the visually impaired and subtitling for the hard of hearing, are thriving
areas of research (Snyder 2005; Díaz Cintas, Matamala, and Neves 2010; Fryer 2016;
Remael and Reviers 2018). The social semiotic emphasis of recent thinking about
multimodality has been highly beneficial in focusing attention on the individuals or
groups who determine which signs are deployed and combined in these kinds of
communicative exchanges, especially those in which ensembles are converted into
ensembles. As Kress puts it, “[s]ign-makers and their agency as social actors are in
the foreground and with them the social environments in which they make signs”
(Kress 2010, 34).
Nonetheless, future research will need to determine how social semiotic theo-
ries of multimodality (which are based on a notion of human social interaction)
cope with the ever-increasing phenomenon of non-social (i.e., non-human-
produced) translations. Machine translation systems (such as Google Translate)
convert a source text into a target text automatically, so translation – in one
form or another – undoubtedly occurs. Yet the translation is not directly pro-
duced as a result of human social semiotic activity (though technologists were
involved in writing the underlying software). And when a machine translation
system is combined with an image-generation system, then the resulting “system
of systems” creates multimodal ensembles (i.e., combinations of written text and
visual images). Once again, while these technologies are designed by humans
who certainly have agency as social actors, the systems do not have agency in the
same sense. They make decisions that have social consequences (e.g., they output
sequences of words and images that convey meanings), but these are determined
by mathematical models such as recurrent neural networks – and the latter gen-
erate the output. Systems of this kind clearly lack consciousness and an under-
standing of the meanings they generate, whether expressed via visual designs or
linguistic structures. Nonetheless, since they are undoubtedly able to create mean-
ingful multimodal ensembles, they can and do function as social actors (in some
sense), and therefore they can have a significant impact on the social semiotic
environments in which they operate. With the burgeoning use of virtual personal
202 Ángeles Carreres and María Noriega-Sánchez
assistants like Siri, Cortana, and Alexa, such scenarios are becoming increasingly
prevalent in contemporary society, and therefore they merit greater considera-
tion in discussions of multimodal communicative exchanges when translation
provides the primary analytical focus.2
Writing in 2014, Pérez- González predicted that multimodality “is bound
to become even more central to translation scholarship in future years” (Pérez-
González 2014b, 127), and echoed Tymoczko’s claim that, as part of this process,
the field of translation studies would need to engage in the re-theorisation of
some of its principles (Tymoczko 2005, 1090). Key to that endeavour is the re-
conceptualisation of translation as a process that involves not just – and often not
even primarily – language, but also a broad range of modes and semiotic resources.
While translation studies has made incipient, but significant, attempts in recent
years to engage with the field of multimodal studies, this interest has not been
extensively reciprocal. The re-positioning of translation as a process of cultural
mediation that engages with multimodality not as a curious add-on, but as part of
its normal operation, will go a long way towards making translation – and transla-
tion studies – more relevant to multimodality. By bringing together expertise from
both fields, this book has sought to encourage the re-theorisation of translation
studies along multimodal lines, while also alerting proponents of multimodality to
the potential for interdisciplinary dialogue that the theory and practice of transla-
tion offer.
Notes
1 A recently published special issue of Linguistica Antverpiensia (Vol. 17, 2018), entitled Meth-
ods for the Study of Multimodality in Translation, edited by Catalina Jiménez Hurtado, Tiina
Tuominen, and Anne Ketola, illustrates this sweeping scope with articles that range from
webcomics and GIFs in political satire to audio description for art museums, and multi-
modal patient information guides (Jiménez Hurtado et al. 2018).
2 Multimodal machine translation is an emerging and promising field, as demonstrated
by a number of recent publications and conferences (Shah, Wang, and Specia 2016; Spe-
cia, Frank, Sima’an, and Elliott 2016; Elliott and Kádár 2017; Qian, Zhong, and Zhou
2017). The Third Conference on Machine Translation, which formed part of the Empiri-
cal Methods in Natural Language Processing Conference 2018, featured a shared task
specifically on multimodal (i.e., image + text) translation: http://statmt.org/wmt18/
multimodal-task.html. For a discussion about the task, see https://aclweb.org/anthology/
W18-6402 (last accessed 4 April 2019).
References
Díaz Cintas, Jorge, Anna Matamala, and Josélia Neves. 2010. New Insights into Audiovisual
Translation and Media Accessibility. Amsterdam and New York: Rodopi.
Elliott, Desmond, and Ákos Kádár. 2017. “Imagination Improves Multimodal Translation.” In
Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume
1: Long Papers), 130–41, Taipei, Taiwan.
Fryer, Louise. 2016. An introduction to Audio Description: A Practical Guide. London and New
York: Routledge.
Gambier,Yves. 2006. “Multimodality and Audiovisual Translation.” In Audiovisual Translation

Scenarios: Proceedings of the Marie Curie Euroconferences MuTra: Audiovisual Translation Sce-
narios, Copenhagen 1–5 May, edited by Mary Carroll, Heidrun Gerzymisch-Arbogast,
and Sandra Nauert, Denmark.
Jiménez Hurtado, Catalina,Tiina Tuominen, and Anne Ketola, eds. 2018. Methods for the Study
of Multimodality in Translation. Linguistica Antverpiensia 17.
Kaindl, Klaus. 2013. “Multimodality and Translation.” In Routledge Handbook of Translation
Studies, edited by Carmen Millán and Francesca Bartrina, 257–69. London and New
York: Routledge.
Pérez-González, Luis. 2014a. Audiovisual Translation: Theories, Methods and Issues. London and
———. 2014b. “Multimodality in Translation and Interpreting Studies. Theoretical and
Methodological Perspectives.” In A Companion to Translation Studies, edited by Sandra
Bermann and Catherine Porter, 119–31. Chichester: Wiley-Blackwell.
Qian, Xin, Ziyi Zhong, and Jieli Zhou. 2017. “Multimodal Machine Translation with Rein-
forcement Learning.” The Computing Research Repository, abs/1805.02356.
Remael, Aline, and Nina Reviers. 2018. “Multimodality and Audiovisual Translation: Cohe-
sion in Accessible Films.” In The Routledge Handbook of Audiovisual Translation, edited by
Luis Pérez-González, Chapter 17. London and New York: Routledge.
Shah, Kashif, Josiah Wang, and Lucia Specia. 2016. “Shef-multimodal: Grounding Machine
Translation on Images.” Proceedings of the First Conference on Machine Translation, 660–65.
Snyder, Joel. 2005. “Audio Description: The Visual Made Verbal Across Arts Disciplines
Across the Globe.” Translating Today 4: 15–17.
Specia, Lucia, Stella Frank, Khalil Sima’an, and Desmond Elliott. 2016. “A Shared Task on
Multimodal Machine Translation and Crosslingual Image Description.” Proceedings of the
First Conference on Machine Translation, 543–53.
Taylor, Christopher. 2004. “Multimodal Text Analysis and Subtitling.” In Perspectives on Mul-
timodality, edited by Eija Ventola, Cassily Charles, and Martin Kaltenbacher, 153–72.
Amsterdam: John Benjamins.
———. 2013. “Multimodality and Audiovisual Translation.” In Handbook of Translation Stud-
ies, edited by Yves Gambier and Luc van Doorslaer, Vol. 4, 98–104. Amsterdam: John
Benjamins.
———. 2016. “The Multimodal Approach in Audiovisual Translation.” In Audiovisual Trans-
lation Theoretical and Methodological Challenges, edited by Yves Gambier and Sara Ramos
Pinto. Special issue of Target 28 (2): 222–36.
Tymoczko, Maria. 2005. “Trajectories of Research in Translation Studies.” Meta 50 (4):
1082–97.
INDEX
accented cinema 101; see also diaspora film; Bracquemond, Félix 146
Naficy, Hamid Browning, Robert 123
action theory 54 Burnard, Pamela 160, 164
activist subtitling see subtitling
Adami, Elisabetta 11, 17 – 18, 55, 95, captioning see digital culture
199 – 200 Carroll, Lewis 4
adaptation see translation censorship 10
advertising 4 – 5, 17, 56, 80, 83, 85, 201 children’s literature 4, 9, 54, 82, 135, 201
affinity spaces 105 China 107, 111, 187
affordances 4, 13, 15 – 16, 26, 28 – 33, 37 – 8, Chinese Whispers 24 – 7
45, 81, 88, 96, 104 – 8, 110, 137, 139, choreomusicology 158 – 66, 170
145, 150 cinema of attractions 98, 103
Albright, Daniel 159, 162 Cocteau, Jean 173 – 4
alif 127 code see semiotics
anime 73 – 4, 77, 82, 86, 90n1, 104, 107 coherence 14, 32 – 3, 74
anthropology 9, 89 Cold War, the 7, 61
Apollinaire, Guillaume 19, 140 – 53 comics 10, 16, 71, 78, 82, 136, 201
Arabic see language Commedia 18, 119 – 20, 124 – 5, 128 – 9
artificial intelligence see technology communication: channels of 12, 51,
assistive mediation 201 53; codification of 75 – 7, 85, 90;
audiovisual translation see translation communicative aim 55, 72, 100,
108; in films 96 – 7, 102; intercultural
ballet 3 – 4, 19, 140, 162 – 75 11, 15, 71 – 2, 75, 89, 104, 160 – 1,
Bannerman, Henrietta 163, 174 164; intermodal 16, 58, 78 (see also
Bassnett, Susan 6 – 10, 53, 160 multimodality); interpersonal 28, 80,
Beharry, Shauna 193 102 – 3; and language (see language);
Berger, John 181 limits of 136; in literature 117, 124,
Berman, Antoine 122 134 – 6; materiality of 95, 104; and
Bezemer, Jeff 5, 12 – 13 meaning 1, 3 – 5, 12, 28, 160 (see also
“black” music see music meaning); monomodal 11, 14, 94 (see
Blake, William 135, 145 also monomodality); multimodal 3, 5,
blindness 187 – 8, 190 11 – 13, 30, 37, 45, 51 – 6, 59, 64, 77, 110,
blues, the see music 135 – 6, 142, 160, 163, 198 – 9 (see also
Boethius 2 multimodality); nonverbal (see modes);
Index 205
purpose of 51; and social interaction discourse analysis 5, 56

12 – 13, 27 – 8, 31 – 2, 36, 45, 51, 71, Divine Comedy, the see Commedia
75, 103 – 4, 201; theories of 25, 36, domesticating translations see translation
51, 59 – 61; and translation 1 (see also studies
translation); visual 89 drawing 25, 34 – 5, 38, 42, 47, 179 – 97
computer games see technology Dryden, John 122 – 3, 136
context: communicative 57; cultural 2, 5 – 6, dubbing see subtitling
9 – 10, 15, 59, 55, 62, 64, 72; disciplinary Dufy, Raoul 19, 144, 146, 150 – 1
6; linguistic 1, 10; multimodal 3, 5 Dürer, Albrecht 146
(see also multimodality); nonverbal 71;
re-contextualisation 38; social 8, 12, Eastwood, Clint 141
50 – 2, 55, 62, 64; temporal, medial 50 – 1, Eco, Umberto 52, 195
58; mediation 57 ekphrasis 6, 13, 139
co-text see text types e-literature see digital culture
crowdsourcing 10, 111 emojis see digital culture
cultural mediation 199 ensembles: audiovisual 111; cultural context
culture: digital (see digital culture); and of 18, 96, 138; decodification of 96;
globalisation 4; and language (see definition of 13, 135; dimensionality
language); and modes 135, 140; and of 138 – 9, 142 – 4, 148 – 9, 152, 154;
multimodality 72 – 5; national 2, 29, 88; and films 96; and machine translation
and semiotics (see social semiotics); and 201; and meaning 32 – 3, 38 – 9, 201; as
thought 118; and translation 6, 8 – 12, 15, multimodal complexes 11 – 13, 18 – 19,
51 – 5, 76 – 7, 97, 160 – 3, 166, 172 – 5; see 33 – 4, 46, 73, 79, 135 – 8, 147 – 55, 200;
also context orchestration of 14, 135, 148, 152; as
texts 73; translation of 19, 38 – 9, 74,
dance 3, 6, 19, 28, 31, 63, 121, 154, 158 – 75, 83 – 5, 135 – 8, 142, 147 – 54, 200 – 1 (see
200 – 1 also translation)
danmu 18, 96, 106 – 11, 200 epistemology 31, 37 – 8, 45, 74, 77, 86
Dante: the character and the man 117; equivalence see translation studies
De vulgari eloquentia 118, 120, 125, 129; ethics 8, 10, 106, 200
original language 117 – 19, 126 – 7; and ethnocentrism 9
translation 18, 117 – 31, 153; see also eye(s) 87, 97 – 9, 137, 168, 170, 179, 181 – 5,
Commedia 188 – 95
Debussy, Claude 3, 16, 19, 166 – 9
denotation 74, 136, 142, 154 faithfulness see translations studies
Derrida, Jacques 57, 118, 122 – 3, 128, 136, fansubbing see subtitling
140, 180 – 1, 188 – 91 feminism 6, 9 – 10, 191; see also translation
de Saussure, Ferdinand 13 studies
Diaghilev, Serge 3, 166, 173 feminist translation see translation studies
dialects see language film sound 95 – 6, 98 – 101
diaspora film 193 film studies 51, 82, 98
Dickens, Charles 9, 135 Foucault, Michel 52, 147, 150
diegesis 81 – 2, 96 – 111 French see language
digital culture: cameras 57, 74, 100, 146,
188; captioning 4, 102 – 3, 106; and Gauguin, Paul 146
corpora 87, 200; democratic nature of gender 2, 6, 10, 53, 56
18, 95 – 6, 105; e-literature 4, 138 (see also Genette, Gérard 6, 102
poetry); emojis 4; hypertextual fiction genre: and culture 57, 60 – 2, 64 – 5;
201; memes 4, 11, 80, 85; participatory definition of 57 – 8, 65, 199; in film 97,
media 103 – 11, 200; social media 4 – 5, 11, 102, 104; in font 141; and language 120;
103, 106; and subtitling (see subtitling); of multimodal texts 6, 14, 39, 43, 82 – 3;
virtual communities 104, 109 in music 60 – 3; and national codification
digital technologies see technology and circulation 85 – 7; and plurality 192;
digital texts see text types and translation 32, 58 – 9, 63 – 5, 84, 148,
digressions 179 – 80 162, 172
Dionisi, Giovanni Iacopo 126 – 7, 129 German see language
206 Index
gesture: artistic 175; and dance 169 – 70, 172; 160, 180; German 3, 14, 18, 57 – 8,
and equivalence 163 – 5, 170; gestural 60 – 1, 63, 122, 128 – 9, 139; Italian
analysis 71; gestural arts 164 – 70; gestural 119 – 20, 125 – 8, 130 – 1, 153, 139;
interplay; and meaning 74 – 6, 80, 88, 161 language change 120 – 1; Latin 14 – 16,
(see also modes); musical 161 – 3; non- 120, 122 – 3, 126 – 8; lexis 15, 31, 140,
gestural 5; physical 159 – 60, 163; and 143; and meaning making 119, 130 – 1;
sound 171 metakinetic 163; metalinguistic 163;
globalisation 3 – 4, 10 – 11, 71, 95, 173 as a mode 14, 16 – 17, 65, 122, 199
Goodman, Nelson 121 (see also modes); multilingualism 11,
Gorbman, Claudia 168, 175 75 – 7, 86, 119, 160; and multimodality
(see multimodality); and neuroscience
Halliday, Michael 11 – 13, 28, 32, 72 89; original language 196n5 (see also
haptic visuality 193 – 4 Dante); relation to the world 72 – 3;
Hazlitt, William 134 – 5 register 12, 74, 119 – 20; and relevance
Holmes, James 8 theory 86; as a semiotic exception 76,
Humphrey, Doris 170 – 1 84; Spanish 3, 129, 139; and speech
hypertexts see texts types (see speech); standardisation of 75 – 6,
83, 90, 123; style 74, 108, 119, 148;
iconology 6 textual 159 – 60, 164; and translation
ideology 6, 53, 99 – 100 (see translation); verbal 78, 87 – 8, 127,
image see modes 159 – 60, 163 – 4; writing (see writing)
innovation see technology Latin see language
intelligibility 180 Lefevere, André 6, 8, 53 – 4
interculturality 11, 15, 71 – 2, 75, 89, 104, lexis see language
160 – 4, 166, 172, 175 linguistics 7, 9, 29, 44, 53, 73, 88 – 90, 118
interdisciplinarity 11, 51, 64 – 5, 161, 201 – 2 linguistic standardisation see language
intermediality 50, 56, 59 – 60, 64 literary criticism 5, 135, 137, 141 – 2, 145
intermodality see translation livres d’artistes 19, 96, 136, 144, 146
interpreting see translation localisation see translation
intersemiotic translation see translation logocentrism 14, 78, 142, 147, 160
intertitles see subtitling
Italian see language machine translation see technology
Mallarmé, Stéphane 3, 16, 166 – 9
Jakobson, Roman 7, 31, 46, 49, 53, 58 – 60, Marks, Laura U. 191, 193
65, 65n1 Martinet, André 7
Jewitt, Carey 5, 12 – 13, 52, 54 – 6 meaning: and agency 28, 38, 54, 72, 90,
Jordan, Stephanie 158 – 9, 170 – 2 96, 201; communication of 1, 3 – 5,
Joyce, James 153 – 4 12, 14, 52; (re)constitution of 27, 35,
37 – 9, 41 – 4, 57, 136, 139 – 40, 143, 145,
Kaindl, Klaus 7, 15 – 18, 39, 198 – 200 147 – 8, 154, 198 – 9; construction of 10,
Kloepfer, Rolf 51 12 – 13, 24 – 47, 51 – 2, 59, 160, 165 – 6,
Kosminsky, Peter 101 168, 173 – 5; forms of meaning making
Kress, Gunther 5 – 7, 13 – 18, 50 – 2, 54 – 6, 59, 2, 5, 12, 14, 18, 34, 55, 166, 199 – 200;
78, 86, 96, 99, 106, 118 – 19, 136, 139 – 40, intergeneric 60 – 2; intermedial (see
142, 160 – 1, 165, 198 – 9, 201 intermediality); intermodal 5, 15 – 16, 58,
Kuhn, Thomas 50 60, 78, 81, 87, 96, 111, 139, 148, 151 – 2,
199; intramodal 6, 15 – 16, 60, 142 – 4;
Lacoue-Labarthe, Philippe 192 meaning of 1; and metaphor 30, 34 (see
language: Arabic 82, 101, 126 – 7; and also metaphor); and multimodality (see
the arts 124, 159; body language 121; multimodality); potential for 24, 26,
Chinese 4, 110; and communication 28 – 9, 30, 32 – 3, 38, 57 – 8, 72, 74, 76,
12, 24, 27 – 9, 75, 199; and complexity 106, 141; resources for 14, 17, 26 – 34,
theory 86; and cultural context 2, 37, 39, 41, 45 – 6, 51 – 2, 56, 71 – 90, 94,
6, 12, 15, 71 – 4, 153 – 4; dialects 109, 96, 111, 128, 142, 148 (see also modes);
119 – 20; French 1 – 3, 6, 14 – 16, 60, transduction of 5, 15 – 16, 26, 37, 46,
123 – 4, 128 – 9, 139, 143 – 4, 149, 151, 59, 142, 151, 165, 199; transference
Index 207
of 3, 5 – 6, 8, 19, 53 – 65, 165, 175, 198; multilingualism see language

transformation of 5, 15, 19, 36, 42, 45 – 6, multimediality 18, 50 – 1, 55
59, 64, 94 – 5, 142, 165; translation of multimodal complexes see ensembles
(see translation); transmutation of 58, 60; multimodality: and the arts 175; and
transposition of 8, 17, 27 – 47, 59, 136, audiovisual texts 94 – 111; and
142, 147 – 8, 150, 154, 172, 198 – 9 communication (see communication);
medium 18, 50 – 1, 56 – 65, 85 – 7, 98, 118, and culture (see culture); and digital
161, 163, 170 – 1, 199; see also translation technology (see digital culture;
memes see digital culture technology); dimensionality of ensembles
metaphor 15, 28, 30, 34, 42, 44, 46, 86, 122, (see ensembles); the formal analysis
140, 158 – 9, 163, 170 – 2, 184, 198 of 142 – 52; framing 36 – 7, 100, 139;
metre see poetry and intermodality (see translation);
Minh-ha, Trinh 100, 103 and intramodality (see translation);
Minors, Helen Julia 6, 19, 47, 55, 78, 140, and language (see language); and
198, 200 linguistics (see linguistics); and literary
Mitchell, W. J. T. 121 translation 8 – 9, 18, 122 – 4, 136 – 8,
mobile phones see technology 144 – 5, 148, 153; and multilingualism
modes: affordances of (see affordances); (see language); multimodal complexes
blending of 163, 167 – 8; dance as 3, (see ensembles); nonverbal resources
6, 19, 28, 31, 159 – 61, 85, 168 – 75 (see (see modes); orchestrating ensembles
also dance); definition of 12 – 14, 19, 29, (see ensembles); processes of design,
118 – 19, 137 – 40, 161, 199; development production, and distribution 51 – 2,
of 29, 31 – 2, 86; disaggregation of 144, 58, 73, 85; social semiotic theories of
147, 150, 154; and discourse 12 – 13, (see social semiotics); and semiotics
50 – 2, 55, 58, 159 (see also discourse (see semiotics); and subtitling (see
analysis); distal 137, 139 – 40, 151; subtitling); and transdisciplinarity (see
drawing as 179 – 97 (see also drawing); transdisciplinarity); transcriptions 44, 46,
ensembles of (see ensembles); fonts 109, 165, 200; and transduction 5, 15 – 16, 26,
137, 139 – 42; image as 13 – 15, 25 – 7, 37, 46, 66n16, 165 (see also meaning);
31, 33, 37, 41, 47, 71 – 90, 128, 142, and transformation (see meaning); and
145, 148, 152; latent 137, 151; language translation 24, 71, 75, 83 – 4, 88 – 9,
as (see language); music as (see music); 122 – 31, 180 (see also translation);
non-gestural (see gesture); non-linguistic transmodality 59, 122
17 – 18, 54, 121; nonverbal 4 – 5, 55, Munday, Jeremy 56
72 – 3, 76, 78 – 84, 87 – 90; not countable music: as accompaniment 150 – 2;
119; ontology of 16, 37 – 8, 45 – 6, 95 – 6, arrangement 63, 165; “black” 50, 61 – 2;
136, 138, 142, 154, 199; and partiality blues 62; covers 165; jazz 195n2; as a
27, 32 – 3, 35 – 6, 185; prioritising mode 3, 6, 11, 13, 16, 19, 13, 46, 55, 58,
in translation 145, 147 – 52, 154; 61 – 5, 74 – 7, 88, 118, 135 – 8, 140, 150 – 4,
proximal 137, 139, 144, 150 – 1, 153 – 4; 158 – 75, 200 (see also modes); music-
relationships between 16, 51, 58, 78, dance works 19, 158, 160 – 6, 169 – 70,
81 – 2, 87, 121, 137 – 9, 145, 147, 153, 160, 172 – 3, 175, 200; musicking 164; opera
163 – 5, 170; and semiotic knowledge 39, 47, 57 – 60, 166, 201; performed vs.
75, 77; as semiotic resources 5, 12 – 15, written 150 – 2; popular 18, 61; rock
17, 19, 28 – 32, 34, 41 – 2, 51 – 2, 57, 65, ‘n’roll 18, 49 – 50, 60 – 3; Schlager 18, 60 – 5;
65n3, 72 – 6, 85, 153, 199, 202; singing transcriptions of 46, 165; visualisation of
as 58, 140, 150 – 2; sound as 4 – 5, 30, 165, 170 – 2, 174 – 5; “white” 50, 61 – 2
39, 63 – 4, 78, 80, 82, 121; speech as (see musicology 51, 75, 89, 163
speech); submodes 55, 62, 66n9, 108 – 9,
119, 150 – 4; thought as 118; touch as Naficy, Hamid 98, 100 – 1, 103
39, 41 – 2, 181 – 2, 184, 188 – 91, 193; Nancy, Jean-Luc 179 – 80, 189, 192
visual 55, 122; writing as (see writing); neuroscience 89, 121
monomodality 6, 14, 53 – 4, 94, 137, 141, Newfield, Denise 59
143, 147 – 8, 201 Nida, Eugene 7, 136
Morris, Mark 171 – 2 Nijinsky,Vaslav 3 – 4, 19, 166 – 70
Mounin, George 7 Norwood, Tamarin 5, 19, 33, 78, 200
208 Index
objectivity 98 – 100, 191 Rees, Marc 173 – 4

on-screen titles see subtitling relevance theory 86, 89
ontology 16, 18, 31, 37 – 8, 45 – 6, 95 – 6, Reynolds, Matthew 11, 18, 24, 55, 153, 161,
99 – 103, 105 – 11, 136, 138, 142, 154 198 – 9
opera see music rhyme see poetry
original language see Dante rock ‘n’roll see music
Romanticism 7, 58, 135, 192
page (surface of) 179 – 95 Romero-Fresco, Pablo 101
Palumba, Giovanni Battista 146 Rossetti, Dante Gabriel 1 – 3, 6, 14 – 15
panaesthetics 162
paratexts see text types Sakai, Naoki 123
Peirce, Charles Sanders 13, 50 Satie, Erik 19, 173 – 4
pencil: marks on a page 180, 182 – 5, 188; tip Schlager see music
of 179 – 83, 186, 188 Scott, Clive 144, 152 – 3
Pérez-González, Luis 5, 11, 17 – 18, 95–6, semiotic resources see modes
98, 100 – 1, 103 – 4, 106, 109, 200 – 2 semiotics: applied 52; arbitrariness of the
performance: audience of 174; and sign 13, 35, 52, 72 – 3; and code 26,
citizenship 18, 96, 200; and drawing 195; 51, 72, 96, 100, 108 – 9, 127, 131, 174;
of films 99; and meaning construction dynamic nature of 51 – 2, 72, 75 – 7,
160, 162, 164, 171, 175 (see also 84 – 6, 140, 152, 158, 194, ; iconic
meaning); multimodal impact of 58; signs 13, 72, 121; indexical signs 13;
of music 150 – 1; and rhetors 18, 96, intersemiotic translation (see translation);
110 – 11; spaces 161; spoken word 122, and linguistics 29, 44 (see also linguistics);
159, 201; as text 3, 16 and multimodality 49 – 50 (see also
performing arts 158 – 9, 161 multimodality); semiosis 12, 34, 45,
phenomenology 152, 191 – 2 59, 86 – 9; signs 13, 28, 30 – 45, 49 – 52,
Picasso, Pablo 173 56, 73 – 90, 108, 121, 124, 127, 130,
plays 54, 122, 136, 201 201; signifieds 30, 33 – 6, 43 – 5, 72 – 4,
poetry: adaptation of 3, 5 – 6, 19; animated 102; signifiers 29 – 36, 38, 41 – 5, 73 – 4,
153, 201; and ballet 140, 166 – 9 (see 127; structuralist 51, 72; see also social
also ballet); e-poems 138; and film 100; semiotics
and fonts 140; and knowledge 193; as sensory planes 159
literature 1, 118, 134 – 6; metre 60; and Shakespeare, William 137 – 8
multimodality 5, 120, 122, 135 – 6, 139, sight 180 – 95, 121
169 (see also multimodality); rhyme 2, signifieds see semiotics
6, 14, 137; translation of 1 – 6, 14 – 19, signifiers see semiotics
47, 120 – 4, 140 – 54, 166 – 9 (see also signs see semiotics
translation);Vietnamese 100 singing see modes
Posner, Roland 51 social agency 51, 54
postcolonialism 9 – 10 social media see digital culture
post-impressionism 146 social sciences 9, 11
post-structuralism 9 social semiotics: and cultural context 73,
Poulenc, Francis 150 – 2 84, 88 – 9; the development of 13, 52, 85,
Pousette-Dart, Richard 149 88; interpersonal 80; levels of meaning
power 6, 10, 39, 52 – 3, 56, 99, 102, 117, 119 in 80; and the making of signs 36,
Prager, Emily 187 45; representational 80; and semiotic
pragmatics 17 knowledge 84 (see also semiotics); textual
Presley, Elvis 18, 61 – 3 80; and theories of multimodality 12 – 13,
psychology 5, 51, 82, 89 27, 72 – 4, 87 – 8 (see also multimodality);
publishers 142, 148, 154 and translation 27 – 8, 84, 89 – 90, 198 – 9
Purcell, Henry 171 (see also translation)
sociolinguistics 88, 90
race 6, 53, 172 sociology 51, 88 – 9
Ramos Pinto, Sara 11, 18, 55, 95, 199 – 200 somatic experience 19, 158 – 61, 166
re-constitution of meaning see meaning Sorrentino, Paolo 102
recording technologies see technology sound see modes
Index 209
source 1 – 3, 6 – 8, 19, 53 – 4, 58, 63, 78 – 9, 81, temporal arts 158, 162, 171 – 2, 175
85, 172, 179 – 80 text types: alterity of 9; and context
Spanish see language 8, 73 – 90; conventions of 53, 61;
spatial logic 30, 33, 37, 43, 46, 99, 105 co-text 73 – 5, 77, 89; digital 15, 95 – 6,
speech 5, 11 – 14, 16, 19, 26 – 33, 37, 39, 103 – 11; functional 55; hypertexts 88,
41 – 6, 58, 72 – 8, 88, 118 – 20, 126, 138 – 9, 95, 153, 201; monomodal 6 (see also
144 – 5, 151 – 4, 161, 169 monomodality); multimedial 18, 50 – 1,
Sterne, Laurence 4 – 5, 153, 194 55; multimodal 3 – 4, 6, 11, 16, 18 – 19,
Stravinsky, Igor 19, 166, 169, 180 55, 59, 64, 71, 73, 75, 78 – 9, 82, 85, 87,
submodes see modes 95 – 6, 101, 111 (see also multimodality);
subtitles see subtitling paratexts 6; production of 18, 54 – 6;
subtitling: aesthetics of 10, 94 – 111; as source text (see source); spoken 16,
activism 104 – 6; as an amateur practice 124 – 5, 137, 139; target text (see target);
79, 81 – 2, 96, 103 – 11, 200; and technical 17, 55, 85; verbal 54, 60,
censorship 10; and commercialism 79, 159 – 60; written 16, 38, 97, 108, 111,
96 – 100, 111; as creative practice 101, 112 – 14, 137 – 8, 141 – 2, 144, 150 – 3, 201
104, 111; decotitles 101; diegetic 103; theatre 54, 57 – 8, 123
and digital technologies (see digital Thornton, Big Mama 62
culture); vs. dubbing 78, 81, 96 – 7; Tomalin, Marcus 11, 16, 19, 55, 78, 96,
dynamic nature of 83; ethnographic 198 – 9
approaches 101; and eye-tracking 97; touch see mode
fansubbing 10, 78, 81 – 2, 104 – 6, 200; Tower of Babel 119 – 20, 127 – 8
functional units of 56; for the hard of transcriptions see multimodality
hearing 81, 201; development of 18, 79, transdisciplinarity 11, 18, 72, 89, 94
96 – 100; industrial 18, 97 – 100; intertitles transduction see multimodality
10, 96, 103; on-screen titles/text 10, transference see meaning
97, 101 – 2, 106 – 7, 201 (see also danmu); transformation see multimodality
ontology of 94 – 111; and multimodality translation: as an amateur practice 10,
5, 56, 78, 81, 94 – 111, 122, 201 (see also 78 – 82, 103 – 11, 200; and adaptation
multimodality); as professional practice 58 – 9, 64, 79, 165 – 6 (see also poetry;
81, 100, 106; as social semiotic practice translation studies); artistic 3, 94,
(see social semiotics); and sound 161, 164 – 6, 168 – 9; audiovisual 5,
synchrony/asynchrony 97 – 103, 105, 109; 10, 16, 49, 53 – 4, 78, 81, 94 – 5, 100,
suture 99; and translation 18, 96, 78, 81, 162 – 3, 201; context of 8 – 9, 15, 74,
83, 94 – 111 (see also translation) 77 – 8; classification of (see translation
sufism see alif studies); and cultural identity (see
surface(s) 179 – 95 culture); definition of (see translation
suture subtitling see subtitling studies); domesticating approaches to
synchrony see subtitling (see translation studies); from drawing
synthesising technologies see technology to writing 179 – 81; equivalence in 2,
7 – 10 (see also translation studies); the
target 1 – 3, 6 – 8, 19, 53, 59, 78, 80, 172, 180 ethnography of 88 – 9, 100 – 1, 200;
target culture 54, 64 faithfulness (see translation studies);
Taylor, Christopher 94 – 5, 100 foreignising approaches to (see translation
technology: artificial intelligence 11; and studies); and gender (see translation
cinema 99 – 101; computer games 4; studies); impossibility of 3, 195; (inter)
digital 3 – 4, 10 – 11, 56, 71, 95 – 6, 103 – 7, cultural 81, 160, 165, 172 – 5 (see also
110 – 11; human-computer interfaces 11, interculturality); interlingual 7, 59 – 60,
109, 201; innovation 3, 11; and literature 65; intermodal 16, 60, 78, 81, 87,
138, 144; machine translation 7, 53, 201, 139, 148, 151 – 2, 199; interpretative
202n2; and medium 56; mobile/smart possibilities 3, 38, 45, 125, 158 – 9, 166;
phones 4, 119; and music 164; recording and interpreting 38, 56, 66n5, 79 – 83,
3, 44, 63, 96, 99 – 101, 105, 109, 150, 122 – 30, 150, 162, 201; intersemiotic 7,
152, 169; and semiotics 47, 106 – 7 (see 31, 46 – 7, 60, 65, 76, 165; intralingual
also semiotics); synthesising 96, 103 – 6, 7, 46, 59 – 60; intramodal 15, 60, 142 – 4;
109 – 11; video games 54, 201 and language 7 – 10, 16, 18, 29, 31,
210 Index
76 – 8, 81 – 3, 122 – 3 (see also language); 57, 59, 163; and multimodality 16 – 18,
and linguistics 73 (see also linguistics); 49, 53 – 8, 60, 64 – 5, 94, 163, 198, 200,
localisation 4, 75, 79 – 83; machine 202 (see also multimodality); and music
translation (see technology); multimodal 161 – 3 (see also music); post-translation
(see multimodality); myopic 180 – 1, 189; studies 11; sociological 6, 52 – 3, 59; the
para-translation 58; and perfection 38, translation game 24 – 6; translator training
118, 181, 195; and phenomenology 8 – 9, 65, 72, 77, 81, 84 – 5, 122, 200
(see phenomenology); post-translation translator training see translation studies
11, 58; the practice of 1, 4 – 5, 8 – 11, transmediality 59
14, 18, 29 – 31, 36 – 8, 44, 57, 61, 65, transmutation see meaning
72, 77, 79, 81 – 7, 90, 103 – 11, 122 – 4, transnationality 11, 18, 71 – 2, 75 – 7, 83 – 6
144, 148, 158, 181, 199, 200, 202; as transposition see meaning
re-writing 127; and modes (see modes); transsensorial perception 169
as a professional practice 81 – 2, 84;
representation of 123 – 4; and semiotics Ungaretti, Giuseppe 15
72, 76, 88 – 9 (see also semiotics); sensory
165 – 70, 175; source (see source); and van Leeuwen, Theo 5, 13, 50 – 2, 54 – 6, 96,
subtitling (see subtitling); transmission of 99, 105 – 6
95, 125; transmodal (see multimodality); Venuti, Lawrence 6, 9, 136, 160, 180
transportation/transposition of meaning video games see technology
(see meaning); theories of (see translation Villon, François 1 – 2, 6, 14 – 15
studies) virtual communities see digital culture
translation studies: action-oriented 59; and visualisation see music
adaptation studies 64; classification of von Flotow, Luise 6, 10
6 – 9, 14; definition of 7 – 8, 24, 31 – 2,
39, 42, 45, 47, 77 – 8; descriptive 59; Wei, Wang 3
development of 3, 6 – 17, 49 – 58, 71, 79, “white” music see music
152, 201; domesticating translations 6, woodcuts 19, 135, 144 – 6, 148, 152
9, 90, 146; faithfulness 8, 36, 83, 188, writing 2, 5, 14, 16, 19, 26 – 33, 37 – 42,
194; feminist theories 10; foreignising 46 – 7, 71 – 8, 88, 119 – 20, 126, 136, 139,
translations 2, 6, 64, 90n6; formal and 141 – 7, 153 – 4, 179 – 97
dynamic equivalence 7, 53 – 4, 58, 73,
75 – 6, 78 – 9, 82 – 3; and language 18, Zukofsky, Celia and Louis 123

Boria Et Al (2020) Translation and Multimodality

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Boria Et Al (2020) Translation and Multimodality

Uploaded by

Copyright:

Available Formats

TRANSLATION AND

Monica Boria is Affiliated Lecturer in the Department of Italian at the University

Ángeles Carreres is Senior Language Teaching Officer in the Department of

María Noriega-­Sánchez is Senior Language Teaching Officer in Spanish at the

Marcus Tomalin is Fellow at Trinity Hall, Cambridge, and a Senior Research

1 Transposing meaning: translation in a multimodal semiotic

2 A theoretical framework for a multimodal conception of

3 Meaning-­(re)making in a world of untranslated signs:

4 From the “cinema of attractions” to danmu: a multimodal-­

5 Translating “I”: Dante, literariness, and the inherent

6 The multimodal dimensions of literature in translation 134

7 Translations between music and dance: analysing the

8 Writing drawingly: a case study of multimodal translation

Beyond words: concluding remarks 198

0.1 The squiggle flourish 4

Elisabetta Adami is University Academic Fellow in Multimodal Communication

Monica Boria is Affiliated Lecturer in the Department of Italian at the University

Ángeles Carreres is Senior Language Teaching Officer in the Department of Span-

Klaus Kaindl is Professor of Translation Studies at the University of Vienna, Austria.

meaning (-­making) in contemporary environments, and was instrumental in develop-

María Noriega-­Sánchez is Senior Language Teaching Officer in Spanish at the

Luis Pérez-­González is Professor of Translation Studies and Co-­director of the

Matthew Reynolds is Professor of English and Comparative Criticism at Oxford,

The ensuing chapters were first presented at a conference organised by CCiT

Raoul Dufy, 1877–1953 and Guillaume Apollinaire, 1880–1918, “Le Droma-

1. Translation in a multimodal world

Dictes moy ou, n’en quel paÿs,

If these lines of medieval French function as a source-­language text, then their

Tell me now in what hidden way is

Where’s Hipparchia, and where is Thais,

It is immediately apparent that “meaning” (at least in the rudimentary sense of

binary gender boundaries and creating ambiguities. Should Rossetti’s handling

FIGURE 0.1 The squiggle flourish

version of a French poem) (Kress 2010, 124–31). At present, though, there is no

2. Trends in translation studies

translation should be recognised as a field in its own right. Their excitement at

a view of literature as a complex and dynamic system; a conviction that there

govern the production and reception of translation, in the relation between

Prompted by similar interests, Mary Snell-­Hornby theorised an integrated approach

3. The rise of multimodality

1 Meaning is made with different semiotic resources, each offering distinct

Therefore, ensembles are said to be “orchestrated” by at least one meaning-­maker

This observation raises several important issues. When it is recognised, for

moved across modal boundaries. For example, compare Giuseppe Ungaretti’s

It is not possible (other than metaphorically) to analyse the image in terms of

4. An overview of this book

therefore it is not clear how it should be pronounced. These translation-­related

Boethius. 2001 [524]. Consolation of Philosophy. Translated by Joel C. Relihan. Indianapolis

1. Preamble: the question of translation in the

2. A slightly odd take on translation: “Chinese Whispers”

FIGURE 1.1 The translation game

the Master of Ceremonies (MC) of the evening to participate in a game of “Chi-

difference of personal “interest” of the participants, leading to difference in focus,

3. A brief sketch of a social semiotic theory and its tools

a conceptual/theoretical frame capable of dealing with changes in the making of

As a consequence, modes – together with their elements and the relations of

The recognition of a wide range of modes, and an awareness of the changed

4. The partiality of the semiotic resource of modes

available to realise coherence include pronouns, (other) lexical items, syntactic

5. Sign-­making: interest, criteriality, aptness

FIGURE 1.2 Drawing by a three-­year-­old: “this is a car”

chunkily visible wheels, especially at the three-­year-­old observer’s height), we

such as professional or other considerations and demands, which shape attention

6. Communication in the frame of social semiotic theory

María Noriega-Sánchez is Senior Language Teaching Officer in Spanish at the

3 Meaning-(re)making in a world of untranslated signs:

4 From the “cinema of attractions” to danmu: a multimodal-

6 The multimodal dimensions of literature in translation 134

Beyond words: concluding remarks 198

0.1 The squiggle flourish 4

meaning (-making) in contemporary environments, and was instrumental in develop-

María Noriega-Sánchez is Senior Language Teaching Officer in Spanish at the

Luis Pérez-González is Professor of Translation Studies and Co-director of the

1. Translation in a multimodal world

If these lines of medieval French function as a source-language text, then their

2. Trends in translation studies

Prompted by similar interests, Mary Snell-Hornby theorised an integrated approach

3. The rise of multimodality

Therefore, ensembles are said to be “orchestrated” by at least one meaning-maker

4. An overview of this book

therefore it is not clear how it should be pronounced. These translation-related

1. Preamble: the question of translation in the

2. A slightly odd take on translation: “Chinese Whispers”

3. A brief sketch of a social semiotic theory and its tools

4. The partiality of the semiotic resource of modes

5. Sign-making: interest, criteriality, aptness

FIGURE 1.2 Drawing by a three-year-old: “this is a car”

chunkily visible wheels, especially at the three-year-old observer’s height), we

6. Communication in the frame of social semiotic theory

7. The transposition of the “undocumented” and of

2. Multimodality as academic rock ‘n’ roll

phenomenon by combining and re-evaluating previously researched aspects, which

3. On the (bumpy) road to multimodal translation studies

communicative-functional aspects through medial technical-social implications to

4. Mode, medium, and genre in the translation of

• What are the (multi)modal culture-specific textual conventions?

2. Issues for a way forward in social semiotic research

2.1 The “context” vs. “co-text” approach

2.2.1 Language as an exception: National codification and

2.3 Meaning-making in a world of untranslated signs

3. Issues for a way forward in translation

in the target context when co-occurring with non-mediated nonverbal resources.

4. Conclusion: A joint research agenda

picture of generalizable significance for TS as a whole and day-to-day translation

4.2.1 Distribution of semiotic knowledge: modes, media, genres,

all sign-making practices that we encounter, as well as on actual translations and

2. Subtitling in the era of recording technologies

example 6.1 in Pérez-González 2014a, 188–90), it is generally assumed that view-

3. Subtitling during the period of ontological transition

4. Subtitling in the era of synthesising technologies