Semiotics in Media

Frank Nack (ISLA - UvA)

Semiotics - Definition
Semiotics is the study of sign processes (semiosis), signs and
symbols, or signification and communication. It is usually
divided into the three following branches:
Semantics: Relation between signs and the things to
which they refer
Syntactics: Relations among signs in formal structures
Pragmatics: Relation between signs and their effects on
the people who use them
A fragment of semiotic history
Image by Gran Sonesson: The semiotic function and the genesis of pictorial meaning,
http://www.arthist.lu.se/kultsem/sonesson/ImatraCourseTx1.html
Context of pictorial semiotics
Image by Gran Sonesson: The semiotic function and the genesis of pictorial
meaning, http://www.arthist.lu.se/kultsem/sonesson/ImatraCourseTx1.html
Communication or the creation of meaning
Fundamentals of Communication
Signal Source Signal Observer
p c
p = perceive
c = conceive
Communication chain
Expedient Percipient
p c p c
Communication
is a process of transferring information from one entity to another
is sign-mediated interaction between at least two agents
both agents share a repertoire of signs and semiotic rules.
Communication chain
p c
p c
Sign Repertoires
Description chain
p c p p
Reality
Description
Query chain
Describer Archiver Enquirer
Communication semantics
Task
knowledge
Expedient
structures
Shared cultural
Shared social
structures
Personality
attributes
Organismic
attributes e.g.
male, adult, etc.
Outside cultural
attributes
Outside social
attributes
Percipient Media
Expression
knowledge
Thematic
knowledge
Meaning
Media semiotics
Text Image Video Audio Tactile
Semiotic concepts
14
The Sign - Saussure
Concept
Mental Perception
of Media
SIGN
Signifier
Signified
beauty
15
The Sign - Peirce
SIGN
Representamen
(symbol)
Object
(referent)
Interpretant
(thought)
active process
physical or
mental entity?
psychological or ontological status?
referred to on a particular occasion?
typical or ideal representation?
16
The Sign - Arbitrariness
Conventional in the Saussurean sense means
that the relationship between the signifier and
the signified dependents on social and cultural
conventions.
The Saussurean model supports the notion of
arbitrariness of the sign by proposing the
autonomy of language in relation to reality. Its
emphasis on internal structures within a sign
system assumes that language does not
reflect reality but rather constructs it.
Semantics Index, Icon Symbol (Peirce)
Icon A sign which represents its object
mainly through its similarity with some
properties of the object, based on the
reproduction of perceptual conditions.
Symbol A sign with an arbitrary link to its
object (the representation is based
on convention).
Index A sign which represents its object by
an inherent relationship.
Semiotics Text, Image, Video
Text a sign system
Approaching text
A (Alphabet)!
Saussure, Ferdinand de - (1857-1913)
Swiss linguist. His Course in General
Linguistics (1916, posthumous) is generally
considered to be the foundation of modern
linguistics. He envisaged the development of
semiology as a science of signs.
Peirce, Charles S. - (1839-1914) American
scientist and philosopher. One of the
foremost philosophers of 'pragmatism' - no
object or concept possesses validity or
importance in its own right. Its significance
lies only in the practical effects of its use or
application. For Communication and Media
students, his importance lies primarily in his
development of semiotics.
B (Logogram)!
Approaching text
Encoding - decoding
Encoding : framework of knowledge
relation of production
technical infrastructure
Decoding : syntactic
recognition of the sign (in
relation to other signs);
semantic
comprehension of the intended
meaning of the sign;
pragmatic
interpretation of the sign in terms
of relevance, agreement etc.
dominant reading
negotiated reading
oppositional reading
p c
p c
Sign
Text a sign system I
Text a sign system II
Syntagms are often defined as 'sequential' (and
thus temporal - as in speech and music), but
they can represent spatial relationships. The
plane of the syntagm is that of the combination
of 'this-and-this-and-this' (syntax).
A paradigmatic structure represents
potential substitutions in which a range of
candidates can take the place of a sign in
the syntagmatic structure. The plane of the
paradigm is that of the selection of 'this-or-
this-or-this' (semantics).
shoes socks pants sweater scarf hat
knickers
short
kilt
tights
Example:
Text a sign system III
Syntagm
Spatial relations (horizontal and vertical axi,
centre and margin)!
Logical order (grammar)!
Exposition (proposition, evidence, justification
Narrative space (exposition, retardation,
digression, omission, redundancy)!
Narrative time (ellipses, compression, insertion,
dilation)!
Paradigm
clusters (e.g. synonyms)!
doublets (e.g. oppositions)!
proportional series ( e.g. a series of
oppositional doublets such as
female - male, passive - active, etc.)!
=> Taxonomy
hierarchies (ordered semantic units
based on relations of inclusion or
exclusion, e.g.
Pekinese/dog/animal/living thing).
=> Thesaurus
Semantic field: '...a
conceptual structure
which organises
potential meanings in
relation to others'
=> Conceptual graph,
semantic network,
ontology
processes
Representation and Transformation mechanisms
Text a sign system IV
A code is a rule-governed system of signs,
whose rules and conventions are shared
amongst members of a culture, and which is
used to generate and circulate meanings in
and for that culture. Fiske, J. (1987, p.4)
A set of signs
that carry
meaning.
A set of agreed rules
for combining those
signs together
! Perceptual (e.g. Typography)!
! Syntagmatic (e.g. Grammar)!
! Paradigmatic (e.g. Ontology)!
! Social (e.g. Word use)!
Text a sign system summary I
Text is a sign system strong on arbitrariness,
proposing the autonomy of language in relation
to reality.
Text emphasis on internal structures and thus
does not 'reflect' reality but rather constructs it.
Text is conventional with an emphasis on the
types index and symbol.
Text a sign system summary II
Representing Text in a media-based system:
Conceptual models for:
! Typography
! Layout
! Writing system (e.g. Alphabet)!
! Syntax (e.g. grammar, markup languages, ....)!
! Dictionaries
! Semantics (e.g. taxonomy, thesaurus, ontology,
conceptual graph, etc.)!
! Style (e.g. frame, template, script,....)!
! Genre (e.g. template, conceptual graph)!
Interpretation depends on the task:
! Search (e.g. text understanding, word matching
and/or ranking)!
! Generation (e.g. text understanding, question-
answering, ....)!
! Comparison (e.g. Syntax (pattern matching) or
semantics (clustering, distance evaluation, etc.)!
Text Example use
Twitoems
Syntagma
Syntagma +
IPA (code)
Aesthetic codes
Rhetorical codes
Stylistic codes
Perceptional Codes
29
Image a sign system
" Legend of Orpheus & Eurydice ", 2001,
The Werner Collection
http://www.wernercollection.com/
WorldView1.htm
Approaching an image
Concept
Mental perception
of media
SIGN
Signifier
(material)!
Signified
(meaning)!
?
31
Approaching an image
Colour
Object
Mise en scene
Framing
Genre
Meaning
Materiality
Distance
(foreground - background)!
32
Image a sign system I
Perceptual codes
perceptive codes (establish the condition for effective
perception)
recognition codes which are blocks of signifieds we use to
recognize objects
transmission codes which construct the determining
conditions for the perception of an image (dots that make
up a newspaper image)
Textual codes
tonal codes address the prosodic features by
connoting them with particular intonation of the sign
Iconic codes (figures, signs, semes)
Iconographic codes connote more complex and
culturalized semes that are immediately identifiable and
classifiable, such as "the four horsemen of the
Apocalypse".
33
Image a sign system II
Social codes
verbal language
bodily codes (bodily contact, physical orientation, gaze,
gestures and posture);
commodity codes (fashions, clothing, cars);
behavioural codes (protocols, rituals, role-playing, games)
ideological codes (encoding' and 'decoding' information by
using theories such as individualism, liberalism, feminism,
materialism, capitalism, socialism, etc.)
Syntagmatic - paradigmatic codes
scientific codes, including mathematics;
aesthetic codes (poetry, drama, painting, sculpture, music,
etc.)
genre, rhetorical and stylistic codes (e.g. in narrative: plot,
character, action, dialogue, setting, etc.),
mass media codes (e.g. in photography, TV, film, radio,
newspaper and magazine, etc.)
34
Image a sign system III
Denotation describes the 'literal' or 'obvious'
meaning of a sign. Thus, denotation of a
representational visual image is what all
viewers from any culture and at any time would
recognize the image as depicting.
Denotation is the first level of signification.
Perceptual
codes
Textual codes
Social codes
Sensory system Media
Sign I
(denotative sign with signifier and signified)!
35
Image a sign system IV
Connotation refers to the socio-cultural and
'personal' associations (ideological,
emotional etc.) of the sign. These are
typically related to the interpreter's class,
age, gender, ethnicity and so on.
Connotation is the second level of signification.
Sign I
+
Signified
Social codes
Syntagmatic codes + experiences
Paradigmatic codes associations
Sign II
(connotative sign with signifier and signified)!
36
Image a sign system VI
The third level of signification.
Sign III
(valued signifier and signified)!
Sign II
Social codes
Syntagmatic codes
Paradigmatic codes
value
37
Image a sign system IV
Signification difference between text and image
On the 1
st
step:
Text => provides an index as a signified
Image => sets the signified
The reader replaces each
index (word) of the
provided order with a
signified for his or her
liking =>
the 3
rd
step of signification
does not cause a problem,
as it is already matched in
the first step
The viewer has to
establish the order of
importance (using step 2)
=>
particular the 3
rd
step of
signification becomes
important as it is the
comparison with the own
sign system (comparison
with what is not shown)
that determines how the
perceiver values and
thus understands the
material.
38
Image Description methods
" Legend of Orpheus &
Eurydice ", 2001,
The Werner Collection
39
Image a sign system summary
An image is a a dominantly iconic
sign system, proposing a union in
relation to reality.
The denotative power of an image,
the optical pattern, communicates a
precise knowledge, which releases
the audience from the process of
decision making but leaves a
problem of interpretation
(signification process).
40
Image a sign system summary II
Representing an Image in a media-based system:
Conceptual models for:
! quantitative or qualitative characterization of optical
pattern (feature extraction (colour, texture, light,
angle, etc.), pattern recognition (line, shape region,
etc.), multi-scale signal analysis, ...)!
! Spatial dimensions
=> textual metadata
! Semantics (e.g. taxonomy, thesaurus, ontology, etc.)!
! Semantic markers (key word, tag, schema, ....)!
to express higher semantics , such as forms, styles,
genres, aesthetics, social codes.
Interpretation depends on the task:
! Search (e.g. retrieval by example)!
! Generation (e.g. Qualitative support on features
and higher semantics)!
! Presentation (e.g. browsing through collage)!
! Automatic art generation
Image Example use
Semiotic tagging (collaboration with Ansgar Scherp, West, University Koblenz-Landau)
Icon
Index
Symbol
Classification of tags
while annotating
Identifying the meaning of the
image in the query based on
classification slots
Step A
Step B
Rhetorical tropes
Image from Luis Buuels Un Chien Andalou
Rhetorical tropes
A trope describes the use of a word or
expression as changed from the original
signification to another.
Tropes are mainly of four kinds:
Trope
metaphor,
metonymy,
synecdoche,
and irony.
Rhetorical tropes
A metaphor involves one signified acting as a
signifier referring to a different signified.
A metaphor is a figure of speech in which an
expression is used to refer to something that it
does not literally denote in order to suggest a
similarity.
Metaphor
Rhetorical tropes
Metonymy
Metonymy is a function which involves using
one signified to stand for another signified
which is directly related to it or closely
associated with it in some way.
(Chandler, http://www.aber.ac.uk/media/Documents/S4B/sem07.html)
Metonyms are based on various indexical
relationships between signifieds, notably the
substitution of
effect for cause
object for user
substance for form
place for event
place for person
place for institution
institution for people
See also: Catherine R. Langans article Intertextuality
in Advertisements for Silk Cut Cigarettes, available at
http://www.aber.ac.uk/media/Students/crl9502.html
Rhetorical tropes
Synecdoche
A synecdoche substitutes a part of a thing for
the whole, or the whole for a part, the species
for the genus or vice versa.
Examples:
part for whole (fifty sails for fifty ships )
whole for part (the market for customers)
species for genus - hypernymy (cutthroat for
assassin)
genus for species - hyponymy (vehicle for
car).
In visual media a close-up is a simple
synecdoche - a part representing the whole.
(Jakobson & Halle 1956, p. 92)
Image from Jean-Luc Godard's La Chinoise'
Rhetorical tropes
Irony
Irony involves incongruity between what is
expected and what occurs.
Irony involves a shift in modality. The evaluation
of the ironic sign requires the retrospective
assessment of its modality status. Re-
evaluating an apparently literal sign for ironic
cues requires reference to perceived intent and
to truth status. Thus, an ironic sign is 'double-
coded'.
The signifier of the ironic sign seems to signify
one thing but another signifier establishes that
it actually signifies something very different.
Where it means the opposite of what it says
(as it usually does) it is based on binary
opposition.
48
Video a temporal visual sign system
Video Examples for content repurpose
The shining a romantic comedy
Titanic The sequel
Schindler's list the romantic comedy
Mary Poppins the scary original
All images from YouTube (www.youtube.com)!
50
Video Internal and external context
An image is an index to a story
Images from Stanley Kubrick's 'Shining'
A video is an iconic representation
of a story
51
Video Internal and external context
Media-dependent context
! Spatial
! Temporal
Meta-semantic context
! Plot structure
! Genre
! References
! Reviews
! Personal preferences
! ....
Images from Stanley Kubrick's 'Shining'
Video Media-dependent context - temporal
t
Shot context: temporal relationship
between frames
Scene context: temporal relationship
between shots, e.g. insert
Episode context: temporal relationship between scenes
Montage
Metric (absolute length)!
Rhythmic
Tonal
Compression
Expansion
Insertion
Deletion
Images from Stanley Kubrick's 'Shining'
Temporal
Spatial
t
Video Media-dependent context - summary
All frames from The Shining (1980)!
Knowledge
representation
54
Video Internal and external context
Media-dependent context
! Spatial
! Temporal
Meta-semantic context
! Plot structure
! Genre
! References
! Reviews
! Personal preferences
! ....
Images from Stanley Kubrick's 'Shining'
Video Meta-Semantic context - Narration III
Characteristic objects
and actions from the real
world
Fabula
Style
Genre
a loose set of criteria
for a category of
composition
Theme
as a set of possible
narrative objects
(comedy, tragedy, etc.)!
Plot
Structure
Catalyst Conflict -
Consequence
Images from Stanley Kubrick's 'Shining'
Video Meta-Semantic context - Narration IV
Images from Stanley Kubrick's 'Shining'
Syntagma in video
Autonomous
sequence
Metz's syntagmatic
categories (Monaco,
1981, p. 188).
Autonomous shot
(establishing shot, insert)
Syntagms
Parallel syntagm (montage of motifs)
Chronological
syntagms
Achronological
syntagms Bracket syntagm (montage of brief shots)
Descriptive syntagm (sequence describing one moment)
Narrative
syntagms
Alternative narrative syntagm
(two sequences alternating)
Linear
narrative
syntagms
Scene (shots implying temporal
continuity)
Sequences
proper
Episodic sequence
(organized
discontinuity of shots)
Ordinary sequence
(temporal with some
compression)
Syntagmatic structures of
construction (Monaco,
1981, p. 145).
Space (Frame)
Time (shot, scene, sequence)
Construction
Syntagms
Syntagms and paradigms
Paradigm - Examples
shoes socks pants sweater scarf hat
Paradigmatic structures
of clothing (Monaco,
1981, p. 341).
skirt
knickers
short
kilt
culottes
tights
Choice process example in film: Casting
Image from Charlie Kaufmans Synecdoche New York'
59
Video Context - summary
MPEG - 1
AVI
Video
Time
60
Video a temporal visual sign system - summary
! A video is a compositional unit with
individualised semantics.
! The semantics may change if a shot is
juxtaposed with another shot.
! A distinction between filmic (codify the
relation to reality) and cinematic codes
(codify narrative communication) must be
made.
! Video, though based on common human
content and thematic structures, provides its
own realities of time and space which are
interwoven in the narrative structure.
! A story is a representational system based
on two main layers, structure and content,
each serving two distinct purposes (form and
substance) simultaneously.
Video Example use
AUTEUR (1996)
well described video components
generation of humorous
sequences from clip database
rules for continuity editing,
action generation, themes, and
presentation generation
Interactive Production Generator Narrative Model
Nack (1996)
AUTEUR - Aim
Automatic composition of
visual slapstick sequences.
Use an existing, arbitrary data
base.
Investigate machine creativity.
AUTEUR Generation Example I
Motivation
Narrative rules (joke structure)!
Access via content representation
Editing rules (continuity)!
Narrative rules (continuity)!
Access via content representation
Editing rules (continuity)!
Realisation
Narrative rules (joke structure)!
Access via content representation
Editing rules (continuity)!
AUTEUR Generation Example II
Resolution
Narrative rules (joke structure)!
Access via content representation
Editing rules (continuity)!
AUTEUR Generation Example III
Result
AUTEUR Generation Example IV
Video Example use
Narranotations
An approach for an association-based
story environment, in which a priori
unrelated experiences represented in
images, are stitched together to guide
users through interesting city spaces.
Interesting spaces are described as
hypespot, which facilitates linking the real
world with the structure of the story.
Narranotations (annotation to the image)
provide information about how an
expression can be used as an element
within a story.
Video Example use
N
a
r
r
a
n
o
t
a
t
I
o
n
s
Video Example use
N
a
r
r
a
n
o
t
a
t
i
o
n
s
Discussion
Image from Charlie Kaufmans Synecdoche New York'
References
Image from Charlie Kaufmans Synecdoche New York'
Bibliography - Fundamentals
! Andersen, P. B. (1997). Semiotic Approaches to Constrution and
Assessment Of Computer Systems. Cambridge Universty Press
! Barthes, R. Image Music Text. Hill and Wang, New York, 1977.
! Bolter, J. G. & Grusin, R. (1999). Remediation - Understanding New
Media. Cambridge, Massachusetts: MIT Press.
! Chandler, D. (2003). Semiotics for Beginners - http://www.aber.ac.uk/
media/Documents/S4B/semiotic.html
! Corner, John (1980). 'Codes and Cultural Analysis', Media, Culture and
Society 2: 73-86 Davis, H. & Walton P. (Eds.) (1983). Language,
Image, Media. Oxford: Basil Blackwell
! Derrida, J. Writing and Difference. Routledge & Kegan Paul, London,
1978.
! Ducrot, O. and Todorov, T. Encyclopedic Dictionary of the Sciences of
Language. The Johns Hopkins University Press, Baltimore, Maryland,
1979.
Bibliography - Fundamentals
! Eco, U. A Theory of Semiotics. Indiana University Press, Bloomington,
Indiana, 1976.
! Fiske, J. (1982). Introduction to Communication Studies. London:
Routledge
! Greimas, J. (1983). Structural Semantics: An Attempt at a Method.
Lincoln: University of Nebraska Press.
! Gumpert, G. & Cathcar, R. (1985). Media Grammars, Generations and
Media Gaps. Critical Studies in Mass Communication 2: 23-35
! Hawkes, T. Structuralism and Semiotics. University of California Press,
Berkeley, 1977.
! Hodge, R. & Kress, G. (1988): Social Semiotics. Cambridge: Polity
! Kendon, A. (1981). Nonverbal Communication, Interaction, and
Gesture - Selections from Semiotica. The Hague: Mouton Publishers.
! Lvi-Strauss, C. (1968). Structural Anthropology Volume 1, translated
by Clair Jacobson, Brooke Grundfest Schoepf. London: Allen Lane.
! Lvi-Strauss, C. (1977). Structural Anthropology Volume 2, translated
by Monique Layton. London: Allen Lane.
! Lippmann, W. (1934). Public Opinion. New York: The MacMillan
Company.
Bibliography - Fundamentals
! McLuhan, M. The Gutenberg Galaxy: The Making of Typographic Man.
University of Toronto Press, Toronto, 1962.
! McLuhan, M. (1962). The Gutenberg Galaxy. Toronto: University of
Toronto Press
! McLuhan, M. & Quentin Fiore (1967). The Medium is the Massage. New
York: Bantam
! McLuhan, M. (1970). Counterblast. London: Rapp & Whiting
! McLuhan, M. Understanding Media: The Extensions of Man. The MIT
Press, Cambridge, Massachusetts, 1994.
! Metz, C. Film Language: A Semiotics of Cinema. The University of
Chicago Press, Chicago, 1974.
! Mick, D. G. (1986): 'Consumer Research and Semiotics: Exploring the
Morphology of Signs, Symbols and Significance', Journal of Consumer
Research 13(2): 196-213
! Minsky, M. L. (1988). The Society of mind. London: Picador.
! Nth, W. (1990). Handbook of Semiotics. Bloomington, IN: Indiana
University Press
Bibliography - Fundamentals
! Packer, R. & Jordan, K. (editors) (2001). multiMEDIA - From Wagner to
Virtual Reality. New York, W.W.Norton & Company
! Peirce, C. S. (1960). The Collected Papers of Charles Sanders Peirce - 1
Principles of Philosophy and 2 Elements of Logic, Edited by Charles
Hartshorne and Paul Weiss. Cambridge, Massachusetts: The Belknap
Press of Harvard University Press.
! Reddy, M. The Conduit Metaphor: A Case of Frame Conflict in Our
Language about Language. in Ortony, A. ed. Metaphor and Thought,
Cambridge University Press, New York, 1979, 284-324.
! Saussure, F.d. Course in General Linguistics. McGraw-Hill, New York,
1983.
! Sebeok, T. A. & Danesi, M. (2000). The Forms of Meaning Modeling
Systsems Theory and Semiotic Analysis, Mouton de Gryter, Berlin
! Sowa, J. F. (1984). Conceptual Structures: Information Processing in Mind
and Machine. Reading, MA: Addison-Wesley Publishing Company.
! Stam, R., Burgoyne, R. and Flitterman-Lewis, S. New Vocabularies in Film
Semiotics: Structuralism, Post-Structuralism, and Beyond. Routledge,
London, 1992.
Bibliography - Fundamentals
! Sturrock, J. (1986): Structuralism. London: Paladin
! Todorov, T. Theories of the Symbol. Cornell University Press, 1977.
! Wilden, A. (1987). The Rules Are No Game: The Strategy of
Communication. London: Routledge & Kegan Paul
! Wolff, C. (1972). A Psychology of Gesture. New York: Arno Press.
Bibliography - Text
! Ball, M. (1997). Narratology - Introduction to the Theory of Narrative
(2nd edition). Toronto: University of Toronto Press.
! Barrett, E. (ed.), The Society of Text: Hypertext, Hypermedia, and the
Social Construction of Information. The MIT Press, Cambridge,
Massachusetts, 1989.
! Barrett, E. (ed.), Text, ConText, and HyperText: Writing with and for the
Computer. The MIT Press, Cambridge, Massachusetts, 1988.
! Barthes, R. Elements of Semiology. Hill and Wang, New York, 1967.
! Barthes, R. (1974). S/Z with a preface by Richard Howard. London:
Jonathan Cape.
! Barthes, R. (1977). Image, Music, Text - Essays selected and
translated by Stephen Heath. London: Fontana Press.
! Bernstein, M., Patterns of Hypertext. in ACM Hypertext'98, (Pittsburg,
Pennsylvania, 1998), ACM Press, 21-29.
! Berwick, R.C. Computational Structure of Language.
Bibliography - Text
! Berwick, R.C. and Weinberg, A.S. The Grammatical Basis of Linguistic
Performance: Language Use and Acquisition. MIT Press, Cambridge, Mass.,
1984.
! Bordwell, D. (1989). Making Meaning - Inference and Rhetoric in the
Interpretation of Cinema. Cambridge, Massachusetts: Harvard University
Press.
! Brady, M., Berwick, R.C. and Allen, J. Computational Models of Discourse.
MIT Press, Cambridge, Massachusetts, 1983.
! Brusilovsky, P., Kobsa, A., and Vassileva J.(ed.) Adaptive Hypertext and
Hypermedia. Kluwer Academic Publishers, Dordrecht, The Netherlands, 1998.
! Bush, V. As We May Think. Atlantic Monthly, 176 (July). 101-108.
! Chatman, S.B. Story and Discourse: Narrative Structure in Fiction and Film.
Cornell University Press, Ithaca, N.Y., 1978.
! Chatman, S.B. Coming to Terms The Rhetoric of Narrative in Fiction and
Film. Cornell University Press, Ithaca, N.Y., 1990.
Bibliography - Text
! Chomsky, N. Aspects of the Theory of Syntax. The MIT Press, Cambridge,
1965.
! Chomsky, N. Language and Mind. Harcourt Brace & World, Inc, New York,
1968.
! Chomsky, N. Studies on Semantics in Generative Grammar. Mouton, The
Hague, 1972. Chomsky, N. Rules and Representations, 1980.
! Chomsky, N. Lectures on Government and Binding: The Pisa Lectures.
Foris Publications, Holland, 1981.
! Chomsky, N. The Logical Structure of Linguistic Theory. University of
Chicago Press, Chicago, 1985.
! Chomsky, N. Language and Thought. Moyer Bell, Wakefield, Rhode Island,
1993.
! Chomsky, N. On Language: Chomsky's Classic Works "Language and
Responsibility" and "Reflections on Language" in One Volume. The New
Press, New York, 1998.
Bibliography - Text
! Delany, P. and Landow, G.P. Hypermedia and Literary Studies, 1991.
! Eco, U. (1966). 'Narrative Structure in Fleming'. In E. del Buono & Umberto
Eco (Eds.): The Bond Affair. London: Macdonald
! Eco, U. (1981). The Role of the Reader. London: Hutchinson
! Eco, U. (1984). Semiotics and the Philosophy of Language. Bloomington, IN:
Indiana University Press
! Engelbart, D. A Conceptual Framework for the Augmentation of Human
Intellect. in Howerton, P.D., Weeks, D.C ed. Vistas in Information Handling,
Spartan Books, Washington DC, 1963.
! Goodman, N. Languages of Art. Hackett Publishing Company, Inc., 1976.
! Greimas, A.J. Structural Semantics: An Attempt at a Method. University of
Nebraska Press, Lincoln, 1983.
! Holdcroft, D. (1991). Saussure: Signs, Systems and Arbitrariness.
Cambridge: Cambridge University Press
! Illich, I. and Sanders, B. ABC: The Alphabetization of the Popular Mind. North
Point Press, San Francisco, 1988.
Bibliography - Text
! Iser, W. (1989). Prospecting: From Reader Response in Literary Anthropology.
Baltimore, Maryland: The John Hopkins University Press.
! Iser, W. (1993). The Fictive and the Imaginary: Charting Literary Anthropology.
Baltimore, Maryland: The John Hopkins University Press.
! Jakobson, R. (1971): 'Language in Relation to Other Communication Systems'.
In Roman Jakobson (Ed.): Selected Writings, Vol. 2. Mouton: The Hague, pp.
570-79
! Jakobson, R.H., M. Fundamentals of Language. Mouton Publishers, The
Hague, 1980.
! Kolb, D. Socrates in the Labyrinth: Hypertext, Argument, Philosophy. Eastfgate
Systems, Cambridge, 1995.
! Kumpf, E.P. Visual Metadiscourse: Designing the Considerate Text. Technical
Communication Quarterly, 9 (4). 401-424.
! Lakoff, G. P. (1972). Structural complexity in fairy tales. The study of man, 1,
128 - 190.
! Lakoff, G. & Johnson, M. (1980). Metaphors We Live By. Chicago: University of
Chicago Press
Bibliography - Text
! Leiss, W., Kline, S. & Jhally, S. (1990). Social Communication in
Advertising: Persons, Products and Images of Well-Being (2nd Edn.).
London: Routledge
! Lemon, L. T., & Reis, M. J. (1965). Russian Formalist Criticism - Four
Essays. Lincoln: University of Nebraska Press.
! McQuarrie, E. F. & Mick, D. G. (1999). Visual Rhetoric in Advertising: Text-
Interpretative, Experimental and Reader-Response Analyses, Journal of
Consumer Research 26(1): 37-54
! Miles, A. Hypertext Syntagmas: Cinematic Narration with Links. Journal of
Digital Information, 1 (7).
! Moulthrop, S., Beyond the Electronic Book: A Critique of Hypertext Rhetoric.
in 4th ACM conference on Hypertext, (San Antonio, USA, 1991), 291 - 298.
! Moulthrop, S. and Cohen, S. The Color of Television, 1996, A multi-
threaded, verbal/graphic Web fiction in collaboration with the writer and
designer Sean Cohen.
! Nelson, T. Computer Lib and Dream Machines. Microsoft Press, Redmond,
Washington, 1987.
! Nelson, T. Literary Machines, 1981.
Bibliography - Text
! Nielsen, J. Hypertext & Hypermedia. Academic Press, Inc.,, Boston, 1990.
! Nielson, J. Multimedia and Hypertext: The Internet and Beyond. Academic
Press, Inc, Boston, 1990.
! Nyce, J.M. and Kahn, P. (eds.). From Memex to Hypertext: Vannevar Bush
and the Mind's Machine. Academic Press, Boston, 1991.
! Ong, W. Interfaces of the Word. Cornell University Press, Ithaca, 1977.
! Ong, W. The Presence of the Word. Yale University Press, New Haven, 1967.
! Ong, W.J. Orality and Literacy: The Technologizing of the Word. Methuen,
London, 1982.
! Price, G. (1973). A grammar of story: An introduction. The Hague: Mouton.
! Propp, V. W. (1968). Morphology of the Folktale. University of Texas Press.
! Raskin, V. (1985). Semantic Mechanisms of Humor. Dordrecht: D. Reidel
Publishing Company.
! Ricoeur, P. (1985). Time and Narrative. Chicago: The University of Chicago
Press.
Bibliography - Text
! Sampson, G. Writing Systems: A Linguistic Introduction. Stanford
University Press, Stanford, California, 1985.
! Suleiman, S. R. (1983). Authoritarian Fictions: The Ideological Novel As a
Literary Genre. New York: Columbia University Press.
! Suls, J. M. (1972). A two-stage model for the appreciation of jokes and
cartoons: An information-processing analysis. In J. H. Goldstein & P. E.
McGhee (Eds.), The Psychology of Humour (pp. 81 - 100). New York,
London: Academic Press.
! Saussure, F. d. (1966). Course in General Linguistics - edited by Charles
Balley, Albert Sechehaye and Albert Riedlinger. New York: McGraw-Hill.
! Segre, C. (1979). Structure and Time - Narration, Poetry, Models. Chicago:
The University of Chicago Press.
! Tosca, S.P., A Pragmatics of Links. in 11th ACM conference on Hypertext
and Hypermedia, (San Antonio, Texas, USA, 2000), pp. 77 - 84.
! Winograd, T. Language as a Cognitive Process: Syntax. Addison-Wesley,
Massachusetts, 1983.
! Winograd, T. Understanding Natural Language. Academic Press, 1972.
Bibliography - Text
! Zellweger, P.T., Chang, B-W., and Mackinlay, J. D., Fluid Links for
Informed and Incremental Link Transitions. in 9th ACM conference on
Hypertext and Hypermedia, (Pittsburgh, USA, 1998), 50 - 57.
Bibliography - Image
! Arnheim, R. Art and Visual Perception: A Psychology of the Creative
Eye. University of California Press, Berkeley, 1974.
! Arnheim, R. (1983). Film as Art. London: Faber & Faber. Argyle, M.
(1988): Bodily Communication (2nd edn.). London: Methuen
! Aumont, J. The Image. British Film Institute, London, 1997.
! Barthes, R. Elements of Semiology. Hill and Wang, New York, 1967.
! Barthes, R. Image Music Text. Hill and Wang, New York, 1977.
! Barthes, R. S/Z, 1974.
! Bazin, A. (1967). The Ontology of the Photographic Image. In H. Gray
(Eds.), What is Cinema - essays selected and translated by Hugh Gray
(pp. 9 - 16). Berkeley: University of California Press.
Bibliography - Image
! Cook, G. (1992). The Discourse of Advertising. London: Routledge
! Del Bimbo, A. (1999). "Visual Information Retrieval", Morgan Kaufmann
Ed, San Francisco, USA
! Eco, U. (1982). Critique of the Image. In Burgin, V. (Ed.), Thinking
Photography. London: Macmillan
! Forceville, C. (1996). Pictorial Metaphor in Advertising. London: Routledge
! Galtung, J. & Ruge, E. (1981). Structuring and Selecting News. In Cohen,
S. & Young, J. (Eds.) (1981): The Manufacture of News: Social Problems,
Deviance and the Mass Media. London: Constable ., pp. 52-63
! Gibson, J. J. (1950). Perception of the Visual World. Boston, Ma:
Houghton Mifflin.
! Gibson, J. J. (1971). The Information available in Pictures. Leonardo(4), 27
- 35.
Bibliography - Image
! Gombrich, E. H (1974). The Visual Image'. In David R Olson (Ed.): Media
and Symbols: The Forms of Expression, Communication and Education.
Chicago, IL: University of Chicago Press, pp. 255-8; first published in
Scientific American 227 (September 1971): 82-96
! Gombrich, E. H (1977). Art and Illusion: A Study in the Psychology of
Pictorial Representation. London: Phaidon
! Gombrich, E.H. Image and Code. Horace H. Rackham School of
Graduate Studies, Ann Arbor, MI, 1981.
! Gombrich, E. H (1982). The Image and the Eye: Further Studies in the
Psychology of Pictorial Representation. London: Phaidon
! Gombrich, E. H (1995). The Story of Art. London: Phaidon
! Goodman, N. Languages of Art. Hackett Publishing Company, Inc., 1976.
! Grayson, K. (1998). The Icons of Consumer Research: Using Signs to
Represent Consumers' Reality. In Stern, B. B (Ed) (1998): Representing
Consumers: Voices, Views and Visions. London: Routledge, pp. 27-43.
Bibliography - Image
! Hall, S. (1981). The Determinations of News Photographs. In Cohen, S.
& Young, J. (Eds.) (1981): The Manufacture of News: Social Problems,
Deviance and the Mass Media. London: Constable , pp. 226-43
! Hochberg, J.E. Perception. Prentice-Hall, New Jersey, 1978.
! Kennedy, J.M. A Psychology of Picture Perception. Jossey-Bass, San
Francisco, 1974.
! Kress, G. & van Leeuwen, T. (1996): Reading Images: The Grammar of
Visual Design. London: Routledge
! Kress, G. & van Leeuwe, T. (1998): 'Front Pages: (The Critical) Analysis
of Newspaper Layout'. In Allan Bell & Peter Garrett (Eds.): Approaches
to Media Discourse. Oxford: Blackwell, pp. 186-219
! Marr, D. Vision: A Computational Investigation into the Human
Representation and Processing of Visual Information. Freeman, San
Francisco, 1982.
! McCloud, S. Understanding Comics: The Invisible Art. Tundra
Publishing, Northampton, Massachusetts, 1993.
Bibliography - Image
! McLuhan, M. Counter-Blast. Rapp & Whiting, London, 1970.
! McLuhan, M. The Gutenberg Galaxy: The Making of Typographic Man.
University of Toronto Press, Toronto, 1962.
! McLuhan, M. and Fiore, Q. The Medium is the Massage. Bantam Books,
New York, 1967.
! Panofsky, E. Meaning in the Visual Arts: Papers in and on Art History.
Doubleday, Garden City, N.Y., 1955.
! Pentland, A.P., Picard, R.W. and Sclaroff, S. Photobook: Tools for
Content-Based Manipulation of Image Databases, Massachusetts
Institute of Technology Media Laboratory, 1994.
! Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A. and Jain, R.
Content-Based Image Retrieval at the End of the Early Years. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 22 (12).
1349-1380.
! Sonesson, G. (1992). The semiotic function and the genesis of pictorial
meaning, in Tarasti, Eero, ed.,Center/Periphery in representations and
institutions. Proceedings from the 3rd Annual Meeting and Congress of
The International Semiotics Insitute, Imatra, Finland, July 16-21, 1990.:
International Semiotics Institute, Imatra 1992; ss 211-256.
Bibliography - Image
! Strothotte, S.S., T. Seeing Between the Pixels. Springer, Berlin, 1997.
! Strothotte, T. Computational Visualization - Graphics, Abstraction and
Interactivity. Springer, Berlin, 1998.
! Tudor, A. Image and Influence. George Allen & Unwin Ltd., London,
1974.
! Tufte, E.R. Envisioning Information. Graphics Press, Cheshire,
Connecticut, 1990.
! Tufte, E.R. The Visual Display of Quantitative Information. Graphics
Press, Cheshire, 1983.
! Tufte, E.R. Visual Explanations: Images And Quantities, Evidence And
Narrative. Graphics Press, Cheshire, Connecticut, 1977.
Bibliography Temporal Media
! Andrew, J.D. The Major Film Theories: An Introduction. Oxford
University Press, Oxford, 1976.
! Andrew, D. Concepts in Film Theory. Oxford University Press, Oxford,
1984.
! Arnheim, R. Film as Art. Faber & Faber, London, 1983.
! Baggaley, J. & Duck S. (1976). Dynamics of Television. Farnborough:
Saxon House
! Balzs, B. Theory of the Film: Character and Growth of a New Art.
Dover Publications, Inc., 1970.
! Barthes, R. The Sequences of Actions. in Strelka, J. ed. Patterns of
Literary Style, State University of Pennsylvania Press, University Park,
Pennsylvania, 1971.
! Bazin, A. What is Cinema? (Volume I). University of California Press,
Berkeley, 1971.
! Bazin, A. What is Cinema? (Volume II), 1971.
Bibliography Temporal Media
! Bellour, R. To Analyze, to Segment. Quarterly Review of Film Studies, 1, 3
(August). 331-354.
! Bettetini, G. The Language and Technique of the Film, 1973.
! Bloch, G.R. Elements d'une Machine de Montage Pour l'Audio-Visuel, Ecole
Nationale Suprieure Des Tlcommunications, 1986.
! Bloch, G.R. From Concepts to Film Sequences, Yale University Department of
Computer Science, 1987.
! Bordwell, D., Staiger, J. and Thompson, K. The Classical Hollywood Cinema:
Film Style and Mode of Production to 1960. Columbia University Press, New
York, 1985.
! Bordwell, D. Narration in the Fiction Film. University of Wisconsin Press,
Madison, 1985.
! Bordwell, D. Making Meaning: Inference and Rhetoric in the Interpretation of
Cinema. Harvard University Press, Cambridge, Massachusetts, 1989.
! Bordwell, D. The Cinema of Eisenstein. Harvard University Press, Cambridge,
Massachusetts, 1993.
Bibliography Temporal Media
! Bordwell, D. On the History of Film Style. Harvard University Press,
Cambridge, Massachusetts, 1997.
! Bordwell, D. Planet Hong Kong: Popular Cinema and the Art of
Entertainment. Harvard University Press, Cambridge, 2000.
! Bordwell, D. and Thompson, K. Film Art: An Introduction. McGraw Hill, New
York, 2003.
! Burch, N. Theory of Film Practice. Princeton University Press, Princeton,
1969.
! Buckland, W. (2000). The Cognitive Semiotics of Film. Cambridge, UK:
Cambridge University Press.
! Carroll, J.M. Toward a Structural Psychology of Cinema. Mouton
Publishers, The Hague, 1980.
! Chatman, S.B. Story and Discourse: Narrative Structure in Fiction and Film.
Cornell University Press, Ithaca, N.Y., 1978.
! Davis, D. (1960). The Grammar of Television Production (revised by John
Elliot). London: Barrie & Rockliff
Bibliography Temporal Media
! Eco, U. (1965). 'Towards a Semiotic Enquiry into the Television
Message', In Corner & Hawthorn (Eds.) (1980), pp. 131-50
! Eco, U. Articulations of the Cinematic Code. in Nichols, B. ed. Movies and
Methods: An Anthology, University of California Press, Berkeley, 1976,
590-607.
! Eco, U. A Theory of Semiotics. Indiana University Press, Bloomington,
Indiana, 1976.
! Eisenstein, S.M. The Film Sense. Harcourt Brace Jovanovich, Publishers,
San Diego, 1947.
! Eisenstein, S.M. Film Form: Essays in Film Theory. Harcourt Brace
Jovanovich, Publishers, San Diego, 1949.
! Eisenstein, S.M. Film Essays and a Lecture. Princeton University Press,
Princeton, New Jersey, 1982.
! Eisenstein Selected Works: Writings 1922 - 1934. BFI Publishing,
London, 1988.
Bibliography Temporal Media
! Eisenstein Selected Works: Towards a Theory of Montage. BFI Publishing,
London, 1991.
! Fiske, J. (1987). Television Culture. London: Routledge
! Fiske, J. & John H. (1978): Reading Television. London: Methuen
! Frith, U., & Robson, J. E. (1975). Perceiving the language of film. Perception,
4, 97 - 103.
! Greenway, T. and Mouchawar, R., A Visual Language for the Management of
Digital Film and Video (Motion Imagery) Archives. in 136th Technical
Conference and World Media Expo of the Society of Motion Picture and
Television Engineers (SMPTE), (Los Angeles, California, 1994), Society of
Motion Picture and Television Engineers, Inc., 1-15.
! Gregory, J. R. (1961) Some Psychological Aspects of Motion Picture
Montage. Ph.D. Thesis, University of Illinois.
! Gregory, R. L. (1971). The Intelligent Eye. London: Weidenfeld&Nicolson.
Bibliography Temporal Media
! Hartley, J. (1996). Popular Reality: Journalism, Modernity, Popular Culture.
London: Arnold
! Hedges, I. (1991). Breaking the Frame - Film Language and the Experience of
Limits. Bloomington: Indiana University Press.
! Holtzman, S.R. Digital Mantras - The Languages of Abstract and Virtual Worlds.
MIT Press, Cambridge, Massachusetts, 1995.
! Isenhour, J.P. The Effects of Context and Order in Film Editing. AV
Communications Review, 23 (1). 69-80.
! Jenkins, H. Textual Poachers: Television Fans & Participatory Culture.
Routledge, New York, 1992.
! Jordan, T. H. (1975). The Anatomy Of Cinematic Humor. New York: The
Revisionist Press.
! Katz, S. D. (1991). Shot by Shot - Visualizing from concept to screen.
Stoneham, MA: Michael Wiese Productions in conjunction with Focal Press.
! Kracauer, S. (1960). Theory of Film: The Redemption of Physical Reality. New
York: Oxford University Press.
Bibliography Temporal Media
! Kuleshov, L. (1974). Kuleshov on Film - Writing of Lev Kuleshov. Berkeley:
University of California Press.
! Manovich, L. The Language of New Media. The MIT Press, Cambridge,
Massachusetts, 2001.
! McAleese, R. (1985). Knowledge and Information Mapping and Interactive
Video. Aberdeen: University Teaching Centre, University of Aberdeen.
! Metz, C. (1974). Film Language: A Semiotic Of The Cinema. New York:
Oxford University Press.
! Metz, C. (1976a). Current Problems of Film Theory: Mitry's L'Esthtique et
Psychologie du Cinema. Vol. II. In B. Nichols (Eds.), Movies and Methods (pp.
568 - 578). Berkeley: University of California Press.
! Metz, C. (1976b). On the Notion of Cinematographic Language. In B. Nichols
(Eds.), Movies and Methods (pp. 582 - 589). Berkeley: University of California
Press.
! Metz, C. (1981). Methodological Propositions for the Analysis of Film. In:
Eaton, M. (Ed.) Cinema and Semiotics (Screen Reader 2). London: Society
for Education in Film and Television, op. cit., pp. 86-98
Bibliography Temporal Media
! Metz, C. (1982). The Imaginary Signifier - Psychoanalysis and the cinema.
Bloomington: Indiana University Press.
! Monaco, J. How to Read a Film: The World of Movies, Media, and
Multimedia: Language, History, Theory. Oxford University Press, New York,
2000.
! Neale, S., & Krutnik, F. (1990). Popular Film and Television Comedy.
London: Routledge.
! Nichols, B. (editor) (1976). Movies and Methods I. Berkeley;University of
California Press.
! Nichols, B. (editor) (1985). Movies and Methods II. Berkeley;University of
California Press.
! Oldham, G. First Cut: Conversations with Film Editors. University of
California Press, Berkeley, California, 1992.
! Packer, R. and Jordan, K. Multimedia: From Wagner to Virtual Reality.
Norton, New York, 2001.
Bibliography Temporal Media
! Peterson, J. (1994). Dreams of Chaos, Vision of Order - Understanding the
American Avant-garde Cinema. Detroit: Wayne State University Press.
! Petric, V. Constructivism in Film. Cambridge University Press, Cambridge,
1987.
! Pudovkin, V.I. Film Technique and Film Acting. Bonanza Books, New York,
1949.
! Rabinger, M. Directing - Film Techniques and Aesthetics. Focal Press., Boston,
1989.
! Reisz, K., & Millar, G. The Technique of Film Editing. Focal/Hastings House,
New York, 1969.
! Rieser, M. & Zapp, A. (2002). New Screen Media - Cinema/Art/Narrative.
London: BFI.
! Rosenblum, R. and Karen, R. When The Shooting Stops, The Cutting Begins -
A Film Editor's Story. Da Capo Paperback, New York, 1979.
! Rush, M. (1999). New Media in Late 20th-Century Art. London:Thames &
Hudson
Bibliography Temporal Media
! Salomon, G. and Cohen, A.A. Television Formats, Mastery of Mental
Skills, and the Acquisition of Knowledge. Journal of Educational
Psychology (69). 612 - 619.
! Salt, B. Film Style and Technology: History and Analysis. Starword,
London, 1992.
! Spottiswode, R. J. (1955). A Grammar Of The Film - an analysis of film
technique. London: Faber & Faber.
! Stam, R., Burgoyne, R. and Flitterman-Lewis, S. New Vocabularies in
Film Semiotics: Structuralism, Post-Structuralism, and Beyond.
Routledge, London, 1992.
! Weber, K., The Work Practice of Moving Image Indexing. in AAAI-94
Workshop on Indexing and Reuse in Multimedia Systems, (Seattle,
Washington, 1994).
! Wells, P. Understanding Animation. Routledge, London, 1998.
! Woods, W. A. (1985). What's in a link: Foundations for Semantic
Networks. In R. J. Brachman & H. J. Levesque (Eds.), Readings in
Knowledge Representation (pp. 218 - 241). San Mateo, California:
Morgan Kaufmann Publishers.
Bibliography Temporal Media
! Wollen, P. Signs and Meaning in the Cinema. Indiana University Press,
Indiana, 1972.
! Young, B. M (1990): Television Advertising and Children. Oxford:
Clarendon Press
! Zettl, H. Essentials of Applied Media Aesthetics. in Dorai, C. and
Venkatesh, S. eds. Media Computing: Computational Media
Aesthetics, Kluwer Academic Publishers, Boston, 2001, 11-38.
! Zettl, H. Sight, Sound, Motion: Applied Media Aesthetics. Wadsworth
Publishing. Company, Belmont, California, 1999.