Professional Documents
Culture Documents
Defining Multimedia
Defining Multimedia
Defining Multimedia
Abstract-iconic
Individual
text is created according to a particular code and
Augmented transmitted via a device. The receiver of the text
decodes it to extract meaning, choosing to impose
Temporal a particular code on the text to interpret it.
Syntax
An image serves merely as a sign to the extent to ❙ Augmentation: An extension to the individual
which it stands for a particular content without arrangement. Like the individual syntax, only
reflecting its characteristic visually…. Images are a single object is used, augmented with one or
pictures to the extent to which they portray things more additional features that contribute addi-
located at a lower level of abstractness than they are tional meaning to the sign’s interpretation.
themselves. They do their work by grasping and
rendering some relevant qualities—shape, color, ❙ Temporal: Time is important to this arrange-
movement—of the objects or activities they depict…. ment—the message cannot be interpreted if
An image acts as a symbol to the extent to which it only a snapshot is taken. Like the individual
portrays things which are at a higher level of syntax, only a single concept is communicat-
abstractness than is the symbol itself (pp. 136-138). ed by a single object, and the interpretation
does not change over time.
If icons occupy one end of an “abstraction con-
tinuum” and symbols the other, a further catego- ❙ Linear: Linear systems place the objects in a
ry of sign—proposed here—falls between the two, purely sequential manner, and interpretation of
classifying icons as either concrete or abstract. the message depends on the objects being con-
Icons perceived as identical to the concept that sidered in this linear arrangement over time.
they represent (like photographs) have a very low However, unlike the temporal arrangement,
level of abstraction, hence are here termed con- more than one concept may be communicated.
crete-icons. Icons sensed as similar, but not iden- This is similar to Heller and Martin’s definition
tical, to the concept lie higher up the abstraction of “temporal,”8 which includes the notion of
January–March 1998
scale. These abstract-icons are not as abstract as the message’s content changing over time.
symbols, since a perceptual relationship still exists
between object and concept, but neither are they ❙ Schematic: Schematic representational systems
perceived as identical to the concept.7 An exam- use spatial indicators to show the information’s
ple of an abstract-icon is a road sign warning structure. They represent relationships between
against falling rocks in the area. the concepts associated with the individual
The three values for the first dimension of the objects in a 2D or 3D manner according to a
model proposed here therefore relate to the nature conventional code.
9
Authorized licensed use limited to: b-on: Instituto Politecnico de Beja. Downloaded on April 13,2024 at 11:37:48 UTC from IEEE Xplore. Restrictions apply.
.
Figure 2. A shaped
photograph illustrating
the use of a concrete- The third dimension: the modality This third dimension in the model therefore
iconic augmented text. The third dimension requires a clear distinc- has two values—visual and aural—relating to
The shape adds tion between the oft-confused terms multimedia the two senses most commonly used for
information about the and multimodal. In its common use, multimedia communication.
status of the person in refers to the nature of the text used in communi-
the photo, in this case a cation, both as output from a technological sys- Examples
popular singer. tem (video, sound, and graphics) and, less Having defined the three dimensions to classi-
obviously, the human input (touch and speech). fy texts, validating these dimensions requires asso-
Multimodal, however, relates specifically to the ciating all cells in the resulting model with an
senses used by the receiver of the text (visual, existing representational system. Distinct exam-
auditory, and tactile).9 ples of media types exist for all 30 cells. Table 1
10
Authorized licensed use limited to: b-on: Instituto Politecnico de Beja. Downloaded on April 13,2024 at 11:37:48 UTC from IEEE Xplore. Restrictions apply.
.
Figure 5. Three road signs, with color indicating whether the road is local,
national, or a motorway. This example illustrates the use of an abstract-iconic
individual text.
Aural modality
When applying this model to the aural modal-
ity, two additional issues need to be considered.
First, the difference between concrete-iconic
and abstract-iconic texts is the difference between
recordings of real sounds and sounds that have
been artificially synthesized. This distinction thus
proves less useful in the aural modality than in
the visual modality.
Second, since by nature the aural modality is
temporal, it is difficult to define individual
objects of aural communication without consid-
ering the temporal dimension. Taking into
January–March 1998
11
Authorized licensed use limited to: b-on: Instituto Politecnico de Beja. Downloaded on April 13,2024 at 11:37:48 UTC from IEEE Xplore. Restrictions apply.
.
Table 2 shows examples in the aural modality. arranging more than one object. A composite-text
Note that in the aural modality, the syntax of aug- will therefore have a temporal, linear, or schemat-
mentation focuses predominantly on the individ- ic syntax arrangement associated with it.
ual sign’s tone, amplitude, or timbre. Also, the For example, an instructional video (a com-
schematic arrangement in the aural modality is posite-text that uses a linear syntax) may include
defined with respect to the frequency domain; the following component-texts:
thus, the second dimension of spatial aural com-
munication is not space (as in the visual modali- ❙ film of a lecturer explaining a problem (con-
ty), but frequency. It’s not easy for humans to crete-iconic, linear, visual),
distinguish the many different individual fre-
quencies in a complex aural text like speech, and ❙ the soundtrack for the film of the lecturer
the examples are limited to aural texts with only explaining the problem (symbolic, linear,
two perceivable different frequency bands. aural),
tational system and may use more than one Extending the syntax dimension: the network
modality. category
The component-texts must themselves be The final syntactic category—network—does
arranged according to a syntax within the com- not describe the syntax for individual objects. It’s
posite-text. The individual and augmentation syn- only used for the arrangement of component-
tactical arrangements are inappropriate for texts within a composite-text. Unlike the first five
12
Authorized licensed use limited to: b-on: Instituto Politecnico de Beja. Downloaded on April 13,2024 at 11:37:48 UTC from IEEE Xplore. Restrictions apply.
.
syntactic categories, it does not restrict the order not suggest a single definition—it can be used to
in which the user receives the component-texts. create different definitions of varying inclusive-
In the network arrangement, component-texts ness, as shown by the following examples.
connect together in a network structure of nodes
and links, with related component-texts linked to 1. The production, transmission, and interpreta-
each other. There is an implicit lack of linearity tion of a composite-text, when at least two of
and no predefined sequence of receiving the entire the component-texts use different representa-
composite-text. Thus, in receiving the text, a com- tional systems.
ponent-text may be followed by any one of the
other component-texts associated with it. (Note This definition of multimedia is very broad.
that the network syntax corresponds to the hyper- Examples of multimedia communication under
prefix used in the terms hypertext and hyperme- this definition include a wall poster that includes
dia.) Table 3 shows examples of the visual and a photograph, some written paragraphs, and a
aural modalities in the network arrangement. map; a sentence that uses more than one font;
and an audio novel with a single narrator and at
The principle of synchronicity least one sound effect.
When considering composite-texts that use The use of “two” in the definition arises from
more than one modality, it becomes possible (and the simple principle “two is greater than one, and
indeed, sometimes essential) to transmit more one component-text does not comprise a com-
than one message at once, as the receiver can now posite-text.” Of course, the definition could adapt
receive messages through each of the modalities to any number (n) replacing “two,” but this could
used. The synchronous messages in the different cause problems in defining composite texts with
modalities may be considered independent of n − 1 component-texts using different representa-
each other from a perceptual point of view, tional systems.
although the receiver will usually make a cogni-
tive link between them (for example, associating 2. The production, transmission, and interpreta-
a “beep” from a computer with a visually percep- tion of a composite-text, where at least two of
tible error). Note that cognitive links may some- the component-texts use different representa-
times be made between two unintentionally tional systems in different modalities.
perceptibly synchronous messages.
A synchronous-text, therefore, is a special type This definition extends the previous one,
of composite-text containing an aural component- emphasizing a need to have more than one
text and a visual component-text (which may be modality in the text, with a similar justification
composite-texts themselves). The two component- for the choice of the number “two.” Examples
January–March 1998
texts are transmitted simultaneously, with the include an audio tour of an art gallery; a television
intention that the receiver make a cognitive link commercial with a “voice-over” (the voice of an
between the aural and visual perception. unseen narrator); and a greeting card that plays a
tune when opened.
Defining multimedia communication
This concrete and well-defined model of media 3. The production, transmission, and interpreta-
objects serves as a fundamental basis for defining tion of a composite-text, where the network
multimedia communication. The model itself does syntax is used at least once.
13
Authorized licensed use limited to: b-on: Instituto Politecnico de Beja. Downloaded on April 13,2024 at 11:37:48 UTC from IEEE Xplore. Restrictions apply.
.
This definition concentrates on the hyper- pre- For example, a cartoon strip may be hand
fix: If a network syntax is used, and the receiver drawn or computer generated, a piece of music
has some choice over the order in which some or may be transmitted via an audio recording or an
all of the component-texts are transmitted, the orchestra, and a newspaper may be read from
text is considered multimedia. Note that the defi- beginning to end, or selected articles may be cho-
nition includes no restrictions on the nature of the sen at will. All these considerations are external to
component-texts: they may all be of exactly the the definition of the text’s nature and can there-
same representational system. Examples include a fore be considered separately. The model does not
book; telephone touch-tone menus of recorded tie definitions of multimedia to specific hardware
spoken information; and a text-based interactive or to production and interpretation methods.
fiction story. This is the model’s power rather than a limita-
tion. It acknowledges that multimedia communi-
4. The production, transmission, and interpreta- cation is a complex process that cannot be simply
tion of a composite-text, where the network defined.
syntax is used at least once, and at least two of Thus, while the existing model produces much
the component-texts use different representa- broader definitions than those usually employed
tional systems. (concentrating as they do on the nature of the
text, and ignoring technology and methods), fur-
Like the previous definition, this one empha- ther defining the nature of the production, trans-
sizes the hyper- aspect of the text, but in this case mission, and interpretation permits further
it imposes restrictions on the nature of the com- refining of these definitions.
ponent-texts. Examples include a hypertext sys- In addition, models of text production, trans-
tem that emits a “beep” whenever a link is mission, and interpretation may be related to this
traversed; a pictorial encyclopedia; and an existing media model. Appropriate inter-model
indexed compact disk of a variety of different mappings may also be defined. For example, any
sound effects. multimedia text including a component-text using
a network syntax requires an interactive device.7
5. The production, transmission, and interpreta-
tion of a composite-text, where at least one of Related work
the component-texts is a synchronous-text. Two theoretical classifications of representa-
tional systems for multimedia communication
This definition resembles the second one, in have already been proposed, both based on exist-
that the use of more than one modality is impor- ing terminology that defines existing media types
tant. Examples include a picture of a waterfall, (graphics, beep, written sentence, sound).
accompanied by the sound of running water; a The model presented here is comparable to
film with a sound track; and an interactive film that defined by Arens et al.10 in two respects: the
with a sound track. augmentation syntax relates to Arens’ concept of
signs (or information carriers) having “channels,”
The model’s scope and potential and his definitions of complex exhibits and sim-
These definitions of multimedia communica- ple exhibits are similar to the definitions of com-
tion are important both for what they include and posite-text and single-text presented here.
for what they omit. The model permits defining The taxonomy proposed by Heller and Martin8
multimedia communication in terms of the text’s consists of two dimensions. The media expression
nature, with respect to a well-grounded semiotic dimension is related to a concrete-abstract con-
basis and the representational systems the text tinuum of expression: Elaboration media provide
uses. a “real world” depiction of the message (such as a
The model does not provide for consideration photograph), Representation media allow for the
of other general human communication issues, communication of a more abbreviated, more
IEEE MultiMedia
encompassed within the sample definitions’ “stylized” version of the original information
phrase “production, transmission and interpreta- (such as a map), and Abstraction media rely on
tion.” The model makes no reference to the metaphor and require that cultural context and
nature of this production, transmission, or inter- experience be considered (for example, a well-
pretation, although each of these steps may have known symbol). These three media expression
varying features. categories are similar to the three categories of
14
Authorized licensed use limited to: b-on: Instituto Politecnico de Beja. Downloaded on April 13,2024 at 11:37:48 UTC from IEEE Xplore. Restrictions apply.
.
defining name in the common parlance of multi- ligent Multimedia Interfaces, M.T. Maybury, ed., MIT
media studies. Press, Cambridge, Mass., 1993, pp. 280-306.
11. G.L. Lohse et al., “A Classification of Visual Representa-
Conclusion tions,” CACM, Vol. 37, No. 12, Dec. 1994, pp. 36-49.
The model presented here avoids the
temptation of using labels associated with existing Contact Purchase at Dept. of Computer Science and
text types. Instead it considers the basic compo- Electrical Engineering, The University of Queensland,
nents of the text and how they are arranged. St. Lucia 4072, Australia, e-mail hcp@csee.uq.edu.au.
15
Authorized licensed use limited to: b-on: Instituto Politecnico de Beja. Downloaded on April 13,2024 at 11:37:48 UTC from IEEE Xplore. Restrictions apply.