Chapter 2:

Multimedia Basics and

Representation
•Digital multimedia characteristics
•Audio format and MIDI
•Image format and Color models
•Video format and Color models

chapter2: Multimedia Basics and Representation 1

Characteristics of a Multimedia
Systems
• A Multimedia systems has four basic
characteristics:
• Multimedia systems must be computer
controlled.
• Multimedia systems are integrated.
• The information they handle must be
represented digitally.
• The interface to the final presentation of media
is usually interactive.

2
 Characteristics of a Multimedia
Systems
Computer Controlled
• Producing the content of the information – e.g. by
using the authoring tools, image editor, sound and
video editor
• Storing the information – providing large and
shared capacity for multimedia information.
Transmitting the information – through the
network.
• Presenting the information to the end user – make
direct use of computer peripheral such as display
device (monitor) or sound generator (speaker).

chapter1: Introduction to Multimedia 3

 Characteristics of a Multimedia
Systems
Integrated
• All multimedia components (audio, video, text,
graphics) used in the system must be somehow
integrated.
• Every device, such as microphone and camera is
connected to and controlled by a single
computer.
• A single type of digital storage is used for all
media type.
• Video sequences are shown on computer screen
instead of TV monitor.
chapter1: Introduction to Multimedia 4
 Characteristics of a Multimedia
Systems
Interactivity
• Level 1: Interactivity strictly on information
delivery. Users select the time at which the
presentation starts, the order, the speed and the
form of the presentation itself.
• Level 2: Users can modify or enrich the content of
the information, and this modification is recorded.
• Level 3: Actual processing of users input and the
computer generate genuine result based on the
users input.

chapter1: Introduction to Multimedia 5

 Characteristics of a Multimedia
Systems
• Digitally Represented
• Digitization: process involved in transforming
an analog signal to digital signal.

chapter1: Introduction to Multimedia 6

 Audio Format: Digital Audio
Application of Digital Audio - Selected Examples
• Music Production
– Hard Disk Recording
– Sound Synthesis
– Samplers
– Effects Processing
• Video
- Audio Important Element: Music and Effects
• Web
- Many uses on Web
– Spice up Web Pages
– Listen to Cds
– Listen to Web Radio
• Many More Uses
- try and think of some?
7
 Digitization of Sound
• Sound is a continuous wave that travels
through the air
• The wave is made up of pressure
differences. Sound is detected by
measuring the pressure level at a
particular location.
• Sound waves have normal wave
properties (reflection, refraction,
diffraction, etc.).

8
 Digitizing Audio
• That is the basic idea of digitizing a sound
unfortunately things are (practically speaking)
not so simple.

• Questions for producing digital audio

(Analogue-to-Digital Conversion):
1. How often do you need to sample the signal?
2. How good is the signal?
3. How is audio data formatted?

9
 Computer Manipulation of Sound
• Once Digitised processing the digital sound is essentially
straightforward although it depends on the processing
you wish to do (e.g. volume is easier to code than
accurate reverb)
Sample Rates and Bit Size
How do we store each sample value (Quantisation)?
• 8 Bit Value
– (0-255)
• 16 Bit Value
– (Integer) (0-65535)
How many Samples to take?
– 11.025 KHz Speech (Telephone 8KHz)
– 22.05 KHz Low Grade Audio
(WWW Audio, AM Radio)
– 44.1 KHz CD Quality

10
 Implications of Sample Rate and
Bit Size
Affects Quality of Audio
• Ears do not respond to sound in a linear fashion
• Decibel (dB) a logarithmic measurement of
sound
• 16-Bit has a signal-to-noise ratio of 98 dB -
virtually inaudible
• 8-bit has a signal-to-noise ratio of 50 dB
• Therefore, 8-bit is roughly 8 times as noisy
– 6 dB increment is twice as loud

11
 Typical Audio Formats
• Popular audio file formats include .au (Unix
workstations), .aiff (MAC, SGI), .wav (PC,
DEC workstations)

• A simple and widely used audio

compression method is Adaptive Delta Pulse
Code Modulation (ADPCM). Based on past
samples, it predicts the next sample and
encodes the difference between the actual
value and the predicted value.
12
 Computer Music - MIDI

• Sound waves, whether occurred natural or man-

made, are often very complex, i.e., they consist
of
many frequencies. Digital sound is relatively
straight forward to record complex sound.
However, it is quite difficult to generate (or
synthesize) complex sound.
• There is a better way to generate high quality
music. This is known as MIDI — Musical
Instrument Digital Interface.

13
 MIDI
• It is a communication standard developed in the
early 1980s for electronic instruments and
computers.
• It specifies the hardware connection between
equipments as well as the format in which the
data are transferred between the equipments.
• Common MIDI devices include electronic music
synthesisers, modules, and MIDI devices in
common sound cards.
• General MIDI is a standard specified by MIDI
Manufacturers Association. To be GM compatible,
a sound generating device must meet the General
MIDI system level 1 performance requirement.

14
 MIDI
• Musical Instrument Digital Interface
a protocol that enables computer, synthesizers, keyboards, and other
musical devices to communicate with each other.
 Setup:
 MIDI OUT of synthesizer is
connected to MIDI IN of sequencer.
 MIDI OUT of sequencer is connected
to MIDI IN of synthesizer and
THRU IN OUT IN OUT
"through" to each of the additional
Synthesizer/Keyboard MIDI Interface/Sound Card sound modules.
(Sequencer)  Working:
IN THRU
MIDI Module A  During recording, the keyboard-
equipped synthesizer is used to send
IN THRU MIDI message to the sequencer,
MIDI Module B which records them.
Etc.  During play back, messages are sent
Typical Sequencer setup out from the sequencer to the sound
modules and the synthesizer which
15
will play back the music.
 MIDI: Data Format
• Information traveling through the hardware is encoded
in MIDI data format.
• The encoding includes note information like beginning of
note, frequency and sound volume; upto 128 notes
• The MIDI data format is digital
• The data are grouped into MIDI messages
• Each MIDI message communicates one musical event
between machines. An event might be pressing keys,
moving slider controls, setting switches and adjusting
foot pedals.
• 10 mins of music encoded in MIDI data format is about
200 Kbytes of data. (compare against CD-audio!)
16
 MIDI files
When using computers to play MIDI music, the
MIDI data are often stored in MIDI files. Each
MIDI files contains a number of chunks. There are
two types of chunks:
• Header chunk — contains information about the
entire file: the type of MIDI file, number of tracks
and the timing.
• Track chunk — the actual data of MIDI track.

There three types of MIDI file:

• 0 single multi-channel track
• 1 one or more simultaneous track of a sequence
• 2 one or more sequentially independent single-
track patterns
17
 Tracks, channels and patches

• Multiple tracks can be played at the same

time.
• Each track can be assigned to a different
channel.
• Each channel can accept more than one
track.
• Each channel is assigned a patch,
therefore generates sound of a particular
instrument.
18
How MIDI Sounds Are Synthesized

A simplistic view is that:

• the MIDI device stores the characteristics
of sounds produced by different sound
sources;
• the MIDI messages tell the device which
kind of sound, at which pitch is to be
generated, how long the sound is played
and other attributes the note should have.

19
There are two ways of synthesizing sounds:
• FM Synthesis (Frequency Modulation)—Using
one sine wave to modulate another sine wave,
thus generating a new wave which is rich in
timbre. It consists of the two original waves, their
sum and difference and harmonics.
The drawbacks of FM synthesis are: the generated
sound is not real; there is no exact formula for
generating a particular sound.
• Wave-table synthesis— It stores representative
digital sound samples. It manipulates these
samples, e.g., by changing the pitch, to create the
complete range of notes.
20
 MIDI Sound Attributes

• The shape of the amplitude envelop has

great influence on the resulting character
of sound. There are two different types of
envelop:
• Diminishing sound — gradually die out;
• Continuing sound — sustain until turned
off.

21
 Image Format and color model:
Analog versus bitmap images
• The visual world is analog, which is to say that
real-world images is a continuous spectrum of
colors.

• Infinity is uncomfortably large to a computer. A

computer-stored image is split into little bits of
light.

• A bitmap is a matrix of colored dots.

.
chapter 2: Image format 22
 Bitmaps, Pixels & Colors
• The colored dots that make up a bitmap are
properly called "pixels".

• Color is represented on a computer by using

varying amounts of red, green and blue light.

• In the simplest sort of bitmapped image, each

pixel is represented by three numbers to store
the amounts of red, green and blue light that
define the color of the pixel in question.

chapter 2: Image format 23

 True Color Bitmap
• Each pixel requires one byte for each color index
for a total of three bytes per pixel.

• As a byte represents eight bits, each pixel

requires 24 bits to store all its color information.

• This defines the maximum number of discrete

colors this sort of bitmap can represent as 224, or
16,777,216.

• Such graphics are referred to as “True Color"

images, or just as "24-bit" graphics.
chapter 2: Image format 24
Bit-Depth = Color-Depth
• Number of Colors = 2^(Bit-depth)
• Bit-depth is the number of bits.
– It is also called “Color resolution”.

Bit depth Color resolution Calcuation

1-bit 2 colors 2^1 = 2
2-bit 4 colors 2^2 = 4
3-bit 8 colors 2^3 = 8
4-bit 16 colors 2^4 = 16
8-bits 256 colors 2^8 = 256
16-bits 65,536 colors 2^16 = 65536
24-bits 16,777,215 colors 2^24 = 16.7 million

chapter 2: Image format 25

 Vector Graphics
• Vector graphics define pictures as collections of
lines, ellipses, triangles, polygons and other basic
graphic "primitives."

• Vector graphics are limited to storing mechanical

art

• Vector graphic formats include

– Corel Draw files, which use the extension CDR,
– AutoCAD drawings, which use the extension
DXF.
chapter 2: Image format 26
Bitmap vs. Vector graphics

chapter 2: Image format 27

 Compression
Graphic file formats can be regarded as being of three types.
• The first type stores graphics uncompressed.
– Windows BMP (.bmp) files are an example of this sort of
format.
• The second type is called "non-lossy“ or “lossless”
compression formats.
– Most graphic formats use lossless compression - the GIF
formats are among them.
• The third type of bitmapped graphic file formats is called
"lossy" compression.
– the details are what prevent areas from being all the same
color, and as such from responding well to compression.
– perhaps too subtle to be discernable by your eye
chapter 2: Image format 28
 Image File Formats
Examples of common interchange formats are:
•- GIF - Graphics Interchange Format
•- PNG - Portable Network Graphics
•- JFIF - JPEG (Joint Photographic Experts Group) File
Interchange Format
•- TIFF - Tagged Image File Format
•- PGM - Portable Gray Map
• -PBM - Portable Bit Map
• -PPM - Portable Pix Map
• - FITS - Flexible Image Transport System
chapter 2: Image format 29
 Test Image in Various Formats
• JPG 67 kb
• GIF 185 kb (256 colors, optimized)
• BMP 899 kb (640*479*3/1024 = 898.125)
• TIF 902 kb
• PNG 418 kb
• RAW 900 kb

chapter 2: Image format 30

GIF (Graphics Interchange Format )
• GIF creates a table of up to 256 colors from a pool of 16
million.
• If the image has fewer than 256 colors, GIF can render
the image exactly.
• GIF is "lossless" only for images with 256 colors or less.
For a rich, true color image, GIF may "lose" 99.998% of
the colors.
• The GIF format can contain animations. Its internal structure is
such that it can store multiple images and the controls to make
them appear as real time animation
– animated GIF.
• The GIF format also allows a special color as to be specified as
"using the background." This results in the image looks like
transparent
– transparent GIF.
chapter 2: Image format 31
 Animated GIF
• Advantages:
– No need other software or plugins
– GIF is the standard format on the Web.
– GIF animated tools are available to use.
• The file contains layered frames on
top of each other.
• Tips:
– Avoid more than one animated GIF on
a page.
– Avoid animated GIF on text-rich pages.
– Examine the pause between repetitions.

chapter 2: Image format 32

 Transparency GIF
• The image can be shapes other than rectangles.
• One position of the color palette is designated as
“Transparent”.
• All pixels of the image that have this particular color
index will be painted as transparent when viewing.

a transparent GIF

chapter 2: Image format 33

 JPG/JPEG Format
• JPEG – Joint Photographic Experts Group
• The JPEG format uses "lossy" compression to get more
graphics into a smaller file than would otherwise be
possible.
• An image written to the JPEG format will be degraded.
• The amount of degradation, the "quality factor," can usually
be set in a graphics software.
– If the value is set to 100, almost no degradation will occured.
– If it is set to a value close to zero, the resulting image will be a very
small file but unrecognizable.
– The default value of 75 is usually a good compromise.

chapter 2: Image format 34

 Progressive JPEG
• There are two types of JPEG files extant as of this writing,
called "sequential" and "progressive".
– A sequential JPEG file stores its image as a simple
bitmap.

– A progressive JPEG files stores its image such that it can

appear initially out of focus when it begins to download
to a web page, and resolve itself as more of the image is
received by your web browser.

• Advantage: Provide indication of the whole image to the viewer

before the entired image is loaded.
• Disadvantage: Require more computational power to display.

35
chapter 2: Image format
 GIF Versus JPG
• For photographic images JPG is better both for quality
and file size
• For line art GIF is better both for quality and file size
• The future of line art is probably in vector file formats
like SVG (Scalable Vector Graphics)
• VML (Vector Markup Language) is Microsoft backed
GIF JPEG
Best application Line Art, Image with few Photographs, Image with
color text many colors
How to reduce Interlace Interlace (Progressive)
display time?
Display speed Fast Slower, more computation
Benefits Transparency, Greatest compress for
Animation photographs, more color
Max. color 256 16.7 million
chapter 2: Image format 36
 TIFF (Tagged Image File Format )
• TIFF can be lossless or lossy.
• The details of the image storage algorithm are
included as part of the file.
• TIFF is used almost exclusively as a lossless image
storage format that uses no compression at all.
• Most graphics programs that use TIFF do not use
compression .
• File sizes are quite big.
– (Sometimes LZW is used, but it is not universally
supported.)

chapter 2: Image format 37

 PNG ( Portable Network Graphics )
• PNG is a lossless storage format.
• It looks for patterns in the image that it can use to compress file size.
• The compression is exactly reversible, so the image is recovered exactly.
• Support three image type: true color, grayscale, palette-based (8-bit).
• JPEG supports the first 2.
• GIF supports the 3rd one.
Advantages
– Better Compression
• Deflate is an improved version of the Lempel-Ziv compression algorithm.
– Improve Interlacing
• Display image quicker than Interlaced GIF.
– True Color and Transparency
• Support 16-bit (Grey scale) or 48-bit (True Color)
• 16-bit for alpha channel (Transparency).
– Gamma storage
• Store the gamma setting of the platform of the creator.
Disadvantages
– Not support by old browsers (Netscape 2,3,4 and IE 2,3,4)

chapter 2: Image format 38

BMP bitmap (DIB) file format /bitmap
• BMP is an uncompressed proprietary format invented
by Microsoft.
• There is really no reason to ever use this format.

PSD
•PSD is a proprietary format used by Photoshop.
•Working format to edit images in the software
•This package uses layers to build complex images, and layer
information may be lost in the nonproprietary formats such as
TIFF and JPG.
•However, it is best to save the product as a TIFF or JPG, so it
can be viewed in the future when software changes.
chapter 2: Image format 39
 Number of colors
• Images have differing numbers of
color within them.
• Black and white images take up
the least amount of space with 1
bit of information.
• Current video cards are able to
display 24 bits of color per pixel.
• This makes it possible for 16
million colors to be displayed (224
colors).
• This requires a significant amount
of disk space.
chapter 2: Image format 40
 8-bit and 24-bit photos
• Can you see the difference?

chapter 2: Image format 41

 Color Palettes
The store images having a maximum of 16,777,216 colours, is
one which can store images having a maximum of 256
colours.
• Such files are referred to as using "palette-colour“. The
colours in a palette-colour file are derived from a potential
palette of 16,777,216 colours, but no more than 256 of them
can be used in any one image.
The Web Palette
• Color in Browser – 24-bit color displays on an 8-bit monitor
display.
• Need a common palette (Web Palette) for browser, guarantees the
image will look the same on different platforms.
• If the browser is in 24-bit display, the palette does not affect the
images.
• Web Palette consists of 216 colors
• Also called: Web-safe Palette, Non-dithering Palette, 6x6x6 cube.
chapter 2: Image format 42
 Which Color Palette?

Black (0,0,0) Blue (0,0,255)

Green (0,255,0)

Red (255,0,0)

White
(FF,FF,FF)
(255,255,255)
chapter 2: Image format 43
6x6x6 Color Palette

chapter 2: Image format 44

 Define RGB Values
• Decimal
– Ranging from 0 to 255.
– 51-51-255 means red value is 51, green value is 51, blue value is
255.
• Hexadecimal
– Base-16 number system
– Only six characters to describe RGB color.
– 51-51-255 is 3333FF.
• Percentage
– 51-51-51 is 20%-20%-100%
– 00,33,66,99,CC,FF – 0%,20%,40%,60%,80%,100%
– Some old Macintosh programs use it.

chapter 2: Image format 45

 Gamma control
• the overall brightness of the display of a computer monitor.
• The relationship of voltage and the light intensity.
• Default gamma setting varies from platform to platform
– Macintosh – 1.8 Gamma
– PC – 2.2 Gamma
– UNIX – 2.4 Gamma
• The higher the gamma value, the darker the display.
• Image created on PC looks wash out on Macintosh.

Different Gamma
Value

chapter 2: Image format 46

 Hue - Ink Mixing
• Subtractive mix of
transparent inks
• Primary: CMY(Cyan,
Magenta, yellow)
• Secondary: RGB
• ~Black: C + M + Y
• Actually use CMYK
to get true black

Assumption: ink printed on pure white paper

CMY = White – RGB:

C = 1 – R, M = 1 – G, Y = 1 – B
CMYK from CMY (K is black ink): K = min(C, M, Y)
C = C – K, M = M – K, Y = Y - K
chapter 2: Image format 47
Hue - Light Mixing

• Additive mix of colored

lights
• Primary: RGB
• Secondary: CMY
• White = R + G + B
• Show demonstration of
optical mixing

chapter 2: Image format 48

 Video Format

• Video formats involve two distinct, and very

different technology
• concepts: containers (sometimes called wrappers)
& codecs (short for coder/decoder).
• Codecs are used inside of a container and because
of this video formats can be confusing.

chapter 2: Image format 49

• Container

The container describes the structure of the file: where the

various pieces are stored, how they are interleaved, and
which codecs are used by which pieces. It may specify an
audio codec as well as video. It is used to package the
video & its components (audio/metadata) and is identified
(usually) by a file extension such as .AVI, .MP4 or .MOV.

• Codec

A codec (short for "coder/decoder") is a way of encoding

audio or video into a stream of bytes. It is the method used
to encode the video and is the chief determiner of quality.
chapter 2: Image format 50
 How do they work together?
Think of the container as the file itself.
Think of the codec as its contents.
The important thing to realize is that most good container
formats can hold many codecs.
For example a .MOV container can hold almost any kind
of codec data. The same goes for .MP4 and even .AVI files
can hold a wide variety of codecs as their contents. In no
way does the container decide the quality or features of
the video itself, that is up to the codec.

The proper way to describe video is to indicate both: A

.MOV file containing H.264 data. An .AVI file containing
DivX data. Professionals use shorthand’s for this, saying
things like "Give me anchapter
H.264 Quicktime file (.mov)".
2: Image format 51
Why is it so complicated?
At first this seems straightforward, but it's not. The
problem lies in the confusion and imprecision in the
common use of these terms. To make matters worse,
software companies try to simplify their documentation
and instructions by ignoring the difference altogether.
The result is that people believe that phrases like "I'll
give you a MOV file," or "An MP4 file will be fine" are
legitimate ways to talk about video.

To make life even more confusing, some names, such as

"mpeg-‐4", describe both a codec and a container, so it's
not always clear from context, which is being used. You
could have a movie encoded with an mpeg-‐4 codec
inside an avi container, for example, or a movie encoded
with the Sorenson codec inside an mpeg-‐4 container.

chapter 2: Image format 52

 What Video Format Should I Use?
File size & quality
Digital Storage Space -‐ To calculate the amount of storage space you will
need for a project, digital video requires approximately 200 MB per
minute of footage, or roughly 12 GB per hour. Of course this varies
according to your recording device and the quality it is set to record at.

Frames per Second – The standard for FPS is 29.97, increasing the FPS allows
for more images per second thus a smoother image. Decreasing FPS will
make the video a bit choppy and not nearly as smooth.

Video Bitrate -‐ Bitrate is a measurement of the number of bits that are

transmitted over a set length of time. Your overall bitrate is a
combination of your video stream, audio stream & metadata in your file
with the majority coming from your video stream. The higher the bit rate
the better the quality the bigger it will be.
Resolution – this is the number of pixels present in the images of the video. This
determines whether your video is standard definition or high definition. The
higher the resolution the clearer the image the bigger the file.
chapter 2: Image format 53
 List of Most Common Codecs
• MPEG (Moving Pictures Expert Group): three video formats, MPEG 1, 2,
and 4.
• MPEG-‐1: Old, supported by everything (at least up to 352x240),
reasonably efficient. A good format for the web.
• MPEG-‐2: A version of MPEG-‐1, with better compression. 720x480. Used
in HDTV, DVD, and SVCD.
• MPEG-‐4: A family of codecs, some of which are open, others Microsoft
proprietary.
• H.264: Most commonly used codecs for videos uploaded to the web. Part
of the MPEG-‐4 codec.
• MPEG spinoffs: mp3 (for music) and VideoCD.
• MJPEG (Motion JPEG): A codec consisting of a stream of JPEG images.
Common in video from digital cameras, and a reasonable format for
editing videos, but it doesn't compress well, so it's not good for web
distribution.
• DV (Digital Video): Usually used for video grabbed via firewire off a
video camera. Fixed at 720x480 @ 29.97FPS, or 720x576 @ 25 FPS. Not
very highly compressed. chapter 2: Image format 54
 List of Most Common Codecs
• WMV (Windows Media Video): A collection of Microsoft proprietary
video codecs. Since version 7, it has used a special version of MPEG4.
• RM (Real Media): a closed codec developed by Real Networks for
streaming video and audio.
• DivX: in early versions, essentially an ASF (incomplete early MPEG-‐4)
codec inside an AVI container; DivX 4 and later are a more full MPEG-
‐4 codec…no resolution limit. Requires more horsepower to play than
mpeg1, but less than mpeg2. Hard to find mac and windows players.
• Sorenson 3: Apple's proprietary codec, commonly used for distributing
movie trailers (inside a Quicktime container).
• Quicktime 6: Apple's implementation of an MPEG4 codec.
• RP9: a very efficient streaming proprietary codec from Real (not
MPEG4).
• WMV9: a proprietary, non-‐MPEG4 codec from Microsoft.
• Ogg Theora: A relatively new open format from Xiph.org.
• Dirac: A very new open format under development by the BBC.

chapter 2: Image format 55

 List of Most Common Containers
• AVI (Audio Video Interleave): a Windows' standard multimedia
container.
• MPEG-‐4 Part 14 (known as .mp4): is the standardized container for
MPEG-‐4.
• FLV (Flash Video): the format used to deliver MPEG video through
Flash Player.
• MOV: Apple's QuickTime container format.
• OGG, OGM & OGV: open-‐standard containers.
• MKV (Mastroska): another open-‐specification container that
you've seen if you've ever downloaded anime.
• VOB (DVD Video Object): It's DVD's standard container.
• ASF: a Microsoft format designed for WMV and WMA—files can
end in .wmv or .asf

chapter 2: Image format 56

 Sample Bit Rates
• 16 kbit/s – videophone quality (minimum necessary for a consumer-‐acceptable
“talking head” picture using various video compression schemes)
• 128 – 384 kbit/s – business-‐oriented video conferencing quality using video
compression
• 1.5 Mbit/s max – VCD quality (using MPEG1 compression)
• 3.5 Mbit/s typ — Standard-‐definition television quality (with bit-‐rate reduction
from MPEG-‐2 compression)
• 9.8 Mbit/s max – DVD (using MPEG2 compression)
• 8 to 15 Mbit/s typ – HDTV quality (with bit-‐rate reduction from MPEG-‐4 AVC
compression)
• 19 Mbit/s approximate — HDV 720p (using MPEG2 compression)
• 24 Mbit/s max — AVCHD (using MPEG4 AVC compression)
• 25 Mbit/s approximate — HDV 1080i (using MPEG2 compression)
• 29.4 Mbit/s max – HD DVD
• 40 Mbit/s max – Blu-‐ray Disc (using MPEG2, AVC or VC-‐1 compression)

chapter 2: Image format 57