The MPEG-7 Standard

- A Brief Tutorial -

Ali Tabatabai
Sony US Research Laboratories
February 27, 2001

Outline

Objectives of the MPEG-7 Standard

Main Elements of MPEG-7
Scope of MPEG-7
MPEG-7 Application Areas
MPEG 7s relation with other standards

Why do we need MPEG-7 ?

Support for Advanced Query

Fast & Accurate Access

Need

Personalized Content
Production and
Consumption
Content Management
Automation

Visual
Audio
Sketch
3

MPEG: A Brief History (1)

MPEG: Moving Picture Experts Group

Established in 1988

ISO / IEC/JTC1/SC29/WG11
A Working Group of ISO/IEC in charge of the
Development of Standards for
Coded Representation of Digital Audio and Video

MPEG: A Brief History (2)

MPEG-1: Interactive CD and MP3

11 / 1992

MPEG-2: DTV, STB, DVD 11 / 1994

MPEG-4: Web and Mobility

ver1: 09 /1998

ver2: 11 /1999
MPEG-7:

???

08 / 2001

MPEG-21: Multimedia Framework

11 / 2001

MPEG-7: What Is It ?
THE MPEG
MPEG 77
THE
STANDARD
STANDARD
IS NOT a COMPRESSION Standard
similar to MPEG-1/2/4 or their
Extension

IS NOT a STANDARD for

FEATURE
EXTRACTION/MATCHING

Content Description of
Various Audio Visual
Information
Types of Audio Visual Information
Audio, Speech
Moving video, still pictures, graphics
Information on how objects are
combined in scenes

MPEG-7: Application Areas

Storage and retrieval of audiovisual databases (image, film,
radio archives)

Broadcast media selection (radio, TV programs)

Surveillance (traffic control, surface transportation, production
chains)

E-commerce and Tele-shopping (searching for clothes / patterns)

Remote sensing (cartography, ecology, natural resources
management)

Entertainment (searching for a game, for a karaoke)

Cultural services (museums, art galleries)
Journalism (searching for events, persons)
Personalized news service on Internet (push media filtering)
Intelligent multimedia presentations
Educational applications
Bio-medical applications

MPEG-7
Description Scope for AV Content
Description Granularity
Low-level
High-level

Form
Access
Classification
Link
Context

MPEG-7: Main Elements

Descriptors (D)
syntax and semantics of each feature representation

Description Schemes (DS)

structure and semantics of the relationships between
components

Description Definition Language (DDL)

creation of new DSs
modification/extension of existing DSs

MPEG-7: Major Functionalities

Systems

(ISO / IEC 15938 - 1)

Description Definition Language (ISO / IEC 15938 2)

Visual

(ISO / IEC 15938 - 3)

Audio

(ISO / IEC 15938 - 4)

Multimedia Description Schemes (ISO / IEC 15938

- 5)
Reference Software

(ISO / IEC 15938 - 6)

10

MPEG-7: Main Elements (2)

DDL

Instantiation
DS1

DS2

DS3

<Object>
<Label/>
<Definition/>
.
.
</Object>

DS2

D1

D2

D3

Systems
0001100

11

MPEG-7: Systems
It defines tools to:
provide for efficient storage and transport
synchronize between content and description
manage and protect intellectual property

12

MPEG-7: DDL and its Components

Description Definition Language:
Creation of the Ds and DSs: XML Schema & MPEG-7 Extensions
Instantiation of XML

XML Schema:
Data types
Simple and Complex types
Elements, attributes
Inheritance, Abstract types

MPEG-7 extensions:
Array and Matrix data type

13

MPEG-7: Audio
Sound Effects
Music Instrument Timbre
Spoken Content
Melody Contour

14

MPEG-7: Visual (1)

Color
quantization, dominant, scalable, color-structure, layout,
GoF/GoP

Texture
Shape
region-based, contour-based, 3D

Motion
camera motion, motion trajectory, parametric motion,
motion activity

15

MPEG-7: Visual (2)

Localization
spatio temporal

Others
face recognition

16

MPEG-7: Basic Visual Structures

Grid Layout
2D-3D Multiple View
Time Series
Spatial 2D Coordinates
Temporal Interpolation

17

Low level Audio Visual descriptors

Video segments

Still regions

Color
Camera motion
Motion activity
Mosaic

Moving regions

Color
Shape
Position
Texture

Audio segments

Color
Motion trajectory
Parametric motion
Spatio-temporal
shape

Spoken content
Spectral
characterization
Music: timbre,
melody

18

MPEG-7: MMDS Basic Elements

Root, Top-level
elements,
Packages

Time, Duration,
Medialocators

Language
Annotation,Person, Place

Basic elements
Datatype &
structures

Schema
tools

Link & media

localization

Basic DSs

19

MPEG 7: Content Management and

Description
Format, Coding, Instances,
Identification, Transcoding
Hint, etc.
(Several instances)

Title, Creator, Creation

location & date, Purpose,
Classification, Genre,
Review, Parental guidance,
etc. (Author generated)
Rights holder, Access rights,
Usage Record, Financial aspects,
etc. (Evolution)

Creation &
production
Media

Content
Usage

Content management
Content description
Structural
aspects

Conceptual
aspects

Viewpoint of the structure: Segments

Basic
elements
Spatial
/ temporal structure
Audio,&video low-level
Ds
Schema
Datatype
Elementary
tools
structures semantic information.

Viewpoint of conceptual notions

Events, objects, abstract concepts, and
Link & media
their
relation
Basic
DSs
localization

20

Example of Segment trees

SR1:
Creation, Usage meta
information
Media description
Textual annotation
Color histogram, Texture

SR6:
Color Histogram
Textual annotation

Background

SR3:
Shape
Color Histogram
Textual annotation

Foreground

SR2:
Shape
Color Histogram
Textual annotation
SR5:
Shape
Textual annotation
SR4:
Shape
Color Histogram
Textual annotation

21

Time
Axis

Segment Tree
Shot1
Segment 1
Sub-segment 1

Shot2

Semantic DS (Events)

Shot3

Introduction
Summary

Sub-segment 2

Sub-segment 3

Program logo

Studio
Overview

Sub-segment 4

News Presenter

segment 2

News Items
Segment 3

International
Clinton Case
Pope in Cuba

Segment 4

National
Segment 5

Segment 6

Twins
Sports
Closing

Segment 7

22

MPEG 7: Navigation and Access

Creation &
Efficient
support of: discovery,
production
browsing,
navigation, visualization
Media

Content
Usage

Content management

Navigation &
Access

Summary

Content description
Structural
aspects
Basic elements
Datatype &
structures

Conceptual
aspects

Substitution of the original content

Adaptation to terminal, network, or
Schema user preferences
Link & media
tools

localization

Variation

Basic DSs

23

MPEG 7: Hierarchical summary

Hierarchical
Summary

HighlightLevel

Highlight
Segment

Highlight
Segment

Highlight
Segment

HighlightLevel

Highlight
Segment

Highlight
Segment

Highlight
Segment

Highlight
Segment

A-V
Data

24

MPEG 7: Sequential summary

Frame
Property

Sequential
Summary

Text
Property

Sound
Property

Frame
Property

Frame
Property

Text
Property

Sound
Property

Sound
Property

A-V
Data

25

MPEG 7: Variation
Universal Multimedia Access
Adapt delivery to network and terminal characteristics (QoS)

Fidelity
H

Variation

Source
E

VIDEO

IMAGE

TEXT

Modality

AUDIO

26

MPEG-7: Content Organization

Collection &
Classification

Content organization

Description and organization of

Creation &
collection of documents

Analytic
Model

Navigation &
Access

production
Media

Content
Usage

Content management

Summary

Content description
Structural
aspects

Conceptual
aspects

Variation

Basic elements
Datatype &
structures

Schema
tools

Link & media

localization

Basic DSs

27

MPEG-7: Collection

28

MPEG 7: User Interaction

Collection &
Classification

Content organization

Analytic
Model

Navigation &
Access

Creation &
production
Media

Content
Usage

Content management

Summary

Content description
Structural
aspects

Conceptual
aspects

User
preferences

Variation

User identification and preferences:

Filtering, search and browsing

Basic elements
Datatype &
structures

User
Interaction

Schema
tools

Link & media

localization

Basic DSs

29

MPEG-7
Its Relation with other standards
AHG on Metadata harmonization:
SMPTE: Metadata dictionary, KLV encoding
Dublin Core Metadata Initiative
European Broadcast Union

AHG on TV AnyTime Application

Large number of Liaisons:
SMPTE
Dublin Core
W3C (XML Schema)
etc.

30

MPEG-7: TimeLine - The Work Plan

1996

Competition:
Divergence
Individual work
Definition
scope and
r
1998

Call for proposals

Convergence
1999

Working draft

2000

Committee draft
Final committee
draft

2001

International
standard
Draft international
standard

Conclusions on AV Content Description

and MPEG-7
MPEG-7:
AV content description for interoperable
application

Description Definition Language:

XML Schema (flexibility) + Binary version
(efficiency)

Description Schemes:
Library of description tools
Covers a wide range of generic needs

32