Module 1

INTRODUCTION TO MULTIMEDIA DATABASES

Prof. Dr. Naomie Salim Faculty of Computer Science & Information Systems Universiti Teknologi Malaysia

The Explosion of Digital Multimedia Information

We interact with multimedia everyday Large amount of text, images, speech & video converted to digital form Advantages of digitized data over analog
Easy storage Easy processing Easy sharing

Give examples of multimedia applications that deals with storing, retrieving, processing and sharing of multimedia data

Eg 1. Journalism
Journalist to write article about influence of alcohol on driving Investigation involved:
Collect news articles about accidents, scientific reports, television commercials, police interviews, medical experts interviews

Illustration:
Search photo archives, stock footage companies for good photos shocking, funny, etc.

Other examples
Searching movies
Based on taste of movies already seen Based on movies a friend favor

Searching on web
Eg. searching Australian Open website (http://www.ausopen.org) Integrate conceptual terms + interesting events give info about video segments showing female American tennis players going to the net

Retrieval problems
EMPLOYEE (Name: char(20), City: Char(20), Photo: Image)
How do you select employees in Skudai? How do you select employees that wear tudung, wear glasses, fair and have a mole under the lips?

Characteristics of Media Data

Medium - Information representation
Alphanumeric Representation of audio, video and image

Static vs dynamic
Static: do not have time dimensions (alphanumeric data, images, graphics) Dynamic: have time dimensions (video, animation, audio)

Multimedia
Collection of media types used together At least one media types must be non-alphanumeric

Digital representation of text

OCR techniques convert analog text to digital text Eg. of digital representation: ASCII
Use 8 bits Chinese char requires more space Storage requirements depend on number of characters

Structured documents becoming more popular

Docs consist of titles, chapters, sections, paragraphs, etc. Standards like HTML and XML used to encode structured information

Compression of text
Huffman, arithmetic coding Since storage requirements not too high, less important than multimedia data

Digital representation of audio

Audio
air pressure waves with frequency, amplitude Human hears 20-20,000 Hertz Low amplitude soft sound

Digitizing pressure waveforms

Transform into electrical signal (by microphone) Convert into discrete values
Sampling: continuous time axis divided into small, fixed intervals Quantization: determination of amplitude of video signals at beginning of each time interval Human cannot notice difference between analog & digital with enough high sampling rate and precise quantization

Audio storage requirements

Example of a CD audio
16 bits per sample 44,000 samples per second Two (stereo) channels Requirements = 16 * 44,000 * 2 bits = 1.4 Mbit per second

Compression (examples)
Masking: Discard soft sound because not audible by louder sound Speech: coding of lower frequency sounds only MPEG: audio compression standards

Digital representation of image

Scan analog photos & pictures using scanner
Analog image approximated by rectangle of small dots In digital camera, ADC is built-in

Image consists of many small dots or picture elements (pixels)

Gray scale: 1 byte (8 bits) per pixel Color: 3 color (RGB) of one byte each Data required for 1 rectangular screen
A = xyb A:number of bytes needed, x: # pixels per horizontal line, y: # horizontal lines, b: # bytes per pixel

Image compression
Exploit redundancy in image & properties of human perception
Spatial redundancy: pixels in certain area often appear similar (golden sand, blue sky) Human tolerance: error still allows effective communication

Eg. of image compression

Transform coding Fractal image coding

Digital representation of video

Sequence of frames or images presented at fixed rate
Digital video obtained by digitizing analog videos or digital cameras Playing 25 frames per second gives illusion of continuous view

Amount of data to represent video

1 second, image: 512 lines, 512 pixels per line, 24 bits per pixel, 25 frames per second 512 * 512 * 3 * 25 = 19 Mbytes

Compression of video
Compressing frames of videos: similar to image
Reduce redundancy & exploit human perception properties

Temporal redundancy: neighboring frames normally similar, remove by applying motion estimation & compression
Each image divided into fixed-sized blocks For each block in image, the most similar block in previous image is determined & pixel difference computed Together with displacement between the two blocks, this difference stored or transmitted

MPEG-1 (VHS, pixel based coding): coding of video data up to speed of 1.5 Mbits per second MPEG-2 (pixel based coding): coding of video data up to speed of 10 Mbits per second MPEG-4 (multimedia data, object based coding) : coding of video data up to speed of 40 Mbits per second, tools for decoding & representing video objects, support content-based indexing & retrieval

How to search for images or multimedia data?

Analyze one by one? No! Takes too long! Have to use metadata instead of searching directly, search for metadata that have been added to it Metadata requirements to be valuable for searching:
Description of multimedia object should be as complete as possible Storage of metadata must not take too much overhead Comparison of two metadata values must be fast

Metadata of Multimedia Objects

Descriptive data
Give format or factual info about multimedia object Eg.: author name, creation date, length of multimedia object, representation technique Eg. standard for descriptive data: Dublin core Can use SQL (metadata condition in WHERE clause)

Metadata of Multimedia Objects (cont.)

Annotations
Textual description of contents of objects Eg.: photo description in Facebook Either free format or sequence of keywords Manual text annotations allow Information Retrieval techniques to be used but
Time consuming, expensive Subjective, incomplete

Structured concepts (eg semantic web, ER-like schema) can be used to describe content through concepts, their relationships to each other & MM object but
Also slow and expensive

Metadata of Multimedia Objects (cont.)

Features
Derive characteristics from MM object itself Need language to describe features, eg. MPEG-7 Process to capture features from MM object is called feature extraction
Performed automatically, sometimes with human support

Two feature classes

Low-level features High-level features

Low-level Features
Grasp data patterns & statistics of MM object Depend strongly on medium Extraction performed automatically Eg. for text
List of keywords with frequency indicators

Eg. for audio

Representation
Amplitude-time sequence: quantification of air pressure at each sample Silence:0, > silence:+ve amplitude, < silence:-ve amplitude

Eg. Low-level features derived

Energy (loudness of signal), ZCR(zero crossing rate-frequency of sign change)-high indicate speech, silence ratio(low indicates music)

Low-level features (cont.)

Eg. for images
Color histograms: # pixels having color of certain range Spatial relationships: eg. blue patterns appears above yellow (beach photo), Contrast: # dark spots neighboring light spots

Eg. for video

Use low-level features for image Eg. of temporal dimension: shot change-when pixel difference between two images is higher than certain threshold
Shot- sequence of images taken with same camera position

High-level features
Features which are meaningful to end user, such as golf course, forest How can we bridge semantic gap between low level and high level features
High level feature extraction from low level features Eg. text containing words football, referee football match text Eg. Speech to text translators (low level audio features to text) Eg. Video-Domain specific: loud sound from crowd, round object passing white line, followed by sharp whistle-goal

Multimedia Information Retrieval System (MIRS)

Component of MIRS - Archiving

MM data stored separately from its metadata
Voluminous Visible or audible delays in playback unacceptable

MM data managed separately in MM content server

Objects get identification to be used by other parts of MIRS at storage time Have to deal with compression and protection

Component of MIRS Feature Extraction (Indexing)

Extraction of metadata (annotations, descriptions, features) from incoming multimedia object Algorithms have to consider extraction dependencies. Eg.:
Video object segmented, choose key frame for each segment Extract low-level features from key frame Based on low-level features, classify into shots of audience, fields, close-ups For field shots, detect positions of players Extract body related features of players Determine where net playing begins and ends

Have to consider incremental maintenance (modification of MM objects, extractors, extraction dependencies)

Incremental Maintenance in ACOI Feature Extraction Architecture

Component of MIRS - Searching

Multimedia queries are diverse, can be specified in many different ways No exact match, many ways to describe MM objects Specifying information need
Direct user specifies info. need herself Indirect user relies on other users

Possible Querying Scenarios

Possible Querying Scenarios (cont.)

Queries based on Profile
Users expose preferences in one way or another Preferences stored in user profile in MIRS Can use profile of a friend if not sure & trusted

Queries based on Descriptive Data

Based on format and fact about MM object Eg. all movies with Director = Steven Spielberg

Possible Querying Scenarios (cont.)

Queries based on Annotations
Text-based: keywords or natural language Eg. Show me video in which Barack Obama shakes hand with Mahathir Mohamad
Set of keywords derived from query & compared with keywords in annotations of movies

Queries based on Features

content-based queries features derived (semi) automatically from content of MM object Low & high level features used Eg. Find all photos with color distribution like this photo Eg. Give me all football videos which a goal is scored within last ten minutes
goal is high-level feature that must be known to MIRS

Possible Querying Scenarios (cont.)

Query by example
Give example MM object MIRS extract all kinds of features from the MM object Resulting query based on these features

Similarity
Degree to which query & MM object of MIRS are similar Similarity calculated by MIRS based on metadata of MM object & query Try to estimate value of relevance of MM object to user Output is list of MM objects in descending order of similarity value

General Retrieval Model

Relevance Feedback

Helps when user doesnt know exactly what he is looking for, causing problem in query formulation Interactive approach
User issue starting query, MIRS compose result set, user judge output (relevant/not), MIRS uses feedback to improve retrieval process

Component of MIRS - Browsing

User sometimes cannot precisely specify what they want, but can recognize what they want when they see it Browsing let user scans through objects
Exploits hyperlinks which lead user from one object to other When object shown, user judge its relevance & proceed accordingly If objects are huge, icons are used

Starting point
query that describe info need or system provide starting point User can ask for another starting point if not satisfied Can classify object based on topics & subtopics

Component of MIRS Output Presentation (Play)

When MIRS returns list of objects, system has to decide whether user has right to see them User interface should be able to show all kinds of MM data What if objects are huge and result set large?
Give user perception of content of object Extract & present essential info for user to browse & select objects
Text: title, summary, places where keywords occur Audio: tune, start of song Images: summary of images thumbnails Video: cut into scene n choose for each scene a prime image

Component of MIRS Output Presentation (cont.)

Streaming
Content sent to client at specific rate and except for buffering, played directly Audio & video is delivered as continuous stream of packets When resource become scarce
Use switched Ethernet instead of shared Ethernet Use disk stripping Skip frames during play-back Fragment content over several content servers (need logical component between client & servers to direct client request to corresponding server)

Quality of MIRS
Recall
r/R
r: # of relevant objects returned by system, n: # objects retrieved, R: # relevant objects in collection

Precision
r/n

Relevance judged by humans, refer to TREC (Text Retrieval Conference)

Exercise
Discuss the role of DBMS in storing MM objects Discuss the role of Information Retrieval systems in storing MM objects

End of Module 1