Multimedia Information Retrieval

Prof Stefan Rüger

Multimedia and Information Systems
Knowledge Media Institute
The Open University
http://kmi.open.ac.uk/mmis
kmi.open.ac.uk
kmi.open.ac.uk
 kmi.open.ac.uk

Since 1995: 117 projects & 67 technologies

Current year
17 live projects , typically per year
£2.5m (¥300m) ext, £1m (¥120m) internal
• 10 EU
• 3 UK
• 1 US
• 3 internal (iTunes U, SocialLearn)
 Multimedia Information
Retrieval

1. What are multimedia queries?

2. Fingerprinting
3. Metadata & piggy-back retrieval
4. Automated image annotation
5 Visual content-based retrieval I
6 Visual content-based retrieval II
7. Evaluation
8. Browsing, search and geography
 Multimedia Information
Retrieval

1. What are multimedia queries?

- What is multimedia?
- Query by image
- Current best practice for image search
- Snaptell/Google goggles
- Shazam
- Discussion: Challenges and difficulties
2. Fingerprinting
3. Metadata & piggy-back retrieval
4. Automated image annotation
5 Visual content-based retrieval I
6 Visual content-based retrieval II
7. Evaluation
8. Browsing, search and geography
 What is Multimedia?

Within this lecture:

One or more media
Possibly interlinked
Digital
For communication
(not only entertainment)‫‏‬
 Multimedia queries

Sensō-ji ( Kinryū-zan Sensō-ji?) is an ancient Buddhist temple

located in Asakusa, Taitō, Tokyo, Japan. It is Tokyo's oldest temple, and one
of its most significant. Formerly associated with the Tendai sect, it became
independent after World War II. Adjacent to the temple is a Shinto shrine,
the Asakusa Shrine [http://en.wikipedia.org/wiki/Sensō-ji]
 Web-based
image searching

“Tokyo temple”

Google Images
Bing Images
Flickr
Yahoo Images
Яндекс
 Web-based
image searching

Best current practice is a text search:

Find text in filename, anchor text, caption, ...

Text search works by creating a large index:

Google
Tokyo temple
Bing
Tokyo temple
Flickr
Tokyo temple
Yahoo
Tokyo temple
Yandex
Tokyo temple
 New search types

query

humming
Example
location
speech

motion
image

sound doc
text

text conventional
video textroar
you retrieval
and
images get a wildlife
speech documentary
type “floods”
music and
humget BBC
a tune
radio
and getnews
a
sketches
music piece
multimedia
 Exercise

Organise yourself in groups

Discuss with neighbours
- Two Examples for different query/doc modes?
- How hard is this? Which techniques are involved?
- One example combining different modes
 Exercise

query Discuss

humming
location

- 2 examples
speech

motion
image

sound doc
text

- How hard is it?

text - 1 combination
video
images
speech
music
sketches
multimedia
Near-duplictate detection:
Cool access mode!
Snaptell: Book, CD and DVD
covers
Snaptell: Book, CD and DVD
covers
Snaptell: Book, CD and DVD
covers
Snaptell: Book, CD and DVD
covers
Snaptell: Book, CD and DVD
covers
 Link from real world to
databases

doi: 10.2200/S00244ED1V01Y200912ICR010
 The Open Univerity's
Spot & Search

Scott Forrest: E=MC squared

"Between finished surface texture
and raw quarried stone. Between
hard materials and soft concepts.
Between text and context."
More information

[with Suzanne Little]

 Spot & Search

[with Suzanne Little]

 Near duplicate detection

Works well in 2d: CD covers, wine labels, signs, ...

Less so in near 2d: buildings, vases, …
Not so well in 3d: faces, complex objects, ...
Shazam

Rueger, Multimedia IR, 2010

explains it all! Buy it now 
 Near duplicate detection
Exercise

Find applications for near-duplicate detection

- be imaginative: the more “outragous” the better
- can be other media types (audio, smells, haptic, ...)
- can be hard to do
 Near-duplicate detection
Where are the challenges?

[Victoria and Albert museum, London, ceramics collection, 2010]

 Leaf detection
What are the challenges?

[with Natural History Museum, London, and Goldsmiths]