Data Mining:

Principles and Algorithms

Chapter 10.1
Mining Object, Spatial, and Multimedia Data

Jiawei Han Department of Computer Science University of Illinois at Urbana-Champaign www.cs.uiuc.edu/~hanj

09/11/2012 Data Mining: Principles and Algorithms 1

09/11/2012

Data Mining: Principles and Algorithms

Mining Object, Spatial and Multi-Media Data

Mining object data sets Mining spatial databases and data warehouses Spatial DBMS Spatial Data Warehousing Spatial Data Mining Spatiotemporal Data Mining Mining multimedia data Summary

09/11/2012

Data Mining: Principles and Algorithms

Mining Complex Data Objects: Generalization of Structured Data

Set-valued attribute Generalization of each value in the set into its corresponding higher-level concepts Derivation of the general behavior of the set, such as the number of elements in the set, the types or value ranges in the set, or the weighted average for numerical data E.g., hobby = {tennis, hockey, chess, violin, PC_games} generalizes to {sports, music, e_games} List-valued or a sequence-valued attribute Same as set-valued attributes except that the order of the elements in the sequence should be observed in the generalization
09/11/2012 Data Mining: Principles and Algorithms 4

Generalizing Spatial and Multimedia Data

Spatial data: Generalize detailed geographic points into clustered regions, such as business, residential, industrial, or agricultural areas, according to land usage Require the merge of a set of geographic areas by spatial operations Image data: Extracted by aggregation and/or approximation Size, color, shape, texture, orientation, and relative positions and structures of the contained objects or regions in the image Music data: Summarize its melody: based on the approximate patterns that repeatedly occur in the segment Summarized its style: based on its tone, tempo, or the major musical instruments played
09/11/2012 Data Mining: Principles and Algorithms 5

Generalizing Object Data

Object identifier generalize to the lowest level of class in the class/subclass hierarchies Class composition hierarchies generalize only those closely related in semantics to the current one Construction and mining of object cubes Extend the attribute-oriented induction method Apply a sequence of class-based generalization operators on different attributes Continue until getting a small number of generalized objects that can be summarized as a concise in high-level terms Implementation Examine each attribute, generalize it to simple-valued data Construct a multidimensional data cube (object cube) Problem: it is not always desirable to generalize a set of values to single-valued data
09/11/2012 Data Mining: Principles and Algorithms 6

Ex.: Plan Mining by Divide and Conquer

Plan: a sequence of actions E.g., Travel (flight): <traveler, departure, arrival, d-time, a-time, airline, price, seat> Plan mining: extraction of important or significant generalized (sequential) patterns from a planbase (a large collection of plans) E.g., Discover travel patterns in an air flight database, or find significant patterns from the sequences of actions in the repair of automobiles Method Attribute-oriented induction on sequence data A generalized travel plan: <small-big*-small> Divide & conquer:Mine characteristics for each subsequence E.g., big*: same airline, small-big: nearby region
09/11/2012 Data Mining: Principles and Algorithms 7

A Travel Database for Plan Mining

Example: Mining a travel planbase
Travel plan table
plan# 1 1 1 1 2 . . . action# 1 2 3 4 1 . . . departure ALB JFK ORD LAX SPI . . . depart_time 800 1000 1300 1710 900 . . . arrival JFK ORD LAX SAN ORD . . . arrival_time 900 1230 1600 1800 950 . . . airline TWA UA UA DAL AA . . . . . .

Airport info table

airport_code 1 1 1 1 2 . . .
09/11/2012

city 1 2 3 4 1 . . .

state ALB JFK ORD LAX SPI . . .

region

airport_size 800 1000 1300 1710 900 . . .

. . .
8

Data Mining: Principles and Algorithms

Multidimensional Analysis
Strategy Generalize the planbase in different directions Look for sequential patterns in the generalized plans Derive high-level plans A multi-D model for the planbase

09/11/2012

Data Mining: Principles and Algorithms

Multidimensional Generalization
Multi-Dimensional generalization of the planbase
Plan# 1 2
. . .

Loc_Seq ALB - JFK - ORD - LAX - SAN SPI - ORD - JFK - SYR
. . .

Size_Seq S-L-L-L-S S-L-L-S

State_Seq N-N-I-C-C I-I-N-N

. . .

Merging consecutive, identical actions in plans

Plan# 1 2 . . . Size_Seq S - L+ - S S - L+ - S State_Seq N+ - I - C+ I+ - N+ . . . Region_Seq E+ - M - P+ M+ - E+ . . .

flight ( x, y, ) airport _ size( x, S ) airport _ size( y, L) region( x) = region( y ) [75%]

09/11/2012 Data Mining: Principles and Algorithms 10

Generalization-Based Sequence Mining

Generalize planbase in multidimensional way using dimension tables Use # of distinct values (cardinality) at each level to determine the right level of generalization (levelplanning) Use operators merge +, option [] to further generalize patterns Retain patterns with significant support

09/11/2012

Data Mining: Principles and Algorithms

11

Generalized Sequence Patterns

AirportSize-sequence survives the min threshold (after applying merge operator): S-L+-S [35%], L+-S [30%], S-L+ [24.5%], L+ [9%] After applying option operator: [S]-L+-[S] [98.5%] Most of the time, people fly via large airports to get to final destination Other plans: 1.5% of chances, there are other patterns: S-S, L-S-L

09/11/2012

Data Mining: Principles and Algorithms

12

Mining Object, Spatial and Multi-Media Data

Mining object data sets Mining spatial databases and data warehouses Spatial DBMS Spatial Data Warehousing Spatial Data Mining Spatiotemporal Data Mining Mining multimedia data Summary

09/11/2012

Data Mining: Principles and Algorithms

13

What Is a Spatial Database System?

Geometric, geographic or spatial data: space-related data Example: Geographic space (2-D abstraction of earth surface), VLSI design, model of human brain, 3-D space representing the arrangement of chains of protein molecule. Spatial database system vs. image database systems. Image database system: handling digital raster image (e.g., satellite sensing, computer tomography), may also contain techniques for object analysis and extraction from images and some spatial database functionality. Spatial (geometric, geographic) database system: handling objects in space that have identity and well-defined extents, locations, and relationships.
09/11/2012 Data Mining: Principles and Algorithms 14

GIS (Geographic Information System)

GIS (Geographic Information System)
Analysis and visualization of geographic data

Common analysis functions of GIS

Search (thematic search, search by region) Location analysis (buffer, corridor, overlay) Terrain analysis (slope/aspect, drainage network) Flow analysis (connectivity, shortest path) Distribution (nearest neighbor, proximity, change detection) Spatial analysis/statistics (pattern, centrality, similarity, topology) Measurements (distance, perimeter, shape, adjacency, direction)
09/11/2012 Data Mining: Principles and Algorithms 15

Spatial DBMS (SDBMS)

SDBMS is a software system that supports spatial data models, spatial ADTs, and a query language supporting them supports spatial indexing, spatial operations efficiently, and query optimization can work with an underlying DBMS Examples Oracle Spatial Data Catridge ESRI Spatial Data Engine
09/11/2012 Data Mining: Principles and Algorithms 16

Modeling Spatial Objects

What needs to be represented? Two important alternative views Single objects: distinct entities arranged in space each of which has its own geometric description modeling cities, forests, rivers Spatially related collection of objects: describe space itself (about every point in space) modeling land use, partition of a country into districts
09/11/2012 Data Mining: Principles and Algorithms 17

Modeling Single Objects: Point, Line and Region

Point: location only but not extent Line (or a curve usually represented by a polyline, a sequence of line segment): moving through space, or connections in space (roads, rivers, cables, etc.) Region: Something having extent in 2D-space (country, lake, park). It may have a hole or consist of several disjoint pieces.
09/11/2012 Data Mining: Principles and Algorithms 18

Modeling Spatially Related Collection of Objects

Modeling spatially related collection of objects: plane partitions and networks. A partition: a set of region objects that are required to be disjoint (e.g., a thematic map). There exist often pairs of objects with a common boundary (adjacency relationship). A network: a graph embedded into the plane, consisting of a set of point objects, forming its nodes, and a set of line objects describing the geometry of the edges, e.g., highways. rivers, power supply lines. Other interested spatially related collection of objects: nested partitions, or a digital terrain (elevation) model.
09/11/2012 Data Mining: Principles and Algorithms 19

Spatial Data Types and Models

(0,4) y Field-based model: raster data (0,2) framework: partitioning of space Object-based model: vector (0,0) model point, line, polygon, Objects, Attributes Object Viewpoint of Forest Stands Area-ID FS1 FS2 FS3 Dominant Tree Species Pine Fir Oak Pine

Fir

Oak

(2,0)

(4,0) x

(a) Field Viewpoint of Forest Stands "Pine," 2 x x x 4;2 2; 0 4; 0 y y y 4 2 2

Area/Boundary [(0,2),(4,2),(4,4),(0,4)] [(0,0),(2,0),(2,2),(0,2)] [(2,0),(4,0),(4,2),(2,2)] (b) (c) f(x,y) "Fir," 0 "Oak," 2

09/11/2012

Data Mining: Principles and Algorithms

20

Spatial Query Language

Spatial query language Spatial data types, e.g. point, line segment, polygon, Spatial operations, e.g. overlap, distance, nearest neighbor, Callable from a query language (e.g. SQL3) of underlying DBMS SELECT S.name FROM Senator S WHERE S.district.Area() > 300 Standards SQL3 (a.k.a. SQL 1999) is a standard for query languages OGIS is a standard for spatial data types and operators Both standards enjoy wide support in industry
09/11/2012 Data Mining: Principles and Algorithms 21

Spatial Data Types by OGIS

09/11/2012

Data Mining: Principles and Algorithms

22

Query Processing
Efficient algorithms to answer spatial queries Common Strategy: filter and refine Filter: Query Region overlaps with MBRs (minimum bounding rectangles) of B, C, D Refine: Query Region overlaps with B, C

MBR A Query Region B FILTER B

C D

C D

Data Object

REFINE

09/11/2012

Data Mining: Principles and Algorithms

23

Join Query Processing

Determining Intersection Rectangle Plane Sweep Algorithm Place sweep filter identifies 5 intersections for refinement step

sweep line (T.xu, T.yu)

y-axis

R2

y-axis

S3 S2 R4 S1 x-axis (a) R1

T (T.xl, T.yl)

R3

x-axis (b)

R4

S2

S1

R1 (c)

S3

R2

R3

09/11/2012

Data Mining: Principles and Algorithms

24

File Organization and Indices

SDBMS: Dataset is in the secondary storage, e.g. disk Space Filling Curves: An ordering on the locations in a multi-dimensional space Linearize a multi-dimensional space Helps search efficiently

09/11/2012

Data Mining: Principles and Algorithms

25

File Organization and Indices

Spatial Indexing B-tree works on spatial data with space filling curve R-tree: Heighted balanced extention of B+ tree
Objects are represented as MBR provides better performance
A e d B g f j h
09/11/2012 Data Mining: Principles and Algorithms 26

A B C C i d e f g h i j

Spatial Query Optimization

A spatial operation can be processed using different strategies Computation cost of each strategy depends on many parameters Query optimization is the process of ordering operations in a query and selecting efficient strategy for each operation based on the details of a given dataset
09/11/2012 Data Mining: Principles and Algorithms 27

Spatial Data Warehousing

Spatial data warehouse: Integrated, subject-oriented, time-variant, and nonvolatile spatial data repository Spatial data integration: a big issue Structure-specific formats (raster- vs. vector-based, OO vs. relational models, different storage and indexing, etc.) Vendor-specific formats (ESRI, MapInfo, Integraph, IDRISI, etc.) Geo-specific formats (geographic vs. equal area projection, etc.) Spatial data cube: multidimensional spatial database Both dimensions and measures may contain spatial components
09/11/2012 Data Mining: Principles and Algorithms 28

Dimensions and Measures in Spatial Data Warehouse

Dimensions
non-spatial e.g. 25-30 degrees generalizes tohot (both are strings) spatial-to-nonspatial e.g. Seattle generalizes to description Pacific Northwest (as a string) spatial-to-spatial e.g. Seattle generalizes to Pacific Northwest (as a spatial region)
09/11/2012

Measures
numerical (e.g. monthly revenue of a region) distributive (e.g. count, sum) algebraic (e.g. average) holistic (e.g. median, rank) spatial collection of spatial pointers (e.g. pointers to all regions with temperature of 25-30 degrees in July)
29

Data Mining: Principles and Algorithms

Spatial-to-Spatial Generalization
Generalize detailed geographic points into clustered regions, such as businesses, residential, industrial, or agricultural areas, according to land usage Requires the merging of a set of geographic areas by spatial operations
09/11/2012

Dissolve Merge Clip Intersect Union

30

Data Mining: Principles and Algorithms

Example: British Columbia Weather Pattern Analysis

Input
A map with about 3,000 weather probes scattered in B.C. Daily data for temperature, precipitation, wind velocity, etc. Data warehouse using star schema

Output
A map that reveals patterns: merged (similar) regions

Goals
Interactive analysis (drill-down, slice, dice, pivot, roll-up) Fast response time Minimizing storage space used

Challenge
A merged region may contain hundreds of primitive regions (polygons)
09/11/2012 Data Mining: Principles and Algorithms 31

Star Schema of the BC Weather Warehouse

Spatial data warehouse Dimensions region_name time temperature precipitation Measurements region_map area count
Dimension table
09/11/2012 Data Mining: Principles and Algorithms

Fact table
32

Dynamic Merging of Spatial Objects

Materializing (precomputing) all?too much storage space On-line merge?slow, expensive Precompute rough approximations? accuracy trade off A better way: object-based, selective (partial) materialization

09/11/2012

Data Mining: Principles and Algorithms

33

Methods for Computing Spatial Data Cubes

On-line aggregation: collect and store pointers to spatial objects in a spatial data cube expensive and slow, need efficient aggregation techniques Precompute and store all the possible combinations huge space overhead Precompute and store rough approximations in a spatial data cube accuracy trade-off Selective computation: only materialize those which will be accessed frequently a reasonable choice
09/11/2012 Data Mining: Principles and Algorithms 34

Spatial Association Analysis

Spatial association rule: A B [s%, c%] A and B are sets of spatial or non-spatial predicates
Topological relations: intersects, overlaps, disjoint, etc. Spatial orientations: left_of, west_of, under, etc. Distance information: close_to, within_distance, etc.

s% is the support and c% is the confidence of the rule

Examples
1) is_a(x, large_town) ^ intersect(x, highway) adjacent_to(x, water)

[7%, 85%]
2) What kinds of objects are typically located close to golf courses?

09/11/2012

Data Mining: Principles and Algorithms

35

Progressive Refinement Mining of Spatial Association Rules

Hierarchy of spatial relationship:
g_close_to: near_by, touch, intersect, contain, etc.
First search for rough relationship and then refine it

Two-step mining of spatial association:

Step 1: Rough spatial computation (as a filter) Using MBR or R-tree for rough estimation Step2: Detailed spatial algorithm (as refinement) Apply only to those objects which have passed the rough spatial association test (no less than min_support)

09/11/2012

Data Mining: Principles and Algorithms

36

Mining Spatial Co-location Rules

Co-location rule is similar to association rule but explore more relying spatial auto-correlation It leads to efficient processing It can be integrated with progressive refinement to further improve its performance Spatial co-location mining idea can be applied to clustering, classification, outlier analysis and other potential mining tasks

09/11/2012

Data Mining: Principles and Algorithms

37

Spatial Autocorrelation
Spatial data tends to be highly self-correlated Example: Neighborhood, Temperature Items in a traditional data are independent of each other, whereas properties of locations in a map are often auto-correlated. First law of geography: Everything is related to everything, but nearby things are

more related than distant things.

09/11/2012

Data Mining: Principles and Algorithms

38

Spatial Autocorrelation (contd)

09/11/2012

Data Mining: Principles and Algorithms

39

Spatial Classification
Methods in classification Decision-tree classification, Nave-Bayesian classifier + boosting, neural network, logistic regression, etc. Association-based multi-dimensional classification Example: classifying house value based on proximity to lakes, highways, mountains, etc. Assuming learning samples are independent of each other Spatial auto-correlation violates this assumption! Popular spatial classification methods Spatial auto-regression (SAR) Markov random field (MRF)
09/11/2012 Data Mining: Principles and Algorithms 40

Spatial Auto-Regression
Linear Regression Y=X + Spatial autoregressive regression (SAR) Y = WY + X + W: neighborhood matrix. models strength of spatial dependencies error vector The estimates of and can be derived using maximum likelihood theory or Bayesian statistics
09/11/2012 Data Mining: Principles and Algorithms 41

Markov Random Field Based Bayesian Classifiers

Bayesian classifiers MRF A set of random variables whose interdependency relationship is represented by an undirected graph (i.e., a symmetric neighborhood matrix) is called a Markov Random Field.

Pr(X | Ci, Li) Pr(Ci | Li) Pr(Ci | X, Li) = Pr (X)

Li denotes set of labels in the neighborhood of si excluding labels at si Pr(Ci | Li) can be estimated from training data by examine the
ratios of the frequencies of class labels to the total number of locations Pr(X|Ci, Li) can be estimated using kernel functions from the observed values in the training dataset
09/11/2012 Data Mining: Principles and Algorithms

42

SAR v.s. MRF

Pr(X | Ci, Li) , Pr(Ci | Li)

Pr(Ci | X, Li)
Pr(X | Ci, Li) , Pr(Ci | Li)

09/11/2012

Data Mining: Principles and Algorithms

43

Spatial Trend Analysis

Function Detect changes and trends along a spatial dimension Study the trend of non-spatial or spatial data changing with space Application examples Observe the trend of changes of the climate or vegetation with increasing distance from an ocean Crime rate or unemployment rate change with regard to city geo-distribution
09/11/2012 Data Mining: Principles and Algorithms 44

Spatial Cluster Analysis

Mining clustersk-means, k-medoids, hierarchical, density-based, etc. Analysis of distinct features of the clusters

09/11/2012

Data Mining: Principles and Algorithms

45

Constraints-Based Clustering
Constraints on individual objects
Simple selection of relevant objects before clustering

Clustering parameters as constraints

K-means, density-based: radius, min-# of points

Constraints specified on clusters using SQL aggregates

Sum of the profits in each cluster > $1 million

Constraints imposed by physical obstacles

Clustering with obstructed distance
09/11/2012 Data Mining: Principles and Algorithms 46

Constrained Clustering: Planning ATM Locations

C2 C1 River

C3

Mountain Spatial data with obstacles

09/11/2012

C4 Clustering without taking obstacles into consideration

47

Data Mining: Principles and Algorithms

Spatial Outlier Detection

Outlier Global outliers: Observations which is inconsistent with the rest of the data Spatial outliers: A local instability of non-spatial attributes Spatial outlier detection Graphical tests Variogram clouds Moran scatterplots Quantitative tests Scatterplots Spatial Statistic Z(S(x)) Quantitative tests are more accurate than Graphical tests
09/11/2012 Data Mining: Principles and Algorithms 48

Spatial Outlier Detection Variogram Clouds

Graphical method For each pair of locations, the square-root of the absolute difference between attribute values at the locations versus the Euclidean distance between the locations are plotted Nearby locations with large attribute difference indicate a spatial outlier Quantitative method Compute spatial statistic Z(S(x))
09/11/2012 Data Mining: Principles and Algorithms 49

Spatial Outlier DetectionMoran Scatterplots

Graphical tests A plot of normalized attribute value Z against the neighborhood average of normalized attribute values (WZ)
Z [ f (i)] = f (i ) u f

The upper left and lower right quadrants indicate a spatial outlier Computation method Fit a linear regression line Select points (e.g. P, Q, S) which are from the regression line greater than specified residual error
09/11/2012 Data Mining: Principles and Algorithms 50

Mining Spatiotemporal Data

Spatiotemporal data Data has spatial extensions and changes with time Ex: Forest fire, moving objects, hurricane & earthquakes Automatic anomaly detection in massive moving objects Moving objects are ubiquitous: GPS, radar, etc. Ex: Maritime vessel surveillance Problem: Automatic anomaly detection
09/11/2012 Data Mining: Principles and Algorithms 51

Analysis: Mining Anomaly in Moving Objects

Raw analysis of collected data does not fully convey anomaly information More effective analysis relies on higher semantic features Examples: A speed boat moving quickly in open water A fishing boat moving slowly into the docks A yacht circling slowly around landmark during night hours
09/11/2012 Data Mining: Principles and Algorithms 52

Framework: Motif-Based Feature Analysis

Motif-based representation A motif is a prototypical movement pattern View a movement path as a sequence of motif expressions Motif-oriented feature space Automated motif feature extraction Semantic-level features Classification Anomaly detection via classification High dimensional classifier
09/11/2012 Data Mining: Principles and Algorithms 53

Movement Motifs
Prototypical movement of object

Right-turn, U-turn
Can be either defined by an expert or discovered automatically from data Defined in our framework Extracted in movement paths Path becomes a set of motif expressions

09/11/2012

Data Mining: Principles and Algorithms

54

Motif Expression Attributes

Each motif expression has attributes (e.g., speed, location, size) Attributes express how a motif was expressed Conveys semantic information useful for classification

a tight circle at 30mph near landmark Y. A tight circle at 10mph in location X

09/11/2012 Data Mining: Principles and Algorithms 55

Motif-Oriented Feature Space

Attributes describe how motifs are expressed Let there be A attributes, each path is a set of (A+1)-tuples {(mi, v1, v2, , vA), (mj, v1, v2, , vA)} Nave Feature space construction 1. Let each distinct (mj, v1, v2, , vA) be a feature 2. If path exhibits a particular motif-expression, its value is 1. Otherwise, its value is 0.

09/11/2012

Data Mining: Principles and Algorithms

56

Analyzing Nave Feature Space

Let there be M distinct motifs and V different possible values for each of the A attributes Size of feature space is M * VA V is usually very large due to high granularity of measurements E.g., seconds for time or meters for location Modest values for A and M could lead to extremely high dimensional feature space
09/11/2012 Data Mining: Principles and Algorithms 57

More on Nave Feature Space

High dimensional feature space could make effective learning hard More importantly, high granular features make generalization impossible!
(mj, v1, 10:01am, , vA) vs (mj, v1, 10:02am, , vA) Learning on one feature has no effect on another feature

Intuition: should have features that describe general high-level concepts

Early Morning instead of 2:03am, 2:04am, Near Location X instead of 50m west of Location X

Solution: Clustering on nave feature space

09/11/2012 Data Mining: Principles and Algorithms 58

Motif Feature Extraction

For each motif attribute, cluster values to form higher level concepts Frequency and distribution in learning data dictates the final clusters Hierarchical micro-clustering Small clusters so concepts are not merged unnecessarily Hierarchy allows flexibility in describing objects For example: afternoon vs. early afternoon and late afternoon
09/11/2012 Data Mining: Principles and Algorithms 59

Feature Clustering
Rough, fast micro-clustering method based on BIRCH (SIGMOD96) A micro-cluster is represented by a CF Vector: CF = (n, LS, SS) Centroid and radius can be calculated from CF vector CF Additive Theorem allows two CF Vectors to be combined quickly and losslessly CF Tree is a hierarchy of CF Vectors A parent CF Vector holds information for all descendent CF Vectors Leaf CF Vector corresponds to a set of actual points
09/11/2012 Data Mining: Principles and Algorithms 60

More on Feature Clustering

Build CF Tree from raw data, much like B-tree Two parameters in clustering B: branching factor of CF Tree T: radius threshold of CF Vector Parameters control how fine micro-clusters are constructed Hierarchical agglomerative clustering on leaves of CF Tree Entire process is efficient: O(N)

09/11/2012

Data Mining: Principles and Algorithms

61

Extracted Feature Space

Leaf nodes in final clustering become the new features More general than the original nave feature space Dimensionality could still be moderately high Use Support Vector Machine for classification

09/11/2012

Data Mining: Principles and Algorithms

62

Experiments
Synthetic Data Generated at motif-expression level Abnormal paths are injected with abnormal motif-expressions Classifiers SVM using nave feature space SVM using extracted feature spaces of varying refinement levels

09/11/2012

Data Mining: Principles and Algorithms

63

Experiment

09/11/2012

Data Mining: Principles and Algorithms

64

Experiment (2)

09/11/2012

Data Mining: Principles and Algorithms

65

Summary: Moving Object Anomaly Detection

Higher level semantic analysis of moving objects yields better results Automated feature extraction Future work Automatic determination of t parameter Better use of feature space hierarchy Other analysis, such as clustering and local outlier detection for anomaly detection Mining other knowledge for moving objects
09/11/2012 Data Mining: Principles and Algorithms 66

Mining Object, Spatial and Multi-Media Data

Mining object data sets Mining spatial databases and data warehouses Spatial DBMS Spatial Data Warehousing Spatial Data Mining Spatiotemporal Data Mining Mining multimedia data Summary

09/11/2012

Data Mining: Principles and Algorithms

67

Similarity Search in Multimedia Data

Description-based retrieval systems Build indices and perform object retrieval based on image descriptions, such as keywords, captions, size, and time of creation Labor-intensive if performed manually Results are typically of poor quality if automated Content-based retrieval systems Support retrieval based on the image content, such as color histogram, texture, shape, objects, and wavelet transforms
09/11/2012 Data Mining: Principles and Algorithms 68

Queries in Content-Based Retrieval Systems

Image sample-based queries Find all of the images that are similar to the given image sample Compare the feature vector (signature) extracted from the sample with the feature vectors of images that have already been extracted and indexed in the image database Image feature specification queries Specify or sketch image features like color, texture, or shape, which are translated into a feature vector Match the feature vector with the feature vectors of the images in the database
09/11/2012 Data Mining: Principles and Algorithms 69

Approaches Based on Image Signature

Color histogram-based signature The signature includes color histograms based on color composition of an image regardless of its scale or orientation No information about shape, location, or texture Two images with similar color composition may contain very different shapes or textures, and thus could be completely unrelated in semantics Multifeature composed signature Define different distance functions for color, shape, location, and texture, and subsequently combine them to derive the overall result
09/11/2012 Data Mining: Principles and Algorithms 70

Wavelet Analysis
Wavelet-based signature Use the dominant wavelet coefficients of an image as its signature Wavelets capture shape, texture, and location information in a single unified framework Improved efficiency and reduced the need for providing multiple search primitives May fail to identify images containing similar objects that are in different locations.
09/11/2012 Data Mining: Principles and Algorithms 71

One Signature for the Entire Image?

Walnus: [NRS99] by Natsev, Rastogi, and Shim Similar images may contain similar regions, but a region in one image could be a translation or scaling of a matching region in the other

Wavelet-based signature with region-based granularity Define regions by clustering signatures of windows of varying sizes within the image Signature of a region is the centroid of the cluster Similarity is defined in terms of the fraction of the area of the two images covered by matching pairs of regions from two images
09/11/2012 Data Mining: Principles and Algorithms 72

Multidimensional Analysis of Multimedia Data

Multimedia data cube Design and construction similar to that of traditional data cubes from relational data Contain additional dimensions and measures for multimedia information, such as color, texture, and shape The database does not store images but their descriptors Feature descriptor: a set of vectors for each visual characteristic
Color vector: contains the color histogram MFC (Most Frequent Color) vector: five color centroids MFO (Most Frequent Orientation) vector: five edge orientation centroids

Layout descriptor: contains a color layout vector and an edge layout vector
09/11/2012 Data Mining: Principles and Algorithms 73

Multi-Dimensional Search in Multimedia Databases

09/11/2012

Data Mining: Principles and Algorithms

74

Multi-Dimensional Analysis in Multimedia Databases

Color histogram Texture layout

09/11/2012

Data Mining: Principles and Algorithms

75

Mining Multimedia Databases

Refining or combining searches Search for airplane in blue sky
(top layout grid is blue and keyword = airplane)

Search for blue sky and green meadows Search for blue sky
(top layout grid is blue) (top layout grid is blue and bottom is green)

09/11/2012

Data Mining: Principles and Algorithms

76

Mining Multimedia Databases

The Data Cube and the Sub-Space Measurements

By Size By Format By Format & Size

RED WHITE BLUE

Cross Tab
JPEG GIF RED WHITE BLUE

By Colour

By Colour & Size Sum

By Format & Colour By Colour

Group By
Colour
RED WHITE BLUE

By Format Sum

Measurement

Sum

Format of image Duration Colors Textures Keywords Size Width Height Internet domain of image Internet domain of parent pages Image popularity
77

09/11/2012

Data Mining: Principles and Algorithms

Mining Multimedia Databases in

09/11/2012

Data Mining: Principles and Algorithms

78

Classification in MultiMediaMiner

09/11/2012

Data Mining: Principles and Algorithms

79

Mining Associations in Multimedia Data

Special features: Need # of occurrences besides Boolean existence, e.g., Two red square and one blue circle implies theme air-show Need spatial relationships Blue on top of white squared object is associated with brown bottom Need multi-resolution and progressive refinement mining It is expensive to explore detailed associations among objects at high resolution It is crucial to ensure the completeness of search at multi-resolution space
09/11/2012 Data Mining: Principles and Algorithms 80

Mining Multimedia Databases

Spatial Relationships from Layout
property P1 on-top-of property P2 property P1 next-to property P2

Different Resolution Hierarchy

09/11/2012

Data Mining: Principles and Algorithms

81

Mining Multimedia Databases

From Coarse to Fine Resolution Mining

09/11/2012

Data Mining: Principles and Algorithms

82

Challenge: Curse of Dimensionality

Difficult to implement a data cube efficiently given a large number of dimensions, especially serious in the case of multimedia data cubes Many of these attributes are set-oriented instead of single-valued Restricting number of dimensions may lead to the modeling of an image at a rather rough, limited, and imprecise scale More research is needed to strike a balance between efficiency and power of representation
09/11/2012 Data Mining: Principles and Algorithms 83

Summary
Mining object data needs feature/attribute-based generalization methods Spatial, spatiotemporal and multimedia data mining is one of important research frontiers in data mining with broad applications Spatial data warehousing, OLAP and mining facilitates multidimensional spatial analysis and finding spatial associations, classifications and trends Multimedia data mining needs content-based retrieval and similarity search integrated with mining methods
09/11/2012 Data Mining: Principles and Algorithms 84

References on Spatial Data Mining

H. Miller and J. Han (eds.), Geographic Data Mining and Knowledge Discovery, Taylor and Francis, 2001. Ester M., Frommelt A., Kriegel H.-P., Sander J.: Spatial Data Mining: Database Primitives, Algorithms and Efficient DBMS Support, Data Mining and Knowledge Discovery, 4: 193-216, 2000. J. Han, M. Kamber, and A. K. H. Tung, "Spatial Clustering Methods in Data Mining: A Survey", in H. Miller and J. Han (eds.), Geographic Data Mining and Knowledge Discovery, Taylor and Francis, 2000. Y. Bedard, T. Merrett, and J. Han, "Fundamentals of Geospatial Data Warehousing for Geographic Knowledge Discovery", in H. Miller and J. Han (eds.), Geographic Data Mining and Knowledge Discovery, Taylor and Francis, 2000 K. Koperski and J. Han. Discovery of spatial association rules in geographic information databases. SSD'95. Shashi Shekhar and Sanjay Chawla, Spatial Databases: A Tour , Prentice Hall, 2003 (ISBN 013-017480-7). Chapter 7.: Introduction to Spatial Data Mining X. Li, J. Han, and S. Kim, Motion-Alert: Automatic Anomaly Detection in Massive Moving Objects, IEEE Int. Conf. on Intelligence and Security Informatics (ISI'06).
09/11/2012 Data Mining: Principles and Algorithms 85

09/11/2012

Data Mining: Principles and Algorithms

86