Feature Reduction and Selection

for Automatic Image Annotation

Guan-Bin Chen and Been-Chian Chien

Department of Computer Science and Information Engineering

National University of Tainan
Hualien, Taiwan
Dec. 12-14, 2012

2012/12/12 ICS 2012 1

 Outline
• Image Annotation
• The Problems of High-dimension Features
• The Proposed Methods
– Feature Reduction Method
– Feature Selection Method
• Experimental Results and Discussion
• Conclusion

2012/12/12 ICS 2012 2

 Introduction to Image Annotation
• Image Annotation
– Add textual tags to images
– Easy to search and manage large image data set.
• Annotating large number of images is a tedious job and
hard work for human.
– Annotate Image Automatically

 lake
 water
 tree
 reflection

2012/12/12 ICS 2012 3

 Annotation Approach
• We treat the image annotation problem as the image
classification problem.
• Each annotated tag is a labeled class.

Training
Images

 Visual feature
Feature Extraction
 Textual feature
Machine Learning

Testing Classifier Evaluation

Images
Annotating labels
2012/12/12 ICS 2012 4
 Problems
• High-dimension features
– Take a long time to learn classification models
– Require large memory space
 Feature reduction
• Select the effective features for annotating concepts
– E. g. The color based feature is effective to find the concept
“sea”.

 Feature selection
2012/12/12 ICS 2012 5
 Improved Annotation Approach
Training Validation
Our methods Images Images

 Visual feature
Features Extraction
 Textual feature

Visual Feature Reduction Textual Feature Reduction

Feature Selection

Machine Learning

Testing Classifier Evaluation

Images
Annotating labels
ICS 2012 6
2012/12/12
 Visual & Textual Features Reduction
• Visual Feature Reduction
– Feature attributes are numerical descriptors
– Statistics and concept similarity

• Textual Feature Reduction

– Feature attributes are term frequency
– Use discriminant coefficient method
Term book food like news subject
Appear times 0 4 1 2 1

2012/12/12 ICS 2012 7

 Visual Feature Reduction (1/3)

Feature attributes

A1 A2 A3 A4 A5 A6
I1 1.1 2.3 1.0 0.0 0.0 2.7
I2 2.2 3.1 0.0 0.0 0.0 0.0 C1
I3 1.0 1.0 0.3 3.0 1.0 1.0
Concepts
Images I4 0.0 1.5 0.0 3.7 0.0 1.0
C2 (Classes)
I5 1.0 0.0 2.0 4.2 1.0 0.0
I6 0.0 0.1 0.0 1.0 0.0 3.2
C3
I7 1.6 1.0 1.0 0.0 3.7 2.0

2012/12/12 ICS 2012 8

 Visual Feature Reduction (2/3)
A1 A2 A3 A4 A5 A6
I1 1.1 2.3 1.0 0.0 0.0 2.7
I2 2.2 3.1 0.0 0.0 0.0 0.0 C1
I3 1.0 1.0 0.3 3.0 1.0 1.0
I4 0.0 1.5 0.0 3.7 0.0 1.0
C2
I5 1.0 0.0 2.0 4.2 1.0 0.0
I6 0.0 0.1 0.0 1.0 0.0 3.2
C3
I7 1.6 1.0 1.0 0.0 3.7 2.0

m   x  
2


f li 
j 1
exp   
ij
   lj
lj 




 

Concept similarity
2012/12/12 ICS 2012 9
 Visual Feature Reduction (3/3)
A1 A2 A3 A4 A5 A6
I1 1.1 2.3 1.0 0.0 0.0 2.7
I2 2.2 3.1 0.0 0.0 0.0 0.0 C1
I3 1.0 1.0 0.3 3.0 1.0 1.0
I4 0.0 1.5 0.0 3.7 0.0 1.0
C2
I5 1.0 0.0 2.0 4.2 1.0 0.0
I6 0.0 0.1 0.0 1.0 0.0 3.2
C3
I7 1.6 1.0 1.0 0.0 3.7 2.0
reduction

6-dimension 3-dimension

2012/12/12 ICS 2012 10

 Textual Feature Reduction
• Original form : term frequency matrix
• Normalize textual features and transform into the
concept domain
w
I i Cl
ij max {wij }
1 j  m
u lj 
Cl

wij : the term frequency of the term j for the image Ii.
• Then, apply the discriminant coefficient method to reduce
the dimension of features
Y. X. Lin and B. C. Chien, Feature Reduction for High-Precision Text Classification, in
Proceedings of 2011 National Computer Symposium, Dec. 2-3, 2011, Chia-Yi, Taiwan,
pp. 36-45.
2012/12/12 ICS 2012 11
 Feature Selection

Training feature1 feature2 ... featurej

Images

RF Classifier RF Classifier ... RF Classifier

Validation F1 Evaluation F1 Evaluation ... F1 Evaluation

Images

 l  max{ lj }  2 l
j

keep the largest four lj features

Selected features

2012/12/12 ICS 2012 12

 Experiments (1/3)
• Data set: ImageCLEF 2012 Flickr photo annotation task
– 10,000 images for train (2,500 validation)
– 5,000 images for test
– 94 visual concepts (e.g. dog, strangers, happy)
http://www.imageclef.org/2012/photo-flickr

timeofday_day timeofday_day timeofday_day

weather_clearsky quantity_biggroup age_baby
fauna_bird age_adult quality_noblur
quantity_none relation_coworkers view_portrait
quality_motionblur sentiment_happy sentiment_happy
view_closeupmacro sentiment_active
sentiment_happy
2012/12/12 ICS 2012 13
 Experiments (2/3)
• Visual Features
– Extracted by LIRE JAVA library
– Each feature reduced to 94 dimensions (if original
dimension > 94)
Color Shape Texture Spatial Dimensions
Auto Color Correlogram   256
CEDD    144
Color Layout  120
Edge Histogram  80
FCTH    192
Gabor  60
Jpeg Coefficient Histogram   192

2012/12/12 ICS 2012 14

 Experiments (3/3)
• Textual Features
– Source: Flickr tags (more than 60 different languages)
– Tool: Google translation, WVTool
– 36,702 words in English
– Reduction: 36,702 dimensions are reduced to 94
dimensions
• Evaluation measure
– Precision
– Recall
– MicroF1

2012/12/12 ICS 2012 15

 Experimental Cases

Set up 3 Experiments Learning from

Single feature Original Reduced

features features

Combine all features No selection OL RL

Original
OS-OL OS-RL
Selection features
Combine all feature &
feature selection from Reduced
RS-OL RS-RL
features

2012/12/12 ICS 2012 16

 Experimental Results
Features MicroP MicroR MicroF1
AutoColorCorrelogram (ACC) 0.4963 0.3977 0.4416
CEDD 0.5225 0.4086 0.4586
ColorLayout (CL) 0.4805 0.4023 0.4379
Visual EdgeHistogram (EH) 0.4283 0.4649 0.4459
FCTH 0.4814 0.4271 0.4526
Gabor 0.3446 0.3518 0.3482
JpegCoefficientHistogram (JCH) 0.5027 0.4330 0.4652
Textual Tags_reduction 0.5010 0.4702 0.4851
Original features (OL) 0.5562 0.4714 0.5103
Reduced features (RL) 0.4990 0.4973 0.4982
OS-OL 0.5439 0.5048 0.5236
Mixed
OS-RL 0.4905 0.5241 0.5068
RS-OL 0.5577 0.4974 0.5258
RS-RL 0.4774 0.5212 0.4983

2012/12/12 ICS 2012 17

 Discussions
• Discussions
– Using mixed features are better than using a single feature
– Using feature selection are better than non feature selection
– RS-OL method has the best result

• The selection rates of image features

Tags FCTH CEDD JCH ACC CL EH Gabor
OS 0.42 0.10 0.12 0.10 0.07 0.06 0.08 0.05
RS 0.32 0.16 0.14 0.13 0.11 0.05 0.05 0.04

2012/12/12 ICS 2012 18

 Conclusion
• Our methods can improve the performance of image
annotation.
• The feature reduction method can speed up the
selection process.
• The RS-OL method is the best
– Applying reduced efficient and select effective.

2012/12/12 ICS 2012 19

 Thank you for your Attention!

2012/12/12 ICS 2012 20