Professional Documents
Culture Documents
Features Election
Features Election
Section 4.2
Genetic Algorithms
Chapter 7 (Duda et al.) – Section 7.5
dimensionality
reduction
• Feature Selection
– When classifying novel patterns, only a small number of features
need to be computed (i.e., faster classification).
– New features is just a subset of the original features.
Feature Selection Steps
• Feature selection is an
optimization problem.
– Step 1: Search the space of
possible feature subsets.
Search strategies
– Exhaustive
– Heuristic
– Randomized
Evaluation strategies
- Filter methods
- Wrapper methods
Search Strategies
• Assuming n features, an exhaustive search would
require:
n
– Examining all possible subsets of size d
d
6
Evaluation Strategies
• Filter Methods
– Evaluation is independent of
the classification algorithm.
• Wrapper Methods
– Evaluation uses criteria
related to the
classification algorithm.
• Disadvantage
– Correlation among features is not considered.
– The best pair of features may not even contain the best
individual feature.
Sequential forward selection (SFS)
(heuristic search)
• First, the best single feature is selected (i.e.,
using some objective function).
• Then, pairs of features are formed using one of
the remaining features and this best feature, and
the best pair is selected.
• Next, triplets of features are formed using one
of the remaining features and these two best
features, and the best triplet is selected.
• This procedure continues until a predefined
number of features are selected.
SFS performs
best when the
optimal subset is
small.
12
Example
features added at
each iteration
Population of
• What are GAs? encoded solutions
– A global optimization technique for searching
very large spaces.
– Inspired by the biological mechanisms of
natural selection and reproduction.
10010110…
01100011…
• Main characteristics of GAs
– Search probabilistically using a population of 01100011…
possible solutions.
10100100…
– Each solution is encoded as a string of
symbols.
– Use an objective (or fitness) function to
evaluate the “goodness” of each solution.
– Do not require derivatives.
Genetic Algorithms (GAs) (cont’d)
• In contrast to traditional search optimization
methods (e.g., gradient descent), GAs work with a
population of solutions.
– More effective to escape local minima
Encoding
• Each solution in the search space is represented as a
finite length string (chromosome) over some finite
set of symbols.
e.g., using binary encoding
Fitness f (decode(chromosome))
Searching
Population of Population of
encoded solutions encoded solutions
10010110… 10010110…
01100010… GA operators: 01100010…
10100100... 10100100…
10010010… Selection Crossover Mutation 01111001…
01111101… 10011101…
Fitness
1001 0.1 1001
1101 0.9 1101
1000 0.01 1101
0001 0.01 1101
Crossover
Explore new solutions:
10011110 10010010
10110010 10111110
Mutation
Explore new solutions:
10011110 10011010
mutated bit
Steps
Feature Selection using GAs
(randomized search)
Feature
Selection
(GA)
Feature Selection Using GAs
(cont’d)
• Binary encoding: 1 means “choose feature” and 0
means “do not choose” feature.
1 N
Z. Sun, G. Bebis, X. Yuan, and S. Louis, "Genetic Feature Subset Selection for Gender
Classification: A Comparison Study", IEEE Workshop on Applications of Computer Vision,
pp. 165-170, Orlando, December 2002.
Feature Extraction Using PCA
• Use PCA to represent faces in terms of the “best”
eigenvectors:
Which eigenvectors encode
mostly gender information?
0.25 22.4%
ERM
0.2 17.7%
ERG
14.2%
0.15 11.3% 13.3%
9% 8.9%
0.1 6.7%
4.7%
0.05
0
NN Bayes LDA SVMs SBFS+SVM
5
4
3
2
1
0
10 30 50 70 90 110 130 150 170 190 210 230 250
(a) LDA (b) Bayes
Original images
Top 30 EVs
EVs selected
by SVM+GA
EVs selected
by SVM+SFBS
Case Study 2: Vehicle Detection
Z. Sun, G. Bebis, and R. Miller, "Object Detection Using Feature
low light
Subset Selection", camera
Pattern Recognition, vol. 37, pp. 2165-2176,
2004.
rear views
6.49%
SFBS-SVM GA-SVM
Number of eigenvectors Number of eigenvectors
selected by SBFS: 87 selected by GA: 46
(43.5% information) (23% information)
Vehicle Detection
Original
Top 50 EVs
EVs selected
by SFBS
EVs selected
by GAs
• Thermal IR spectrum
– Low resolution, face heat patterns, aging, and the presence
of eyeglasses.
– Robust to illumination changes and facial expressions.
glass is
opaque to
thermal IR
How should we fuse information from
visible and thermal IR?
Feature Reconstruct
Fusion Using
Extraction Image
Genetic Algorithms
Fused
Image
Frontal
Illumination
Lateral
illumination
EFnG EFG
DATA
ELnG ELG
EnG EG
No Glasses Glasses
Experiments
glasses no glasses
test
EG
glasses ELG
EFG
EnG
no ELnG
glasses
EFnG
Eyeglasses Tests
0
10
20
30
40
50
60
70
80
90
ELG-ELnG
EFG-EFnG
ELnG-ELG
EFnG-EFG
eyeglasses
Presence of
ELG-EFG
EFG-ELG
ELnG-EFnG
EFnG-ELnG
Illumination direction
ELG-EFnG
EFG-ELnG
ELnG-EFG
EFnG-ELG
Eyeglasses and
Results
illumination direction
EG-EnG
EnG-EG
ELnG-EG
EFnG-EG
EG-ELnG
EG-EFnG
ELG-EnG
EFG-EnG
EnG-ELG
Eyeglasses and mixed illumnation
EnG-EFG
fused
visible
infrared
Overall Accuracy - Eyeglasses
Test 2: Facial Expression
Frontal Frontal
Illumination Illumination
EF VF
EA VA
Smile, Frown Lateral Speaking
& Surprise illumination Vowels
EL VL
Results
100
90 infrared
80
70 visible
60 fused
50
40
30
20
10
0
VL-VF
EL-VL
VL-EL
VF-VL
EL-VF
VL-EF
EF-VL
VF-EL
EF-VF
EL-EF
VF-EF
EA-VA
VA-EA
EF-EL