Unit 2 :

Feature Selection
 Why Data Preprocessing?
Data in the real world is dirty
incomplete: lacking attribute values, lacking certain attributes of
interest, or containing only aggregate data
noisy: containing errors or outliers
inconsistent: containing discrepancies in codes or names
No quality data, no quality mining results!
Quality decisions must be based on quality data
Data warehouse needs consistent integration of quality data
A multi-dimensional measure of data quality:
A well-accepted multi-dimensional view:
accuracy, completeness, consistency, timeliness, believability, value
added, interpretability, accessibility
Broad categories:
intrinsic, contextual, representational, and accessibility.
 Major Tasks in Data Preprocessing
Data cleaning
Fill in missing values, smooth noisy data, identify or remove outliers, and resolve
inconsistencies
Data integration
Integration of multiple databases, data cubes, files, or notes
Data transformation
Normalization (scaling to a specific range)
Aggregation
Data reduction
Obtains reduced representation in volume but produces the same or similar
analytical results
Data discretization: with particular importance, especially for numerical data
Data aggregation, dimensionality reduction, data compression,generalization
Forms of data preprocessing
 Agenda

Why preprocess the data?

Data cleaning
Data integration and transformation
Data reduction
Discretization and concept hierarchy generation
Summary
 Data Cleaning

Data cleaning tasks

Fill in missing values
Identify outliers and smooth out noisy data
Correct inconsistent data
 Missing Data
Data is not always available
E.g., many tuples have no recorded value for several attributes, such as
customer income in sales data
Missing data may be due to
equipment malfunction
inconsistent with other recorded data and thus deleted
data not entered due to misunderstanding
certain data may not be considered important at the time of entry
not register history or changes of the data
Missing data may need to be inferred
 How to Handle Missing Data?
Ignore the tuple: usually done when class label is missing (assuming the task
is classification—not effective in certain cases)
Fill in the missing value manually: tedious + infeasible?
Use a global constant to fill in the missing value: e.g., “unknown”, a
new class?!
Use the attribute mean to fill in the missing value
Use the attribute mean for all samples of the same class to fill in
the missing value: smarter
Use the most probable value to fill in the missing value: inference-
based such as regression, Bayesian formula, decision tree
 Scikit-learn Dataset
• Scikit-learn provides some built-in datasets
that can be used for testing purposes.
• They are all available in the package
sklearn.datasets and have a common
structure:
o The data instance variable contains the whole
input set X.
o Target contains the labels for classification or
target values for regression.
 Scikit-learn Dataset

In this case, we have 506 samples with 13

features and a single target value.
 Creating training & test sets
When dataset is large enough, it is good practice to split it
into training and test sets; the former is used for training the
model and latter is used to test its performances
 Creating training & test sets
There are two main rules in performing such an
operation:
• Both datasets must reflect the originial
distribution
• The original dataset must be randomly shuffled
before the split phase in order to avoid a
correlation between consequent elements.
 Creating training & test sets
With scikit-learn, this can be achieved using the
train_test_split() function:

The parameter test_size (as well as training_size) allows specifying

the percentage of elements to put into the training /test dataset.
In this e.g, the ratio is 75 percent for training and 25 percent for
test phase
 Managing categorical data
• In many classification problems, the target
dataset is made up of categorical labels which
cannot immediately be processed by any
algorithm.
• An encoding is needed and scikit-learn offers
at least two valid options.
 Managing categorical data
• Approach 1: Integer Encoding

T-Shirt Size

S -> 1
Small (S)
M -> 2

Ordinal L -> 3
Medium (M)
Relationship

S<M<L
Large (L)
 Managing categorical data
• Approach 2: One Hot Encoding

Colors
R G B

Red (R) 1 0 0

Non -
Green (G) 0 1 0
Ordinal
Labels
0 0 1
Blue (B)

3 Unique classes
 Managing categorical data
Let's consider a very small dataset made of 10 categorical
' samples with two features each:

In this example, random.uniform (low, high, size) is used to

draw samples from uniform distribution. Where 0.0 indicates
lower boundary of the output interval & 1.0 indicates upper
boundary of the output interval.
 Managing categorical data
Approach 1: Integer Encoding
The first option is to use the LabelEncoder class, which
adopts a dictionary-oriented approach, associating to each
category label a progressive integer number, that is an
index of an instance array called classes_:
 Managing categorical data
Approach 2: One Hot Encoding
 Managing missing features
Sometimes a dataset can contain missing
features, so there are a few options that can be
taken into account:

1. Removing the whole line

2. Creating sub-model to predict those
features
3. Using an automatic strategy to input them
according to the other known values
 Managing missing features
1. Removing the whole line:

This option is the most drastic one and should

be considered only when the dataset is quite
large, the number of missing features is high,
and any prediction could be risky.
 Managing missing features
2. Creating sub-model to predict those features:

This option is much more difficult because it's

necessary to determine a supervised strategy to
train a model for each feature and, finally, to
predict their value.
 Managing missing features
3. Using an automatic strategy to input them
according to the other known values

Considering all pros and cons, the third option is

likely to be the best choice. scikit-learn offers the
class Imputer, which is responsible for filling the
holes. Missing values can be imputed/replaced with
a provided constant value, or using the statistics
(mean, median or most frequent) of each column in
which the missing values are located.
Managing missing features
1. Replacing Missing Value by Mean:
 Managing missing features
2. Replacing Missing Value by Median:

Median Formula:Odd Number of Observations: Median = {(n+1)/2}thterm

Even Number of Observations:Median=[(n/2)th term+{(n/2)+1th }]/2
 Managing missing features
3. Replacing Missing Value by Most Frequent Value:
 Data scaling and normalization
• A generic dataset (we assume here that it is
always numerical) is made up of different values
which can be drawn from different distributions,
having different scales and, sometimes, there are
also outliers.
• A machine learning algorithm isn't naturally able
to distinguish among these various situations,
and therefore, it's always preferable to
standardize datasets before processing them.
 Data scaling and normalization
A very common problem derives from having a non-zero
mean and a variance greater than one. In the following figure,
there's a comparison between a raw dataset and the same
dataset scaled and centered:

This result can be achieved using the StandardScaler class.

Data scaling and normalization
 Data scaling and normalization
If you need a more powerful scaling feature, with a superior
control on outliers and the possibility to select a quantile
range, there's also the class RobustScaler. Here are some
examples with different quantiles:
 Data scaling and normalization

Other options include MinMaxScaler and MaxAbsScaler, which scale data by removing
elements that don't belong to a given range (the former) or by considering a maximum
absolute value (the latter). Prepared by Prof. Priyanka Shahane
 Data scaling and normalization
scikit-learn also provides a class for per-sample
normalization, Normalizer. It can apply max, L1 and L2
norms to each element of a dataset. In a Euclidean space,
they are defined in the following way,
 Feature selection and filtering
An unnormalized dataset with many features contains information
proportional to the independence of all features and their variance.
Let's consider a small dataset with three features, generated with random
Gaussian distributions:

Even without further analysis, it's obvious that the central line (with
the lowest variance) is almost constant and doesn't provide any
useful information.
 Feature selection and filtering
A variance threshold is, therefore, a useful approach to remove all
those elements whose contribution (in terms of variability and so,
information) is under a predefined level. scikit-learn provides the class
Variance Threshold that can easily solve this problem. By applying it on
the previous dataset, we get the following result:

The third feature has been completely removed because its variance is under the selected
threshold (1.5 in this case).
 Feature selection and filtering
There are also many univariate methods that can be used in order to select the
best features according to specific criteria based on F-tests and p-values, such
as chi-square or ANOVA.
Two examples of feature selection that use the classes SelectKBest (which
selects the best K high-score features) and SelectPercentile (which selects only
a subset of features belonging to a certain percentile) are shown next. It's
possible to apply them both to regression and classification datasets
Feature selection and filtering
 Principal component analysis
• In many cases, the dimensionality of the input
dataset X is high and so is the complexity of
every related machine learning algorithm.
• Moreover, the information is seldom spread
uniformly across all the features and there will
be high entropy features together with low
entropy ones, which, of course, don't
contribute dramatically to the final outcome.
 Principal component analysis
• In general, if we consider a Euclidean space,
we have:

So each point is expressed using an orthonormal basis made

of m linearly independent vectors.
 Principal component analysis
Now, considering a dataset X, a natural question arises: is it possible
to reduce m without a drastic loss of information? Let's consider the
following figure (without any particular interpretation):
 Principal component analysis
• It doesn't matter which distributions generated
X=(x,y), however, the variance of the horizontal
component is clearly higher than the vertical one.
• As discussed, it means that the amount of
information provided by the first component is
higher and, for example, if the x axis is stretched
horizontally keeping the vertical one fixed, the
distribution becomes similar to a segment where
the depth has lower and lower importance.
 Principal component analysis
In order to assess how much information is
brought by each component, and the correlation
among them, a useful tool is the covariance
matrix (if the dataset has zero mean, we can use
the correlation matrix):