Scribd Logo
Upload

Welcome to Scribd!

  • 15 views

    Foundations of Machine Learning: Sudeshna Sarkar IIT Kharagpur

    Uploaded by

    Venkatesh Gaduputi
    This document discusses feature extraction and dimensionality reduction techniques. It begins by defining feature extraction as mapping a set of features to a reduced feature set that maximizes a learner's ability to classify patterns. It then discusses principal component analysis (PCA) and how it works to find a projection of data to a lower dimensional space while preserving as much of the variance as possible. However, PCA does not consider class labels. Linear discriminant analysis (LDA) is then introduced as a technique that finds projections that maximize the distance between class means while minimizing within-class variance, with the goal of better separating classes.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Flag for inappropriate content

    You might also like

    Foundations of Machine Learning: Sudeshna Sarkar IIT Kharagpur

    Uploaded by

    Venkatesh Gaduputi
    15 views19 pages
    This document discusses feature extraction and dimensionality reduction techniques. It begins by defining feature extraction as mapping a set of features to a reduced feature set that maximizes a learner's ability to classify patterns. It then discusses principal component analysis (PCA) and how it works to find a projection of data to a lower dimensional space while preserving as much of the variance as possible. However, PCA does not consider class labels. Linear discriminant analysis (LDA) is then introduced as a technique that finds projections that maximize the distance between class means while minimizing within-class variance, with the goal of better separating classes.

    Original Description:

    Original Title

    3c Feature Extraction

    Copyright

    © © All Rights Reserved

    Available Formats

    PPTX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    This document discusses feature extraction and dimensionality reduction techniques. It begins by defining feature extraction as mapping a set of features to a reduced feature set that maximizes a learner's ability to classify patterns. It then discusses principal component analysis (PCA) and how it works to find a projection of data to a lower dimensional space while preserving as much of the variance as possible. However, PCA does not consider class labels. Linear discriminant analysis (LDA) is then introduced as a technique that finds projections that maximize the distance between class means while minimizing within-class variance, with the goal of better separating classes.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pptx, pdf, or txt
    15 views19 pages

    Foundations of Machine Learning: Sudeshna Sarkar IIT Kharagpur

    Uploaded by

    Venkatesh Gaduputi
    This document discusses feature extraction and dimensionality reduction techniques. It begins by defining feature extraction as mapping a set of features to a reduced feature set that maximizes a learner's ability to classify patterns. It then discusses principal component analysis (PCA) and how it works to find a projection of data to a lower dimensional space while preserving as much of the variance as possible. However, PCA does not consider class labels. Linear discriminant analysis (LDA) is then introduced as a technique that finds projections that maximize the distance between class means while minimizing within-class variance, with the goal of better separating classes.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pptx, pdf, or txt
    of 19

    Foundations of Machine Learning

    Module 3: Instance Based Learning and Feature


    Reduction

    Part C: Feature Extraction

    Sudeshna Sarkar
    IIT Kharagpur
    Feature extraction - definition
    ••  Given a set of features
    the Feature Extraction(“Construction”) problem is
    is to map to some feature set that maximizes the
    learner’s ability to classify patterns
    Feature Extraction
    • Find
      a projection matrix w from N-dimensional to M-
    dimensional vectors that keeps error low
    • Assume that N features are linear combination of vectors

    • What we expect from such basis


    – Uncorrelated, cannot be reduced further
    – Have large variance or otherwise bear no information
    Geometric picture of principal components (PCs)
    Geometric picture of principal components (PCs)
    Geometric picture of principal components (PCs)
    Algebraic definition of PCs
    Given a sample of p observations on a vector of N variables

     x , x ,, x   
    1 2 p
    N

    define the first principal component of the sample


    by the linear transformation
    N
    z1  w x j   wi1 xij ,
    T
    1 j  1,2,  , p.
    i 1

    where the vector w1  ( w11 , w21 , , wN 1 )


    x j  ( x1 j , x2 j ,, x Nj )
    is chosen such that var[ z1 ] is maximum.
    PCA
    PCA
    • Choose
      directions such that a total variance of data
    will be maximum
    1. Maximize Total Variance
    • Choose directions that are orthogonal
    2. Minimize correlation

    • Choose orthogonal directions which maximize total


    variance
    PCA
    • -dimensional
      feature space
    • symmetric covariance matrix estimated from
    samples
    • Select largest eigenvalue of the covariance matrix
    and associated eigenvectors
    • The first eigenvector will be a direction with largest
    variance
    PCA for image compression

    p=1 p=2 p=4 p=8

    Original
    p=16 p=32 p=64 p=100 Image
    Is PCA a good criterion for classification?

    • Data variation
    determines the
    projection direction
    • What’s missing?
    – Class information
    What is a good projection?
    Two classes
    • Similarly, what is a
    overlap
    good criterion?
    – Separating different
    classes

    Two classes are


    separated
    What class information may be useful?

    • Between-class distance
    – Distance between the centroids
    of different classes

    Between-class distance
    What class information may be useful?
    • Between-class distance
    – Distance between the centroids of
    different classes
    • Within-class distance
    • Accumulated distance of an instance
    to the centroid of its class

    • Linear discriminant analysis (LDA) finds


    most discriminant projection by
    • maximizing between-class distance
    • and minimizing within-class distance Within-class distance
    Linear Discriminant Analysis
    • Find
      a low-dimensional space such that when
    is projected, classes are well-separated
    Means and Scatter after projection
    Good Projection
    • Means are as far away as possible
    • Scatter is small as possible
    • Fisher Linear Discriminant

     m1  m2 
    2

    J  w 
    s s
    2
    1
    2
    2
    Thank You

    You might also like