Scribd Logo
Upload

Welcome to Scribd!

  • Tree-Based Methods

    Uploaded by

    Thành Cao Đức
    22 views32 pages
    This document discusses different tree-based machine learning methods including classification trees, regression trees, random forests, and boosted trees. Tree-based methods work by recursively splitting data into purer groups based on predictor variables. Random forests and boosted trees are ensemble methods that combine many decision trees to improve predictive performance by reducing variance.

    Original Description:

    Original Title

    12. TREE-BASED METHODS

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    This document discusses different tree-based machine learning methods including classification trees, regression trees, random forests, and boosted trees. Tree-based methods work by recursively splitting data into purer groups based on predictor variables. Random forests and boosted trees are ensemble methods that combine many decision trees to improve predictive performance by reducing variance.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pdf or txt
    22 views32 pages

    Tree-Based Methods

    Uploaded by

    Thành Cao Đức
    This document discusses different tree-based machine learning methods including classification trees, regression trees, random forests, and boosted trees. Tree-based methods work by recursively splitting data into purer groups based on predictor variables. Random forests and boosted trees are ensemble methods that combine many decision trees to improve predictive performance by reducing variance.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pdf or txt
    of 32

    PHUONG NGUYEN

    TREE-BASED METHODS
    FORESTS: The Wisdom of Crowds?
    CONTENT
    1. CLASSIFICATION TREES

    2. REGRESSION TREES

    3. RANDOM FORESTS

    4. BOOSTED TREES
    TREES AND RULES
    ▪

    ▪
    ▪

    ▪
    RECURSIVE PARTITIONING

    ▪
    ▪

    ▪
    ▪

    ▪
    Income Lot_Size Ownership

    EXAMPLE:
    60.0 18.4 owner
    85.5 16.8 owner
    64.8 21.6 owner

    RIDING MOWERS 61.5


    87.0
    20.8
    23.6
    owner
    owner
    110.1 19.2 owner
    ▪ 108.0 17.6 owner
    82.8 22.4 owner
    69.0 20.0 owner
    93.0 20.8 owner
    51.0 22.0 owner
    81.0 20.0 owner
    75.0 19.6 non-owner
    52.8 20.8 non-owner
    64.8 17.2 non-owner
    43.2 20.4 non-owner
    84.0 17.6 non-owner
    49.2 17.6 non-owner
    59.4 16.0 non-owner
    66.0 18.4 non-owner
    47.4 16.4 non-owner
    33.0 18.8 non-owner
    51.0 14.0 non-owner
    63.0 14.8 non-owner
    HOW TO SPLIT
    ▪
    ▪

    ▪
    ▪
    ▪
    NOTE: CATEGORICAL VARIABLES
    ▪

    ▪
    ▪
    ▪
    ▪
    ▪
    THE FIRST SPLIT: INCOME = 60
    SECOND SPLIT: LOT SIZE = 21
    AFTER ALL SPLITS
    MEASURING IMPURITY: GINI INDEX
    ▪

    I(A) = 1 - σ𝑚 𝑝
    𝑘=1 𝑘
    2



    MEASURING IMPURITY: ENTROPY MEASURE
    ▪
    𝑚

    𝑒𝑛𝑡𝑟𝑜𝑝𝑦 𝐴 = − 𝑝𝑘 𝑙𝑜𝑔2 (𝑝𝑘 )


    𝑘=1

    ▪
    EXP: GINI & ENTROPY INDEX
    IMPURITY AND RECURSIVE PARTITIONING
    ▪

    ▪
    THE TREE: FIRST SPLIT
    THE TREE: AFTER THREE SPLITS
    The dominant class
    in this portion of the
    first split (those with
    income >= 60) is
    If Income < 60 AND
    “owner” – 11 owners
    Lot Size < 21,
    and 5 non-owners
    classify as “Nonowner”
    The next split for
    this group of 16 will
    be on the basis of
    lot size, splitting at
    Decision nodes 20
    Leaves
    (terminal/leaf nodes)

    The first split is on Income, then the next split is on Lot Size for both the low
    income group (at lot size 21) and the high income split (at lot size 20)
    DETERMINING LEAF NODE LABEL
    ▪

    ▪
    OVERFITTING PROBLEM
    ▪
    ▪

    ▪
    OVERFITTING PROBLEM

    Overfitting produces poor predictive performance – past a certain point in tree


    complexity, the error rate on new data starts to increase
    PRUNING
    ▪

    ▪
    PRUNING
    CC(T) = Err(T) +  L(T)
    ▪
    ▪
    ▪ 

    →
    REGRESSION TREES FOR REGRESSION
    ▪
    ▪
    ▪

    ▪
    ▪

    ▪
    ADVANTAGES/DISADVANTAGES OF TREES
    ▪
    ▪
    ▪
    ▪
    ▪
    ▪
    ▪
    ▪

    ▪
    ENSEMBLE LEARNING
    ▪

    ▪
    ▪
    ▪

    The Wisdom of Crowds


    BAGGING
    BOOSTING
    RANDOM FORESTS AND BOOSTED TREES
    ▪
    ▪

    ▪
    RANDOM FORESTS

    ▪
    ▪
    BOOSTED TREES
    SUMMARY
    ▪

    You might also like