Scribd Logo
Upload

Welcome to Scribd!

  • Lec 19

    Uploaded by

    ABHIRAJ E
    22 views9 pages
    Random forests are an ensemble method that grows many classification trees. Each tree classifies a new vector and the forest chooses the most common classification. Randomness is introduced through bagging, where each tree uses a bootstrap sample of training data, and by selecting a random subset of variables to consider at each node. The strength and diversity of individual trees impacts the forest's accuracy, with more accurate and less correlated trees reducing error. The algorithm builds many trees, estimates out-of-bag error, and classifies new data through majority voting of the trees.

    Original Description:

    Original Title

    lec 19

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    Random forests are an ensemble method that grows many classification trees. Each tree classifies a new vector and the forest chooses the most common classification. Randomness is introduced through bagging, where each tree uses a bootstrap sample of training data, and by selecting a random subset of variables to consider at each node. The strength and diversity of individual trees impacts the forest's accuracy, with more accurate and less correlated trees reducing error. The algorithm builds many trees, estimates out-of-bag error, and classifies new data through majority voting of the trees.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pdf or txt
    22 views9 pages

    Lec 19

    Uploaded by

    ABHIRAJ E
    Random forests are an ensemble method that grows many classification trees. Each tree classifies a new vector and the forest chooses the most common classification. Randomness is introduced through bagging, where each tree uses a bootstrap sample of training data, and by selecting a random subset of variables to consider at each node. The strength and diversity of individual trees impacts the forest's accuracy, with more accurate and less correlated trees reducing error. The algorithm builds many trees, estimates out-of-bag error, and classifies new data through majority voting of the trees.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pdf or txt
    of 9

    Random Forests

    Prof. Navneet Goyal


    Random Forests
    • Ensemble method specifically designed for decision
    tree classifiers
    • Random Forests grows many classification trees
    (that is why the name!)
    • Ensemble of unpruned decision trees
    • Each base classifier classifies a “new” vector
    • Forest chooses the classification having the most
    votes (over all the trees in the forest)
    Random Forests
    • Introduce two sources of randomness: “Bagging”
    and “Random input vectors”
    – Each tree is grown using a bootstrap sample of training
    data
    – At each node, best split is chosen from random sample of
    variables instead of all variables
    Random Forests
    Random Forest Algorithm
    • M input variables, a number m<<M is specified such that at
    each node, m variables are selected at random out of the M
    and the best split on these m is used to split the node.
    • m is held constant during the forest growing
    • Each tree is grown to the largest extent possible
    • There is no pruning
    • Bagging using decision trees is a special case of random
    forests when
    m=M
    Random Forest Algorithm
    In the original paper on random forests, it was shown
    that the forest error rate depends on two things:
    • The correlation between any two trees in the forest.
    Increasing the correlation increases the forest error
    rate.
    • The strength of each individual tree in the forest. A
    tree with a low error rate is a strong classifier.
    Increasing the strength of the individual trees
    decreases the forest error rate.
    Random Forest Algorithm
    Step 1 – Build as many trees as u want! (Say P)
    Building a tree:
    Step 1 – take a 0.632 bootstrap sample of size N (P times)
    Step 2 – randomly select sqrt (M) features (at each decision node)
    while using a DT induction algorithm to build the tree
    Step 2 – Estimating Error rate
    Step 1 – take union of all OOB* data of all DTs
    Step 2 – test the accuracy of P DTs using all points in the union
    Step 3 – take average over all DTs
    Step 3 – Classify new data point
    Step 1 – classify each OOB data using each DT
    Step 2 – use majority voting to assign class label

    * OOB (out-of-bag): the training examples not selected in 0.632 bootstrap


    Random Forest
    Bagging Reduces Variance
    Two categories of
    samples: blue, red
    Two predictors: x1 and x2
    Diagonal separation ..
    hardest case for tree-based
    classifier
    Single tree decision
    boundary in orange.
    Bagged predictor
    decision boundary in
    red.

    Source: Albert A. Montillo, Ph.D.


    University of Pennsylvania, Radiology
    Rutgers University, Computer Science

    Guest lecture: Statistical Foundations of


    Data Analysis
    Temple University
    4-2-2009
    Random
    Random Forest
    Single tree decision boundary
    Bagging Reduces Variance

    100 bagged trees..

    Source: Albert A. Montillo, Ph.D.


    University of Pennsylvania, Radiology
    Rutgers University, Computer Science

    Guest lecture: Statistical Foundations of Data Analysis


    Temple University
    4-2-2009
    Random

    You might also like