TYL

Machine Learning
Classification- Decision Tree

Rijo Jackson Tom, 9500 191 494, rijo.j@cmrit.ac.in

 05/03/2020
Decision Tree
• Decision tree builds classification or regression
models in the form of a tree structure.
• It breaks down a dataset into smaller and smaller
subsets while at the same time an associated
decision tree is incrementally developed.
• The final result is a tree with decision
nodes and leaf nodes.

Rijo Jackson Tom

• Decision trees can handle both categorical and
numerical data. 

2
 Decision Trees

Rijo Jackson Tom 05/03/2020

3
Our brain is hardwired to
understand visuals better than
text.

people remember up to 

80% of what they see,
compared to only about 20%
of what they read. 
 Rijo Jackson Tom 05/03/2020
5
 Money heist Decision
Tree

05/03/2020
• METICULOUS PLANNING — “DECISION
TREES”
• This is an extension of the “chess treatment”
argument — since the extensive research
helps “The Professor” envision all possible
outcomes — it allows him to create  
decision trees — a set of small plans
and (exhaustive) outcomes. And then he just
trains his team on how to achieve each sub-
plan.
• Eg: Tokyo’s activities:
• Cause and decisions--------

Rijo Jackson Tom

 Plan  Plan
Cameroon Chernobyl

6
 05/03/2020
Algorithm
• The core algorithm for building decision trees called ID3  by J. R.
Quinlan 

Rijo Jackson Tom

7
 05/03/2020
Algo
• ID3 uses Entropy and Information
Gain to construct a decision tree
• Entropy: ID3 algorithm uses entropy
to calculate the homogeneity of a
sample.
• If the sample is completely
homogeneous the entropy is zero and
if the sample is an equally divided it
has entropy of one.

Rijo Jackson Tom

• To build a decision tree, we need to
calculate two types of entropy using
frequency tables as follows:

8
 05/03/2020
a) Entropy using the frequency
table of one attribute:

Rijo Jackson Tom

9
 05/03/2020
b) Entropy using the frequency
table of two attributes:

Rijo Jackson Tom

10
 05/03/2020
Information Gain
• The information gain is based on the decrease in
entropy after a dataset is split on an attribute.
Constructing a decision tree is all about finding
attribute that returns the highest information gain
(i.e., the most homogeneous branches).
• Step 1: Calculate entropy of the target.

Rijo Jackson Tom

11
 05/03/2020
• The dataset is then split on the different attributes. The entropy for
each branch is calculated. Then it is added proportionally, to get
total entropy for the split. The resulting entropy is subtracted from

Step 2: the entropy before the split. The result is the Information Gain, or
decrease in entropy. 

Rijo Jackson Tom

12
 Step 3: Choose attribute with the

05/03/2020
•
largest information gain as the
decision node, divide the dataset
by its branches and
• repeat the same process on
every branch.

Rijo Jackson Tom

13
 Rijo Jackson Tom 05/03/2020
14
 05/03/2020
Rijo Jackson Tom
Step 5: The ID3 algorithm is run recursively on the non-leaf branches, until all data is
classified.
15
Decision Tree to Decision Rules

05/03/2020
• A decision tree can easily be transformed to a set of rules by mapping
from the root node to the leaf nodes one by one.

Rijo Jackson Tom

16
 05/03/2020
Notable decision tree algorithms
include:
• ID3 (Iterative Dichotomiser 3)
• C4.5 (successor of ID3)
• CART (Classification And Regression Tree) [4]
• Chi-square automatic interaction detection (CHAID). Performs multi-
level splits when computing classification trees. [12]
• MARS: extends decision trees to handle numerical data better.
• Conditional Inference Trees. Statistics-based approach that uses non-
parametric tests as splitting criteria, corrected for multiple testing to

Rijo Jackson Tom

avoid overfitting. This approach results in unbiased predictor selection
and does not require pruning

17
Thank you !
Python implementation