Classification: Decision Tree Algorithm

Dr. Rudra Kalyan Nayak

Assistant Professor
School of CSE
VIT Bhopal University

12 April 2022 1
 To be Covered
 What is Classification ?
 Examples
 Types of Classification
 Classification Use case
 What is Decision Tree ?
 Terminologies associated to a Decision Tree
 Visualizing a Decision Tree
 Strengths and Weaknesses of Decision Tree

12 April 2022 2
 What is Classification ?

12 April 2022 3
 What is Classification Contd.

12 April 2022 4
 Example

12 April 2022 5
 Example Contd.

12 April 2022 6
 Example Contd.

12 April 2022 7
 Example Contd.

12 April 2022 8
 Types of Classification
 Decision Tree
 Random Forest
 Naïve Bayes
 KNN
 SVM

12 April 2022 9
 What is Decision Tree ?

12 April 2022 10
 Types of Decision Tree

 Categorical Variable Decision Tree

 Continuous Variable Decision Tree

12 April 2022 11
 Understanding a Decision Tree

12 April 2022 12
 Understanding a Decision Tree.. Contd.

12 April 2022 13
 Decision Tree Terminologies

12 April 2022 14
 How Does Decision Tree Work?
The algorithm selection is also based on the type of target
variables. Let us look at some algorithms used in Decision
Trees:
 ID3 (Iterative Dichotomiser 3)
 C4.5 (successor of ID3)
 CART (Classification And Regression Tree)
 CHAID (Chi-square automatic interaction detection
Performs multi-level splits when computing
classification trees)

12 April 2022 15
 Let’s First Visualize the Decision Tree

12 April 2022 16
 Let’s First Visualize the Decision Tree Contd.

12 April 2022 17
 Let’s First Visualize the Decision Tree Contd.

12 April 2022 18
 Let’s First Visualize the Decision Tree Contd.

12 April 2022 19
 Let’s First Visualize the Decision Tree Contd.

12 April 2022 20
 Let’s First Visualize the Decision Tree Contd.

12 April 2022 21
 How does a Tree Decide Where to Split?
(Attribute Selection Measures)

12 April 2022 22
12 April 2022 23
12 April 2022 24
12 April 2022 25
12 April 2022 26
 What is Entropy?

12 April 2022 27
 To be Covered

12 April 2022 28
 To be Covered

12 April 2022 29
 To be Covered

12 April 2022 30
 To be Covered

12 April 2022 31
 What is Information Gain?

12 April 2022 32
12 April 2022 33
12 April 2022 34
12 April 2022 35
12 April 2022 36
12 April 2022 37
12 April 2022 38
12 April 2022 39
12 April 2022 40
What is Classification ?
 Types of Classification
 Classification Use case
 What is Decision Tree ?
 Terminologies associated to a Decision Tree
 Visualizing a Decision Tree

12 April 2022 41
12 April 2022 42
 What is Pruning?
Pruning is a technique in machine learning and search
algorithms that reduces the size of decision trees by
removing sections of the tree that provide little power to
classify instances. Pruning reduces the complexity of the
final classifier, and hence improves predictive accuracy by
the reduction of over fitting.

12 April 2022 43
12 April 2022 44
 Strengths of Decision Tree approach
 Decision trees are able to generate understandable
rules.
 Decision trees perform classification without requiring
much computation.
 Decision trees are able to handle both continuous and
categorical variables.
 Decision trees provide a clear indication of which fields
are most important for prediction or classification.

12 April 2022 45
 Weaknesses of Decision Tree approach
 Decision trees are less appropriate for estimation tasks
where the goal is to predict the value of a continuous
attribute.
 Decision trees are prone to errors in classification
problems with many class and relatively small number
of training examples.
 Decision tree can be computationally expensive to
train.

12 April 2022 46
 Practice
Q. Calculate information gain for Temperature
and Humidity for the Play Tennis dataset given
in slide no.33.

12 April 2022 47