Performance Measures

Machine Learning Process

 Difference between Regression and Classification
Sl.No Regression Algorithm
1 In Regression, the output variable must be of
continuous nature or real value.
2 The task of the regression algorithm is to map
the input value (x) with the continuous output
variable(y).
3 Regression Algorithms are used with continuous
data.
4 In Regression, we try to find the best fit line,
which can predict the output more accurately.
5 Regression algorithms can be used to solve the
regression problems such as Weather
Prediction, House price prediction, etc.
6 The regression Algorithm can be further divided
into Linear and Non-linear Regression.
Classification Algorithm
In Classification, the output variable must be a discrete
value.
The task of the classification algorithm is to map the
input value(x) with the discrete output variable(y).
Classification Algorithms are used with discrete data.
In Classification, we try to find the decision boundary,
which can divide the dataset into different classes.
Classification Algorithms can be used to solve
classification problems such as Identification of spam
emails, Speech Recognition, Identification of cancer
cells, etc.
The Classification algorithms can be divided into Binary
Classifier and Multi-class Classifier.
 Confusion Matrix
• Need for confusion Matrix
• What is a confusion Matrix?
• Confusion Matrix example
• Metrics in confusion Matrix
• Confusion Matrix for multiclass classification
• Key Points in Confusion Matrix
Health care sector
Confusion Matrix
Health care sector
What is Confusion Matrix?
 Metrics in confusion Matrix
Performance Metrics
•Accuracy
•Precision
•Recall
•Specificity
•AUC
•F1 Score
Performance Charts
•ROC Curve
•Precision/Recall Curve
Accuracy
It is most common performance metric for classification
algorithms. It may be defined as the number of correct
predictions made as a ratio of all predictions made.
 Precision

• To overcome the limitations of Accuracy, Data Scientists usually use Precision,

Recall and Specificity. 
• Precision tells what proportion of positive predictions was actually correct. 
• It achieves this by counting the samples correctly predicted as positive (TP) and
dividing it by the total positive predictions, correct or incorrect (TP, FP).
 Recall
• Recall = Sensitivity = True Positive Rate = Hit Rate
• It does so by dividing the correctly predicted positive samples (TP) by
the total number of positives, either correctly predicted as positive or
incorrectly predicted as negative (TP, FN).
 Specificity
• Specificity = True Negative Rate = Selectivity
• Symmetrically to Recall (also known as Sensitivity), Specificity aims
at measuring what proportion of actual negatives was identified
correctly.
 F1 Score

• The F1 score is a less known performance metric, indicating the

harmonic mean of Precision and Recall.
• The highest value of an F1 Score is 1, indicating perfect Precision and
Recall.
• The lowest possible value is 0 if either the Precision or the Recall is
zero.

OR
𝑭𝟏 = 𝟐 ∗ (𝒑𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏 ∗ 𝒓𝒆𝒄𝒂𝒍𝒍) / (𝒑𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏 + 𝒓𝒆𝒄𝒂𝒍𝒍)
F-measure:
• If two models have low precision and high recall or vice versa, it is
difficult to compare these models.
• So, for this purpose, we can use F-score. This score helps us to
evaluate the recall and precision at the same time. The F-score is
maximum if the recall is equal to the precision.
Misclassification rate

Error rate=
Confusion Matrix for Multiclass
Classification
Adelie
Problem 1
 Problem 2
1. Suppose we train a model to predict whether an email is Spam or Not Spam. After training the model, we
apply it to a test set of 200 new email messages (also labeled) and the model produces the contingency
table below.

    True Class
    Spam Not Spam
Predicted Class Spam 70 30
Not Spam 70 330

[A] Compute the precision of this model with respect to the Spam class.
[B] Compute the recall of this model with respect to the Not Spam class.
[C] Suppose we have two users (Emily and Simon) with the following preferences.

Emily hates seeing spam emails in her inbox! However, she doesn’t mind periodically checking the “Junk” directory for
genuine emails incorrectly marked as spam.
Simon doesn’t even know where the “Junk” directory is. He would much prefer to see spam emails in his inbox than to miss
genuine emails without knowing!
 
Which user is more likely to be satisfied with this classifier? Why?
Problem 3
 Problem 4
• Consider the following 3-class confusion Matrix . Calculate precision
and recall per class .Also calculate weighted average ,precision and
recall for classifier.

Predicted Total

15 2 3

Actual 7 15 8

2 3 45

Total
 Predicted Total

20
15 2 3

30
Actual 7 15 8

50
2 3 45

100
Total 24 20 56
 Precision
• Precision=

• Class A
• Class B
• Class C
 Recall
• Recall =

• Class A
• Class B
• Class C
Accuracy
• Accuracy =

• Weighted Average Precision =

(Actual class A instances *precision of class A ) + (Actual class B
instances *precision of class B ) + (Actual class c instances *precision of
class c)
• Weighted Average Precision = (Actual class A instances *Recall of class
A ) + (Actual class B instances * Recall of class B ) + (Actual class c
instances * Recall of class c)
 Problem 5CAT
Predicted class
DOG RABBIT
DOG 23 12 27
Actual Class
CAT 11 29 13
RABBIT 4 10 24

FIND ,

TP(DOG) FP(DOG) FN(DOG) TN(DOG)

TP(CAT) FP(CAT) FN(CAT) TN(CAT)
TP(RABBIT) FP(RABBIT) FN(RABBIT) TN(RABBIT)
TP(OVERALL) FP(OVERALL) FN(OVERALL) TN(OVERALL)