Evaluation metrics..

•Evaluation is essential part of model building

pipeline.
• Model may give satisfying results when
evaluated using some metric say
accuracy_score but may give poor results
when evaluated against some other metrics
such as logarithmic_loss.

29/04/2021 Dr Geetishree Mishra 1

Types:
• Classification Accuracy
• Log loss/Binary cross entropy
• Jaccard Similarity Index
• Confusion Matrix
• AUC-ROC

29/04/2021 Dr Geetishree Mishra 2

• Accuracy=
(No of correct predictions)/(Total no. of
predictions made)

• It works well only if there are equal number of

samples belonging to each class.

• Risk of false sense of achieving high accuracy.

29/04/2021 Dr Geetishree Mishra 3

 • Log Loss:
• Logarithmic Loss or Log Loss also called cross entropy, works by
penalizing the false classifications.
• It works well for multi-class classification.
• When working with Log Loss, the classifier must assign probability
to each class for all the samples.
• Suppose, there are N samples belonging to M classes, then the Log
Loss is calculated as:

29/04/2021 Dr Geetishree Mishra 4

• y_ij, indicates whether sample i belongs to class j
or not
• p_ij, indicates the probability of sample i
belonging to class j
• Log Loss has no upper bound and it exists on the
range [0, ∞).
• Log Loss nearer to 0 indicates higher accuracy,
whereas if the Log Loss is away from 0 then it
indicates lower accuracy.
• In general, minimising Log Loss gives greater
accuracy for the classifier.

29/04/2021 Dr Geetishree Mishra 5

• Jaccard Index:
•Given predicted values as (y )and actual values as y, the
Jaccard index can be defined as :

29/04/2021 Dr Geetishree Mishra 6

Confusion matrix..

29/04/2021 Dr Geetishree Mishra 7

AUC-ROC- Area Under the Curve- Receiver
Operating Characteristics
• AUC - ROC curve is a performance measurement
for the classification problems at various
threshold settings.
• ROC is a probability curve and AUC represents the
degree or measure of separability.
• Higher the AUC, the better the model is at
predicting 0s as 0s and 1s as 1s.
• AUC = ∫ TPR * d(FPR)

29/04/2021 Dr Geetishree Mishra 8

 AUC-ROC..

As evident, AUC has a range of [0, 1]. The greater the value, the better is the
performance
29/04/2021 of our model. Dr Geetishree Mishra 9
29/04/2021 Dr Geetishree Mishra 10
29/04/2021 Dr Geetishree Mishra 11
29/04/2021 Dr Geetishree Mishra 12