ICT 4304 MACHINE LEARNING TOOLS AND TECHNOLOGIES [3 0 0 3]

(Open Elective)

Evaluation Metrics for ML Models

Machine Learning metrics for Regression and
Classification problems
• Classification:
• Accuracy
• Logarithmic Loss
• Confusion Matrix
• F1 Score
• Confusion Matrix
• Specificity / Precision / True Negative Rate
• Sensitivity / Recall / True Positive Rate
• Regression Metrics:
• Root Mean Squared Error (RMSE)
• Mean Squared Error
Classification Accuracy
• It is the ratio of number of correct predictions to the total number of
input samples.
Logarithmic Loss
Logarithmic Loss or Log Loss, works by penalising the false classifications. It works well for
multi-class classification. When working with Log Loss, the classifier must assign probability to
each class for all the samples. Suppose, there are N samples belonging to M classes
Confusion Matrix
• True Positives : The cases in which we predicted YES and the actual
output was also YES.
• True Negatives : The cases in which we predicted NO and the actual
output was NO.
• False Positives : The cases in which we predicted YES and the actual
output was NO.
• False Negatives : The cases in which we predicted NO and the actual
output was YES.
• Confusion Matrixes compare the number of data points in the actual class
(on the y-axis) to the number in the predicted class (on the x-axis). The
matrix can consist of one, two (binary classification), or multiple classes
(>2). Every prediction can be categorized in one of 4 ways:
• True means your prediction is accurate:
• True Positives: Predicted yes and actual was yes.
• True Negatives: Predicted no and actual was no.
• False means your prediction is inaccurate:
• False Positives: Predicted yes but actual was no.
• False Negatives: Predicted no but actual was yes.
• Accuracy for the matrix can be calculated by taking average of the
values lying across the “main diagonal” i.e

Accuracy = 100+ 50/ 165 = 0.91

Area Under Curve
• Area Under Curve(AUC) is one of the metrics for evaluation. It is used
for binary classification problem. AUC of a classifier is equal to the
probability that the classifier will rank a randomly chosen positive
example higher than a randomly chosen negative example.
• True Positive Rate (Sensitivity) : True Positive Rate is defined as TP/
(FN+TP).

• True Negative Rate (Specificity)

F1 Score
• F1 Score is used to measure a test’s accuracy. F1 Score is the Harmonic
Mean between precision and recall. The range for F1 Score is [0, 1].
• Precision : It is the number of correct positive results divided by the
number of positive results predicted by the classifier.

• Recall : It is the number of correct positive results divided by the

number of all relevant samples (all samples that should have been
identified as positive).
• Mean Absolute Error
• Mean Absolute Error is the average of the difference between the
Original Values and the Predicted Values.

• Mean Squared Error

Example
• Actual values = [‘dog’, ‘cat’, ‘dog’, ‘cat’, ‘dog’, ‘dog’, ‘cat’, ‘dog’, ‘cat’, ‘dog’]
Predicted values = [‘dog’, ‘dog’, ‘dog’, ‘cat’, ‘dog’, ‘dog’, ‘cat’, ‘cat’, ‘cat’, ‘cat’]
 Accuracy:
Accuracy = (TP + TN) / (TP + TN + FP + FN) =
(3+4)/(3+4+2+1) = 0.70
Recall:
Recall = TP / (TP + FN) = 3/(3+1) = 0.75
Precision:
Precision = TP / (TP + FP) = 3/(3+2) = 0.60
F-score:
F-score = (2*Recall*Precision)/(Recall+Presision) =
(2*0.75*0.60)/(0.75+0.60) = 0.67
Specificity:
Specificity = TN / (TN + FP) = 4/(4+2) = 0.67
• Consider we have to model a classifier that classifies 2 kinds of fruits. We
have 2 types of fruits – apples and grapes – and we want our machine-
learning model to identify or classify the given fruit as an apple or grape.
• Sample Size: 15 samples of 2 fruits
• Distribution: 8 belong to Apples, and 7 belong to the Grapes class. We will
represent Apple as 1 and Grape as 0 class.
• The actual class for 8 apples and 7 grapes can be represented as:
Actual = [1,1,1,1,1,1,1,1,0,0,0,0,0,0,0] The classifier model
predicts 1 for Apple and 0 for grape.
• Assume that the classifier takes all 15 inputs and makes the following
predictions:
• Out of 8 apples, it will classify 5 correctly as apples and wrongly predict 3 as grapes.
• Out of 7 grapes, it will classify 5 correctly as grapes and wrongly predicts 2 as apples
• The prediction of the classifier may be as follows:
• Prediction = [1,0,0,0,1,1,1,1,0,0,0,0,0,1,1]
TP=5, FN=3, FP=2, TN=5
• https://www.analyticsvidhya.com/blog/2021/06/confusion-matrix-
for-multi-class-classification/#What_Is_a_Confusion_Matrix?