PYTHON PROGRAMMING & DATA SCIENCE

EVALUATION METRICS
 Evaluation Metrics
Evaluation Metrics
Evaluation metrics are used to measure the quality of the statistical or 
machine learning model.
 There are many different types of evaluation metrics available to test a
model.
Some Examples are:
1. classification accuracy
2. logarithmic loss
3.  confusion matrix, etc
 Classification accuracy is the ratio of the number of correct predictions to
the total number of input samples.
Logarithmic loss, also called log loss, works by penalizing the false
classifications.
 Evaluation Metrics
Evaluation Metrics
A confusion matrix gives us a matrix as output and describes the complete
performance of the model.
 Evaluation metrics involves using a combination of these individual
evaluation metrics to test a model or algorithm. 
 Evaluation Metrics
Evaluation metrics for classification,Regression & Clustering
For Classification some of the important evaluation matrices are:
1. Confusion Matrix
2. Recall,Sensitivity & Specificity
3. F1 Score
4. AUC-ROC(Area under ROC curve)

AUC-ROC(Area under ROC curve)

A ROC curve (receiver operating characteristic curve) is a graph showing
the performance of a classification model at all classification thresholds.
 Evaluation Metrics
Evaluation metrics for classification
AUC-ROC(Area under ROC curve)
This curve plots two parameters:
1. True Positive Rate- Number of True positive divided by the sum of the
number of True positive and the number of false negatives. It describes how
good the model is at predicting the positive class when the actual outcome
is positive.
2. False Positive Rate- The number of false positives divided by the sum of the
number of false positives and the number of true negatives.
The ROC curves plot the graph between True positive rate and false-positive
rate. These plots are generated at different classification thresholds.
 Evaluation Metrics
Evaluation metrics for classification
AUC-ROC(Area under ROC curve)
The following figure shows a typical ROC curve.
AUC is an acronym for Area under the curve.
It computes the entire 2D area under the ROC
curve.

it is a plot of FPR(False positive rate) on the x-axis and TPR(True positive rate)
on the y-axis for different threshold ranging from 0.0 to 1.
 Evaluation Metrics
Evaluation metrics for classification
AUC-ROC(Area under ROC curve)
 The AUC ROC Plot is one of the most popular metrics used for determining
machine learning model predictive capabilities.
some reasons for using AUC ROC plot-
1. The Different Machine learning models curves can be checked with
different thresholds.
2. Model predictive capability is summarized by the area under the
curve(AUC).
3. AUC is considered to be scaled variant, it measures the rank of predictions
rather than its absolute values
4. AUC always focuses on the quality of the Model’s skills on prediction
irrespective of what threshold has been chosen.
 Evaluation Metrics
Evaluation metrics for classification
Logarithmic Loss
 Logarithmic Loss or Log loss works by penalizing the false classification.
It is nothing but a negative average of the log of corrected predicted
probabilities for each instance.
 Log loss mostly suits the multi-class classification problem.
Log loss takes the probabilities for all classes present in the sample.
 If we minimize the log loss for a particular classifier we get better
performance metrics.
 Evaluation Metrics
Evaluation metrics for classification
Logarithmic Loss
 The math formula is given below

y_ij, indicates whether sample i belongs to class j or not

p_ij, indicates the probability of sample i belonging to class j
Log loss has a range [0, ∞). Moreover, it has no upper bound limit. We can
interpret log loss as it is nearer to 0 have higher accuracy whereas if log loss
moves away from 0 indicates lower accuracy.
 Evaluation Metrics
Evaluation metrics for Regression
For Regression some of the important evaluation matrices are:
1. Mean Absolute Error
2. Mean Squared Error
3. R Squared
R Squared
R squared is a statistical measure of how close the data point is fitted to the
regression line.
 It is also known as the coefficient of determination.
 R-Squared is defined by the explained variation divided by total variation
that is explained by the linear model.
 Evaluation Metrics
Evaluation metrics for Regression
R Squared
R squared value always lies between 0% to 100 %
0% indicates none of the variability of the response data around its mean
and 100 % shows model explains all the variability of the response data around
its mean.
This clearly means a higher R square value model perfect your model is.
R-squared = Explained variation / Total variation
 Evaluation Metrics
Evaluation metrics for Clustering
Clustering
it is difficult to figure out the quality of results from clustering. Evaluation
metric cannot depend on the labels but only on the goodness of split.
For Clustering some of the important evaluation matrices are:
1. Adjusted Rand Score
2. Adjusted mutual information
3. Silhouette
 Evaluation Metrics
Evaluation metrics for Clustering
Adjusted Rand Score
 Adjusted Rand score does not depend on the label values but on the cluster
split.
The formula for Adjusted Rand score is given by

Where ‘n’ be the number of observations in a sample,

a to be the number of observation pairs with the same labels and located in
the same cluster, and
b to be the number of observations with different labels and located in
different clusters.
 Evaluation Metrics
Evaluation metrics for Clustering
Adjusted mutual information
  Adjusted mutual information is defined by entropy function and
interpreted by sample split which is the likelihood of assigning a cluster.

Here K is a clustering result and C is the initial split.

 h evaluates whether each cluster is composed of same class objects
 c measures how well the same class fits the clusters.
 Evaluation Metrics
Evaluation metrics for Clustering
Silhouette
 The coefficient of silhouette score is calculated by mean intra-cluster
distance and the mean nearest-cluster distance for each sample.

The Silhouette distance shows up to which extent the distance between the
objects of the same class differs from the mean distance between the objects
from different clusters.
Values of silhouette score lie between -1 to +1 .
if the value is close to 1 then it corresponds to good clustering results having
dense and well-defined clusters
 Evaluation Metrics
Evaluation metrics for classification,Regression & Clustering
Silhouette
 if the value is close to -1 then it represents bad clustering.
Therefore, the higher the silhouette value is, the better the results from
clustering.