Confusion Matrix

A confusion matrix is a table used in machine learning and statistics to assess the performance
of a classification model. It summarizes the results of classification by showing the counts of
true positive, true negative, false positive, and false negative predictions. These values help
evaluate a model’s accuracy, precision, recall, and F1-score, which are crucial metrics for
understanding how well the model classifies data into different categories.

How to use the Confusion Matrix in ML?

To use a confusion matrix in machine learning in 4 steps:
1. Train a machine learning model. This can be done using any machine learning
algorithm, such as logistic regression, decision tree, or random forest.
2. Make predictions on a test dataset. This is a dataset of data that the model has not been
trained on.
3. Construct a confusion matrix. This can be done using a Python library such as Scikit-
learn.
4. Analyze the confusion matrix. Look at the diagonal elements of the matrix to see how
many instances the model predicted correctly. Look at the off-diagonal elements of the
matrix to see how many instances the model predicted incorrectly.

In summary, a confusion matrix tells you what your machine learning algorithm did
right and it did wrong.

An example of the confusion matrix we may obtain with the trained model is shown below for
a medical dataset for patients who Have Heart Disease (+ve) and those who Do Not Have
Heart Disease (-ve). This gives us a lot more information than just the accuracy of the model.

Adding the numbers in the first column, we see that the total samples in the positive class are
45+15=60. Similarly, adding the numbers in the second column gives us the number of
samples in the negative class, which is 40 in this case. The sum of the numbers in all the boxes
gives the total number of samples evaluated. Further, the correct classifications are the
diagonal elements of the matrix—45 for the positive class and 32 for the negative class.
1
Now, 15 samples (bottom-left box) that were expected to be of the positive class were
classified as the negative class by the model. So it is called “False Negatives” because the
model predicted “negative,” which was wrong. Similarly, 8 samples (top-right box) were
expected to be of negative class but were classified as “positive” by the model. They are thus
called “False Positives.” We can evaluate the model more closely using these four different
numbers from the matrix.

In general, we can get the following quantitative evaluation metrics from this binary class
confusion matrix:
1. Accuracy. The number of samples correctly classified out of all the samples present in the test
set.
Accuracy = (all correct / all) = (TP + TN) / (TP + TN + FP + FN)

2. Misclassification=Error Rate = (all incorrect / all) = (FP+FN)/(TP+TN+FP+FN)

3. Precision (for the positive class) aka Positive Predicted Value (PPV). Precision (for the
negative class) aka Negative Predicted Value (NPV). The number of samples actually
belonging to the positive/negative class out of all the samples that were predicted to be of the
positive class by the model.
Precision = (true positives / predicted positives) = PPV =TP / (TP + FP)
Precision = (true negatives / predicted negatives) = NPV = TN / (TN + FN

4. Recall (for the positive class) aka Sensitivity. The number of samples predicted correctly to
be belonging to the positive class out of all the samples that actually belong to the positive
class.
Sensitivity aka Recall (true positives / all actual positives) = TP / (TP + FN)

5. F1-Score (for the positive class). The harmonic mean of the precision and recall scores
obtained for the positive class.
F1-Score = (2 X Precision X Recall) / (Precision + Recall)

6. Specificity. The number of samples predicted correctly to be in the negative class out of all the
samples in the dataset that actually belong to the negative class.
Specificity = (true negatives / all actual negatives) = TN / (TN + FP)

2
 Confusion Matrix Exercise
Suppose we have a dataset consisting of two class labels Class 1: CAT and Class 2: DOG,
with three features (A, B and C). The Actual Class and Predicted class based on KNN
classifier are given in the table below.

Calculate the following measures to evaluate the KNN classification model, where Class 1
is the positive class.
Predicted
Data ID A B C Actual Class
Class
1 6 1 5 CAT CAT
2 4 1 6 CAT CAT
3 13 2 7 CAT DOG
4 12 3 4 DOG DOG
5 14 4 5 CAT CAT
6 36 5 4 DOG DOG
7 2 8 1 DOG DOG
8 20 3 2 CAT CAT
9 18 6 4 DOG CAT
10 9 6 6 CAT DOG

Calculate: (1) Accuracy, (2) Precision, (3) F-measure, (4) Specificity, (5) Sensitivity, and
(6) Recall.

Solution
First, we select the data with classes:
Table 1_1
Data ID Actual Class Predicted Class
1 CAT CAT
2 CAT CAT
3 CAT DOG
4 DOG DOG
5 CAT CAT
6 DOG DOG
7 DOG DOG
8 CAT CAT
9 DOG CAT
10 CAT DOG

3
We build the following table (Table 2_1) and need to know the values of TP, FP, FN, and TN:
Table 2_1
Actual (known)

CAT DOG
CAT TP FP

Predicted
DOG FN TN

We now:
1. Modify Table 1 by adding a new column to the right, and fill the new values into Table
2 to become as in Table 2_2 below.
2. Use the values in the Class column of Table 1_2 to fill Table 2 to become as shown in
Table 2_2 below.
Table 1_2 Table 2_2
Data Actual Predicted Actual (known)
Class
ID Class Class CAT DOG
1 CAT CAT TP CAT 4 1 5
2 CAT CAT TP Predicte
d
DOG 2 3 5
3 CAT DOG FN 6 4 10
4 DOG DOG TN
5 CAT CAT TP
6 DOG DOG TN
7 DOG DOG TN
8 CAT CAT TP
9 DOG CAT FP
10 CAT DOG FN

Now, we use Table 2_2 to calculate the required measurements values:

Measurement Formula Value
Accuracy = (TP+TN)/(Population)=(4+3)/10=7/10 0.700
Precision_PPV = TP/(TP+FP) = 4/(4+1)=4/5 0.800
Precision_NPV = TN/(TN+FN)=3/(3+2)=3/5 0.600
Sensitivity/Recall = TP/(TP+FN)=4/(4+2)=4/6 0.667
F1-Score = (2 X Precision X Recall) / (Precision + Recall) 0.727
Specificity = TN/(TN+FP)=3/(3+1)=3/4 0.750
Misclassification = (FP + FN) / (Population)=(1-Accuracy)=(1+2)/10 0.300