Consider a classifier that acts on a statistical system with 4 labels i = 1, 2, 3, 4.

Consider
a particular test set that has x1 = x2 = x3 = x4 = 100 instances from each class. Further,
suppose the classifier makes the following number of correct predictions, y1 = 100, y2 =
75, y3 = 50, y4 = 100. Then, the in-class accuracy is going to be

y1 y2 y3 y4
= 1, = 0.75 = 0.5 =1 (0.1)
x1 x2 x3 x4

These will be the diagonals of the plotted confusion matrices.

The overall accuracy, which is plotted in the epoch graph is
P
y 325
P i = = 0.8125 (0.2)
xi 400

This is larger than some diagonal elements.