Confusing matrix: on rows, we have predicted outcomes, on columns we have actual

outcomes (or vice-versa). This matrix is the result of applying the trained model to the
validation test set to have a measure of the model performance.
Based on confusion matrix:

• Sensitivity / recall: if the % of true positives over the actual positives (sum on the
column of positives).
o How good is it to identify positives?
o A test can cheat and maximize this by always returning "positive".
o With more values, we have to calculate sensitivity for each category.
• Specificity: if the % of true negatives over the actual negatives (sum on the column of
negatives).
o How good is it to identify negatives?
o A test can cheat and maximize this by always returning "negative".
o With more values, we have to calculate specificity for each category.
• Precision: proportion of positive results that were correctly classified over the results
that were classified as positive.
o If there are a lot of negative samples, it makes sense to use precision rather
than false positive rate. This is because precision does not include the number
of true negatives in its calculation, and is not affected by the imbalance.
• ROC (Receiver Operator Characteristic): provides a simple way to summarize all the
confusion matrices tried (i.e. when choosing the threshold for the logistic regression).
o On the Y axis you have True Positive Rate (sensitivity).
o On the X axis you have False Positive Rate (1 - specificity).
o The diagonal line shows where the true positive rate = false positive rate. The
proportion of correctly classified Positives samples is the same as the
proportion of incorrectly classified samples that are Negative.
o If you measure the area under the curve (AUC) you can compare different
models and parameters at the same time.
o There are other matrices that attempt to do the same. For example, people
often replace False Positive Rate with Precision.