Precision, recall, and the confusion matrix

The confusion matrix — the foundation

Every other metric in this section is computed from the confusion matrix, so we start there. For binary classification, the matrix is 2×2:

                    Predicted
                  No        Yes
            ┌─────────┬─────────┐
   No       │   TN    │   FP    │   ← Actual: No (the model said Yes wrongly)
            ├─────────┼─────────┤
   Yes      │   FN    │   TP    │   ← Actual: Yes (the model said No wrongly)
            └─────────┴─────────┘

Four cells, four important names:

• TP (True Positive) — model said yes, truth said yes. ✓
• TN (True Negative) — model said no, truth said no. ✓
• FP (False Positive) — model said yes, truth said no. ✗ (“false alarm”)
• FN (False Negative) — model said no, truth said yes. ✗ (“missed it”)

Accuracy is (TP + TN) / (TP + TN + FP + FN). The 95%-accurate-but-useless model from the last section has high TN, low TP, low FP, and high FN. The accuracy averages over them, hiding the fact that recall is terrible.

In scikit-learn:

from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_test, predictions))
# [[955  10]
#  [ 11 139]]

The output is [[TN, FP], [FN, TP]]. With confusion_matrix(..., labels=[1, 0]) you can flip the order if you want positives on top — useful when “positive” is the rare class you care about.

Precision and recall, formally

Two ratios computed from the confusion matrix, each with a specific meaning:

precision = TP / (TP + FP)    →  of the cases the model said "yes", what fraction were actually yes?
recall    = TP / (TP + FN)    →  of the cases that actually were "yes", what fraction did the model find?

Both range from 0 to 1. Both equal 1 in the perfect case.

In plain language:

• Precision answers “when the model raises an alarm, how often is it right?” Higher precision = fewer false alarms.
• Recall answers “how many of the real positives does the model actually catch?” Higher recall = fewer misses.

Examples to make it stick:

• A spam classifier with high precision rarely flags ham as spam — but might miss some real spam. The user trusts every flag (precision), but some spam still gets through (low recall).
• A loan-default classifier with high recall catches almost every actual defaulter — but also flags many non-defaulters as risky. The lender catches every bad loan (recall), but reviews many false alarms (low precision).

You almost never optimise for both simultaneously. You trade off.

The trade-off, made visible

For any probabilistic classifier (logistic regression, random forest, etc.), the prediction is “yes” when the predicted probability exceeds a threshold (default 0.5). Raising the threshold makes the model more conservative (higher precision, lower recall). Lowering the threshold makes it more aggressive (lower precision, higher recall).

import numpy as np
from sklearn.metrics import precision_score, recall_score

probabilities = model.predict_proba(X_test)[:, 1]   # probability of positive class

for threshold in [0.3, 0.5, 0.7]:
    preds = (probabilities >= threshold).astype(int)
    p = precision_score(y_test, preds)
    r = recall_score(y_test, preds)
    print(f"threshold={threshold}: precision={p:.3f}  recall={r:.3f}")

Output (illustrative):

threshold=0.3: precision=0.62  recall=0.88
threshold=0.5: precision=0.71  recall=0.62
threshold=0.7: precision=0.84  recall=0.41

Same model. Three threshold choices. Three different precision/recall trade-offs. The “right” threshold depends on which mistake costs more — which is a business question, not a technical one.

When to optimise for what

A short rules-of-thumb table:

Application	Care more about	Why
Disease screening (cheap follow-up test)	Recall	Missing a real case is much worse than a false alarm
Fraud flagging (manual review queue)	Precision	False alarms waste analyst time
Spam filter	Balance	False positives lose real email; false negatives are annoying
Loan default flagging (high-trust lender)	Recall	A missed default loses real money
Loan default flagging (high-volume retailer)	Precision	Too many false flags slow down approval
Resume screening	Precision (and audit per-group)	False positives waste interviews; false negatives reject good candidates
Search engine ranking	Precision at the top	Users rarely look past the first few results

Notice that almost every line is context-dependent. There is no universal “you should care about precision” or “you should care about recall.” Defining which one matters more — and saying so explicitly — is part of framing the problem.

F1: one number when you need just one

When you need to report a single quality number — say, for a hyperparameter search that compares dozens of models — use the F1 score, the harmonic mean of precision and recall:

F1 = 2 * (precision * recall) / (precision + recall)

In scikit-learn:

from sklearn.metrics import f1_score
f1 = f1_score(y_test, predictions)

F1 ranges 0 to 1 like precision and recall, and rewards models that have both high precision AND high recall — it punishes a model that gets one right but tanks the other. It’s the right single number for imbalanced classification when you don’t yet know what threshold you’ll deploy at.

For multi-class problems, scikit-learn computes F1 per class and averages, with two flavors:

• average="macro" — unweighted average across classes. Treats all classes equally.
• average="weighted" — weighted by support (number of samples per class). Reflects overall performance.

For imbalanced multi-class, macro-F1 is usually more informative.

classification_report — the daily-driver summary

scikit-learn’s classification_report computes precision, recall, F1, and support per class in one call:

from sklearn.metrics import classification_report
print(classification_report(y_test, predictions, target_names=["ham", "spam"]))

              precision    recall  f1-score   support
         ham       0.99      0.99      0.99       965
        spam       0.94      0.93      0.93       150

    accuracy                           0.98      1115
   macro avg       0.97      0.96      0.96      1115
weighted avg       0.98      0.98      0.98      1115

Read this. The ham class is easy (99% precision and recall). The spam class is harder (94% precision, 93% recall). Both are well above baseline. F1 of 0.93 on spam is a strong result for a balanced precision/recall trade-off.

The classification report is the right thing to read first on every classification project. It tells the whole story in eight lines.

ROC curves, AUC, and PR curves (briefly)

Two more visualisations worth knowing, both about the precision/recall/threshold story:

• ROC curve — plots true positive rate (= recall) against false positive rate for all thresholds. The area under the curve (AUC) is a single number summarising the model’s discriminative ability across all thresholds. AUC of 0.5 = random guessing. AUC of 1.0 = perfect. AUC of 0.8-0.9 is strong.

• Precision-Recall curve — plots precision against recall for all thresholds. More informative than ROC when classes are heavily imbalanced. The area under it (AP, average precision) is the corresponding single-number summary.

from sklearn.metrics import roc_auc_score, average_precision_score
probs = model.predict_proba(X_test)[:, 1]
print(f"ROC AUC: {roc_auc_score(y_test, probs):.3f}")
print(f"PR AUC:  {average_precision_score(y_test, probs):.3f}")

Use AUC when you need a single threshold-independent quality number. For imbalanced problems, prefer PR-AUC.

Check your understanding

Quick check

—

A clinic deploys a TB screening classifier where every flagged patient gets a free cheap follow-up test. The clinic asks you to optimise the model. Which metric should you focus on, and why?

1. APrecision, because false positives waste resources.
2. BRecall, because in screening contexts, missing a real TB case is much worse than a false alarm. The cost of a follow-up test is small and recoverable; the cost of an undiagnosed TB case is much larger and may compound (continued transmission, delayed treatment). When the cost of false negatives substantially exceeds the cost of false positives, you optimise for recall — and you accept a lower threshold to catch more positives, even at the price of more false alarms.
3. CF1 alone, because it balances precision and recall.
4. DAccuracy, because it summarises everything.

Check answer→

Quick check

—

A classification report shows precision=0.40, recall=0.95 for the positive class. What is the model doing?

1. AThe model is performing badly and should be discarded.
2. BThe model is being aggressive — it catches 95% of real positives (high recall) but only 40% of its predicted positives are actually positive (the other 60% are false alarms). Useful in screening contexts where a false alarm costs little. The decision threshold is probably set quite low. Whether this is good or bad depends entirely on the application: high false-alarm cost = bad, high false-negative cost = good.
3. CThe model has a bug; precision and recall should always be equal.
4. DThe model is overfitting.

Check answer→

What comes next

You can evaluate a model honestly. The last section of Chapter 4 covers the failure mode that hides from precision/recall too if you don’t know where to look: overfitting. The phenomenon where a model learns the training data too well — including its noise — and fails on new data. We’ll meet learning curves, train/test gap diagnostics, and the small disciplines that keep your models honest.