Accuracy, and why it lies

What accuracy is, exactly

For a classifier, accuracy is the fraction of predictions that match the ground-truth label:

accuracy = (number correct) / (total predictions)

That’s all. It is a single number between 0 and 1 (often shown as a percentage). Higher is better.

In code:

from sklearn.metrics import accuracy_score
accuracy = accuracy_score(y_test, predictions)

So far so familiar. The trouble starts when you read the number without understanding what it can and cannot tell you.

The baseline accuracy you have to beat

Before you celebrate any accuracy number, compute the baseline accuracy — the score a trivial model would get by always predicting the most common class.

For a balanced binary classification (50/50 split), the baseline is 50%. Anything below that is worse than a coin flip.

For a 95/5 imbalanced classification (95% class 0, 5% class 1), the baseline is 95% — achieved by predicting class 0 for everything. A model that reports 94% accuracy is worse than the baseline, despite the impressive-looking number.

import pandas as pd

# Baseline: always predict majority class
majority_class = y_train.value_counts().idxmax()
baseline_accuracy = (y_test == majority_class).mean()
print(f"Baseline (majority-class) accuracy: {baseline_accuracy:.3f}")

Always compute this. Always compare against it. A model that doesn’t beat the baseline is not a model — it is a roundabout way of predicting the majority class.

The three classic ways accuracy lies

1. Class imbalance. As above. The textbook trap; every beginner falls in. A loan-default dataset with 5% defaulters, a disease screening with 1% positives, a fraud-detection dataset with 0.1% fraud — all let a “predict no” model score over 95% accuracy while being completely useless.

2. Cost asymmetry. A model that misses 10 actual cancers but correctly classifies 990 non-cancers has 99% accuracy. A model that misses 0 actual cancers but flags 50 false positives has 95% accuracy. In a real screening application, the second model is far more useful — false positives lead to one more test, false negatives lead to a missed cancer. Accuracy treats both kinds of mistake as equal; reality does not.

3. Distribution shift. Test accuracy is computed on the test set sampled from the same distribution as training. If the deployment distribution differs (recall the Lalitpur-vs-Karnali example from Course 02), test accuracy overstates real-world accuracy — sometimes dramatically.

Three different lies, three different responses. We address (1) and (2) in the next section with precision/recall and the confusion matrix. We address (3) in Course 02 Chapter 6 territory — per-group evaluation and representative datasets.

A worked example of the trap

import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, classification_report
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification

# Make a 95/5 imbalanced binary classification problem
X, y = make_classification(
    n_samples=10000, n_features=10, weights=[0.95, 0.05],
    random_state=42,
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42,
)

# Baseline: always predict majority
baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)
print(f"Baseline accuracy: {accuracy_score(y_test, baseline.predict(X_test)):.3f}")

# Actual model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
print(f"Model accuracy:    {accuracy_score(y_test, model.predict(X_test)):.3f}")

# Honest report
print(classification_report(y_test, model.predict(X_test)))

Output (illustrative):

Baseline accuracy: 0.950
Model accuracy:    0.952

              precision    recall  f1-score   support
           0       0.96      0.99      0.97      1900
           1       0.61      0.21      0.31       100

    accuracy                           0.95      2000

The model is barely above baseline (0.952 vs 0.950). Reading the classification report reveals why: the recall on class 1 is 21% — the model is catching one-fifth of the minority cases. Most of the model’s “accuracy” comes from correctly classifying the abundant majority class, which the baseline does too.

If you stopped at the 95.2% number, you’d be lying to yourself and to whoever consumed your report. The classification report tells the truth.

A small philosophical note

Accuracy is not bad. It is a valid metric, deeply intuitive, and the right thing to report when:

• The classes are balanced (or close to it).
• All misclassifications cost roughly the same.
• The training and deployment distributions match.

When those three conditions hold, accuracy is a clean summary of model performance. The mistake is treating accuracy as universally informative — it isn’t, and the conditions above almost never all hold in real-world problems.

The grown-up move is to always look at the classification report (which we’ll unpack in the next section), always compare against a baseline, and always evaluate per-group if there are groups that matter.

Check your understanding

Quick check

—

A team reports a tuberculosis-screening classifier with 99% test accuracy on a dataset where 1% of patients have TB. Which is the most thoughtful first response?

1. A99% is excellent; ship the model.
2. BA model that predicts 'no TB' for everyone scores 99% on this dataset. So 99% accuracy may simply be the baseline — the model could be learning nothing. Ask for the classification report or confusion matrix to see recall on the TB-positive class. If recall is also high (say 80%+), the model is real. If recall is near zero, the model is useless despite the impressive accuracy number.
3. CIncrease the training data.
4. DSwitch to a different ML library.

Check answer→

Quick check

—

A teammate argues: 'we don't need a baseline, the model gets 89% accuracy.' What's the right response, given this section's reasoning?

1. A89% is high enough to skip the baseline check.
2. B89% accuracy is informative only when you know the baseline. If the dataset is balanced (50/50), 89% is a strong result. If the dataset is 90/10 imbalanced, 89% is below baseline — a useless model. We don\'t know whether 89% is good or bad without the comparison; computing the baseline is two lines of code.
3. CAccuracy is the only metric that matters.
4. DBaselines are an academic concept and don\'t apply to production models.

Check answer→

What comes next

Accuracy lies. The next section meets the metrics that don’t: precision, recall, F1, and the confusion matrix. Together they let you read what a classifier is doing, on which class, and where its mistakes live. This is the vocabulary of every honest ML report you will ever produce.