Overfitting: when a model just memorises

The diagnostic, in two lines

The most reliable overfitting test in machine learning:

print(f"Train accuracy: {model.score(X_train, y_train):.3f}")
print(f"Test accuracy:  {model.score(X_test, y_test):.3f}")

If they’re close (say, within 0.03), your model is generalising well. If train is much higher than test (say, train 0.99, test 0.78), your model is overfitting — it has learned the training data too well, including its noise, and that learning doesn’t transfer.

This single comparison, done after every model fit, catches 90% of overfitting in practice. Make it a reflex.

Why overfitting happens

A model has capacity — roughly, how many distinguishable patterns it can represent. A logistic regression has small capacity (linear boundaries only). A decision tree with depth 3 has moderate capacity. A decision tree with no depth limit has enormous capacity. A neural network with millions of parameters has effectively unlimited capacity for typical tabular datasets.

A model with enough capacity to memorise the training data exactly will do so, unless something stops it. Memorising training data is, in fact, the cheapest way to achieve high training accuracy. The problem is that the memorised noise — random fluctuations in any real dataset that don’t carry signal — gets memorised right alongside the real patterns. Then on new data, the noise is different, the memorised noise patterns don’t fit, and the model is wrong.

In short: overfitting is what happens when capacity exceeds signal.

A useful mental picture: imagine a regression problem where 100 training points lie roughly on a straight line, with some scatter due to noise. A linear regression fits the line and predicts well on new points. A high-degree polynomial fits every training point exactly — passing through each one — and produces wildly wavy predictions between them. The polynomial has higher training accuracy and worse test accuracy. It memorised; it didn’t learn.

The three cures

Once you’ve diagnosed overfitting (big train/test gap), three reliable cures:

Cure 1 — reduce model capacity

The simplest fix. Use a smaller, less flexible model. Examples:

• Decision tree: set max_depth to a smaller value (3-8 for most tabular problems).
• k-NN: increase n_neighbors (each prediction averages over more points → smoother).
• Random forest / gradient boosting: reduce n_estimators (fewer trees), max_depth, or min_samples_split.
• Linear/logistic regression: remove features, or use Lasso to drive feature weights to zero.

Try a few capacity settings and watch the train/test gap close:

from sklearn.tree import DecisionTreeClassifier

for depth in [3, 5, 10, 20, None]:
    model = DecisionTreeClassifier(max_depth=depth, random_state=42)
    model.fit(X_train, y_train)
    train_acc = model.score(X_train, y_train)
    test_acc = model.score(X_test, y_test)
    print(f"depth={str(depth):4s}: train={train_acc:.3f}  test={test_acc:.3f}  gap={train_acc - test_acc:.3f}")

depth=3   : train=0.834  test=0.821  gap=0.013
depth=5   : train=0.852  test=0.831  gap=0.021
depth=10  : train=0.901  test=0.818  gap=0.083
depth=20  : train=0.978  test=0.762  gap=0.216
depth=None: train=1.000  test=0.731  gap=0.269

Sweet spot is around depth 5 — small gap, decent test accuracy. Beyond that, training accuracy keeps climbing but test accuracy falls. Classic overfitting curve.

Cure 2 — regularisation

Mathematically penalise model complexity during training. The model still has high capacity but is encouraged to use less of it.

• Linear/logistic regression: Ridge, Lasso, or ElasticNet (sklearn’s LogisticRegression has penalty='l2' by default).
• Neural networks: weight decay, dropout (not in scikit-learn; see Course 04 for the PyTorch version).
• Tree-based ensembles: min_samples_split, min_samples_leaf, learning rate × trees combinations.

In scikit-learn’s regression context:

from sklearn.linear_model import Ridge

for alpha in [0.01, 1.0, 100.0]:
    model = Ridge(alpha=alpha)
    model.fit(X_train, y_train)
    print(f"alpha={alpha:6.2f}: train={model.score(X_train, y_train):.3f}  test={model.score(X_test, y_test):.3f}")

Higher alpha → more regularisation → smaller training accuracy, smaller train/test gap. You’re trading some training fit for better generalisation.

Cure 3 — more data

The most expensive cure, but often the most effective. Every additional training point makes it harder for the model to fit noise (noise averages out across larger datasets) and easier to learn the underlying signal.

A useful experiment when you’re not sure if overfitting is fixable: try training on 25%, 50%, 75%, 100% of your data and see what happens.

import numpy as np

for fraction in [0.25, 0.5, 0.75, 1.0]:
    n = int(len(X_train) * fraction)
    model = DecisionTreeClassifier(max_depth=10, random_state=42)
    model.fit(X_train[:n], y_train[:n])
    train_acc = model.score(X_train[:n], y_train[:n])
    test_acc = model.score(X_test, y_test)
    print(f"{fraction*100:.0f}% of data: train={train_acc:.3f}  test={test_acc:.3f}  gap={train_acc - test_acc:.3f}")

If the test accuracy keeps climbing with more data, collect more data. If it has plateaued, more data won’t help — only regularisation or smaller models will.

The opposite problem: underfitting

A model can also be too simple — it doesn’t have enough capacity to fit even the training data well. The diagnostic is the opposite: both train and test accuracy are mediocre, and adding more training data doesn’t help.

Examples of underfitting:

• Linear regression on a U-shaped problem — Section 3.3 territory.
• Decision tree with max_depth=1 — only allows one split, can barely express anything.
• k-NN with n_neighbors=200 on a 500-row dataset — every prediction is roughly the global mean.

The cure for underfitting is the opposite of the cure for overfitting: more capacity, less regularisation, more features.

The art of model tuning is finding the sweet spot — enough capacity to fit the signal, not so much that you fit the noise. Chapter 5 (cross-validation) gives you a principled way to find this sweet spot.

Learning curves — the visual version

The diagnostic from this section, made into a chart:

import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve

train_sizes, train_scores, test_scores = learning_curve(
    DecisionTreeClassifier(max_depth=10, random_state=42),
    X_train, y_train,
    train_sizes=np.linspace(0.1, 1.0, 10),
    cv=5,
    scoring="accuracy",
)

plt.plot(train_sizes, train_scores.mean(axis=1), label="Train")
plt.plot(train_sizes, test_scores.mean(axis=1), label="Test")
plt.xlabel("Training set size")
plt.ylabel("Accuracy")
plt.legend()
plt.title("Learning curve")
plt.show()

You see two curves. If both rise and converge, you have a well-calibrated model — more data still helps. If they’re separated (overfitting), regularise or shrink the model. If both are low and flat (underfitting), add capacity.

Learning curves are one of the most informative single charts you can produce for a project. Always make one before declaring a model done.

A short summary

The whole shape of this chapter:

• Accuracy alone lies when classes are imbalanced or costs are asymmetric.
• Precision and recall tell different stories; the right trade-off depends on your application.
• Confusion matrix shows where mistakes live.
• F1 gives a single summary number when you need one.
• Train/test gap diagnoses overfitting; learning curves visualise it.
• Cures: smaller models, regularisation, more data — in that order of cost.

Carry this with you. Almost every “the model worked in training and failed in production” story is some combination of the issues this chapter named. With these tools, you avoid most of them.

Check your understanding

Quick check

—

A teammate reports a model with 99.5% training accuracy and 72% test accuracy. They are about to deploy it. What's the careful response?

1. A72% is fine — deploy.
2. BA train-test gap of 27 percentage points is classic overfitting. The model has memorised the training data and generalises poorly. Do not deploy. Try (1) reducing model capacity (smaller max_depth, fewer features, simpler model), (2) adding regularisation, (3) collecting more training data, then re-run the train/test comparison. Look for a gap below ~3-5 percentage points. The 72% test accuracy understates how badly this will behave in production.
3. CUse a larger model.
4. DSwitch random_state and try again.

Check answer→

Quick check

—

A model has 75% training accuracy and 73% test accuracy — small gap. The team wants higher accuracy overall. What's the right next step?

1. ARegularise more — the model is overfitting.
2. BThe small gap with mediocre accuracy on both is the signature of *underfitting*: the model isn't complex enough to fit the signal in the data. Try increasing capacity (more flexible model: random forest instead of logistic regression, deeper tree, more features), and watch what happens to the train/test gap. If train accuracy rises substantially while test rises a little, you've found signal. If train rises but test falls, you've gone past the sweet spot into overfitting.
3. C75% is good; don't try to improve.
4. DSwitch to predict_proba and adjust threshold.

Check answer→

What comes next

You can evaluate honestly and diagnose overfitting. Chapter 5 closes the loop on model quality with three reliable practices to actually make models better: thoughtful feature engineering, hyperparameter tuning, and cross-validation that earns trust in the numbers. After Chapter 5, you’ll have everything you need to handle a complete ML project from end to end — which is what Chapter 6 is about.