Cross-validation: trusting your numbers · Machine Learning, Hands-On

Every accuracy number you’ve reported so far was computed on a single train/test split. The number depends on which rows ended up in test — change the split, get a slightly different number. For small datasets, that variation can be five or ten percentage points. Cross-validation is the technique that averages over many splits to give you a more stable, more trustworthy estimate. By the end of this section you’ll have the discipline to never trust a single-split evaluation number again — and the tools to produce numbers worth quoting.

The single-split problem

A small thought experiment. Suppose you train a model and report a test accuracy of 0.84. What does the 0.84 actually tell you?

It tells you the model achieved 84% on that specific test set — the rows that happened to land in test when you split with random_state=42. With a different random seed, you might get 0.81 or 0.87. With a small test set (say, 200 rows), this variation can easily be 5 percentage points.

Reporting a single number suggests precision that isn’t really there. The honest version is “84% ± something” — but to know the something, you need multiple evaluations.

k-fold cross-validation: the standard

The standard approach:

Split the training data into k equal-sized folds (typically k=5 or k=10).
For each fold: a. Hold out that fold. b. Train the model on the other (k-1) folds. c. Score on the held-out fold.
Average the k scores. Report the average and the standard deviation.

In scikit-learn:

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=1000)
scores = cross_val_score(model, X_train, y_train, cv=5, scoring="accuracy")

print(f"CV scores: {scores}")
print(f"Mean: {scores.mean():.3f}  Std: {scores.std():.3f}")

Output:

CV scores: [0.823 0.841 0.818 0.835 0.829]
Mean: 0.829  Std: 0.008

The mean (0.829) is your point estimate of accuracy. The standard deviation (0.008) tells you how stable that estimate is. A small std (under ~0.02) means the model is reliably around 83%. A large std (above ~0.04) means a single-split number from this dataset can be misleading by several percentage points.

When to use cross-validation

Three situations where cross-validation makes the biggest difference:

1. Small datasets. A 200-row dataset with an 80/20 split leaves only 40 test rows. One unlucky row can swing accuracy by 2.5 percentage points. Cross-validation lets you use every row for both training and evaluation, multiplied across folds.

2. Hyperparameter tuning. GridSearchCV and RandomizedSearchCV use cross-validation internally — that’s how they choose hyperparameters without leaking the test set. The CV score is what they optimise.

3. Comparing models. When you’re deciding between, say, logistic regression and random forest, the difference might be smaller than the single-split variation. Cross-validation gives you mean scores you can actually compare honestly.

Stratified cross-validation

For classification with imbalanced classes, the default cross_val_score may produce folds where the rare class is under-represented or absent. Use StratifiedKFold to preserve class proportions in each fold:

from sklearn.model_selection import StratifiedKFold

scores = cross_val_score(
    model, X_train, y_train,
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
    scoring="f1",
)

In practice, scikit-learn’s classifier scorers default to stratified CV automatically. But for regression, or when you need control, explicitly specify it.

Time-series cross-validation

For temporal data (NEPSE prices, daily transactions, etc.), the random-split version of cross-validation is wrong — every fold would have future data in training. Use TimeSeriesSplit:

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5)
scores = cross_val_score(model, X_train, y_train, cv=tscv, scoring="neg_mean_absolute_error")
print(f"Negative MAE per fold: {scores}")

TimeSeriesSplit creates folds where each training set is followed by a held-out test fold in time order — so the model is always tested on the future of its training data. This matches how the model will be used in production.

Grouped cross-validation

For data where the same entity appears in multiple rows (the loan dataset with multiple loans per borrower from Course 02 Chapter 5), use GroupKFold:

from sklearn.model_selection import GroupKFold

gkf = GroupKFold(n_splits=5)
scores = cross_val_score(
    model, X_train, y_train,
    cv=gkf.split(X_train, y_train, groups=df_train["borrower_id"]),
    scoring="f1",
)

This ensures each borrower lands entirely in one fold (train or test), never split across folds. Same logic as Course 02 Chapter 5’s leakage discipline, applied to CV.

Putting it together: cross-validation inside hyperparameter tuning

The GridSearchCV and RandomizedSearchCV we used last section already do cross-validation internally. The full pattern combines everything:

from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000)),
])

param_grid = {
    "clf__C": [0.01, 0.1, 1.0, 10.0, 100.0],
    "clf__penalty": ["l1", "l2"],
    "clf__solver": ["liblinear"],
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

grid = GridSearchCV(
    pipeline,
    param_grid=param_grid,
    cv=cv,
    scoring="f1",
    n_jobs=-1,
)

grid.fit(X_train, y_train)

print(f"Best params: {grid.best_params_}")
print(f"Best CV F1: {grid.best_score_:.3f}")
print(f"Test F1: {grid.score(X_test, y_test):.3f}")

Three honest numbers come out:

The best CV F1 score (mean across the 5 folds). This is your tuned model’s expected F1.
The test F1 score. This is the truly honest evaluation on data the tuning never saw.
(Implicit) the spread of CV scores, which you can also access via grid.cv_results_ for a sense of stability.

If the CV F1 and test F1 are similar (within ~0.02), the model is robust. If they differ a lot, something is wrong — usually a data leak, distribution shift, or too-small test set.

A small worked example

On the spam dataset:

from sklearn.model_selection import cross_validate

scoring = ["accuracy", "precision", "recall", "f1"]
results = cross_validate(
    model, X_train, y_train,
    cv=5, scoring=scoring,
    return_train_score=True,
)

import pandas as pd
df_results = pd.DataFrame(results)
print(df_results[["test_accuracy", "test_precision", "test_recall", "test_f1"]].describe().round(3))

Output (illustrative):

       test_accuracy  test_precision  test_recall  test_f1
count          5.000           5.000        5.000    5.000
mean           0.974           0.913        0.881    0.897
std            0.005           0.018        0.023    0.014
min            0.968           0.895        0.860    0.880
max            0.980           0.937        0.913    0.917

You see five evaluations — one per fold. The mean is what you’d report. The standard deviation tells you how stable the estimate is. The min/max gives you a sense of worst-case performance across folds. This is a much richer picture than “98% accuracy” on its own.

The honest takeaway from this chapter

Chapter 5 had three big moves: pick the right features, tune the hyperparameters, validate with cross-validation. Together they routinely take a 75%-accurate baseline to an 85%-accurate model that you can defend to anyone. Most of the gain isn’t from clever modelling — it’s from disciplined practice.

The final number you report should be:

Model: Random Forest (best params from RandomizedSearchCV on training set) 5-fold CV F1 on training: 0.86 ± 0.013 Held-out test F1: 0.85 Per-group F1: Bagmati 0.87, Karnali 0.78 (recommend additional Karnali data before deployment).

Five numbers. Each one earned. None of them lying.

Check your understanding

Quick check

—

A teammate runs cross-validation and gets fold scores of [0.82, 0.69, 0.85, 0.91, 0.68]. They report 'mean accuracy 0.79.' What does the large spread tell you?

The mean is what matters; the spread is irrelevant.
A spread of 0.69 to 0.91 (range 0.22) means the model's performance is highly unstable across different splits of the data. A single-split number from this dataset would be untrustworthy — could be 0.69 or 0.91 depending on luck. Worth investigating: (1) the dataset may be small enough that 5 folds is too few; (2) there may be sub-populations that perform very differently (check per-group performance); (3) the model may be overfitting to specific characteristics of training folds. Don't deploy yet.
The model is perfect on the high folds; only train on those.
Switch to a different scoring metric.

Quick check

—

A teammate trains a NEPSE close-price prediction model and validates with the default 5-fold cross-validation. They report 'MAE = 4 NPR' on cross-validation. You suspect this is wrong. What's the likely issue?

The MAE is impossibly small; the model is too good.
Default 5-fold CV uses random splits — which mixes future and past dates across folds. For time-series data, this is leakage: every fold has 'future' data in training. Use TimeSeriesSplit instead, which respects temporal ordering. The honest MAE will likely be much higher, and the report from default CV is dangerously misleading.
NEPSE data is too complex for cross-validation.
Use a different model.

What comes next

You can build models, evaluate them, and tune them. The last chapter of Course 03 is the capstone: a complete, end-to-end ML project, framed from a real Nepali problem, built and iterated on, and presented honestly. By the end of Chapter 6 you’ll have one real, finished, shareable ML project — exactly like Course 01 ended with the NEPSE summariser. The shape of the work is now familiar; this time the model is something you trained yourself.