Splitting data: train, validation, test

Three pieces, three jobs

A clean experimental setup has three subsets of the data:

Training set. The data the model fits on. Usually 60-80% of the total. The model learns patterns here.

Validation set. The data you use to decide things during development — which model, which hyperparameters, which feature set works best. Usually 10-20% of the total. You look at this many times during a project.

Test set. The data you use to judge the final model. You look at this once, at the end. Usually 10-20% of the total. Treat it as sacred.

If you only have two splits (train and test), the test set ends up serving both jobs — and you accidentally tune your decisions to it. After fifteen rounds of “let me try this hyperparameter and see what test accuracy I get”, the test set is no longer testing anything. It became part of training.

The mechanics: a clean three-way split

Scikit-learn does not have a built-in three-way splitter, but two calls give you what you need:

from sklearn.model_selection import train_test_split

# First split: 80% temp / 20% test
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42,
)

# Second split: of the 80%, take 75% train / 25% val (so the original is 60/20/20)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, random_state=42,
)

print(len(X_train), len(X_val), len(X_test))

random_state=42 makes the split deterministic — re-running the script produces the same split. Essential for reproducibility.

For most projects, 60-20-20 or 70-15-15 is fine. When data is small (under 1,000 rows), consider a larger training share. When data is huge, you can shrink the val/test percentages — 80-10-10 of a million rows still gives you 100,000 in each evaluation set.

The four traps in splitting

Trap 1: Splitting time-series data randomly. A model predicting future NEPSE prices that was trained on data from 2025 interspersed with 2026 is cheating: it saw rows from later dates during training. The test rows from 2026 look “in distribution” because similar rows are in the training set. In production, the model sees only the past. The right split is by time:

df = df.sort_values("date")
cutoff = "2026-06-01"
train = df[df["date"] < cutoff]
test = df[df["date"] >= cutoff]

For time-series, always split chronologically. Always.

Trap 2: Stratification failures. When the target variable is imbalanced (e.g. only 5% of loans default), a random split can land all the defaults in the training set and none in the test. Use stratify=:

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42,
)

stratify=y ensures the class proportions are preserved in both splits. Always use it for classification problems with imbalanced classes.

Trap 3: Grouped data leaks. If the same borrower appears in multiple rows (multiple loans over time), a random split puts some of their rows in train and some in test. The model “knows” the borrower from training, and predicting their behaviour in test is too easy. Use GroupShuffleSplit:

from sklearn.model_selection import GroupShuffleSplit

splitter = GroupShuffleSplit(test_size=0.2, random_state=42)
train_idx, test_idx = next(splitter.split(X, y, groups=df["borrower_id"]))

This ensures all rows for a given borrower land in either train or test, never both. Use whenever your data has natural groups (borrowers, students, customers, devices).

Trap 4: Distribution shift. Even with a clean split, the training distribution might not match production. A model trained only on 2024 Kathmandu loans, evaluated on 2024 Kathmandu loans, may still fail on 2026 Karnali loans. The fix is not in the split — it is in collecting data that represents the deployment population. The split can only verify what you have; it cannot create representativeness that was not collected.

A complete preparation pipeline

Putting Chapter 5 together. A clean preparation pass from raw data to model-ready splits:

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

# 1. Load cleaned data (output of Chapter 3 / 4)
df = pd.read_csv("data/clean/loans.csv")

# 2. Engineer features
df["log_loan_amount"] = np.log1p(df["loan_amount"])
df["loan_to_income_ratio"] = df["loan_amount"] / df["monthly_income"]
df["application_date"] = pd.to_datetime(df["application_date"])
df["application_month"] = df["application_date"].dt.month

# 3. Choose feature columns and target
numeric = ["log_loan_amount", "loan_to_income_ratio", "application_month", "monthly_income"]
categorical = ["district", "occupation_category"]
target = "defaulted"

X = df[numeric + categorical]
y = df[target]

# 4. Split — stratified, with a held-out test set
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42,
)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, stratify=y_temp, random_state=42,
)

# 5. Build the preprocessing pipeline (fit only on train)
preprocess = ColumnTransformer([
    ("num", StandardScaler(), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])

# 6. Save the splits — you'll re-use these for every model you try
import joblib
joblib.dump((X_train, y_train), "data/splits/train.joblib")
joblib.dump((X_val, y_val),     "data/splits/val.joblib")
joblib.dump((X_test, y_test),   "data/splits/test.joblib")
joblib.dump(preprocess,         "data/splits/preprocess.joblib")

When you start training models in Course 03, you load these splits and use the same train/val/test boundaries for every experiment. Comparing models with different splits is comparing nothing.

What “honest” looks like, in the end

A trustworthy report on a model has this structure:

• Trained on data/splits/train.joblib (n=4,800 rows).
• Tuned hyperparameters using validation set (data/splits/val.joblib, n=1,600 rows).
• Final model evaluated once on test set (data/splits/test.joblib, n=1,600 rows).
• Test accuracy: 0.87. Test F1: 0.72.
• Training accuracy: 0.91 (slightly higher than test, as expected — modest overfit).
• Validation accuracy: 0.88 (very close to test, suggesting validation-set guidance was fair).

When the gap between training and test is huge (0.99 vs 0.62), you have overfitting and should not ship. When the gap between validation and test is huge (0.92 vs 0.68), you have overfit your hyperparameter search and should investigate. Honest evaluation reveals these gaps; sloppy splitting hides them.

Check your understanding

Quick check

—

You are building a model to predict next-week NEPSE closes. You use `train_test_split` with `random_state=42` to randomly split your daily price data 80/20. Your test accuracy is 94%, which seems great. What is most likely wrong?

1. ANothing — 94% is a strong result.
2. BTime-series data was split randomly, so the model trained on dates *after* some of the test dates — i.e., it saw the 'future' during training. The test set is contaminated. The fix is to split chronologically: train on data up to a cutoff date, test on data after. Real test accuracy will be much lower.
3. CUse a different random_state.
4. DTrain on more data.

Check answer→

Quick check

—

A loan dataset has 50,000 rows representing 5,000 unique borrowers (some borrowers have multiple loans). You do a random 80/20 split. The model achieves 91% test accuracy. A teammate suspects leakage. Is the suspicion justified?

1. ANo — random splitting always works.
2. BYes — the same borrower can appear in both train and test, so the model can effectively memorise individual borrower patterns from training data and re-recognise them in test, which is unfairly easy. Use `GroupShuffleSplit` with `groups=borrower_id` to ensure each borrower lands entirely in train or entirely in test.
3. CNo — having more loans per borrower helps the model.
4. DYes, but switching to a different model architecture will solve it.

Check answer→

What comes next

You have a clean, split, prepared dataset. The mechanical work is done. The last chapter of Course 02 steps back from the keyboard and looks at the human dimensions of all of this — where bias enters, what privacy means when your data is about real people in Nepal, and how to document a dataset so that someone you have never met can trust your work. The technical preparation matters. Doing it responsibly is what makes the work matter.