Tuning a model's settings

Parameters vs hyperparameters

A useful distinction:

• Parameters are the knobs inside the model that training adjusts — the weights in a logistic regression, the splits in a decision tree, the cluster centres in k-means. You don’t set these directly; model.fit(...) does.

• Hyperparameters are the settings of the model that you choose before training — max_depth, learning_rate, n_neighbors. You set these explicitly; training does not touch them.

Tuning means finding the hyperparameter values that make the trained model best (highest cross-validation accuracy, lowest cross-validation MSE, etc.).

The wrong way

A common beginner approach:

# DON'T DO THIS
for depth in [3, 5, 10, 20]:
    model = DecisionTreeClassifier(max_depth=depth)
    model.fit(X_train, y_train)
    test_acc = model.score(X_test, y_test)
    print(f"depth={depth}: {test_acc:.3f}")

# Then deploy with the best depth

The problem: you tuned on the test set. The “best” depth is the one that happens to perform best on the specific test set you have. That number is no longer an honest estimate of how the model will generalise — you optimised the test-set score directly.

The correct approach is to evaluate hyperparameters on a separate validation set (or via cross-validation, the better version), keep the test set untouched until the final report, and only then check honest performance.

The right way: GridSearchCV

scikit-learn’s GridSearchCV does the heavy lifting:

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

param_grid = {
    "n_estimators": [50, 100, 200],
    "max_depth": [3, 5, 10, None],
    "min_samples_split": [2, 5, 10],
}

grid = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid=param_grid,
    cv=5,           # 5-fold cross-validation on the training data
    scoring="accuracy",
    n_jobs=-1,      # use all CPU cores
    verbose=1,
)

grid.fit(X_train, y_train)

print(f"Best params: {grid.best_params_}")
print(f"Best CV score: {grid.best_score_:.3f}")

# Honest evaluation on untouched test set, once, at the end
print(f"Test accuracy: {grid.score(X_test, y_test):.3f}")

What just happened:

• GridSearchCV trains a fresh model for every combination of hyperparameters (here, 3×4×3 = 36 combinations).
• For each combination, it does 5-fold cross-validation on the training data, computing the average score across the 5 folds.
• It keeps the combination with the highest average CV score.
• grid.best_estimator_ is the model trained on all the training data with those best hyperparameters.
• You evaluate on the test set once at the end. The test set was never touched during tuning.

This is the gold standard for tuning. The only downside: with many hyperparameters and many values, the grid grows fast. A 36-combination grid with 5-fold CV trains 180 models. A 1000-combination grid trains 5000 models. On large datasets, this gets expensive.

RandomizedSearchCV — when the grid is too big

When the hyperparameter space is large, RandomizedSearchCV samples random combinations instead of trying every one:

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform

param_dist = {
    "n_estimators": randint(50, 500),
    "max_depth": randint(3, 20),
    "min_samples_split": randint(2, 20),
    "max_features": uniform(0.1, 0.9),
}

search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    param_distributions=param_dist,
    n_iter=50,        # try 50 random combinations
    cv=5,
    scoring="accuracy",
    n_jobs=-1,
    random_state=42,
)

search.fit(X_train, y_train)
print(search.best_params_)

Counterintuitively, RandomizedSearch with 50 trials often finds nearly the same best hyperparameters as a full grid with 200+ trials. The reason: most hyperparameters have a few key values where the model performs well, and 50 random tries are usually enough to land on (or near) at least one of them.

For most projects, RandomizedSearchCV with 30-50 iterations is more practical than GridSearchCV. Use GridSearch when you have a small, well-understood hyperparameter space.

Which hyperparameters to tune?

A short guide for the models in this course:

Logistic regression: C (inverse of regularisation strength, smaller = more regularisation), penalty (l1 vs l2), class_weight.

Random forest: n_estimators (more trees = better, with diminishing returns), max_depth (controls overfitting), min_samples_split, max_features (how many features each tree considers per split — a key knob for forest diversity).

Gradient boosting (GradientBoostingClassifier or xgboost): learning_rate, n_estimators, max_depth. These three interact strongly; tune them together.

K-NN: n_neighbors, weights (uniform vs distance), metric.

Decision tree: max_depth, min_samples_split, min_samples_leaf.

For most tabular projects, you do not need to tune all hyperparameters. Picking the 2-3 that matter most for your model class is enough to capture 80% of the gain.

A worked example — tuning a random forest

Starting from a default random forest with test accuracy 0.78, we’ll tune and see how far we can get:

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint
import time

# Baseline
default = RandomForestClassifier(random_state=42)
default.fit(X_train, y_train)
print(f"Default: train={default.score(X_train, y_train):.3f}  test={default.score(X_test, y_test):.3f}")

# Tuned
param_dist = {
    "n_estimators": randint(100, 500),
    "max_depth": randint(3, 30),
    "min_samples_split": randint(2, 20),
    "min_samples_leaf": randint(1, 10),
    "max_features": ["sqrt", "log2", 0.3, 0.5],
}

start = time.time()
search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    param_distributions=param_dist,
    n_iter=40,
    cv=5,
    scoring="f1",
    n_jobs=-1,
    random_state=42,
)
search.fit(X_train, y_train)
elapsed = time.time() - start

print(f"Search took {elapsed:.1f}s")
print(f"Best params: {search.best_params_}")
print(f"Best CV F1: {search.best_score_:.3f}")
print(f"Tuned test F1: {search.score(X_test, y_test):.3f}")

Typical result:

Default: train=0.998  test=0.778
Search took 67.4s
Best params: {'max_depth': 11, 'max_features': 'sqrt', 'min_samples_leaf': 4, 'min_samples_split': 14, 'n_estimators': 312}
Best CV F1: 0.804
Tuned test F1: 0.811

The tuned model is 3 percentage points better on test F1, the train/test gap closed (less overfitting), and the model is more robust. Sixty seconds of compute time bought 3-5 points of real performance.

Tuning works inside a pipeline

You can tune hyperparameters of any step inside a Pipeline using the step_name__hyperparameter syntax:

pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000)),
])

param_grid = {
    "clf__C": [0.01, 0.1, 1.0, 10.0, 100.0],
    "clf__penalty": ["l1", "l2"],
    "clf__solver": ["liblinear"],
}

grid = GridSearchCV(pipeline, param_grid, cv=5, scoring="f1")
grid.fit(X_train, y_train)

The double underscore (clf__C) tells the grid search “tune the C hyperparameter of the clf step in the pipeline.” This lets you combine preprocessing and tuning cleanly, with no leakage.

A note on time budgets

Hyperparameter search is constrained by your compute budget. A useful rule of thumb:

• Quick exploration: RandomizedSearchCV with 20-30 iterations, 5-fold CV. Hours-to-overnight on a laptop.
• Production model: GridSearchCV (or a more careful RandomizedSearch with 100+ iterations) on a focused subset of hyperparameters identified from quick exploration.
• Very large models / very long training: don’t tune everything. Use defaults from published configurations, tune one or two hyperparameters that matter most, and accept that further tuning has diminishing returns.

The tuning-budget conversation deserves to be explicit with your team and your stakeholders.

Check your understanding

Quick check

—

A teammate runs a `for` loop over different max_depth values, training each model on the training set and scoring it on the test set. They pick the depth with the highest test score and report that test score as 'the model's accuracy.' What's wrong?

1. ANothing — this is how tuning works.
2. BBy tuning hyperparameters against the test set, the reported test score is no longer an honest estimate of generalisation — it's the score on the test set that was *chosen because it was high*. The model may not perform as well on truly unseen data. Fix: use cross-validation on the training set to tune hyperparameters (e.g. GridSearchCV or RandomizedSearchCV with cv=5), and only score on the test set once, at the very end.
3. CThe for loop should iterate over more depth values.
4. DUse a larger model.

Check answer→

Quick check

—

You have 30 hyperparameters to potentially tune, each with 5 plausible values. A full grid search would require 5^30 trials — impossibly many. What's the practical approach?

1. ATrain every combination overnight; it will finish eventually.
2. BUse RandomizedSearchCV — sampling 30-100 random combinations from the full grid almost always finds a competitive (often near-best) configuration without exhaustive search. Alternatively, identify the 3-5 hyperparameters that matter most for your model class (e.g. for gradient boosting: learning_rate, n_estimators, max_depth) and grid-search only those, leaving the rest at defaults. Both approaches are standard practice.
3. CReduce the dataset until grid search becomes feasible.
4. DTry every combination by hand.

Check answer→

What comes next

You can pick hyperparameters honestly. The last section of Chapter 5 makes formal the technique we’ve been quietly using throughout: cross-validation. The discipline of training and evaluating on multiple splits to get a more honest estimate of model performance — and the small reasons it matters even when you “have lots of data.”