Decision trees you can read · Machine Learning, Hands-On

A decision tree is a model you can draw on paper. It asks a series of yes/no questions about the input — is loan_to_income_ratio greater than 2.5? is district Karnali? — and follows the answers down a branching path until it reaches a leaf, where it predicts a label. The structure is intuitive: it is how a thoughtful human might explain a decision to a friend. And because the model is its tree, you can print the tree, read the rules, and challenge them — which makes decision trees one of the most explainable models in machine learning.

The shape, drawn

Picture a tree. At the top is the root, asking one question:

                  loan_to_income_ratio > 2.5 ?
                         /             \
                       no                yes
                       /                   \
              monthly_income > 15000     district = Karnali?
                  /        \                 /          \
                yes         no              no           yes
                |           |               |            |
            no-default   no-default     default       default

Each non-leaf node asks a question. Each leaf assigns a label. To predict for a new loan, start at the root, answer the question, follow the appropriate branch, repeat until you hit a leaf, and report that leaf’s label.

The training procedure for a decision tree is: pick the question that best splits the data into two groups (the most “no-defaults” on one side, the most “defaults” on the other), recurse on each side, stop when the groups are pure (one label dominates) or when the tree gets too deep.

Training a tree on the loan data

Same pipeline pattern as before:

from sklearn.tree import DecisionTreeClassifier
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder

numeric = ["loan_amount", "monthly_income", "loan_to_income_ratio"]
categorical = ["district"]

model = Pipeline([
    ("preprocess", ColumnTransformer([
        ("num", "passthrough", numeric),   # trees don't need scaling
        ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
    ])),
    ("classifier", DecisionTreeClassifier(max_depth=5, random_state=42)),
])

model.fit(X_train, y_train)
predictions = model.predict(X_test)

Two things to notice:

"passthrough" for numeric features. Decision trees compare values directly — is X > 2.5? — so scaling them adds nothing. Trees are immune to the scale problem that bedevils k-NN. This is one of the trees’ practical advantages.
max_depth=5. The most important knob on a decision tree. Without it, the tree grows until every leaf is pure, which means it memorises the training data (overfit). With it, the tree stays shallow and generalises better. Chapter 4 returns to this.

Reading the tree

scikit-learn lets you print the tree in plain text:

from sklearn.tree import export_text

classifier = model.named_steps["classifier"]
preprocess = model.named_steps["preprocess"]
feature_names = preprocess.get_feature_names_out()

tree_text = export_text(classifier, feature_names=list(feature_names))
print(tree_text)

Output (truncated):

|--- num__loan_to_income_ratio <= 2.45
|   |--- num__monthly_income <= 14500.00
|   |   |--- num__loan_amount <= 25000.00
|   |   |   |--- class: 0
|   |   |--- num__loan_amount > 25000.00
|   |   |   |--- class: 1
|   |--- num__monthly_income > 14500.00
|   |   |--- class: 0
|--- num__loan_to_income_ratio > 2.45
|   |--- cat__district_Karnali <= 0.50
|   |   |--- class: 1
|   |--- cat__district_Karnali > 0.50
|   |   |--- class: 1

Read this. The model is telling you, in plain rules: if loan-to-income ratio is high, predict default. If it’s moderate but income is low and the loan is large, also predict default. Otherwise, predict no-default. You can read this rule set to a microfinance officer and they will recognise it as a reasonable heuristic. The model is not a black box — it is a small rule book the algorithm wrote for you.

Visualising a small tree

For small trees (depth 3 or 4), you can render them as a picture:

import matplotlib.pyplot as plt
from sklearn.tree import plot_tree

fig, ax = plt.subplots(figsize=(16, 8))
plot_tree(
    classifier,
    feature_names=list(feature_names),
    class_names=["no-default", "default"],
    filled=True,
    rounded=True,
    fontsize=10,
)
plt.show()

The result is a colour-coded tree with each split labelled. For real projects with deep trees this gets unwieldy, but for max_depth=3 or max_depth=4 it produces a beautiful summary worth pasting into a slide.

What `max_depth` does — the central knob

A decision tree without max_depth (or with a very large one) grows until every leaf is pure. On 12,000 training examples, this might produce a tree with thousands of leaves, each carving out tiny corners of feature-space. Training accuracy will be near 100%. Test accuracy will be much lower, because the tree has memorised noise.

A shallow tree (max_depth=3, max_depth=5) forces simpler rules. Some training points end up misclassified — and that is fine, because the rules generalise.

Try several values and watch what happens:

for depth in [3, 5, 10, 20, None]:
    model = make_tree_pipeline(max_depth=depth)
    model.fit(X_train, y_train)
    train_acc = model.score(X_train, y_train)
    test_acc = model.score(X_test, y_test)
    print(f"depth={depth}: train={train_acc:.3f}  test={test_acc:.3f}")

You will see something like:

depth=3:    train=0.834  test=0.821
depth=5:    train=0.847  test=0.831
depth=10:   train=0.902  test=0.819
depth=20:   train=0.987  test=0.762
depth=None: train=1.000  test=0.731

Depth 5 is the sweet spot here — small enough to generalise, large enough to capture the real structure. Beyond that, training accuracy keeps climbing while test accuracy falls. This is overfitting, and Chapter 4 will name it formally.

Trees alone vs. trees as building blocks

A single decision tree is decent. It is rarely the best model on its own, because individual trees can be brittle — one shifted threshold and the predictions change. The real power of trees comes when you combine many of them.

Two of the strongest classical ML models — both built from trees — are:

Random forests. Train hundreds of trees on different random samples of the data, with each split considering only a random subset of features. Average their votes.
Gradient boosting (XGBoost, LightGBM, scikit-learn’s GradientBoostingClassifier). Train trees sequentially, with each new tree correcting the errors of the previous ones.

Both are in Chapter 5. For now, hold in mind: the decision tree you just learned is the building block of those much more powerful models. Mastering the simple one earns you the more sophisticated ones for almost free.

Check your understanding

Quick check

—

A teammate trains a decision tree with no max_depth and reports 99.8% training accuracy. They are about to deploy it. What is the warning sign?

99.8% is excellent; deploy.
Without a max_depth, the tree grows until almost every leaf is pure, which means the tree has memorised the training data including its noise. The 99.8% training accuracy is therefore not a measure of skill; it is a measure of memorisation. The test accuracy will almost certainly be much lower — the model is overfitting. Set a max_depth (5 to 10 for most problems), or use cross-validation to pick a depth, before reporting performance.
Decision trees always overfit at any depth; switch to k-NN.
Increase training data tenfold before deploying.