Classifying with nearest neighbours · Machine Learning, Hands-On

In Section 3 we used logistic regression — a model with one weight per feature and a clean equation. This section meets a fundamentally different approach: k-nearest neighbours, abbreviated k-NN. It has no equation, no weights, no training in the usual sense. It just memorises the training data, and when asked to predict on a new row, finds the k most similar rows in memory and copies what they did. It feels like cheating. It works surprisingly well. And it teaches a useful intuition about what classification really is.

The idea, in one sentence

To predict the label for a new row, find the k training rows most similar to it and use majority vote.

That is the whole algorithm. With k=5, you find the five training rows closest to the new row, look at their labels, and predict whichever label appears most often among those five.

A small example. We have ten past loans, each with loan_to_income_ratio and a defaulted label. A new loan comes in with ratio = 3.2. The five training loans with ratios closest to 3.2 are:

ratio  defaulted?
3.0     yes
3.1     yes
3.3     no
3.4     yes
3.5     yes

Four “yes” out of five → predict “defaulted = yes” for the new loan. Done. No training, no optimiser, no loss function.

”Similar” means distance

What does “most similar” mean when a row has multiple features? Mathematically, the row is a point in feature-space, and “similar” means small distance. The default is Euclidean distance — the straight-line distance you computed in school geometry:

For two rows A and B with two features each:

distance(A, B) = sqrt( (A1 - B1)^2 + (A2 - B2)^2 )

For three features, add another squared term. For ten features, add nine more. The formula generalises cleanly to any number of features.

This is why scaling matters enormously for k-NN. If loan_amount ranges 0 to 5,000,000 and borrower_age ranges 18 to 70, the loan-amount differences will dominate the distance computation — and age effectively contributes nothing. Always scale before k-NN.

Trying it on the loan data

Almost the same code as Section 3, with one class swapped:

from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder

numeric = ["loan_amount", "monthly_income", "loan_to_income_ratio"]
categorical = ["district"]

model = Pipeline([
    ("preprocess", ColumnTransformer([
        ("num", StandardScaler(), numeric),
        ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
    ])),
    ("classifier", KNeighborsClassifier(n_neighbors=5)),
])

model.fit(X_train, y_train)
predictions = model.predict(X_test)

That is it. The interface is identical to logistic regression. The behaviour is very different.

Choosing k

The n_neighbors parameter (the k in k-NN) is the one knob you have to choose. Some intuition:

Small k (1, 3, 5) — the model is highly sensitive to individual training points. Risky if some training points are mislabeled or outliers — those points have an outsized voice. Tends to overfit.
Large k (50, 100) — the model averages over more neighbours, so predictions are smoother and noise-robust. But too large, and the model averages over points so distant from the query that they’re not really similar.
Odd k for binary classification so majority voting can’t tie.

There is no universal best k. For loan-style datasets in the thousands of rows, k=5 to k=30 usually performs well. Chapter 5 introduces cross-validation — a principled way to compare several k values and pick the best.

What k-NN is good at, and bad at

A short, honest list.

Good at:

Small datasets where there isn’t enough data for fancier models to learn from.
Non-linear decision boundaries — k-NN naturally produces curvy boundaries that follow the data’s shape.
Cases where you have no idea what model to use — k-NN as a first baseline is often shockingly competitive.

Bad at:

Large datasets. To predict one new row, k-NN has to compute distances to every training row. With a million training rows, every prediction is slow. There are optimisations (KD-trees, ball-trees), but the algorithm fundamentally does not scale like a model with fixed parameters.
High-dimensional data. Once you have hundreds of features, distances become weirdly uninformative — every point is roughly equidistant from every other. This is the famous “curse of dimensionality”.
Datasets with lots of categorical features. One-hot encoding helps, but distance in categorical space is fuzzy.

For Course 03’s purposes — datasets in the 1,000-100,000 range with a handful of features — k-NN is a great baseline and sometimes the best model. We’ll keep it in our toolkit.

Reading k-NN predictions

Unlike logistic regression, k-NN has no coefficients to read. You can, however, ask it which neighbours it consulted for any given prediction:

classifier = model.named_steps["classifier"]
preprocess = model.named_steps["preprocess"]

# Get the preprocessed test row
sample_row_processed = preprocess.transform(X_test.iloc[[0]])

# Find its neighbours among the training set
distances, indices = classifier.kneighbors(sample_row_processed, n_neighbors=5)

print("Distances:", distances[0])
print("Neighbour rows in X_train:")
print(X_train.iloc[indices[0]])
print("Their labels:")
print(y_train.iloc[indices[0]].values)

For one test loan, this prints the 5 most similar training loans, their labels, and the distances. Working through a few examples like this is the fastest way to build intuition for what k-NN is doing — and to catch obvious problems (e.g. all the nearest neighbours come from one district, which might mean your scaling or your features need rethinking).

A small, honest comparison

Trained on the loan dataset, with k=15:

Logistic regression test accuracy: 0.83
k-NN (k=15)        test accuracy: 0.81

Close. Sometimes k-NN beats logistic regression. Sometimes logistic regression wins. There is no universal answer about which is better for any given problem — and that is exactly why Chapter 5 teaches you to try several models and compare them honestly.

Check your understanding

Quick check

—

A teammate trains a k-NN classifier (k=5) on raw loan data without scaling. `loan_amount` ranges 1,000 to 5,000,000 NPR; `borrower_age` ranges 18 to 70. They report that the model's predictions seem to ignore borrower age entirely. What is the most likely explanation?

The k-NN model doesn't support age as a feature.
Without scaling, the loan_amount differences (on the order of millions) dominate the Euclidean distance calculation. Borrower age differences (on the order of tens) contribute almost nothing to similarity, so the 'nearest neighbours' are basically 'nearest by loan amount alone'. Fix: scale the features before k-NN, e.g. wrap in a Pipeline with StandardScaler.
k-NN cannot handle numeric features at all.
Use a larger k value — that will make the model attend to all features.

Quick check

—

You set k=1 in a KNeighborsClassifier and find the training accuracy is 100% but the test accuracy is much lower. What's going on?

k=1 is the optimal choice and the model is working correctly.
With k=1, each training point's prediction is itself (it's the closest point to itself in memory), so training accuracy is trivially 100% — it tells you nothing about generalisation. The low test accuracy reveals the truth: the model is overfitting to noise in individual training points. Try a larger k (say 5, 15, 50) and use cross-validation to pick a good value.
The training data is corrupted.
The model needs to be retrained with more iterations.

What comes next

k-NN classifies by similarity. The next section meets a very different approach: decision trees, which classify by asking a series of yes/no questions and following the answers down a branching tree. Trees are interpretable (you can literally print the tree and read the rules), often very accurate, and form the basis of one of the most powerful model families we’ll meet in Chapter 5 — gradient boosting.