Encoding categories and scaling numbers

Why encoding matters

Almost every interesting column in a real dataset is categorical — district, occupation, vendor, school type. A model cannot multiply “Morang” by anything. You have to convert these to numbers, and how you convert determines what the model can learn.

Three encoding strategies cover almost every case.

Strategy 1: one-hot encoding

The default for small-cardinality categorical columns. Each unique value becomes its own 0/1 column:

import pandas as pd

df = pd.DataFrame({"district": ["Morang", "Kaski", "Morang", "Bhaktapur"]})
encoded = pd.get_dummies(df, columns=["district"], drop_first=False)
print(encoded)

Output:

   district_Bhaktapur  district_Kaski  district_Morang
0               False           False             True
1               False            True            False
2               False           False             True
3                True           False            False

Three districts become three new boolean columns. The model can now learn “rows where district_Morang is True default 8% of the time”.

When to use: small categorical columns (typically up to ~30 unique values). For Nepal’s 7 provinces or 14 zones (the legacy administrative division) — one-hot is fine. For all 77 districts — one-hot is borderline. For thousands of vendors — one-hot is wrong.

The drop_first=True argument drops one column (the model can infer it from the others), reducing dimensionality slightly and avoiding “perfect multicollinearity” for some linear models. Useful but not always necessary.

Strategy 2: ordinal encoding

When the categories have a natural order — education level (primary, secondary, tertiary), Likert scale (strongly disagree → strongly agree) — assign integers that respect the order:

edu_order = {"primary": 1, "secondary": 2, "tertiary": 3}
df["education"] = df["education"].map(edu_order)

The model now sees 1, 2, 3 and can learn that 3 is “more” than 2. This is correct when the order is real and wrong when it is not. Encoding district as ordinal integers (Morang=1, Kaski=2, ...) is wrong — there is no ordering, and the model will mistakenly learn “Morang is less than Kaski” as if that meant something.

Strategy 3: target encoding

For high-cardinality categoricals (thousands of vendors, thousands of borrowers), one-hot is impractical and ordinal is meaningless. Target encoding replaces each category with the average value of the target variable for that category — a single column instead of thousands.

For loan default prediction:

default_rate_by_vendor = df.groupby("vendor")["defaulted"].mean()
df["vendor_default_rate"] = df["vendor"].map(default_rate_by_vendor)

Each vendor becomes the historical default rate for that vendor — a number between 0 and 1. The model can now use it.

For Course 02, the takeaway is: one-hot for small categoricals, ordinal for ordered ones, target encoding (carefully) for high-cardinality.

Why scaling matters

A loan amount (range 0 to 5,000,000) and a borrower age (range 18 to 70) are on wildly different scales. Some models — anything based on distances (k-NN, k-means, SVM, neural networks) — treat the loan amount as a thousand times more important than the age, simply because the numbers are larger. Tree-based models (decision trees, random forests, gradient boosting) are mostly immune.

Two scaling strategies cover almost everything.

Strategy A: standardisation

Subtract the mean, divide by the standard deviation. The result has mean 0 and standard deviation 1:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
df[["loan_amount", "age"]] = scaler.fit_transform(df[["loan_amount", "age"]])

After this, both columns are centred around 0, with about 68% of values within ±1 and almost all within ±3. The model now treats them as comparable.

Use when: the column is roughly symmetric (normal-ish). Most numeric columns work fine with standardisation.

Strategy B: min-max scaling

Squish into the [0, 1] range:

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
df[["loan_amount", "age"]] = scaler.fit_transform(df[["loan_amount", "age"]])

After this, the minimum value becomes 0 and the maximum becomes 1, with everything else proportionally in between.

Use when: you want bounded values (some neural networks expect this), or when the distribution is heavily skewed and you do not want a few extremes to dominate. Less common than standardisation for general use.

A third option for skewed columns: log transform

For right-skewed columns (loan amount, income, transaction value), a log transform is often the right move before scaling. It compresses the long tail and makes the distribution more symmetric:

import numpy as np
df["log_loan_amount"] = np.log1p(df["loan_amount"])

np.log1p(x) computes log(1 + x) — the +1 lets you handle zeros without errors (log(0) is undefined). After log-transforming a skewed column, standardisation works beautifully on the result.

The most-stepped-on landmine in this whole section

A profound, easy-to-make mistake: fit the scaler on the entire dataset, then split into train and test. This is wrong because the scaler’s mean and standard deviation incorporate information from the test set, leaking knowledge that should be unavailable at training time.

The right pattern:

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)   # fit only on train
X_test = scaler.transform(X_test)         # transform test using train's parameters

fit_transform on train (computes mean and std, then applies them). transform only on test (applies the same parameters, does not recompute).

The same rule applies to encoders, imputers, and any other preprocessing that depends on the data: fit on train only, transform train and test separately.

Putting it together with Pipeline

Scikit-learn’s Pipeline makes the right pattern automatic:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier

numeric = ["loan_amount", "age"]
categorical = ["district", "occupation"]

preprocessor = ColumnTransformer([
    ("num", StandardScaler(), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])

model = Pipeline([
    ("preprocess", preprocessor),
    ("classifier", RandomForestClassifier()),
])

model.fit(X_train, y_train)            # fits everything on train only
predictions = model.predict(X_test)    # transforms test correctly

The Pipeline ensures the scaler and encoder are fit on training data and applied to test data automatically — no leakage, no mistakes. Use this pattern as the default starting structure for any modelling project, even before you know which model you will use.

Check your understanding

Quick check

—

A new builder one-hot encodes a column of 5,000 unique vendor IDs and is surprised the model trains for hours and overfits dramatically. What is the right diagnosis?

1. AThe model needs a bigger GPU.
2. BOne-hot encoding 5,000 unique values produces 5,000 new columns — most of which appear in only a handful of rows. The model has too many sparse features and not enough signal per feature, so it overfits. For high-cardinality categoricals like this, use target encoding (carefully, with out-of-fold splits) or group rare vendors into an 'other' category before encoding.
3. CLower the learning rate.
4. DAdd more training data.

Check answer→

Quick check

—

A teammate writes: scaler = StandardScaler() X = scaler.fit_transform(X) X_train, X_test = train_test_split(X) model.fit(X_train, ...) model.score(X_test, ...) What is the leakage problem here, and what is the fix?

1. AThere is no leakage; this is standard.
2. BThe scaler was fit on the *entire* dataset before splitting, which means the test-set values influenced the scaler's mean and standard deviation. The test 'evaluation' is therefore optimistic. The fix is to split first, then fit the scaler on train only and apply it to test using `transform` — or wrap the whole thing in a sklearn Pipeline that handles this automatically.
3. CThe scaler should be applied after the model, not before.
4. DStandardScaler should be replaced by MinMaxScaler.

Check answer→

What comes next

You have features, encoded and scaled. The last and most important preparation step is how you split the data into train, validation, and test sets. Splitting is mechanical, but doing it badly is the most common cause of “the model looked great in validation and failed in production”. The next section is the careful version of this single most-skipped step.