Choosing and combining features

The two halves of feature work

Feature work splits into two distinct activities:

Feature engineering — creating new features from existing ones. We covered this in Course 02 Chapter 5 (“turning reality into columns”) and use it constantly. The four moves: decomposition (date → year, month, dayofweek), combination (loan_amount / monthly_income), aggregation (count of prior loans per borrower), and external context (was 2023 a drought year?).

Feature selection — picking which features to actually feed the model from a longer list of candidates. Some features hurt — they add noise, encourage overfitting, slow training. Removing them honestly often improves performance.

This section focuses on selection, since engineering was already covered. By the end you’ll have three reliable methods for narrowing a long candidate feature list down to the right working set.

Why feature selection matters

You can keep adding features forever. After about 20-30 features in a tabular project, every additional one carries roughly equal risk and reward:

• Adds a little potential signal.
• Adds a lot of potential overfitting risk.
• Adds noise that confuses the model’s optimiser.
• Adds maintenance cost (one more column to clean, encode, monitor).

A model with 15 carefully chosen features routinely beats one with 80 features, where the extra 65 are noise. Pruning is a real skill.

Method 1 — filter selection (cheap, fast)

Score each feature independently against the target, keep the top k. Three common scoring functions:

from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif

# F-test (linear association)
selector = SelectKBest(f_classif, k=10)
X_selected = selector.fit_transform(X_train, y_train)

# Mutual information (captures non-linear relationships too)
selector = SelectKBest(mutual_info_classif, k=10)
X_selected = selector.fit_transform(X_train, y_train)

# Which were kept?
kept_mask = selector.get_support()
kept_features = [name for name, kept in zip(X_train.columns, kept_mask) if kept]
print(kept_features)

For regression, use f_regression or mutual_info_regression.

Filter selection is fast (no model training required), simple, and a good first pass. The downside is that it scores each feature in isolation — it doesn’t know that two correlated features carry the same signal, so it may keep both and reject a better-but-different feature.

Method 2 — wrapper selection (slow but principled)

Train the model with different feature subsets and pick the subset that performs best. The most common variant in scikit-learn is Recursive Feature Elimination (RFE):

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=1000)
rfe = RFE(model, n_features_to_select=10, step=1)
rfe.fit(X_train, y_train)

kept = [name for name, kept in zip(X_train.columns, rfe.support_) if kept]
print(kept)

RFE works by training the model, looking at the weights (or feature importances), removing the weakest features, and repeating until the desired number remain. It accounts for feature interactions in a way filter selection cannot.

The downside is that it can be slow on large datasets — every step requires training the full model. For models that fit in milliseconds (logistic regression on a few thousand rows), this is fine. For models that take an hour to train, RFE is impractical.

Method 3 — embedded selection (models that select for you)

Some models do feature selection as part of their training. The two most useful:

Lasso (L1 regularisation) drives the weights of uninformative features to exactly zero:

from sklearn.linear_model import LassoCV

lasso = LassoCV(cv=5, random_state=42).fit(X_train, y_train)
nonzero_features = [name for name, w in zip(X_train.columns, lasso.coef_) if w != 0]
print(f"Lasso kept {len(nonzero_features)} features:")
print(nonzero_features)

Tree-based feature importances identify which features the model relied on most:

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=200, random_state=42).fit(X_train, y_train)
importances = sorted(zip(X_train.columns, rf.feature_importances_), key=lambda x: x[1], reverse=True)
for name, imp in importances[:15]:
    print(f"  {name:40s} {imp:.4f}")

Both give you a ranked list of features by their contribution to the model. You can then train a smaller model on the top-N features and check whether performance is preserved.

Combining features — when interaction matters

Sometimes the right feature isn’t a single column but a combination of two. A loan’s risk might depend on loan_amount * borrower_age — a young borrower with a large loan is riskier than the sum of the parts would suggest. Linear and even some tree models can’t capture this kind of interaction unless you make it explicit.

Two patterns:

Polynomial features (covered in Section 3.3) — automatically generate all squares and pairwise interactions:

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
X_poly = poly.fit_transform(X)

interaction_only=True skips the squared terms (which you usually don’t want for categorical-style features) and keeps only the products.

Hand-crafted interactions — sometimes you know the interaction that matters from domain knowledge:

df["amount_per_year_of_age"] = df["loan_amount"] / df["borrower_age"]
df["urban_high_amount"] = (df["is_urban"] == 1) & (df["loan_amount"] > 100000)

Hand-crafted interactions are usually better than polynomial ones for tabular data — you bring domain knowledge that the polynomial expansion does not.

A useful sanity check — correlation between features

Highly correlated features are redundant. Two features that move together carry roughly the same signal, but you pay double for the noise. A quick check:

import seaborn as sns
import matplotlib.pyplot as plt

corr = X_train.corr()
sns.heatmap(corr, annot=False, cmap="coolwarm", center=0)
plt.show()

Look for off-diagonal cells with high absolute correlation (above 0.9, say). If two features are 0.95-correlated, consider dropping one. The model’s predictions will barely change, but the model will be simpler and more robust.

A worked example — pruning the loan dataset

Let’s prune the loan-default dataset from Chapter 1. Suppose we have 30 candidate features after engineering, and we want to find the working set:

import pandas as pd
from sklearn.feature_selection import SelectKBest, mutual_info_classif
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import Pipeline

# Step 1: filter to 20 with mutual information
filter_selector = SelectKBest(mutual_info_classif, k=20)
filter_selector.fit(X_train, y_train)
filter_kept = X_train.columns[filter_selector.get_support()]
print(f"After filter ({len(filter_kept)} features):", list(filter_kept))

# Step 2: train a random forest on those 20, rank by importance
X_train_filtered = X_train[filter_kept]
rf = RandomForestClassifier(n_estimators=200, random_state=42).fit(X_train_filtered, y_train)
top_15 = sorted(zip(filter_kept, rf.feature_importances_), key=lambda x: x[1], reverse=True)[:15]
top_15_names = [name for name, _ in top_15]
print(f"Top 15 by importance:", top_15_names)

# Step 3: train final model on top 15, compare accuracy
model_full = Pipeline([("pp", make_preprocessor(filter_kept)), ("clf", LogisticRegression(max_iter=1000))])
model_pruned = Pipeline([("pp", make_preprocessor(top_15_names)), ("clf", LogisticRegression(max_iter=1000))])

for name, m, cols in [("Full (30)", model_full, X_train.columns), ("Pruned (15)", model_pruned, top_15_names)]:
    m.fit(X_train[cols], y_train)
    score = m.score(X_test[cols], y_test)
    print(f"{name:15s} test accuracy: {score:.3f}")

You typically see something like:

Full (30)       test accuracy: 0.831
Pruned (15)     test accuracy: 0.829

The pruned model is 2 percentage points (relatively) cheaper to train, easier to interpret, and just as good. That is the win — not always higher accuracy, but the same accuracy with less complexity. In production, that matters.

What not to do

Two anti-patterns worth naming:

• Don’t pick features based on test-set performance. Tuning features against the test set leaks information into your model selection. Use cross-validation (next section) instead.
• Don’t blindly drop “low-importance” features that have domain meaning. If is_first_time_borrower is low-importance but stakeholders need to see that the model considered it, keep it. Feature selection is a technical lever, not a substitute for judgment.

Check your understanding

Quick check

—

A teammate proudly adds 50 features to their model from various automated sources. Test accuracy goes up by 0.3 percentage points but training time increases tenfold. What's the right read?

1. A0.3pp improvement is worth the cost; deploy.
2. BA tiny accuracy gain at large cost in training time, interpretability, and overfitting risk is usually a bad trade. Most of those 50 features are probably adding noise. Apply feature selection (filter → importance → train) to find the 15-20 that actually drive the signal. The smaller model will be cheaper to train, easier to debug, less likely to overfit on new data, and probably no worse on accuracy — often slightly better.
3. CMore features always helps; add 100 more.
4. DUse a deep neural network instead.

Check answer→

Quick check

—

A teammate uses filter selection (SelectKBest) and keeps the top 10 features by F-score. They notice 5 of the top 10 are very similar variants of the same underlying signal (loan_amount, log_loan_amount, loan_amount_per_1000, loan_amount_squared, loan_amount_zscored). What's wrong?

1. ANothing — all 5 are useful.
2. BFilter selection scores each feature independently, so it can't see that the 5 'amount' variants are essentially the same signal in different forms. The model gets 5 redundant features and only 5 distinct ones, losing diversity. Use wrapper selection (RFE) or tree importances after a first-pass filter, both of which account for feature redundancy. Or do a correlation check and manually drop duplicates.
3. CFilter selection is broken; never use it.
4. DAdd 5 more variants of the loan_amount to get 10 amount-features.

Check answer→

What comes next

Good features in hand. The next section tackles the other lever: hyperparameters — the knobs you set before training (max_depth, n_neighbors, alpha) that shape how the model learns. Setting these well can be the difference between a 75%-accurate and 85%-accurate model on the same data. We meet GridSearchCV and RandomizedSearchCV, the standard tools for tuning them.