Linear regression in code · Machine Learning, Hands-On

A classifier picks a category. A regressor predicts a number. The shape — fit, predict, evaluate — is the same; the loss function and the metrics differ. This section introduces linear regression, the simplest and most-used regressor. It assumes the target is a weighted sum of the features plus an offset. It is fast, interpretable, and works surprisingly well as a baseline. Even when you eventually need something fancier, you fit a linear regression first to know what “easy” looks like.

The mental model

Linear regression assumes:

predicted_value = w1 * feature1 + w2 * feature2 + ... + wn * featuren + b

The weights w1, w2, ..., wn and the offset b are the knobs. Training finds the values for them that minimise the loss — typically mean squared error (MSE), the average of (actual - predicted)² across the training set.

That is the whole picture. For Kathmandu rent prediction with two features:

predicted_rent = w_size * size_sqft + w_location * location_score + b

A model with weights w_size = 30, w_location = 12000, b = -5000 would predict rent as:

predicted_rent = 30 * size_sqft + 12000 * location_score + 5000

For a 600 sqft flat with location_score 3:

predicted_rent = 30 * 600 + 12000 * 3 - 5000 = 18000 + 36000 - 5000 = 49000 NPR/month

Plausible. Whether the model is good depends on whether those weights actually fit the data — which is what .fit() figures out.

Training a linear regression

scikit-learn class: LinearRegression. Same interface as everything else:

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import numpy as np

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42,
)

model = LinearRegression()
model.fit(X_train, y_train)

predictions = model.predict(X_test)

print(f"MAE:  {mean_absolute_error(y_test, predictions):.0f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_test, predictions)):.0f}")
print(f"R²:   {r2_score(y_test, predictions):.3f}")

Three metrics, each useful for a different question:

MAE (Mean Absolute Error) — average size of prediction errors, in the same units as the target. For rent prediction in NPR, “MAE = 4500” means typical prediction errors are around Rs 4,500/month. Intuitive.
RMSE (Root Mean Squared Error) — like MAE but penalises large errors more heavily. Also in the target’s units. Always ≥ MAE.
R² (R-squared, the “coefficient of determination”) — proportion of variance in the target explained by the model. Ranges typically from 0 (model adds nothing) to 1 (perfect fit). 0.6 to 0.8 is common in noisy real-world problems.

For an interpretable rent model, MAE in NPR is the metric to report to non-technical stakeholders. R² is what ML practitioners check first.

Reading the coefficients

for name, coef in zip(X.columns, model.coef_):
    print(f"  {name:25s} {coef:+10.2f}")
print(f"  {'intercept':25s} {model.intercept_:+10.2f}")

Output (illustrative):

  size_sqft                    +28.45
  bedrooms                  +5234.10
  bathrooms                 +3812.55
  location_score           +11342.78
  has_balcony               +2104.30
  intercept                 -3420.00

Read this aloud. Each extra square foot of size adds about Rs 28 to predicted rent. Each extra bedroom adds Rs 5,234. Each unit of location-score adds Rs 11,343. A balcony is worth Rs 2,104. The intercept is what you’d predict for a hypothetical flat with all features at zero — a meaningless number on its own, but a calibration the model needs.

A useful sanity check: do the signs and magnitudes match your intuition? If “bedrooms” had a negative weight, something is wrong — either with the data, with the features, or with how they correlate (more bedrooms might correlate with farther from city centre, and that correlation might dominate). Investigate.

When linear regression is enough — and when it isn’t

A short, honest list.

Linear regression is enough when:

The relationship between features and target is roughly linear (straight-line, not curved). For many things — rent vs size, income vs experience, distance vs travel time — it really is roughly linear.
You want an interpretable model with a small number of features.
You need a fast baseline you can train in seconds on a laptop.
Your dataset is small (under a few thousand rows) — flexible models will overfit.

Linear regression struggles when:

The relationship is genuinely non-linear (rent vs distance from city centre is U-shaped, not linear).
Features interact (effect of size depends on whether you’re in Thamel or Lalitpur).
The target is bounded (linear regression can predict negative rent, which is impossible).

Chapter 3 Section 3 will show what to reach for in the non-linear cases. For now, linear regression is your starting point for every regression problem.

A note on regularisation

The plain LinearRegression can overfit when you have many features relative to your number of rows. Two common variants add a small penalty for large weights, which improves generalisation:

Ridge — penalises the sum of squared weights. Shrinks all weights smoothly.
Lasso — penalises the sum of absolute weights. Drives weights of less-useful features to exactly zero — useful for feature selection.

Same interface:

from sklearn.linear_model import Ridge, Lasso

ridge = Ridge(alpha=1.0)
ridge.fit(X_train, y_train)

lasso = Lasso(alpha=0.1)
lasso.fit(X_train, y_train)

The alpha parameter controls the strength of the penalty. Larger alpha → more shrinkage → simpler model. We’ll cover hyperparameter tuning (how to pick alpha) properly in Chapter 5.

For most projects, plain LinearRegression is the right starting point; reach for Ridge when you have many correlated features and Lasso when you want feature selection.

A tiny worked example

Generate some fake data to play with:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(42)
n = 200

size_sqft = rng.uniform(200, 1500, size=n)
bedrooms = rng.integers(1, 5, size=n)
location_score = rng.uniform(1, 5, size=n)

# True relationship + noise
rent = 25 * size_sqft + 4000 * bedrooms + 10000 * location_score + rng.normal(0, 5000, size=n)

X = pd.DataFrame({"size_sqft": size_sqft, "bedrooms": bedrooms, "location_score": location_score})
y = pd.Series(rent, name="rent")

model = LinearRegression()
model.fit(X, y)

print("Recovered weights:")
for name, coef in zip(X.columns, model.coef_):
    print(f"  {name:20s} {coef:+8.2f}")
print(f"  {'intercept':20s} {model.intercept_:+8.2f}")

You should see weights very close to 25, 4000, 10000 — the model recovers the true relationship from noisy data. This is the magic of regression in a controlled setting. On real data, of course, the “true relationship” is messier and the weights have to balance many things; but the procedure is the same.

Check your understanding

Quick check

—

A teammate trains a linear regression for Kathmandu rent prediction. The reported MAE is 4,200 NPR. They want to explain this to a real-estate client. Which of the following is the best explanation?

The model is 4,200% accurate.
On average, the model's predicted rent for a flat is about Rs 4,200 different from the actual rent. So if it predicts Rs 50,000 for a flat, the real answer is typically within Rs 45,800 to Rs 54,200. This is a property of the test-set evaluation, not a guarantee — but it's what 'typical error' looks like for this model.
The model fails 4,200 times out of every measurement.
MAE has no business meaning; only R² matters.