Chapter 04 · Section III · 14 min read
Regression
Predicting a number — house prices in Bhainsepati, paddy yields in Chitwan, electricity demand on a winter night.
Classification predicts a category. Regression predicts a number. How much will this house in Bhainsepati sell for? How many tons of paddy will Chitwan produce this season? What will Kathmandu’s electricity demand be at 7pm tonight? All three are regression problems. All three are far more common in industry than the deep-learning systems that get headlines.
The simplest form is linear regression: fit a straight line through your data. It is the workhorse of applied AI in industry — far more common than transformers, and far more often actually deployed.
The setup
Suppose we have data on 200 recent house sales in Bhainsepati. For each house, we know:
- The size in square feet.
- The number of bedrooms.
- The distance to the nearest chowk with a school.
- The sale price.
We want to predict the sale price of a new house from its size, bedrooms, and distance.
This is supervised learning — every training example has an input (size, bedrooms, distance) and a target (price). It is regression because the target is a continuous number (a price), not a category.
Linear regression in one paragraph
Linear regression assumes the price is approximately a weighted sum of the input features, plus a constant:
predicted price ≈ w₁ × size + w₂ × bedrooms + w₃ × distance + b
The numbers w₁, w₂, w₃, b are the weights. The algorithm’s job is to find values for these weights such that, on the training data, the predictions are as close as possible to the actual prices.
That is the entire model. Multiply each feature by its weight, add them up, add a constant, and you have your prediction.
How the weights are found
We want predictions to match actual prices on the training data. So we define the error of a model as the sum of squared differences between predicted and actual price across all training examples:
error = Σ (predicted_i − actual_i)²
The weights that minimise this error are the “best fit” linear weights. For linear regression, there is a beautiful piece of mathematics — the normal equation — that gives you the optimal weights in one shot, no iteration needed.
In practice, with very large datasets, we use gradient descent: start with random weights, see how the error changes when you nudge each weight a little, and step in the direction that reduces error. Repeat until no more reduction is possible. Gradient descent is the same algorithm that trains every neural network you have ever heard of, just at a much larger scale.
A worked example
Imagine our 200 Bhainsepati houses gave us these weights:
- w₁ = 4,500 (per square foot)
- w₂ = 200,000 (per bedroom)
- w₃ = −150,000 (per kilometre to nearest school chowk)
- b = 500,000
The negative weight on distance makes sense: further from a school, the price drops.
Now a new listing comes in: 1,200 sq ft, 3 bedrooms, 0.8 km from the nearest school chowk. The predicted price is:
4,500 × 1,200 + 200,000 × 3 + (−150,000) × 0.8 + 500,000 = 5,400,000 + 600,000 − 120,000 + 500,000 = Rs. 6,380,000
That’s the entire prediction. Multiply features by weights. Add them. Done.
What linear regression cannot do
The “linear” assumption is the limitation. The relationship between features and price is assumed to be a sum of single-feature effects. Real life is rarely so kind.
- Two bedrooms in 800 sq ft is cramped, but two bedrooms in 2,000 sq ft is spacious. The value of an extra bedroom depends on the square footage. Linear regression cannot represent this without help.
- A house 100m from a busy main road might be more valuable (convenience) than one 500m away (more isolated); but a house 2 metres from the road is less valuable (noise). The relationship is not monotonic in distance.
- Prices change suddenly across neighbourhood borders. A house just inside a more prestigious neighbourhood is worth significantly more, with no smooth gradient.
Workarounds exist. You can hand-craft interaction features (size × bedrooms), apply non-linear transforms (square root of distance), or use richer models entirely (decision trees, neural networks). These all improve fit at the cost of simplicity and interpretability.
Overfitting: the universal regression trap
If you add enough features, you can make linear regression fit any training data exactly. This sounds good — perfect predictions! — but the model then also fits the noise in your training data, and predicts terribly on new examples it has not seen.
This is called overfitting, and it shows up everywhere in machine learning, not just regression. The standard defence is to hold out a portion of your data, train on the rest, and check performance on the held-out part. If the model is good on training data but bad on held-out data, it is overfitting. Make the model simpler — fewer features, smaller weights — and try again.
This is also why “the model gets 100% accuracy on training data” is not a good sign. It usually means the model is memorising rather than learning.
Beyond linear regression
Once you understand linear regression, the rest of supervised learning is mostly the same shape with bigger machinery:
- Logistic regression uses the same weighted-sum idea but transforms the output to a probability — it is regression repurposed for classification.
- Decision trees split the data on one feature at a time and predict a number (or category) at each leaf. They handle non-linearities natively.
- Neural networks stack many weighted sums with non-linear functions in between. The training is still “find weights that reduce error” — just on many more weights.
All three are taught in the next two chapters. None of them will surprise you, because you have already seen the core idea.
Check your understanding
Quick check
—A municipal water utility wants to predict next year's monthly water demand from population, average temperature, and the number of new construction permits. What kind of problem is this?
What comes next
We close the machine-learning chapter here. Chapter 5 moves to neural networks — the technique behind ChatGPT, image generation, and modern Devanagari OCR. Spoiler: the core idea is still “find weights that reduce error.” Just more weights.