Where bias enters a dataset · Working with Data

Bias in datasets is rarely deliberate. It is almost always the cumulative result of small, defensible-at-the-time choices: which population to survey, which form fields to include, which records to digitise, which outliers to remove. Each choice introduces a slight tilt. By the time the data reaches a model, the tilts have compounded into systematic distortion — and the model, learning faithfully from the data it was given, becomes a fast machine for amplifying those distortions at scale. This section is a map of where bias enters and how to recognise it before it leaves your laptop.

Bias has stages

Recall the dataset’s life from Chapter 1, Section 3 — generation, capture, storage, extraction, preparation, training. Bias can enter at every single one of those stages. Knowing which stage is the source of a problem tells you where to fix it.

At generation: The events themselves are unevenly distributed. A loan officer in Lalitpur grants more small-business loans than a loan officer in Doti — not because the model is wrong, but because Lalitpur has more small businesses. Any dataset of “loans we have made” will reflect this.

At capture: The form doesn’t have the right options. A field for “occupation” with three dropdown choices forces a tea-stall owner and a vegetable seller and a tailor into the same "small business" bucket. The model sees one category where there were three.

At storage: Certain records are kept, others are not. A loan officer’s notes about the borrower’s family situation might be discarded after six months by the database. Information that was collected is gone before the data reaches you.

At extraction: A query filter excludes some records. “Show me all loans from the last two years” silently excludes longer-tenor loans whose origination is older than two years even if they are still active.

At preparation: Your cleaning decisions remove or modify rows. Dropping outliers, normalising names, imputing missing values — each of these is a small bias choice. Most are defensible. Document them.

At training and inference: Even with clean data, a model’s choices about which patterns to learn and how to weigh them are themselves a kind of choice. We will not cover this here; Course 03 addresses it.

The key insight: by the time you, the builder, receive a dataset, it has already passed through four or five stages of choices. Your work is downstream of all of them.

The five common biases, named

A working vocabulary for bias that you can carry into any project.

Selection bias

The data was collected from a non-representative subset of the real population. A microfinance dataset of successful loans (because failed applications were not recorded) tells you nothing about who would have been rejected. A NEPSE-trader dataset of registered traders tells you nothing about informal trading.

How to spot it: ask “who or what is missing from this dataset?”. The data does not announce its absences; you have to imagine them.

Survivorship bias

The records you have are only those that survived to be recorded. A dataset of “successful exporters” hides every business that tried to export and failed. A dataset of “graduates” hides every student who dropped out. The model learns “patterns of survival” but is interpreted as “patterns of success”.

How to spot it: ask “what would have to be true for an entity to be in this dataset?”. The conditions of inclusion shape what the model sees.

Confirmation bias

The data was collected, or labelled, by someone with a hypothesis to support. A teacher assessing “engagement” in students may unconsciously rate students like themselves more highly. A loan officer assessing “trustworthiness” may unconsciously favour borrowers who remind them of past successful borrowers.

How to spot it: ask “who made the judgments, and what were they trying to find?”. Any human-labelled dataset carries the labellers’ assumptions.

Sampling bias

The way data was sampled differs across groups. A school survey conducted only in Bagmati gives you Bagmati data, full stop — but it may be reported as “Nepal data” if the framing is loose.

How to spot it: ask “what was the sampling procedure, and are all groups represented in proportion to their actual size?”. The answer is rarely “yes”.

Measurement bias

The instrument or procedure for measuring something is uneven across groups. A facial recognition system trained mostly on light-skinned faces is technically measuring “faces” but is actually measuring “ability to detect faces with the kind of features over-represented in training”. The measurement itself is biased.

How to spot it: ask “is the measurement procedure the same for every group I will deploy to?”. If accuracy varies systematically across groups, the measurement is biased.

A short Nepali walkthrough

A microfinance institution in Lalitpur wants to deploy a default-prediction model across Karnali Province. Walk through each bias:

Selection bias: The Lalitpur dataset includes only Lalitpur-area borrowers. Karnali has different economies (more agriculture, less retail), different default patterns, and a sparser banking history. The training data does not represent the deployment population.
Survivorship bias: The dataset includes only loans that were approved. Anyone the Lalitpur officer rejected — who might have been a perfectly good Karnali borrower — is not in the data.
Confirmation bias: “Trustworthiness” assessments by Lalitpur officers reflect Lalitpur cultural norms — what a “trustworthy borrower” looks and sounds like. Karnali norms differ.
Sampling bias: Even within Lalitpur, urban borrowers were oversampled because the officer’s branch is in town. Rural Lalitpur borrowers — who are demographically closer to Karnali borrowers — are underrepresented.
Measurement bias: Default was measured by an event (missed payments) that depends on the officer’s follow-up frequency. Karnali deployment will have less follow-up, so the same underlying default behaviour will be measured differently.

Each of these is a real, named, addressable problem. Acknowledging them is the first step. Some can be fixed (collect Karnali data before deployment). Some can be mitigated (use group-aware evaluation). Some are intrinsic and require honesty in reporting (limitations sections in your write-up).

What you can actually do

A few concrete habits that catch bias early:

Document the data’s origin. In your data/README.md, state where every column came from, who collected it, when, and what was excluded. Future-you, and anyone else reading the data, needs this.

Evaluate the model per-group. A model with 87% overall accuracy might be 92% on Bagmati and 71% on Karnali. Always compute accuracy by subgroup before reporting an overall number.

for province in y_test.index.get_level_values("province").unique():
    mask = y_test.index.get_level_values("province") == province
    print(province, accuracy_score(y_test[mask], predictions[mask]))

Look at confusion matrices, not just accuracy. Accuracy hides asymmetries. A model that predicts “no default” for everyone will be 95% accurate if defaults are 5% of the data — and completely useless. Per-class precision, recall, and F1 reveal this.

Talk to a domain expert. Someone who actually works with the population the data represents will spot biases you cannot. Half an hour with a Karnali loan officer reviewing the model’s predictions on Karnali borrowers will catch more bias than a week of statistical analysis.

Check your understanding

Quick check

—

A team trains a school-dropout prediction model on data from Bagmati schools and reports 86% accuracy. They are about to deploy it nationally. What is the most important question to ask before deploying?

What architecture is the model using?
What is the accuracy on each province separately, especially Karnali, Sudurpaschim, and Madhesh — provinces with very different schooling realities that may have been under-represented in Bagmati training data? An overall 86% accuracy can hide an 89% on Bagmati and 58% on Karnali, which means the model is unsafe for national deployment without further work.
How fast does the model run?
What is the cost to retrain the model?