Where bias comes from — data, designers, deployment · AI, Ethics, and Society

When a model behaves unfairly, the first instinct of almost every team is to look at the model. They print the weights, they tweak a threshold, they argue about which loss function is more “neutral.” The actual sources of unfairness are usually behind them, not in front of them — in the data that was collected long before the model existed, in the choices the designers made about what was worth predicting, and in the rooms and phones and languages the model eventually meets. Bias is not one thing the model has caught, like a virus. It is three different things, leaking in through three different doors, and a team that has only locked one of them is the team most confident the house is secure.

1. The data: models learn the world they are shown, not the world

A model has no opinion about Nepal. It has no opinion about anything. What it has is a very large pile of examples, and a habit of reproducing the patterns inside that pile. If the pile leans, the model leans. If the pile is silent on something, the model is silent — or worse, confidently wrong — on that thing.

Consider a loan-default model trained on a decade of records from a bank with most of its branches inside Kathmandu Valley. The historical data will reflect the historical book: customers who were approved in Lalitpur and Bhaktapur, customers who were denied in Saptari and Bajura, repayment patterns shaped by who got the loan in the first place. The model, asked to “predict default risk,” will learn that being from outside the Valley correlates with worse outcomes — not because rural borrowers are riskier, but because the bank historically extended smaller loans on worse terms to them and then watched the results. The model is not making a prediction. It is laundering the bank’s old preferences into a number that looks objective.

This is the most under-appreciated source of bias, because it pre-exists the model. You can hire the most careful engineer in Bagmati, give them the cleanest pipeline, and they will still build something that reproduces the past faithfully. Faithful reproduction of an unjust past is not neutrality. It is the most efficient way to preserve injustice yet invented.

2. The designers: somebody chose what counts

Every model carries a thousand small human decisions in its bones, and most of them were made before any code was written. Somebody decided what problem was worth solving. Somebody decided what success would look like. Somebody decided which features to collect, which to drop, which proxy to accept when the real signal was unavailable. None of these decisions are “in” the model in a way you can inspect — they are upstream of it, baked into the dataset and the loss function and the deployment plan.

Take a hiring screen for a Kathmandu software firm. A designer decides that “good engineer” will be operationalised as “stayed at the company more than two years and got promoted at least once.” That sounds reasonable. But women in Nepali tech leave earlier — not because they are worse engineers, but because they marry, move with husbands, take maternity leave at companies that don’t quite welcome them back. The designer’s choice of label has quietly written a gender penalty into the model before any data was looked at. The model is doing what it was told. The unfairness lives in what it was told.

The same applies to feature choices. A model that uses “graduated from a college on this list” as a feature has, in effect, decided that the graduates of Tribhuvan-affiliated rural campuses are a different category from the graduates of Kathmandu University. A model that uses “years of continuous employment” has decided that a woman who took three years off to raise children is a worse candidate than a man who did not. The designer did not write be unfair to women. They wrote a feature that correlates with being a woman, and the model did the rest.

3. The deployment: a model is the room it ends up in

A model that performs beautifully in the lab can be deeply unfair the moment it is shipped. The reason is that fairness is not a property of the model alone — it is a property of the model meeting a population. Change the population, change the fairness.

A customer-support chatbot trained on English support tickets and deployed inside Khalti is, in pure technical terms, a fine model. Asked a question in English, it answers well. The problem is that a meaningful fraction of Khalti’s user base is more comfortable writing in romanised Nepali, in Devanagari, or in a code-switched mix. Those users get worse answers — slower, more confused, more likely to escalate — than the English-comfortable user. The model didn’t change. The unfairness emerged from where it was put.

Face-recognition models tell the same story even more starkly. A model trained predominantly on lighter-skinned faces — as most large public face datasets are — will simply be less accurate on darker skin. Deploy that model at a border checkpoint in Birgunj and you have built a system that detains Madheshi citizens more often than Khas-Arya citizens, not by intent, but by physics of training data meeting a different population. The lab numbers were fine. The deployment numbers are an injustice.

”We dropped the variable” is not a defence

The most common, most reassuring, most wrong sentence in this whole conversation is: we removed the gender column from the training data, so the model cannot discriminate by gender. The same sentence is offered with caste, with region, with religion. It is wrong every time, and it is wrong for the same reason: information about a protected attribute almost never lives only in the column labelled with that attribute. It lives in proxies — features that correlate with the attribute, sometimes very strongly, even when the attribute itself is invisible.

Drop gender, and the model learns from “list of extracurriculars,” because boys and girls in Nepali schools are pushed into different ones. Drop caste, and the model learns from surname, from village postcode, from the name of the school attended. Drop region, and the model learns from the dialect markers in the applicant’s written Nepali, or from the IP range the application came from. The protected attribute is not a single column in a spreadsheet — it is woven through almost every other column, and a model that is rewarded for predicting outcomes will follow the weave wherever it goes.

The only honest response is the harder one: measure the model’s behaviour by group, after training, on data you trust. If it treats groups differently, you have a problem — and dropping the column is not a solution, it is a way to make the problem invisible while it continues.

Check your understanding

Quick check

—

A Nepali bank trains a loan-default model on ten years of its own approval and repayment records. The model recommends rejecting a higher share of applicants from Karnali than from Bagmati. Which source of bias is most clearly at work here?

Deployment bias — the model is being used in a context it was not trained for.
Training-data bias — the historical record reflects the bank’s past lending patterns, and the model is reproducing them.
No bias — the model is making a neutral statistical prediction.
User bias — Karnali applicants are filling the forms in incorrectly.

Quick check

—

A team building a hiring tool says: “We removed gender, caste, and region from our features, so the model cannot be biased on those attributes.” Why is this defence usually wrong?

Because removing columns also reduces model accuracy.
Because information about protected attributes leaks through proxy variables — school, surname, postcode, extracurriculars, writing style — that the model can still learn from.
Because the model can secretly re-create the columns from random noise.
Because Nepal’s laws require those columns to be included.

What comes next

Knowing the three doors bias walks through is the easy part. The harder part is recognising the specific shapes bias takes in this country — surname-as-caste-proxy, Bagmati-centric hiring data, language-model gaps on Maithili and Tamang, fraud models that flag Madhesh transactions more aggressively. The next section makes those shapes concrete.