Chapter 02 · Section III · 15 min read
What "fair" even means
"Fair" is not one thing — it is at least three mutually incompatible mathematical definitions, and choosing among them is a political judgment dressed up as a calculation.
The most disorienting fact about algorithmic fairness — and the one that almost no business meeting in Nepal has yet absorbed — is that there is no single thing called “fair.” There are several incompatible things, each of which someone reasonable could call fair, and a model can satisfy any one of them only by violating the others. This is not a temporary state of the science waiting on a better definition. It is a theorem: mathematicians have proven that, except in the degenerate case where the groups you are comparing are identical, you cannot satisfy more than one of the main fairness definitions at the same time. The moment your bank, your hospital, your university accepts that it is going to use a model on a real population, somebody has to choose which fairness it wants. And every choice silently rules out the other fairnesses, and the people they would have protected.
Three definitions, briefly
The literature on algorithmic fairness has dozens of formal definitions, but three of them carry most of the weight. Each is intuitive on its own. The trouble starts when you ask for two of them at once.
1. Equal acceptance rates (demographic parity). The model accepts the same fraction of applicants from each group. If 30% of male applicants are approved for a loan, 30% of female applicants are approved too. This is what most people mean when they say “the model treats men and women the same.”
2. Equal error rates (equalised odds). The model makes mistakes at the same rate for each group. If 5% of qualified male applicants are wrongly rejected, 5% of qualified female applicants are wrongly rejected. And if 8% of unqualified male applicants are wrongly approved, 8% of unqualified female applicants are wrongly approved.
3. Equal calibration. When the model says “70% chance of repayment,” it means 70% for every group. A 0.7 score from a Khas-Arya applicant and a 0.7 score from a Dalit applicant should both correspond, in the real world, to seven-in-ten actually repaying. The number means what it says, regardless of who it is said about.
Each of these definitions is something you might call “fair.” Each captures a real moral intuition. And the result that broke the field — the impossibility theorem by Chouldechova, Kleinberg and others — is that you cannot have all three at once, unless the base rates of the outcome are already identical across groups. In any real population where they aren’t, the model has to violate at least one of these definitions to satisfy any other.
A worked example: a Nepali bank credit model
Imagine a bank in Kathmandu building a credit-default model that will be used across Nepal. Suppose the historical default rate is meaningfully different between two groups — say, applicants from Bagmati and applicants from Karnali — not because of anything inherent, but because of the structural differences in income volatility, harvest cycles, and prior access to formal credit. The model has to predict who will default. The bank wants to be “fair.”
If the bank chooses equal acceptance rates, it will approve the same fraction of Karnali applicants as Bagmati applicants. To do this, given different base rates of default, the model has to use different thresholds for the two groups — a Karnali applicant with a 0.55 predicted-repayment score might be approved, while a Bagmati applicant with the same score is rejected. The bank will face two complaints. Bagmati applicants will say they are being held to a stricter standard. The bank’s risk officers will say the approved Karnali book is going to default more, on average, than the approved Bagmati book.
If the bank chooses equal error rates, it accepts a different deal. The model is calibrated so that the rate of wrongly-rejected qualified borrowers, and wrongly-approved unqualified borrowers, is the same for both groups. The bank may end up approving slightly different fractions of each group, which will look like demographic disparity from the outside. But within each group, the mistakes are evenly distributed.
If the bank chooses equal calibration, it commits to making the scores mean the same thing across groups — a 0.7 is a 0.7 whether the applicant is from Bajura or Bhaktapur. This sounds neutral, and it is the easiest definition to defend in a regulator’s office. But it permits substantial differences in acceptance rates and in error rates across groups. If the base rates of default are different, the same calibrated model will end up rejecting more applicants from one group than the other — and within each group, the error patterns may diverge.
There is no fourth option that escapes the tradeoff. Every metric the bank could pick will, by mathematical necessity, treat at least one group worse than another by at least one definition. The question is not whether the model will be unfair by some metric; it is which unfairness the bank is willing to defend in writing.
Why this matters in Nepal specifically
The Nepali institutions most likely to deploy these models — banks, telcos, insurance providers, the government’s social-protection registries, the new private health-insurance schemes, the universities that will inevitably adopt automated screening — have no public convention yet for which fairness definition they hold themselves to. NRB has begun talking about model governance for the banking sector, but the actual choice of fairness metric is not in the regulation. The result is that each institution, today, is making this choice in private — usually without naming it, usually by accident, usually by defaulting to whatever the vendor’s dashboard reports.
The vendor’s dashboard, in turn, almost always reports calibration. Calibration is the easiest metric to compute, the easiest to display, and the easiest to call “neutral” because it looks like the model is just being honest about probabilities. What it hides is everything else — the acceptance gap, the error gap, the way the model’s “honest” probabilities still produce a different deal for a Karnali applicant than for a Bagmati one.
The healthier convention, when an institution can be persuaded to adopt it, is to publish all three numbers for any high-stakes model — the acceptance rate by group, the error rates by group, and the calibration by group — and to name, in writing, which one the institution is optimising for and why. Naming the tradeoff is not the same as resolving it. But it makes the politics visible, which is the only condition under which the public, the regulator, and the affected groups can argue about whether the institution chose well.
A short note on “the model is just neutral”
If you take one thing from this section, take this: anyone who tells you a model is “just neutral” or “just learning the data” is either confused or selling you something. Every deployed model embeds a fairness choice, whether or not the people who built it know which one. The choice is in the threshold. It is in the loss function. It is in which group the data over-represents. It is in the metric the dashboard happened to print. Neutrality is a posture, not a property. The honest version of this work names the choice and defends it. The dishonest version pretends there was no choice to make.
Check your understanding
Quick check
—Which of the following statements about algorithmic fairness is true?
What comes next
Bias and fairness are only one face of how AI meets people. The next chapter turns to the other face — privacy, data, and sovereignty: who gets to collect what about whom, where that data sits, whose laws govern it, and what it means for a country like Nepal that its citizens’ data is overwhelmingly stored, processed, and modelled on infrastructure it does not own.