Standardising names, places, and units · Working with Data

Of all the cleaning work in a typical project, this is the one that beginners under-estimate the most. A column of “district names” entered by twelve different officers across five years can contain two hundred distinct strings that refer to seventy-seven actual districts. Until you fix that, every group-by, every join, every count is wrong by a slowly-varying factor. The work is patient, partly judgmental, and produces enormous downstream payoff. This section walks the patterns that turn the chaos into clean columns.

The shape of the problem

Imagine a column that has — over five years of data entry by different officers — accumulated these forms of a single district name:

Morang
morang
MORANG
Morang District
Morang district
Morang Dist.
मोरङ
Morang जिल्ला
 Morang
Morang
Mrng

All eleven refer to the same place. To a computer, they are eleven different strings. To a groupby("district"), they become eleven groups, each with a fraction of the total, and your analysis is silently nonsense.

Your job is to map all eleven to a single canonical form. We will pick "Morang" and stick with it.

The four cleaning passes

A repeatable approach that handles most cases:

Pass 1 — whitespace and case. The cheapest pass, catches roughly half the variants:

import pandas as pd

df["district"] = df["district"].astype(str).str.strip().str.lower()

After this, Morang, morang, MORANG, Morang, Morang all become morang. We will re-capitalise to a canonical form (Morang) at the end.

Pass 2 — known suffixes and prefixes. Strip things like “District”, “Dist.”, “जिल्ला”, “Municipality”:

suffixes_to_remove = [" district", " dist.", " जिल्ला", " municipality"]
for s in suffixes_to_remove:
    df["district"] = df["district"].str.replace(s, "", regex=False)
df["district"] = df["district"].str.strip()

After Pass 1 + 2, Morang District, morang dist., Morang जिल्ला all become morang.

Pass 3 — explicit mappings. Handle the irregular cases with a dictionary:

manual_mapping = {
    "मोरङ": "morang",
    "mrng": "morang",
    "morng": "morang",
    # ... add as you discover them ...
}

df["district"] = df["district"].replace(manual_mapping)

This is the place to put one-off fixes — abbreviations, transliterations, common misspellings. Build the dictionary incrementally as you find new cases.

Pass 4 — canonicalise the casing. Map the cleaned lowercase form to the official capitalisation:

canonical = {
    "morang": "Morang",
    "kaski": "Kaski",
    "kathmandu": "Kathmandu",
    # ... all 77 districts ...
}

df["district"] = df["district"].replace(canonical)

Now your column has at most 77 distinct values, each in the canonical form. groupby("district") works as expected.

When to bring fuzzy matching in

For the truly irregular cases — "Krcnj" for Karnali, hand-written typos — exact matching breaks down. Fuzzy matching lets you find the closest known value for any input.

pip install rapidfuzz

from rapidfuzz import process

canonical_list = ["Morang", "Kaski", "Kathmandu", "Karnali", "Sunsari", ...]

def best_match(name: str) -> str:
    match, score, _ = process.extractOne(name, canonical_list)
    return match if score > 80 else name

df["district_guessed"] = df["district"].apply(best_match)

process.extractOne returns the closest match and a score from 0 to 100. The threshold (> 80 here) controls how aggressive you are about accepting matches. Set it low and you catch more typos but also more false positives.

A safer pattern: review the matches before applying them:

review = df[df["district"] != df["district_guessed"]][["district", "district_guessed"]].drop_duplicates()
print(review)

Read the table. Approve or reject each fuzzy match by hand. Then apply them. Two hours of review saves months of misattribution.

Standardising vendor and product names

The same patterns apply beyond places. A column of vendor names:

Khalti
khalti
KHALTI
Khalti Inc.
Khalti Pvt. Ltd.
खल्ती

Same four-pass approach:

Whitespace + lowercase.
Strip suffixes (" inc.", " pvt. ltd.").
Explicit map ("खल्ती" → "khalti").
Canonical form ("khalti" → "Khalti").

The skill transfers across domains — district names, vendor names, product SKUs, school IDs.

Standardising units

The slightly different but equally important problem: a column that mixes units.

loan_amount
50000
5 lakh
500000
5,00,000     (Indian numbering — 5 lakh)
0.5 million
500000.00

Six rows, all the same amount. The fix is parsing the units explicitly:

import re

def parse_amount(s: str) -> float:
    if pd.isna(s):
        return None
    s = str(s).lower().strip()
    s = s.replace(",", "")    # remove all comma separators (Indian + Western)
    # Handle "lakh" and "million" suffixes
    if "lakh" in s:
        return float(re.sub(r"[^\d.]", "", s)) * 100000
    if "million" in s:
        return float(re.sub(r"[^\d.]", "", s)) * 1000000
    if "crore" in s:
        return float(re.sub(r"[^\d.]", "", s)) * 10000000
    try:
        return float(s)
    except ValueError:
        return None

df["loan_amount_npr"] = df["loan_amount"].apply(parse_amount)

Now all six rows become the float 500000.0. The original column is preserved so you can re-check; the new column is what downstream analysis uses.

A short note on what “canonical” means

The word canonical in data work means “the one form we have agreed to use”. You — the builder — choose the canonical form. The choice should be:

Unambiguous. "Morang" is canonical because there is exactly one place that means.
Reusable. Pick the form that other systems also use; if downstream consumers expect "Morang District", use that.
Documented. Write the choice down in your project’s data/README.md. Future-you, and anyone else who reads your data, needs to know.

Once you have written one canonical-mapping pass for one column, you have written the pattern for all of them. Do it carefully the first time and reuse.

Check your understanding

Quick check

—

A district column has 240 unique strings that refer to Nepal's 77 districts. After applying whitespace stripping, lowercasing, and removing the suffix 'District', you still see 110 unique values. What is the most useful next step?

Give up and treat each variant as a separate district.
Inspect the remaining variants by frequency (`df['district'].value_counts()`), build an explicit mapping dictionary for the most common misspellings and transliterations (e.g. 'मोरङ' → 'morang'), and then use fuzzy matching with a high threshold and *manual review* for the long tail of rare typos.
Use fuzzy matching with a threshold of 50 on all values automatically.
Drop all rows with non-matching districts.

Quick check

—

A loan amount column contains the values `'5 lakh'`, `'500000'`, `'5,00,000'`, and `'0.5 million'`. What is the most accurate description of the cleaning work needed?

No cleaning needed — pandas will figure it out.
These are four different amounts and should be kept separate.
All four refer to the same amount (Rs 500,000) but use different unit conventions and separators. Build a small parsing function that strips commas, recognises `lakh`/`crore`/`million` as multipliers, and returns a uniform numeric value. Keep the original column too, for traceability.
Convert to scientific notation and let the model handle it.

What comes next

You can now load, parse, and normalise almost any dataset you will meet in Nepal. Chapter 4 takes the next step: understanding the data you have just cleaned. Looking at distributions, spotting outliers, asking the right questions of a table before the model gets near it. The cleaning gave you a dataset. Understanding tells you what the dataset means — and whether you should trust it.