Fixing types, dates, and Nepali text encoding

Diagnosing types

The first move on any messy dataset is asking pandas what type it thinks each column is:

import pandas as pd

df = pd.read_csv("data/messy_ministry_export.csv")
print(df.dtypes)

Output might look like:

district     object
school_id    object
students     object
date         object
dtype: object

Everything is object — pandas’s name for “string or mixed”. That is your first signal something is wrong: a column called students that should be integers is sitting as strings.

A second look:

df["students"].head()

0      245
1     1,250
2      178
3       NA
4      820
Name: students, dtype: object

The comma in 1,250 is what blocked pandas from auto-detecting the column as numeric. And NA is a string, not a missing value.

Converting types deliberately

The pattern to convert a string column to numeric:

df["students"] = (
    df["students"]
    .str.replace(",", "", regex=False)
    .replace("NA", pd.NA)
    .astype("Int64")
)

Three steps in one chain:

• .str.replace(",", "", regex=False) removes commas. The regex=False is faster and avoids treating the comma as a regex pattern.
• .replace("NA", pd.NA) turns the string "NA" into pandas’s real missing-value marker.
• .astype("Int64") (capital-I) converts to a nullable integer type — unlike plain int, this can hold missing values without converting to float.

After this:

print(df["students"].head())
print(df["students"].sum())

You get real integers, with pd.NA for the missing ones, and arithmetic works.

A more permissive conversion that does not fail on bad values:

df["students"] = pd.to_numeric(df["students"], errors="coerce")

errors="coerce" turns anything pandas cannot parse into NaN. Useful for first-pass conversions when you do not yet know what bad values lurk in the column.

Parsing dates — including BS

Dates are the second universal headache. A typical Nepali ministry CSV might have:

date
2026-06-23
2026/06/24
23-06-2026
2080-03-09

The fourth row is in Bikram Sambat. The others are AD, with three different formats.

The first step is always: ask the data owner which calendar each column uses. If you cannot ask, look for context — surrounding fiscal-year mentions, BS-specific month names (Shrawan, Bhadra), or a sample where the year is clearly above 2070 (almost certainly BS) or in the 1990s-2030s (almost certainly AD, though 2080 is plausible in both).

Once you know the calendar, parsing:

df["date_ad"] = pd.to_datetime(
    df["date"],
    errors="coerce",     # rows that fail to parse become NaT (Not a Time)
    format="mixed",      # let pandas try multiple formats per row
    dayfirst=False,      # treat ambiguous DD-MM-YYYY vs MM-DD-YYYY as MM-DD by default
)

pd.to_datetime handles the common AD formats. For mixed dayfirst (Nepali context is often DD-MM-YYYY), set dayfirst=True if you are confident.

For BS dates, you need a converter library. The cleanest in 2026 is nepali_datetime:

pip install nepali-datetime

from nepali_datetime import date as nepali_date

bs = nepali_date(2080, 3, 9)
ad = bs.to_datetime_date()
print(ad)   # datetime.date(2023, 6, 23)

To convert a whole column, write a small function and apply it:

from nepali_datetime import date as nepali_date

def bs_string_to_ad(s):
    try:
        y, m, d = map(int, s.split("-"))
        return nepali_date(y, m, d).to_datetime_date()
    except Exception:
        return None

df["date_ad"] = df["date_bs"].apply(bs_string_to_ad)

The encoding problem, finally

The third universal headache: your CSV shows ?????? or पà¥à¤à¤¤à¥à¤ª where Devanagari should be.

This is an encoding mismatch. The file was written in one character encoding and is being read in another. The fix is to read with the right encoding.

# Common offender 1: Excel for Windows saves CSV as 'cp1252' or 'utf-8-sig'
df = pd.read_csv("data/file.csv", encoding="utf-8-sig")

# Common offender 2: legacy systems use Latin-1
df = pd.read_csv("data/file.csv", encoding="latin-1")

# Modern default and the right answer for 95% of cases
df = pd.read_csv("data/file.csv", encoding="utf-8")

How to figure out which to use:

1. Try encoding="utf-8" first. If it raises a UnicodeDecodeError, the file is not UTF-8.
2. Try encoding="utf-8-sig". This is “UTF-8 with a BOM” — what Excel for Windows often produces. Most files that fail UTF-8 succeed here.
3. Try encoding="latin-1". This never errors but may produce wrong characters; only use if 1 and 2 fail.
4. As a last resort, install chardet and let it guess:

pip install chardet

import chardet

with open("data/file.csv", "rb") as f:
    raw = f.read(10000)
guess = chardet.detect(raw)
print(guess)   # {'encoding': 'utf-8-sig', 'confidence': 0.99, 'language': ''}

Even when the file is read in the right encoding, you might find that earlier processing already corrupted the text. Once Devanagari has been written as ?????? to a file, the original characters are gone. There is no fix from that point. The only path is to re-export from the source.

Writing files in the right encoding

The reverse problem: when you write a CSV that downstream tools (Excel for Windows, say) will open, write it with the encoding those tools expect.

df.to_csv("data/out.csv", index=False, encoding="utf-8-sig")

utf-8-sig writes a UTF-8 file with a leading byte order mark, which is what makes Devanagari display correctly in Excel for Windows. Plain utf-8 works for most other tools.

A complete cleaning pass — types, dates, encoding

A single script that does all three:

import pandas as pd
from nepali_datetime import date as nepali_date


def bs_string_to_ad(s):
    try:
        y, m, d = map(int, str(s).split("-"))
        return nepali_date(y, m, d).to_datetime_date()
    except Exception:
        return None


# 1. Read in the right encoding
df = pd.read_csv("data/messy_ministry_export.csv", encoding="utf-8-sig")

# 2. Fix numeric column
df["students"] = (
    df["students"]
    .astype(str)
    .str.replace(",", "", regex=False)
    .replace({"NA": pd.NA, "": pd.NA})
    .astype("Int64")
)

# 3. Parse BS dates to AD
df["date_ad"] = df["date_bs"].apply(bs_string_to_ad)
df["date_ad"] = pd.to_datetime(df["date_ad"])

# 4. Save in a downstream-friendly encoding
df.to_csv("data/clean/ministry.csv", index=False, encoding="utf-8-sig")

Sixteen lines of cleaning code. They will, with small adaptations, work on most Nepali government and NGO datasets you ever meet.

Check your understanding

Quick check

—

You open a CSV in Python with `pd.read_csv('data/file.csv')` and the Devanagari column shows as `पà¥à¤à¤¤à¥à¤ª`. You try `encoding='utf-8'` and get a `UnicodeDecodeError`. What is the most likely fix?

1. AThe Devanagari is permanently corrupted and cannot be recovered.
2. BTry `encoding='utf-8-sig'` first — Excel for Windows often saves CSVs with a UTF-8 BOM and this is the single most common cause. If that fails, try `encoding='latin-1'`. If both fail, use `chardet` to detect the encoding from the file bytes.
3. CSwitch to a different file format like JSON.
4. DPandas does not support Devanagari; you need a special library.

Check answer→

Quick check

—

A CSV column called `students` contains values like `'1,250'`, `'NA'`, `'845'`. You want a numeric column you can sum. Which approach is the most robust?

1. Adf['students'] = df['students'].astype(int)
2. Bdf['students'] = pd.to_numeric(df['students'])
3. Cdf['students'] = df['students'].str.replace(',', '', regex=False).replace('NA', pd.NA).astype('Int64') — strip commas, replace NA strings with pd.NA, and convert to nullable Int64.
4. Ddf['students'] = df['students'] * 1

Check answer→

What comes next

You can now read any file, parse any column, and translate any date. The last universal cleaning task is standardising values — turning Khalti, khalti, KHALTI, Khalti , and खल्ती all into the same "Khalti". The shape is similar but the judgment is subtle, especially for Nepali names and places. That is the next section.