Documenting a dataset others can trust · Working with Data

The work of cleaning, understanding, and preparing a dataset takes weeks. The work of documenting that dataset takes an hour. And almost no one does it. The result is that every project starts from scratch — even projects in the same organisation, even projects using the same underlying data — because no one wrote down what was true. This section closes Course 02 with the documentation discipline that turns one-off cleaning work into a reusable asset.

Why undocumented data is worthless six months later

A scenario: six months from now, a teammate inherits the loan dataset you cleaned. They open data/clean/loans.csv. They see a column called log_loan_amount. They have three questions immediately:

What is the underlying unit (NPR? lakhs?)?
What does “log” mean here — natural log? Base-10?
Why was this transformation applied?

Without documentation, they re-derive the answer by trial-and-error, or worse, by guessing. Six hours of their time, multiplied by every column on which they have the same question.

With documentation, they read a single line and continue. Six minutes.

This is the actual cost of skipping documentation. It is paid by everyone except the person who chose to skip it.

Two documents that cover almost everything

A complete documentation discipline only needs two artefacts.

Document 1 — the Dataset Card. A single Markdown file (data/clean/DATASET_CARD.md) that explains the dataset as a whole. Half a page, structured.

Document 2 — the Data Dictionary. A table (often a CSV or a section of the Dataset Card) listing every column with type, units, source, and explanation. One row per column.

Together they take an hour to write the first time, fifteen minutes to update after later edits, and answer 90% of questions a future reader will have.

The Dataset Card template

Use this as a starting point. Save as data/clean/DATASET_CARD.md:

# Loans — Cleaned Microfinance Dataset

## Source

Loans data from [Institution Name], received as `loans_raw.xlsx`
on 2026-03-12 from [Contact Person]. Original file contained
12,847 loan applications across 7 provinces, dates 2022-01 to
2026-02. Original file preserved at `data/raw/loans_raw.xlsx`.

## What this dataset is

One row per loan application. Includes borrower characteristics
(pseudonymised), loan terms, and outcome flags. The pseudonymisation
mapping is in `data/secrets/pseudonym_map.csv` (NOT in version control).

## Cleaning applied

1. Whitespace stripped and case normalised on `district` and `vendor`.
2. Manual district-name mapping applied (see `data/mappings/district_canonical.csv`).
3. 142 rows with negative loan amounts flagged in `is_suspicious`;
   not removed.
4. 38 exact duplicates dropped (probably double-entered).
5. BS dates converted to AD using `nepali-datetime`; original BS
   string preserved in `date_bs`.
6. Loan amounts converted to consistent NPR (some rows were in lakhs).

## Known limitations

- Records before 2022-01 were not included in the source export.
- Karnali Province is under-represented (only 4% of rows vs ~6%
  population share). Inferences about Karnali should be treated
  cautiously.
- The `occupation` column has 1,247 unique strings and has not yet
  been mapped to a small canonical set; some analyses will need
  the manual mapping pass first.
- 'defaulted' is the institution's own classification, not an
  external definition; it includes some rows where the loan was
  formally rolled over rather than written off.

## Personal data and permissions

- Identifying columns (name, citizenship number, phone) were
  dropped during cleaning. They survive only in `data/raw/`,
  which is gitignored and access-controlled.
- The institution's data-sharing agreement permits use for
  internal modelling and academic research. Commercial use
  requires renewed permission.

## How to use

Splits are pre-computed in `data/splits/`. The recommended starting
script is `notebooks/02_load_and_split.ipynb`.

## Contact

For questions, contact [Your Name] <you@example.com>.
Last updated: 2026-06-27.

That is the entire format. Eight short sections, each answering questions a future reader will ask.

The Data Dictionary template

A simple table, one row per column. Save as data/clean/DATA_DICTIONARY.csv:

column,type,unit,source,description
borrower_id,string,,derived,Pseudonymised borrower identifier (SHA-256 of citizenship + salt)
loan_amount,integer,NPR,raw,Loan principal in Nepali rupees
log_loan_amount,float,log(NPR+1),derived,Natural log of (loan_amount + 1) — for skew correction
district,string,,raw,Canonical district name (see data/mappings/district_canonical.csv)
province,string,,derived,Province inferred from district lookup
occupation,string,,raw,Free-text occupation as recorded by loan officer
application_date_ad,date,AD,derived,Converted from date_bs; primary date column for analysis
application_date_bs,string,BS,raw,Original Bikram Sambat string from source
defaulted,boolean,,raw,True if loan classified as defaulted by source institution
is_suspicious,boolean,,derived,True if loan_amount < 0 (likely data error; not auto-removed)

Now the reader knows, for any column, what type it is, what unit, where it came from, and what it means. Five minutes to type up; thousands of minutes saved across the project’s life.

The README that ties it together

A data/README.md at the top of the data folder, even shorter, that orients a new reader:

# Data — Loans project

Cleaned: data/clean/loans.csv  (see DATASET_CARD.md)
Raw:     data/raw/loans_raw.xlsx  (gitignored — request from team lead)
Splits:  data/splits/  (joblib files; load with joblib.load)

Quick start: open notebooks/02_load_and_split.ipynb.

Five lines. Tells a new person what is where and where to start. A whole career’s worth of project onboarding can be made friendlier by the presence of a file like this.

Versioning and updates

Datasets change. The cleaning script gets improved. A new export from the source institution arrives with one more month of data. A bug is found and fixed.

Three small habits handle this:

1. Version the dataset’s name or path. data/clean/loans_v2.csv instead of overwriting loans.csv. When a model was trained on v1, you can still reproduce it.

2. Date the DATASET_CARD. Every change to the dataset gets a line in a “changelog” section at the bottom.

## Changelog

- 2026-06-27 (v2): added 2026-03 to 2026-05 data; recomputed
  district mappings to fix Sunsari/Sunsari Pi confusion.
- 2026-03-12 (v1): initial cleaning of source data.

3. Snapshot the data dictionary alongside the data. When loans_v2.csv is created, DATA_DICTIONARY_v2.csv lives alongside it. Future-you can compare the two and see what changed.

A note on Dataset Cards as an industry standard

What we have just written is often called a datasheet or a dataset card — formats proposed in academic papers (Timnit Gebru’s “Datasheets for Datasets”, 2018) and adopted by Hugging Face, Google, and other major data publishers. The exact format is less important than the practice. Use the template above as a starting point; adapt it for your team’s needs.

For datasets you publish externally — to Hugging Face, to a public repo, in a paper — using the established formats makes your work discoverable and reusable by people you will never meet. The same minimum information; just laid out the way researchers and other builders expect.

Closing Course 02

This is the end of Course 02. You started the course as someone who knew how to call APIs and parse JSON. You leave as someone who knows where data comes from, how it goes wrong, how to clean it, how to understand it, how to prepare it for a model, and how to do all of this in a way that respects the people behind the data and the people who will work with the data after you.

The work is not chronologically first in a project — you usually start by talking to stakeholders and choosing a problem — but it is the work that decides whether everything downstream is sound. Course 03 takes the prepared data and finally puts it in front of models. The models will look impressive. By now you know that they are only as impressive as the data they were given. You did the hard part.

Course 03 begins

In Course 03 — Machine Learning, Hands-On — you will finally train your first real models. Classifiers that decide whether an SMS is spam. Regressors that predict rent in Kathmandu. The models will be small at first, then bigger. Each will use the prepared, cleaned, split, documented data that this course was about. Each will succeed or fail based on the quality of the work you have just learned to do.

When you are ready, turn the page.