A dataset's life: collection to model · Working with Data

Datasets do not appear. They are produced — by people, by systems, by sensors — and they pass through a recognisable sequence of stages on their way from origin to model. Most data problems you will meet in this course can be traced to a specific stage. Recognising which stage is the source of trouble cuts diagnosis time from days to minutes. This section walks the six stages, with one Nepali microfinance loan-default record as our running example.

Stage 1 — Generation

A real thing happens in the world. A loan officer in Birgunj sits with a borrower and approves a Rs 80,000 loan. The borrower has a tea stall. The repayment schedule is monthly over 18 months. This is the data: a single approval, a single contract, a single repayment plan.

Notice what is true at this stage:

The data is anchored to a specific real-world event.
It exists, but it is not yet “data” in a Python sense — it is paperwork, conversation, a handshake.
Anything written down at this stage will reflect what the officer chose to record, and what the form had space for. Anything not written down here is permanently lost.

Stage 2 — Capture

Someone writes it down. The officer fills out a paper form, or types into a tablet, or dictates to a clerk. The Rs 80,000 becomes a number in a cell labelled “amount”. The borrower’s district becomes a string in a cell labelled “district”.

Capture is where most of the dirty-data problems are born:

The officer types 80000 in one row and 80,000 in another and 80k in a third.
The district is sometimes Parsa and sometimes Birgunj (a city in Parsa district).
The tea-stall business is recorded as tea stall, Tea Stall, chiya pasal, and चिया पसल across different rows.
The repayment schedule, which is structured information, gets crammed into a single free-text comment field.

Almost every problem this course teaches you to fix starts here.

Stage 3 — Storage

The captured data goes somewhere. A spreadsheet. A SQL database. A handwritten ledger that gets digitised once a month. A photo of a paper form sitting in a Google Drive folder.

Storage is where the data acquires its shape — what columns exist, what types they have, whether dates are strings or proper dates, whether amounts are integers or strings with commas in them. Storage decisions made by someone who left the project two years ago will haunt you forever.

What can go wrong here:

The database column for amount is VARCHAR(20) instead of INTEGER, so amounts get stored as text and arithmetic stops working.
Encoding is set to Latin-1 instead of UTF-8, and Devanagari names get corrupted into ?????? or पà¥à¤à¤¤à¥à¤ª.
A “deleted” flag is added later in the project but old rows have NULL in it, which means the same thing as False but is treated differently by some queries.

You will write code to read this storage; the code will be only as good as your understanding of how the storage works.

Stage 4 — Extraction

Someone exports the data into a file you can work with. This is usually the moment the data lands on your desk. A CSV download from the database, a .xlsx someone sent over Signal, a JSON dump from an API.

Extraction has its own set of failure modes:

Truncation. The export tool limited to 65,000 rows because that is what older Excel could handle, even though there are 80,000 rows.
Format conversion losses. Dates get reformatted into the locale of the machine that did the export — 4/5/2026 could be April 5 (US) or May 4 (UK).
Silent encoding choices. The CSV is technically UTF-8 but was saved by Windows Excel, which inserts a BOM (Byte Order Mark) that some Python tools choke on.
Sampling without telling you. Someone gives you “the dataset” but actually gave you “a sample of the dataset” because the full one was too big to email.

A discipline that saves real time: always ask, in writing, what was done to the data between its native form and the file you received. Even an honest answer like “I just clicked Export” tells you something useful (it was a default export, not a custom query).

Stage 5 — Preparation

You — the builder — clean the data. Fix the dirty things from stages 2 and 3. Handle the export quirks from stage 4. Normalise text, parse dates, handle missing values, validate ranges, dedupe. Engineer features for the model.

This is the work of Chapters 3 and 5 of this course. It is the largest single time-cost in almost every AI project. Senior data engineers, asked to estimate how long a new project will take, mentally allocate 60-80% of the budget to this stage and have been doing so for decades. The number does not go down.

Stage 6 — Training and inference

The clean, prepared data finally meets the model. The model trains on it, learns patterns, and is later asked to predict on new data. Months of work in stages 1-5 either succeed or fail at this stage — but the success or failure of stage 6 is entirely determined by what happened in stages 1-5.

If you forget unit-checking in stage 5, your model misjudges defaults. If you missed the encoding bug in stage 3, your model never learned anything about Devanagari names. If you didn’t ask about the sampling in stage 4, your “amazing” model is amazing only on the subset that was sampled and may collapse in production.

The model is the cheap part. The data work is the expensive, slow, important part. This sequence is the reason.

Why this stage map matters

When something is wrong in stage 6 — your model isn’t working, accuracy is bad, predictions are weird — the instinct is to fix the model. Bigger network, more compute, different architecture. Almost always, the actual problem is upstream: dirty capture (stage 2), broken storage (stage 3), botched export (stage 4), or skipped preparation (stage 5). The fix is to walk back through the stages until you find where the truth got lost.

The senior people you will work with carry this map in their heads. They diagnose data problems by mentally walking back through the stages until they find the gap. Get this map into your head now and you will diagnose faster than peers who only think about the model.

Check your understanding

Quick check

—

Your loan-default model is producing very low-confidence predictions on Karnali borrowers. Walking back through the data stages, which question would most likely surface the root cause?

What architecture is the model using? Could a transformer help?
Were Karnali borrowers represented in the *generation* and *capture* stages — i.e. did the field officers in Karnali use the same forms, and were their loan records actually included in the dataset we extracted?
What GPU is the model running on?
Have we increased the training epochs enough?