Exploratory analysis: asking questions of a table · Working with Data

Before any model, before any feature engineering, before any clever idea, you sit with the data. Not for hours. For thirty minutes. You ask it questions — small, specific questions whose answers you can guess in advance — and you check whether the data agrees. This is Exploratory Data Analysis, or EDA, and skipping it is the single most common cause of “the model is producing strange results” three weeks later. This section is the EDA habit, condensed.

The thirty-minute habit

The structure of every EDA session looks roughly the same:

Size and shape — how many rows, how many columns?
Types — is each column the type you expected?
Summary — what do head(), tail(), describe() show?
Missing — what is missing, and where?
Categorical balance — for each categorical column, what are the values and how often do they each appear?
Three guess-and-check questions — pick three things you think you know about the data, and verify.

Half an hour is enough to do this for a typical small dataset. The payoff is enormous: bugs surface, assumptions get corrected, and you walk into the modelling stage already knowing what the data does and does not contain.

The first four questions

Working with a hypothetical school-attendance dataset:

import pandas as pd

df = pd.read_csv("data/schools.csv")

print(df.shape)        # (rows, columns)
print(df.dtypes)       # types of each column
print(df.head())       # first 5 rows
print(df.describe())   # summary stats of numeric columns

A typical session:

df.shape returns (12847, 8). Plausible — there are roughly 12,000-30,000 schools in Nepal depending on what you count.
df.dtypes shows students is int64, district is object, date is object. The date being object is a clue: I have not parsed it yet.
df.head() shows the first five schools — readable, no immediate weirdness.
df.describe() shows students ranges from 0 to 2,847 with mean 156. The 0 is suspicious — a school with 0 students? The 2,847 is also suspicious — far above the mean. Both need investigating.

You have already learned something from four lines.

Missing — and where

Always look at missing values with the structure they appear in:

df.isna().sum()
df.isna().mean() * 100   # percentage missing per column

If students is 0.4% missing — fine. If district is 30% missing — something is structurally wrong.

A useful pattern: look at missingness by another column:

df.groupby("province")["students"].apply(lambda s: s.isna().mean() * 100)

This tells you whether the missingness is evenly distributed (random missingness, often safe to impute) or clumped in one province (structural — that province’s data was collected differently and you need to understand why before doing anything).

Categorical balance

For every categorical column, run:

print(df["province"].value_counts())
print(df["province"].value_counts(normalize=True) * 100)

The first shows raw counts. The second shows percentages. Together they reveal:

Whether the categories are balanced (each province ~14% — Nepal has 7 provinces).
Whether the categories you expected actually appear (all 7 provinces present? or only 6?).
Whether there is a “long tail” of weird values you missed in cleaning (a province column that has eight entries instead of seven — the eighth being "province 1" lowercase, surviving Chapter 3 because you forgot one normalisation step).

Run value_counts() on every categorical column. Read the output. Note anything strange.

Three guess-and-check questions

The most useful part of EDA is asking questions whose answers you should be able to predict. If your prediction is right, you build confidence. If it is wrong, you have found something important.

For our schools dataset:

Question 1: Does the average school size differ by province?

Guess: Bagmati (urban) probably has larger average schools than Karnali (rural). Maybe 200 vs 100.

df.groupby("province")["students"].mean()

If the answer matches your guess — good. If Karnali shows up as having larger schools than Bagmati, something is off. Maybe the dataset only includes private boarding schools in Karnali. Maybe a unit error.

Question 2: How are schools distributed by district within Bagmati?

Guess: Kathmandu has the most schools by far. Maybe 20-30% of Bagmati’s schools.

bagmati = df[df["province"] == "Bagmati"]
print(bagmati["district"].value_counts().head(5))

Check the top five and the percentage they take.

Question 3: Is the date column dense, or does it have gaps?

Guess: A “school survey” dataset is probably collected once a year; we should see one date per school, repeated across many schools.

print(df["date"].value_counts().head(10))

If we see 365 unique dates with even distribution across the year, the data was not collected on a single survey day — it was collected continuously, which changes how to interpret it.

This is the rhythm. Three small predictions, three small checks, thirty minutes of work.

The `pandas-profiling` shortcut

For first-pass EDA on small datasets, the library ydata-profiling produces an HTML report with most of the above automatically:

pip install ydata-profiling

from ydata_profiling import ProfileReport
profile = ProfileReport(df, title="Schools dataset")
profile.to_file("eda/schools_report.html")

The report includes column distributions, missingness, correlations, and basic warnings. Useful as a starting point — never as a replacement for asking your own questions. The library does not know what you are trying to learn; you do.

Writing it down

A small habit that compounds: keep an eda/ folder in your project. After each EDA session, save:

The notebook (eda/schools_first_pass.ipynb) — the actual cells you ran.
A short text summary (eda/schools_first_pass.md) — three to five bullet points of what you learned, three to five surprises, and three to five questions for the data owner.

The text summary is what you re-read three weeks later when you cannot remember whether you ever checked the date column. The notebook is what you re-run when you want to re-check. Together they make the EDA work cumulative across the project.

Check your understanding

Quick check

—

You loaded a new dataset of 50,000 microfinance loans and ran `df.describe()`. The `loan_amount` column shows min = -3,500 and max = 45,000,000. What is the right interpretation?

These are both valid extreme values; move on to modelling.
Both numbers are suspicious and worth investigating before any modelling. A negative loan amount is almost certainly an error (a refund recorded with the wrong sign, or a flipped data field). A Rs 4.5 crore loan is plausible for a large business loan but unusual for microfinance — confirm with the data owner whether such loans should be in this dataset. EDA is the place these surprises surface; ignoring them is exactly how silent bugs reach production.
Drop the affected rows automatically.
Rescale the column to make the numbers smaller.