Distributions, outliers, and what they hide

Looking at distributions

The fastest way to see a column’s distribution is to plot a histogram. A histogram divides the range of values into bins and counts how many values fall in each bin.

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv("data/clean/loans.csv")
df["amount"].hist(bins=50)
plt.xlabel("Loan amount (NPR)")
plt.ylabel("Number of loans")
plt.title("Loan amount distribution")
plt.show()

The shape that appears tells you a lot in one glance.

The shapes you will recognise

Symmetric (bell-shaped). A “normal” distribution — most values cluster near the mean, with roughly equal numbers above and below, tapering symmetrically. Heights, body temperatures, measurement errors often look this way. Mean and median are roughly equal.

Right-skewed. Most values are small, with a long tail of large values. Income, transaction amounts, file sizes, loan amounts almost always look this way in real datasets. The mean is higher than the median because the long tail pulls the mean up.

Left-skewed. The mirror image — most values are large, with a long tail of small values. Less common; you sometimes see it in time-to-event data (e.g. customer survival before churn).

Bimodal. Two peaks. Often a sign that you have two populations mixed in one column — small personal loans and large business loans in the same amount column, or test scores from two schools with very different averages combined.

Uniform. Values spread roughly evenly across the range. Sometimes natural (e.g. lottery numbers), but in real data often a sign that the column is junk or that you are looking at IDs by accident.

The shape of the distribution suggests the right next step. Symmetric → mean is meaningful, standard deviation is meaningful. Skewed → median is more meaningful, percentiles are more honest than mean. Bimodal → you probably need to split the data into the two underlying populations before any modelling.

A small Nepali example

A microfinance loan dataset. We plot the amount distribution and see:

Most loans cluster between Rs 10,000 and Rs 50,000.
A second smaller cluster sits between Rs 200,000 and Rs 500,000.
A long thin tail extends up to Rs 4,500,000.

Three signals in one chart:

1. The two clusters are a bimodal distribution — likely “personal microfinance” and “small-business” loans, mixed in one column. Worth splitting in the next stage.
2. The Rs 4.5M tail — a few values up to 4.5 million — is probably real but unusual. Either real high-value business loans, or rows that belong to a different dataset entirely. Investigate.
3. The mass concentration in the first cluster is right-skewed within itself. The mean of the whole column will be misleading; the median tells a much truer story of “what a typical loan looks like”.

You read all three of these in fifteen seconds from one histogram.

Outliers — the four kinds

An outlier is a value far from the rest of the distribution. The phrase covers four very different situations, and the right action is different for each.

1. A data-entry error. Someone typed 1245 when they meant 124.5. The value is impossible in context. Action: correct or remove with a flag.

2. A unit error. A loan amount of 5 when others are 500000 — almost certainly entered in lakhs but stored in rupees. The value is real but the unit is wrong. Action: detect and convert (the patterns from Chapter 3 Section 3).

3. A rare but legitimate extreme. A real Rs 4.5 crore loan in a dataset of mostly small microfinance loans. The value is real and correct. Action: decide whether to include in the model based on what the model is for. If the model is meant to predict microfinance defaults, the Rs 4.5 crore loan is the wrong population — exclude. If the model is meant to predict all bank loans, include but possibly scale.

4. A signal of a structural problem. All outliers cluster in one district, or one month. Something is happening there that does not happen elsewhere. The value is real but reveals a story. Action: investigate before modelling.

Telling the four apart is judgmental work. Tools help you find them; only thinking decides what to do.

Detecting outliers cheaply

Two standard rules of thumb. Neither is universal — both are starting points.

The IQR rule. The interquartile range (IQR) is the spread between the 25th and 75th percentile. Anything more than 1.5×IQR below Q1 or above Q3 is flagged as an outlier.

q1 = df["amount"].quantile(0.25)
q3 = df["amount"].quantile(0.75)
iqr = q3 - q1

mask = (df["amount"] < q1 - 1.5 * iqr) | (df["amount"] > q3 + 1.5 * iqr)
outliers = df[mask]
print(f"{len(outliers)} potential outliers")
print(outliers[["amount", "vendor", "district"]].head(10))

The IQR rule is robust to skewed distributions and is the right first pass for most loan, transaction, and revenue columns.

The z-score rule. A value’s z-score is how many standard deviations it sits from the mean. Anything with |z| > 3 is flagged.

mean = df["amount"].mean()
std = df["amount"].std()
df["z"] = (df["amount"] - mean) / std
outliers = df[df["z"].abs() > 3]

Z-scores are reasonable for symmetric distributions but get fooled by skewed ones. Prefer IQR for skewed columns; either works for symmetric ones.

A scatter plot, briefly, for outliers

A scatter plot of one column against another — e.g. loan amount against repayment days — shows outliers in two dimensions at once. A row that is unusual in both columns (very high amount AND very few repayment days) stands out, and is more interesting than a row that is just unusual in one.

df.plot.scatter(x="amount", y="repayment_days", alpha=0.3)
plt.show()

The alpha=0.3 (transparency) is essential for dense scatter plots — without it, the dots overlap and you cannot see where the mass actually is.

What to do, in the end

A useful decision flowchart:

1. Plot the distribution. Note the shape.
2. If bimodal, investigate — likely two populations mixed.
3. Compute IQR-based outlier flags. Look at the flagged rows.
4. For each cluster of outliers, ask: data error, unit error, real extreme, or structural signal?
5. Document the decision for each cluster. Save flagged rows separately rather than deleting them.

You do this once per column per dataset. The first time it takes an hour. The tenth time it takes ten minutes. It is the discipline that keeps real datasets honest.

Check your understanding

Quick check

—

You plot a histogram of customer ages in a Nepali e-commerce dataset and see two distinct peaks — one around age 22 and another around age 38, with a clear dip between. What is the most likely interpretation?

1. AThe dataset is broken; rerun the histogram.
2. BThe bimodal distribution likely indicates two distinct customer populations mixed in one column — perhaps young first-time online shoppers and a separate cohort of established middle-aged buyers. Investigate by splitting and analysing each peak separately; a single model lumping both will likely underperform on both.
3. CIgnore — it is just statistical noise.
4. DDrop all customers in the dip.

Check answer→

Quick check

—

A loan dataset has a right-skewed loan_amount column with a long tail of large values. The IQR rule flags about 8% of rows as outliers — mostly the large-value tail. What is the most thoughtful action?

1. ADelete the flagged 8% — they are clearly outliers.
2. BLook at the flagged rows. Are they data errors (delete or fix), unit errors (convert), real extremes (decide whether they belong in this analysis), or a structural signal (investigate)? In a right-skewed financial dataset, many 'outliers' by the IQR rule are real and legitimate values — the rule's threshold is a suggestion to look, not a verdict to delete.
3. CApply a log transform and call it done.
4. DDrop the column entirely.

Check answer→

What comes next

You can read a single column’s distribution. The next section adds the second dimension: relationships between columns. How does loan amount relate to repayment days? How does student count relate to district? Scatter plots, correlation, and the careful eye that tells real signal from coincidence — that is the last piece of Chapter 4.