Visualising relationships before modelling · Working with Data

A model that “predicts loan defaults from borrower characteristics” is, mechanically, looking for relationships between columns. If repayment days correlates with default, the model finds that. If district correlates with default, the model finds that. But not every pattern the model finds is real — some are coincidences, some are caused by a third variable, and some are artefacts of the data collection. The way to tell them apart starts with looking. This section is about seeing relationships clearly enough to know what your future model is going to find.

Two numeric columns: the scatter plot

The simplest relationship view is a scatter plot — one column on the x-axis, another on the y-axis, one dot per row.

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv("data/clean/loans.csv")

df.plot.scatter(x="amount", y="repayment_days", alpha=0.3)
plt.xlabel("Loan amount (NPR)")
plt.ylabel("Repayment period (days)")
plt.title("Amount vs repayment period")
plt.show()

The alpha=0.3 makes overlapping points partially transparent — essential for any dense scatter plot. Without it, a thousand dots in one place look the same as one dot in one place.

What to look for in the result:

A diagonal line up to the right → strong positive relationship. Larger loans get longer repayment periods. Probably real, probably what you expected.

A diagonal line down to the right → negative relationship. As one increases, the other decreases. Sometimes real, sometimes a clue that one variable is a proxy for another.

A cloud with no clear pattern → little or no relationship. The two columns vary independently. Sometimes informative (you expected a relationship and there is not one) and sometimes just noise.

Distinct clusters or stripes → categorical structure showing through. Often the data is being divided by some categorical column you have not added to the picture.

A bent or curved pattern → non-linear relationship. Linear models will under-fit this; tree-based models will catch it. Worth knowing before modelling.

Adding a third dimension with colour

A two-axis scatter plot can encode a third column through colour:

districts = df["district"].unique()
colours = plt.cm.tab10(range(len(districts)))

for i, d in enumerate(districts[:5]):  # top 5 districts only
    subset = df[df["district"] == d]
    plt.scatter(
        subset["amount"], subset["repayment_days"],
        c=[colours[i]], label=d, alpha=0.5,
    )

plt.xlabel("Amount")
plt.ylabel("Repayment days")
plt.legend()
plt.show()

Now you can see whether the relationship between amount and repayment days is the same in every district — or whether some districts have systematically different patterns. The latter is often the more interesting finding.

Correlation — the number behind the picture

A correlation coefficient is a single number that summarises how strongly two numeric columns vary together:

df["amount"].corr(df["repayment_days"])

The result is a number between -1 and +1:

+1.0 — perfect positive relationship; one column moves up exactly when the other does.
+0.5 — moderate positive.
0.0 — no linear relationship (still might have a non-linear one).
-0.5 — moderate negative.
-1.0 — perfect negative.

For all numeric columns at once:

df.select_dtypes(include="number").corr()

This produces a correlation matrix — useful but easy to misread.

What correlation does not tell you

Three universal cautions:

Correlation is not causation. Loan amount and repayment days are correlated. Does loan amount cause longer repayment? Probably yes (longer-term contracts are written for larger amounts). But correlation alone cannot prove it. Two columns may both depend on a third — for example, the loan officer’s seniority might cause both larger loans and longer terms, and the direct relationship between amount and term might be much weaker.

Correlation only catches linear relationships. A column that follows a parabola (y = x²) has near-zero linear correlation with the underlying x, even though the relationship is perfect. Always plot before trusting a correlation number.

Correlation is sensitive to outliers. A single Rs 4.5 crore loan in a microfinance dataset can swing the correlation coefficient by 0.3 or more. Look at the scatter plot before reporting any correlation number.

Categorical vs numeric: the box plot

When you want to compare a numeric column across a categorical one — say, loan amount by district — a scatter plot is the wrong tool. Use a box plot:

df.boxplot(column="amount", by="province", figsize=(12, 6))
plt.title("Loan amount by province")
plt.xticks(rotation=45)
plt.show()

A box plot for each category shows the median (line in the box), the interquartile range (the box), the typical range (the whiskers), and the outliers (dots). Side-by-side, they let you compare distributions across categories at a glance.

What to look for:

Boxes at very different heights → the categorical variable has a strong relationship with the numeric one.
Boxes at similar heights but very different widths → the spread differs by category, even if the medians are similar.
One box that is much taller (long whiskers, many dots) than the others → that category has unusually variable values; investigate.

A short note on confounders

A confounder is a hidden third variable that explains an apparent relationship between two others. They are the most common reason a beginner’s “we found a strong predictor” turns out to be wrong.

Classic example: in a Nepal-wide schools dataset, you find that higher altitude correlates with lower student counts. Causal? Almost certainly not. The confounder is population density — high-altitude districts (Karnali, Mustang) have low population density, low population density → small schools. Altitude is a coincidence; population density is the cause.

The discipline: before trusting any correlation as causal, ask “what third variable might explain both?” If you can think of one, control for it in your analysis (e.g. by computing the correlation within each population-density bucket).

Sitting with the chart

A practice that distinguishes careful builders from rushed ones: when you produce a chart, look at it for a full minute before moving on. Note what you see. Note what you do not see. Note what you expected to see but is missing. Most of the value of a chart is in the looking, not in the producing.

This sounds obvious. It is rare. Most beginners produce a chart, glance at it for two seconds, decide “yes that looks fine”, and move on. They miss the bimodal hint, the unexpected outlier, the cluster that does not belong. A minute per chart, applied to ten charts, costs ten minutes and catches problems worth weeks of debugging.

Check your understanding

Quick check

—

A scatter plot of repayment days vs loan amount shows a clear positive trend overall. When you colour the points by province, you discover that within each province the trend is negative. What is happening?

You made a plotting mistake.
This is a classic case of Simpson's paradox: a trend that appears in aggregated data reverses when you break it out by group. Province is acting as a confounder — larger loans tend to be issued in provinces with shorter repayment cultures, but within each province, larger loans actually get longer repayment. The within-province trend is the truer relationship for modelling, but the aggregated trend is what someone naïvely looking at the data would report. Worth flagging in your analysis.
Ignore the province colours and trust the overall trend.
The dataset is bad and should be discarded.