Building, evaluating, and iterating

The eight-step shape of every project

Every ML project from now on follows the same eight steps. Internalise the order; the details vary, the shape doesn’t.

1. Frame — the previous section’s deliverable.
2. Acquire — fetch and load raw data.
3. Explore — EDA (Course 02 Ch 4).
4. Clean — handle missing/duplicate/bad data (Course 02 Ch 3).
5. Engineer features — create the columns the model will use (Course 02 Ch 5).
6. Split, fit, evaluate — your first honest performance number (Course 03 Ch 1).
7. Iterate — try other models, tune hyperparameters, prune features (Course 03 Ch 5).
8. Cross-validate and finalise — get a trustworthy number (Course 03 Ch 5 Section 3).

The last section of Chapter 6 covers the ninth step — presenting results honestly — but for the build itself, eight steps.

Step 2 — Acquire

For NEPSE: download a CSV of daily index values from one of the public sources (NEPSE Alpha, Sharesansar — verify scraping permissions; see Course 02 Ch 2 Section 3). Save as data/raw/nepse_daily.csv.

import pandas as pd

df = pd.read_csv("data/raw/nepse_daily.csv", parse_dates=["date"])
df = df.sort_values("date").reset_index(drop=True)
print(df.shape, df["date"].min(), df["date"].max())

Five years of daily data is roughly 1,250 trading rows. Modest but workable.

Step 3 — Explore

Apply the 30-minute habit from Course 02 Ch 4 Section 1.

print(df.dtypes)
print(df.describe())
df["close"].plot(figsize=(12, 4))

# Distribution of daily returns
df["return"] = df["close"].pct_change()
df["return"].hist(bins=80)

You’ll see classic financial-data patterns: roughly normal returns with fatter-than-normal tails, occasional volatility regimes, and long stretches of slow drift.

Three guess-and-check questions:

• Q: What fraction of days does the index go up? Probably close to 52% — markets drift up slowly. Check.
• Q: Are there clusters of high volatility? Plot return.abs().rolling(20).mean() and look. Almost certainly yes.
• Q: Are the highest-return days clustered? Check whether the 50 largest gain days fall near the 50 largest loss days. Almost certainly yes — volatility clusters.

This 15 minutes shapes your feature engineering. You learn the data has volatility regimes, which suggests rolling-volatility features will help.

Step 4 — Clean

For daily index data, cleaning is light:

• Remove rows with NaN close prices (data feed errors).
• Verify there are no duplicate dates.
• Ensure the date column is properly sorted.

df = df.dropna(subset=["close"]).reset_index(drop=True)
assert df["date"].is_monotonic_increasing
assert not df["date"].duplicated().any()

Three lines. For some projects the cleaning will be a hundred lines; for daily NEPSE, it’s three. Adjust per project.

Step 5 — Engineer features

This is where domain knowledge buys you the most. For predicting tomorrow’s direction:

import numpy as np

df["return_1d"] = df["close"].pct_change()
df["return_3d"] = df["close"].pct_change(3)
df["return_5d"] = df["close"].pct_change(5)
df["return_20d"] = df["close"].pct_change(20)

df["return_1d_lag1"] = df["return_1d"].shift(1)
df["return_1d_lag2"] = df["return_1d"].shift(2)

df["volatility_5d"] = df["return_1d"].rolling(5).std()
df["volatility_20d"] = df["return_1d"].rolling(20).std()

df["above_ma_20"] = (df["close"] > df["close"].rolling(20).mean()).astype(int)
df["above_ma_50"] = (df["close"] > df["close"].rolling(50).mean()).astype(int)

df["day_of_week"] = df["date"].dt.dayofweek
df["month"] = df["date"].dt.month

# Target: did the index go up tomorrow?
df["target_up_tomorrow"] = (df["close"].shift(-1) > df["close"]).astype(int)

# Drop rows with NaN from rolling features and the future-shifted target
df = df.dropna().reset_index(drop=True)

A dozen features — recent returns at various horizons, rolling volatility, position relative to moving averages, calendar features. Domain experts in finance will recognise these as the standard short-horizon technical features.

Step 6 — First fit and evaluate

The full pipeline with TimeSeriesSplit (because this is time series):

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score, TimeSeriesSplit
from sklearn.metrics import f1_score, classification_report, confusion_matrix

features = [
    "return_1d", "return_3d", "return_5d", "return_20d",
    "return_1d_lag1", "return_1d_lag2",
    "volatility_5d", "volatility_20d",
    "above_ma_20", "above_ma_50",
    "day_of_week", "month",
]

X = df[features]
y = df["target_up_tomorrow"]

# Chronological train/test
cutoff = int(len(df) * 0.8)
X_train, X_test = X.iloc[:cutoff], X.iloc[cutoff:]
y_train, y_test = y.iloc[:cutoff], y.iloc[cutoff:]

model = GradientBoostingClassifier(random_state=42)
model.fit(X_train, y_train)

predictions = model.predict(X_test)
print(classification_report(y_test, predictions))
print(confusion_matrix(y_test, predictions))
print(f"Baseline (always up): {y_test.mean():.3f}")

Sample output:

              precision    recall  f1-score   support
           0       0.58      0.51      0.54       115
           1       0.62      0.68      0.65       135

    accuracy                           0.60       250
   macro avg       0.60      0.59      0.60       250
weighted avg       0.60      0.60      0.60       250

[[ 59  56]
 [ 43  92]]

Baseline (always up): 0.540

The model is 6 percentage points above baseline. For NEPSE direction prediction, that’s a decent first pass — markets are hard. Better than coinflip, not magic.

Step 7 — Iterate

Two or three obvious iterations to try, using Chapter 5:

# Try several models
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

models = {
    "Logistic":  LogisticRegression(max_iter=1000),
    "Forest":    RandomForestClassifier(n_estimators=200, random_state=42),
    "GradBoost": GradientBoostingClassifier(random_state=42),
}

for name, m in models.items():
    m.fit(X_train, y_train)
    pred = m.predict(X_test)
    print(f"{name:12s} F1={f1_score(y_test, pred):.3f}")

Then tune the winner with RandomizedSearchCV:

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform

param_dist = {
    "n_estimators": randint(50, 300),
    "max_depth": randint(2, 8),
    "learning_rate": uniform(0.01, 0.3),
    "subsample": uniform(0.6, 0.4),
}

tscv = TimeSeriesSplit(n_splits=5)
search = RandomizedSearchCV(
    GradientBoostingClassifier(random_state=42),
    param_distributions=param_dist,
    n_iter=30,
    cv=tscv,
    scoring="f1",
    n_jobs=-1,
    random_state=42,
)
search.fit(X_train, y_train)

print(f"Best params: {search.best_params_}")
print(f"CV F1: {search.best_score_:.3f}")
print(f"Test F1: {f1_score(y_test, search.predict(X_test)):.3f}")

You’ll typically see 1-3 percentage points of improvement. Not magic, but meaningful — and earned by disciplined search.

Step 8 — Cross-validate the finalist

The CV inside RandomizedSearchCV already gave you the trustworthy CV score. The held-out test F1 gives you the truly honest evaluation. Sanity-check:

# CV score from the search
cv_score = search.best_score_

# Test set score (the truly held-out one)
test_score = f1_score(y_test, search.predict(X_test))

# If they disagree by more than ~0.03, investigate
gap = abs(cv_score - test_score)
print(f"CV F1: {cv_score:.3f}  Test F1: {test_score:.3f}  Gap: {gap:.3f}")

A gap above ~0.05 typically suggests distribution shift between training and test (likely with time-series data — markets change), or a small/unrepresentative test set, or hidden leakage.

A short reflection — what just happened

In one section, you went from data/raw/nepse_daily.csv to a tuned gradient-boosting classifier with an honest CV evaluation and a test-set F1 score above baseline. The whole thing is, depending on your typing speed, two to four hours of work.

That is the shape of a working ML project. The same shape — frame, acquire, explore, clean, engineer, fit, iterate, validate — applies to every project you’ll build for the rest of your career, regardless of domain. The specifics change. The pattern stays.

Check your understanding

Quick check

—

In the NEPSE example, the target is `close.shift(-1) > close`. A teammate suggests adding `close.shift(-1)` itself as a feature, arguing it would help the model. What's wrong?

1. ANothing — using more features is always better.
2. BIncluding tomorrow's close as a feature is target leakage. The model would 'predict' the direction by literally looking at tomorrow's close, which is exactly the information you don't have at prediction time. Training F1 would jump to ~1.0 and the model would be useless in production — there is no future data on the day the prediction is needed. Always ask 'could this feature value have been known at the time the prediction needs to be made?' before including it.
3. CThe model would be too slow to train.
4. DTomorrow's close has no useful information.

Check answer→

Quick check

—

After tuning a NEPSE direction predictor, the cross-validated F1 is 0.62 and the held-out test F1 is 0.49. What is the most likely cause and what's the response?

1. A0.49 is fine; deploy.
2. BA large gap between CV and test scores in a time-series context almost always indicates distribution shift — the test period had different market dynamics than the training period. This is a feature of financial data, not a bug in your code. Investigate by plotting model F1 over time (rolling 20-day windows in the test set), check if the model is failing during specific regimes (high volatility, policy events), and consider retraining more frequently in production. The 0.49 is the more honest expected production performance — adjust stakeholder expectations accordingly.
3. CThe test set is corrupted; rerun the split.
4. DUse a different cross-validation strategy.

Check answer→

What comes next

The model is built and evaluated. The last step — the easiest to skip and the highest-leverage — is presenting the results honestly. Numbers, charts, limitations, and what you’d build next. That is the next and final section of Chapter 6.