Safe rollouts and rollback

The staging environment

Before anything hits production, it runs in staging — an identical replica of production that no real user sees. Staging exists so that you catch the “it works on my laptop but not on the deployed environment” bugs before real users do.

Structure:

Environments:
- dev         (your laptop)
- staging     (a full replica, deployed at staging.nepse-mitra.np)
- production  (the real thing, at nepse-mitra.np)

Every deploy goes to staging first. You test the change end-to-end on staging (using the same eval set from Chapter 4). Only after it passes does it go to production.

Cost: staging typically runs at 10-25% the scale of production, so infrastructure cost is modest. The saved outages easily justify it.

Canary and staged rollout

Even after staging tests, a change may misbehave on real traffic. The pattern: gradual rollout.

1. Canary deploy: the new version goes live for 5% of users.
2. Watch for 30-60 minutes: check error rate, latency, thumbs-down, cost.
3. If metrics are healthy, expand: 25% → 50% → 100% over hours.
4. If metrics degrade at any point, roll back immediately.

This mechanism means at worst, 5% of users saw a broken version — for 30 minutes. Not 100% of users for 12 hours.

On managed platforms (Fly.io, Kubernetes, Vercel), staged rollout is a built-in feature. On simpler platforms, do it yourself with an application-level flag:

NEW_PROMPT_ROLLOUT_PCT = 5  # start at 5%, raise as confidence grows

def prompt_for_user(user_id: str) -> str:
    if hash_percent(user_id) < NEW_PROMPT_ROLLOUT_PCT:
        return NEW_PROMPT
    return CURRENT_PROMPT

Adjust NEW_PROMPT_ROLLOUT_PCT progressively (5 → 25 → 100). When you reach 100 and the metrics have been stable for 24 hours, promote the new prompt to CURRENT_PROMPT and remove the flag.

The instant rollback

Every change needs a plan to reverse it in under 5 minutes:

• Prompt change: revert the constant, redeploy. Or (if using feature flags) drop the rollout percentage to 0.
• Model version change: change the pinned version back, redeploy.
• Data / index change: point the retrieval code at the previous index (keep the last few indexes around).
• Infrastructure change: platform-level rollback (Fly.io: flyctl deploys rollback; Kubernetes: kubectl rollout undo).

Practise the rollback before you need it. On day one of a new deployment, deliberately roll back once — just to verify the mechanism works. A rollback that only exists in theory is not a rollback.

Blue/green deploys

Advanced pattern for infrastructure changes: run two full copies of production simultaneously. Route all traffic to “blue” while you deploy the new version to “green.” When green is healthy, switch traffic. If green fails, switch back to blue.

For a small Nepal-scale product, blue/green is overkill. For anything serving thousands of users, or where downtime is measured in dollars, it becomes worthwhile. Managed platforms provide it as a checkbox.

The rollout checklist

Before every non-trivial deploy:

• Passes the automated tests (unit + integration).
• Passes the eval set on staging.
• Change log entry written.
• Rollback plan documented (specifically for this change).
• Metrics dashboards open, ready to watch.
• Deploy at a reasonable hour (not late night; not right before a holiday).
• Team member awake and available for the first 30-60 minutes post-deploy.

Skip any step at your peril. Every step is small; skipping any of them costs more than doing all of them combined.

Change fatigue

Frequent, safe deploys are the goal. The trap is that when small deploys become routine, the discipline erodes. Someone deploys without checking, someone skips the change log, someone reduces staged rollouts to “just push to 100%.”

Preserve the discipline by keeping deploys friction-low but ceremony-explicit. Every deploy announcement in Slack:

Deploying prompt v1.5 to production.
Change: added Nepali script normalisation instruction.
Rollout: 5% canary for 30 min, then 100% if healthy.
Rollback: revert commit abc123, redeploy.
Deployer: @nishan
Metrics: dashboard.example.com/nepse-mitra

Five lines. Two minutes to write. Signals to the team that the change is happening, is bounded, and has a plan.

Rolling back is not failure

Culturally, teams need to internalise: rolling back is a success, not a failure. The alternative — leaving a broken change live to preserve pride — is where trust gets destroyed.

Every team should have a norm: whoever notices the regression rolls back, no permission needed. Post-mortems happen after users are protected, not before.

The rollout maturity ladder

Where is your team today?

Level 1 — Cowboy. Deploys directly to production. No staging. No rollback plan. Breakages are frequent and hard to fix. (Most day-1 projects.)

Level 2 — Staged. Has a staging environment. Deploys go to staging first, tested manually, then to production. Fewer regressions. Rollback is possible but not fast.

Level 3 — Canary. Staged rollout for every real change. Fast rollback (<5 minutes). Every deploy tracked. Metrics watched actively during rollout.

Level 4 — Automated. Rollouts are automated. Metrics automatically abort a canary that misbehaves. Every deploy is a routine event, not a nerve-wracking one.

Most Nepal-scale AI products should aim for Level 3. Level 4 is worth it once you have a team of 4+ and traffic large enough that regressions cost real money.

What comes next

You know how to roll changes out safely. But some changes come from outside your control — Anthropic updates a model, OpenAI deprecates an API, a Nepal-specific data source changes format. The last section of Chapter 5 is about handling that specific class of change: model provider drift, and the operational discipline it requires.