Handling model provider drift

The kinds of drift

Provider-side changes come in a few shapes:

1. Silent model updates. The same model name behaves differently. Rare on Anthropic (they usually announce), more common on OpenAI’s “gpt-4o” family.
2. Announced deprecations. The provider says “we’re retiring model X on date Y.” You have 3-12 months to migrate.
3. API contract changes. New request fields, new response formats, sometimes breaking changes (rare, but they happen).
4. Pricing changes. Rate goes up or down; your cost model changes overnight.
5. Availability / rate limits. New usage tiers, new regional restrictions.

Each requires a different response. All require knowing they happened.

Staying informed

Watching for provider drift is a small monthly habit:

• Subscribe to provider changelogs.

  • Anthropic: docs.anthropic.com/en/release-notes and their status page.
  • OpenAI: their changelog page and email announcements.
  • Every serious provider has an announcement channel; subscribe.

• Check the model pinning. Once a month, look at your MODEL = "..." constants. Are the pinned versions still fresh? Any deprecation notices attached to them?

• Watch pricing pages. Providers sometimes lower prices (rare but wonderful) or raise them (usually with notice). Update your cost projections accordingly.

• Monitor community forums. Reddit, HackerNews, and Discord servers surface behaviour changes before providers officially acknowledge them. Not primary sources, but useful early signals.

Fifteen minutes a month. Prevents the “why is our service broken and no one knows?” scenario.

Testing a new model version

When Anthropic or OpenAI releases a new model (or updates an existing pinned version), run your eval set against it:

# In your eval runner
CANDIDATE_MODELS = {
    "current":    "claude-3-5-sonnet-20241022",
    "candidate":  "claude-3-5-sonnet-20250115",
}

for name, model in CANDIDATE_MODELS.items():
    results = run_eval(eval_set, model=model)
    print(f"{name}: correctness={results['correctness']}, cost=${results['cost']}")

Three questions:

1. Is quality maintained or improved?
2. Is cost lower, equal, or higher?
3. Is latency comparable?

If quality is equal or better, cost is equal or lower, and latency is comparable, migrate. If any of those regress, decide: is the improvement in the other dimensions worth the regression?

Never migrate blindly because “the newest model is best.” Real projects have seen quality drops on nominally-better models because the new training data (or new safety fine-tuning) changed behaviour in ways specific to their use case.

Handling deprecation notices

Anthropic and OpenAI give ample notice (usually 3-12 months) before retiring models. When you receive one:

1. Add the migration to the roadmap. Not “eventually” — a specific target date, at least a month before the deprecation.
2. Test the recommended replacement. Run the eval set on the new model.
3. Test on staging first. Deploy the new model to staging for a week. Monitor metrics.
4. Roll out to production with a canary. 5%, 25%, 50%, 100% over a week.
5. Retire the old model reference from code. Once fully migrated, remove the old pinned version from configuration.

The window between “deprecation announced” and “deprecation happens” is ample. Failed migrations are almost always failed because the team ignored the notice until the last week.

Handling API contract changes

Rarer, but real. Occasionally a provider changes the API’s request or response shape.

Mitigation:

• Pin the SDK version. Just like the model, pin the SDK to a specific release in requirements.txt (anthropic==0.40.0, not anthropic). SDK upgrades are their own scheduled work.
• Read the SDK changelog before upgrading. New SDK versions sometimes have breaking changes; the changelog will list them.
• Have integration tests. A small test that calls the real API and asserts the response shape catches API-contract changes the moment they happen. Run these nightly.

When the provider drift is bad

Sometimes the drift genuinely damages your product:

• The new model version scores 10% lower on your eval.
• The pricing doubles.
• The provider deprecates a feature your product depends on.

Options:

• Stay on the old version (if it’s not deprecated). Sometimes the answer is “we won’t upgrade.”
• Migrate to a different provider. Cross-provider migration is real work, but it’s the strongest defence against long-term drift dependence.
• Migrate to open-source. Local models (Llama, Mistral, Qwen) are increasingly viable for many workloads. The trade-off is you take on serving infrastructure.

Provider drift is why some teams take on the burden of multi-provider infrastructure from day one — the abstraction lets them switch when they must. For Nepal-scale products, single-provider is usually fine, but keep the escape hatch in mind.

The migration playbook

When you have to switch models, providers, or embedding models, the concrete steps:

1. Freeze the change window. No other unrelated changes during the migration.
2. Extend the eval set. Add specific test cases that stress the areas you’re worried might regress.
3. Test in isolation. Run the new model against the eval set. Score every dimension.
4. Deploy behind an A/B test. 5% initially.
5. Compare production metrics. Real-user thumbs, latency, cost.
6. Ramp up. 25% → 50% → 100% over a week if metrics look healthy.
7. Remove the old code path. Only after you’re sure the new version is fully deployed and stable.
8. Write a change log entry. Record the migration for future you.

The whole playbook: 2-4 weeks of calendar time for a well-managed migration. Rushed migrations are where regressions come from.

The trust budget

Every migration burns a small amount of user trust — some queries will get slightly different answers, some will regress. Budget for it.

Announce meaningful model migrations to users if your product’s tone permits it: “Starting Monday, we upgraded to a newer model. You may see slightly different answers as we tune the behaviour. Let us know if anything degrades.” Users who understand the change accept small variations. Users who don’t understand assume it’s a bug.

The one-off external event

Beyond regular drift, occasionally there is a big event: a provider outage lasting a day, a legal issue prompting a rapid API change, a security incident requiring a forced model migration.

For these:

• Have a fallback provider. Even if you rarely use it, having Claude and OpenAI both integrated means you can switch in an hour when one has a bad day.
• Have a public status page. Users appreciate honesty. status.nepse-mitra.np with “Currently degraded due to provider outage; expected resolution in 4 hours” beats silence.
• Have a “Nepal-safe” fallback if your product depends on external services. If Anthropic or OpenAI is unreachable from Nepal (VPN block, provider outage, submarine cable cut), what does your service do? Even a static “We’re temporarily unavailable” page beats a dead URL.

Closing chapter 5

You now have the discipline for changing an AI system safely. Version each axis. Test each change. Roll out gradually. Watch for external drift. Migrate deliberately when you must. Rollback plans for every axis.

None of this is glamorous. All of it is what separates an AI system that lasts three years from an AI system that quietly rots after month six.

What comes next

Chapter 6 is the final chapter of Course 05. It shifts from operations engineering to the specific realities of running an AI product in Nepal: trust and privacy expectations, the legal and regulatory landscape (as it stands in 2026), and the practitioner’s playbook that ties everything together and closes the course.