Prompts, models, and data — three axes of change · Shipping AI Systems

A classical web app changes when someone changes the code. An AI system changes for three reasons at once: someone edited the prompt, someone updated the model, or someone re-indexed the data. Any of the three can break the others, and any of them can happen without a code commit. This section is the mental model that turns that chaos into discipline — and the small habits that make “what changed” always answerable.

The three axes

Every LLM system has three independent things that can change:

Prompt. The system prompt, the few-shot examples, the retrieval-context format, the output format.
Model. The provider, the model name, the model version, the parameters (temperature, max_tokens).
Data. The documents in your RAG index, the embeddings, the chunk size, the retrieval strategy.

A change on any axis can affect behaviour. A change on more than one axis simultaneously makes it hard to know which change caused the observed effect. Small habits keep this manageable.

Why this matters

Real story from real teams: on a Tuesday, someone tightened a system prompt. On the same day, someone re-indexed the RAG corpus with a new chunker. On Wednesday, thumbs-down rate rose. The team spent a week arguing whether the prompt or the chunker was to blame. Neither could be reversed cleanly, because “the change” was two changes tangled together.

Versioning and disciplined rollout separate the axes. Change one thing at a time. Know which change caused which effect. Roll back with confidence.

Version the prompt

Prompts belong in code, not in a Google doc:

# prompts.py
PROMPT_V1 = """You are a helpful assistant for Nepal's passport office FAQs.
Answer questions using only the documents provided..."""

PROMPT_V2 = """You are niyam, a helpful assistant for Nepal's passport office.
Follow these rules strictly:
1. Answer in the same language as the question.
2. Cite the document name after each fact.
..."""

CURRENT_PROMPT = PROMPT_V2
CURRENT_PROMPT_VERSION = "v2.0"

In your endpoint, always log the prompt version:

response = client.messages.create(
    model=CURRENT_MODEL,
    system=CURRENT_PROMPT,
    ...
)
log_event("model_call",
          request_id=request_id,
          prompt_version=CURRENT_PROMPT_VERSION,
          model=CURRENT_MODEL)

When you experiment with a new prompt, add it as PROMPT_V3 alongside the old one; use A/B test infrastructure to route a fraction of traffic. If it wins, promote it to CURRENT_PROMPT and delete V1. If it loses, delete V3.

Never edit a prompt in place without bumping the version. The version tag ties prompt content to observed metrics.

Version the model

Model version is the most easily forgotten. "claude-3-5-sonnet-latest" is convenient — until Anthropic updates what “latest” points to, and your service silently changes behaviour.

Rule: pin the exact model version in production code.

# Not this:
MODEL = "claude-3-5-sonnet-latest"

# This:
MODEL = "claude-3-5-sonnet-20241022"

The trade-off: pinned models don’t get automatic improvements. That’s the point. When Anthropic ships a new version, you evaluate the new version against your eval set on your own timeline, and switch when you decide.

Same logic for OpenAI, Gemini, and every provider. gpt-4o-2024-11-20, not gpt-4o.

Version the data

RAG systems change every time the underlying documents change or the index rebuilds. Every version of the index is a version of the system’s behaviour.

Track:

Document version. Each source document has a version (hr-policy-v3.pdf, or a git SHA of the source repo).
Chunking strategy. Chunk size, overlap, and boundary rules are part of the version.
Embedding model. The embedding model used (text-embedding-3-small-v1, or a versioned local model).
Index build timestamp. When was this specific index built?

Tag each index build with a version identifier:

INDEX_VERSION = "2026-06-15-hr-v3-embed-3s"

In every request, log which index version served it:

log_event("retrieval",
          request_id=request_id,
          index_version=INDEX_VERSION,
          top_k=3,
          top_score=hits[0]["score"])

When you re-index, produce a new versioned index and A/B test the new one against the current one before promoting.

The change log

For every AI system, maintain a change log — a file (or a dashboard) that records every change on every axis, with timestamps:

2026-05-01  10:14  MODEL     Switched sonnet-3-5-20240620 → sonnet-3-5-20241022
2026-05-08  15:32  PROMPT    v1.2 → v1.3 (adjusted refusal wording)
2026-05-15  09:00  DATA      Reindexed HR corpus (v3.1 → v4.0, new chunker)
2026-05-22  14:11  PROMPT    v1.3 → v1.4 (added Nepali script normalisation instruction)
2026-06-01  11:00  MODEL     A/B test: 50% traffic to haiku-4-5 for cost savings

When metrics move, the change log is the first place you look. “Thumbs-down rose on 2026-05-16 — what changed?” → “We reindexed on 2026-05-15.” → investigation.

Without this log, every regression investigation is archaeology. With it, it’s a database query.

The rollback contract

For each axis, know how to roll back:

Prompt rollback. Trivial — revert the constant, redeploy.
Model rollback. Change the pinned version back, redeploy. Two-minute recovery.
Data rollback. Keep the previous index alongside the new one for at least 7 days. If the new index causes regressions, swap the pointer back to the old one.

If any axis lacks a rollback plan, you don’t have full versioning. Fix that before shipping the next change.

The dependency graph

Some changes cross axes:

Changing the embedding model requires re-embedding all documents (the index becomes incompatible).
Changing the model provider may require rewriting parts of the prompt (Anthropic and OpenAI differ in what they respond well to).
Changing the chunk size invalidates the specific top-K retrieval behaviour.

Note these dependencies. When a cross-axis change is unavoidable, plan it explicitly: one deploy that changes only the coupled axes together, followed by a stabilisation period before any other change.

The one-thing-at-a-time habit

The concrete rule in day-to-day work:

No deploy changes more than one axis unless explicitly required.
Each change ships with an incremented version tag.
Each change is A/B tested when feasible.
Each change is logged in the change log.
Metrics are watched for 24-48 hours after each ship.

Every one of these adds friction. That friction is the price of a system whose behaviour is understandable over months. Skipping the friction is how AI systems become mysteries their own creators cannot debug.

What “good versioning” feels like

A well-versioned AI system has these properties:

Given any answer from any date, you can say exactly which prompt, which model, which index produced it.
When metrics move, you can identify which change (or which combination of changes) caused it within an hour of investigation.
Rolling back any single change takes minutes, not days.
Comparing “how the system behaves today” vs. “how it behaved a month ago” is a queryable question, not a story anyone has to remember.

If your system has these properties, you can safely iterate. If it doesn’t, every change is a gamble against the invisible state of everything else.

What comes next

You know how to version the three axes. The next section is how to roll out changes to those axes safely — canary deploys, staged rollouts, and the specific patterns that prevent a bad change from reaching all users at once.