A/B testing prompts and models

The setup

An A/B test compares two versions of your system, each serving a fraction of users:

• Variant A — the current version (control).
• Variant B — the change you want to test.

For a period (a day, a week), each incoming request is randomly assigned to A or B, and the results for each are tracked separately. At the end, you compare the metrics and decide.

Everything else stays the same: same eval set, same logging, same infrastructure. The only difference between the two groups is the specific change being tested.

What to A/B test

Every non-trivial change to an LLM system deserves an A/B test:

• Prompts. A rewritten system prompt, a new instruction, a better few-shot example.
• Models. Switching from Sonnet to Haiku (cheaper), or Sonnet to Opus (more powerful), or a different provider.
• Retrieval. New chunk size, new embedding model, adding a re-ranker.
• Temperature / max_tokens. Behavioural parameters.
• Feature additions. Adding conversation memory, adding tool use.

Every one of these can go either way. The A/B test protects you from shipping regressions.

A minimal implementation

Simple hash-based assignment:

import hashlib

def variant_for_user(user_id: str, test_name: str) -> str:
    """Return 'A' or 'B' deterministically for this user."""
    payload = f"{test_name}::{user_id}"
    h = hashlib.sha256(payload.encode()).hexdigest()
    return "B" if int(h, 16) % 100 < 50 else "A"


@app.post("/summarise")
def summarise(req, user_id: str = Depends(get_current_user)):
    variant = variant_for_user(user_id, "prompt-v2-test")

    if variant == "A":
        prompt = build_prompt_v1(req)
    else:
        prompt = build_prompt_v2(req)

    response = call_model(prompt)

    log_event("ab_test",
              test="prompt-v2-test",
              variant=variant,
              user_id=user_id,
              request_id=request_id,
              latency_ms=...,
              tokens_out=...,
              cost=...,
              output=response)

    return response

Two important properties:

1. Deterministic. Same user always sees the same variant. Prevents mid-session flipping.
2. Logged. Every request’s variant is logged so you can slice metrics later.

Choosing the sample size

The trap: run the test for a day, see A > B, ship A. But with 50 users a day, a day of data has too much noise to distinguish real differences from random variance.

Rule of thumb for LLM A/B tests:

• Behavioural metrics (thumbs, refusal rate): need ~500-1000 samples per variant for confidence.
• Latency and cost: need ~100-200 samples per variant.
• Quality (LLM-judged): need ~50-100 samples per variant.

At 100 daily active users, a 50/50 split gives 50 samples per variant per day. A meaningful test for behavioural metrics needs 10-20 days.

For low-traffic products, this is the honest constraint. Do not ship changes based on 20 samples per variant. It doesn’t work.

What to measure

The metrics you compare between A and B:

Direct quality (via eval set).

• Run the same eval set against A and B. Compare mean scores per dimension (correctness, completeness, faithfulness).
• Statistical test: paired t-test on per-question scores.

User behaviour on real traffic.

• Thumbs-up rate.
• Thumbs-down rate.
• Session length.
• Refusal rate.
• User retry-rate (did they rephrase because the answer was bad?).

Operational metrics.

• Latency (p50, p95).
• Cost per call.
• Error rate.

For most changes, quality (#1) and cost (#3) are the deciding factors. If B is 30% cheaper and quality is within 3% of A, ship B. If B is 5% better quality but 2× the cost, consider whether the users care about the difference.

An honest example

You want to test a shorter system prompt (400 tokens instead of 1200) for a Nepali policy bot.

Setup: 50/50 split for 2 weeks, 500 daily users. That’s ~7,000 samples per variant.

Results:

Metric              Variant A (1200-token prompt)   Variant B (400-token prompt)   Difference
Correctness (0-2)               1.84                          1.79                    -3%
Completeness (0-2)              1.72                          1.65                    -4%
Faithfulness (0-2)              1.91                          1.90                    -1%
Thumbs-up rate                  67%                           64%                     -3%
Latency p50 (ms)                1420                          1120                    -21%
Latency p95 (ms)                2400                          1950                    -19%
Cost per call ($)              $0.010                        $0.006                   -40%

Interpretation: B is meaningfully faster and cheaper (-40% cost, -20% latency). Quality is slightly worse (~3-4% drop on correctness/completeness).

Decision depends on your product priorities. If you’re cost-constrained and users generally rate the bot 4/5, the 3-4% quality drop may be worth the 40% cost saving. If you’re a premium product where every answer matters, keep A.

The point is that the decision is now numeric and honest, not vibes-based.

Multi-arm tests

You can test more than two variants at once — a 50/25/25 split for A/B/C, or a 33/33/33 for three variants. Same principles, more traffic needed for each arm to reach confidence.

Do not run 10 concurrent tests. Every additional test lowers the statistical confidence of each and makes results harder to interpret. Rule: at most 1-2 concurrent A/B tests per endpoint, with clear priorities.

Guardrails

An A/B test can silently damage users on the losing variant. Guardrails:

• Kill switch. A config flag that lets you drop B to 0% instantly if it turns out to cause serious issues (safety, cost, errors).
• Auto-abort thresholds. If B’s error rate exceeds A’s by 2× for 15 minutes, automatically route all traffic to A.
• Bounded blast radius. Don’t run the test on 50% of traffic if you’re unsure of the change. Start with 5% and grow only if metrics look healthy.

Ending a test

An A/B test ends when one of three things is true:

1. Statistical significance is reached. The difference between A and B on the primary metric is beyond the noise floor.
2. The test has run long enough that “no significant difference” is a valid conclusion. Ship the cheaper / simpler variant.
3. A guardrail fired. Abort.

Once decided, ship the winner as the new baseline. Delete the losing variant from the code. Do not let old test variants linger — they add complexity and confuse the next test.

The A/B mindset

For teams new to A/B testing, the shift is uncomfortable at first. You have to accept:

• Your intuition about “which is better” is often wrong.
• Small changes rarely produce dramatic differences (and dramatic differences are usually bugs).
• The right answer is usually revealed by the numbers, not by team debate.
• Some tests are inconclusive — that’s a real result, not a failure.

The discipline pays off in months. Products that A/B test their improvements ship better products than products that don’t — not because they’re smarter, but because they let evidence win over confidence.

What comes next

You know how to test individual changes. The last section of Chapter 4 is about turning the flood of real user interactions into a feedback loop — the ongoing improvement of your service from what real users actually do, want, and complain about.