Continuous evaluation · Shipping AI Systems

You tested carefully before launch. Weeks later, quality has drifted, and nobody noticed. This is the most common quiet failure mode of production AI: the launch was solid, real usage was solid, but the model provider updated something, or your prompts drifted, or the queries shifted, and now users get worse answers than they used to. This section is the small discipline that catches that drift: continuous evaluation, run every day, on a fixed benchmark, tracked over time.

The problem

An AI system’s quality is not a fixed number. It moves for many reasons:

Provider updates. Anthropic or OpenAI silently update the underlying model. Behaviour changes.
Prompt edits. Someone adjusts a system prompt for one case; a different case regresses.
Data updates. New documents in the RAG index change what’s retrieved.
Query drift. Users start asking different things; the corpus doesn’t cover them anymore.

Any one of these can cause quality to slowly deteriorate. Without a fixed benchmark, none is visible — until users complain, at which point damage is already done.

Continuous evaluation is the fix: a benchmark you run on a schedule, that produces the same number under the same conditions, so changes in that number are the signal to investigate.

The eval set

A good eval set has three properties:

Fixed. Same questions, same expected answers, forever. Don’t add or edit unless you’re deliberately updating the benchmark.
Representative. Covers the actual distribution of user queries — including the edge cases and Nepal-specific patterns.
Sized right. 50-200 items. Enough for statistical stability, small enough to run cheaply and often.

For a Nepali policy bot, a starter eval set might look like:

30 policy questions in Nepali
30 policy questions in English
10 mixed-language queries
10 out-of-scope queries (should be refused)
5 ambiguous queries (should ask for clarification)
5 adversarial / prompt-injection attempts

Total: 90 items. Each item has:

The input query.
The expected behaviour (correct answer, or “refuse”, or “clarify”).
A grading rubric with 2-3 criteria (correctness, source citation, format).

Grading — the hard part

Human grading is the gold standard but doesn’t scale. LLM-based grading — where you use another LLM to grade the answer — is the practical compromise.

An LLM-as-judge prompt:

JUDGE_PROMPT = """You are grading an AI assistant's answer.

Question:
{question}

Reference answer (from the source document):
{reference_answer}

Model's answer:
{model_answer}

Grade the model's answer on three dimensions, each 0-2:

1. Correctness: does the answer say the right thing?
   0 = wrong, 1 = partially correct, 2 = correct
2. Completeness: does it cover the key facts?
   0 = misses key facts, 1 = missing some, 2 = complete
3. Faithfulness: are the facts supported by the reference?
   0 = hallucinated facts, 1 = some drift, 2 = fully supported

Return a JSON object with keys 'correctness', 'completeness', 'faithfulness', and 'comment'.
"""

def grade(question, reference, model_answer):
    prompt = JUDGE_PROMPT.format(
        question=question,
        reference_answer=reference,
        model_answer=model_answer,
    )
    response = anthropic_client.messages.create(
        model="claude-3-5-sonnet-latest",
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}],
    )
    return json.loads(response.content[0].text)

Use a different model for grading than the model being graded. This reduces the risk that the grader is biased in favour of its own outputs.

Running the eval

Once a day (or on every deploy), run the full eval set through your service:

def run_eval(eval_set, service_url):
    results = []
    for item in eval_set:
        answer = call_service(service_url, item["query"])
        grade = grade_answer(item["query"], item["reference"], answer)
        results.append({
            "id": item["id"],
            "query": item["query"],
            "answer": answer,
            **grade,
        })

    scores = {
        "correctness": sum(r["correctness"] for r in results) / len(results),
        "completeness": sum(r["completeness"] for r in results) / len(results),
        "faithfulness": sum(r["faithfulness"] for r in results) / len(results),
    }
    return scores, results

Save the scores to a table with a timestamp. Chart the trend over time:

Date       Correctness  Completeness  Faithfulness
2026-05-01     1.83         1.71         1.92
2026-05-08     1.81         1.68         1.91
2026-05-15     1.85         1.72         1.93
2026-05-22     1.72         1.65         1.85   ← drop
2026-05-29     1.68         1.60         1.82   ← keeps dropping

The drop between 2026-05-15 and 2026-05-22 is the signal. Something changed in that week — a deploy, a provider update, a data change. Investigate before users report it.

Per-slice tracking

The overall score can hide a regression in a specific slice. Break down by:

Language — Nepali vs. English separately. Regressions often hit one language and not the other.
Query type — factoids vs. reasoning vs. summarisation.
Topic — different sections of your documents / different policy areas.

Date        Overall  Nepali  English  Refusals
2026-05-22   1.72     1.55    1.89     1.30    ← Nepali dropped
2026-05-29   1.68     1.48    1.88     1.30    ← Nepali still worse

The overall score is stable-ish; Nepali is falling. The overall metric would have missed this. Slice-by-slice tracking wouldn’t.

Investigating a regression

When the eval score drops:

Look at the failing items. Which specific questions regressed? Read the model’s new answers vs. its old answers.
Look at recent changes. What deployed between the last two evals? Any provider announcements? Any data updates?
Reproduce. Can you reproduce the regression on a single failing query?
Bisect. Try older versions of your prompt / model / retrieval to isolate which change caused it.
Fix or accept. Either fix the regression (rollback a change, improve a prompt) or accept the new baseline (some quality trade-offs are worth other benefits).

The eval set is your instrument. The investigation is where the actual work happens.

Building the eval set

Where do the questions come from?

Real user queries. The best source. Take a random sample from your logs, hand-write reference answers, mark expected behaviours.
Synthetic queries. Ask an LLM to generate 30 questions about your documents. Curate ruthlessly.
Adversarial queries. Hand-craft the edge cases: prompt injection, ambiguous, out-of-scope.
Regression cases. When a user reports a bug and you fix it, add that query to the eval set. Now the fix is protected against future regression.

Start with 30-50 items and grow monthly. Reject items that are ambiguous, poorly graded, or don’t discriminate (everyone always gets them right).

The eval discipline

Three habits:

Automate the run. Every night, a scheduled job runs the eval and posts the scores to Slack. No one has to remember.
Chart the trend. Score-over-time is the most useful view. Sudden drops trigger investigation.
Grow the set. Whenever you fix a user-reported bug, add a corresponding eval item. The set becomes your regression suite.

Once you have this, you know when quality changes. That knowledge is the substrate for every other improvement in Chapter 4.

Check your understanding

Quick check

—

A team runs an eval only at launch, not continuously. Three months later, users start complaining about wrong answers. What operational tool would have caught this earlier?

Better pre-launch testing.
A daily (or per-deploy) automated eval on a fixed benchmark, with the scores charted over time. Quality drift in production has many causes — provider updates, prompt edits, data updates, query drift — none visible from logs alone. A continuous eval turns 'quality changed' from a mystery to a scheduled measurement. When the score drops, the timestamp tells you when to look, and comparing the drop to your deploy history usually identifies what to fix. Without continuous evals, quality regressions are invisible until users complain — by which time trust has already eroded.
A larger vector database.
A more powerful model.

Quick check

—

Why should the LLM used to grade an eval be different from the LLM being graded?

It costs less.
To reduce self-preference bias: LLMs tend to grade their own style / patterns / phrasing more favourably than a human would. Using a different provider (e.g. grading Claude with GPT, or vice versa) produces more independent judgments and catches regressions that a same-model grader would miss. This isn't perfect independence — both models share training-data patterns — but it's a meaningful safeguard against bias, and it's cheap to implement. In high-stakes contexts, add periodic human grading alongside.
The grader model is smarter.
It runs faster.

What comes next

Continuous evaluation tells you when quality changed. The next section is how to change quality on purpose: A/B testing prompt and model variants. Trying a new prompt against the old one on real traffic, measuring the difference, and deciding whether to keep the change — the standard discipline for shipping improvements safely.