5.1 Testing your prompt: build a small eval set

What an eval set is

An eval set is a small, hand-curated collection of inputs where you know what the right output looks like. For a classification prompt, that’s 10–50 example inputs each with the correct label. For a summarization prompt, it’s 5–15 source articles each with a short note on what a good summary must include.

You don’t need thousands of examples. You need enough to cover:

• The common case (the input you expect 90% of the time).
• The edge cases you’ve already seen fail (the specific bugs the current prompt fixes).
• A few adversarial inputs (empty input, very long input, unusual language, prompt-injection attempts).

Fifteen well-chosen examples beat five hundred random ones.

Where the examples come from

• Your own history. If the prompt has been running for a week, pull inputs from actual usage logs. Real inputs are messier than the ones you’d invent.
• The specific bugs. Every time the prompt fails in production, add that input to the eval set with the correct output. This is called a regression case — you’re pinning the bug down so it can never come back silently.
• Deliberate stress tests. Empty strings, single-word inputs, inputs in the wrong language, inputs that look like SQL injection, inputs the size of a book.

The format

For classification, a CSV or JSONL of { input, expected_label }:

{"input": "App keeps crashing when I try to send money.", "expected": "app_crash"}
{"input": "OTP not arriving even after 5 minutes.", "expected": "login_or_auth"}
{"input": "Uploaded citizenship 3 days ago, still pending.", "expected": "kyc_or_verification"}
{"input": "", "expected": "other"}
{"input": "hi", "expected": "other"}

For extraction, { input, expected_json }. For summarization, { input, must_contain: [...] } — a short list of phrases or facts that must appear in a good summary, and optionally must_not_contain: [...].

The run

Loop over the eval set, run the prompt on each input, compare against expected:

• For classification and extraction, comparison can be exact (predicted == expected).
• For summarization and rewriting, use rubric checks (does the output contain "NPR 240 million"?) or a second model as judge (feed the output plus rubric to a second prompt that returns pass/fail per criterion).

You get a single number: accuracy. That’s your baseline. Now every time you change the prompt, re-run and compare.

What the eval set is not

• It is not a benchmark. You’re not comparing your prompt against a published leaderboard; you’re comparing this version of your prompt against the last version.
• It is not a formal test suite. It doesn’t need to cover every possible input. It needs to cover the inputs you actually care about not getting wrong.
• It is not one-and-done. It grows every time production surprises you.

The trap of overfitting

If you iterate too aggressively on the same 15 examples, you’ll end up with a prompt that scores 100% on your eval set and 60% on real data. Defenses:

• Keep a held-out set of 5–10 examples you don’t tune against. Only run them occasionally to sanity-check that your main eval score isn’t misleading you.
• Add fresh production examples regularly. If the eval set is frozen, it stops reflecting reality.
• Watch for “the prompt now has X because example 7 needed it” spirals. Those specific additions rarely help outside example 7.

Fifteen minutes is enough

You do not need Weights & Biases, a vector store, or a testing framework. A spreadsheet with two columns (input, expected) and a small script that loops through and prints pass/fail is enough for 95% of real use. Start there; add tooling only when the eval set outgrows a spreadsheet.

Check your understanding

Quick check

—

Your prompt is passing all 15 of its eval-set examples but customers are still complaining about wrong outputs in production. Which is the MOST likely explanation?

1. AThe eval framework is broken.
2. BThe eval set is stale or unrepresentative — the 15 examples do not reflect the shape of real customer inputs, and you have been iterating against them rather than reality.
3. CThe model is wrong; contact the provider.
4. DCustomers are wrong.

Check answer→