2.2 Run-until-done: stop when the goal is proven

What “done” looks like

The whole trick is turning a fuzzy goal (“make it good”) into a checkable condition. Some examples that work:

• Tests pass. pytest exits 0. Trivially verifiable.
• Endpoint returns 200. After deployment, curl the health check.
• A file exists. After extraction, report_2026-07-22.pdf is on disk.
• A JSON field is present. After parsing, { status: "approved" } is in the output.
• A count reaches a target. After tagging, at least 95% of items have a label.

Some that don’t work as stop conditions:

• “The result is good.” Subjective. The loop will loop forever, or worse, will confidently claim done when it isn’t.
• “The user is happy.” There is no user in a run-until-done loop by definition.
• “The model thinks it’s done.” Self-declared completion, unverified, is the same failure mode.

The two-part safety net

Every serious run-until-done loop pairs the stop condition with a budget:

Stop when tests pass, OR after 20 cycles, whichever comes first.

The budget prevents the runaway. If the tests never pass, the loop still stops eventually, and you get a report (“gave up after 20 cycles; tests still failing on test_login_flow”) instead of a phone bill.

Common budgets:

• Cycle count. “Try up to 20 iterations.”
• Wall-clock time. “Give up after 30 minutes.”
• Token spend. “Stop once we’ve spent $5 on model calls.”
• No-progress detection. “Stop if the last three cycles produced no diff.”

Pick at least one. A run-until-done with no budget is just a bug waiting for someone to notice.

What changes vs live loops

Two things:

• You leave. By definition. Which means every design decision in Chapter 3 (sandbox, verifier, safe tool use) now matters more. Nobody is watching cycle 12 when it decides to delete something.
• You need a report at the end. Not just “did it finish?” but “what did it do?” A summary of what changed, what was tried, and what didn’t work. Without it, the return-and-review step takes as long as watching would have.

When to reach for run-until-done

Run-until-done is the right shape when:

• The goal has a clear pass condition.
• The work is bounded (you can guess how many cycles it might take).
• The cost of a runaway is capped by a budget.
• You want to reclaim the time between “I clicked start” and “the work is done.”

It is the wrong shape when:

• The goal is subjective. Use a live loop and stay in the chair.
• The work never converges in your test runs. Fix the design first; automating it won’t help.
• You want it to happen on a schedule (not on your keypress). That’s the next section.

The failure mode nobody catches until it happens

The pathological case is a loop that reports success but shouldn’t have. The stop condition was too weak — “the output file exists” — but the file is empty, or malformed, or contains an error message that happens to be plausible-looking. The loop stops, marks itself done, and moves on. You come back to a green check and shipped garbage.

Two defences: (1) make the stop condition specific (“the output file exists and parses as valid JSON and has at least 10 items”), and (2) include the verifier from 3.4. Do-and-verify is not just for output quality; it’s the last line of defence against a satisfied-with-nothing stop.

Check your understanding

Quick check

—

Which of the following is the WEAKEST stop condition for a run-until-done loop?

1. A"Stop when `pytest` exits with code 0."
2. B"Stop when the output file exists AND parses as valid JSON AND has at least 10 items."
3. C"Stop when the model believes the task is complete."
4. D"Stop after 20 cycles OR when the health-check endpoint returns 200, whichever comes first."

Check answer→