Chapter 06 · Section II · 10 min read
6.2 Trust boundaries, human gates, and knowing when to shut it down
Where the loop can go on its own, where a human has to sign off, and the specific signals that mean it's time to pull the plug.
Every action a loop takes sits somewhere on a trust spectrum. Some are safe to do autonomously; some need a human’s sign-off; some should not be automated at all. Getting the boundaries right is what separates a loop you trust in production from one you have to babysit forever — or worse, one you thought you could leave alone until it did something you couldn’t undo.
The trust spectrum
Sort every action the loop can take onto three tiers:
Tier 1 — Fully autonomous. Reversible, low-blast-radius, cheap-to-undo. Draft a summary, tag a ticket, generate a report, run a test. If the loop does it wrong, you notice, you fix, you move on. Nothing external happened.
Tier 2 — Autonomous with review. External but recoverable. Post a comment on an internal PR, file a draft response, update a non-critical database record. If wrong, you can un-comment, un-file, un-update. The audit trail exists but the damage is small.
Tier 3 — Human gate required. Irreversible, external, high-blast-radius. Send a customer email, charge a payment, delete a file, close a ticket for a real customer, deploy to production. If wrong, it’s out in the world.
Every tool the loop has needs to sit clearly on one of these tiers. Confusion about which tier a tool is on is where the worst mistakes come from.
What a good human gate looks like
Not: “the loop pauses forever until someone happens to notice.” That’s not a gate; that’s a stall.
A good gate is:
- A specific person or channel. “Post to
#loop-approvalsSlack channel, ping the on-call engineer.” Not “notify the team.” - Enough context to decide. The pending action, a one-line summary of why the loop wants to take it, a link to the loop’s log for the cycle.
- A clear input mechanism. A button, a Slack reaction, a form. Not “reply and I’ll parse your reply.”
- A timeout with a default. If nobody responds in N hours, either escalate louder or take a safe default action (usually: do nothing and log the timeout).
- A record of who approved. For any Tier-3 action, the log should show which human said yes, when. Even if the human was you.
The specific actions that should almost never be Tier 1
- Anything a customer sees. Emails, chat messages, invoices, receipts. Even if the loop drafts well, review before send. Wrong customer messages compound reputationally.
- Anything a regulator sees. Filings, disclosures, audit responses. Beyond blast radius — legally binding.
- Anything financial. Payments, refunds, credits, charges. Even small amounts, because “small amounts, high volume” is where the real damage hides.
- Anything with
rm,DELETE,DROP,--force. Reversibility is the whole game here. Destructive operations skip Tier 1 by default. - Anything that touches auth, permissions, or access control. A wrong permission grant is not caught by ordinary review; it becomes a security incident weeks later.
Even in mature loops with excellent do-and-verify, these categories default to gated. The verifier catches most mistakes; the human catches the rest.
When to shut it down
Sometimes the right move isn’t to gate more, or add more verifiers, or write a better prompt. It’s to turn the loop off.
Shut down when:
- The failure mode is unbounded. A loop that produces occasional wrong output is a design problem to fix. A loop that could conceivably produce catastrophic output — because a single bad cycle could cause damage measured in weeks, dollars, or lives — should not run while you’re waiting for the fix.
- The verifier is guessing. If your do-and-verify criteria have become so loose that the verifier accepts everything, the check has degraded to theater. Kill the loop; rewrite the criteria; restart.
- You’ve lost the intent. If the paragraph from 6.1 no longer matches what the loop actually does, the loop is running on old intent. Stop, re-align, restart — don’t just let it drift.
- Nobody’s watching. If the owner left the team and no one took over, the loop is running unowned. Stop until someone owns it.
Turning a loop off is not a failure of design. It’s a normal maintenance move. Loops are software; software gets retired.
The dial you always have
At any moment, for any loop, there’s a dial from “aggressive autonomy” to “conservative gating.” Reasonable positions on the dial:
- New loop, first week: everything gated, high friction, lots of human review. You’re calibrating.
- Mature loop, low stakes: most things autonomous, only irreversible actions gated. High throughput.
- Mature loop, high stakes: verifier for everything, human gate for anything external, kill switch obvious and tested.
- Loop under investigation: back to everything gated until the incident is understood.
Move the dial deliberately, in one direction at a time. “Autonomy creep” — quietly moving gated actions to auto over months without a review — is how loops end up doing things nobody would have approved if asked directly.
What to leave the reader with
Loop engineering, at the end of the day, is not primarily about models. Models are a component. Loop engineering is a design discipline: what triggers each cycle, where the work happens safely, how the loop knows things, what verbs it has, who checks its work, and what it remembers. The models get more capable every year; the discipline stays roughly the same.
The two moves that will keep you out of trouble no matter how the models change:
- Design the loop, then decide what’s inside each cycle. Six parts before any prompt.
- Keep intent and accountability human. Everything else can move to the machine.
Everything in this course was one of these two.
Check your understanding
Quick check
—