01The day my agents approved their own work
For months the setup was simple. One agent drafts, I skim, it ships. Fast, and wrong more often than I admitted. Invented numbers that read fine. A file quietly moved out of the folder it was told to stay in. "Looks good" at the end of every run.
None of those were failures of the model. They were failures of the setup. Nobody in it was allowed to say no, so nothing was ever stopped. The draft was the decision.
My company runs on an AI operator: scheduled jobs, agents, and one assistant that handles the day's messages with me approving anything that sends or spends. More than 100 scheduled jobs, 34 of them AI agents. When that many things run while you sleep, "I skim it" is not a control. This guide is the set of controls I built instead, in the order I built them, with the numbers from the weeks they went in.
If you run agents on real work, the question worth answering is this: who, in your setup, is allowed to say no?
02What a gate is
A gate is a stop, not a review step. A review step is a place where someone may look. A gate is a place where nothing passes until someone does.
In my setup the gate is one typed word. An agent prepares the action, writes a card with the evidence, and waits. I type the word, the action runs. I do not type it, the action never runs, and the card expires. There is no third state where the agent decides the wait was long enough.
An example of the difference. My outreach drafts arrive as cards in a channel. If I reply "nice" under one, nothing happens. If I reply "send" under one, the message goes, once, and the card records the word, the time and the read-back from the provider that proves the message is ours. A second "send" on the same card does nothing, because the card is spent. The agent that drafted the message never had the sending function; a separate script, which cannot draft, does the sending, and only when it finds my word on a card it did not write.
Three properties make it a gate rather than a habit:
- It is in the code path, not in the instructions. The agent cannot reach the sending function without the approval token. Telling an agent "always ask first" is a request. Removing its ability to send is a gate.
- It is typed, not inferred. A thumbs up in a thread, a "sounds good" in passing, a message that mentions the draft: none of those count. One exact word, from me, on the card for that action.
- The gate cannot be edited by what it gates. The settings, the hooks, the allow lists and anything that sends are out of the agents' reach by construction. An agent that can rewrite its own gate has no gate.
03Where the gate sits
Not everything needs one. Put a gate on everything and the humans stop reading the cards, which is worse than no gate. I draw the line by what the action can reach.
The four actions that need a gate:
- Send. Anything that reaches a person outside the machine: a message, an email, a post, an application. The most common one, and the one where a wrong send cannot be unsent.
- Spend. Any call that costs money beyond a cap: a paid API, an ad budget, a purchase. Caps catch the runaway; the gate catches the single expensive mistake.
- Delete. Anything that removes data or history. A deletion is a send to the past.
- Rule change. Any edit to the agent's own instructions, allow lists or settings. The one most setups forget, and the one that quietly removes the other three.
The actions that do not need one: reading, drafting, scoring, summarising, planning. They touch nothing outside the machine, so a wrong result costs a retry, not a customer. Let those run free, and log them.
04The agent whose only job is to refute
The gate stops the action. It does not make the draft good. For that I added a second agent to every batch with one job: refute.
It has to open every file the first agent cited, run the tests, and find at least one real problem before it may say anything else. It reads the files bottom up, so it does not inherit the drafter's story about what the code does. A finding that two reviewers hit gets promoted a level. The output is block, concerns or clean, with the line of evidence.
What changed when it went in:
- Silent errors caught before I saw the output: invented numbers, moved files, skipped tests.
- My own bias checked. The reviewer does not know what I hoped the draft would say.
- A verdict I can act on. Block, concerns or clean, with evidence, instead of a paragraph of praise.
- Fewer things built twice. Two of my nine "we need this" calls that day were flipped to "we already have it".
The rule that makes it work: the refuter has to find at least one real problem before it may say anything else. A reviewer allowed to approve for free will approve for free.
05Measure the operator, not the vibes
For months my operator got better the only way most of these setups do: I corrected it by hand, and hoped the correction stuck. That is not a system. You cannot improve a thing you do not measure, and "it feels better this week" is not a measurement.
So every Monday a scorecard reads the week's replies and counts my corrections per hundred, by kind:
- An unverified claim. It said "sent", "live", "verified" without a tool result behind the word.
- A misread intent. It did a thing, not the thing I asked.
- Did not act. It asked me to do something it could have done itself.
- A format slip. The right answer in the wrong shape.
- A missed memory. It failed to check something it had already been told.
It also replays a fixed set of golden questions with known answers, the dates, the ids, the prices, the names that must always come back right. A model swap or a lost context shows up there first, as a wrong answer to a question that was right last week.
The number is small. That is not the point. The point is that it is a number, with a kind attached, so the next step can aim at the biggest kind instead of at a feeling. In that first week the biggest kind was "did not act": the operator asking me to run a command it could have run itself.
The hunch the scorecard killed
I built the scorecard because I was sure of something: the operator got worse every time I switched the underlying model, and the hour after a switch was where the mistakes lived. It felt obvious. Every bad reply I remembered seemed to sit near a switch.
The first study ran over two weeks of transcripts and binned every correction by model and by the hour after a switch. The corrections did not cluster after a switch at all. The hunch was wrong, and I had been about to spend days building a guard against it. That is what "measure the operator, not the vibes" costs you: the stories you were sure of. It is also what it saves you.
Two more rules came out of that study. A correction has to be judged, not matched by a regex: "why" in a message is often frustration or a question, not a correction, and the first regex pass counted sixteen where a judge found about twelve real ones. And a correction has to be bound to the reply it corrects, or the count drifts toward whatever the owner was annoyed about that week.
06Let the system propose, never apply
Once you have the scorecard, the tempting move is to let the agent fix itself. Don't. Let it propose. The top recurring kind becomes one proposed rule change, at most fifteen lines, to one file, written as a diff with a prediction attached: which number should move, and by how much.
Before I ever see it, the change is replayed. The golden questions and five of the week's real incidents run with and without the rule, graded by a cheaper model. The edit must fix at least most of the incidents that failed without it and break none of the golden answers. If the replay cannot tell the difference, the card says so, and the decision stays with me.
Then it lands on my desk as a card: the evidence in three lines, the diff, the prediction, the replay table, and the rollback line. One typed word applies it. Nothing lands without that word.
Two scorecards later the loop reports whether the predicted number moved. Better, same, or worse, and worse carries the rollback. A change only counts as kept after that.
Three rules keep it honest:
- One change in flight at a time.
- A change may add or clarify a rule, never delete one without my explicit word.
- The loop cannot touch its own gates: settings, hooks, allow lists and anything that sends are out of its reach by construction.
07The card template
Every gated action and every proposed change arrives in the same shape. The shape matters more than the content: when every card looks the same, a missing line is visible, and a human can decide in under a minute.
CARD <id> · kind: send | spend | delete | rule change EVIDENCE (three lines, each one a fact with where it came from) 1. … 2. … 3. … THE ACTION (the exact message, amount, path or diff, nothing paraphrased) … PREDICTION which number moves, and by how much, by when REPLAY golden: n/n before → n/n after · incidents: fixed n, regressed n ROLLBACK the one command that undoes it WORD type "approve <id>" to run it · the card expires at <time>
Two of those lines are the ones people skip. The prediction, because it can be wrong in public. The rollback, because nobody plans to need it. They are the two that turn an approval into a measured decision instead of a yes.
08The checklist
Ten yes-or-no questions. If you run agents on real work and answer no to any of them, that is the next thing to build.
- Can an agent reach a sending, spending or deleting function without a human token in the code path?
- Is the approval one exact typed word, on the card for that action, rather than a reaction or a reply in a thread?
- Can any agent edit its own instructions, allow lists, hooks or settings?
- Does a second agent have to find one real problem before it may call a draft clean?
- Does the reviewer open the files the drafter cited, or does it grade the drafter's summary of them?
- Do you count your corrections per hundred replies, by kind, every week?
- Do you replay a fixed set of golden questions that must always come back right?
- Does every proposed rule change carry a prediction of which number will move?
- Is every change replayed against real incidents before a human sees it?
- Does every card carry the one command that undoes it?
09What it costs and what it saves
The refuter costs one extra agent run per item. On a batch of seven that is seven more runs, each a few cents and a few minutes. It saves me the hour I used to spend finding what it finds, and it finds things I would not have: the moved file, the number that reads fine and is wrong.
The scorecard costs one scheduled job a week and a cheaper model to judge the corrections. The replay costs a handful of headless runs per proposed change: the first card took twenty calls and about twelve minutes to replay, and there is at most one change in flight. The whole loop runs for less than the price of one bad send.
Put a number on the other side. One wrong send to a customer costs the apology, the correction, and the trust that made the automation worth running. One wrong spend is the amount plus the hour spent finding where it went. One deleted record is a restore, if there is a backup, and a conversation if there is not. Against those, a few cents per item for a refuter and twelve minutes per rule change is not a cost. It is the cheapest insurance in the stack.
The real cost is a slower first draft. Nothing in that first batch shipped on the first pass, and one scorer per deliverable is now the rule. For the kind of work where a wrong send reaches a customer, a slower first pass is the feature.
10Where it fails
The gate fails when the human stops reading the cards. Too many cards, or cards with no evidence, and the word gets typed on reflex. The fix is fewer gates on the right actions, and a card shape where a missing line is obvious.
The replay fails when its bar is wrong. The first card my loop produced was denied by an absolute bar, "fix three of five incidents", on a week where only one incident of that kind had failed. The gate now judges against the baseline instead: did it fix what failed, did it break what passed. A gate that can be wrong is fine, as long as it is measured and can be corrected. A gate that is never checked is a ritual.
The scorecard fails when the corrections are not real. A frustrated "why" is not a correction; a re-ask is not a correction. The judge has to bind each one to the reply it corrects, or the number inflates and the loop aims at noise.
And the whole thing fails if the loop can reach its own gates. That one is not a tuning problem. Keep the settings, the hooks, the allow lists and the senders where no agent can write, and the rest can be wrong safely.
The loop is allowed to be wrong, as long as it is measured.
Sources: this guide expands two shorter pieces, I stopped letting my AI agents approve their own work (4 October 2026) and The loop that rewrites my operator's rules (4 October 2026), and the first weekly scorecard of my own operator (week of 27 September to 4 October 2026). The system itself is described on The Operating System and The loop that runs my company.