How I let agents do hard backend work without watching them

The idea

I don't trust an agent until it can prove it's done.

Agents are good at code that looks right. The dangerous case is a change that reads clean, passes a thin suite, and quietly breaks an invariant the tests never touched. On a novel problem the agent has no memory of your retry semantics, your write ordering, or how your pipeline fails, so it invents architecture that looks incredible on paper and is unusable in reality. And the failures that matter only show up under concurrency, a partial write, or crash recovery.

So I stopped trying to get a smarter agent and gave it a finish line it can't move. Before any code exists, I write the tests that define "done". I watch them fail. Then I lock them. The agent works until they pass, and it can't edit them to get there.

I call that finish line the oracle. The rest of the pipeline exists to feed it and check it.

The problem

The scary part isn't bad code. It's architecture that looks done.

AI also comes up with architecture that looks confident and actually pushable or mergeable. But it wasn't always relevant to my product workflow: it holds up on paper, but breaks in reality.

The pain point isn't just the code. It's whether the problem was the right one to solve, whether the system still holds when a write dies halfway, whether someone knew what to kill, and whether the evals would catch a real failure instead of a flattering one. AI is good at producing code that ships. The hard part is everything that decides whether it should.

The pipeline

Five steps. Plain code runs the process, models do the thinking.

The orchestrator is a few hundred lines of ordinary code, no framework. Models only get the parts that need judgment: planning, writing, reviewing. If something can be a rule, it's a rule.

Spec. I write the goal, the limits, and what must never happen. The agent picks a mode. A typo or a UI tweak is quick: no plan, just the existing checks. Everything else goes full. One rule overrides that: anything touching migrations, auth, billing, webhooks or personal data is always full and always waits for me to merge. I listed those paths once, and a match forces it. The model doesn't get to decide how risky something feels.

Oracle. One planning pass writes a short plan and the tests. Every rule that must hold and every way things can go wrong gets a test: a retry, two requests at once, a write that dies halfway. The tests run first and they have to fail. A test that passes before any code exists is either empty or the work is already done, and I want to know which. Concurrency tests send real parallel requests to a real Postgres database, never a mock. Then the tests freeze.

Implement. The agent gets its own branch, the spec, the plan and only the files that matter. It edits, runs every check, reads the failures, and fixes. If the same failure shows up twice, it drops that approach and replans once. If that fails too, it stops and asks me. Each run has a token budget and a clock.

Review. A second model reads the spec, the plan and the diff, never the implementer's reasoning. A fresh reader catches what the author can't see: security gaps, ways to lose data, races the tests missed, work outside the plan. Plain code adds its own flags: an edited frozen test, a weakened old one, a new dependency, a file nobody planned to touch.

Ship. Rebase, rerun the checks, merge. Risky paths are the exception. They open a pull request with the plan, the test results, the review and a five-line "what to check", and I merge it myself.

Where I step in

Two places. Everything else is the agent's job.

The agent asks me when it can't safely go on: the spec can't be turned into a test, it's stuck twice, it ran out of budget, or the next step can't be undone, like dropping data or a force-push. When it asks, it brings what it tried, where things stand, one specific question, and the answer it would pick.

The other place is the merge on risky paths. The agent can run alone. It can't make a decision I can't take back.

Evidence

One early run. A webhook handler.

Agent wanted to mark the event processed and return 200 → oracle failed on a duplicate delivery → had to replan with an idempotency key and a conditional update → final result: duplicate delivery returns the cached response, no second side effect. A normal loop would have shipped the first version. The frozen test is what made it replan instead of paper over.

SuperFlow

Turn one promise into a test.

SuperFlow is the macOS dictation and meeting transcription app I'm building. It runs locally. Take its audio journal. "Add crash recovery" is a task an agent will happily call finished. The real promise is narrower: after a crash, I get back every second of audio that was safely written to disk, and no transcript segment shows up twice.

That sentence is already a test. Write known audio, kill the process halfway, replay the journal, then check that nothing durable is missing and nothing is duplicated. There's a limit too: audio that never reached disk can be lost, and the spec says so. The test checks the promise I actually make, not a bigger one.

A migration works the same way. Start with realistic data, run it forward, check what has to stay true, then run it backward.

What actually matters

The pipeline is the easy part. The work is deciding what's worth building.

An agent will happily build the wrong thing, and build it well. So most of my effort lands before any agent runs: who this is for, what actually hurts them, and what must never happen. That's why the pipeline stops when a requirement can't become a test. Usually the spec is vague, or I haven't understood the problem yet, and a pipeline shouldn't paper over that.

Hard backend work is mostly about what happens when something goes wrong: a retry, two requests at once, a crash halfway through a write, a payment provider that times out. The happy path never takes a system down, so the planning step starts from the failures. Cost and speed get the same treatment. A pipeline that's right but spends full-run effort on a typo isn't one I'd keep. And I keep exactly one record per run, on purpose. I'd rather read one honest file than trust a dashboard I built to look good.

Most of the design is what I left out. No state-machine library, no model scoring how risky a change feels, no separate agent for every concern. The risk rule is a list of paths because a list doesn't have a bad day. Two modes, not five. If a step doesn't catch a real failure, it goes. That restraint is the taste part: knowing what's good enough to trust, and stopping there.

It also has limits. It works when "done" can be written as a test. It doesn't help when the call is taste: whether a screen feels right, whether a product bet is smart. Those need a person. A pipeline that pretends otherwise is just a demo.

Proof

Today this is a design. The evals are what make it a system.

There's no pass rate to show yet. Here's what will tell me whether it works. The plan is 15 to 20 real tasks from my own past work, each with tests the pipeline never sees. Every change to a prompt, a model or a rule runs against all of them, and a change that lowers the pass rate doesn't ship. I start with three tasks, before building the review step or the PR path, then add the hard ones: a race condition, a migration on a table full of data, a webhook that must never run twice, a query that got slow.

Which tasks go in matters more than how many. A few real, messy ones beat fifty invented ones, and a flattering eval is worse than none because it hands you confidence you haven't earned. When the pipeline fails one, the fix is usually a sharper spec or a better test, not more data.

Build order

Smallest useful loop first.

Orchestrator, run record and quick mode. Then the implement loop with its budget and stuck rule. Then the oracle: the red check, frozen tests, a throwaway Postgres. Then the first three evals, before anything else gets added. Review and the PR path come after that, and the full eval set last. Nothing new goes in until an eval shows a failure it would fix.

The goal is simple. Give an agent room to work, and make the evidence strong enough that I can trust what comes back.