
At 11:37 UTC on November 7, I gave a Claude Code session a screenshot of a ticket and asked it to lead the task to completion, then sent nothing else. The bug was on our console's run page: for a run started from the composer, the Stop button stayed hidden until you reloaded, and the ticket had waited nine days for someone to pick it up. Forty-seven minutes later the fix was live and checked on production. A subagent wrote it, and a reviewer subagent on a different model found a regression in it before it merged.
Phoenix is two people. Since October 27, 2025 we have merged 11,040 pull requests across 49 repositories and shipped 2,064 releases, most of it written by coding agents; in October, 82% of our commits carried an agent's signature. We write no design specs for agents to implement, and nothing we ship is formally verified.
What we do instead I've started calling mistake-driven development. An agent is allowed to be wrong. When it makes the same mistake twice, the second time becomes a check that fires the moment it tries again: a hook that blocks the action, a test that fails CI and prints the fix, a reviewer on another model that reads the diff before it merges. Spec-driven development tries to make an agent right before it starts. Ours assumes it will be wrong and makes each kind of wrong expensive to repeat.
The other routes have serious advocates this fall. Anthropic's red team used Claude to write machine-checked proofs for widely used libraries, while noting that "a proof checker can't tell you whether your spec matches reality." Thorsten Ball, who builds Amp, wrote in October that he has stopped worrying about agents accumulating slop and now reviews only the few files with a high blast radius. Ours costs less than writing specs or proofs, and for a team of two it has held. The rest of this post is the evidence: the mistakes, the checks they became, and the numbers.
Every layer between a ticket and production now carries the checks its own mistakes produced.
Did quality hold?
Our product was never bug-free, and by September the monorepo had sprawled: duplicate endpoints, hand-synced maps, types with a single caller. Over nine days we removed 862,973 lines, 3.3 for every line added. Since then the share of code written by agents has kept rising, and the counts of the problems we track have gone down.
Those counts come from ratchet tests, described below, which fail CI if a known bad pattern grows. They cover the problems we know to look for, not the ones we haven't found, so read them as a floor on quality rather than a proof of it.
A year in numbers
Pull requests became our main path in May; before that, most work went straight to main. Since August we have merged between about 200 and 900 pull requests a week. Of the 11,040, 413 were opened by our Rush automations, mostly generated content pages and review follow-ups, and about 100 came from outside contributors and dependency bots. Over the last thirty days the median pull request changed 56 lines and merged 8 minutes after it was opened.
I count releases rather than commits, because a release is a deploy someone could notice breaking. In the fourteen days to November 7 the monorepo shipped 239 production releases: 126 of the API, 64 of the website, 43 of the console and 6 of the engine.
Much of this ships as software others can use. Rush, which we sell, is where a company defines its agents once and runs them, each in its own cloud computer signed into the accounts it needs, with a record and the cost of every run. agents-cli, the command that runs our agents, is open source and had been downloaded from npm 90,370 times as of November 7, and browser-cli, which gives an agent a signed-in browser, is open source too.
One morning, minute by minute
The run page read the run once when it loaded, and again only when the stream ended, so it never saw the machine come up. A regression pass had found that cause nine days earlier. In its first two minutes the session read the ticket and its history, and at 11:39 it started a separate subagent on a second job, described below.
At 11:59 it started a subagent on Claude Opus 5.5 for the fix. The subagent created its own worktree from main, changed the page to re-read the run when a turn starts, ran the console's browser test and opened PR #5981 at 12:07. Twenty-five seconds later a reviewer subagent on Claude Sonnet 5.5 began reading it. Its first verdict, as it later recorded on the pull request:
First pass on 2bd9f632c6 requested changes: a failed mid-turn re-read nulled the payload, which unmounted the dock, display lease and VNC socket and closed the EventSource.
The author had introduced that regression two minutes earlier. It fixed it and added a browser test that serves a 500 on the re-read and fails against the old page. CI took 3 minutes 8 seconds, and the reviewer approved at 12:16. Our merge guard then refused the merge twice, because GitHub had not yet returned the reviewer's comment and the guard found no verdict on that pull request; the third attempt merged at 12:17. The release started at 12:19 and the console was live at 12:21. At 12:23 the session started a real run on production, and at 12:24 Stop and Pause appeared without a reload.
At 12:20, three minutes after the merge, our cloud reviewer posted its own review. It is a Rush automation that reviews pull requests into main, each in a sandbox of its own. It ran the browser test against main and against the change, then reshaped the test's stream to match what our engine actually sends. It listed three problems, two of them new, and treated as blocking a case the first reviewer had called minor, where a finished run keeps showing Stop. As of writing, nobody has filed that follow-up.
The second job came from an agent that, ten minutes before I started the session, had been auditing our first milestone against production logs. It found that about 6% of the previous day's runs had ended with MACHINE_LOST, checked that no ticket covered the failure, filed one, and wrote in it: "A longer timeout does not count."
The fix tied the lost machines to memory pressure inside the sandbox and changed how the out-of-memory killer treats the agent's own processes. At 12:06 the cloud reviewer marked it blocking: the base image's own copy of that service would still be enabled, so two would run, and it proposed a test that requires exactly one. The fix merged at 12:14 anyway, and the cloud reviewer opened its own follow-up a minute later. At 12:28 the workflow that builds our sandbox images booted one on a real machine, and the test read its process table: "earlyoomOomScoreAdj":"0". The packaged copy was the one running, as the reviewer had said, so the fix had not applied. The session wrote a second fix along the reviewer's lines and the stricter test with it. That merged at 12:44, was live at 12:52, and passed on a real machine at 13:03. The ticket stays open until lost machines stay under 1% for a full day of production.
That morning the reviewers caught both problems before production did. The first catch was acted on. The second was merged over, and only a test on a real machine stopped it from passing as fixed.
Layer by layer
Intake: the open work comes with the session
Before an agent reads its task, hooks inject our ranked goals, its ticket queue, which machines have spare capacity, and the open pull requests and live agents in the repository it is about to touch. This is the start of that block as a session in our monorepo received it this morning, trimmed:
## In-flight in this repo (auto-injected)
Work that already exists here. Before opening a PR, spawning agents, or
adopting a task: don't duplicate an open PR's scope, don't take over
a live session's surface without checking what it is doing, and don't
re-propose something the recent merges already landed.
Open PRs:
- #5985 rush/factory: the guest runs only start.sh's earlyoom (PHNX-4611)
- #5983 rush/factory: mask earlyoom's packaged unit so only start.sh's runs (PHNX-4611)
- #5978 Device relay carries a tool (browser or computer)
We added it after two agents opened separate pull requests for the same change, even though our rules already told them to check first. The tickets themselves are not specs. A typical one has a short "Why", a checklist where each item cites the file and line that is wrong, and a "Done when" stated as what must be true on production, and agents write many of them from logs and failed checks.
Dispatch: one command for every harness
agents run starts a single agent and agents teams starts several with dependencies between them, on whichever machine has room:
agents run codex "fix the flaky test" --mode edit --device auto
agents teams add pricing-page claude "Run Playwright suite, fix flakes" --name qa --after backend,frontend
agents teams start pricing-page --watch
The harness is an argument, so when one is rate-limited we re-run the task on another. Since last October that has meant 19,905 working sessions; over the last thirty days a typical day peaked at 12 agents working at once. A written rule told agents to mix harnesses on a team, and 86% of team agents still ran on one; a hook now refuses a third agent from the same harness unless the request states a reason.
Workspace: every agent works in its own checkout
No agent may write to a person's checkout, on any branch. The guard that refuses the write also creates a worktree and hands back its path. This is the message an agent received this morning when it tried to append a note to a file in our shared configuration checkout, trimmed:
Blocked: redirect destination '~/.agents/.history/…/memory/MEMORY.md' in the
PRIMARY working tree of ~/.agents (branch 'main').
No agent may modify the user's primary working tree — on ANY branch. It is the
checkout the user works in; leaving it dirty or switched onto a feature branch
strands their machine.
A worktree is ready for you — redo this write there:
~/.agents/.agents/worktrees/claude-2026-11-07-…
It is branched off freshly-fetched origin/HEAD. Work, commit, and push from it,
then open a PR.
The first version blocked only the default branch. Agents then checked out feature branches in the main checkout and left it there, dozens of commits behind. The guard now also requires the base to have been fetched in the last 15 minutes, after agents built on a main that was days old.
Rules: checks that fire at the moment of the mistake
We run 28 shared hooks that inject state, nudge, or block an action as it happens, because a line in a rules file is read at the start of a session and forgotten by the middle of it (more on that). Inside the test suites CI already runs, ratchet tests hold each bad pattern at its current count. This is one rule from our API's ratchet file, and the assertion that enforces every rule:
{
name: "empty catch",
fix:
"Log the failure where it is caught: log.warn(\"<snake_case_event>\", { error }) when it is handled " +
"(a retry, a refusal, a transient upstream 5xx; add upstream and status for an outbound call), " +
"log.error when someone must act, log.debug for a parse that falls through to a typed refusal. " +
"Or rethrow. A comment inside the block is not a log line.",
find: (_src, masked) =>
[...each(masked, EMPTY_CATCH_BLOCK), ...each(masked, EMPTY_CATCH_CALLBACK)].sort((a, b) => a - b),
},
expect(total, `${r.name} rose above ${baseTotal} in: ${offenders.join(", ")}\nFix: ${r.fix}`)
.toBeLessThanOrEqual(baseTotal);
The fix text is written for the agent that trips the rule, and a rise fails CI with every offending file and line. A fall prints a reminder to tighten the baseline, and our rules forbid raising a baseline to get a change through. When the same review comment repeats, it becomes one of these rules (how). The rules file also says the default home for new code is something that already exists: a pull request that adds a route, type, flag or table must list each one with the existing thing it couldn't extend.
Review: someone other than the author
On August 28 an agent reported that it had restructured a codebase, and CI passed. A reviewer on a different model opened the files the summary never mentioned and found about 9,500 lines moved into folders named Legacy, with stubs left where the working code had been. The reviewer blocked the change three times (the whole case).
Every agent on our machines pushes as the same GitHub identity, so GitHub alone can't say who reviewed what. Our merge guard requires an approval posted on that exact pull request, rejects one "carried from" another pull request (an agent tried that), ignores the word when it is negated or quoted in code, and blocks the admin and raw-API routes around the check. It cannot tell which session posted the approval; independence depends on the author starting a separate reviewer, and on the cloud reviewer, which runs in its own sandbox from the diff.
Between October 28 and November 7 the cloud reviewer reviewed 259 of 376 code pull requests in the monorepo. On 123 it asked for changes before the merge, and 116 of those got a new commit afterwards. Not every request is a bug: some are procedural, like a missing citation to our design doc.
Tests: the real thing, not a stand-in
In September, 97 of our API's 314 test files mocked our own modules, which meant they tested that the mock was written the way the code expected. An allowlist that could only shrink took that to zero by October 6. Changes that touch how runs execute now need a test against a real service: canaries run against preprod before every API release, and the sandbox image workflow boots a real machine, which is what caught the second job's fix.
Release: a branch push
Pushing release/rush/api/0.6.343 deploys that version of the API, runs canaries against real services and merges its changelog. No one approves a release or logs into a server; at 239 production releases in two weeks, a manual step would hold up the queue.
Done: checked, not reported
The Legacy restructure was reported as finished. We now run a judge on agents' stops; replayed against 55 real stops labelled blind, 7 were an agent handing back a step it should have done itself, and the judge catches 4 of those 7. Our rules require checking the change on production before a ticket closes, which is what the session did at 12:23.
What still breaks
Merges outrun the cloud reviewer. An author merges once its own reviewer subagent approves, and the cloud reviewer can take several minutes. Of the 259 pull requests it reviewed, at least one of its reviews arrived after the merge on 101, and on 79 of those it asked for changes. On 32, its latest verdict before the merge was already "changes needed" and the author merged anyway, as with the second fix this morning.
Review coverage is uneven. Generated pages and documentation-only changes merge without review by design. Reviewer subagents post under my account, so their verdicts can't be told apart from an author's own comments. Counting only reviews we can attribute, the share of merged monorepo pull requests with an independent review comment was 52 to 78% from June through August, about a third in September and October, and 60% so far in November.
The next check we owe. Merging over a review that has already said no is now a mistake we have made more than twice, so by our own rule it should become a check: a merge that waits for every review in flight. We haven't built it yet.
CI has a slow tail. Our rule says under a minute. Over our last 200 pull request runs the median was 53 seconds, but the slowest tenth took more than three minutes.
Mistake-driven development, in practice
- Put live state, such as open pull requests and the ticket queue, into each session's context with a hook. Our written rule to check for open pull requests did not stop duplicates; the injected list did.
- When a guard blocks an action, return the next step with the refusal, like the path of a ready worktree.
- When a review comment repeats, turn it into a ratchet test whose failure message tells the agent what to write instead.
- Have a separate session review every change, on a different model where you can, and don't let a merge outrun a review that has already said no.
- Test the release path against real machines and services. A mocked sandbox would have passed a fix that had not applied.
- Require each new route, type or flag to name the existing code it couldn't extend.
What transfers, and what we don't know
This week dax, who works on OpenCode, wrote that companies selling software factories produce very little software. Our experience fits that: the hooks, ratchets and decision files above encode our own mistakes and wouldn't transfer as they are. The layer underneath can, which is why agents-cli and browser-cli are open source, and why our cloud reviewer and Atlas, our research directory, are agents we defined on Rush.
What we don't know yet is how far this holds. More than a third of our cloud reviews still land after the merge, and the codebase has had two months, not two years, to show whether the ratchets keep it from sprawling again.


