On August 15 I counted what our agents had done with one tool over the previous 30 days. They started 227 browser tasks and closed 92. They typed the same environment-variable export 1,376 times. At least 18 sessions died on a single error line (our own session index, 30 days to August 15, 2026).
Nobody wrote a test case for any of that. Nobody filed a bug. The agents hit a wall in our tooling, found a way around it, and moved on, and the only record was the transcript each session left behind.
Every session an agent runs is a labelled example of where the harness failed it. Mining your own transcripts finds the failures that matter faster than any benchmark, and it turns capture into evaluation without writing a single synthetic task.
Thirty days of one tool
The tool was the browser command in agents-cli, the source-available CLI our agents use to drive a real browser. Every page action belonged to a named task, and the command needed to know which one. You passed it as a --task flag or set it once in an environment variable. If you did neither, the command printed this and exited (source before the fix, merged August 15, 2026):
No task specified. Pass --task <name> or set AGENTS_BROWSER_TASK in your shell.
Tip: T=$(agents browser start --profile <p>) && export AGENTS_BROWSER_TASK=$T
The tip assumed one long-lived shell, the way a person works in a terminal. An agent's tool calls rarely share one. An export in one call is gone by the next, so the agents did the only thing that worked: they re-exported the variable inline, 1,376 times in a month. When they remembered the flag instead, they passed --task 713 times. When they forgot both, the session stopped on that line, and at least 18 did.
The start and done counts tell the second half. A browser task holds open tabs in a real browser profile until something marks it done. Our agents started 227 tasks and closed 92. The other 135 stayed open, tabs and all, because nothing in the tool closed a task its agent had forgotten.
A benchmark would have scored most of these sessions as passes. The agent got the page, took the screenshot, finished the job. The failure lived in the seam between the agent and our tool, and it showed up as cost: extra shell, extra tokens (the browser cost that compounds, as we found when we benchmarked two browser tools), a dead session now and then, and a browser that filled with tabs nobody owned.
Why a benchmark never finds this
A benchmark measures a model on tasks someone wrote down. The failures that cost us the most were in our harness: the shape of our commands, the defaults we picked, the error messages we printed. No public task suite knows that our browser command wanted a task name, so no public task suite could tell us it was a problem.
The people who evaluate model products for a living reached the same conclusion from the other side. Hamel Husain calls error analysis "the single most valuable activity in AI development," and argues for a bottom-up approach that "forces you to look at actual data and let metrics naturally emerge." In one team's logs he walks through, "just three issues accounted for over 60% of all problems" (A Field Guide to Rapidly Improving AI Products, March 24, 2025).
Agent transcripts are that data, and they are already captured. Each session records the request, every tool call with its arguments, every result, every error, and what the agent did next. A failed call followed by a workaround is a labelled example: here is the input, here is where the harness got in the way, here is what it cost. Nobody had to write the case. The agent wrote it by running into it.
Capture, classify, fix, re-measure
What we run now is a loop with four steps, and only one of them involves writing code.
Capture. Every harness already writes a transcript. The trouble is that each writes its own format to its own folder. agents-cli keeps one index over all of them, so one query searches Claude, Codex, Gemini and OpenCode sessions together (Search every agent transcript from one place, July 17, 2026). That index is what made a 30-day count across harnesses one query. A session you cannot find is a label you cannot count, the same lesson as an agent you can't find.
Classify. Start by counting, and save the model for later. The error string "No task specified" is exact, so counting sessions that printed it is exact. The workaround is exact too: an inline export of one variable name. Counting the expensive workaround next to the failure is what turns a log into a ranking. On August 26 we went wider and had an agent read about three months of our sessions and sort every recurring issue into classes before we chose what to fix next. That list, ranked by how often each issue comes back, is the eval set.
Fix the tool. The tempting fix was a sentence in the agents' instructions: always set the task, always call done. The 1,376 exports are the evidence against it. The agents had the instruction printed in front of them on every failure, followed it, and paid for it on every call after. An instruction moves the cost onto the agent; a change to the tool removes it. So we changed the tool. The same day as the count, the browser command learned to find the caller's task from the session that called it, page commands started creating a task when none existed, and done stopped needing a name (agi-cli #2736, merged August 15, 2026). Earlier that day a separate change gave the background service a sweep every five minutes that closes tasks whose session has exited or that have sat idle past a window, 30 minutes by default (agi-cli #2655, merged August 15, 2026). The error line no longer exists in the code. An agent cannot hit it.
Re-measure. This is the step most teams skip, and it is the step that makes the loop an evaluation. The test for the change is the same query over the 30 days after it: inline exports of the task variable should fall toward zero, "No task specified" should not appear at all, and closed tasks should catch up with started ones. Twelve days in, we have not run that count yet, so this post makes no claim that the fix worked. The data for it is being written right now, by the same agents, one session at a time.
What transcripts cannot tell you
These numbers come from one 30-day sample of our own agents on our own machines, counted once. They describe our harness. Yours will have different walls.
A transcript also records what the agent did, and it is weaker on whether the result was right. An agent that returns a confident wrong answer leaves a clean transcript. Mining finds friction, dead ends and expensive workarounds, the failures the agent noticed and routed around. Finding the failures nobody noticed takes a second check against the result itself: tests, review, a person reading the output. The transcripts tell you where to look first, because they cost nothing extra and they are already there.
The same caution applies to the loop. A count only means something if the thing you count cannot be gamed by the fix. "Fewer error lines" can be faked by swallowing the error. "Fewer workarounds" is much harder to fake, which is why we track the workaround next to the failure.
Where Rush fits
Every Rush run keeps its trajectory: the harness's own session record, captured when the run finishes and stored next to the result, the PR or the files. You can read it whole or as a slice, only the tool calls or only the last few turns, through the same API that started the run. A run that ends without its proof ends as needs_review. Rush does not score them for you. The runs give you the transcripts; today the counting is still yours, the same way it is for us.
The eval set you already have
Most of the talk about evaluating agents is about which model to run and which benchmark to trust. The most useful evaluation we ran this summer was a count over sessions we already had, of a failure no benchmark could have known to include. Demos hide the cost of failure; transcripts record it, one session at a time, whether you read them or not.
The cheapest eval set an agent team will ever have is the one its agents are writing today. Read it before you write a synthetic task, fix the tool it points to, and count again next month.
