What the agent does
The agent clones the library into its own cloud sandbox and reads how it is tested before it writes a benchmark. In a package like encode/httpx, pyproject.toml builds with hatchling, requires Python 3.9 or newer, depends on certifi, httpcore, anyio and idna, and configures pytest with -rxXs and filterwarnings = ["error"], mypy in strict mode with an override for tests.*, and coverage that includes httpx/* and tests/*. scripts/coverage enforces --fail-under=100, so any file placed under tests/ must itself be fully covered. requirements.txt pins the tooling exactly, pytest==8.4.1, coverage[toml]==7.10.6, ruff==0.12.11, mypy==1.17.1, and installs the package with every extra. .github/workflows/test-suite.yml runs scripts/install, scripts/check, scripts/build, scripts/test and scripts/coverage across Python 3.9 to 3.13. There is no benchmark anywhere: no benchmarks/ directory, no pytest-benchmark, no timing in CI. The package's own modules show where time goes: _urlparse.py and _urls.py for URL parsing, _models.py for Headers, _content.py and _multipart.py for request bodies, _decoders.py for gzip, brotli and zstd, and _client.py for the request path.
It creates a benchmarks/ directory, deliberately outside tests/ so the 100 percent coverage floor and the tests.* mypy override are untouched, and adds a pinned pytest-benchmark line to requirements.txt in the style of the other tools. Each benchmark is a small pytest function taking the benchmark fixture and timing one operation on a fixed input: parsing a URL with a query string and an internationalised host, building a Headers object from a list and looking up a key case-insensitively, encoding a multipart body with a file field, decoding a gzip and a zstd response body of fixed size, and a full Client.get through httpx.MockTransport so the client's own overhead is measured without a socket. Inputs are built once at module level, not inside the timed function, and every function has a docstring saying what it measures and which module it exercises.
A benchmark that only prints numbers is noise, so the agent makes the comparison meaningful. It adds scripts/benchmark, following the repository's scripts/ convention, which runs pytest benchmarks --benchmark-only and accepts a --save or --compare name. In a new .github/workflows/benchmarks.yml on pull_request, a single job on one Python version checks out the merge base into a git worktree, runs the benchmarks there with --benchmark-save=base, then runs them on the head with --benchmark-compare against that saved run and --benchmark-compare-fail=mean:20%, so a hot path that got a fifth slower on the same machine in the same job fails the check, while the ordinary drift between runners does not, because base and head are measured back to back on one runner. The results table is written to the job summary.
It keeps the new code to the project's own standards and proves it in the sandbox. benchmarks/ is added to SOURCE_FILES in scripts/check so ruff format --diff, ruff check and mypy cover it, with a benchmarks.* mypy override mirroring the one for tests. The agent runs scripts/install, scripts/check, the full scripts/test to show the suite still passes with the new dependency, then scripts/benchmark --save=first and a second run comparing against it to show the comparison and the fail threshold work. It also runs pytest benchmarks --benchmark-disable to show the benchmark functions pass as plain tests, which is what the matrix job can run cheaply if the maintainers want them exercised on every Python version.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add performance benchmarks to this Python library. Read pyproject.toml, requirements.txt, every script under scripts, the workflows under .github/workflows and the layout of the package and tests first, and name the modules where time is spent. Add a benchmarks directory outside tests so the coverage floor and mypy overrides are unaffected, pin pytest-benchmark in requirements.txt like the other tools, and write one benchmark per hot path with fixed inputs built outside the timed call and a docstring naming the module it measures; include a full request through the mock transport. Add scripts/benchmark that saves or compares a run, add the benchmarks directory to the files scripts/check lints and type-checks with a mypy override like the tests one, and add a pull_request workflow on one Python version that benchmarks the merge base and the head on the same runner and fails when a mean regresses more than twenty percent. Run install, check, the full test script, a saved benchmark run and a comparison run, and the benchmarks with --benchmark-disable, then open a pull request with the results table and the command output.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request adding a benchmarks/ directory with one module per area, scripts/benchmark, a pinned line in requirements.txt, a benchmarks.* override in pyproject.toml, one word in scripts/check and the new .github/workflows/benchmarks.yml. Nothing under httpx/ or tests/ changes. The description lists each benchmark, the module it measures and the input it uses, and shows the comparison table from the two sandbox runs.
The run's record keeps scripts/check, the full scripts/test and scripts/coverage output proving the suite and the coverage floor are intact, the saved and compared benchmark runs, and the --benchmark-disable pass, so a reviewer sees the threshold mechanism work before it guards a real pull request.
Before merging, a reviewer decides whether twenty percent is the right threshold for the first weeks, whether the benchmark workflow should be a required check or informational while the numbers settle, and which of the measured paths matter enough to keep in a short list that stays fast to run.
Run it on every pull request that touches a hot path
The workflow runs on every pull request once merged. The agent's recurring job is the reading: a team starts one run per pull request that changes the parsing, headers, content or decoder modules from an automation with a github trigger on pull_request.opened and pull_request.synchronize, in the Project that holds the library, with a prompt asking it to read the comparison table, profile the slowed benchmark with cProfile in its sandbox, and post a review naming the function where the time went.
The trigger adds the pull request after the prompt, each run works in its own sandbox on that branch, and only work by an owner, member or collaborator fires it; a pull request from a fork never does. An automation run has its own max_runs_per_hour cap.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Benchmarks are easy to write and hard to make trustworthy, and the trust comes from the details: inputs built outside the timed call, base and head measured on the same machine, a threshold wide enough to ignore runner noise and tight enough to catch a real regression. The agent works those out in a sandbox where it can run the comparison several times and show the spread, and the pull request carries that evidence instead of a single number.
A library with a 100 percent coverage floor and strict mypy has conventions a drive-by contribution tends to break. The agent reads scripts/check and pyproject.toml first and fits the new directory into them, so the pull request passes the existing checks on the first run, and the record shows every one of them passing.
Frequently asked questions
Why not put the benchmarks under tests/?
Because coverage includes tests/* and scripts/coverage fails under 100 percent, so a benchmark module would have to be fully covered by the ordinary test run, and the tests.* mypy override would apply to it. A separate benchmarks/ directory keeps both rules exactly as they are.
Will runner noise fail pull requests at random?
The workflow measures the merge base and the head in the same job on the same runner, back to back, so the comparison is relative, not against a number stored weeks ago. A twenty percent threshold on the mean leaves room for the jitter that remains; the pull request shows the spread from the sandbox so a reviewer can tune it.
Does this slow down the test matrix?
No. The benchmarks run in their own workflow on one Python version. The matrix can optionally run them with --benchmark-disable, which executes each function once as a plain test in well under a second, to catch a benchmark that stopped working after a refactor.
What does the mock transport benchmark measure?
The client's own work per request: building the request, merging headers and cookies, running hooks, constructing the response, with httpx.MockTransport returning a fixed response and no network involved. It is the number that moves when _client.py changes.
Can we see trends over time?
Each run's results table goes to the job summary, and --benchmark-save writes JSON under .benchmarks/. The pull request keeps that directory out of git and notes that uploading the JSON as a workflow artifact, or to a benchmark tracking service, is the follow-up if a history is wanted.