# Add Benchmarks That Catch Performance Regressions in a Python Library

Yes. An agent writes pytest-benchmark cases for a library's hot paths, adds a workflow that compares a PR against its base on one runner, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-benchmarks-to-ci/python/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent clones the library into its own cloud sandbox and reads how it is tested before it writes a benchmark. In a package like encode/httpx, `pyproject.toml` builds with hatchling, requires Python 3.9 or newer, depends on `certifi`, `httpcore`, `anyio` and `idna`, and configures pytest with `-rxXs` and `filterwarnings = ["error"]`, mypy in strict mode with an override for `tests.*`, and coverage that includes `httpx/*` and `tests/*`. `scripts/coverage` enforces `--fail-under=100`, so any file placed under `tests/` must itself be fully covered. `requirements.txt` pins the tooling exactly, `pytest==8.4.1`, `coverage[toml]==7.10.6`, `ruff==0.12.11`, `mypy==1.17.1`, and installs the package with every extra. `.github/workflows/test-suite.yml` runs `scripts/install`, `scripts/check`, `scripts/build`, `scripts/test` and `scripts/coverage` across Python 3.9 to 3.13. There is no benchmark anywhere: no `benchmarks/` directory, no `pytest-benchmark`, no timing in CI. The package's own modules show where time goes: `_urlparse.py` and `_urls.py` for URL parsing, `_models.py` for `Headers`, `_content.py` and `_multipart.py` for request bodies, `_decoders.py` for gzip, brotli and zstd, and `_client.py` for the request path.

It creates a `benchmarks/` directory, deliberately outside `tests/` so the 100 percent coverage floor and the `tests.*` mypy override are untouched, and adds a pinned `pytest-benchmark` line to `requirements.txt` in the style of the other tools. Each benchmark is a small pytest function taking the `benchmark` fixture and timing one operation on a fixed input: parsing a URL with a query string and an internationalised host, building a `Headers` object from a list and looking up a key case-insensitively, encoding a multipart body with a file field, decoding a gzip and a zstd response body of fixed size, and a full `Client.get` through `httpx.MockTransport` so the client's own overhead is measured without a socket. Inputs are built once at module level, not inside the timed function, and every function has a docstring saying what it measures and which module it exercises.

A benchmark that only prints numbers is noise, so the agent makes the comparison meaningful. It adds `scripts/benchmark`, following the repository's `scripts/` convention, which runs `pytest benchmarks --benchmark-only` and accepts a `--save` or `--compare` name. In a new `.github/workflows/benchmarks.yml` on `pull_request`, a single job on one Python version checks out the merge base into a `git worktree`, runs the benchmarks there with `--benchmark-save=base`, then runs them on the head with `--benchmark-compare` against that saved run and `--benchmark-compare-fail=mean:20%`, so a hot path that got a fifth slower on the same machine in the same job fails the check, while the ordinary drift between runners does not, because base and head are measured back to back on one runner. The results table is written to the job summary.

It keeps the new code to the project's own standards and proves it in the sandbox. `benchmarks/` is added to `SOURCE_FILES` in `scripts/check` so `ruff format --diff`, `ruff check` and `mypy` cover it, with a `benchmarks.*` mypy override mirroring the one for tests. The agent runs `scripts/install`, `scripts/check`, the full `scripts/test` to show the suite still passes with the new dependency, then `scripts/benchmark --save=first` and a second run comparing against it to show the comparison and the fail threshold work. It also runs `pytest benchmarks --benchmark-disable` to show the benchmark functions pass as plain tests, which is what the matrix job can run cheaply if the maintainers want them exercised on every Python version.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add performance benchmarks to this Python library. Read pyproject.toml, requirements.txt, every script under scripts, the workflows under .github/workflows and the layout of the package and tests first, and name the modules where time is spent. Add a benchmarks directory outside tests so the coverage floor and mypy overrides are unaffected, pin pytest-benchmark in requirements.txt like the other tools, and write one benchmark per hot path with fixed inputs built outside the timed call and a docstring naming the module it measures; include a full request through the mock transport. Add scripts/benchmark that saves or compares a run, add the benchmarks directory to the files scripts/check lints and type-checks with a mypy override like the tests one, and add a pull_request workflow on one Python version that benchmarks the merge base and the head on the same runner and fails when a mean regresses more than twenty percent. Run install, check, the full test script, a saved benchmark run and a comparison run, and the benchmarks with --benchmark-disable, then open a pull request with the results table and the command output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request adding a `benchmarks/` directory with one module per area, `scripts/benchmark`, a pinned line in `requirements.txt`, a `benchmarks.*` override in `pyproject.toml`, one word in `scripts/check` and the new `.github/workflows/benchmarks.yml`. Nothing under `httpx/` or `tests/` changes. The description lists each benchmark, the module it measures and the input it uses, and shows the comparison table from the two sandbox runs.

The run's record keeps `scripts/check`, the full `scripts/test` and `scripts/coverage` output proving the suite and the coverage floor are intact, the saved and compared benchmark runs, and the `--benchmark-disable` pass, so a reviewer sees the threshold mechanism work before it guards a real pull request.

Before merging, a reviewer decides whether twenty percent is the right threshold for the first weeks, whether the benchmark workflow should be a required check or informational while the numbers settle, and which of the measured paths matter enough to keep in a short list that stays fast to run.

## Run it on every pull request that touches a hot path

The workflow runs on every pull request once merged. The agent's recurring job is the reading: a team starts one run per pull request that changes the parsing, headers, content or decoder modules from an automation with a `github` trigger on `pull_request.opened` and `pull_request.synchronize`, in the Project that holds the library, with a prompt asking it to read the comparison table, profile the slowed benchmark with `cProfile` in its sandbox, and post a review naming the function where the time went.

The trigger adds the pull request after the prompt, each run works in its own sandbox on that branch, and only work by an owner, member or collaborator fires it; a pull request from a fork never does. An automation run has its own `max_runs_per_hour` cap.

## Why run it on Rush?

Benchmarks are easy to write and hard to make trustworthy, and the trust comes from the details: inputs built outside the timed call, base and head measured on the same machine, a threshold wide enough to ignore runner noise and tight enough to catch a real regression. The agent works those out in a sandbox where it can run the comparison several times and show the spread, and the pull request carries that evidence instead of a single number.

A library with a 100 percent coverage floor and strict mypy has conventions a drive-by contribution tends to break. The agent reads `scripts/check` and `pyproject.toml` first and fits the new directory into them, so the pull request passes the existing checks on the first run, and the record shows every one of them passing.

## Frequently asked questions

### Why not put the benchmarks under tests/?

Because `coverage` includes `tests/*` and `scripts/coverage` fails under 100 percent, so a benchmark module would have to be fully covered by the ordinary test run, and the `tests.*` mypy override would apply to it. A separate `benchmarks/` directory keeps both rules exactly as they are.

### Will runner noise fail pull requests at random?

The workflow measures the merge base and the head in the same job on the same runner, back to back, so the comparison is relative, not against a number stored weeks ago. A twenty percent threshold on the mean leaves room for the jitter that remains; the pull request shows the spread from the sandbox so a reviewer can tune it.

### Does this slow down the test matrix?

No. The benchmarks run in their own workflow on one Python version. The matrix can optionally run them with `--benchmark-disable`, which executes each function once as a plain test in well under a second, to catch a benchmark that stopped working after a refactor.

### What does the mock transport benchmark measure?

The client's own work per request: building the request, merging headers and cookies, running hooks, constructing the response, with `httpx.MockTransport` returning a fixed response and no network involved. It is the number that moves when `_client.py` changes.

### Can we see trends over time?

Each run's results table goes to the job summary, and `--benchmark-save` writes JSON under `.benchmarks/`. The pull request keeps that directory out of git and notes that uploading the JSON as a workflow artifact, or to a benchmark tracking service, is the follow-up if a history is wanted.

## More Python jobs

- [Add Test Coverage Reporting to Python CI](https://getrush.ai/run/add-coverage-to-ci/python/)
- [Fail CI on Sphinx Warnings and Broken Links for a Python Library](https://getrush.ai/run/add-docs-build-check/python/)
- [Add the Latest Python Release to a Python Library's Support Matrix](https://getrush.ai/run/add-latest-python-support/python/)

## More jobs like this: Write the missing tests

- [Add chart-testing and a kind Install Test to a Helm Chart's CI](https://getrush.ai/run/add-chart-testing/helm/)
- [Add Coverage Thresholds to a Jest Suite](https://getrush.ai/run/add-coverage-thresholds/jest/)
- [Add Coverage Thresholds to a Mocha Suite With nyc](https://getrush.ai/run/add-coverage-thresholds/mocha/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
