# Add Criterion Benchmarks to a Tokio Server

Yes. An agent adds Criterion benches for your Tokio server's frame parser and a client round trip against an in-process server, builds them in CI, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-criterion-benchmarks/tokio/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads the crate's public surface before it writes a benchmark, because a `benches/` target is an external crate that sees only what `src/lib.rs` exports. In a server like tokio-rs/mini-redis, `Cargo.toml` is edition 2018 at version 0.4.1 with `tokio` 1 on `features = ["full"]`, `bytes`, `clap` 4 with `derive`, `tracing` and `tracing-subscriber`, an optional `otel` feature, one dev-dependency, `tokio` with `test-util`, and no `[[bench]]` section; there is no `benches/` directory. `src/lib.rs` makes `clients::{Client, BlockingClient, BufferedClient}`, `cmd::Command`, `Connection`, `frame::Frame`, `server` and `DEFAULT_PORT` public, while `Db`, `DbDropGuard`, `Parse` and `Shutdown` stay `pub(crate)` or private, so the key-value store cannot be timed on its own. `tests/server.rs` already shows how to stand a server up in-process: a `start_server` helper binds `127.0.0.1:0` and spawns `server::run` with a shutdown future, and the tests speak raw RESP bytes to it. `.github/workflows/ci.yml` runs `cargo build`, `cargo test`, both again with `--features otel`, and `cargo fmt --all --check` on Ubuntu, macOS and Windows.

The agent adds `criterion` as a dev-dependency with the `async_tokio` feature, at the current release it reads from crates.io, and two `[[bench]]` entries with `harness = false`, which Criterion needs because it supplies its own `main` through `criterion_main!`. `benches/frame.rs` is synchronous, which Criterion's own guide prefers for small functions: it builds a `Cursor` over the bytes of a `SET` array frame and of a bulk string of a size taken from the prompt, and benchmarks `Frame::check` followed by `Frame::parse`, the same check-then-parse order `Connection::read_frame` uses, with inputs passed through `std::hint::black_box` so the compiler cannot fold them. `Connection::write_frame` needs a socket, so the encoding side is measured through the round trip instead.

`benches/roundtrip.rs` builds a Tokio runtime, binds `127.0.0.1:0` and spawns `server::run` with a `std::future::pending()` shutdown the way `tests/server.rs` does, connects one `Client`, and uses `b.to_async(&runtime).iter(…)` to time `ping`, `set` and `get` on that one connection, so the number is the cost of a command through `Connection`, the `cmd` module and the `Db` mutex rather than of opening sockets. Criterion's guide says async benchmarks carry more measurement overhead than sync ones, so the description states what each number includes and keeps the `ping` bench as the floor the others are read against. The server task is dropped with the runtime at the end of the group.

CI builds the benches but does not run them, since timings on a shared runner are noise: `ci.yml` gains a `cargo bench --no-run` step on the Ubuntu job, which is what catches a bench that no longer compiles when `Frame` or `Client` changes, because `cargo test` does not build `[[bench]]` targets by default. `cargo fmt --all --check` already covers `benches/`. The README gains a short section with `cargo bench`, `cargo bench -- --save-baseline before` and `--baseline before` for comparing a change, and the note that `html_reports` is an opt-in feature if someone wants the charts. The agent runs `cargo bench --no-run`, `cargo bench` once to see every bench complete, `cargo test` and `cargo fmt --all --check` in its sandbox and keeps the output.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add Criterion benchmarks to this Tokio server. Read Cargo.toml, src/lib.rs, src/frame.rs, src/connection.rs, src/server.rs, src/clients/client.rs, tests/server.rs and the CI workflow first. Add criterion as a dev-dependency at its current release with the async_tokio feature and two [[bench]] targets with harness = false. In benches/frame.rs, benchmark Frame::check followed by Frame::parse on a SET array frame and on a 4 KiB bulk string through std::hint::black_box. In benches/roundtrip.rs, start the server in-process on 127.0.0.1:0 with a pending shutdown future as tests/server.rs does, connect one Client, and benchmark ping, set and get on that connection with to_async on a Tokio runtime. Do not benchmark Db, which is not public, and say so in the PR. Add cargo bench --no-run to the Ubuntu CI job without running the benches in CI, and document cargo bench and baselines in the README. Run cargo bench --no-run, cargo bench, cargo test and cargo fmt --all --check, include the output, and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that adds `benches/frame.rs` and `benches/roundtrip.rs`, and changes `Cargo.toml`, `Cargo.lock`, `.github/workflows/ci.yml` and `README.md`. Nothing under `src/` changes, and the runtime dependency list is the same; Criterion is a dev-dependency only.

The description explains why `Db` is out of reach, what each round-trip number includes, and why CI only builds the benches, and quotes the `cargo bench --no-run`, `cargo test` and `cargo fmt` output with the list of benches that completed. The commands and their full output are in the transcript at `GET /o/{org}/sessions/{id}/messages`.

Before merging, a reviewer runs `cargo bench` on a quiet machine to get a first baseline worth keeping and decides whether `html_reports` should be on. A follow-up message to the same session can add a bench for `BufferedClient`, a `publish` and `subscribe` pair, or a parametrised set of bulk sizes.

## Run it on every Tokio crate with a hot path and no numbers

A team with several Tokio services sends one `POST /o/{org}/sessions` per repository with this prompt as `input` and the repository in `resources`. Each run reads its own `lib.rs`, so a crate whose parser is private gets its benches written against the public type that wraps it, and one that already has a `benches/` directory gets new files in the same style.

Send a `request_id` that is a UUID v5 of the repository's full name, so a retried call returns the run it already started. The key's `webhook_url` reports when each run the key starts finishes, with `session_url` for the pull request link and the bench output.

## Why run it on Rush?

The hard part of a first benchmark is choosing what it can honestly measure: the store is crate-private, the encoder needs a socket, and an async bench carries runtime overhead that a sync one does not. The agent reads the visibility of every type and the way the tests start a server, picks the measurable paths, and runs every bench to completion in a sandbox of its own, with the output kept.

The pull request carries the benches, the CI step that keeps them compiling and the README on baselines, so the next performance change starts from a number rather than a guess. Several crates can be done at once, each in its own sandbox with its own pull request.

## Frequently asked questions

### Why not benchmark Db directly?

`Db` is `pub(crate)` in `src/lib.rs`, and a `benches/` file is compiled as a separate crate that only sees public items. Making it public for a benchmark would change the library's API, so the round trip measures it through `Client` instead; say so if you would rather expose it.

### Will benchmarks run in CI?

They are built with `cargo bench --no-run` so a broken bench fails the pull request, but not run: timings on a shared runner vary too much to compare. Run `cargo bench` locally with `--save-baseline` before a change and `--baseline` after.

### Does this change the runtime dependencies?

No. Criterion is a dev-dependency, so `cargo build` and the binaries are unaffected; only `Cargo.lock` grows with its tree. The `otel` feature and its optional crates are untouched.

### Why run a real server for the round trip?

Because that is the path a client pays for: frame encoding in `Connection`, the command dispatch in `cmd`, and the `Db` mutex. Starting it on port 0 inside the bench, as `tests/server.rs` does, needs no running process and no fixed port.

### How do I compare before and after a change?

`cargo bench -- --save-baseline before` on the current commit, make the change, then `cargo bench -- --baseline before`; Criterion prints the change per bench with its own noise estimate. The README section the agent writes has those two commands.

## More Tokio jobs

- [Add Dependabot to a Tokio Project](https://getrush.ai/run/add-dependabot/tokio/)
- [Write a Dockerfile for a Tokio Server](https://getrush.ai/run/add-dockerfile/tokio/)
- [Add Fuzz Tests to a Tokio Protocol Parser](https://getrush.ai/run/add-fuzz-tests/tokio/)

## The same job on other stacks

- [Add Criterion Benchmarks to a Rust Workspace](https://getrush.ai/run/add-criterion-benchmarks/rust/)
- [Add Criterion Benchmarks to a Rust CLI](https://getrush.ai/run/add-criterion-benchmarks/rust-cli/)

## More jobs like this: Hand off a long task

- [Schedule Database Jobs With pg_cron on Supabase](https://getrush.ai/run/add-cron-jobs/supabase/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
