All jobs

Add Criterion Benchmarks to a Tokio Server

Yes. An agent adds Criterion benches for your Tokio server's frame parser and a client round trip against an in-process server, builds them in CI, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the crate's public surface before it writes a benchmark, because a benches/ target is an external crate that sees only what src/lib.rs exports. In a server like tokio-rs/mini-redis, Cargo.toml is edition 2018 at version 0.4.1 with tokio 1 on features = ["full"], bytes, clap 4 with derive, tracing and tracing-subscriber, an optional otel feature, one dev-dependency, tokio with test-util, and no [[bench]] section; there is no benches/ directory. src/lib.rs makes clients::{Client, BlockingClient, BufferedClient}, cmd::Command, Connection, frame::Frame, server and DEFAULT_PORT public, while Db, DbDropGuard, Parse and Shutdown stay pub(crate) or private, so the key-value store cannot be timed on its own. tests/server.rs already shows how to stand a server up in-process: a start_server helper binds 127.0.0.1:0 and spawns server::run with a shutdown future, and the tests speak raw RESP bytes to it. .github/workflows/ci.yml runs cargo build, cargo test, both again with --features otel, and cargo fmt --all --check on Ubuntu, macOS and Windows.

The agent adds criterion as a dev-dependency with the async_tokio feature, at the current release it reads from crates.io, and two [[bench]] entries with harness = false, which Criterion needs because it supplies its own main through criterion_main!. benches/frame.rs is synchronous, which Criterion's own guide prefers for small functions: it builds a Cursor over the bytes of a SET array frame and of a bulk string of a size taken from the prompt, and benchmarks Frame::check followed by Frame::parse, the same check-then-parse order Connection::read_frame uses, with inputs passed through std::hint::black_box so the compiler cannot fold them. Connection::write_frame needs a socket, so the encoding side is measured through the round trip instead.

benches/roundtrip.rs builds a Tokio runtime, binds 127.0.0.1:0 and spawns server::run with a std::future::pending() shutdown the way tests/server.rs does, connects one Client, and uses b.to_async(&runtime).iter(…) to time ping, set and get on that one connection, so the number is the cost of a command through Connection, the cmd module and the Db mutex rather than of opening sockets. Criterion's guide says async benchmarks carry more measurement overhead than sync ones, so the description states what each number includes and keeps the ping bench as the floor the others are read against. The server task is dropped with the runtime at the end of the group.

CI builds the benches but does not run them, since timings on a shared runner are noise: ci.yml gains a cargo bench --no-run step on the Ubuntu job, which is what catches a bench that no longer compiles when Frame or Client changes, because cargo test does not build [[bench]] targets by default. cargo fmt --all --check already covers benches/. The README gains a short section with cargo bench, cargo bench -- --save-baseline before and --baseline before for comparing a change, and the note that html_reports is an opt-in feature if someone wants the charts. The agent runs cargo bench --no-run, cargo bench once to see every bench complete, cargo test and cargo fmt --all --check in its sandbox and keeps the output.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add Criterion benchmarks to this Tokio server. Read Cargo.toml, src/lib.rs, src/frame.rs, src/connection.rs, src/server.rs, src/clients/client.rs, tests/server.rs and the CI workflow first. Add criterion as a dev-dependency at its current release with the async_tokio feature and two [[bench]] targets with harness = false. In benches/frame.rs, benchmark Frame::check followed by Frame::parse on a SET array frame and on a 4 KiB bulk string through std::hint::black_box. In benches/roundtrip.rs, start the server in-process on 127.0.0.1:0 with a pending shutdown future as tests/server.rs does, connect one Client, and benchmark ping, set and get on that connection with to_async on a Tokio runtime. Do not benchmark Db, which is not public, and say so in the PR. Add cargo bench --no-run to the Ubuntu CI job without running the benches in CI, and document cargo bench and baselines in the README. Run cargo bench --no-run, cargo bench, cargo test and cargo fmt --all --check, include the output, and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds benches/frame.rs and benches/roundtrip.rs, and changes Cargo.toml, Cargo.lock, .github/workflows/ci.yml and README.md. Nothing under src/ changes, and the runtime dependency list is the same; Criterion is a dev-dependency only.

The description explains why Db is out of reach, what each round-trip number includes, and why CI only builds the benches, and quotes the cargo bench --no-run, cargo test and cargo fmt output with the list of benches that completed. The commands and their full output are in the transcript at GET /o/{org}/sessions/{id}/messages.

Before merging, a reviewer runs cargo bench on a quiet machine to get a first baseline worth keeping and decides whether html_reports should be on. A follow-up message to the same session can add a bench for BufferedClient, a publish and subscribe pair, or a parametrised set of bulk sizes.

Run it on every Tokio crate with a hot path and no numbers

A team with several Tokio services sends one POST /o/{org}/sessions per repository with this prompt as input and the repository in resources. Each run reads its own lib.rs, so a crate whose parser is private gets its benches written against the public type that wraps it, and one that already has a benches/ directory gets new files in the same style.

Send a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run it already started. The key's webhook_url reports when each run the key starts finishes, with session_url for the pull request link and the bench output.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

The hard part of a first benchmark is choosing what it can honestly measure: the store is crate-private, the encoder needs a socket, and an async bench carries runtime overhead that a sync one does not. The agent reads the visibility of every type and the way the tests start a server, picks the measurable paths, and runs every bench to completion in a sandbox of its own, with the output kept.

The pull request carries the benches, the CI step that keeps them compiling and the README on baselines, so the next performance change starts from a number rather than a guess. Several crates can be done at once, each in its own sandbox with its own pull request.

Frequently asked questions

Why not benchmark Db directly?

Db is pub(crate) in src/lib.rs, and a benches/ file is compiled as a separate crate that only sees public items. Making it public for a benchmark would change the library's API, so the round trip measures it through Client instead; say so if you would rather expose it.

Will benchmarks run in CI?

They are built with cargo bench --no-run so a broken bench fails the pull request, but not run: timings on a shared runner vary too much to compare. Run cargo bench locally with --save-baseline before a change and --baseline after.

Does this change the runtime dependencies?

No. Criterion is a dev-dependency, so cargo build and the binaries are unaffected; only Cargo.lock grows with its tree. The otel feature and its optional crates are untouched.

Why run a real server for the round trip?

Because that is the path a client pays for: frame encoding in Connection, the command dispatch in cmd, and the Db mutex. Starting it on port 0 inside the bench, as tests/server.rs does, needs no running process and no fixed port.

How do I compare before and after a change?

cargo bench -- --save-baseline before on the current commit, make the change, then cargo bench -- --baseline before; Criterion prints the change per bench with its own noise estimate. The README section the agent writes has those two commands.

More Tokio jobs

The same job on other stacks

More jobs like this: Hand off a long task

Keep reading