All jobs

Add Criterion Benchmarks to a Rust CLI

Yes. An agent adds Criterion benchmarks that time your CLI binary against seeded databases of three sizes, keeps CI green, and opens a PR with a baseline.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the crate is laid out before it decides what a benchmark can call. In a CLI shaped like ajeetdsouza/zoxide, Cargo.toml names version 0.10.0, edition = "2024", rust-version = "1.88.0", one binary and no library target: src/main.rs declares mod cmd, mod config, mod db and the rest privately and only exports main. Dev dependencies are assert_cmd, rstest, rstest_reuse and tempfile, and the release profile sets lto, strip and one codegen unit. There is no benches/ directory and no [[bench]] target. Because nothing is exported, a Criterion bench cannot import Database or Stream; it has to drive the binary the way a shell does, which is also what users feel.

The code shows what is worth timing. src/db/mod.rs reads db.txt from the data directory into a Vec<Dir>, each line a timestamp, a rank and a path separated by whitespace, and sort_by_score orders them by Dir::score, which multiplies the rank by 4, 2, 0.5 or 0.25 by age. src/db/stream.rs walks that list from the top, matching keywords against the path from the right, then filtering by base directory and exclude globs, and last checking that the directory exists on disk, with a comment that exists checks are slow and must come last; a directory that no longer exists and has not been seen in three months is removed as a side effect. src/cmd/query.rs runs that stream for zoxide query, printing the first match, every match with --list, or skipping the exists check with --all. src/cmd/add.rs canonicalizes each path, bumps its rank, ages the database when the total passes _ZO_MAXAGE and writes the file back. src/config.rs reads _ZO_DATA_DIR, which must be absolute, and that is the knob the benchmarks turn.

The agent adds criterion under [dev-dependencies] at the newest release whose minimum Rust version is at or below 1.88.0, a [[bench]] entry named cli with harness = false, and benches/cli.rs. The bench writes a db.txt of N entries into a tempfile::TempDir, with paths that exist (subdirectories it creates) and paths that do not, ranks spread across the clamp range and timestamps spread across the four age bands, for N of 1,000, 10,000 and 100,000. It then times std::process::Command::new(env!("CARGO_BIN_EXE_zoxide")) with _ZO_DATA_DIR pointed at that directory for four cases: query --list (sort, keyword filter and an exists check per row), query --list --all (the same without the filesystem), query <keyword> for the first match, and add <dir>, which mutates the file and so runs under iter_batched with a fresh copy of the data directory per iteration. The per-process startup cost is measured once with query --list on an empty database and named in the report so the reader can subtract it.

Then it runs what justfile and ci.yml run. just lint includes cargo clippy --all-features --all-targets -- -Dwarnings and cargo udeps --all-features --all-targets, so the bench file is written to lint clean and criterion has to be used by a target or udeps flags it; cargo msrv verify means the chosen Criterion must compile on 1.88.0. just test is cargo nextest run --all-features --no-fail-fast --workspace, which compiles bench targets without running them. The agent runs those, commits the Cargo.lock change, runs cargo bench --bench cli once in the sandbox, pastes Criterion's summary lines for every case and size into the pull request, and adds a bench entry to justfile beside test.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add Criterion benchmarks to this Rust CLI and open a pull request. Read Cargo.toml, Cargo.lock, src/main.rs, src/config.rs, src/db/mod.rs, src/db/dir.rs, src/db/stream.rs, src/cmd/query.rs, src/cmd/add.rs, justfile, shell.nix and .github/workflows/ci.yml first. The crate has no library target, so do not restructure it: write benches/cli.rs as a harness = false bench that runs the built binary through env!(\"CARGO_BIN_EXE_zoxide\") with _ZO_DATA_DIR set to a tempfile directory. Seed db.txt with 1,000, 10,000 and 100,000 entries that mix existing and missing directories, ranks across the allowed range and timestamps across the hour, day, week and older bands. Time query --list, query --list --all, a single-keyword query, and add with a fresh copy of the data directory per iteration; measure process startup once on an empty database and report it. Add criterion to dev-dependencies at the newest release whose minimum Rust version is at or below rust-version, and commit the Cargo.lock change. Run just lint and just test (or the cargo commands they wrap if nix is unavailable, and say so), then cargo bench --bench cli once, and paste the check results and Criterion's summary lines in the pull request. Add a bench entry to the justfile. Do not change anything under src.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request with criterion in [dev-dependencies], a [[bench]] entry, benches/cli.rs, the Cargo.lock update and a bench entry in justfile. Nothing under src/ changes. The description holds the clippy, udeps, msrv and nextest results and Criterion's summary line for each of the four cases at each of the three sizes, labelled as one baseline on the sandbox's machine, with the empty-database startup time beside them.

The repository's ci workflow runs on the pull request: just lint test inside the pinned nix shell, so the bench target is compiled and linted on the same toolchain the project uses, even though no job runs the benchmarks. The run's record keeps every command the agent ran with its output, including the cargo bench run the summary lines came from.

A reviewer checks that the seeded database exercises the exists check (a mix of real and missing directories), that add runs on a fresh copy each iteration rather than growing one file, and that the numbers are read as relative between sizes rather than absolute. Then they decide whether a dedicated machine should run just bench before trusting a comparison across commits.

Run it on every release candidate

Once the bench target exists, a team runs it on each release branch: one POST /o/{org}/sessions with the branch URL, https://github.com/<owner>/<repo>/tree/<branch>, in resources and a prompt that asks for cargo bench --bench cli and a comparison against the baseline committed in the pull request. The run clones that branch, runs the benchmarks in its own sandbox and reports the Criterion output as an answer rather than a pull request.

Give the API key a webhook_url and run.completed arrives with data.result holding that answer, up to 16 KiB, and a session_url for the full record. Several CLIs can be benchmarked at once, each in its own sandbox.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

A benchmark that spawns a process a few thousand times, against a hundred thousand seeded directories, is exactly the job that should not run on the laptop someone is typing on, and the numbers it produces there are noise. In a Rush sandbox the agent builds the release binary, seeds the databases, runs every case once and keeps Criterion's output in the run's record, so the pull request carries a baseline with its conditions stated rather than a bench file nobody has run.

Driving the binary rather than internal functions means no module becomes pub for the sake of a benchmark, the crate's layout is untouched, and the timings include what users feel: startup, reading db.txt and the exists checks. The same request works on any binary-only Rust crate.

Frequently asked questions

Why spawn the binary instead of benchmarking Database and Stream directly?

src/main.rs keeps every module private and there is no lib.rs, so a bench crate cannot import them. Cargo sets CARGO_BIN_EXE_zoxide for benches and integration tests, and timing the process is also what a shell hook pays on every cd.

Will the benches break just lint or cargo msrv verify?

The bench file is compiled by clippy under --all-targets -Dwarnings, cargo udeps checks that criterion is used, and cargo msrv verify builds on 1.88.0, so the Criterion release is chosen to compile there. The agent runs all three before opening the pull request.

How is the database seeded?

db.txt is a text file, one line per directory with a timestamp, a rank and a path separated by whitespace, which src/db/mod.rs parses. The bench writes it directly into a temporary directory and points _ZO_DATA_DIR at it, creating some of the listed directories so the exists check has work to do.

Are the numbers in the pull request reliable?

They are one run on the sandbox's hardware, so they compare the cases and sizes against each other, not against another machine. The pull request says so, and the bench entry exists so a dedicated box can produce the baseline you compare future commits against.

More Rust CLI jobs

The same job on other stacks

More jobs like this: Hand off a long task

Keep reading