What the agent does
The agent reads the workspace before it picks what to measure. In a workspace shaped like rust-lang/mdBook, the root Cargo.toml lists the mdbook binary crate, crates/* (mdbook-core, mdbook-markdown, mdbook-html, mdbook-driver, mdbook-renderer, mdbook-preprocessor, mdbook-summary, xtask) and two example members, sets edition 2024 and rust-version 1.88.0 for every member, pins every dependency under [workspace.dependencies], and marks pulldown-cmark 0.13.4 as part of the public API. The root crate already has a [[test]] target named gui with harness = false, so a custom harness is an accepted pattern here. .github/workflows/main.yml runs cargo test --workspace --locked on stable, beta, nightly, three targets and an msrv job on 1.88.0, then cargo clippy --workspace --all-targets --no-deps -- -D warnings and cargo fmt --check. There is no benches directory in any crate.
Two things are worth timing and both have a public entry point. mdbook_markdown::new_cmark_parser(text, &MarkdownOptions) turns a chapter into pulldown events, and mdbook::MDBook::load(root) followed by build() renders a whole book; the render_markdown function in mdbook-html is pub(crate), so the agent does not reach for it. The agent adds criterion to [workspace.dependencies] at the newest release whose minimum Rust fits 1.88.0, as a dev dependency of mdbook-markdown with benches/parse.rs declared under [[bench]] with harness = false, which parses every chapter under guide/src and drains the events, and of the root crate with benches/build_guide.rs, which loads guide/ and builds it into a tempfile directory, tempfile already being a dev dependency there. The corpus is the repository's own 35 guide chapters, so the numbers mean something to the maintainers.
Because CI passes --locked, the agent commits the Cargo.lock update that adds Criterion and its tree, and because clippy runs with --all-targets -D warnings, the bench files are written to lint clean under the workspace's [workspace.lints], including missing_docs on anything public. It runs cargo test --workspace --locked and cargo test --workspace --no-default-features to confirm bench targets do not disturb the test builds, cargo +1.88.0 check --workspace --all-targets for the MSRV job, cargo clippy --workspace --all-targets --no-deps -- -D warnings and cargo fmt --check.
Then it runs cargo bench --workspace once in the sandbox to produce a baseline and pastes Criterion's summary lines for both benchmarks into the pull request, and adds a short Benchmarks section to CONTRIBUTING.md beside the existing Tests section with the two commands and a note that cargo xtask test-all leaves benches out.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "codex",
"input": "Add Criterion benchmarks to this Rust workspace and open a pull request. Read Cargo.toml, Cargo.lock, crates/mdbook-markdown/Cargo.toml, crates/mdbook-markdown/src/lib.rs, crates/mdbook-driver/src/mdbook.rs, guide/book.toml, CONTRIBUTING.md and .github/workflows/main.yml first. Add criterion to workspace.dependencies at the newest release whose minimum Rust version is at or below the workspace rust-version. In crates/mdbook-markdown add benches/parse.rs as a harness-less bench target that parses every Markdown file under guide/src with new_cmark_parser and drains the events. In the root crate add benches/build_guide.rs that loads guide/ with MDBook::load and builds it into a tempfile directory. Use only public APIs. Commit the Cargo.lock change. Run cargo test --workspace --locked, cargo test --workspace --no-default-features, cargo check --workspace --all-targets on the rust-version toolchain, cargo clippy --workspace --all-targets --no-deps -- -D warnings and cargo fmt --check, then cargo bench --workspace once, and paste the check results and Criterion's summary lines in the pull request description. Add a Benchmarks section to CONTRIBUTING.md next to Tests. Do not change any source file under src or crates/*/src and do not update pulldown-cmark.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request with criterion in the root Cargo.toml and in the two crates' dev dependencies, two [[bench]] entries with harness = false, crates/mdbook-markdown/benches/parse.rs, benches/build_guide.rs at the root, the Cargo.lock update, and a Benchmarks section in CONTRIBUTING.md. The description holds the output of each check the agent ran and Criterion's summary lines for the parse and build benchmarks from the sandbox, labelled as a single baseline on that machine.
The repository's CI workflow runs on the pull request: the test matrix including the msrv job, the clippy job with --all-targets, rustfmt and the typos check, so the bench targets are compiled and linted on every toolchain the project supports even though no job runs them.
A reviewer checks that both benchmarks call only public functions, that pulldown-cmark did not move in Cargo.lock, that the bench files pass the workspace lints without allow attributes, and that the corpus is the guide rather than a synthetic string. Then they decide whether to run cargo bench on a dedicated machine before trusting numbers across commits.
Run it on every new workspace crate
A team that maintains several Rust workspaces starts one run per repository with POST /o/{org}/sessions, this prompt and the repository in resources. Each run reads that workspace's own public API and CI, so a crate with a [[test]] harness = false precedent gets the same shape and one with an MSRV job gets its Criterion version chosen to match, and each run ends in its own pull request.
The key's webhook_url receives run.completed for every run it starts, with a session_url for the result, so one script can collect the baseline summaries from each description into a single page.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
Adding benchmarks to a locked workspace touches the things CI is strict about: the lockfile, --all-targets clippy, the MSRV job and the public API boundary. In a Rush sandbox the agent runs every one of those checks the way main.yml does, on the toolchains the workflow names, and then runs the benchmarks once; every command and its output is kept in the run's transcript, so the pull request carries the evidence rather than a promise.
The result is a pull request that CI compiles on every supported target before a reviewer opens it, with a baseline attached and the limits of that baseline stated. The same request works on every Rust workspace the team owns, each in its own sandbox at the same time.
Frequently asked questions
Why not benchmark the HTML renderer directly?
Because render_markdown in mdbook-html is pub(crate), and a bench target is a separate crate that only sees public items. The agent benchmarks the public parser in mdbook-markdown and the public MDBook::load and build path, which covers the renderer end to end without widening the API.
Will the benches break the locked build or the MSRV job?
The agent commits the Cargo.lock change so --locked resolves, and picks the Criterion release whose minimum Rust version fits the workspace's rust-version, checking it with cargo check --workspace --all-targets on that toolchain before pushing.
Are the numbers in the pull request reliable?
They are one run on one sandbox, labelled as such. Criterion stores its baseline under target/criterion, which is not committed, so comparisons across commits need the same machine. The pull request gives the maintainers the harness and corpus; the stable numbers come from running it where they choose.
Does cargo test now run the benchmarks?
No. cargo test does not build bench targets unless asked with --benches or --all-targets, and harness = false benches are not test binaries. The agent runs both cargo test --workspace --locked and the --no-default-features variant to show the test builds are unchanged.