All jobs

Add Mutation Testing to a Tokio Project

Yes. An agent adds cargo-mutants with a config that skips the binaries, runs it on changed code in every pull request, and opens a PR listing the missed mutants.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the crate before deciding what to mutate. In a project laid out like tokio-rs/mini-redis, Cargo.toml describes version 0.4.1 on edition 2018 with two binaries, mini-redis-cli at src/bin/cli.rs and mini-redis-server at src/bin/server.rs, tokio with the full feature, bytes, clap 4 with derive, tracing and tracing-subscriber, and an optional otel feature that pulls in the OpenTelemetry crates. The library is src/lib.rs with connection.rs, db.rs, frame.rs, parse.rs, server.rs, shutdown.rs, the commands under src/cmd/ and the three clients under src/clients/. Tests live in tests/client.rs, tests/buffered_client.rs, tests/frame_validation.rs and tests/server.rs, which bind a server on a local port and drive it over TCP, with a few #[test] blocks inside modules. .github/workflows/ci.yml runs cargo build, cargo build --features otel, cargo test, cargo test --features otel and cargo fmt --all --check on Ubuntu, macOS and Windows.

The tool is cargo-mutants, currently at v27.1.0. It rewrites one function at a time, replacing a body with a default value or flipping an operator, and runs the test suite against each mutant; a mutant that survives is a gap in the tests. The agent writes .cargo/mutants.toml with exclude_globs for src/bin/*.rs and examples/**/*.rs, because the binaries parse command lines and print, and no test exercises them; mutating them only produces noise. It sets timeout_multiplier above the default so the TCP tests, which wait on sockets, are not reported as timeouts, and adds skip_calls for the tracing macros so a deleted debug! line is not counted as a mutant. The library modules stay in scope: frame.rs and parse.rs are where a flipped comparison in a length check should be caught, and db.rs has the expiry logic.

CI gets one new workflow rather than a change to ci.yml. On pull_request, the job checks out with fetch-depth: 0, writes git diff origin/${{ github.base_ref }}.. | tee git.diff, installs the tool with taiki-e/install-action@v2 and tool: cargo-mutants, and runs cargo mutants --no-shuffle -vV --in-diff git.diff, so only functions touched by the pull request are mutated and the job finishes in a fraction of a full run. On push to the default branch it runs cargo mutants -vV --in-place over the whole library. Both upload mutants.out as an artifact with if: always(), and the tool's own GitHub annotations mark each missed mutant on the diff. A Swatinem/rust-cache@v2 step keeps the baseline build warm.

Before the pull request, the agent runs the full set once in the sandbox with cargo mutants -vV --in-place and reads mutants.out/missed.txt. For mini-redis the likely survivors are small: a replaced return in src/clients/blocking_client.rs, which the integration tests do not cover, or a negated branch in src/cmd/set.rs around the expiry argument. The agent does not write tests for all of them; it lists them in the pull request so the team decides which ones matter, and it adds exactly as many exclude_re entries as are needed for mutants that cannot be caught, such as a Debug impl. The config, the workflow and that list are the whole change.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add mutation testing with cargo-mutants to this Tokio project. Read Cargo.toml, src/lib.rs, the modules under src/, the tests under tests/ and .github/workflows/ci.yml first. Write .cargo/mutants.toml that excludes src/bin/*.rs and examples/**/*.rs with exclude_globs, raises timeout_multiplier for the TCP tests, and lists the tracing macros in skip_calls. Add .github/workflows/mutants.yml with two jobs: on pull requests, check out with fetch-depth 0, write git diff origin/${{ github.base_ref }}.. to git.diff, install cargo-mutants with taiki-e/install-action, and run cargo mutants --no-shuffle -vV --in-diff git.diff; on pushes to the default branch run cargo mutants -vV --in-place; both upload mutants.out with if: always(). Install cargo-mutants in the sandbox, run the full set once, and open a pull request that quotes the summary line and lists every missed mutant from mutants.out/missed.txt with the file and function, without changing any test.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request adds .cargo/mutants.toml and .github/workflows/mutants.yml and changes nothing under src/ or tests/. Its description quotes the summary line from the sandbox run, the count of mutants tested, caught, missed and unviable as the tool printed it, and lists each missed mutant by file, function and replacement, copied from mutants.out/missed.txt.

A reviewer checks that the excluded paths are only the binaries and examples, that the timeout multiplier is high enough for the socket tests on a slow runner, and that the pull-request job uses --in-diff so a two-line change does not run the whole suite dozens of times. They decide which missed mutants deserve a test, and those become follow-up issues rather than part of this change.

From then on, each pull request's checks include a mutants job that fails when a function the pull request touched can be broken without a test noticing, with the surviving mutant annotated on the diff.

Run it on every pull request

The workflow in the pull request is the per-event mechanism, but a Rush automation can go further than a red check. With a github trigger on pull_request.opened and pull_request.synchronize, an automation checks out the branch, runs cargo mutants --in-diff on the same diff, and posts one review comment that names each missed mutant and the test file that would most plausibly catch it, pointing at the existing tests in tests/server.rs or tests/client.rs that already drive the command in question. An automation's run is read-only, so it comments and pushes nothing.

A team that wants the gaps closed as well starts a second run from a handler on its own server, sending this body with the pull request URL in resources and a request_id that is a UUID v5 of the head commit SHA, so a retried delivery returns the run already started. That run writes the tests for the mutants the reviewer marked and pushes to the branch.

The use case this job follows, with its automation and trigger: Review every pull request.

Why run it on Rush?

A full mutation run builds and tests the crate once per mutant, which is minutes of CPU on a laptop that nobody wants to give up. The agent does that run in its own sandbox, keeps mutants.out and the command output with the pull request, and the reviewer reads a list instead of waiting for a local run to finish.

The config is specific to this crate: which paths are binaries, which tests bind sockets, which macros are logging. Running the setup through an agent that reads Cargo.toml and the test files, rather than copying a workflow from another repository, is what keeps the first run from being a wall of false positives.

Frequently asked questions

Does this slow down every pull request?

The pull-request job runs with --in-diff git.diff, so only functions the pull request changed are mutated, and the baseline build is cached. The whole-crate run happens on pushes to the default branch, where its length does not block a merge.

Will the TCP integration tests cause spurious timeouts?

They can, because a mutant that breaks the frame parser can leave a test waiting on a socket. The agent raises timeout_multiplier in .cargo/mutants.toml so the per-mutant limit is a multiple of the measured baseline, and the pull request names the value it chose.

What about the otel feature?

The CI workflow builds and tests with --features otel as a second pass. The mutants run uses the default features, since the OpenTelemetry code paths are initialisation in src/bin/server.rs, which is excluded. A reviewer who wants both can add additional_cargo_args to the config.

Does the agent fix the missed mutants?

Not in this pull request. It lists them with file and function so the team can decide which gaps matter. Writing the tests is a separate run, which keeps this change to configuration and lets the first mutants job pass on a known baseline.

More Tokio jobs

The same job on other stacks

Keep reading