What the agent does
The agent clones the repository into its own cloud sandbox and reads the command surface first. In a CLI like ajeetdsouza/zoxide (version 0.10.0, edition 2024, rust-version = "1.88.0"), src/cmd/cmd.rs defines the clap subcommands: add (one or more paths and an optional --score), query (keywords plus --all, --interactive, --list, --score, --exclude and --base-dir, with --interactive and --list declared as conflicting), remove, edit, import (from atuin, autojump, fasd, z, z.lua or zsh-z, with a global --merge) and init (a shell, --no-cmd, --cmd defaulting to z, and --hook of none, prompt or pwd). assert_cmd, tempfile and rstest are already dev-dependencies, so nothing new is needed to start.
Then it finds out what runs under a plain cargo test. The repository's tests are real but mostly behind a feature: tests/completions.rs starts with #![cfg(feature = "nix-dev")], and the shell tests in src/shell.rs are #[cfg(feature = "nix-dev")] too. They render every init template across combinations of command name, hook, echo and symlink resolution and check each one with the real shell and with linters such as shellcheck and shfmt. CI runs them through just lint test, which calls cargo nextest run --all-features inside nix-shell. Without that feature and those shells, the subcommands' behaviour on a real database is untested.
It writes integration tests under tests/ that run the built binary with assert_cmd::Command::cargo_bin("zoxide"). Each test points _ZO_DATA_DIR at a tempfile::TempDir, which matters twice over: src/config.rs requires that path to be absolute, and without it the tests would write into the real data directory. Because query filters out paths that no longer exist on disk, each test creates real directories inside the temp dir before adding them. The cases cover add then query returning the best match, query --list --score printing every entry with its score, --exclude dropping the current directory, remove deleting an entry, --interactive and --list together failing with clap's conflict error, and each importer reading a fixture file through its own variable (_Z_DATA for z, _ZL_DATA for z.lua) with and without --merge.
For each shell's init output it adds snapshot tests: one snapshot per shell and per --hook value, plus --cmd and --no-cmd variants, so a template edit in templates/*.txt shows up as a reviewable diff. The repository has no snapshot library, so adding one such as insta is a new dev-dependency, and the agent names it as such; the alternative is expected output files under tests/ compared by hand. It runs cargo test until it passes, and cargo clippy --all-targets -- -D warnings because the lint recipe in the justfile does the same.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Write integration tests with assert_cmd and tempfile for this Rust CLI's add, query, remove and import commands, and snapshot tests for each shell's init script. Read the clap definitions first and list every subcommand, flag and flag conflict. Run the built binary in each test, point the data directory environment variable at a temporary directory so no test touches real user data, and create real directories for any command that checks paths exist. Cover each importer with a small fixture file in the format it reads, with and without --merge. Snapshot the init output for every shell and every hook value, plus the command-name options. If snapshotting needs a new dev-dependency, say which and why. Note which existing tests only run behind a cargo feature. Run cargo test and cargo clippy --all-targets -- -D warnings until both pass, and open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request on its own branch. For a CLI like zoxide, the diff adds files under tests/ (one per subcommand or a shared helper module plus test files), fixture files for each import format, a snapshot folder with one file per shell and option combination, and, if the agent chose a snapshot crate, one line in [dev-dependencies] in Cargo.toml and the matching Cargo.lock change. The description maps each test to the subcommand and flag it covers and lists the existing feature-gated tests it left alone.
The session transcript keeps every cargo test and clippy run with its output, so the reviewer can see which cases failed first and what was fixed. The pull request's checks run the repository's own workflow, which for zoxide is just lint test on its CI matrix, running the new tests beside the shell tests.
Before merging, a reviewer reads the snapshots, since they are now the definition of correct init output, checks that every test sets _ZO_DATA_DIR to a temp path, and decides whether the new dev-dependency is acceptable.
Run it on every release tag
Before each release, one run can check that the tests still cover every subcommand: your release script sends POST /o/{org}/sessions with the repository in resources and a prompt to add tests for any flag or subcommand added since the last tag. The agent opens a pull request when something is missing.
Set a webhook_url on the API key and Rush posts run.completed or run.failed to it when the turn ends; the release script can wait for that event and read the session's pull request before it tags.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Integration tests for a tool that keeps a database of the user's directories are the kind that go wrong on a laptop: one test that forgets _ZO_DATA_DIR adds temp paths to the developer's own zoxide history. In a Rush sandbox the agent builds the binary and runs every test on a machine that exists for this run, with no shell history or data directory to damage.
Each cargo build and cargo test is kept with its output, so the reviewer sees the snapshots being generated and the full passing run without checking out the branch. Several CLIs in one organization can each get a run at once, each in its own sandbox.
Frequently asked questions
Will the tests touch my real zoxide database?
No. Every test sets _ZO_DATA_DIR to a tempfile::TempDir, which zoxide requires to be an absolute path, and the directory is deleted when the test ends. The same applies to the import fixtures, which are read through their own environment variables.
Does it need every shell installed?
Not for snapshot tests, which compare the text zoxide init prints. The existing tests that run the generated scripts in bash, fish, zsh and other shells stay behind the nix-dev feature, which the repository's CI turns on inside nix-shell.
Why add a snapshot crate instead of comparing strings?
One snapshot per shell and option combination means dozens of expected outputs, and a snapshot tool writes and updates them for you and shows a readable diff when a template changes. If you would rather not add a dependency, say so in input and the agent compares against files under tests/ instead.
Does it work for CLIs built with something other than clap?
Yes. assert_cmd runs the compiled binary, so it does not matter how arguments are parsed. The agent reads whichever parser the CLI uses to list its subcommands and flags before writing tests.