All jobs

Add Snapshot Tests for the Shell Scripts a Rust CLI Generates

Yes. An agent adds insta snapshot tests that pin the shell scripts your Rust CLI prints, one per shell and option set, and commits them so any drift fails CI.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent clones the repository into its own cloud sandbox and reads what the CLI generates and how it is tested today. In a CLI like ajeetdsouza/zoxide (version 0.10.0, edition 2024, rust-version = "1.88.0"), zoxide init <shell> prints a shell script rendered by askama from templates/: nine templates between 82 lines (tcsh.txt) and 218 lines (bash.txt), chosen by the InitShell enum in src/cmd/cmd.rs, with --cmd (default z), --no-cmd and --hook none|prompt|pwd as the options, plus two more read from the environment through config::echo() and config::resolve_symlinks(). src/shell.rs already has a test module, but it is gated behind the nix-dev feature: an rstest matrix of 24 option combinations per shell that renders each script and runs it through the real shell from shell.nix, which lists bash, dash, elvish, fish, ksh, nushell, powershell, tcsh, xonsh and zsh. Those tests prove each script parses and runs; they say nothing about whether its content changed, and they need the Nix shell, which just test enters with cargo nextest run --all-features.

The agent adds insta as a dev-dependency and writes tests/init.rs, an integration test that runs the built binary with assert_cmd (Command::cargo_bin("zoxide")) rather than calling the templates directly, so the snapshot covers the real init command including its argument parsing. For each of the nine shells it records four variants: the defaults, --cmd j, --no-cmd and --hook prompt, with _ZO_ECHO and _ZO_RESOLVE_SYMLINKS cleared from the environment so a developer's own settings cannot change the output. Each case asserts insta::assert_snapshot! under a name like init_bash_no_cmd, which stores the script in tests/snapshots/init__init_bash_no_cmd.snap. The templates branch on cfg!(windows) in a few places, so the test is compiled for non-Windows targets only and the pull request says why.

Snapshots are committed, 36 files in all, and the agent reads each one once to confirm it is the script it expects rather than an error message or an empty string; build.rs already lists tests/ and templates/ under rerun-if-changed, so a template edit rebuilds and the next cargo test shows the diff. The test needs no shells and no Nix, so it runs with a plain cargo test --test init on any machine, and under the existing just test task as well, because nextest runs integration tests the same way. In CI, insta refuses to write new snapshots when the CI variable is set, so a template change without an updated snapshot fails the ci.yml job with the diff in the log.

The agent runs cargo test --test init twice in the sandbox, once to create the snapshots and once to show they pass unchanged, then runs cargo fmt --all --check and cargo clippy --all-targets -- -Dwarnings as the justfile's lint task does, and adds a short section to the README's development notes on cargo insta review for accepting an intended change. It changes no template and nothing under src/.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add snapshot tests for the shell scripts this Rust CLI generates with its init command. Read src/cmd/cmd.rs, src/shell.rs, the templates directory and the existing tests first. Use insta with assert_cmd to run the built binary for every supported shell with the default options and a few option variants, clear the environment variables that change the output, commit the snapshots, and make sure the tests run with plain cargo test without the Nix shell the existing shell tests need. Run the tests, fmt and clippy, add a note on updating snapshots to the contributor docs, and open a pull request. Change no template.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request on its own branch that adds tests/init.rs and 36 files under tests/snapshots/, adds insta to [dev-dependencies] in Cargo.toml with the matching Cargo.lock entries, and adds a paragraph to the README. The description lists the shells and variants covered, shows the two cargo test runs from the sandbox, and shows fmt and clippy passing.

The run's record keeps every command and its output, so a reviewer can see the test count, that the second run passed with no snapshot written, and that no file under src/ or templates/ changed. Each snapshot file is reviewable as the exact script a user gets from zoxide init, which doubles as documentation of the generated output.

Before merging, a reviewer decides whether four variants per shell is the right number or whether the full option matrix from src/shell.rs should be mirrored, and whether the snapshot test should also run on the Windows job, which would need a second set of snapshots for the cfg!(windows) branches.

Run it on every pull request that touches a template

Once the snapshots are in, the existing CI job does the ongoing work: a change to any file in templates/ that is not accompanied by an updated snapshot fails cargo test with the diff. To have an agent look at those failures, add an automation in the Project with a github trigger on pull_request.opened and pull_request.synchronize and a prompt that says to read the failing snapshot diff, decide whether the change is intended, and either update the snapshot or comment on what looks wrong. The trigger appends the pull request event after the prompt, so the agent knows which pull request to read.

For a nightly check that the snapshots still match what the release build prints, an automation with a schedule trigger (cron for 03:00 in your timezone) runs the same test against a fresh clone and opens a pull request only when something drifted.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

Snapshot tests are mostly typing: one case per shell and variant, a name for each, and a careful read of 36 generated files to make sure the first snapshot is right. The agent does the typing and the reading in one sitting, with the binary built from your branch, and leaves you a review of the scripts rather than a review of boilerplate.

The sandbox runs cargo test, fmt and clippy for real and the run's record keeps the output, so the pull request shows the tests passing twice. Several CLIs, or several branches, can be done at once in separate sandboxes.

Frequently asked questions

Does the existing shell matrix in src/shell.rs not already cover this?

It covers a different property. Those tests run each rendered script through the real shell to prove it executes, and they need the Nix shell with ten interpreters installed. The snapshots pin the script's content, so a reordering, a changed comment or a dropped function shows up as a diff, and they run anywhere cargo test runs.

How do I update the snapshots when I change a template on purpose?

Run cargo test --test init, then cargo insta review to accept or reject each diff, or set INSTA_UPDATE=always for one run to accept all of them. The new .snap files go in the same commit as the template change. The README paragraph the agent adds says the same.

Why are the snapshot tests not compiled on Windows?

The templates use cfg!(windows) for a few lines, such as the cygpath call in the bash script, so the generated output differs by target and one set of snapshots cannot match both. The pull request notes that a Windows job would need its own snapshot files with a target suffix.

How many snapshots are there and how big are they?

Nine shells times four variants, 36 files, each holding one generated script of between roughly 80 and 220 lines plus insta's small header. They are plain text and diff cleanly in a pull request.

More Rust CLI jobs

The same job on other stacks

Keep reading