What the agent does
A CLI's CI fails in one of several jobs that have nothing to do with each other, so the agent first reads which. In a project shaped like casey/just, .github/workflows/ci.yaml sets RUSTFLAGS: --deny warnings for every job and runs four: lint (cargo clippy --all --all-targets, cargo fmt --all -- --check, ./bin/forbid, and shellcheck www/install.sh after installing ripgrep and shellcheck), msrv (cargo check on the 1.89.0 toolchain that Cargo.toml names as rust-version), pages (cargo run --package generate-book, then mdbook build for book/en and book/zh with mdbook 0.4.52 and mdbook-linkcheck 0.7.7 downloaded by version), and test, a matrix over Ubuntu, macOS and Windows that runs cargo test --all and then bash www/install.sh --to /tmp --tag 1.25.0 followed by /tmp/just --version. The agent reads gh run view <id> --log-failed, notes the job, the matrix OS and the first failing command, and whether sibling jobs passed, which already separates a lint from a regression from a release-asset problem.
Then it reads how the project makes those commands strict, because the fix has to satisfy the same settings. Cargo.toml is edition 2024, a workspace of the root crate plus crates/action-versions and crates/generate-book, and its [lints] tables deny clippy::all and clippy::pedantic, unreachable_pub and unexpected_cfgs, with a short allow list (needless_pass_by_value, similar_names, struct_excessive_bools among them); rustfmt.toml sets max_width = 100 and tab_spaces = 2; clippy.toml raises the cognitive complexity threshold and orders source items. bin/forbid is a ripgrep for dbg! or todo anywhere but the changelog, so a leftover debug macro fails lint before clippy does. Tests are integration tests: autotests = false and tests/lib.rs lists well over a hundred modules, each built on the Test type in tests/test.rs, which writes a justfile into a temp directory, runs the built just binary from CARGO_BIN_EXE_just with arguments, and compares stdout, stderr and the exit status, with pretty_assertions for the diff.
Reproduction is per job. For lint the agent runs cargo clippy --all --all-targets with RUSTFLAGS=--deny warnings, cargo fmt --all -- --check, ./bin/forbid and shellcheck www/install.sh, in that order, and the first one that fails is the one the log named. For msrv it installs the 1.89.0 toolchain with rustup and runs cargo +1.89.0 check, which catches a dependency or a language feature newer than the declared minimum. For pages it downloads the two mdbook binaries at the pinned versions and runs the same three commands; a broken link in book/ fails mdbook-linkcheck, not the Rust build. For test on Ubuntu it runs cargo test --all and, when a single test is named, cargo test --all <name> to read the expected-versus-actual diff. A failure only on the Windows or macOS leg cannot be rerun in a Linux sandbox, so the agent reads the test's expectation (MAIN_SEPARATOR, EXE_SUFFIX and the windows.rs module exist for that reason), the relevant source, and the log, and says in the pull request that the fix is reasoned from the log rather than reproduced.
The fix is the smallest change that makes the failing command pass without weakening a check. A clippy pedantic lint is fixed in the code, not allowed in Cargo.toml; a formatting failure is cargo fmt --all; a forbid hit is the removal of the dbg!; an msrv failure is either a code change back to what 1.89.0 accepts or, when a dependency raised its own minimum, a pinned dependency version with the reason stated. A test failure is fixed in src/ unless the test's expected output is what changed on purpose in the same commit, in which case the agent updates the Test expectation and says why. After the fix it reruns the full job's commands, not just the one that failed, and also cargo test --all when it touched src/, since lint passing says nothing about behaviour. The pull request names the run, quotes the failing lines, and pastes each command it ran with its output.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "The CI workflow for this Rust command-line tool failed; the failed workflow run follows this prompt. Read .github/workflows/ci.yaml, Cargo.toml including its [lints] tables, rustfmt.toml, clippy.toml, bin/forbid and tests/test.rs, then read the failed job's log with gh run view <run id> --log-failed. Identify the job, the matrix OS if any, the first failing command and whether sibling jobs passed. Reproduce that job in the sandbox with the same commands, flags and RUSTFLAGS the workflow sets: for msrv, install the toolchain Cargo.toml names as rust-version and run cargo check with it; for pages, download the mdbook and mdbook-linkcheck versions the workflow pins; for test, run cargo test --all and then the one named test for its diff. If the failure is only on a Windows or macOS leg, reason from the log and the test's expectations and say so. Make the smallest change that fixes the cause: fix lints in code rather than allowing them, change a test's expected output only when the behaviour change was intended and explain it, and pin a dependency only when it raised its minimum Rust version past the declared one. Rerun the whole failed job's commands and cargo test --all if you touched src/. Open a pull request that names the failed run, quotes the failing lines, explains the cause in one paragraph and pastes every command you ran with its output.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request names the failed run and quotes the lines from its log that decided the diagnosis. The diff is in one place: a few lines in a src/ module for a lint or a regression, an expectation in one tests/ module when behaviour changed on purpose, a version in Cargo.toml for an MSRV break, or a link in book/ for a pages failure. The body pastes the commands it reran with their output, and says plainly when a Windows-only or macOS-only failure was fixed from the log without a rerun.
The repository's ci.yaml runs on the pull request: lint, msrv, pages and the three-OS test matrix, with RUSTFLAGS=--deny warnings throughout. That is the check that matters, because the agent reran one job on one OS; the matrix reruns all of them, including the platform it could not reach.
A reviewer reads the cause against the quoted log, confirms that no allow was added to Cargo.toml or in a #[allow] attribute to make a lint go away, and reads any change to a test's expected stdout or stderr as a behaviour change that needs its own justification. A dependency pin added for MSRV is checked against that dependency's changelog before merge.
Run it on every failed CI run on the default branch
Add an automation to the Project for this repository with the prompt above and a github trigger on workflow_run.failure, with branch_patterns set to your default branch (master in just) so a contributor's pull request does not start a run. The trigger appends the run event after the prompt, so the agent reads the run id, workflow and commit from it instead of listing runs; CI runs from forks never fire. max_runs_per_hour caps what a bad afternoon can start.
From your own server instead, a handler for GitHub's workflow_run webhook with conclusion: failure sends POST /o/{org}/sessions with this prompt and the run id in input, the repository in resources, and a request_id that is a UUID v5 of the run id so a redelivery returns the same run. A webhook_url on the key receives run.completed with data.session_url to read the pull request from.
The use case this job follows, with its automation and trigger: Fix a bug from an alert.
Why run it on Rush?
Reproducing a Rust CI failure locally means matching the job: the pinned MSRV toolchain, --deny warnings, the exact mdbook build, ripgrep and shellcheck on the path. Most developers run cargo test and hope the rest follows. In a Rush sandbox the agent installs exactly what the failing job installs and runs exactly its commands, and every command with its output is in the transcript, so a reviewer sees the same failure the runner saw and then sees it pass.
A CLI with a hundred-plus integration test modules fails for reasons that look alike in a red badge and differ entirely in cause. Reading the log, the lint tables and the test harness together, then fixing the one file and letting the full three-OS matrix prove the rest, is a job that fits an agent and a sandbox better than a laptop between meetings.
Frequently asked questions
What if the failure is only on Windows or macOS?
The sandbox is Linux, so the agent cannot rerun that leg. It reads the log, the test's expectation (the harness has MAIN_SEPARATOR, EXE_SUFFIX and a windows.rs module for exactly these cases) and the source, makes the fix, and says in the pull request that it was reasoned from the log; the matrix on the pull request is the rerun.
Will it silence a clippy lint to get green?
No. Cargo.toml denies clippy::all and clippy::pedantic on purpose, and the prompt tells the agent to fix lints in code rather than add allow attributes or entries. A reviewer can grep the diff for allow to confirm.
Why does it install an old toolchain?
The msrv job runs cargo check on the rust-version from Cargo.toml, 1.89.0 in just, to make sure the crate still builds for users on that compiler. A failure there is often a dependency that raised its own minimum, and the fix is a pin with a stated reason, not a bump of rust-version.
Can it change a test's expected output?
Only when the behaviour change was intended in the commit that broke CI, and it has to say so. The tests compare the binary's stdout, stderr and exit status byte for byte, so an expectation change is a behaviour change, and the pull request explains it as one.