What the agent does
A role's CI usually fails on one distribution out of many, so the agent first works out which one and at which task. In a role like geerlingguy/ansible-role-docker, .github/workflows/ci.yml runs on pull requests, on pushes to master and on a weekly schedule, 0 7 * * 0, checks the role out into a geerlingguy.docker directory so the role name resolves, and has two jobs: lint, which installs yamllint on Python 3.13 and runs it, and molecule, a matrix over rockylinux10, ubuntu2404, ubuntu2204, debian13, debian12, fedora43 and opensuseleap15 that installs ansible, molecule, molecule-plugins[docker] and docker and runs molecule test with MOLECULE_DISTRO set. The agent reads gh run view <id> --log-failed, notes the distro from the matrix, the Molecule stage (converge, idempotence or verify) and the task name in the Ansible output, and whether the same run passed on the sibling distros, which tells it whether the break is in a shared task or in one OS family's file.
Then it reads how the scenario is built, because a failure in molecule test is as often about the test image as about the role. molecule/default/molecule.yml uses the Docker driver with the image geerlingguy/docker-${MOLECULE_DISTRO:-rockylinux9}-ansible:latest, privileged, with cgroupns_mode: host and /sys/fs/cgroup mounted so systemd can run, and pre_build_image: true, so a change in that upstream image arrives in CI without any commit here; the default distro in that file is rockylinux9 while the matrix now starts at rockylinux10. converge.yml refreshes the apt cache on Debian-family hosts and waits for systemctl is-system-running to report running or degraded before applying the role, and verify.yml runs docker version, checks systemctl is-active docker and pulls hello-world, with the docker run step commented out because of Docker-in-Docker quirks on GitHub Actions. A failure in verify at the pull step therefore points at the registry or at the daemon not starting, not at the install tasks.
The role itself is small enough to read whole. tasks/main.yml loads vars/<distribution>.yml, vars/<os_family>.yml or vars/main.yml through first_found, includes setup-RedHat.yml, setup-Suse.yml or setup-Debian.yml by family, installs docker_packages with allow_downgrade on RedHat and Debian, installs the compose plugin, writes daemon.json when docker_daemon_options is set, patches the service's ExecStart when asked, starts the service, flushes handlers, and adds docker_users to the group. setup-Debian.yml removes old key and list files, installs ca-certificates and python3-debian, and adds the repository with deb822_repository using ansible_facts.distribution_release as the suite and the key from docker_apt_gpg_key. setup-RedHat.yml fetches the repo file into /etc/yum.repos.d, disables the nightly section and has a RHEL 8 block for containerd.io. setup-Suse.yml adds a zypper repository whose URL vars/Suse.yml builds from a release mapping, with a comment noting that the numeric paths match distribution_version for Leap and SLES 15 and that only Tumbleweed and SLES 12 SP5 need special cases. defaults/main.yml lists docker-ce, docker-ce-cli, docker-ce-rootless-extras, containerd.io and docker-buildx-plugin, and vars/Debian.yml and vars/RedHat.yml hold the obsolete packages each family removes first.
With the failing task located, the agent reproduces it on the one distro rather than all seven: MOLECULE_DISTRO=<distro> molecule converge in the sandbox's own Docker, then molecule verify and molecule idempotence as the failing stage dictates. A suite named by distribution_release that Docker's repository does not publish yet, a package renamed in the distribution's own repository, a repository path that moved, or a new image where a service name changed each lead to a different file: a variable in vars/, a when: on a task, or a default in defaults/main.yml. The agent makes the smallest change that passes the stage that failed, reruns molecule test for that distro and for one distro from each other family it touched, runs yamllint . because the lint job does, and keeps .ansible-lint's skip list in mind so it does not reformat tasks the maintainer chose to leave. It changes the matrix only when the failure is that the image itself no longer exists, and says so.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "The CI workflow for this Ansible role failed; the failed workflow run follows this prompt. Read .github/workflows/ci.yml, molecule/default/molecule.yml, converge.yml and verify.yml, then tasks/main.yml, the setup-*.yml file for the OS family that failed, defaults/main.yml, vars/ and meta/main.yml. Read the failed job's log with gh run view <run id> --log-failed and identify the matrix distro, the Molecule stage and the exact task that failed, and whether the sibling distros passed. Install ansible, molecule, molecule-plugins[docker] and docker as the workflow does. Reproduce on the one distro that failed with MOLECULE_DISTRO=<distro> molecule converge, then the stage that failed. Find the cause in the role's tasks, variables or defaults and make the smallest change that fixes it; change the matrix or the test image only if the image no longer exists, and explain why. Rerun molecule test for that distro and for one distro from each other OS family whose files you changed, and run yamllint . as the lint job does. Do not reformat tasks or change the .ansible-lint skip list. Open a pull request that names the failed run, quotes the failing task output, explains the cause in one paragraph and pastes the molecule and yamllint commands you ran with their output.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that names the failed run and quotes the task output that broke, explains the cause in one paragraph, and changes the file where the cause lives: a line in vars/Suse.yml or vars/Debian.yml, a when: or a module argument in one of the setup-*.yml files, or a default in defaults/main.yml. The body pastes the molecule converge, molecule verify or molecule test output for the distro that failed and for the sibling it reran, and the yamllint . output.
The repository's ci.yml runs on the pull request: yamllint and the full seven-distro Molecule matrix. That is the check that matters, since the agent reran one distro per family and the matrix reruns all of them, including the ones that passed before the change.
A reviewer reads the cause against the quoted log, checks that a change to a shared task did not alter behaviour for the families that were passing, and, when the agent changed the matrix or the image tag, reads its reason: an image that no longer exists is a test-infrastructure change, not a role change, and belongs in its own commit.
Run it on every failed Molecule workflow run
Add an automation to the Project that holds the role with this prompt and a github trigger on workflow_run.failure. The weekly schedule in ci.yml means a failure can arrive with no commit behind it, when an upstream image or repository changes; a run from a fork never fires, and neither does one whose sender is a bot. The trigger appends the run event after the prompt, so the agent reads the run id and branch from it rather than listing runs first. max_runs_per_hour on the automation caps how many failures it picks up in an hour.
From your own server instead, a handler for GitHub's workflow_run webhook with conclusion: failure sends POST /o/{org}/sessions with this prompt and the run id in input, the repository in resources, and a request_id that is a UUID v5 of the run id so a redelivery returns the same run. A webhook_url on the key receives run.completed with data.session_url to read the pull request from.
The use case this job follows, with its automation and trigger: Fix a bug from an alert.
Why run it on Rush?
Reproducing a Molecule failure on a laptop means a privileged systemd container for the one distro that broke, the same Ansible and Molecule versions the runner installed, and patience while the image pulls, usually on a Monday morning after the weekend schedule turned red. In a Rush sandbox the agent has its own Docker daemon and installs exactly what ci.yml installs, and every molecule command and its Ansible output stays in the transcript.
A role that supports seven distributions breaks for reasons outside its own commits: a repository path moves, a release name appears before the vendor publishes packages for it, an image is rebuilt. The agent reads the log, the role and the scenario together, fixes the file the cause lives in, and leaves the full matrix to prove the rest.
Frequently asked questions
What if the failure is in the test image, not the role?
The agent says so. molecule.yml pulls geerlingguy/docker-<distro>-ansible:latest with pre_build_image: true, so a rebuilt image can change systemd or package behaviour with no commit in this repository. When the fix is a different image tag or a matrix entry, the pull request puts it in its own commit and explains that the role's tasks were not the cause.
Does the agent run all seven distros?
No. It reproduces on the one that failed and reruns one distro from each other family whose files it changed, because molecule test builds and converges a full container each time. The workflow on the pull request runs the whole matrix, which is the check that counts.
Why can verify fail when converge passed?
verify.yml checks docker version, systemctl is-active docker and pulls hello-world from the registry. A daemon that installed but did not start, or a pull refused by the registry, fails there with the install tasks all green. The agent reads which of the three steps failed before touching an install task.
Will it loosen the lint rules to get green?
No. .yamllint extends the default rules with a long-line warning and .ansible-lint has a short skip list the maintainer chose. The prompt forbids changing either, and the agent runs yamllint . on its own change as the lint job does.