What the agent does
The agent reads the role's variables before it describes them, because an argument spec that disagrees with defaults/main.yml breaks every playbook that uses the role. In a role shaped like geerlingguy/ansible-role-docker, defaults/main.yml declares about thirty variables: docker_edition (ce or ee), the docker_packages list built from it, docker_packages_state, the docker_obsolete_packages list, the service switches docker_service_manage, docker_service_state, docker_service_enabled, docker_service_start_command and docker_restart_handler_state, the Compose plugin settings docker_install_compose_plugin, docker_compose_package and docker_compose_package_state, the standalone Compose settings docker_install_compose, docker_compose_version, docker_compose_arch, docker_compose_url and docker_compose_path, the repository settings docker_add_repo, docker_repo_url, docker_apt_release_channel, docker_apt_ansible_distribution, docker_apt_repo_url, docker_apt_gpg_key, docker_apt_filename, docker_yum_repo_url, docker_yum_repo_enable_test and docker_yum_gpg_key, and the two operators actually reach for, docker_users (a list) and docker_daemon_options (a dict). meta/main.yml has galaxy_info with min_ansible_version: 2.15.1 and dependencies: [], and there is no meta/argument_specs.yml.
The tasks show which types and values matter. tasks/main.yml tests docker_install_compose_plugin | bool, docker_service_manage | bool and docker_install_compose | bool, compares docker_service_start_command with the empty string, calls docker_daemon_options.keys(), and reads docker_users | length; tasks/docker-users.yml loops over docker_users with the user module; tasks/docker-compose.yml regex-matches docker_compose_version after stripping a leading v. The README.md documents the allowed values in prose: docker_edition is ce or ee, docker_packages_state is present, absent or latest, docker_apt_release_channel is stable or nightly. Nothing checks any of that at run time, so docker_users: alice (a string where a list is expected) or docker_packages_state: installed fails late inside a task, or not at all.
Role argument validation has existed since ansible-core 2.11: when meta/argument_specs.yml exists, a validation task is inserted at the start of the role and the role fails before its first real task if a variable has the wrong type or an unknown value. The agent writes that file with one main entry point, a short_description, the author from galaxy_info, and an options entry per variable: type (str, bool, list, dict) with elements: str for the lists, default copied from defaults/main.yml, choices where the README names them (docker_edition, docker_packages_state, docker_compose_package_state, docker_apt_release_channel, and docker_service_state as started or stopped), and a one-sentence description taken from the README's wording. Variables whose default is a Jinja expression, such as docker_compose_arch and docker_apt_repo_url, are declared with their type and no default, with the template shown in the description, because the spec validates the value a caller passes, not the expression. The file passes yamllint with the repository's .yamllint (line length 200) and ansible-lint with its .ansible-lint skip list.
Proof is the existing Molecule scenario. molecule/default/molecule.yml converges the role on geerlingguy/docker-${MOLECULE_DISTRO:-rockylinux9}-ansible:latest and verify.yml checks docker version, the systemd unit and a docker pull hello-world. The agent runs molecule test once with the spec in place to show a converge with default values still passes, then runs ansible-playbook against the converged instance twice more with deliberately bad values, docker_users: alice and docker_packages_state: installed, and keeps the output showing the role refused on the validation task with the variable named. It runs ansible-doc -t role -r . docker to show the spec renders as role documentation, and adds a short README note that the role now validates its arguments and which ansible-core that needs.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add role argument validation to this Ansible role and open a pull request. Read defaults/main.yml, meta/main.yml, README.md, every file under tasks/, vars/ and handlers/, molecule/default/molecule.yml, converge.yml and verify.yml, .ansible-lint, .yamllint and .github/workflows/ci.yml first. Write meta/argument_specs.yml with a main entry point and one option per variable in defaults/main.yml: the type (str, bool, list with elements, or dict), the default copied from defaults, choices wherever the README or the tasks restrict the value, and a one-sentence description in the README's words. For variables whose default is a Jinja expression, declare the type and describe the template instead of copying it as a default. Do not change defaults/main.yml or any task. Run yamllint and ansible-lint with the repository's configs, run molecule test with the default distro, then against the converged instance run the role with docker_users set to a plain string and with docker_packages_state set to an invalid value and keep the output showing validation fail before the first task. Run ansible-doc -t role on the role and paste the output. Add a README note on validation and the minimum ansible-core it needs. Paste every result in the pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request with one new file, meta/argument_specs.yml, and a few lines in README.md. defaults/main.yml, meta/main.yml and every task file are untouched, so a playbook that passes valid values behaves exactly as before. The description lists each variable with the type and choices the spec gives it and the README sentence it was taken from, and pastes the ansible-doc -t role output so a reviewer sees the documentation the spec now generates.
The run's record keeps the yamllint and ansible-lint runs, the full molecule test with its converge, idempotence and verify stages, and the two deliberate failures with the message naming the offending variable and the validation task that stopped the role. On a repository with this role's ci.yml, the lint job and the seven-distro Molecule matrix run on the pull request as well.
Before merging, a reviewer checks the choices against what they actually pass from their playbooks (an ee edition, a nightly channel, a pinned package version in docker_packages), because the spec turns an undocumented value that used to work into a refusal, and decides whether min_ansible_version in meta/main.yml needs a bump to the ansible-core that validates specs.
Run it on every role released to Galaxy
A team that maintains several roles starts one run per repository from the terminal with POST /o/{org}/sessions, this prompt and the role's repository in resources. Each run reads that role's own defaults, tasks and README, so a role with two entry points gets two entries in the spec, and each opens its own pull request.
With a webhook_url on the API key, run.completed arrives with data.session_url for each run, so the pull request links collect in one place. The release.yml workflow that imports the role into Galaxy on a tag is untouched; the spec ships with the next tag like any other file.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
Writing an argument spec is reading: thirty variables across defaults, tasks and a README, each with a type the tasks assume and values the prose allows. The failure mode is a spec that is subtly stricter than the role, which only shows up when a playbook that used to work is refused. In a Rush sandbox the agent converges the role through its own Molecule scenario with the spec in place, then feeds it bad values on purpose and keeps the refusals in the run's record, so the pull request shows the validation doing its job rather than describing it.
Molecule needs a privileged container with systemd and a distro image pull, which is slow and messy on a laptop and should not share a Docker daemon with anything else. The sandbox is its own machine, torn down afterwards, and the same request works on every role the team publishes.
Frequently asked questions
Will the spec change how the role behaves with valid values?
No. Validation is one task inserted at the start of the role, and the spec copies the defaults and the documented choices, so a playbook that passes valid values runs the same tasks as before. Only a wrong type or an undocumented value is refused.
What about variables whose default is a Jinja template?
docker_compose_arch, docker_compose_url, docker_apt_repo_url and the yum URLs default to expressions over facts. The spec declares their type and describes the template; the value is still computed from defaults/main.yml when the caller passes nothing.
Which Ansible version does this need?
Role argument validation arrived in ansible-core 2.11, and meta/argument_specs.yml is the file to use for compatibility with older controllers, which ignore it. This role's min_ansible_version is 2.15.1, so every supported controller validates.
Does it touch the Molecule scenario?
No. The default scenario is run as it is to show a normal converge still passes. The two bad-value runs are ad hoc ansible-playbook invocations against the converged instance, kept in the run's record, not new scenario files.
More Ansible jobs
More jobs like this: Hand off a long task
- Add Build Provenance Attestations to a Rust CLI's Release Workflow (Rust CLI)
- Add Are the Types Wrong to CI for a TypeScript Library (TypeScript library)
- Add an Audit Log to a Fastify API (Fastify)