What the agent does
The agent reads every Dockerfile and the workflow that builds it before writing a test. In a repository like dockersamples/example-voting-app there are five: vote/Dockerfile is a three-stage Python image from python:3.11-slim with base, dev and final stages, installs curl for the health check, and ends in gunicorn app:app -b 0.0.0.0:80 with EXPOSE 80; result/Dockerfile is node:18-slim with curl and tini, a global nodemon for development, npm ci followed by a move of node_modules to /node_modules, ENV PORT=80 and ENTRYPOINT ["/usr/bin/tini", "--"]; worker/Dockerfile builds with the .NET 7 SDK on $BUILDPLATFORM and dotnet publish -a $TARGETARCH, then copies into mcr.microsoft.com/dotnet/runtime:7.0 with ENTRYPOINT ["dotnet", "Worker.dll"]. seed-data/Dockerfile and result/tests/Dockerfile are tooling images, the second on node:8.9-slim with PhantomJS. The only existing test, result/tests/tests.sh, posts a vote and greps a rendered page for 1 vote through docker-compose.test.yml, and result/package.json has a test script that exits 1. Three workflows, call-docker-build-{vote,result,worker}.yaml, call a reusable workflow in dockersamples/.github that builds for linux/amd64,linux/arm64,linux/arm/v7 and pushes to GHCR and Docker Hub, each filtered to its own directory.
It writes one structure-test.yaml per application image, in the container-structure-test schema 2.0.0, from what each Dockerfile promises. For vote: a metadataTest that the exposed port is 80, the workdir is /usr/local/app and the command starts with gunicorn; commandTests that python --version reports 3.11 and gunicorn --version and curl --version succeed, since the compose health check depends on curl; fileExistenceTests for app.py, requirements.txt and the templates directory. For result: /usr/bin/tini and /node_modules/express exist, the entrypoint is tini, the environment carries PORT=80, and node --version is 18. For worker: /app/Worker.dll exists, the entrypoint is dotnet Worker.dll, and dotnet --list-runtimes lists the 7.0 runtime and no SDK, which is the check that the final stage is the runtime image and not the build stage. Each test file sits beside its Dockerfile, so the path filters in the existing workflows keep working.
The sandbox has no Docker, so the agent installs it first, sudo apt-get install -y docker.io, starts dockerd, and builds each image with docker build from its own directory, which on the worker image exercises the BUILDPLATFORM and TARGETARCH arguments on the native platform. It installs the container-structure-test binary and runs container-structure-test test --image <tag> --config <dir>/structure-test.yaml for each, reading the pass and fail lines and fixing a test that asserted something the image does not do, such as a dev stage assumption that does not hold in the final stage.
Then it adds .github/workflows/structure-tests.yaml: a matrix over vote, result and worker that builds the image with docker/build-push-action using load: true and no push, installs the test binary, and runs the same command. It runs on pull requests and on pushes to main, with the same per-directory path filters as the existing workflows plus the test file itself, so a change to vote/Dockerfile runs only the vote tests. The reusable build workflow in the other repository is left alone; the new workflow runs beside it.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Write container structure tests for the Docker images in this repository. Read every Dockerfile, the compose files, the health check scripts and every workflow under .github/workflows first. For each application image, write a structure-test.yaml beside its Dockerfile in the container-structure-test 2.0.0 schema: metadata tests for the exposed ports, workdir, entrypoint, command and environment the Dockerfile sets, command tests for the runtime version and the tools later stages depend on, and file existence tests for the application files and anything the Dockerfile copies or moves. Leave tooling images without tests. Install Docker in the sandbox, build each image, install container-structure-test, run it against each image and fix any test that does not match what the image actually does. Add a GitHub Actions workflow with a matrix over the images that builds each one without pushing and runs its tests, with path filters matching the existing build workflows. Open a pull request that shows each test run's output.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that adds vote/structure-test.yaml, result/structure-test.yaml, worker/structure-test.yaml and .github/workflows/structure-tests.yaml, and changes no Dockerfile. The description lists each image's tests in a short table, which Dockerfile line each one checks, and the pass and fail counts from the runs in the sandbox.
The run's record keeps the docker build output for each image and the full container-structure-test output, so a reviewer can see that the worker test found the 7.0 runtime and no SDK, and that result's /node_modules/express check passed because the Dockerfile moves the directory. The tooling images are named as left out, with the reason.
Before merging, a reviewer looks at the new workflow's run on the pull request, where the matrix shows three green jobs, and reads one test file to confirm the assertions are the ones that would catch a regression they care about, such as a base image bump that drops curl and silently breaks the compose health check.
Run it on every service with a Dockerfile
A team with many image-building repositories sends one POST /o/{org}/sessions per repository with this body and the repository in resources; a monorepo with services added over time sends one per new service directory, naming it in input. Each run reads its own Dockerfiles and writes tests for what they actually set, so an image that uses tini gets an entrypoint test and one that does not gets a command test instead. A request_id that is a UUID v5 of the repository and directory makes a retried call return the run it already started.
The key's webhook_url reports each run the key starts, with the pull request URL in result, so a script can open the pull requests in one list without polling the sessions.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
A structure test is only worth having if it was run against the real image, and running it means building the image and installing Docker and the test tool, which the agent does in its own sandbox with the commands and their output kept. A test written from reading the Dockerfile alone would assert a dev stage's tools in a final stage image, and the run catches that before a reviewer does.
Several images, several repositories, one request each: the agent builds and tests them in parallel sandboxes and comes back with one pull request per repository, each with proof that the tests pass on the image as built today.
Frequently asked questions
Why not test the images through the existing reusable workflow?
That workflow lives in another repository and builds for three platforms with push. A separate workflow that builds locally with load: true keeps the tests fast and runs them on pull requests from forks too, where the registry secrets are not available.
Do the tests run on arm64 images?
In the sandbox and the new workflow they run on the native linux/amd64 build. The test file is the same for any platform, so a team that wants arm coverage adds a platform to the build step and a QEMU setup step; the pull request names where.
What happens when a base image bump breaks a test?
The job fails with the test name and the command or path it checked, such as curl --version on the vote image. That is the point: the compose health check and the result image's tini entrypoint depend on packages a slimmer base could drop.
Why were the seed-data and test images skipped?
They are run by hand to generate load and to drive the old PhantomJS check, and nothing deploys them. The pull request says they were left out and what a test for them would look like if the team wants one.
Does the agent change my Dockerfiles?
No. It writes tests for what the Dockerfiles already do. Where it sees something worth changing, such as a build stage's tools reaching the final image, it says so in the description and leaves the change for a separate pull request.
More Docker jobs
More jobs like this: Write the missing tests
- Write Tests for Apache Airflow DAGs (Apache Airflow)
- Write dbt Data Tests (dbt)
- Write Tests for a Django App (Django)