What the agent does
The agent starts by reading how the suite already runs. In an application like hexpm/hexpm it opens mix.exs and finds elixir: "~> 1.20", a test alias that runs ecto.create --quiet, ecto.load --skip-if-loaded and ecto.migrate before test, test-only dependencies such as mox, stream_data, lasso and ex_machina, and no coverage dependency and no test_coverage key. It reads test/test_helper.exs, which starts Hexpm.Store.Memory, builds a registry and puts Hexpm.RepoBase into manual sandbox mode, so coverage has to wrap the whole mix test alias rather than a bare mix test.
Then it reads .github/workflows/main.yml. The Test Hexpm job pins otp: 29.1 and elixir: 1.20.4, starts a postgres:18 service with a health check, restores deps and _build from a cache keyed on mix.lock and config/test.exs, runs mix deps.get and mix compile, then mix test. A separate Format job runs mix compile --warnings-as-errors and mix format --check-formatted. The agent keeps all of that and changes only the test step, so the cache keys and the service container stay as they are.
It runs the suite in its sandbox against a Postgres it starts itself, first as mix test to see the baseline pass, then as mix test --cover to get the built-in test_coverage report and its per-module percentages. From that output it picks a threshold a little under the current total and adds test_coverage: [threshold: <that number>, summary: [threshold: <that number>]] to project/0 in mix.exs, and adds cover/ to .gitignore next to the existing _build and deps entries. When the prompt asks for an HTML report or a Codecov upload, it adds excoveralls as a test-only dependency and uses mix coveralls.json instead, uploading cover/excoveralls.json with codecov/codecov-action and the token from a repository secret.
Finally it changes the Run tests step in main.yml to mix test --cover, which fails the job when total coverage drops below the threshold, and uploads cover/ as a workflow artifact so the report can be read from the run. It runs mix format --check-formatted on its own change to mix.exs, pushes the branch, opens the pull request and reads the Test Hexpm job on it. If the job fails, for example because the threshold it chose is too close to the real figure on a different seed, it adjusts and pushes again until the job is green.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add test coverage to this Elixir project's CI. Read mix.exs, test/test_helper.exs and .github/workflows/*.yml first. Run the test suite with mix test --cover in your sandbox and use the result to pick a coverage threshold slightly under the current total. Add test_coverage with that threshold to mix.exs, add cover/ to .gitignore, change the CI test step to run with --cover so the job fails under the threshold, and upload the cover directory as a workflow artifact. Do not change the cache keys, the service containers or the pinned OTP and Elixir versions. Run mix format --check-formatted on your change. Open a pull request and make sure the test job passes on it; if it fails, fix the cause and push again.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request on a new branch with three files changed: mix.exs gains a test_coverage entry with the threshold, .gitignore gains cover/, and .github/workflows/main.yml runs mix test --cover and uploads the cover/ directory as an artifact. When excoveralls was requested, mix.exs and mix.lock also gain that dependency and the workflow uploads cover/excoveralls.json to Codecov. The pull request description quotes the coverage total the agent measured in its sandbox and explains why the threshold sits where it does.
The proof is the Test Hexpm job on the pull request itself, which now prints the coverage summary and fails under the threshold, plus the uploaded cover/ artifact a reviewer can download from that run. The session's trajectory keeps the full mix test --cover output from the sandbox, including each module's percentage, so the number in the description can be checked against what the agent saw.
Before merging, a reviewer checks that the threshold leaves room for normal variance between runs, that the modules with the lowest percentages are ones the team is comfortable leaving for now, and that the artifact retention fits how long they want reports kept. The threshold is a floor to raise over time, not a target the pull request claims to have met.
Run it on every Elixir repository without a coverage number
Run this once per repository: have a script or your server send the same request with each repository's URL in resources, with a fresh UUID as request_id so a retried request returns the run it already started instead of a second pull request. Each run has its own sandbox, its own Postgres and its own branch, so ten repositories get ten pull requests at once. The Rush GitHub App must be installed on each; GET /o/{org}/github lists the repositories it can reach.
Set a webhook URL on the API key and Rush POSTs run.completed or run.failed when each run's turn finishes, with session_url so your server can post the pull request link and the measured coverage total to the team's channel. A larger backfill across Phoenix apps needs no queue of its own: a key runs max_concurrent at a time, 4 by default and up to 100, and each request beyond that answers 202 with status: "queued" and starts, oldest first, when a slot frees.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
Picking a coverage threshold means running the whole suite with coverage against a real database, reading the summary, and then watching a CI run to see whether the number holds. On a laptop that is a Postgres you have to have running, a suite that takes the machine over, and a push-and-wait loop on a branch you cannot leave. On Rush the agent runs the suite in its own sandbox with its own Postgres, reads the real total, and does the push-and-fix loop itself.
The result comes with its evidence. The job on the pull request prints the coverage summary, the uploaded artifact holds the report, and the trajectory keeps the sandbox run the threshold was chosen from. Because each run is independent, the same request can add coverage to every Elixir service in the organization in parallel, each behind a pull request a person reviews before anything changes on the default branch.
Frequently asked questions
Do I need excoveralls, or is the built-in coverage enough?
Mix's built-in test_coverage prints a per-module summary, fails under a threshold, and writes an HTML report to cover/ with no new dependency, which is what the agent adds by default. Ask for excoveralls in the prompt when you want JSON or LCOV output for Codecov or Coveralls, and the agent adds it as a test-only dependency and uses mix coveralls.json instead.
Will it change my cache keys or service containers?
No. The prompt tells it not to, and the only change to the workflow is the test step and a new artifact upload step. The deps and _build cache keyed on mix.lock and config/test.exs, the postgres:18 service and the pinned OTP and Elixir versions stay exactly as they were, so the first run on the pull request still hits the cache.
How does it choose the threshold?
From the total it measures by running mix test --cover in its sandbox, set a little below that figure so normal variance does not fail the job on day one. The exact number and the reasoning are in the pull request description, and a reviewer can raise or lower it in the diff before merging.
My test alias runs ecto.create and ecto.migrate first. Does coverage still work?
Yes. mix test --cover runs the test alias from mix.exs, so ecto.create --quiet, ecto.load --skip-if-loaded and ecto.migrate run before the suite exactly as they do today, and coverage is collected for the test run that follows. The agent checks this in its sandbox before changing the workflow.
More Elixir jobs
The same job on other stacks
- Add Test Coverage to CI for a Filament App (Filament)
- Add Test Coverage Reporting to a Flutter App's CI (Flutter)
- Add Coverage Reporting to a GitHub Action's CI (GitHub Actions)