What the agent does
The agent first gets the suite running exactly as CI runs it. In a Phoenix app like hexpm/hexpm it reads .github/workflows/main.yml: Elixir 1.20.4 on OTP 29.1, MIX_ENV=test, a postgres:18 service, and mix test, which mix.exs aliases to ecto.create --quiet, ecto.load --skip-if-loaded, ecto.migrate and test. It starts Postgres in its sandbox and runs the named test once to confirm it passes, then runs it in a loop, mix test path/to/file_test.exs:LINE --seed 0 and with random seeds, recording each seed that fails. A failure that appears only with the whole file or the whole suite points at another test.
Then it looks for the shared thing. test/test_helper.exs puts the Ecto sandbox in :manual mode and DataCase and ConnCase check out a connection per test, so the database is isolated, but plenty else is not. Around 190 test files are async: true, and some of them touch global state: install_controller_test.exs calls Application.put_env(:hexpm, :cdn_url, …) without declaring async: false, several files call Mox.set_mox_global(), and the GitHub secret-scanning controller test runs :ets.delete(Hexpm.Cache). Any async test that reads the same config key, mock or table can see another test's value.
Time is the other common source. cache_test.exs is async: true and waits with Process.sleep(100) and Process.sleep(200) around entries stamped with System.monotonic_time(:second); the device-code tests compute expiry from DateTime.utc_now(); and test/support/test_helpers.ex already carries align_to_throttle_bucket, a sleep that dodges a time-bucket boundary, a sign of one flake fixed by waiting. A sleep that is long enough on a fast laptop and too short on a loaded CI runner is a race.
Once it has a failing seed, it fixes the cause rather than the symptom: mark a test that mutates application env or uses global Mox as async: false, restore env in on_exit, inject the clock or pass the time in as an argument, replace a sleep with assert_receive on a message the code sends when it is done, or give each test its own ETS table name. Then it runs the test 50 times, with the failing seed among them, and the whole file and the suite once, and records the counts. It never adds a retry or a longer sleep.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "The ExUnit test named below fails intermittently in CI. Start Postgres and run the suite as CI does. Run the test in a loop, alone and with its whole file, with random seeds, until it fails, and record the failing seed. Find why: async tests sharing application env, Mox global mode, ETS tables or other global state, wall-clock time, a sleep standing in for synchronization, randomness, or test order. Fix the cause in the test or the code. Then run the test 50 times, including the failing seed, plus the whole file and the full suite once. Never add a retry, a longer sleep or a skip. Open a pull request with the cause, the failing seed, and the failure counts before and after.\n\nTest: {test_file}:{line}",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that changes as little as the cause needs: often one test module's use line (async: true to async: false), an on_exit that restores Application env, or an assert_receive in place of a Process.sleep, and sometimes the code under test, when the race is real and production could hit it too. The description states the cause in a sentence, gives the seed that reproduced it so anyone can rerun mix test --seed, and reports how many of the loop runs failed before the fix and after.
The pull request's own checks run the suite in CI on the fix. The trajectory keeps every loop the agent ran with its output, so a reviewer can see the failure reproduce, the investigation that ruled out other causes, and the clean runs afterwards.
Before merging, a reviewer checks that the fix is not a disguised retry (a bigger timeout on assert_receive is still a wait), that switching a module to async: false did not hide a bug the code should not have, and whether the same pattern exists in sibling tests that will flake next.
Run it on every test flagged as flaky
Most teams already notice flakes: a test that passed on rerun in CI, a label on an issue, a flaky-test report. Wherever that signal lives, have it send this request with the test's file and line in input and the repository in resources, one run per test. Use the test's file and line as the Idempotency-Key header so the same flake reported twice starts one fix.
Give the API key a webhook URL and Rush POSTs run.completed or run.failed when the turn ends. If the agent could not reproduce the failure in its loops, it says so in result and opens no pull request, which is worth knowing too: the flake may depend on CI's hardware or load.
The use case this job follows, with its automation and trigger: Fix a bug from an alert.
Why run it on Rush?
Finding a flake takes repetition: hundreds of runs, several seeds, the whole file and the whole suite, while a laptop is needed for other work. On Rush the agent loops the test in its own sandbox with its own Postgres for as long as it takes, and keeps every run's output, so the failing seed and the before and after counts in the pull request come from runs a reviewer can read.
Each flake is its own run, so a team can work down a list of them in parallel, each fix reviewed on its own merit. And the result is a fix rather than a retry annotation, which keeps the suite honest: a test that passes 50 times in a row with the seed that used to break it is evidence, not hope.
Frequently asked questions
What if the test only fails in CI, never locally?
The agent runs it under the same Elixir and OTP versions and Postgres image as CI and with the whole suite, which reproduces most order and shared-state flakes. For flakes that depend on CPU load, it can run the suite with --max-cases raised or under artificial load. If it still cannot reproduce it, it reports the loops it ran and the suspects it found instead of guessing a fix.
Will it just make every test async: false?
No. It changes the module that shares state, and only when isolating the state is not practical. Where a test can own its state (its own ETS table name, a per-test Mox expectation in private mode, a clock passed in), that is the better fix and keeps the suite fast.
Does it work with Mox and Ecto sandbox setups?
Yes, and those are where many Elixir flakes live. A test in Mox global mode must not be async, and a process that outlives its test can hold a sandbox connection; the agent checks both. It reads your DataCase and ConnCase first to see how checkouts and stubs are set up.
Why 50 runs?
It is enough to show a test that used to fail in a few percent of runs now passes repeatedly, and short enough to finish quickly for a single test. Raise the number in the prompt for a rarer flake; the agent reports the counts it actually ran.