What the agent does
In a Vue app like mutoe/vue3-realworld-example-app, the first thing the agent reads is where retries hide the flake. playwright.config.ts sets retries: isCI ? 1 : 0, workers: isCI ? 1 : undefined and fullyParallel: false, and the test:e2e script is pnpm build && cross-env CI=true playwright test, so in CI a test that fails once and passes on its retry still reports green with a trace saved on-first-retry. The unit side has no retries: vite.config.ts runs Vitest over src/**/*.spec.ts in happy-dom with globals: true, a setupFiles entry at src/setup-tests.ts, and coverage through vitest-monocart-coverage, which the test script merges with the Playwright coverage in scripts/merge-coverage.ts.
Then it reproduces the failure the alert names, with retries forced off. For an end-to-end spec it installs with pnpm@10.33.0 from packageManager, runs pnpm playwright install --with-deps for Chromium, builds once, and loops CI=true pnpm playwright test playwright/specs/home.spec.ts --retries=0 --repeat-each=20 until the failure shows, reading the trace Playwright saved. For a unit spec it loops npx vitest run src/pages/Article.spec.ts with a fixed --sequence.seed and then with random seeds, because a test that only fails after another test is an ordering bug, not a timing one.
The causes it looks for are the ones this test setup makes likely. The e2e specs intercept the API with fixtures: playwright/specs/home.spec.ts registers conduit.intercept routes for articles?tag=, articles?limit, articles/ and tags in beforeEach, so a route pattern that matches too late, or a request the page makes before the intercept is registered, answers from the real VITE_API_HOST instead of the fixture. The autoTestFixture in playwright/extends.ts starts JS and CSS coverage on every page, which adds time to the first navigation, so an assertion with a short timeout can be racing the coverage hook. On the unit side the Pinia store in src/store/user.ts persists the token to localStorage through the Storage utility, and a spec that signs in and never clears it leaks state into the next spec in the same file.
The fix goes where the cause is. A late intercept moves ahead of the navigation or widens its pattern; an assertion that read a value before the page settled becomes a web-first expect on the element rather than a sleep; leaked localStorage gets cleared in the spec's afterEach; a component test that depended on a sibling's mock gets its own. When the test caught a real bug in a component or composable, that code changes instead. Then the agent runs the spec with --repeat-each=20 and retries off to show it passes every time, runs the whole suite once, and opens the pull request with both counts.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "The test named in the alert below fails intermittently. Install with the repository's package manager, and for an end-to-end spec install the Playwright browsers and build the app as the test:e2e script does. Run that one spec with retries turned off and repeated at least 20 times until it fails, and read the trace, screenshot or video Playwright saved; for a Vitest spec, repeat it with a fixed sequence seed and then with random seeds to separate ordering from timing. Find the cause: a route intercept registered too late or matching too narrowly, an assertion racing a navigation or the coverage hook, state leaking through localStorage or a Pinia store between specs, shared fixture data, time or randomness. Fix the test or the code behind it, run the spec 20 more times with retries off to show it now passes every time, run the full suite once, and open a pull request with the before and after counts and the cause in one paragraph. Never raise retries, add a fixed wait or skip the test.\n\nAlert: <paste the failing spec, test title and CI log here>",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request changes the spec and, where the test caught a real rendering or state bug, the component, composable or store behind it. The description names the cause in one paragraph, shows the loop that reproduced it with retries off and how many of the runs failed, and the 20 passing runs afterwards, both as the commands the agent ran. When Playwright saved a trace or screenshot of the failure, the agent returns it as a file on the session so the reviewer can see what the page looked like.
A reviewer checks that the fix is the cause and not a wider timeout, that no test.skip, retries or fixed waitForTimeout was added, and that the full-suite run passed. The intercept and afterEach changes are small and local, which makes the diff short to read.
The session read returns the repository and the pull request, and the transcript holds every run in order: the install, the browser download, the build, each repeated spec run with its result, and the final suite.
Run it on every test that passed on retry
The signal is a CI job that went green on its retry. Add an automation to the Project with this prompt and a webhook trigger; a handler on your server receives your CI provider's job-completed event, keeps only runs where a test passed on retry, and forwards the spec name and log to the trigger URL with Authorization: Bearer <its secret>. The payload reaches the agent after the prompt as data, never as instructions, and each forwarded alert starts its own run on its own branch.
A schedule trigger with a nightly cron in your timezone is the other way in: a prompt that runs the whole suite 5 times with retries off and fixes whatever fails at least once, so flakes are found before they block someone's pull request.
The use case this job follows, with its automation and trigger: Fix a bug from an alert.
Why run it on Rush?
A flaky test is proved fixed by running it many times, not by reading it, and 20 Playwright runs with a browser and a build in front of them is time nobody wants to spend on a laptop. The sandbox has Chromium, the build and the suite, and the transcript keeps every repeated run, so the before and after counts in the pull request are commands a reviewer can see rather than claims.
The run never touches the developer's own checkout or their CI configuration. It cannot raise retries in playwright.config.ts, cannot skip the test, and leaves a branch with one small fix and the evidence behind it, while the developer who was paged carries on with what they were doing.
Frequently asked questions
Why turn retries off when CI keeps them on?
Because retries: 1 in CI is what hides the failure rate. With retries off and --repeat-each=20, the agent sees how often the spec really fails and gets the trace from the first failure rather than from a retry. CI keeps its own setting; the pull request does not change it.
Can it run Playwright end-to-end tests in the cloud?
Yes. The sandbox installs the Chromium that pnpm playwright install --with-deps asks for, builds the app with pnpm build and serves it with pnpm serve through the webServer block in playwright.config.ts, exactly as the test:e2e script does, with CI=true set so the config takes its CI branch.
Does it handle flaky Vitest unit tests too?
Yes. For a spec under src/**/*.spec.ts the agent repeats it with a fixed --sequence.seed and then with random seeds, which separates a test that fails only after a sibling from one that fails on timing. State leaking through localStorage or a Pinia store between specs in the same file is the common cause in this app.
What if the test is flaky because the real API is slow?
In this app the e2e specs intercept the API with fixtures under playwright/fixtures/, so a request that reaches the real VITE_API_HOST is itself the bug: an intercept registered after the navigation or a pattern that does not match. The fix is to the intercept, not a longer timeout.
Why 20 runs instead of 100?
Twenty is enough to catch a flake that fails more than a few percent of the time while keeping the run short, and the prompt says at least 20. If the alert says the failure is rarer than that, raise the number in the prompt and the agent repeats accordingly.