# Fix Flaky Playwright Tests

Yes. The agent repeats each spec until the flaky ones show, finds the cause in the test or its config, fixes it without adding retries and opens a pull request.

[Run this on Rush](https://cloud.getrush.ai/run/fix-flaky-tests/playwright/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

A flaky Playwright test is one the configuration is hiding, so the agent reads the configuration before any spec. In a suite laid out like MarcusFelling/demo.playwright, one root `package.json` pins `@playwright/test` and twelve folders each carry their own `playwright.config.ts`. Most of them set `retries: process.env.CI ? 2 : 0`, `workers: process.env.CI ? 1 : undefined` and `trace: 'on'`. Two retries and a single worker in CI mean a test that fails one time in three still ends green, and a test that only breaks beside another test never runs beside one. The reusable workflow `.github/workflows/template-test.yml` goes further: it runs `npx playwright test --retries 0` under `continue-on-error: true`, so a red test does not fail the job at all. The agent writes these down first, because they decide what a failure in CI can and cannot tell anyone.

Then it makes the flakiness show. It runs `npm ci` and `npx playwright install --with-deps` for the browsers the configs name, and for each folder runs `npx playwright test --repeat-each=20 --retries=0 --reporter=json`, once with one worker and once with several. A test that fails in some repeats and passes in others is flaky by measurement, and the JSON report gives its file, line, project and error for every repeat. A test that fails every time is a broken test and goes on a separate list. A test that fails only with several workers shares something with its neighbours.

The causes fall into families, and the agent names the family before it edits. A promise nobody awaited: in `todomvc/tests/example.spec.ts` the helper `checkNumberOfCompletedTodosInLocalStorage(page, 1)` is called without `await` just before `page.reload()`, while every other call to those helpers is awaited, so its `page.waitForFunction` can settle after the page it was watching is gone. State outside the test: `github-api/tests/example.spec.ts` creates a repository named `Test-Repo-1` in `beforeAll` and deletes it in `afterAll`, so two runs at once, or a retry in a fresh worker, meet a repository that already exists. A fixed port: the Lighthouse test in `performance/tests/example.spec.ts` launches Chromium with `--remote-debugging-port=9222` and asserts a performance score of 90 against a live site. A page the team does not own: `visual-comparison/tests/example.spec.ts` screenshots `.heroTitle_ohkl`, a generated class name on playwright.dev, with `maxDiffPixels: 1`.

Each family has a fix that removes the cause. The missing `await` is added. A fixed name gets a suffix from `testInfo` so parallel runs cannot collide. A fixed port becomes one derived from `workerIndex`, the way `fixtures/tests/express-test.ts` in the same suite already computes `3000 + workerInfo.workerIndex`. A locator tied to a generated class becomes a role or text locator. A call to a third-party API the test only needs an answer from is fulfilled with `page.route`. The agent does not add `waitForTimeout`, raise a timeout or raise `retries`; when a test depends on a live site's speed and cannot be made deterministic, it says so and proposes moving it out of the blocking run. It then repeats the same `--repeat-each` command for every test it touched and keeps both sets of numbers.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Find and fix the flaky tests in this Playwright suite. Read package.json, every playwright.config.ts, every workflow under .github/workflows and every spec, fixture and page object first, and write down the retries, workers, timeouts and trace settings each config uses and whether a failing test can fail CI. Run npm ci and npx playwright install --with-deps for the browsers the configs name. For each folder with a config, run npx playwright test --repeat-each=20 --retries=0 --reporter=json with one worker and again with four, and list every test that passes in some repeats and fails in others, with its file, line, project and error. List tests that fail every time separately and do not treat them as flaky. For each flaky test name the cause from the code: a promise that is not awaited, an assertion that does not wait, state shared between tests or runs (a fixed name, port or file), a dependency on a site or API we do not control, or a selector tied to generated markup. Fix the cause: await the promise, use a web-first assertion, make names and ports unique per worker or test, mock third-party responses with page.route when the test does not exist to exercise them, and use role or text locators. Do not add waitForTimeout, do not raise timeouts or retries, do not skip or delete a test, and do not run tests that need credentials we have not given you; list those as not run. Rerun the same repeat command for every test you changed. Open a pull request with one section per test: the failure before, the cause, the change, and the pass count before and after. Add a final section for tests you could not make deterministic and why.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

A pull request with one section per flaky test. Each section quotes the error from a failing repeat, names the cause in a sentence that points at a line, shows the change, and gives two counts: how many of the repeats passed before and how many pass after. The diff stays inside spec files, fixtures and page objects, plus a config when the cause was there. No `retries` value goes up and no test is skipped.

Two more lists close the description. One holds tests that fail on every repeat, which are broken tests and need a different decision. The other holds tests the agent could not make deterministic or could not run: a Lighthouse score measured against someone else's server, a suite that needs an API token or a mail account, a job the workflow sends to a macOS runner. For each it says what it would take, such as serving a local copy of the page or moving the test to a scheduled job that does not block merges.

A reviewer reads each fix for whether the test still tests the same thing. A mocked response is right when the test is about the page and wrong when it is about the API. A reviewer also checks the workflow: if CI runs with `continue-on-error: true`, the repeats in the description are the only evidence the fix holds, and whether to let failures fail the job from now on is a decision for the team.

## Run it on every test reported flaky in CI

With retries on, Playwright marks a test that failed and then passed as `flaky` in its JSON report, and that is the signal to act on. Add the `json` reporter to the config, and add a step at the end of the test job that reads the report and, when any test has that status, posts the list of titles and files with curl to the URL of a `webhook` trigger on an automation in the Project. The step sends the trigger's secret, kept as an Actions secret, as a bearer token in the `Authorization` header, and gives the trigger's URL an `Idempotency-Key` header set to the workflow run id, so a re-run job starts nothing twice. The posted list arrives after the saved prompt as data, and the agent starts from those tests.

A suite that has never been measured starts with a schedule instead: a `schedule` trigger with a `cron` for one night a week runs this prompt over the whole suite. The automation's max runs per hour bounds how many runs a bad day in CI can start.

## Why run it on Rush?

Measuring flakiness means running the same tests many times in several browsers, with one worker and with many, and then again after each fix. On a laptop that is an afternoon of a machine nobody can use, and the result depends on what else that laptop was doing. A sandbox has nothing else running, installs the browsers itself, and several of them can work on different folders of a suite at once.

The numbers come back with the diff. Every repeat command and its output are in the run's transcript at `GET /o/{org}/sessions/{id}/messages`, so a claim that a test went from failing some repeats to failing none can be checked line by line, and a reviewer who doubts a fix can send a follow-up to the same session asking for more repeats.

## Frequently asked questions

### Will it fix flakiness by adding retries?

No. The prompt forbids raising `retries` or timeouts and adding `waitForTimeout`, because each of those hides the cause and slows the suite. When a test cannot be made deterministic, the agent reports that in the pull request and leaves the retry decision to the team.

### Our tests run against a staging site. Can it still reproduce the failures?

It can if the sandbox can reach the site over the public internet with the same `baseURL` the config reads. A site behind a VPN or an allowlist is out of reach, and the agent then works from the reports and traces CI uploaded and says the fix was reasoned from them.

### Does it use the Playwright Docker image our CI uses?

No. The sandbox has no Docker, so the agent installs the pinned `@playwright/test` with `npm ci` and the matching browsers with `npx playwright install --with-deps`. The browser builds are the ones that Playwright version ships, the same as in the `mcr.microsoft.com/playwright` image with that tag; system fonts can differ, which matters for screenshot tests and is called out when one is involved.

## More Playwright jobs

- [Review Pull Requests in a Playwright Test Suite](https://getrush.ai/run/review-pull-requests/playwright/)
- [Upgrade Playwright and Its Browsers in a Test Suite](https://getrush.ai/run/upgrade-playwright/playwright/)
- [Add Accessibility Tests to a Playwright Suite](https://getrush.ai/run/add-accessibility-tests/playwright/)

## The same job on other stacks

- [Fix pytest Tests That Pass Once and Fail on the Next Run](https://getrush.ai/run/fix-flaky-tests/pytest/)
- [Fix Flaky Tests in a Rails App](https://getrush.ai/run/fix-flaky-tests/rails/)
- [Fix Flaky Tests in a React App](https://getrush.ai/run/fix-flaky-tests/react/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
