All jobs

Fix Flaky Playwright Tests

Yes. The agent repeats each spec until the flaky ones show, finds the cause in the test or its config, fixes it without adding retries and opens a pull request.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

A flaky Playwright test is one the configuration is hiding, so the agent reads the configuration before any spec. In a suite laid out like MarcusFelling/demo.playwright, one root package.json pins @playwright/test and twelve folders each carry their own playwright.config.ts. Most of them set retries: process.env.CI ? 2 : 0, workers: process.env.CI ? 1 : undefined and trace: 'on'. Two retries and a single worker in CI mean a test that fails one time in three still ends green, and a test that only breaks beside another test never runs beside one. The reusable workflow .github/workflows/template-test.yml goes further: it runs npx playwright test --retries 0 under continue-on-error: true, so a red test does not fail the job at all. The agent writes these down first, because they decide what a failure in CI can and cannot tell anyone.

Then it makes the flakiness show. It runs npm ci and npx playwright install --with-deps for the browsers the configs name, and for each folder runs npx playwright test --repeat-each=20 --retries=0 --reporter=json, once with one worker and once with several. A test that fails in some repeats and passes in others is flaky by measurement, and the JSON report gives its file, line, project and error for every repeat. A test that fails every time is a broken test and goes on a separate list. A test that fails only with several workers shares something with its neighbours.

The causes fall into families, and the agent names the family before it edits. A promise nobody awaited: in todomvc/tests/example.spec.ts the helper checkNumberOfCompletedTodosInLocalStorage(page, 1) is called without await just before page.reload(), while every other call to those helpers is awaited, so its page.waitForFunction can settle after the page it was watching is gone. State outside the test: github-api/tests/example.spec.ts creates a repository named Test-Repo-1 in beforeAll and deletes it in afterAll, so two runs at once, or a retry in a fresh worker, meet a repository that already exists. A fixed port: the Lighthouse test in performance/tests/example.spec.ts launches Chromium with --remote-debugging-port=9222 and asserts a performance score of 90 against a live site. A page the team does not own: visual-comparison/tests/example.spec.ts screenshots .heroTitle_ohkl, a generated class name on playwright.dev, with maxDiffPixels: 1.

Each family has a fix that removes the cause. The missing await is added. A fixed name gets a suffix from testInfo so parallel runs cannot collide. A fixed port becomes one derived from workerIndex, the way fixtures/tests/express-test.ts in the same suite already computes 3000 + workerInfo.workerIndex. A locator tied to a generated class becomes a role or text locator. A call to a third-party API the test only needs an answer from is fulfilled with page.route. The agent does not add waitForTimeout, raise a timeout or raise retries; when a test depends on a live site's speed and cannot be made deterministic, it says so and proposes moving it out of the blocking run. It then repeats the same --repeat-each command for every test it touched and keeps both sets of numbers.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Find and fix the flaky tests in this Playwright suite. Read package.json, every playwright.config.ts, every workflow under .github/workflows and every spec, fixture and page object first, and write down the retries, workers, timeouts and trace settings each config uses and whether a failing test can fail CI. Run npm ci and npx playwright install --with-deps for the browsers the configs name. For each folder with a config, run npx playwright test --repeat-each=20 --retries=0 --reporter=json with one worker and again with four, and list every test that passes in some repeats and fails in others, with its file, line, project and error. List tests that fail every time separately and do not treat them as flaky. For each flaky test name the cause from the code: a promise that is not awaited, an assertion that does not wait, state shared between tests or runs (a fixed name, port or file), a dependency on a site or API we do not control, or a selector tied to generated markup. Fix the cause: await the promise, use a web-first assertion, make names and ports unique per worker or test, mock third-party responses with page.route when the test does not exist to exercise them, and use role or text locators. Do not add waitForTimeout, do not raise timeouts or retries, do not skip or delete a test, and do not run tests that need credentials we have not given you; list those as not run. Rerun the same repeat command for every test you changed. Open a pull request with one section per test: the failure before, the cause, the change, and the pass count before and after. Add a final section for tests you could not make deterministic and why.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request with one section per flaky test. Each section quotes the error from a failing repeat, names the cause in a sentence that points at a line, shows the change, and gives two counts: how many of the repeats passed before and how many pass after. The diff stays inside spec files, fixtures and page objects, plus a config when the cause was there. No retries value goes up and no test is skipped.

Two more lists close the description. One holds tests that fail on every repeat, which are broken tests and need a different decision. The other holds tests the agent could not make deterministic or could not run: a Lighthouse score measured against someone else's server, a suite that needs an API token or a mail account, a job the workflow sends to a macOS runner. For each it says what it would take, such as serving a local copy of the page or moving the test to a scheduled job that does not block merges.

A reviewer reads each fix for whether the test still tests the same thing. A mocked response is right when the test is about the page and wrong when it is about the API. A reviewer also checks the workflow: if CI runs with continue-on-error: true, the repeats in the description are the only evidence the fix holds, and whether to let failures fail the job from now on is a decision for the team.

Run it on every test reported flaky in CI

With retries on, Playwright marks a test that failed and then passed as flaky in its JSON report, and that is the signal to act on. Add the json reporter to the config, and add a step at the end of the test job that reads the report and, when any test has that status, posts the list of titles and files with curl to the URL of a webhook trigger on an automation in the Project. The step sends the trigger's secret, kept as an Actions secret, as a bearer token in the Authorization header, and gives the trigger's URL an Idempotency-Key header set to the workflow run id, so a re-run job starts nothing twice. The posted list arrives after the saved prompt as data, and the agent starts from those tests.

A suite that has never been measured starts with a schedule instead: a schedule trigger with a cron for one night a week runs this prompt over the whole suite. The automation's max runs per hour bounds how many runs a bad day in CI can start.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

Measuring flakiness means running the same tests many times in several browsers, with one worker and with many, and then again after each fix. On a laptop that is an afternoon of a machine nobody can use, and the result depends on what else that laptop was doing. A sandbox has nothing else running, installs the browsers itself, and several of them can work on different folders of a suite at once.

The numbers come back with the diff. Every repeat command and its output are in the run's transcript at GET /o/{org}/sessions/{id}/messages, so a claim that a test went from failing some repeats to failing none can be checked line by line, and a reviewer who doubts a fix can send a follow-up to the same session asking for more repeats.

Frequently asked questions

Will it fix flakiness by adding retries?

No. The prompt forbids raising retries or timeouts and adding waitForTimeout, because each of those hides the cause and slows the suite. When a test cannot be made deterministic, the agent reports that in the pull request and leaves the retry decision to the team.

Our tests run against a staging site. Can it still reproduce the failures?

It can if the sandbox can reach the site over the public internet with the same baseURL the config reads. A site behind a VPN or an allowlist is out of reach, and the agent then works from the reports and traces CI uploaded and says the fix was reasoned from them.

Does it use the Playwright Docker image our CI uses?

No. The sandbox has no Docker, so the agent installs the pinned @playwright/test with npm ci and the matching browsers with npx playwright install --with-deps. The browser builds are the ones that Playwright version ships, the same as in the mcr.microsoft.com/playwright image with that tag; system fonts can differ, which matters for screenshot tests and is called out when one is involved.

More Playwright jobs

The same job on other stacks

Keep reading