# Add Test Retries to a Playwright Suite

Yes. An agent turns on retries for CI only, keeps the first failure's trace, makes flaky passes visible, fixes the timing bugs it finds, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-test-retries/playwright/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads how the suite runs today before it changes a line. In a Next.js chat app laid out like vercel/chatbot, `playwright.config.ts` sets `retries: 0` under a comment that says "Retry on CI only", `fullyParallel: true`, `workers: 2` whether or not `CI` is set, a single `e2e` project on Desktop Chrome matching `e2e/.*.test.ts`, `trace: "retain-on-failure"`, the `html` reporter, and a `webServer` that runs `pnpm dev` and waits on `/ping`. Both the test timeout and `expect.timeout` are 240 seconds. `package.json` runs the suite as `export PLAYWRIGHT=True && pnpm exec playwright test` on `@playwright/test` 1.61, and that flag is what `lib/constants.ts` reads as `isTestEnvironment`, so `lib/ai/providers.ts` swaps the gateway for the mock models in `lib/ai/models.mock.ts`. `.github/workflows/playwright.yml` runs it on every push and pull request to `main`, installs Chromium with its system dependencies, and uploads `playwright-report/` for seven days.

It then looks for the reasons a test would fail once and pass on the rerun, because a retry hides those rather than fixing them. `tests/helpers.ts` builds each test user's email as `test-<unix seconds>@playwright.com`: with two workers and `fullyParallel`, two registrations in the same second get the same address, and the second one fails on a duplicate email. `tests/e2e/chat.test.ts` has a stop-button test that clicks with a five-second timeout and swallows the error, which passes whether or not the button ever appeared. The agent changes the helper to include the worker index and a random suffix, and rewrites the stop test to assert what it can see with the mock model's word-by-word stream.

Then the retry policy itself. `retries` becomes `process.env.CI ? 2 : 0`, so a developer running locally still sees a failure the first time. `trace` becomes `on-first-retry`, which records the retry of a test that already failed once, and the reporter becomes a list on CI: `html` as before, `github` so a failing test is annotated on the pull request's checks tab, and `json` written to `playwright-report/results.json`. The JSON report marks each test's `outcome` as `expected`, `unexpected`, `flaky` or `skipped`, and the agent adds a short Node script, run as a workflow step after the suite, that prints every `flaky` test's title so a pass that needed a retry is still read by someone.

It runs the suite in its sandbox with `PLAYWRIGHT=True` and a PostgreSQL it installs there, since the app's auth and chat routes write to the database through `lib/db/queries.ts`, first with `retries: 0` to see which tests fail at all and then with the new configuration, and keeps both outputs for the pull request. It also runs `pnpm check`, the Biome configuration behind `ultracite`, and `tsc --noEmit`.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add Playwright retries for CI only. Read playwright.config.ts, package.json, .github/workflows/playwright.yml, tests/fixtures.ts, tests/helpers.ts and every file under tests/e2e first. Set retries to 2 on CI and 0 locally, record traces on the first retry, and on CI use the html, github and json reporters, writing the JSON report to playwright-report/results.json. Add a script, run as a workflow step after the tests, that reads that file and prints the title of every test whose outcome is flaky, so a retried pass is visible. Before turning retries on, look for tests that are flaky by construction: unique test data that is not unique across parallel workers, and assertions that swallow their own failure. Fix those rather than relying on the retry. Run the suite with PLAYWRIGHT=True against a local PostgreSQL once with retries 0 and once with the new config, run pnpm check and tsc --noEmit, and open a pull request with both outputs in the description.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that changes `playwright.config.ts`, `tests/helpers.ts`, one test in `tests/e2e/chat.test.ts`, `.github/workflows/playwright.yml` and adds the small report script. The description lists each configuration key it changed and why, the test it rewrote, and the two suite outputs: the run with retries off, which shows which tests fail without help, and the run with the new configuration, which shows which of those passed on a retry.

The run's record keeps every command and its output, so a reviewer can read the first run's failures rather than take the summary's word for them. A reviewer checks three things: that the fix to the helper really makes emails unique across workers, that the rewritten stop test asserts something the mock stream guarantees, and that the flaky list step runs even when the suite fails, since that is when it matters.

A repository whose tests all pass with retries off gets the configuration change and the report step only, and the description says so.

## Run it on every repository whose CI reruns on red

A team with several Playwright suites starts one run per repository with one `POST /o/{org}/sessions` each, this prompt as `input` and the repository in `resources`. Each run reads its own config, so a suite that already retries on CI gets only the reporters and the flaky list, and one with no CI workflow gets the config change and a description saying where the step would go.

Send a `request_id` that is a UUID v5 of the repository's full name so a retried call returns the run it already started. The key's `webhook_url` reports when each run finishes, with `session_url` to read the pull request from.

## Why run it on Rush?

Turning on retries is one line, and that line is how a suite quietly stops telling the truth: a test that fails one time in three passes CI two times in three, and nobody reads the HTML report to find the word flaky. The useful work is the part before the line, reading each test for the race it contains, and the part after, making a retried pass something a person sees. The agent does the reading in its sandbox with the suite actually running, against a real database and the project's own mock models, and keeps the first run's failures as evidence.

A company with many suites runs this on all of them at once, each in its own sandbox, and gets one pull request per repository that says which tests were flaky by construction and which were only slow.

## Frequently asked questions

### Why not retry locally too?

Because a developer who sees the failure the first time fixes it, and one who sees it on the third run of a retry stops noticing. Retries on CI absorb the infrastructure noise a shared runner adds; a laptop run should stay strict. The config reads `process.env.CI`, which GitHub Actions sets, so no script changes.

### What does on-first-retry change compared with retain-on-failure?

`retain-on-failure` records a trace for every test and keeps it only when the test fails, which costs time on every passing test. `on-first-retry` records only when a test runs again after failing, so the trace exists exactly for the runs a reviewer opens. With retries at 0 locally no trace is recorded at all, and the agent says so in the description.

### Will the flaky list fail the build?

Not in this pull request. The step prints the titles so they show in the job log, and the agent explains in the description how to make it exit non-zero instead, which is one line, if the team wants a retried pass to block merging. Playwright's own exit code stays what it is: a test that passes on a retry passes the run.

### Does this touch the tests' timeouts?

No. The 240-second test and expect timeouts stay, because they cover real model latency outside the mocked environment. The agent reports them in the description as something to revisit once retries are in place, since a hung test now costs up to three times that long on CI.

## More Playwright jobs

- [Shard Playwright Tests Across GitHub Actions Runners](https://getrush.ai/run/add-test-sharding/playwright/)
- [Set Test, Expect and Global Timeouts in a Playwright Suite](https://getrush.ai/run/add-test-timeouts/playwright/)
- [Add Visual Regression Tests With Playwright](https://getrush.ai/run/add-visual-regression-tests/playwright/)

## The same job on other stacks

- [Add Test Retries for Flaky Jest Tests in CI](https://getrush.ai/run/add-test-retries/jest/)

## More jobs like this: Write the missing tests

- [Add Test Timeouts to a pytest Suite](https://getrush.ai/run/add-test-timeouts/pytest/)
- [Roll Back the Database After Each pytest Test in a FastAPI App](https://getrush.ai/run/add-transactional-test-fixtures/pytest/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
