# Fix Flaky Tests in a Cypress Suite

Yes. The agent reruns each Cypress spec until it fails, names the cause (live network, fixed waits, timers, server start), fixes it and proves it with clean reruns.

[Run this on Rush](https://cloud.getrush.ai/run/fix-flaky-tests/cypress/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

Flaky end-to-end tests have a short list of causes, and a suite's own files usually show which apply. In a suite like cypress-io/cypress-example-kitchensink, `cypress.config.js` holds a `projectId` and an empty `e2e: {}`: no `baseUrl`, no `retries`, default timeouts. Each of the twenty spec files under `cypress/e2e` visits `http://localhost:8080/...` by full URL. The `test:ci` script is `run-p --race start e2e`, which launches the static server and Cypress side by side, while `local:run` uses `start-test 8080 cy:run`, which waits for the port first. And the app under test, in `app/assets/js/scripts.js`, sends its comment requests to `https://jsonplaceholder.cypress.io`, a server the suite does not control.

The agent measures before it edits. It runs `npm ci`, installs the Linux packages Cypress's install guide lists, Xvfb among them, and checks the binary with `npx cypress verify`. Then, with `npm start` serving `app/` on port 8080, it runs each spec on its own twenty times with `npx cypress run --spec <file>` and records the exit code and failing test title of every pass. A test that fails some of the time is flaky; one that fails every time is broken and goes in a separate list. It repeats the loop with `blockHosts` set to the external host, which shows at once which tests only pass when that server answers.

Reading the specs explains most of what the loop finds. Three files, `aliasing.cy.js`, `waiting.cy.js` and `network_requests.cy.js`, register `cy.intercept('GET', '**/comments/*')` with no reply and then `cy.wait` on the alias, so the request still travels to the live server; two of them accept a status of 200 or 304 and `aliasing.cy.js` insists on 200. `network_requests.cy.js` also counts rows with `cy.request`, and its own comment says the server sometimes returns one extra, which is why the assertion allows 500 or 501. `spies_stubs_clocks.cy.js` schedules two calls with `setTimeout` at 500 and 2500 ms and leans on the default retry window to see the second. `files.cy.js` writes `cypress/fixtures/users.json` from a live response during the run.

Each cause has its own fix, and none of them is a retry setting. An intercepted app request gets a stubbed reply from a fixture, the way `files.cy.js` already answers the same route with `example.json`, so the wait no longer depends on a remote status. An assertion on a shared server's row count becomes an assertion on shape, or on data the test created. A test that waits on real timers moves to `cy.clock()` and `cy.tick()`, which the two clock tests in that file use. A CI script that races the server against the runner switches to the `start-test` form, and `baseUrl` goes into `cypress.config.js` so Cypress checks the server before the first spec. A fixed `cy.wait(1000)` becomes an assertion on what the wait was for; where the sleep is the point of the test, as in `waiting.cy.js`, whose first test is titled for the command it demonstrates, the agent leaves it and notes why. It runs `npm run lint`, which the pre-commit hook also runs, and then the twenty-pass loop again on every spec it touched.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Some tests in this Cypress suite fail intermittently. Read package.json, cypress.config.js, cypress/support, every spec under cypress/e2e, the app code the specs exercise and AGENTS.md if there is one. Run npm ci, install the system packages Cypress needs on Linux including Xvfb, and run npx cypress verify. Start the app with npm start and run each spec on its own 20 times with npx cypress run --spec, recording for every pass the exit code and the failing test. Repeat with blockHosts set to any external host the specs or the app call. List tests that fail sometimes separately from tests that always fail. For each flaky test name the cause and fix it at the cause: stub intercepted requests with a fixture instead of waiting on a live server, assert on shape or on data the test created instead of a shared server's counts, use cy.clock and cy.tick instead of real timers, replace fixed cy.wait(ms) with an assertion on the thing awaited, make the CI script wait for the port, and set baseUrl. Do not add retries to make a test pass, do not skip or delete a test, and leave a wait in place where demonstrating the wait is the test's purpose, saying so. Run the linter the repository uses. Rerun every spec you changed 20 times and the full suite once. Open a pull request with a table: test, failures before out of 20, cause, fix, failures after out of 20.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

A pull request whose description is a table, one row per flaky test: the spec and test title, how many of twenty passes failed before, the cause in a phrase (live server, row count on shared data, real timer, server not ready, fixed sleep), the fix, and how many of twenty failed after. Tests that failed every time are listed under it as broken, with the error, and are not counted as fixed flakes. Tests the agent chose to leave, such as a deliberate `cy.wait`, are listed with the reason.

The diff is confined to the specs, `cypress/fixtures`, `cypress.config.js` and the npm scripts. A reviewer reads it for two things. First, that no assertion was weakened to get a pass: a stubbed reply should carry the same fields the test checks, and a count that became a shape check should still fail when the list is empty. Second, that `retries` was not added, since retries turn a failing test into a slow passing one and hide the next flake.

Stubbing has a cost the description states plainly: a test that no longer calls the real service no longer notices when that service changes. Where a team wants one contract test against the live server, the agent keeps a single such test, tags it in the table and leaves the rest stubbed.

## Run it on every failed Cypress run on the default branch

An automation in the Project that holds the repository, with this prompt and a `github` trigger on `workflow_run.failure`, starts a run when a workflow fails. The trigger puts the failed run's event after the prompt, so the agent knows which workflow and commit to look at and begins its loop with the specs that failed there. Its max runs per hour keeps a bad afternoon from starting a dozen investigations; events past the cap wait as queued runs, and CI runs from forks never fire it.

A team that records which tests needed a rerun can be more exact. A handler on their own server sends `POST /o/{org}/sessions` with the spec path and test title in `input`, the repository in `resources` and a `request_id` that is a UUID v5 of the spec path, so the same flake seen in ten builds starts one run.

## Why run it on Rush?

Finding a flake means running the same spec dozens of times and keeping every result, with a browser, a display server and the app all up. That is hours of a laptop doing nothing else, which is why flaky tests usually get a retry setting and no diagnosis. A sandbox has no other job: the agent runs the loops to the end, and the pass-by-pass results are in the run's transcript, not in someone's memory of a terminal.

It also separates the suite from the network. In its own sandbox the agent can block the external host and watch which tests fall over, an experiment nobody wants to run against the office connection. The before and after counts in the pull request come from those runs, so the claim that a test is fixed is a number a reviewer can check.

## Frequently asked questions

### Why not just turn on retries?

Retries change what a failure costs, not whether it happens. A test that needs two attempts still depends on a server or a timer it does not control, and it will need three attempts on a slower day. The agent removes the dependency and leaves the `retries` setting as it found it.

### How many reruns are enough?

The prompt asks for twenty per spec, before and after. A test that fails one time in fifty can slip through that, so the table reports counts and does not claim certainty; raise the number in the prompt for a suite with rare failures. Each rerun is a full `cypress run` of that spec, not an in-process repeat.

### Does it run the tests in Chrome and Firefox too?

By default it uses Electron, the browser bundled with the Cypress binary, which is what `cypress run` picks with no `--browser` flag. The repository's scripts have `e2e:chrome` and `e2e:firefox` forms; ask for them in the prompt and the agent installs those browsers in its sandbox first and reports per browser.

### What if the flake comes from the app and not the test?

Then the row says so. A race in application code, such as a handler attached after the first click can arrive, shows up as a test that fails with timers and network both controlled. The agent fixes it in the app when the fix is small and local, and otherwise reports the race with the steps that reproduce it.

## More Cypress jobs

- [Migrate a Cypress Test Suite to Playwright](https://getrush.ai/run/migrate-cypress-to-playwright/cypress/)
- [Upgrade Cypress to the Current Major Version](https://getrush.ai/run/upgrade-cypress/cypress/)
- [Write Cypress Component Tests for the Untested Forms in a React App](https://getrush.ai/run/write-component-tests/cypress/)

## The same job on other stacks

- [Fix Flaky Tests in an Elixir App](https://getrush.ai/run/fix-flaky-tests/elixir/)
- [Fix Flaky Tests in a Laravel App](https://getrush.ai/run/fix-flaky-tests/laravel/)
- [Fix Flaky Playwright Tests](https://getrush.ai/run/fix-flaky-tests/playwright/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
