All jobs

Add Test Retries for Flaky Jest Tests in CI

Yes. An agent switches Jest to the runner that supports retries, retries only in CI, reports every retried test so flakiness stays visible, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the suite and the runner before it touches retries, because on an older Jest they depend on each other. In an Express API laid out like hagopj13/node-express-boilerplate, package.json pins jest at ^26.0.1 and yarn.lock resolves it to 26.6.3 with jest-jasmine2 and no jest-circus; jest.config.js sets testEnvironment: 'node', restoreMocks: true and coverage options and names no testRunner, so the default jasmine2 runs. jest.retryTimes is implemented in jest-circus only, so on this configuration a call to it would do nothing. The test script runs jest -i --colors --verbose --detectOpenHandles, in band, against a real MongoDB: tests/utils/setupTestDB.js connects with mongoose.connect in beforeAll and empties every collection in beforeEach. The three integration files under tests/integration hold most of the suite, forty-five tests in auth.test.js and forty-two in user.test.js, each hitting the app with supertest; four unit files cover the error middleware, two model plugins and the user model. CI is .travis.yml with a MongoDB service and yarn lint then yarn test.

It adds jest-circus at the same 26.6.3 release as a devDependency and sets testRunner: 'jest-circus/runner' in jest.config.js, the smallest change that makes retries work without moving Jest majors. Then it adds a setup file, listed in setupFilesAfterEnv, that calls jest.retryTimes with a small count only when process.env.CI is set, so a developer running yarn test locally still sees a flaky test fail on the first try. The count and the condition sit in one place with a comment that says why.

A retry that passes silently hides the problem, so the agent makes retried tests visible. It adds a small custom reporter under tests/utils that reads each test result's invocations and prints the name and file of every test that needed more than one, and registers it in reporters beside the default. In CI that list is the flakiness backlog; the pull request explains that a test on it for two weeks should be fixed or quarantined rather than retried forever.

To run the suite the agent installs a MongoDB server in its sandbox from MongoDB's own package repository, since the Ubuntu archive does not carry one, points MONGODB_URL at it the way .env.example does, and runs yarn test twice: once as a developer with CI unset, confirming retries are off, and once with CI=true to confirm the runner loads, the retry count applies and the reporter prints nothing when every test passes first time. It makes one test fail deliberately on its first invocation to see the reporter list it, reverts that, and runs yarn lint so the new files pass the airbnb and jest ESLint rules the project uses.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add CI-only test retries to this Jest suite. Read package.json, yarn.lock for the resolved jest and jest-jasmine2 versions, jest.config.js, tests/utils/setupTestDB.js, the files under tests/integration and tests/unit, .travis.yml and .env.example first. If jest.retryTimes needs jest-circus on this Jest release, add jest-circus at the same version and set testRunner. Add a setup file in setupFilesAfterEnv that calls jest.retryTimes with a small count only when CI is set, and a custom reporter that prints every test whose invocations were more than one so retried tests stay visible. Install a MongoDB server in the sandbox, run yarn test with CI unset and with CI=true, make one test fail on its first invocation to see the reporter list it, revert that, run yarn lint, and open a pull request that explains the policy and pastes both runs.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds one devDependency, two lines to jest.config.js, a setup file and a reporter under tests/utils, and a paragraph in the README on what a retried test means. The description pastes the yarn test output from the sandbox with CI unset and with it set, and the reporter's line from the deliberate first-try failure.

The run's record keeps the MongoDB install, both suite runs and the lint, so a reviewer can see the runner switch did not change which tests pass.

Before merging, a reviewer sets the retry count and decides where the reporter's output should go: the job log is the minimum, and a step that posts it on the pull request is a follow-up the description suggests.

Run it on every repository whose CI reruns on red

A team with several Node services on this boilerplate starts one run per repository with one POST /o/{org}/sessions each, this prompt as input and the repository in resources. Each run reads its own Jest version and config, so a service already on Jest 27 or later skips the runner change and gets only the setup file and the reporter.

Send a request_id that is a UUID v5 of the repository's full name so a retried call returns the run it already started. The key's webhook_url reports when each run finishes, with session_url to read the pull request from.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

Retries are easy to add and easy to add wrong: on Jest 26 the call is a no-op under the default runner, and a retry that passes quietly turns a flaky suite into a slow one nobody looks at. The agent checks the runner from the lockfile, proves the retry fires with a deliberate failure, and leaves a reporter that names every test that needed a second try, all in a sandbox with a real MongoDB behind the integration tests.

The pull request is a runner switch, a setup file and a reporter, so review is short, and the two pasted runs show the behaviour with and without CI. Several services get the same treatment at once, each in its own sandbox with its own pull request.

Frequently asked questions

Why not just upgrade Jest?

An upgrade from 26 changes the default runner, the testURL default and several timer and module behaviours, and is its own pull request with its own risks. Adding jest-circus at the same release gets retries working today; the pull request notes that a later upgrade makes the explicit testRunner line unnecessary.

Why only retry in CI?

A developer who sees a test fail once and pass once learns something; a retry hides it. Keying jest.retryTimes on process.env.CI keeps local runs strict and makes CI resilient to a transient MongoDB connection or timing problem, while the reporter still records the retry.

Do retries interfere with the database reset?

No. setupTestDB.js clears every collection in beforeEach, and Jest runs the hooks again for each invocation, so a retried test starts from an empty database like any other. The agent's deliberate first-try failure confirms the second invocation ran the hooks.

What about tests that fail every time?

They still fail. Retries only help a test that passes on a later invocation; a real regression fails every attempt and the job goes red as before, just a little later. The reporter output shows that no invocation passed, which tells a reviewer it was not flakiness.

More Jest jobs

The same job on other stacks

More jobs like this: Write the missing tests

Keep reading