# Fix pytest Tests That Pass Once and Fail on the Next Run

Yes. The agent reruns the suite alone, twice and in reverse to find shared database state, fixes the fixtures, and proves every order now passes.

[Run this on Rush](https://cloud.getrush.ai/run/fix-flaky-tests/pytest/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

A test that fails only sometimes is failing for a reason that holds every time; the agent's job is to find the conditions. It reads how the suite is meant to run before it runs anything. In an app like Donnype/fastapi-template, `api/test_all.py` holds five tests, four marked `integration`, and the `Makefile` runs them through Docker Compose after a `cleantest` target that kills, removes and rebuilds the containers. Each CI run therefore gets a new, empty PostgreSQL. That explains why the pipeline is green, and it is the first hint about where the trouble is. The sandbox has no Docker, so the agent installs PostgreSQL with apt, sets the `DB_STRING` variable the code reads, and runs `pytest` directly. A database that survives between runs is what a developer's machine has, and it is the condition CI never tests.

Then it runs four experiments and records each result. The full suite on an empty database: 5 passed. The full suite again without resetting anything: 1 failed, `test_contract_test`, on an `IntegrityError`. `test_api` alone, twice: passed, then failed. `test_api` followed by `test_repository` in one invocation: the second fails. And on a database with no tables at all, `test_api` alone fails with `relation "todo" does not exist`. The three repository tests, run together and repeated, pass every time. So the fault is around `test_api`, in both directions: it depends on something an earlier test did, and it leaves something behind for later ones.

The code confirms it. Migrations run inside the `todo_repository` fixture, which calls `alembic upgrade head` and, after the test, truncates every table. `test_api` does not request that fixture. It posts a row with the key `testkey` through the real application and never removes it, so the table exists only if another test ran first, and the row outlives the run. The suite passes in file order because `test_api` is last. The same fixture, and `test_api` itself, each begin with `time.sleep(1)`, a second of waiting per test in place of knowing the database is ready.

The fix is structural, not a retry. Migration moves to a session-scoped fixture that runs once. A function-scoped fixture truncates the tables before and after each test, so a row left by an aborted run cannot poison the next. `todo_repository` builds on it, and `test_api` requests it. No assertion changes. The agent then repeats every experiment: `test_api` alone on an empty database, twice; the full suite twice; the tests in reverse order; and the suite with a stale `testkey` row inserted by hand first. All pass. The suite also gets about three seconds shorter, because four one-second sleeps became one.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Some tests in this repository fail intermittently or only on a second run. Find out why and fix the cause. Read the test files, conftest and fixtures, the pytest configuration, and how CI and the Makefile start the tests and their database; note whether CI always starts from an empty database. Docker may not be available here: if the tests need PostgreSQL, install it with apt, start it, and run pytest directly with the connection settings the code reads. Do not start by changing code. Run these experiments and record the exact result of each: the full suite on an empty database; the full suite a second time without resetting anything; each test alone on a database with no tables; each test alone twice in a row; the tests in reverse order. From the results, state for every failing case which test depends on state from another and which test leaves state behind, then confirm it in the code: which fixture creates the schema, which cleans up, and which tests bypass them. Report every sleep used in place of a readiness check. Fix the fixtures so that each test starts from a known state and leaves none: create the schema once per session, clean before and after each test, and make every test that touches the database use that path. Do not add retries, reruns, ordering plugins or skips, and do not change what any test asserts. Repeat all the experiments after the fix, plus one with a leftover row inserted by hand before the run, and require every one to pass. Open a pull request with a before and after table of the experiments, the cause in two sentences, and the run time before and after.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

A pull request that changes the test file and nothing in the application. In a suite like the one above the diff is one file: two new fixtures, one helper for the truncate, and `test_api` given a fixture argument. The description states the cause in two sentences, naming the test that leaked a row and the fixture it bypassed.

Most of the description is a table. Its rows are the experiments: full suite on an empty database, full suite repeated, each test alone, each test alone twice, reverse order, and a run that starts with a leftover row. Its columns are the result before and the result after. Before, most of those rows contain a failure; after, none does. A reviewer does not have to trust that the flakiness is gone, and can rerun any row with the command printed beside it.

The agent also lists what it noticed and left alone. The remaining `time.sleep(1)` in the session fixture stands in for a readiness check on the database container, and replacing it means changing how Compose starts the service. The tests ran against the PostgreSQL version apt installs, which is not the version the Compose file pins. Both are stated so that nobody assumes more was verified than was.

## Run it on every failed workflow run

Flaky tests announce themselves as a red build on a commit that did not touch them. An automation in the Project that holds the repository, with a `github` trigger on `workflow_run.failure`, starts a run each time a workflow fails. Its saved prompt is the instruction above with one line added: if the failing job's log shows a deterministic failure caused by the commit, say so in one paragraph and change nothing. Rush puts the event after the prompt, so the agent knows which run failed. Runs from forks and from bot senders do not fire the trigger.

Set the automation's hourly run limit low, since one bad merge can fail many workflows at once; an event past the limit waits until the hour has room, and once an hour's worth is waiting, further events are refused. To start the job by hand for a test a teammate has complained about, send this body to `POST /o/{org}/sessions` with a `request_id` that is a UUID made for the job and the test's name added to the prompt.

## Why run it on Rush?

Order and repetition bugs are found by running the suite many times under different conditions, each from a known starting state. That is slow and awkward on a laptop, where the database holds last week's rows and every reset risks someone's local data. The run's sandbox starts with nothing, the agent installs the database it needs, drops and recreates state as often as the experiments require, and keeps each command and its output in the transcript at `GET /o/{org}/sessions/{id}/messages`.

The common shortcut is a rerun plugin, which turns a real defect into a slower green build. This job comes back with the opposite: the cause named, the fixture corrected, and a table showing that the tests pass alone, repeated, reversed and after a dirty start. If the reviewer wants another ordering tried, a follow-up to `POST /o/{org}/sessions/{id}/messages` wakes the same sandbox with the database still running.

## Frequently asked questions

### Why do my tests pass in CI but fail locally on the second run?

Usually because CI starts from an empty database every time and your machine does not. A test that writes a row and never removes it passes on a new database and collides with its own leftovers on the next run. In Donnype/fastapi-template the `Makefile` rebuilds the database container before every test run, which hides exactly this.

### Will it just add pytest-rerunfailures?

No. The prompt forbids retries, reruns, ordering plugins and skips. A rerun makes an order-dependent test pass by chance and leaves the shared state in place. The agent changes the fixtures so that the state is not shared.

### Does cleaning the database before every test slow the suite down?

A truncate on a small schema takes milliseconds. In this fixture the fixed suite is about three seconds faster, because running migrations once per session removed three of the four one-second sleeps. On a large schema the agent would use a transaction rolled back per test and say why.

## More pytest jobs

- [Fix "AttributeError: module 'pytest' has no attribute 'config'" After Upgrading pytest](https://getrush.ai/run/fix-pytest-config-attribute-error/pytest/)
- [Fix "PytestRemovedIn10Warning: @pytest.yield_fixture is deprecated" Before pytest 10 Removes It](https://getrush.ai/run/fix-pytest-yield-fixture-is-deprecated-warning/pytest/)
- [Add Coverage Thresholds to a pytest Suite](https://getrush.ai/run/add-coverage-thresholds/pytest/)

## The same job on other stacks

- [Fix Flaky Tests in a Rails App](https://getrush.ai/run/fix-flaky-tests/rails/)
- [Fix Flaky Tests in a React App](https://getrush.ai/run/fix-flaky-tests/react/)
- [Fix Flaky Tests in a Vue App](https://getrush.ai/run/fix-flaky-tests/vue/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
