# Add Random Test Order to a pytest Suite

Yes. An agent adds pytest-randomly to your test dependencies, runs the suite shuffled until it finds any order it depends on, fixes that, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-random-test-order/pytest/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent starts with how the suite is run. In a project laid out like pallets/itsdangerous, tests live under `tests/test_itsdangerous/`, `pyproject.toml` sets `testpaths = ["tests"]` and `filterwarnings = ["error"]`, and the suite is run by `tox` with `uv-venv-lock-runner`, the `tests` dependency group of `freezegun` and `pytest`, and the command `pytest -v --tb=short --basetemp={env_tmp_dir}` plus whatever posargs the caller adds. `.github/workflows/tests.yaml` runs `uv run --locked tox run -e py3.13` and the rest of the matrix, and `.pre-commit-config.yaml` has the `uv-lock` hook, so a dependency added without updating `uv.lock` fails the style job. The tests are classes that inherit from each other: `TestURLSafeTimedSerializer` extends both `TestURLSafeSerializer` and `TestTimedSerializer`, `test_timed.py` has an autouse `freeze` fixture that pins the clock with `freeze_time`, and `test_serializer.py` parametrizes a fixture over `Serializer` and a pickle-backed `Serializer`.

It adds the plugin where the other test dependencies are. `pytest-randomly` goes into the `tests` group in `pyproject.toml`, `uv lock` updates `uv.lock` so the pre-commit hook and `--locked` stay green, and nothing is added to the pytest command: the plugin shuffles module, class and function order on every run, reseeds Python's `random` per test from the run's seed, and prints `Using --randomly-seed=NNN` at the top of the output so a failing order can be replayed with the same flag. The `tox` command gains `-p randomly` explicitly, so a reader of `pyproject.toml` can see the suite is shuffled on purpose, and the `tox.env_run_base` description says so.

Then it looks for order dependence. The agent runs the suite twenty times with different seeds and reads each run's failures. In this suite the likely place is the autouse clock: a test that imports a value computed at module load under the frozen time, or a fixture whose state leaks across the inherited classes. Whatever it finds, it fixes the test and not the order: a module-level timestamp becomes a fixture, a shared mutable default becomes a fresh object per test, and it does not mark anything `@pytest.mark.random_order(disabled=True)`. If twenty shuffled runs all pass, the pull request says so and the fix list is empty.

In its sandbox the agent installs `uv`, runs `uv run --locked tox run -e py3.13` the way CI does, then `uv run pytest -p randomly -p no:cacheprovider` with `--randomly-seed=1` through `--randomly-seed=20`, and quotes the seed line and the result of each run in a short table in the pull request.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add random test ordering to this pytest suite. Read pyproject.toml, .github/workflows/tests.yaml, .pre-commit-config.yaml and every file under tests/ first. Add pytest-randomly to the tests dependency group, run uv lock so uv.lock is updated, and add -p randomly to the tox test command so the shuffling is visible in config. Then run the suite twenty times with --randomly-seed=1 through 20. If any order fails, fix the test that depends on order, not the plugin settings, and do not disable random order for any module. Run uv run --locked tox run -e py3.13 the way CI does. In the pull request put a table of the twenty seeds and their results, the uv lock diff summary, and a list of tests you changed with the reason for each. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request touching `pyproject.toml`, `uv.lock` and whichever test files needed a fix. The description holds a twenty-row table of seed and result, the `Using --randomly-seed=` line from one run so a reviewer can replay it, the `tox` output for `py3.13`, and for each changed test a sentence naming what it shared with its neighbour and how it was separated.

The run's record keeps all twenty runs in full, so the exact failing order, if there was one, is there to read along with the traceback that led to the fix. A reviewer can see that `filterwarnings = ["error"]` stayed and the plugin raised no warning under it.

Before merging, a reviewer decides whether CI should pin a seed. The pull request leaves it random, so every CI run is a new order and a new chance to catch a dependence; pinning it with `--randomly-seed` in the tox command trades that for reproducible runs.

## Run it on every nightly

The job is worth running more than once, because a suite that is clean today acquires an order dependence the next time someone adds a module-level value. In the Project for the repository, an automation with the prompt above and a `schedule` trigger runs it nightly against `main`; a run that finds nothing opens no pull request and the schedule's record shows twenty green seeds.

A run that finds a failing order opens a pull request on its own branch with the seed in the title, so the person reviewing it can replay the failure locally before reading the fix. The same automation on several repositories runs in parallel, one sandbox each.

## Why run it on Rush?

Adding the plugin is one line; finding out what it breaks is twenty runs, each a full suite under a different order, and reading which test leaked into which. On a laptop that is a morning, and the usual shortcut is to disable the plugin for the module that fails, which keeps the dependence and hides it. In a Rush sandbox the agent runs the twenty orders, reads the tracebacks, and fixes the test; the pull request shows every seed.

The runs stay with the record, so when a seed fails in CI six months later the next person can compare it to the orders that passed. Across many Python repositories the job runs in parallel, one sandbox each, with the same prompt and a different table in every result.

## Frequently asked questions

### Will random order change what pytest -k and -x do?

No. Selection and stopping on first failure work as before; only the order of the selected tests changes. `-p no:randomly` turns it off for one run, and `--randomly-dont-reorganize` keeps the plugin's seeding of `random` while leaving the order alone.

### Why is the plugin not disabled for the module that failed?

Because a module that only passes in file order has a shared piece of state, and leaving it in place means the next test added to that module inherits the problem. The fix is in the test: a fixture instead of a module-level value, a fresh object instead of a shared default.

### Does this work with freezegun and the autouse clock?

Yes. Each test's `freeze_time` context is set up and torn down by its own fixture, so shuffling does not change the time a test sees. What shuffling exposes is a value computed outside a fixture, at import time, under whichever test happened to be frozen first.

### Why run twenty seeds and not more?

Twenty different orders catch the common case, one pair of tests that leak into each other, with high probability on a suite this size. A larger suite gets more seeds; say the number in the prompt and the agent runs that many and reports them all.

### How do I reproduce a failure CI reports?

The first line of the pytest output says `Using --randomly-seed=NNN`. Run `pytest -p randomly --randomly-seed=NNN` locally and the order is identical, including the per-test reseeding of `random`.

## More pytest jobs

- [Add Test Timeouts to a pytest Suite](https://getrush.ai/run/add-test-timeouts/pytest/)
- [Roll Back the Database After Each pytest Test in a FastAPI App](https://getrush.ai/run/add-transactional-test-fixtures/pytest/)
- [Fix "Fixtures are not meant to be called directly" in a pytest Suite](https://getrush.ai/run/fix-fixtures-are-not-meant-to-be-called-directly/pytest/)

## More jobs like this: Write the missing tests

- [Add Replay Tests to Temporal Workflows](https://getrush.ai/run/add-replay-tests/temporal/)
- [Add a Playwright Smoke Suite to a Next.js App](https://getrush.ai/run/add-smoke-tests/playwright/)
- [Add Snapshot Tests to a Go CLI's Rendered Output](https://getrush.ai/run/add-snapshot-tests/go-cli/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
