What the agent does
The agent starts with how the suite is run. In a project laid out like pallets/itsdangerous, tests live under tests/test_itsdangerous/, pyproject.toml sets testpaths = ["tests"] and filterwarnings = ["error"], and the suite is run by tox with uv-venv-lock-runner, the tests dependency group of freezegun and pytest, and the command pytest -v --tb=short --basetemp={env_tmp_dir} plus whatever posargs the caller adds. .github/workflows/tests.yaml runs uv run --locked tox run -e py3.13 and the rest of the matrix, and .pre-commit-config.yaml has the uv-lock hook, so a dependency added without updating uv.lock fails the style job. The tests are classes that inherit from each other: TestURLSafeTimedSerializer extends both TestURLSafeSerializer and TestTimedSerializer, test_timed.py has an autouse freeze fixture that pins the clock with freeze_time, and test_serializer.py parametrizes a fixture over Serializer and a pickle-backed Serializer.
It adds the plugin where the other test dependencies are. pytest-randomly goes into the tests group in pyproject.toml, uv lock updates uv.lock so the pre-commit hook and --locked stay green, and nothing is added to the pytest command: the plugin shuffles module, class and function order on every run, reseeds Python's random per test from the run's seed, and prints Using --randomly-seed=NNN at the top of the output so a failing order can be replayed with the same flag. The tox command gains -p randomly explicitly, so a reader of pyproject.toml can see the suite is shuffled on purpose, and the tox.env_run_base description says so.
Then it looks for order dependence. The agent runs the suite twenty times with different seeds and reads each run's failures. In this suite the likely place is the autouse clock: a test that imports a value computed at module load under the frozen time, or a fixture whose state leaks across the inherited classes. Whatever it finds, it fixes the test and not the order: a module-level timestamp becomes a fixture, a shared mutable default becomes a fresh object per test, and it does not mark anything @pytest.mark.random_order(disabled=True). If twenty shuffled runs all pass, the pull request says so and the fix list is empty.
In its sandbox the agent installs uv, runs uv run --locked tox run -e py3.13 the way CI does, then uv run pytest -p randomly -p no:cacheprovider with --randomly-seed=1 through --randomly-seed=20, and quotes the seed line and the result of each run in a short table in the pull request.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add random test ordering to this pytest suite. Read pyproject.toml, .github/workflows/tests.yaml, .pre-commit-config.yaml and every file under tests/ first. Add pytest-randomly to the tests dependency group, run uv lock so uv.lock is updated, and add -p randomly to the tox test command so the shuffling is visible in config. Then run the suite twenty times with --randomly-seed=1 through 20. If any order fails, fix the test that depends on order, not the plugin settings, and do not disable random order for any module. Run uv run --locked tox run -e py3.13 the way CI does. In the pull request put a table of the twenty seeds and their results, the uv lock diff summary, and a list of tests you changed with the reason for each. Open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request touching pyproject.toml, uv.lock and whichever test files needed a fix. The description holds a twenty-row table of seed and result, the Using --randomly-seed= line from one run so a reviewer can replay it, the tox output for py3.13, and for each changed test a sentence naming what it shared with its neighbour and how it was separated.
The run's record keeps all twenty runs in full, so the exact failing order, if there was one, is there to read along with the traceback that led to the fix. A reviewer can see that filterwarnings = ["error"] stayed and the plugin raised no warning under it.
Before merging, a reviewer decides whether CI should pin a seed. The pull request leaves it random, so every CI run is a new order and a new chance to catch a dependence; pinning it with --randomly-seed in the tox command trades that for reproducible runs.
Run it on every nightly
The job is worth running more than once, because a suite that is clean today acquires an order dependence the next time someone adds a module-level value. In the Project for the repository, an automation with the prompt above and a schedule trigger runs it nightly against main; a run that finds nothing opens no pull request and the schedule's record shows twenty green seeds.
A run that finds a failing order opens a pull request on its own branch with the seed in the title, so the person reviewing it can replay the failure locally before reading the fix. The same automation on several repositories runs in parallel, one sandbox each.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Adding the plugin is one line; finding out what it breaks is twenty runs, each a full suite under a different order, and reading which test leaked into which. On a laptop that is a morning, and the usual shortcut is to disable the plugin for the module that fails, which keeps the dependence and hides it. In a Rush sandbox the agent runs the twenty orders, reads the tracebacks, and fixes the test; the pull request shows every seed.
The runs stay with the record, so when a seed fails in CI six months later the next person can compare it to the orders that passed. Across many Python repositories the job runs in parallel, one sandbox each, with the same prompt and a different table in every result.
Frequently asked questions
Will random order change what pytest -k and -x do?
No. Selection and stopping on first failure work as before; only the order of the selected tests changes. -p no:randomly turns it off for one run, and --randomly-dont-reorganize keeps the plugin's seeding of random while leaving the order alone.
Why is the plugin not disabled for the module that failed?
Because a module that only passes in file order has a shared piece of state, and leaving it in place means the next test added to that module inherits the problem. The fix is in the test: a fixture instead of a module-level value, a fresh object instead of a shared default.
Does this work with freezegun and the autouse clock?
Yes. Each test's freeze_time context is set up and torn down by its own fixture, so shuffling does not change the time a test sees. What shuffling exposes is a value computed outside a fixture, at import time, under whichever test happened to be frozen first.
Why run twenty seeds and not more?
Twenty different orders catch the common case, one pair of tests that leak into each other, with high probability on a suite this size. A larger suite gets more seeds; say the number in the prompt and the agent runs that many and reports them all.
How do I reproduce a failure CI reports?
The first line of the pytest output says Using --randomly-seed=NNN. Run pytest -p randomly --randomly-seed=NNN locally and the order is identical, including the per-test reseeding of random.
More pytest jobs
More jobs like this: Write the missing tests
- Add Replay Tests to Temporal Workflows (Temporal)
- Add a Playwright Smoke Suite to a Next.js App (Playwright)
- Add Snapshot Tests to a Go CLI's Rendered Output (Go CLI)