All jobs

Run a pytest Suite in Parallel With pytest-xdist

Yes. An agent adds pytest-xdist, checks the suite's fixtures and ports are safe per worker, measures the gain in CI's own command, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the suite is run before it adds a worker flag, because parallel tests fail in the places a serial suite never noticed. In a project laid out like pallets/werkzeug, pyproject.toml keeps the test tools in a tests dependency group (pytest, pytest-timeout, ephemeral-port-reserve, watchdog, cffi, cryptography), runs pytest through tox with uv-venv-lock-runner and the flags -v --tb=short --basetemp={env_tmp_dir}, and sets filterwarnings = ["error"]. .github/workflows/tests.yaml runs uv run --locked --no-default-groups --group dev tox run across nine matrix rows, including Windows, Mac, a free-threaded 3.15t and PyPy. The suite has 588 tests in tests/, and the slow ones are the dev-server tests: tests/conftest.py defines DevServerClient, which starts live_apps/run.py as a subprocess on a port from ephemeral_port_reserve.reserve, logs into the test's tmp_path, and is marked dev_server; 35 such calls sit in test_serving.py and 4 in test_debug.py.

That reading answers the question the plugin cannot: is the suite safe to split across processes. The dev-server tests reserve a free port per start and write under the test's own tmp_path, so two workers never share a socket or a log file. The unix-socket test uses tmp_path_factory.mktemp("sock"), again per test. No test writes a fixed path in the repository, and the reloader tests copy their app into tmp_path before touching it. The one thing to watch is filterwarnings = error combined with xdist's own DeprecationWarning on some Python versions, which the agent checks by running once with the new flag and reading the first failure, if any.

The change is then small. pytest-xdist joins the tests group in pyproject.toml, uv lock updates uv.lock, which the uv-lock pre-commit hook checks, and -n auto joins the tox commands next to --basetemp, which xdist already makes per-worker. Nothing changes in the workflow: --locked installs the new pin, and tox run carries the flag to every matrix row. The 3.15t row keeps running: xdist spawns processes, not threads, so it does not depend on the free-threading build's plugin support, and the commands_pre that uninstalls watchdog on that row is left as it is.

In its sandbox the agent runs uv sync and times uv run --locked --no-default-groups --group dev tox run -e py3.13 before and after the flag, on the same machine, and quotes both wall-clock lines and the worker count pytest printed. It runs the parallel suite three times to see that it is stable, since a test that passes serially and fails once in three under xdist is the signal this change exists to surface. A dev_server test that proves flaky is quoted with its name and the agent's reading of why, and --dist loadgroup with an xdist_group mark is applied to that file only, rather than disabling parallelism for the suite.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Run this pytest suite in parallel with pytest-xdist. Read pyproject.toml, uv.lock, .pre-commit-config.yaml, .github/workflows/tests.yaml, tests/conftest.py and tests/test_serving.py first. Check that fixtures, ports, temporary directories and any module-level state are safe per worker process, and say what you found. Add pytest-xdist to the tests dependency group, run uv lock, and add -n auto to the tox pytest command. Do not change the workflow. Run uv sync, then time tox run -e py3.13 before and after the change on this machine and quote both. Run the parallel suite three times; if a test is flaky under xdist, name it, explain why from its code, and pin it with an xdist_group mark and --dist loadgroup rather than turning parallelism off. Open a pull request with the timings and the stability runs.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request with two files changed, pyproject.toml and uv.lock: one line in the tests group, one flag in the tox command, and the lock entries for xdist and execnet. No test file changes unless a group mark was needed, and no workflow change. The description quotes two timings of the same tox environment on the same machine, serial and with -n auto, with the worker count pytest reported, and the three parallel runs all green.

The description also records what the agent checked and found safe: per-test ports from ephemeral_port_reserve, per-test tmp_path for the dev-server logs and sockets, and no shared files in the repository tree, so a reviewer sees the reasoning rather than a flag.

Before merging, a reviewer decides whether CI should pin a worker count instead of auto, since GitHub's runners report more cores than a dev-server test can use well, and whether the Windows row, where process start-up is slowest, wants -n 2 on its own.

Run it on every nightly

A parallel suite is where order-dependent tests show up, so a team runs it unattended. A schedule trigger with cron set to a quiet hour in the trigger's timezone starts a run that executes the suite with -n auto on the latest dependency versions, the way the tox update-requirements environment refreshes uv.lock, and opens a pull request only when a test fails, quoting the test's name, the worker it ran on and the stack.

A test that passes alone and fails beside another is reported with both names from pytest's -v output, which is the clue that a fixture leaks state. A green night produces no pull request at all; the run's needs_review state, when the suite does not pass, is the signal to read.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

Adding a worker flag is one line, and finding out it is safe is the whole job. A suite that starts servers, writes files and reserves ports has to be read before it is split, and the right time to read it is before the first flaky night in CI. In a Rush sandbox the agent reads the fixtures, runs the suite serially and in parallel on the same machine, and repeats the parallel run to prove it holds.

The record of the run keeps every run and both timings, and the pull request quotes them, so the reviewer reads the actual gain on this suite rather than a plugin's claim. Across every Python repository a team maintains, the same request runs in parallel, one sandbox and one pull request each.

Frequently asked questions

How does the agent know the dev-server tests are safe to parallelise?

By reading tests/conftest.py. DevServerClient reserves a free port with ephemeral_port_reserve for every start and writes its log and any unix socket under the test's own tmp_path, so two workers cannot collide. The pull request quotes those lines.

What if a test is flaky only under xdist?

The agent runs the parallel suite three times and names any test that fails, with its reading of the cause from the test's code. Such a test gets an xdist_group mark so the tests that share state run on one worker, with --dist loadgroup, rather than switching parallelism off for everyone.

Does this change the CI workflow?

No. The workflow runs tox with --locked, so it installs the new pin from uv.lock, and tox carries the -n auto flag to every matrix row, including the free-threaded and PyPy rows. The only files in the pull request are pyproject.toml and uv.lock.

Will the suite be faster?

The pull request quotes the serial and parallel wall-clock times on the same machine so the reviewer can see. The gain comes from the dev-server tests, which each wait on a subprocess; the many fast tests gain little, and on a two-core runner the result is smaller than on the sandbox.

More pytest jobs

The same job on other stacks

More jobs like this: Write the missing tests

Keep reading