All jobs

Add Mutation Testing to a pytest Suite

Yes. An agent adds mutmut to a pytest project, runs it against the real suite in its sandbox, writes tests for the mutants that survive and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent starts from how the suite is built and run, because mutation testing multiplies that cost. In a library laid out like pallets/itsdangerous, pyproject.toml declares a flit_core build, the package under src/itsdangerous/ (signer.py, serializer.py, timed.py, url_safe.py, encoding.py, exc.py, _json.py), the tests under tests/test_itsdangerous/ in five files, testpaths = ["tests"] and filterwarnings = ["error"]. Dependencies are uv groups: tests holds pytest and freezegun, pinned in uv.lock (pytest 8.3.5, freezegun 1.5.2), and every check is a tox environment run through tox-uv; .github/workflows/tests.yaml calls uv run --locked tox run -e py3.13 and friends across a CPython and PyPy matrix. The agent also notices that [tool.coverage.run] names source = ["jinja2", "tests"], a line copied from a sibling project, so the coverage configuration is not a reliable guide to what the tests exercise; it reads the modules itself instead.

It adds mutmut to the tests dependency group and re-locks with uv lock, then writes a [tool.mutmut] table with paths_to_mutate set to src/itsdangerous/ and tests_dir to tests/, so mutants are made from the package and not from the tests. mutmut writes its working copies under a mutants/ directory, which the agent adds to .gitignore next to .tox/ and htmlcov/. Because filterwarnings turns every warning into an error, a mutant that merely provokes a DeprecationWarning is killed by the existing configuration, which the agent leaves as it is.

Then it runs the whole thing once: uv sync, uv run mutmut run, and uv run mutmut results to list what survived, module by module. For each survivor it reads the mutant's diff with mutmut show and sorts it into one of two piles. A real gap gets a test in the matching file: a mutant that moves the age boundary in timed.py survives only if no test signs a value exactly at max_age, and the fix is a test that freezes time with freezegun, which the suite already imports, and checks both sides of the limit; a mutant that changes the default separator in signer.py survives only if every test passes sep explicitly, and the fix is one that relies on the default. An equivalent mutant, one that changes behaviour no caller can observe, such as a string used only in a repr, goes into the configuration's exclusions with a one-line reason, never into a test that asserts an implementation detail.

Last it wires a mutation environment into [tool.tox], using the tests dependency group and running mutmut run followed by mutmut results, and a mutation job in tests.yaml that calls uv run --locked tox run -e mutation on one interpreter only, the way the existing typing job reads its version from pyproject.toml. The job reports on the first merge rather than failing, and the pull request says which single line turns a new surviving mutant into a failure once the team has agreed on the baseline. The misnamed coverage source is pointed out in the description and left alone: it is outside this job.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add mutation testing with mutmut to this pytest project. Read pyproject.toml (the dependency groups, the pytest and tox configuration), uv.lock, .gitignore, .github/workflows/tests.yaml and every module under src/ and tests/ first. Add mutmut to the tests dependency group and re-lock with uv, configure it in pyproject.toml to mutate the package under src/ and run the tests under tests/, and ignore its working directory in .gitignore. Run the full mutation run in the sandbox and list the survivors. For each surviving mutant, either write a test in the matching test file that kills it, or, when it is equivalent and no caller could observe it, exclude it with a one-line reason. Add a tox environment and a CI job that runs it on one interpreter. Do not change the coverage configuration or existing tests. Open a pull request whose description quotes the mutmut results before and after your tests, per module, and lists each excluded mutant with its reason.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that changes pyproject.toml (the dependency group, the mutmut table, the tox environment), uv.lock, .gitignore and .github/workflows/tests.yaml, and adds tests to files under tests/test_itsdangerous/. The description quotes the mutmut results summary before and after the new tests, per module, so a reviewer sees which files were under-tested and by how much, and it lists every excluded mutant with its diff and the reason it is equivalent.

The run's record holds the complete first mutation run and the second one after the tests landed, including each mutant's diff and the pytest output that killed it, so a reviewer can open any survivor the agent excluded and judge the reasoning rather than take the summary on trust.

Before merging, a reviewer reads the new tests for assertions that pin an implementation detail instead of a behaviour, checks the exclusions for a mutant that a user could in fact observe, and looks at how long the mutation job took in the run, since that is the cost every push will pay once the job is made to fail.

Run it on every nightly

A mutation run belongs on a schedule rather than on every push: in the console, add an automation to the Project with a schedule trigger set to every 1d and this prompt, and the agent runs the whole mutant set overnight, writes tests for whatever survived since the last run and opens one pull request by morning. The automation starts at most max_runs_per_hour runs in an hour, and each run works in its own sandbox with the repository freshly cloned.

A team with several Python packages sends this body once per repository from a script instead, with the repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started. A key starts at most max_concurrent runs at once, 4 by default; the rest answer 202 with status: "queued" and start as slots free, which suits a job that takes as long as a mutation run does.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

A mutation run executes the test suite once per mutant, which for a small library is hundreds of pytest invocations. On a laptop that means an afternoon with the fans on and a terminal nobody can close. In a Rush sandbox the agent runs it to the end, reads every survivor, and writes the tests while the results are still in front of it, without touching anything on the developer's machine.

The result carries its own proof. The record keeps the mutmut output from before and after the new tests, each mutant's diff and the test that killed it, so the reviewer judges the tests against the mutants they were written for rather than against a score in a description. Several packages run in parallel, each in its own sandbox with its own interpreter and lockfile.

Frequently asked questions

Will mutation testing slow down my CI?

Only if you let it run on every push. The job the agent adds runs on one interpreter, reports rather than fails on the first merge, and the pull request says how long the mutation run takes in the sandbox. Most teams move it to a nightly schedule or a manual trigger and keep the ordinary suite on pull requests.

Does it work with uv and tox?

Yes. The agent adds mutmut to the tests dependency group so uv lock records it, and runs it through a tox environment the same way the existing py3.13 and typing environments run, with uv run --locked tox run -e mutation. No second tool chain is introduced.

What is an equivalent mutant, and why not write a test for it?

A mutant whose change no caller can observe, such as a different wording in a repr or a default argument every code path overrides. A test that kills it has to assert the implementation detail itself, which makes the suite brittle for no safety. The agent excludes such mutants with a reason and the reviewer can disagree on any of them.

Will it change my existing tests?

No. It adds tests to the existing files and leaves the ones that are there untouched. If a surviving mutant shows an existing test asserting the wrong thing, the agent says so in the pull request rather than editing it.

Why does it leave the coverage configuration alone?

Because fixing it is a separate change. The agent reads [tool.coverage.run], sees it names another package, and uses the modules under src/ directly to decide what to mutate; the misnaming goes in the description for a reviewer to fix in its own pull request.

More pytest jobs

The same job on other stacks

Keep reading