# Add Mutation Testing to a pytest Suite

Yes. An agent adds mutmut to a pytest project, runs it against the real suite in its sandbox, writes tests for the mutants that survive and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-mutation-testing/pytest/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent starts from how the suite is built and run, because mutation testing multiplies that cost. In a library laid out like pallets/itsdangerous, `pyproject.toml` declares a `flit_core` build, the package under `src/itsdangerous/` (`signer.py`, `serializer.py`, `timed.py`, `url_safe.py`, `encoding.py`, `exc.py`, `_json.py`), the tests under `tests/test_itsdangerous/` in five files, `testpaths = ["tests"]` and `filterwarnings = ["error"]`. Dependencies are uv groups: `tests` holds pytest and freezegun, pinned in `uv.lock` (pytest 8.3.5, freezegun 1.5.2), and every check is a tox environment run through `tox-uv`; `.github/workflows/tests.yaml` calls `uv run --locked tox run -e py3.13` and friends across a CPython and PyPy matrix. The agent also notices that `[tool.coverage.run]` names `source = ["jinja2", "tests"]`, a line copied from a sibling project, so the coverage configuration is not a reliable guide to what the tests exercise; it reads the modules itself instead.

It adds `mutmut` to the `tests` dependency group and re-locks with `uv lock`, then writes a `[tool.mutmut]` table with `paths_to_mutate` set to `src/itsdangerous/` and `tests_dir` to `tests/`, so mutants are made from the package and not from the tests. mutmut writes its working copies under a `mutants/` directory, which the agent adds to `.gitignore` next to `.tox/` and `htmlcov/`. Because `filterwarnings` turns every warning into an error, a mutant that merely provokes a `DeprecationWarning` is killed by the existing configuration, which the agent leaves as it is.

Then it runs the whole thing once: `uv sync`, `uv run mutmut run`, and `uv run mutmut results` to list what survived, module by module. For each survivor it reads the mutant's diff with `mutmut show` and sorts it into one of two piles. A real gap gets a test in the matching file: a mutant that moves the age boundary in `timed.py` survives only if no test signs a value exactly at `max_age`, and the fix is a test that freezes time with freezegun, which the suite already imports, and checks both sides of the limit; a mutant that changes the default separator in `signer.py` survives only if every test passes `sep` explicitly, and the fix is one that relies on the default. An equivalent mutant, one that changes behaviour no caller can observe, such as a string used only in a `repr`, goes into the configuration's exclusions with a one-line reason, never into a test that asserts an implementation detail.

Last it wires a `mutation` environment into `[tool.tox]`, using the `tests` dependency group and running `mutmut run` followed by `mutmut results`, and a `mutation` job in `tests.yaml` that calls `uv run --locked tox run -e mutation` on one interpreter only, the way the existing `typing` job reads its version from `pyproject.toml`. The job reports on the first merge rather than failing, and the pull request says which single line turns a new surviving mutant into a failure once the team has agreed on the baseline. The misnamed coverage `source` is pointed out in the description and left alone: it is outside this job.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add mutation testing with mutmut to this pytest project. Read pyproject.toml (the dependency groups, the pytest and tox configuration), uv.lock, .gitignore, .github/workflows/tests.yaml and every module under src/ and tests/ first. Add mutmut to the tests dependency group and re-lock with uv, configure it in pyproject.toml to mutate the package under src/ and run the tests under tests/, and ignore its working directory in .gitignore. Run the full mutation run in the sandbox and list the survivors. For each surviving mutant, either write a test in the matching test file that kills it, or, when it is equivalent and no caller could observe it, exclude it with a one-line reason. Add a tox environment and a CI job that runs it on one interpreter. Do not change the coverage configuration or existing tests. Open a pull request whose description quotes the mutmut results before and after your tests, per module, and lists each excluded mutant with its reason.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that changes `pyproject.toml` (the dependency group, the mutmut table, the tox environment), `uv.lock`, `.gitignore` and `.github/workflows/tests.yaml`, and adds tests to files under `tests/test_itsdangerous/`. The description quotes the `mutmut results` summary before and after the new tests, per module, so a reviewer sees which files were under-tested and by how much, and it lists every excluded mutant with its diff and the reason it is equivalent.

The run's record holds the complete first mutation run and the second one after the tests landed, including each mutant's diff and the pytest output that killed it, so a reviewer can open any survivor the agent excluded and judge the reasoning rather than take the summary on trust.

Before merging, a reviewer reads the new tests for assertions that pin an implementation detail instead of a behaviour, checks the exclusions for a mutant that a user could in fact observe, and looks at how long the `mutation` job took in the run, since that is the cost every push will pay once the job is made to fail.

## Run it on every nightly

A mutation run belongs on a schedule rather than on every push: in the console, add an automation to the Project with a `schedule` trigger set to `every` `1d` and this prompt, and the agent runs the whole mutant set overnight, writes tests for whatever survived since the last run and opens one pull request by morning. The automation starts at most `max_runs_per_hour` runs in an hour, and each run works in its own sandbox with the repository freshly cloned.

A team with several Python packages sends this body once per repository from a script instead, with the repository in `resources` and a `request_id` that is a UUID v5 of the repository's full name, so a retried call returns the run already started. A key starts at most `max_concurrent` runs at once, 4 by default; the rest answer `202` with `status: "queued"` and start as slots free, which suits a job that takes as long as a mutation run does.

## Why run it on Rush?

A mutation run executes the test suite once per mutant, which for a small library is hundreds of pytest invocations. On a laptop that means an afternoon with the fans on and a terminal nobody can close. In a Rush sandbox the agent runs it to the end, reads every survivor, and writes the tests while the results are still in front of it, without touching anything on the developer's machine.

The result carries its own proof. The record keeps the mutmut output from before and after the new tests, each mutant's diff and the test that killed it, so the reviewer judges the tests against the mutants they were written for rather than against a score in a description. Several packages run in parallel, each in its own sandbox with its own interpreter and lockfile.

## Frequently asked questions

### Will mutation testing slow down my CI?

Only if you let it run on every push. The job the agent adds runs on one interpreter, reports rather than fails on the first merge, and the pull request says how long the mutation run takes in the sandbox. Most teams move it to a nightly schedule or a manual trigger and keep the ordinary suite on pull requests.

### Does it work with uv and tox?

Yes. The agent adds mutmut to the `tests` dependency group so `uv lock` records it, and runs it through a tox environment the same way the existing `py3.13` and `typing` environments run, with `uv run --locked tox run -e mutation`. No second tool chain is introduced.

### What is an equivalent mutant, and why not write a test for it?

A mutant whose change no caller can observe, such as a different wording in a `repr` or a default argument every code path overrides. A test that kills it has to assert the implementation detail itself, which makes the suite brittle for no safety. The agent excludes such mutants with a reason and the reviewer can disagree on any of them.

### Will it change my existing tests?

No. It adds tests to the existing files and leaves the ones that are there untouched. If a surviving mutant shows an existing test asserting the wrong thing, the agent says so in the pull request rather than editing it.

### Why does it leave the coverage configuration alone?

Because fixing it is a separate change. The agent reads `[tool.coverage.run]`, sees it names another package, and uses the modules under `src/` directly to decide what to mutate; the misnaming goes in the description for a reviewer to fix in its own pull request.

## More pytest jobs

- [Run a pytest Suite in Parallel With pytest-xdist](https://getrush.ai/run/add-parallel-tests/pytest/)
- [Add Property-Based Tests with Hypothesis to a pytest Suite](https://getrush.ai/run/add-property-tests/pytest/)
- [Add Random Test Order to a pytest Suite](https://getrush.ai/run/add-random-test-order/pytest/)

## The same job on other stacks

- [Add Mutation Testing to a Tokio Project](https://getrush.ai/run/add-mutation-testing/tokio/)
- [Add Mutation Testing to a TypeScript Library](https://getrush.ai/run/add-mutation-testing/typescript-library/)
- [Add Mutation Testing to a .NET Solution](https://getrush.ai/run/add-mutation-testing/dotnet/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
