# Add Coverage Thresholds to a pytest Suite

Yes. An agent adds coverage to your pytest lockfile, fixes the measured source, sets fail_under from a real run, adds a tox env and CI job, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-coverage-thresholds/pytest/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads the test setup end to end before it adds a threshold, because a threshold on the wrong measurement is worse than none. In a library laid out like pallets/itsdangerous, `pyproject.toml` keeps the test tools in a `tests` dependency group (`freezegun` and `pytest`), runs them through tox configured in `[tool.tox]` with the `uv-venv-lock-runner`, `package = "wheel"` and the command `pytest -v --tb=short --basetemp={env_tmp_dir}`, and sets `filterwarnings = ["error"]`. `.github/workflows/tests.yaml` runs `uv run --locked tox run -e py3.13` and the other environments on a matrix with Windows, Mac and PyPy rows, and a `uv-lock` pre-commit hook keeps `uv.lock` honest. Coverage is half configured and never run: `[tool.coverage.run]` sets `branch = true` and `source = ["jinja2", "tests"]`, a package name carried over from another Pallets project, `[tool.coverage.paths]` maps `*/site-packages` back to `src`, and `[tool.coverage.report]` has three `exclude_also` patterns and no `fail_under`. Neither `coverage` nor `pytest-cov` appears in any dependency group or in `uv.lock`.

First it makes the measurement correct. `source` becomes `["itsdangerous", "tests"]`, matching `[tool.flit.module]`, so coverage measures the package under `src/` rather than a module that is not installed. The `paths` mapping already does the right thing for a wheel install: tox installs the built wheel into the environment, and `*/site-packages` is folded back to `src` in the report, so line numbers point at the repository. `coverage` joins the `tests` group and `uv lock` updates the lockfile, which the pre-commit hook will check.

Then it measures before it sets a number. A new `coverage` tox environment runs `coverage run -m pytest` with the same flags the base command has, then `coverage report`; the agent runs it in its sandbox with `uv run --locked tox run -e coverage` and reads the total, with `branch = true` meaning the figure counts branches as well as lines. `fail_under` goes into `[tool.coverage.report]` at that total rounded down to a whole number, so the suite passes today and the first pull request that lowers coverage fails. The number is the measured one, written into the pull request with the report that produced it, and the per-file table is quoted so a maintainer can see where the uncovered lines are.

Last it adds one job to `tests.yaml`, `coverage`, on `ubuntu-latest` only, with the same `setup-uv` and `setup-python` steps the `typing` job uses and the command `uv run --locked tox run -e coverage`. It does not add coverage to the matrix rows: a Windows-only branch or a PyPy-only path would make the same threshold pass on one row and fail on another. Before pushing, the agent runs `uv run --locked tox run -e py3.13`, the new `coverage` environment, and `pre-commit run --all-files` through the `style` environment, and quotes each, including the `uv-lock` hook passing on the updated lockfile.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add coverage thresholds to this pytest project. Read pyproject.toml, uv.lock, .pre-commit-config.yaml and .github/workflows/tests.yaml first. Fix [tool.coverage.run] source to name this package, add coverage to the tests dependency group and run uv lock. Add a coverage tox environment in [tool.tox] that runs coverage run -m pytest with the base command's flags and then coverage report. Run it, read the total with branch coverage on, and set fail_under in [tool.coverage.report] to that total rounded down; do not invent the number. Add a coverage job to tests.yaml on ubuntu-latest using the same uv and Python setup steps as the typing job; do not add coverage to the matrix rows, and say why. Run tox -e py3.13, the new environment and pre-commit run --all-files, and quote the results including the per-file coverage table. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request with three files changed: `pyproject.toml` (the `source` fix, `coverage` in the `tests` group, the `coverage` tox environment and `fail_under`), `uv.lock`, and `.github/workflows/tests.yaml` with the new job. The description quotes the coverage report with its per-file table and total, the `py3.13` run passing with the plugin absent from the matrix rows, and the pre-commit run with `uv-lock` green.

The run's record keeps the first coverage run, the one made before `fail_under` existed, so a reviewer can see the number was measured and not chosen.

Before merging, a reviewer decides whether the threshold should sit below the measured value to leave room for a refactor, and whether `tests` should stay in `source`, which makes an unexecuted test file count against the total, a choice the Pallets config already made and the agent kept.

## Run it on every pull request

Once `fail_under` is set, the `coverage` job fails any pull request that lowers the total, and the useful follow-on is the explanation. A `github` trigger on `pull_request.opened` and `pull_request.synchronize` starts a run for each pull request a person opens, with a prompt asking the agent to run the `coverage` tox environment on that branch and on the base, and to post one review that lists the lines the pull request added without a test, by file and line, when the total fell. The trigger adds the pull request after the prompt, so the prompt names no branch.

A pull request a bot opens never fires the trigger, so a handler on the reader's server starts those runs with the `…/pull/<n>` URL in `resources` and a `request_id` that is a UUID v5 of the head commit SHA, so a synchronize after a rebase starts a new run and a redelivered webhook does not. A pull request that keeps or raises the total gets no review.

## Why run it on Rush?

A coverage threshold is a number, and the only honest way to pick it is to measure first, with the right package under measurement. Here the config measured a package that is not installed, so any threshold set from it would have been set against zero. In a Rush sandbox the agent fixes the source, runs the real suite from the lockfile, reads the total and writes that into the config, with the report in the record.

The pull request carries the tox environment and the CI job so the number is enforced on every pull request from then on, and the lockfile change passes the hook the project uses to guard it. Across every Python repository a team maintains, the job runs in parallel, one sandbox and one pull request each, with each threshold measured on its own suite.

## Frequently asked questions

### Why coverage instead of pytest-cov?

Because the repository already configures `coverage` directly in `pyproject.toml`, and `coverage run -m pytest` uses that config as it is. `pytest-cov` adds a plugin and flags for the same measurement; it works too, and switching is a one-line change in the tox command.

### Why not run coverage in every matrix row?

Because the total differs by platform: a branch that runs only on Windows or only on PyPy is covered on one row and not another, so one `fail_under` cannot hold on all of them. One `ubuntu-latest` job gives a single comparable number.

### Does branch coverage change the number?

Yes, `branch = true` is already set, so partial branches count against the total and the measured figure is lower than line coverage alone. The agent keeps the setting and measures with it on, so the threshold matches what the report shows.

### What if the threshold is too high for a refactor?

Lower `fail_under` in the same pull request as the refactor, with the new report quoted. The setting is one line, and the CI job will show the new total next to it.

## More pytest jobs

- [Add Test Coverage to CI for a pytest Project](https://getrush.ai/run/add-coverage-to-ci/pytest/)
- [Run Docstring Examples as Tests in CI With pytest](https://getrush.ai/run/add-doctests-to-ci/pytest/)
- [Add Mutation Testing to a pytest Suite](https://getrush.ai/run/add-mutation-testing/pytest/)

## The same job on other stacks

- [Add Coverage Thresholds to a Vitest Suite](https://getrush.ai/run/add-coverage-thresholds/vitest/)
- [Add Coverage Thresholds to a Jest Suite](https://getrush.ai/run/add-coverage-thresholds/jest/)
- [Add Coverage Thresholds to a Mocha Suite With nyc](https://getrush.ai/run/add-coverage-thresholds/mocha/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
