What the agent does
The agent reads the test setup end to end before it adds a threshold, because a threshold on the wrong measurement is worse than none. In a library laid out like pallets/itsdangerous, pyproject.toml keeps the test tools in a tests dependency group (freezegun and pytest), runs them through tox configured in [tool.tox] with the uv-venv-lock-runner, package = "wheel" and the command pytest -v --tb=short --basetemp={env_tmp_dir}, and sets filterwarnings = ["error"]. .github/workflows/tests.yaml runs uv run --locked tox run -e py3.13 and the other environments on a matrix with Windows, Mac and PyPy rows, and a uv-lock pre-commit hook keeps uv.lock honest. Coverage is half configured and never run: [tool.coverage.run] sets branch = true and source = ["jinja2", "tests"], a package name carried over from another Pallets project, [tool.coverage.paths] maps */site-packages back to src, and [tool.coverage.report] has three exclude_also patterns and no fail_under. Neither coverage nor pytest-cov appears in any dependency group or in uv.lock.
First it makes the measurement correct. source becomes ["itsdangerous", "tests"], matching [tool.flit.module], so coverage measures the package under src/ rather than a module that is not installed. The paths mapping already does the right thing for a wheel install: tox installs the built wheel into the environment, and */site-packages is folded back to src in the report, so line numbers point at the repository. coverage joins the tests group and uv lock updates the lockfile, which the pre-commit hook will check.
Then it measures before it sets a number. A new coverage tox environment runs coverage run -m pytest with the same flags the base command has, then coverage report; the agent runs it in its sandbox with uv run --locked tox run -e coverage and reads the total, with branch = true meaning the figure counts branches as well as lines. fail_under goes into [tool.coverage.report] at that total rounded down to a whole number, so the suite passes today and the first pull request that lowers coverage fails. The number is the measured one, written into the pull request with the report that produced it, and the per-file table is quoted so a maintainer can see where the uncovered lines are.
Last it adds one job to tests.yaml, coverage, on ubuntu-latest only, with the same setup-uv and setup-python steps the typing job uses and the command uv run --locked tox run -e coverage. It does not add coverage to the matrix rows: a Windows-only branch or a PyPy-only path would make the same threshold pass on one row and fail on another. Before pushing, the agent runs uv run --locked tox run -e py3.13, the new coverage environment, and pre-commit run --all-files through the style environment, and quotes each, including the uv-lock hook passing on the updated lockfile.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add coverage thresholds to this pytest project. Read pyproject.toml, uv.lock, .pre-commit-config.yaml and .github/workflows/tests.yaml first. Fix [tool.coverage.run] source to name this package, add coverage to the tests dependency group and run uv lock. Add a coverage tox environment in [tool.tox] that runs coverage run -m pytest with the base command's flags and then coverage report. Run it, read the total with branch coverage on, and set fail_under in [tool.coverage.report] to that total rounded down; do not invent the number. Add a coverage job to tests.yaml on ubuntu-latest using the same uv and Python setup steps as the typing job; do not add coverage to the matrix rows, and say why. Run tox -e py3.13, the new environment and pre-commit run --all-files, and quote the results including the per-file coverage table. Open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request with three files changed: pyproject.toml (the source fix, coverage in the tests group, the coverage tox environment and fail_under), uv.lock, and .github/workflows/tests.yaml with the new job. The description quotes the coverage report with its per-file table and total, the py3.13 run passing with the plugin absent from the matrix rows, and the pre-commit run with uv-lock green.
The run's record keeps the first coverage run, the one made before fail_under existed, so a reviewer can see the number was measured and not chosen.
Before merging, a reviewer decides whether the threshold should sit below the measured value to leave room for a refactor, and whether tests should stay in source, which makes an unexecuted test file count against the total, a choice the Pallets config already made and the agent kept.
Run it on every pull request
Once fail_under is set, the coverage job fails any pull request that lowers the total, and the useful follow-on is the explanation. A github trigger on pull_request.opened and pull_request.synchronize starts a run for each pull request a person opens, with a prompt asking the agent to run the coverage tox environment on that branch and on the base, and to post one review that lists the lines the pull request added without a test, by file and line, when the total fell. The trigger adds the pull request after the prompt, so the prompt names no branch.
A pull request a bot opens never fires the trigger, so a handler on the reader's server starts those runs with the …/pull/<n> URL in resources and a request_id that is a UUID v5 of the head commit SHA, so a synchronize after a rebase starts a new run and a redelivered webhook does not. A pull request that keeps or raises the total gets no review.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
A coverage threshold is a number, and the only honest way to pick it is to measure first, with the right package under measurement. Here the config measured a package that is not installed, so any threshold set from it would have been set against zero. In a Rush sandbox the agent fixes the source, runs the real suite from the lockfile, reads the total and writes that into the config, with the report in the record.
The pull request carries the tox environment and the CI job so the number is enforced on every pull request from then on, and the lockfile change passes the hook the project uses to guard it. Across every Python repository a team maintains, the job runs in parallel, one sandbox and one pull request each, with each threshold measured on its own suite.
Frequently asked questions
Why coverage instead of pytest-cov?
Because the repository already configures coverage directly in pyproject.toml, and coverage run -m pytest uses that config as it is. pytest-cov adds a plugin and flags for the same measurement; it works too, and switching is a one-line change in the tox command.
Why not run coverage in every matrix row?
Because the total differs by platform: a branch that runs only on Windows or only on PyPy is covered on one row and not another, so one fail_under cannot hold on all of them. One ubuntu-latest job gives a single comparable number.
Does branch coverage change the number?
Yes, branch = true is already set, so partial branches count against the total and the measured figure is lower than line coverage alone. The agent keeps the setting and measures with it on, so the threshold matches what the report shows.
What if the threshold is too high for a refactor?
Lower fail_under in the same pull request as the refactor, with the new report quoted. The setting is one line, and the CI job will show the new total next to it.