# Add Test Coverage to CI for a pytest Project

Yes. An agent adds pytest-cov to your dev extras, measures real coverage per file, sets a threshold the suite already meets, reports it on every PR, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-coverage-to-ci/pytest/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads how the tests already run before it adds coverage, because in a repository like shiva-sripelly/connect4-ai the workflow is the contract. `.github/workflows/ci.yml` runs on every push to `main` and every pull request, on a matrix of Python 3.10, 3.11 and 3.12, with `pip install -e ".[dev]"`, `ruff check .` and `pytest -v`. `pyproject.toml` declares `fastapi` and `uvicorn[standard]` as dependencies, a `dev` extra of `pytest`, `httpx` and `ruff`, `testpaths = ["tests"]` with `pythonpath = ["."]` under `[tool.pytest.ini_options]`, and `packages = ["connect4"]` for setuptools. There is no coverage tool anywhere: not in the extras, not in the workflow, not in `.gitignore`.

The package is five modules: `engine.py` (the board, rules and validation), `ai.py` (negamax with alpha-beta pruning), `api.py` (a FastAPI app with `POST /api/move`, `GET /api/health` and `/`), `cli.py` (an argparse loop that reads moves from `input()`), and `__main__.py`. The three test files under `tests/` cover the engine, the AI's tactics and the API through `fastapi.testclient.TestClient`; `pytest -v` in the sandbox reports 44 passed, the count the README promises, with one `StarletteDeprecationWarning` about using `httpx` with the test client. `ruff check .` passes.

With `pytest-cov` installed, `pytest --cov=connect4 --cov-branch --cov-report=term-missing` puts a number on each module: `ai.py` 98%, `api.py` and `engine.py` 97%, each with one or two uncovered lines and a partial branch, and `cli.py` and `__main__.py` at 0%, because nothing drives the terminal game. Total coverage is 84% with or without branch measurement, so a guess like `--cov-fail-under=90` turns the suite red on its first run (`FAIL Required test coverage of 90% not reached. Total coverage: 83.97%`). The agent sets the threshold just under the measured total, so it blocks regressions without inventing work, and writes the per-file table into the pull request so the team can see that the gap is the CLI, not the engine.

The change is small: `pytest-cov` joins the `dev` extra, a `[tool.coverage.run]` section names `source = ["connect4"]` with `branch = true`, `[tool.coverage.report]` carries `fail_under` and `show_missing = true`, `.gitignore` gains `.coverage` and `coverage.xml`, and the workflow's test step becomes `pytest --cov --cov-report=term-missing --cov-report=xml`. The XML report is uploaded with `actions/upload-artifact` from one matrix leg only, so a pull request gets one report rather than three. The agent runs the exact command the workflow will run, on the Python versions the matrix names, before it opens the pull request.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add test coverage to CI for this Python project. Read pyproject.toml, .github/workflows/ci.yml, .gitignore, the modules under connect4/ and the tests under tests/ first, then install the dev extra and run pytest -v and ruff check . to confirm the suite passes as committed. Add pytest-cov to the dev extra, configure coverage in pyproject.toml with the package as source and branch coverage on, measure coverage with --cov-report=term-missing, and set fail_under to a threshold the current suite meets with a small margin rather than a round number it does not. Change the workflow's test step to run pytest with --cov, --cov-report=term-missing and --cov-report=xml, and upload coverage.xml as an artifact from one matrix leg. Add the coverage files to .gitignore. Do not write new tests and do not omit files from the measurement; report the per-file table in the pull request and name the modules with no coverage. Open a pull request that pastes the pytest output with the coverage table.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that changes `pyproject.toml`, `.github/workflows/ci.yml` and `.gitignore`, and nothing under `connect4/` or `tests/`. The description pastes the `pytest --cov` run from the sandbox: 44 passed, the per-file table with `cli.py` and `__main__.py` at 0% and the other three modules at 97% or 98%, the total, and the `fail_under` it chose with the line that shows the threshold passing.

The description names the uncovered modules as a finding rather than a fix: `cli.py` reads moves with `input()` in a loop and nothing exercises it, and `__main__.py` only calls it. A follow-up could drive `cli.main` with a patched `input` and `capsys`; the pull request leaves that choice to the team and says so. It also quotes the deprecation warning from the test client so the team knows it predates this change.

Before merging, a reviewer checks that `fail_under` is a floor the suite clears on every Python in the matrix, not only the one the agent ran, decides whether the XML artifact is enough or a coverage service should receive it, and confirms the workflow's first run on the pull request prints the same table the description quotes.

## Run it on every pull request that touches the test workflow

An automation in the Project with a `github` trigger on `pull_request.opened` and `pull_request.synchronize` runs this prompt against a pull request that changes `.github/workflows/ci.yml` or `pyproject.toml`; the trigger adds the pull request to the prompt, and the run reads the branch, reruns the suite with coverage and posts the per-file table where the team reads it. Pull requests from forks never start a run.

A team that wants coverage on every Python repository at once instead sends one `POST /o/{org}/sessions` per repository from a script, with the repository in `resources` and a `request_id` that is a UUID v5 of its full name, so a retried call returns the run already started. Each run opens at most one pull request.

## Why run it on Rush?

The hard part of a coverage threshold is picking a number that is true. A round number copied from a blog turns the build red on the first run, and a number set to pass silently excuses the modules nobody tests. The agent measures the suite in its own sandbox first, reads the per-file table, and sets the floor where the code is, with the uncovered modules named in the pull request instead of hidden by an `omit` line.

Every command and its output stays on the run, so a reviewer reads the coverage table and the passing threshold without installing anything. Several Python repositories can be measured at once, each in its own sandbox with its own interpreter versions, and a suite that fails under coverage is reported with the output rather than papered over by a lower number.

## Frequently asked questions

### Will the threshold break the build?

Not on the first run. The agent measures the suite before it sets `fail_under` and chooses a value just under the measured total, which for this project was 84% with `cli.py` and `__main__.py` at 0%. A threshold of 90% would have failed immediately; the pull request quotes that failure so the reviewer can see why the lower number was chosen.

### Why not exclude cli.py from the measurement?

Because an `omit` line hides the gap instead of reporting it. `cli.py` is real code the project ships as the `connect4` console script, and the table showing it at 0% is the most useful line in the pull request. The agent leaves the measurement honest and names the module as a candidate for a test that patches `input`.

### Does the matrix upload three coverage reports?

No. Every leg runs `pytest --cov` and enforces the threshold, but the `actions/upload-artifact` step is conditioned on one Python version, so a pull request gets one `coverage.xml`. Sending that file to a coverage service instead is a one-step change the reviewer can ask for.

### Does branch coverage change the number?

It can only lower it, since a partially taken `if` counts against the total once `branch = true` is set. The agent sets `fail_under` from a run with branch coverage already on, so the threshold in the pull request matches the measurement the workflow will make.

### Will this work with uv or Poetry instead of pip?

Yes, as long as the test command is the same. This project installs with `pip install -e ".[dev]"`, so `pytest-cov` goes in the `dev` extra; under uv it would go in a `dev` dependency group and under Poetry in `[tool.poetry.group.dev.dependencies]`. The coverage configuration lives in `pyproject.toml` either way and the workflow step is unchanged.

## More pytest jobs

- [Run Docstring Examples as Tests in CI With pytest](https://getrush.ai/run/add-doctests-to-ci/pytest/)
- [Add Mutation Testing to a pytest Suite](https://getrush.ai/run/add-mutation-testing/pytest/)
- [Run a pytest Suite in Parallel With pytest-xdist](https://getrush.ai/run/add-parallel-tests/pytest/)

## The same job on other stacks

- [Add Test Coverage Reporting to Python CI](https://getrush.ai/run/add-coverage-to-ci/python/)
- [Add Test Coverage Reporting to Quarkus CI](https://getrush.ai/run/add-coverage-to-ci/quarkus/)
- [Add Test Coverage Reporting to Rust CI](https://getrush.ai/run/add-coverage-to-ci/rust/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
