What the agent does
The agent reads how the tests already run before it adds coverage, because in a repository like shiva-sripelly/connect4-ai the workflow is the contract. .github/workflows/ci.yml runs on every push to main and every pull request, on a matrix of Python 3.10, 3.11 and 3.12, with pip install -e ".[dev]", ruff check . and pytest -v. pyproject.toml declares fastapi and uvicorn[standard] as dependencies, a dev extra of pytest, httpx and ruff, testpaths = ["tests"] with pythonpath = ["."] under [tool.pytest.ini_options], and packages = ["connect4"] for setuptools. There is no coverage tool anywhere: not in the extras, not in the workflow, not in .gitignore.
The package is five modules: engine.py (the board, rules and validation), ai.py (negamax with alpha-beta pruning), api.py (a FastAPI app with POST /api/move, GET /api/health and /), cli.py (an argparse loop that reads moves from input()), and __main__.py. The three test files under tests/ cover the engine, the AI's tactics and the API through fastapi.testclient.TestClient; pytest -v in the sandbox reports 44 passed, the count the README promises, with one StarletteDeprecationWarning about using httpx with the test client. ruff check . passes.
With pytest-cov installed, pytest --cov=connect4 --cov-branch --cov-report=term-missing puts a number on each module: ai.py 98%, api.py and engine.py 97%, each with one or two uncovered lines and a partial branch, and cli.py and __main__.py at 0%, because nothing drives the terminal game. Total coverage is 84% with or without branch measurement, so a guess like --cov-fail-under=90 turns the suite red on its first run (FAIL Required test coverage of 90% not reached. Total coverage: 83.97%). The agent sets the threshold just under the measured total, so it blocks regressions without inventing work, and writes the per-file table into the pull request so the team can see that the gap is the CLI, not the engine.
The change is small: pytest-cov joins the dev extra, a [tool.coverage.run] section names source = ["connect4"] with branch = true, [tool.coverage.report] carries fail_under and show_missing = true, .gitignore gains .coverage and coverage.xml, and the workflow's test step becomes pytest --cov --cov-report=term-missing --cov-report=xml. The XML report is uploaded with actions/upload-artifact from one matrix leg only, so a pull request gets one report rather than three. The agent runs the exact command the workflow will run, on the Python versions the matrix names, before it opens the pull request.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add test coverage to CI for this Python project. Read pyproject.toml, .github/workflows/ci.yml, .gitignore, the modules under connect4/ and the tests under tests/ first, then install the dev extra and run pytest -v and ruff check . to confirm the suite passes as committed. Add pytest-cov to the dev extra, configure coverage in pyproject.toml with the package as source and branch coverage on, measure coverage with --cov-report=term-missing, and set fail_under to a threshold the current suite meets with a small margin rather than a round number it does not. Change the workflow's test step to run pytest with --cov, --cov-report=term-missing and --cov-report=xml, and upload coverage.xml as an artifact from one matrix leg. Add the coverage files to .gitignore. Do not write new tests and do not omit files from the measurement; report the per-file table in the pull request and name the modules with no coverage. Open a pull request that pastes the pytest output with the coverage table.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that changes pyproject.toml, .github/workflows/ci.yml and .gitignore, and nothing under connect4/ or tests/. The description pastes the pytest --cov run from the sandbox: 44 passed, the per-file table with cli.py and __main__.py at 0% and the other three modules at 97% or 98%, the total, and the fail_under it chose with the line that shows the threshold passing.
The description names the uncovered modules as a finding rather than a fix: cli.py reads moves with input() in a loop and nothing exercises it, and __main__.py only calls it. A follow-up could drive cli.main with a patched input and capsys; the pull request leaves that choice to the team and says so. It also quotes the deprecation warning from the test client so the team knows it predates this change.
Before merging, a reviewer checks that fail_under is a floor the suite clears on every Python in the matrix, not only the one the agent ran, decides whether the XML artifact is enough or a coverage service should receive it, and confirms the workflow's first run on the pull request prints the same table the description quotes.
Run it on every pull request that touches the test workflow
An automation in the Project with a github trigger on pull_request.opened and pull_request.synchronize runs this prompt against a pull request that changes .github/workflows/ci.yml or pyproject.toml; the trigger adds the pull request to the prompt, and the run reads the branch, reruns the suite with coverage and posts the per-file table where the team reads it. Pull requests from forks never start a run.
A team that wants coverage on every Python repository at once instead sends one POST /o/{org}/sessions per repository from a script, with the repository in resources and a request_id that is a UUID v5 of its full name, so a retried call returns the run already started. Each run opens at most one pull request.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
The hard part of a coverage threshold is picking a number that is true. A round number copied from a blog turns the build red on the first run, and a number set to pass silently excuses the modules nobody tests. The agent measures the suite in its own sandbox first, reads the per-file table, and sets the floor where the code is, with the uncovered modules named in the pull request instead of hidden by an omit line.
Every command and its output stays on the run, so a reviewer reads the coverage table and the passing threshold without installing anything. Several Python repositories can be measured at once, each in its own sandbox with its own interpreter versions, and a suite that fails under coverage is reported with the output rather than papered over by a lower number.
Frequently asked questions
Will the threshold break the build?
Not on the first run. The agent measures the suite before it sets fail_under and chooses a value just under the measured total, which for this project was 84% with cli.py and __main__.py at 0%. A threshold of 90% would have failed immediately; the pull request quotes that failure so the reviewer can see why the lower number was chosen.
Why not exclude cli.py from the measurement?
Because an omit line hides the gap instead of reporting it. cli.py is real code the project ships as the connect4 console script, and the table showing it at 0% is the most useful line in the pull request. The agent leaves the measurement honest and names the module as a candidate for a test that patches input.
Does the matrix upload three coverage reports?
No. Every leg runs pytest --cov and enforces the threshold, but the actions/upload-artifact step is conditioned on one Python version, so a pull request gets one coverage.xml. Sending that file to a coverage service instead is a one-step change the reviewer can ask for.
Does branch coverage change the number?
It can only lower it, since a partially taken if counts against the total once branch = true is set. The agent sets fail_under from a run with branch coverage already on, so the threshold in the pull request matches the measurement the workflow will make.
Will this work with uv or Poetry instead of pip?
Yes, as long as the test command is the same. This project installs with pip install -e ".[dev]", so pytest-cov goes in the dev extra; under uv it would go in a dev dependency group and under Poetry in [tool.poetry.group.dev.dependencies]. The coverage configuration lives in pyproject.toml either way and the workflow step is unchanged.