All jobs

Add Test Coverage to CI for an aiohttp Project

Yes. An agent adds pytest-cov to your aiohttp project's test target, combines coverage across every demo, publishes the report from CI, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the tests already run, because coverage has to be collected the same way. In a project laid out like aio-libs/aiohttp-demos, the root Makefile has a test target that is nine separate pytest invocations, one per demo directory under demos/, each with its own pytest.ini, its own package (aiohttpdemo_polls, aiohttpdemo_blog, aiohttpdemo_chat, graph, imagetagger, moderator, moderator_bot, motortwit, shortify) and its own tests/ directory. The root requirements-dev.txt pins pytest and pytest-aiohttp and nothing for coverage, and make install pip-installs that file plus each demo's requirements. .github/workflows/ci.yml has a test job with Postgres and Redis service containers that runs PYTHONASYNCIODEBUG=1 make test on Python 3.12, a lint job for make lint and the docs spelling check, and a doc job. Nothing measures coverage, and nothing uploads anything.

Nine invocations mean nine separate measurements unless they share a data file, so the agent adds pytest-cov to the test section of requirements-dev.txt, writes a root .coveragerc whose [run] section sets source to demos, branch to true and omits every tests/ directory, and changes each line of the test target to pass --cov --cov-append --cov-report= so the invocations accumulate into one .coverage file and print nothing until the end. A new coverage target then runs coverage report -m and coverage xml. Before it commits to that, the agent reads each demo's pytest.ini: demos/polls/pytest.ini turns every warning into an error with filterwarnings = error, so a warning the coverage plugin emits, for instance when a package is imported before measurement starts, would fail that demo's run, and the agent checks for exactly that when it runs the target.

In ci.yml the test job gains three steps after make test: make coverage, a step that appends the text report to $GITHUB_STEP_SUMMARY so the numbers show on the workflow run page, and actions/upload-artifact for coverage.xml and the .coverage data file. The agent does not add a Codecov upload or a minimum threshold, because the first number is the baseline and both are decisions the team makes once they have seen it; the description shows where a --cov-fail-under would go.

Then it runs the whole thing in its sandbox the way CI does: make install, a Postgres and a Redis started locally for the demos that need them, PYTHONASYNCIODEBUG=1 make test and make coverage. The per-file report is pasted into the pull request, so a reviewer sees which demos the tests reach and which modules in demos/ have no test touching them at all.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add test coverage to CI for this aiohttp project. Read Makefile, requirements-dev.txt, .flake8, .github/workflows/ci.yml and every demos/*/pytest.ini first. Add pytest-cov to requirements-dev.txt, write a root .coveragerc that measures demos with branch coverage and omits the tests directories, and change each pytest line of the Makefile test target to pass --cov --cov-append --cov-report= so the nine demos accumulate into one data file. Add a coverage target that prints coverage report -m and writes coverage xml. In ci.yml, after make test, run make coverage, append the report to the step summary and upload coverage.xml as an artifact. Do not add a threshold or a third-party upload. Install the requirements, start Postgres and Redis in the sandbox, run make test and make coverage exactly as CI does, check that no demo with filterwarnings set to error fails on a coverage warning, and open a pull request with the full report.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request changes Makefile, requirements-dev.txt and .github/workflows/ci.yml, adds .coveragerc, and adds .coverage and coverage.xml to .gitignore. Its description pastes the coverage report -m table from the sandbox run, demo by demo, and notes which demos' pytest.ini files have filterwarnings = error and that they passed with the plugin on.

The CI workflow runs on the pull request, and the test job's summary page now carries the same table, with coverage.xml attached as an artifact a reviewer can download. The lint and doc jobs are unchanged.

Before merging, a reviewer decides whether to add --cov-fail-under once the baseline is agreed, whether to send coverage.xml to a coverage service, and whether the demos with the lowest numbers in the table deserve tests before the threshold goes in.

Run it on every nightly

Once the report is in CI, the useful question is what changed, and that is a job for a schedule. An automation on the Project with a schedule trigger set to a nightly slot in the team's timezone and the prompt "run make test and make coverage, compare the per-file report with the one in the latest run of the CI workflow on the default branch, and if any module in demos/ dropped, comment on the most recent pull request that touched it" posts a comment only when something moved. Automation runs are read-only, so the comment is all it posts.

A team that wants the gaps closed starts one run per demo from its own server, each a POST /o/{org}/sessions with the repository in resources, a prompt that names the demo directory and asks for tests on its uncovered functions, and a request_id that is a UUID v5 of the demo's path, so a retried call returns the run already started rather than opening a second pull request.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

Coverage across nine separately invoked test suites is a configuration problem before it is a testing problem: without --cov-append the last demo overwrites the rest, without --cov-report= nine partial tables scroll past, and a pytest.ini that turns warnings into errors can fail a demo for a plugin message nobody asked for. The agent reads those files, runs the target in its own sandbox with the same services CI uses, and fixes what breaks before it opens anything.

The install, the nine test runs and the final report stay with the run, and the pull request quotes the table. A team that keeps several aiohttp repositories gets the same change on each at once, with each repository's own baseline rather than a number copied from a sibling.

Frequently asked questions

Why one data file instead of a report per demo?

Because the Makefile runs the demos as nine pytest commands and the team wants one answer. --cov-append makes each command add to the same .coverage file instead of replacing it, and --cov-report= keeps each command quiet, so make coverage prints the single table once at the end.

Will filterwarnings = error break the run?

It can, which is why the agent reads every demos/*/pytest.ini and runs the target before committing. A coverage plugin warning in a demo with that setting fails that demo. If one appears, the agent fixes its cause, usually the order in which a package is imported relative to measurement, rather than silencing the warning.

Why no Codecov and no threshold?

Because the first run produces the baseline, and both decisions depend on it. A threshold set before anyone has seen the number is either too low to matter or fails the next pull request. The pull request shows the table and names where --cov-fail-under goes, so adding it later is one line.

Do the tests need Postgres and Redis?

Some do, which is why ci.yml already starts both as service containers for the test job. The agent starts the same two in its sandbox so make test runs the way CI runs it, and the coverage it reports is the coverage CI will report.

More aiohttp jobs

The same job on other stacks

Keep reading