All jobs

Add Coverage to CI for a Litestar App

Yes. An agent makes your Litestar workflow's coverage step measure the real app, sets a floor it measured, puts the table on the job summary, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the workflow before it adds anything, because in an app laid out like litestar-org/litestar-hello-world coverage is already named and measures nothing. app.py at the repository root defines two handlers, GET /async and GET /sync, and builds app = Litestar(route_handlers=[...]); tests/test_app.py calls both through litestar.testing.TestClient. pyproject.toml lists pytest, pytest-asyncio and pytest-cov in the PDM test group, sets [tool.coverage.run] omit = ["*/tests/*"] and an exclude_lines list, and [tool.pdm.scripts] test = "pytest". .github/workflows/ci.yaml runs a validate job with pre-commit and a test matrix on Python 3.8 to 3.11 through the reusable .github/workflows/test.yaml, passing coverage: true only on 3.11.

In test.yaml the coverage step is pdm run pytest docs/examples tests --cov=src --cov-report=xml, followed by actions/upload-artifact@v4 of coverage.xml. Two things are wrong with that line. The repository has no docs/examples directory, and pytest exits with an error when a path it is given does not exist, so the step cannot pass as written. And --cov=src names a package that does not exist either: the application is the top-level module app, so even a successful run would report no measured lines. The sonar job downstream downloads coverage-xml and rewrites paths with a sed for litestar/litestar, another repository's name, and the Makefile's coverage target repeats --cov=src. [tool.pyright] include = ["src"] has the same stale path.

The fix is small and the agent keeps it that way. The step becomes pdm run pytest tests --cov=app --cov-report=xml --cov-report=term-missing --cov-fail-under=90, [tool.coverage.run] gains source = ["app"] so a local pytest --cov measures the same thing, the Makefile target and the pyright include point at app.py, and the sed in the sonar job replaces the real repository path. A new step appends the term-missing table to $GITHUB_STEP_SUMMARY, so the number is on the job page without downloading the artifact. The threshold is set from what the two tests cover: both handlers run, so the agent reports the measured percentage in the pull request and picks a floor just under it rather than a round number nobody checked.

Before opening the pull request the agent runs the workflow's own commands in its sandbox: pdm install, pdm run pytest tests --cov=app --cov-report=term-missing, and pre-commit run --all-files, since validate runs ruff with select = ["ALL"], codespell, mypy and pyright over app.py and tests, and a YAML change that trips prettier or a pyproject.toml change that trips pdm-lock-check would fail that job. The output of each is kept with the run and quoted in the pull request.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Make the coverage step in this Litestar repository's CI real. Read pyproject.toml, Makefile, .pre-commit-config.yaml, app.py, tests/test_app.py and every workflow under .github/workflows first. In test.yaml, change the coverage step to run pytest tests with --cov=app, an XML and a term-missing report and a --cov-fail-under floor just under the measured percentage; drop the docs/examples path, which does not exist. Set [tool.coverage.run] source to app in pyproject.toml, fix the Makefile coverage target and the pyright include to point at app.py, fix the sed path in the sonar job to this repository's name, and append the term-missing table to GITHUB_STEP_SUMMARY. Run pdm install, pdm run pytest tests --cov=app --cov-report=term-missing and pre-commit run --all-files, and quote the output. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request changes .github/workflows/test.yaml (the coverage step, the summary step), .github/workflows/ci.yaml (the sed path in sonar), pyproject.toml (source, the pyright include) and Makefile (the coverage target). Its description quotes the term-missing table from the agent's own run, names the floor it chose and why, and says plainly that the previous step named a directory that does not exist and a package that does not exist.

A reviewer checks the floor against the table, confirms the validate job's hooks still pass on the changed files, and decides whether coverage: true should stay on 3.11 only or move to the newest version in the matrix. Once merged, the next pull request's 3.11 job shows the coverage table in its summary and goes red if the number drops under the floor.

No test is added or changed. The two tests in tests/test_app.py are what the step measures, and the pull request is about making that measurement true, not about adding more of it.

Run it on every new Python repository

A platform team that wants every service to publish a coverage number starts this from its own server: a handler lists the repositories, and for each one whose workflow has a --cov flag that names a path missing from the tree, or no coverage step at all, sends this body to POST /o/{org}/sessions with that repository in resources. A request_id that is a UUID v5 of the repository's full name makes a retried call return the run it already started, so a cron that runs the handler weekly opens no duplicate.

A Rush automation with a github trigger on workflow_run.failure can watch the same repository and, because its runs are read-only, comment on the failing run with the line in test.yaml that caused it. The run that opens the pull request is the one above.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

A coverage step that names the wrong package is worse than none, because the badge and the artifact look fine while measuring nothing. The agent runs the exact command from the workflow in its own sandbox, reads the term-missing table, and sets the floor from a number it saw. The command output is kept with the pull request, so the reviewer compares the floor to evidence rather than to a guess.

Small CI fixes pile up across a team's repositories. Running this as a request per repository, all at once, means each pull request is specific to that workflow: one repository gets a source fix, another gets the missing pytest-cov dependency, and each description says which.

Frequently asked questions

Why does --cov=app work when there is no package?

pytest-cov takes a module or package name for --cov, and app.py at the repository root imports as the module app. Coverage then reports app.py alone, which is the whole application in this layout. [tool.coverage.run] source = ["app"] makes a plain coverage run agree.

Will the floor break the build on the next pull request?

Only if coverage drops under it. The agent sets the floor just under what the two existing tests measure, says the number in the pull request, and a reviewer can raise or lower it in one flag. A new handler without a test lowers the percentage, which is the point.

Does this change what pre-commit checks?

No. The validate job still runs ruff, codespell, mypy, pyright and prettier through .pre-commit-config.yaml. The agent runs pre-commit run --all-files before opening the pull request because prettier formats the workflow YAML and pdm-lock-check watches pyproject.toml.

What about the SonarCloud job?

It downloads the coverage-xml artifact and rewrites paths with sed before uploading. The agent fixes that sed to this repository's name and leaves the job's if as it is, since it is meant to run only for the upstream organization.

More Litestar jobs

The same job on other stacks

Keep reading