What the agent does
The agent reads the workflow and the test settings before it adds anything. In a Wagtail site like wagtail/bakerydemo, .github/workflows/ci.yml has three jobs: lint-server, which installs requirements/development.txt with uv and runs make lint-server (ruff format and check, curlylint --parse-only, djhtml --check); lint-client, which runs the npm linters on the Node version in .nvmrc; and test-server, which sets DJANGO_SETTINGS_MODULE=bakerydemo.settings.test, installs the same requirements with uv, and runs ./manage.py check, ./manage.py makemigrations --check --noinput and ./manage.py test. requirements/base.txt pins Django 6.0 and Wagtail 8.0, and development.txt adds only ruff, curlylint and djhtml, so coverage is not installed anywhere. bakerydemo/settings/test.py imports the base settings, switches the cache to local memory, the password hasher to MD5 and the task backend to ImmediateBackend, and without a DATABASE_URL the base settings fall back to SQLite, which is how the suite runs on GitHub. The tests themselves are bakerydemo/base/tests/test_home_page.py and test_seo.py, WagtailPageTestCase classes that render the home page and check the SEO metadata and robots.txt; the blog, breads, locations, people and search apps have no tests. Nothing reads a coverage number.
The change is three files. coverage goes into requirements/development.txt beside the linters, pinned. A .coveragerc sets source = bakerydemo, branch = true and an omit list for */migrations/*, */tests/*, bakerydemo/settings/* and manage.py, so the number measures the models, blocks, views and template tags rather than generated code. In ci.yml, the test-server job's last step becomes uv run coverage run ./manage.py test, followed by uv run coverage report --fail-under=<measured> and uv run coverage html, and an actions/upload-artifact step uploads htmlcov/ so a reviewer can open the report from the workflow run. The check and makemigrations --check steps stay as they are, and the two lint jobs are untouched.
The measured number comes from the sandbox. The agent creates the uv virtual environment the workflow does, installs requirements/development.txt, exports the test settings module, runs coverage run ./manage.py test and coverage report, and takes the TOTAL line. With two test modules that render the home page, the figure covers base/models.py, base/blocks.py and the template tags the home page uses and leaves most of blog and breads uncovered, so the threshold is set at that total rounded down, not at a target: the pull request passes on the code as it is, and only a drop fails later. It also adds htmlcov/ and .coverage to .gitignore.
The agent does not write tests in this run and does not change settings/test.py or the Makefile; make lint-server still passes because .coveragerc is not a Python file and ruff has no opinion on it. If the team runs tests against PostgreSQL locally by setting DATABASE_URL, the coverage command is the same, since coverage run wraps manage.py whatever the database.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "codex",
"input": "Add test coverage reporting with a threshold to this Wagtail project's CI and open a pull request. Read .github/workflows/ci.yml, requirements/base.txt, requirements/development.txt, bakerydemo/settings/test.py, Makefile and every file under bakerydemo/*/tests first. Add a pinned coverage to requirements/development.txt and a .coveragerc with source = bakerydemo, branch = true, and an omit list for migrations, tests, the settings package and manage.py. In the sandbox create the uv environment the test-server job uses, set DJANGO_SETTINGS_MODULE=bakerydemo.settings.test, run coverage run ./manage.py test and coverage report, and read the TOTAL. In the test-server job replace the test step with uv run coverage run ./manage.py test, then uv run coverage report --fail-under set to that total rounded down, then uv run coverage html, and upload htmlcov as an artifact. Add htmlcov and .coverage to .gitignore. Do not add tests and do not change the lint jobs, settings/test.py or the Makefile. Put the measured total and the coverage report in the pull request description.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that changes .github/workflows/ci.yml, requirements/development.txt and .gitignore and adds .coveragerc. The description states the total measured in the sandbox, quotes the coverage report table with its per-file lines, and names the Django, Wagtail and Python versions it was measured with. The session keeps the full test output, and the workflow run on the pull request attaches htmlcov/ as an artifact.
A reviewer checks that the --fail-under value equals the total in the description, that the omit list leaves out only generated and test code, and that check and makemigrations --check still run before the suite. The pull request's own test-server job is the first run of the threshold on GitHub and should pass, because the number came from the same command on the same code; the two lint jobs run unchanged.
Run it on every new Wagtail site repository
Run it once per site rather than once: a script that reads GET /o/{org}/github for the installation's repositories and calls POST /o/{org}/sessions with this body and each repository's URL in resources, with a request_id per repository so a retry returns the run already started. Each site gets its own sandbox, its own measurement and its own threshold.
Give the key a webhook_url and Rush POSTs run.completed, run.failed or run.cancelled when each turn ends, signed per Standard Webhooks, with data.session_url to read for the pull request. A key runs at most max_concurrent sessions at once, 4 by default; a batch over many Wagtail sites can still be sent in full, since each site past that number is queued (202, status: "queued") and its run starts, oldest first, when a slot frees.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
A threshold set by guessing is either ignored or fails on the first pull request. The agent gets the real number by running the real suite under coverage in its own sandbox, with the same uv environment and the same settings module the workflow uses, and the coverage report table is on the session, so the value in the workflow has a recorded source and the per-file lines show where the gaps are.
The job ends at a pull request whose own CI run has executed the new check, not at a suggested YAML block. The same request body works on every Wagtail site a team maintains, each with its own measured figure, and a follow-up on the same parked session can write the first tests for the blog or breads app using the report to choose where.
Frequently asked questions
Where does the threshold come from?
From running coverage run ./manage.py test and coverage report in the sandbox with bakerydemo.settings.test, and reading the TOTAL line. It is rounded down and written as --fail-under in the workflow, and the pull request description states it with the full table.
Will it write tests to raise coverage?
Not in this run. It measures what the existing page-rendering tests cover and stops a drop. The htmlcov/ artifact and the report table show that blog, breads, locations, people and search have no tests, which is where a follow-up would start.
Does it need PostgreSQL?
No. Without DATABASE_URL the base settings use SQLite, which is what the test-server job already relies on, and coverage run wraps manage.py either way. A team that sets DATABASE_URL in the job gets the same measurement against PostgreSQL.
Does it change the lint jobs or the Makefile?
No. lint-server and lint-client are untouched, and .coveragerc is not something ruff, curlylint or djhtml read. The only workflow change is inside the test-server job, after the check and makemigrations --check steps.
More Wagtail jobs
The same job on other stacks
- Add Test Coverage to CI for an aiohttp Project (aiohttp)
- Add Test Coverage Reporting to Angular CI (Angular)
- Add Test Coverage Reporting to ASP.NET Core CI (ASP.NET Core)