All jobs

Add Test Coverage to CI for a Django REST Framework API

Yes. An agent adds pytest-cov to your Django REST Framework workflow, measures the suite against Postgres, sets a floor from the real number and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent starts with how the tests run now. In an API like HackSoftware/Django-Styleguide-Example, requirements/base.txt pins Django==5.2.7, djangorestframework==3.16.1 and psycopg2==2.9.11, and requirements/local.txt adds pytest==8.4.2, pytest-django==4.11.1, factory-boy, mypy==1.18.2 and ruff==0.14.0, with no coverage package anywhere. pytest.ini sets DJANGO_SETTINGS_MODULE = config.django.test and the file patterns; config/django/test.py speeds the suite up with the MD5 password hasher, eager Celery and a local-memory cache. Fourteen test_*.py files sit in per-app tests/ packages, from users/tests/services/test_user_create.py to files/tests/flows/test_direct_upload.py.

.github/workflows/django.yml runs on push and has two jobs that both execute the suite. docker_build builds the Compose stack and runs ruff, mypy and py.test through docker compose run django. build starts a postgres:latest service, installs requirements/local.txt on Python 3.12.4 with two layers of pip caching, then runs ruff check ., mypy --config mypy.ini styleguide_example/ and pytest; config/django/base.py switches DATABASES to that service whenever the GITHUB_WORKFLOW variable is set. A third job, deploy_to_heroku, is disabled with if: false. The workflow has no pull_request trigger, so today a fork's pull request runs nothing.

The agent adds pytest-cov to requirements/local.txt with an exact pin like its neighbours, and configures coverage in pyproject.toml, which so far holds only the [tool.ruff] tables: [tool.coverage.run] with source = ["styleguide_example"] and an omit list for */migrations/*, */tests/* and */apps.py, and [tool.coverage.report] with show_missing = true and skip_covered = true. pytest.ini gets an addopts line that turns on --cov, a terminal report with missing lines, and --cov-report=xml, so both jobs produce coverage without a workflow edit. In the build job the agent uploads coverage.xml with actions/upload-artifact@v4 and appends the terminal table to $GITHUB_STEP_SUMMARY, and it adds pull_request to on: so the checks run where reviews happen.

The floor is the part that has to be measured rather than typed. In the sandbox the agent starts Postgres, exports DATABASE_URL the way docker-compose.yml does, installs requirements/local.txt and runs pytest with coverage once to see the real percentage of styleguide_example. Only then does it write fail_under in [tool.coverage.report], a whole number just below what the suite achieves, so the floor stops regressions without failing the first build. The pull request pastes the terminal table and says where the number came from.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "codex",
  "input": "Add test coverage to the CI of this Django REST Framework project and open a pull request. Read .github/workflows/django.yml, pytest.ini, pyproject.toml, requirements/local.txt, config/django/base.py, config/django/test.py and docker-compose.yml first. Add pytest-cov to requirements/local.txt with an exact pin. In pyproject.toml add [tool.coverage.run] with source styleguide_example and omit for migrations, tests and apps.py, and [tool.coverage.report] with show_missing and skip_covered. In pytest.ini add addopts that enable --cov with a term-missing report and an xml report, so both the docker_build and build jobs produce coverage without further changes. In the build job upload coverage.xml as an artifact and write the terminal table to the job summary, and add pull_request to the workflow triggers. Start Postgres in the sandbox, set DATABASE_URL as docker-compose.yml does, install requirements/local.txt, run pytest once with coverage, and set fail_under in [tool.coverage.report] to a whole number just below the percentage you measured. Paste the coverage table in the pull request and say how the floor was chosen. Do not add Codecov or any service that needs a token, and leave the deploy_to_heroku job as it is.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request with one added line in requirements/local.txt, two new tables in pyproject.toml, an addopts line in pytest.ini, and in .github/workflows/django.yml a pull_request trigger plus two steps in the build job: the artifact upload and the summary. The description shows the coverage table from the sandbox run, per module with missing line numbers, and names the measured percentage the fail_under value sits under.

Because pytest.ini carries the options, the docker_build job gains the same report with no change to its docker compose run django py.test step, and the build job's pytest step fails if coverage drops below the floor. The coverage.xml artifact is attached to every run for any tool that wants it later.

A reviewer checks that the omit list leaves out only generated and test code, that the floor is close enough to the real number to catch a regression, and that the summary table reads well in the Actions UI. Then they decide whether the modules the table shows at the bottom deserve tests before the floor is raised.

Run it on every new Django service

A team running several Django APIs sends one POST /o/{org}/sessions per repository with this prompt and that repository in resources. Each run reads its own pytest.ini and workflow, so a project that runs tests through manage.py test instead of pytest gets coverage run wired in instead, and a project whose settings already read DATABASE_URL from the environment gets the sandbox database the same way. Each run opens its own pull request with its own measured floor.

Give the API key a webhook_url and Rush POSTs run.completed when each turn ends, with the session_url to read the pull request from, so the coverage numbers from every service can be compared in one place before any floor is raised.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

The configuration is small; the number is not. A fail_under picked by hand is either too low to mean anything or fails the first build. In a Rush sandbox the agent brings up Postgres, installs the project and runs the whole suite with coverage before writing the floor, and the run keeps the command and its table, so the pull request shows where the value came from.

The repository also has two jobs that run the same tests in different ways, one through Compose and one on the runner, and changes that help one and break the other are easy to make on a laptop with one of them. Putting the options in pytest.ini keeps the two in step, and the sandbox is where that is checked.

Frequently asked questions

Why not Codecov?

It needs a token and an account, and the prompt rules it out. coverage.xml is uploaded as a workflow artifact and the table is in the job summary, so the numbers are visible on every run; a Codecov or Coveralls step can read the same file later if the team wants trend graphs.

Does the docker_build job change?

Its workflow step does not. The coverage options live in pytest.ini, so docker compose run django py.test produces the same report and enforces the same floor. The requirements/local.txt change reaches it because docker/local.Dockerfile installs that file.

How is the floor chosen?

By measurement. The agent runs the suite once with coverage against Postgres in the sandbox and sets fail_under to a whole number just below that result, and says so in the description. The point is to stop coverage dropping, not to claim a target the code does not meet.

Will mypy or ruff complain?

The change adds no Python code, only configuration, and mypy.ini scopes type checking to styleguide_example/. Both run in the sandbox as part of checking the branch, and their output is in the run if anything did.

What about branch coverage?

The first pass measures line coverage, which is what the suite's size justifies. Turning on branch = true in [tool.coverage.run] lowers the number and should come with a newly measured floor, which is a follow-up run with one more line in the prompt.

More Django REST Framework jobs

The same job on other stacks

Keep reading