# Add Missing Tests to Every Pull Request in a Flask App

Yes. An agent reads each pull request to your Flask app, writes pytest tests for the challenges API code it changes, runs pytest and pushes them.

[Run this on Rush](https://cloud.getrush.ai/run/test-every-pull-request/flask/), in a Project that has your repository.

## What the agent does

The agent reads the pull request's diff and maps it onto the API. In a Flask app like CTFd the challenges API is one Flask-RESTX namespace in `CTFd/api/v1/challenges.py`, with a resource class per route: `ChallengeList` on `/api/v1/challenges`, `Challenge` on `/<challenge_id>`, `ChallengeAttempt` on `/attempt`, and smaller resources for a challenge's solves, files, tags, topics, hints, flags, requirements, ratings and solution. For each method the pull request changes, such as a new visibility rule in `ChallengeList.get` or a new check in `ChallengeAttempt.post`, the agent lists the request and the user state that would reach it and checks whether a test already sends that request.

Then it reads how CTFd tests the API. `tests/api/v1/test_challenges.py` is a long file of plain pytest functions, each building its own app with `create_ctfd()` and tearing it down with `destroy_ctfd(app)`. Setup comes from `tests/helpers.py`: `gen_challenge`, `gen_flag`, `gen_hint`, `gen_solve`, `gen_fail`, `gen_tag`, `gen_team`, `gen_user`, `register_user` and `login_as_user`, with `set_config` toggling settings like `challenge_visibility`, and `freezegun` freezing the clock for start and end times. The agent writes each new test in that shape and in that file, with a docstring that states the question it answers, as the existing `test_api_challenges_get_visibility_public` does.

It runs the suite the way the repository does. `TestingConfig` in `CTFd/config.py` reads `TESTING_DATABASE_URL` and falls back to an in-memory `sqlite://`, uses a simple cache and no Redis, so the SQLite workflow needs no services. The Makefile's `test` target runs `pytest -rf --cov=CTFd` with `-n auto` for parallel workers, and the workflows call it as `uv run make test` on Python 3.11. The agent runs the new tests first with `uv run pytest tests/api/v1/test_challenges.py -k <name>`, then the whole challenges file, so a test that only passes alone does not get pushed.

A new test that fails because the challenge API does the wrong thing, for example letting a hidden challenge's details through to a non-admin, is not edited until it passes and is not committed red. The prompt forbids changing application code, so the agent describes the failure and the exact request in its comment, which is often the most useful part of the run.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "This pull request changes the challenges API. Read the diff against the base branch and list the new or changed behavior in CTFd/api/v1/challenges.py, and in any plugin or helper it calls, that no existing test covers. Write pytest tests for each item in tests/api/v1/test_challenges.py, following the existing style: create_ctfd and destroy_ctfd per test, the gen_* helpers and login_as_user from tests/helpers.py, set_config for settings and freezegun for time, with a docstring stating what each test checks. Run the new tests with uv run pytest, then the whole file, until they pass. Do not change application code: if a test fails because the code is wrong, leave it out and describe the failure. Push the tests to this pull request's branch and post a comment listing each test and the behavior it covers.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app/pull/123"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a commit on the pull request's branch that adds test functions to `tests/api/v1/test_challenges.py`, or to a file under `tests/challenges/` when the change is in a challenge type rather than the API. Each test builds its own CTFd app, generates the users and challenges it needs, makes the request through `app.test_client()` or `login_as_user`, and asserts the status code and the fields in `r.get_json()["data"]`. No application file changes.

The comment the agent posts names each test, the behavior it covers ("a team member sees a challenge as solved after a teammate solves it") and the pytest commands it ran. When a test found a real defect, the comment includes the request, the response it got and the line in `challenges.py` it points to. CTFd's workflows then run on the pushed commit against SQLite, and MySQL, MariaDB and Postgres where you run those, which shows whether the tests hold on each database.

Before merging, a reviewer checks that each test sets only the config it needs, so it does not depend on defaults another test changed, and that frozen times match the CTF start and end the test describes. They also check that tests for user mode and team mode are not mixed up.

## Run it on every pull request opened

In the Project bound to your Flask repository, create an automation with this prompt and a GitHub trigger on `pull_request.opened`. Set `branch_patterns` to the branches you merge into. Every pull request opened by a person with write access then gets its own run; the run's prompt carries the pull request's number, URL, title, author and base branch. Pull requests from forks, common on an open source project like this, never fire it.

To decide in your own code which pull requests get a run, listen to GitHub's `pull_request` webhook and send this request with the pull request's URL in `resources` only for the ones you choose, such as those that touch `CTFd/api/`. Give the API key a webhook URL and Rush tells your server when each run finishes.

## Why run it on Rush?

CTFd's suite is large, and `make test` runs it with parallel workers and coverage, which keeps a laptop busy while you write tests for someone else's branch. On Rush the agent gets a sandbox with its own clone at the pull request's branch, installs the locked dependencies with `uv`, and runs the challenges tests as often as it needs, while your checkout and your local CTF instance stay untouched.

Its claims come with their evidence. The trajectory keeps every pytest command and its output, so a reviewer can see which tests failed on the way and why one was left out. The repository's database matrix reruns the pushed tests on its own runners, and the tests reach review inside the pull request whose behavior they check.

## Frequently asked questions

### Does it need MySQL or Postgres to run the tests?

No. `TestingConfig` falls back to an in-memory SQLite database when `TESTING_DATABASE_URL` is unset, which is how the SQLite workflow runs. Your CI's MySQL, MariaDB and Postgres jobs still check the pushed tests against those databases.

### Will it use pytest fixtures instead of create_ctfd?

It writes tests the way the file it adds to already does. CTFd builds an app per test with `create_ctfd()` and `destroy_ctfd()`, so the new tests do too. In a Flask app whose tests use a `conftest.py` with an `app` and `client` fixture, the agent uses those.

### Can it test a plugin's challenge type?

Yes, when the pull request changes one. Challenge types register through CTFd's plugin system, and the existing tests under `tests/challenges/` exercise them through the same API, so the agent adds tests there and runs them the same way.

### What if the existing suite already fails on the branch?

The agent runs the file before and after adding its tests and reports any test that was failing before it started, by name. That keeps a broken base from being blamed on the new tests, and tells the author about it.