What the agent does
The agent reads the repository's own pins first. In CTFd/CTFd it opens requirements.txt and finds flask==2.1.3 among the compiled pins, with flask-restx building the API layer. The Makefile's test target runs pytest -rf --cov=CTFd --cov-context=test --cov-report=xml, and its lint step runs ruff check CTFd/ migrations/ tests/.
The API lives under CTFd/api/v1 with a module per resource: challenges.py, flags.py, submissions.py, scoreboard.py and the rest, backed by the models in CTFd/models and the plugin hooks under CTFd/plugins. The agent checks out the pull request branch, installs the requirements into its environment, and runs pytest with coverage as the Makefile does.
It reads the diff in the changed resource with the surrounding model and schema code. A challenges endpoint that filters on the wrong column, a missing 404 when the model lookup returns nothing, a marshmallow schema that no longer matches what the view returns: findings cite the module and line in CTFd/api/v1, and each is checked against the test files under tests/ that exercise the resource.
One GitHub review comes back per run. The summary quotes the pytest and coverage result, separates blocking issues from suggestions, and every finding carries file and line. The prompt forbids pushing commits, and none are pushed.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Review this pull request. Check it out, install dependencies with pip, run `pytest`, and read the diff with the surrounding code in the challenges API. Post one GitHub review that cites file and line for every finding, separates blocking issues from suggestions, and includes the test result. Do not push commits.",
"resources": [
{
"url": "https://github.com/your-org/your-app/pull/42"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a review on the pull request. The summary quotes the pytest result and the coverage figure, lists the findings with file and line in CTFd/api/v1, and marks which block merge and which are suggestions. Inline comments sit on the cited lines.
The session's trajectory keeps the pip install, the pytest invocation with coverage, and the model, schema and test files the agent opened while checking each finding. When a finding claims a schema mismatch, the trajectory shows the schema and the view's return side by side.
Before merging, a reviewer reproduces the blocking findings with pytest on the branch and opens each cited line. Suggestions can be taken or dismissed without holding the merge.
Run it on every pull request opened
Create an automation with a GitHub trigger on pull_request.opened and pull_request.synchronize, scoped to the branch your team merges to, and this prompt. Every pull request then gets its own run in its own sandbox, and the review appears without a developer installing the requirements and waiting on pytest.
Set a webhook URL on the API key and Rush POSTs run.completed when each run's turn finishes, with the session URL. Your server can post the review link to the channel that watches the repository. A key runs at most max_concurrent sessions at once, 4 by default, and answers 429 past it, so queue the starts when a milestone branch opens many pull requests together.
The use case this job follows, with its automation and trigger: Review every pull request.
Why run it on Rush?
A Flask review on a laptop means checking out the branch, building an environment that matches the compiled requirements and running the suite before any line can be commented on. On Rush the agent does that in its own sandbox and keeps every command and its output in the trajectory, so the review quotes an executed pytest run with coverage rather than asserting that one happened.
One sandbox per pull request keeps the reviews parallel when several resources change in one release: each pull request gets its own install, suite and review, and the maintainers keep their checkouts. The result is checkable, file and line for every finding, with the test result quoted at the top.
Frequently asked questions
Does the suite need Redis or a database server?
CTFd's own CI runs the pytest suite against service containers such as MySQL or Postgres plus Redis, with the database URL passed as an environment variable. Name the services your fork's tests expect in the prompt, or a Project secret for the connection string, and the agent mirrors that setup in the sandbox.
Can it review changes under CTFd/plugins?
Yes. Plugin code is part of the tree, and a diff in a plugin is read with the plugin's hooks and the API modules it attaches to, so a finding can cite the plugin file and the resource it changes behaviour in.
Will it flag the pinned Flask version as a problem?
No. The review is of the change, judged against the dependencies the repository pins. If the diff itself bumps the Flask pin, the agent checks the bump against the code the pin affects and reports what it finds.
What does it do when pytest fails on an unrelated test?
The review quotes the failing test separately from the findings, so a pre-existing failure does not read as a defect in the diff. A reviewer confirms quickly by running the same command on the base branch.