Turn Sentry Errors Into Fix Pull Requests for a Flask App

The run starts when a Sentry alert rule posts to your handler and the handler sends this request with the alert payload after the prompt. The agent reads the payload for the issue id, then uses the Sentry connection to fetch the issue and its latest event: the exception type and message, the culprit, the release and environment tags, and every in-app frame with its filename, function and line. For a Flask app the frames name files under the package, such as CTFd/api/v1/challenges.py, so the agent knows where to look before it opens the repository.

In a Flask app like CTFd/CTFd it then reads the code around those frames in its own clone. pyproject.toml pins Flask 2.1.3, Werkzeug 2.2.3, flask-restx 1.3.0 and SQLAlchemy 1.4.54 and requires Python 3.11; uv.lock is the source of truth and requirements.txt is exported from it, so the agent installs with uv. The challenges API lives in CTFd/api/v1/challenges.py as flask-restx resources: ChallengeList, Challenge, ChallengeAttempt at /api/v1/challenges/attempt, plus solves, files, hints, flags and ratings. ChallengeAttempt.post reads challenge_id from JSON or form data, loads the challenge with first_or_404, checks the hidden and locked states, checks prerequisites against the user's solves, and applies the anti-bruteforce window from the incorrect_submissions_per_min and max_attempts_timeout settings before it hands the request to the class get_chal_class returns. A KeyError there, when a plugin's challenge type is not installed, is what Challenge.get turns into a 500 whose message names the type.

Before changing anything the agent writes a test that reproduces the exception, in tests/api/v1/test_challenges.py next to test_api_challenge_attempt_post_private and the other attempt tests. The helpers in tests/helpers.py build the fixture: create_ctfd() for an app with a fresh database, gen_challenge and gen_flag for the data, register_user and login_as_user for a client, destroy_ctfd at the end. The test sends the request the Sentry event's request data shows and asserts the response the code should have given. It runs that one test with pytest -k and keeps the failure output; the suite uses pytest 8.4.2 with pytest-xdist, pytest-randomly and freezegun from the dev dependency group.

Then it fixes the cause, in the API module or in CTFd/plugins/challenges/__init__.py, whose BaseChallenge.attempt, solve and fail class methods are what a custom challenge type overrides, and runs the test again until it passes. It runs make test, the target the sqlite.yml workflow runs with TESTING_DATABASE_URL set to sqlite://, and the Python half of make lint: ruff check CTFd/ migrations/ tests/ and ruff format --check. The MySQL, MariaDB and Postgres workflows run the same suite against a database service; the agent says in the pull request which of those it could run in the sandbox.

What comes back is a pull request on its own branch with two kinds of change: the fix in CTFd/api/v1/challenges.py or the plugin module, and a new test function in tests/api/v1/test_challenges.py that fails on the old code. The description links the Sentry issue, quotes the exception line and the in-app frames it fixed, shows the test failing before and passing after, and lists the commands it ran. Your repository's own workflows run on the branch as they do on any pull request.

The session keeps the record: each Sentry call the agent made, every pytest and ruff command with its full output, and every file it read. GET /o/{org}/sessions/{id}/messages returns that transcript, and the run.completed webhook carries the agent's final summary as data.result, so your handler can post the pull request link back to the Sentry issue or to a channel.

Before merging, a reviewer checks three things: that the test fails for the reason Sentry reported and not a nearby one, that the fix does not widen a try block or swallow the exception, and that a change to the attempt flow keeps the state, prerequisite and rate-limit checks in the same order. If the right fix is different, a follow-up message to the same session asks for it and the agent continues on the same branch.

In Sentry, give the project an alert rule whose action is a webhook and point it at a small handler on your server. The handler sends this request with the alert's JSON after the prompt and the Sentry issue id as the Idempotency-Key header, so a resent alert returns the run it already started instead of a second one. One alert becomes one run and one pull request.

Set a webhook URL on the API key and Rush POSTs run.completed to it when the turn finishes, with session_url and the result. Or have that handler call a webhook trigger on an automation in the Project with this prompt: it forwards the alert's JSON as the body with the trigger's secret as Authorization: Bearer, because Sentry signs its own webhooks and cannot send that header. The body becomes the event text after the prompt, and max_runs_per_hour caps an issue that keeps firing.

Fixing a Sentry issue by hand means reading the stack trace in one tab, finding the commit in another, setting up a database to reproduce it, and then convincing a reviewer that the test you wrote is the same failure. In a Rush sandbox the agent does that in one place: it reads the issue through your connection, installs with uv, runs the suite against SQLite the way CI does, and nothing on anyone's laptop changes.

The pull request carries its proof with it: the failing test, the passing test and the transcript of every command. Several issues can be worked at the same time in separate sandboxes, up to the key's max_concurrent runs, so the issues that pile up after a release each get a proposed fix instead of waiting in a queue for one person.

How does the agent reach my Sentry project?

Does it need MySQL to run the CTFd tests?

What if the error is in a challenge plugin?

What happens when it cannot reproduce the exception?