# Fix a Django App From a Grafana Alert

Yes. The alert starts a run that queries its metric and Loki logs through Grafana, finds the failing view and opens a pull request with a TestCase.

[Run this on Rush](https://cloud.getrush.ai/run/fix-grafana-alerts/django/), in a Project that has your repository.

## What the agent does

The run starts when a Grafana Cloud alert posts to a Rush webhook trigger. Its payload arrives after the prompt as untrusted data: the alert rule's name, its labels (often `job`, `route` or `view`), `startsAt` and the `generatorURL` back to the rule. The agent uses the Grafana connection to read the rule's query and threshold, runs that PromQL over the window around `startsAt`, and then queries Loki for error lines from the same service in the same minutes, so it sees a traceback rather than only a rising 5xx ratio.

In a Django site like `django/djangoproject.com`, it maps what the logs name to code. A failing `/en/<version>/search/` request lands in `search_results` in `docs/views.py`, which calls `DocumentRelease.objects.get_by_version_and_lang` and the ranked full-text search in `DocumentQuerySet.search` in `docs/models.py`, where `uses_trigram` changes which query runs. A failing blog permalink lands in `BlogDateDetailView` in `blog/views.py` and the `EntryQuerySet.published()` and `active()` filters in `blog/models.py`. The traceback's last frame inside the project tells the agent which of these to read first.

To reproduce it, the agent sets up the sandbox the way `.github/workflows/tests.yml` does: Python 3.12 (`requires-python = "==3.12.*"`), `requirements/tests.txt`, gettext, a PostgreSQL server with the `djangoproject` and `code.djangoproject` roles and databases, `tracdb/trac.sql` loaded, and a `conf/secrets.json` one directory above the checkout. It writes a `django.test.TestCase` in `docs/tests/test_views.py` or `blog/tests.py` that sends the request from the logs and fails on the same exception.

Then it fixes the view, queryset or model in the app the evidence points to, reruns the new test, runs `python manage.py test docs blog`, and then `make ci`, which also runs `compilemessages` and `collectstatics`, so the fix does not break the translations or static files the site builds.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "A Grafana alert fired for this Django service; its payload follows this prompt. Use the Grafana connection to read the alert rule, run its Prometheus query around the firing time, and query the Loki error logs for the same service and window. Find the cause in the docs and blog apps, write a django.test.TestCase that reproduces it, fix it, and run python manage.py test. Open a pull request that includes the PromQL and LogQL queries, the log lines behind the diagnosis, and the test output before and after the fix. If the evidence does not point at code, say what it points at instead.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request with three parts in its description: the PromQL the rule runs and the values around the firing time, the LogQL query and the traceback lines it returned, and the new `TestCase` failing before the fix and passing after it. The diff is the test and the fix in the one app the logs pointed at.

A reviewer reads the evidence first, then checks that the test exercises the same URL and exception the logs show, that the fix handles the input rather than catching the exception and returning a 200, and that `docs/models.py` search changes still use the PostgreSQL features the site depends on.

When the queries show something code cannot fix, such as a database that stopped answering or a deploy that was rolling at the time, the run says that with the numbers and lines it found and opens no pull request.

## Run it on every Grafana alert firing

Create an automation in the Project with this prompt and a `webhook` trigger; Rush shows its URL and secret once. In Grafana Cloud, add a webhook contact point with that URL and an `Authorization` header of `Bearer <secret>`, and route the alert rules for this service to it with a notification policy. Each notification, up to 64 KB, becomes the event text under the prompt.

Turn on the contact point's option to skip resolved notifications so a recovery does not start a run, and set the policy's repeat interval longer than a run takes, because each delivery without an `Idempotency-Key` header starts a new one.

## Why run it on Rush?

The first half hour of an alert is usually the same work: open the rule, rerun its query, search the logs for the matching traceback, then find the view and reproduce it locally against a database that looks like production. A Rush run does that in its own sandbox with each query, log search and test run kept in the session, so whoever looks next starts from evidence, not from a page.

Alerts fire at bad times and often in pairs. Each one gets its own run on its own branch at once, without anyone's laptop or production shell, and each ends with a pull request whose test proves the fix or a short account of what the metrics and logs show.

## Frequently asked questions

### How does the agent read my Grafana data?

Through the Grafana MCP server, which searches dashboards, queries Prometheus and Loki, and reads alerts in Grafana Cloud. Connect it once with `POST /me/connections` and `"auth": {"mode": "oauth"}`, finish the sign-in, then name it in `connections` on the run or the saved agent the automation uses.

### Does it need my production database?

No. It builds a PostgreSQL database in the sandbox the way the repository's test workflow does and reproduces the failure with a `TestCase` against it. Production is read only through Grafana's metrics and logs.

### What if the alert is noise?

Then the queries show it, for example a short spike with no matching errors in Loki, and the run says so without a pull request. That answer, with its queries, is still useful when deciding whether to tune the rule.

### Will it change my alert rules?

No. It reads rules and queries through the Grafana connection and changes only the repository, through a pull request a person reviews.