# Fix a Django App Bug From a Datadog Monitor Alert

Yes. A Datadog monitor starts a run, the agent reads the logs and traces behind it, fixes the Django code with a TestCase and opens a pull request quoting the logs.

[Run this on Rush](https://cloud.getrush.ai/run/fix-datadog-alerts/django/), in a Project that has your repository.

## What the agent does

The monitor's webhook carries little more than a title, a link and tags, so the agent starts by asking Datadog for the rest. Through the Datadog connection it reads the monitor's query, then searches error logs and APM traces for the service in the 30 minutes around the alert, grouping them by resource name and error type. In a Django service like `healthchecks/healthchecks`, the traces point at one of two places: the ping endpoints `ping` and `ping_by_slug` in `hc/api/views.py`, which every cron job and script calls, or the notification path that runs in the separate `sendalerts` management command.

It reads the code those traces name, with the repository cloned in its own sandbox. On the ping side it follows how `ping` turns the request into an action: an exit status above 0 becomes `fail`, a check whose `methods` is `POST` ignores a GET, and `failure_kw`, `success_kw` and `start_kw` are matched against the body when `filter_http_body` is set, before `check.ping` records it. On the alert side it follows `Channel.notify` in `hc/api/models.py`, which saves a `Notification`, calls the transport's `notify` and records the `TransportError` message, and the integration under `hc/integrations/`, such as `webhook/transport.py`, whose `prepare` fills in `$CODE`, `$STATUS` and the other placeholders.

With a suspect line found, it writes a Django `TestCase` that fails the way production did, next to the tests already there: `hc/api/tests/test_ping.py` for the ping view, or the integration's own `tests/test_notify.py`. It runs that module, makes the fix, then runs `python manage.py test` for the whole suite on SQLite, the default database. The repository's GitHub workflow runs the same command against SQLite, Postgres, MySQL and MariaDB, so the agent avoids SQL that only one of them accepts.

Last, it opens a pull request. The description quotes the log lines and the trace's resource name that the fix addresses, links the monitor, explains the cause in two or three sentences and lists the test that now covers it. If the logs point somewhere the agent cannot reproduce, it says so and opens no pull request.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "codex",
  "input": "A Datadog monitor on this Django service fired; its webhook payload follows this prompt. Use the Datadog connection to read the monitor's query and the error logs and APM traces from the 30 minutes around the alert. Find the code path that produced them, starting from the ping and ping_by_slug views in hc/api/views.py and Channel.notify in hc/api/models.py with the integration transports under hc/integrations/. Reproduce it with a Django TestCase next to the existing tests, fix it, run python manage.py test, and open a pull request that quotes the log lines the fix addresses and links the monitor. If you cannot reproduce it, open no pull request and report what you checked.\n\nMonitor payload:\n{payload}",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ],
  "connections": [
    "datadog"
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request on your repository. Its diff has the fix, usually a few lines in a view, a model method or one integration's `transport.py`, and a new test in the matching `tests/` module. Its description quotes the Datadog log lines and names the trace resource, so a reviewer can match the fix to what production reported without opening Datadog. Your own CI runs on it as on any pull request.

The session's `result`, also in `data.result` on the `run.completed` webhook, is the agent's summary: the monitor, the cause, the files changed and the test commands it ran. The transcript keeps every Datadog query with its arguments and the output of each `manage.py test` run, so the claim that the suite passed is something you can read rather than take on trust.

Before merging, a reviewer checks three things: that the new test fails without the fix, that the change does not swallow the error the monitor was built to catch, and, for a fix in the alert path, that a failing transport still records its error on the channel instead of hiding it.

## Run it on every Datadog monitor alert

In the Project that holds this repository, add an automation with a webhook trigger and this prompt, and name the Datadog connection on the saved agent it runs. The trigger has its own URL and secret. In Datadog's Webhooks integration, point a webhook at that URL with the secret as a bearer token, and add `@webhook-<name>` to the monitor's message. The request body becomes the event text after the prompt, and `max_runs_per_hour` caps how many runs a flapping monitor can start.

If you would rather keep the logic on your own server, receive Datadog's webhook there and call `POST /o/{org}/sessions` with the body above, the payload appended to `input`. Use the alert id as the `Idempotency-Key` so a retried webhook starts one run, and set a webhook on the key to hear when the pull request is open.

## Why run it on Rush?

An alert at night usually ends with someone reading logs, guessing at a cause and leaving the fix for the morning. A Rush run does the reading and the guessing check in its own sandbox: it queries Datadog with your connection, clones the repository, runs the Django test suite for real and keeps the output. The person on call wakes up to a pull request with evidence instead of a page.

Monitors rarely fire one at a time. Each alert gets its own run and its own sandbox, so two monitors on the ping API and a webhook integration are investigated in parallel without stepping on each other's database or branch. Every run's record shows which logs it read and which tests it ran, which is what a reviewer needs to merge a fix that nobody watched being written.

## Frequently asked questions

### Which Datadog site does the connection reach?

The Datadog connection in the catalog points at the US1 site. If your account is on another site, add that site's MCP server as a custom connection and name it in `connections` instead of `datadog`.

### Will it run the tests against Postgres?

It runs `python manage.py test` with the default SQLite settings unless your prompt says otherwise. Your own CI then runs the pull request against whichever databases your workflow covers, as the healthchecks workflow does with four.

### Can it change monitors or mute alerts?

The prompt asks it to read the monitor, logs and traces, and the transcript records every Datadog call it made. If you do not want the agent able to write to Datadog at all, connect it with an account or key that has read-only access.

### What if the alert is noise rather than a bug?

Then it opens no pull request and says why in its result, for example that the errors came from one client sending malformed pings. That answer is worth having too: it tells you the monitor's query needs work, not the code.