What the agent does
The monitor's webhook carries little more than a title, a link and tags, so the agent starts by asking Datadog for the rest. Through the Datadog connection it reads the monitor's query, then searches error logs and APM traces for the service in the 30 minutes around the alert, grouping them by resource name and error type. In a Django service like healthchecks/healthchecks, the traces point at one of two places: the ping endpoints ping and ping_by_slug in hc/api/views.py, which every cron job and script calls, or the notification path that runs in the separate sendalerts management command.
It reads the code those traces name, with the repository cloned in its own sandbox. On the ping side it follows how ping turns the request into an action: an exit status above 0 becomes fail, a check whose methods is POST ignores a GET, and failure_kw, success_kw and start_kw are matched against the body when filter_http_body is set, before check.ping records it. On the alert side it follows Channel.notify in hc/api/models.py, which saves a Notification, calls the transport's notify and records the TransportError message, and the integration under hc/integrations/, such as webhook/transport.py, whose prepare fills in $CODE, $STATUS and the other placeholders.
With a suspect line found, it writes a Django TestCase that fails the way production did, next to the tests already there: hc/api/tests/test_ping.py for the ping view, or the integration's own tests/test_notify.py. It runs that module, makes the fix, then runs python manage.py test for the whole suite on SQLite, the default database. The repository's GitHub workflow runs the same command against SQLite, Postgres, MySQL and MariaDB, so the agent avoids SQL that only one of them accepts.
Last, it opens a pull request. The description quotes the log lines and the trace's resource name that the fix addresses, links the monitor, explains the cause in two or three sentences and lists the test that now covers it. If the logs point somewhere the agent cannot reproduce, it says so and opens no pull request.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "codex",
"input": "A Datadog monitor on this Django service fired; its webhook payload follows this prompt. Use the Datadog connection to read the monitor's query and the error logs and APM traces from the 30 minutes around the alert. Find the code path that produced them, starting from the ping and ping_by_slug views in hc/api/views.py and Channel.notify in hc/api/models.py with the integration transports under hc/integrations/. Reproduce it with a Django TestCase next to the existing tests, fix it, run python manage.py test, and open a pull request that quotes the log lines the fix addresses and links the monitor. If you cannot reproduce it, open no pull request and report what you checked.\n\nMonitor payload:\n{payload}",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
],
"connections": [
"datadog"
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request on your repository. Its diff has the fix, usually a few lines in a view, a model method or one integration's transport.py, and a new test in the matching tests/ module. Its description quotes the Datadog log lines and names the trace resource, so a reviewer can match the fix to what production reported without opening Datadog. Your own CI runs on it as on any pull request.
The session's result, also in data.result on the run.completed webhook, is the agent's summary: the monitor, the cause, the files changed and the test commands it ran. The transcript keeps every Datadog query with its arguments and the output of each manage.py test run, so the claim that the suite passed is something you can read rather than take on trust.
Before merging, a reviewer checks three things: that the new test fails without the fix, that the change does not swallow the error the monitor was built to catch, and, for a fix in the alert path, that a failing transport still records its error on the channel instead of hiding it.
Run it on every Datadog monitor alert
In the Project that holds this repository, add an automation with a webhook trigger and this prompt, and name the Datadog connection on the saved agent it runs. The trigger has its own URL and secret. In Datadog's Webhooks integration, point a webhook at that URL with the secret as a bearer token, and add @webhook-<name> to the monitor's message. The request body becomes the event text after the prompt, and max_runs_per_hour caps how many runs a flapping monitor can start.
If you would rather keep the logic on your own server, receive Datadog's webhook there and call POST /o/{org}/sessions with the body above, the payload appended to input. Use the alert id as the Idempotency-Key so a retried webhook starts one run, and set a webhook on the key to hear when the pull request is open.
The use case this job follows, with its automation and trigger: Fix a bug from an alert.
Why run it on Rush?
An alert at night usually ends with someone reading logs, guessing at a cause and leaving the fix for the morning. A Rush run does the reading and the guessing check in its own sandbox: it queries Datadog with your connection, clones the repository, runs the Django test suite for real and keeps the output. The person on call wakes up to a pull request with evidence instead of a page.
Monitors rarely fire one at a time. Each alert gets its own run and its own sandbox, so two monitors on the ping API and a webhook integration are investigated in parallel without stepping on each other's database or branch. Every run's record shows which logs it read and which tests it ran, which is what a reviewer needs to merge a fix that nobody watched being written.
Frequently asked questions
Which Datadog site does the connection reach?
The Datadog connection in the catalog points at the US1 site. If your account is on another site, add that site's MCP server as a custom connection and name it in connections instead of datadog.
Will it run the tests against Postgres?
It runs python manage.py test with the default SQLite settings unless your prompt says otherwise. Your own CI then runs the pull request against whichever databases your workflow covers, as the healthchecks workflow does with four.
Can it change monitors or mute alerts?
The prompt asks it to read the monitor, logs and traces, and the transcript records every Datadog call it made. If you do not want the agent able to write to Datadog at all, connect it with an account or key that has read-only access.
What if the alert is noise rather than a bug?
Then it opens no pull request and says why in its result, for example that the errors came from one client sending malformed pings. That answer is worth having too: it tells you the monitor's query needs work, not the code.