What the agent does
The agent begins with the window you give it, for example 09:40 to 10:15 UTC, and reads Datadog through the Datadog connection. It searches logs for the service in that window and groups them by status code, host and path, reads the request error-rate and latency metrics for the same span, and lists any monitors that triggered. Every query it runs is written down with its time range, so the report can show where each number came from.
Then it reads the code those errors point at. In a multi-app site like django/djangoproject.com, pyproject.toml pins Python to 3.12 and django to 6.1.1, and djangoproject/hosts.py splits traffic by hostname with django-hosts: www to the root URLconf, docs to djangoproject.urls.docs and dashboard to its own app. A spike of 500s on the docs host leads to docs/views.py, where the document view resolves the language and version, raises Http404 when the path is not ASCII or the release is missing, loads the page JSON and globalcontext.json from the built docs root, reads the last_build timestamp and sets a Surrogate-Control header for the CDN; outside DEBUG that view is wrapped in cache_page on the docs-pages cache. A spike on the blog leads to blog/views.py, five archive views over the Entry model whose published queryset filters on pub_date. A spike on search leads to the search_results and search_suggestions views and the Document queryset's search and rank methods in docs/models.py. The agent follows the host and path Datadog shows, not the app the ticket guesses.
It compares the window with what changed. It runs git log --since='7 days ago' -- docs/ blog/ djangoproject/ and reads each diff that touches the failing view, its template or its model. It also checks the management commands that rebuild the docs, update_docs, build_doc_release and purge_docs_cache under docs/management/commands, because a bad docs build or a cache purge that did not run looks like a code regression but is not one. The settings add context: djangoproject/settings/common.py logs django.request at ERROR, prod.py adds a syslog handler to it and initialises sentry-sdk with the Django and Redis integrations when a DSN is in the secrets file, so the agent knows which errors reach Datadog as logs and which are only in Sentry. It ties a commit to the incident only when its time and the symptom match, and says how confident it is.
It changes no code. To check a hypothesis it may run the tests for one app, make test APP_LIST=docs, which the Makefile turns into coverage run over manage test; that needs the PostgreSQL databases .github/workflows/tests.yml creates, djangoproject and code.djangoproject with the Trac dump loaded, and a conf/secrets.json beside the checkout, and the agent sets those up in the sandbox or says it could not. The report lists every command and query, and every claim carries the query or diff behind it.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Error rates for this Django site rose between 09:40 and 10:15 UTC today. Use the Datadog connection to read that window: error logs grouped by host, path and status, the request error-rate and latency metrics, and any monitors that triggered. Map the failing host through djangoproject/hosts.py to its URLconf and read the view, template and model the failing paths resolve to, then compare the window with the commits to docs/, blog/ and djangoproject/ in the last 7 days, reading the diff of each one that touches those files, and with the docs rebuild and cache purge commands. You may run make test APP_LIST=docs against a local PostgreSQL to check a hypothesis. Write incident.md with: a summary, a timeline in UTC, the most likely cause with the commit and file, the evidence for each claim (the Datadog query or command and what it returned), other causes you ruled out, and every query you ran. Change no code and change nothing in Datadog.",
"connections": [
"datadog"
],
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The result is one file, incident.md, attached to the session as an artifact with a download link. It opens with a two-line summary, then a timeline in UTC that interleaves deploys, commits, docs rebuilds, the first error, monitor triggers and recovery. The cause section names a commit, a file and a line range, such as a change to the document view in docs/views.py or to a template it renders, and says what in the logs matches it.
Each claim carries its evidence: the Datadog log or metric query, its time range and what it returned, or the git or make test command and its output. A section lists the causes the agent checked and ruled out, such as a stale docs build, a CDN cache that kept serving an old page, or a database connection problem, with the query that ruled each one out. The last section lists every query, so an on-call engineer can rerun any of them in Datadog.
Nothing changed in the repository or in Datadog, which the session's changed-files list and tool calls confirm. Before acting on the report, the engineer reruns the query behind the cause, checks the commit time against the deploy time, and decides whether to revert, fix forward or rebuild the docs.
Run it on every Datadog monitor alert
Create an automation in the Project with this prompt and a webhook trigger. Datadog cannot call that trigger directly, because the trigger needs an Authorization header with its own secret, so point the Datadog webhook at a small handler on your server that forwards the monitor payload to the trigger URL with that header. Each alert starts a run with the payload after the prompt as data, so reword the prompt to read the window and host from it. A redelivered alert forwarded with the same Idempotency-Key is deduplicated.
Set the automation's max runs per hour so a flapping monitor starts one investigation, not ten. Give the API key a webhook URL and Rush POSTs run.completed when the report is ready, with the session_url; your handler can post the incident.md link into the incident channel.
The use case this job follows, with its automation and trigger: Investigate an incident.
Why run it on Rush?
During an incident, an on-call engineer switches between Datadog, git log and the editor while the clock runs. An agent can do that first pass in its own sandbox, with its own clone and its own connection to Datadog, while the engineer works on mitigation. It runs tests and reads diffs there, so nothing on the engineer's machine or the production hosts changes.
The report is easy to check because every line cites its source: a Datadog query with its range, or a command with its output, all kept in the session's record. The prompt and the session's changed files show it altered no code. Several incidents at once get several runs, each in a sandbox of its own.
Frequently asked questions
Can the agent mute monitors or change dashboards?
The prompt forbids changing anything in Datadog, and the session records every tool call, so you can confirm it only read. The connection acts as the person who connected it, so it can never do more than that person could in Datadog.
Our errors go to Sentry, not Datadog logs. Does this still work?
The settings in this site send django.request errors to syslog and to Sentry when a DSN is configured, so Datadog sees what your log shipper forwards. The agent works from the request logs and metrics Datadog has, says which errors it could not see, and a Sentry connection can be added to the run for the rest.
What about the docs that are built outside the request path?
The agent reads the update_docs and build_doc_release commands and the last_build timestamp the document view uses, so a failed rebuild or a purge that never ran shows up in the timeline as an event rather than being mistaken for a code regression.
Will it run the site against production data?
No. It creates the two PostgreSQL databases the test workflow creates, loads the Trac dump from the repository, and runs the test suite there. It never connects to your production database unless you give it a connection that reaches it.