What the agent does
The agent reads Grafana before it reads code. Through your Grafana connection it searches dashboards for the services in the window and queries Prometheus for the metrics those dashboards plot. In a deployment of open-telemetry/opentelemetry-demo the provisioned Demo Dashboard (src/grafana/provisioning/dashboards/demo/demo-dashboard.json) is built on span metrics from the collector: request rate from traces_span_metrics_duration_milliseconds_count by span_name, error rate from traces_span_metrics_calls_total{status_code="STATUS_CODE_ERROR"}, and latency from histogram_quantile over traces_span_metrics_duration_milliseconds_bucket, all filtered by service_name. The agent runs those queries minute by minute across the window for every service, so the first series to turn red names the first service to fail, and the span names say which operation.
It then follows the dependency. The demo's services (frontend, checkout, cart, payment, product-catalog, currency, shipping, recommendation, ad, email, quote, with accounting, fraud-detection and Kafka in compose.full.yaml) call each other through gRPC and HTTP, and the dashboard's Service Dependency node graph and the APM dashboard (Jaeger, Prometheus, OpenSearch) show the edges. A checkout error rate that rises one minute after payment errors is a downstream symptom, not a cause, and the report says so. Where the connection reaches Loki, the agent queries log lines for the failing service in the same window; the demo as checked in sends logs to OpenSearch (webstore-logs, index otel-logs-*), which the Grafana connection does not query, so for that deployment the report marks log evidence as not reachable rather than inventing it.
The repository explains the mechanism. The demo's outages are feature flags in src/flagd/demo.flagd.json, each off by default: paymentFailure (fail a percentage of charge requests), paymentUnreachable, cartFailure, productCatalogFailure (one product), productCatalogLockContention, recommendationCacheFailure, adHighCpu, adManualGc, emailMemoryLeak, kafkaQueueProblems, kafkaConsumerDead, imageSlowLoad, loadGeneratorFloodHomepage and failedReadinessProbe. The agent reads the flag whose description matches the failing service, finds where the service evaluates it, and matches the error text in the span status to the code path, so the timeline ends with a cause a person can verify by reading the flag's state in flagd-ui.
It writes incident.md with a summary, a UTC timeline at one-minute resolution, the first failing service and operation, the propagation to callers, the most likely mechanism with the file that implements it, and an appendix with every PromQL query and the numbers it returned. It changes nothing: no code, no dashboards, no flag values.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "An outage happened in the window below. Through the Grafana connection, find the dashboards for these services and query Prometheus for their span metrics in that window: request rate, error rate and p50 and p95 duration per service and span name, at one-minute steps. Where the connection reaches Loki, query the failing service's logs for the same window; if the logs are not reachable, say so. Build a minute-by-minute timeline, name the first service to fail and the operation, trace how the failure spread to callers, and use the repository to explain the mechanism, including any feature flag that produces it. Write incident.md with a summary, the timeline, the evidence for each claim, and every query with its result. Change nothing.\n\nWindow: 14:00 to 14:30 UTC today.",
"connections": [
"grafana"
],
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is incident.md as the run's artifact, with a url on the session that downloads it for an hour and a fresh one each time the session is read. Its sections are the summary, the timeline, the first failure, the propagation, the mechanism, what was ruled out, and the query appendix. The result on the session repeats the one-paragraph summary, and the transcript holds each Grafana call and what it returned.
Before acting on it, a reviewer checks that the first failure is supported by the error-rate series and not by a latency bump alone, that the propagation order matches the service dependency graph, that any feature flag named in the mechanism is one the code of the failing service actually reads, and that every number in the timeline appears in the appendix. A claim with no query behind it is the thing to question first.
Nothing in Grafana or the repository changes. The report is input for the on-call engineer and the postmortem, written while the metrics are still in Prometheus's retention window.
Run it on every Grafana alert firing
Create an automation in the Project with this prompt and a webhook trigger. Grafana's webhook contact point cannot send the trigger's bearer token, so point the contact point at a small handler on your server that verifies Grafana's request and forwards the alert JSON to the trigger URL with Authorization: Bearer <its secret>. Each firing starts one run with the alert's labels and values after the prompt, so the window and the service come from the alert.
An alert that stays firing resends, so set the automation's max_runs_per_hour low; extra fires are skipped rather than starting duplicate investigations. Give the API key a webhook URL to receive run.completed with the session_url when incident.md is ready.
The use case this job follows, with its automation and trigger: Investigate an incident.
Why run it on Rush?
During an outage the person on call has the dashboards open and no time to turn them into a timeline. On Rush the agent runs the same PromQL the dashboards use, service by service and minute by minute, in a sandbox that also has the repository, and hands back a document with the queries attached. The engineer reads conclusions with evidence instead of hunting for the first red panel.
The record matters afterwards too. Every query and result is in the transcript and the appendix, so the postmortem can cite them, and a second run over a wider window or another service is one more request. Nothing is run against production except read-only Grafana queries through a connection you granted.
Frequently asked questions
What does the Grafana connection let the agent do?
Search dashboards, query Prometheus and Loki, and read alerts and incidents, through Grafana's own MCP server, signed in with your Grafana account. It does not edit dashboards or silence alerts, and the prompt tells it to change nothing.
The demo stores logs in OpenSearch, not Loki. Does the agent still work?
Yes, on metrics and traces-derived span metrics, which are enough to name the first failing service and operation. The report states plainly that log lines were not reachable through the connection. A deployment that ships logs to Loki gets log evidence in the same report.
Does it need the repository?
Not to build the timeline. It needs it to explain the mechanism: which flag or code path produces the error the spans record. Leave resources out and the report stops at what the metrics show.
Can it turn a feature flag off to confirm the cause?
No. The prompt forbids changes and the connection is to Grafana, not to flagd. The report names the flag and where it is evaluated so a person can check and change it.
How far back can it look?
As far as your Prometheus keeps data; the demo's Prometheus is configured with seven days of retention and a 60-second scrape interval, so one-minute steps are the finest resolution that means anything there.