What the agent does
The agent first learns what the service reports. In a Rust service like rust-lang/crates.io, src/metrics/otel.rs exports OpenTelemetry metrics over OTLP when otel.metrics_enabled is set, and installs a no-op provider otherwise, so a dashboard with no data can mean the exporter was off rather than a quiet service. src/metrics/consts.rs names the instruments: http.server.active_requests, db.client.connection.count, crates_io.db.client.connection.fallbacks, crates_io.background_jobs, crates_io.background_jobs.oldest_age and crates_io.crates_total. Through the Grafana connection the agent queries those series for your window, allowing for the name changes Grafana Cloud makes when it stores OTLP metrics as Prometheus series, and reads the matching logs in Loki.
Then it maps symptoms to the two controllers in question. Downloads go through download_version in src/controllers/version/downloads.rs, which serves GET /api/v1/crates/{name}/{version}/download with a 302 to the storage location, or JSON {"url": ...} when the client asks for JSON. Publishing goes through src/controllers/krate/publish.rs, 1,124 lines that check the publisher's rate limit and max_upload_size, write the version, then enqueue RenderAndUploadReadme, SyncToGitIndex, SyncToSparseIndex and the RSS feed jobs. So a publish incident can show up as a growing crates_io.background_jobs.oldest_age long after the request itself returned 200, and the agent reads the job queue metrics alongside request errors.
It compares the window with the history: git log --since='7 days ago' -- src/controllers/krate/publish.rs src/controllers/version/ src/worker/ and the diffs of each commit that touches the failing path. A rise in crates_io.db.client.connection.fallbacks next to a publish-path change, or download 5xx after a change to how the storage key is built, are the kinds of pairings it looks for, and it says how strong each one is.
It changes no code. To check an idea it installs the toolchain pinned in rust-toolchain.toml (1.99.1), creates the test database the way the repository's AGENTS.md describes, with createdb --lc-collate=C --lc-ctype=C -T template0 cargo_registry_test and TEST_DATABASE_URL, and runs one test module, such as those under src/tests/krate/publish/ (rate_limit.rs, max_size.rs, tarball.rs). The command and its output go into the report.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Error rates for our crates.io-based Rust service rose between 03:40 and 04:25 UTC today. Use the Grafana connection to read that window: request errors and latency for the publish and download routes, http.server.active_requests, db.client.connection.count, crates_io.db.client.connection.fallbacks, crates_io.background_jobs and crates_io.background_jobs.oldest_age (as stored in Prometheus), plus the service's error lines in Loki and any alerts that fired. Compare the window with the commits to src/controllers/krate/publish.rs, src/controllers/version/ and src/worker/ in the last 7 days. You may run a single test module with `cargo test` against a local Postgres. Write incident.md with a summary, a UTC timeline, the most likely cause with commit and file, the evidence for each claim (query or command and result), causes ruled out, and every query you ran. Change no code and nothing in Grafana.",
"connections": [
"grafana"
],
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The session returns incident.md as an artifact. Its summary separates the request path from the background path: whether publishes and downloads failed at request time, or succeeded and then left jobs queued. The timeline in UTC lists the deploy, the first failing request or the start of the job backlog, connection pool fallbacks, alerts and recovery.
The cause names a commit and a file, for example in publish.rs or a job under src/worker/, and every claim cites its query or command: the PromQL with its range and result, the LogQL behind any quoted log line, or the cargo test run with its output. Ruled-out causes come with their own evidence, such as db.client.connection.count staying below the pool size, which argues against pool exhaustion.
No file in the repository changed and Grafana received only reads, as the session's changed files and tool calls show. The engineer on call reruns the main query, checks the commit against the deployed revision and decides whether to roll back, pause a background job type, or fix forward.
Run it on every background job backlog alert
Add an automation to the Project with this prompt and a webhook trigger. In Grafana, create a webhook contact point that posts to the trigger's URL with its secret as the bearer token, and route the alert on crates_io.background_jobs.oldest_age and the API error-rate alert to it. Each firing starts one run with the alert's labels after the prompt as data, so the agent knows which job type or route alerted.
A backlog alert stays firing until the queue drains, and Grafana repeats the notification. Keep the automation's max runs per hour low so repeats are skipped, and give the API key a webhook URL so Rush POSTs run.completed with the session_url when the report exists.
The use case this job follows, with its automation and trigger: Investigate an incident.
Why run it on Rush?
Building crates.io for a quick check means a Rust 1.99.1 toolchain, a Postgres database with C collation and a long first compile, which is a lot to set up on a laptop mid-incident. The agent does it in its own Rush sandbox, with its own clone and target directory, while the people on call look after the service.
Its report is checkable line by line: each claim has the PromQL, LogQL or command that supports it, kept with its output on the session. The run is read-only by instruction and by record, and two alerts on two services become two runs in separate sandboxes.
Frequently asked questions
Our metric names in Grafana use underscores, not dots. Will the agent find them?
Yes. It reads the instrument names from src/metrics/consts.rs and then looks up how they are stored in your Prometheus data source, where OTLP names are usually rewritten with underscores and unit suffixes. It lists the exact series names it queried in the report.
Does it run the full test suite?
No, only the module that tests the suspected path, such as the publish tests under src/tests/krate/publish/. The full suite is large and needs a database per test; one module is enough to confirm or reject a hypothesis.
What if OpenTelemetry metrics were disabled during the incident?
Then otel.rs used a no-op provider and there are no metrics for that window. The report says so, explains the empty panels, and works from Loki logs and request data instead.
Can it look at the frontend too?
The same repository has a SvelteKit frontend under svelte/ with Playwright tests in e2e/. If the errors are in the browser rather than the API, the agent says so and lists what it would need, but this prompt keeps it on the backend.