# Investigate a Rust Service Incident With Grafana

Yes. An agent reads your Rust service's OpenTelemetry metrics and logs in Grafana, ties the incident to recent commits and writes incident.md.

[Run this on Rush](https://cloud.getrush.ai/run/investigate-incidents-with-grafana/rust/), in a Project that has your repository.

## What the agent does

The agent first learns what the service reports. In a Rust service like rust-lang/crates.io, `src/metrics/otel.rs` exports OpenTelemetry metrics over OTLP when `otel.metrics_enabled` is set, and installs a no-op provider otherwise, so a dashboard with no data can mean the exporter was off rather than a quiet service. `src/metrics/consts.rs` names the instruments: `http.server.active_requests`, `db.client.connection.count`, `crates_io.db.client.connection.fallbacks`, `crates_io.background_jobs`, `crates_io.background_jobs.oldest_age` and `crates_io.crates_total`. Through the Grafana connection the agent queries those series for your window, allowing for the name changes Grafana Cloud makes when it stores OTLP metrics as Prometheus series, and reads the matching logs in Loki.

Then it maps symptoms to the two controllers in question. Downloads go through `download_version` in `src/controllers/version/downloads.rs`, which serves `GET /api/v1/crates/{name}/{version}/download` with a 302 to the storage location, or JSON `{"url": ...}` when the client asks for JSON. Publishing goes through `src/controllers/krate/publish.rs`, 1,124 lines that check the publisher's rate limit and `max_upload_size`, write the version, then enqueue `RenderAndUploadReadme`, `SyncToGitIndex`, `SyncToSparseIndex` and the RSS feed jobs. So a publish incident can show up as a growing `crates_io.background_jobs.oldest_age` long after the request itself returned 200, and the agent reads the job queue metrics alongside request errors.

It compares the window with the history: `git log --since='7 days ago' -- src/controllers/krate/publish.rs src/controllers/version/ src/worker/` and the diffs of each commit that touches the failing path. A rise in `crates_io.db.client.connection.fallbacks` next to a publish-path change, or download 5xx after a change to how the storage key is built, are the kinds of pairings it looks for, and it says how strong each one is.

It changes no code. To check an idea it installs the toolchain pinned in `rust-toolchain.toml` (1.99.1), creates the test database the way the repository's `AGENTS.md` describes, with `createdb --lc-collate=C --lc-ctype=C -T template0 cargo_registry_test` and `TEST_DATABASE_URL`, and runs one test module, such as those under `src/tests/krate/publish/` (`rate_limit.rs`, `max_size.rs`, `tarball.rs`). The command and its output go into the report.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Error rates for our crates.io-based Rust service rose between 03:40 and 04:25 UTC today. Use the Grafana connection to read that window: request errors and latency for the publish and download routes, http.server.active_requests, db.client.connection.count, crates_io.db.client.connection.fallbacks, crates_io.background_jobs and crates_io.background_jobs.oldest_age (as stored in Prometheus), plus the service's error lines in Loki and any alerts that fired. Compare the window with the commits to src/controllers/krate/publish.rs, src/controllers/version/ and src/worker/ in the last 7 days. You may run a single test module with `cargo test` against a local Postgres. Write incident.md with a summary, a UTC timeline, the most likely cause with commit and file, the evidence for each claim (query or command and result), causes ruled out, and every query you ran. Change no code and nothing in Grafana.",
  "connections": [
    "grafana"
  ],
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

The session returns `incident.md` as an artifact. Its summary separates the request path from the background path: whether publishes and downloads failed at request time, or succeeded and then left jobs queued. The timeline in UTC lists the deploy, the first failing request or the start of the job backlog, connection pool fallbacks, alerts and recovery.

The cause names a commit and a file, for example in `publish.rs` or a job under `src/worker/`, and every claim cites its query or command: the PromQL with its range and result, the LogQL behind any quoted log line, or the `cargo test` run with its output. Ruled-out causes come with their own evidence, such as `db.client.connection.count` staying below the pool size, which argues against pool exhaustion.

No file in the repository changed and Grafana received only reads, as the session's changed files and tool calls show. The engineer on call reruns the main query, checks the commit against the deployed revision and decides whether to roll back, pause a background job type, or fix forward.

## Run it on every background job backlog alert

Add an automation to the Project with this prompt and a `webhook` trigger. In Grafana, create a webhook contact point that posts to the trigger's URL with its secret as the bearer token, and route the alert on `crates_io.background_jobs.oldest_age` and the API error-rate alert to it. Each firing starts one run with the alert's labels after the prompt as data, so the agent knows which job type or route alerted.

A backlog alert stays firing until the queue drains, and Grafana repeats the notification. Keep the automation's max runs per hour low so repeats are skipped, and give the API key a webhook URL so Rush POSTs `run.completed` with the `session_url` when the report exists.

## Why run it on Rush?

Building crates.io for a quick check means a Rust 1.99.1 toolchain, a Postgres database with C collation and a long first compile, which is a lot to set up on a laptop mid-incident. The agent does it in its own Rush sandbox, with its own clone and target directory, while the people on call look after the service.

Its report is checkable line by line: each claim has the PromQL, LogQL or command that supports it, kept with its output on the session. The run is read-only by instruction and by record, and two alerts on two services become two runs in separate sandboxes.

## Frequently asked questions

### Our metric names in Grafana use underscores, not dots. Will the agent find them?

Yes. It reads the instrument names from `src/metrics/consts.rs` and then looks up how they are stored in your Prometheus data source, where OTLP names are usually rewritten with underscores and unit suffixes. It lists the exact series names it queried in the report.

### Does it run the full test suite?

No, only the module that tests the suspected path, such as the publish tests under `src/tests/krate/publish/`. The full suite is large and needs a database per test; one module is enough to confirm or reject a hypothesis.

### What if OpenTelemetry metrics were disabled during the incident?

Then `otel.rs` used a no-op provider and there are no metrics for that window. The report says so, explains the empty panels, and works from Loki logs and request data instead.

### Can it look at the frontend too?

The same repository has a SvelteKit frontend under `svelte/` with Playwright tests in `e2e/`. If the errors are in the browser rather than the API, the agent says so and lists what it would need, but this prompt keeps it on the backend.