# Investigate a Go Service Incident With Grafana

Yes. An agent reads your Go service's Prometheus metrics and logs in Grafana, matches the incident to recent commits and writes incident.md.

[Run this on Rush](https://cloud.getrush.ai/run/investigate-incidents-with-grafana/go/), in a Project that has your repository.

## What the agent does

The agent starts from the metrics your Go service already exports. In a server like navidrome/navidrome, `core/metrics/prometheus.go` registers them when `prometheus.enabled` is true (it defaults to false) and serves them at `/metrics`: `http_request_count` labelled by `endpoint`, `method`, `client` and `status`, `http_request_latency` by `endpoint`, `method` and `client`, `media_scans` and `media_scan_last` by `success`, `db_model_totals` by `model`, and the plugin request counters. Through the Grafana connection the agent queries those series in Prometheus for your window and reads the matching log lines in Loki.

Those labels decide where it looks. A rise in `http_request_count` with a 5xx `status` on one Subsonic `endpoint`, broken down by `client`, shows whether every app failed or only one. A `media_scans{success="false"}` increase points at the scanner instead. In Navidrome the scanner runs in phases, in `scanner/phase_1_folders.go`, `phase_2_missing_tracks.go`, `phase_3_refresh_albums.go` and `phase_4_playlists.go`, with `controller.go` exposing `ScanAll` and `server/subsonic/api.go` routing `getScanStatus` and the admin-only `startScan` to it, so a scan failure and a Subsonic API error lead to different files.

It checks the window against recent changes with `git log --since='7 days ago' -- scanner/ server/subsonic/` and reads the diffs that touch the failing endpoint's handler, a scan phase, or `server/subsonic/auth_limiter.go`, which caps failed Subsonic logins per key. A spike of failed logins from one `client` right after a change to the limiter or the auth middleware in `middlewares.go` is a different story from timeouts in `browsing.go`, and the report says which one the data supports.

It changes no code. To test a theory it runs the Ginkgo suites the way the Makefile does, `go test -tags netgo,sqlite_fts5 ./scanner/` or `./server/subsonic/`, with the Go version from `go.mod`. The command and its output go into the report next to the claim they support.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "codex",
  "input": "Error rates for our Navidrome-based Go service rose between 20:15 and 21:00 UTC today. Use the Grafana connection to read that window: in Prometheus, http_request_count by endpoint, client and status, http_request_latency by endpoint, and media_scans and media_scan_last by success; in Loki, the service's error and warning lines. List any alerts that fired. Compare the window with the commits to scanner/ and server/subsonic/ in the last 7 days and read the diff of each one that touches a failing endpoint, a scan phase or auth_limiter.go. You may run `go test -tags netgo,sqlite_fts5` on a package to check a hypothesis. Write incident.md with a summary, a UTC timeline, the most likely cause with commit and file, evidence for each claim (PromQL, LogQL or command and its result), causes ruled out, and every query you ran. Change no code and nothing in Grafana.",
  "connections": [
    "grafana"
  ],
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

The run produces `incident.md`, attached to the session. It says first whether the incident was in serving requests or in scanning the library, because the metrics separate the two, then gives a UTC timeline: the deploy, the first rise in failed requests or failed scans, alerts, and recovery. The cause names a commit and a file, such as a change in `phase_3_refresh_albums.go` or in a Subsonic handler.

Each claim sits beside its proof: the PromQL with its range and result, the LogQL for the log lines it quotes, or the `git show` and `go test` command with output. Ruled-out causes are listed with the query that excluded them, for example a `db_model_totals` series that stayed flat, which argues against a library that suddenly grew.

The session's changed-files list shows no edits and its tool calls show only reads from Grafana. The engineer reading the report reruns the main query in Grafana Explore and decides whether to roll back, disable the scan schedule for now, or fix forward.

## Run it on every Prometheus alert in Grafana

Create an automation in the Project with this prompt and a `webhook` trigger. In Grafana, add a webhook contact point that posts to the trigger's URL with the trigger's secret as its bearer token, and route the error-rate and failed-scan alert rules to it. Each firing starts a run, with the alert's labels after the prompt as data, so the run knows whether a scan or an endpoint alerted.

For a regular check instead of an alert, give the automation a schedule trigger with a `cron` such as `0 7 * * *` in your `timezone`, and ask for a short report on the last 24 hours that flags failed scans and endpoints whose error ratio rose.

## Why run it on Rush?

The labels on Navidrome's metrics already say a lot, but only if someone knows which code emits them and how the scanner's phases fit together. The agent reads that code and queries Grafana in one sandbox of its own, so it can run the Ginkgo suites with the right build tags and Go version without disturbing the engineer's local setup.

Its claims are checkable: every PromQL and LogQL query and every command is in the report with its result, and the full record stays on the session. It changes nothing, and two alerts on two services become two independent runs.

## Frequently asked questions

### Our service does not export Prometheus metrics. Can the agent still help?

Yes, with less precision. It reads the logs in Loki instead and groups them by message and time. In Navidrome, turning on `prometheus.enabled` adds the per-endpoint and per-scan series that make the cause much easier to place.

### Why does it need build tags to run the tests?

Navidrome's Makefile runs `go test -tags netgo,sqlite_fts5`, and some packages expect SQLite's full-text search to be compiled in. Running the tests without the same tags can fail for reasons that have nothing to do with the incident.

### Can it tell which client app was affected?

Yes, when the `client` label is present on `http_request_count`. The report then says whether failures came from every Subsonic client or one, which separates a server bug from a single app sending requests the server now rejects.

### Will it trigger a library scan to test its idea?

No. It reads scan results from the `media_scans` and `media_scan_last` series and the scanner's log lines, and it tests ideas with the scanner's Ginkgo suites in its own sandbox. Calling `startScan` on your server would change state, and the prompt forbids that.