# Fix a Go Service From a Grafana Alert Yes. The alert starts a run that queries its Prometheus metric and Loki logs, finds the failing handler and opens a pull request with a Go test. [Run this on Rush](https://cloud.getrush.ai/run/fix-grafana-alerts/go/), in a Project that has your repository. ## What the agent does A Grafana Cloud alert posting to a Rush webhook trigger starts the run, with the notification after the prompt as untrusted data. The agent reads the rule through the Grafana connection, reruns its PromQL around `startsAt` broken down by the route or handler label, and queries Loki for the service's error lines in the same window. In a Gin service the access log line carries the method, path and status, so the agent can tell whether the errors come from one route or all of them. In an API like `gothinkster/golang-gin-realworld-example-app`, routes map to two packages. `GET /api/articles` is `ArticleList` in `articles/routers.go`, which calls `FindManyArticle` in `articles/models.go`; that function parses `limit` and `offset` with `strconv.Atoi`, opens a transaction with `db.Begin()`, and takes different query paths for `tag`, `author` and `favorited`. Login and registration are `UsersLogin` and `UsersRegistration` in `users/routers.go`, with the JWT middleware in `users/middlewares.go`. The agent follows the failing path from the log line to the branch of code it took. The sandbox needs no external database: `common/database.go` opens SQLite through GORM at `DB_PATH` (default `./data/gorm.db`) and tests use `TEST_DB_PATH`, with helpers in `common/test_helpers.go`. The module says `go 1.21` and CI tests on 1.21, 1.22 and 1.23. The agent writes a table case in `articles/unit_test.go` or `users/unit_test.go`, next to tests such as `TestFindManyArticle` and `TestGetArticleFeed`, that sends the request from the logs and fails the way production did. It then fixes the handler or model, runs `go test ./...`, and runs `scripts/gofmt.sh`, which CI runs before `scripts/coverage.sh`, so the change passes the same checks as any other pull request. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "claude", "input": "A Grafana alert fired for this Go service; its payload follows this prompt. Use the Grafana connection to read the alert rule, run its Prometheus query around the firing time broken down by route, and query the Loki error logs for the same window. Find the cause in the users and articles packages, add a Go test that reproduces it, fix it, and run go test ./... and gofmt. Open a pull request that includes the PromQL and LogQL queries, the log lines behind the diagnosis, and the test output before and after the fix. If the evidence does not point at code, say what it points at instead.", "resources": [ { "url": "https://github.com/your-org/your-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back What comes back is a pull request whose description shows the rule's PromQL with the route that crossed the threshold, the LogQL query with the error lines it returned, and the new test failing and then passing with `go test ./...`. The diff is the test and the fix in `articles/` or `users/`. A reviewer checks that the test sends the same request the logs show, that a fix in `FindManyArticle` keeps the transaction committed or rolled back on every path, and that error handling returns the right status rather than turning a failure into an empty `200`. CI runs the change on all three Go versions. When the evidence points away from code, for example at a full disk under the SQLite file or a deploy in progress, the run says so with the numbers and log lines it found. ## Run it on every Grafana alert on the API Add an automation to the Project with this prompt and a `webhook` trigger. In Grafana Cloud, create a webhook contact point at the trigger's URL with `Authorization: Bearer `, and point this service's alert rules at it. The notification Grafana sends becomes the event text the agent reads under the prompt. Skip resolved notifications on the contact point, and keep the notification policy's repeat interval longer than a run, since a delivery with no `Idempotency-Key` header is always a new run. The automation's runs-per-hour limit is the backstop for a flapping rule. ## Why run it on Rush? Turning an alert into a fix takes the same steps each time: rerun the rule's query, find the matching log lines, follow the route to the handler, write a test that fails the same way. A Rush run does those steps in its own sandbox and keeps every query and `go test` output in the session, so the diagnosis can be checked, not just trusted. When a bad release trips alerts on several routes at once, each alert gets its own run in parallel, none of them touches a production host, and each ends in a pull request whose test proves the fix or an account of what the data shows instead. ## Frequently asked questions ### My service does not expose a route label. Does this still work? Yes, with less precision. The agent reruns the rule's query as written and leans on the Loki access and error lines to find the path, which in a Gin service include the method, path and status. ### Does the test need a real database? For this API, no: it uses SQLite through GORM at `TEST_DB_PATH`, so tests run in the sandbox as they do in CI. A service on PostgreSQL gets a database started in the sandbox. ### Which Go version does it use? The one `go.mod` names, and CI then tests the pull request on every version in its matrix, 1.21 to 1.23 here. ### Can it read traces as well as metrics and logs? The Grafana connection is described as querying Prometheus and Loki and reading alerts, incidents and dashboards. This page's prompt uses metrics and logs; a service that only reports failures through traces is outside what it asks for.