All jobs

Fix a Go Service From a Grafana Alert

Yes. The alert starts a run that queries its Prometheus metric and Loki logs, finds the failing handler and opens a pull request with a Go test.

Run this on RushIn a Project that has your repository.

What the agent does

A Grafana Cloud alert posting to a Rush webhook trigger starts the run, with the notification after the prompt as untrusted data. The agent reads the rule through the Grafana connection, reruns its PromQL around startsAt broken down by the route or handler label, and queries Loki for the service's error lines in the same window. In a Gin service the access log line carries the method, path and status, so the agent can tell whether the errors come from one route or all of them.

In an API like gothinkster/golang-gin-realworld-example-app, routes map to two packages. GET /api/articles is ArticleList in articles/routers.go, which calls FindManyArticle in articles/models.go; that function parses limit and offset with strconv.Atoi, opens a transaction with db.Begin(), and takes different query paths for tag, author and favorited. Login and registration are UsersLogin and UsersRegistration in users/routers.go, with the JWT middleware in users/middlewares.go. The agent follows the failing path from the log line to the branch of code it took.

The sandbox needs no external database: common/database.go opens SQLite through GORM at DB_PATH (default ./data/gorm.db) and tests use TEST_DB_PATH, with helpers in common/test_helpers.go. The module says go 1.21 and CI tests on 1.21, 1.22 and 1.23. The agent writes a table case in articles/unit_test.go or users/unit_test.go, next to tests such as TestFindManyArticle and TestGetArticleFeed, that sends the request from the logs and fails the way production did.

It then fixes the handler or model, runs go test ./..., and runs scripts/gofmt.sh, which CI runs before scripts/coverage.sh, so the change passes the same checks as any other pull request.

Start it with one API call

POST https://api.prix.dev/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "A Grafana alert fired for this Go service; its payload follows this prompt. Use the Grafana connection to read the alert rule, run its Prometheus query around the firing time broken down by route, and query the Loki error logs for the same window. Find the cause in the users and articles packages, add a Go test that reproduces it, fix it, and run go test ./... and gofmt. Open a pull request that includes the PromQL and LogQL queries, the log lines behind the diagnosis, and the test output before and after the fix. If the evidence does not point at code, say what it points at instead.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request whose description shows the rule's PromQL with the route that crossed the threshold, the LogQL query with the error lines it returned, and the new test failing and then passing with go test ./.... The diff is the test and the fix in articles/ or users/.

A reviewer checks that the test sends the same request the logs show, that a fix in FindManyArticle keeps the transaction committed or rolled back on every path, and that error handling returns the right status rather than turning a failure into an empty 200. CI runs the change on all three Go versions.

When the evidence points away from code, for example at a full disk under the SQLite file or a deploy in progress, the run says so with the numbers and log lines it found.

Run it on every Grafana alert on the API

Add an automation to the Project with this prompt and a webhook trigger. In Grafana Cloud, create a webhook contact point at the trigger's URL with Authorization: Bearer <secret>, and point this service's alert rules at it. The notification Grafana sends becomes the event text the agent reads under the prompt.

Skip resolved notifications on the contact point, and keep the notification policy's repeat interval longer than a run, since a delivery with no Idempotency-Key header is always a new run. The automation's runs-per-hour limit is the backstop for a flapping rule.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

Turning an alert into a fix takes the same steps each time: rerun the rule's query, find the matching log lines, follow the route to the handler, write a test that fails the same way. A Rush run does those steps in its own sandbox and keeps every query and go test output in the session, so the diagnosis can be checked, not just trusted.

When a bad release trips alerts on several routes at once, each alert gets its own run in parallel, none of them touches a production host, and each ends in a pull request whose test proves the fix or an account of what the data shows instead.

Frequently asked questions

My service does not expose a route label. Does this still work?

Yes, with less precision. The agent reruns the rule's query as written and leans on the Loki access and error lines to find the path, which in a Gin service include the method, path and status.

Does the test need a real database?

For this API, no: it uses SQLite through GORM at TEST_DB_PATH, so tests run in the sandbox as they do in CI. A service on PostgreSQL gets a database started in the sandbox.

Which Go version does it use?

The one go.mod names, and CI then tests the pull request on every version in its matrix, 1.21 to 1.23 here.

Can it read traces as well as metrics and logs?

The Grafana connection is described as querying Prometheus and Loki and reading alerts, incidents and dashboards. This page's prompt uses metrics and logs; a service that only reports failures through traces is outside what it asks for.