# Fix a Spring Boot App From a Grafana Alert

Yes. When a Grafana alert fires, an agent queries its Prometheus and Loki data, finds the failing controller and opens a pull request with a fix and a JUnit test.

[Run this on Rush](https://cloud.getrush.ai/run/fix-grafana-alerts/spring-boot/), in a Project that has your repository.

## What the agent does

The run starts from the alert itself. A Grafana contact point of type webhook posts its JSON (`status`, and per alert the `labels`, `annotations`, `startsAt` and `generatorURL`) to a small handler on your server, which appends that payload to the prompt and sends the request on this page. The agent clones the repository into its own cloud sandbox and, through your Grafana Cloud connection, opens the alert rule the payload names to see the exact query and threshold that fired, rather than guessing from the alert title.

Next it reads the evidence around the firing time. For a Spring Boot service the request metric Micrometer exports is `http_server_requests_seconds_count`, labelled by `uri`, `status`, `outcome` and `exception`, so the agent queries Prometheus for the 5xx rate by `uri` and by `exception` over the window before `startsAt`. That separates "every route is slow" from "one route throws". Then it queries Loki for the error lines from the same service and window and pulls the stack traces. Upstream spring-projects/spring-petclinic ships `spring-boot-starter-actuator` with `management.endpoints.web.exposure.include=*` but no Prometheus registry, so if your fork exports metrics another way the agent says which series it found and which it could not.

With a `uri` and a stack trace in hand it reads the code they point at. In PetClinic the owner and visit paths share two lookups that throw on bad data: `OwnerController.findOwner` throws `IllegalArgumentException("Owner not found with id: ...")` from `owners.findById(ownerId).orElseThrow(...)`, and `VisitController.loadPetWithVisit`, a `@ModelAttribute` that runs before every visit handler, throws when `owner.getPet(petId)` returns null. There is no `@ControllerAdvice` in `src/main`, so either exception becomes a 500 rendered by `templates/error.html`, and a `uri="/owners/{ownerId}/pets/{petId}/visits/new"` spike with `exception="IllegalArgumentException"` points straight at that method. The agent also rules out `CrashController`'s `/oups`, which throws on purpose to demo the error page.

It then writes a failing test before the fix, in the style the repository already uses: `VisitControllerTests` and `OwnerControllerTests` are `@WebMvcTest` slices with `OwnerRepository` as a `@MockitoBean`, so a new `@Test` that stubs the repository with the data from the logs and asserts the status reproduces the alert without a database. It changes the controller until that test passes, runs `./mvnw test` for the whole suite, and opens a pull request whose description quotes the PromQL and LogQL it ran and the log lines that led to the diagnosis.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "A Grafana alert fired for this Spring Boot service; its webhook payload follows this prompt. Use the Grafana connection to read the alert rule and its query. Query Prometheus for http_server_requests_seconds_count by uri, status and exception for the 30 minutes before startsAt, and query Loki for the service's error logs and stack traces in the same window. Find the cause in the owner and visit controllers (OwnerController, VisitController and the @ModelAttribute methods they run first). Write a JUnit test in the existing @WebMvcTest style that fails the way production failed, fix the code, and run ./mvnw test until it passes. Open a pull request that quotes the PromQL and LogQL you ran, the log lines behind the diagnosis, and the alert's generatorURL. If the data does not point at code, say so in the session result and open no pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request on a new branch. In a service shaped like PetClinic it touches one controller under `src/main/java/.../owner/`, for example `VisitController.java` returning a 404 or a redirect with a message instead of throwing when a pet is not found, and adds one test method to `VisitControllerTests.java` or `OwnerControllerTests.java`. The description has three parts: the alert (rule, threshold, firing time and link), the evidence (each query and the series or log lines it returned), and the change with the reason it stops the errors the alert counted.

The pull request's own checks run your workflows; in PetClinic that is `maven-build.yml` running `./mvnw -B verify`, which adds the javaformat and nohttp Checkstyle checks on top of the tests. The session keeps every query, every command and its output, so a reviewer can rerun the same PromQL in Grafana and see the same spike.

Before merging, a reviewer checks that the fix handles the bad input rather than hiding it (a swallowed exception would also make the alert go quiet), that the new test fails on the old code, and that the status the endpoint now returns is one the alert rule treats as healthy. If the evidence pointed at a database or a dependency rather than the code, the session result says so and no pull request is opened.

## Run it on every Grafana alert that fires

Add a webhook contact point in Grafana Alerting and route the service's alert rules to it with a notification policy. Your handler checks that `status` is `firing`, keeps each alert's `fingerprint` so a repeated notification for the same alert starts no second run, and puts the payload at the end of `input`. Name your Grafana connection in the request's `connections`, by its id or the name you gave it, so the agent can query the stack.

Give the API key a webhook URL and Rush POSTs `run.completed` or `run.failed` with the `session_url` when the agent's turn finishes, so the handler can post the pull request link back to the on-call channel. A key runs at most `max_concurrent` sessions at once (4 by default) and answers `429` past that, which also caps a noisy alert storm.

## Why run it on Rush?

Diagnosing from an alert means running queries, reading code and running the build in one loop. On a laptop that is an engineer switching between Grafana, an IDE and a terminal at the moment the page went off. In a Rush sandbox the agent does the loop on its own clone with JDK, Maven and the Grafana connection side by side, and it starts the second the alert fires instead of when someone acknowledges it.

The result carries its own proof: the queries and their output, the stack trace it matched, the test that fails before the fix and passes after, and `./mvnw test` in the transcript. A reviewer who was asleep when the alert fired can read the whole diagnosis in the morning and decide whether to merge, and several alerts firing at once each get their own sandbox.

## Frequently asked questions

### Does my service need to export Prometheus metrics?

It helps. Spring Boot's actuator exports `http_server_requests_seconds` when `micrometer-registry-prometheus` is on the classpath, and that metric is what ties the alert to a route. Upstream PetClinic does not include that registry, so a fork that sends metrics through OpenTelemetry or another path should say so in the prompt and name the metric.

### Why is the Grafana connection not in the request on this page?

The request shows only the task and the repository. A key run gets only the connections it names, so add `"connections": ["grafana"]` (or the connection's id) after you connect Grafana Cloud once with `POST /me/connections`. Without it the agent has only the payload and the code.

### What if the alert is caused by the database, not the code?

The prompt tells the agent to open no pull request when the evidence does not point at code. The session result then says what it found, such as connection pool errors from Hikari in Loki, so the on-call engineer starts from a diagnosis instead of a blank screen.

### Will it touch the alert rule or silence the alert?

No. The Grafana connection is used to read the rule, the metrics and the logs. The change is in the repository and goes through a pull request, so nothing reaches production until someone merges it.