# Fix the Cause of a Grafana Alert in Kubernetes Yes. The alert starts a run that queries Prometheus and Loki through Grafana, finds the failing service in the cluster and opens a pull request with the fix. [Run this on Rush](https://cloud.getrush.ai/run/fix-grafana-alerts/kubernetes/), in a Project that has your repository. ## What the agent does The alert reaches a Rush webhook trigger and its payload follows the prompt as untrusted data, with labels such as `namespace`, `pod`, `container` or `service`. Through the Grafana connection the agent reads the rule and runs its PromQL around the firing time, then queries Loki with a selector built from those labels, for example `{namespace="default", container="server"}` filtered to error lines. In a cluster of many services this step decides which directory in the repository to open. In a microservices repository like `GoogleCloudPlatform/microservices-demo` (Online Boutique), there are eleven services in `src/` in five languages, deployed from `kubernetes-manifests/`, `helm-chart/` or `kustomize/`. `checkoutservice` logs JSON through logrus with `severity` and `message` fields, which makes its Loki lines easy to filter. Its `PlaceOrder` in `src/checkoutservice/main.go` calls the cart, product catalog, currency, shipping, payment and email services in turn and wraps each failure in a gRPC status, such as `failed to charge card` with `codes.Internal` or `shipping error` with `codes.Unavailable`, so the error text tells the agent which hop failed. Some alerts are about the pod, not the code. `kubernetes-manifests/checkoutservice.yaml` requests `64Mi` and limits memory to `128Mi`; `cartservice.yaml` limits it to `128Mi` with `300m` of CPU. An alert on restarts with `OOMKilled` in the container's last state points at those numbers, and the agent compares them with the memory series before the kill rather than raising the limit blindly. When the cause is code, it writes a test next to the failing function, such as `money_test.go` for `src/checkoutservice/money`, or `shippingservice_test.go`, and makes it fail first. It then fixes the service or the manifest, runs that service's own tests (`go test` in `shippingservice`, `productcatalogservice` and `frontend/validator`, or `dotnet test src/cartservice/`, as `ci-pr.yaml` does), and checks that a manifest change also lands in `helm-chart/` and `kustomize/` when those carry the same value. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "claude", "input": "A Grafana alert fired; its payload is below. Through the Grafana connection, read the alert rule, run its Prometheus query around the firing time, and query the Loki logs for the namespace, pod and container in the alert's labels. Find which service and code path, or which manifest setting, produces it. Fix it with a test where the cause is code, run that service's tests, and keep Kubernetes manifests, the Helm chart and Kustomize in agreement. Open a pull request that quotes the queries and their results. If the cause is not in this repository, say what it is instead.", "resources": [ { "url": "https://github.com/your-org/your-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back What comes back is a pull request whose description names the service, quotes the PromQL and the series values around the firing time, and quotes the LogQL selector with the lines it returned. For a code fix it shows the new test failing and then passing; for a resource or probe change it shows the metric that justifies the new number. A reviewer checks the evidence matches the alert's labels, that a code fix handles the failing call instead of swallowing its error, and that any change to a request, limit or probe appears in every deployment format the repository ships. The pull request's own CI runs the service tests. Nothing is applied to the cluster. The change is a pull request, and when the cause lies outside the repository, such as a node under pressure or a dependency's outage, the run reports it with the queries and opens nothing. ## Run it on every Grafana alert in the cluster Make an automation in the Project with this prompt and a `webhook` trigger, then add a Grafana webhook contact point at the trigger's URL with an `Authorization` header of `Bearer `. Route the cluster's alert rules to it with a notification policy; group by `namespace` and `alertname` so one bad rollout sends one notification rather than one per pod. Resolved notifications can be turned off on the contact point, and a repeat interval longer than a run keeps a still-firing alert from starting a second run, because deliveries without an `Idempotency-Key` header are never merged. ## Why run it on Rush? In a cluster with many services, most of the work behind an alert is finding which service it is about: the query, the labels, the right Loki selector, then the right directory and language. A Rush run does that in its own sandbox with each query and its result kept, without a kubeconfig on anyone's laptop and without touching the cluster. Alerts arrive in clusters of their own when a dependency fails. Each gets its own run at the same time on its own branch, and each ends in a pull request backed by queries and a test, or in an answer that says the cause is outside the code. ## Frequently asked questions ### Does the agent get access to my cluster? No. It reads metrics and logs through the Grafana connection and changes only the repository. Nothing in this flow runs `kubectl` against your cluster. ### Will it just raise memory limits when a pod is OOMKilled? Only when the memory series supports it, and the pull request quotes that series. A leak shows as steady growth to the limit, and then the fix is in the code. ### We deploy with Helm, not raw manifests. Does it still work? Yes. It edits whichever format the repository uses, and when a value appears in more than one, as in Online Boutique's manifests, Helm chart and Kustomize base, it changes all of them. ### What does the agent need from Grafana? The Grafana MCP server connected once with `POST /me/connections` and an OAuth sign-in, then named in `connections`. It queries Prometheus and Loki and reads alerts in Grafana Cloud.