What the agent does
The alert reaches a Rush webhook trigger and its payload follows the prompt as untrusted data, with labels such as namespace, pod, container or service. Through the Grafana connection the agent reads the rule and runs its PromQL around the firing time, then queries Loki with a selector built from those labels, for example {namespace="default", container="server"} filtered to error lines. In a cluster of many services this step decides which directory in the repository to open.
In a microservices repository like GoogleCloudPlatform/microservices-demo (Online Boutique), there are eleven services in src/ in five languages, deployed from kubernetes-manifests/, helm-chart/ or kustomize/. checkoutservice logs JSON through logrus with severity and message fields, which makes its Loki lines easy to filter. Its PlaceOrder in src/checkoutservice/main.go calls the cart, product catalog, currency, shipping, payment and email services in turn and wraps each failure in a gRPC status, such as failed to charge card with codes.Internal or shipping error with codes.Unavailable, so the error text tells the agent which hop failed.
Some alerts are about the pod, not the code. kubernetes-manifests/checkoutservice.yaml requests 64Mi and limits memory to 128Mi; cartservice.yaml limits it to 128Mi with 300m of CPU. An alert on restarts with OOMKilled in the container's last state points at those numbers, and the agent compares them with the memory series before the kill rather than raising the limit blindly. When the cause is code, it writes a test next to the failing function, such as money_test.go for src/checkoutservice/money, or shippingservice_test.go, and makes it fail first.
It then fixes the service or the manifest, runs that service's own tests (go test in shippingservice, productcatalogservice and frontend/validator, or dotnet test src/cartservice/, as ci-pr.yaml does), and checks that a manifest change also lands in helm-chart/ and kustomize/ when those carry the same value.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "A Grafana alert fired; its payload is below. Through the Grafana connection, read the alert rule, run its Prometheus query around the firing time, and query the Loki logs for the namespace, pod and container in the alert's labels. Find which service and code path, or which manifest setting, produces it. Fix it with a test where the cause is code, run that service's tests, and keep Kubernetes manifests, the Helm chart and Kustomize in agreement. Open a pull request that quotes the queries and their results. If the cause is not in this repository, say what it is instead.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request whose description names the service, quotes the PromQL and the series values around the firing time, and quotes the LogQL selector with the lines it returned. For a code fix it shows the new test failing and then passing; for a resource or probe change it shows the metric that justifies the new number.
A reviewer checks the evidence matches the alert's labels, that a code fix handles the failing call instead of swallowing its error, and that any change to a request, limit or probe appears in every deployment format the repository ships. The pull request's own CI runs the service tests.
Nothing is applied to the cluster. The change is a pull request, and when the cause lies outside the repository, such as a node under pressure or a dependency's outage, the run reports it with the queries and opens nothing.
Run it on every Grafana alert in the cluster
Make an automation in the Project with this prompt and a webhook trigger, then add a Grafana webhook contact point at the trigger's URL with an Authorization header of Bearer <secret>. Route the cluster's alert rules to it with a notification policy; group by namespace and alertname so one bad rollout sends one notification rather than one per pod.
Resolved notifications can be turned off on the contact point, and a repeat interval longer than a run keeps a still-firing alert from starting a second run, because deliveries without an Idempotency-Key header are never merged.
The use case this job follows, with its automation and trigger: Fix a bug from an alert.
Why run it on Rush?
In a cluster with many services, most of the work behind an alert is finding which service it is about: the query, the labels, the right Loki selector, then the right directory and language. A Rush run does that in its own sandbox with each query and its result kept, without a kubeconfig on anyone's laptop and without touching the cluster.
Alerts arrive in clusters of their own when a dependency fails. Each gets its own run at the same time on its own branch, and each ends in a pull request backed by queries and a test, or in an answer that says the cause is outside the code.
Frequently asked questions
Does the agent get access to my cluster?
No. It reads metrics and logs through the Grafana connection and changes only the repository. Nothing in this flow runs kubectl against your cluster.
Will it just raise memory limits when a pod is OOMKilled?
Only when the memory series supports it, and the pull request quotes that series. A leak shows as steady growth to the limit, and then the fix is in the code.
We deploy with Helm, not raw manifests. Does it still work?
Yes. It edits whichever format the repository uses, and when a value appears in more than one, as in Online Boutique's manifests, Helm chart and Kustomize base, it changes all of them.
What does the agent need from Grafana?
The Grafana MCP server connected once with POST /me/connections and an OAuth sign-in, then named in connections. It queries Prometheus and Loki and reads alerts in Grafana Cloud.