Investigate a Kubernetes Outage with Grafana and Loki

The agent starts in its own cloud sandbox with your repository cloned and Grafana Cloud reachable through the Grafana connection, which searches dashboards, queries Prometheus and Loki, and reads alerts and incidents. It begins with the manifests, because they say which services exist and who calls whom. In a system like GoogleCloudPlatform/microservices-demo, Online Boutique, kubernetes-manifests/ holds one file per service: frontend.yaml reads PRODUCT_CATALOG_SERVICE_ADDR, CURRENCY_SERVICE_ADDR, CART_SERVICE_ADDR, RECOMMENDATION_SERVICE_ADDR, SHIPPING_SERVICE_ADDR, CHECKOUT_SERVICE_ADDR and AD_SERVICE_ADDR, and checkoutservice.yaml reads the product catalog, shipping, payment, email, currency and cart addresses. From those the agent draws the call graph it will use to tell a cause from a symptom.

It also records each Deployment's limits, because they explain many failures. cartservice.yaml gives the cart server cpu: 200m and memory: 64Mi requested with limits of 300m and 128Mi, and it depends on a redis-cart Deployment running redis:alpine with its data on a redis-data volume; frontend.yaml and checkoutservice.yaml limit at 200m and 128Mi. A pod at its memory limit is OOM killed and restarted, and a single Redis that restarts takes every cart request with it.

Then it queries the window in the prompt. Through Grafana it reads the error rate and latency per service from the dashboards' Prometheus queries, container restarts and OOMKilled terminations from kube-state-metrics when the cluster exports it, and the Loki logs per pod. It builds the timeline minute by minute and, for each minute, writes which services were failing and the query that shows it. The first service whose errors rise, while the services it calls stay healthy, is the first to fail; the services that call it fail after it, and the report says so instead of listing them all as causes.

When the logs give a reason, such as Redis connection refused in the cart service or a gRPC deadline in the frontend, the agent quotes the log line and goes back to the code under src/ that writes it to explain what the service was doing. It changes nothing in the cluster or the repository.

What comes back is incident.md as an artifact on the session, which the session read lists with a download link. It starts with a one-paragraph summary naming the first service to fail and the reason, then the minute-by-minute timeline. Each row says what changed in that minute and carries the PromQL or LogQL query that shows it, with the result the agent read, so the timeline can be checked without trusting it.

After the timeline comes the call graph drawn from the manifests, with the failing services marked, and a section on the cause: the evidence that the first service failed on its own, such as a restart, an OOMKilled termination or a dependency that stopped answering, and the line in the manifest or code that relates to it, such as a 128Mi memory limit. The session's transcript keeps every Grafana call the agent made and every file it read.

Before acting on it, the on-call engineer reruns the key queries in Grafana, checks the limits against what the pods actually used and decides what to change. A follow-up message to the same session can ask the agent to raise a limit or add a probe on a branch and open a pull request.

Grafana alerting can send a notification to a webhook contact point. Point it at a handler on your server that sends this request with the alert's start time and the services it names, and each firing alert gets a timeline written while people are still joining the call.

Set a webhook URL on the API key and Rush POSTs run.completed when the turn finishes, with the session_url. The handler takes incident.md from the session's artifacts and posts it in the incident channel or attaches it to the Grafana incident.

In an outage across many services, the slow part is not one query but dozens: per service, per minute, metrics and logs, and keeping straight which failure came first. On a laptop that means many browser tabs and notes. In a Rush sandbox the agent runs the queries through your Grafana connection and reads the manifests on its own disk, and the engineer keeps their attention on recovery.

Every row of the timeline carries the query that produced it, and the transcript keeps the answers, so the post-incident review starts from evidence. Separate outages, or the same window seen from different clusters, can each get their own run at once, up to the key's concurrent run limit.

Does the agent need access to my cluster?

How do I give it Grafana access?

What if my manifests are in Helm or Kustomize?

Can it tell a cause from a symptom?