# Investigate CrashLoopBackOff Pods in Kubernetes Yes. An agent reads your manifests, service code and recent commits, then writes an incident report ranking each suspect with file and line evidence. [Run this on Rush](https://cloud.getrush.ai/run/investigate-crashloopbackoff/kubernetes/), in a Project that has your repository. ## What the agent does The agent clones the repository into its own cloud sandbox and starts from the alert you paste into the prompt: which Deployment restarts, how often, the exit code and when it began. Exit code 137 means the container was sent SIGKILL, which in Kubernetes is usually the kernel's OOM killer at the memory limit or the kubelet killing a container that failed its liveness probe. The agent keeps both explanations open and looks for evidence of each, rather than assuming the first. In a repository like GoogleCloudPlatform/microservices-demo it reads `kubernetes-manifests/cartservice.yaml` first. The `server` container there requests `memory: 64Mi` and is limited to `memory: 128Mi`, with `cpu: 300m`, while `helm-chart/values.yaml` gives `cartService` a `256Mi` limit. So the same service gets half the memory depending on which deployment path was used, and the agent notes which one the rollout applied. It then reads `src/cartservice/src/Dockerfile`: a .NET 10 self-contained, trimmed single-file publish running on a chiseled `runtime-deps` image, with no `DOTNET_GCHeapHardLimit` or other GC setting in the manifest's `env`, which only sets `REDIS_ADDR` to `redis-cart:6379`. For the probe explanation, it reads the `livenessProbe`: gRPC on port 7070, `initialDelaySeconds: 15`, `periodSeconds: 10` and the default failure threshold. It follows the probe into `services/HealthCheckService.cs`, where `Check` returns `Serving` when `_cartStore.Ping()` is true, and then into `cartstore/RedisCartStore.cs`, where `Ping()` returns `true` without touching Redis. That clears a Redis outage as a probe-failure cause, and the agent says so in the report instead of leaving it as a suspect. It also reads `Startup.cs`, which picks Redis, Spanner, AlloyDB or an in-memory cache from environment variables, because an overlay from `kustomize/components/` (such as `memorystore` or `spanner`) that changes `REDIS_ADDR` changes which store runs. Last, it reads the 30 most recent commits with `git log -p` on the manifests, the Helm chart, the Kustomize components and `src/cartservice`, and lines each change up against the alert's start time: a lowered limit, a base image bump, a package version in `cartservice.csproj` (Grpc.AspNetCore, StackExchangeRedis, Npgsql), or a new probe setting. Each suspect goes into `incident.md` with its file and line, why it fits exit code 137 and a 90-second restart cycle, and the one check that confirms or clears it. It changes no code and opens no pull request. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "claude", "input": "Incident: Grafana alerts that the cartservice pods restart every 90 seconds in CrashLoopBackOff with exit code 137 since this morning's rollout. Read the code, the configuration and the last 30 commits, find the most likely causes, and write incident.md with a timeline, each suspect with file and line evidence, and the check that would confirm or clear it. Change no code.", "resources": [ { "url": "https://github.com/your-org/your-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back What comes back is `incident.md`, attached to the session as an artifact with a download link that works for an hour. It opens with a timeline built from the commit history: each change to the cartservice manifests, chart values, overlays and source in the window, with its commit and time, set against the alert you described. Then come the suspects, ranked. Each one names the file and line (the `limits.memory` line in `cartservice.yaml`, the `cartService.resources` block in `values.yaml`, a probe setting, a dependency bump), explains how it produces a SIGKILL every cycle, and gives the check that settles it, such as `kubectl describe pod` showing `Last State: Terminated, Reason: OOMKilled`, `kubectl get events` showing `Liveness probe failed`, or `container_memory_working_set_bytes` for the pod climbing to its limit before each restart. The report also lists what the agent ruled out and why, such as the Redis-backed health check that cannot fail because `Ping()` always returns true. That keeps an on-call engineer from spending the first twenty minutes on a dead end. The session's transcript keeps every command the agent ran, `git log`, `grep` and each file it read, so any line of the report can be traced back to its source. Before acting on it, the engineer runs the confirming checks against the live cluster, which the agent did not have, and decides on the fix: raise the limit, set a GC heap limit, roll back one commit or change a probe. A follow-up message to the same session can ask the agent to prepare that change as a pull request once the cause is confirmed. ## Run it on every CrashLoopBackOff alert Point a Grafana alert's webhook contact point at a small handler on your server that turns the alert into this request: the Deployment name, the restart interval and the exit code go into the prompt, and the repository that holds the manifests goes in `resources`. Each alert then starts its own run in its own sandbox, and the first pass is under way before anyone is paged out of bed. Set a webhook URL on the API key, and Rush POSTs `run.completed` with the `session_url` when the turn finishes. Your handler reads the session, takes the `incident.md` link from its artifacts and posts it in the incident channel or on the alert itself. ## Why run it on Rush? A CrashLoopBackOff investigation usually starts with someone cloning three repositories, grepping manifests and reading Helm values while also watching the pods. In a Rush sandbox the agent does the reading side in parallel with the people on call: it has the whole repository, the full commit history and a shell, and it cannot touch the cluster or change the deployment while it works. The report is only useful if each claim can be checked, so the agent's work is kept: every file it opened and every command it ran with its output is in the session's transcript. When several services crash at once, each alert can start its own run, up to the key's concurrent run limit, and each comes back with a report of its own. ## Frequently asked questions ### Does the agent connect to my cluster or run kubectl? Not in this request. It reads the repository and the commit history, and the report names the kubectl and metrics checks a person runs against the cluster. If your Grafana Cloud stack is connected, the agent can query Prometheus and Loki through that connection as well. ### We deploy with Helm and Kustomize overlays. Does it understand which values apply? It reads the base manifests, the chart's `values.yaml` and the overlays in the repository, and reports where they disagree, such as a memory limit that differs between the raw manifest and the chart. It cannot see which one your pipeline applied unless that is in the repository or the prompt, so say so in the prompt when you know. ### Will it change my manifests or open a pull request? No. The prompt says to change no code, and the result is a Markdown report attached to the session. If you want the fix as a pull request, send a follow-up to the same session once you have confirmed the cause. ### How does it tell an OOM kill from a liveness probe kill? Both end in exit code 137, so the agent looks for evidence of each in the code: memory limits and runtime settings for one, probe timing and what the health check actually tests for the other. The report gives the check that separates them, the `Reason` in the pod's last state or the probe failures in its events.