What the agent does
The agent clones the repository into its own cloud sandbox and reads the chart the incident names. In prometheus-community/helm-charts that is charts/kube-prometheus-stack: Chart.yaml gives the chart version (92.10.1 at the time of writing) and its dependencies, kube-state-metrics 8.6.0, prometheus-node-exporter 4.60.0, grafana 14.0.1 and prometheus-windows-exporter 0.12.*. It installs Helm in the sandbox and runs helm template with your values file, so it reads the Prometheus custom resource and the ServiceMonitors the chart actually produces, not only the defaults.
Then it reads the parts of values.yaml that decide memory. Under prometheus.prometheusSpec, resources: {} sets no request or limit, with memory: 400Mi only in a comment, so the memory limit the pod is killed at comes from your values or a namespace default; retention: 10d and an empty retentionSize bound the TSDB on disk but not the head block in memory, which grows with active series. It reads serviceMonitorSelectorNilUsesHelmValues: true, which decides whether Prometheus picks up only the chart's ServiceMonitors or every one in the cluster, and the kubelet section, where cAdvisor: true, cAdvisorInterval: 10s and a cAdvisorMetricRelabelings list that drops metrics such as container_cpu_load_average_10s, container_memory_mapped_file and container_memory_failures_total at scope="hierarchy".
Because the incident started after a chart upgrade with active series doubling, the agent compares the two chart versions. It reads the 30 most recent commits with git log and git show on the chart directory, and renders the chart at the old and new versions with the same values, then diffs the output. A removed relabel rule, a newly enabled ServiceMonitor, a changed scrape interval or a new label on a high-cardinality metric each shows up in that diff with the line that caused it.
For each suspect it writes the check that confirms it in Grafana or Prometheus: prometheus_tsdb_head_series over the upgrade, the top series counts by job and by metric name from count by (__name__), and the container's memory against its limit. It changes no code and touches no cluster.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Incident: Grafana shows the Prometheus pod from kube-prometheus-stack OOM killed every few hours since a chart upgrade, with active series doubling. Read charts/kube-prometheus-stack, its values.yaml and the last 30 commits. Render the chart with helm template at the version before and after the upgrade with the same values and diff the output. Find the most likely causes, and write incident.md with a timeline, each suspect with file and line evidence, the template diff that supports it, and the PromQL check that would confirm or clear it. Change no code.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is incident.md as an artifact on the session, with a download link the session read returns. It opens with a timeline of the chart changes between the two versions, then ranks the suspects: each with the file and line in values.yaml or a template, the part of the rendered diff that shows the change, why it adds series or memory, and the PromQL query that would confirm it, such as head series by job before and after the upgrade.
It also states what the chart leaves to you, such as the empty prometheusSpec.resources, and what that means for the limit the pod is killed at. The session's transcript keeps every helm template run and every diff, so the claims about what changed can be checked by rerunning the same commands.
Before acting on it, the engineer runs the PromQL checks, confirms which jobs gained series, and decides between restoring a relabel rule, dropping a ServiceMonitor or sizing the pod up. A follow-up message to the same session can ask the agent to write that change to your values file on a branch and open a pull request.
Run it on every kube-prometheus-stack version bump
Run it before the upgrade, not after. A Renovate or Dependabot pull request that bumps the chart version can trigger a small job that sends this request with the old and new versions in the prompt, so the series and memory impact of an upgrade is known before it reaches the cluster.
Set a webhook URL on the API key and Rush POSTs run.completed when the turn finishes. Your handler reads the session, takes incident.md from its artifacts and comments it on the bump's pull request.
The use case this job follows, with its automation and trigger: Investigate an incident.
Why run it on Rush?
Comparing two versions of a chart this size means rendering it twice with your values, diffing thousands of lines of YAML and reading release notes for the parts that matter. On a laptop that is a context switch in the middle of an incident. In a Rush sandbox the agent installs Helm, renders and diffs on its own disk, and never touches your cluster.
The report ties each suspect to a line in the rendered diff and a query to confirm it, and the transcript keeps every command, so the result is easy to check and easy to hand over. Several clusters with different values files can be compared at once, one run each, up to the key's concurrent run limit.
Frequently asked questions
Does the agent connect to my cluster?
No. It renders the chart with helm template in the sandbox, which needs no cluster, and writes the PromQL checks for you to run. Connect Grafana and name it on the run if you want the agent to query the metrics itself.
Can it use my own values file?
Yes. Keep the values file in the repository you name in resources, or attach it to the request as a file, and say in the prompt which one you deploy with. The agent renders both versions with it.
Why doesn't retention fix the memory problem?
retention and retentionSize limit how much data is kept on disk. Prometheus memory is driven mostly by the number of active series in the head block, which grows with targets and labels, so the report looks for what added series.
Will it change my values or open a pull request?
Not in this run; the prompt says to change no code. After you have read the report, a follow-up message to the same session can ask it to make the change and open a pull request.