All jobs

Write an Incident Postmortem for a Kubernetes Outage

An agent reads your Grafana logs, the commits deployed in the window and the incident channel export, and writes a blameless postmortem with a sourced timeline.

Run this on RushIn a Project that has your repository.

What the agent does

The run starts with three sources named. The Grafana connection lets the agent query logs and metrics itself; the repository is cloned so it can read the manifests and the commits deployed in the window; and the incident channel export attaches as a file block, so the human side of the timeline arrives with the run rather than from memory. Each source lands in the prompt, and the report has to cite which one every line came from.

In a system like GoogleCloudPlatform/microservices-demo, Online Boutique, the manifests are the map. The release file holds twelve deployments and twelve services, eleven application services across four languages plus a redis-cart, and the agent uses it to translate Grafana results into service names instead of pod hashes. The Go services log JSON through logrus with timestamp, severity and message fields, and the frontend logs request-path lines such as home, order placed with the order id, and request error, so a Loki query grouped by service and severity gives the error spike a shape and a start time. Tracing through the OpenTelemetry collector is an opt-in component in the helm chart and kustomize overlays, and the report notes when it was on, because a trace id is stronger evidence than a log correlation.

The commit window comes from the deploy times: git log between the last deploy before the incident and the one after it, cross-checked against the image tags in the manifests. The agent reads the diffs in that range with the failing service already known, and separates commits that touched the failing path from commits that merely shipped beside them. The channel export fills the human timeline: who noticed, what was tried, when the page went out.

The postmortem itself follows a blameless structure: a timeline where every entry names its source, a root cause stated as the change and the mechanism, the impact in requests and time, what went well, and action items with owners left blank for the review meeting. No line of the timeline rests on "someone said" when a log query or a commit can carry it; where the sources disagree, the report says so rather than smoothing it over.

Start it with one API call

POST https://api.prix.dev/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Write a blameless postmortem for the incident in the window below. Use the Grafana logs and metrics, the commits in this repository deployed in that window, and the incident channel export attached as a file. Include a timeline with sources, root cause, impact, what went well and action items with owners left blank.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is the postmortem document, the run's artifact. Its timeline is a table of entries, each with a timestamp, what happened, and the source: the Loki or metrics query behind it, the commit sha, or a line from the channel export. The root cause section names the change and the mechanism, with the diff and the log evidence quoted beside it, and the impact section states what the metrics show about the window.

The action items are written to be assigned: each is one sentence, verifiable, and points at the check that would have caught the incident earlier. Owners stay blank on purpose; filling them is the review meeting's job. The what-went-well section is sourced the same way, which keeps it honest instead of ceremonial.

Before the review, someone checks that each timeline entry's source reproduces: the query returns the entry, the commit exists in the window, the export line is quoted correctly. The trajectory holds every query and its raw output, so the postmortem can be audited line by line, and the record of how it was assembled is kept alongside it.

Run it on every incident closes

Close the loop in your incident tool: when an incident resolves, have your server post this same request with the incident window, and attach the channel export as a file block. One resolved incident starts one run, so the postmortem is drafted while the logs are still in retention and the commits are still fresh, instead of a week later from a quieter memory.

Set a webhook URL on the API key and Rush posts run.completed or run.failed when the run's turn finishes, carrying the session URL; your server can file the draft into the incident record. Name the Grafana connection on every run, since a key run reads only the connections it names.

The use case this job follows, with its automation and trigger: Investigate an incident.

Why run it on Rush?

A postmortem written by hand drifts: the timeline is reconstructed from chat scrollback, the root cause from whoever remembers loudest, and the action items from whatever survives the meeting. On Rush the agent assembles the same document from the sources directly, in its own sandbox, with every query and diff kept in the trajectory, so each line of the timeline is a fact with a citation rather than a recollection.

It also scales in the way incident-heavy weeks demand. Every closed incident gets one request and one run, so three incidents in a day produce three drafted postmortems in parallel, each waiting for a human review and owner assignment. The blameless structure holds because the evidence, not the seniority, carries each claim.

Frequently asked questions

How does the agent query my logs and metrics?

You connect Grafana once from the console and name the connection on the run; a key run reads only the connections it names. The agent queries Loki and Prometheus through it, and the postmortem quotes each query next to the timeline entries it supports.

How do I attach the incident channel export?

Send the run's input as blocks: a text block with the prompt and a file block carrying the export. Rush places it in the sandbox and lists it in the prompt, and the agent quotes lines from it into the timeline with the rest of the sources.

We self-host and our manifests differ from Online Boutique. Does it still work?

Yes. Your repository's own manifests and helm values are the map the agent uses; the Online Boutique layout above shows the read for a twelve-service system. Single-cluster or multi-cluster, the method is the same: manifests to name services, logs for the window, commits for the deployed changes.

Will it assign blame to whoever merged the change?

No. The postmortem is blameless by structure: the root cause names the change and the mechanism, not the person, and the action items target detection and process gaps. The evidence is kept so the review stays factual.

What if the evidence is incomplete?

The report says so where it happens: a timeline gap names the source that should have covered it and that retention or configuration prevented it. An honest gap with the missing source named is more useful to the review than a smoothed-over guess.