# Investigate a Phoenix App Incident With Grafana Yes. An agent queries Prometheus and Loki in Grafana for the incident window, ties the spike to recent commits in your Phoenix stats API, and writes incident.md. [Run this on Rush](https://cloud.getrush.ai/run/investigate-incidents-with-grafana/phoenix/), in a Project that has your repository. ## What the agent does The agent reads the window after the prompt and uses the Grafana connection to query it. From Prometheus it pulls the request rate, error ratio and duration histograms the app exports; from Loki it pulls the log lines for the same minutes, filtered to 5xx responses and grouped by request path; and it lists the alert rules that fired. It reads a margin on either side of the window to see whether the error ratio stepped up at one minute, which points at a deploy, or climbed with traffic, which points at capacity. In a Phoenix app like plausible/analytics the metrics have a known shape, because `lib/plausible/prom_ex.ex` registers the PromEx `Application`, `Beam`, `Phoenix`, `PhoenixLiveView`, `Ecto` and, on the right nodes, `Oban` plugins plus a custom `Plausible.PromEx.Plugins.PlausibleMetrics`, with `grafanacloud-prom` as the datasource id. `mix.exs` pins `phoenix` to `~> 1.8.2`, `prom_ex` to `~> 1.12`, `sentry` to `~> 13.5` and the `opentelemetry_phoenix`, `opentelemetry_ecto` and `opentelemetry_oban` packages, so traces exist beside the metrics. `lib/plausible_web/endpoint.ex` adds `Plug.RequestId` and `Plug.Telemetry` with `log: false`, which means the request id is the key that joins a Loki line to a trace. The stats API is the usual suspect, so the agent maps the failing paths to code through `lib/plausible_web/router.ex`. Under `scope "/api"` the dashboard calls `StatsController` for `query`, `csv_export`, `current_visitors`, `filter_suggestions`, `funnel` and `exploration_next`; under `/api/v1/stats` the public `ExternalStatsController` serves `aggregate`, `breakdown`, `timeseries` and `realtime_visitors`; and `/api/v2` routes to `ExternalQueryApiController.query`. All of them end in `lib/plausible/stats/`: `query.ex`, `query_builder.ex`, `query_optimizer.ex`, `query_runner.ex`, `table_decider.ex` and the SQL under `sql/`, running against ClickHouse through `Plausible.ClickhouseRepo`. A 429 is `PlausibleWeb.Api.RateLimit` doing its job; a 500 from `query_runner.ex` is not. It then runs `git log --since=7.days` over `lib/plausible_web/controllers/api` and `lib/plausible/stats` and reads the diff of each commit that touches a module on the failing path. To test a hypothesis it can run the matching ExUnit files, for example `mix test test/plausible_web/controllers/api/stats_controller/exploration_test.exs` or the `external_stats_controller` folder, with the Erlang 28.5 and Elixir 1.20 versions `.tool-versions` names, after `mix do ecto.create, ecto.migrate` as the `elixir.yml` workflow does. It changes no code and nothing in Grafana, and keeps every PromQL and LogQL query for the report. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "claude", "input": "Error rates for the Phoenix app rose between 09:20 and 09:55 UTC today. Use the Grafana connection to read that window: the PromEx Phoenix request rate, error ratio and duration by route, Loki log lines with status 5xx grouped by request path and request id, and any alert rules that fired. Open the repository and compare the window with the commits of the last 7 days to lib/plausible_web/controllers/api and lib/plausible/stats; read the diff of every commit that touches a controller action or a stats query module on a failing path. You may run the matching ExUnit files under test/plausible_web/controllers/api to check a hypothesis. Write incident.md with a summary, a UTC timeline, the most likely cause with commit and file, the evidence for each claim (the PromQL, LogQL or command and its result), the causes you ruled out and every query you ran. Change no code and change nothing in Grafana.", "connections": [ "grafana" ], "resources": [ { "url": "https://github.com/your-org/your-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back What comes back is `incident.md` on the session's artifacts and the transcript beside it. The summary names the route and the failure in one sentence, for example 500s on `POST /api/v2/query` for breakdown queries with a `time:day` dimension, while `/api/v1/stats/aggregate` kept its normal error ratio. The timeline lists the deploy, the first 5xx line in Loki with its request id, each alert rule as it fired and the moment the ratio returned to baseline. The cause section names a commit and a file under `lib/plausible/stats/`, quotes the changed clause and explains how it produces the exception the logs show. A ruled-out section follows, with the PromQL that showed ClickHouse query duration flat, or the `git log` that showed no change to `rate_limit.ex`. The last section is the full list of queries, ready to paste into Explore. A reviewer checks three things before trusting it: that the quoted LogQL returns the same lines in their own Grafana, that the blamed commit landed before the first failing line, and that the ExUnit file the agent ran actually exercises the code path named. The report recommends a next step but opens no pull request. ## Run it on every Grafana alert firing Create a Grafana contact point of type webhook that posts to a handler on your server, and route the error-ratio alert rule to it. The handler sends this request with the alert's labels, values and `startsAt` appended after the prompt, naming your Grafana connection in `connections`, and uses the alert fingerprint as `Idempotency-Key` so a repeated notification returns the run already started. The handler can instead forward the payload to a `webhook` trigger on an automation in the Project, adding the trigger's secret as `Authorization: Bearer`. Grafana resends while an alert stays firing, so give that automation a low `max_runs_per_hour`; further fires are reported as `skipped` with `rate_limited`. Set a webhook URL on the API key to receive `run.completed` with the `session_url` when the report exists, and post that link into the incident channel. ## Why run it on Rush? An on-call engineer with a Phoenix app has to hold three tools open at once: Grafana for the signal, the repository for the diff, and a terminal to compile and run the one test that would settle the question. The agent does the third part in its own sandbox with the right Erlang, Elixir and ClickHouse versions, and leaves the first two to the engineer, so the investigation runs in parallel rather than in sequence. Everything the agent concludes carries its evidence: the PromQL and LogQL it ran, the commit hashes it read, the `mix test` output it saw. Several windows or several hypotheses can be separate sessions started at once. The report is the kind a postmortem can cite, because each line can be checked by someone who was not there. ## Frequently asked questions ### Does the app have to export PromEx metrics for this to work? The metrics side does. In plausible/analytics `PROMEX_DISABLED` defaults to true in `config/runtime.exs`, so a deployment that wants Grafana dashboards sets it to false and scrapes the endpoint. Without metrics the agent still reads Loki logs and alerts, and says in the report that no request-rate series was available. ### Can it read traces as well as logs and metrics? If your Grafana stack has Tempo and the app's OpenTelemetry exporter points at it, yes: the Grafana connection reaches the datasources your account can see. The request id from `Plug.RequestId` in the log line is what links a Loki line to its trace. ### Will it touch production ClickHouse or Postgres? No. It reads Grafana and the repository, and any `mix test` it runs uses the test databases the sandbox creates with `mix do ecto.create, ecto.migrate`, as the project's own CI does. The prompt forbids code changes and changes in Grafana. ### How does it tell a rate-limit response from a real failure? By status code and module. `PlausibleWeb.Api.RateLimit` enforces an hourly and a burst limit and answers 429 with a message for the caller; those are expected under load. The report counts them separately from 500s raised inside `lib/plausible/stats/`. ### What is in the report when the cause is not in the code? The report says the diff of the last 7 days explains nothing on the failing path, shows the queries that point elsewhere, such as ClickHouse query duration rising without a deploy, and names what to check next. It does not invent a commit to blame.