# Fix a Phoenix App From a Grafana Alert Yes. The alert starts a run that reruns its PromEx and Loki queries, finds the failing action in your Phoenix stats API and opens a pull request with an ExUnit test. [Run this on Rush](https://cloud.getrush.ai/run/fix-grafana-alerts/phoenix/), in a Project that has your repository. ## What the agent does A Grafana Cloud alert posting to a Rush webhook trigger starts the run, with the notification after the prompt as data. The agent reads the rule through the Grafana connection, reruns its PromQL around `startsAt`, and queries Loki for the app's error lines in the same window. A Phoenix app that uses PromEx exposes request metrics per route, so the first query usually names the controller action. In an app like plausible/analytics, `lib/plausible/prom_ex.ex` loads the PromEx `Application`, `Beam`, `PhoenixLiveView` and `Ecto` plugins, the `Phoenix` plugin bound to `PlausibleWeb.Router` and `PlausibleWeb.Endpoint`, `Oban` in the environments that run it, and the app's own `Plausible.PromEx.Plugins.PlausibleMetrics`, which adds counters for buffered and dropped ingestion events, a cache hit counter and distributions for the ingestion pipeline steps. The dashboard's stats API lives in `lib/plausible_web/controllers/api/stats_controller.ex`: `POST /api/stats/:domain/query` is `query`, which parses the body with `Dashboard.QueryParser.parse`, builds a `Query` with `QueryBuilder.build` and answers `Plausible.Stats.query(site, query)`, turning a `QueryError` into a 400; the same controller has `csv_export`, `funnel`, `current_visitors`, `filter_suggestions` and the `exploration_*` actions, behind `date_validation_plug` and `validate_required_filters_plug`. The agent follows the route label and the Loki lines into one of those actions and the query modules under `lib/plausible/stats`. Tests for that controller are already split by area in `test/plausible_web/controllers/api/stats_controller/` (`top_stats_test.exs`, `pages_test.exs`, `sources_test.exs`, `main_graph_test.exs`, `query_controller_test.exs` and others). Each uses `PlausibleWeb.ConnCase`, which checks out the SQL sandbox on `Plausible.Repo`, and seeds ClickHouse with `populate_stats(site, [build(:pageview, ...)])` before posting to `/api/stats/#{site.domain}/query`. The agent adds a test in the matching file that sends the query the logs show, with the filters and date range from the request, and fails it the way production did. The sandbox needs what `.github/workflows/elixir.yml` provides: Elixir 1.20.4 on OTP 28 from `.tool-versions`, Postgres and ClickHouse (the workflow uses `postgres:18` and `clickhouse/clickhouse-server:25.11.5.8-alpine`, and the `Makefile` has `postgres` and `clickhouse` targets that start the same with Docker), `mix deps.get --only test`, `mix compile --warnings-as-errors --all-warnings` and `mix do ecto.create, ecto.migrate`. The `test` alias in `mix.exs` creates and migrates the database and cleans ClickHouse after the run. The agent runs the new file alone first, then `mix test --include slow --include migrations --max-failures 1 --warnings-as-errors` as the `ce_test` job does, and `mix format --check-formatted`. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "claude", "input": "A Grafana alert fired for this Phoenix service; its payload follows this prompt. Use the Grafana connection to read the alert rule, rerun its PromQL around the firing time broken down by the PromEx route or action label, and query the Loki error logs for the same window. Find the cause in lib/plausible_web/controllers/api/stats_controller.ex and the query modules under lib/plausible/stats it calls. Start Postgres and ClickHouse as the Elixir CI workflow does, run mix deps.get and mix do ecto.create, ecto.migrate, add an ExUnit test in the matching file under test/plausible_web/controllers/api/stats_controller/ that posts the logged query and reproduces the failure, fix it, and run that file, then mix test --include slow --include migrations --max-failures 1 --warnings-as-errors and mix format --check-formatted. Open a pull request that includes the PromQL and LogQL queries, the log lines behind the diagnosis, and the test output before and after the fix. If the evidence does not point at code, say what it points at instead.\n\nAlert payload:\n{payload}", "resources": [ { "url": "https://github.com/your-org/your-app" } ], "connections": [ "grafana" ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back What comes back is a pull request whose description shows the rule's PromQL with the route that crossed the threshold, the LogQL query with the lines it returned, and the new test failing and then passing under `mix test`. The diff is the test and the fix in the stats controller or a module under `lib/plausible/stats`, with nothing in `config/`. A reviewer checks that the test's `populate_stats` data and query body match what the logs show, that a fix in `query` still answers a `QueryError` with a 400 rather than letting it raise, and that the change passes `--warnings-as-errors`. The Elixir workflow then runs its six partitions on the pull request in both `test` and `ce_test`. When the evidence points away from code, for example at ClickHouse timing out under one site's export or at an Oban queue falling behind, the run says so with the series and the log lines it found and opens no pull request. ## Run it on every Grafana alert on the stats API Add an automation to the Project with this prompt and a `webhook` trigger; Rush shows the URL and secret once. In Grafana Cloud, make a webhook contact point at that URL with `Authorization: Bearer ` and route the service's alert rules to it. The notification body becomes the event under the prompt. Skip resolved notifications and set a repeat interval longer than a run, because each delivery without an `Idempotency-Key` header is a separate run. The automation's `max_runs_per_hour` caps a rule that flaps during a deploy. ## Why run it on Rush? Reproducing a stats API failure by hand means two databases, a ClickHouse seed and the exact query body from production, which few developers keep set up. A Rush run starts both stores in its own sandbox from the same images the workflow uses, replays the query in an ExUnit test, and keeps every `mix` command with its output in the session. The pull request carries the PromQL, the LogQL and the test output, so the diagnosis can be checked line by line. Alerts on several routes run in parallel, each in its own sandbox, and none of them touches the production ClickHouse. ## Frequently asked questions ### My Phoenix app does not use PromEx. Can it still find the route? Yes, from the logs. Phoenix's request logging names the controller and action, so the agent reads those from Loki for the alert window and continues the same way; only the first PromQL breakdown is less precise. ### Does it need ClickHouse in the sandbox? For this app, yes: the stats controller tests seed ClickHouse through `populate_stats`, and the `test` alias cleans it afterwards. The agent starts the version the workflow names and says in the pull request which image it used. ### Which Elixir and OTP does it use? The ones `.tool-versions` pins, Elixir 1.20.4 on OTP 28, which is also what the workflow installs through `erlef/setup-beam`. `mix.exs` requires Elixir 1.18 or later. ### Will it run the whole suite with all six partitions? It runs the new file alone first, then the full `mix test` with the same include flags as the `ce_test` job in one partition. The pull request's own workflow run covers the `test` environment with MinIO, which the agent does not start.