What the agent does
A Grafana Cloud alert posting to a Rush webhook trigger starts the run, with the notification after the prompt as data. The agent reads the rule through the Grafana connection, reruns its PromQL around startsAt, and queries Loki for the app's error lines in the same window. A Phoenix app that uses PromEx exposes request metrics per route, so the first query usually names the controller action.
In an app like plausible/analytics, lib/plausible/prom_ex.ex loads the PromEx Application, Beam, PhoenixLiveView and Ecto plugins, the Phoenix plugin bound to PlausibleWeb.Router and PlausibleWeb.Endpoint, Oban in the environments that run it, and the app's own Plausible.PromEx.Plugins.PlausibleMetrics, which adds counters for buffered and dropped ingestion events, a cache hit counter and distributions for the ingestion pipeline steps. The dashboard's stats API lives in lib/plausible_web/controllers/api/stats_controller.ex: POST /api/stats/:domain/query is query, which parses the body with Dashboard.QueryParser.parse, builds a Query with QueryBuilder.build and answers Plausible.Stats.query(site, query), turning a QueryError into a 400; the same controller has csv_export, funnel, current_visitors, filter_suggestions and the exploration_* actions, behind date_validation_plug and validate_required_filters_plug. The agent follows the route label and the Loki lines into one of those actions and the query modules under lib/plausible/stats.
Tests for that controller are already split by area in test/plausible_web/controllers/api/stats_controller/ (top_stats_test.exs, pages_test.exs, sources_test.exs, main_graph_test.exs, query_controller_test.exs and others). Each uses PlausibleWeb.ConnCase, which checks out the SQL sandbox on Plausible.Repo, and seeds ClickHouse with populate_stats(site, [build(:pageview, ...)]) before posting to /api/stats/#{site.domain}/query. The agent adds a test in the matching file that sends the query the logs show, with the filters and date range from the request, and fails it the way production did.
The sandbox needs what .github/workflows/elixir.yml provides: Elixir 1.20.4 on OTP 28 from .tool-versions, Postgres and ClickHouse (the workflow uses postgres:18 and clickhouse/clickhouse-server:25.11.5.8-alpine, and the Makefile has postgres and clickhouse targets that start the same with Docker), mix deps.get --only test, mix compile --warnings-as-errors --all-warnings and mix do ecto.create, ecto.migrate. The test alias in mix.exs creates and migrates the database and cleans ClickHouse after the run. The agent runs the new file alone first, then mix test --include slow --include migrations --max-failures 1 --warnings-as-errors as the ce_test job does, and mix format --check-formatted.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "A Grafana alert fired for this Phoenix service; its payload follows this prompt. Use the Grafana connection to read the alert rule, rerun its PromQL around the firing time broken down by the PromEx route or action label, and query the Loki error logs for the same window. Find the cause in lib/plausible_web/controllers/api/stats_controller.ex and the query modules under lib/plausible/stats it calls. Start Postgres and ClickHouse as the Elixir CI workflow does, run mix deps.get and mix do ecto.create, ecto.migrate, add an ExUnit test in the matching file under test/plausible_web/controllers/api/stats_controller/ that posts the logged query and reproduces the failure, fix it, and run that file, then mix test --include slow --include migrations --max-failures 1 --warnings-as-errors and mix format --check-formatted. Open a pull request that includes the PromQL and LogQL queries, the log lines behind the diagnosis, and the test output before and after the fix. If the evidence does not point at code, say what it points at instead.\n\nAlert payload:\n{payload}",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
],
"connections": [
"grafana"
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request whose description shows the rule's PromQL with the route that crossed the threshold, the LogQL query with the lines it returned, and the new test failing and then passing under mix test. The diff is the test and the fix in the stats controller or a module under lib/plausible/stats, with nothing in config/.
A reviewer checks that the test's populate_stats data and query body match what the logs show, that a fix in query still answers a QueryError with a 400 rather than letting it raise, and that the change passes --warnings-as-errors. The Elixir workflow then runs its six partitions on the pull request in both test and ce_test.
When the evidence points away from code, for example at ClickHouse timing out under one site's export or at an Oban queue falling behind, the run says so with the series and the log lines it found and opens no pull request.
Run it on every Grafana alert on the stats API
Add an automation to the Project with this prompt and a webhook trigger; Rush shows the URL and secret once. In Grafana Cloud, make a webhook contact point at that URL with Authorization: Bearer <secret> and route the service's alert rules to it. The notification body becomes the event under the prompt.
Skip resolved notifications and set a repeat interval longer than a run, because each delivery without an Idempotency-Key header is a separate run. The automation's max_runs_per_hour caps a rule that flaps during a deploy.
The use case this job follows, with its automation and trigger: Fix a bug from an alert.
Why run it on Rush?
Reproducing a stats API failure by hand means two databases, a ClickHouse seed and the exact query body from production, which few developers keep set up. A Rush run starts both stores in its own sandbox from the same images the workflow uses, replays the query in an ExUnit test, and keeps every mix command with its output in the session.
The pull request carries the PromQL, the LogQL and the test output, so the diagnosis can be checked line by line. Alerts on several routes run in parallel, each in its own sandbox, and none of them touches the production ClickHouse.
Frequently asked questions
My Phoenix app does not use PromEx. Can it still find the route?
Yes, from the logs. Phoenix's request logging names the controller and action, so the agent reads those from Loki for the alert window and continues the same way; only the first PromQL breakdown is less precise.
Does it need ClickHouse in the sandbox?
For this app, yes: the stats controller tests seed ClickHouse through populate_stats, and the test alias cleans it afterwards. The agent starts the version the workflow names and says in the pull request which image it used.
Which Elixir and OTP does it use?
The ones .tool-versions pins, Elixir 1.20.4 on OTP 28, which is also what the workflow installs through erlef/setup-beam. mix.exs requires Elixir 1.18 or later.
Will it run the whole suite with all six partitions?
It runs the new file alone first, then the full mix test with the same include flags as the ce_test job in one partition. The pull request's own workflow run covers the test environment with MinIO, which the agent does not start.