# Find Noisy Grafana Alerts

An agent reads Grafana alert rules and available firing history, then returns a ranked noise audit with queries and proposed changes.

[Run this on Rush](https://cloud.getrush.ai/run/alert-noise-audit/grafana/).

## What the agent does

The agent starts with your Grafana stack, the review window and the services you want covered. It reads alert rule definitions, evaluation intervals, pending durations, labels and notification routing through the Grafana connection. Rush's connector catalog describes dashboard search, Prometheus and Loki queries, alerts and incidents. Before sending the minimal request below, add your connected Grafana MCP server's id to the submitted body's `connections` array. Naming Grafana in the prompt alone does not attach it.

For the requested 30-day window, it looks for firing and resolution transitions in the history your connection can actually read. The report records the time zone, earliest available event, pagination and any retention gaps. It groups events by rule identifier and alert-instance labels, rather than treating every label combination as a separate rule. Repeated evaluations of one continuing incident must not inflate the number of firings. If the server cannot expose historical transitions, the agent reports that limitation instead of reconstructing a month from today's status.

The audit measures firing counts, observed durations and the share of episodes that resolve within the prompt's five-minute cutoff. It separates completed episodes from alerts still firing at the window boundary. It follows notification policies before calling a rule unowned: an alert can inherit a contact point without naming one directly. A missing routing permission becomes an unknown, not a finding that nobody receives the alert. Where useful, the agent compares the rule's query with Prometheus values or relevant Loki logs to distinguish a brief spike from a sustained service problem.

For the ten leading candidates requested in the prompt, it proposes a threshold or `for` duration and includes the existing expression, the evidence behind the change and a way to validate it. A short alert can still catch a real outage, so duration alone never justifies disabling a rule. The agent writes `alert-audit.md` with separate sections for frequent alerts, short-lived episodes, routing questions and tuning proposals. It leaves rules, silences, contact points and incident state untouched.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Use the attached Grafana MCP connection to audit the authorized stack's alert rules and available firing history for the last 30 days. Write alert-audit.md ranking rules by distinct firing episodes, showing the fraction resolving within 5 minutes and checking inherited notification routing before flagging a missing contact point. For the top 10 candidates, include the original query, a proposed threshold or for duration, supporting observations and validation risks. State retention, permissions and tool limits; do not invent missing history. Read only: do not edit rules, create silences, change routing or update incidents."
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

`alert-audit.md` gives the on-call reviewer a ranked table with rule identifiers, service labels, firing counts, duration summaries, routing evidence and coverage limits. Each recommendation points back to the query and observed interval that supports it. Open episodes and incomplete history remain visible, so a partial month cannot masquerade as a complete comparison. The report distinguishes a measured observation from the agent's proposed explanation.

The reviewer checks whether the suggested pending duration would hide a short but serious outage, whether a threshold still covers the service's failure mode and whether routing assumptions match the notification policy. There is no configuration change to merge in this task. Read the answer and artifact list from `GET /o/{org}/sessions/{id}`; the session's messages expose the tool calls and outputs behind the conclusions.

## Run it on every monthly alert review

Have your own scheduler submit one session per authorized Grafana stack after the reporting window closes. Put the stack identifier, service filter and explicit start and end times in `input`, and attach that stack's connection to the call. Reuse the same `Idempotency-Key` for a retry of that request, then use a new key for the next review.

Your server can read the session when it settles or receive the key's configured completion webhook. Keep the report alongside the prior audit and compare rules whose frequency changed. A recurring request starts another analysis; it does not give the agent permission to apply last month's suggested edits.

## Why run it on Rush?

An alert review involves more than exporting a list of rule names. The agent needs space to join transitions, check label grouping and compare the expression with the metric range it describes. Rush gives that work its own sandbox, where the intermediate calculations and report can live without mixing with a developer's local investigation files.

The transcript keeps the queries and their returned evidence available for review, which matters when someone challenges a recommendation to reduce paging. You can submit separate stack audits concurrently within your API key's concurrency limit. Each result remains tied to its own request and connection, rather than becoming another untraceable spreadsheet on an on-call laptop.

## Frequently asked questions

### Does a quickly resolving alert count as noise?

It counts as a candidate for review. A brief error burst can still affect users, and a high-severity service may need a short response time. The report pairs episode duration with the query and service context before proposing a change.

### What if Grafana has less than a month of history?

The agent states the interval it can observe and leaves unavailable periods out of the calculation. It can still inspect current rule definitions and available transitions. It must not label the result a complete 30-day firing audit when retention or permissions prevent that.

### Will the agent change alerts or silence them?

This request asks for a read-only report and explicitly excludes edits and silences. Use read-scoped provider access where your setup supports it; a prompt is not a permission boundary. A tuning proposal needs a separate authorized change and review.

### Do I need to connect a GitHub repository?

No repository is involved, so the request leaves out `resources` and needs no GitHub installation. Data access comes from the Grafana MCP connection you explicitly attach through `connections`. Check that the saved connection is connected before starting the audit.