# Investigate a FastAPI App Incident With Grafana

Yes. An agent reads the window in Grafana, maps failing paths to your FastAPI meal-plan and shopping-list routers and recent commits, and writes incident.md.

[Run this on Rush](https://cloud.getrush.ai/run/investigate-incidents-with-grafana/fastapi/), in a Project that has your repository.

## What the agent does

The agent starts with the window after the prompt and the Grafana connection. From Loki it pulls the app's access and error lines for those minutes; from Prometheus, if the deployment exports request metrics, the rate, error ratio and latency per path; and from the alerting API the rules that fired. It widens the window a few minutes each way to see whether errors began at a point or grew with load.

In a FastAPI app like Mealie the log lines are plain text, so the agent reads `mealie/core/logger/logconf.prod.json` first to know what to parse. The access formatter is `uvicorn.logging.AccessFormatter` with the pattern `[client_addr] status_code "request_line"`, the `simple` formatter is `levelname asctime - message`, and the root logger writes to stderr at WARNING, to stdout at `LOG_LEVEL` and to a rotating `${DATA_DIR}/mealie.log`. A LogQL pattern on `request_line` and `status_code` is enough to group 5xx lines by method and path. `mealie/routes/handlers.py` logs a `Start 422 Error` block only outside production, so a 422 in production logs has no detail, which the report says plainly.

The paths named in the prompt map to two routers under `mealie/routes/households/`. `controller_mealplan.py` registers `/households/mealplans` with `get_all`, `create_one`, `/today`, `/random`, `get_one`, `update_one` and `delete_one`, with rules in `controller_mealplan_rules.py` and the calendar feed in `controller_mealplan_ical.py` backed by `mealie/services/household_services/mealplan_ical_service.py`. `controller_shopping_lists.py` registers `/households/shopping/lists` with `get_all`, `create_one`, `update_one`, `delete_one`, `update_label_settings` and the ingredient-import routes, and a second router at `/households/shopping/items` with bulk `create_many`, `update_many` and `delete_many`; `publish_list_item_events` fans each change out to the event bus. The data layer is `mealie/repos/repository_shopping_list.py` and the models under `mealie/db/models/household/`.

With the path and handler known, the agent runs `git log --since=7.days` over those routers, the repository module, `mealie/schema/meal_plan` and `mealie/alembic/versions`, and reads each diff; a migration such as the one adding `mealplan_ical_token` to households is exactly the kind of change that explains a 500 on one path after a deploy. To test a hypothesis it runs the matching files under `tests/integration_tests/user_household_tests/`, such as `test_group_mealplan.py` or `test_group_shopping_list_items.py`, with `uv run pytest` on the Python 3.14 that `pyproject.toml` requires. It changes no code and nothing in Grafana.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Error rates for the FastAPI app rose between 18:10 and 18:45 UTC today. Use the Grafana connection to read that window: Loki lines from the app parsed with the uvicorn access format ([client_addr] status_code \"request_line\"), grouped by status and path, the error-level lines around them, request rate and latency from Prometheus if the deployment exports them, and any alert rules that fired. Open the repository and compare the window with the commits of the last 7 days to mealie/routes/households/controller_mealplan.py, controller_mealplan_rules.py, controller_mealplan_ical.py, controller_shopping_lists.py, mealie/repos/repository_shopping_list.py, mealie/schema/meal_plan and mealie/alembic/versions; read the diff of each commit that touches a failing path. You may run the matching files under tests/integration_tests/user_household_tests with uv run pytest to check a hypothesis. Write incident.md with a summary, a UTC timeline, the most likely cause with commit and file, the evidence for each claim (the query or command and its result), the causes you ruled out and every query you ran. Change no code and change nothing in Grafana.",
  "connections": [
    "grafana"
  ],
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is `incident.md` on the session's artifacts, with the transcript of every query and command beside it. The summary names the path and the failure, for example 500s on `PUT /api/households/shopping/items` for bulk updates while single-item updates stayed at their usual rate. The timeline puts the deploy, the first 5xx access line, the first error-level line that followed it, each alert and the recovery in UTC order.

The cause section names a commit and a file, such as a change to `update_many` in `controller_shopping_lists.py` or to `repository_shopping_list.py`, quotes the lines, and explains how they produce the error-level message the logs show. The ruled-out section lists what the agent checked and found clean, for example no change to the meal-plan router and flat latency on `/households/mealplans/today`. The last section holds every LogQL and PromQL query.

A reviewer confirms that the LogQL returns the same lines in their own Grafana, that the blamed commit was deployed before the first failing line, and that the pytest file the agent ran covers the handler named. The report ends with a recommended next step and no code change.

## Run it on every Grafana alert firing

Add a Grafana contact point of type webhook that posts to a handler on your server, and route the app's error-ratio rule to it. The handler sends this request with the alert's labels, values and `startsAt` after the prompt, your Grafana connection in `connections`, and the alert fingerprint as `Idempotency-Key`, so Grafana's repeat notifications return the run already started. The handler can also forward the payload to a `webhook` trigger on an automation in the Project, adding the trigger's secret as `Authorization: Bearer`.

Because a firing alert renotifies on an interval, keep the automation's `max_runs_per_hour` low; further fires come back as `skipped` with `rate_limited`. With a webhook URL on the API key, Rush sends `run.completed` with the `session_url` when `incident.md` exists.

## Why run it on Rush?

A self-hosted FastAPI app is often run by a small team, and the person on call is also the person who knows the code. Handing the first pass to an agent means the Loki parsing, the `git log` reading and the targeted `pytest` run all happen in a sandbox at once, while the human watches the dashboard and decides whether to roll back. Several suspects can be separate sessions.

What makes the report usable is that every line in it is checkable: the LogQL is pasted, the commit is a hash, the pytest command and its summary are in the transcript. A report like that can go straight into the incident ticket, and when it is wrong, it is wrong in a way the reader can see and correct.

## Frequently asked questions

### Mealie logs plain text, not JSON. Can the agent still group by path?

Yes. The access formatter in `logconf.prod.json` writes `[client_addr] status_code "request_line"`, so a LogQL `pattern` or `regexp` stage extracts the status and the method and path from `request_line`. The report shows the exact parsing stage it used.

### Does the app export Prometheus metrics?

Not out of the box; the repository configures logging but no metrics exporter. If your deployment adds one, or your ingress exports per-path metrics, the agent uses them. Otherwise the report builds the rate and error ratio from the Loki access lines and says so.

### Why does the prompt name the alembic versions directory?

Because a schema change that ships with a route change is a common cause of a 500 on one path right after a deploy. Reading the recent migrations beside the router diff lets the agent say whether the data model or the handler changed first.

### Will it run the whole test suite?

No. The project's `Taskfile.yml` runs `uv run pytest -n auto --dist loadfile` across everything, which is too slow for an incident. The agent runs the one or two files under `tests/integration_tests/user_household_tests/` that cover the suspect handler, and only to confirm or rule out a hypothesis.

### Can the same run also open the fix?

This run is read-only by design, so the report is an investigation and nothing more. When the team agrees with the cause, a second session can take `incident.md` as input, add a failing test and open a pull request.