# Fix a FastAPI App Bug From a Datadog Monitor Alert

Yes. An agent reads the Datadog monitor, logs and traces behind an alert, finds the FastAPI route at fault, fixes it with a pytest test and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/fix-datadog-alerts/fastapi/), in a Project that has your repository.

## What the agent does

The agent begins with the monitor payload your server passes in and asks Datadog, through its MCP server, for the rest: the monitor's query and thresholds, the error logs for that service and environment in the half hour around the alert, and the APM traces for the failing requests. For a FastAPI service, a trace's resource name carries the route template, such as `GET /api/v1/items/{id}`, and its error span carries the exception type and the Python traceback. The agent groups those by the innermost frame in your own code, so ten thousand identical 500s become one problem with a sample request.

It then finds the route. In an app built on fastapi/full-stack-fastapi-template the routers live in `backend/app/api/routes/` (`items.py`, `users.py`, `login.py`, `private.py`, `utils.py`), mounted under `API_V1_STR = "/api/v1"` from `app/core/config.py`. A failing `GET /api/v1/items/{id}` maps to `read_item`, which calls `session.get(Item, id)` and raises `HTTPException(status_code=404, detail="Item not found")` or `403` with `"Not enough permissions"`. An unhandled 500 there means something before those checks broke, so the agent reads the SQLModel models in `app/models.py`, the helpers in `app/crud.py` and the latest Alembic revision before it decides where the fault is.

It reproduces the failure as a pytest test before it fixes anything. The template's suite in `backend/tests/` uses FastAPI's `TestClient`, a session-scoped `db` fixture that calls `init_db` and deletes every `Item` and `User` at the end, and `superuser_token_headers` and `normal_user_token_headers` fixtures, so the new test in `tests/api/routes/test_items.py` or `test_users.py` sends the same request the trace shows, with the same kind of user. The tests need Postgres (`compose.yml` runs `postgres:18`), so the agent brings the database up, runs `alembic upgrade head` and `python app/initial_data.py` the way `scripts/prestart.sh` does, and installs with uv for the Python 3.14 that `.python-version` names.

With the test failing for the same reason the alert fired, the agent makes the smallest fix that handles the input class, runs the new test, then `bash scripts/tests-start.sh`, which runs the whole suite under coverage. The template's CI fails under 90 percent coverage (`coverage report --fail-under=90`), so the test also keeps the new branch covered.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "A Datadog monitor on this FastAPI service fired; its webhook payload is at the end of this prompt. Use the Datadog connection to read the monitor, the error logs and the APM traces for its service and environment from 15 minutes before to 15 minutes after it fired. Group the errors by exception and the innermost frame in this repository, map each trace's route template to its router in backend/app/api/routes, and find the code path that produced them. Start the database the tests need, apply the Alembic migrations, and write a pytest regression test with the existing TestClient and token fixtures that sends the request the traces show and fails for the same reason. Fix the cause, run the full backend test suite, and open a pull request that quotes the log lines and trace ids the fix addresses. If the cause is not in this code, change nothing and answer with what the logs show.\n\nPayload: <Datadog webhook body>",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

When the fault is in the service, the agent opens a pull request with the fix in the router, CRUD helper or model it traced, and a regression test in `backend/tests/api/routes/`. Its description names the monitor, quotes the Datadog log lines and trace ids it reproduced, shows the request the test sends, and lists the pytest and coverage commands with their result. If the fix needs a schema change, the pull request includes a new Alembic revision and says so at the top.

When the logs show a cause outside the code, such as Postgres refusing connections or a missing `SECRET_KEY` in one environment, the run makes no change and its answer explains what Datadog showed and where to look. That answer arrives on your webhook with the run, so the alert channel still learns what happened.

Before merging, a reviewer checks that the test fails on the current main for the alert's reason, that the fix does not widen what a normal user can read or change (the `403` checks in `items.py` are the line to watch), and that any migration runs forward cleanly on a copy of production data.

## Run it on every Datadog error-rate alert

Create a webhook in Datadog's webhook integration pointing at your server and mention it in each monitor you want handled, such as an error-rate monitor on `service:backend`. When it moves to Alert, your server adds the payload to this prompt and sends the request with the repository in `resources`. Ignore Recovered and No Data transitions so only real failures start a run.

Rush POSTs `run.completed` or `run.failed` to the webhook URL set on your API key when the turn ends, with the answer and the session URL, which your server can forward to the incident or the on-call channel. Keep the alert id with the returned `execution_id` so a monitor that re-notifies during the same incident does not start a second run.

## Why run it on Rush?

Reproducing a production error locally means matching Python, uv, Postgres and migrations to what the service runs, and that is exactly the work nobody wants to do at night. The agent does it in its own sandbox: its own clone, its own database, its own Datadog session. Your laptop and your local database are not involved, and the run continues whether or not anyone is awake.

Every claim in the pull request can be checked. The trajectory keeps each Datadog query and its response, each migration and each pytest run with its output, and the regression test proves the fix against the request the traces captured. Several alerts in one incident each get their own run and their own sandbox, and a person still decides what merges.

## Frequently asked questions

### How does the agent read Datadog?

With Datadog's own MCP server, one of the connectors Rush lists. Connect it once through `POST /me/connections` with `"type": "mcp"` and OAuth, sign in, and add `"connections": ["datadog"]` to the request body. The listed server is the US1 site; another Datadog site is added as your own MCP server with its URL.

### Will it touch my production database?

No. It reads logs and traces through Datadog, and runs the tests against a database it starts in its own sandbox. It never needs your database credentials for this job.

### We use Sentry for errors. Does that matter?

The template initialises `sentry_sdk` when `SENTRY_DSN` is set, and that can run beside Datadog. This job reads Datadog; if your errors are only in Sentry, connect Sentry's MCP server instead and change the prompt to read issues from it.

### What if the route is async and the error is a timeout?

It reads the trace's span durations to find which call ran long, such as a query or an outbound `httpx` request, and checks whether the code awaits it without a limit. A timeout caused by load or an upstream outage is reported in the answer rather than patched over with a longer timeout.