# Investigate Database Connection Pool Exhaustion in a FastAPI App Yes. An agent traces how your FastAPI app opens and holds database connections and writes incident.md with ranked, file:line-cited suspects. [Run this on Rush](https://cloud.getrush.ai/run/investigate-connection-pool-exhaustion/fastapi/), in a Project that has your repository. ## What the agent does The agent starts with the number that matches the symptom. "QueuePool limit reached" and requests that time out after 30 s point at SQLAlchemy's pool: the default `QueuePool` holds 5 connections plus 10 overflow, and a request waits `pool_timeout`, 30 seconds by default, for one before it raises. In an app built on the official full-stack-fastapi-template, the engine is one line, `backend/app/core/db.py` line 7: `create_engine(str(settings.DATABASE_URL), pool_pre_ping=True)`, with no `pool_size`, `max_overflow` or `pool_timeout`, so every default applies. The agent records that as the ceiling before looking for what fills it. Next it traces how a request gets and keeps a connection. `backend/app/api/deps.py` defines `get_db` as `with Session(engine) as session: yield session` and `SessionDep` as a dependency on it, so the session, and the connection it checks out on its first query, lives until the response is sent. Every route in `app/api/routes/` is a plain `def`, which FastAPI runs in a thread pool, so many requests can hold sessions at once while the pool has 15 connections per process. `backend/Dockerfile` ends with `CMD ["fastapi", "run", "--workers", "4"]`, four processes with a pool each, which the agent multiplies out against the database's `max_connections` and any connection limit in front of it. Then it looks for routes that hold a connection while waiting on something else. `recover_password` in `routes/login.py` (line 54) and `create_user` in `routes/users.py` (line 57) take a `SessionDep`, query, then call `send_email`, which in `app/utils.py` line 56 does `message.send(to=email_to, smtp=smtp_options)` with the `emails` package; the agent checks whether that SMTP call sets a timeout and whether the query before it has already checked out the connection. A slow mail server then turns every password reset into a connection held for as long as SMTP takes. It also reads `read_items` in `routes/items.py`, which runs a `count` and a paged `select` per request, for how long each holds its connection under load. Last, it reads the last 30 commits with their diffs for anything that changed the engine, the session dependency, a route's signature from `async def` to `def` or back, the worker count or the Postgres image in `compose.yml`. With the Sentry connection named on the run, it reads the error events and the traces Sentry collects (`sentry_sdk.init(..., enable_tracing=True)` in `app/main.py` line 19) to see which endpoints were slow when the errors started. It writes `incident.md` and changes no code. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "codex", "input": "Incident: Sentry shows a burst of QueuePool limit reached errors from the API under modest load, and requests time out after 30 s. Read how the engine and its pool are configured, how a request gets and releases a session, which routes hold a session while doing other slow work such as sending email, how many worker processes run, and the last 30 commits with their diffs. If the Sentry connection is available, read the error events and the slowest transactions around the start of the burst. Find the most likely causes and write incident.md with a timeline, each suspect ranked with file and line evidence, the arithmetic of connections per process against the database limit, and the exact query, setting or test that would confirm or clear each one. Change no code.", "resources": [ { "url": "https://github.com/your-org/your-fastapi-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back What comes back is `incident.md` as an artifact on the session, with a download link, and the agent's answer naming the leading suspect. The report has a timeline of relevant commits and, when Sentry was read, when the errors began and which endpoints were slow first. It has a short capacity section that does the arithmetic from the code: pool size and overflow per process, processes per container, containers if the repository says, against Postgres's `max_connections`. Then the suspects, ranked, each anchored to a file and line: the default pool at `core/db.py` line 7, the session held across `send_email` in `routes/login.py` and `routes/users.py`, the thread pool that lets more `def` routes run at once than the pool has connections. Each ends with its check: `SELECT state, count(*) FROM pg_stat_activity GROUP BY state` during load to see connections idle in a transaction, `engine.pool.status()` logged per request, a load test against `/api/v1/login/password-recovery/{email}` with a blackholed SMTP host to see whether it reproduces the error, or the Sentry trace for that endpoint. The trajectory keeps every file the agent read and every command it ran, so a reviewer can trace each claim to its source. The report recommends where to look first; it does not change the pool settings or the code, because a bigger pool can hide a connection held too long rather than fix it. ## Run it on every Sentry issue for QueuePool errors Add a Sentry alert rule for new issues matching `QueuePool limit` or `TimeoutError` and send its webhook to your own server, which starts this request with the issue's title, count and first-seen time in `input` and the repository in `resources`. The report is then ready when the on-call engineer opens the alert, and every burst gets its own run, so a recurrence a week later is compared against fresh code and commits. Give the API key a webhook URL and Rush POSTs `run.completed` when the turn finishes, with the agent's answer and the `session_url`. Your server can read the session to get the `incident.md` artifact link and post it on the Sentry issue or in the incident channel. ## Why run it on Rush? Pool exhaustion is a question about how the whole request path holds connections, which means reading the engine, the dependency, every route and the deploy config together while the errors are still coming in. On Rush that reading runs in the agent's own sandbox with its own clone, alongside the people working the incident, and it touches nothing in production. The report is checkable at every line. Each suspect cites a file and line, the capacity math is shown, and each check is a query or command someone can run. The trajectory records what the agent read for each claim. Several services, or several incidents, can be investigated at once in separate sandboxes, each producing its own report. ## Frequently asked questions ### Does the agent connect to our database? No. It reads the repository and, if you name it, the Sentry connection. The queries in the report, such as the `pg_stat_activity` one, are written for someone with database access to run. That keeps the investigation read-only. ### How does it read Sentry? Sentry is in Rush's MCP catalog: connect it once with `POST /me/connections`, then name it in the request's `connections`. A key run gets only the connections it names. Without it, the agent works from the code and from the error text and numbers you put in `input`. ### Won't it just say "increase pool_size"? The request asks for causes with evidence, and the report says why a larger pool is a mitigation rather than a fix when a route holds its connection across slow work. If the arithmetic shows the pool really is too small for the traffic, it says that too, with the numbers. ### Does it matter whether my routes are def or async def? Yes, and the agent checks it. A plain `def` route runs in FastAPI's thread pool, so many can hold a session at once; an `async def` route that calls a blocking database driver can stall the event loop instead. The report says which shape your routes have and what that means for the pool.