# Investigate GenServer Timeouts in a Phoenix App

Yes. An agent reads your Phoenix code, config and recent commits and writes an incident report ranking each GenServer timeout suspect with file:line evidence.

[Run this on Rush](https://cloud.getrush.ai/run/investigate-genserver-timeouts/phoenix/), in a Project that has your repository.

## What the agent does

The agent clones the repository into its own cloud sandbox and starts from the two errors in the prompt: exits with `GenServer.call` timing out after 5000 ms, and `DBConnection.ConnectionError`, both on the package publishing path since the morning's deploy. In a Phoenix app like hexpm, the code behind hex.pm, it reads `mix.exs` (Elixir `~> 1.20`, `phoenix`, `ecto_sql`, `postgrex`, `oban ~> 2.23`) and follows a publish from `lib/hexpm_web/controllers/api/release_controller.ex` into `Hexpm.Repository.Releases.publish/8` in `lib/hexpm/repository/releases.ex`, which builds an `Ecto.Multi` and commits it with `Repo.transaction(timeout: @publish_timeout)`, where `@publish_timeout` is `60_000`.

Then it lists every `GenServer.call` in `lib/` and checks which ones a publish reaches and what timeout each one has. A call with no third argument uses the 5000 ms default, which is the number in the error. In hexpm, `Hexpm.TmpDir.track/1` in `lib/hexpm/tmp_dir.ex` makes exactly that call, `GenServer.call(__MODULE__, {:monitor, pid})`, every time `tmp_file/1` or `tmp_dir/1` runs, and the release controller calls `Hexpm.TmpDir.tmp_dir("release-tarball")` to unpack each tarball. `TmpDir` is a single named process whose `handle_info({:DOWN, …})` deletes a finished process's files with `File.rm_rf` before it answers the next message, so a slow delete of a large unpacked tarball holds every caller queued behind it. The agent records that as a suspect with the lines, and says what would confirm it: the `TmpDir` process's message queue length during the spike.

For the connection errors it reads the pool. `lib/hexpm/repo.ex` takes `HEXPM_DATABASE_POOL_SIZE` from the environment in `init/2`, so the production pool size is set outside the repository, and the agent says so instead of guessing a number. It then counts what holds connections during a publish: the 60-second publish transaction, and Oban jobs such as `Hexpm.Repository.RegistryWorker`, which runs each build inside `Repo.transaction` with a timeout of up to 30 minutes for a full build and waits up to `registry_lock_wait` (`60_000` in `config/config.exs`) for its advisory locks. A burst of publishes that each enqueue registry work can leave the web requests waiting for a connection, which surfaces as `DBConnection.ConnectionError`.

Last it reads the 30 most recent commits with `git log` and `git show`, looking for changes to these files, to pool or queue settings in `config/prod.exs` and `config/runtime.exs`, and to anything new on the publish path. It changes no code. Where a suspect needs production data to confirm (Sentry's breadcrumbs, the pool's queue time, Oban's queue length), it writes the exact query or check rather than claiming the answer.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Incident: Sentry shows a spike of exits with GenServer.call timeout after 5000ms and DBConnection.ConnectionError from the package publishing path since this morning's deploy. Read the code, the configuration and the last 30 commits, find the most likely causes, and write incident.md with a timeline, each suspect with file and line evidence, and the check that would confirm or clear it. List every GenServer.call the publish path reaches and its timeout, and every place that holds a database connection during a publish. Change no code.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is `incident.md` as an artifact on the session, which the session read returns with a download link that is good for an hour. It opens with a timeline built from the commits since the last deploy and the times in the prompt, then ranks the suspects. Each suspect names the file and line, quotes the code, explains how it would produce a 5000 ms `GenServer.call` exit or a connection error, and ends with the check that confirms or clears it, such as reading `Process.info(pid, :message_queue_len)` for the named process or the pool's queue time from telemetry.

It also carries two tables the on-call engineer can use directly: every `GenServer.call` the publish path reaches with its timeout, and every place that holds a database connection during a publish with how long it can hold it. The session's transcript keeps every command the agent ran, every file it read and the `git log` output, so each line in the report can be traced to where it came from.

Before acting on it, the engineer runs the confirming checks against production, starting with the top suspect, and decides on the fix. If one is confirmed, a follow-up message to the same session can ask the agent to write the change and open a pull request, since its sandbox still has the repository and everything it learned.

## Run it on every Sentry alert on a timeout spike

Sentry can send an alert to a webhook when an issue's event count crosses a threshold. A small handler on your server turns that alert into this request, with the error, the time window and the deploy in the prompt, so the report is waiting by the time someone opens the page.

Set a webhook URL on the API key and Rush POSTs `run.completed` when the turn finishes, with the `session_url`. The handler reads the session, takes the link to `incident.md` from its artifacts and posts it to the incident channel or the Sentry issue.

## Why run it on Rush?

During an incident the on-call engineer's laptop is busy with dashboards and a shell to production. Reading a large Phoenix codebase for every `GenServer.call`, every `Repo.transaction` timeout and every config change since the deploy is slow work to do by hand at the same time. In a Rush sandbox the agent does that reading on its own copy of the repository while the engineer keeps watching production.

The report is useful because it is cited: each suspect points at a line and comes with a check, and the transcript shows every command behind it. Several investigations can run at once, for example one per error class or one per recent deploy, up to the key's concurrent run limit, and none of them touches production or your code.

## Frequently asked questions

### Does the agent connect to my production system?

No. It reads the repository, its configuration and its git history in a sandbox, and writes the queries and checks for you to run. If you want it to read Sentry directly, connect the Sentry MCP server and name it on the run.

### Why focus on the 5000 ms number?

It is the default timeout of `GenServer.call/2` in Elixir. A call that names no timeout uses it, so the agent lists those calls first. A call with an explicit timeout of 5000, such as `TmpDir.await_cleanup/1` in hexpm, is listed too.

### Will it change any code?

No. The prompt says to change no code and the result is a Markdown report. Once you know the cause, you can ask the same session for a fix and a pull request.

### What if the pool size is not in the repository?

The report says where it is read from, such as `HEXPM_DATABASE_POOL_SIZE` in `lib/hexpm/repo.ex`, and asks you to check the live value. It does not assume a number it cannot read.