Investigate GenServer Timeouts in a Phoenix App

The agent clones the repository into its own cloud sandbox and starts from the two errors in the prompt: exits with GenServer.call timing out after 5000 ms, and DBConnection.ConnectionError, both on the package publishing path since the morning's deploy. In a Phoenix app like hexpm, the code behind hex.pm, it reads mix.exs (Elixir ~> 1.20, phoenix, ecto_sql, postgrex, oban ~> 2.23) and follows a publish from lib/hexpm_web/controllers/api/release_controller.ex into Hexpm.Repository.Releases.publish/8 in lib/hexpm/repository/releases.ex, which builds an Ecto.Multi and commits it with Repo.transaction(timeout: @publish_timeout), where @publish_timeout is 60_000.

Then it lists every GenServer.call in lib/ and checks which ones a publish reaches and what timeout each one has. A call with no third argument uses the 5000 ms default, which is the number in the error. In hexpm, Hexpm.TmpDir.track/1 in lib/hexpm/tmp_dir.ex makes exactly that call, GenServer.call(__MODULE__, {:monitor, pid}), every time tmp_file/1 or tmp_dir/1 runs, and the release controller calls Hexpm.TmpDir.tmp_dir("release-tarball") to unpack each tarball. TmpDir is a single named process whose handle_info({:DOWN, …}) deletes a finished process's files with File.rm_rf before it answers the next message, so a slow delete of a large unpacked tarball holds every caller queued behind it. The agent records that as a suspect with the lines, and says what would confirm it: the TmpDir process's message queue length during the spike.

For the connection errors it reads the pool. lib/hexpm/repo.ex takes HEXPM_DATABASE_POOL_SIZE from the environment in init/2, so the production pool size is set outside the repository, and the agent says so instead of guessing a number. It then counts what holds connections during a publish: the 60-second publish transaction, and Oban jobs such as Hexpm.Repository.RegistryWorker, which runs each build inside Repo.transaction with a timeout of up to 30 minutes for a full build and waits up to registry_lock_wait (60_000 in config/config.exs) for its advisory locks. A burst of publishes that each enqueue registry work can leave the web requests waiting for a connection, which surfaces as DBConnection.ConnectionError.

Last it reads the 30 most recent commits with git log and git show, looking for changes to these files, to pool or queue settings in config/prod.exs and config/runtime.exs, and to anything new on the publish path. It changes no code. Where a suspect needs production data to confirm (Sentry's breadcrumbs, the pool's queue time, Oban's queue length), it writes the exact query or check rather than claiming the answer.

What comes back is incident.md as an artifact on the session, which the session read returns with a download link that is good for an hour. It opens with a timeline built from the commits since the last deploy and the times in the prompt, then ranks the suspects. Each suspect names the file and line, quotes the code, explains how it would produce a 5000 ms GenServer.call exit or a connection error, and ends with the check that confirms or clears it, such as reading Process.info(pid, :message_queue_len) for the named process or the pool's queue time from telemetry.

It also carries two tables the on-call engineer can use directly: every GenServer.call the publish path reaches with its timeout, and every place that holds a database connection during a publish with how long it can hold it. The session's transcript keeps every command the agent ran, every file it read and the git log output, so each line in the report can be traced to where it came from.

Before acting on it, the engineer runs the confirming checks against production, starting with the top suspect, and decides on the fix. If one is confirmed, a follow-up message to the same session can ask the agent to write the change and open a pull request, since its sandbox still has the repository and everything it learned.

Sentry can send an alert to a webhook when an issue's event count crosses a threshold. A small handler on your server turns that alert into this request, with the error, the time window and the deploy in the prompt, so the report is waiting by the time someone opens the page.

Set a webhook URL on the API key and Rush POSTs run.completed when the turn finishes, with the session_url. The handler reads the session, takes the link to incident.md from its artifacts and posts it to the incident channel or the Sentry issue.

During an incident the on-call engineer's laptop is busy with dashboards and a shell to production. Reading a large Phoenix codebase for every GenServer.call, every Repo.transaction timeout and every config change since the deploy is slow work to do by hand at the same time. In a Rush sandbox the agent does that reading on its own copy of the repository while the engineer keeps watching production.

The report is useful because it is cited: each suspect points at a line and comes with a check, and the transcript shows every command behind it. Several investigations can run at once, for example one per error class or one per recent deploy, up to the key's concurrent run limit, and none of them touches production or your code.

Does the agent connect to my production system?

Why focus on the 5000 ms number?

Will it change any code?

What if the pool size is not in the repository?