# Investigate a Goroutine Leak in a Go Service Yes. An agent reads your Go service and its recent commits, finds the goroutines that can block forever, and reports each with its file, line and a pprof check. [Run this on Rush](https://cloud.getrush.ai/run/investigate-goroutine-leak/go/), in a Project that has your repository. ## What the agent does The agent clones the repository into its own cloud sandbox and starts from the shape of the metric you give it: `go_goroutines` climbing from 300 to 40,000 over two days, memory following, then an OOM kill. Steady growth that tracks traffic means some goroutine starts per request or per message and never returns. The agent's job is to find every place a goroutine can wait forever, on a channel nobody sends to, a network call with no deadline, or a context that is never cancelled. In a service like evrone/go-clean-template it reads `go.mod` first to learn the transports: Fiber v2 for HTTP, gRPC, RabbitMQ through `amqp091-go`, NATS, and pgx v5 for Postgres. Then it reads the RPC plumbing in `pkg/`. In `pkg/rabbitmq/rmq_rpc/client/client.go` it notes that `notify` is created unbuffered at line 81, so the message loop started at line 216 blocks on `c.notify <- err` at line 219 until some caller reads it, and that `RemoteCall` starts its span from the client's own `c.ctx` at line 127 rather than the caller's context, so a caller's cancellation never reaches the wait in `remoteCallWait`. It checks how pending calls are cleaned up: `addCall` and the deferred `deleteCall` at lines 157 and 158, and the timer at line 197. It reads every outbound call on a request path the same way. `internal/repo/webapi/translation_google.go` builds a new `go-googletrans` translator on every call at line 35 and calls `trans.Translate` at line 37 with no context, so the request's deadline does not reach the HTTP call underneath; if translate.google.com stalls, each Fiber handler waiting on it stays parked. The agent checks the library's HTTP client for a timeout and notes what it finds. It also reads `pkg/postgres/postgres.go`, where `_defaultMaxPoolSize` is 1 and `.env.example` sets `PG_POOL_MAX=2`, because goroutines waiting for a pool connection pile up the same way. Last, it reads the 30 most recent commits with `git log -p` and marks any new `go` statement, channel, `context.Background()` on a request path, or dependency bump. Where it can, it runs the service's own tests (`go test ./...`) and a short load test against a handler in the sandbox while reading `runtime.NumGoroutine()` or `/debug/pprof/goroutine?debug=2`, to show which path leaves goroutines behind. It writes `incident.md` and changes no code. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "codex", "input": "Incident: Grafana shows go_goroutines on the service rising from 300 to 40,000 over two days and memory following it, until the pod is OOM killed and restarts. Read the code, the configuration and the last 30 commits, find the most likely causes, and write incident.md with a timeline, each suspect with file and line evidence, and the check that would confirm or clear it. Change no code.", "resources": [ { "url": "https://github.com/your-org/your-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back What comes back is `incident.md`, attached to the session as an artifact. After a timeline of the commits in the window, each suspect gets its own section: the file and line where the goroutine starts, the line where it can block, what it is waiting for and why that may never arrive. Each one ends with the check that settles it, usually a goroutine profile from a running pod (`go tool pprof` on `/debug/pprof/goroutine`, grouped by stack) showing thousands of goroutines parked at that exact line, or a metric such as in-flight upstream calls rising with `go_goroutines`. If the agent reproduced a leak in the sandbox, the report includes the test or load command and the goroutine counts before and after, marked as sandbox measurements. The session's transcript keeps every command and file read, so the evidence behind each suspect can be checked. Before fixing anything, an engineer captures one goroutine profile from production and compares its top stacks with the report's suspects. That one profile usually confirms or clears the top suspect, and a follow-up to the same session can ask the agent to write the fix, such as passing the request context through or adding a deadline, as a pull request. ## Run it on every goroutine count alert An alert on the rate of `go_goroutines` (for example, growth over six hours with no matching traffic growth) can call a handler on your server that sends this request with the service name and the numbers in the prompt. Leaks grow slowly, so an alert on the trend starts the investigation a day before the OOM kill rather than after it. Set a webhook URL on the API key and Rush POSTs `run.completed` when the report is ready; your handler reads the session and attaches `incident.md` to the alert or the incident ticket. ## Why run it on Rush? A goroutine leak investigation is mostly careful reading: every `go` statement, every channel send and receive, every outbound call and whether a context reaches it. The agent does that reading across the whole repository in its own sandbox, with the Go toolchain installed, so it can also run the tests and a small load test to watch the count grow, without touching your laptop's Go setup or the running service. Every claim in the report points to a file and line, and every measurement to a command and its output in the session's transcript. Several services can be investigated at once, one run each, up to the key's concurrent run limit. ## Frequently asked questions ### Can it read a goroutine dump from production? Yes, if you give it one. Attach the output of `/debug/pprof/goroutine?debug=2` as a file in the request's input and the agent matches its stacks to the code. Without one, it reports the check that would produce it. ### Does it understand errgroup, worker pools and context cancellation? It reads them as code: where an `errgroup` limit is set, which goroutine owns a channel, and whether a context passed in is the one that reaches the blocking call. In go-clean-template, for example, it notes that the RabbitMQ client and servers call `SetLimit(1)` on their errgroups, so their message loops cannot multiply. ### Will it add pprof or change my code? No. The prompt says to change no code. If the service does not expose `net/http/pprof`, the report says so and suggests where to mount it, and you can ask for that change in a follow-up.