# Investigate a Latency Regression in a Go Service

An agent compares your Datadog traces before and after the rise, finds the span that grew, maps it to the Go code, and writes a cited report.

[Run this on Rush](https://cloud.getrush.ai/run/investigate-latency-regression/go/), in a Project that has your repository.

## What the agent does

The agent starts from the traces, not from a guess. Through the Datadog connection named on the run, it searches APM for the service over the week before the rise and the days after, compares p95 duration grouped by resource name, and finds the span that grew. It reads the flame graph for that span to see whether the time sits in the handler, in a middleware, or in a downstream call, and it copies the exact queries it ran into the report so anyone can re-run them.

Then it maps the grown span to the code in a repository like pocketbase/pocketbase. PocketBase serves HTTP from a custom router in tools/router that wraps the standard library http.ServeMux, and the middleware stack is bound in order in apis/base.go: an activityLogger that stamps the request start and logs execTime in milliseconds, panicRecover, rateLimit, loadAuthToken (a database lookup per request when a token is present), superuserIPsWhitelist, securityHeaders, and a 32 MB body limit. A span that grew for every route points at that stack; a span that grew for one route points at the handler file in apis/, one file pair per endpoint such as apis/record_crud.go or apis/realtime.go.

For a handler-level regression, it reads where the time actually goes. PocketBase keeps its data in SQLite through the pure-Go modernc.org/sqlite driver, with pool defaults of 120 max open and 15 idle connections for the main database in core/base.go and a second nonconcurrentDB pinned to a single connection, so a grown span with most of its time waiting maps to pool contention, not CPU. Long-lived work sits in the realtime SSE broker with per-connection idle and max timers, the crontab-like service in tools/cron, and a pooled goja runtime for JS hooks that creates extra one-off VMs when the pool is busy; the agent checks whether the regression window lines up with traffic on those paths.

The prompt says to change no code, and it doesn't. It writes a report that names the span, the before and after numbers, the flame graph reading, the file and function the span maps to, and the likely cause ranked against the alternatives, with the Datadog queries quoted so the on-call engineer can verify each claim in two minutes.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "p95 latency for the service below rose this week. Through the Datadog connection, compare traces before and after the rise, find the span that grew, map it to the code in this repository, and write a report with the queries, the numbers and the likely cause. Change no code.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a report, the run's artifact. It opens with the Datadog queries the agent ran and the time windows they cover, then a per-span table of p95 before and after the rise that isolates the one span that grew. A flame graph reading follows, then the mapping to code: the middleware stack or handler file, the database pool settings, and any background worker whose timers line up with the regression window.

Each suspect carries its evidence: the query that shows it, the file and line in the repository, and the check that would confirm or clear it, such as comparing pool wait time against query time in Datadog. The trajectory of the run keeps every query and its raw answer, so a reviewer can replay the investigation step by step instead of trusting a summary.

Before acting on it, the on-call engineer checks that the cited files exist at the lines named, that the numbers in the report reproduce from the quoted queries, and that the cause explains the timing, not just the symptom. If the fix is wanted, a follow-up run with the same repository and a change request takes the report as its starting point.

## Run it on every latency alert fires

Point a Datadog monitor at the service's p95 and give it a webhook to your server. When the monitor fires, your server posts this same request with the alert window in the prompt, so every regression gets its own investigation within minutes of the curve moving instead of waiting for someone to notice a dashboard.

Set a webhook URL on the API key and Rush posts run.completed or run.failed to it when the run's turn finishes, with the session URL and the report; your server can thread it back into the alert or the incident channel. Name the Datadog connection on every run, since a key run reads only the connections it names.

## Why run it on Rush?

On a laptop this investigation is a slog of browser tabs: the APM query, the flame graph, the repository checkout, the middleware file, each context switch losing the thread. On Rush the agent does it in one sandbox with the repository cloned and the Datadog connection live, and every query it runs is kept with its output in the trajectory, so the report cites evidence rather than memory.

The result is checkable by design. The report names the queries, the numbers and the file:line for each suspect, and the trajectory holds the raw answers, which is what a reviewer needs before paging someone at night. And because one request starts one run, a fleet of services with a rising p95 each gets its own investigation at the same time, each with its own report, instead of queueing behind whoever is on call.

## Frequently asked questions

### How does the agent reach my Datadog traces?

You connect Datadog once from the console, then name the connection on the run; a key run reads only the connections it names. The agent uses it to search APM traces, metrics and the flame graph for the service, and the report quotes every query it ran so you can re-run them yourself.

### Will it change any code?

No. The prompt says to change no code, and the report is the whole result. If you want the fix afterwards, start a follow-up run with the repository and describe the change; it works from the same evidence.

### What if the regression is not in my code?

The flame graph reading separates handler time from middleware time from downstream calls. If the grown span is a database wait, the agent reports the pool settings and contention evidence; if it is an upstream dependency, it says which call grew and what to check on that side. It does not force a code answer onto an infrastructure cause.

### My service is not PocketBase. Does the page still apply?

Yes. The shape is the same for any Go service: find the span that grew, read its flame graph, map it to the router, middleware, handlers and pool configuration in your repository. The PocketBase details above are an example of the specificity the report reaches for the repository you name in resources.

### How current does the trace data need to be?

The comparison needs APM retention to cover both the before and after windows, which for a regression that started this week is comfortably inside Datadog's default retention. The prompt takes the window from you, so set it to span the last known-good period and the rise.