# Investigate a Go Service Incident With Datadog Yes. An agent reads the incident window in Datadog, lines it up with recent commits to your Go handlers and writes incident.md with cited evidence. [Run this on Rush](https://cloud.getrush.ai/run/investigate-incidents-with-datadog/go/), in a Project that has your repository. ## What the agent does The agent begins with the window you give it, for example 14:05 to 14:40 UTC, and reads Datadog through the Datadog connection. It searches logs for the service in that window and groups them by status code and route, reads the error-rate and latency metrics for the same span, and lists any monitors that triggered or incidents that were declared. Every query it runs is written down with its time range, so the report can show where each number came from. Then it reads the code those errors point at. In a Gin service like gothinkster/golang-gin-realworld-example-app, `hello.go` builds the router with `gin.Default()`, mounts everything under `/api`, and registers routes in a fixed order: `users.UsersRegister` for `/api/users` and `/api/users/login`, then `users.AuthMiddleware(false)` for the anonymous article, tag and profile reads, then `users.AuthMiddleware(true)` for `/api/user`, follows, and article writes from `articles.ArticlesRegister`. A spike of 401s on `POST /api/articles/:slug/favorite` and a spike of 500s on `GET /api/articles` lead to different files, `users/middlewares.go` and `articles/models.go`, and the agent follows the one Datadog shows. It compares the window with what changed. It runs `git log --since='7 days ago' -- users/ articles/ common/` and reads each diff that touches the failing route's handler, its validator in `validators.go` or its query in `models.go`. A change to how `AuthMiddleware` extracts the token from the `Authorization` header, or to `r.RedirectTrailingSlash = false` (the comment above it says it stops POST bodies being lost on redirects), lines up with a specific symptom in the logs. The agent ties a commit to the incident only when its time and the symptom match, and says how confident it is. It changes no code. It may run the package tests (`go test ./users/ ./articles/`, which the repository keeps in each package's `unit_test.go`) or start the server on SQLite through `gorm.io/driver/sqlite` to confirm a hypothesis, and it records the output. The repository ships no Datadog tracer, so if your service runs `dd-trace-go`, the agent also reads the traces for the failing spans; without it, logs and host metrics are the evidence. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "claude", "input": "Error rates for the conduit-api Go service rose between 14:05 and 14:40 UTC today. Use the Datadog connection to read that window: error logs grouped by route and status, the request error-rate and latency metrics, and any monitors that triggered. Then compare it with the commits to users/, articles/ and common/ in the last 7 days, reading the diff of each one that touches a failing route's handler, middleware or model. You may run `go test ./...` to check a hypothesis. Write incident.md with: a summary, a timeline in UTC, the most likely cause with the commit and file, the evidence for each claim (the Datadog query or command and what it returned), other causes you ruled out, and every query you ran. Change no code and change nothing in Datadog.", "connections": [ "datadog" ], "resources": [ { "url": "https://github.com/your-org/your-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back The result is one file, `incident.md`, attached to the session as an artifact with a download link. It opens with a two-line summary, then a timeline in UTC that interleaves deploys, commits, the first error, monitor triggers and recovery. The cause section names a commit, a file and a line range, such as a change to `users/middlewares.go`, and says what in the logs matches it. Each claim carries its evidence: the Datadog log query or metric query, its time range and what it returned, or the `git` or `go test` command and its output. A section lists the causes the agent checked and ruled out, such as a database lock or a traffic spike, with the query that ruled each one out. The last section lists every query, so an on-call engineer can rerun any of them in Datadog. Nothing changed in the repository or in Datadog, which the session's changed-files list and tool calls confirm. Before acting on the report, the engineer reruns the query behind the cause, checks the commit time against the deploy time, and decides whether to revert or fix forward. ## Run it on every Datadog monitor alert Create an automation in the Project with this prompt and a `webhook` trigger, then add its URL as a Datadog webhook with the trigger's secret in an `Authorization: Bearer` header, and mention that webhook in the monitor's message. Each alert starts a run, with the monitor's payload after the prompt as data, so reword the prompt to read the window and service from it. A redelivered alert with the same `Idempotency-Key` is deduplicated. Set the automation's max runs per hour so a flapping monitor starts one investigation, not ten. Give the API key a webhook URL and Rush POSTs `run.completed` when the report is ready, with the `session_url`; your bot can post the `incident.md` link into the incident channel. ## Why run it on Rush? During an incident, an on-call engineer switches between Datadog, `git log` and the editor while the clock runs. An agent can do that first pass in its own sandbox, with its own clone and its own connection to Datadog, while the engineer works on mitigation. It runs `go test` and reads diffs there, so nothing on the engineer's machine changes. The report is easy to check because every line cites its source: a Datadog query with its range, or a command with its output, all kept in the session's record. The prompt and the session's changed files show it altered no code. Several incidents at once get several runs, each in a sandbox of its own. ## Frequently asked questions ### Can the agent mute monitors or change dashboards? The prompt forbids changing anything in Datadog, and the session records every tool call, so you can confirm it only read. The connection acts as the person who connected it, so it can never do more than that person could in Datadog. ### We are on a Datadog site other than US1. Does this work? The connector catalog lists Datadog's MCP server for the US1 site. For another site, connect your site's MCP address as a custom server with `POST /me/connections` and name that connection in `connections` instead of `datadog`. ### Our Go service has no APM tracing. Is there enough to go on? Usually yes. `gin.Default()` logs each request with its method, path, status and latency, and those logs in Datadog are enough to find the route and the minute errors began. Traces from `dd-trace-go` add the failing span and its database call when you have them. ### Will it run the service against production data? No. It can start the service on a local SQLite database in its sandbox, which this repository supports through `gorm.io/driver/sqlite`, to test a hypothesis. It never connects to your production database unless you give it a connection that reaches it.