# Fix a Go Service Bug From a Datadog Monitor Alert

Yes. A Datadog monitor starts a run; the agent follows its logs and traces into your Go code, adds a regression test, runs go test and opens a pull request.

[Run this on Rush](https://cloud.getrush.ai/run/fix-datadog-alerts/go/), in a Project that has your repository.

## What the agent does

Go services that log with `log/slog` give an agent structured fields to search on, and the run uses them. Through the Datadog connection it reads the monitor and then the error and warning logs from the 30 minutes around the alert. In a Go service like `miniflux/v2`, which emits JSON logs when `LOG_FORMAT` is set to `json`, the feed refresh path writes `Unable to fetch feed` and `Feed is rate limited` with `feed_id`, `feed_url` and `error` fields from `internal/reader/handler/handler.go`. The agent groups the lines by message and error so it can see whether one feed or every feed is failing.

From there it reads the code in its sandbox. `RefreshFeed` in `handler.go` builds a request, wraps the answer in `fetcher.NewResponseHandler`, and checks `IsRateLimited`, `ParseRetryDelay` and `LocalizedError` before it parses the body. Those methods live in `internal/reader/fetcher/response_handler.go`, alongside the logic for `Expires`, `Cache-Control` max age, `ETag` and `Last-Modified` and the network and TLS error checks. A spike in one error type points at one of these functions, and the agent reads the trace's span tags to see which upstream status codes came back.

It writes a table-driven test in the package's existing `_test.go` file, such as `response_handler_test.go`, which already has `TestIsModified`, `TestRetryDelay` and `TestCacheControlMaxAgeInMinutes`, using `httptest` or a hand-built `http.Response` to feed the exact headers production saw. It runs that test, fixes the code, then runs `go test -cover -race -count=1 ./...`, the repository's `make test` target. Tests under `internal/api` need a Postgres database and the `make integration-test` setup, so the agent runs them only when the fix touches that package.

The pull request it opens quotes the log lines, with their `feed_url` redacted if you ask, names the span the trace pointed at, and explains the cause. If the logs show the failures came from the feeds themselves, a site returning 429 for everyone, it reports that and changes nothing.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "codex",
  "input": "A Datadog monitor on this Go service fired; its webhook payload follows this prompt. Use the Datadog connection to read the monitor and the error and warning logs and traces from the 30 minutes around the alert, grouped by message and error. Find the code path that produced them, starting with RefreshFeed in internal/reader/handler/handler.go and the response handler in internal/reader/fetcher/response_handler.go. Add a table-driven regression test in the package's existing _test.go file, fix the code, run go test -race ./..., and open a pull request that quotes the log lines the fix addresses and links the monitor. If the cause is outside this code, open no pull request and say what you found.\n\nMonitor payload:\n{payload}",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ],
  "connections": [
    "datadog"
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

You get a pull request with a focused diff in one package and a new row or case in its `_test.go` file. The description has the quoted `slog` lines with their fields, the trace span, the monitor link and the commands the agent ran with their summary lines. Your CI runs on it as usual.

The session's `result`, sent as `data.result` in the `run.completed` webhook, is the short account: which monitor, which function, which test. The full transcript keeps the Datadog queries and the raw `go test` output, including the race detector's, so a reviewer can see the suite passed with `-race` rather than read that it did.

Before merging, check that the new test fails on the old code, that a retry or backoff change still respects the upstream's `Retry-After`, and that the fix does not turn a real fetch error into a silent skip, which would stop the monitor from firing the next time.

## Run it on every Datadog monitor alert

Add an automation with a webhook trigger to the Project for this service, using this prompt and a saved agent that names your Datadog connection. Point a webhook in Datadog's Webhooks integration at the trigger's URL with its secret as the bearer token, and mention `@webhook-<name>` in the monitor's message. Each alert's body becomes the event text after the prompt; `max_runs_per_hour` limits a monitor that flaps.

Or handle the webhook in your own service and call `POST /o/{org}/sessions` with this body, the payload added to `input` and the alert id as `Idempotency-Key`. Give the key a webhook URL and Rush posts `run.completed` back when the pull request is ready.

## Why run it on Rush?

A Go fix is quick to write and slow to trust: you want the exact headers that broke it in a test and the race detector green before anyone merges. A Rush run does that with a real toolchain in its own sandbox, reads Datadog with your connection, and keeps the `go test` output with the session, so the evidence goes with the pull request.

Each alert gets its own run, so a refresh failure and an unrelated handler panic are worked on at the same time in separate clones. Nobody has to stop what they are doing to open Datadog, and the record of what the agent read and ran is there for the person who reviews it.

## Frequently asked questions

### Do my logs need to be JSON?

It helps. Structured `slog` fields such as `feed_id` and `error` let the agent group and count lines precisely. Plain text logs work, with more searching.

### Will it run integration tests that need Postgres?

Only when the fix touches that code and you allow it in the prompt. The sandbox can run Postgres, and the agent follows the repository's own setup, here the `integration-test` target in the `Makefile`.

### Which Go version does it use?

The one `go.mod` declares. The agent reads the `go` line and uses a matching toolchain, so the tests run on the same version your CI uses.

### Can it act on Datadog for me?

The prompt limits it to reading the monitor, logs and traces, and every call it makes is in the transcript. For a hard limit, connect Datadog with a read-only key.