# Fix a Phoenix App Bug From a Datadog Monitor Alert

Yes. An agent reads the logs and traces behind a Datadog alert, finds the failing Phoenix controller, fixes it with an ExUnit test and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/fix-datadog-alerts/phoenix/), in a Project that has your repository.

## What the agent does

The agent reads the alert before it reads the code. Your server passes Datadog's webhook payload into the prompt, and the agent uses the Datadog MCP server to pull the monitor definition, the error logs for the service in the 30 minutes around the alert, and the traces of the failing requests. Phoenix logs crashes with the exception, the controller and action, and an Elixir stacktrace; with JSON logging those land in Datadog as attributes, so the agent can group a burst of errors by exception and the first frame in your own modules, and keep one sample of each with its request id.

Then it maps the errors to controllers. In hexpm/hexpm, the Phoenix app behind hex.pm, API controllers live in `lib/hexpm_web/controllers/api/`, and package publishing goes through `HexpmWeb.API.ReleaseController.publish/2`. That action reads the tarball's metadata with `release_metadata(body_path)`, logs it with `log_tarball` using the `x-request-id` header, and calls `Releases.publish` with `replace: Map.get(params, "replace", true)` and the workload identity context. Before it runs, `plug :authorize` checks the `api:write` or `package` domain, `package_owner` and `organization_billing_active`. A failing publish can break at any of those steps, so the agent follows the stacktrace into `lib/hexpm/repository/releases.ex` and the policy modules instead of stopping at the controller.

It writes the regression test first, in the style the suite already uses. `test/hexpm_web/controllers/api/release_controller_test.exs` is `use HexpmWeb.ConnCase, async: true`, builds its world with ExMachina factories (`insert(:user)`, `insert(:package, package_owners: [...])`, `insert(:release, ...)`), and groups cases under `describe "POST /api/packages/:name/releases"`. The repository also ships `test/support/log_lines.ex`, a `:logger` handler that sends every line a test process logs back to it as `{Hexpm.LogLines, level, fields}`, so the new test can assert that the log line from the alert is no longer written, not only that the response code changed.

To run it, the agent starts Postgres (`docker-compose.yaml` uses `postgres:14`), fetches dependencies for the Elixir version `mix.exs` requires (`~> 1.20`), and runs `mix test`, whose alias creates, loads and migrates the test database before the suite. It runs the new test alone first to see it fail, then the controller's test file and the full suite after the fix.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "A Datadog monitor on this Phoenix service fired; its webhook payload is at the end of this prompt. Use the Datadog connection to read the monitor, then the error logs and traces for its service and environment from 15 minutes before to 15 minutes after it fired. Group the errors by exception and the first frame in this repository, map them to the controller actions under lib/hexpm_web/controllers, and follow the stacktrace into the context modules to find the cause. Write an ExUnit regression test in the matching controller test file, using the existing ConnCase and factories, that sends the request the logs show and fails for the same reason; assert on the log line too where the test helpers allow it. Fix the cause, run mix test, and open a pull request that quotes the log lines and trace ids the fix addresses. If the cause is outside this code, change nothing and answer with what the logs show.\n\nPayload: <Datadog webhook body>",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

A pull request with the fix in the context or controller module the stacktrace led to, and a new test in the matching file under `test/hexpm_web/controllers/api/`, such as a case in the `POST /api/packages/:name/releases` block. The description names the monitor, quotes the Datadog log lines with their request ids and trace ids, shows the request the test reproduces, and lists the `mix test` commands with their result. The session's trajectory keeps each Datadog query and every test run with its full output.

If the logs point outside the app, such as S3 errors from `ex_aws_s3` during a tarball upload or a queue backing up behind `broadway_sqs`, the agent leaves the code alone and its answer explains what it found and which dependency or service to check. That answer comes back on your webhook with the run.

Before merging, a reviewer checks that the test fails on the old code for the alert's reason, that the change does not loosen the `:authorize` plug or the owner and billing checks on publish, and that a fix to `Releases.publish` keeps the `replace` behavior existing clients rely on.

## Run it on every Datadog monitor alert

Set up a Datadog webhook that posts to your server and mention it in the monitors that cover the app, such as an error-count monitor on the publish endpoint. When one moves to Alert, your server appends the payload to this prompt and sends the request with the repository in `resources`; skip Recovered notifications and alerts tagged for infrastructure, which a code change will not fix.

Store the alert id beside the `execution_id` Rush returns, so a monitor that re-notifies in the same incident is matched to the run already working on it. When the turn finishes, Rush POSTs `run.completed` or `run.failed` to the webhook URL on your API key with the answer and the session URL, ready to forward to the incident.

## Why run it on Rush?

Getting an Elixir app with Postgres, S3 and SQS dependencies to the point where a failing request reproduces takes setup most people only have on one machine. The agent does that setup in its own sandbox with its own clone, its own database and its own Datadog session, and none of it touches your laptop, your local `_build` or your shell history. It starts when the alert fires, not when someone reads it.

The pull request is evidence a reviewer can check: the quoted log lines and trace ids, a test that fails without the fix, and a trajectory with every query and `mix test` run and its output. When an incident trips several monitors, each one gets its own run in parallel, and nothing merges until a person approves it.

## Frequently asked questions

### How does the agent connect to Datadog?

Through Datadog's MCP server, listed among Rush's connectors. Connect it once with `POST /me/connections`, `"type": "mcp"` and OAuth, sign in, then add `"connections": ["datadog"]` to the request so the run can query it. The listed server is the US1 site; add another site's MCP URL as your own server.

### Our logs are JSON for another platform. Will Datadog's attributes be there?

The agent reads whatever attributes the log lines carry. hexpm configures `logger_json` with its Google Cloud formatter in `config/prod.exs`; the same library has a Datadog formatter, which puts the request id and trace ids where Datadog links logs to traces. Without that link, the agent matches logs and traces by request path and time.

### Can it break publishing for everyone?

Only if a reviewer merges a change that does. The fix arrives as a pull request with a test, and the prompt tells it not to change code when the cause is outside the app. Review changes to the authorization plug with extra care.

### Does it run the whole ExUnit suite?

Yes, after the targeted test file. The suite runs `async: true` where the tests allow it, and the pull request lists each `mix test` command it ran so you can see what was covered.