# Fix a Rails App Bug From a Datadog Monitor Alert

Yes. A Datadog monitor starts a run; the agent traces failing background jobs to your Rails code, adds a Minitest test, runs bin/rails test and opens a pull request.

[Run this on Rush](https://cloud.getrush.ai/run/fix-datadog-alerts/rails/), in a Project that has your repository.

## What the agent does

Many Rails alerts come from background jobs, not web requests, and the agent reads them that way. Through the Datadog connection it reads the monitor and then the job errors, logs and traces from the 30 minutes around the alert. Datadog's Ruby tracer reports each Sidekiq job under its class name, so a monitor on job failures points at a class such as `SendDailyTriageEmailJob` or `SendSingleTriageEmailJob`, with the exception and backtrace on the error span.

In a Rails app like `codetriage/CodeTriage` (Ruby 4.0.5 per `.ruby-version`, Rails 8.1.3.1 in the `Gemfile`, Sidekiq for jobs), issue assignment runs through a short chain. `SendDailyTriageEmailJob#perform` skips users for reasons it names in `reason_for_skip`, then calls `user.issue_assignments_to_deliver`, which returns undelivered assignments up to `daily_issue_limit` or calls `IssueAssigner#assign!` to make new ones. `IssueAssigner` in `app/models/issue_assigner.rb` walks each subscription and stops at its `email_limit`. After the mail is queued with `deliver_later`, the job marks assignments `delivered` and stamps `last_sent_at`. The agent reads each step against the backtrace, because a failure after the mail is queued and before `update_all` sends the same issues again.

It reproduces the failure in Minitest beside the existing job tests in `test/jobs/`, such as `send_daily_triage_email_job_test.rb`, an `ActiveJob::TestCase` that calls `perform` directly and checks the mail job with `assert_enqueued_with`, using the fixtures in `test/fixtures/`. `IssueAssigner` takes `can_access_network: !Rails.env.test?`, so tests never call GitHub; the agent keeps it that way. The repository's CI creates the test database with `bin/rails db:create db:schema:load` on Postgres and runs `bin/rake test`, and the agent does the same in its sandbox, first the one file with `bin/rails test`, then everything.

It opens a pull request quoting the job's error log lines and the trace, linking the monitor, and saying what users saw: no email, a duplicate email, or the same issue twice. If the cause is GitHub's API rate limit rather than the code, it says so and changes nothing.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "A Datadog monitor on this Ruby on Rails service fired; its webhook payload follows this prompt. Use the Datadog connection to read the monitor and the failing Sidekiq job errors, logs and traces from the 30 minutes around the alert, grouped by job class and exception. Find the code path that produced them in the issue-assignment jobs: app/jobs/send_daily_triage_email_job.rb, send_single_triage_email_job.rb, User#issue_assignments_to_deliver and app/models/issue_assigner.rb. Write a Minitest regression test in test/jobs/ with the existing fixtures and no network calls, set up the Postgres test database, fix the code, run bin/rails test, and open a pull request that quotes the log lines the fix addresses and links the monitor.\n\nMonitor payload:\n{payload}",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ],
  "connections": [
    "datadog"
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

You get a pull request with the fix in a job or in `IssueAssigner`, and a new test in `test/jobs/` that fails on the old code. The description has the quoted job errors, the job class and exception from the trace, the monitor link and the test run's summary. Your CI runs it against Postgres as usual.

The session's `result`, in `data.result` on the `run.completed` webhook, states the cause and the change briefly. The transcript keeps each Datadog query, the database setup and every test run's output, so the reviewer can confirm the suite passed and that no test reached GitHub.

Review the order of operations most carefully. A job that sends mail and then marks it delivered must not send twice when it retries, and a fix that moves `update_all` should come with a test for the retry. Also check that the skip reasons in `reason_for_skip` still hold, since they decide who gets email at all.

## Run it on every Datadog monitor alert

Add an automation with a webhook trigger to the Project for this app, with this prompt and a saved agent that names the Datadog connection. In Datadog's Webhooks integration, send to the trigger's URL with the trigger's secret as a bearer token, and add `@webhook-<name>` to the job-failure monitor. Its body follows the prompt as event text; `max_runs_per_hour` caps runs from a monitor that keeps re-alerting while Sidekiq retries.

Or receive the webhook in your own app and call `POST /o/{org}/sessions` with this request, the payload appended to `input` and the alert id as the `Idempotency-Key`, so Datadog's retries start one run. A webhook on the key tells you when it finishes.

## Why run it on Rush?

Job bugs surface hours after the code that caused them, usually when a user says they got an email twice or not at all. A Rush run starts when the monitor fires, reads the failing jobs with your Datadog connection, sets up Ruby and Postgres in its own sandbox and runs the real suite, with every command's output kept on the session.

The fix comes with proof: the production errors quoted, a test that reproduces them, and a transcript that shows nothing called GitHub or touched real users. Several monitors can fire at once and each gets its own run and branch.

## Frequently asked questions

### We report errors to Sentry too. Does that matter?

No. CodeTriage itself reports errors to Sentry through a `SentryJob`, and the agent reads Datadog because that is where the monitor and traces are. If you want it to read both, connect Sentry as well and name both connections.

### Will it send email while testing?

No. `config/environments/test.rb` sets the mailer's `delivery_method` to `:test` and Active Job's `queue_adapter` to `:test`, so mail is only enqueued and asserted on. The agent works on a clone with fixture users.

### Does it need Redis for Sidekiq?

Not for the tests, which use Active Job's test adapter and call `perform` directly. If a fix depends on Sidekiq's retry behaviour, the agent can start Redis in the sandbox and says so in the pull request.

### Can it retry the failed jobs in production?

No. It fixes code and opens a pull request. Re-enqueueing failed jobs after the fix is deployed is a decision for your team.