# Fix a Rails App From a Grafana Alert

Yes. The alert starts a run that reruns its Loki and Prometheus queries, follows the JSON request logs to the Rails controller at fault and opens a pull request.

[Run this on Rush](https://cloud.getrush.ai/run/fix-grafana-alerts/rails/), in a Project that has your repository.

## What the agent does

A Grafana Cloud alert posting to a Rush webhook trigger starts the run, with the notification after the prompt as data. The agent reads the alert rule through the Grafana connection, reruns its query around `startsAt`, and pulls the matching Loki lines for the same window. For a Rails app the useful lines are the request logs: in lobsters/lobsters, `config/initializers/lograge.rb` writes one JSON object per request to `log/action.log` in production with `controller`, `action`, `status`, `path`, `request_token`, filtered `params`, `remote_ip`, `user_id`, `username` and, when the request raised, `exception` and `exception_message`. A LogQL query such as `{app="lobsters"} | json | exception =~ ".+"` therefore gives the agent the controller, the action, the parameters and the exception class for every failing request, which is most of the diagnosis.

lobsters has no Prometheus exporter in its `Gemfile`, so the alert rule is usually a Loki metric query over those lines, `rate` of `status=~"5.."` by `controller`, or a metric from the reverse proxy in front of Puma. Either way the agent ends up in `app/controllers`. `StoriesController#show` loads by `short_id`, redirects a story merged into another, renders `_missing` with a 404 when `can_be_seen_by_user?` fails, redirects to the canonical title path, then loads `Comment.story_threads(@story).for_presentation.load_async` and renders HTML or JSON. `CommentsController#create` answers 400 with `can't find story` when the story is gone, checks the parent comment belongs to the same story, silently accepts a duplicate resubmission, applies `breaks_speed_limit?`, and on save touches `last_comment_at` and enqueues `NotifyCommentJob`. The agent follows the logged `action` and `params` into exactly one of these branches.

The suite is RSpec (`rspec-rails` 8.0.2) with factories in `spec/factories`, request specs in `spec/requests/stories_spec.rb` and `spec/requests/comments_spec.rb`, and controller specs such as `spec/controllers/comments_controller_spec.rb`. `spec/rails_helper.rb` turns on transactional fixtures, cleans with `DatabaseCleaner`, and seeds a `System` user and a placeholder tag before the suite. Setup matches `.github/workflows/check.yml`: Ruby 4.0.0 from `.ruby-version`, `libvips` for `ruby-vips`, `cp .github/workflows/check/* config/` for the test credentials, `cp config/database.yml.sample config/database.yml`, which points every connection (primary, cache, queue and `rack_attack`) at SQLite files under `db/test`, and `./bin/rails db:schema:load db:migrate`. No MySQL or Postgres is needed in the sandbox.

The agent writes a request spec that sends the request the log line recorded, with the same parameters and the same kind of user from the factories, fails it the way production did, and then fixes the controller or the model it calls. It reruns the one file, then `bundle exec rspec`, `bundle exec standardrb` and `bundle exec brakeman --ensure-ignore-notes`, the three commands the workflow's `test`, `lint` and `security` jobs run, under the workflow's `RUBYOPT` of `--yjit --enable-frozen-string-literal --debug-frozen-string-literal` so a frozen string mutation shows up here rather than in CI.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "A Grafana alert fired for this Ruby on Rails service; its payload follows this prompt. Use the Grafana connection to read the alert rule, rerun its query around the firing time, and query Loki for the lograge JSON request lines in the same window, filtering on exception and status. Find the cause in app/controllers/stories_controller.rb or app/controllers/comments_controller.rb and the models they call. Set up the test database the way .github/workflows/check.yml does (copy the check credentials and config/database.yml.sample, then bin/rails db:schema:load db:migrate), add a request spec under spec/requests that sends the logged request and reproduces the failure, fix it, and run bundle exec rspec, bundle exec standardrb and bundle exec brakeman --ensure-ignore-notes. Open a pull request that includes the LogQL and PromQL queries, the log lines behind the diagnosis, and the spec output before and after the fix. If the evidence does not point at code, say what it points at instead.\n\nAlert payload:\n{payload}",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ],
  "connections": [
    "grafana"
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request whose description shows the alert rule's query with the series that crossed its threshold, the LogQL query with the JSON lines it returned, and the new spec failing and then passing under `bundle exec rspec`. The diff is the spec plus the fix in one controller or model, nothing in `config/`.

A reviewer checks that the spec sends the parameters the log recorded, that a change in `StoriesController#show` keeps the merged-story redirect and the 404 for stories the user may not see, that `CommentsController#create` still answers its 400s rather than raising, and that `standardrb` and `brakeman` stayed clean. The workflow then runs the same four jobs on the pull request.

When the lines point away from code, for example at `Rack::Attack` answering 429s to one `remote_ip` or at the SQLite `timeout` of 1000 ms under a burst, the run says so with the queries and the lines it found and opens nothing.

## Run it on every Grafana alert on the Rails app

Add an automation to the Project with this prompt and a `webhook` trigger; Rush shows its URL and secret once. In Grafana Cloud, create a webhook contact point at that URL with `Authorization: Bearer <secret>` and route the app's alert rules to it. The notification Grafana sends becomes the event text the agent reads under the prompt.

Turn off resolved notifications on the contact point and keep the repeat interval longer than a run, because a delivery with no `Idempotency-Key` header is a new run each time. The automation's `max_runs_per_hour` is the backstop for a rule that flaps.

## Why run it on Rush?

Turning a Rails alert into a fix means the same loop every time: rerun the rule's query, find the request lines, read the controller branch they took, write a spec that fails the same way. A Rush run does that loop in its own sandbox with its own SQLite databases, and keeps every query, every spec run and every lint result in the session, so the diagnosis can be checked rather than trusted.

When one deploy trips alerts on stories and comments at once, each alert gets its own run in parallel, none touches a production host, and each ends in a pull request whose spec proves the fix or in an account of what the logs show instead.

## Frequently asked questions

### My Rails app does not use lograge. Does this still work?

Yes, with more reading. Without JSON request lines the agent matches the default Rails log by `Started` and `Completed` pairs and the exception block between them. The result is the same spec and fix; the description quotes plain lines instead of JSON fields.

### Does the test need MySQL or Postgres?

Not for this app: `config/database.yml.sample` puts every connection on SQLite under `db/test`, so the suite runs in the sandbox as it does in CI. An app on Postgres or MySQL gets that server started in the sandbox the way its workflow's `services` block does.

### Will it edit the credentials or database config?

No. It copies the test credentials from `.github/workflows/check/` and the sample database file exactly as the workflow does, and the pull request touches neither. A fix that needs a new credential is described in the pull request for a person to add.

### What if the rule is on a proxy metric and not on the logs?

The agent still has the request lines for the same minutes. It joins the proxy's path and status to the `path`, `controller` and `action` fields in the lograge JSON and proceeds from there; the pull request shows both queries so the reviewer can see the join.