All jobs

Fix a Rails App From a Grafana Alert

Yes. The alert starts a run that reruns its Loki and Prometheus queries, follows the JSON request logs to the Rails controller at fault and opens a pull request.

Run this on RushIn a Project that has your repository.

What the agent does

A Grafana Cloud alert posting to a Rush webhook trigger starts the run, with the notification after the prompt as data. The agent reads the alert rule through the Grafana connection, reruns its query around startsAt, and pulls the matching Loki lines for the same window. For a Rails app the useful lines are the request logs: in lobsters/lobsters, config/initializers/lograge.rb writes one JSON object per request to log/action.log in production with controller, action, status, path, request_token, filtered params, remote_ip, user_id, username and, when the request raised, exception and exception_message. A LogQL query such as {app="lobsters"} | json | exception =~ ".+" therefore gives the agent the controller, the action, the parameters and the exception class for every failing request, which is most of the diagnosis.

lobsters has no Prometheus exporter in its Gemfile, so the alert rule is usually a Loki metric query over those lines, rate of status=~"5.." by controller, or a metric from the reverse proxy in front of Puma. Either way the agent ends up in app/controllers. StoriesController#show loads by short_id, redirects a story merged into another, renders _missing with a 404 when can_be_seen_by_user? fails, redirects to the canonical title path, then loads Comment.story_threads(@story).for_presentation.load_async and renders HTML or JSON. CommentsController#create answers 400 with can't find story when the story is gone, checks the parent comment belongs to the same story, silently accepts a duplicate resubmission, applies breaks_speed_limit?, and on save touches last_comment_at and enqueues NotifyCommentJob. The agent follows the logged action and params into exactly one of these branches.

The suite is RSpec (rspec-rails 8.0.2) with factories in spec/factories, request specs in spec/requests/stories_spec.rb and spec/requests/comments_spec.rb, and controller specs such as spec/controllers/comments_controller_spec.rb. spec/rails_helper.rb turns on transactional fixtures, cleans with DatabaseCleaner, and seeds a System user and a placeholder tag before the suite. Setup matches .github/workflows/check.yml: Ruby 4.0.0 from .ruby-version, libvips for ruby-vips, cp .github/workflows/check/* config/ for the test credentials, cp config/database.yml.sample config/database.yml, which points every connection (primary, cache, queue and rack_attack) at SQLite files under db/test, and ./bin/rails db:schema:load db:migrate. No MySQL or Postgres is needed in the sandbox.

The agent writes a request spec that sends the request the log line recorded, with the same parameters and the same kind of user from the factories, fails it the way production did, and then fixes the controller or the model it calls. It reruns the one file, then bundle exec rspec, bundle exec standardrb and bundle exec brakeman --ensure-ignore-notes, the three commands the workflow's test, lint and security jobs run, under the workflow's RUBYOPT of --yjit --enable-frozen-string-literal --debug-frozen-string-literal so a frozen string mutation shows up here rather than in CI.

Start it with one API call

POST https://api.prix.dev/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "A Grafana alert fired for this Ruby on Rails service; its payload follows this prompt. Use the Grafana connection to read the alert rule, rerun its query around the firing time, and query Loki for the lograge JSON request lines in the same window, filtering on exception and status. Find the cause in app/controllers/stories_controller.rb or app/controllers/comments_controller.rb and the models they call. Set up the test database the way .github/workflows/check.yml does (copy the check credentials and config/database.yml.sample, then bin/rails db:schema:load db:migrate), add a request spec under spec/requests that sends the logged request and reproduces the failure, fix it, and run bundle exec rspec, bundle exec standardrb and bundle exec brakeman --ensure-ignore-notes. Open a pull request that includes the LogQL and PromQL queries, the log lines behind the diagnosis, and the spec output before and after the fix. If the evidence does not point at code, say what it points at instead.\n\nAlert payload:\n{payload}",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ],
  "connections": [
    "grafana"
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request whose description shows the alert rule's query with the series that crossed its threshold, the LogQL query with the JSON lines it returned, and the new spec failing and then passing under bundle exec rspec. The diff is the spec plus the fix in one controller or model, nothing in config/.

A reviewer checks that the spec sends the parameters the log recorded, that a change in StoriesController#show keeps the merged-story redirect and the 404 for stories the user may not see, that CommentsController#create still answers its 400s rather than raising, and that standardrb and brakeman stayed clean. The workflow then runs the same four jobs on the pull request.

When the lines point away from code, for example at Rack::Attack answering 429s to one remote_ip or at the SQLite timeout of 1000 ms under a burst, the run says so with the queries and the lines it found and opens nothing.

Run it on every Grafana alert on the Rails app

Add an automation to the Project with this prompt and a webhook trigger; Rush shows its URL and secret once. In Grafana Cloud, create a webhook contact point at that URL with Authorization: Bearer <secret> and route the app's alert rules to it. The notification Grafana sends becomes the event text the agent reads under the prompt.

Turn off resolved notifications on the contact point and keep the repeat interval longer than a run, because a delivery with no Idempotency-Key header is a new run each time. The automation's max_runs_per_hour is the backstop for a rule that flaps.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

Turning a Rails alert into a fix means the same loop every time: rerun the rule's query, find the request lines, read the controller branch they took, write a spec that fails the same way. A Rush run does that loop in its own sandbox with its own SQLite databases, and keeps every query, every spec run and every lint result in the session, so the diagnosis can be checked rather than trusted.

When one deploy trips alerts on stories and comments at once, each alert gets its own run in parallel, none touches a production host, and each ends in a pull request whose spec proves the fix or in an account of what the logs show instead.

Frequently asked questions

My Rails app does not use lograge. Does this still work?

Yes, with more reading. Without JSON request lines the agent matches the default Rails log by Started and Completed pairs and the exception block between them. The result is the same spec and fix; the description quotes plain lines instead of JSON fields.

Does the test need MySQL or Postgres?

Not for this app: config/database.yml.sample puts every connection on SQLite under db/test, so the suite runs in the sandbox as it does in CI. An app on Postgres or MySQL gets that server started in the sandbox the way its workflow's services block does.

Will it edit the credentials or database config?

No. It copies the test credentials from .github/workflows/check/ and the sample database file exactly as the workflow does, and the pull request touches neither. A fix that needs a new credential is described in the pull request for a person to add.

What if the rule is on a proxy metric and not on the logs?

The agent still has the request lines for the same minutes. It joins the proxy's path and status to the path, controller and action fields in the lograge JSON and proceeds from there; the pull request shows both queries so the reviewer can see the join.