Investigate Slow Requests in a Rails App

The agent starts from the controller named in the alert and works outward. In a Rails app like lobsters/lobsters it reads app/controllers/home_controller.rb, whose index action answers through get_from_cache; that serves from Rails.cache only for a logged-out visitor with no tag filters, and even then with expires_in of 45 seconds, so any logged-in user runs the full query on every request. The file also declares caches_page for the index action behind a CACHE_PAGE flag from actionpack-page_caching, so the agent checks whether that flag is set before treating the page as cached at all.

Next it reads the query the uncached path runs. In app/models/story.rb the hottest scope chains base(user) (which includes :hidings, :story_text and :user), not_hidden_by, filter_tags, positive_ranked and order(:hotness), and app/models/stories_paginator.rb wraps it with limit 26, an offset and for_presentation, which adds includes for domain, origin, tags and taggings. In db/schema.rb, stories.hotness is a decimal(20,10) with hotness_idx and an index on merged_story_id and hotness, but no index covers the front-page predicate merged_story_id IS NULL with is_deleted = false and score >= 0, so the sort can scan far more rows than the 25 shown. Story.recent also builds where.not(id: front_page.ids), a subquery that re-runs the hot ranking, and StoriesPaginator fires Vote, HiddenStory and SavedStory lookups per logged-in viewer.

The third pass is configuration and recent change. config/environments/production.rb sets config.cache_store = :solid_cache_store with shards: [:cache] and a page_cache_directory under public/cache, and config/cache.yml caps entries at 13 weeks and 2 gigabytes, so a cache that stops expiring changes exactly this controller's behavior. Gemfile.lock pins rails at 8.1.3.1 with .ruby-version at 4.0.0, and config/database.yml.sample shows separate sqlite3 databases for primary, cache, queue and rack_attack. Then git log over the last 30 commits with --stat, reading each diff that touches app/models/story.rb, db/migrate or config, lines the merges up against the week p95 climbed from 180 ms to 1.9 s.

Where it can, it confirms rather than guesses. In its sandbox the agent loads the schema from db/schema.rb and runs the hottest query under bin/rails runner with .explain, so it can write whether the order(:hotness) sort scans or uses hotness_idx with a filter step, and it greps for where CACHE_PAGE is set. It changes no code: incident.md carries a timeline and, per suspect, the file and line it rests on plus the check that would confirm or clear it.

What comes back is incident.md as a published artifact on the session. In a repository shaped like lobsters/lobsters it opens with a timeline that ties the p95 climb to merges from git log --stat, then a ranked suspect list. A typical entry names the uncached path in get_from_cache at app/controllers/home_controller.rb, the order(:hotness) sort in the hottest scope at app/models/story.rb against the missing partial index in db/schema.rb, and the per-request Vote, HiddenStory and SavedStory lookups in app/models/stories_paginator.rb, each with the file, line and EXPLAIN or grep behind it.

Every suspect ends with the check that settles it: the EXPLAIN ANALYZE to run in production for the order(:hotness) query, the cache hit ratio for the stories keys, whether CACHE_PAGE is set, and the puma thread count in config/puma.rb against the pool in config/database.yml.sample. A final section lists what it ruled out, so the on-call engineer opens Datadog with a few queries to run instead of a blank page.

The artifact's url on api.prix.dev downloads it with no key for an hour, and the transcript at GET /o/{org}/sessions/{id}/messages keeps every event with its command output, so each claim traces to the git log, EXPLAIN or file read that produced it. If one suspect is worth digging into, a follow-up POST to /o/{org}/sessions/{id}/messages wakes the same sandbox for a deeper pass on just that one.

Point your monitoring webhook, for example a Datadog monitor on controller p95, at your server and have it POST this same request when the alert fires, with the alert text pasted into input. Each alert gets its own run in its own sandbox and its own incident.md, so a regression is being read before the on-call engineer opens the dashboard. The run names your GitHub repository in resources, so the Rush GitHub App has to be installed on it for the clone to work.

Set a webhook URL on the API key and Rush POSTs run.completed or run.failed when the run's turn finishes, with the session URL, so your server can page the engineer with the incident.md link. A key runs at most max_concurrent sessions at once (4 by default) and answers 429 past that; if a deploy fans alerts out across services, queue the requests.

Reading a Rails incident on a laptop means checking out the repo, loading a schema, running EXPLAIN by hand and keeping notes, all while the pager is still open. On Rush the agent does that in its own cloud sandbox with its own clone: it runs git log --stat, loads db/schema.rb, runs the hottest query under bin/rails runner with .explain, and keeps every command with its output in the transcript. Your machine and your working tree are not involved, and the run keeps going if you close the laptop.

The report is checkable rather than asserted. Every suspect in incident.md carries a file, a line and the exact check that confirms or clears it, and the trajectory shows the command behind each claim. Because each run is its own sandbox, one request can investigate several services or alerts at once, each with its own report.

Will it change any code?

Can it use our Datadog data, not just the alert text?

What if the cause is a recent deploy?

How do I verify the report before acting on it?

Does it work on a private repository?