# Investigate a Rails App Incident With Datadog Yes. An agent reads the incident window in Datadog, matches it against your Rails request logs, jobs and recent commits, and writes incident.md with cited evidence. [Run this on Rush](https://cloud.getrush.ai/run/investigate-incidents-with-datadog/rails/), in a Project that has your repository. ## What the agent does The agent begins with the window you give it, for example 21:10 to 21:35 UTC, and reads Datadog through the Datadog connection. It searches logs for the app in that window and groups them by controller, action and status, reads the request error-rate and latency metrics for the same span, and lists any monitors that triggered. Every query it runs is written down with its time range, so the report can show where each number came from. Then it reads how the app logs, because that decides what Datadog can show. In a community site like lobsters/lobsters, config/initializers/lograge.rb formats every request as one JSON line in log/action.log with a timestamp, the request_token, the filtered params, the exception class and message, the remote_ip, and a custom payload of user_id, username and path; config/environments/production.rb reads the level from RAILS_LOG_LEVEL and broadcasts the Rails log to stdout and log/rails.log, and the solid_queue log to stdout and log/solid_queue.log. With ENABLE_SLOW_QUERY_LOGS set, config/initializers/slow_query_log.rb writes any query over SLOW_QUERY_THRESHOLD_MS, 100 by default, to log/slow_query.log as JSON with the request_token and the SQL. So the agent can join a 500 in the action log to the slow queries of the same request by request_token, if those files reach Datadog. config/initializers/rack_attack.rb adds the other common shape of an incident: a 429 with RateLimit headers that is working as designed. It follows the controller the logs name. A spike in StoriesController or CommentsController leads to app/controllers/stories_controller.rb and comments_controller.rb, where the before_action chains such as require_logged_in_user_or_400 and find_user_story run before create, update, upvote, flag, hide and save, and a spike in a job leads to app/jobs, where cache_page_job, refill_story_page_cache_job, notify_comment_job, send_webmention_job and wal_checkpoint_job run on solid_queue. Storage is four SQLite databases under storage, primary, cache, queue and rack_attack, so a stalled WAL checkpoint or a locked database shows up as latency across every controller at once rather than in one action. The agent runs git log --since='7 days ago' -- app/ config/ db/ and reads each diff that touches the controller, job or initializer the symptom points at, and ties a commit to the incident only when its time and the symptom match. It changes no code. To check a hypothesis it may run the request spec for the controller, bundle exec rspec spec/requests/stories_spec.rb, after copying config/database.yml.sample into place and loading the schema the way .github/workflows/check.yml does, and it records the output. The repository ships sentry-rails for exceptions and no Datadog tracer, so if your deploy runs the Datadog agent the traces add the failing span and its SQL; without them, the action log, the slow query log and the host metrics are the evidence, and the report says which it had. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "claude", "input": "Error rates and latency for this Rails app rose between 21:10 and 21:35 UTC today. Use the Datadog connection to read that window: error logs grouped by controller, action and status, the request error-rate and latency metrics, slow query and job logs if they are shipped, and any monitors that triggered. Read config/initializers/lograge.rb, slow_query_log.rb and rack_attack.rb and config/environments/production.rb first so you know what each log line carries, join action log and slow query log lines by request_token, then read the controller or job the logs name and compare the window with the commits to app/, config/ and db/ in the last 7 days, reading each diff that touches it. You may run the matching request spec to check a hypothesis after setting up the SQLite databases as .github/workflows/check.yml does. Write incident.md with: a summary, a timeline in UTC, the most likely cause with the commit and file, the evidence for each claim (the Datadog query or command and what it returned), other causes you ruled out, and every query you ran. Change no code and change nothing in Datadog.", "connections": [ "datadog" ], "resources": [ { "url": "https://github.com/your-org/your-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back The result is one file, incident.md, attached to the session as an artifact with a download link. It opens with a two-line summary, then a timeline in UTC that interleaves deploys, commits, job runs, the first error, monitor triggers and recovery. The cause section names a commit, a file and a line range, such as a change to a before_action in stories_controller.rb or to a job under app/jobs, and says what in the logs matches it, with the request_token of an example request. Each claim carries its evidence: the Datadog log or metric query, its time range and what it returned, or the git or rspec command and its output. A section lists the causes the agent checked and ruled out, such as rack-attack throttling a crawler, a WAL checkpoint holding the primary database, or a solid_queue backlog, with the query that ruled each one out. The last section lists every query, so an on-call engineer can rerun any of them in Datadog. Nothing changed in the repository or in Datadog, which the session's changed-files list and tool calls confirm. Before acting on the report, the engineer reruns the query behind the cause, checks the commit time against the deploy time, and decides whether to revert or fix forward. ## Run it on every Datadog monitor alert Create an automation in the Project with this prompt and a webhook trigger. Datadog cannot call that trigger directly, because the trigger needs an Authorization header with its own secret, so point the Datadog webhook at a small handler on your server that forwards the monitor payload to the trigger URL with that header. Each alert starts a run with the payload after the prompt as data, so reword the prompt to read the window and controller from it. A redelivered alert forwarded with the same Idempotency-Key is deduplicated. Set the automation's max runs per hour so a flapping monitor starts one investigation, not ten. Give the API key a webhook URL and Rush POSTs run.completed when the report is ready, with the session_url; your handler can post the incident.md link into the incident channel. ## Why run it on Rush? During an incident, an on-call engineer switches between Datadog, git log and the editor while the clock runs. An agent can do that first pass in its own sandbox, with its own clone and its own connection to Datadog, while the engineer works on mitigation. It runs specs and reads diffs there, so nothing on the engineer's machine or the production host changes. The report is easy to check because every line cites its source: a Datadog query with its range, or a command with its output, all kept in the session's record. The prompt and the session's changed files show it altered no code. Several incidents at once get several runs, each in a sandbox of its own. ## Frequently asked questions ### Can the agent mute monitors or change dashboards? The prompt forbids changing anything in Datadog, and the session records every tool call, so you can confirm it only read. The connection acts as the person who connected it, so it can never do more than that person could in Datadog. ### We only ship the Rails log, not action.log or slow_query.log. Is that enough? Often. The Rails log still carries each request and its exception. The joins by request_token and the per-query timings need the lograge and slow query files, so the report says which it had and what it could not confirm without them. ### Will it confuse rack-attack throttling with an outage? No. The initializer answers 429 with RateLimit headers on purpose, and the agent reads it before the logs. A burst of 429s from one address is reported as throttling working, separate from the 500s that started the alert. ### Will it run the app against production data? No. It creates the SQLite databases from config/database.yml.sample in its sandbox, loads the schema and runs the request specs there. It never touches the storage directory of your production host.