What the agent does
The agent starts where the workers are defined. In a Rails app laid out like codetriage/CodeTriage, Procfile runs bundle exec sidekiq -q default -q mailers -c ${SIDEKIQ_CONCURRENCY:-5}, config/application.rb sets config.active_job.queue_adapter = :sidekiq, and config/initializers/sidekiq.rb points Sidekiq at REDIS_URL and nothing else: no queue weights, no per-job limits, no concurrency above the default of five threads. Queues listed with -q and no weights are checked in strict order, so a worker takes a mailers job only when default is empty. The agent writes that down first, because every later finding depends on it.
Next it finds out what lands on each queue. app/jobs/application_job.rb has queue_as :default and no job overrides it. Mail goes there too: config/application.rb loads the 8.1 defaults, nothing sets deliver_later_queue_name, and test/jobs/send_daily_triage_email_job_test.rb asserts that ActionMailer::MailDeliveryJob is enqueued on default. So the mailers queue the Procfile reserves is empty, and every email waits behind every other job. The agent greps for perform_later and deliver_later and lists each enqueue site: the subscription controllers, Repo#update_repo_info!, User#subscribe_docs! and DocMailerMaker.
Then it reads the fan-out. lib/tasks/schedule.rake defines the tasks an external scheduler runs. schedule:triage_emails enqueues one SendDailyTriageEmailJob per user with a subscription, and its own TODO says the filtering should happen before the jobs reach Redis; each of those jobs that sends then enqueues a mail job. schedule:populate_issues enqueues one PopulateIssuesJob per active repository, and each one walks every page of GitHub issues with fetcher.call(retry_on_bad_token: 3) before calling force_issues_count_sync!. schedule:process_repos enqueues PopulateDocsJob, which clones the repository with fetcher.clone and parses it inside ActiveRecord::Base.uncached, the slowest work in the app, on the same five threads. The agent notes that the times these tasks run are set in the scheduler, not in the repository, so the report asks for them rather than guessing.
Last, it checks what would keep a backlog from clearing. app.json and Procfile describe one worker process, the Gemfile has rails-autoscale-web with no Sidekiq adapter beside it, and Sidekiq's default retry behaviour is untouched, so a job that fails on GitHub errors comes back later in the same queue. The agent also reads config/initializers/sentry.rb, which drops Sidekiq::Shutdown, and config/routes.rb, which mounts Sidekiq::Web at /sidekiq for admins, so the report can point at a page the team already has. It changes no code.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Incident: the Sidekiq default queue on this Rails app backs up every day and triage emails go out hours late. Read Procfile, Procfile.concurrency, app.json, config/application.rb, config/initializers/sidekiq.rb, every file in app/jobs, lib/tasks/schedule.rake and every perform_later and deliver_later call. Write incident.md with what each queue receives, the fan-out each scheduled rake task causes, each suspect ranked with file and line evidence, and the exact Sidekiq API or console command that would confirm or clear it. Change no code.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is incident.md, attached to the session as an artifact. It opens with a table of queues: default receiving every application job and every mail delivery, mailers receiving nothing, and the worker reading them in strict order with five threads. A second table lists each scheduled task in schedule.rake with the job it enqueues, how many it enqueues (one per subscribed user, one per active repository) and what each of those jobs calls in turn.
The suspects follow in order, each with its file and line: mail sharing default with repository work, PopulateDocsJob clones holding threads, the per-user fan-out in triage_emails, and retries of failing GitHub calls landing back in the same queue. Each carries the check that settles it, such as Sidekiq::Queue.new("default").latency and Sidekiq::Queue.new("mailers").size from rails c, a count of jobs in the queue grouped by class, the Busy tab at /sidekiq during the morning run, or the scheduler's list of task times.
The report ends with the changes it would propose, each as a separate follow-up rather than a patch: give mail its own queue with deliver_later_queue_name, weight or split the queues, move clones to a dedicated queue with a lower concurrency, and filter users before enqueuing. A reviewer runs the checks against production before choosing any of them.
Run it on every queue latency alert
A Sidekiq latency check is a few lines on your side: a scheduled job or monitor reads Sidekiq::Queue.new("default").latency, and when it passes your threshold a handler on your server sends this request with the number and the time in the prompt. Grafana contact points can also call a Rush webhook trigger directly, since they can send an Authorization: Bearer header with the trigger's secret.
Each run reads the repository as it is that day, so a report written after a new job or a new scheduled task names it. With a webhook URL on the API key that starts the run, Rush POSTs run.completed with the session_url when the turn ends, and the report is waiting for whoever is on call.
The use case this job follows, with its automation and trigger: Investigate an incident.
Why run it on Rush?
A queue backlog investigation is mostly reading: the Procfile, the jobs, the rake tasks and the Rails defaults that decide where mail goes. On a laptop that reading competes with the incident itself. In its own sandbox the agent reads the whole repository, greps every enqueue site and keeps each command and its output in the run's record, so the claim that mail lands on default comes with the test file that proves it.
The result is a document with evidence, not a fix pushed under pressure. The run changes nothing, and each suspect carries a check the team can run against the live Redis before deciding. When the same queue backs up again after a deploy, a second run reads the new code and the two reports can be compared line by line.
Frequently asked questions
Does the agent connect to our Redis or our Sidekiq dashboard?
No. It reads the repository only and writes the commands to run against production. Giving a run access to live Redis would be a separate decision, and the report is written so that someone with that access can confirm each suspect directly.
Why would mail land on the default queue when the Procfile lists mailers?
Since Rails 6.1, deliver_later uses the job's default queue unless config.action_mailer.deliver_later_queue_name is set, and this app sets it nowhere. The repository's own test asserts the mail job is enqueued on default, so the -q mailers flag watches a queue nothing writes to.
What does strict queue ordering mean for a backlog?
When Sidekiq is started with -q default -q mailers and no weights, a thread only takes from mailers when default is empty. Any queue listed after a busy one can starve, which is why the report lists queue order before anything else.
Can it tell when the scheduled tasks run?
Not from this repository. schedule.rake defines the tasks but an external scheduler decides when they start, so the report lists the task names and asks for their times. If your app uses sidekiq-cron or a schedule file, the agent reads that file instead.