# Investigate a Sidekiq Queue Backlog in a Rails App

Yes. An agent reads the Procfile, the jobs and the scheduled rake tasks, finds what floods the queue and writes a report with the checks that confirm each cause.

[Run this on Rush](https://cloud.getrush.ai/run/investigate-queue-backlog/sidekiq/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent starts where the workers are defined. In a Rails app laid out like codetriage/CodeTriage, `Procfile` runs `bundle exec sidekiq -q default -q mailers -c ${SIDEKIQ_CONCURRENCY:-5}`, `config/application.rb` sets `config.active_job.queue_adapter = :sidekiq`, and `config/initializers/sidekiq.rb` points Sidekiq at `REDIS_URL` and nothing else: no queue weights, no per-job limits, no concurrency above the default of five threads. Queues listed with `-q` and no weights are checked in strict order, so a worker takes a `mailers` job only when `default` is empty. The agent writes that down first, because every later finding depends on it.

Next it finds out what lands on each queue. `app/jobs/application_job.rb` has `queue_as :default` and no job overrides it. Mail goes there too: `config/application.rb` loads the 8.1 defaults, nothing sets `deliver_later_queue_name`, and `test/jobs/send_daily_triage_email_job_test.rb` asserts that `ActionMailer::MailDeliveryJob` is enqueued on `default`. So the `mailers` queue the Procfile reserves is empty, and every email waits behind every other job. The agent greps for `perform_later` and `deliver_later` and lists each enqueue site: the subscription controllers, `Repo#update_repo_info!`, `User#subscribe_docs!` and `DocMailerMaker`.

Then it reads the fan-out. `lib/tasks/schedule.rake` defines the tasks an external scheduler runs. `schedule:triage_emails` enqueues one `SendDailyTriageEmailJob` per user with a subscription, and its own TODO says the filtering should happen before the jobs reach Redis; each of those jobs that sends then enqueues a mail job. `schedule:populate_issues` enqueues one `PopulateIssuesJob` per active repository, and each one walks every page of GitHub issues with `fetcher.call(retry_on_bad_token: 3)` before calling `force_issues_count_sync!`. `schedule:process_repos` enqueues `PopulateDocsJob`, which clones the repository with `fetcher.clone` and parses it inside `ActiveRecord::Base.uncached`, the slowest work in the app, on the same five threads. The agent notes that the times these tasks run are set in the scheduler, not in the repository, so the report asks for them rather than guessing.

Last, it checks what would keep a backlog from clearing. `app.json` and `Procfile` describe one worker process, the Gemfile has `rails-autoscale-web` with no Sidekiq adapter beside it, and Sidekiq's default retry behaviour is untouched, so a job that fails on GitHub errors comes back later in the same queue. The agent also reads `config/initializers/sentry.rb`, which drops `Sidekiq::Shutdown`, and `config/routes.rb`, which mounts `Sidekiq::Web` at `/sidekiq` for admins, so the report can point at a page the team already has. It changes no code.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Incident: the Sidekiq default queue on this Rails app backs up every day and triage emails go out hours late. Read Procfile, Procfile.concurrency, app.json, config/application.rb, config/initializers/sidekiq.rb, every file in app/jobs, lib/tasks/schedule.rake and every perform_later and deliver_later call. Write incident.md with what each queue receives, the fan-out each scheduled rake task causes, each suspect ranked with file and line evidence, and the exact Sidekiq API or console command that would confirm or clear it. Change no code.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is `incident.md`, attached to the session as an artifact. It opens with a table of queues: `default` receiving every application job and every mail delivery, `mailers` receiving nothing, and the worker reading them in strict order with five threads. A second table lists each scheduled task in `schedule.rake` with the job it enqueues, how many it enqueues (one per subscribed user, one per active repository) and what each of those jobs calls in turn.

The suspects follow in order, each with its file and line: mail sharing `default` with repository work, `PopulateDocsJob` clones holding threads, the per-user fan-out in `triage_emails`, and retries of failing GitHub calls landing back in the same queue. Each carries the check that settles it, such as `Sidekiq::Queue.new("default").latency` and `Sidekiq::Queue.new("mailers").size` from `rails c`, a count of jobs in the queue grouped by class, the Busy tab at `/sidekiq` during the morning run, or the scheduler's list of task times.

The report ends with the changes it would propose, each as a separate follow-up rather than a patch: give mail its own queue with `deliver_later_queue_name`, weight or split the queues, move clones to a dedicated queue with a lower concurrency, and filter users before enqueuing. A reviewer runs the checks against production before choosing any of them.

## Run it on every queue latency alert

A Sidekiq latency check is a few lines on your side: a scheduled job or monitor reads `Sidekiq::Queue.new("default").latency`, and when it passes your threshold a handler on your server sends this request with the number and the time in the prompt. Grafana contact points can also call a Rush `webhook` trigger directly, since they can send an `Authorization: Bearer` header with the trigger's secret.

Each run reads the repository as it is that day, so a report written after a new job or a new scheduled task names it. With a webhook URL on the API key that starts the run, Rush POSTs `run.completed` with the `session_url` when the turn ends, and the report is waiting for whoever is on call.

## Why run it on Rush?

A queue backlog investigation is mostly reading: the Procfile, the jobs, the rake tasks and the Rails defaults that decide where mail goes. On a laptop that reading competes with the incident itself. In its own sandbox the agent reads the whole repository, greps every enqueue site and keeps each command and its output in the run's record, so the claim that mail lands on `default` comes with the test file that proves it.

The result is a document with evidence, not a fix pushed under pressure. The run changes nothing, and each suspect carries a check the team can run against the live Redis before deciding. When the same queue backs up again after a deploy, a second run reads the new code and the two reports can be compared line by line.

## Frequently asked questions

### Does the agent connect to our Redis or our Sidekiq dashboard?

No. It reads the repository only and writes the commands to run against production. Giving a run access to live Redis would be a separate decision, and the report is written so that someone with that access can confirm each suspect directly.

### Why would mail land on the default queue when the Procfile lists mailers?

Since Rails 6.1, `deliver_later` uses the job's default queue unless `config.action_mailer.deliver_later_queue_name` is set, and this app sets it nowhere. The repository's own test asserts the mail job is enqueued on `default`, so the `-q mailers` flag watches a queue nothing writes to.

### What does strict queue ordering mean for a backlog?

When Sidekiq is started with `-q default -q mailers` and no weights, a thread only takes from `mailers` when `default` is empty. Any queue listed after a busy one can starve, which is why the report lists queue order before anything else.

### Can it tell when the scheduled tasks run?

Not from this repository. `schedule.rake` defines the tasks but an external scheduler decides when they start, so the report lists the task names and asks for their times. If your app uses sidekiq-cron or a schedule file, the agent reads that file instead.

## More Sidekiq jobs

- [Review Pull Requests That Change Sidekiq Jobs in Rails](https://getrush.ai/run/review-pull-requests/sidekiq/)
- [Add Retry Policies to Sidekiq Jobs in a Rails App](https://getrush.ai/run/add-retry-policies/sidekiq/)

## The same job on other stacks

- [Investigate a Queue Backlog in a Laravel App](https://getrush.ai/run/investigate-queue-backlog/laravel/)

## More jobs like this: Investigate an incident

- [Investigate a Sentry Error Spike in a Django App](https://getrush.ai/run/investigate-sentry-spike/django/)
- [Investigate a FastAPI App Incident With Sentry](https://getrush.ai/run/investigate-sentry-spike/fastapi/)
- [Investigate a Laravel App Incident With Sentry](https://getrush.ai/run/investigate-sentry-spike/laravel/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
