All jobs

Add Retry Policies to Sidekiq Jobs in a Rails App

Yes. An agent sorts each Active Job on Sidekiq into retry, discard or retry-once, adds retry_on and discard_on with tests, and opens a pull request.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent first learns how jobs reach Sidekiq. In a Rails app like codetriage/CodeTriage, config/application.rb sets config.active_job.queue_adapter = :sidekiq, every job under app/jobs inherits from ApplicationJob, and no job sets retry_on, discard_on or sidekiq_options. That means every exception falls through to Sidekiq's default: up to 25 retries with exponential backoff spread over about three weeks, whether the error can ever succeed or not. The agent lists each job, what it calls and what it raises, and sorts them into three groups: errors worth retrying, errors that will never pass, and jobs where a retry repeats a side effect.

The second group is the easiest to see. RepoBasedJob and UserBasedJob look up their record in an around_perform with Repo.find and User.find, so a repository or user deleted after the job was enqueued raises ActiveRecord::RecordNotFound 25 times. UpdateRepoInfoJob and BackgroundInactiveEmailJob receive the record itself, so the same deletion surfaces as ActiveJob::DeserializationError before perform runs. The agent adds discard_on ActiveRecord::RecordNotFound, ActiveJob::DeserializationError to the two base classes and to those jobs, each with a one-line log so a discarded job is still visible.

The first group is GitHub. PopulateIssuesJob#populate_issues raises a bare RuntimeError reading Error grabbing issues status when the response body is not an array, which is what a GitHub outage or a secondary rate limit looks like. The agent replaces the bare raise with a named error class defined in the job, then adds retry_on for it with wait: :polynomially_longer and a small attempts count, so a short outage clears on its own and a long one stops retrying within hours rather than weeks. fetcher.call(retry_on_bad_token: 3) already retries a bad token in place, and the agent leaves that loop alone.

The third group is mail. SendDailyTriageEmailJob#send_email calls deliver_later first and then marks assignments delivered and stamps last_sent_at, so a failure in either update_all makes the retry send the same email again. The agent moves both updates into one transaction before the delivery is enqueued and explains the trade in the pull request: a lost email instead of a duplicate one. To test it all, the agent installs the Ruby that .ruby-version names and PostgreSQL in its sandbox, copies config/database.ci.yml as the CI workflow does, runs bin/rake db:create db:schema:load, writes tests in test/jobs under the :test adapter from config/environments/test.rb, and runs bin/rails test and bundle exec standardrb.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add retry policies to the Active Job classes that run on Sidekiq. Read config/application.rb, config/initializers/sidekiq.rb, Procfile, every file in app/jobs and the models they call. Discard jobs whose record was deleted (RecordNotFound in RepoBasedJob and UserBasedJob, DeserializationError for jobs given a record) with a log line. Give the GitHub failure in PopulateIssuesJob a named error class and retry it with polynomial backoff and a small attempts count. Make SendDailyTriageEmailJob safe to retry without sending the same email twice. Add tests in test/jobs, run bin/rails test and bundle exec standardrb, and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that changes app/jobs/repo_based_job.rb, user_based_job.rb, update_repo_info_job.rb, background_inactive_email_job.rb, populate_issues_job.rb and send_daily_triage_email_job.rb, and adds tests beside the existing ones in test/jobs. The description has a table with one row per job: what it raised before, what it does now (retry with how many attempts, discard, or none), and why.

The new tests use ActiveJob::TestCase: one deletes the record after enqueueing and asserts the job is discarded without raising, one stubs the GitHub issues request with WebMock to return a non-array body and asserts the job is enqueued again with a wait, and one forces the assignment update to fail and asserts no mail job was enqueued. The bin/rails test and standardrb output are in the description, and the repository's Main workflow runs the same steps on the branch.

Before merging, a reviewer checks the attempt count against how long a GitHub outage usually lasts for them, and decides whether a missed daily email is acceptable or whether they want the stricter fix: an idempotency check on the assignment rather than an ordering change.

Run it on every Linear issue labelled jobs

Create an automation in the Project that holds the repository, with this page's prompt made general and a linear trigger on linear.issue.labeled with a label such as jobs in config.labels. The issue is added after the prompt, so the prompt says to read the job named in the issue that follows, decide whether its errors should be retried, discarded or neither, write the test and open a pull request that links the issue.

A job that fails in Sentry 25 times for a deleted record becomes one labelled issue and one small pull request. An automation run has its own max_runs_per_hour cap, so a burst of labels from a cleanup session queues rather than starting all at once.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

A retry policy is only right if the test shows the failure it is for. That test needs Postgres, the repository's fixtures, WebMock and the Active Job test adapter, and on a laptop it often waits for someone to set them up. The sandbox installs them, loads the schema and keeps every test command with its output in the run's record, so the pull request shows the discard and the retry happening instead of saying they would.

Sorting jobs by what their errors mean is careful reading across app/jobs, the models and the mailers, and it is the kind of change that goes stale in a branch. A run does the reading and opens a pull request the team can review in one sitting, and a second run for the next app's jobs starts alongside it without sharing anything.

Frequently asked questions

Should I use retry_on or sidekiq_options retry?

retry_on is handled by Active Job before Sidekiq sees the error, so it can pick the wait, the attempt count and the exceptions per job, and it shows up in Active Job tests. When its attempts run out the error is raised again and Sidekiq's own retry takes over, so the agent also sets sidekiq_options retry: on jobs where that second round would be wrong.

Will discard_on hide real bugs?

It drops only the exceptions it names, and the agent limits those to a record that no longer exists. Each discard writes a log line with the job class and the missing id, so a sudden rise is still visible in your logs.

Does this change jobs already sitting in the retry set?

Jobs already scheduled for retry run with the new code when they come up, so a discard rule applies to them too. Nothing in the pull request clears the retry set; that stays a decision for whoever runs production.

Does it need Redis to run the tests?

No. config/environments/test.rb uses the :test adapter, which records jobs in memory, so the tests need Postgres and the Ruby gems only.

More Sidekiq jobs

The same job on other stacks

Keep reading