All jobs

Add Retry Policies to Oban Workers in an Elixir App

Yes. An agent reads each Oban worker's options and return values, makes failures retry, cancel or snooze on purpose, adds tests and opens a pull request.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads every use Oban.Worker line and every perform/1 return. In an Elixir app like thechangelog/changelog.com, most workers take Oban's defaults, 20 attempts with exponential backoff, and MailDeliverer sets max_attempts: 1 with unique: [fields: [:args, :queue], period: 60]. Retry settings only matter if a failure reaches Oban, so the agent checks what each perform/1 returns. MailDeliverer.perform/1 calls the mailer through apply/3 and returns :ok no matter what Mailer.deliver/1 answered, and SocialPoster.perform/1 calls Bsky.post, Social.post and Zulip.post in a row and returns :ok even when one of them returns {:error, _}. In both, a failed send is not a failed job.

Mail is the first fix. The agent makes each mailer function's result the job's result, so an SMTP error from the relay in config/prod.exs becomes {:error, reason} and Oban retries it. It raises max_attempts to a small number and keeps the uniqueness window. A record that no longer exists is different: Repo.get/2 returns nil and the email builder crashes on it, so the agent returns {:cancel, :not_found} for that case, which stops the job without retrying. The pull request names the trade: a relay that accepted the message and then timed out could send it twice, which is why the attempt count stays low.

Social posts are the second. Making SocialPoster return the first error would retry the whole job and post again to the networks that already succeeded. So the agent splits it: SocialPoster.queue/1 inserts one job per network with a "network" key in the args, each perform/1 clause posts to one network and returns its result, and the existing uniqueness style keeps a double click from enqueuing twice. OvercastPinger gets a catch-all clause, because today it matches a 200 and a 429 only and any other status raises a CaseClauseError, and a backoff/1 callback that waits longer after a 429 than Oban's default would.

Last, the noise. Changelog.ObanReporter sends every [:oban, :job, :exception] event to Sentry, one per attempt, so more retries would mean more Sentry events for one problem. The agent changes handle_event/4 to report when attempt has reached max_attempts, using the fields it already takes from the job. Tests go in test/changelog/oban_workers with Oban.Testing and perform_job/3, Mock for the outside clients and the mailer, and the agent runs mix test after installing the Erlang and Elixir versions mise.toml pins and a Postgres that matches config/test.exs.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add deliberate retry behaviour to the Oban workers. Read config/config.exs, config/prod.exs, config/test.exs, lib/changelog/oban_reporter.ex and every worker in lib/changelog/oban_workers first. Make MailDeliverer return the mailer's result, retry a few times, and cancel when its record is gone. Split SocialPoster into one job per network so a retry never posts twice. Give OvercastPinger a catch-all clause and a longer backoff after a 429. Make ObanReporter report to Sentry only on the final attempt. Write tests with Oban.Testing for each change, run mix test, and open a pull request that explains each trade-off.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that changes lib/changelog/oban_workers/mail_deliverer.ex, social_poster.ex, overcast_pinger.ex and lib/changelog/oban_reporter.ex, updates the callers of SocialPoster.queue/1 in lib/changelog/notifier.ex only if the signature changes, and adds tests for each worker. The description has one row per worker: attempts and backoff before, after, what a failure returns now, and why.

The tests show each behaviour directly: perform_job/3 on MailDeliverer returns {:error, _} when the mailer fails and {:cancel, :not_found} for a deleted person, SocialPoster.queue/1 enqueues three jobs whose "network" args assert_enqueued can see, and OvercastPinger returns an error tuple for a 500 instead of raising. The mix test output from the sandbox is in the description, and the repository's GitHub workflow runs the suite on the branch.

Before merging, a reviewer decides the attempt count for mail with the relay's behaviour in mind, checks that jobs already in the oban_jobs table with the old SocialPoster args are still handled by a clause, and confirms that Sentry alert rules relying on per-attempt events are adjusted.

Run it on every GitHub issue labelled oban

In the Project that holds the repository, create an automation with a general form of this prompt and a github trigger on issues.labeled with oban in its labels. The issue is added after the prompt, so the prompt says to read the worker the issue that follows names, decide whether its failures should retry, cancel or snooze, write the test and open a pull request that links the issue.

A Sentry issue about a worker that failed 20 times on a deleted record becomes one labelled GitHub issue and one small pull request. The trigger fires only for a label added by a repository owner, member or collaborator, and the run uses the Project's repository.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

Retry behaviour in Oban is decided by return values as much as by options, and getting it right means reading each worker against the modules it calls. Proving it needs Postgres for the jobs table and the pinned Erlang and Elixir, which the sandbox installs; every test run is kept in the run's record with its output, so the reviewer sees {:cancel, :not_found} come back rather than reading that it would.

The changes are small but each one is a judgement about duplicates versus losses, and a pull request that states those trades in a table is easier to review than one written in a hurry after a missed email. A run for each of several Elixir services can start at once, each in its own sandbox with its own database.

Frequently asked questions

What is the difference between {:error, reason} and {:cancel, reason}?

{:error, reason} marks the attempt as failed and Oban schedules another until max_attempts is reached. {:cancel, reason} stops the job at once and records it as cancelled, which fits a failure that can never succeed, such as a record that was deleted.

Why split SocialPoster instead of retrying it?

Posting is not idempotent. If Bluesky succeeds and Zulip fails, retrying the whole job posts to Bluesky again. One job per network lets each retry on its own without repeating the others.

Does a custom backoff replace Oban's default for every worker?

No. backoff/1 is a callback on one worker module, so only OvercastPinger waits longer. Every other worker keeps Oban's default exponential backoff.

What happens to jobs already queued with the old args?

Oban stores args as JSON, so a SocialPoster job inserted before the change still has the old shape. The agent keeps a perform/1 clause that accepts it and fans it out, and says in the pull request when that clause can be removed.

More Oban jobs

The same job on other stacks

Keep reading