What the agent does
The agent first decides which tasks should retry at all. In an app like miguelgrinberg/flask-celery-example, app.py has two. send_async_email is declared with a bare @celery.task and calls mail.send(msg) against smtp.googlemail.com on port 587, so a dropped connection or a busy server raises once, the task ends in FAILURE and the email is gone. The index route calls delay or apply_async(..., countdown=60) and throws the result object away, so nobody would notice. long_task only sleeps and reports progress with update_state; it talks to nothing outside the worker and gets no retry policy. The agent writes that reasoning into the pull request, because a retry on a task that cannot fail transiently only hides bugs.
Next it sorts the errors. A refused or reset connection, a timeout, SMTPServerDisconnected and a server reply in the 400 range are worth another attempt. SMTPAuthenticationError and SMTPRecipientsRefused are not: the same message will fail the same way, and retrying a bad password can lock an account. Since autoretry_for takes exception classes and cannot look at a reply code, the agent uses it for the plain connection errors and, for SMTPResponseException, binds the task and calls self.retry(exc=exc) only when smtp_code is below 500. It checks the pinned version before choosing options: requirements.txt freezes celery==4.4.1, and it confirms from that release's documentation which of retry_backoff, retry_backoff_max, retry_jitter and max_retries the version accepts, so the change does not smuggle in an upgrade.
Then it looks at what a retry can break. An email task is not idempotent: if the server accepted the message and the connection dropped before the reply, a retry sends it twice. The agent states that risk, keeps max_retries low, and leaves acknowledgement settings as they are unless asked, because acks_late trades lost tasks for duplicates. It also compares the longest delay a retry can reach, and the existing 60 second countdown, with the Redis transport's visibility timeout, since a task held longer than that timeout is delivered again.
It proves the policy two ways. Unit tests run the task in process with mail.send replaced: failing twice with a connection error and then succeeding must end in success after three calls, an authentication error must fail after one, and a reply code of 451 must retry while 550 must not. Then it installs and starts Redis in the sandbox, runs a worker from a small script that points the mail settings at a closed port on loopback, queues one email and reads the worker log for the retry lines and their growing delays until the task gives up.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add retries with exponential backoff to this app's Celery tasks that call outside services, and only to those. Read every Python file, requirements.txt and the README first. For each task, say what it calls outside the worker, which exceptions that call can raise, and how the task is queued. Give no retry policy to a task that does no outside I/O, and say why. For the tasks that need one, split the exceptions into transient (connection refused or reset, timeouts, server disconnects, temporary server replies) and permanent (authentication failures, refused recipients, validation errors). Retry only the transient ones: use autoretry_for for exception classes, and bind the task and call self.retry for cases that need the error's code. Use backoff with jitter and a low max_retries. Before using an option, check that the pinned Celery version supports it; do not upgrade Celery. Do not change acks_late or other delivery settings; instead write down whether the task is safe to run twice, and compare the longest retry delay and any countdown in the code with the broker's visibility timeout. Add pytest tests that replace the outside call: transient failures followed by success, a permanent failure that must not retry, and exhaustion of max_retries. Then prove it against a real broker: install and start Redis, run a worker with the outside service pointed at a closed loopback port, queue one task, and capture the worker log showing each retry and its delay. Open a pull request with the reasoning, the tests and the log.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request is a small diff with a long description. In app.py the decorator on send_async_email gains its retry options and the body gains the one branch that inspects the SMTP reply code; long_task is untouched. A new test file covers the three behaviours, and pytest goes into a dev requirements file so the pinned runtime set does not change. The description has a table of each exception with its class, whether it retries and why, the delay sequence the options produce, the test output, and the worker log from the sandbox with one line per attempt.
It ends with the decisions that are yours. Whether a duplicate email is acceptable, whether a final failure should alert someone, since the route discards the result and the only trace is the worker log, and whether to turn on late acknowledgement are listed with the trade-off for each and no code. A reviewer checks the exception table against how their mail provider actually fails, confirms the total retry window is shorter than the broker's visibility timeout, and reads the log to see that the delays grow and that the task stops.
Run it on every Linear issue labelled reliability
Create an automation in the Project that holds the repository with a linear trigger on linear.issue.labeled and reliability in its labels. It needs a Linear connection for whoever runs the automation, made in the console under Settings → Connections, and it fires only when the label is added by someone in your Rush org. The trigger adds the issue after the saved prompt and fills nothing in, so the prompt is this page's request plus one line: limit the work to the task the issue names and mention the issue in the pull request.
An on-call engineer who sees a task fail on a timeout files the issue with the traceback and adds the label. The run reads the traceback as part of the issue, finds the task, and comes back with a policy for that one failure and a test that reproduces it. Several labelled issues start several runs, up to the automation's max runs per hour, and the rest wait as queued.
The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.
Why run it on Rush?
Retry settings are easy to type and hard to trust. The decorator accepts the options without complaint whether or not the delays grow, the right errors are caught or the task ever stops. Finding out needs a broker, a worker and an outside service that fails on demand, which is more setup than most people do for a few keyword arguments. The agent builds that in its sandbox, points the task at a port nothing listens on, and keeps the worker log in the run's record, so the pull request shows the attempts and their spacing as they happened in that sandbox.
The judgment is in what not to retry, and the agent writes it down. Each exception gets a row and a reason, each task without a policy gets a sentence, and the delivery settings it declined to touch are listed with what they would cost. That makes the review a matter of disagreeing with specific rows. A backlog of flaky tasks across several services becomes one labelled issue each, and each run reads its own task and its own pinned version.
Frequently asked questions
Will it retry every exception?
No. Only errors that can succeed on a later attempt, such as a refused connection, a timeout or a temporary server reply. Authentication failures, refused recipients and anything raised by your own validation fail at once, and the pull request lists which is which.
Could a retry send the same email twice?
It can, when the server accepted the message and the connection failed before the reply arrived. The agent says so, keeps the retry count low and does not change acknowledgement settings. If duplicates are unacceptable, ask for a deduplication key stored before the send and it adds that as a follow-up.
Does this need a newer Celery?
The agent checks the options against the version your requirements pin and uses only what that release supports. If an option you ask for needs a newer release, it says which one and stops there; the upgrade is a separate pull request.
How does it test retries without waiting for the delays?
The unit tests run the task in process, where a retry runs again immediately, and assert on the number of calls and the final state. The delays themselves are read from the worker log in the run against real Redis, with short values set for that one run.