All jobs

Make Celery Tasks Survive a Worker Crash With acks_late

Yes. An agent turns on task_acks_late and reject_on_worker_lost, decides per task whether a redelivery is safe, keeps email at most once, tests it and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads what happens to a task when a worker dies. In a Django app laid out like pretalx/pretalx, src/pretalx/celery_app.py creates Celery("pretalx"), reads every CELERY_* setting from Django with config_from_object, and connects a task_failure handler that emails the admins except on WorkerLostError, with a comment saying those are assumed to come from restarts sending SIGTERM. src/pretalx/settings.py sets, when [celery] broker is configured, CELERY_BROKER_URL, CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True, CELERY_RESULT_BACKEND, CELERY_RESULT_BACKEND_THREAD_SAFE = True and CELERY_TASK_ALWAYS_EAGER = False, and nothing about acknowledgement. Celery's default is to acknowledge a message when a worker takes it, so a worker killed by the OOM killer, a kill -9 or a host reboot in the middle of a task loses that task: the broker has already forgotten it. For a queued mail the trace is visible in src/pretalx/mail/domain/send.py, which sets the row to SENDING and then calls task_send_draft.apply_async, and in src/pretalx/mail/domain/queue.py, where expire_stale_queued_mails resets rows left in SENDING for over an hour back to DRAFT with a TimeoutError note, run from the periodic_task signal with a fifteen-minute minimum interval. A crash costs that mail an hour and a periodic run.

The setting itself is two lines: CELERY_TASK_ACKS_LATE = True and CELERY_TASK_REJECT_ON_WORKER_LOST = True, placed beside the other CELERY_* settings but outside the broker condition so they also hold in tests, where src/pretalx/common/settings/test_settings.py runs everything eagerly. With them, a message is acknowledged after the task finishes, and a task whose worker process is lost is requeued rather than marked failed. The agent pairs them with CELERY_WORKER_PREFETCH_MULTIPLIER = 1, since a worker that has prefetched a batch and dies now hands the whole batch back, and one-at-a-time prefetch keeps that small. The pretalx.example.cfg and doc/administrator/configure.rst entries for the [celery] section need no change, because these are code settings rather than operator configuration.

The real work is the audit, because late acknowledgement means a task can run twice. The agent reads each of the fifteen tasks registered with @app.task and sorts them. pretalx.process_image, pretalx.generate_thumbnails and pretalx.cleanup_file in common/tasks.py re-read the model row and check the file still matches before acting, so a second run is harmless. pretalx.export_table, pretalx.agenda.export_schedule_html, pretalx.submission.export_question_files, pretalx.submission.recalculate_review_scores, pretalx.schedule.update_unreleased_schedule_changes, pretalx.event.periodic_event_services and pretalx.common.update_check recompute from the database and overwrite, so they are safe to repeat. The mail tasks are not: pretalx.mail.send_draft in mail/tasks.py hands the row to SMTP and only then calls mark_sent, so a worker lost between the two would make a redelivery send the same message again, and pretalx.mail.send_transient has no row at all. For those, and for send_outbox_mails, generate_mails with skip_queue and pretalx.submission.send_initial_mails, the agent sets acks_late=False on the decorator, so email keeps today's at-most-once behaviour, and writes the reason as a comment above each one.

Tests follow the pattern in src/tests/mail/test_tasks.py, which calls tasks directly or with .apply() under pytestmark = [pytest.mark.django_db]. A new test module asserts app.conf.task_acks_late and app.conf.task_reject_on_worker_lost are true, and that each mail task's acks_late attribute is false, so a future task added without a decision shows up as a failing assertion rather than a surprise in production. pyproject.toml turns every warning into an error under filterwarnings, so the agent runs the suite with uv run pytest src/tests/mail src/tests/common and the new module. For a live check it installs Redis in the sandbox, points PRETALX_CELERY_BROKER and PRETALX_CELERY_BACKEND at it as common/settings/config.py reads them, starts celery -A pretalx.celery_app worker -l info -c 1, enqueues pretalx.generate_thumbnails for a seeded image, kills the worker's child process with SIGKILL while it runs, starts a worker again and quotes the log line showing the task delivered a second time.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Make Celery tasks survive a worker crash. Read src/pretalx/celery_app.py, src/pretalx/settings.py, src/pretalx/common/settings/config.py, src/pretalx/common/settings/test_settings.py, every tasks.py under src/pretalx, src/pretalx/mail/domain/send.py, src/pretalx/mail/domain/queue.py and src/tests/mail/test_tasks.py first. Set CELERY_TASK_ACKS_LATE, CELERY_TASK_REJECT_ON_WORKER_LOST and CELERY_WORKER_PREFETCH_MULTIPLIER of 1 outside the broker condition. Audit every registered task: leave late acknowledgement on for the ones that re-read state or recompute, and set acks_late to False on the decorator of every task that sends email, with a comment saying why. Add a test module that asserts the app settings and each mail task's acks_late attribute. Run the mail and common test suites with pytest, start a worker against a local Redis, kill its child during a thumbnail task and show the redelivery in the log, quote the results and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that changes src/pretalx/settings.py, adds acks_late=False with a comment to the decorators in src/pretalx/mail/tasks.py and src/pretalx/submission/tasks.py, and adds one test module under src/tests/common/. No task body changes, and the operator-facing configuration files keep their current keys.

The description holds the audit as a table, one row per task name with its verdict and the line that justifies it, the pytest summary for the suites the agent ran, and the two worker logs: the first ending with the SIGKILL and the second showing the same task id received and finished. The repository's Tests workflow runs its Python and database matrix on the branch and appears beside the agent's record.

Before merging, a reviewer decides whether send_draft should instead become safe to repeat by recording an attempt before the SMTP call, which would let email move to at-least-once too, and whether the stale-mail expiry window of one hour still makes sense once crashed tasks are redelivered. The agent leaves both as they are and names them.

Run it on every GitHub issue labelled reliability

In the Project that holds the repository, create an automation with a general form of this page's prompt and a github trigger on issues.labeled with reliability in its labels. The trigger puts the issue after the prompt and fills in nothing, so the prompt says: read the issue that follows, make the Celery change it describes, audit every task the change affects, prove it against a local worker, and open a pull request that links the issue. It fires only when the label is added by someone who can write to the repository.

An on-call engineer who finds thumbnails missing after a node reboot files the issue with the worker log; the run reads it, makes the change and opens the pull request with the audit and the redelivery log in it. The next reliability issue, say time limits for the export tasks, starts its own run on its own branch.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

Proving that a task survives a crash means a broker, a worker, a task in flight, a SIGKILL at the right moment and a second worker to read the redelivery, which no one sets up on a laptop for a two-line settings change. The sandbox installs Redis, runs the worker, kills it and keeps both logs, so the pull request opens with the evidence in it.

The run's record holds the pytest output and the two worker logs, and the audit lists each task with the line that decided it, so a reviewer checks reasons rather than a claim that the tasks are idempotent. Across a company's Celery services the same request runs at once, one sandbox each, with each pull request auditing that service's own task list.

Frequently asked questions

Why keep acks_late off for email?

task_send_draft sends over SMTP and only afterwards marks the row sent, so a worker lost between those two steps would make a redelivered task send the message twice. Keeping the default for mail tasks means a crash costs one delayed message, which the hourly stale-mail expiry already handles, rather than a duplicate.

Does this change how tests run?

No. test_settings.py keeps CELERY_TASK_ALWAYS_EAGER = True, so tasks still run in-process. The new settings sit outside the broker condition only so the assertion test can read them from app.conf in that mode.

Why set the prefetch multiplier at the same time?

With late acknowledgement, every message a worker has prefetched but not finished goes back to the queue when it dies. A multiplier of 1 means one message in flight per process, so a crash returns one task rather than a batch.

What about tasks a worker restart interrupts on purpose?

A SIGTERM lets Celery finish the running task before exiting, which is what the WorkerLostError comment in celery_app.py assumes for restarts. Redelivery applies to a process that disappears, as after SIGKILL or an out-of-memory kill, not to a warm shutdown.

More Celery jobs

More jobs like this: Turn a Linear label into a PR

Keep reading