# Make Celery Tasks Survive a Worker Crash With acks_late

Yes. An agent turns on task_acks_late and reject_on_worker_lost, decides per task whether a redelivery is safe, keeps email at most once, tests it and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-acks-late/celery/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads what happens to a task when a worker dies. In a Django app laid out like pretalx/pretalx, `src/pretalx/celery_app.py` creates `Celery("pretalx")`, reads every `CELERY_*` setting from Django with `config_from_object`, and connects a `task_failure` handler that emails the admins except on `WorkerLostError`, with a comment saying those are assumed to come from restarts sending SIGTERM. `src/pretalx/settings.py` sets, when `[celery] broker` is configured, `CELERY_BROKER_URL`, `CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True`, `CELERY_RESULT_BACKEND`, `CELERY_RESULT_BACKEND_THREAD_SAFE = True` and `CELERY_TASK_ALWAYS_EAGER = False`, and nothing about acknowledgement. Celery's default is to acknowledge a message when a worker takes it, so a worker killed by the OOM killer, a `kill -9` or a host reboot in the middle of a task loses that task: the broker has already forgotten it. For a queued mail the trace is visible in `src/pretalx/mail/domain/send.py`, which sets the row to `SENDING` and then calls `task_send_draft.apply_async`, and in `src/pretalx/mail/domain/queue.py`, where `expire_stale_queued_mails` resets rows left in `SENDING` for over an hour back to `DRAFT` with a `TimeoutError` note, run from the `periodic_task` signal with a fifteen-minute minimum interval. A crash costs that mail an hour and a periodic run.

The setting itself is two lines: `CELERY_TASK_ACKS_LATE = True` and `CELERY_TASK_REJECT_ON_WORKER_LOST = True`, placed beside the other `CELERY_*` settings but outside the broker condition so they also hold in tests, where `src/pretalx/common/settings/test_settings.py` runs everything eagerly. With them, a message is acknowledged after the task finishes, and a task whose worker process is lost is requeued rather than marked failed. The agent pairs them with `CELERY_WORKER_PREFETCH_MULTIPLIER = 1`, since a worker that has prefetched a batch and dies now hands the whole batch back, and one-at-a-time prefetch keeps that small. The `pretalx.example.cfg` and `doc/administrator/configure.rst` entries for the `[celery]` section need no change, because these are code settings rather than operator configuration.

The real work is the audit, because late acknowledgement means a task can run twice. The agent reads each of the fifteen tasks registered with `@app.task` and sorts them. `pretalx.process_image`, `pretalx.generate_thumbnails` and `pretalx.cleanup_file` in `common/tasks.py` re-read the model row and check the file still matches before acting, so a second run is harmless. `pretalx.export_table`, `pretalx.agenda.export_schedule_html`, `pretalx.submission.export_question_files`, `pretalx.submission.recalculate_review_scores`, `pretalx.schedule.update_unreleased_schedule_changes`, `pretalx.event.periodic_event_services` and `pretalx.common.update_check` recompute from the database and overwrite, so they are safe to repeat. The mail tasks are not: `pretalx.mail.send_draft` in `mail/tasks.py` hands the row to SMTP and only then calls `mark_sent`, so a worker lost between the two would make a redelivery send the same message again, and `pretalx.mail.send_transient` has no row at all. For those, and for `send_outbox_mails`, `generate_mails` with `skip_queue` and `pretalx.submission.send_initial_mails`, the agent sets `acks_late=False` on the decorator, so email keeps today's at-most-once behaviour, and writes the reason as a comment above each one.

Tests follow the pattern in `src/tests/mail/test_tasks.py`, which calls tasks directly or with `.apply()` under `pytestmark = [pytest.mark.django_db]`. A new test module asserts `app.conf.task_acks_late` and `app.conf.task_reject_on_worker_lost` are true, and that each mail task's `acks_late` attribute is false, so a future task added without a decision shows up as a failing assertion rather than a surprise in production. `pyproject.toml` turns every warning into an error under `filterwarnings`, so the agent runs the suite with `uv run pytest src/tests/mail src/tests/common` and the new module. For a live check it installs Redis in the sandbox, points `PRETALX_CELERY_BROKER` and `PRETALX_CELERY_BACKEND` at it as `common/settings/config.py` reads them, starts `celery -A pretalx.celery_app worker -l info -c 1`, enqueues `pretalx.generate_thumbnails` for a seeded image, kills the worker's child process with SIGKILL while it runs, starts a worker again and quotes the log line showing the task delivered a second time.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Make Celery tasks survive a worker crash. Read src/pretalx/celery_app.py, src/pretalx/settings.py, src/pretalx/common/settings/config.py, src/pretalx/common/settings/test_settings.py, every tasks.py under src/pretalx, src/pretalx/mail/domain/send.py, src/pretalx/mail/domain/queue.py and src/tests/mail/test_tasks.py first. Set CELERY_TASK_ACKS_LATE, CELERY_TASK_REJECT_ON_WORKER_LOST and CELERY_WORKER_PREFETCH_MULTIPLIER of 1 outside the broker condition. Audit every registered task: leave late acknowledgement on for the ones that re-read state or recompute, and set acks_late to False on the decorator of every task that sends email, with a comment saying why. Add a test module that asserts the app settings and each mail task's acks_late attribute. Run the mail and common test suites with pytest, start a worker against a local Redis, kill its child during a thumbnail task and show the redelivery in the log, quote the results and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that changes `src/pretalx/settings.py`, adds `acks_late=False` with a comment to the decorators in `src/pretalx/mail/tasks.py` and `src/pretalx/submission/tasks.py`, and adds one test module under `src/tests/common/`. No task body changes, and the operator-facing configuration files keep their current keys.

The description holds the audit as a table, one row per task name with its verdict and the line that justifies it, the pytest summary for the suites the agent ran, and the two worker logs: the first ending with the SIGKILL and the second showing the same task id received and finished. The repository's `Tests` workflow runs its Python and database matrix on the branch and appears beside the agent's record.

Before merging, a reviewer decides whether `send_draft` should instead become safe to repeat by recording an attempt before the SMTP call, which would let email move to at-least-once too, and whether the stale-mail expiry window of one hour still makes sense once crashed tasks are redelivered. The agent leaves both as they are and names them.

## Run it on every GitHub issue labelled reliability

In the Project that holds the repository, create an automation with a general form of this page's prompt and a `github` trigger on `issues.labeled` with `reliability` in its `labels`. The trigger puts the issue after the prompt and fills in nothing, so the prompt says: read the issue that follows, make the Celery change it describes, audit every task the change affects, prove it against a local worker, and open a pull request that links the issue. It fires only when the label is added by someone who can write to the repository.

An on-call engineer who finds thumbnails missing after a node reboot files the issue with the worker log; the run reads it, makes the change and opens the pull request with the audit and the redelivery log in it. The next reliability issue, say time limits for the export tasks, starts its own run on its own branch.

## Why run it on Rush?

Proving that a task survives a crash means a broker, a worker, a task in flight, a SIGKILL at the right moment and a second worker to read the redelivery, which no one sets up on a laptop for a two-line settings change. The sandbox installs Redis, runs the worker, kills it and keeps both logs, so the pull request opens with the evidence in it.

The run's record holds the pytest output and the two worker logs, and the audit lists each task with the line that decided it, so a reviewer checks reasons rather than a claim that the tasks are idempotent. Across a company's Celery services the same request runs at once, one sandbox each, with each pull request auditing that service's own task list.

## Frequently asked questions

### Why keep acks_late off for email?

`task_send_draft` sends over SMTP and only afterwards marks the row sent, so a worker lost between those two steps would make a redelivered task send the message twice. Keeping the default for mail tasks means a crash costs one delayed message, which the hourly stale-mail expiry already handles, rather than a duplicate.

### Does this change how tests run?

No. `test_settings.py` keeps `CELERY_TASK_ALWAYS_EAGER = True`, so tasks still run in-process. The new settings sit outside the broker condition only so the assertion test can read them from `app.conf` in that mode.

### Why set the prefetch multiplier at the same time?

With late acknowledgement, every message a worker has prefetched but not finished goes back to the queue when it dies. A multiplier of 1 means one message in flight per process, so a crash returns one task rather than a batch.

### What about tasks a worker restart interrupts on purpose?

A SIGTERM lets Celery finish the running task before exiting, which is what the `WorkerLostError` comment in `celery_app.py` assumes for restarts. Redelivery applies to a process that disappears, as after SIGKILL or an out-of-memory kill, not to a warm shutdown.

## More Celery jobs

- [Add a Periodic Task with Celery Beat](https://getrush.ai/run/add-periodic-tasks/celery/)
- [Add Prometheus Metrics to a Celery App](https://getrush.ai/run/add-prometheus-metrics/celery/)
- [Cancel a Running Celery Task](https://getrush.ai/run/add-task-cancellation/celery/)

## More jobs like this: Turn a Linear label into a PR

- [Add an Output to a Custom GitHub Action from a Linear Issue](https://getrush.ai/run/add-action-output/github-actions/)
- [Add Activity Heartbeats to a Temporal Workflow](https://getrush.ai/run/add-activity-heartbeats/temporal/)
- [Add Analytics to an Expo App](https://getrush.ai/run/add-analytics/expo/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
