Investigate a Celery Task Backlog in a Django App

The agent starts from the symptom you give it (the default queue at 50,000 tasks, the oldest waiting 40 minutes, workers reporting idle) and from how the workers are started, because idle workers next to a full queue usually means they are not consuming it. In a Django app laid out like HackSoftware's Django Styleguide Example, that is three places that must agree: the Procfile (worker: ... celery -A styleguide_example.tasks worker -l info --without-gossip --without-mingle --without-heartbeat), docker/celery_entrypoint.sh and the celery service in docker-compose.yml, all pointing at RabbitMQ through CELERY_BROKER_URL. It notes that heartbeats are off, which matters for spotting a broker connection that dropped without the worker noticing.

Then it reads the Celery settings and every task. config/settings/celery.py sets CELERY_RESULT_BACKEND = "django-db" (every task writes a result row to Postgres through django_celery_results), CELERY_TASK_SOFT_TIME_LIMIT = 20 and CELERY_TASK_TIME_LIMIT = 30, and CELERY_TASK_MAX_RETRIES = 3, a name the agent checks against Celery 5.5.3's settings list to see whether it does anything at all. In styleguide_example/emails/tasks.py, email_send calls self.retry(exc=exc, countdown=5) on any exception, and config/settings/email_sending.py defines EMAIL_SENDING_FAILURE_TRIGGER with a default EMAIL_SENDING_FAILURE_RATE of 0.2, so with the trigger on in an environment, a fifth of sends fail and come back five seconds later. email_send_all in emails/services.py enqueues one task per email from a loop with transaction.on_commit, so a bulk send is a burst of thousands of messages at once.

It also reads the less obvious paths. styleguide_example/blog_examples/celery_non_concurrent/tasks.py wraps a task in a decorator that calls inspect().active() at line 21 on every run, a broadcast that waits for every worker to reply before the task does its work. Beat runs with django_celery_beat.schedulers:DatabaseScheduler, and setup_periodic_tasks deletes and recreates every schedule, so a re-run during a deploy can enqueue overdue tasks together. Then it reads the last 30 commits with git log -p, looking for changes to worker startup, settings and task signatures; in this repository that includes the structlog work merged in #549, which adds DjangoStructLogInitStep to app.steps["worker"] in styleguide_example/tasks/celery.py.

Finally it ranks the suspects by how well each explains all three symptoms at once, not only the queue depth, and writes incident.md. With the Datadog connection named on the run, it also reads the queue depth, consumer count and worker metrics around the start of the incident and puts them on the timeline. It changes no code.

What comes back is incident.md, attached to the session as an artifact with a download link, plus the agent's answer summarising the top suspect. The report opens with a timeline: commits that touched worker startup, settings or tasks in the window, and, when Datadog was read, when the queue started growing and whether the consumer count changed at the same moment. Then come the suspects, ranked, each with the file and line it rests on, such as the disabled heartbeat in Procfile line 3, the retry on every exception in emails/tasks.py line 29, or the broadcast in celery_non_concurrent/tasks.py line 21.

Each suspect ends with the check that would confirm or clear it, written to be run as-is: celery -A styleguide_example.tasks inspect active_queues to see which queues each worker consumes, rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers to see whether the queue has any consumers, celery inspect reserved to find tasks a worker prefetched but never started, or a query on django_celery_results_taskresult grouped by task_name and status to see which task is failing and retrying. The trajectory keeps every file the agent read and every command it ran, so each claim in the report can be traced.

The report is a first pass, not a verdict. An on-call engineer runs the top two checks, crosses off what they clear, and decides the fix; the agent did not restart, purge or change anything.

Point a Datadog monitor on queue depth or task age at your own server with a webhook, and have it send this request with the monitor's numbers in input and the repository in resources. The investigation then starts when the alert fires, and the report is waiting by the time someone opens the page, instead of the first ten minutes going to reading celery.py from memory.

Set a webhook URL on the API key and Rush POSTs run.completed when the turn finishes, with the agent's answer (up to 16 KiB) and the session_url; your server can read the session for the incident.md artifact's link and post both into the incident channel. Send the alert's id as the Idempotency-Key so a monitor that re-notifies does not start a second run.

During an incident, the people on call are needed on the dashboards and the broker, not on reading every task module and the last 30 diffs. On Rush the agent does that reading in its own sandbox with its own clone, so the first pass happens in parallel with the people, and nothing it does touches production: it reads code, history and, if connected, metrics.

The report is built to be checked. Every suspect carries a file and line, every check is a command someone can paste, and the trajectory shows what the agent read to reach each claim. Two incidents at once, or the same question asked of three services, are separate runs in separate sandboxes, each with its own report.

Does the agent connect to RabbitMQ or our workers?

How does it get the Datadog metrics?

Can it tell a code bug from an infrastructure problem?

Where does incident.md end up?