All jobs

Add Task Routing to a Celery App

Yes. An agent reads your tasks, routes the slow ones to their own queue with task_routes, proves it with workers in the sandbox, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent starts with what the workers are doing. In an app laid out like miguelgrinberg/flask-celery-example, everything lives in app.py: a Celery(app.name, broker=...) built from CELERY_BROKER_URL and CELERY_RESULT_BACKEND on redis://localhost:6379/0, two tasks, and the Flask routes that call them. send_async_email sends one message through Flask-Mail, either .delay() right away or .apply_async(countdown=60) from the form, and long_task is bind=True, loops for ten to fifty seconds calling self.update_state(state='PROGRESS', ...) with time.sleep(1) between steps, and /status/<task_id> polls its AsyncResult. Both go through the one default celery queue, so a handful of long_task runs started from the page hold every worker process and the email a visitor asked for waits behind them. The README's worker command is celery -A app.celery worker --loglevel=info, and requirements.txt pins celery==4.4.1 with redis==3.4.1.

It splits the queues by what each task costs. task_routes in the Celery config maps app.long_task to a slow queue and app.send_async_email to a mail queue, written with the module-qualified names Celery registers (the agent reads them from celery -A app.celery inspect registered rather than guessing), with task_default_queue left at celery so a task added later still has somewhere to go. The config moves from app.config uppercase keys into celery.conf.update(...) with the lowercase names Celery 4 reads, which is also where task_acks_late=True and worker_prefetch_multiplier=1 go for the slow queue's worker, so a ten-second task is not reserved four at a time by a process that then dies. The countdown=60 path is untouched: ETA tasks sit in the worker that reserved them, and prefetch_multiplier=1 on the mail worker keeps that from starving the queue.

The README's one worker line becomes two, celery -A app.celery worker -Q mail --concurrency 2 and celery -A app.celery worker -Q slow --concurrency 4, plus a note that a worker started with no -Q serves only the default queue and will no longer pick up either task. A Procfile-style run-workers.sh beside run-redis.sh starts both for local use.

In its sandbox the agent installs the pinned requirements, starts Redis with run-redis.sh, starts both workers, and sends one long_task.apply_async() and one send_async_email.delay(...) with MAIL_SUPPRESS_SEND set so Flask-Mail builds the message without an SMTP connection. It reads celery -A app.celery inspect active_queues to show each worker consuming only its queue, and redis-cli LLEN slow before the worker takes the job, and pastes both into the pull request.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add task routing to this Celery app. Read app.py, requirements.txt, run-redis.sh and the README first. Route long_task to a slow queue and send_async_email to a mail queue using task_routes with the names Celery registers; check them with celery -A app.celery inspect registered rather than guessing. Move the Celery settings into celery.conf.update with the lowercase Celery 4 names, keep task_default_queue at celery, and set task_acks_late and worker_prefetch_multiplier 1. Do not change the countdown=60 path or the progress reporting in long_task. Update the README's worker command to one worker per queue and add run-workers.sh beside run-redis.sh. In the sandbox start Redis and both workers, send one task of each kind with MAIL_SUPPRESS_SEND set, and quote inspect active_queues and the redis-cli queue length in the pull request. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request touching app.py, README.md and a new run-workers.sh. The description quotes the inspect registered output that the route names were taken from, the inspect active_queues block showing the mail worker on mail and the slow worker on slow, and the worker log lines where each task was received and succeeded.

The run's record keeps both workers' full logs and the Redis server output, so a reviewer can see the long_task progress updates arriving on the slow worker while the mail worker stays idle. Nothing was sent: MAIL_SUPPRESS_SEND kept Flask-Mail from opening an SMTP connection, and the pull request says so.

Before merging, a reviewer sets the concurrency per worker for the real host, and decides whether task_acks_late is right for send_async_email: with it on, a worker killed mid-send delivers the message twice, and the pull request names that trade-off next to the setting.

Run it on every Linear issue labelled queues

A team with several Celery services makes this a rule. In the Project for each repository, an automation with the prompt above and a linear trigger on linear.issue.labeled for the label queues starts a run when someone files "move the report export off the default queue". The trigger appends the issue after the prompt, so the run reads which task the issue names and routes that one, keeping everything else where it was.

Each run opens its own pull request on its own branch and links the issue. For a repository that already has task_routes, the run extends the existing map rather than adding a second one, and the README's worker list gains one line.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

Routing is a dict, and the mistakes are in the names and the worker flags: a route keyed on the function name instead of the registered app.long_task, a worker started without -Q that silently serves nothing you routed, a prefetch multiplier left at 4 so one worker reserves forty seconds of work. Each one is invisible in a diff and obvious with a worker running. In a Rush sandbox the agent starts Redis and the workers, reads the registered names from Celery itself, and the pull request quotes what the broker saw.

The worker logs stay with the run, so the next person who adds a queue can read exactly what active_queues looked like when the split was made. Across a dozen services on the same pattern the job runs in parallel, one sandbox each, and each pull request has its own proof.

Frequently asked questions

Will the progress bar on the page still work?

Yes. long_task still calls self.update_state and /status/<task_id> still reads the result backend; only the queue the task waits in changes. The result backend stays on the same Redis database.

What happens to the one worker command I run today?

A worker started with no -Q consumes only the default celery queue, and after this change neither task goes there, so it would sit idle. The README and run-workers.sh start one worker per queue; a single worker with -Q celery,mail,slow also works for a laptop.

Why keep task_default_queue?

So a task added next month without a route still has a queue and a worker to run it. The pull request leaves it at celery and the README says a worker for it is still needed if new tasks are added without a route.

Does this need a newer Celery?

No. task_routes, task_acks_late and worker_prefetch_multiplier are the lowercase names Celery 4 reads, and the project pins 4.4.1. An upgrade is a different pull request.

Can the slow queue have a rate limit too?

Yes. Say so in the prompt and the agent adds task_annotations with a rate_limit for app.long_task, which is per worker process, and notes that in the README next to the concurrency flag.

More Celery jobs

More jobs like this: Turn a Linear label into a PR

Keep reading