# Add Rate Limits to Celery Tasks

Yes. An agent adds per-task rate limits to a Celery app, checks them against a real worker and Redis broker in its sandbox, and opens a PR with the worker log.

[Run this on Rush](https://cloud.getrush.ai/run/add-task-rate-limits/celery/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads how tasks are defined and configured before touching a limit. In an app laid out like miguelgrinberg/flask-celery-example, everything is in `app.py`: a Flask app, a Celery instance built with `Celery(app.name, broker=...)` followed by `celery.conf.update(app.config)`, so configuration arrives as uppercase `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND` keys pointing at `redis://localhost:6379/0`. Two tasks: `send_async_email`, which sends through Flask-Mail with Gmail's SMTP host and credentials from `MAIL_USERNAME` and `MAIL_PASSWORD`, and `long_task`, bound, which reports progress through `update_state` for a web page that polls `/status/<task_id>`. The index route calls `send_async_email.delay(...)` or `apply_async(args=[...], countdown=60)`. `requirements.txt` pins Celery 4.4.1, Flask 1.1.1 and redis 3.4.1; `run-redis.sh` builds Redis from source; there are no tests and no README note about the mail provider's sending limits.

A rate limit belongs on the task that talks to a third party, which is the mail task. Celery takes it as the `rate_limit` argument of the task decorator, in its `n/s`, `n/m` or `n/h` form, so the agent changes the decorator to `@celery.task(rate_limit=os.environ.get('MAIL_RATE_LIMIT'))`: unset means no limit, as before, and the value is whatever the provider allows, which the reader knows and the agent does not. It puts it in the decorator rather than in `task_annotations` on purpose: this app configures Celery with old-style uppercase keys, and Celery 4 refuses a configuration that mixes old and new names, so adding one new-style key would break startup. `long_task` gets no limit; it is bound by the page that polls it, not by a provider.

Then it explains what the limit does and does not do, in the README and the pull request, because this is where rate limits surprise people. A Celery rate limit is per worker process, not per cluster: two workers each send at the configured rate. A limited task that cannot run yet sits in the worker's prefetch buffer, so when emails pile up they can delay `long_task` on the same worker; the agent routes mail to its own queue with `CELERY_ROUTES` in the same uppercase style, and documents starting one worker with `-Q mail` and one for the rest, so a backlog of emails never holds a progress bar. A `countdown` still applies: the task is scheduled first and limited when it becomes due. And the limit can be changed on a running worker with `celery -A app.celery control rate_limit app.send_async_email <rate>` without a restart, which the README now says.

In its sandbox the agent installs `redis-server` from apt instead of running `run-redis.sh`, creates a virtualenv from `requirements.txt`, sets `MAIL_RATE_LIMIT`, starts a worker with `celery -A app.celery worker --loglevel=info -Q mail`, and enqueues a burst of `send_async_email` calls from a small script that sets `app.config['MAIL_SUPPRESS_SEND'] = True`, which makes Flask-Mail skip the SMTP connection, since no mail credentials exist in the sandbox. The worker log then shows each task received at once and succeeded at the spacing the limit dictates, and that log is the proof. It adds a `tests/test_rate_limit.py` that imports `app` with the variable set and asserts `send_async_email.rate_limit` equals it, so the wiring cannot silently disappear.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add a rate limit to the task in this Celery app that sends email. Read app.py, requirements.txt, run-redis.sh, templates/index.html and the README first, and note that Celery is configured through uppercase CELERY_* keys passed with conf.update. Set the limit with the task decorator's rate_limit argument, read from a MAIL_RATE_LIMIT environment variable in Celery's n/m form, unset meaning unlimited; do not add new-style configuration keys beside the old-style ones. Route the mail task to its own queue in the same configuration style and document running a worker per queue. Leave the progress task unlimited. In the sandbox, install Redis, start a worker on the mail queue with the variable set, enqueue a burst of mail tasks with Flask-Mail's MAIL_SUPPRESS_SEND on so nothing is sent, and keep the worker log. Add a test that asserts the task's rate_limit equals the variable. Explain in the README that the limit is per worker process and how to change it at run time with celery control. Open a pull request whose description quotes the worker log lines that show the spacing.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that changes `app.py` and the README and adds `tests/test_rate_limit.py`. The description quotes the worker log from the sandbox run, with the `received` lines arriving together and the `succeeded` lines spaced as the limit dictates, names the queue the mail task now uses and the worker commands for each queue, and states that the limit is per worker process.

The run's record keeps the complete worker session and the script that enqueued the burst, so a reviewer can read every timestamp rather than trust a description of the spacing.

Before merging, a reviewer sets `MAIL_RATE_LIMIT` to what the mail provider actually permits, decides how many mail workers will run since each has its own limit, and confirms the deployment starts the second worker for the new queue, because a queue nobody consumes is a silent backlog.

## Run it on every Linear issue labelled celery

In the console, add an automation to the Project with a `linear` trigger on `linear.issue.labeled` and the label `celery`, with this prompt. When someone files an issue about a provider refusing bursts of mail and labels it, the run starts with the issue as its context, makes the change in the Project's repository and opens a pull request on its own branch linked to the issue. A label added by an app integration never fires the trigger, and the automation starts at most `max_runs_per_hour` runs in an hour.

To start it from a support tool instead, send this body to `POST /o/{org}/sessions` with the issue text appended to the prompt and a `request_id` that is a UUID v5 of the issue id, so a retried delivery returns the run already started.

## Why run it on Rush?

A rate limit is a line of code and a paragraph of understanding, and the paragraph is what goes wrong: the limit is set in a configuration style the app does not use and the worker refuses to start, or it is set per worker and three workers send three times the rate, or limited emails sit in front of other tasks on the same queue. In a Rush sandbox the agent runs a real broker and a real worker, enqueues a burst and reads the timestamps, so the behaviour in the pull request is observed rather than described.

The worker log stays with the run, so the reviewer sees the spacing between tasks and not a claim about it. A team with several Celery services runs the same job on each in parallel, each in its own sandbox with its own Redis and its own pull request.

## Frequently asked questions

### Why read the limit from an environment variable?

Because the right number belongs to the mail provider's terms and the account's tier, which differ between the sandbox, staging and production. The decorator reads `MAIL_RATE_LIMIT` at import time, unset means unlimited as before, and `celery control rate_limit` changes it on a live worker when the provider changes the terms.

### Is the limit shared across workers?

No. Celery enforces a task's rate limit in each worker process separately, so the total sending rate is the limit multiplied by the number of worker processes consuming the mail queue. The README the agent writes says this and the reviewer sizes the value with it in mind.

### Why a separate queue for mail?

A limited task that cannot run yet waits inside the worker that already reserved it, ahead of whatever that worker reserved after it. With mail on its own queue and its own worker, a backlog of emails cannot delay the progress task the web page is polling.

### Does it work with the countdown the app already uses?

Yes. `apply_async(countdown=60)` schedules the task for a minute later; when it becomes due the worker applies the rate limit like any other execution. The two compose, and the test the agent adds only checks the limit is wired, since timing with a countdown is what the sandbox log shows.

### How is this checked without sending mail?

Flask-Mail skips the SMTP connection when `MAIL_SUPPRESS_SEND` is set, so the task runs to completion with no credentials and nothing leaves the sandbox. The worker log still records when each task was received and when it succeeded, which is all the limit affects.

## More Celery jobs

- [Add Retries with Backoff to Celery Tasks](https://getrush.ai/run/add-task-retries/celery/)
- [Add Task Routing to a Celery App](https://getrush.ai/run/add-task-routing/celery/)
- [Add Task Time Limits to a Flask Celery App](https://getrush.ai/run/add-task-timeouts/celery/)

## More jobs like this: Turn a Linear label into a PR

- [Add Retries with Backoff to RQ Jobs](https://getrush.ai/run/add-task-retries/rq/)
- [Add Team Invitations to a Filament Panel](https://getrush.ai/run/add-team-invitations/filament/)
- [Shard a Jest Suite Across Parallel CI Jobs for an Express API](https://getrush.ai/run/add-test-sharding/jest/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
