All jobs

Upgrade Celery 4 to Celery 5

Yes. An agent moves a Celery 4 app to Celery 5 with its pinned companions, starts a real broker and worker to prove the tasks still run, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent starts from the pins, because Celery does not upgrade alone. In an app like miguelgrinberg/flask-celery-example, requirements.txt freezes fifteen packages, among them celery==4.4.1 with its own libraries kombu==4.6.8, amqp==2.5.2, vine==1.3.0 and billiard==3.6.3.0, the client redis==3.4.1, and Flask==1.1.1 with Click==7.0. It reads what the current Celery release asks for on PyPI: version 5.6.3 needs Python 3.9 or later, kombu 5.6 or later, billiard 4.2.1 or later, vine 5.1 or later and click from 8.1.2. That last line collides with the Click pin Flask 1.1.1 was frozen beside, so the plan has to say which other packages move and why.

Before changing anything it records a baseline. It creates a virtual environment on the sandbox's Python, installs the file exactly as pinned and notes whether that still installs and imports. It installs a Redis server from the system packages, since run-redis.sh downloads and compiles one, starts it, and runs the worker with the command in the README, celery -A app.celery worker --loglevel=info. That command is already written in Celery 5's order, with -A before worker, while the pin is still 4.4.1.

Then it upgrades in the smallest steps that resolve. The first commit moves Celery, its four libraries, the Redis client and Click, and removes pins Celery no longer uses or adds the ones it now needs, such as click-didyoumean, click-repl and click-plugins. If Flask 1.1.1 no longer imports beside the new Click, a second commit moves Flask with Werkzeug, Jinja2, MarkupSafe, itsdangerous, blinker and Flask-Mail to a set that resolves together. In app.py it changes how the app is configured: Celery(app.name, broker=app.config['CELERY_BROKER_URL']) followed by celery.conf.update(app.config) passes Flask's whole config, where CELERY_RESULT_BACKEND is an old-style setting name. The agent passes the broker and the result backend explicitly under the lowercase names broker_url and result_backend, and reports the change.

The proof is the app's own two tasks, run against the real broker. With Redis and a worker up, it posts to /longtask, follows the Location header to /status/<task_id> and polls until the state leaves PROGRESS. That exercises self.update_state with a custom state, the result backend, and task.info in the status route, and it should end with result 42. For send_async_email it sets MAIL_SUPPRESS_SEND so Flask-Mail builds the message without an SMTP server, submits the form both ways, immediate and with countdown=60, and waits for the worker log to show each task received and succeeded. It reads the worker's startup banner and log for deprecation warnings and lists them.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Upgrade this app from Celery 4 to the current Celery 5 release. Read requirements.txt, app.py, the README's setup steps and any script that starts the broker first. Look up on PyPI what the current celery release requires (Python floor, kombu, billiard, vine, click and the click plugins) and what the current releases of the other pinned packages require, and write down which pins must move together. Record a baseline: create a virtual environment, install the pinned file as it is, and note whether it installs and imports on this Python. Install and start a Redis server from system packages. Upgrade in separate commits: first celery with kombu, amqp, vine, billiard, redis and click, adding or removing pins as the new release needs; then, only if the web framework no longer imports or serves beside them, the framework and its own dependencies. Keep the file fully pinned. In the code, stop passing the whole Flask config to Celery: set broker_url and result_backend by their lowercase names, and change nothing else about the tasks. Prove it: start a worker with the README's command, POST /longtask and poll its status URL until it finishes, and send the email task both immediately and with a countdown with MAIL_SUPPRESS_SEND set, confirming from the worker log that each task was received and succeeded. List every deprecation warning the worker prints. Update the README where a command or version changed. Open a pull request with the old and new pins as a table, the commands and their output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request changes requirements.txt, a few lines of app.py and, where a step changed, the README. Its description has a table of every pin with the old and new version and the reason it moved: required by Celery, required by Flask, or left alone. Under it are the baseline result, the install output for the new set, the worker's startup banner showing the broker and result backend it connected to, the sequence of status responses for the long task from PENDING through PROGRESS to the final state, and the worker log lines for both email submissions.

It also says what was not proven. No email left the sandbox, because sending was suppressed, so real SMTP delivery with your credentials is still yours to check. Tasks that were queued by Celery 4 and consumed by Celery 5 were not tested unless you ask for it. A reviewer reads the configuration change in app.py first, since a result backend that is silently not configured only shows up when something reads a task's state. Then they check that the Python floor of the new pins matches what production runs, and decide whether to drain the queue before deploying.

Run it on every Monday morning

After the major upgrade the same job is small, and a schedule keeps it small. Create an automation in the Project that holds the repository with a schedule trigger whose cron is 0 9 * * 1 in your timezone, and a prompt that says: check PyPI for a newer Celery, kombu or Redis client than the pins, and if there is one, upgrade, run the broker and worker checks from this page and open a pull request with the output; if there is none, end with one line saying so. The automation's max runs per hour applies to it, not an API key's limit.

For a company with many worker services, a script on your own server sends this body once per repository to POST /o/{org}/sessions, with that repository in resources and a request_id that is a UUID v5 of the repository's full name and the target Celery version. A retried call then returns the run it already started, and each run reads its own pins.

The use case this job follows, with its automation and trigger: Upgrade dependencies weekly.

Why run it on Rush?

A Celery upgrade cannot be checked by importing the module. The failures are at the edges: a worker that starts and never receives a task, a result backend that was configured under a name the new version reads differently, a countdown that is accepted and never fires. Seeing those needs a broker, a worker and a web process running together, which is three terminals on a laptop and usually gets skipped. The agent's sandbox runs all three, and the run's record keeps the worker log and every status response, so the pull request shows tasks completing and not just a resolver that was satisfied.

The dependency reasoning is also the kind people shortcut. Fifteen frozen pins have to move as a few consistent sets, and the agent reads each package's real requirements instead of deleting the pins and taking whatever installs. When ten services need the same move, ten runs start at once and each works out its own set.

Frequently asked questions

Do my task definitions have to change?

Usually not. Decorators such as @celery.task and @celery.task(bind=True), delay, apply_async with countdown and update_state are the same calls in Celery 5, and the agent leaves them alone. What changes is configuration and the command line, and the pull request lists each line it touched.

Why does it also upgrade Click and Flask?

Celery 5 builds its command line on Click and current releases need Click 8.1.2 or later, which a frozen Click==7.0 forbids. Flask is only moved if it stops working beside the new Click, and that is a separate commit so you can see the two decisions apart.

Will tasks already in the queue survive the upgrade?

The run does not test that by default. If you want it checked, say so in the prompt: the agent queues tasks with the old environment, switches to the new one and reports whether the new worker consumes them. Draining the queue before the deploy is the cautious path either way.

What if my broker is RabbitMQ and not Redis?

The agent reads the broker URL in your configuration and installs that broker in the sandbox instead. The checks are the same: a worker connects, receives a task, stores its result and reports it.

Does it need my mail credentials?

No. It sets MAIL_SUPPRESS_SEND so the email task runs to completion without contacting an SMTP server, and it says in the pull request that delivery was not tested.

More Celery jobs

More jobs like this: Upgrade dependencies weekly

Keep reading