What the agent does
The agent starts by reading how tasks are declared and run, because Celery's own metrics come from events the worker only sends when asked. In an app laid out like miguelgrinberg/flask-celery-example, app.py builds the Celery instance from the Flask config with celery.conf.update(app.config), using the old-style CELERY_BROKER_URL and CELERY_RESULT_BACKEND keys pointed at redis://localhost:6379/0, and declares two tasks: send_async_email, which sends through Flask-Mail inside app.app_context(), and long_task, bound so it can call self.update_state with a PROGRESS state and a current and total in its meta. requirements.txt pins celery==4.4.1, redis==3.4.1 and Flask==1.1.1; the README starts the worker with celery -A app.celery worker --loglevel=info, and /status/<task_id> polls AsyncResult for the web page.
The metrics come from celery-exporter, which listens to the worker's events and serves them on a port Prometheus scrapes, so no task code changes. The agent sets worker_send_task_events and task_send_sent_event in the Celery config next to the broker URL, so the worker reports received, started, succeeded, failed and retried tasks and the sender reports each dispatch, and adds a Procfile-style entry or a line in the README that starts celery-exporter with the same broker URL. It pins the exporter in requirements.txt beside the existing pins, keeping the celery 4 line since the exporter reads events, not the Celery API version.
One detail comes from reading rather than from the exporter's README. long_task sleeps one second per step for a random ten to fifty steps, so its runtime histogram needs buckets up to about a minute rather than the exporter's default spread; the agent sets the exporter's bucket flag to match. And because send_async_email is also scheduled with countdown=60 from the form's second button, the gap between a task's sent and received events is a real metric here, not noise, and the pull request names it.
In its sandbox the agent installs the pinned requirements on a Python that celery 4.4.1 still runs on, starts a Redis server with apt-get install -y redis-server, since the repository's run-redis.sh builds Redis from source, starts the worker with -E, starts the exporter, then dispatches long_task.apply_async() from a Python shell and reads /metrics on the exporter's port before and after. It quotes the lines that moved: celery_task_sent_total, celery_task_received_total, celery_task_succeeded_total with the task name label, and the celery_task_runtime histogram buckets. The mail task is not sent, since there are no mail credentials in the sandbox; the agent dispatches it anyway and reads the failed counter move, which proves the failure path reports too.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add Prometheus metrics to this Flask and Celery app with celery-exporter. Read app.py, requirements.txt, run-redis.sh and README.md first. Enable worker_send_task_events and task_send_sent_event in the Celery config in app.py, pin celery-exporter in requirements.txt, and document starting it beside the worker with the same broker URL in the README, with runtime histogram buckets that cover long_task's ten to fifty one-second steps. Do not change the tasks or the Flask routes. In the sandbox install the requirements, install and start Redis with apt-get, start the worker with -E and the exporter, dispatch long_task and send_async_email, and quote the /metrics lines that changed in the pull request, including the failed counter for the mail task with no credentials. Open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request with a few lines in app.py (the two event settings), one pin in requirements.txt, and a README section on running the exporter and pointing Prometheus at it. No task body changes and no route changes. The description quotes the exporter's /metrics output before and after the two dispatches: the sent, received, started and succeeded counters for app.long_task, the failed counter for app.send_async_email, and the runtime histogram's bucket lines.
The run's record keeps the worker log with the task events, the exporter's startup line naming its port, and the Redis install, so a reviewer can see the worker was really started with events on rather than reading a config diff and hoping.
Before merging, a reviewer checks the bucket boundaries against the real long_task durations in production, decides whether the queue length gauge the exporter can also serve is worth its broker polling, and adds the exporter's port to the Prometheus scrape config, which lives outside this repository.
Run it on every Linear issue labelled observability
A team with several Celery services makes this a rule. In the Project for each repository, an automation with the prompt above and a linear trigger on linear.issue.labeled for the label observability starts a run when someone files "we cannot see how long the report task takes". The trigger adds the issue after the prompt, so the run reads which task and which number the issue cares about and sizes the histogram for it.
Each run opens its own pull request on its own branch and links the issue. A service whose worker already runs with events on needs only the exporter, and the agent says so in the description instead of adding a second setting.
The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.
Why run it on Rush?
The usual failure is a dashboard with every counter at zero: the exporter runs, the worker runs, and nobody turned on task events, or the sender events are off so sent never moves. The only way to know is to start all three, dispatch a task and read /metrics, which on a laptop means a Redis, a worker and an exporter in three terminals. In a Rush sandbox the agent does that in one run, with the repository's own tasks, and quotes the lines that moved.
The worker log and the metrics output stay with the run, so the reviewer sees the task-succeeded event and the counter that followed it rather than a config diff. Across several services the job runs in parallel, one sandbox each, each with its own proof.
Frequently asked questions
Why celery-exporter rather than instrumenting the tasks?
Because the worker already emits the events once asked, and the exporter turns them into counters and histograms with the task name as a label without a line in any task. A metric the app itself defines, such as emails sent per recipient domain, is a prometheus_client counter inside the task and a separate change.
Will the mail task's failure in the sandbox break anything?
No. It fails because MAIL_USERNAME and MAIL_PASSWORD are unset, which is exactly the failed event the exporter should count. The pull request quotes that counter so the failure path is proven, and nothing is sent anywhere.
Does this work with Celery 4.4?
Yes. The exporter reads the event stream over the broker, which Celery 4 and 5 emit the same way. The pin stays at celery==4.4.1 so the pull request changes one thing; upgrading Celery is its own job.
What does the PROGRESS state in long_task become?
Nothing on its own: update_state writes to the result backend, not to the event stream, so it is not a metric. The runtime histogram covers the whole task; if progress steps need a gauge, that is an app-defined metric inside the task.
Where do the queue length numbers come from?
The exporter can poll the broker for queue lengths as a gauge, which is a Redis LLEN per queue on an interval. It is off by default here; the pull request names the flag so a reviewer can turn it on with the interval they want.
More Celery jobs
The same job on other stacks
- Add Prometheus Metrics to a chi API (chi)
- Add Prometheus Metrics to a Django REST Framework API (Django REST Framework)
- Add Prometheus Metrics to an Echo API (Echo)