# Add Prometheus Metrics to a Celery App

Yes. An agent turns on Celery's task events, runs celery-exporter beside the worker, checks the counters and runtime histograms it scrapes, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-prometheus-metrics/celery/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent starts by reading how tasks are declared and run, because Celery's own metrics come from events the worker only sends when asked. In an app laid out like miguelgrinberg/flask-celery-example, `app.py` builds the Celery instance from the Flask config with `celery.conf.update(app.config)`, using the old-style `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND` keys pointed at `redis://localhost:6379/0`, and declares two tasks: `send_async_email`, which sends through Flask-Mail inside `app.app_context()`, and `long_task`, bound so it can call `self.update_state` with a `PROGRESS` state and a `current` and `total` in its meta. `requirements.txt` pins `celery==4.4.1`, `redis==3.4.1` and `Flask==1.1.1`; the README starts the worker with `celery -A app.celery worker --loglevel=info`, and `/status/<task_id>` polls `AsyncResult` for the web page.

The metrics come from `celery-exporter`, which listens to the worker's events and serves them on a port Prometheus scrapes, so no task code changes. The agent sets `worker_send_task_events` and `task_send_sent_event` in the Celery config next to the broker URL, so the worker reports received, started, succeeded, failed and retried tasks and the sender reports each dispatch, and adds a `Procfile`-style entry or a line in the README that starts `celery-exporter` with the same broker URL. It pins the exporter in `requirements.txt` beside the existing pins, keeping the celery 4 line since the exporter reads events, not the Celery API version.

One detail comes from reading rather than from the exporter's README. `long_task` sleeps one second per step for a random ten to fifty steps, so its runtime histogram needs buckets up to about a minute rather than the exporter's default spread; the agent sets the exporter's bucket flag to match. And because `send_async_email` is also scheduled with `countdown=60` from the form's second button, the gap between a task's `sent` and `received` events is a real metric here, not noise, and the pull request names it.

In its sandbox the agent installs the pinned requirements on a Python that celery 4.4.1 still runs on, starts a Redis server with `apt-get install -y redis-server`, since the repository's `run-redis.sh` builds Redis from source, starts the worker with `-E`, starts the exporter, then dispatches `long_task.apply_async()` from a Python shell and reads `/metrics` on the exporter's port before and after. It quotes the lines that moved: `celery_task_sent_total`, `celery_task_received_total`, `celery_task_succeeded_total` with the task name label, and the `celery_task_runtime` histogram buckets. The mail task is not sent, since there are no mail credentials in the sandbox; the agent dispatches it anyway and reads the `failed` counter move, which proves the failure path reports too.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add Prometheus metrics to this Flask and Celery app with celery-exporter. Read app.py, requirements.txt, run-redis.sh and README.md first. Enable worker_send_task_events and task_send_sent_event in the Celery config in app.py, pin celery-exporter in requirements.txt, and document starting it beside the worker with the same broker URL in the README, with runtime histogram buckets that cover long_task's ten to fifty one-second steps. Do not change the tasks or the Flask routes. In the sandbox install the requirements, install and start Redis with apt-get, start the worker with -E and the exporter, dispatch long_task and send_async_email, and quote the /metrics lines that changed in the pull request, including the failed counter for the mail task with no credentials. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request with a few lines in `app.py` (the two event settings), one pin in `requirements.txt`, and a README section on running the exporter and pointing Prometheus at it. No task body changes and no route changes. The description quotes the exporter's `/metrics` output before and after the two dispatches: the `sent`, `received`, `started` and `succeeded` counters for `app.long_task`, the `failed` counter for `app.send_async_email`, and the runtime histogram's bucket lines.

The run's record keeps the worker log with the task events, the exporter's startup line naming its port, and the Redis install, so a reviewer can see the worker was really started with events on rather than reading a config diff and hoping.

Before merging, a reviewer checks the bucket boundaries against the real `long_task` durations in production, decides whether the queue length gauge the exporter can also serve is worth its broker polling, and adds the exporter's port to the Prometheus scrape config, which lives outside this repository.

## Run it on every Linear issue labelled observability

A team with several Celery services makes this a rule. In the Project for each repository, an automation with the prompt above and a `linear` trigger on `linear.issue.labeled` for the label `observability` starts a run when someone files "we cannot see how long the report task takes". The trigger adds the issue after the prompt, so the run reads which task and which number the issue cares about and sizes the histogram for it.

Each run opens its own pull request on its own branch and links the issue. A service whose worker already runs with events on needs only the exporter, and the agent says so in the description instead of adding a second setting.

## Why run it on Rush?

The usual failure is a dashboard with every counter at zero: the exporter runs, the worker runs, and nobody turned on task events, or the sender events are off so `sent` never moves. The only way to know is to start all three, dispatch a task and read `/metrics`, which on a laptop means a Redis, a worker and an exporter in three terminals. In a Rush sandbox the agent does that in one run, with the repository's own tasks, and quotes the lines that moved.

The worker log and the metrics output stay with the run, so the reviewer sees the `task-succeeded` event and the counter that followed it rather than a config diff. Across several services the job runs in parallel, one sandbox each, each with its own proof.

## Frequently asked questions

### Why celery-exporter rather than instrumenting the tasks?

Because the worker already emits the events once asked, and the exporter turns them into counters and histograms with the task name as a label without a line in any task. A metric the app itself defines, such as emails sent per recipient domain, is a `prometheus_client` counter inside the task and a separate change.

### Will the mail task's failure in the sandbox break anything?

No. It fails because `MAIL_USERNAME` and `MAIL_PASSWORD` are unset, which is exactly the `failed` event the exporter should count. The pull request quotes that counter so the failure path is proven, and nothing is sent anywhere.

### Does this work with Celery 4.4?

Yes. The exporter reads the event stream over the broker, which Celery 4 and 5 emit the same way. The pin stays at `celery==4.4.1` so the pull request changes one thing; upgrading Celery is its own job.

### What does the PROGRESS state in long_task become?

Nothing on its own: `update_state` writes to the result backend, not to the event stream, so it is not a metric. The runtime histogram covers the whole task; if progress steps need a gauge, that is an app-defined metric inside the task.

### Where do the queue length numbers come from?

The exporter can poll the broker for queue lengths as a gauge, which is a Redis `LLEN` per queue on an interval. It is off by default here; the pull request names the flag so a reviewer can turn it on with the interval they want.

## More Celery jobs

- [Cancel a Running Celery Task](https://getrush.ai/run/add-task-cancellation/celery/)
- [Add Rate Limits to Celery Tasks](https://getrush.ai/run/add-task-rate-limits/celery/)
- [Add Retries with Backoff to Celery Tasks](https://getrush.ai/run/add-task-retries/celery/)

## The same job on other stacks

- [Add Prometheus Metrics to a chi API](https://getrush.ai/run/add-prometheus-metrics/chi/)
- [Add Prometheus Metrics to a Django REST Framework API](https://getrush.ai/run/add-prometheus-metrics/django-rest-framework/)
- [Add Prometheus Metrics to an Echo API](https://getrush.ai/run/add-prometheus-metrics/echo/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
