# Add Prometheus Metrics to Broadway SQS Pipelines With PromEx

Yes. An agent adds PromEx's Broadway plugin and a failure counter by reason to an app's SQS pipelines, tests both against real telemetry and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-prometheus-metrics/broadway/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads the pipelines first. In an Elixir app like hexpm/hexpm, `lib/hexpm/preview/queue.ex` and `lib/hexpm/hexdocs/queue.ex` are Broadway pipelines fed by `BroadwaySQS.Producer` in `config/prod.exs` and by `Broadway.DummyProducer` in `config/config.exs`, each with `visibility_timeout: 300` and processor concurrency from config. `Hexpm.Preview.Queue.handle_message/3` decodes an S3 event, turns each record into Oban jobs inside one `Repo.transaction`, and calls `Broadway.Message.failed/2` on anything it cannot handle. A failed SQS message is not deleted, so it comes back after the visibility timeout. Nothing measures any of this: `Hexpm.PromEx.plugins/0` lists Application, Beam, Phoenix, Ecto, the app's own counters, DB connections, outbound HTTP, email and Oban, and no Broadway plugin.

Then it reads how this codebase adds metrics, because it has firm conventions. Each upstream plugin is wrapped in a module under `lib/hexpm/prom_ex/plugins/` when it needs a change: `Hexpm.PromEx.Plugins.Oban` keeps the upstream metric names so the upstream dashboard still works and replaces only the histogram buckets, with a moduledoc explaining why. `oban_plugins/0` returns the Oban plugin only on nodes that run queues, and `initialize_counters/0` reports a zero for every counter series with a bounded label set so Prometheus sees the first real increase. The pipelines themselves start only when `worker_mode?` is true in `lib/hexpm/application.ex`, from `worker_after_oban_children/0`. The agent writes all of this down before adding a line.

The change has two parts. It adds `PromEx.Plugins.Broadway` to `plugins/0` under the same worker-mode condition the application uses to start the pipelines, and `{:prom_ex, "broadway.json"}` to `dashboards/0`, wrapping the plugin only if its histogram buckets do not fit a handler whose real work is a short transaction. Then it adds a counter of failed messages per pipeline and reason. The reasons are a closed set read from the code: the first element of each error tuple the handlers return (`unsupported_preview_message` or `unsupported_hexdocs_message`, `unsupported_s3_record`, `malformed_s3_object`), plus a JSON decode error and a raised exception, which is how a failed insert transaction surfaces. The agent maps each `{:error, reason}` to one of those labels, emits a telemetry event from the `else` and `rescue` branches, and adds the counter to `initialize_counters/0` with every label at zero.

Tests follow the existing ones. `test/hexpm/prom_ex/plugins/oban_test.exs` compares the wrapper's metric names with upstream's, and `test/hexpm/prom_ex_test.exs` checks when a plugin is returned, so the agent adds a test that the Broadway plugin appears only in worker mode and one that each failure reason in `test/hexpm/preview/queue_test.exs` increments its own label. It installs PostgreSQL and the OTP and Elixir versions `.github/workflows/main.yml` names, runs `mix test`, then `mix format --check-formatted` and `mix compile --warnings-as-errors`, which the same workflow enforces.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add Prometheus metrics for the Broadway pipelines. Read lib/hexpm/preview/queue.ex, lib/hexpm/hexdocs/queue.ex, config/config.exs, config/prod.exs, lib/hexpm/application.ex, lib/hexpm/prom_ex.ex, every module in lib/hexpm/prom_ex/plugins and their tests first, and follow their conventions. Add PromEx.Plugins.Broadway and its dashboard only on nodes that run the pipelines. Add a counter of failed messages per pipeline and reason, with reasons taken from the error tuples the handlers return, zero-initialized like the other counters. Add tests beside the existing PromEx and queue tests, run mix test, mix format --check-formatted and mix compile --warnings-as-errors, and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that changes `lib/hexpm/prom_ex.ex`, both queue modules and the module that owns the new counter, and adds tests under `test/hexpm/prom_ex/` and to the two queue tests. The description lists every new metric with its labels, shows that each label is a closed set taken from a function clause in the code, and names the dashboard it adds.

It includes the `mix test`, format and compile output from the sandbox, and the repository's CI workflow runs the same checks on the branch. It says what the metrics will show and what they will not: a message that fails comes back after its 300 second visibility timeout and counts again, so a poison message appears as a steady rate on one reason rather than a single spike, and whether it is ever moved to a dead-letter queue is set on the SQS queue, outside this repository.

A reviewer checks that no label can carry a key, a package name or an error message, which would make the series unbounded, and that the plugin stays off on web nodes where no pipeline runs.

## Run it on every Linear issue labelled observability

In the Project that holds the repository, an automation with a `linear` trigger on `linear.issue.labeled` and `observability` in `config.labels` starts a run when someone labels an issue such as "we cannot see SQS failures". The issue is added after the prompt, so a general prompt that says to read the component the issue names and add metrics in the codebase's existing style covers the next request too.

To run the same work from your own tooling, send this body to `POST /o/{org}/sessions` with the repository in `resources`. A key with a webhook URL receives `run.completed` with the `session_url` when the turn ends, and the session carries the pull request link.

## Why run it on Rush?

Metrics work in a codebase with strong conventions is mostly reading: which plugins are wrapped and why, how counters are initialized, which nodes report what. The agent reads all of it before writing, and the sandbox runs the real test suite against Postgres with the pinned OTP and Elixir, so the pull request shows the new tests passing beside the old ones. Every command and its output stay in the run's record.

Label sets are the part that hurts later, because one unbounded label can multiply a Prometheus series count. Taking the reasons from function clauses and proving each with a test keeps that set closed, and a reviewer can check the claim against the code in one pass.

## Frequently asked questions

### What does PromEx's Broadway plugin report?

It reports metrics from Broadway's own telemetry in three groups: pipeline init, message processing and batch processing. Adding it to `plugins/0` and `broadway.json` to `dashboards/0` is all the upstream setup asks for; the agent wraps it only if the buckets need changing, as the Oban plugin is wrapped here.

### Why add a separate failure counter?

The plugin measures processing, but the reason a message failed lives in this app's code: an unsupported record, a malformed object, a failed insert. A counter labelled by those reasons tells an operator which failure is growing without reading logs.

### Will it add a dead-letter queue?

No. A dead-letter queue is a setting on the SQS queue, which this repository does not manage. The pull request explains how failures repeat after the visibility timeout and leaves the queue setting to whoever owns the infrastructure.

### Does it need AWS credentials to test?

No. Outside production the pipelines use `Broadway.DummyProducer`, and the existing queue tests build a `%Broadway.Message{}` with `Broadway.NoopAcknowledger` and call the handler with sample S3 events, so the tests run with Postgres only.

## The same job on other stacks

- [Add Prometheus Metrics to a Celery App](https://getrush.ai/run/add-prometheus-metrics/celery/)
- [Add Prometheus Metrics to a chi API](https://getrush.ai/run/add-prometheus-metrics/chi/)
- [Add Prometheus Metrics to a Django REST Framework API](https://getrush.ai/run/add-prometheus-metrics/django-rest-framework/)
- [Add Prometheus Metrics to an Echo API](https://getrush.ai/run/add-prometheus-metrics/echo/)
- [Add Prometheus Metrics to a Fastify App](https://getrush.ai/run/add-prometheus-metrics/fastify/)
- [Add Prometheus Metrics to a Fiber App](https://getrush.ai/run/add-prometheus-metrics/fiber/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
