# Add Prometheus ServiceMonitors to a Kustomize App

Yes. An agent adds a ServiceMonitor per scrapeable service to a Kustomize base, names the Service ports, validates the build against the CRD and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-service-monitors/kustomize/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads the whole base, because a ServiceMonitor is only right if it matches a Service that exists and a port that is named. In an app like the `sock-shop` directory of argoproj/argocd-example-apps, `kustomization.yaml` lists twenty-nine resources under `base/`: fourteen Deployments and their Services for `front-end`, `carts`, `catalogue`, `orders`, `payment`, `queue-master`, `shipping`, `user`, their MongoDB, MySQL and Redis stores, and `rabbitmq`. Every Service selects on a `name` label, and all the application Services serve port 80, except `front-end`, which maps 80 to `targetPort: 8079`. Only `rabbitmq-svc.yaml` names its ports, `rabbitmq` on 5672 and `exporter` on 9090, and `rabbitmq-dep.yaml` runs a second container, `kbudde/rabbitmq-exporter`, on that port. The only scrape hints are annotations: `queue-master-svc.yaml` carries `prometheus.io/path: "/prometheus"`, `rabbitmq-dep.yaml`'s pod template says `prometheus.io/scrape: "false"`, and `session-db-dep.yaml` has a misspelt `prometheus.io.scrape`. Nothing is `monitoring.coreos.com/v1`, and the images are pinned `weaveworksdemos/*` tags from `0.3.0` to `0.4.8`.

It writes `base/monitoring/`, one `ServiceMonitor` per service that exposes metrics, each selecting the Service by its `name` label and scraping one named port. Since a ServiceMonitor endpoint refers to a Service port by name, the agent adds `name: http` to each unnamed application Service port, a one-line change in eight Service files that changes no routing. The paths come from what the manifests say and what each service's own documentation says: `/prometheus` for `queue-master` as its annotation states and for the other Spring services, `/metrics` for the Go and Node services, with each path recorded in the pull request alongside where it came from and which ones the agent could not confirm from the repository alone. The `rabbitmq` monitor scrapes the `exporter` port, and the agent removes the `scrape: "false"` annotation on the pod template because the exporter sidecar is the thing to scrape, and fixes the typo on `session-db`. The databases get no monitor, because none of them runs an exporter, and the description lists the three exporters that would add them.

The monitors carry no `release` label, because which label the Prometheus Operator selects on is a property of the cluster, not the app. The agent instead documents the two ways to make them visible: a `commonLabels` entry in an overlay, or `serviceMonitorSelectorNilUsesHelmValues: false` on a kube-prometheus-stack install, and shows the overlay form in the pull request so a team with a `release: monitoring` convention adds one line. `kustomization.yaml` gains the new directory under `resources`.

Then it checks the build. `kustomize build sock-shop` renders every resource, and `kubeconform -strict` validates them with the default Kubernetes schemas plus the ServiceMonitor schema from the CRDs catalog, so a wrong `apiVersion` or an `endpoints` entry with both `port` and `targetPort` fails in the sandbox and not in Argo CD. It also greps the rendered output to confirm every monitor's selector matches exactly one Service and every named port exists on it, and shows that table in the pull request.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add Prometheus Operator ServiceMonitors to this Kustomize application. Read kustomization.yaml and every file it lists first, and note which Services name their ports, which pods and Services carry prometheus.io annotations, and which Deployments run an exporter sidecar. Write one ServiceMonitor per service that exposes metrics, in a monitoring directory added to the kustomization, selecting each Service by the label it already has and scraping a named port; add a port name to any Service port the monitor needs, and use the metrics path the manifests or the service's own documentation name, recording each path's source in the pull request and saying which you could not confirm from the repository. Scrape exporter sidecars on their port and remove scrape annotations that would hide them; fix any misspelt annotation you find. Add no monitor for a datastore without an exporter, but list the exporters that would. Do not add a release label; document how an overlay or the Prometheus install selects the monitors. Run kustomize build and validate the output with kubeconform in strict mode using the ServiceMonitor CRD schema, check that every monitor's selector matches exactly one Service with that port name, and open a pull request with that table.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that adds nine files under `base/monitoring/`, one per application service and one for the RabbitMQ exporter, edits eight Service files to name their port, edits `rabbitmq-dep.yaml` and `session-db-dep.yaml` for the annotations, and adds the directory to `kustomization.yaml`. The description has a table with one row per monitor: the Service it selects, the port name, the path and where that path came from.

The run's record keeps the `kustomize build` output and the `kubeconform` summary with the CRD schema, so a reviewer sees that the rendered manifests validate as ServiceMonitors and that nothing else in the base changed shape. The pull request says which paths were confirmed from the repository and which came from the services' documentation.

Before merging, a reviewer syncs the application to a cluster with the Prometheus Operator installed, applies the overlay label if their install needs one, and opens the Prometheus targets page: each monitor appears with its endpoint, and a target whose path was guessed wrong shows up as down with a 404, which the table makes easy to fix in one line.

## Run it on every Linear issue labelled observability

A team that plans platform work in Linear adds an automation with a `linear` trigger on `linear.issue.labeled` for the label `observability`, the Project that has this repository, and this prompt. Labelling the issue that asks for service metrics starts one run with the issue's text after the prompt, and the pull request links back to the issue.

The trigger adds the issue after the same prompt each time, so the next observability issue, such as a `PrometheusRule` with alerts for the queue depth the RabbitMQ exporter now reports, runs through the same automation with the agent reading the base fresh.

## Why run it on Rush?

A ServiceMonitor that selects nothing fails silently: no target appears, no error is raised, and someone notices weeks later. The agent reads every Service and port in its own sandbox, names what needs naming, renders the base and validates it against the CRD schema, and proves each selector matches exactly one Service before a reviewer opens Argo CD.

The commands and their output stay in the run's record, and the pull request separates what the repository proved from what the services' documentation says, so a reviewer knows which rows of the table to watch on the first sync.

## Frequently asked questions

### Why did the Service ports need names?

A ServiceMonitor's `endpoints[].port` is a port name, not a number. Only the RabbitMQ Service named its ports, so the agent added `name: http` to the others. Nothing about routing changes; the Service still maps the same port to the same target.

### How do I make my Prometheus pick them up?

Either add the label your Prometheus selects on through `commonLabels` in an overlay, or configure the operator to select all monitors in the namespace. The pull request shows both and takes no side, because the label is the cluster's convention.

### Are the metrics paths verified?

The `queue-master` path comes from its own Service annotation and the RabbitMQ exporter's port from its Deployment. The remaining paths come from the services' documentation, and the pull request lists them as such. The first scrape on a real cluster confirms each one; a wrong path shows as a 404 target.

### Why no monitors for the databases?

MongoDB, MySQL and Redis expose no Prometheus endpoint by themselves. Each needs an exporter sidecar first, and the description names the three so a follow-up pull request can add them with their monitors.

### Does this change the Argo CD sync?

The ServiceMonitor CRD must exist on the cluster, or the sync fails on an unknown kind. Clusters with the Prometheus Operator have it; the pull request says so, and `kubeconform` in the sandbox used the same schema.

## More Kustomize jobs

- [Add a Staging Overlay with Kustomize from a Linear Issue](https://getrush.ai/run/add-staging-overlay/kustomize/)
- [Fix "# Warning: 'bases' is deprecated. Please use 'resources' instead" Across a Kustomize Repository](https://getrush.ai/run/fix-kustomize-bases-is-deprecated/kustomize/)
- [Pin and Upgrade Container Images in Kustomize Manifests](https://getrush.ai/run/pin-image-versions/kustomize/)

## The same job on other stacks

- [Add a ServiceMonitor to a Helm Chart](https://getrush.ai/run/add-service-monitors/helm/)

## More jobs like this: Turn a Linear label into a PR

- [Add Shell Completion to a Go CLI from a Linear Issue](https://getrush.ai/run/add-shell-completion/go-cli/)
- [Add Shell Completion to a Textual App's CLI](https://getrush.ai/run/add-shell-completion/textual/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
