All jobs

Add Prometheus ServiceMonitors to a Kustomize App

Yes. An agent adds a ServiceMonitor per scrapeable service to a Kustomize base, names the Service ports, validates the build against the CRD and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the whole base, because a ServiceMonitor is only right if it matches a Service that exists and a port that is named. In an app like the sock-shop directory of argoproj/argocd-example-apps, kustomization.yaml lists twenty-nine resources under base/: fourteen Deployments and their Services for front-end, carts, catalogue, orders, payment, queue-master, shipping, user, their MongoDB, MySQL and Redis stores, and rabbitmq. Every Service selects on a name label, and all the application Services serve port 80, except front-end, which maps 80 to targetPort: 8079. Only rabbitmq-svc.yaml names its ports, rabbitmq on 5672 and exporter on 9090, and rabbitmq-dep.yaml runs a second container, kbudde/rabbitmq-exporter, on that port. The only scrape hints are annotations: queue-master-svc.yaml carries prometheus.io/path: "/prometheus", rabbitmq-dep.yaml's pod template says prometheus.io/scrape: "false", and session-db-dep.yaml has a misspelt prometheus.io.scrape. Nothing is monitoring.coreos.com/v1, and the images are pinned weaveworksdemos/* tags from 0.3.0 to 0.4.8.

It writes base/monitoring/, one ServiceMonitor per service that exposes metrics, each selecting the Service by its name label and scraping one named port. Since a ServiceMonitor endpoint refers to a Service port by name, the agent adds name: http to each unnamed application Service port, a one-line change in eight Service files that changes no routing. The paths come from what the manifests say and what each service's own documentation says: /prometheus for queue-master as its annotation states and for the other Spring services, /metrics for the Go and Node services, with each path recorded in the pull request alongside where it came from and which ones the agent could not confirm from the repository alone. The rabbitmq monitor scrapes the exporter port, and the agent removes the scrape: "false" annotation on the pod template because the exporter sidecar is the thing to scrape, and fixes the typo on session-db. The databases get no monitor, because none of them runs an exporter, and the description lists the three exporters that would add them.

The monitors carry no release label, because which label the Prometheus Operator selects on is a property of the cluster, not the app. The agent instead documents the two ways to make them visible: a commonLabels entry in an overlay, or serviceMonitorSelectorNilUsesHelmValues: false on a kube-prometheus-stack install, and shows the overlay form in the pull request so a team with a release: monitoring convention adds one line. kustomization.yaml gains the new directory under resources.

Then it checks the build. kustomize build sock-shop renders every resource, and kubeconform -strict validates them with the default Kubernetes schemas plus the ServiceMonitor schema from the CRDs catalog, so a wrong apiVersion or an endpoints entry with both port and targetPort fails in the sandbox and not in Argo CD. It also greps the rendered output to confirm every monitor's selector matches exactly one Service and every named port exists on it, and shows that table in the pull request.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add Prometheus Operator ServiceMonitors to this Kustomize application. Read kustomization.yaml and every file it lists first, and note which Services name their ports, which pods and Services carry prometheus.io annotations, and which Deployments run an exporter sidecar. Write one ServiceMonitor per service that exposes metrics, in a monitoring directory added to the kustomization, selecting each Service by the label it already has and scraping a named port; add a port name to any Service port the monitor needs, and use the metrics path the manifests or the service's own documentation name, recording each path's source in the pull request and saying which you could not confirm from the repository. Scrape exporter sidecars on their port and remove scrape annotations that would hide them; fix any misspelt annotation you find. Add no monitor for a datastore without an exporter, but list the exporters that would. Do not add a release label; document how an overlay or the Prometheus install selects the monitors. Run kustomize build and validate the output with kubeconform in strict mode using the ServiceMonitor CRD schema, check that every monitor's selector matches exactly one Service with that port name, and open a pull request with that table.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds nine files under base/monitoring/, one per application service and one for the RabbitMQ exporter, edits eight Service files to name their port, edits rabbitmq-dep.yaml and session-db-dep.yaml for the annotations, and adds the directory to kustomization.yaml. The description has a table with one row per monitor: the Service it selects, the port name, the path and where that path came from.

The run's record keeps the kustomize build output and the kubeconform summary with the CRD schema, so a reviewer sees that the rendered manifests validate as ServiceMonitors and that nothing else in the base changed shape. The pull request says which paths were confirmed from the repository and which came from the services' documentation.

Before merging, a reviewer syncs the application to a cluster with the Prometheus Operator installed, applies the overlay label if their install needs one, and opens the Prometheus targets page: each monitor appears with its endpoint, and a target whose path was guessed wrong shows up as down with a 404, which the table makes easy to fix in one line.

Run it on every Linear issue labelled observability

A team that plans platform work in Linear adds an automation with a linear trigger on linear.issue.labeled for the label observability, the Project that has this repository, and this prompt. Labelling the issue that asks for service metrics starts one run with the issue's text after the prompt, and the pull request links back to the issue.

The trigger adds the issue after the same prompt each time, so the next observability issue, such as a PrometheusRule with alerts for the queue depth the RabbitMQ exporter now reports, runs through the same automation with the agent reading the base fresh.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

A ServiceMonitor that selects nothing fails silently: no target appears, no error is raised, and someone notices weeks later. The agent reads every Service and port in its own sandbox, names what needs naming, renders the base and validates it against the CRD schema, and proves each selector matches exactly one Service before a reviewer opens Argo CD.

The commands and their output stay in the run's record, and the pull request separates what the repository proved from what the services' documentation says, so a reviewer knows which rows of the table to watch on the first sync.

Frequently asked questions

Why did the Service ports need names?

A ServiceMonitor's endpoints[].port is a port name, not a number. Only the RabbitMQ Service named its ports, so the agent added name: http to the others. Nothing about routing changes; the Service still maps the same port to the same target.

How do I make my Prometheus pick them up?

Either add the label your Prometheus selects on through commonLabels in an overlay, or configure the operator to select all monitors in the namespace. The pull request shows both and takes no side, because the label is the cluster's convention.

Are the metrics paths verified?

The queue-master path comes from its own Service annotation and the RabbitMQ exporter's port from its Deployment. The remaining paths come from the services' documentation, and the pull request lists them as such. The first scrape on a real cluster confirms each one; a wrong path shows as a 404 target.

Why no monitors for the databases?

MongoDB, MySQL and Redis expose no Prometheus endpoint by themselves. Each needs an exporter sidecar first, and the description names the three so a follow-up pull request can add them with their monitors.

Does this change the Argo CD sync?

The ServiceMonitor CRD must exist on the cluster, or the sync fails on an unknown kind. Clusters with the Prometheus Operator have it; the pull request says so, and kubeconform in the sandbox used the same schema.

More Kustomize jobs

The same job on other stacks

More jobs like this: Turn a Linear label into a PR

Keep reading