# Add Drift Detection to Flux Helm Releases

Yes. An agent turns on helm-controller drift detection on your HelmReleases, adds ignore rules for fields other controllers own, validates them and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-drift-detection/flux/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads every HelmRelease and how it reaches the cluster before changing one. In a repository laid out like fluxcd/flux2-kustomize-helm-example, `apps/base/podinfo/release.yaml` is a `helm.toolkit.fluxcd.io/v2` HelmRelease with `interval: 50m`, a `RetryOnFailure` strategy for install and upgrade, a chart from the `podinfo` HelmRepository and values that enable Redis and an `httpRoute` on the `envoy` Gateway; `apps/staging/podinfo-values.yaml` and `apps/production/podinfo-values.yaml` are Kustomize patches that set the chart version constraint, `>=1.0.0-alpha` and `>=1.0.0`, and the hostname per cluster. `infrastructure/controllers/` holds two more releases, `cert-manager` and `envoy-gateway`, each pulled as an OCI chart through `chartRef`. None of the three has a `driftDetection` field, so a change made with `kubectl edit` or by a colleague's Helm upgrade stays until the next chart version. Validation already exists: `scripts/validate.sh` runs the Flux Schema plugin against `.fluxschema.yml`, the `test` workflow runs it on every pull request and then `flux migrate -f . --yes` to catch deprecated API versions, and the `e2e` workflow boots a kind cluster, installs Flux with the `source-watcher` component and waits for `helmrelease/podinfo` to become ready.

The field is `spec.driftDetection`, documented in helm-controller's HelmRelease spec. With `mode: warn`, the controller compares the manifest in Helm storage against the cluster with a server-side dry-run after each successful reconcile and emits an event when they differ; with `mode: enabled` it also patches the drifted resources back. The agent adds `driftDetection` to the base release so both clusters get it, then uses the overlays the repository already has to stage the rollout: `enabled` in `apps/staging/podinfo-values.yaml` and `warn` in `apps/production/podinfo-values.yaml`, since a Kustomize patch on a HelmRelease is exactly how those two files already set version and hostname. Production moves to `enabled` in a later pull request once its events show nothing unexpected.

Ignore rules are where the job needs care, because a controller that legitimately changes a resource would otherwise be fought forever. The podinfo chart's `values.yaml` has `hpa.enabled: false` by default, so the release's Deployment keeps its `replicaCount`, but the agent still adds `ignore: [{paths: ["/spec/replicas"], target: {kind: Deployment}}]` so that enabling the chart's HPA later does not turn into a correction loop; the spec's own example uses that path for the same reason. For `cert-manager`, whose `cainjector` writes the `caBundle` into the chart's webhook configurations at runtime, the agent adds an ignore for `/webhooks/0/clientConfig/caBundle` on `ValidatingWebhookConfiguration` and `MutatingWebhookConfiguration`, and sets that release to `warn` first. `envoy-gateway` gets `warn` with no ignore rules, so a week of events shows which of its resources, if any, another controller touches.

Then the agent checks what it can without a cluster. It runs `scripts/validate.sh`, which builds every Kustomize overlay with `--load-restrictor=LoadRestrictionsNone` and validates the output against the ecosystem schemas, so a misspelt `driftDetection` key or a wrong `target` field fails here rather than on the cluster. It runs `kustomize build apps/staging` and `apps/production` and diffs the rendered HelmReleases to show that only `driftDetection` changed. The kind-based `e2e` workflow needs Docker, which the sandbox does not have; the agent runs it only when asked, installing `docker.io` with `apt-get` and starting `dockerd` first, and otherwise leaves it to the pull request's own checks, where it already runs on every push.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Enable Flux drift detection on the HelmReleases in this repository. Read every HelmRelease under apps/ and infrastructure/, the Kustomize overlays that patch them, .fluxschema.yml, scripts/validate.sh and the GitHub Actions workflows first. Add spec.driftDetection to each release: mode enabled for the application release in the staging overlay and mode warn in the production overlay, warn for the infrastructure releases. Add ignore rules only for fields another controller writes: /spec/replicas on Deployments that a HorizontalPodAutoscaler may scale, and the webhook caBundle paths that cert-manager's cainjector fills in. Run scripts/validate.sh and kustomize build on each overlay, diff the rendered HelmReleases against main to show only driftDetection changed, and do not run the kind cluster unless asked. Open a pull request that explains each ignore rule, how to read the Drifted condition and DriftDetected events, and when to move production from warn to enabled.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

The pull request changes `apps/base/podinfo/release.yaml`, the two `podinfo-values.yaml` overlays, `infrastructure/controllers/cert-manager.yaml` and `infrastructure/controllers/envoy-gateway.yaml`, and nothing under `clusters/`. Its description quotes the `validate.sh` summary line, the `kustomize build` diff per overlay, and a short guide to reading the result on the cluster: `kubectl -n podinfo describe helmrelease podinfo` shows the `Drifted` condition, `True` with reason `DriftDetected` or `False` with `NoDriftDetected`, and the controller's events carry a summary of what differed and, in `enabled` mode, what it corrected.

A reviewer checks that every ignore rule names a field some controller really owns, that production starts in `warn`, and that the staging release is the one they want to see corrected first. After merge, Flux applies the change on the next reconcile of the `apps` and `infra-controllers` Kustomizations, and the first drift event, or the absence of one over a few intervals, tells the team whether the rules are complete before anyone enables correction in production.

## Run it on every new HelmRelease in the repository

A platform team adds an automation to the Project with a `github` trigger on `pull_request.opened` and a prompt that asks the agent to read any HelmRelease the pull request adds and comment with the `driftDetection` block and ignore rules it would add, naming the controllers that own each ignored field. An automation's runs are read-only, so the comment is the result; the author adds the block or asks for it. Only pull requests by repository members fire the trigger, and the automation's `max_runs_per_hour` caps a busy day.

To apply the change across several GitOps repositories at once, a script on the team's own server sends this body once per repository with a `request_id` that is a UUID v5 of the repository's full name, so a retry returns the run already started, and the key's `webhook_url` reports each pull request URL in `result`.

## Why run it on Rush?

Drift detection is one field, but the ignore rules are a judgment about every controller in the cluster, and getting them wrong means Flux and an autoscaler, or Flux and cert-manager, rewriting the same object every interval. The agent reads the charts' default values and the repository's own overlays, writes the rules with the reason for each, and validates every rendered manifest against the Flux schemas in its own sandbox, with the commands and diffs kept in the run's record, so the reviewer sees exactly what will reach each cluster.

The repository's structure sets the rollout, and the agent follows it rather than flattening it: the base gets the field, the overlays decide `warn` or `enabled` per cluster, the same way they already decide the chart constraint and hostname. A fleet of GitOps repositories gets a pull request each, fitted to its releases and controllers.

## Frequently asked questions

### What is the difference between warn and enabled?

In `warn` the controller detects drift with a server-side dry-run and reports it as an event and a `Drifted` condition, changing nothing. In `enabled` it also creates and patches the drifted resources back to the Helm manifest and reports what it did. The pull request starts production in `warn` so the first events can be read before any correction happens.

### Will Flux fight my HorizontalPodAutoscaler?

Not with the ignore rule the pull request adds: `/spec/replicas` on Deployments is excluded from detection and correction, which the helm-controller spec gives as the example for exactly this case. The podinfo chart ships with `hpa.enabled: false`, so the rule costs nothing today and protects the release if that value changes.

### Why does cert-manager need an ignore rule?

Its `cainjector` writes the CA bundle into the chart's webhook configurations after they are applied, so the live object always differs from the rendered manifest at that one path. Ignoring `/webhooks/0/clientConfig/caBundle` on both webhook configuration kinds keeps that from showing up as drift on every interval.

### Can I exclude a whole resource instead of a path?

Yes. The spec allows an ignore rule whose path is the empty JSON Pointer, `""`, with a `target` selector, which covers the whole document, or the annotation `helm.toolkit.fluxcd.io/driftDetection: disabled` applied through a `postRenderers` Kustomize patch. The pull request prefers paths because they keep the rest of the resource under detection.

### How do I see that it is working?

Describe the HelmRelease: the `Drifted` condition reads `True` with reason `DriftDetected` or `False` with `NoDriftDetected`, and the controller emits an event with a summary of the differences. Running `kubectl scale` on the podinfo Deployment in staging and watching the next reconcile is the simplest live check, and the pull request describes it.

## More Flux jobs

- [Add Image Update Automation to a Flux Repository](https://getrush.ai/run/add-image-automation/flux/)
- [Add Kyverno Policies to a Flux Repository](https://getrush.ai/run/add-kyverno-policies/flux/)
- [Publish Manifests as OCI Artifacts for Flux](https://getrush.ai/run/add-oci-artifacts/flux/)

## The same job on other stacks

- [Add Drift Detection to a Pulumi Program](https://getrush.ai/run/add-drift-detection/pulumi/)
- [Add Terraform Drift Detection to an EKS Configuration](https://getrush.ai/run/add-drift-detection/terraform/)

## More jobs like this: Hand off a long task

- [Add a --dry-run Flag to a Cobra CLI](https://getrush.ai/run/add-dry-run-flag/cobra/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
