All jobs

Add Kyverno Policies to a Flux Repository

Yes. An agent installs Kyverno through a Flux HelmRelease, adds cluster policies that audit in staging and enforce in production, proves it on kind, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the clusters are wired before it adds a controller. In a repository laid out like fluxcd/flux2-kustomize-helm-example, clusters/staging and clusters/production each hold an ArtifactGenerator that splits the monorepo into infrastructure and apps artifacts, and three Flux Kustomization objects: infra-controllers from ./controllers with wait: true, infra-configs from ./configs with dependsOn: infra-controllers, and apps with dependsOn: infra-configs. infrastructure/controllers installs cert-manager and Envoy Gateway as OCIRepository plus HelmRelease pairs, each namespace labelled toolkit.fluxcd.io/tenant: sre-team, each release with RetryOnFailure install and upgrade strategies; infrastructure/configs holds the ClusterIssuer, GatewayClass and Gateway, and the staging cluster patches the issuer's ACME server in clusters/staging/infrastructure.yaml. apps/base/podinfo runs the podinfo chart with a pinned Redis image tag and an HTTPRoute. scripts/validate.sh builds every overlay and validates it with the Flux schema plugin against the ecosystem catalog in .fluxschema.yml, test.yaml runs that and flux migrate, and e2e.yaml boots a kind cluster, installs Flux, reconciles ./clusters/staging and waits for infra-controllers, apps and the podinfo release before curling through the gateway.

Kyverno joins the controllers layer. infrastructure/controllers/kyverno.yaml adds a kyverno namespace with the sre-team tenant label, a HelmRepository for https://kyverno.github.io/kyverno/ and a HelmRelease for the kyverno chart on a 3.x semver range with the same RetryOnFailure strategies as its neighbours. Its values extend config.webhooks so the webhook namespaceSelector excludes flux-system as well as the chart's default kube-system: if Kyverno is down, Flux must still be able to reconcile the cluster, including Kyverno itself. The Kyverno namespace is excluded by the chart's config.excludeKyvernoNamespace default. The file is listed in the controllers kustomization.yaml, and because infra-controllers already sets wait: true, the Kyverno CRDs and webhooks are ready before infra-configs applies anything that depends on them.

The policies live in the configs layer, where the dependsOn chain guarantees the ClusterPolicy kind exists. infrastructure/configs/policies.yaml carries two: require-tenant-label, which validates that every Namespace carries toolkit.fluxcd.io/tenant, the convention this repository already follows for cert-manager, envoy-gateway-system, podinfo and now kyverno; and disallow-latest-tag, which validates that every container in a Pod names an image tag other than latest, which podinfo and its pinned Redis already satisfy. Both are written with Audit as the failure action in the base so a violation is reported, not blocked, and clusters/production/infrastructure.yaml gets a patch on the infra-configs Kustomization that sets both to Enforce, mirroring how the ACME server is patched per cluster. Both exclude kube-system and flux-system so the cluster's own components are never caught.

The agent proves the change the way the repository does. It runs scripts/validate.sh with the Flux schema plugin so every overlay builds and the new objects validate or are skipped as unknown kinds, and flux migrate -f . to confirm no deprecated API. The sandbox has no Docker, so it installs it with sudo apt-get install -y docker.io, starts dockerd, creates a kind cluster and follows e2e.yaml step by step: flux install, a GitRepository and Kustomization for ./clusters/staging, waits for infra-controllers, infra-configs and apps, checks the kyverno release is ready, reads the PolicyReport objects to see podinfo pass both rules, and applies a throwaway namespace without the tenant label to see an audit entry appear. It repeats the reconcile with the production path to confirm the same namespace is rejected by the webhook, deletes the throwaway objects, and opens the pull request with those outputs.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add Kyverno with cluster policies to this Flux repository and open a pull request. Read clusters/staging, clusters/production, infrastructure/controllers, infrastructure/configs, apps/base/podinfo, scripts/validate.sh, .fluxschema.yml and both workflows under .github/workflows first. Add infrastructure/controllers/kyverno.yaml with a kyverno namespace carrying the tenant label used elsewhere, a HelmRepository for the Kyverno chart and a HelmRelease on a 3.x range with the RetryOnFailure strategies the other releases use, and values that exclude flux-system from the webhook namespace selector alongside kube-system; list it in the controllers kustomization. Add infrastructure/configs/policies.yaml with a require-tenant-label policy for namespaces and a disallow-latest-tag policy for pods, both in Audit mode and both excluding kube-system and flux-system, and list it in the configs kustomization. Patch the infra-configs Kustomization in the production cluster to switch both policies to Enforce. Run scripts/validate.sh and flux migrate, then install docker.io, start dockerd, boot a kind cluster, install Flux, reconcile the staging cluster path, wait for the controllers, configs and apps, read the policy reports for podinfo, apply a namespace without the label to see it audited, reconcile the production path to see it rejected, and clean up. Open a pull request with the validate output, the policy reports and the webhook rejection.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request adds infrastructure/controllers/kyverno.yaml and infrastructure/configs/policies.yaml, and changes the two kustomization.yaml files beside them and clusters/production/infrastructure.yaml. Its description lists the two policies with the resources they match and their failure action per cluster, quotes the validate.sh summary, the PolicyReport lines for podinfo from the kind run, the audit entry for the unlabelled namespace in staging and the webhook's rejection message in production.

It explains the two exclusions as the safety of the design: flux-system is outside the webhooks so a Kyverno outage cannot stop Flux from reconciling, and both policies skip kube-system and flux-system so no cluster component is ever reported or blocked. It also notes that Audit in the base means a new cluster starts by reporting, and only the production patch turns reporting into refusal.

Before merging, a reviewer checks the chart's semver range against the Kyverno release that introduced the policy fields in use, decides whether the admission controller should run more than one replica in production, and chooses which further policies, such as requiring resource limits or a non-root security context, should follow once the reports from staging are read.

Run it on every policy report

Once the policies run, a scheduled automation on the Project, daily, with the prompt "read the PolicyReport objects on the staging cluster through the kubeconfig in this Project's secrets, and for every new failure open a Linear issue naming the resource, the rule and the file in this repository that defines it" starts one run a day. The run has the same cluster access an engineer would, held as a Project secret rather than in the prompt, and finds nothing to file on a day with no new violation.

A platform team that runs several Flux repositories sends this body once per repository from its own server, each POST /o/{org}/sessions with that repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started. Each run reads its own layout: a repository without a configs layer gets the policies in a new Kustomization that depends on the controllers, and one that already has Kyverno gets only the policies and the per-cluster patch.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

An admission controller in a GitOps repository is a dependency puzzle: the CRDs must exist before the policies, the policies must not fire on the reconciler that installs them, and a rule that enforces on day one blocks workloads nobody has audited. The agent reads the dependsOn chain, the wait flags and the per-cluster patches that this repository already uses, places each new object where the chain makes it safe, and then boots a cluster in its sandbox to watch the whole thing reconcile, which is the only way to know a webhook and a policy behave.

The validate output, the kind reconcile, the policy reports and the rejection message all stay with the run, so a reviewer reads what happened on a real cluster instead of reasoning about YAML. A team with several Flux repositories sends the same request to each and reads one pull request per repository, each proved on its own clusters' paths.

Frequently asked questions

Why exclude `flux-system` from the webhooks?

Because Flux installs and upgrades Kyverno. If the webhook is unreachable while Kyverno is restarting and flux-system were in scope, kustomize-controller's own applies could be refused and the cluster could not heal itself. Excluding the namespace keeps the reconciler outside the thing it reconciles.

Why Audit in the base and Enforce only in production?

So the first reconcile on any cluster reports rather than refuses, and the team reads the reports before anything is blocked. The production patch follows the pattern already used for the ACME server: the base holds the safe default, and the cluster directory holds the stricter value.

Does `scripts/validate.sh` know the Kyverno kinds?

It validates with the Flux schema plugin against the ecosystem catalog named in .fluxschema.yml, and skipMissingSchemas is on, so a ClusterPolicy is validated when the catalog carries its schema and skipped otherwise. The kind run is what proves the policies behave either way.

Will podinfo keep deploying?

Yes. Its namespace carries the tenant label and its chart pins the application and Redis image tags, so both policies pass, which the PolicyReport quoted in the pull request shows. A future chart value that pulled latest would be reported in staging and refused in production.

More Flux jobs

The same job on other stacks

More jobs like this: Hand off a long task

Keep reading