All jobs

Add Cosign Signature Verification to Flux Helm Sources

Yes. An agent checks which Flux Helm sources are cosign-signed, adds verify blocks only where it verified a signature, validates the manifests and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent starts by finding out what is signed, because a verify block on an unsigned source stops reconciliation. In a repository laid out like fluxcd/flux2-kustomize-helm-example there are three chart sources: apps/base/podinfo/repository.yaml is a HelmRepository over HTTPS at https://stefanprodan.github.io/podinfo, consumed by the HelmRelease in release.yaml through chart.spec.sourceRef; infrastructure/controllers/cert-manager.yaml and envoy-gateway.yaml each pair an OCIRepository (oci://quay.io/jetstack/charts/cert-manager with semver: "1.x", oci://docker.io/envoyproxy/gateway-helm with semver: ">= 1.8.0-rc.0", both with a layerSelector for the chart content media type) with a HelmRelease that points at it through chartRef. No manifest in the repository carries a verify field. The agent downloads the current cosign release into its sandbox and runs cosign verify against each registry: the podinfo chart published at ghcr.io/stefanprodan/charts/podinfo verifies keyless, with the certificate issuer https://token.actions.githubusercontent.com and a subject of the form https://github.com/stefanprodan/podinfo/.github/workflows/release.yml@refs/tags/<version>; the cert-manager chart carries a .sig tag that verifies only with cert-manager's published public key together with --signature-digest-algorithm sha512 and --insecure-ignore-tlog; the Envoy Gateway chart has no signature at all.

Flux's documentation says chart verification is available only for charts fetched from an OCI registry, so the HTTPS podinfo repository cannot be verified as it stands. The agent replaces it with an OCIRepository named podinfo whose url is oci://ghcr.io/stefanprodan/charts/podinfo, with the same layerSelector the two infrastructure sources already use, a ref.semver range, and a verify block with provider: cosign and no secretRef, which is how the OCIRepository API asks for keyless verification, plus one matchOIDCIdentity entry whose issuer is ^https://token.actions.githubusercontent.com$ and whose subject is ^https://github.com/stefanprodan/podinfo.*$, the identity the agent read off the signature. The HelmRelease moves from chart.spec to chartRef, the shape cert-manager.yaml already uses, which also means the version: ">=1.0.0-alpha" and ">=1.0.0" patches in apps/staging/podinfo-values.yaml and apps/production/podinfo-values.yaml no longer have a chart.spec to patch; the agent moves those ranges into the ref.semver of the OCIRepository per environment, so staging still follows pre-releases and production stable versions, and checks that the chart versions published to the OCI repository match the ones in the HTTPS index.

For cert-manager the agent does not guess. The OCIRepository verify block takes a provider, a secretRef holding .pub keys, matchOIDCIdentity matchers and a trusted root secret, and nothing that selects a SHA-512 signature digest or skips the transparency log, which is what the cosign command line needed. So the pull request leaves cert-manager without a verify block, quotes the exact cosign verify invocation that succeeds and the one that fails, and suggests trying a secretRef on the staging cluster where a failed verification only pauses one HelmRelease. Envoy Gateway is recorded as unsigned, with the cosign tree output that shows no related artifacts, so nobody adds a block that could never pass.

The repository already validates itself, and the agent runs that machinery rather than inventing its own: scripts/validate.sh builds every kustomize overlay and checks each resource against the Flux Schema ecosystem catalog named in .fluxschema.yml, which is what the test workflow runs through the fluxcd/flux-schema actions, and the same workflow runs flux migrate -f . --yes and fails on any diff. The agent installs the flux CLI and flux-schema, runs both, and builds apps/staging and apps/production to read the rendered OCIRepository and HelmRelease side by side. The e2e workflow starts a kind cluster, installs Flux with the source-watcher extra component and waits for helmrelease/podinfo to become ready; the sandbox has no Docker, so the agent installs it with sudo apt-get install -y docker.io, starts dockerd, and repeats those steps with its branch, which is the only way to see the podinfo artifact fetched and verified by source-controller before helm-controller installs it.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add cosign signature verification to the Helm chart sources this Flux repository reconciles. Read every HelmRepository, OCIRepository and HelmRelease under apps/ and infrastructure/, the overlays under apps/staging and apps/production, clusters/, .fluxschema.yml, scripts/validate.sh and the workflows under .github/workflows first. Install cosign and run cosign verify against each chart source to learn which are signed, keyless or with a key, and which are not. Where a chart is only available over HTTPS but the same chart is published and signed in an OCI registry, switch the source to an OCIRepository with the same layerSelector the infrastructure sources use, move the HelmRelease to chartRef, and keep the per-environment version ranges. Add a verify block with provider cosign and matchOIDCIdentity for keyless signatures; do not add verification to a source you could not verify yourself, and say why in the pull request with the cosign output. Run scripts/validate.sh, flux migrate -f . --yes, and the e2e workflow's steps on a kind cluster, then open a pull request with the commands and their output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request changes apps/base/podinfo/repository.yaml and release.yaml, the two podinfo-values.yaml overlays and the README section that shows the HelmRelease shape. Its description has one table with a row per chart source: registry, how it is signed, the cosign command that proved it and whether a verify block was added. Below it are the scripts/validate.sh summary, the empty flux migrate diff, and the flux tree ks flux-system listing from the kind cluster with the podinfo HelmRelease ready.

A reviewer checks the identity regexes against the signature subject, decides whether production should pin a narrower semver range than >=1.0.0, and reads the cert-manager note to decide whether a key-based secretRef is worth a staging trial. After merging, source-controller verifies every new podinfo chart version before helm-controller sees it; a chart whose signature does not match the identity stays unfetched and the OCIRepository reports the failure in its conditions.

Run it on every Linear issue labelled supply-chain

An automation in the Project that holds the cluster repository takes a linear trigger on linear.issue.labeled with the label supply-chain. Its runs are read-only, so the saved prompt asks for the survey only: the run runs cosign verify and cosign tree against every source, lists which can be verified and how, and posts that as a comment on the issue, where the team decides which sources to lock down. A handler on the team's own server then starts the run that writes the pull request, sending this body with the issue's URL added to resources and a request_id that is a UUID v5 of the issue id, so a retried start returns the run already started.

A platform team with one repository per cluster sends one request per repository, each with its own resources entry; a source that is signed in one registry and unsigned in another gets a different answer per repository.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

The decision in this job is made by querying registries, not by reading YAML: whether a chart is signed, with which identity, and whether the signature is one Flux's controllers can check. The agent runs cosign against each registry in its own sandbox, keeps every command and its output in the run's record, and writes only the verify blocks those outputs support. A block copied from documentation without that check would stop the release on the first reconcile.

It also proves the change the way the repository proves itself: the schema validation and migration check from the test workflow, and a kind cluster running the e2e steps, so the review reads a ready HelmRelease rather than a promise. Several cluster repositories get the same survey at once, each with its own table of sources.

Frequently asked questions

Why switch podinfo from the HTTPS repository to an OCI one?

Flux verifies chart signatures only for charts fetched from an OCI registry; a HelmRepository over HTTPS has no verify field that works. The same chart is published, signed, to ghcr.io/stefanprodan/charts/podinfo, so the switch costs one resource change and the HelmRelease takes the chartRef shape the infrastructure releases already use.

What happens when a new chart version is not signed?

Source-controller refuses to produce an artifact for it and reports the failed verification on the OCIRepository; the running release keeps the version it has. That is the point of the block, and it is also why the agent adds one only where it verified a signature itself.

Does the identity regex need updating on every release?

No. The subject matcher ^https://github.com/stefanprodan/podinfo.*$ matches the release workflow at any tag, and the issuer matcher pins GitHub's OIDC provider. A chart signed from a fork or another workflow would not match, which is the protection you want.

Why not add a public key for cert-manager?

The agent found that cert-manager's chart signature verifies only with a SHA-512 digest and with the transparency log ignored, and the OCIRepository verify block exposes neither choice. Rather than ship a block that might never pass, the pull request records the working cosign command and suggests a staging trial.

More Flux jobs

More jobs like this: Turn a Linear label into a PR

Keep reading