All jobs

Add Canary Rollouts to a Flux Repository

Yes. An agent installs Flagger through Flux, turns your Helm-released app into a Canary behind your Gateway, validates every manifest and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the repository's layout before it adds a controller, because in a Flux monorepo the order things reconcile is written in the files. In a repository like fluxcd/flux2-kustomize-helm-example, clusters/staging/artifacts.yaml holds an ArtifactGenerator that splits the monorepo into an infrastructure artifact from infrastructure/** and an apps artifact from apps/base/** and apps/staging/**. clusters/staging/infrastructure.yaml defines two Flux Kustomizations on that artifact, infra-controllers with wait: true and infra-configs that dependsOn it, and clusters/staging/apps.yaml defines apps, which depends on infra-configs. infrastructure/controllers/ installs cert-manager and Envoy Gateway as OCIRepository plus HelmRelease pairs, and infrastructure/configs/gateway.yaml creates a GatewayClass and a Gateway named envoy in envoy-gateway-system with an HTTP listener on port 80. apps/base/podinfo/release.yaml is a HelmRelease for the podinfo chart with httpRoute.enabled: true, parentRefs pointing at that gateway and a hostname of podinfo.local, which apps/staging/podinfo-values.yaml and apps/production/podinfo-values.yaml patch to podinfo.staging and podinfo.production. .github/workflows/test.yaml validates every manifest with fluxcd/flux-schema and fails on deprecated API versions through flux migrate, scripts/validate.sh does the same locally, and e2e.yaml boots a kind cluster on every push and waits for infra-controllers, apps and the podinfo HelmRelease to become ready.

Flagger goes in beside the other controllers. The agent adds infrastructure/controllers/flagger.yaml with a flagger-system namespace, an OCIRepository for oci://ghcr.io/fluxcd/charts/flagger with semver: "1.x", and a HelmRelease with install.crds and upgrade.crds set to CreateReplace, as Flagger's own Flux install guide shows, and values of meshProvider: gatewayapi:v1, since the chart lists that provider for any Gateway API implementation, and prometheus.install: true, which the chart's values offer for environments with no Prometheus yet and points metricsServer at the one it installs. A second file installs Flagger's load tester from the oci://ghcr.io/fluxcd/flagger-manifests artifact with a Flux Kustomization on its tester path, targeting the podinfo namespace. Both are listed in infrastructure/controllers/kustomization.yaml, so the existing infra-controllers Kustomization with wait: true registers the Canary CRD before apps reconciles, which is what the dependsOn chain already guarantees.

The app side is one new resource and one changed value. apps/base/podinfo/canary.yaml defines a Canary whose targetRef is the podinfo Deployment the chart creates under releaseName: podinfo, a service on port 9898 with gatewayRefs naming the envoy Gateway in envoy-gateway-system and hosts of podinfo.local, and an analysis with interval: 1m, threshold: 5, maxWeight: 50 and stepWeight: 10. Flagger creates and owns the HTTPRoute for a Canary, so the agent sets httpRoute.enabled: false in release.yaml and moves the hostname patches: each cluster's podinfo-values.yaml keeps its HelmRelease patch, and a new podinfo-canary.yaml patch per overlay, registered in that overlay's kustomization.yaml with a Canary target, sets podinfo.staging or podinfo.production on the Canary's hosts. Two MetricTemplates in flagger-system query the installed Prometheus for podinfo's own http_request_duration_seconds histogram, one for the share of 5xx responses and one for the p99 latency, and the analysis lists them with thresholdRange limits and a load-test webhook that asks the load tester to run hey against the canary's hostname during each step.

Validation is the repository's own. The agent installs the Flux CLI, the flux-schema plugin and kustomize in its sandbox and runs scripts/validate.sh, which builds every overlay with LoadRestrictionsNone, validates the output against the ecosystem schema catalog and reports how many resources were valid, and flux migrate -f . --yes followed by git diff --stat to prove no API version is deprecated. It builds apps/staging and apps/production with kustomize build and reads the Canary each produces to confirm the hostname patch landed. The kind end-to-end run needs Docker, which the sandbox does not ship; the agent installs docker.io, starts dockerd and runs the steps from e2e.yaml when the prompt asks for it, and otherwise notes that the workflow runs on the push of the branch. Pushing the branch starts that workflow in the reader's CI, where the new HelmReleases and the Canary have to become ready for the job to pass.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add canary rollouts with Flagger to this Flux repository. Read README.md, every file under clusters, infrastructure and apps, .fluxschema.yml, scripts/validate.sh and .github/workflows first. Install Flagger under infrastructure/controllers as an OCIRepository for oci://ghcr.io/fluxcd/charts/flagger at semver 1.x and a HelmRelease with CRDs set to CreateReplace, meshProvider gatewayapi:v1 and the chart's bundled Prometheus, plus Flagger's load tester from oci://ghcr.io/fluxcd/flagger-manifests into the podinfo namespace, and list both in the controllers kustomization. Add a Canary for the podinfo Deployment under apps/base/podinfo with gatewayRefs on the existing envoy Gateway, port 9898, an analysis of interval 1m, threshold 5, maxWeight 50 and stepWeight 10, MetricTemplates on podinfo's http_request_duration_seconds histogram for error share and p99 latency, and a load-test webhook; set httpRoute.enabled to false in the HelmRelease because Flagger owns the route, and add per-cluster patches so the Canary's hosts match the staging and production hostnames. Run scripts/validate.sh, flux migrate -f . --yes with git diff --stat, and kustomize build on both app overlays, and open a pull request quoting the output and explaining what the e2e workflow will check on push.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request adds infrastructure/controllers/flagger.yaml, a load tester file, apps/base/podinfo/canary.yaml, two MetricTemplates and a podinfo-canary.yaml patch under apps/staging and apps/production, and changes infrastructure/controllers/kustomization.yaml, apps/base/podinfo/kustomization.yaml, release.yaml and both overlay kustomization.yaml files. Its description quotes the validation script's summary line, the empty git diff --stat after flux migrate, and the rendered Canary from each overlay with its hostname.

A reviewer checks the dependency chain, that Flagger lands in infra-controllers so its CRD exists before apps applies the Canary, and the analysis numbers, which decide how fast a bad release gets traffic and how many failed checks roll it back. The end-to-end workflow on the branch is the proof that the whole set reconciles on a fresh cluster; a HelmRelease that never turns ready fails that job before merge.

Run it on every Flux repository with a Helm-released app

A platform team hands this off from the terminal once per repository: one POST /o/{org}/sessions with this body and the repository in resources, naming the application and its Gateway in input when the repository holds more than one. Each run reads its own layout, so a repository that already runs a Prometheus gets metricsServer pointed at it rather than a second one, and a repository with no Gateway API gets a note instead of a Canary. A request_id that is a UUID v5 of the repository's full name means a retried request returns the run it already started.

The key's webhook_url reports each run the key starts with the pull request URL in result, so a script that sends one request per cluster repository collects the pull requests without polling.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

A canary setup is one controller and one resource, but it has to land in the right order and agree with manifests that already exist: the Gateway it attaches to, the chart's own route, the hostname patch per cluster, and the validation the repository already enforces. The agent reads all of it in its own sandbox, runs the repository's validation script, the migration check and the overlay builds, and keeps every command and its output in the run's record.

Progressive delivery is also where numbers matter, and the agent writes them into the Canary where a reviewer can argue with them: the step size, the ceiling, the failure count and the metric thresholds. The pull request names each one and what it does, so the on-call engineer knows what a rollback will look like before the first release goes through it.

Frequently asked questions

Why turn off the chart's HTTPRoute?

Flagger creates and manages the HTTPRoute for a Canary, splitting weight between the primary and canary Services it also creates. Leaving httpRoute.enabled: true in the HelmRelease would give the Gateway two routes for one hostname, so the agent disables the chart's and moves the hostname into the Canary's hosts.

Does this work with Envoy Gateway?

Yes. Flagger's gatewayapi:v1 provider works with any Gateway API implementation, and the repository already installs Envoy Gateway and a Gateway named envoy, which is what the Canary's gatewayRefs names. The load tester sends its requests through that Gateway with the canary's Host header.

Where do the metrics come from?

From the Prometheus the Flagger chart installs when prometheus.install is on, which scrapes pods that expose metrics, and from podinfo's own http_request_duration_seconds histogram. A repository with a Prometheus already gets metricsServer pointed at it instead, and the two MetricTemplates query whichever one is configured.

What happens when a release fails analysis?

Flagger stops advancing when a check fails, counts failures against threshold, and after five failed checks routes all traffic back to the primary and marks the Canary failed. The Deployment keeps the new image, the primary keeps the old one, and the next commit starts a new analysis.

Can the end-to-end test run in the sandbox?

It needs a kind cluster, which needs Docker, and the sandbox has no Docker installed. Ask for it in input and the agent installs docker.io, starts dockerd, creates the cluster and runs the steps from e2e.yaml; otherwise it runs the schema validation and overlay builds and leaves the cluster test to the workflow on push.

More Flux jobs

The same job on other stacks

More jobs like this: Hand off a long task

Keep reading