All jobs

Fix the Manifest Behind a Failed Argo CD Sync

Yes. Argo CD Notifications posts the failed sync to a Rush webhook; an agent reproduces the render, fixes the manifest or chart and opens a pull request.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The run starts with the notification payload after the prompt: the Application name, its source.path, the revision and the message from status.operationState. The agent clones the repository into its own cloud sandbox at that revision and goes to the path the Application points at. In a repository laid out like argoproj/argocd-example-apps, that path is one of many shapes: guestbook is plain YAML, kustomize-guestbook and sock-shop are Kustomize, helm-guestbook and helm-dependency are Helm charts, jsonnet-guestbook is Jsonnet, sync-waves and pre-post-sync carry argocd.argoproj.io/hook and sync-wave annotations, and plugins/ needs a config management plugin. The agent reads apps/values.yaml to see how the Application is declared, including its tool block, before it renders anything.

It then reproduces the failure the way Argo CD would have hit it. For a Kustomize path it runs kustomize build and notes, for example, that pre-post-sync/kustomization.yaml still uses the deprecated bases: key. For a chart it runs helm dependency build and helm template: helm-dependency/Chart.yaml pulls wordpress 9.0.3 from https://charts.helm.sh/stable, the archived stable repository, and its values.yaml sets wordpress.image.tag: invalid with a comment that this app is broken on purpose. The agent reads that comment and reports it instead of fixing it, because a deliberate break is not a bug. Rendered manifests go through kubectl apply --dry-run=client and a schema check, and a hook Job with a generateName, as in sync-waves/manifests.yaml, is checked against the hook delete policy it needs.

When the cause is in the repository, the agent makes the smallest change that renders cleanly: a field name, an API version, a missing namespace on a resource, a chart dependency moved to a repository that still serves it, a hook annotation that the sync phase needs. It commits one logical change, re-runs the same render and dry-run to show the error is gone, and leaves every other path untouched. When the message points outside the repository, a cluster permission, an image pull failure, a resource that another controller owns, the agent writes up what it found with the exact render output and opens no pull request.

The wiring is Argo CD Notifications, which the repository does not yet have. The agent adds an argocd-notifications-cm with a service.webhook.rush whose url is the Rush trigger's URL and whose headers carry Authorization with Bearer $rush-trigger-secret, a template.rush-sync-failed with method: POST and a JSON body built from .app.metadata.name, .app.spec.source.path, .app.status.sync.revision and .app.status.operationState.message, and a trigger.on-sync-failed whose condition is app.status.operationState.phase in ['Error', 'Failed']. In apps/templates/applications.yaml it adds the notifications.argoproj.io/subscribe.on-sync-failed.rush annotation so every generated Application subscribes, and a stub argocd-notifications-secret with the secret's key named and its value left for you.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "An Argo CD sync failed; the notification payload follows with the Application name, source path, revision and the operation message. Clone this repository at that revision, read how the Application is declared and go to its source path. Reproduce the failure with the same tool Argo CD uses for that path: kustomize build, helm dependency build and helm template, or kubectl apply --dry-run=client for plain YAML. If a file in the repository is the cause, make the smallest fix, re-run the render to show it passes, and open a pull request with the before and after output. If the path is broken on purpose, or the cause is outside the repository, report what you found and open nothing.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request on its own branch that touches only the failing path: one manifest, one kustomization.yaml or one Chart.yaml and values.yaml pair. Its description quotes the operation message from the notification, the render command that reproduced it with its error, the change, and the same command passing afterwards, so a reviewer sees the fix against the failure rather than a description of it.

The run's record keeps the clone at the failing revision and every render and dry-run with its output. When the agent decides not to open a pull request, because the comment in helm-dependency/values.yaml says the break is intentional or because the message names something in the cluster, the run's result says so with the output that led there.

Before merging, a reviewer confirms the change is to the file Argo CD actually renders for that Application, not a sibling in another tool's format, and that the chart or overlay still renders the same resources apart from the fix. Once merged, Argo CD picks the commit up on its next refresh and the failed operation is retried as any other change would be.

Run it on every failed Argo CD sync

Create an automation in the Project with this prompt and a webhook trigger. Rush shows the trigger's URL and its secret once; the Argo CD Notifications service.webhook.rush in the ConfigMap sends Authorization: Bearer <secret> as a header, so Argo CD calls the trigger's URL directly with no handler in between, and the JSON body reaches the agent after the prompt as data, never as instructions.

Argo CD Notifications records what it has already sent for an Application's current state in an annotation, so one failed operation produces one notification and one run rather than one per reconcile loop. A run per Application is independent: two Applications failing on the same commit become two sandboxes and two pull requests, and the automation's max_runs_per_hour bounds a repository-wide breakage.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

A failed sync is read in the Argo CD UI, reproduced on someone's laptop with whichever version of helm or kustomize is installed there, and fixed from memory of what the message meant. The agent reproduces it at the exact revision with the exact render command, and the pull request carries both runs of that command, so the reviewer compares outputs rather than recollections.

The sandbox is one per failure and disposable: a chart with a dependency from an archived repository, a Kustomize overlay with a deprecated key and a plain manifest with a wrong API version can be worked on at once without three people being paged. Everything the agent ran stays in the run's record, and nothing is applied to any cluster.

Frequently asked questions

Can Argo CD call the Rush webhook directly?

Yes. The Notifications webhook service takes a headers list, so the ConfigMap sets Authorization to Bearer and the trigger's secret, read from argocd-notifications-secret as $rush-trigger-secret. No forwarding handler is needed, and the token never appears in the repository.

What if the sync failed because of the cluster, not the repository?

The agent reports it and opens nothing. An image that cannot be pulled, a namespace it is not allowed to create or a resource owned by another controller shows up in the render as passing and in the operation message as failing, and the run's result says which it is, with the render output.

Will it fix an Application I broke on purpose?

No. The prompt tells it to read comments and READMEs at the source path, and the example repository marks a chart's image.tag: invalid as deliberate. The agent names such a path in its result and leaves the file alone.

Does the agent change what Argo CD deploys?

Only through the pull request. Nothing is applied to a cluster from the sandbox. The change lands when a reviewer merges it, and Argo CD syncs it on its next refresh like any other commit.

More Argo CD jobs

More jobs like this: Fix a bug from an alert

Keep reading