# Fix the Manifest Behind a Failed Argo CD Sync

Yes. Argo CD Notifications posts the failed sync to a Rush webhook; an agent reproduces the render, fixes the manifest or chart and opens a pull request.

[Run this on Rush](https://cloud.getrush.ai/run/fix-failed-syncs/argo-cd/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The run starts with the notification payload after the prompt: the Application name, its `source.path`, the revision and the message from `status.operationState`. The agent clones the repository into its own cloud sandbox at that revision and goes to the path the Application points at. In a repository laid out like argoproj/argocd-example-apps, that path is one of many shapes: `guestbook` is plain YAML, `kustomize-guestbook` and `sock-shop` are Kustomize, `helm-guestbook` and `helm-dependency` are Helm charts, `jsonnet-guestbook` is Jsonnet, `sync-waves` and `pre-post-sync` carry `argocd.argoproj.io/hook` and `sync-wave` annotations, and `plugins/` needs a config management plugin. The agent reads `apps/values.yaml` to see how the Application is declared, including its `tool` block, before it renders anything.

It then reproduces the failure the way Argo CD would have hit it. For a Kustomize path it runs `kustomize build` and notes, for example, that `pre-post-sync/kustomization.yaml` still uses the deprecated `bases:` key. For a chart it runs `helm dependency build` and `helm template`: `helm-dependency/Chart.yaml` pulls `wordpress` `9.0.3` from `https://charts.helm.sh/stable`, the archived stable repository, and its `values.yaml` sets `wordpress.image.tag: invalid` with a comment that this app is broken on purpose. The agent reads that comment and reports it instead of fixing it, because a deliberate break is not a bug. Rendered manifests go through `kubectl apply --dry-run=client` and a schema check, and a hook Job with a `generateName`, as in `sync-waves/manifests.yaml`, is checked against the hook delete policy it needs.

When the cause is in the repository, the agent makes the smallest change that renders cleanly: a field name, an API version, a missing `namespace` on a resource, a chart dependency moved to a repository that still serves it, a hook annotation that the sync phase needs. It commits one logical change, re-runs the same render and dry-run to show the error is gone, and leaves every other path untouched. When the message points outside the repository, a cluster permission, an image pull failure, a resource that another controller owns, the agent writes up what it found with the exact render output and opens no pull request.

The wiring is Argo CD Notifications, which the repository does not yet have. The agent adds an `argocd-notifications-cm` with a `service.webhook.rush` whose `url` is the Rush trigger's URL and whose `headers` carry `Authorization` with `Bearer $rush-trigger-secret`, a `template.rush-sync-failed` with `method: POST` and a JSON body built from `.app.metadata.name`, `.app.spec.source.path`, `.app.status.sync.revision` and `.app.status.operationState.message`, and a `trigger.on-sync-failed` whose condition is `app.status.operationState.phase in ['Error', 'Failed']`. In `apps/templates/applications.yaml` it adds the `notifications.argoproj.io/subscribe.on-sync-failed.rush` annotation so every generated Application subscribes, and a stub `argocd-notifications-secret` with the secret's key named and its value left for you.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "An Argo CD sync failed; the notification payload follows with the Application name, source path, revision and the operation message. Clone this repository at that revision, read how the Application is declared and go to its source path. Reproduce the failure with the same tool Argo CD uses for that path: kustomize build, helm dependency build and helm template, or kubectl apply --dry-run=client for plain YAML. If a file in the repository is the cause, make the smallest fix, re-run the render to show it passes, and open a pull request with the before and after output. If the path is broken on purpose, or the cause is outside the repository, report what you found and open nothing.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request on its own branch that touches only the failing path: one manifest, one `kustomization.yaml` or one `Chart.yaml` and `values.yaml` pair. Its description quotes the operation message from the notification, the render command that reproduced it with its error, the change, and the same command passing afterwards, so a reviewer sees the fix against the failure rather than a description of it.

The run's record keeps the clone at the failing revision and every render and dry-run with its output. When the agent decides not to open a pull request, because the comment in `helm-dependency/values.yaml` says the break is intentional or because the message names something in the cluster, the run's result says so with the output that led there.

Before merging, a reviewer confirms the change is to the file Argo CD actually renders for that Application, not a sibling in another tool's format, and that the chart or overlay still renders the same resources apart from the fix. Once merged, Argo CD picks the commit up on its next refresh and the failed operation is retried as any other change would be.

## Run it on every failed Argo CD sync

Create an automation in the Project with this prompt and a `webhook` trigger. Rush shows the trigger's URL and its secret once; the Argo CD Notifications `service.webhook.rush` in the ConfigMap sends `Authorization: Bearer <secret>` as a header, so Argo CD calls the trigger's URL directly with no handler in between, and the JSON body reaches the agent after the prompt as data, never as instructions.

Argo CD Notifications records what it has already sent for an Application's current state in an annotation, so one failed operation produces one notification and one run rather than one per reconcile loop. A run per Application is independent: two Applications failing on the same commit become two sandboxes and two pull requests, and the automation's `max_runs_per_hour` bounds a repository-wide breakage.

## Why run it on Rush?

A failed sync is read in the Argo CD UI, reproduced on someone's laptop with whichever version of `helm` or `kustomize` is installed there, and fixed from memory of what the message meant. The agent reproduces it at the exact revision with the exact render command, and the pull request carries both runs of that command, so the reviewer compares outputs rather than recollections.

The sandbox is one per failure and disposable: a chart with a dependency from an archived repository, a Kustomize overlay with a deprecated key and a plain manifest with a wrong API version can be worked on at once without three people being paged. Everything the agent ran stays in the run's record, and nothing is applied to any cluster.

## Frequently asked questions

### Can Argo CD call the Rush webhook directly?

Yes. The Notifications webhook service takes a `headers` list, so the ConfigMap sets `Authorization` to `Bearer` and the trigger's secret, read from `argocd-notifications-secret` as `$rush-trigger-secret`. No forwarding handler is needed, and the token never appears in the repository.

### What if the sync failed because of the cluster, not the repository?

The agent reports it and opens nothing. An image that cannot be pulled, a namespace it is not allowed to create or a resource owned by another controller shows up in the render as passing and in the operation message as failing, and the run's result says which it is, with the render output.

### Will it fix an Application I broke on purpose?

No. The prompt tells it to read comments and READMEs at the source path, and the example repository marks a chart's `image.tag: invalid` as deliberate. The agent names such a path in its result and leaves the file alone.

### Does the agent change what Argo CD deploys?

Only through the pull request. Nothing is applied to a cluster from the sandbox. The change lands when a reviewer merges it, and Argo CD syncs it on its next refresh like any other commit.

## More Argo CD jobs

- [Review Argo CD App Changes in Pull Requests](https://getrush.ai/run/review-kubernetes-manifests/argo-cd/)
- [Add AppProjects to Restrict Each Team's Argo CD Applications](https://getrush.ai/run/add-app-projects/argo-cd/)
- [Convert an Argo CD App-of-Apps Chart Into an ApplicationSet](https://getrush.ai/run/add-applicationset/argo-cd/)

## More jobs like this: Fix a bug from an alert

- [Fix a Failing CI Build in an Angular App](https://getrush.ai/run/fix-failing-ci/angular/)
- [Fix a Failing Molecule CI Run for an Ansible Role](https://getrush.ai/run/fix-failing-ci/ansible/)
- [Fix Failing GitHub Actions Checks in an ASP.NET Core App](https://getrush.ai/run/fix-failing-ci/aspnet-core/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
