# Add Retry Policies With Exponential Backoff to Airflow DAGs

Yes. An agent sets retries, exponential backoff and a delay cap per DAG, overrides them per task where a retry would repeat work, tests the DagBag, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-retry-policies/airflow/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads every DAG before changing a default. In a project like astronomer/airflow-example-dags, the `Dockerfile` builds on `quay.io/astronomer/astro-runtime:6.0.0`, `packages.txt` adds `postgresql`, `requirements.txt` is empty, and `.astro-registry.yaml` publishes five DAGs. `dags/s3_upload_dag.py` and `dags/s3_upload_copy_dag.py` share a `default_args` block with `retries` 1 and `retry_delay` of five minutes, `max_active_runs` 1, `catchup` off and the schedule `0 12 8-14,22-28 * 6`; the first loops five `PythonOperator` tasks that write a local file and call `S3Hook.load_file` with `replace=True` on the `astro-s3-workshop` connection, the second loops ten uploads each followed by an `S3CopyObjectOperator`, with a stub `on_failure_callback`. `dags/kube_pod.py` runs one `KubernetesPodOperator` with the `hello-world` image under the same single retry. `dags/databricks_sample.py` has no `retries` at all in its args, only `email_on_failure`, and chains an `S3KeySensor` into a `DatabricksRunNowOperator`. `dags/example_external_task_sensor.py` is commented out end to end. So a transient S3 error gets exactly one more try five minutes later, a Databricks job gets none, and a sensor and a notebook run are treated the same as an idempotent upload.

The agent sets a policy per DAG and then overrides it where a retry is not free. In the two S3 DAGs the `default_args` become `retries` 3, `retry_delay` of thirty seconds, `retry_exponential_backoff` on and `max_retry_delay` of ten minutes, so the gap doubles with jitter from thirty seconds and stops growing at ten minutes; the uploads and the copy are safe to repeat because `load_file` replaces and `S3CopyObjectOperator` overwrites the same key. `kube_pod.py` gets `retries` 2 with the same backoff, since a `hello-world` pod is harmless to rerun and the operator names each pod uniquely. In `databricks_sample.py` the sensor gets `mode='reschedule'` and `retries` 0, because poking already handles waiting and a retry on a sensor only hides a broken connection, while `DatabricksRunNowOperator` gets `retries` 1 with a note that a notebook that writes output must be idempotent before that number goes higher. The `on_failure_function` stub stays; beside it the agent adds an `on_retry_callback` in `include/callbacks.py` that logs the task id, `try_number` and `max_tries` from the context, so a retry is visible in the task log rather than only in the grid.

Each DAG keeps its schedule, its task ids and its dependencies, because the ids are what the registry listing, the grid view and any downstream `ExternalTaskSensor` refer to. The change is confined to argument dictionaries and operator keyword arguments, and the agent leaves the deprecated `dummy_operator` and `python_operator` import paths alone, since replacing them is a different job and would widen the diff.

The test is a `DagBag` the project did not have. `tests/dags/test_retry_policies.py` loads `dags/` with `include_examples=False`, asserts there are no import errors, and for each DAG checks that every task has `retry_exponential_backoff` set, a `max_retry_delay` at or under ten minutes, and the expected `retries` by operator type: at least 2 for the S3 operators and the pod, 0 for the sensor and 1 for the Databricks run. The agent reads the runtime image tag in the `Dockerfile`, installs the Airflow release that image carries with its constraints file and the `amazon`, `cncf.kubernetes` and `databricks` providers the DAGs import, and runs `pytest tests/dags` in the sandbox the way `astro dev pytest` would in the project's own tooling.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add retry policies to the DAGs in this Airflow project. Read the Dockerfile, requirements.txt, packages.txt, .astro-registry.yaml, airflow_settings.yaml and every file under dags/ first. In each DAG's default_args set retries, a short retry_delay, retry_exponential_backoff and a max_retry_delay of ten minutes, then override per task: sensors get reschedule mode and no retries, operators that run a notebook or job get one retry with a comment about idempotency, operators that upload or copy to S3 with replace get three, and a pod operator gets two. Add an on_retry_callback in include/callbacks.py that logs the task id, try number and max tries, and wire it beside the existing failure callback. Keep every schedule, task id and dependency unchanged and leave the deprecated import paths alone. Write tests/dags/test_retry_policies.py that loads the DagBag without import errors and asserts the backoff, cap and per-operator retry counts. Install the Airflow release the runtime image carries with its constraints and the amazon, cncf.kubernetes and databricks providers, run pytest tests/dags, and open a pull request with a table of DAG, task type and retry count.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that edits the four live DAG files, adds `include/callbacks.py` and `tests/dags/test_retry_policies.py`, and leaves the commented-out sensor example as it is. The description has one table with each DAG, each task type, its `retries`, the backoff and the cap, and a sentence per override explaining why a sensor gets none and a notebook run gets one.

The run's record keeps the Airflow and provider installation with the constraints file it used, the `pytest tests/dags` output with each assertion named, and a `DagBag` load showing zero import errors, so a reviewer can see the DAGs still parse under the runtime's Airflow release rather than only under the agent's reading of them.

Before merging, a reviewer deploys to a development deployment, breaks the `astro-s3-workshop` connection on purpose, triggers `s3_upload`, and watches the grid: three retries with growing gaps, the `on_retry_callback` lines in the task log, and a failure after the fourth attempt. Restoring the connection and clearing the task shows the retry succeeding without a duplicate upload.

## Run it on every DAG whose first failure pages someone

The setup is one `POST /o/{org}/sessions` with this body and the repository in `resources`, started from a terminal; the run continues in its sandbox after you close the window, and a follow-up through `POST /o/{org}/sessions/{id}/messages` can raise the cap or add a Slack call to the retry callback without starting over. A team with several Astro projects sends one request per repository, each with a fresh UUID as `request_id`, so a retried script returns the run it already started.

Set a `webhook_url` on the API key and Rush posts `run.completed` or `run.failed` when a run's turn finishes, with `session_url` so your server can read the pull request link and the test output.

## Why run it on Rush?

A retry policy is a dictionary, and the harm is in applying one dictionary to every task: a sensor that retries hides a dead connection for an hour, a notebook that retries writes its output twice, and a cap that is missing turns exponential backoff into a task that sleeps until tomorrow. The agent reads what each operator does, including whether the S3 hook replaces, and sets the number per task rather than per project, then loads the DAGs under the real Airflow release in its own sandbox to show they parse.

Every command and its output stay with the run, so the pull request can point at the `DagBag` load and the per-operator assertions, and a reviewer who does not have the providers installed locally can still check the claim against the record.

## Frequently asked questions

### Why do the sensors get no retries?

A sensor already waits by poking, and in `reschedule` mode it frees its worker slot between pokes. A retry on a sensor restarts the wait after an exception, which usually means a broken connection that a retry will not fix and that should fail loudly. The `S3KeySensor` in the Databricks DAG keeps its timeout as the limit instead.

### How long can a task wait between attempts?

The delay starts at `retry_delay`, doubles on each attempt with jitter, and never exceeds `max_retry_delay`, which the agent sets to ten minutes in every DAG. With three retries the longest a task spends retrying is bounded, and the test asserts the cap is present on every task.

### Does this change when the DAGs run?

No. Schedules, `catchup`, `max_active_runs`, task ids and dependencies are untouched, and `.astro-registry.yaml` still lists the same five files. Only the argument dictionaries, the operator keyword arguments and the new callback module change.

### Can the Databricks run get more than one retry?

Only once the notebook is idempotent, which the agent cannot see from the DAG. `DatabricksRunNowOperator` starts a job run on the workspace, and a second run of a notebook that appends to a table appends twice. The comment beside `retries` 1 says what to check before raising it.

## More Airflow jobs

- [Move Airflow Connections to a Secrets Backend](https://getrush.ai/run/add-secrets-backend/airflow/)
- [Add Slack Alerts on Task Failure to Airflow DAGs](https://getrush.ai/run/add-slack-failure-alerts/airflow/)
- [Group Generated Tasks into TaskGroups in an Airflow DAG](https://getrush.ai/run/add-task-groups/airflow/)

## The same job on other stacks

- [Add Retry Policies to the Flaky Assets in a Dagster Project](https://getrush.ai/run/add-retry-policies/dagster/)
- [Turn On Safe Retries for Event-Driven Firebase Cloud Functions](https://getrush.ai/run/add-retry-policies/firebase-functions/)
- [Add Retry Policies to Oban Workers in an Elixir App](https://getrush.ai/run/add-retry-policies/oban/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
