What the agent does
The agent reads every DAG before changing a default. In a project like astronomer/airflow-example-dags, the Dockerfile builds on quay.io/astronomer/astro-runtime:6.0.0, packages.txt adds postgresql, requirements.txt is empty, and .astro-registry.yaml publishes five DAGs. dags/s3_upload_dag.py and dags/s3_upload_copy_dag.py share a default_args block with retries 1 and retry_delay of five minutes, max_active_runs 1, catchup off and the schedule 0 12 8-14,22-28 * 6; the first loops five PythonOperator tasks that write a local file and call S3Hook.load_file with replace=True on the astro-s3-workshop connection, the second loops ten uploads each followed by an S3CopyObjectOperator, with a stub on_failure_callback. dags/kube_pod.py runs one KubernetesPodOperator with the hello-world image under the same single retry. dags/databricks_sample.py has no retries at all in its args, only email_on_failure, and chains an S3KeySensor into a DatabricksRunNowOperator. dags/example_external_task_sensor.py is commented out end to end. So a transient S3 error gets exactly one more try five minutes later, a Databricks job gets none, and a sensor and a notebook run are treated the same as an idempotent upload.
The agent sets a policy per DAG and then overrides it where a retry is not free. In the two S3 DAGs the default_args become retries 3, retry_delay of thirty seconds, retry_exponential_backoff on and max_retry_delay of ten minutes, so the gap doubles with jitter from thirty seconds and stops growing at ten minutes; the uploads and the copy are safe to repeat because load_file replaces and S3CopyObjectOperator overwrites the same key. kube_pod.py gets retries 2 with the same backoff, since a hello-world pod is harmless to rerun and the operator names each pod uniquely. In databricks_sample.py the sensor gets mode='reschedule' and retries 0, because poking already handles waiting and a retry on a sensor only hides a broken connection, while DatabricksRunNowOperator gets retries 1 with a note that a notebook that writes output must be idempotent before that number goes higher. The on_failure_function stub stays; beside it the agent adds an on_retry_callback in include/callbacks.py that logs the task id, try_number and max_tries from the context, so a retry is visible in the task log rather than only in the grid.
Each DAG keeps its schedule, its task ids and its dependencies, because the ids are what the registry listing, the grid view and any downstream ExternalTaskSensor refer to. The change is confined to argument dictionaries and operator keyword arguments, and the agent leaves the deprecated dummy_operator and python_operator import paths alone, since replacing them is a different job and would widen the diff.
The test is a DagBag the project did not have. tests/dags/test_retry_policies.py loads dags/ with include_examples=False, asserts there are no import errors, and for each DAG checks that every task has retry_exponential_backoff set, a max_retry_delay at or under ten minutes, and the expected retries by operator type: at least 2 for the S3 operators and the pod, 0 for the sensor and 1 for the Databricks run. The agent reads the runtime image tag in the Dockerfile, installs the Airflow release that image carries with its constraints file and the amazon, cncf.kubernetes and databricks providers the DAGs import, and runs pytest tests/dags in the sandbox the way astro dev pytest would in the project's own tooling.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add retry policies to the DAGs in this Airflow project. Read the Dockerfile, requirements.txt, packages.txt, .astro-registry.yaml, airflow_settings.yaml and every file under dags/ first. In each DAG's default_args set retries, a short retry_delay, retry_exponential_backoff and a max_retry_delay of ten minutes, then override per task: sensors get reschedule mode and no retries, operators that run a notebook or job get one retry with a comment about idempotency, operators that upload or copy to S3 with replace get three, and a pod operator gets two. Add an on_retry_callback in include/callbacks.py that logs the task id, try number and max tries, and wire it beside the existing failure callback. Keep every schedule, task id and dependency unchanged and leave the deprecated import paths alone. Write tests/dags/test_retry_policies.py that loads the DagBag without import errors and asserts the backoff, cap and per-operator retry counts. Install the Airflow release the runtime image carries with its constraints and the amazon, cncf.kubernetes and databricks providers, run pytest tests/dags, and open a pull request with a table of DAG, task type and retry count.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that edits the four live DAG files, adds include/callbacks.py and tests/dags/test_retry_policies.py, and leaves the commented-out sensor example as it is. The description has one table with each DAG, each task type, its retries, the backoff and the cap, and a sentence per override explaining why a sensor gets none and a notebook run gets one.
The run's record keeps the Airflow and provider installation with the constraints file it used, the pytest tests/dags output with each assertion named, and a DagBag load showing zero import errors, so a reviewer can see the DAGs still parse under the runtime's Airflow release rather than only under the agent's reading of them.
Before merging, a reviewer deploys to a development deployment, breaks the astro-s3-workshop connection on purpose, triggers s3_upload, and watches the grid: three retries with growing gaps, the on_retry_callback lines in the task log, and a failure after the fourth attempt. Restoring the connection and clearing the task shows the retry succeeding without a duplicate upload.
Run it on every DAG whose first failure pages someone
The setup is one POST /o/{org}/sessions with this body and the repository in resources, started from a terminal; the run continues in its sandbox after you close the window, and a follow-up through POST /o/{org}/sessions/{id}/messages can raise the cap or add a Slack call to the retry callback without starting over. A team with several Astro projects sends one request per repository, each with a fresh UUID as request_id, so a retried script returns the run it already started.
Set a webhook_url on the API key and Rush posts run.completed or run.failed when a run's turn finishes, with session_url so your server can read the pull request link and the test output.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
A retry policy is a dictionary, and the harm is in applying one dictionary to every task: a sensor that retries hides a dead connection for an hour, a notebook that retries writes its output twice, and a cap that is missing turns exponential backoff into a task that sleeps until tomorrow. The agent reads what each operator does, including whether the S3 hook replaces, and sets the number per task rather than per project, then loads the DAGs under the real Airflow release in its own sandbox to show they parse.
Every command and its output stay with the run, so the pull request can point at the DagBag load and the per-operator assertions, and a reviewer who does not have the providers installed locally can still check the claim against the record.
Frequently asked questions
Why do the sensors get no retries?
A sensor already waits by poking, and in reschedule mode it frees its worker slot between pokes. A retry on a sensor restarts the wait after an exception, which usually means a broken connection that a retry will not fix and that should fail loudly. The S3KeySensor in the Databricks DAG keeps its timeout as the limit instead.
How long can a task wait between attempts?
The delay starts at retry_delay, doubles on each attempt with jitter, and never exceeds max_retry_delay, which the agent sets to ten minutes in every DAG. With three retries the longest a task spends retrying is bounded, and the test asserts the cap is present on every task.
Does this change when the DAGs run?
No. Schedules, catchup, max_active_runs, task ids and dependencies are untouched, and .astro-registry.yaml still lists the same five files. Only the argument dictionaries, the operator keyword arguments and the new callback module change.
Can the Databricks run get more than one retry?
Only once the notebook is idempotent, which the agent cannot see from the DAG. DatabricksRunNowOperator starts a job run on the workspace, and a second run of a notebook that appends to a table appends twice. The comment beside retries 1 says what to check before raising it.
More Airflow jobs
The same job on other stacks
- Add Retry Policies to the Flaky Assets in a Dagster Project (Dagster)
- Turn On Safe Retries for Event-Driven Firebase Cloud Functions (Firebase Cloud Functions)
- Add Retry Policies to Oban Workers in an Elixir App (Oban)