# Add Timeouts to Airflow DAGs

Yes. An agent reads each DAG's schedule, sensors and retries, sets a dagrun_timeout per DAG and an execution_timeout per task, tests DAG loading and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-dag-timeouts/airflow/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads every DAG file and the project around it before choosing a single value. In astronomer/airflow-example-dags, the `Dockerfile` starts from `quay.io/astronomer/astro-runtime:6.0.0`, `packages.txt` adds `postgresql`, `requirements.txt` is empty, and `.astro-registry.yaml` lists five DAG files under `dags/`. `databricks_sample.py` defines `adb_pipeline` on `30 4 * * *` with `catchup=False`: a `DummyOperator`, then an `S3KeySensor` with `poke_interval` 45, `timeout` 600 and `wildcard_match`, then four pairs of `DatabricksRunNowOperator` and `PostgresOperator`, with `email_on_failure` on in `default_args`. `kube_pod.py` defines `example_kubernetes_pod` on `@once` with a `KubernetesPodOperator` requesting 200m CPU and 1Gi memory, `retries` 1 and `retry_delay` of 5 minutes. `s3_upload_dag.py`, `s3_upload_copy_dag.py` and `s3_upload_copy_delete.py` define `s3_upload`, `s3_upload_copy` and `s3_upload_copy_delete` on `0 12 8-14,22-28 * 6` with `max_active_runs=1` and `catchup=False`, generating 5, 10 and 10 upload tasks with copy and delete steps behind them, each with `retries` 1 and `retry_delay` 5 minutes; `s3_upload_copy_dag.py` wires an empty `on_failure_function` as `on_failure_callback`. `example_external_task_sensor.py` is entirely commented out. No DAG sets `dagrun_timeout`, no task sets `execution_timeout`, the imports use the deprecated `airflow.operators.dummy_operator`, `python_operator` and `postgres_operator` paths, and the DAGs use `schedule_interval`. There is no `tests/` directory and `.drone.yml` only builds and pushes the image.

Two kinds of timeout go in, and the agent keeps them apart in the pull request. `dagrun_timeout` is a `timedelta` on each `DAG` and bounds a whole run; the agent derives it from the schedule gap, so the daily `adb_pipeline` gets hours rather than minutes and the twice-monthly S3 DAGs, whose `max_active_runs=1` means a hung run would block the next scheduled one, get a bound comfortably inside their gap. It also notes the rule that `dagrun_timeout` only acts on scheduled runs and only once the number of active runs has reached `max_active_runs`, which is exactly the situation these three DAGs are in. `execution_timeout` goes into each file's `default_args` so every task inherits it, with explicit overrides where a task is allowed to take longer: the `S3KeySensor`, whose own `timeout` of 600 bounds how long it pokes, needs an `execution_timeout` above that or the task is killed before the sensor gives up, and the agent also switches it to `mode='reschedule'` so a worker slot is not held for the whole wait; the Databricks runs get their own larger value; the `KubernetesPodOperator`, which streams logs with `get_logs=True`, gets one that lets Airflow kill the pod when it expires. Every value is written as a named `timedelta` at the top of the file with a comment saying where it came from, and the description lists them in a table.

A timeout only helps if someone hears about it, so the agent checks what happens when one fires. `execution_timeout` raises `AirflowTaskTimeout`, which counts as a failure, so the existing `retries` 1 applies and `email_on_failure` in `adb_pipeline` sends mail; the empty `on_failure_function` in `s3_upload_copy_dag.py` gets a log line naming the task and its duration so the callback does something. The agent does not add SLAs, because the Airflow version in this runtime handles them separately and the project has no SLA callback.

To prove the DAGs still load, the agent adds `tests/dags/test_dag_integrity.py`: it builds a `DagBag` from `dags/`, asserts `import_errors` is empty, and asserts that every DAG has a `dagrun_timeout` and every task an `execution_timeout`, with the sensor's value above its poke `timeout`. Running it needs Airflow and the providers the DAGs import (`amazon`, `databricks`, `cncf.kubernetes`, `postgres`). The sandbox has no Docker, so rather than `astro dev pytest` the agent installs the Airflow version that `astro-runtime:6.0.0` ships, with its constraints file and those providers, into a virtual environment and runs `pytest tests/`; the pull request says which path it took so a reader with the Astro CLI can run `astro dev pytest` instead.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add timeouts to every DAG in this Airflow project. Read the Dockerfile, requirements.txt, packages.txt, .astro-registry.yaml and every file under dags first, and list each DAG's schedule, catchup, max_active_runs, retries, sensors and callbacks. Set a dagrun_timeout on each DAG derived from its schedule gap, and an execution_timeout in each file's default_args with explicit larger values for sensors, Databricks runs and Kubernetes pods, keeping the sensor's execution timeout above its own poke timeout and switching it to reschedule mode. Define every value as a named timedelta with a comment on where it came from. Make the empty failure callback log the task and its duration. Do not add SLAs. Add a DagBag integrity test that asserts no import errors, a dagrun_timeout on every DAG and an execution_timeout on every task, install the Airflow version the runtime image ships with its constraints and the providers the DAGs import, run the test, and open a pull request with a table of every DAG, task group and timeout chosen.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that edits the five DAG files and adds `tests/dags/test_dag_integrity.py`. The description has one table row per DAG with its schedule, the `dagrun_timeout` chosen and the reasoning, and one per task kind with its `execution_timeout`, plus a note on the sensor's switch to `reschedule` mode and what the failure callback now logs.

The run's record keeps the `pip install` of Airflow and the four providers, and the `pytest` output showing the DagBag loaded all five DAGs with no import errors and every timeout assertion passed.

Before merging, a reviewer compares the chosen values with the real run durations in their Airflow metadata or Astro deployment, since the agent could only reason from schedules and sensor settings, adjusts the two or three that matter, and lets the integrity test keep the rest honest from then on.

## Run it on every Airflow project without timeouts

A platform team with several Astro projects can start this job for each one from a short script: list the repositories the GitHub App reaches with `GET /o/{org}/github`, keep those whose default branch has a `dags/` directory, and send this body once per repository as the `resources` URL. Give each request a `request_id` that is a UUID v5 of the repository's full name, so a rerun of the script returns the runs it already started.

A team that prefers a standing check adds an automation with a `schedule` trigger whose `cron` fires on the first day of each month and a prompt that first asks whether any DAG or task lacks a timeout and opens a pull request only when one does.

## Why run it on Rush?

Choosing timeouts is reading work: every DAG's schedule, every sensor's own timeout, every retry setting, and the one rule about `dagrun_timeout` and `max_active_runs` that decides whether the setting does anything at all. The agent does that reading across every file, writes the value next to the reason, and proves the DAGs still parse with a test that installs the real Airflow version rather than guessing from the source.

Started from a terminal and left to run, it comes back as a pull request with a table a data engineer can correct in two minutes from their run history, and an integrity test that fails the next time someone adds a DAG without a timeout.

## Frequently asked questions

### When does dagrun_timeout actually fire?

Only for scheduled runs, and only once the DAG's active runs have reached `max_active_runs`. The three S3 DAGs set `max_active_runs=1`, so a hung run is exactly the case it covers: the next scheduled run cannot start, the timeout marks the stuck one failed, and the schedule continues. Manually triggered runs are not bounded by it, which the pull request notes.

### Why does the sensor get a larger execution_timeout than its own timeout?

The `S3KeySensor`'s `timeout` of 600 seconds is how long it keeps poking before failing on its own terms. If `execution_timeout` were shorter, Airflow would kill the task first with a generic timeout error and the sensor's reason would be lost. Setting it above 600 lets the sensor fail first, and `mode='reschedule'` means the wait no longer holds a worker slot.

### Why no SLAs?

An SLA in this Airflow version is a separate alerting mechanism that needs an `sla_miss_callback` and does not stop a task, and the project has neither. Timeouts stop runaway work; SLAs report lateness. The prompt asks for the first and the pull request says the second is a different change if the team wants it.

### How are the tests run without Docker?

The sandbox has no Docker, so `astro dev pytest` is not available there. The agent installs the Airflow release that `astro-runtime:6.0.0` ships, with its constraints file and the `amazon`, `databricks`, `cncf.kubernetes` and `postgres` providers, into a virtual environment and runs `pytest tests/` there; the pull request says so and the same test runs under `astro dev pytest` for anyone with the CLI.

## More Airflow jobs

- [Add Data-Aware Scheduling to Airflow DAGs](https://getrush.ai/run/add-data-aware-scheduling/airflow/)
- [Add Deferrable Operators to an Airflow Project](https://getrush.ai/run/add-deferrable-operators/airflow/)
- [Convert Airflow DAG Loops to Dynamic Task Mapping](https://getrush.ai/run/add-dynamic-task-mapping/airflow/)

## More jobs like this: Hand off a long task

- [Add Dark Mode to a Blazor Web App](https://getrush.ai/run/add-dark-mode/blazor/)
- [Add a Dark Mode Toggle to an Eleventy Site](https://getrush.ai/run/add-dark-mode/eleventy/)
- [Add Dark Mode to a Jekyll Site](https://getrush.ai/run/add-dark-mode/jekyll/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
