All jobs

Add Timeouts to Airflow DAGs

Yes. An agent reads each DAG's schedule, sensors and retries, sets a dagrun_timeout per DAG and an execution_timeout per task, tests DAG loading and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads every DAG file and the project around it before choosing a single value. In astronomer/airflow-example-dags, the Dockerfile starts from quay.io/astronomer/astro-runtime:6.0.0, packages.txt adds postgresql, requirements.txt is empty, and .astro-registry.yaml lists five DAG files under dags/. databricks_sample.py defines adb_pipeline on 30 4 * * * with catchup=False: a DummyOperator, then an S3KeySensor with poke_interval 45, timeout 600 and wildcard_match, then four pairs of DatabricksRunNowOperator and PostgresOperator, with email_on_failure on in default_args. kube_pod.py defines example_kubernetes_pod on @once with a KubernetesPodOperator requesting 200m CPU and 1Gi memory, retries 1 and retry_delay of 5 minutes. s3_upload_dag.py, s3_upload_copy_dag.py and s3_upload_copy_delete.py define s3_upload, s3_upload_copy and s3_upload_copy_delete on 0 12 8-14,22-28 * 6 with max_active_runs=1 and catchup=False, generating 5, 10 and 10 upload tasks with copy and delete steps behind them, each with retries 1 and retry_delay 5 minutes; s3_upload_copy_dag.py wires an empty on_failure_function as on_failure_callback. example_external_task_sensor.py is entirely commented out. No DAG sets dagrun_timeout, no task sets execution_timeout, the imports use the deprecated airflow.operators.dummy_operator, python_operator and postgres_operator paths, and the DAGs use schedule_interval. There is no tests/ directory and .drone.yml only builds and pushes the image.

Two kinds of timeout go in, and the agent keeps them apart in the pull request. dagrun_timeout is a timedelta on each DAG and bounds a whole run; the agent derives it from the schedule gap, so the daily adb_pipeline gets hours rather than minutes and the twice-monthly S3 DAGs, whose max_active_runs=1 means a hung run would block the next scheduled one, get a bound comfortably inside their gap. It also notes the rule that dagrun_timeout only acts on scheduled runs and only once the number of active runs has reached max_active_runs, which is exactly the situation these three DAGs are in. execution_timeout goes into each file's default_args so every task inherits it, with explicit overrides where a task is allowed to take longer: the S3KeySensor, whose own timeout of 600 bounds how long it pokes, needs an execution_timeout above that or the task is killed before the sensor gives up, and the agent also switches it to mode='reschedule' so a worker slot is not held for the whole wait; the Databricks runs get their own larger value; the KubernetesPodOperator, which streams logs with get_logs=True, gets one that lets Airflow kill the pod when it expires. Every value is written as a named timedelta at the top of the file with a comment saying where it came from, and the description lists them in a table.

A timeout only helps if someone hears about it, so the agent checks what happens when one fires. execution_timeout raises AirflowTaskTimeout, which counts as a failure, so the existing retries 1 applies and email_on_failure in adb_pipeline sends mail; the empty on_failure_function in s3_upload_copy_dag.py gets a log line naming the task and its duration so the callback does something. The agent does not add SLAs, because the Airflow version in this runtime handles them separately and the project has no SLA callback.

To prove the DAGs still load, the agent adds tests/dags/test_dag_integrity.py: it builds a DagBag from dags/, asserts import_errors is empty, and asserts that every DAG has a dagrun_timeout and every task an execution_timeout, with the sensor's value above its poke timeout. Running it needs Airflow and the providers the DAGs import (amazon, databricks, cncf.kubernetes, postgres). The sandbox has no Docker, so rather than astro dev pytest the agent installs the Airflow version that astro-runtime:6.0.0 ships, with its constraints file and those providers, into a virtual environment and runs pytest tests/; the pull request says which path it took so a reader with the Astro CLI can run astro dev pytest instead.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add timeouts to every DAG in this Airflow project. Read the Dockerfile, requirements.txt, packages.txt, .astro-registry.yaml and every file under dags first, and list each DAG's schedule, catchup, max_active_runs, retries, sensors and callbacks. Set a dagrun_timeout on each DAG derived from its schedule gap, and an execution_timeout in each file's default_args with explicit larger values for sensors, Databricks runs and Kubernetes pods, keeping the sensor's execution timeout above its own poke timeout and switching it to reschedule mode. Define every value as a named timedelta with a comment on where it came from. Make the empty failure callback log the task and its duration. Do not add SLAs. Add a DagBag integrity test that asserts no import errors, a dagrun_timeout on every DAG and an execution_timeout on every task, install the Airflow version the runtime image ships with its constraints and the providers the DAGs import, run the test, and open a pull request with a table of every DAG, task group and timeout chosen.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that edits the five DAG files and adds tests/dags/test_dag_integrity.py. The description has one table row per DAG with its schedule, the dagrun_timeout chosen and the reasoning, and one per task kind with its execution_timeout, plus a note on the sensor's switch to reschedule mode and what the failure callback now logs.

The run's record keeps the pip install of Airflow and the four providers, and the pytest output showing the DagBag loaded all five DAGs with no import errors and every timeout assertion passed.

Before merging, a reviewer compares the chosen values with the real run durations in their Airflow metadata or Astro deployment, since the agent could only reason from schedules and sensor settings, adjusts the two or three that matter, and lets the integrity test keep the rest honest from then on.

Run it on every Airflow project without timeouts

A platform team with several Astro projects can start this job for each one from a short script: list the repositories the GitHub App reaches with GET /o/{org}/github, keep those whose default branch has a dags/ directory, and send this body once per repository as the resources URL. Give each request a request_id that is a UUID v5 of the repository's full name, so a rerun of the script returns the runs it already started.

A team that prefers a standing check adds an automation with a schedule trigger whose cron fires on the first day of each month and a prompt that first asks whether any DAG or task lacks a timeout and opens a pull request only when one does.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

Choosing timeouts is reading work: every DAG's schedule, every sensor's own timeout, every retry setting, and the one rule about dagrun_timeout and max_active_runs that decides whether the setting does anything at all. The agent does that reading across every file, writes the value next to the reason, and proves the DAGs still parse with a test that installs the real Airflow version rather than guessing from the source.

Started from a terminal and left to run, it comes back as a pull request with a table a data engineer can correct in two minutes from their run history, and an integrity test that fails the next time someone adds a DAG without a timeout.

Frequently asked questions

When does dagrun_timeout actually fire?

Only for scheduled runs, and only once the DAG's active runs have reached max_active_runs. The three S3 DAGs set max_active_runs=1, so a hung run is exactly the case it covers: the next scheduled run cannot start, the timeout marks the stuck one failed, and the schedule continues. Manually triggered runs are not bounded by it, which the pull request notes.

Why does the sensor get a larger execution_timeout than its own timeout?

The S3KeySensor's timeout of 600 seconds is how long it keeps poking before failing on its own terms. If execution_timeout were shorter, Airflow would kill the task first with a generic timeout error and the sensor's reason would be lost. Setting it above 600 lets the sensor fail first, and mode='reschedule' means the wait no longer holds a worker slot.

Why no SLAs?

An SLA in this Airflow version is a separate alerting mechanism that needs an sla_miss_callback and does not stop a task, and the project has neither. Timeouts stop runaway work; SLAs report lateness. The prompt asks for the first and the pull request says the second is a different change if the team wants it.

How are the tests run without Docker?

The sandbox has no Docker, so astro dev pytest is not available there. The agent installs the Airflow release that astro-runtime:6.0.0 ships, with its constraints file and the amazon, databricks, cncf.kubernetes and postgres providers, into a virtual environment and runs pytest tests/ there; the pull request says so and the same test runs under astro dev pytest for anyone with the CLI.

More Airflow jobs

More jobs like this: Hand off a long task

Keep reading