All jobs

Add Deferrable Operators to an Airflow Project

Yes. An agent finds the sensors and long-polling operators in your DAGs, switches them to deferrable mode, confirms the DAGs still parse and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent clones the repository into its own cloud sandbox and reads how the project is built before it touches a DAG. In an Astronomer project like astronomer/airflow-example-dags, the whole Dockerfile is one line, FROM quay.io/astronomer/astro-runtime:6.0.0, requirements.txt is empty, packages.txt adds only postgresql, and .astro-registry.yaml lists the five DAGs the project publishes. There is no test directory and no GitHub workflow; the only pipeline is a .drone.yml that builds the image and pushes it to a registry. So the Runtime image decides which provider versions the DAGs run against, and anything the agent adds to requirements.txt layers on top of it.

It then reads each DAG for a task that holds a worker slot while it waits. dags/databricks_sample.py is the one that matters: its adb_pipeline DAG starts with an S3KeySensor on bucket_key='globetelecom/copy_*' with wildcard_match=True, poke_interval=45 and timeout=600, and then fans out into four DatabricksRunNowOperator tasks, each followed by a PostgresOperator. With the default mode that sensor occupies a worker for up to ten minutes doing nothing but polling, and each Databricks task occupies one for as long as the notebook runs. The other DAGs have nothing to defer: s3_upload_dag.py, s3_upload_copy_dag.py and s3_upload_copy_delete.py do their work in PythonOperator callables against S3Hook and the S3 copy and delete operators, kube_pod.py runs a single KubernetesPodOperator, and example_external_task_sensor.py is commented out from top to bottom.

The change is small and deliberate: deferrable=True on the S3KeySensor, keeping timeout=600 so the trigger gives up at the same point the poking sensor did, and deferrable=True on each DatabricksRunNowOperator so the worker is released while the triggerer watches the run state through the adb_workspace connection that default_args already names. The poke_interval stays as the polling cadence for the trigger. Before writing any of that, the agent checks that the Amazon and Databricks providers in the pinned Runtime image accept the argument; when they predate it, it pins a provider release that has it in requirements.txt rather than moving the project to a newer Runtime, and says which in the pull request.

To prove the DAGs still load, it installs Airflow and the two providers in a virtualenv in the sandbox, points AIRFLOW_HOME at a scratch directory and runs a DagBag import of the dags/ folder, which fails on any import or argument error. It also runs python -m py_compile over the edited files and leaves the deprecated airflow.operators.dummy_operator and python_operator import paths alone, noting them as a separate cleanup so the diff stays about one thing. The pull request links the Linear issue, lists each task that changed mode and states the one operational requirement: a triggerer process must be running in the deployment, or deferred tasks never resume.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Convert the sensors and long-running operators in this Airflow project to deferrable mode. Read the Dockerfile, requirements.txt and every file under dags/ first. For each S3KeySensor, ExternalTaskSensor, DatabricksRunNowOperator or similar task that polls, add deferrable=True and keep its existing timeout and poke_interval. Check that the provider versions in the pinned Runtime image support the argument; if not, pin a provider release in requirements.txt that does, and say so. Do not change DAG ids, schedules or task ids. Verify the DAGs still parse with a DagBag import in a virtualenv and paste the output into the pull request. Open a pull request that lists each task you changed and states that a triggerer must be running for deferred tasks to resume.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that edits dags/databricks_sample.py and, when the pinned Runtime needs it, adds one or two provider pins to requirements.txt. The description lists the five tasks that moved to deferrable mode, the sensor and the four Databricks runs, the timeout and poke_interval each kept, and the DagBag import output showing no import errors across the five DAG files.

The run's record keeps the virtualenv install, the provider version check and the import, so a reviewer can see which provider release the agent tested against and that nothing else in dags/ changed. The three S3 DAGs and the Kubernetes pod DAG are untouched, and the diff says why.

Before merging, a reviewer confirms the deployment runs a triggerer, decides whether to accept a provider pin on top of the Runtime image or to bump the image instead, and checks that the adb_workspace and astro-s3-workshop connections exist in the environment the deferred tasks will poll from.

Run it on every Linear issue labelled airflow

Teams that add DAGs faster than they review them label the work instead of doing it by hand. An automation with a linear trigger on linear.issue.labeled and the label airflow starts one run per labelled issue, with the issue's title and description after the prompt above. An issue that says which DAG file and which tasks to defer gives the agent the scope; one that only says to defer everything gets the inventory step first and a pull request that explains what it left alone.

Each run opens one pull request on its own branch that links back to the issue. A run whose DagBag import fails ends needs_review rather than opening a broken pull request, and the trajectory shows the import error so the fix is a follow-up comment on the issue, not a fresh investigation.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

Deferrable operators are a one-line change per task with a long tail of questions: does the pinned provider support the argument, does the DAG still import, does the deployment have a triggerer. A run on Rush answers the first two with real commands in its own sandbox, a virtualenv with the exact provider versions and a DagBag load, and writes the third into the pull request where the person who owns the deployment will read it. Nothing is installed on anyone's laptop and the sandbox is discarded when the run ends.

An organisation with many Astronomer projects can send the same request to each repository and get one pull request per project, each built from that project's own Dockerfile and dags/, with its own import check in the record. The projects that have nothing to defer come back as a short pull request that says so, which is still worth knowing.

Frequently asked questions

What does the agent do if the Runtime image's providers are too old for deferrable=True?

It pins a provider release that has the argument in requirements.txt, which the Astronomer build installs on top of the image, and says in the pull request which version and why. It does not bump the Runtime image on its own, because that changes the Airflow version for every DAG in the project.

Will the sensor's timeout behave the same?

Yes. The agent keeps timeout=600 on the S3KeySensor, so a deferred wait still fails after the same ten minutes; the difference is that the wait happens in the triggerer instead of on a worker slot. The poke_interval becomes the polling cadence of the trigger.

Does it touch the S3 upload DAGs?

No. Their work runs inside PythonOperator callables against S3Hook, and the copy and delete operators finish in one call, so there is no wait to move. The pull request names them as reviewed and unchanged.

Do I need a triggerer?

Yes. A deferred task hands its wait to the triggerer and is only resumed by it. The pull request states this as a requirement; the agent does not change your deployment configuration, so confirming the triggerer runs in each environment is the reviewer's step.

More Airflow jobs

More jobs like this: Turn a Linear label into a PR

Keep reading