# Add Deferrable Operators to an Airflow Project

Yes. An agent finds the sensors and long-polling operators in your DAGs, switches them to deferrable mode, confirms the DAGs still parse and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-deferrable-operators/airflow/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent clones the repository into its own cloud sandbox and reads how the project is built before it touches a DAG. In an Astronomer project like astronomer/airflow-example-dags, the whole `Dockerfile` is one line, `FROM quay.io/astronomer/astro-runtime:6.0.0`, `requirements.txt` is empty, `packages.txt` adds only `postgresql`, and `.astro-registry.yaml` lists the five DAGs the project publishes. There is no test directory and no GitHub workflow; the only pipeline is a `.drone.yml` that builds the image and pushes it to a registry. So the Runtime image decides which provider versions the DAGs run against, and anything the agent adds to `requirements.txt` layers on top of it.

It then reads each DAG for a task that holds a worker slot while it waits. `dags/databricks_sample.py` is the one that matters: its `adb_pipeline` DAG starts with an `S3KeySensor` on `bucket_key='globetelecom/copy_*'` with `wildcard_match=True`, `poke_interval=45` and `timeout=600`, and then fans out into four `DatabricksRunNowOperator` tasks, each followed by a `PostgresOperator`. With the default mode that sensor occupies a worker for up to ten minutes doing nothing but polling, and each Databricks task occupies one for as long as the notebook runs. The other DAGs have nothing to defer: `s3_upload_dag.py`, `s3_upload_copy_dag.py` and `s3_upload_copy_delete.py` do their work in `PythonOperator` callables against `S3Hook` and the S3 copy and delete operators, `kube_pod.py` runs a single `KubernetesPodOperator`, and `example_external_task_sensor.py` is commented out from top to bottom.

The change is small and deliberate: `deferrable=True` on the `S3KeySensor`, keeping `timeout=600` so the trigger gives up at the same point the poking sensor did, and `deferrable=True` on each `DatabricksRunNowOperator` so the worker is released while the triggerer watches the run state through the `adb_workspace` connection that `default_args` already names. The `poke_interval` stays as the polling cadence for the trigger. Before writing any of that, the agent checks that the Amazon and Databricks providers in the pinned Runtime image accept the argument; when they predate it, it pins a provider release that has it in `requirements.txt` rather than moving the project to a newer Runtime, and says which in the pull request.

To prove the DAGs still load, it installs Airflow and the two providers in a virtualenv in the sandbox, points `AIRFLOW_HOME` at a scratch directory and runs a `DagBag` import of the `dags/` folder, which fails on any import or argument error. It also runs `python -m py_compile` over the edited files and leaves the deprecated `airflow.operators.dummy_operator` and `python_operator` import paths alone, noting them as a separate cleanup so the diff stays about one thing. The pull request links the Linear issue, lists each task that changed mode and states the one operational requirement: a triggerer process must be running in the deployment, or deferred tasks never resume.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Convert the sensors and long-running operators in this Airflow project to deferrable mode. Read the Dockerfile, requirements.txt and every file under dags/ first. For each S3KeySensor, ExternalTaskSensor, DatabricksRunNowOperator or similar task that polls, add deferrable=True and keep its existing timeout and poke_interval. Check that the provider versions in the pinned Runtime image support the argument; if not, pin a provider release in requirements.txt that does, and say so. Do not change DAG ids, schedules or task ids. Verify the DAGs still parse with a DagBag import in a virtualenv and paste the output into the pull request. Open a pull request that lists each task you changed and states that a triggerer must be running for deferred tasks to resume.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that edits `dags/databricks_sample.py` and, when the pinned Runtime needs it, adds one or two provider pins to `requirements.txt`. The description lists the five tasks that moved to deferrable mode, the sensor and the four Databricks runs, the `timeout` and `poke_interval` each kept, and the `DagBag` import output showing no import errors across the five DAG files.

The run's record keeps the virtualenv install, the provider version check and the import, so a reviewer can see which provider release the agent tested against and that nothing else in `dags/` changed. The three S3 DAGs and the Kubernetes pod DAG are untouched, and the diff says why.

Before merging, a reviewer confirms the deployment runs a triggerer, decides whether to accept a provider pin on top of the Runtime image or to bump the image instead, and checks that the `adb_workspace` and `astro-s3-workshop` connections exist in the environment the deferred tasks will poll from.

## Run it on every Linear issue labelled airflow

Teams that add DAGs faster than they review them label the work instead of doing it by hand. An automation with a `linear` trigger on `linear.issue.labeled` and the label `airflow` starts one run per labelled issue, with the issue's title and description after the prompt above. An issue that says which DAG file and which tasks to defer gives the agent the scope; one that only says to defer everything gets the inventory step first and a pull request that explains what it left alone.

Each run opens one pull request on its own branch that links back to the issue. A run whose `DagBag` import fails ends `needs_review` rather than opening a broken pull request, and the trajectory shows the import error so the fix is a follow-up comment on the issue, not a fresh investigation.

## Why run it on Rush?

Deferrable operators are a one-line change per task with a long tail of questions: does the pinned provider support the argument, does the DAG still import, does the deployment have a triggerer. A run on Rush answers the first two with real commands in its own sandbox, a virtualenv with the exact provider versions and a `DagBag` load, and writes the third into the pull request where the person who owns the deployment will read it. Nothing is installed on anyone's laptop and the sandbox is discarded when the run ends.

An organisation with many Astronomer projects can send the same request to each repository and get one pull request per project, each built from that project's own `Dockerfile` and `dags/`, with its own import check in the record. The projects that have nothing to defer come back as a short pull request that says so, which is still worth knowing.

## Frequently asked questions

### What does the agent do if the Runtime image's providers are too old for deferrable=True?

It pins a provider release that has the argument in `requirements.txt`, which the Astronomer build installs on top of the image, and says in the pull request which version and why. It does not bump the Runtime image on its own, because that changes the Airflow version for every DAG in the project.

### Will the sensor's timeout behave the same?

Yes. The agent keeps `timeout=600` on the `S3KeySensor`, so a deferred wait still fails after the same ten minutes; the difference is that the wait happens in the triggerer instead of on a worker slot. The `poke_interval` becomes the polling cadence of the trigger.

### Does it touch the S3 upload DAGs?

No. Their work runs inside `PythonOperator` callables against `S3Hook`, and the copy and delete operators finish in one call, so there is no wait to move. The pull request names them as reviewed and unchanged.

### Do I need a triggerer?

Yes. A deferred task hands its wait to the triggerer and is only resumed by it. The pull request states this as a requirement; the agent does not change your deployment configuration, so confirming the triggerer runs in each environment is the reviewer's step.

## More Airflow jobs

- [Convert Airflow DAG Loops to Dynamic Task Mapping](https://getrush.ai/run/add-dynamic-task-mapping/airflow/)
- [Add GitHub Actions CI to an Airflow Project](https://getrush.ai/run/add-github-actions-ci/airflow/)
- [Add Pools to Limit Concurrent Tasks in Airflow](https://getrush.ai/run/add-pools/airflow/)

## More jobs like this: Turn a Linear label into a PR

- [Add a Dev Container to a .NET Solution](https://getrush.ai/run/add-dev-containers/dotnet/)
- [Add Dialyzer to an Elixir App from a Linear Issue](https://getrush.ai/run/add-dialyzer/elixir/)
- [Add a Discount Function to a Shopify App](https://getrush.ai/run/add-discount-function/shopify-app/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
