# Add Data-Aware Scheduling to Airflow DAGs

Yes. An agent declares Datasets for the S3 keys your DAGs write, adds outlets on the producers, schedules consumers on them instead of a shared cron, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-data-aware-scheduling/airflow/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent clones the Astro project into its own cloud sandbox and reads what runs where. In a project like astronomer/airflow-example-dags, the `Dockerfile` is `FROM quay.io/astronomer/astro-runtime:6.0.0`, the runtime that ships Airflow 2.4, the first release with Datasets; `requirements.txt` is empty and `packages.txt` holds only `postgresql`. Five DAGs live under `dags/`. Three of them, `s3_upload`, `s3_upload_copy` and `s3_upload_copy_delete`, share the same `schedule_interval` of `0 12 8-14,22-28 * 6`, `max_active_runs=1` and `catchup=False`, and each loops a `PythonOperator` that writes `globetelecom/<name>_file_<i>.txt` into the `astro-workshop-bucket` bucket through `S3Hook` with the `astro-s3-workshop` connection. The copy DAG follows each upload with an `S3CopyObjectOperator` to `globetelecom/copy_<name>_file_<i>.txt`. In `databricks_sample.py`, `adb_pipeline` runs at `30 4 * * *` and starts with an `S3KeySensor` on `globetelecom/copy_*` (`poke_interval=45`, `timeout=600`) before its `DatabricksRunNowOperator` tasks. `example_external_task_sensor.py` is entirely commented out: an older way to chain DAGs with `ExternalTaskSensor`.

The dependency between these DAGs is real but implicit: the copy only works after the upload, and `adb_pipeline` polls S3 for forty-five seconds at a time to find out whether copies exist. The agent makes it explicit with Datasets. A new `dags/datasets.py` defines one `Dataset` for the uploaded prefix and one for the copied prefix, both URIs under `s3://astro-workshop-bucket/globetelecom/`, imported from `airflow.datasets`. In `s3_upload_dag.py` every `generate_file_*` task gets `outlets=[uploads]`; in `s3_upload_copy_dag.py` the DAG's cron becomes `schedule=[uploads]` and each `copy_*` task gets `outlets=[copies]`; in `databricks_sample.py` `adb_pipeline` moves to `schedule=[copies]`, and the `S3KeySensor` either stays as a guard or goes, as the prompt says. The changed DAGs also move from the deprecated `schedule_interval` argument to `schedule`, which Airflow 2.4 introduced for exactly this purpose.

`s3_upload_copy_delete` is a self-contained variant of the same workshop that uploads, copies and deletes its own files, so the agent leaves it on its cron and explains why in the pull request rather than wiring it to a Dataset it does not depend on. The upload DAG keeps its cron, since it is the start of the chain, and the agent notes that `max_active_runs=1` on the consumers still applies to Dataset-triggered runs.

To check the result without an Airflow deployment, the agent installs `apache-airflow` 2.4 in the sandbox with the matching constraints file plus the `amazon` and `databricks` providers the changed DAGs import, loads `dags/` through `DagBag` and prints its `import_errors`, then reads each changed DAG's `dataset_triggers` to show that `s3_upload_copy` and `adb_pipeline` now trigger on the Datasets rather than a cron. It never connects to S3 or Databricks: the connections named in the DAGs stay untouched and the sandbox only parses.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Replace the shared cron dependency between the S3 DAGs with data-aware scheduling. Read every file under dags/ and the Dockerfile first. Define Datasets for the uploaded files and the copied files under the globetelecom/ prefix of the astro-workshop-bucket bucket in a new dags/datasets.py, add outlets on the tasks in s3_upload_dag.py that write the uploads and on the copy tasks in s3_upload_copy_dag.py, schedule s3_upload_copy on the uploads Dataset and adb_pipeline in databricks_sample.py on the copies Dataset, keep the S3KeySensor in adb_pipeline as a guard, and switch every changed DAG from schedule_interval to schedule. Leave s3_upload_copy_delete on its cron and say why in the pull request. Install Airflow 2.4 with its constraints file and the providers the changed DAGs import, load the DAG folder with DagBag to show no import errors, print each changed DAG's dataset triggers, and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that adds `dags/datasets.py` and changes `dags/s3_upload_dag.py`, `dags/s3_upload_copy_dag.py` and `dags/databricks_sample.py`. The description names the two Dataset URIs, shows the before and after schedule of each DAG in a short table, pastes the `DagBag` import check and the printed `dataset_triggers`, and explains why the delete DAG kept its cron.

The run's record keeps the `pip install` with the constraints file, the `DagBag` load and every file read, so a reviewer can see that the three DAGs still parse under the runtime's Airflow version before deploying with the Astro CLI.

Before merging, a reviewer checks that the Dataset URIs match the real bucket and prefix in the environment the DAGs deploy to, decides whether `adb_pipeline` still needs its `S3KeySensor` now that it only runs after copies were written, and, after the deploy, opens the Datasets view in the Airflow UI to see the producer and consumer graph the pull request describes.

## Run it on every Linear issue labelled airflow

In the console, an automation in the Project that has the repository, with a `linear` trigger on `linear.issue.labeled` and `labels` set to `airflow`, starts one run per issue with the issue as its context after the prompt; the saved prompt is the whole instruction and the trigger fills in no placeholder. The prompt asks the run to apply the issue's request, such as a new consumer DAG, on top of the Datasets `dags/datasets.py` already declares, so each new dependency arrives as an outlet and a `schedule` list rather than a new cron.

From your own tracker, a handler sends one `POST /o/{org}/sessions` with this body, the repository in `resources` and a `request_id` that is a UUID v5 of the issue id, so a repeated webhook delivery returns the run already started. A webhook URL on the API key reports `run.completed` with the `session_url` when the pull request is up.

## Why run it on Rush?

Turning a cron coincidence into a declared dependency touches every DAG in the chain, and the failure mode is quiet: a consumer whose `schedule` lists a Dataset that no task has as an outlet never runs again. The agent reads every producer and consumer, declares the Datasets in one module both sides import, and parses the whole folder with the runtime's own Airflow version before anything is deployed.

The sandbox is its own computer where Airflow and the providers install cleanly from the constraints file, with every command and its output kept in the run, and a team with several Astro projects can start one run per project at once, each ending in its own pull request.

## Frequently asked questions

### Does this work on Airflow 3, where Datasets became Assets?

The agent reads the `Dockerfile` to learn which runtime the project is on and uses that version's API. On Astro Runtime 6 that is `airflow.datasets.Dataset`; on a project whose runtime carries Airflow 3 it writes `Asset` from `airflow.sdk` and the same `outlets` and `schedule` shape, because the mechanism is the same.

### Will the upload DAG still run on its cron?

Yes. It is the start of the chain, so it keeps `0 12 8-14,22-28 * 6`. Only the consumers change: the copy DAG runs once every upload task in a run has finished and updated the Dataset, and `adb_pipeline` runs once the copy tasks have updated theirs.

### Does the run need S3 or Databricks credentials?

No. It installs Airflow and the providers so the DAG files import, and parses them with `DagBag`. The `astro-s3-workshop` and `adb_workspace` connections are only names in the code to the sandbox; nothing reads or writes a bucket.

### Why keep the S3KeySensor if the DAG now waits for the Dataset?

A Dataset update says a task that declares the outlet completed, not that the object exists with the key the Databricks job expects. The prompt keeps the sensor as a cheap guard; the pull request points out that it can go once the team trusts the outlet, which removes the forty-five second polling loop entirely.

## More Airflow jobs

- [Add Deferrable Operators to an Airflow Project](https://getrush.ai/run/add-deferrable-operators/airflow/)
- [Convert Airflow DAG Loops to Dynamic Task Mapping](https://getrush.ai/run/add-dynamic-task-mapping/airflow/)
- [Add GitHub Actions CI to an Airflow Project](https://getrush.ai/run/add-github-actions-ci/airflow/)

## More jobs like this: Turn a Linear label into a PR

- [Add Database Indexes to a Drizzle Schema from a Linear Issue](https://getrush.ai/run/add-database-indexes/drizzle/)
- [Add Missing Foreign Key Indexes With an Ecto Migration](https://getrush.ai/run/add-database-indexes/ecto/)
- [Add Database Indexes to GORM Models](https://getrush.ai/run/add-database-indexes/gorm/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
