All jobs

Add Data-Aware Scheduling to Airflow DAGs

Yes. An agent declares Datasets for the S3 keys your DAGs write, adds outlets on the producers, schedules consumers on them instead of a shared cron, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent clones the Astro project into its own cloud sandbox and reads what runs where. In a project like astronomer/airflow-example-dags, the Dockerfile is FROM quay.io/astronomer/astro-runtime:6.0.0, the runtime that ships Airflow 2.4, the first release with Datasets; requirements.txt is empty and packages.txt holds only postgresql. Five DAGs live under dags/. Three of them, s3_upload, s3_upload_copy and s3_upload_copy_delete, share the same schedule_interval of 0 12 8-14,22-28 * 6, max_active_runs=1 and catchup=False, and each loops a PythonOperator that writes globetelecom/<name>_file_<i>.txt into the astro-workshop-bucket bucket through S3Hook with the astro-s3-workshop connection. The copy DAG follows each upload with an S3CopyObjectOperator to globetelecom/copy_<name>_file_<i>.txt. In databricks_sample.py, adb_pipeline runs at 30 4 * * * and starts with an S3KeySensor on globetelecom/copy_* (poke_interval=45, timeout=600) before its DatabricksRunNowOperator tasks. example_external_task_sensor.py is entirely commented out: an older way to chain DAGs with ExternalTaskSensor.

The dependency between these DAGs is real but implicit: the copy only works after the upload, and adb_pipeline polls S3 for forty-five seconds at a time to find out whether copies exist. The agent makes it explicit with Datasets. A new dags/datasets.py defines one Dataset for the uploaded prefix and one for the copied prefix, both URIs under s3://astro-workshop-bucket/globetelecom/, imported from airflow.datasets. In s3_upload_dag.py every generate_file_* task gets outlets=[uploads]; in s3_upload_copy_dag.py the DAG's cron becomes schedule=[uploads] and each copy_* task gets outlets=[copies]; in databricks_sample.py adb_pipeline moves to schedule=[copies], and the S3KeySensor either stays as a guard or goes, as the prompt says. The changed DAGs also move from the deprecated schedule_interval argument to schedule, which Airflow 2.4 introduced for exactly this purpose.

s3_upload_copy_delete is a self-contained variant of the same workshop that uploads, copies and deletes its own files, so the agent leaves it on its cron and explains why in the pull request rather than wiring it to a Dataset it does not depend on. The upload DAG keeps its cron, since it is the start of the chain, and the agent notes that max_active_runs=1 on the consumers still applies to Dataset-triggered runs.

To check the result without an Airflow deployment, the agent installs apache-airflow 2.4 in the sandbox with the matching constraints file plus the amazon and databricks providers the changed DAGs import, loads dags/ through DagBag and prints its import_errors, then reads each changed DAG's dataset_triggers to show that s3_upload_copy and adb_pipeline now trigger on the Datasets rather than a cron. It never connects to S3 or Databricks: the connections named in the DAGs stay untouched and the sandbox only parses.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Replace the shared cron dependency between the S3 DAGs with data-aware scheduling. Read every file under dags/ and the Dockerfile first. Define Datasets for the uploaded files and the copied files under the globetelecom/ prefix of the astro-workshop-bucket bucket in a new dags/datasets.py, add outlets on the tasks in s3_upload_dag.py that write the uploads and on the copy tasks in s3_upload_copy_dag.py, schedule s3_upload_copy on the uploads Dataset and adb_pipeline in databricks_sample.py on the copies Dataset, keep the S3KeySensor in adb_pipeline as a guard, and switch every changed DAG from schedule_interval to schedule. Leave s3_upload_copy_delete on its cron and say why in the pull request. Install Airflow 2.4 with its constraints file and the providers the changed DAGs import, load the DAG folder with DagBag to show no import errors, print each changed DAG's dataset triggers, and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds dags/datasets.py and changes dags/s3_upload_dag.py, dags/s3_upload_copy_dag.py and dags/databricks_sample.py. The description names the two Dataset URIs, shows the before and after schedule of each DAG in a short table, pastes the DagBag import check and the printed dataset_triggers, and explains why the delete DAG kept its cron.

The run's record keeps the pip install with the constraints file, the DagBag load and every file read, so a reviewer can see that the three DAGs still parse under the runtime's Airflow version before deploying with the Astro CLI.

Before merging, a reviewer checks that the Dataset URIs match the real bucket and prefix in the environment the DAGs deploy to, decides whether adb_pipeline still needs its S3KeySensor now that it only runs after copies were written, and, after the deploy, opens the Datasets view in the Airflow UI to see the producer and consumer graph the pull request describes.

Run it on every Linear issue labelled airflow

In the console, an automation in the Project that has the repository, with a linear trigger on linear.issue.labeled and labels set to airflow, starts one run per issue with the issue as its context after the prompt; the saved prompt is the whole instruction and the trigger fills in no placeholder. The prompt asks the run to apply the issue's request, such as a new consumer DAG, on top of the Datasets dags/datasets.py already declares, so each new dependency arrives as an outlet and a schedule list rather than a new cron.

From your own tracker, a handler sends one POST /o/{org}/sessions with this body, the repository in resources and a request_id that is a UUID v5 of the issue id, so a repeated webhook delivery returns the run already started. A webhook URL on the API key reports run.completed with the session_url when the pull request is up.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

Turning a cron coincidence into a declared dependency touches every DAG in the chain, and the failure mode is quiet: a consumer whose schedule lists a Dataset that no task has as an outlet never runs again. The agent reads every producer and consumer, declares the Datasets in one module both sides import, and parses the whole folder with the runtime's own Airflow version before anything is deployed.

The sandbox is its own computer where Airflow and the providers install cleanly from the constraints file, with every command and its output kept in the run, and a team with several Astro projects can start one run per project at once, each ending in its own pull request.

Frequently asked questions

Does this work on Airflow 3, where Datasets became Assets?

The agent reads the Dockerfile to learn which runtime the project is on and uses that version's API. On Astro Runtime 6 that is airflow.datasets.Dataset; on a project whose runtime carries Airflow 3 it writes Asset from airflow.sdk and the same outlets and schedule shape, because the mechanism is the same.

Will the upload DAG still run on its cron?

Yes. It is the start of the chain, so it keeps 0 12 8-14,22-28 * 6. Only the consumers change: the copy DAG runs once every upload task in a run has finished and updated the Dataset, and adb_pipeline runs once the copy tasks have updated theirs.

Does the run need S3 or Databricks credentials?

No. It installs Airflow and the providers so the DAG files import, and parses them with DagBag. The astro-s3-workshop and adb_workspace connections are only names in the code to the sandbox; nothing reads or writes a bucket.

Why keep the S3KeySensor if the DAG now waits for the Dataset?

A Dataset update says a task that declares the outlet completed, not that the object exists with the key the Databricks job expects. The prompt keeps the sensor as a cheap guard; the pull request points out that it can go once the team trusts the outlet, which removes the forty-five second polling loop entirely.

More Airflow jobs

More jobs like this: Turn a Linear label into a PR

Keep reading