All jobs

Convert Airflow DAG Loops to Dynamic Task Mapping

Yes. An agent finds every DAG that builds tasks in a Python loop, rewrites each loop as a mapped task with expand, parses the DAGs again and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent clones the repository into its own cloud sandbox and reads the project before touching a DAG. In an Astro project like astronomer/airflow-example-dags it starts with the Dockerfile, which pins quay.io/astronomer/astro-runtime:6.0.0, and reads requirements.txt, packages.txt and airflow_settings.yaml to learn which providers and connections the DAGs expect. It then finds the loops: dags/s3_upload_dag.py runs for i in range(0,5) and creates a PythonOperator per iteration with a formatted task_id, dags/s3_upload_copy_dag.py and dags/s3_upload_copy_delete.py do the same over range(0,10) and chain an S3CopyObjectOperator and an S3DeleteObjectsOperator behind each upload, and dags/databricks_sample.py loops over a job_info list to create a DatabricksRunNowOperator and a PostgresOperator per job. Each loop bakes the task count into the DAG file, and the comment above the first one already says generates 10 tasks where the range gives five.

For each loop the agent decides what the mapped input is and rewrites the operators with partial and expand. The upload becomes one PythonOperator.partial(task_id='generate_file', python_callable=upload_to_s3) expanded over op_kwargs, a list of {'file_name': i} dictionaries built from the same range, so the Airflow UI shows generate_file[0] through generate_file[4] instead of generate_file_workshop_0 and friends. The copy and delete operators need several arguments that vary together, source key and destination key, so the agent uses expand_kwargs over a list of dictionaries rather than expand over two lists, which would produce a cross product; expand_kwargs arrived in Airflow 2.4, so the agent confirms the Airflow version the runtime image ships before choosing it and falls back to a single mapped dictionary argument otherwise. In the Databricks DAG the job_info list becomes the mapped input for both operators, with json and job_id drawn from each entry and the SQL for the follow-up query mapped alongside it, and the S3KeySensor stays as a single upstream task.

The same pass fixes what the loops hid. The upload callable is only ever given the loop index, so the agent keeps its signature and leaves S3Hook, the bucket constant and the connection id alone. The test_config2 resource dictionary in s3_upload_copy_delete.py is defined and never passed to an operator, so the agent either attaches it as executor_config on the mapped task or removes it, and says which in the pull request. Imports from airflow.operators.dummy_operator and airflow.operators.python_operator are the pre-2.0 module paths, and DummyOperator was renamed EmptyOperator in 2.3; the agent moves them to airflow.operators.empty and airflow.operators.python only when the runtime's Airflow version supports both, because the point of this change is mapping, not a framework upgrade. It leaves schedule_interval, default_args and catchup=False exactly as they are.

To check the result the agent installs the Airflow version matching the runtime image into the sandbox with the Amazon, Databricks and Kubernetes providers, runs airflow db migrate against a local SQLite metadata database, and loads the dags/ folder through DagBag, asserting that there are no import errors and that each DAG now has one mapped task where it had five or ten literal ones. It calls airflow dags show for each DAG to render the new graph and airflow tasks render on a mapped index to prove the templated keys still resolve to the same S3 paths as before. It cannot run the uploads, because the astro-s3-workshop connection exists only in the reader's deployment, and the pull request says so.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Convert the DAGs in this Airflow project from Python loops that create one operator per iteration to dynamic task mapping. Read the Dockerfile, requirements.txt, packages.txt, airflow_settings.yaml and every file under dags/ first, and work out which Airflow version the runtime image ships. For each loop, rewrite the operators with partial() and expand(), or expand_kwargs() where several arguments vary together, so the task ids become mapped indexes; keep every callable, connection id, bucket name, schedule and default_args unchanged. Attach or remove any resource configuration the loops define but never use, and say which you did. Move deprecated operator import paths to their current modules only if the runtime's Airflow version supports them. Install that Airflow version with the providers the DAGs import, load the dags/ folder with DagBag, confirm there are no import errors, render the mapped task graph with airflow dags show and airflow tasks render, and open a pull request that lists each DAG, how many literal tasks it had and the mapped task that replaces them, with the DagBag and render output included.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that changes the four DAG files and nothing else. Each for loop that created operators is gone, replaced by a partial call that holds the arguments shared by every copy and an expand or expand_kwargs call that holds the list the loop used to iterate over. The description has one row per DAG: the literal task ids it used to generate, the mapped task id that replaces them, and the number of mapped instances the UI will show.

The run's record holds the pip install of Airflow and its providers, the DagBag load with its import error list, and the airflow dags show and airflow tasks render output, so a reviewer can see that the DAGs parse and that a rendered mapped index points at the same S3 key the old literal task did. Where the agent chose expand_kwargs, the description quotes the Airflow version it checked and why expand over two lists would have produced a cross product.

Before merging, a reviewer checks that the mapped task ids are acceptable, because the old literal ids disappear from the task instance history, decides whether to keep or drop the unused executor resource dictionary the agent flagged, and confirms the deployment's Airflow version matches the one the agent tested against.

Run it on every DAG file that builds tasks in a loop

In the console, an automation in the Project that has the repository, with a github trigger on pull_request.opened, starts one run per pull request with the event after the prompt. The prompt asks the run to finish at once unless a file under dags/ changed, and otherwise to look for a for loop that constructs operators, rewrite it as a mapped task on the pull request's branch and leave a review comment with the diff, so a new DAG never reaches the main branch with a task count hard-coded into its source.

From your own server, send this body as POST /o/{org}/sessions with the repository in resources for each Airflow project you own, with a request_id that is a UUID v5 of the repository name so a retried call returns the run it already started. Set a webhook_url on the key to receive run.completed with data.session_url when each pull request is open.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

Rewriting a loop as a mapped task is small per DAG and risky in aggregate, because the proof that nothing changed is a parse of every DAG under a specific Airflow version with the right providers installed, and that environment rarely exists on a laptop beside a different project's virtualenv. The agent's sandbox is a fresh machine where it can install the exact Airflow release the runtime image ships, load the DAG folder and render the mapped graph, and every one of those commands and its output stays in the run's record next to the pull request.

The same request runs across every Airflow repository in an organisation at once, each in its own sandbox with its own runtime version check, which is how a team with a dozen Astro projects of different ages gets a consistent set of pull requests instead of a migration that stalls after the first one. A pull request the agent could not make parse is never opened; the run says why instead.

Frequently asked questions

Will the mapped task ids break my task instance history?

The history for the old ids stays in the metadata database but is no longer attached to a task in the DAG, so it disappears from the grid view. The pull request lists every old id beside its new mapped id so you can decide per DAG whether to keep a loop for one more release or accept the break. Nothing is deleted.

What about loops whose arguments vary together, like a source key and a destination key?

Those use expand_kwargs over a list of dictionaries, one per iteration, so the pairs stay together. Mapping two lists through expand would run every source against every destination. expand_kwargs needs Airflow 2.4 or later, which is why the agent reads the runtime version before choosing it.

Does the agent also move my DAGs to the TaskFlow API?

No. This job keeps the classic operators and changes only how many of them are created. Mapping works on both styles, and mixing a decorator migration into the same pull request would make the diff harder to review. Run the TaskFlow migration as a separate job if you want it.

Can it run the mapped tasks end to end?

Not against S3 or Databricks, because the connections in airflow_settings.yaml are placeholders and the real ones live in your deployment. The agent proves the DAGs parse and that templated fields render to the same values, and leaves the first real run to your deployment.

Will it bump my Astro Runtime or Airflow version?

No. It reads the version from the Dockerfile and tests against that. If a mapping feature it wants is missing on that version, it uses the older form and notes which upgrade would allow the shorter one.

More Airflow jobs

More jobs like this: Hand off a long task

Keep reading