# Convert Airflow DAG Loops to Dynamic Task Mapping

Yes. An agent finds every DAG that builds tasks in a Python loop, rewrites each loop as a mapped task with expand, parses the DAGs again and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-dynamic-task-mapping/airflow/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent clones the repository into its own cloud sandbox and reads the project before touching a DAG. In an Astro project like astronomer/airflow-example-dags it starts with the `Dockerfile`, which pins `quay.io/astronomer/astro-runtime:6.0.0`, and reads `requirements.txt`, `packages.txt` and `airflow_settings.yaml` to learn which providers and connections the DAGs expect. It then finds the loops: `dags/s3_upload_dag.py` runs `for i in range(0,5)` and creates a `PythonOperator` per iteration with a formatted `task_id`, `dags/s3_upload_copy_dag.py` and `dags/s3_upload_copy_delete.py` do the same over `range(0,10)` and chain an `S3CopyObjectOperator` and an `S3DeleteObjectsOperator` behind each upload, and `dags/databricks_sample.py` loops over a `job_info` list to create a `DatabricksRunNowOperator` and a `PostgresOperator` per job. Each loop bakes the task count into the DAG file, and the comment above the first one already says `generates 10 tasks` where the range gives five.

For each loop the agent decides what the mapped input is and rewrites the operators with `partial` and `expand`. The upload becomes one `PythonOperator.partial(task_id='generate_file', python_callable=upload_to_s3)` expanded over `op_kwargs`, a list of `{'file_name': i}` dictionaries built from the same range, so the Airflow UI shows `generate_file[0]` through `generate_file[4]` instead of `generate_file_workshop_0` and friends. The copy and delete operators need several arguments that vary together, source key and destination key, so the agent uses `expand_kwargs` over a list of dictionaries rather than `expand` over two lists, which would produce a cross product; `expand_kwargs` arrived in Airflow 2.4, so the agent confirms the Airflow version the runtime image ships before choosing it and falls back to a single mapped dictionary argument otherwise. In the Databricks DAG the `job_info` list becomes the mapped input for both operators, with `json` and `job_id` drawn from each entry and the SQL for the follow-up query mapped alongside it, and the `S3KeySensor` stays as a single upstream task.

The same pass fixes what the loops hid. The upload callable is only ever given the loop index, so the agent keeps its signature and leaves `S3Hook`, the bucket constant and the connection id alone. The `test_config2` resource dictionary in `s3_upload_copy_delete.py` is defined and never passed to an operator, so the agent either attaches it as `executor_config` on the mapped task or removes it, and says which in the pull request. Imports from `airflow.operators.dummy_operator` and `airflow.operators.python_operator` are the pre-2.0 module paths, and `DummyOperator` was renamed `EmptyOperator` in 2.3; the agent moves them to `airflow.operators.empty` and `airflow.operators.python` only when the runtime's Airflow version supports both, because the point of this change is mapping, not a framework upgrade. It leaves `schedule_interval`, `default_args` and `catchup=False` exactly as they are.

To check the result the agent installs the Airflow version matching the runtime image into the sandbox with the Amazon, Databricks and Kubernetes providers, runs `airflow db migrate` against a local SQLite metadata database, and loads the `dags/` folder through `DagBag`, asserting that there are no import errors and that each DAG now has one mapped task where it had five or ten literal ones. It calls `airflow dags show` for each DAG to render the new graph and `airflow tasks render` on a mapped index to prove the templated keys still resolve to the same S3 paths as before. It cannot run the uploads, because the `astro-s3-workshop` connection exists only in the reader's deployment, and the pull request says so.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Convert the DAGs in this Airflow project from Python loops that create one operator per iteration to dynamic task mapping. Read the Dockerfile, requirements.txt, packages.txt, airflow_settings.yaml and every file under dags/ first, and work out which Airflow version the runtime image ships. For each loop, rewrite the operators with partial() and expand(), or expand_kwargs() where several arguments vary together, so the task ids become mapped indexes; keep every callable, connection id, bucket name, schedule and default_args unchanged. Attach or remove any resource configuration the loops define but never use, and say which you did. Move deprecated operator import paths to their current modules only if the runtime's Airflow version supports them. Install that Airflow version with the providers the DAGs import, load the dags/ folder with DagBag, confirm there are no import errors, render the mapped task graph with airflow dags show and airflow tasks render, and open a pull request that lists each DAG, how many literal tasks it had and the mapped task that replaces them, with the DagBag and render output included.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that changes the four DAG files and nothing else. Each `for` loop that created operators is gone, replaced by a `partial` call that holds the arguments shared by every copy and an `expand` or `expand_kwargs` call that holds the list the loop used to iterate over. The description has one row per DAG: the literal task ids it used to generate, the mapped task id that replaces them, and the number of mapped instances the UI will show.

The run's record holds the `pip install` of Airflow and its providers, the `DagBag` load with its import error list, and the `airflow dags show` and `airflow tasks render` output, so a reviewer can see that the DAGs parse and that a rendered mapped index points at the same S3 key the old literal task did. Where the agent chose `expand_kwargs`, the description quotes the Airflow version it checked and why `expand` over two lists would have produced a cross product.

Before merging, a reviewer checks that the mapped task ids are acceptable, because the old literal ids disappear from the task instance history, decides whether to keep or drop the unused executor resource dictionary the agent flagged, and confirms the deployment's Airflow version matches the one the agent tested against.

## Run it on every DAG file that builds tasks in a loop

In the console, an automation in the Project that has the repository, with a `github` trigger on `pull_request.opened`, starts one run per pull request with the event after the prompt. The prompt asks the run to finish at once unless a file under `dags/` changed, and otherwise to look for a `for` loop that constructs operators, rewrite it as a mapped task on the pull request's branch and leave a review comment with the diff, so a new DAG never reaches the main branch with a task count hard-coded into its source.

From your own server, send this body as `POST /o/{org}/sessions` with the repository in `resources` for each Airflow project you own, with a `request_id` that is a UUID v5 of the repository name so a retried call returns the run it already started. Set a `webhook_url` on the key to receive `run.completed` with `data.session_url` when each pull request is open.

## Why run it on Rush?

Rewriting a loop as a mapped task is small per DAG and risky in aggregate, because the proof that nothing changed is a parse of every DAG under a specific Airflow version with the right providers installed, and that environment rarely exists on a laptop beside a different project's virtualenv. The agent's sandbox is a fresh machine where it can install the exact Airflow release the runtime image ships, load the DAG folder and render the mapped graph, and every one of those commands and its output stays in the run's record next to the pull request.

The same request runs across every Airflow repository in an organisation at once, each in its own sandbox with its own runtime version check, which is how a team with a dozen Astro projects of different ages gets a consistent set of pull requests instead of a migration that stalls after the first one. A pull request the agent could not make parse is never opened; the run says why instead.

## Frequently asked questions

### Will the mapped task ids break my task instance history?

The history for the old ids stays in the metadata database but is no longer attached to a task in the DAG, so it disappears from the grid view. The pull request lists every old id beside its new mapped id so you can decide per DAG whether to keep a loop for one more release or accept the break. Nothing is deleted.

### What about loops whose arguments vary together, like a source key and a destination key?

Those use `expand_kwargs` over a list of dictionaries, one per iteration, so the pairs stay together. Mapping two lists through `expand` would run every source against every destination. `expand_kwargs` needs Airflow 2.4 or later, which is why the agent reads the runtime version before choosing it.

### Does the agent also move my DAGs to the TaskFlow API?

No. This job keeps the classic operators and changes only how many of them are created. Mapping works on both styles, and mixing a decorator migration into the same pull request would make the diff harder to review. Run the TaskFlow migration as a separate job if you want it.

### Can it run the mapped tasks end to end?

Not against S3 or Databricks, because the connections in `airflow_settings.yaml` are placeholders and the real ones live in your deployment. The agent proves the DAGs parse and that templated fields render to the same values, and leaves the first real run to your deployment.

### Will it bump my Astro Runtime or Airflow version?

No. It reads the version from the `Dockerfile` and tests against that. If a mapping feature it wants is missing on that version, it uses the older form and notes which upgrade would allow the shorter one.

## More Airflow jobs

- [Add GitHub Actions CI to an Airflow Project](https://getrush.ai/run/add-github-actions-ci/airflow/)
- [Add Pools to Limit Concurrent Tasks in Airflow](https://getrush.ai/run/add-pools/airflow/)
- [Add Retry Policies With Exponential Backoff to Airflow DAGs](https://getrush.ai/run/add-retry-policies/airflow/)

## More jobs like this: Hand off a long task

- [Turn On DynamoDB Point-in-Time Recovery in a Serverless Framework Service](https://getrush.ai/run/add-dynamodb-point-in-time-recovery/serverless/)
- [Add a DynamoDB TTL to a Serverless Framework Service](https://getrush.ai/run/add-dynamodb-ttl/serverless/)
- [Add an EAS Build Workflow on GitHub Actions to an Expo App](https://getrush.ai/run/add-eas-build-workflow/expo/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
