# Migrate Airflow DAGs From PythonOperator to the TaskFlow API

Yes. An agent rewrites your Airflow DAGs' PythonOperator tasks as @task functions, keeps task ids and dependencies identical, proves it with a DagBag diff, in a PR.

[Run this on Rush](https://cloud.getrush.ai/run/migrate-to-taskflow-api/airflow/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads the DAGs and the runtime they target. In a project like astronomer/airflow-example-dags, the `Dockerfile` is one line, `FROM quay.io/astronomer/astro-runtime:6.0.0`, an image built on Airflow 2.4, `requirements.txt` is empty and `packages.txt` adds `postgresql`. Under `dags/` are six files. `s3_upload_dag.py`, `s3_upload_copy_dag.py` and `s3_upload_copy_delete.py` share one pattern: a module-level `upload_to_s3(file_name)` function that writes a small file and calls `S3Hook(aws_conn_id='astro-s3-workshop').load_file`, a `DummyOperator` named `start`, and a `for` loop that creates a `PythonOperator` per index with `op_kwargs={'file_name': i}` and a dynamic `task_id` such as `generate_file_workshop_0`, chained with `>>` to `S3CopyObjectOperator` and, in the third file, `S3DeleteObjectsOperator`. `kube_pod.py` and `databricks_sample.py` use provider operators and no Python callables, and `example_external_task_sensor.py` is entirely commented out.

Two things in those files are already deprecated on Airflow 2.4 and are part of the same clean-up: the imports `airflow.operators.dummy_operator` and `airflow.operators.python_operator` are compatibility shims, and `schedule_interval=` has given way to `schedule=`. The agent moves `DummyOperator` to `EmptyOperator` from `airflow.operators.empty`, imports `task` from `airflow.decorators`, and converts `upload_to_s3` into a `@task` function in each file, carrying the `on_failure_callback` that `s3_upload_copy_dag.py` passes to its `PythonOperator` into the decorator's arguments. Inside the loop the call becomes `upload_to_s3.override(task_id=f"generate_file_{name}_{i}")(i)`, which Airflow 2.4 supports and which keeps every task id exactly as it was, so run history and the Grid view stay attached to the same names.

The dependency lines stay as `>>` chains, with the TaskFlow task's return value standing in for the operator object, which is how decorated and traditional tasks are mixed. `S3CopyObjectOperator` and `S3DeleteObjectsOperator` are left as operators, since TaskFlow is for Python callables, and the connection ids, bucket name, `catchup=False`, `max_active_runs=1` and the cron expressions are untouched. The `KubernetesPodOperator` and Databricks DAGs get only the import and `schedule` updates, and the commented-out file is left alone and named in the pull request.

The proof is structural. In the sandbox the agent installs Airflow 2.4 with its constraints file and the `amazon`, `cncf-kubernetes`, `databricks` and `postgres` providers the imports need, loads `dags/` through `DagBag(include_examples=False)` on the original code and on the converted code, asserts `import_errors` is empty both times, and diffs each DAG's task ids and each task's `downstream_task_ids`. An empty diff is the evidence the pull request leads with. It also runs `python -W error::DeprecationWarning` over the import of each DAG file to show the shim warnings are gone, and leaves a `tests/dags/test_dag_integrity.py` with the DagBag check so the next change is held to the same bar.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Migrate the DAGs in this Airflow project to the TaskFlow API. Read the Dockerfile, requirements.txt, packages.txt and every file under dags/ first, and note the Airflow version the runtime image carries. Convert each module-level function used by a PythonOperator into a @task function from airflow.decorators, carrying over on_failure_callback and other operator arguments, and call it inside the existing loops with override(task_id=...) so every task id stays identical. Replace DummyOperator with EmptyOperator from airflow.operators.empty, replace the deprecated operator import paths, and change schedule_interval to schedule. Leave provider operators such as S3CopyObjectOperator, S3DeleteObjectsOperator and KubernetesPodOperator as operators and keep connection ids, bucket names and schedules unchanged. Install Airflow at the runtime's version with its constraints and the providers the imports need, load dags/ with DagBag before and after, assert no import errors, diff task ids and downstream_task_ids per DAG, confirm no DeprecationWarning on import, add a DagBag integrity test under tests/dags/, and open a pull request that includes the diff.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request on its own branch with the three S3 DAG files rewritten around `@task`, import and `schedule` updates in `kube_pod.py` and `databricks_sample.py`, a new `tests/dags/test_dag_integrity.py`, and a `requirements.txt` entry only if a provider the DAGs import is not in the runtime image. The `Dockerfile`, `packages.txt` and `airflow_settings.yaml` are unchanged.

The description leads with the DagBag comparison: for each DAG, the task ids and dependencies before and after, and the empty diff between them, followed by the `import_errors` check and the deprecation-warning run. The install and test commands with their output are in the transcript at `GET /o/{org}/sessions/{id}/messages`.

Before merging, a reviewer reads one converted file against its original, checks that `on_failure_callback` and `op_kwargs` carried over, and deploys to a development deployment to watch one scheduled run of `s3_upload` complete. A follow-up message to the same session can convert the per-index loops into dynamic task mapping with `expand`, which changes task ids and is therefore kept out of this pull request.

## Run it on every terminal hand-off

A data engineer who has been converting DAGs by hand one at a time starts this from the terminal with this body and goes back to the pipeline that is actually on fire. The run reads the real `dags/` directory and the real runtime version, so what it converts and what it leaves is decided by the code, not by a template.

Send a `request_id`, a UUID, with the body so a retried start returns the run it already began. The pull request is on the run's `GET /o/{org}/sessions/{id}` answer once it is open.

## Why run it on Rush?

A TaskFlow migration is easy to do badly: a task id changes and the Grid view loses its history, a callback silently falls off, or a DAG stops parsing in production because a shim was removed. The agent proves the structure is identical with a DagBag diff on a real Airflow install at the runtime's version, in a sandbox with the providers the imports need, before a reviewer reads a line.

The pull request carries that diff and the parse results, and a test that keeps them true. Several Airflow projects can be migrated at once, each in its own sandbox, each with one pull request.

## Frequently asked questions

### Will my task ids or run history change?

No. Each `@task` call uses `override(task_id=...)` with the same f-string the `PythonOperator` used, so `generate_file_workshop_0` stays `generate_file_workshop_0`, and the DagBag diff in the pull request shows the ids and dependencies unchanged. Dynamic task mapping, which would change ids, is deliberately left for a separate run.

### Why also replace DummyOperator and the import paths?

`airflow.operators.dummy_operator` and `airflow.operators.python_operator` are compatibility shims on Airflow 2, and `EmptyOperator` replaced `DummyOperator` from 2.3. The migration touches those lines anyway, and the deprecation-warning check only passes once they are gone.

### What about the operators that are not PythonOperator?

They stay. `S3CopyObjectOperator`, `S3DeleteObjectsOperator`, `S3KeySensor`, `DatabricksRunNowOperator` and `KubernetesPodOperator` wrap no Python callable of yours, so TaskFlow does not apply. The `>>` chains mix the new `@task` outputs with those operators, which Airflow supports.

### Does the agent run the DAGs against S3?

No. The connections `astro-s3-workshop`, `adb_workspace` and `prod_postgres` exist only in your deployment, so the check is structural: parsing, task ids, dependencies and warnings. Running one DAG on a development deployment after merge is the step the pull request asks the reviewer to take.

## More Apache Airflow jobs

- [Upgrade an Astro Project's DAGs to Airflow 3](https://getrush.ai/run/upgrade-to-airflow-3/airflow/)
- [Write Tests for Apache Airflow DAGs](https://getrush.ai/run/write-dag-tests/airflow/)
- [Add DAG and Task Documentation to Airflow DAGs](https://getrush.ai/run/add-dag-docs/airflow/)

## More jobs like this: Hand off a long task

- [Migrate a Python Package to PyPI Trusted Publishing](https://getrush.ai/run/migrate-to-trusted-publishing/python/)
- [Migrate a Django REST Framework Project to uv](https://getrush.ai/run/migrate-to-uv/django-rest-framework/)
- [Migrate a Flask Project to uv](https://getrush.ai/run/migrate-to-uv/flask/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
