What the agent does
The agent reads the DAGs and the runtime they target. In a project like astronomer/airflow-example-dags, the Dockerfile is one line, FROM quay.io/astronomer/astro-runtime:6.0.0, an image built on Airflow 2.4, requirements.txt is empty and packages.txt adds postgresql. Under dags/ are six files. s3_upload_dag.py, s3_upload_copy_dag.py and s3_upload_copy_delete.py share one pattern: a module-level upload_to_s3(file_name) function that writes a small file and calls S3Hook(aws_conn_id='astro-s3-workshop').load_file, a DummyOperator named start, and a for loop that creates a PythonOperator per index with op_kwargs={'file_name': i} and a dynamic task_id such as generate_file_workshop_0, chained with >> to S3CopyObjectOperator and, in the third file, S3DeleteObjectsOperator. kube_pod.py and databricks_sample.py use provider operators and no Python callables, and example_external_task_sensor.py is entirely commented out.
Two things in those files are already deprecated on Airflow 2.4 and are part of the same clean-up: the imports airflow.operators.dummy_operator and airflow.operators.python_operator are compatibility shims, and schedule_interval= has given way to schedule=. The agent moves DummyOperator to EmptyOperator from airflow.operators.empty, imports task from airflow.decorators, and converts upload_to_s3 into a @task function in each file, carrying the on_failure_callback that s3_upload_copy_dag.py passes to its PythonOperator into the decorator's arguments. Inside the loop the call becomes upload_to_s3.override(task_id=f"generate_file_{name}_{i}")(i), which Airflow 2.4 supports and which keeps every task id exactly as it was, so run history and the Grid view stay attached to the same names.
The dependency lines stay as >> chains, with the TaskFlow task's return value standing in for the operator object, which is how decorated and traditional tasks are mixed. S3CopyObjectOperator and S3DeleteObjectsOperator are left as operators, since TaskFlow is for Python callables, and the connection ids, bucket name, catchup=False, max_active_runs=1 and the cron expressions are untouched. The KubernetesPodOperator and Databricks DAGs get only the import and schedule updates, and the commented-out file is left alone and named in the pull request.
The proof is structural. In the sandbox the agent installs Airflow 2.4 with its constraints file and the amazon, cncf-kubernetes, databricks and postgres providers the imports need, loads dags/ through DagBag(include_examples=False) on the original code and on the converted code, asserts import_errors is empty both times, and diffs each DAG's task ids and each task's downstream_task_ids. An empty diff is the evidence the pull request leads with. It also runs python -W error::DeprecationWarning over the import of each DAG file to show the shim warnings are gone, and leaves a tests/dags/test_dag_integrity.py with the DagBag check so the next change is held to the same bar.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Migrate the DAGs in this Airflow project to the TaskFlow API. Read the Dockerfile, requirements.txt, packages.txt and every file under dags/ first, and note the Airflow version the runtime image carries. Convert each module-level function used by a PythonOperator into a @task function from airflow.decorators, carrying over on_failure_callback and other operator arguments, and call it inside the existing loops with override(task_id=...) so every task id stays identical. Replace DummyOperator with EmptyOperator from airflow.operators.empty, replace the deprecated operator import paths, and change schedule_interval to schedule. Leave provider operators such as S3CopyObjectOperator, S3DeleteObjectsOperator and KubernetesPodOperator as operators and keep connection ids, bucket names and schedules unchanged. Install Airflow at the runtime's version with its constraints and the providers the imports need, load dags/ with DagBag before and after, assert no import errors, diff task ids and downstream_task_ids per DAG, confirm no DeprecationWarning on import, add a DagBag integrity test under tests/dags/, and open a pull request that includes the diff.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request on its own branch with the three S3 DAG files rewritten around @task, import and schedule updates in kube_pod.py and databricks_sample.py, a new tests/dags/test_dag_integrity.py, and a requirements.txt entry only if a provider the DAGs import is not in the runtime image. The Dockerfile, packages.txt and airflow_settings.yaml are unchanged.
The description leads with the DagBag comparison: for each DAG, the task ids and dependencies before and after, and the empty diff between them, followed by the import_errors check and the deprecation-warning run. The install and test commands with their output are in the transcript at GET /o/{org}/sessions/{id}/messages.
Before merging, a reviewer reads one converted file against its original, checks that on_failure_callback and op_kwargs carried over, and deploys to a development deployment to watch one scheduled run of s3_upload complete. A follow-up message to the same session can convert the per-index loops into dynamic task mapping with expand, which changes task ids and is therefore kept out of this pull request.
Run it on every terminal hand-off
A data engineer who has been converting DAGs by hand one at a time starts this from the terminal with this body and goes back to the pipeline that is actually on fire. The run reads the real dags/ directory and the real runtime version, so what it converts and what it leaves is decided by the code, not by a template.
Send a request_id, a UUID, with the body so a retried start returns the run it already began. The pull request is on the run's GET /o/{org}/sessions/{id} answer once it is open.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
A TaskFlow migration is easy to do badly: a task id changes and the Grid view loses its history, a callback silently falls off, or a DAG stops parsing in production because a shim was removed. The agent proves the structure is identical with a DagBag diff on a real Airflow install at the runtime's version, in a sandbox with the providers the imports need, before a reviewer reads a line.
The pull request carries that diff and the parse results, and a test that keeps them true. Several Airflow projects can be migrated at once, each in its own sandbox, each with one pull request.
Frequently asked questions
Will my task ids or run history change?
No. Each @task call uses override(task_id=...) with the same f-string the PythonOperator used, so generate_file_workshop_0 stays generate_file_workshop_0, and the DagBag diff in the pull request shows the ids and dependencies unchanged. Dynamic task mapping, which would change ids, is deliberately left for a separate run.
Why also replace DummyOperator and the import paths?
airflow.operators.dummy_operator and airflow.operators.python_operator are compatibility shims on Airflow 2, and EmptyOperator replaced DummyOperator from 2.3. The migration touches those lines anyway, and the deprecation-warning check only passes once they are gone.
What about the operators that are not PythonOperator?
They stay. S3CopyObjectOperator, S3DeleteObjectsOperator, S3KeySensor, DatabricksRunNowOperator and KubernetesPodOperator wrap no Python callable of yours, so TaskFlow does not apply. The >> chains mix the new @task outputs with those operators, which Airflow supports.
Does the agent run the DAGs against S3?
No. The connections astro-s3-workshop, adb_workspace and prod_postgres exist only in your deployment, so the check is structural: parsing, task ids, dependencies and warnings. Running one DAG on a development deployment after merge is the step the pull request asks the reviewer to take.
More Apache Airflow jobs
More jobs like this: Hand off a long task
- Migrate a Python Package to PyPI Trusted Publishing (Python)
- Migrate a Django REST Framework Project to uv (Django REST Framework)
- Migrate a Flask Project to uv (Flask)