What the agent does
The agent clones the Airflow project and reads the DAG files before it touches one. In a project like astronomer/airflow-example-dags, dags/s3_upload_copy_delete.py loops for i in range(0,10) and creates three tasks per iteration, generate_file_workshop_{i}, copy_workshop_file_{i} and delete_workshop_file_{i}, all chained from a single start task, so the graph view shows thirty parallel tasks with nothing that says which copy belongs to which upload. dags/s3_upload_copy_dag.py does the same with two tasks per iteration, dags/s3_upload_dag.py with one, and dags/databricks_sample.py creates a DatabricksRunNowOperator and a PostgresOperator for each of four entries in job_info. The agent also notes what the files import: airflow.operators.dummy_operator and airflow.operators.python_operator, both old module paths, and schedule_interval, while the Dockerfile pins quay.io/astronomer/astro-runtime:6.0.0 and requirements.txt is empty, so the providers come from the runtime image.
It then rewrites the loops around TaskGroup so each unit of work is one collapsible node. In s3_upload_copy_delete.py every iteration becomes with TaskGroup(group_id=f"file_{i}") as group: holding its generate, copy and delete tasks with the dependency chain inside the group, and t0 >> group outside it, which keeps the fan-out but makes the graph read as ten files rather than thirty boxes. In databricks_sample.py the group is one per Databricks job, named from job_id, holding the run and the follow-up query. The agent decides the task id question explicitly: Airflow prefixes a grouped task's id with the group id by default, which renames every task and orphans its history, so it either keeps the ids by setting prefix_group_id=False on each group or accepts the rename and says so, and it picks the first unless the prompt says otherwise. Where it is already editing a file it replaces DummyOperator with EmptyOperator and the python_operator import with airflow.operators.python, both available in the Airflow 2.4 line the runtime image ships.
To prove the DAGs still load, the agent installs the same Airflow line in its sandbox with the matching constraints file, plus the Amazon, Databricks and Kubernetes providers the imports need, and runs a short script that builds a DagBag from dags/ and fails on any import error. It then compares the set of task ids per DAG before and after the change, so the pull request can state that no id changed, and renders each DAG's dependency list to show the chains are intact inside their groups. dags/example_external_task_sensor.py is entirely commented out and dags/kube_pod.py has one task, so both are left alone.
The pull request carries a table per DAG: group id, the tasks inside it and whether the ids were preserved. It also lists the deprecated imports it saw in files it did not otherwise edit, with the one-line replacement each needs, rather than widening the change.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Organise the DAGs in this Airflow project with TaskGroups. Read every file under dags/ and the Dockerfile first, and list each DAG that creates tasks in a loop or per entry of a list. Wrap each unit of work in a TaskGroup with a clear group_id and keep the dependency chain inside the group; keep existing task ids unchanged by setting prefix_group_id=False and say so in each DAG's docstring. While editing a file, replace DummyOperator with EmptyOperator and old operator import paths with their current ones. Install the Airflow version the runtime image pins with its constraints file and the providers the DAGs import, load the dags/ folder with a DagBag and fail on any import error, and compare task ids before and after. Open a pull request with one table per DAG and the DagBag output.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that changes the DAG files with loops, adds a TaskGroup import and a short docstring note to each, and leaves single-task and commented-out DAGs untouched. The description has one table per DAG with each group id and its member tasks, and a line per DAG saying the task ids are unchanged.
The run's record holds the install of the pinned Airflow line, the DagBag load with zero import errors, and the before-and-after task id sets, so a reviewer can see the proof instead of taking the description's word for it.
Before merging, a reviewer opens the graph view in a development deployment to confirm the groups collapse the way the team wants, checks that any dashboard or alert that references a task id still matches, and decides whether the deprecated imports listed for untouched files should become a second pull request.
Run it on every Airflow project
A data engineer hands this off from the terminal: one POST /o/{org}/sessions with this body and the project repository in resources. The key's webhook_url reports the finished run with the pull request in result, and a retry sends the same request_id, a UUID, so one request starts one run.
A team with several Airflow repositories sends one request per repository. Each run reads that project's own Dockerfile or requirements.txt to pick the Airflow version it tests against, so a project on a newer runtime is parsed with that runtime's Airflow rather than a guessed one.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
Grouping tasks is mechanical to describe and easy to get wrong in the details: a group id that renames every task, a dependency that ends up outside the group, an import path that no longer exists in the Airflow the image ships. The agent reads every DAG, makes the id decision explicit, and loads the result with the same Airflow line as the runtime image before a person reviews it.
On Rush that check is part of the run, not a note asking the reviewer to try it locally. The installed Airflow, the DagBag output and the task id comparison are kept in the run's record beside the pull request, and the same request works across every Airflow repository a team owns.
Frequently asked questions
Will my task ids change?
Not unless you ask. Airflow prefixes a task id with its group id by default, which breaks task history and any external reference to the id. The agent sets prefix_group_id=False on each group and compares the id sets before and after to prove nothing was renamed.
Does it use the @task_group decorator or the TaskGroup context manager?
The context manager, because the DAGs here build tasks with classic operators inside loops. The decorator form fits TaskFlow functions; converting to TaskFlow is a different change the agent does not make.
How does it know which Airflow version to test with?
From the project itself. A Dockerfile that pins an Astro Runtime image maps to an Airflow line, and a requirements.txt that pins apache-airflow is used directly. The run installs that line with its constraints file so the import check matches what runs in production.
Will it also fix the deprecated imports?
Only in files it is already editing. For the others it lists the file, the old import and the replacement in the pull request, so the team can decide whether that cleanup is worth a separate change.
More Apache Airflow jobs
More jobs like this: Hand off a long task
- Add Telemetry to a VS Code Extension (VS Code extension)
- Add Tenant Isolation to a Flux Repository (Flux)
- Generate Terraform Module Docs With terraform-docs and Keep Them Current in CI (Terraform)