# Upgrade an Astro Project's DAGs to Airflow 3

Yes. An agent runs the Airflow 3 ruff rules on your DAGs, rewrites removed imports and arguments, bumps the runtime image, adds a DAG parse test and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/upgrade-to-airflow-3/airflow/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent clones the repository into its own cloud sandbox and reads what Airflow version the project runs before it changes a DAG. In a project like astronomer/airflow-example-dags, the `Dockerfile` is one line, `FROM quay.io/astronomer/astro-runtime:6.0.0`, an Airflow 2 image; `requirements.txt` is empty, `packages.txt` lists `postgresql`, and `airflow_settings.yaml` is the blank connections, pools and variables template. `dags/` holds six files. `databricks_sample.py` imports `DummyOperator` from `airflow.operators.dummy_operator`, `PostgresOperator` from `airflow.operators.postgres_operator`, `DatabricksRunNowOperator` and `S3KeySensor` from their providers, and declares `adb_pipeline` with `schedule_interval='30 4 * * *'`. `kube_pod.py` imports `KubernetesPodOperator` from `airflow.providers.cncf.kubernetes.operators.kubernetes_pod`, does `from airflow import configuration as conf` and reads `conf.get('kubernetes', 'NAMESPACE')` at parse time to decide `in_cluster`. The three `s3_upload*` DAGs import `PythonOperator` from `airflow.operators.python_operator`, use `S3Hook` and `S3CopyObjectOperator`, generate tasks in a loop and set `schedule_interval='0 12 8-14,22-28 * 6'` with `max_active_runs=1` and `catchup=False`. `example_external_task_sensor.py` is commented out from top to bottom. There are no tests and no CI.

The agent starts with the tool the Airflow 3 upgrade guide recommends: `ruff check dags/ --select AIR301,AIR302` with a current ruff, the first for breaking changes and the second for operators that moved to provider packages, then `AIR311` and `AIR312` for the recommended ones. It reads the findings file by file rather than applying `--fix` blindly, because the fixes that matter here are not all mechanical. `schedule_interval=` becomes `schedule=` in every DAG; `DummyOperator` becomes `EmptyOperator` from `airflow.providers.standard.operators.empty`; `PythonOperator` moves to `airflow.providers.standard.operators.python`; `PostgresOperator` becomes `SQLExecuteQueryOperator` from `airflow.providers.common.sql.operators.sql` with `conn_id='prod_postgres'`; `KubernetesPodOperator` moves to `airflow.providers.cncf.kubernetes.operators.pod`; and `DAG` is imported from `airflow.sdk`. The parse-time `conf.get` in `kube_pod.py` gets its own paragraph in the pull request: DAG files in Airflow 3 are parsed by the DAG processor and run under the Task SDK, so the agent reads the namespace from an environment variable with the old lookup quoted in a comment, and flags it for the reviewer rather than pretending the behaviour is identical.

Then the image. The agent lists the `astro-runtime` tags that ship Airflow 3 and pins the newest one in the `Dockerfile`, adds the provider distributions the DAGs now import to `requirements.txt` where the image does not already bundle them, and checks `catchup=False` is explicit on every DAG, since `catchup_by_default` changed and a DAG that relied on the old default would backfill. It leaves the fully commented `example_external_task_sensor.py` alone and says so, because deleting a file is a maintainer's decision.

To prove the DAGs still parse, the agent installs Airflow 3 with those providers into a virtual environment in the sandbox and writes `tests/test_dag_integrity.py`: a `DagBag` over `dags/` with examples off, asserting `import_errors` is empty and that the five DAG ids are present. It runs `pytest`, runs the ruff rules again to show a clean result, and adds `.github/workflows/dags.yml` that does the same on every pull request so the next regression is caught before deploy.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Upgrade the DAGs in this Astro project to Airflow 3. Read the Dockerfile, requirements.txt, packages.txt and every file under dags/ first. Run ruff check dags/ --select AIR301,AIR302 and then AIR311,AIR312 with a current ruff and read every finding. Replace removed import paths with their provider or airflow.sdk equivalents, change schedule_interval to schedule, replace removed operators with their successors, and make catchup explicit. Where a DAG reads Airflow configuration or the metadata database at parse time, rewrite it and explain the change in the pull request. Pin the Dockerfile to the newest astro-runtime image that ships Airflow 3 and add any provider packages the DAGs now need to requirements.txt. Install Airflow 3 with those providers in a virtual environment, add a DagBag parse test under tests/ and a GitHub Actions workflow that runs it and the ruff rules, run both, and open a pull request that lists each file with what changed and why. Do not delete DAG files.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request on its own branch that edits the five live DAG files, the `Dockerfile` and `requirements.txt`, and adds `tests/test_dag_integrity.py` and `.github/workflows/dags.yml`. The description has one section per DAG listing the old and new import for every operator, the `schedule` change, and in `kube_pod.py` the namespace lookup it replaced, followed by the ruff output before and after and the `pytest` output from the sandbox.

The run's record keeps the `ruff` findings as they were first printed, the `pip install` of Airflow 3 and the providers with their resolved versions, and the parse test run, so a reviewer can see which version the DAGs were parsed under without rerunning anything.

Before merging, a reviewer checks the runtime tag against what the deployment target supports, confirms the environment variable that now supplies the Kubernetes namespace is set in each deployment, and reads the `SQLExecuteQueryOperator` change against the queries in `databricks_sample.py`, since the replacement operator's return handling differs from the old `PostgresOperator`.

## Run it on every Astro project

A team with many Astro projects starts one run per repository from its own server: one `POST /o/{org}/sessions` with this body and the repository in `resources`, with a `request_id` that is a UUID v5 of the repository name so a retried call returns the first run. Each run reads that repository's own `Dockerfile` and DAGs, so the import rewrites and the runtime tag match what that project uses.

A webhook URL on the API key receives `run.completed` or `run.failed` for each run with `session_url`, and the server collects the pull request links so the data platform team can review the upgrades in one sitting.

## Why run it on Rush?

An Airflow 3 upgrade is many small, well-documented edits plus a few that need judgement, like a DAG that reads configuration at parse time. Running the `AIR3` rules is the easy part; reading each finding against the file, choosing the successor operator, and proving the result parses under a real Airflow 3 install is the work. The agent does all of it in one sandbox and keeps the ruff output, the install log and the parse test in the record.

The parse test and workflow it adds are what keep the upgrade from regressing when the next DAG is written against an old tutorial. Several projects can be upgraded at the same time in separate sandboxes, each against its own image tag and providers.

## Frequently asked questions

### Does it run the DAGs?

No. It parses them with a `DagBag` under Airflow 3 and runs the ruff rules; it does not start a scheduler or execute tasks, and the pull request says so. Running a DAG against S3, Databricks or a cluster is a deployment step with real credentials.

### Why not just apply ruff --fix?

The agent uses the fixes where they are mechanical, but some findings need a decision: a parse-time configuration read, a removed operator whose successor returns something different, or a schedule that relied on data intervals. Those get a sentence each in the pull request instead of a silent rewrite.

### What happens to the commented-out DAG file?

Nothing. The prompt forbids deleting DAG files, so `example_external_task_sensor.py` stays as it is and the pull request notes that it contains no live code.

### Which runtime tag does it choose?

The newest `astro-runtime` tag that ships Airflow 3 at the time of the run, named in the pull request with the Airflow version it carries. If your deployment pins a different one, say so in `input` and it uses that.

### Will catchup behaviour change?

Not for these DAGs: each one sets `catchup=False` explicitly already, and the agent makes it explicit anywhere it was missing, because the Airflow 3 default is now `False`. The pull request lists any DAG where that was added.

## More Airflow jobs

- [Write Tests for Apache Airflow DAGs](https://getrush.ai/run/write-dag-tests/airflow/)
- [Add DAG and Task Documentation to Airflow DAGs](https://getrush.ai/run/add-dag-docs/airflow/)
- [Add Params to Airflow DAGs](https://getrush.ai/run/add-dag-params/airflow/)

## More jobs like this: Hand off a long task

- [Add Accessibility Labels to an Expo App's Screens and Components](https://getrush.ai/run/add-accessibility-labels/expo/)
- [Add Accessibility Labels to a Flutter Cupertino App](https://getrush.ai/run/add-accessibility-labels/flutter/)
- [Add Account Deletion to an AdonisJS API](https://getrush.ai/run/add-account-deletion/adonisjs/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
