# Add DAG and Task Documentation to Airflow DAGs

Yes. An agent wires each DAG's docstring into doc_md, adds descriptions, tags and task docs that show in the Airflow UI, and checks every DAG still imports, in a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-dag-docs/airflow/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads what the DAGs already say about themselves. In a project like astronomer/airflow-example-dags, the `Dockerfile` is one line, `FROM quay.io/astronomer/astro-runtime:6.0.0`, `packages.txt` holds `postgresql`, `requirements.txt` is empty, and `airflow_settings.yaml` is the Astro CLI template with blank connections, pools and variables. Under `dags/`, `s3_upload_dag.py`, `s3_upload_copy_dag.py`, `s3_upload_copy_delete.py`, `databricks_sample.py` and `kube_pod.py` each open with a Markdown module docstring, a `###` heading and a paragraph, but none passes it to the DAG: `s3_upload_dag.py` builds `DAG('s3_upload', ...)` with `schedule_interval`, `max_active_runs=1` and `catchup=False` and never mentions `doc_md`, so the Airflow UI shows nothing on the DAG's page. `example_external_task_sensor.py` is entirely commented out. The tasks are built in loops, `generate_file_{name}_{i}` through `PythonOperator` calling `upload_to_s3`, `copy_{name}_file_{i}` through `S3CopyObjectOperator`, and in `databricks_sample.py` an `S3KeySensor` followed by `DatabricksRunNowOperator` and `PostgresOperator` per job. Owners are `airflow` or `Airflow`; no DAG has `tags` or `description`.

It wires the words that exist before adding new ones. Each DAG gets `doc_md=__doc__`, so the docstring that is already maintained at the top of the file renders on the DAG details page, plus a one-line `description` for the list view and `tags` such as `s3`, `databricks` and `kubernetes` so the UI filter works. The docstrings and comments are tidied where they disagree with the code: in `s3_upload_dag.py` the docstring says five files and the loop runs `range(0,5)`, but the comment on that loop says ten tasks, so the agent makes the comment match the `range` it reads. Owners become a team name read from the prompt rather than the default `airflow`, with the change applied through `default_args` so every task inherits it.

Tasks get their own notes where a reader would stop. The `upload_to_s3` function's docstring becomes the `doc_md` of each generated `PythonOperator`, the `S3KeySensor` in `databricks_sample.py` gets a `doc_md` naming the bucket, the wildcard key and the `timeout` it waits for, and the `KubernetesPodOperator` in `kube_pod.py` gets one explaining the `in_cluster` switch on the detected namespace and the resource requests. Operators built in a loop share a doc with the loop index interpolated, so each task's page says which file it handles. The commented-out `example_external_task_sensor.py` is left alone and named in the pull request as a file with no DAG to document.

Then it checks every DAG still parses. The runtime image pins the Airflow version, so the agent installs that Airflow with its constraints file into a virtual environment, runs `python -m py_compile` over `dags/`, then loads a `DagBag` from `dags/` and asserts `import_errors` is empty and each DAG's `doc_md`, `description` and `tags` are set, as a small `pytest` file under `tests/` that the pull request keeps. It reads the output before opening the pull request and does not start a scheduler, since the connections in `airflow_settings.yaml` are blank.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add documentation to the Airflow DAGs in this repository so it shows in the UI. Read Dockerfile, requirements.txt, packages.txt, airflow_settings.yaml and every file under dags/ first. For each DAG pass its module docstring as doc_md, add a one-line description and tags that match what it touches, and set the owner in default_args to the data-platform team. Fix a docstring that disagrees with the code it describes, such as a file count that does not match the loop. Add doc_md to tasks where a reader needs context: the upload function's docstring on each PythonOperator, the bucket, key pattern and timeout on the S3KeySensor, and the in_cluster logic and resources on the KubernetesPodOperator, interpolating the loop index where tasks are generated. Leave example_external_task_sensor.py, which is fully commented out, unchanged and say so. Install the Airflow version the astro-runtime image pins with its constraints file, run py_compile over dags/, and add a pytest under tests/ that loads a DagBag, asserts no import errors and asserts each DAG has doc_md, description and tags. Run it, then open a pull request that lists each DAG with its tags and the tasks that gained docs.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that edits the five DAG files under `dags/`, adds `tests/test_dag_docs.py`, and adds a `requirements-dev.txt` or an equivalent note naming the Airflow version and constraints it installed to run the check. The description holds a table of DAG id, description, tags and owner, lists the tasks that gained `doc_md`, quotes the comment correction it made, and names `example_external_task_sensor.py` as untouched.

The run's record keeps the `py_compile` output and the `pytest` run with the DagBag assertions named, so a reviewer can see that every DAG imported under the pinned Airflow and that the documentation fields were present on each.

Before merging, a reviewer runs `astro dev start`, opens one DAG's details page to see the rendered Markdown, clicks a generated task to see its doc, and filters the DAG list by a new tag.

## Run it on every DAG folder where the UI shows no documentation

The setup is one `POST /o/{org}/sessions` with this body and the repository in `resources`, started from a terminal; the run continues in its sandbox after the window closes, and a follow-up through `POST /o/{org}/sessions/{id}/messages` can ask for `owner_links` or a different tag scheme without starting over. A team with several Astro projects sends one request per repository, each with its own UUID as `request_id`, so a rerun of the script returns the run it already started.

Give the API key a `webhook_url` and Rush POSTs `run.completed` or `run.failed` when a run's turn finishes, with `session_url` so your server can read the pull request link and the test output.

## Why run it on Rush?

Documentation changes are the ones nobody tests, and a DAG file with a stray comma in `tags` or a `doc_md` keyword on an operator that does not accept it fails only when the scheduler parses it. The agent installs the exact Airflow the runtime image pins, loads the DagBag in its own sandbox, and reads the import errors, so the pull request is checked against the same version that will parse it.

The commands and their output stay with the run, so the pull request can point at the DagBag test that saw zero import errors and every documentation field set, and a reviewer only has to open one DAG page to see the result.

## Frequently asked questions

### Why doc_md=__doc__ instead of writing new text?

The module docstrings already exist and are maintained with the code; passing them to `doc_md` renders them on the DAG page without a second copy to keep in sync. The agent edits a docstring only where it disagrees with the code.

### Which Airflow version does it test against?

The one the `astro-runtime` tag in the `Dockerfile` ships. The agent reads the runtime's release notes for that mapping, installs that Airflow with its constraints file, and names both in the pull request rather than testing against whatever `pip` would pick today.

### Does it need my connections or an S3 bucket?

No. Loading a DagBag parses the files without running a task, and the agent starts no scheduler. The blank `airflow_settings.yaml` stays blank.

### Will the tasks generated in a loop each get the same doc?

They share a template with the loop index and file name interpolated, so `generate_file_workshop_3` says it writes file 3. The agent avoids a single generic paragraph that would be identical on every task page.

## More Airflow jobs

- [Add Params to Airflow DAGs](https://getrush.ai/run/add-dag-params/airflow/)
- [Add Timeouts to Airflow DAGs](https://getrush.ai/run/add-dag-timeouts/airflow/)
- [Add Data-Aware Scheduling to Airflow DAGs](https://getrush.ai/run/add-data-aware-scheduling/airflow/)

## More jobs like this: Hand off a long task

- [Add Dark Mode to a Blazor Web App](https://getrush.ai/run/add-dark-mode/blazor/)
- [Add a Dark Mode Toggle to an Eleventy Site](https://getrush.ai/run/add-dark-mode/eleventy/)
- [Add Dark Mode to a Jekyll Site](https://getrush.ai/run/add-dark-mode/jekyll/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
