All jobs

Write Tests for Apache Airflow DAGs

Yes. An agent adds DAG integrity and structure tests to your Airflow project, runs them with the same Airflow version your runtime image ships, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the project and every DAG before it writes a test. In an Astro project like astronomer/airflow-example-dags, the Dockerfile is one line, FROM quay.io/astronomer/astro-runtime:6.0.0, requirements.txt is empty, packages.txt adds postgresql, airflow_settings.yaml holds empty placeholders for connections, pools and variables, and .drone.yml builds and pushes the image on master, dev and release-* branches without running anything. There is no tests/ directory. Under dags/, databricks_sample.py defines adb_pipeline on 30 4 * * * with catchup=False: a DummyOperator, an S3KeySensor on bucket astro-workshop-bucket with a wildcard key and a 600-second timeout, then four DatabricksRunNowOperator tasks, spark_job_one to spark_job_four, each followed by a PostgresOperator on prod_postgres. kube_pod.py defines example_kubernetes_pod with one KubernetesPodOperator and reads conf.get('kubernetes', 'NAMESPACE') at import time to decide in_cluster. s3_upload_dag.py, s3_upload_copy_dag.py and s3_upload_copy_delete.py build 5, 20 and 30 tasks in loops on the cron 0 12 8-14,22-28 * 6 with max_active_runs=1, one retry and a five-minute retry_delay; the task ids embed a name variable that is still your_name_here in one file. example_external_task_sensor.py is entirely commented out. All five import from airflow.operators.dummy_operator, airflow.operators.python_operator and airflow.operators.postgres_operator, the deprecated paths, and the Amazon, Databricks and Kubernetes providers come from the runtime image, not from requirements.txt.

It writes tests/dags/test_dag_integrity.py first: a DagBag over dags/ with examples off, an assertion that import_errors is empty, and for every DAG a check that it has no cycles, that every task has an owner and retries set, and that catchup and schedule_interval are what the file says. A conftest.py points AIRFLOW_HOME at a temporary directory, initialises the SQLite metadata database once, turns example DAGs off, and sets the kubernetes namespace setting so kube_pod.py imports outside a cluster the way it does on Astronomer.

Then one test module per DAG pins its structure: adb_pipeline has ten tasks, check_for_file is downstream of kick_off_dag, each spark_job_* task has exactly one downstream post_*_query task, and the sensor's poke_interval and timeout match the file. s3_upload has six tasks, s3_upload_copy twenty-one and s3_upload_copy_delete thirty-one, each generate_file_* task leads to its copy_* task and, in the last DAG, to a delete_* task, and the upload_to_s3 callable writes the file it uploads under the expected key when the S3Hook is replaced with a mock. The your_name_here task ids are asserted as they are and flagged in the pull request as a placeholder, not silently renamed. The import-time check on the deprecated operator paths is recorded as a warning count so an upgrade later can see it change.

To run the suite against the right Airflow, the agent installs Docker in its sandbox with sudo apt-get install -y docker.io and starts dockerd, pulls the runtime image from the Dockerfile, reads airflow version from it, and runs pytest tests/ inside the image with the project mounted, which is what astro dev pytest does. It also records the matching apache-airflow and provider versions so the same suite runs in a plain virtual environment. It adds a test step to .drone.yml that runs pytest in the freshly built image before the push step, and opens a pull request with the pytest output.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Write tests for the DAGs in this Airflow project and open a pull request. Read the Dockerfile, requirements.txt, packages.txt, airflow_settings.yaml, .drone.yml, .astro/config.yaml and every file under dags first. Add tests/dags/test_dag_integrity.py that loads a DagBag over dags with examples off, asserts no import errors, no cycles, an owner and retries on every task, and the catchup and schedule each file sets; add a conftest.py that points AIRFLOW_HOME at a temporary directory, initialises the metadata database once and sets the Kubernetes namespace setting the kube_pod DAG reads at import. Add one test module per DAG that pins task counts, task ids and dependencies as the files define them, and tests the upload_to_s3 callable with the S3Hook mocked. Flag placeholder task ids in the pull request rather than renaming them. The sandbox has no Docker: install docker.io and start dockerd, pull the runtime image from the Dockerfile, read its Airflow version and run pytest inside it with the project mounted. Add a pytest step to .drone.yml before the push step. Change no DAG code. Open a pull request with the pytest output and the Airflow and provider versions the tests ran on.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request on its own branch that adds tests/conftest.py, tests/dags/test_dag_integrity.py and one test module per DAG, and adds a test step to .drone.yml. The description quotes the pytest summary with every test named, the airflow version read from the runtime image, the deprecation warnings the DAG imports produce, and a short list of what the tests pin for each DAG, with the your_name_here task ids called out as a placeholder for the team to resolve.

Before merging, a reviewer checks that the structure tests pin what the team means to keep stable, such as task counts in the looped DAGs, rather than incidental details, and whether the Drone step should run on pull requests as well as the branches it builds today. They also decide what to do about the commented-out external task sensor DAG, which the tests ignore.

Run it on every new or changed DAG

Add an automation to the Project with a github trigger on pull_request.opened and a saved prompt that asks for tests for the DAGs the pull request adds or changes under dags/; the trigger appends the pull request event after the prompt, so the agent reads the changed files from it. Each run opens its own pull request against the feature branch with tests for just those DAGs, so a new pipeline arrives with its integrity and structure tests rather than months later.

For the backlog, a schedule trigger once a week with the same prompt picks the next untested DAGs each time. Give the API key a webhook URL and read run.completed to post each pull request link where the team reviews tests.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

DAG tests fail for reasons that have nothing to do with the DAGs: a provider missing from the virtual environment because it comes with the runtime image, an Airflow version that differs from production, a config section a file reads at import time, a metadata database that was never initialised. On Rush the agent runs the suite inside the same image the pipeline builds, in its own sandbox with Docker installed, and the transcript shows each failed import and the setting that fixed it.

Because the result is a pull request with the pytest output and the image's Airflow version in its record, the review is about what the tests pin and not whether they run. The same request runs on every Airflow project a team owns at once, each in its own sandbox, without a local Airflow install on anyone's laptop.

Frequently asked questions

Do the tests need AWS, Databricks or Kubernetes credentials?

No. A DagBag parses the files and builds the task graph without talking to any connection, and the upload_to_s3 test replaces S3Hook with a mock. The connections named in the DAGs, such as astro-s3-workshop and prod_postgres, never have to exist for the suite to pass.

Why run pytest inside the runtime image?

Because requirements.txt is empty and the Amazon, Databricks and Kubernetes providers the DAGs import come from astro-runtime:6.0.0, so a plain virtual environment would need the exact Airflow and provider versions that image ships. Running inside the image is what astro dev pytest does, and the pull request records the versions so a virtual environment can match them.

Will it fix the deprecated operator imports?

No. The prompt forbids changing DAG code, so the tests load the files as they are and record the deprecation warnings. Moving to airflow.operators.empty, airflow.operators.python and the Postgres provider's operator is a separate change the pull request lists as a follow-up.

What about the DAG that is fully commented out?

example_external_task_sensor.py defines nothing, so the DagBag ignores it and no test refers to it. The pull request names it so the team can delete it or restore it; either way the integrity test keeps passing.

More Apache Airflow jobs

More jobs like this: Write the missing tests

Keep reading