All jobs

Add Pools to Limit Concurrent Tasks in Airflow

Yes. An agent defines Airflow pools for your S3 and Databricks tasks, assigns every task that hits those services, adds a DAG test, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the DAGs to find the fan-outs that hammer one external service. In a project like astronomer/airflow-example-dags, dags/s3_upload_copy_delete.py loops for i in range(0,10) and builds, for each i, a PythonOperator that calls S3Hook.load_file through the astro-s3-workshop connection, an S3CopyObjectOperator and an S3DeleteObjectsOperator, all fanned out from one DummyOperator named start, with max_active_runs=1 on the DAG and KubernetesExecutor resource requests in test_config2. dags/s3_upload_dag.py does the same upload five times. dags/databricks_sample.py waits on an S3KeySensor with poke_interval=45 and timeout=600, then starts four DatabricksRunNowOperator tasks against the adb_workspace connection, each followed by a PostgresOperator on prod_postgres. None of the tasks names a pool, so all of them share the default pool and the only limit on concurrent S3 or Databricks calls is the executor's. airflow_settings.yaml holds the Astro CLI's empty template for connections, pools and variables, Dockerfile is FROM quay.io/astronomer/astro-runtime:6.0.0, requirements.txt is empty and packages.txt adds postgresql.

The agent defines three pools and assigns them where the calls happen. airflow_settings.yaml gets s3_workshop (a few slots, for the hook, copy and delete tasks), databricks_jobs (two slots, one short of the four jobs so the fan-out is visibly queued), and sensors (one slot), each with a pool_description, so astro dev start creates them locally. Every S3 task in both S3 DAGs gets pool='s3_workshop', the four DatabricksRunNowOperator tasks get pool='databricks_jobs', and the S3KeySensor gets pool='sensors' with mode='reschedule', so it gives its slot back between pokes instead of holding it for up to the 600-second timeout. The PostgresOperator tasks stay in the default pool, because the four small queries are not the bottleneck.

Pools in airflow_settings.yaml exist only for the local Astro CLI, so the agent also adds include/pools.json in the shape airflow pools import reads, with the same three pools, and a README note on importing it in a deployment or creating the pools in the UI under Admin, Pools. A task that names a pool the scheduler cannot find does not run and the scheduler logs why, which the note says plainly.

The test is a DAG-level check rather than a run. The agent adds tests/dags/test_pools.py, which loads every file under dags/ into a DagBag with include_examples=False, asserts there are no import errors, and asserts that every task whose operator is one of the S3 operators or whose callable is upload_to_s3 has pool == 's3_workshop', every DatabricksRunNowOperator has pool == 'databricks_jobs', and the sensor is in sensors with reschedule mode. In its sandbox the agent installs the Airflow release that runtime 6.0.0 ships, with the amazon, databricks, postgres and cncf-kubernetes providers, runs pytest tests/dags and airflow dags list-import-errors, and includes the output; the commented-out example_external_task_sensor.py imports nothing and is left alone.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add Airflow pools to this project so tasks that call S3 and Databricks are limited in how many run at once. Read every file under dags/, airflow_settings.yaml, Dockerfile, requirements.txt and packages.txt first. Define pools in airflow_settings.yaml with descriptions: one for S3 tasks, one for Databricks jobs with fewer slots than the fan-out, and one for sensors. Set pool on every S3 hook, copy and delete task in both S3 DAGs, on every DatabricksRunNowOperator, and on the S3KeySensor, switching the sensor to reschedule mode. Leave the Postgres tasks in the default pool. Add include/pools.json in the format airflow pools import reads and a README note on creating the pools in a deployment. Add tests/dags/test_pools.py that loads the DagBag, asserts no import errors, and asserts the pool of each task by operator type. Install the Airflow version the runtime image ships with the needed providers, run the tests and airflow dags list-import-errors, include the output, and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that changes airflow_settings.yaml (three pools), dags/s3_upload_copy_delete.py and dags/s3_upload_dag.py (pool= on each S3 task), dags/databricks_sample.py (pool= on the sensor and the Databricks tasks, mode='reschedule' on the sensor), and adds include/pools.json, tests/dags/test_pools.py and a README note. Dockerfile, requirements.txt, packages.txt, kube_pod.py and the task ids and dependencies are unchanged.

The description lists each pool with its slots and the tasks assigned, quotes the pytest summary and the empty import-error list, and says how to create the pools in a deployment where airflow_settings.yaml is not read. The full command output is in the transcript at GET /o/{org}/sessions/{id}/messages.

Before merging, a reviewer sets the slot counts to the limits the S3 bucket and the Databricks workspace actually tolerate, and creates the pools in the deployment before the DAGs with pool= reach it. A follow-up message to the same session can add priority_weight so the copy and delete tasks go ahead of new uploads.

Run it on every Airflow project with fan-out tasks and no pools

A data platform team with several Astro projects sends one POST /o/{org}/sessions per repository, with this prompt as input and the repository in resources. Each run reads that project's own DAGs, so a project whose fan-out is a Snowflake operator gets a snowflake pool and the right operator class in the test, rather than the S3 and Databricks pools of this example.

A request_id that is a UUID v5 of the repository's full name makes a retried call return the run already started. The key's webhook_url reports run.completed with the session_url, so a handler can collect the pull request links for the team's review queue.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

Pools are easy to describe and tedious to apply: every operator in every loop needs the argument, the sensor needs its mode changed or it holds a slot while sleeping, and nothing in a DAG file tells you whether the pool exists where the DAG will run. The agent makes the edits, writes the test that keeps them true, and runs the DagBag check with the matching Airflow release in a sandbox of its own.

The pull request carries the test and the import check, so a reviewer decides slot counts rather than reading loops for a missing argument. Several projects can go through at once, each in its own sandbox with its own pull request.

Frequently asked questions

What happens if the pool does not exist in production?

Tasks that name it are not scheduled and the scheduler logs that the pool is missing, so the DAG stalls rather than failing loudly. The pull request includes include/pools.json for airflow pools import and tells the reviewer to create the pools before merging; airflow_settings.yaml only applies to the local Astro CLI.

Why put the sensor in its own pool and switch it to reschedule mode?

In the default poke mode a sensor holds its worker slot for the whole wait, up to its 600-second timeout here, and a few of them can fill a pool meant for real work. In reschedule mode the task frees its slot between pokes, and a one-slot sensors pool keeps sensors from competing with the S3 and Databricks tasks at all.

Is this different from max_active_tasks on the DAG?

Yes. max_active_tasks limits one DAG's running tasks regardless of what they call; a pool limits every task across every DAG that names it. Two DAGs uploading to the same bucket share the s3_workshop pool, which is the limit that matters to the bucket.

Will the task ids or dependencies change?

No. Each task keeps its id and its place in the t0 >> generate >> copy >> delete chain or the t1 >> t2 >> spark >> query chain; only the pool argument and the sensor's mode are added, so history and the UI graph read the same.

More Airflow jobs

More jobs like this: Hand off a long task

Keep reading