# Add Pools to Limit Concurrent Tasks in Airflow

Yes. An agent defines Airflow pools for your S3 and Databricks tasks, assigns every task that hits those services, adds a DAG test, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-pools/airflow/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads the DAGs to find the fan-outs that hammer one external service. In a project like astronomer/airflow-example-dags, `dags/s3_upload_copy_delete.py` loops `for i in range(0,10)` and builds, for each `i`, a `PythonOperator` that calls `S3Hook.load_file` through the `astro-s3-workshop` connection, an `S3CopyObjectOperator` and an `S3DeleteObjectsOperator`, all fanned out from one `DummyOperator` named `start`, with `max_active_runs=1` on the DAG and KubernetesExecutor resource requests in `test_config2`. `dags/s3_upload_dag.py` does the same upload five times. `dags/databricks_sample.py` waits on an `S3KeySensor` with `poke_interval=45` and `timeout=600`, then starts four `DatabricksRunNowOperator` tasks against the `adb_workspace` connection, each followed by a `PostgresOperator` on `prod_postgres`. None of the tasks names a pool, so all of them share the default pool and the only limit on concurrent S3 or Databricks calls is the executor's. `airflow_settings.yaml` holds the Astro CLI's empty template for `connections`, `pools` and `variables`, `Dockerfile` is `FROM quay.io/astronomer/astro-runtime:6.0.0`, `requirements.txt` is empty and `packages.txt` adds `postgresql`.

The agent defines three pools and assigns them where the calls happen. `airflow_settings.yaml` gets `s3_workshop` (a few slots, for the hook, copy and delete tasks), `databricks_jobs` (two slots, one short of the four jobs so the fan-out is visibly queued), and `sensors` (one slot), each with a `pool_description`, so `astro dev start` creates them locally. Every S3 task in both S3 DAGs gets `pool='s3_workshop'`, the four `DatabricksRunNowOperator` tasks get `pool='databricks_jobs'`, and the `S3KeySensor` gets `pool='sensors'` with `mode='reschedule'`, so it gives its slot back between pokes instead of holding it for up to the 600-second timeout. The `PostgresOperator` tasks stay in the default pool, because the four small queries are not the bottleneck.

Pools in `airflow_settings.yaml` exist only for the local Astro CLI, so the agent also adds `include/pools.json` in the shape `airflow pools import` reads, with the same three pools, and a `README` note on importing it in a deployment or creating the pools in the UI under Admin, Pools. A task that names a pool the scheduler cannot find does not run and the scheduler logs why, which the note says plainly.

The test is a DAG-level check rather than a run. The agent adds `tests/dags/test_pools.py`, which loads every file under `dags/` into a `DagBag` with `include_examples=False`, asserts there are no import errors, and asserts that every task whose operator is one of the S3 operators or whose callable is `upload_to_s3` has `pool == 's3_workshop'`, every `DatabricksRunNowOperator` has `pool == 'databricks_jobs'`, and the sensor is in `sensors` with reschedule mode. In its sandbox the agent installs the Airflow release that runtime `6.0.0` ships, with the `amazon`, `databricks`, `postgres` and `cncf-kubernetes` providers, runs `pytest tests/dags` and `airflow dags list-import-errors`, and includes the output; the commented-out `example_external_task_sensor.py` imports nothing and is left alone.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add Airflow pools to this project so tasks that call S3 and Databricks are limited in how many run at once. Read every file under dags/, airflow_settings.yaml, Dockerfile, requirements.txt and packages.txt first. Define pools in airflow_settings.yaml with descriptions: one for S3 tasks, one for Databricks jobs with fewer slots than the fan-out, and one for sensors. Set pool on every S3 hook, copy and delete task in both S3 DAGs, on every DatabricksRunNowOperator, and on the S3KeySensor, switching the sensor to reschedule mode. Leave the Postgres tasks in the default pool. Add include/pools.json in the format airflow pools import reads and a README note on creating the pools in a deployment. Add tests/dags/test_pools.py that loads the DagBag, asserts no import errors, and asserts the pool of each task by operator type. Install the Airflow version the runtime image ships with the needed providers, run the tests and airflow dags list-import-errors, include the output, and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that changes `airflow_settings.yaml` (three pools), `dags/s3_upload_copy_delete.py` and `dags/s3_upload_dag.py` (`pool=` on each S3 task), `dags/databricks_sample.py` (`pool=` on the sensor and the Databricks tasks, `mode='reschedule'` on the sensor), and adds `include/pools.json`, `tests/dags/test_pools.py` and a README note. `Dockerfile`, `requirements.txt`, `packages.txt`, `kube_pod.py` and the task ids and dependencies are unchanged.

The description lists each pool with its slots and the tasks assigned, quotes the `pytest` summary and the empty import-error list, and says how to create the pools in a deployment where `airflow_settings.yaml` is not read. The full command output is in the transcript at `GET /o/{org}/sessions/{id}/messages`.

Before merging, a reviewer sets the slot counts to the limits the S3 bucket and the Databricks workspace actually tolerate, and creates the pools in the deployment before the DAGs with `pool=` reach it. A follow-up message to the same session can add `priority_weight` so the copy and delete tasks go ahead of new uploads.

## Run it on every Airflow project with fan-out tasks and no pools

A data platform team with several Astro projects sends one `POST /o/{org}/sessions` per repository, with this prompt as `input` and the repository in `resources`. Each run reads that project's own DAGs, so a project whose fan-out is a Snowflake operator gets a `snowflake` pool and the right operator class in the test, rather than the S3 and Databricks pools of this example.

A `request_id` that is a UUID v5 of the repository's full name makes a retried call return the run already started. The key's `webhook_url` reports `run.completed` with the `session_url`, so a handler can collect the pull request links for the team's review queue.

## Why run it on Rush?

Pools are easy to describe and tedious to apply: every operator in every loop needs the argument, the sensor needs its mode changed or it holds a slot while sleeping, and nothing in a DAG file tells you whether the pool exists where the DAG will run. The agent makes the edits, writes the test that keeps them true, and runs the DagBag check with the matching Airflow release in a sandbox of its own.

The pull request carries the test and the import check, so a reviewer decides slot counts rather than reading loops for a missing argument. Several projects can go through at once, each in its own sandbox with its own pull request.

## Frequently asked questions

### What happens if the pool does not exist in production?

Tasks that name it are not scheduled and the scheduler logs that the pool is missing, so the DAG stalls rather than failing loudly. The pull request includes `include/pools.json` for `airflow pools import` and tells the reviewer to create the pools before merging; `airflow_settings.yaml` only applies to the local Astro CLI.

### Why put the sensor in its own pool and switch it to reschedule mode?

In the default poke mode a sensor holds its worker slot for the whole wait, up to its 600-second timeout here, and a few of them can fill a pool meant for real work. In reschedule mode the task frees its slot between pokes, and a one-slot `sensors` pool keeps sensors from competing with the S3 and Databricks tasks at all.

### Is this different from max_active_tasks on the DAG?

Yes. `max_active_tasks` limits one DAG's running tasks regardless of what they call; a pool limits every task across every DAG that names it. Two DAGs uploading to the same bucket share the `s3_workshop` pool, which is the limit that matters to the bucket.

### Will the task ids or dependencies change?

No. Each task keeps its id and its place in the `t0 >> generate >> copy >> delete` chain or the `t1 >> t2 >> spark >> query` chain; only the `pool` argument and the sensor's `mode` are added, so history and the UI graph read the same.

## More Airflow jobs

- [Add Retry Policies With Exponential Backoff to Airflow DAGs](https://getrush.ai/run/add-retry-policies/airflow/)
- [Move Airflow Connections to a Secrets Backend](https://getrush.ai/run/add-secrets-backend/airflow/)
- [Add Slack Alerts on Task Failure to Airflow DAGs](https://getrush.ai/run/add-slack-failure-alerts/airflow/)

## More jobs like this: Hand off a long task

- [Add a Post-Deploy Smoke Test to a Fly.io Workflow](https://getrush.ai/run/add-post-deploy-smoke-test/fly/)
- [Add pre-commit Hooks to an Ansible Role](https://getrush.ai/run/add-pre-commit-hooks/ansible/)
- [Add pre-commit Hooks to a CMake C++ Project](https://getrush.ai/run/add-pre-commit-hooks/cmake/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
