What the agent does
The agent reads where concurrency is already decided before it adds a limit. In a project like dagster-io/hooli-data-eng-pipelines, hooli-data-eng/uv.lock resolves dagster and dagster-cloud to 1.12.1 and dagster-dbt to 0.28.1. dagster.yaml at the root holds only run_retries (enabled: false, max_retries: 1), and the Makefile copies it into ~/.dagster_home for stateful_dev. hooli-data-eng/src/hooli_data_eng/definitions.py merges the loaded defs with dg.multiprocess_executor.configured({"max_concurrent": 3}), which caps steps inside one run and says nothing about runs against each other. dagster_cloud.yaml lists seven code locations, with the data-eng-pipeline location's Kubernetes run pods limited to 1000m CPU and 2G memory. Nothing sets a pool, a tag concurrency limit or a max_concurrent_runs.
It then reads what the assets contend for. resources.py defines Snowflake IO managers for branch and prod deployments on a warehouse named TINY_WAREHOUSE, and the dbt component in defs/dbt/component.yaml splits the models into groups with daily and weekly partitions behind a refresh_analytics_model_job tagged dagster/max_retries: 1 and a dbt_components_job on 0 8 * * *. defs/databricks/assets.py submits a Databricks job with a new i3.xlarge cluster per materialization, defs/kubernetes/assets.py launches a pipes pod in data-eng-prod, defs/scikit_learn/assets.py fits a model and tracks error by MonthlyPartitionsDefinition, and defs/pandas/schedules.py has an asset sensor that launches predict_job whenever orders_augmented updates plus a freshness check job every ten minutes. A backfill of the daily users and orders partitions, a scheduled dbt run and the sensor can all land on the tiny warehouse at once.
The limits go in two layers. In dagster.yaml, the agent adds a concurrency section: runs.max_concurrent_runs so the run queue never launches more pods than the cluster's quota holds, runs.tag_concurrency_limits keyed on dagster/backfill so one backfill cannot take every slot from the schedules, and pools with a default_limit and granularity: op. On the assets it sets pool: snowflake on the dbt assets through the translator in defs/dbt/component.py and on the pandas and scikit-learn assets that read through the Snowflake IO manager, databricks on big_orders and databricks_asset, and k8s on k8s_pod_asset, each with a limit in the pool config that matches what the warehouse or cluster can serve. Because this project deploys to Dagster+, pool limits live in deployment settings rather than in dagster.yaml there, so the agent adds a deployment_settings.yaml with the same concurrency block and the dagster-cloud deployment settings set-from-file command to the README, and keeps dagster.yaml for local dagster dev.
The proof is that the definitions still load and the pools are visible. The agent runs uv sync in hooli-data-eng, dg check defs or dagster definitions validate against hooli_data_eng.definitions, and the existing hooli_data_eng_tests/test_repo_loads.py, which asserts refresh_analytics_model_job exists. It then starts dagster dev with DAGSTER_HOME pointing at a copy of the new dagster.yaml, as the Makefile does, and queries the GraphQL endpoint for the pool names and limits on each asset's op, including that output rather than a screenshot. It does not materialize anything against Snowflake or Databricks.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add concurrency limits to this Dagster project. Read dagster.yaml, the Definitions and executor configuration, every asset and job definition, the schedules and sensors, the resources and IO managers, and the deployment configuration first. Add a concurrency section to dagster.yaml with a run queue limit, a tag concurrency limit for backfills, and pools with limits; assign a pool to each asset that shares a warehouse, cluster or external service, including dbt assets through their translator. If the project deploys to Dagster+, also write the deployment settings file and the command that applies it, and document it. Run the definitions validation and the existing tests, start dagster dev against the new instance config and confirm the pools appear on the assets, and include that output. Do not materialize any asset against a real warehouse. Open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request with a concurrency section in dagster.yaml, a new deployment_settings.yaml for Dagster+ with the command to apply it in the README, a pool on each asset that touches Snowflake, Databricks or Kubernetes, and the dbt translator in defs/dbt/component.py returning the snowflake pool for every model. The executor's max_concurrent stays as it is, with a comment explaining the difference between step and run concurrency.
The description lists each pool with its limit and the assets in it, quotes the definitions validation and the test_repo_loads result, and shows the pool names read back from the running webserver. The commands and their output are in the transcript at GET /o/{org}/sessions/{id}/messages.
Before merging, a reviewer checks each limit against the real warehouse size and the Kubernetes namespace quota, decides whether the backfill tag limit should be lower, and applies the deployment settings to a branch deployment first. A follow-up message to the same session can add a pool for a new external service or change a limit.
Run it on every Dagster code location you deploy
A data platform team with several code locations in one repository, like the seven in this one, starts one run per location with one POST /o/{org}/sessions each, naming the location's directory in input, so each pull request is small enough to review in one sitting. A team with one repository per location sends the repository in resources instead.
A key runs at most max_concurrent runs at once, four by default; a run past that answers 202 with status: "queued" and starts when a slot frees. Send a request_id, a UUID v5 of the location name, so a retried call returns the run it already started.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
The right limits come from reading the whole project: which assets share a warehouse, which schedule overlaps which sensor, what the executor already caps and what the deployment target supports. A limit set from one file is either too loose to matter or tight enough to stall a backfill. The agent reads all of it in a sandbox of its own, writes the limits in the two places they must live, and shows the definitions loading with the pools attached.
The pull request carries the pool table and the validation output, so review is a check of numbers against capacity. Several code locations can be done at once, each in its own sandbox, each with its own pull request.
Frequently asked questions
What is the difference between the executor's max_concurrent and a pool?
multiprocess_executor with max_concurrent: 3 limits steps running in parallel inside one run. A pool limits steps across all runs that name it, so two runs and a backfill together cannot exceed the Snowflake pool's limit. max_concurrent_runs limits how many runs leave the queue at all.
Why does Dagster+ need a separate settings file?
On Dagster+ the run queue and pool limits are deployment settings managed by the platform, not read from dagster.yaml in the code location. The agent writes the same concurrency block into a settings file and documents dagster-cloud deployment settings set-from-file, and keeps dagster.yaml so local dagster dev behaves the same way.
How do dbt models get a pool?
Through the DagsterDbtTranslator subclass the project already uses for descriptions and metadata. The agent overrides the pool method so every dbt asset carries the snowflake pool without editing a model, and the limit is set once in the pool config.
Does this change any schedule or sensor?
No. Schedules and sensors keep firing as before; what changes is that their runs wait in the queue when the limits are reached instead of all starting at once. The description names the overlaps it found, such as a backfill coinciding with the morning dbt job, that the limits now serialize.