# Add Concurrency Limits to a Dagster Deployment

Yes. An agent adds run queue limits and concurrency pools to your Dagster project, tags the assets that share a warehouse or cluster, and opens a PR with the checks.

[Run this on Rush](https://cloud.getrush.ai/run/add-concurrency-limits/dagster/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads where concurrency is already decided before it adds a limit. In a project like dagster-io/hooli-data-eng-pipelines, `hooli-data-eng/uv.lock` resolves `dagster` and `dagster-cloud` to 1.12.1 and `dagster-dbt` to 0.28.1. `dagster.yaml` at the root holds only `run_retries` (`enabled: false`, `max_retries: 1`), and the `Makefile` copies it into `~/.dagster_home` for `stateful_dev`. `hooli-data-eng/src/hooli_data_eng/definitions.py` merges the loaded `defs` with `dg.multiprocess_executor.configured({"max_concurrent": 3})`, which caps steps inside one run and says nothing about runs against each other. `dagster_cloud.yaml` lists seven code locations, with the `data-eng-pipeline` location's Kubernetes run pods limited to `1000m` CPU and `2G` memory. Nothing sets a pool, a tag concurrency limit or a `max_concurrent_runs`.

It then reads what the assets contend for. `resources.py` defines Snowflake IO managers for branch and prod deployments on a warehouse named `TINY_WAREHOUSE`, and the dbt component in `defs/dbt/component.yaml` splits the models into groups with daily and weekly partitions behind a `refresh_analytics_model_job` tagged `dagster/max_retries: 1` and a `dbt_components_job` on `0 8 * * *`. `defs/databricks/assets.py` submits a Databricks job with a new `i3.xlarge` cluster per materialization, `defs/kubernetes/assets.py` launches a pipes pod in `data-eng-prod`, `defs/scikit_learn/assets.py` fits a model and tracks error by `MonthlyPartitionsDefinition`, and `defs/pandas/schedules.py` has an asset sensor that launches `predict_job` whenever `orders_augmented` updates plus a freshness check job every ten minutes. A backfill of the daily `users` and `orders` partitions, a scheduled dbt run and the sensor can all land on the tiny warehouse at once.

The limits go in two layers. In `dagster.yaml`, the agent adds a `concurrency` section: `runs.max_concurrent_runs` so the run queue never launches more pods than the cluster's quota holds, `runs.tag_concurrency_limits` keyed on `dagster/backfill` so one backfill cannot take every slot from the schedules, and `pools` with a `default_limit` and `granularity: op`. On the assets it sets `pool`: `snowflake` on the dbt assets through the translator in `defs/dbt/component.py` and on the pandas and scikit-learn assets that read through the Snowflake IO manager, `databricks` on `big_orders` and `databricks_asset`, and `k8s` on `k8s_pod_asset`, each with a limit in the pool config that matches what the warehouse or cluster can serve. Because this project deploys to Dagster+, pool limits live in deployment settings rather than in `dagster.yaml` there, so the agent adds a `deployment_settings.yaml` with the same `concurrency` block and the `dagster-cloud deployment settings set-from-file` command to the README, and keeps `dagster.yaml` for local `dagster dev`.

The proof is that the definitions still load and the pools are visible. The agent runs `uv sync` in `hooli-data-eng`, `dg check defs` or `dagster definitions validate` against `hooli_data_eng.definitions`, and the existing `hooli_data_eng_tests/test_repo_loads.py`, which asserts `refresh_analytics_model_job` exists. It then starts `dagster dev` with `DAGSTER_HOME` pointing at a copy of the new `dagster.yaml`, as the `Makefile` does, and queries the GraphQL endpoint for the pool names and limits on each asset's op, including that output rather than a screenshot. It does not materialize anything against Snowflake or Databricks.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add concurrency limits to this Dagster project. Read dagster.yaml, the Definitions and executor configuration, every asset and job definition, the schedules and sensors, the resources and IO managers, and the deployment configuration first. Add a concurrency section to dagster.yaml with a run queue limit, a tag concurrency limit for backfills, and pools with limits; assign a pool to each asset that shares a warehouse, cluster or external service, including dbt assets through their translator. If the project deploys to Dagster+, also write the deployment settings file and the command that applies it, and document it. Run the definitions validation and the existing tests, start dagster dev against the new instance config and confirm the pools appear on the assets, and include that output. Do not materialize any asset against a real warehouse. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request with a `concurrency` section in `dagster.yaml`, a new `deployment_settings.yaml` for Dagster+ with the command to apply it in the README, a `pool` on each asset that touches Snowflake, Databricks or Kubernetes, and the dbt translator in `defs/dbt/component.py` returning the `snowflake` pool for every model. The executor's `max_concurrent` stays as it is, with a comment explaining the difference between step and run concurrency.

The description lists each pool with its limit and the assets in it, quotes the definitions validation and the `test_repo_loads` result, and shows the pool names read back from the running webserver. The commands and their output are in the transcript at `GET /o/{org}/sessions/{id}/messages`.

Before merging, a reviewer checks each limit against the real warehouse size and the Kubernetes namespace quota, decides whether the backfill tag limit should be lower, and applies the deployment settings to a branch deployment first. A follow-up message to the same session can add a pool for a new external service or change a limit.

## Run it on every Dagster code location you deploy

A data platform team with several code locations in one repository, like the seven in this one, starts one run per location with one `POST /o/{org}/sessions` each, naming the location's directory in `input`, so each pull request is small enough to review in one sitting. A team with one repository per location sends the repository in `resources` instead.

A key runs at most `max_concurrent` runs at once, four by default; a run past that answers `202` with `status: "queued"` and starts when a slot frees. Send a `request_id`, a UUID v5 of the location name, so a retried call returns the run it already started.

## Why run it on Rush?

The right limits come from reading the whole project: which assets share a warehouse, which schedule overlaps which sensor, what the executor already caps and what the deployment target supports. A limit set from one file is either too loose to matter or tight enough to stall a backfill. The agent reads all of it in a sandbox of its own, writes the limits in the two places they must live, and shows the definitions loading with the pools attached.

The pull request carries the pool table and the validation output, so review is a check of numbers against capacity. Several code locations can be done at once, each in its own sandbox, each with its own pull request.

## Frequently asked questions

### What is the difference between the executor's max_concurrent and a pool?

`multiprocess_executor` with `max_concurrent: 3` limits steps running in parallel inside one run. A pool limits steps across all runs that name it, so two runs and a backfill together cannot exceed the Snowflake pool's limit. `max_concurrent_runs` limits how many runs leave the queue at all.

### Why does Dagster+ need a separate settings file?

On Dagster+ the run queue and pool limits are deployment settings managed by the platform, not read from `dagster.yaml` in the code location. The agent writes the same `concurrency` block into a settings file and documents `dagster-cloud deployment settings set-from-file`, and keeps `dagster.yaml` so local `dagster dev` behaves the same way.

### How do dbt models get a pool?

Through the `DagsterDbtTranslator` subclass the project already uses for descriptions and metadata. The agent overrides the pool method so every dbt asset carries the `snowflake` pool without editing a model, and the limit is set once in the pool config.

### Does this change any schedule or sensor?

No. Schedules and sensors keep firing as before; what changes is that their runs wait in the queue when the limits are reached instead of all starting at once. The description names the overlaps it found, such as a backfill coinciding with the morning dbt job, that the limits now serialize.

## More Dagster jobs

- [Set Up Dependabot for a Dagster Workspace with uv](https://getrush.ai/run/add-dependabot/dagster/)
- [Set Up GitHub Actions CI for a Dagster Project](https://getrush.ai/run/add-github-actions-ci/dagster/)
- [Add Retry Policies to the Flaky Assets in a Dagster Project](https://getrush.ai/run/add-retry-policies/dagster/)

## More jobs like this: Hand off a long task

- [Add a Config File to a Go CLI](https://getrush.ai/run/add-config-file/go-cli/)
- [Add a Config File to a Textual App](https://getrush.ai/run/add-config-file/textual/)
- [Move Deployment Environment Literals Into Kustomize Generators](https://getrush.ai/run/add-configmap-generator/kustomize/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
