# Add Retry Policies to the Flaky Assets in a Dagster Project

Yes. An agent finds the Dagster assets that call external systems without a RetryPolicy, adds one sized to each failure, repairs the tests and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-retry-policies/dagster/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent clones the project and reads every asset that leaves the process. In a project like dagster-io/hooli-data-eng-pipelines, `hooli-data-eng/src/hooli_data_eng/defs/custom_ingest/assets.py` defines two daily-partitioned assets on a `RawDataAPI` resource whose `flaky` flag, on by default in `resources.py`, makes `get_orders` raise `API time out` whenever a random draw from 0 to 10 lands at or below 4, and `get_users` when it lands at or below 2. `orders` already carries `retry_policy=dg.RetryPolicy(max_retries=3, delay=1, backoff=dg.Backoff.LINEAR, jitter=dg.Jitter.FULL)`; `users`, which calls the same resource, has none. `defs/kubernetes/assets.py` launches a pod through `PipesK8sClient` and `defs/databricks/assets.py` submits a one-node cluster job through `PipesDatabricksClient`, both without a policy, while `defs/pandas/assets.py` and `defs/scikit_learn/assets.py` only compute on data frames the IO manager loads. The repository's `dagster.yaml` has `run_retries` with `enabled: false` and `max_retries: 1`, and `hooli-batch-enrichment` shows the other shape, `@op(retry_policy=RetryPolicy(max_retries=2))` and a job tagged `dagster/max_retries: "1"`.

It then adds policies that match each failure rather than one global number. `users` gets the same linear policy as `orders`, because it fails for the same reason at a lower rate. The two Pipes assets get `max_retries=2` with `backoff=dg.Backoff.EXPONENTIAL` and `jitter=dg.Jitter.PLUS_MINUS` and a longer `delay`, since a Databricks cluster or a Kubernetes pod that failed to start is not helped by an immediate second attempt, and the agent adds a comment on each saying why the external job is safe to resubmit. For an error that should never be retried, such as a schema change the column schema checks in the same file would catch, it raises `dg.Failure` with `allow_retries=False` so the policy is skipped. The pandas and scikit-learn assets are left alone with a one-line note: they fail deterministically or not at all.

It treats run-level retries as a separate question and answers it in the pull request. Switching `run_retries.enabled` to true in `dagster.yaml` would re-run whole jobs from where they stopped, which overlaps with the op policies and doubles the cost of a genuinely broken partition, so the agent leaves it off and instead tags the scheduled analytics job with `dagster/max_retries` only if the team asks. The reasoning is in the description, with the file and the line.

To prove the change, it installs the project with `uv sync` inside `hooli-data-eng`, then fixes the test module before adding to it: `hooli_data_eng_tests/test_assets.py` imports `hooli_data_eng.assets.raw_data` and `hooli_data_eng.resources.api`, paths that no longer exist since the move to `defs/`, so the tests cannot even be collected. It points them at `defs.custom_ingest`, keeps their `RawDataAPI(flaky=False)` setup, and adds a test that materializes `users` with `flaky=True` and a resource stub that fails a fixed number of times, asserting the asset succeeds within the policy's retries and that `allow_retries=False` ends it at once. It runs `pytest` and `dg check defs`, which loads every code location in `dg.toml`, and the `ruff check` the deploy workflow runs.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add retry policies to the Dagster assets in this project that call external systems. Read every assets.py and resources.py under the code locations, dagster.yaml and the tests first, and list each asset as external or in-process with the failure it can hit. Add a RetryPolicy to each external asset that lacks one, sized to its failure: short linear retries for a flaky API, fewer exponential retries with jitter for jobs submitted to another system, and raise Failure with allow_retries=False where a retry can never help. Do not enable run-level retries; explain the choice. Repair any test that imports a path that no longer exists, add a test that proves an asset recovers within its policy, run pytest, dg check defs and ruff check, and open a pull request with the asset table and the command output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request changing the custom ingest, Kubernetes and Databricks asset files, the test module, and nothing in `dagster.yaml`. The description has a table of every asset, whether it is external or in-process, the policy it now has and why, and a paragraph on run-level retries and why they stay off.

The run's record keeps the `uv sync`, the `pytest` run in which the repaired tests collect and pass, the `dg check defs` load of every code location, and the `ruff check` output, so a reviewer can see the retry behaviour asserted rather than described.

Before merging, a reviewer confirms with the owners of the Databricks and Kubernetes jobs that resubmitting a failed job is safe, decides on the delay values for those two assets, and agrees whether the scheduled jobs should also carry `dagster/max_retries`.

## Run it on every Dagster code location

A data platform engineer hands this off from the terminal: one `POST /o/{org}/sessions` with this body and the repository in `resources`. A workspace with several code locations in `dg.toml` gets one run per location, each with a `request_id` that is a UUID v5 of the location name, so a retried request returns the run it already started, and the key's `webhook_url` reports each finished run with its pull request.

Every run reads its own location's assets and resources and decides afresh which calls leave the process. An in-process asset never gets a policy just because its neighbour has one.

## Why run it on Rush?

The hard part of retries is the classification, not the decorator: which assets talk to something that can be flaky, whether a second attempt is safe, and where a retry would only hide a real break. The agent reads every asset and resource in its own sandbox, makes that table explicit, and proves the policy with a test that fails without it.

On Rush the proof stays with the change. A reviewer opens the pull request and the run's record side by side, sees the tests collect for the first time since the project moved to `defs/`, and can send a follow-up to the same run to change a delay or add a location without starting over.

## Frequently asked questions

### Why not turn on run_retries in dagster.yaml?

Because run-level retries restart a whole run after any failure, including the ones the op policies already handle, and a partition that is truly broken then fails twice at full cost. Op policies target the assets that can recover. The pull request explains the choice and shows the one-line change if the team wants both.

### Is retrying a Databricks or Kubernetes job safe?

Only if the job can be submitted again without side effects, which the agent cannot know from the code alone. It adds the policy with a longer delay, writes a comment asking that question at the call site, and lists it as the first thing for the reviewer to confirm.

### What about errors that should never be retried?

The agent raises `dg.Failure` with `allow_retries=False` for those, such as a response whose schema no longer matches, so the asset fails immediately and the retry policy is bypassed. The test it adds covers that path too.

### Why did it touch tests that were not about retries?

Because they did not import. The test module referenced the old `assets.raw_data` module path, so pytest could not collect it, and a new retry test next to broken tests would prove nothing. Repairing the imports is the smallest change that makes the suite real again.

## More Dagster jobs

- [Add a Schedule to a Dagster Project](https://getrush.ai/run/add-schedules/dagster/)
- [Add a Sensor to a Dagster Project](https://getrush.ai/run/add-sensors/dagster/)
- [Add Slack Alerts on Failed Runs to a Dagster Project](https://getrush.ai/run/add-slack-failure-alerts/dagster/)

## The same job on other stacks

- [Turn On Safe Retries for Event-Driven Firebase Cloud Functions](https://getrush.ai/run/add-retry-policies/firebase-functions/)
- [Add Retry Policies to Oban Workers in an Elixir App](https://getrush.ai/run/add-retry-policies/oban/)
- [Add Retry Policies to Sidekiq Jobs in a Rails App](https://getrush.ai/run/add-retry-policies/sidekiq/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
