# Upgrade Python Dependencies Weekly for a Dagster Project

Yes. Each week an agent refreshes every code location's uv.lock, moves Dagster and its libraries together, reloads the definitions and opens one pull request.

[Run this on Rush](https://cloud.getrush.ai/run/upgrade-python-dependencies/dagster/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent first works out where versions are actually decided, because in a Dagster workspace it is rarely one file. In a workspace like dagster-io/hooli-data-eng-pipelines there are seven projects, each with its own `pyproject.toml` and `uv.lock`. The main one, `hooli-data-eng`, lists `dagster`, `dagster-dbt`, `dbt-core`, `dagster-snowflake` and some twenty other packages, all but `ruff` with no version at all, so the lockfile is the only pin, and each project's Dockerfile installs from it with `uv sync --frozen`. The Makefile has an `update_packages` target that runs `uv lock --upgrade --directory` for all seven projects in one go. There is no Dependabot or Renovate config under `.github`.

Running that target moves everything at once, which is why upgrades in a repository like this tend to be put off. The agent splits the work. In each project it runs `uv lock --upgrade` on a scratch copy to see what would move, then applies the changes in groups with `uv lock --upgrade-package`. Dagster core and its libraries are one group and cannot be split: the lock shows `dagster` 1.12.14 beside `dagster-dbt` 0.28.14, and each library release requires the core release it shipped with. `dbt-core` and its DuckDB and Snowflake adapters are a second group. Everything else is a third. Each group is its own commit per project.

After each group it proves the code location still loads. It runs `uv sync --extra dev`, parses the dbt project against the DuckDB target (`dbt parse --project-dir=dbt_project --target LOCAL`), sets `DAGSTER_IS_DEV_CLI` so the project picks its local resources, and runs `dg check defs`, `ruff check` and pytest, comparing against the same commands on the untouched branch. This is where a Dagster upgrade usually fails. Three files here import `FreshnessPolicy` from `dagster.preview.freshness`, a preview module, and `pandas/assets.py` imports `build_anomaly_detection_freshness_checks` from `dagster_cloud`. When an import moves, the agent updates the import in the same commit if the new location is unambiguous, and otherwise holds the group back and says what broke.

It also reports the pins the lockfiles do not control. The deploy workflow installs `dagster-dbt`, `dagster-cloud`, `dbt-core` and `"dbt-snowflake<=1.8.4"` with `uv pip install --upgrade` at CI time, outside any lock, while the `hooli-data-eng` lock resolves `dbt-snowflake` to a 1.10 or 1.11 release. The workflow pins `dagster-io/dagster-cloud-action` at `v1.12.14`, which should move with Dagster. Every project declares `requires-python = ">=3.9,<3.13"` while every Dockerfile runs Python 3.12, and that range forces the lock to carry a second, older resolution for Python 3.9 with `dagster` 1.12.1. And `hooli-airlift` pins `dagster==1.10.2` and is commented out of `dg.toml`; the agent leaves it alone and says so.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Upgrade the Python dependencies of this Dagster workspace and open one pull request. Read dg.toml, the Makefile, every project's pyproject.toml, uv.lock and Dockerfile, dbt_project/packages.yml and .github/workflows first, and list where versions are pinned outside the lockfiles. Do not run the Makefile's update_packages target as is. For each project that dg.toml lists, find what uv lock --upgrade would change, then apply it in groups with uv lock --upgrade-package: Dagster core with every dagster-* library together, dbt-core with its adapters, then the rest. After each group run uv sync --extra dev, dbt deps and dbt parse against the LOCAL target where the project loads dbt, set DAGSTER_IS_DEV_CLI, and run dg check defs, ruff check and pytest; compare with the same commands on the base branch so failures that already exist are not blamed on the upgrade. If a group breaks an import or a check, fix it in the same commit only when the fix is a moved import or a renamed argument; otherwise drop that group for that project and record the error. Commit each group separately per project. Leave projects that are commented out of dg.toml untouched. If a pull request from an earlier run of this job is still open, add the new commits to its branch and do not open another. If nothing would change, open no pull request and say so in your answer. In the pull request description give a table of package, old version and new version per project, the groups held back with the error, and the pins outside the lockfiles that a person should move by hand.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

One pull request whose commits read as a list: Dagster group in `hooli-data-eng`, dbt group in `hooli-data-eng`, the rest, then the same for each smaller project. Only `uv.lock` files change, plus any import the upgrade moved. The description has a table per project with each package's old and new version, the output of `dg check defs` and pytest after the last commit, and a section for groups that were held back with the error that stopped them.

A second section lists what the agent did not change and why a person should: the unpinned `uv pip install` in the deploy workflow, the `dagster-cloud-action` tag, the `requires-python` range that keeps a Python 3.9 resolution in every lock, and the project pinned to an older Dagster. Those are decisions about CI and supported versions, not version bumps.

Before merging, a reviewer reads the Dagster changelog for the releases the first group crosses, since a changed default in declarative automation or freshness evaluation loads cleanly and only shows when a schedule ticks. The branch deployment the repository's own workflow creates for the pull request is the place to watch one tick. A group nobody wants this week is one `git revert` of its commit.

## Run it on every Monday morning

Create an automation in the Project that holds the repository, with this prompt and a `schedule` trigger: a `cron` of `0 7 * * 1` and your `timezone`. A weekly run keeps each pull request small. Dagster ships often, and a workspace that upgrades once a quarter crosses many releases in one diff with no way to tell which one moved an import.

A week in which nothing moved ends with a result that says so and no pull request. If last week's pull request is still open, the run says that in its result and adds to the same branch instead of opening a second one, so reviewers never face two competing lockfile diffs.

## Why run it on Rush?

Seven lockfiles, a dbt project and a set of definitions that must be imported to be checked add up to a task nobody wants on their own machine. Each project needs its own environment built from its own lock, and a failed group has to be rolled back cleanly before the next is tried. In its own sandbox the agent builds and discards those environments as often as the grouping needs, and every `uv`, `dbt` and `dg` command it ran is kept with its output in the run's record.

The result is checkable without trusting the agent. The pull request shows which packages moved in which commit; the session shows the command output behind each claim that a location still loads; and the list of held-back groups shows what was tried and failed. A reviewer who doubts a line reads the transcript from `GET /o/{org}/sessions/{id}/messages` and sees the traceback.

## Frequently asked questions

### Why not just run make update_packages every week?

It upgrades every package in every project at once and checks nothing. When the result fails to load there is one large lockfile diff and no indication of which package caused it. Grouped commits with a load check after each turn the same upgrade into steps that can be accepted or reverted one at a time.

### Can Dagster core be upgraded without the dagster-dbt or dagster-snowflake libraries?

No. Each library release is built against one core release and declares it, so the resolver refuses a mixed set. The agent treats core and every `dagster-*` package in a project as a single group for that reason.

### Does it upgrade the dbt packages too?

Only when asked. `dbt_project/packages.yml` pins its packages separately from Python, and bumping them changes the SQL that tests compile to. The prompt has the agent read the file and report its pins; add a sentence to the prompt to have it run `dbt deps --upgrade` as a fourth group.

### What happens to a project that pins an old Dagster on purpose?

It is left as it is. A project whose `pyproject.toml` pins an exact Dagster version, or one the workspace file has commented out, is listed in the description with its pin and not touched, because lifting that pin is a migration, not a weekly bump.

## More Dagster jobs

- [Write Dagster Asset Tests for a Project](https://getrush.ai/run/write-asset-tests/dagster/)
- [Add Asset Checks to a Dagster Project from a Linear Issue](https://getrush.ai/run/add-asset-checks/dagster/)
- [Add Concurrency Limits to a Dagster Deployment](https://getrush.ai/run/add-concurrency-limits/dagster/)

## The same job on other stacks

- [Upgrade Python Dependencies Weekly for a LangGraph Agent](https://getrush.ai/run/upgrade-python-dependencies/langgraph/)
- [Upgrade Python Dependencies Weekly for a Litestar App](https://getrush.ai/run/upgrade-python-dependencies/litestar/)
- [Upgrade Python Dependencies in a Streamlit App](https://getrush.ai/run/upgrade-python-dependencies/streamlit/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
