All jobs

Upgrade Python Dependencies Weekly for a Dagster Project

Yes. Each week an agent refreshes every code location's uv.lock, moves Dagster and its libraries together, reloads the definitions and opens one pull request.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent first works out where versions are actually decided, because in a Dagster workspace it is rarely one file. In a workspace like dagster-io/hooli-data-eng-pipelines there are seven projects, each with its own pyproject.toml and uv.lock. The main one, hooli-data-eng, lists dagster, dagster-dbt, dbt-core, dagster-snowflake and some twenty other packages, all but ruff with no version at all, so the lockfile is the only pin, and each project's Dockerfile installs from it with uv sync --frozen. The Makefile has an update_packages target that runs uv lock --upgrade --directory for all seven projects in one go. There is no Dependabot or Renovate config under .github.

Running that target moves everything at once, which is why upgrades in a repository like this tend to be put off. The agent splits the work. In each project it runs uv lock --upgrade on a scratch copy to see what would move, then applies the changes in groups with uv lock --upgrade-package. Dagster core and its libraries are one group and cannot be split: the lock shows dagster 1.12.14 beside dagster-dbt 0.28.14, and each library release requires the core release it shipped with. dbt-core and its DuckDB and Snowflake adapters are a second group. Everything else is a third. Each group is its own commit per project.

After each group it proves the code location still loads. It runs uv sync --extra dev, parses the dbt project against the DuckDB target (dbt parse --project-dir=dbt_project --target LOCAL), sets DAGSTER_IS_DEV_CLI so the project picks its local resources, and runs dg check defs, ruff check and pytest, comparing against the same commands on the untouched branch. This is where a Dagster upgrade usually fails. Three files here import FreshnessPolicy from dagster.preview.freshness, a preview module, and pandas/assets.py imports build_anomaly_detection_freshness_checks from dagster_cloud. When an import moves, the agent updates the import in the same commit if the new location is unambiguous, and otherwise holds the group back and says what broke.

It also reports the pins the lockfiles do not control. The deploy workflow installs dagster-dbt, dagster-cloud, dbt-core and "dbt-snowflake<=1.8.4" with uv pip install --upgrade at CI time, outside any lock, while the hooli-data-eng lock resolves dbt-snowflake to a 1.10 or 1.11 release. The workflow pins dagster-io/dagster-cloud-action at v1.12.14, which should move with Dagster. Every project declares requires-python = ">=3.9,<3.13" while every Dockerfile runs Python 3.12, and that range forces the lock to carry a second, older resolution for Python 3.9 with dagster 1.12.1. And hooli-airlift pins dagster==1.10.2 and is commented out of dg.toml; the agent leaves it alone and says so.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Upgrade the Python dependencies of this Dagster workspace and open one pull request. Read dg.toml, the Makefile, every project's pyproject.toml, uv.lock and Dockerfile, dbt_project/packages.yml and .github/workflows first, and list where versions are pinned outside the lockfiles. Do not run the Makefile's update_packages target as is. For each project that dg.toml lists, find what uv lock --upgrade would change, then apply it in groups with uv lock --upgrade-package: Dagster core with every dagster-* library together, dbt-core with its adapters, then the rest. After each group run uv sync --extra dev, dbt deps and dbt parse against the LOCAL target where the project loads dbt, set DAGSTER_IS_DEV_CLI, and run dg check defs, ruff check and pytest; compare with the same commands on the base branch so failures that already exist are not blamed on the upgrade. If a group breaks an import or a check, fix it in the same commit only when the fix is a moved import or a renamed argument; otherwise drop that group for that project and record the error. Commit each group separately per project. Leave projects that are commented out of dg.toml untouched. If a pull request from an earlier run of this job is still open, add the new commits to its branch and do not open another. If nothing would change, open no pull request and say so in your answer. In the pull request description give a table of package, old version and new version per project, the groups held back with the error, and the pins outside the lockfiles that a person should move by hand.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

One pull request whose commits read as a list: Dagster group in hooli-data-eng, dbt group in hooli-data-eng, the rest, then the same for each smaller project. Only uv.lock files change, plus any import the upgrade moved. The description has a table per project with each package's old and new version, the output of dg check defs and pytest after the last commit, and a section for groups that were held back with the error that stopped them.

A second section lists what the agent did not change and why a person should: the unpinned uv pip install in the deploy workflow, the dagster-cloud-action tag, the requires-python range that keeps a Python 3.9 resolution in every lock, and the project pinned to an older Dagster. Those are decisions about CI and supported versions, not version bumps.

Before merging, a reviewer reads the Dagster changelog for the releases the first group crosses, since a changed default in declarative automation or freshness evaluation loads cleanly and only shows when a schedule ticks. The branch deployment the repository's own workflow creates for the pull request is the place to watch one tick. A group nobody wants this week is one git revert of its commit.

Run it on every Monday morning

Create an automation in the Project that holds the repository, with this prompt and a schedule trigger: a cron of 0 7 * * 1 and your timezone. A weekly run keeps each pull request small. Dagster ships often, and a workspace that upgrades once a quarter crosses many releases in one diff with no way to tell which one moved an import.

A week in which nothing moved ends with a result that says so and no pull request. If last week's pull request is still open, the run says that in its result and adds to the same branch instead of opening a second one, so reviewers never face two competing lockfile diffs.

The use case this job follows, with its automation and trigger: Upgrade dependencies weekly.

Why run it on Rush?

Seven lockfiles, a dbt project and a set of definitions that must be imported to be checked add up to a task nobody wants on their own machine. Each project needs its own environment built from its own lock, and a failed group has to be rolled back cleanly before the next is tried. In its own sandbox the agent builds and discards those environments as often as the grouping needs, and every uv, dbt and dg command it ran is kept with its output in the run's record.

The result is checkable without trusting the agent. The pull request shows which packages moved in which commit; the session shows the command output behind each claim that a location still loads; and the list of held-back groups shows what was tried and failed. A reviewer who doubts a line reads the transcript from GET /o/{org}/sessions/{id}/messages and sees the traceback.

Frequently asked questions

Why not just run make update_packages every week?

It upgrades every package in every project at once and checks nothing. When the result fails to load there is one large lockfile diff and no indication of which package caused it. Grouped commits with a load check after each turn the same upgrade into steps that can be accepted or reverted one at a time.

Can Dagster core be upgraded without the dagster-dbt or dagster-snowflake libraries?

No. Each library release is built against one core release and declares it, so the resolver refuses a mixed set. The agent treats core and every dagster-* package in a project as a single group for that reason.

Does it upgrade the dbt packages too?

Only when asked. dbt_project/packages.yml pins its packages separately from Python, and bumping them changes the SQL that tests compile to. The prompt has the agent read the file and report its pins; add a sentence to the prompt to have it run dbt deps --upgrade as a fourth group.

What happens to a project that pins an old Dagster on purpose?

It is left as it is. A project whose pyproject.toml pins an exact Dagster version, or one the workspace file has commented out, is listed in the description with its pin and not touched, because lifting that pin is a migration, not a weekly bump.

More Dagster jobs

The same job on other stacks

Keep reading