All jobs

Review Pull Requests on a Dagster Project

Yes. For each pull request the agent loads the changed code locations, runs Ruff, a dbt parse and the tests, and posts one review with the output quoted.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

A Dagster pull request can pass every check a repository has and still break at the next deploy, because most of what holds a pipeline together is strings: asset keys, partition start dates, resource names, cron expressions. So the agent starts by learning how this workspace is wired. In a workspace like dagster-io/hooli-data-eng-pipelines it reads dg.toml, which lists six projects and comments out a seventh, hooli-airlift, with a note that it is currently failing; dagster_cloud.yaml, which names seven code locations and the directory each one builds from; each changed project's pyproject.toml; and .github/workflows/deploy-dagster-cloud.yml. That workflow runs Ruff with check --fix, validates the cloud config, builds an image for each location whose files changed and deploys it. It never runs pytest and never loads the definitions before the image is built.

Then it checks out the pull request and does what the workflow leaves out. In each changed project it runs uv sync --extra dev against that project's uv.lock, which brings in dagster-dg-cli, pytest and Ruff. For hooli-data-eng it runs dbt deps and dbt parse --project-dir=dbt_project --target LOCAL first, because the dbt component needs a manifest and the Makefile's own manifest target parses with --target BRANCH, which dbt_project/profiles.yml defines as Snowflake; LOCAL is DuckDB and needs no credentials. With DAGSTER_IS_DEV_CLI set, so that get_env() in utils/__init__.py resolves to LOCAL, it runs dg check defs and quotes any error. It runs ruff check without --fix and ruff format --check: the workflow's --fix repairs what it can inside the runner and commits nothing, so a fixable violation passes CI and stays in the branch.

It runs pytest on the base branch before the pull request branch. In this repository that matters: hooli_data_eng_tests/test_repo_loads.py imports a hooli_data_eng object from a package whose __init__.py is empty, and test_assets.py imports hooli_data_eng.assets.raw_data, while the assets now live under defs/custom_ingest/assets.py. A suite that cannot be collected on the base branch is reported once, as a standing problem, and not charged to the author.

Last it reads the diff against rules the code implies. Every resources.py under defs/ builds a resource_def dictionary keyed LOCAL, BRANCH and PROD, so a resource added under one key loads on a laptop and fails in the branch deployment. DailyPartitionsDefinition(start_date="2023-05-25") is declared in both custom_ingest/assets.py and dbt/component.py, so a start date changed in one file is checked against the other. A renamed asset or key_prefix is searched for in every place that refers to it by string: the AssetIn(key_prefix=["ANALYTICS"]) inputs in pandas/assets.py, the orders_sensor asset sensor on ["ANALYTICS", "orders_augmented"], the selection in predict_job, and the asset_selection lines in defs/dbt/component.yaml. A new code location has to appear in dg.toml, in dagster_cloud.yaml, and three times in the workflow: the files_yaml filter, the block that builds the LOCATIONS variable, and an image build step.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Review this pull request to our Dagster workspace. Read dg.toml, dagster_cloud.yaml, the Makefile, .github/workflows/deploy-dagster-cloud.yml, dbt_project/profiles.yml and the pyproject.toml, definitions.py and utils of every project the diff touches first. In each changed project run uv sync --extra dev from its own uv.lock. For a project that loads dbt, run dbt deps and dbt parse against the LOCAL target so no warehouse credentials are needed. Set DAGSTER_IS_DEV_CLI so the code resolves its LOCAL resources, run dg check defs, ruff check without --fix, ruff format --check, and pytest, and quote every error with file and line. Run pytest on the base branch too and report failures that exist there separately from failures the pull request introduces. Then read the diff: every resource must exist under the LOCAL, BRANCH and PROD keys of its resource_def; a changed partitions definition must match every other definition of the same partitions; a renamed asset key, key_prefix, job, schedule or sensor must be updated everywhere it is referred to by string, including asset sensors, AssetIn, asset selections and component YAML; a changed cron or automation condition is stated in plain words; a new code location must be added to dg.toml, dagster_cloud.yaml and the workflow's file filter, location list and image build. Do not materialize assets against a warehouse. Post one review on the pull request with file and line for every finding and the command output quoted. Do not push commits.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app/pull/42"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

One review on the pull request. It opens with a short results block: dependency sync, dbt parse, dg check defs, Ruff lint, Ruff format and pytest, each marked pass or fail with the lines that failed, and a separate line for anything that already fails on the base branch. Findings follow by file, each with the line, what breaks and where the other end of the broken reference is, for example the sensor that still watches an asset key the diff renamed. A change to a cron string or an automation condition is restated as a sentence ("midnight on the odd days of the month" for 0 0 1-31/2 * *) so a reviewer can confirm the schedule is the one intended.

The review comments; it does not approve or block. The full evidence stays on the session: GET /o/{org}/sessions/{id}/messages returns every command with its output, including the full dg check defs traceback when a location does not load.

A person still decides what the agent cannot see from code. Whether a new partition start date means a backfill, and who pays for it, depends on the warehouse. Whether the resource the pull request adds under PROD points at the right account is a credentials question the sandbox has no access to. The review lists these as open questions for the author and does not guess.

Run it on every pull request opened

In the Project that holds the repository, create an automation with this prompt, read-only switched on, and a github trigger on pull_request.opened and pull_request.synchronize. Read-only runs can read the repository and comment but cannot push, which is the right permission for a reviewer. The trigger puts the pull request after the prompt and fills in nothing else; set its branch_patterns to the base branch you deploy from.

In a repository that already creates a Dagster+ branch deployment for each pull request, the two run side by side: the deployment shows the changed assets in the Dagster UI, and the review says whether the code loads and what the diff breaks. The trigger fires for work by an owner, member or collaborator only. A pull request from a fork never fires it, so a maintainer who wants one reviewed pushes its commits to a branch in the repository and opens the pull request from there.

The use case this job follows, with its automation and trigger: Review every pull request.

Why run it on Rush?

Loading a Dagster code location is not free. Each project here has its own lockfile, the main one needs a parsed dbt project before its definitions import, and the environment variable that selects resources changes what gets constructed. A reviewer who wants to know whether a branch loads has to rebuild that on their own machine, usually on top of the virtual environment of whatever they were working on. The agent builds it from the lockfile in a fresh sandbox for each pull request, and when several pull requests are open each gets its own sandbox at the same time.

It also gives the author something a green deploy does not: the exact command and output behind every finding. A branch deployment that fails to load shows an error in a UI after an image build; the review shows the same traceback against the diff, next to the string reference that caused it, and the whole record is readable from the session afterwards.

Frequently asked questions

Does the agent need Snowflake or Dagster+ credentials?

No. It loads the definitions with the project's local resources and parses dbt against the DuckDB target, neither of which opens a warehouse connection. Anything that can only be confirmed against the real warehouse or the deployed code location is listed in the review as not checked, with the reason.

Will it materialize my assets?

Not against real storage. The prompt forbids it, and loading definitions does not run them. Tests that execute a job in process against a temporary DuckDB file, as the existing asset tests in this kind of project do, run as part of pytest.

What does it check in dbt model changes?

It parses the project, so a model that references a missing source or macro fails with dbt's own error. It then reads the changed SQL and YAML against the Dagster side: a renamed model is an asset key change, and a model moved between the groups in the dbt component's YAML changes its partitioning and schedule.

Can it fix what it finds?

Not in a read-only automation, by design. To have a finding fixed, start a separate run on the pull request's branch with a prompt that names the finding, and it pushes a commit there for the author to review.

More Dagster jobs

The same job on other stacks

Keep reading