All jobs

Add Asset Checks to a Dagster Project from a Linear Issue

Yes. An agent picks up the Linear issue, adds asset checks to the Dagster assets that have none, runs them locally against DuckDB with pytest, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent starts from the Linear issue and maps which assets already have checks and which do not. In a workspace like dagster-io/hooli-data-eng-pipelines, dg.toml lists six projects and hooli-data-eng/pyproject.toml points dg at hooli_data_eng.definitions, which merges load_defs(hooli_data_eng.defs) with a multiprocess executor capped at 3. Under src/hooli_data_eng/defs/, custom_ingest/assets.py already has check_users on RAW_DATA/users, which warns when a company outside ShopMart, SportTime, FamilyLtd and DiscountStore appears, and build_column_schema_change_checks for orders and users; pandas/assets.py has check_avg_orders, which passes when the average is under 50, plus anomaly-detection freshness checks on avg_orders; the dbt component in defs/dbt/component.py turns dbt tests into checks with enable_asset_checks=True and adds freshness checks for the weekly group named in component.yaml. What has no check: predicted_orders in FORECASTING, min_order and key_product_deepdive in MARKETING, and order_forecast_model and model_stats_by_month in scikit_learn/assets.py.

It writes one check per gap, each a @dg.asset_check next to its asset so dg.load_asset_checks_from_modules in pandas/definitions.py picks it up, and mirrors that loading in the scikit-learn definitions module if it does not already do it. For predicted_orders, which builds a date_range of 30 days from the last order_date in weekly_order_summary, the check asserts that every num_orders is finite and not negative and that order_date is strictly increasing, with the row count and the minimum in metadata. For min_order, a single-row frame holding the minimum of company_perf["n_orders"], it asserts one row and a value of at least zero. For order_forecast_model, a tuple of two coefficients from optimize.curve_fit, it asserts both are finite and a is positive, since model_func is a * exp(b * ...). For model_stats_by_month it asserts the error value is finite, and for key_product_deepdive that the frame for the partition's SKU is not empty. Severity is ERROR for the forecast and model checks, which feed dashboards, and WARN for the deep-dive, matching how check_users is already set.

It then proves the checks run. The Makefile shows the local path: make dependencies installs the project with uv, make deps and make manifest run dbt deps and dbt parse --project-dir=dbt_project --target BRANCH, and dagster dev serves it; resources come from resource_def[get_env()], so a local run uses the DuckDB variants rather than Snowflake. The agent runs dg check defs to confirm the definitions still load, then writes hooli_data_eng_tests/test_asset_checks.py that materializes each asset together with its check through dg.materialize with test resources and asserts on the AssetCheckResult it gets back, both on a frame that passes and on one it builds to fail. It notices that the existing test_assets.py imports hooli_data_eng.assets.raw_data, a module path that no longer exists under defs/, so it keeps its own tests in a new file and names the stale import in the pull request instead of fixing it unasked.

Finally it runs ruff check and ruff format --check on the project, since .github/workflows/deploy-dagster-cloud.yml runs Ruff before deploying, and opens a pull request that links the issue, lists each new check with its asset and what it asserts, and quotes the pytest and dg check defs output. The same workflow builds a Dagster+ branch deployment for the pull request, where the checks appear under each asset in the catalog once a reviewer looks.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "A Linear issue labelled agent says: \"Every asset that feeds a dashboard needs an asset check. Add checks for the forecast, marketing and model assets that have none, and make sure they run in tests.\" Do the work in the hooli-data-eng project of this Dagster workspace. Read dg.toml, hooli-data-eng/pyproject.toml, the Makefile, src/hooli_data_eng/definitions.py and every module under src/hooli_data_eng/defs, including the existing checks in custom_ingest/assets.py and pandas/assets.py and the dbt component, plus the tests under hooli_data_eng_tests. Add an asset_check for each of predicted_orders, min_order, key_product_deepdive, order_forecast_model and model_stats_by_month that asserts on the values the asset returns, with metadata, and make sure each check is loaded by its definitions module. Run dg check defs, write pytest tests that materialize each asset with its check through dg.materialize using local resources and assert on the AssetCheckResult, run them, run ruff check and ruff format --check, and open a pull request that links the issue and lists every check and what it asserts. Do not touch the dbt models or the existing checks.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request on its own branch that adds checks to src/hooli_data_eng/defs/pandas/assets.py and src/hooli_data_eng/defs/scikit_learn/assets.py, loads them from the matching definitions modules where that was missing, and adds hooli_data_eng_tests/test_asset_checks.py. The description has a table of asset, check name, severity and what it asserts, the dg check defs output, the pytest output with each test named, and the Ruff output, and it links the Linear issue. The repository's own deploy workflow then builds a branch deployment for the pull request.

Before merging, a reviewer checks the thresholds the agent chose, such as treating a negative forecast as an error, against what the dashboards tolerate, and whether a failing check should block downstream materialization, which is a blocking flag the pull request leaves off. They also decide what to do with the stale test import the agent pointed out.

Run it on every Linear issue labelled agent

In the console, an automation with a linear trigger on linear.issue.labeled and the label agent starts one run per labelled issue, with the issue added after the prompt, in the Project's repository. Changes made by an app integration never fire it, so the agent's own comment on the issue does not start another run. Each run reads that project's own definitions, so an issue about a different code location in the same workspace gets checks for its assets, and an issue that asks for freshness rather than value checks gets those.

From your own server, the same run is one POST /o/{org}/sessions per issue: put the repository in resources and add the issue's https://linear.app/<workspace>/issue/<ID> URL there too, so Rush reads it through your Linear connection into the prompt. A webhook URL on the API key reports run.completed or run.failed with the session_url when the turn finishes.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

An asset check is only worth having if it runs, and proving that means loading the definitions with their dbt manifest, materializing assets against a local warehouse and reading the check results, not reading the decorator. In its own sandbox the agent installs the project the way the Makefile does, parses the dbt project, runs dg check defs and pytest, and every command and result stays in the run's record beside the diff.

A Linear label becomes a reviewed change with its own tests and a branch deployment, and several data-quality issues can run at once, each in its own sandbox with its own pull request, while the team's laptops stay free of another DuckDB file.

Frequently asked questions

Does it need Snowflake, S3 or Dagster+ credentials?

No. The resources in this project vary by environment through get_env(), and a local run uses the DuckDB and local-file variants, so the checks are exercised without any cloud account. The pull request's branch deployment on Dagster+ is built by the repository's own workflow with the secrets it already has.

Will the checks block downstream assets when they fail?

Not as written. The agent leaves blocking off so a failing check reports in the catalog and in alerts without stopping the run, which matches how check_avg_orders behaves today. Turning it on for a given check is a one-line change you can ask for in review.

What about the dbt models?

They already get checks from their dbt tests through the component's enable_asset_checks, and the weekly group has freshness checks. The agent does not add Python checks on top of them, and the prompt tells it to leave the dbt project alone.

Why not fix the stale test import while it is there?

Because the issue asks for asset checks, and rewriting tests that reference a module path that no longer exists is a separate decision about what those tests should cover now. The agent names the file and the import in the pull request so the team can ask for it as a follow-up.

More Dagster jobs

More jobs like this: Turn a Linear label into a PR

Keep reading