What the agent does
The agent clones the repository into its own cloud sandbox and maps which code locations have tests. In a workspace like dagster-io/hooli-data-eng-pipelines, dg.toml lists six projects, each with its own pyproject.toml, uv.lock and a pytest entry under the dev extra. Only hooli-data-eng has real tests: hooli_data_eng_tests/test_assets.py and test_repo_loads.py. The test_assets.py files in hooli-bi and hooli-airlift are empty, and hooli-basics, hooli-batch-enrichment and hooli-data-ingest have no test directory at all. The deploy workflow, .github/workflows/deploy-dagster-cloud.yml, runs Ruff with check --fix and deploys the code locations whose files changed; it never runs pytest. The one existing test module also imports hooli_data_eng.assets.raw_data, but the package now keeps its assets under defs/ (pandas, dbt, custom_ingest, scikit_learn and others) and has no assets module, so that file cannot import as it stands.
For hooli-batch-enrichment the agent reads assets.py, api.py, warehouse.py and definitions.py. raw_data is an asset that reads a CSV through MyWarehouse, a ConfigurableResource with a path that runs duckdb.sql(...).df(), logs the row count and attaches preview, nrows and user_input metadata from an experimentConfig whose experiment_name defaults to default_config. enriched_data is a graph_asset of three ops: split_rows yields DynamicOutput chunks of number_records_per_batch rows (default 50), process_chunk carries RetryPolicy(max_retries=2) and calls EnrichmentAPI.get_order_details, which fakes http://api.jaffleshop.co/v1/order_details with the responses library and raises API time out on roughly one call in five hundred, and concat_chunk_list joins the frames. The tests the agent writes in hooli-batch-enrichment/hooli_batch_enrichment_tests/ build raw_data with build_asset_context and a MyWarehouse pointed at a small CSV in tmp_path, assert the frame and the metadata keys, check that split_rows yields three mapping keys for a 120-row frame at the default batch size, run process_chunk with a stub api resource that returns a fixed order_center so the random time-out cannot make the suite flaky, materialize raw_data and enriched_data together with materialize and assert the order_center column, and load defs to check that run_etl_pipeline and the run_assets_30min schedule with its */30 * * * * cron exist.
For hooli-basics the whole project is definitions.py: country_stats calls read_html on a web page with the html5lib flavor, renames six columns and computes pop_change as (pop_2023 / pop_2022 - 1) * 100; check_country_stats is an asset_check that always passes; change_model fits a scikit-learn LinearRegression on continent dummies and returns a MaterializeResult whose metadata holds r2_score, rmse, mae, num_training_samples, intercept and coefficients; continent_stats groups by continent. The tests patch read_html to return a six-column frame so nothing touches the network, assert the renamed columns and the pop_change arithmetic on known numbers, check that num_training_samples equals the rows with a pop_change, that coefficients has one key per continent dummy, that the check returns passed=True, and that defs loads three assets and one check. hooli-data-ingest is Sling replication from component.yaml and replication.yaml with no Python logic beyond a schedule, so the agent adds only a definitions-load test there and says why.
The agent runs uv sync --extra dev in each project, since every one has a uv.lock, then uv run pytest there, and formats the new files with ruff format the way the Makefile's ruff target does. It also repairs the stale import in hooli-data-eng/hooli_data_eng_tests/test_assets.py by finding where orders and users now live under defs/, or marks the module skipped with the reason when the assets changed shape, and says which in the pull request. Finally it adds a pytest step to the deploy workflow before the deploy job, scoped to the projects whose files changed using the changed-files outputs the workflow already computes.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Write pytest tests for the Dagster projects in this workspace that have none. Read dg.toml, every project's pyproject.toml and definitions, and the existing tests under hooli-data-eng first, and follow their naming for the new test directories. For hooli-batch-enrichment, cover raw_data with a temporary CSV, split_rows chunking, process_chunk with a stubbed api resource so the random time-out cannot flake, a full materialize of both assets, and a definitions load that finds the job and schedule. For hooli-basics, patch read_html so no test touches the network, cover the column renames and pop_change math, the change_model metadata, the asset check and a definitions load. Fix or skip the existing hooli-data-eng test whose import no longer resolves and explain which you did. Run uv sync and uv run pytest in each project, format with ruff, add a pytest step to the deploy workflow for changed projects, and open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that adds hooli-batch-enrichment/hooli_batch_enrichment_tests/, hooli-basics/hooli_basics_tests/ and a definitions-load test for hooli-data-ingest, changes hooli-data-eng/hooli_data_eng_tests/test_assets.py, and adds a pytest step to .github/workflows/deploy-dagster-cloud.yml. The description lists each test by name with the behaviour it pins, pastes the uv run pytest output per project, and explains the one stub, the api resource, and the one patch, read_html.
The run's record keeps every uv sync and pytest invocation with its output, so a reviewer can see each project's tests pass in an environment built from its own lockfile, not a shared one.
Before merging, a reviewer checks that the stub for EnrichmentAPI does not hide the retry behaviour worth testing, decides whether the read_html patch should become a fixture file committed under the tests, and confirms the new workflow step runs on branch deployments as well as on main.
Run it on every new code location added to dg.toml
In the console, an automation in the Project that has the repository, with a github trigger on pull_request.opened and pull_request.synchronize and branch_patterns for your default branch, starts one run per pull request with the pull request as its context. The prompt tells the run to look for a project in dg.toml whose *_tests directory is missing or empty, write its first tests on the pull request's branch, and comment with nothing to do when every project is covered.
From your own server, the same work is one POST /o/{org}/sessions with this body and the repository in resources, sent with a request_id that is a UUID v5 of the pull request URL so a retried start returns the run already begun. A webhook URL on the API key reports run.completed with the session_url to read the pull request link from.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Dagster tests fail for reasons that have nothing to do with the assets: a resource that reaches the network, a random failure baked into a fake API, an import that moved when the project adopted the defs/ layout. The agent reads the resources and the ops before it writes a test, stubs exactly the two calls that leave the process, and runs every suite in the project's own uv environment, so the pull request carries tests that pass for the reason they should.
The sandbox is its own computer, each uv sync and pytest run stays in the record with its output, and a workspace with six code locations gets all six covered in one run rather than one afternoon per project.
Frequently asked questions
Why stub the enrichment API instead of testing it as written?
Because EnrichmentAPI.get_order_details throws API time out at random, about once in five hundred calls, to demonstrate retries. A test that calls it directly fails now and then for no change in the code. The stub returns a fixed order_center; the RetryPolicy on process_chunk is asserted separately by reading the op definition.
What happens to the existing test that no longer imports?
The agent looks for orders and users under hooli_data_eng/defs/ and repoints the import when the assets still take the same context and resources. When their shape changed too, it marks the module with pytest.mark.skip and the reason, so the suite is green and the pull request names the test that needs a human.
Does the run need Dagster Cloud credentials or a warehouse?
No. The tests build contexts with build_asset_context and materialize against in-process resources: a DuckDB read of a temporary CSV and a stubbed API. DAGSTER_CLOUD_API_TOKEN and the Snowflake, Databricks and Power BI resources in the other projects are never used.
Will the tests run in CI after this?
Yes. The deploy workflow gains a pytest step before the deploy job, limited to the projects whose files changed, using the changed-files outputs the workflow already computes for the deploy itself. A pull request that breaks an asset now fails before a branch deployment is built.
More Dagster jobs
More jobs like this: Write the missing tests
- Write bUnit Tests for a Blazor Web App (Blazor)
- Write Tests for an AWS CDK App (AWS CDK)
- Write Tests for a Go CLI (Go CLI)