All jobs

Add dbt Project Evaluator to a dbt Project

Yes. An agent installs dbt_project_evaluator, runs it on a local build, fixes the missing tests and descriptions it finds, records accepted exceptions, opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the project the way the evaluator will. In a project shaped like dbt-labs/jaffle-shop, dbt_project.yml names jaffle_shop at version 3.0.0, requires dbt >=2.0.0, points test-paths at data-tests, materialises staging as views and marts as tables, and enables the seeds under seeds/jaffle-data only with the load_source_data var. packages.yml has dbt-labs/dbt_utils 1.4.1 and dbt-audit-helper from git, pinned in package-lock.yml. The models are six staging views over one source, ecom in models/staging/__sources.yml, and seven marts: customers, orders, order_items, products, supplies, locations and metricflow_time_spine, each with a YAML file beside it carrying descriptions, data_tests, semantic model fields and, on three of them, unit_tests.

The YAML is uneven in ways the evaluator reports. orders.yml, customers.yml and order_items.yml test their primary keys with not_null and unique and describe most columns; products.yml, supplies.yml and locations.yml describe the model but not one column, and put no test on product_id, supply_uuid or location_id even though each is declared the entity of type primary. orders.yml lists location_id with an entity and no description. supplies.sql is select * from {{ ref('stg_supplies') }} with no transformation. In __sources.yml, only raw_orders and raw_stores carry a loaded_at_field, and no table has a freshness block. .github/workflows/ci.yml runs dbt Cloud jobs through scripts/dbt_cloud_run_job.py with an API key, so the repository's CI cannot run a new command from the branch; .pre-commit-config.yaml runs ruff and whitespace hooks, and .sqlfluff sets the Snowflake dialect.

The agent adds dbt-labs/dbt_project_evaluator to packages.yml, runs dbt deps so package-lock.yml records it, and adds a dbt_project_evaluator block under models: in dbt_project.yml so its models build into a +schema of their own and are disabled outside a target of the reader's choosing. To run it, it starts a Postgres in the sandbox, installs the dbt-postgres adapter, writes a local profile, seeds the raw tables with dbt seed --vars '{"load_source_data": true}' as Taskfile.yml does, runs dbt build, then dbt build --select package:dbt_project_evaluator. The fct_ models that come back with rows are the findings: missing primary key tests on three marts, undocumented columns, a model with no transformation, and sources without freshness. The agent fixes what the project should own: not_null and unique on product_id, supply_uuid and location_id, descriptions for every column in products.yml, supplies.yml, locations.yml and order_items.yml written from the staging SQL that produces them, and a description for location_id in orders.yml.

What it does not fix, it records. seeds/dbt_project_evaluator_exceptions.csv, the package's exception seed, gets a row for supplies under the pass-through finding with a comment that the mart exists to give the semantic layer a stable name, and a row for the sources without a timestamp under the freshness finding, so the evaluator stays green without hiding the decision. The documentation_coverage_target and test_coverage_target vars are left at their defaults so the report keeps asking for full coverage. It runs the evaluator again to show every fct_ test passing, runs dbt build to show the project's own tests still pass with the new ones added, and keeps both outputs along with the dbt deps result.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add dbt_project_evaluator to this dbt project, act on its findings, and open a pull request. Read dbt_project.yml, packages.yml, package-lock.yml, models/staging/__sources.yml, every YAML file under models/, models/marts/supplies.sql, Taskfile.yml, .github/workflows/ci.yml and .pre-commit-config.yaml first. Add dbt-labs/dbt_project_evaluator to packages.yml, run dbt deps, and configure the package's models under models: in dbt_project.yml with their own schema. Start a local Postgres, install dbt-postgres, write a local profile that is not committed, seed with load_source_data true, run dbt build, then dbt build --select package:dbt_project_evaluator. For each failing fct_ test, either fix the project, adding not_null and unique tests on primary key columns and descriptions on undocumented columns written from the staging SQL, or add a row to the exceptions seed with a comment explaining why the finding is accepted. Leave the coverage target vars at their defaults. Run the evaluator and dbt build again and paste both outputs in the pull request, with a table of findings and what was done for each.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds one package to packages.yml and package-lock.yml, a dbt_project_evaluator block in dbt_project.yml, an exceptions seed, tests and descriptions in five model YAML files, and no SQL changes. The description has a table with one row per fct_ finding, the model or column it named, and either the fix or the exception row, followed by the evaluator output before and after and the final dbt build summary from the sandbox.

The repository's CI triggers dbt Cloud jobs by API, so it does not run the evaluator on the pull request; the description names the command to add to the PR job so it runs there from then on.

A reviewer checks that every column description matches what the staging SQL produces, that each exception row has a reason they agree with, that the package's models are disabled where they should not build, and that the new primary key tests pass on their warehouse. They merge and add dbt build --select package:dbt_project_evaluator to the dbt Cloud CI job.

Run it on every dbt project that has grown past its conventions

A team with several dbt projects sends one POST /o/{org}/sessions per repository with this prompt and the repository in resources. Each run installs the evaluator in that project, reads its own findings, and fixes or records each one, so a project with good test coverage but no descriptions gets descriptions, and each run opens its own pull request.

Send a request_id with each call so a retry returns the run already started, and set a webhook_url on the key so run.completed arrives with session_url when the turn ends and your handler posts the pull request where the analytics engineers read it.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

The evaluator is only useful if someone acts on its findings, and acting on them means reading the SQL behind every undocumented column and deciding, finding by finding, between a fix and an exception. In a Rush sandbox the agent builds the project against a real Postgres, runs the evaluator twice, and keeps both reports and every YAML change in the run's record beside the pull request.

On a laptop this is an afternoon of warehouse setup and a long tail of YAML edits that nobody schedules. The agent does the first pass in one run, and a follow-up such as "also add freshness windows for the two sources that have a loaded_at_field" wakes the same parked sandbox and lands on the same branch.

Frequently asked questions

Why run against Postgres and not our warehouse?

The repository's CI reaches its warehouses through dbt Cloud jobs and an API key the sandbox does not have, and macros/cents_to_dollars.sql already carries a postgres__ implementation, so Postgres is a supported target here. The findings are about YAML and project structure, which do not depend on the adapter; the reviewer runs the new tests on their warehouse before merging.

What goes in the exceptions seed?

A finding the team accepts on purpose, with a comment saying why: here the supplies mart that selects straight from its staging model, kept because the semantic layer names it, and the four source tables with no load timestamp. The seed is the package's own mechanism, so the evaluator stays green and the reasons live in the repository.

Will the evaluator's models build in production?

Only if the reader wants them to. The dbt_project.yml block gives them their own schema and an +enabled condition on the target name, so a production job skips them unless it is configured to run them. The CI job is where the command belongs, which the pull request description says.

Does this change any model SQL?

No. Every change is in packages.yml, dbt_project.yml, the model YAML files and one seed. The descriptions are derived from the staging SQL and the tests assert properties the entity: primary declarations already claim, so a failing new test would mean the data does not match what the YAML already says.

More dbt jobs

More jobs like this: Hand off a long task

Keep reading