# Add dbt Project Evaluator to a dbt Project

Yes. An agent installs dbt_project_evaluator, runs it on a local build, fixes the missing tests and descriptions it finds, records accepted exceptions, opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-dbt-project-evaluator/dbt/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads the project the way the evaluator will. In a project shaped like dbt-labs/jaffle-shop, `dbt_project.yml` names `jaffle_shop` at version 3.0.0, requires dbt `>=2.0.0`, points `test-paths` at `data-tests`, materialises `staging` as views and `marts` as tables, and enables the seeds under `seeds/jaffle-data` only with the `load_source_data` var. `packages.yml` has `dbt-labs/dbt_utils` 1.4.1 and `dbt-audit-helper` from git, pinned in `package-lock.yml`. The models are six staging views over one source, `ecom` in `models/staging/__sources.yml`, and seven marts: `customers`, `orders`, `order_items`, `products`, `supplies`, `locations` and `metricflow_time_spine`, each with a YAML file beside it carrying descriptions, `data_tests`, semantic model fields and, on three of them, `unit_tests`.

The YAML is uneven in ways the evaluator reports. `orders.yml`, `customers.yml` and `order_items.yml` test their primary keys with `not_null` and `unique` and describe most columns; `products.yml`, `supplies.yml` and `locations.yml` describe the model but not one column, and put no test on `product_id`, `supply_uuid` or `location_id` even though each is declared the `entity` of type `primary`. `orders.yml` lists `location_id` with an entity and no description. `supplies.sql` is `select * from {{ ref('stg_supplies') }}` with no transformation. In `__sources.yml`, only `raw_orders` and `raw_stores` carry a `loaded_at_field`, and no table has a `freshness` block. `.github/workflows/ci.yml` runs dbt Cloud jobs through `scripts/dbt_cloud_run_job.py` with an API key, so the repository's CI cannot run a new command from the branch; `.pre-commit-config.yaml` runs `ruff` and whitespace hooks, and `.sqlfluff` sets the Snowflake dialect.

The agent adds `dbt-labs/dbt_project_evaluator` to `packages.yml`, runs `dbt deps` so `package-lock.yml` records it, and adds a `dbt_project_evaluator` block under `models:` in `dbt_project.yml` so its models build into a `+schema` of their own and are disabled outside a target of the reader's choosing. To run it, it starts a Postgres in the sandbox, installs the `dbt-postgres` adapter, writes a local profile, seeds the raw tables with `dbt seed --vars '{"load_source_data": true}'` as `Taskfile.yml` does, runs `dbt build`, then `dbt build --select package:dbt_project_evaluator`. The `fct_` models that come back with rows are the findings: missing primary key tests on three marts, undocumented columns, a model with no transformation, and sources without freshness. The agent fixes what the project should own: `not_null` and `unique` on `product_id`, `supply_uuid` and `location_id`, descriptions for every column in `products.yml`, `supplies.yml`, `locations.yml` and `order_items.yml` written from the staging SQL that produces them, and a description for `location_id` in `orders.yml`.

What it does not fix, it records. `seeds/dbt_project_evaluator_exceptions.csv`, the package's exception seed, gets a row for `supplies` under the pass-through finding with a comment that the mart exists to give the semantic layer a stable name, and a row for the sources without a timestamp under the freshness finding, so the evaluator stays green without hiding the decision. The `documentation_coverage_target` and `test_coverage_target` vars are left at their defaults so the report keeps asking for full coverage. It runs the evaluator again to show every `fct_` test passing, runs `dbt build` to show the project's own tests still pass with the new ones added, and keeps both outputs along with the `dbt deps` result.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add dbt_project_evaluator to this dbt project, act on its findings, and open a pull request. Read dbt_project.yml, packages.yml, package-lock.yml, models/staging/__sources.yml, every YAML file under models/, models/marts/supplies.sql, Taskfile.yml, .github/workflows/ci.yml and .pre-commit-config.yaml first. Add dbt-labs/dbt_project_evaluator to packages.yml, run dbt deps, and configure the package's models under models: in dbt_project.yml with their own schema. Start a local Postgres, install dbt-postgres, write a local profile that is not committed, seed with load_source_data true, run dbt build, then dbt build --select package:dbt_project_evaluator. For each failing fct_ test, either fix the project, adding not_null and unique tests on primary key columns and descriptions on undocumented columns written from the staging SQL, or add a row to the exceptions seed with a comment explaining why the finding is accepted. Leave the coverage target vars at their defaults. Run the evaluator and dbt build again and paste both outputs in the pull request, with a table of findings and what was done for each.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that adds one package to `packages.yml` and `package-lock.yml`, a `dbt_project_evaluator` block in `dbt_project.yml`, an exceptions seed, tests and descriptions in five model YAML files, and no SQL changes. The description has a table with one row per `fct_` finding, the model or column it named, and either the fix or the exception row, followed by the evaluator output before and after and the final `dbt build` summary from the sandbox.

The repository's CI triggers dbt Cloud jobs by API, so it does not run the evaluator on the pull request; the description names the command to add to the PR job so it runs there from then on.

A reviewer checks that every column description matches what the staging SQL produces, that each exception row has a reason they agree with, that the package's models are disabled where they should not build, and that the new primary key tests pass on their warehouse. They merge and add `dbt build --select package:dbt_project_evaluator` to the dbt Cloud CI job.

## Run it on every dbt project that has grown past its conventions

A team with several dbt projects sends one `POST /o/{org}/sessions` per repository with this prompt and the repository in `resources`. Each run installs the evaluator in that project, reads its own findings, and fixes or records each one, so a project with good test coverage but no descriptions gets descriptions, and each run opens its own pull request.

Send a `request_id` with each call so a retry returns the run already started, and set a `webhook_url` on the key so `run.completed` arrives with `session_url` when the turn ends and your handler posts the pull request where the analytics engineers read it.

## Why run it on Rush?

The evaluator is only useful if someone acts on its findings, and acting on them means reading the SQL behind every undocumented column and deciding, finding by finding, between a fix and an exception. In a Rush sandbox the agent builds the project against a real Postgres, runs the evaluator twice, and keeps both reports and every YAML change in the run's record beside the pull request.

On a laptop this is an afternoon of warehouse setup and a long tail of YAML edits that nobody schedules. The agent does the first pass in one run, and a follow-up such as "also add freshness windows for the two sources that have a loaded_at_field" wakes the same parked sandbox and lands on the same branch.

## Frequently asked questions

### Why run against Postgres and not our warehouse?

The repository's CI reaches its warehouses through dbt Cloud jobs and an API key the sandbox does not have, and `macros/cents_to_dollars.sql` already carries a `postgres__` implementation, so Postgres is a supported target here. The findings are about YAML and project structure, which do not depend on the adapter; the reviewer runs the new tests on their warehouse before merging.

### What goes in the exceptions seed?

A finding the team accepts on purpose, with a comment saying why: here the `supplies` mart that selects straight from its staging model, kept because the semantic layer names it, and the four source tables with no load timestamp. The seed is the package's own mechanism, so the evaluator stays green and the reasons live in the repository.

### Will the evaluator's models build in production?

Only if the reader wants them to. The `dbt_project.yml` block gives them their own schema and an `+enabled` condition on the target name, so a production job skips them unless it is configured to run them. The CI job is where the command belongs, which the pull request description says.

### Does this change any model SQL?

No. Every change is in `packages.yml`, `dbt_project.yml`, the model YAML files and one seed. The descriptions are derived from the staging SQL and the tests assert properties the `entity: primary` declarations already claim, so a failing new test would mean the data does not match what the YAML already says.

## More dbt jobs

- [Add dbt Exposures for Your Dashboards from a Linear Issue](https://getrush.ai/run/add-exposures/dbt/)
- [Add a GitHub Actions CI Job That Runs dbt build on Every Pull Request](https://getrush.ai/run/add-github-actions-ci/dbt/)
- [Add Model Contracts to a dbt Project](https://getrush.ai/run/add-model-contracts/dbt/)

## More jobs like this: Hand off a long task

- [Add Deep Links to an Expo App](https://getrush.ai/run/add-deep-links/expo/)
- [Add Deep Links to a Flutter App](https://getrush.ai/run/add-deep-links/flutter/)
- [Add Deep Links to a React Native App](https://getrush.ai/run/add-deep-links/react-native/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
