# Add Source Freshness Checks to a dbt Project

Yes. An agent adds freshness thresholds to the sources that have a load timestamp, disables the check where there is none, runs dbt source freshness, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-source-freshness/dbt/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads the sources before it adds a threshold to them. In a project shaped like dbt-labs/jaffle-shop, `models/staging/__sources.yml` declares one source, `ecom` in schema `raw`, with six tables: `raw_customers`, `raw_orders`, `raw_items`, `raw_stores`, `raw_products` and `raw_supplies`. Two of them already carry a `loaded_at_field`, `ordered_at` on `raw_orders` and `opened_at` on `raw_stores`, but no table and no source has a `freshness` block, so `dbt source freshness` has nothing to check. `dbt_project.yml` names the project `jaffle_shop` at version 3.0.0, requires dbt `>=2.0.0`, materialises staging as views and marts as tables, and enables the seeds under `seeds/jaffle-data` only when the `load_source_data` var is set.

The seeds say which timestamps are real. `raw_orders.csv` has an `ordered_at` per row, a time the event happened, which is a reasonable stand-in for a load time on an append-only orders feed. `raw_stores.csv` has `opened_at`, the day a store opened, which is years old by design and would fail any freshness window. `raw_customers.csv` (`id`, `name`), `raw_items.csv` (`id`, `order_id`, `sku`), `raw_products.csv` and `raw_supplies.csv` carry no timestamp at all. `stg_orders.sql` truncates `ordered_at` to the day, and `stg_locations.sql` renames `opened_at` to `opened_date`.

The agent writes the thresholds where they mean something. `raw_orders` gets a `freshness` block with the `warn_after` and `error_after` from the prompt on its existing `ordered_at`; `raw_stores` gets `freshness: null` with a comment that `opened_at` is an opening date, not a load time, so the table is excluded rather than failing forever; the four tables without a timestamp are left out of the check with the same explicit `null` and a note naming the ingestion column to use if the warehouse adds one. The source-level default stays unset so a new table does not inherit a window it cannot meet. The agent adds a `dbt source freshness` line to the README's project setup and leaves `packages.yml`, with `dbt_utils` 1.4.1 and `dbt-audit-helper`, as it is.

Then it runs the check. `dbt parse` has to accept the YAML with no warehouse. For a real run the agent starts a Postgres in its sandbox, installs the `dbt-postgres` adapter, seeds the raw tables with `dbt seed --vars '{"load_source_data": true}'` as the `Taskfile.yml` `seed` task does, and runs `dbt source freshness`. With the shipped seed data, whose newest `ordered_at` is in 2025, the `raw_orders` check reports an error, which is the check working and is what the pull request says; `dbt build` afterwards shows the models and tests are unaffected. The repository's CI runs dbt Cloud jobs through `.github/workflows/scripts/dbt_cloud_run_job.py`, which the agent cannot change, so the description says to add the freshness step to those jobs.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add source freshness checks to this dbt project. Read dbt_project.yml, models/staging/__sources.yml, every file under models/staging, the header row of every CSV under seeds/jaffle-data, packages.yml, Taskfile.yml, README.md and .github/workflows/ci.yml first. For each source table decide whether it has a column that records when data arrived: give raw_orders a freshness block with warn_after 1 day and error_after 3 days on ordered_at; set freshness to null on raw_stores because opened_at is an opening date, and on every table with no timestamp, each with a comment saying why. Leave the source-level freshness unset. Run dbt parse, then start Postgres in the sandbox, install dbt-postgres, seed with load_source_data true, run dbt source freshness and dbt build, and report what the freshness run says about the seed data. Add a dbt source freshness step to the README setup. Do not change dbt Cloud job settings; say in the pull request which job should run the check. Open a pull request listing each table and its decision.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request on its own branch that changes `models/staging/__sources.yml` and `README.md` and nothing else. The YAML diff adds a `freshness` block to `raw_orders` on its existing `ordered_at`, `freshness: null` with a comment on `raw_stores` and on the four tables without a timestamp, and no source-level default. The description lists all six tables with the decision for each, shows the `dbt source freshness` output from the sandbox, and names the dbt Cloud job that should run `dbt source freshness` before the models.

The record is the proof. The trajectory at `GET /o/{org}/sessions/{id}/messages` keeps the `dbt parse` result, the seed run, the freshness run with its per-table status and the `dbt build` summary beside the commands that produced them, so a reviewer sees the check executed rather than reading that it should. The repository's `dbt Cloud CI` workflow runs its PR jobs on the pull request as it does for any change to the project.

Before merging, a reviewer checks the windows against how often the orders feed actually lands, decides whether `ordered_at` is close enough to a load time or whether the warehouse exposes an ingestion column that should replace it, and adds the freshness step to the Cloud job the description names. Nothing changes in the warehouse until the job runs it.

## Run it on every new source added to the project

An analytics engineer starts this from the terminal when a source lands without a freshness block: one `POST /o/{org}/sessions` with the repository in `resources` and this prompt, then walks away. The run clones the project into its own sandbox, seeds a Postgres of its own, runs the checks and opens the pull request; the sandbox parks when the turn ends and a follow-up through `POST /o/{org}/sessions/{id}/messages` wakes it, for a tighter window or a different column.

With a `webhook_url` on the API key, `run.completed` arrives with `data.session_url` to read for the pull request link, and a `request_id` per source keeps a retried request from starting a second run.

## Why run it on Rush?

The honest way to test a freshness threshold is to run `dbt source freshness` against loaded data, and on a laptop that means a warehouse profile, seeded raw tables and an adapter install before the first result. In a Rush sandbox the agent brings up Postgres, installs `dbt-postgres`, seeds from `seeds/jaffle-data` and runs the check, reads the per-table result and fixes the YAML, with nothing left running on your machine and no credentials to your real warehouse in the loop.

The pull request carries its proof: the freshness output and the `dbt build` summary are in the transcript beside the commands that produced them, and every table's decision is written down with its reason. The same prompt runs on a second project in a second sandbox at the same time.

## Frequently asked questions

### Why does the freshness run report an error in the sandbox?

Because the shipped seed data is old: the newest `ordered_at` in `raw_orders.csv` is from 2025, far outside a three-day window, so the check reports `error` for that table. That is the check doing its job on stale data, and the pull request says so rather than hiding it. Against your warehouse's live feed the same window reports `pass` when the data is current.

### Why is raw_stores excluded when it already has a loaded_at_field?

Because `opened_at` records the day a store opened, not when the row arrived, and the stores in the seed opened years ago. A window on that column would error on every run and teach people to ignore the check. The agent sets `freshness: null` with a comment saying exactly that, and keeps `loaded_at_field` so a future ingestion column is a one-line change.

### What about the tables with no timestamp?

`raw_customers`, `raw_items`, `raw_products` and `raw_supplies` have no column that says when data arrived, so a freshness check cannot be computed for them. The agent marks each `freshness: null` with a comment naming the kind of column that would enable it, such as a warehouse ingestion timestamp, and lists them in the pull request so the decision is visible.

### Does this change the dbt Cloud jobs?

No. The jobs are configured in dbt Cloud and started by `.github/workflows/scripts/dbt_cloud_run_job.py`, which the repository does not define the steps of, so the agent cannot add a command there. The pull request names the job and the `dbt source freshness` step to add, and the YAML works as soon as any job runs that command.

## More dbt jobs

- [Add SQLFluff Linting to a dbt Project](https://getrush.ai/run/add-sqlfluff-linting/dbt/)
- [Add Unit Tests to a dbt Project](https://getrush.ai/run/add-unit-tests/dbt/)
- [Convert a dbt Mart to an Incremental Model](https://getrush.ai/run/convert-to-incremental-model/dbt/)

## More jobs like this: Hand off a long task

- [Add a Splash Screen to an Electron App](https://getrush.ai/run/add-splash-screen/electron/)
- [Add SpotBugs to a Maven Build](https://getrush.ai/run/add-spotbugs/maven/)
- [Prefetch tRPC Queries on the Server in a Next.js Pages Router App](https://getrush.ai/run/add-ssr-prefetching/trpc/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
