All jobs

Add an Alembic Migration Drift Check to CI

Yes. An agent adds alembic check to your backend CI job after the migrate step, so a model change with no revision fails the build, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how migrations are generated and applied before it adds a check. In a FastAPI backend laid out like fastapi/full-stack-fastapi-template, backend/alembic.ini points script_location at app/alembic, and app/alembic/env.py sets target_metadata = SQLModel.metadata from app.models, reads the URL from settings.DATABASE_URL and passes compare_type=True in both the offline and online branches. backend/app/alembic/versions/ holds five revisions in one chain, from e2412789c190 (initialize models) to fe56fa70289e (add created_at to User and Item), the current head. uv.lock pins alembic 1.20.0, and pyproject.toml requires Python 3.14 with sqlmodel, psycopg and pydantic-settings.

It then reads where CI applies them. .github/workflows/test-backend.yml starts db and mailpit with docker compose up -d --wait, runs uv run bash scripts/prestart.sh from backend, which is alembic upgrade head followed by python app/initial_data.py, runs the tests through scripts/tests-start.sh under coverage, and ends with coverage report --fail-under=90. Nothing in that job compares the models with the schema the migrations produce: a field added to Item in app/models.py with no revision passes every step, because init_db in app/core/db.py leaves create_all commented out and the tests only touch columns that exist.

alembic check, in Alembic since 1.9, runs the same comparison a revision --autogenerate would, against the database env.py connects to, and exits non-zero with New upgrade operations detected and the operations it found. The agent adds one step after Migrate DB in test-backend.yml, uv run alembic check with working-directory: backend, so it runs against the database prestart.sh just upgraded, with compare_type=True catching a changed column type as well as a new column. The job's timeout-minutes: 5 stays; the step is one more connection to a database that is already up.

In its sandbox the agent starts a Postgres, exports DATABASE_URL and the other settings app/core/config.py requires, runs uv sync, uv run bash scripts/prestart.sh and uv run alembic check, and reads No new upgrade operations detected. It then adds a column to Item in app/models.py, runs the check again to capture the New upgrade operations detected output naming the column, reverts the model, and runs scripts/lint.sh (mypy, ty, ruff check, ruff format --check), since the workflow file is outside the alembic exclusion those tools have. Both outputs go in the description.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add an Alembic migration drift check to this backend's CI. Read backend/alembic.ini, backend/app/alembic/env.py, the revisions under backend/app/alembic/versions, backend/scripts/prestart.sh and .github/workflows/test-backend.yml first. Add a step that runs uv run alembic check from the backend directory right after the Migrate DB step, so a model change without a revision fails the job. Do not add a revision and do not change the models. Start a Postgres, run prestart.sh and the check, and quote the result. Then add a temporary column to Item in app/models.py, run the check again to capture the failure, revert it, run scripts/lint.sh, and open a pull request with both outputs.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request with one file changed, .github/workflows/test-backend.yml, and one step added to it: uv run alembic check in backend, between Migrate DB and Run tests. No revision is added, app/models.py is as it was, and uv.lock does not move, since alembic is already a dependency.

The description quotes the check passing against a freshly migrated database, then the failure it produced with a column added to Item and no revision: the New upgrade operations detected line and the add_column operation beneath it. The run's record holds the Postgres start, prestart.sh, both checks and the lint, so a reviewer reads the real output.

Before merging, a reviewer decides whether the check also belongs in pre-commit; the agent leaves it out, because .pre-commit-config.yaml runs no database and a check that needs one would fail on every laptop without it.

Run it on every failed backend workflow

The check itself now runs on every pull request through test-backend.yml. What a run on Rush adds is the fix: an automation with a github trigger on workflow_run.failure starts a run when that workflow fails, reads the job log for New upgrade operations detected, runs uv run alembic revision --autogenerate with a message taken from the pull request title, reads the generated file for the operations autogenerate cannot see (a renamed column reads as a drop and an add), and opens a pull request with the revision against the failing branch.

The trigger adds the failed workflow run to the prompt after it. A failure for any other reason, a test or the coverage floor, ends with no pull request, which the prompt says to do.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

A migration drift check is short to write and tedious to prove: it needs a Postgres at the right version, the migrations applied, and a deliberate drift to show it fires. On Rush the agent does all of that in a sandbox with the postgres:18 the compose file names, and the pull request opens with the failing output a reviewer would otherwise have to produce by hand.

The commands and their output stay with the run. A reviewer who wants to know whether compare_type=True was honoured reads the operations the check printed, not a sentence claiming it. For a team with several services on this template, the same request runs against each, one sandbox and one pull request apiece.

Frequently asked questions

Does alembic check need a database?

Yes. It connects the way env.py does, through settings.DATABASE_URL, and compares the models with the schema it finds. In this workflow the database is already up and migrated by the time the step runs, so it adds one connection and no new service.

What does it catch, and what does it miss?

A new or dropped table or column, a nullable change, and with compare_type=True a changed type. It does not know a rename from a drop and an add, and it does not see server defaults unless compare_server_default is set, which the agent leaves as it is in env.py.

Will it fail when a revision exists but was not applied?

No. The check compares the models with the database, and prestart.sh upgrades to head first. A revision that is in the chain but not applied would fail alembic upgrade head a step earlier.

Why not run it in pre-commit?

Because it needs a database, and .pre-commit-config.yaml has none. The CI job already has one, so the check lives there.

More Alembic jobs

The same job on other stacks

More jobs like this: Hand off a long task

Keep reading