All jobs

Make CI Fail When SQLModel Models Change Without an Alembic Migration

Yes. An agent adds an alembic check step to your backend workflow, proves it catches a model change with no revision, documents the rule and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how migrations run today before adding a check. In a backend like the one in fastapi/full-stack-fastapi-template, backend/pyproject.toml requires Python 3.14 with alembic>=1.19.1, sqlmodel>=0.0.46 and psycopg[binary], and excludes the alembic directory from both ruff and mypy. backend/alembic.ini points script_location at app/alembic, whose env.py sets target_metadata = SQLModel.metadata from app.models, reads the URL from settings.DATABASE_URL, and passes compare_type=True in both the offline and online paths. app/alembic/versions/ holds five revisions ending at fe56fa70289e, which added created_at to user and item. app/models.py defines two tables, User with a UUID primary key, a unique indexed email of at most 255 characters, hashed_password, created_at and a cascade-deleting items relationship, and Item with a UUID key, title, description, created_at and an owner_id foreign key with ondelete="CASCADE". app/core/db.py leaves SQLModel.metadata.create_all commented out with a note that tables come from Alembic, and scripts/prestart.sh runs alembic upgrade head and then python app/initial_data.py.

Then it reads the workflow the check belongs in. .github/workflows/test-backend.yml runs on pushes to master and on every pull request with a five-minute timeout: it installs Python from .python-version, installs uv, starts db and mailpit with docker compose up -d --wait, runs scripts/prestart.sh from backend/ in a step named Migrate DB, then runs the tests and enforces 90 percent coverage. compose.yml runs postgres:18 with a pg_isready healthcheck. Nothing in the workflow compares the models with the migrated schema, and backend/README.md relies on the developer remembering to run alembic revision --autogenerate and commit the file.

The check itself is one command. Alembic's check runs the same comparison as revision --autogenerate and stops before writing a file: it prints No new upgrade operations detected. and exits zero when the models match the database, and prints FAILED: New upgrade operations detected: with the pending operations and a non-zero code when they do not. The agent adds a Check migrations step right after Migrate DB, running uv run alembic check in backend/, so it compares SQLModel.metadata against a database that upgrade head just brought to the latest revision, inside the same job and the same five minutes. It adds the command to the Migrations section of backend/README.md beside revision --autogenerate and upgrade head, so the rule a pull request will enforce is written where developers already look.

It proves the step works before asking anyone to trust it. The sandbox has no Docker, so the agent installs PostgreSQL with apt, creates the app database, points DATABASE_URL at it and runs scripts/prestart.sh followed by alembic check, reading the clean result. Then it adds a throwaway nickname column to Item in app/models.py, runs alembic check again, reads the add_column operation in the failure, and reverts the change. It runs ruff check, mypy app and ty check app through scripts/lint.sh and actionlint on the workflow before opening the pull request. The generate-frontend-sdk pre-commit hook watches backend/app/*.py, and since the agent changes no application module, the generated client stays as it is.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add a migration drift check to this FastAPI backend. Read backend/pyproject.toml, backend/alembic.ini, backend/app/alembic/env.py, the files in backend/app/alembic/versions, backend/app/models.py, backend/app/core/db.py, backend/scripts/prestart.sh, .github/workflows/test-backend.yml and the Migrations section of backend/README.md first. In test-backend.yml add a step named Check migrations directly after the Migrate DB step that runs uv run alembic check in the backend directory, so the models are compared against the database that upgrade head just migrated. Document the command in the README's Migrations section. Prove the step in the sandbox: install PostgreSQL, set DATABASE_URL, run scripts/prestart.sh and alembic check and record the clean output, then add a temporary column to a model, run alembic check again, record the failure naming the add_column operation, and revert the column. Run scripts/lint.sh and actionlint, change nothing else, and open a pull request that quotes both check outputs.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that changes two files: .github/workflows/test-backend.yml gains a Check migrations step between Migrate DB and Run tests, and backend/README.md gains the alembic check command in its Migrations section. The description quotes the two outputs the agent recorded in its sandbox, No new upgrade operations detected. on the committed models and the FAILED: New upgrade operations detected: block with the add_column on item after the temporary change, so the reviewer sees the step pass and fail without running anything.

The run's record keeps the PostgreSQL setup, the prestart.sh output with the five Running upgrade lines, both alembic check runs, the model edit and its revert, and the lint.sh and actionlint results.

Before merging, a reviewer checks that the step runs after the migrated database exists and inside the job's five-minute timeout, and decides whether compare_server_default=True should be added to env.py so server default changes are caught too; the pull request names that as a separate decision because it can surface existing differences.

Run it on every backend service with an Alembic directory

One service is a hand-off from your terminal; a platform team with many services sends this body once per repository to POST /o/{org}/sessions, each with its repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started instead of opening a second pull request. Each run reads that service's own env.py and workflow, so a service whose workflow has no migration step gets the step added before the check rather than a check that cannot run.

With a webhook_url on the key, each run.completed event carries the result naming the pull request, and a run.failed event names the service whose check could not be made to pass, such as one whose metadata and committed migrations already disagree and needs a person to write the missing revision.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

The drift check is only credible if someone has watched it fail. The agent has its own sandbox with PostgreSQL, a migrated database and the real models, so it can add a column, see alembic check report the add_column, and revert, all in the same run. On a laptop that proof is skipped because it means editing production models on purpose.

The command outputs stay with the run and go into the pull request verbatim, so the reviewer reads the actual failure text the next developer will see, not a description of it, and the only decision left is whether to also compare server defaults.

Frequently asked questions

Will the check fail on the current repository?

The agent runs it against the committed models and the migrated database before opening the pull request and quotes the result. In this template the five revisions match app/models.py and the output is No new upgrade operations detected.; a repository whose models already drifted gets the pending operations listed in the pull request so someone can write the missing revision first.

Does alembic check need a running database?

Yes. It compares SQLModel.metadata with a live schema, which is why the step runs after Migrate DB in the same job, where docker compose up -d --wait db and scripts/prestart.sh have already brought PostgreSQL to the latest revision.

What does it not catch?

Autogenerate does not detect renamed tables or columns, which appear as a drop plus an add, and env.py sets only compare_type=True, so a changed server default is not reported unless compare_server_default is also enabled. The pull request names that option as a follow-up rather than switching it on, because it can flag differences that already exist.

Why not run it as a pre-commit hook?

The repository's hooks are ruff, mypy, ty, typos and the frontend SDK generator, none of which needs a database. A hook that requires PostgreSQL would fail on machines without it, so the check lives in CI, where the database is already started, and the README tells developers how to run it locally.

More Alembic jobs

The same job on other stacks

More jobs like this: Hand off a long task

Keep reading