All jobs

Fix Alembic Multiple Heads After a Merge

Yes. When two merged branches leave Alembic with two heads, an agent reads both revisions, writes the merge, replays each upgrade path on PostgreSQL, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent begins by reproducing the failure, which takes one command and no database. In a backend like fastapi/full-stack-fastapi-template the five files under backend/app/alembic/versions/ form a single chain today, each down_revision naming the one before, from e2412789c190 to fe56fa70289e. Two pull requests that each autogenerate a revision from fe56fa70289e are both green on their own. Once both are merged, the directory holds two revisions with the same parent, alembic heads prints two lines, and backend/scripts/prestart.sh, which runs alembic upgrade head under set -e, stops with Alembic's message that multiple head revisions are present for the argument head. The Migrate DB step of .github/workflows/test-backend.yml runs that script, so the Test Backend workflow fails on master before a single test starts, and the next deployment would stop at the same line.

It then reads both head revisions in full before choosing a fix. If their operations touch different tables or columns, order does not matter, and the agent runs alembic merge with both revision ids and a message. That creates a revision whose down_revision is a tuple of the two and whose upgrade and downgrade are empty, rendered through the repository's own script.py.mako. No existing file changes, which matters when either revision may already have been applied somewhere. If the two revisions do collide, for example both add a column with the same name or one alters what the other drops, an empty merge would hide a real conflict. The agent then stops short of guessing: it reports the collision with the lines from both files and proposes the ordering it would impose by re-parenting one revision onto the other, for a person to confirm.

Every path through the new graph gets replayed. The compose file runs postgres:18 and the sandbox has no Docker, so the agent installs PostgreSQL with sudo apt-get install -y postgresql, notes the version, and creates the role and database that DATABASE_URL in .env describes. It upgrades an empty database to head. It upgrades a second database to the first former head only, then to head, and a third to the second former head, then to head, because environments that deployed between the two merges sit at one or the other. On each it runs alembic check, which compares the models in backend/app/models.py with the schema and reports any operation still pending, then alembic downgrade -1 and alembic upgrade head again, and finally the backend tests through scripts/tests-start.sh.

Last, it adds the guard that turns this from an outage into a red pull request. A test in the backend suite loads the script directory with Alembic's ScriptDirectory.from_config and asserts that get_heads() returns one revision, so the failure names its cause. The pull request also explains why CI missed it: a pull request's workflow tests the merge with the base as it was when the run started, so a branch whose last run predates the other merge stays green. Requiring branches to be up to date before merging, or a merge queue, closes that gap; it is a repository setting, so the agent recommends it and changes nothing there.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "The backend workflow on the default branch is failing at the migration step; the workflow_run event is at the end of this prompt. Run alembic heads in backend/ and confirm whether there is more than one head. Read backend/alembic.ini, backend/app/alembic/env.py, script.py.mako, every file under backend/app/alembic/versions, backend/app/models.py, backend/scripts/prestart.sh, .env, compose.yml and .github/workflows/test-backend.yml. Read both head revisions in full. If their operations do not touch the same tables or columns, create a merge revision with alembic merge and leave every existing revision file unchanged. If they collide, do not merge: report the conflicting lines and the ordering you propose, and stop. Install PostgreSQL in your sandbox and say which version. Replay three paths: an empty database to head, a database stopped at each former head and then upgraded to head. On each run alembic check, downgrade one step and upgrade again, then run the backend tests. Add a test that fails when the script directory has more than one head. Open a pull request with the output of every command, and explain how two green pull requests produced this and which repository setting would prevent it.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

When the two revisions are independent, the result is a pull request with one new revision file and one new test. The description shows alembic heads before and after, the two parent ids, the output of the three replays with alembic check on each, the downgrade and upgrade, and the test run. It ends with the explanation of how the heads diverged and the branch setting that would have caught it.

When the revisions collide, the result is a report and no merge revision: the two conflicting operations quoted with file and line, the order the agent would impose, and what each environment would need depending on which revision it has already applied. A reviewer who knows what production ran answers that in a follow-up to the same session, and the run continues from there with its sandbox and databases as it left them.

Run it on every failed workflow run on the default branch

Create an automation in the Project that holds the repository, with this prompt and a github trigger on workflow_run.failure. Workflows fail for many reasons, so the prompt's first step keeps unrelated failures short: the run checks alembic heads and, when there is a single head, reports that the cause lies elsewhere and ends. The trigger places the event after the prompt, and runs from forks never fire it. Leave read_only off, since this automation has to push a branch.

Set max_runs_per_hour low. A broken default branch fails every workflow that follows, and one run is enough to fix it; events past the cap wait as queued runs. To start it from your own server instead, forward GitHub's workflow_run webhook and send this body with a request_id that is a UUID v5 of the workflow run id, so a redelivery returns the run already started.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

Two heads is a five-minute fix that is easy to get wrong in a hurry: the merge command always succeeds, including when the two revisions cannot both apply. The agent reads both revisions, replays each upgrade path on a real PostgreSQL in its own sandbox, and keeps every command and its output in the run's record, so the pull request shows three databases reaching the same schema.

It also arrives without anyone being paged. The run starts from the failed workflow, works while the team is doing something else, and leaves either a pull request to approve or a specific question about ordering. The sandbox parks with its databases when the turn ends, so answering that question continues the same work.

Frequently asked questions

Why not just run alembic upgrade heads?

It would get one deployment through, and the next person to autogenerate a revision would still have to pick a parent. prestart.sh and every environment's deploy step call upgrade head, which needs a single head. A merge revision restores that for everyone.

Does a merge revision change the database?

An empty merge runs no schema operation. Applying it only updates the alembic_version table so the database is recorded at the new single head. That is why it is the safe choice when the two revisions do not interact.

What if the two revisions conflict?

Then the run does not paper over it. It reports the conflicting operations and proposes an order, and waits for a person to say which revision the deployed environments have already applied. Re-parenting a revision is only safe with that knowledge.

How do we stop this happening again?

The pull request adds a test that fails when there is more than one head, which makes the cause obvious. It also recommends requiring branches to be up to date before merging, or a merge queue, so the second pull request is tested against the first before it lands.

More Alembic jobs

More jobs like this: Fix a bug from an alert

Keep reading