All jobs

Add a Post-Deploy Smoke Test to a Fly.io Workflow

Yes. An agent adds a workflow job that runs after flyctl deploy, curls the health check and runs read-only Playwright specs against the live app, then opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how a deploy happens today. In a React Router app laid out like epicweb-dev/epic-stack, .github/workflows/deploy.yml runs lint, typecheck, vitest and playwright jobs on every push and pull request, a container job on pushes that runs flyctl deploy --build-only --push --image-label ${{ github.sha }} into the staging app on dev and the production app on main, and a deploy job that needs all of them and runs flyctl deploy --image registry.fly.io/<app>:${{ github.sha }}. The app name comes from fly.toml through SebRollen/toml-action, the staging app is that name with -staging, and nothing runs after the deploy. fly.toml declares an HTTP check on /resources/healthcheck every ten seconds and one on /litefs/health every thirty, with auto_rollback = true, so a deploy whose machine never passes its checks is rolled back by Fly. A deploy whose checks pass but whose pages break is not: app/routes/resources/healthcheck.tsx runs SELECT 1 and a HEAD request to the app's own host, which says the server answers, not that the home page renders.

The new smoke job needs deploy, runs only on push like the two jobs before it, and computes the URL from the same toml-action output: https://<app>-staging.fly.dev on dev and https://<app>.fly.dev on main, with ALLOWED_HOSTS already covering ${FLY_APP_NAME}.fly.dev as .env.example notes. Its first step is curl --fail --silent --show-error --retry against /resources/healthcheck, with retries spread over the grace period, expecting the literal OK the route returns. Its second step runs Playwright with a separate config.

The existing end-to-end tests cannot run against a deployed app, and the agent says why in the pull request: playwright.config.ts always starts a webServer with npm run start:mocks on CI, the specs read mocked email through tests/mocks/utils.ts and create users in the local SQLite database through tests/db-utils.ts, and tests/e2e/onboarding.test.ts asserts email.from is hello@epicstack.dev. So the agent adds playwright.smoke.config.ts with baseURL from PLAYWRIGHT_BASE_URL, no webServer, one chromium project and testDir: './tests/smoke', and writes read-only specs there: the home page renders its heading, /login shows the username and password fields, /users answers a search from tests/e2e/search.test.ts's pattern without creating anything, and /resources/healthcheck returns OK. No spec signs up, logs in or posts a form, because a smoke test that writes to production is a bug of its own.

The agent checks the job locally before pushing: npm run build, then npm run start:mocks in the sandbox with PORT set, then PLAYWRIGHT_BASE_URL=http://localhost:3000 npx playwright test --config playwright.smoke.config.ts, which proves the specs pass against a server they did not start. The job uploads playwright-report/ as an artifact on failure the way the existing playwright job does, and the workflow's concurrency group already cancels a superseded run. Rolling back is documented in the job's summary rather than automated: the container job labels every image with the commit SHA, so redeploying the previous commit's image is one flyctl deploy --image with that SHA.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add a post-deploy smoke test to the Fly workflow. Read .github/workflows/deploy.yml, fly.toml, .env.example, app/routes/resources/healthcheck.tsx, playwright.config.ts, tests/e2e/search.test.ts and tests/e2e/static-files.test.ts first. Add a smoke job that needs deploy, runs on push only, derives the staging or production URL from the fly.toml app name, curls /resources/healthcheck with retries, then runs Playwright with a new playwright.smoke.config.ts that has no webServer and takes PLAYWRIGHT_BASE_URL. Write read-only specs under tests/smoke for the home page, the login page, a user search and the health check, and upload the report on failure. Explain in the PR why the existing e2e specs cannot run against a deployed app. Prove the specs locally against npm run start:mocks, quote the results and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that changes .github/workflows/deploy.yml, adds playwright.smoke.config.ts and a tests/smoke/ directory with a handful of specs, and adds a short section to docs/deployment.md describing the job and the redeploy-by-SHA rollback. playwright.config.ts and tests/e2e/ are untouched.

The description holds the local Playwright output for the smoke config against npm run start:mocks, the exact curl line with its retry flags, the list of what each spec asserts and a sentence on why none of them writes, and the two URLs the job will target. The repository's Deploy workflow runs on the branch as a pull request, where the new job is skipped by its push condition, and the YAML is validated by the run itself.

Before merging, a reviewer decides whether a failed smoke job should also post to a channel, which the Fly health checks do not, and whether the production URL in the job should be the custom domain from ALLOWED_HOSTS instead of the fly.dev name. The agent uses the fly.dev name and says so.

Run it on every deploy workflow without a check after it

A team that deploys several apps to Fly starts one run per repository with POST /o/{org}/sessions, this prompt as input and the repository in resources. Each run reads its own deploy.yml and fly.toml, so an app with a custom domain gets that host in the job, an app with no end-to-end suite gets the curl step and a single page spec, and one that already runs something after deploy gets only what is missing.

Start the batch with one request_id per repository, a UUID v5 of its full name, so a retried call returns the run it already started, and read each result from the session_url in the key's webhook_url event when the run finishes.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

A workflow change is only trusted once the specs have run against a server the test did not start, which takes a built app, a mocked production start and a browser, and few people do that on a laptop for a YAML job. The sandbox builds the app, starts it, runs the smoke config against it and keeps the report, so the pull request opens with that evidence attached.

The run's record shows every spec and what it asserted, so a reviewer checks that nothing writes to production rather than taking that on faith. A company with many Fly apps starts the same request across them at once, and each pull request carries that app's own URL and pages.

Frequently asked questions

Why not run the existing e2e tests against the deployed app?

They need the mock server, the local SQLite database and the seeded data: playwright.config.ts starts npm run start:mocks, the specs create users through tests/db-utils.ts and read email from the mock's fixtures. Against a live app they would fail at the first database call or, worse, create real accounts.

What does the job do when the smoke test fails?

It fails the workflow and uploads the Playwright report, so the failure is visible on the commit and in the run. Fly's auto_rollback only covers a deploy whose health checks never pass, so a broken page is rolled back by redeploying the previous commit's image, which the container job labelled with its SHA.

Does this slow the deploy?

The job starts after deploy finishes, so the new version is live either way and the job only decides whether to flag it. The curl step waits through the grace period in fly.toml with retries, and the specs are a few page loads.

Will the job run on pull requests?

No. It carries the same github.event_name == 'push' condition as container and deploy, because a pull request builds and tests but never deploys. On a pull request the job shows as skipped.

More jobs like this: Hand off a long task

Keep reading