# Add a Post-Deploy Smoke Test to a Fly.io Workflow

Yes. An agent adds a workflow job that runs after flyctl deploy, curls the health check and runs read-only Playwright specs against the live app, then opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-post-deploy-smoke-test/fly/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads how a deploy happens today. In a React Router app laid out like epicweb-dev/epic-stack, `.github/workflows/deploy.yml` runs `lint`, `typecheck`, `vitest` and `playwright` jobs on every push and pull request, a `container` job on pushes that runs `flyctl deploy --build-only --push --image-label ${{ github.sha }}` into the staging app on `dev` and the production app on `main`, and a `deploy` job that needs all of them and runs `flyctl deploy --image registry.fly.io/<app>:${{ github.sha }}`. The app name comes from `fly.toml` through `SebRollen/toml-action`, the staging app is that name with `-staging`, and nothing runs after the deploy. `fly.toml` declares an HTTP check on `/resources/healthcheck` every ten seconds and one on `/litefs/health` every thirty, with `auto_rollback = true`, so a deploy whose machine never passes its checks is rolled back by Fly. A deploy whose checks pass but whose pages break is not: `app/routes/resources/healthcheck.tsx` runs `SELECT 1` and a `HEAD` request to the app's own host, which says the server answers, not that the home page renders.

The new `smoke` job needs `deploy`, runs only on `push` like the two jobs before it, and computes the URL from the same toml-action output: `https://<app>-staging.fly.dev` on `dev` and `https://<app>.fly.dev` on `main`, with `ALLOWED_HOSTS` already covering `${FLY_APP_NAME}.fly.dev` as `.env.example` notes. Its first step is `curl --fail --silent --show-error --retry` against `/resources/healthcheck`, with retries spread over the grace period, expecting the literal `OK` the route returns. Its second step runs Playwright with a separate config.

The existing end-to-end tests cannot run against a deployed app, and the agent says why in the pull request: `playwright.config.ts` always starts a `webServer` with `npm run start:mocks` on CI, the specs read mocked email through `tests/mocks/utils.ts` and create users in the local SQLite database through `tests/db-utils.ts`, and `tests/e2e/onboarding.test.ts` asserts `email.from` is `hello@epicstack.dev`. So the agent adds `playwright.smoke.config.ts` with `baseURL` from `PLAYWRIGHT_BASE_URL`, no `webServer`, one `chromium` project and `testDir: './tests/smoke'`, and writes read-only specs there: the home page renders its heading, `/login` shows the username and password fields, `/users` answers a search from `tests/e2e/search.test.ts`'s pattern without creating anything, and `/resources/healthcheck` returns `OK`. No spec signs up, logs in or posts a form, because a smoke test that writes to production is a bug of its own.

The agent checks the job locally before pushing: `npm run build`, then `npm run start:mocks` in the sandbox with `PORT` set, then `PLAYWRIGHT_BASE_URL=http://localhost:3000 npx playwright test --config playwright.smoke.config.ts`, which proves the specs pass against a server they did not start. The job uploads `playwright-report/` as an artifact on failure the way the existing `playwright` job does, and the workflow's `concurrency` group already cancels a superseded run. Rolling back is documented in the job's summary rather than automated: the `container` job labels every image with the commit SHA, so redeploying the previous commit's image is one `flyctl deploy --image` with that SHA.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add a post-deploy smoke test to the Fly workflow. Read .github/workflows/deploy.yml, fly.toml, .env.example, app/routes/resources/healthcheck.tsx, playwright.config.ts, tests/e2e/search.test.ts and tests/e2e/static-files.test.ts first. Add a smoke job that needs deploy, runs on push only, derives the staging or production URL from the fly.toml app name, curls /resources/healthcheck with retries, then runs Playwright with a new playwright.smoke.config.ts that has no webServer and takes PLAYWRIGHT_BASE_URL. Write read-only specs under tests/smoke for the home page, the login page, a user search and the health check, and upload the report on failure. Explain in the PR why the existing e2e specs cannot run against a deployed app. Prove the specs locally against npm run start:mocks, quote the results and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that changes `.github/workflows/deploy.yml`, adds `playwright.smoke.config.ts` and a `tests/smoke/` directory with a handful of specs, and adds a short section to `docs/deployment.md` describing the job and the redeploy-by-SHA rollback. `playwright.config.ts` and `tests/e2e/` are untouched.

The description holds the local Playwright output for the smoke config against `npm run start:mocks`, the exact `curl` line with its retry flags, the list of what each spec asserts and a sentence on why none of them writes, and the two URLs the job will target. The repository's `Deploy` workflow runs on the branch as a pull request, where the new job is skipped by its `push` condition, and the YAML is validated by the run itself.

Before merging, a reviewer decides whether a failed smoke job should also post to a channel, which the Fly health checks do not, and whether the production URL in the job should be the custom domain from `ALLOWED_HOSTS` instead of the `fly.dev` name. The agent uses the `fly.dev` name and says so.

## Run it on every deploy workflow without a check after it

A team that deploys several apps to Fly starts one run per repository with `POST /o/{org}/sessions`, this prompt as `input` and the repository in `resources`. Each run reads its own `deploy.yml` and `fly.toml`, so an app with a custom domain gets that host in the job, an app with no end-to-end suite gets the curl step and a single page spec, and one that already runs something after deploy gets only what is missing.

Start the batch with one `request_id` per repository, a UUID v5 of its full name, so a retried call returns the run it already started, and read each result from the `session_url` in the key's `webhook_url` event when the run finishes.

## Why run it on Rush?

A workflow change is only trusted once the specs have run against a server the test did not start, which takes a built app, a mocked production start and a browser, and few people do that on a laptop for a YAML job. The sandbox builds the app, starts it, runs the smoke config against it and keeps the report, so the pull request opens with that evidence attached.

The run's record shows every spec and what it asserted, so a reviewer checks that nothing writes to production rather than taking that on faith. A company with many Fly apps starts the same request across them at once, and each pull request carries that app's own URL and pages.

## Frequently asked questions

### Why not run the existing e2e tests against the deployed app?

They need the mock server, the local SQLite database and the seeded data: `playwright.config.ts` starts `npm run start:mocks`, the specs create users through `tests/db-utils.ts` and read email from the mock's fixtures. Against a live app they would fail at the first database call or, worse, create real accounts.

### What does the job do when the smoke test fails?

It fails the workflow and uploads the Playwright report, so the failure is visible on the commit and in the run. Fly's `auto_rollback` only covers a deploy whose health checks never pass, so a broken page is rolled back by redeploying the previous commit's image, which the `container` job labelled with its SHA.

### Does this slow the deploy?

The job starts after `deploy` finishes, so the new version is live either way and the job only decides whether to flag it. The curl step waits through the grace period in `fly.toml` with retries, and the specs are a few page loads.

### Will the job run on pull requests?

No. It carries the same `github.event_name == 'push'` condition as `container` and `deploy`, because a pull request builds and tests but never deploys. On a pull request the job shows as skipped.

## More jobs like this: Hand off a long task

- [Add pre-commit Hooks to an Ansible Role](https://getrush.ai/run/add-pre-commit-hooks/ansible/)
- [Add pre-commit Hooks to a CMake C++ Project](https://getrush.ai/run/add-pre-commit-hooks/cmake/)
- [Add dbt-checkpoint Pre-commit Hooks to a dbt Project](https://getrush.ai/run/add-pre-commit-hooks/dbt/)
- [Add Pre-commit Hooks to a Kubernetes Manifest Repository](https://getrush.ai/run/add-pre-commit-hooks/kubernetes/)
- [Add Pre-commit Hooks to a Python Project](https://getrush.ai/run/add-pre-commit-hooks/python/)
- [Add Pre-commit Hooks to a Reflex App](https://getrush.ai/run/add-pre-commit-hooks/reflex/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
