# Add a Playwright Smoke Suite to a Next.js App

Yes. An agent tags the few end-to-end checks that prove the app is up as @smoke, adds a project that runs only those against a deployed URL, and keeps the rest.

[Run this on Rush](https://cloud.getrush.ai/run/add-smoke-tests/playwright/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

In a Next.js app laid out like vercel/chatbot, `playwright.config.ts` defines one project, `e2e`, matching `e2e/.*.test.ts`, with a per-test `timeout` of 240 seconds, the same `expect.timeout`, `retries: 0`, two workers, an `html` reporter, `trace: "retain-on-failure"`, and a `webServer` that runs `pnpm dev` and waits on `/ping`, which `proxy.ts` answers with `pong` before any auth check. The suite is twenty-three tests in four files: `tests/e2e/chat.test.ts`, `api.test.ts`, `auth.test.ts` and `model-selector.test.ts`, with a `chatPage` fixture from `tests/fixtures.ts` over `tests/pages/chat.ts`. The `test` script exports `PLAYWRIGHT=True`, which `lib/constants.ts` reads as `isTestEnvironment`, so `lib/ai/providers.ts` swaps the gateway for the `MockLanguageModelV3` in `lib/ai/models.mock.ts`. `.github/workflows/playwright.yml` runs `pnpm test` on every push and pull request with `AUTH_SECRET`, `POSTGRES_URL`, `BLOB_READ_WRITE_TOKEN` and `REDIS_URL` from secrets, under a thirty minute job timeout. `@playwright/test` is `1.61.1`.

The agent picks the smallest set that proves the app serves, signs in, talks to the database and streams an answer: `home page loads with input field` from `chat.test.ts`, which also exercises the guest redirect in `proxy.ts` and `createGuestUser` in `lib/db/queries.ts`; `login page renders correctly` from `auth.test.ts`; `displays a model button` from `model-selector.test.ts`; and `sends message and receives AI response` from `api.test.ts`, which goes through the chat route, `getMessageCountByUserId`, `saveChat` and the stream. Each gets `{ tag: '@smoke' }` in the details object Playwright's `test()` takes, so `pnpm exec playwright test --grep @smoke` runs them alone. A second project named `smoke` is added to the config with the same `testMatch` and `grep: /@smoke/`, and a `test:smoke` script runs `playwright test --project smoke`. The `e2e` project is unchanged, so the full suite still runs every test, tagged or not.

For a deployed target, the config learns to read `PLAYWRIGHT_TEST_BASE_URL`, a name `lib/constants.ts` already recognizes: when it is set, `use.baseURL` is that address and `webServer` is left out, so the smoke project runs against a preview deployment instead of starting `pnpm dev`. The agent adds a `smoke` job to `playwright.yml` that runs before the full job with `--project smoke`, and the full job gets `needs: smoke`, so a broken deploy fails in the time four tests take rather than at the end of twenty-three. It notes in the pull request that against a real preview the server decides the model from its own environment, so the `receives AI response` check there costs real model tokens, and leaves that test in the set because it is the one that proves the stream.

Proof runs in the sandbox: the agent installs PostgreSQL with `apt-get`, sets `POSTGRES_URL` and `AUTH_SECRET`, runs `pnpm install`, `pnpm db:migrate` and `pnpm exec playwright install --with-deps chromium`, then `pnpm test:smoke` followed by the full `pnpm test`, and quotes the reporter's summary line for each. `REDIS_URL` is left unset, since `lib/ratelimit.ts` only checks the rate limit in production and `getStreamContext` in the chat route tolerates a missing resumable stream. It runs `pnpm check`, the Biome lint the `lint.yml` workflow runs, on the changed files.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add a Playwright smoke suite to this Next.js app. Read playwright.config.ts, package.json, every file under tests/, proxy.ts, lib/constants.ts, lib/ai/providers.ts and .github/workflows/playwright.yml first. Tag the smallest set of existing tests that prove the app serves, renders the login page, shows the model selector and streams an answer with tag: '@smoke', add a smoke project to the config that greps for it, and a test:smoke script. Make the config use PLAYWRIGHT_TEST_BASE_URL as baseURL with no webServer when it is set. Add a smoke job to the Playwright workflow that runs first and make the full job depend on it. Install PostgreSQL in the sandbox, run pnpm install, pnpm db:migrate and playwright install, then run pnpm test:smoke and the full pnpm test and quote both summaries. Run pnpm check. Do not remove or rewrite any existing test. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that adds a tag to four existing tests, a `smoke` project and the `PLAYWRIGHT_TEST_BASE_URL` branch to `playwright.config.ts`, a `test:smoke` script to `package.json`, and a `smoke` job plus a `needs` line to `.github/workflows/playwright.yml`. No test body changes. The description names the four tests and what each proves, quotes the two reporter summaries from the sandbox, and quotes `pnpm check`.

The run's record keeps the PostgreSQL setup, the migration, the browser install and both suite runs with their traces on failure, so a reviewer can see the smoke set pass on its own before the full suite ran.

Before merging, a reviewer decides whether the streaming check belongs in a smoke run against a live preview, where it spends model tokens, or whether the smoke project should stop at the page and the database. Either is one tag.

## Run it on every preview deployment

The smoke project exists to run against a URL, so the natural trigger is a deploy. A handler on the reader's server receives the hosting provider's deployment notification, reads the preview address from it, and calls `POST /o/{org}/sessions` with a prompt that asks the agent to run `pnpm test:smoke` with `PLAYWRIGHT_TEST_BASE_URL` set to that address and to answer with the reporter summary and a screenshot of any failure. The handler sends a `request_id` that is a UUID v5 of the deployment id, so a redelivered notification returns the run already started.

The run needs no repository write, so the prompt asks for an answer and no push. The key's `run.completed` webhook carries the agent's answer, which the handler posts back on the commit or the pull request the deploy came from.

## Why run it on Rush?

A smoke suite is a selection problem more than a writing problem: which four of twenty-three tests prove the app is alive, and what does each one actually touch. In a Rush sandbox the agent reads the tests, the proxy and the provider switch to make that call, then runs both suites against a database it stands up itself, so the pull request shows the smoke set green on its own rather than as a subset nobody ran separately.

Both runs and their traces stay in the run's record and the pull request quotes the summaries, so review is checking the chosen tests and the workflow ordering rather than running Playwright locally. A team with several Next.js apps on the same test layout runs the job on each at once, one sandbox and one pull request each.

## Frequently asked questions

### Why tag existing tests instead of writing new ones?

Because the tests that prove the app is up already exist in `chat.test.ts`, `auth.test.ts`, `model-selector.test.ts` and `api.test.ts`; writing copies would mean two tests to keep in step. A tag is one line, `--grep @smoke` selects it, and the full `e2e` project still runs the test as before.

### Does the smoke run against a preview use the mock model?

No. The mock is chosen on the server by its own environment in `lib/ai/providers.ts`, and a deployed preview does not set `PLAYWRIGHT`. The `receives AI response` test therefore hits the real gateway and spends tokens, which the pull request says plainly so the reviewer can drop that test from the smoke set if they prefer.

### What does the webServer change do?

When `PLAYWRIGHT_TEST_BASE_URL` is set, the config uses it as `baseURL` and defines no `webServer`, so Playwright does not try to start `pnpm dev` on a machine that is only running tests. When it is unset, nothing changes: the dev server starts and waits on `/ping` as today.

### Will the full suite still run on every pull request?

Yes. The full job in `playwright.yml` now has `needs: smoke`, so it runs after the smoke job passes and is skipped when it fails. A pull request that breaks the home page fails in the time four tests take, and one that passes smoke still gets all twenty-three.

## More Playwright jobs

- [Add Test Retries to a Playwright Suite](https://getrush.ai/run/add-test-retries/playwright/)
- [Shard Playwright Tests Across GitHub Actions Runners](https://getrush.ai/run/add-test-sharding/playwright/)
- [Set Test, Expect and Global Timeouts in a Playwright Suite](https://getrush.ai/run/add-test-timeouts/playwright/)

## More jobs like this: Write the missing tests

- [Add Snapshot Tests to a Go CLI's Rendered Output](https://getrush.ai/run/add-snapshot-tests/go-cli/)
- [Add Snapshot Tests for the Shell Scripts a Rust CLI Generates](https://getrush.ai/run/add-snapshot-tests/rust-cli/)
- [Add Snapshot Tests to a Textual App](https://getrush.ai/run/add-snapshot-tests/textual/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
