What the agent does
In a Next.js app laid out like vercel/chatbot, playwright.config.ts defines one project, e2e, matching e2e/.*.test.ts, with a per-test timeout of 240 seconds, the same expect.timeout, retries: 0, two workers, an html reporter, trace: "retain-on-failure", and a webServer that runs pnpm dev and waits on /ping, which proxy.ts answers with pong before any auth check. The suite is twenty-three tests in four files: tests/e2e/chat.test.ts, api.test.ts, auth.test.ts and model-selector.test.ts, with a chatPage fixture from tests/fixtures.ts over tests/pages/chat.ts. The test script exports PLAYWRIGHT=True, which lib/constants.ts reads as isTestEnvironment, so lib/ai/providers.ts swaps the gateway for the MockLanguageModelV3 in lib/ai/models.mock.ts. .github/workflows/playwright.yml runs pnpm test on every push and pull request with AUTH_SECRET, POSTGRES_URL, BLOB_READ_WRITE_TOKEN and REDIS_URL from secrets, under a thirty minute job timeout. @playwright/test is 1.61.1.
The agent picks the smallest set that proves the app serves, signs in, talks to the database and streams an answer: home page loads with input field from chat.test.ts, which also exercises the guest redirect in proxy.ts and createGuestUser in lib/db/queries.ts; login page renders correctly from auth.test.ts; displays a model button from model-selector.test.ts; and sends message and receives AI response from api.test.ts, which goes through the chat route, getMessageCountByUserId, saveChat and the stream. Each gets { tag: '@smoke' } in the details object Playwright's test() takes, so pnpm exec playwright test --grep @smoke runs them alone. A second project named smoke is added to the config with the same testMatch and grep: /@smoke/, and a test:smoke script runs playwright test --project smoke. The e2e project is unchanged, so the full suite still runs every test, tagged or not.
For a deployed target, the config learns to read PLAYWRIGHT_TEST_BASE_URL, a name lib/constants.ts already recognizes: when it is set, use.baseURL is that address and webServer is left out, so the smoke project runs against a preview deployment instead of starting pnpm dev. The agent adds a smoke job to playwright.yml that runs before the full job with --project smoke, and the full job gets needs: smoke, so a broken deploy fails in the time four tests take rather than at the end of twenty-three. It notes in the pull request that against a real preview the server decides the model from its own environment, so the receives AI response check there costs real model tokens, and leaves that test in the set because it is the one that proves the stream.
Proof runs in the sandbox: the agent installs PostgreSQL with apt-get, sets POSTGRES_URL and AUTH_SECRET, runs pnpm install, pnpm db:migrate and pnpm exec playwright install --with-deps chromium, then pnpm test:smoke followed by the full pnpm test, and quotes the reporter's summary line for each. REDIS_URL is left unset, since lib/ratelimit.ts only checks the rate limit in production and getStreamContext in the chat route tolerates a missing resumable stream. It runs pnpm check, the Biome lint the lint.yml workflow runs, on the changed files.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add a Playwright smoke suite to this Next.js app. Read playwright.config.ts, package.json, every file under tests/, proxy.ts, lib/constants.ts, lib/ai/providers.ts and .github/workflows/playwright.yml first. Tag the smallest set of existing tests that prove the app serves, renders the login page, shows the model selector and streams an answer with tag: '@smoke', add a smoke project to the config that greps for it, and a test:smoke script. Make the config use PLAYWRIGHT_TEST_BASE_URL as baseURL with no webServer when it is set. Add a smoke job to the Playwright workflow that runs first and make the full job depend on it. Install PostgreSQL in the sandbox, run pnpm install, pnpm db:migrate and playwright install, then run pnpm test:smoke and the full pnpm test and quote both summaries. Run pnpm check. Do not remove or rewrite any existing test. Open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that adds a tag to four existing tests, a smoke project and the PLAYWRIGHT_TEST_BASE_URL branch to playwright.config.ts, a test:smoke script to package.json, and a smoke job plus a needs line to .github/workflows/playwright.yml. No test body changes. The description names the four tests and what each proves, quotes the two reporter summaries from the sandbox, and quotes pnpm check.
The run's record keeps the PostgreSQL setup, the migration, the browser install and both suite runs with their traces on failure, so a reviewer can see the smoke set pass on its own before the full suite ran.
Before merging, a reviewer decides whether the streaming check belongs in a smoke run against a live preview, where it spends model tokens, or whether the smoke project should stop at the page and the database. Either is one tag.
Run it on every preview deployment
The smoke project exists to run against a URL, so the natural trigger is a deploy. A handler on the reader's server receives the hosting provider's deployment notification, reads the preview address from it, and calls POST /o/{org}/sessions with a prompt that asks the agent to run pnpm test:smoke with PLAYWRIGHT_TEST_BASE_URL set to that address and to answer with the reporter summary and a screenshot of any failure. The handler sends a request_id that is a UUID v5 of the deployment id, so a redelivered notification returns the run already started.
The run needs no repository write, so the prompt asks for an answer and no push. The key's run.completed webhook carries the agent's answer, which the handler posts back on the commit or the pull request the deploy came from.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
A smoke suite is a selection problem more than a writing problem: which four of twenty-three tests prove the app is alive, and what does each one actually touch. In a Rush sandbox the agent reads the tests, the proxy and the provider switch to make that call, then runs both suites against a database it stands up itself, so the pull request shows the smoke set green on its own rather than as a subset nobody ran separately.
Both runs and their traces stay in the run's record and the pull request quotes the summaries, so review is checking the chosen tests and the workflow ordering rather than running Playwright locally. A team with several Next.js apps on the same test layout runs the job on each at once, one sandbox and one pull request each.
Frequently asked questions
Why tag existing tests instead of writing new ones?
Because the tests that prove the app is up already exist in chat.test.ts, auth.test.ts, model-selector.test.ts and api.test.ts; writing copies would mean two tests to keep in step. A tag is one line, --grep @smoke selects it, and the full e2e project still runs the test as before.
Does the smoke run against a preview use the mock model?
No. The mock is chosen on the server by its own environment in lib/ai/providers.ts, and a deployed preview does not set PLAYWRIGHT. The receives AI response test therefore hits the real gateway and spends tokens, which the pull request says plainly so the reviewer can drop that test from the smoke set if they prefer.
What does the webServer change do?
When PLAYWRIGHT_TEST_BASE_URL is set, the config uses it as baseURL and defines no webServer, so Playwright does not try to start pnpm dev on a machine that is only running tests. When it is unset, nothing changes: the dev server starts and waits on /ping as today.
Will the full suite still run on every pull request?
Yes. The full job in playwright.yml now has needs: smoke, so it runs after the smoke job passes and is skipped when it fails. A pull request that breaks the home page fails in the time four tests take, and one that passes smoke still gets all twenty-three.