What the agent does
The agent reads the suite and the pages before adding anything. In an app laid out like trpc/examples-next-prisma-starter, playwright.config.ts sets testDir to ./playwright, a 35-second timeout, reporter: 'github' and retries: 3 on CI, channel: 'chrome' on CI, and a webServer that runs npm run start on CI or npm run dev otherwise on port 3000, reusing a server when TEST_LOCAL=1. playwright/smoke.test.ts has two tests: one opens / and waits for text=Starter; the other fills [name=title] and [name=text] with a random nonce, clicks form [type=submit], waits for networkidle, reloads and waits for the nonce. .github/workflows/main.yml runs the suite in an e2e job with a Postgres service on trust auth, pnpm install, pnpm exec playwright install chromium, pnpm prebuild (which runs prisma generate and prisma migrate deploy), pnpm build and pnpm test-e2e, then uploads playwright/test-results with actions/upload-artifact@v2. The pages are src/pages/index.tsx, with a PostsCard on trpc.post.list.useInfiniteQuery({ limit: 5 }) and a Load more button while hasNextPage, else That's every post, and an AddPostCard whose form has labelled Title and Text fields and shows Couldn't add the post with the error message on failure; src/pages/post/[id].tsx, which renders the title, a Created date, the text and a Raw data card, or an Empty with the httpStatus and the message when post.byId fails; and src/pages/404.tsx with Page not found and a Home link.
From src/server/routers/post.ts it knows what to assert: post.list takes limit 1 to 100 and a cursor, orders by createdAt descending, fetches one extra row to find nextCursor and reverses the page; post.byId throws NOT_FOUND with No post with id '<id>'; post.add validates title 1 to 32 characters and text at least 1 with zod. prisma/seed.ts upserts one First Post, so a fresh database has a single page. The agent writes playwright/posts.test.ts with specs that read like the pages: the home page shows Latest posts and the seeded post; adding a post through the form with getByLabel('Title') and getByLabel('Text') makes it appear in the list without a reload, because AddPostCard invalidates post.list on success; a title of 33 characters shows Couldn't add the post with zod's message and adds nothing; an empty text does the same; clicking a post opens /post/<id> with the title as the heading, the Posts breadcrumb and the Raw data block; /post/does-not-exist shows 404 and No post with id 'does-not-exist' with a Back to posts link; and /nope shows the custom Page not found.
Pagination needs more than five posts and a database every test shares, so the agent writes that spec to create six posts with distinct nonces through the form, then asserts Load more is visible, clicks it, and asserts That's every post once the second page lands; it tags that spec serial and keeps the others independent by using a nonce per test and never asserting on counts. It replaces the smoke file's waitForLoadState('networkidle') and reload with expect(...).toBeVisible() on the new item, which is what the invalidation guarantees, and swaps the [name=...] selectors for getByLabel and getByRole, so a change to a field's markup that keeps its label does not break the suite. The tests read the seeded post's title from prisma/seed.ts rather than hard-coding it twice.
Then it runs the suite the way CI does: Postgres in its sandbox with the trpcdb database and trust auth as main.yml configures, DATABASE_URL and DIRECT_URL as .env shows, pnpm install, pnpm prebuild, pnpm build and pnpm test-e2e with CI=1 so the config starts npm run start against the production build. channel: 'chrome' on CI means the workflow relies on the Chrome that GitHub's runner image ships, which playwright install chromium does not provide, so the agent installs the chrome channel in its sandbox and writes that dependency into the pull request. It runs the suite three times to catch order dependence, pastes the reporter output, and notes that actions/upload-artifact@v2 in the workflow is deprecated, without changing it, since the task is tests.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Write Playwright end-to-end tests for this tRPC + Next.js app. Read playwright.config.ts, playwright/smoke.test.ts, .github/workflows/main.yml, src/pages/index.tsx, src/pages/post/[id].tsx, src/pages/404.tsx, src/server/routers/post.ts and prisma/seed.ts first. Add playwright/posts.test.ts covering: the home page lists the seeded post; adding a post through the form shows it in the list without a reload; a 33-character title and an empty text each show the add error with zod's message and add nothing; clicking a post opens its page with the title, the Posts breadcrumb and the Raw data card; an unknown post id shows 404 and the not-found message with a Back to posts link; an unknown route shows the custom 404 page; and, as a serial spec, six new posts make Load more appear and a click reaches That's every post. Use getByLabel and getByRole, a nonce per test, and no count assertions. In the smoke file, replace networkidle and reload with a visibility assertion and the name selectors with labels. Start Postgres as the workflow does, run pnpm prebuild, pnpm build and pnpm test-e2e with CI=1 three times, and open a pull request with the reporter output; note anything the workflow relies on that the config does not install. Do not change the app or the workflow.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request adds playwright/posts.test.ts with nine specs and edits playwright/smoke.test.ts to use labels and a visibility assertion. Its description shows the Playwright reporter output for three consecutive runs against the production build, the Postgres service and environment it used, and a table of each spec with the page and procedure it exercises.
Nothing changes in the app or the workflow: src/, prisma/ and main.yml are untouched. The suite still runs with pnpm test-e2e under the existing config, locally against npm run dev and on CI against npm run start, and the serial pagination spec is the only one that depends on its own ordering.
Before merging, a reviewer decides whether the pagination spec, which adds six rows to the shared test database per run, should instead seed through Prisma in a beforeAll, which the pull request sketches; and whether to pin the chrome channel dependency in the workflow or switch the config to chromium, since the sandbox needed the channel installed to run as CI does.
Run it on every night
An automation with a schedule trigger, cron 0 2 * * * and your timezone, runs this prompt in the Project's repository each night. The sandbox needs a Postgres it can reach as DATABASE_URL before pnpm prebuild, because the migrations and the seed run against it, so the prompt tells the agent to start one first the way the workflow does.
Each night opens one pull request on its own branch with the specs it added and the reporter output; a night when the pages it was asked about are already covered ends with a short note and no pull request. max_runs_per_hour on the automation keeps a mis-set schedule from starting more than one.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
End-to-end tests for a tRPC app fail for reasons that are not in the diff: a shared database, a list that refetches after a mutation, a production build that behaves differently from the dev server, and a browser channel the config expects. The agent runs the suite in its own sandbox against the real Postgres and the production build, three times, before opening anything, and writes the one dependency the workflow quietly relies on into the pull request.
The transcript keeps every command, the reporter output and the trace of a failing spec, so a reviewer can see what the browser saw without checking out the branch. Several repositories can get their missing flows covered at once, each in its own sandbox, and nothing merges until a person reads the diff.
Frequently asked questions
Why replace the smoke test's reload with an assertion?
AddPostCard invalidates post.list when the mutation succeeds, so the new post appears without a reload, and that is the behaviour worth testing. waitForLoadState('networkidle') and reload hide a broken invalidation and add seconds to every run; expect(getByText(nonce)).toBeVisible() fails if the list did not refetch.
How do the tests stay independent on one shared database?
Every post a test creates carries its own nonce, assertions look for that nonce rather than a count, and only the pagination spec, which needs six rows, is marked serial. The suite passes on an empty, seeded or already-filled database, which the three consecutive sandbox runs show.
Does the suite need the production build?
On CI, yes: the config starts npm run start when CI is set, so pnpm build must run first, which is why the sandbox runs pnpm prebuild and pnpm build the way main.yml does. Locally the same config uses npm run dev, and TEST_LOCAL=1 reuses a server already running.
What is the chrome channel finding?
playwright.config.ts sets channel: 'chrome' on CI, which uses an installed Google Chrome rather than the Chromium that playwright install chromium downloads. It works on GitHub's runners because Chrome is preinstalled there; the sandbox had to install the channel to match, and the pull request says so rather than changing the config.