What the agent does
The agent reads the existing end-to-end setup before adding screenshots to it. In a React app laid out like alan2207/bulletproof-react, apps/react-vite/playwright.config.ts has testDir: './e2e', a setup project matching *.setup.ts, and a chromium project with storageState: 'e2e/.auth/user.json' that depends on it; retries is 2 and workers is 1 on CI, the reporter is html, and webServer runs yarn dev --port 3000 with reuseExistingServer off on CI. e2e/tests/auth.setup.ts registers a user generated by @ngneat/falso through src/testing/data-generators.ts, logs out, logs in and saves the storage state. smoke.spec.ts and profile.spec.ts drive discussions and the profile page. The test-e2e script starts the MSW mock server with pm2 from mock-server.ts, .env.example-e2e points the app at it on port 8080, and .github/workflows/react-vite-ci.yml installs browsers with yarn playwright install --with-deps and uploads playwright-report/ and mocked-db.json.
Two facts shape the tests. The user is random on every run, so any screen that shows the name or initials would differ each time, and the mock database starts empty, so the discussions list is an empty state unless the test creates one. The agent writes e2e/tests/visual.spec.ts with one test per screen: the landing page at /, the login and register pages signed out, and the dashboard, the empty discussions list, the create-discussion drawer and the profile page signed in. Each calls expect(page).toHaveScreenshot() with fullPage: true and a mask for the user menu button that profile.spec.ts already locates by role, and the signed-out tests use a fresh context so the stored state does not apply.
In playwright.config.ts the agent adds expect.toHaveScreenshot defaults of animations: 'disabled' and a small maxDiffPixelRatio, so a sub-pixel antialiasing difference is not a failure and a moved button is. Baselines are platform-specific by design, since the default snapshot path carries the platform suffix, so the agent records them in its Linux sandbox with yarn playwright test visual --update-snapshots, which is what ubuntu-latest in the workflow will compare against. The .png files are committed under e2e/tests/visual.spec.ts-snapshots/.
The CI job's upload-artifact step gains test-results/ beside playwright-report/, because that is where Playwright writes the expected, actual and diff images when a screenshot test fails. The agent runs the whole suite twice in its sandbox, once to record and once to confirm the recorded baselines match, and runs yarn lint, since e2e/.eslintrc.cjs applies the Playwright plugin's rules to the new spec.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add visual regression tests to the Playwright suite in apps/react-vite. Read playwright.config.ts, e2e/tests/auth.setup.ts, e2e/tests/smoke.spec.ts, e2e/tests/profile.spec.ts, e2e/.eslintrc.cjs, .env.example-e2e, mock-server.ts, package.json, .gitignore and .github/workflows/react-vite-ci.yml first. Write e2e/tests/visual.spec.ts with full-page toHaveScreenshot tests for the landing page, the login and register pages in a fresh signed-out context, and the dashboard, empty discussions list, create-discussion drawer and profile page signed in, masking the user menu button. Set animations disabled and a small maxDiffPixelRatio under expect.toHaveScreenshot in the config. Start the mock server as the test-e2e script does, record the baselines with --update-snapshots on Linux, run the suite again to confirm they match, commit the snapshot images, add test-results to the CI artifact upload, run yarn lint, and open a pull request quoting the two runs.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that adds visual.spec.ts and its snapshot directory, changes playwright.config.ts and the workflow's artifact path, and nothing in src/. The description lists each screen and its baseline image, quotes the recording run and the confirming run, and says which element is masked and why.
The run's record keeps both Playwright runs, so a reviewer can see that the second run compared seven baselines and passed rather than assume the recording was stable.
Before merging, a reviewer looks at the committed images once, since they are the contract, and decides whether maxDiffPixelRatio should be tighter for the small screens and whether the screenshots should also be taken at a mobile viewport, which is a second project in the config.
Run it on every pull request
Once the baselines are in, CI compares every push against them in the e2e job as it stands. For the review itself, an automation in the Project with a github trigger on pull_request.opened and pull_request.synchronize runs a prompt that checks out the pull request's branch, runs yarn playwright test visual, and when a screenshot differs posts a review comment that names the screen and attaches the diff image from test-results/ as the run's artifact, so the reviewer sees what moved without opening the job. The trigger adds the event after the prompt.
A team with several Vite apps starts one run per repository with one POST /o/{org}/sessions each, this prompt as input and the repository in resources, with a request_id that is a UUID v5 of the repository's full name so a retry returns the run it already started. Each run reads its own setup project and data generators, so an app whose dashboard shows a date gets that date masked.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Screenshot baselines recorded on a laptop do not match a Linux runner, and tests written without masking fail on the second run because the generated user changed. Both mistakes are found only by running the suite twice in the environment CI uses. The agent records in its own Linux sandbox, confirms, and keeps both outputs.
The pull request is one spec, a config block and a folder of images, so review is looking at seven screens and one ratio rather than checking out the branch and running a browser. Across several apps the job runs in parallel, one sandbox and one pull request each, with baselines recorded for that app's screens.
Frequently asked questions
Why are the baselines recorded on Linux?
Because the workflow runs on ubuntu-latest, and Playwright names each snapshot with the platform it was taken on. A baseline recorded on macOS would never be compared in CI; one recorded in the agent's Linux sandbox is the one the job uses.
What happens when a screen changes on purpose?
The test fails with the expected, actual and diff images in test-results/, which the workflow now uploads. Running yarn playwright test visual --update-snapshots on Linux rewrites the baseline, and the new image in the pull request is the reviewer's evidence of the change.
Why mask the user menu rather than fix the data?
Because auth.setup.ts registers a random user from @ngneat/falso on every run and the rest of the suite depends on that. A mask over the one element that shows the name keeps the screenshot stable without changing how the suite signs in.
Does this slow the e2e job down?
By the time of seven page loads and screenshots on one worker. The suite already runs with workers: 1 on CI, so the screenshot tests queue behind the smoke and profile tests; they share the same dev server and mock API.
More Playwright jobs
The same job on other stacks
- Add Visual Regression Tests to Storybook Stories (Storybook)