What the agent does
The agent reads what already exists so the new check borrows it. In a React app like alan2207/bulletproof-react, apps/react-vite/package.json has storybook at ^8.0.9 with @storybook/react-vite, scripts storybook and build-storybook, @playwright/test at ^1.49.1 as a devDependency, and test-e2e, which starts the mock server with pm2 and runs yarn playwright test. .storybook/main.ts globs ../src/**/*.stories.@(js|jsx|ts|tsx) and lists seven addons; .storybook/preview.tsx wraps every story in a BrowserRouter and imports src/index.css, so the stories render with the app's Tailwind styles. Eleven story files sit under src/components/ui/: button, dialog, confirmation-dialog, drawer, dropdown, form, link, md-preview, notification, spinner and table. playwright.config.ts has a setup project and a chromium project on .spec.ts files in e2e/, retries: 2 on CI, one worker on CI, and a webServer running yarn dev --port 3000. .github/workflows/react-vite-ci.yml runs build, vitest, lint and type checks in one job and the Playwright suite in an e2e job that uploads playwright-report. None of the twelve vitest files takes a screenshot.
The check is built on the static build. yarn build-storybook writes storybook-static/, including index.json, which lists every story id. The agent adds a visual Playwright project in playwright.config.ts with its own testDir and a second webServer entry that serves storybook-static on another port, with http-server added as a devDependency so the command needs no download. The one test file reads index.json, keeps the entries whose type is story, and for each one opens /iframe.html?id=<story id>&viewMode=story, waits for the story root to render, and calls expect(page).toHaveScreenshot("<story id>.png", { animations: "disabled" }). Playwright writes the baseline next to the test on the first run, named for the project and platform, and compares against it on every run after.
CI gets one more job: build Storybook, install Chromium, run yarn playwright test --project=visual, and on failure upload test-results/, where Playwright puts the expected, actual and diff images. Baselines are platform-specific, so the agent takes them on Linux in the sandbox, which matches ubuntu-latest; the pull request says that a developer on a Mac refreshes them by running the project with --update-snapshots in CI or on Linux rather than locally, so the committed files stay comparable. yarn lint runs on the new file too, since the config already loads eslint-plugin-playwright.
Then it proves the check both ways. After yarn install, yarn build-storybook and yarn playwright install --with-deps chromium, the first run writes one baseline per story and the second run passes against them. If the build stops on one of the addons main.ts names that package.json does not list, the agent pins the missing one before going on. Then it changes one value in src/components/ui/button/button.tsx, runs the project again, and shows the button stories failing with diff images, before reverting the change and committing. The pass, the deliberate fail and the lint result all go into the pull request.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add visual regression tests for every Storybook story in apps/react-vite. Build Storybook to storybook-static, add a Playwright project named visual with its own testDir and a webServer that serves storybook-static with http-server, and write one test that reads storybook-static/index.json and screenshots each story through iframe.html with animations disabled, using toHaveScreenshot named by story id. Commit the Linux baselines. Add a CI job to .github/workflows/react-vite-ci.yml that builds Storybook, installs Chromium, runs the visual project and uploads test-results on failure. Prove it: run twice to show a pass against the baselines, then change the button component, show the failing stories and the diff images, and revert. Run yarn lint. Open a pull request quoting both runs.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request adds the test file and its snapshot directory with one PNG per story, the visual project and second webServer in playwright.config.ts, http-server in package.json, and the new job in .github/workflows/react-vite-ci.yml. Its description quotes the passing run, the deliberate failing run with the story names and the path of each diff image, and the lint result.
A reviewer opens a few baselines to see that the stories rendered with the app's styles rather than unstyled, checks that the test disables animations so the spinner and notification stories are stable, and confirms the deliberate change was reverted. The transcript at GET /o/{org}/sessions/{id}/messages has the Storybook build and both Playwright runs in full.
Nothing runs against your CI until the branch is pushed; the first CI run on the pull request is the check running on the committed baselines.
Run it on every pull request
After the merge, the CI job compares every pull request against the baselines on its own. The agent's job repeats when a new component ships without a story, or when a theme change means the baselines need a deliberate refresh: add an automation on the Project with a github trigger on pull_request.opened and pull_request.synchronize and a prompt that runs the visual project, reports which stories changed with their diff images, and refreshes the baselines only when the pull request says the change is intended. The trigger adds the pull request after the prompt; it fills in no placeholder.
From your own server, the same run starts with this body and the pull request URL in resources, with a request_id that is a UUID v5 of the head commit so a retried send returns the run already begun.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Screenshot baselines are only trustworthy when they were taken where CI will compare them, and that is a Linux machine with the same Chromium and the same fonts. A laptop produces Mac baselines that fail on ubuntu-latest on the first run. The sandbox is Linux, the build and both Playwright runs happen there, and the diff images from the deliberate failure are in the transcript as proof the check catches a real change.
A design system with several packages, or several apps on the same component library, is several runs on their own branches, each with its own baseline set, and a reviewer reads one diff per app. Each run also records which stories it could not stabilize, which is the list a team needs before trusting the check.
Frequently asked questions
Why Playwright and not Chromatic or the Storybook test runner?
Playwright is already installed and already runs in CI, so the check adds one project and no service. Chromatic needs an account and uploads every build; the Storybook test runner covers interactions and a11y but needs another package for image comparison. If you use Chromatic, say so and the agent adds its action instead of the Playwright project.
What makes a screenshot test flaky, and how is that handled?
Animations, timers and fonts that load late. The test disables CSS animations through Playwright, waits for the story root to render before capturing, and runs against the static build so no dev server reload moves things. A story that still differs between two runs is named in the pull request rather than hidden with a wide threshold.
Do stories with a router or global styles work?
Yes. iframe.html renders a story with everything preview.tsx sets up, so the BrowserRouter decorator and index.css apply exactly as in Storybook's own UI. Only the toolbar and panels are absent, which is what you want in a baseline.
How do I update baselines after an intended change?
Run the visual project with --update-snapshots on Linux and commit the new PNGs, or ask the agent in a follow-up message to the session to refresh them and show the before and after. Updating on a Mac produces different files, which is why the pull request says not to.
More Storybook jobs
The same job on other stacks
- Add Visual Regression Tests With Playwright (Playwright)