What the agent does
The agent reads the suite before adding a scan, because an accessibility test is only useful on the pages a person actually reaches. In an app laid out like the react-vite app in alan2207/bulletproof-react, playwright.config.ts sets testDir: './e2e', two projects, a setup project matched on .setup.ts and a chromium project that depends on it and loads storageState from e2e/.auth/user.json, a webServer that runs yarn dev --port 3000, fullyParallel, two retries on CI with trace: 'on-first-retry', and the html reporter. e2e/tests/ holds auth.setup.ts, a smoke.spec.ts that walks discussions and comments end to end, and a profile.spec.ts. package.json runs test-e2e by starting the mock API with pm2 and then playwright test, and .github/workflows/react-vite-ci.yml has an e2e job that moves .env.example-e2e to .env, installs browsers with --with-deps, runs it and uploads playwright-report. @axe-core/playwright is not installed.
The scans go in one new file, e2e/tests/accessibility.spec.ts, under the chromium project so they run signed in: one test per screen the smoke test visits, the landing page, /app, /app/discussions, one discussion with a comment, and the profile, each navigating, waiting for the heading the smoke test already waits for, and running new AxeBuilder({ page }).withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa']).analyze(). The assertion is expect(violations).toEqual([]), and on failure the test attaches the violations as JSON with testInfo.attach, so the HTML report that CI already uploads carries the node, the rule id and the help URL for each.
The first run decides the shape of the pull request, which is why the agent reads it rather than committing a green file. Rules that fire on the Radix dialog and drawer used by FormDrawer and ConfirmationDialog are checked against the component before being excluded: an aria-dialog-name finding on a drawer that does have a title is a false positive from a scan run before the animation finishes, which a waitFor on the dialog's heading fixes in the test, not with a rule exclusion. A color-contrast finding on the gray text-gray-500 empty state in comments-list.tsx is a real finding and is listed in the description with the hex values, not fixed, because colour is a design decision.
In its sandbox the agent runs yarn install, yarn playwright install --with-deps, starts the mock server the way test-e2e does, and runs yarn playwright test to see the existing three specs green. It adds @axe-core/playwright as a dev dependency, writes the spec, runs it, reads the violations, adjusts the waits, and runs the whole suite three times to see the scans hold under fullyParallel. The pull request quotes the final run and the violations list from the first run, so a reviewer sees what was found before the waits were fixed.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add accessibility tests to this app's Playwright suite. Read playwright.config.ts, e2e/tests/auth.setup.ts, e2e/tests/smoke.spec.ts, e2e/tests/profile.spec.ts, package.json and .github/workflows/react-vite-ci.yml first. Add @axe-core/playwright as a dev dependency and write e2e/tests/accessibility.spec.ts under the chromium project: one test per screen the smoke test visits, each waiting for that screen's heading, scanning with the wcag2a, wcag2aa, wcag21a and wcag21aa tags, asserting no violations, and attaching the violations JSON to the test on failure. Run yarn install, install browsers, start the mock server as test-e2e does, and run the existing suite first. Run the new spec, read every violation, fix timing with waits in the test and never with a rule exclusion unless the component shows the finding is wrong, and list real findings in the description with the element and rule. Run the whole suite three times. Open a pull request quoting the first run's violations and the final green run.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request with one new spec, one new dev dependency in package.json and the lockfile, and no change to playwright.config.ts or the workflow, because the new file matches the chromium project's .spec.ts pattern and the e2e job already uploads the report the attachments land in. The description quotes the final run, five scans green beside the three existing specs, and the first run's violations with rule ids.
Any finding the agent judged real is in the description with the element, the rule and axe's help link, left for the team rather than patched in a test file: a scan that passes because a rule was excluded is the outcome the request forbids.
Before merging, a reviewer decides whether the color-contrast finding on the empty state is fixed in the stylesheet now or tracked, and whether the scans should also run in the webkit and firefox projects the config does not yet define, which would be two more project entries and a longer CI job.
Run it on every pull request
Once the scans are in, CI runs them on every push to a pull request through the e2e job that exists already. The review on top comes from an automation in the Project with a github trigger on pull_request.opened and pull_request.synchronize and a prompt that reads the uploaded report: a run per push that, when a scan fails, posts a review quoting the rule, the element's selector and the file in src/ that renders it, and when the diff adds a screen under src/app/routes, asks for a scan of it in the same pull request.
Only work by an owner, member or collaborator fires the trigger; a pull request from a fork never does. Each run installs that pull request's branch in its own sandbox with its own mock server, so two reviews never share a mocked-db.json.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
An axe scan is a few lines, and the work is the first run: separating findings that are timing, findings that are the component, and findings that are the design, then deciding which the test owns. Done quickly, the result is a spec full of rule exclusions that passes and checks nothing. In a Rush sandbox the agent runs the scans against the real app and mock server, reads each finding against the component that produced it, and fixes waits rather than hiding rules.
The record of the run keeps the first run's violations and the three final runs, and the pull request quotes them, so the reviewer sees what the scans caught before they went green. Across every frontend a team keeps on Playwright, the same request runs in parallel, one sandbox and one pull request each.
Frequently asked questions
Why run the scans signed in?
Because the screens that matter are behind the login: discussions, comments and the profile. The new spec sits under the chromium project, which depends on the setup project and loads e2e/.auth/user.json, so each scan starts with the same session the smoke test uses.
What happens when a scan finds something?
The test fails and attaches the violations as JSON, so the HTML report CI already uploads shows the rule, the element and axe's help page. The agent reads each finding from the first run: a timing issue gets a wait in the test, a real issue is listed in the description for the team, and nothing is excluded by rule unless the component proves the finding wrong.
Does this slow the e2e job down?
By the five scans, each a page load and an axe pass of about a second. The suite already runs with one worker on CI and two retries, so the added time is seconds, and the pull request quotes the run time of the final suite.
Which WCAG level is checked?
A and AA for WCAG 2.0 and 2.1, through the four tags passed to withTags. Best-practice rules that are not WCAG criteria are left out so a failure always maps to a criterion the team can cite; adding the best-practice tag is one more string if they want it.