All jobs

Add Accessibility Tests to a Lit Component

Yes. An agent adds axe-core accessibility assertions from @open-wc/testing to your Lit component's test suite, runs them in three browsers, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the test setup before it writes a test. In a component repository laid out like lit/lit-element-starter-ts, package.json depends on lit 3 and runs tests with @web/test-runner 0.15 through two scripts, test:dev (wtr) and test:prod (MODE=prod wtr), after npm run build has compiled src/ with tsc into the package root. web-test-runner.config.js picks up ./test/**/*_test.js, launches Chromium, Firefox and WebKit through @web/test-runner-playwright, sets Mocha to the tdd interface with a 60 second timeout, and injects the web components polyfills with @web/dev-server-legacy. The one existing file, src/test/my-element_test.ts, imports fixture and assert from @open-wc/testing 4.0 and checks the shadow DOM of <my-element> with assert.shadowDom.equal for the default render, a set name, a click and the computed padding.

@open-wc/testing 4.0 already depends on chai-a11y-axe 1.5, so no package is added: the assert the file imports gains isAccessible and isNotAccessible, and expect gains to.be.accessible(). Each assertion runs axe-core inside the real browser against the fixture, light DOM and shadow DOM included, and rejects with the rule ids and the offending nodes when a check fails. The agent keeps the file's suite and test style rather than switching to describe, because tsconfig.json sets types: ["mocha"] and the config fixes the tdd interface.

It adds an accessibility suite beside the existing one, with one test per state the component can be in: the default render, a set name, the render after a click (the button's label changes from Click Count: 0 to Click Count: 1), and a fixture with slotted light DOM content like the <p> in dev/index.html. Each test awaits el.updateComplete before await assert.isAccessible(el), since Lit renders asynchronously and axe would otherwise inspect an empty shadow root. A negative test proves the check does something: assert.isNotAccessible on <my-element> with a slotted <img> that has no alt, which axe reports under image-alt. The agent writes no ignoredRules up front; if a rule fails because of the test harness rather than the component (a fixture outside any landmark, for instance), it ignores that one rule with a comment that names it, and lists the decision in the pull request.

Before opening the pull request the agent runs npx playwright install --with-deps so all three browsers exist in its sandbox, then npm run build, npm run lint (ESLint 8 with @typescript-eslint and lit-analyzer) and npm test, which is both the development and the production-mode run across Chromium, Firefox and WebKit. A rule that fails in one engine only is a real finding, and the output names the engine. The starter has no .github directory, so there is no workflow to extend; the pull request says so and quotes the three-browser output instead.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add accessibility tests to this Lit component. Read package.json, web-test-runner.config.js, tsconfig.json, src/my-element.ts and src/test/my-element_test.ts first. @open-wc/testing already includes chai-a11y-axe, so add no package. In src/test/my-element_test.ts add an accessibility suite in the file's existing tdd style: assert.isAccessible on the default render, with a name attribute, after a click, and with slotted light DOM content, each after awaiting el.updateComplete; and one assert.isNotAccessible test with a slotted img that has no alt. Do not add ignoredRules unless a rule fails because of the test harness and not the component; comment the reason beside any you add. Run npx playwright install --with-deps, npm run build, npm run lint and npm test, so the suite runs in Chromium, Firefox and WebKit in both dev and prod mode. Open a pull request that quotes the three-browser output and lists every axe rule the component failed and how it was fixed or why it was ignored.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request changes src/test/my-element_test.ts and, when axe found something, src/my-element.ts. Its description lists every rule id axe reported in any browser, with the node it pointed at and the line that fixed it, and quotes the wtr summary per browser from both the development and the MODE=prod run. A rule that was ignored appears with its reason, so a reviewer can disagree with one line rather than hunt through the diff.

A reviewer checks three things before merging: that the component change, if any, keeps the shadow DOM assertions in the older tests true (the agent updates those expected strings when a fix adds an attribute); that each ignoredRules entry is about the harness and not the component; and that the negative test fails for the right reason, which the description shows by quoting the image-alt failure text from axe.

Because the starter ships no CI workflow, the suite runs wherever npm test runs. A repository that already has a workflow running npm test gets the accessibility checks on every pull request with no further change, and the agent notes that in the description.

Run it on every pull request opened

An automation on the Project with a github trigger on pull_request.opened and the prompt "run the accessibility suite in all three browsers on this branch; if a component gained a new state or attribute in this pull request, add an isAccessible test for it; report every axe rule that fails with the node and the engine" starts one run per pull request. The trigger adds the pull request after the prompt and fills in nothing else, so the run reads the diff itself to find the new state. Only work by an owner, member or collaborator fires it.

A design-system team with several component packages sends this body once per repository from its own server, each POST /o/{org}/sessions with that repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started rather than opening a second pull request. Each run reads its own test runner configuration: a package on @web/test-runner-chrome gets a single-browser run, and one that already imports expect gets to.be.accessible() in that style.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

axe-core has to run inside a browser against the rendered shadow DOM, and the rules that matter for a custom element (focusable content behind tabindex, a button whose label is only an icon, contrast inside :host styles) differ between engines. On a laptop that means installing three Playwright browsers and their system libraries and waiting through two full runs; on Rush the agent does that in its own sandbox, and the per-browser output is kept with the run rather than scrolled past in a terminal.

A reviewer gets the rule ids, the nodes and the engine for every failure, with the exact command that produced them, so the pull request can be checked without re-running anything. A team with many components starts the same request across all of them at once and reads the results as they arrive.

Frequently asked questions

Does this add a dependency?

No. chai-a11y-axe ships as a dependency of @open-wc/testing, which the starter's tests already import. The assert object gains isAccessible and isNotAccessible, and expect gains accessible, with no change to package.json.

Does axe see inside the shadow root?

Yes. axe-core walks open shadow roots, and the fixture helper from @open-wc/testing mounts the element in the document, so a <button> rendered inside <my-element>'s shadow DOM is checked like any other button. Slotted light DOM content is checked too, which is why the suite includes a fixture with a child element.

What if a rule fails because of the test page rather than my component?

The assertion takes ignoredRules and ignoredTags. The agent adds an ignore only when the failing node is outside the component, writes the reason as a comment beside it, and lists it in the pull request so a reviewer can object to that one line.

Do I need all three browsers?

The starter's config launches Chromium, Firefox and WebKit, and the agent keeps that, because some checks (focus order, computed names) differ between engines. BROWSERS=chromium npm test runs one, as the config's comment says, and a project that only lists one launcher gets a one-browser run.

More Lit jobs

The same job on other stacks

Keep reading