All jobs

Shard Playwright Tests Across GitHub Actions Runners

Yes. An agent splits a Playwright suite across parallel GitHub Actions jobs with --shard, merges the blob reports into one HTML report, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent starts from how the suite runs in CI today. In a repository laid out like MarcusFelling/demo.playwright, each folder is its own project with its own playwright.config.ts, and .github/workflows/test.yml runs one job per folder: most of them call the reusable .github/workflows/template-test.yml with TESTGROUP and BASEURL inputs, while chrome-extension, github-api and the disabled android and oauth jobs are written out in full. The template runs on ubuntu-24.04 inside the mcr.microsoft.com/playwright:v1.46.0-jammy container, sets up Node 18 with the npm cache keyed on the root package-lock.json, runs npm ci and npx playwright install, then HOME=/root npx playwright test --retries 0 in the group's directory with continue-on-error: true, and uploads playwright-report as an artifact named after the group, which a GitHub environment links to as testresults.marcusfelling.com/<group>.

Then it reads the slowest group. todomvc/playwright.config.ts sets workers: process.env.CI ? 1 : undefined, two retries on CI, trace: 'on', the list and html reporters, baseURL from the folder's .env, and three projects for Desktop Chrome, Firefox and Safari. todomvc/tests/example.spec.ts holds 25 tests, so CI runs 75 browser tests one at a time on one runner, each recording a trace. svgomg has the same config shape with six tests. The root package.json pins @playwright/test ^1.46.0, which has --shard and the blob reporter with merge-reports. .gitignore already ignores **/playwright-report/ and **/test-results/ but not blob-report.

The change is in the template, so every group can opt in without its own workflow. template-test.yml gains a shards input that defaults to '[1]', a strategy.matrix.shard built with fromJSON(inputs.shards) and fail-fast: false, and the test step runs npx playwright test --retries 0 --shard=${{ matrix.shard }}/${{ strategy.job-total }}. Each shard uploads blob-report as blob-report-<group>-<shard>, and a second job, merge, with needs: test and if: always() so it runs after a shard fails, downloads them with pattern: blob-report-<group>-* and merge-multiple: true into one directory, runs npx playwright merge-reports --reporter html on it, and uploads the result under the group's old artifact name so the environment link keeps pointing at one report. In todomvc/playwright.config.ts and svgomg/playwright.config.ts the reporter becomes blob on CI and stays list plus html locally; workers: 1 stays, since the split is across machines. test.yml passes shards: '[1, 2, 3]' for todomvc and leaves every other call site alone, and .gitignore gains **/blob-report/.

To prove it without a runner, the agent installs Chromium with npx playwright install --with-deps chromium in its sandbox, runs npx playwright test --shard=1/3 --list and --shard=2/3 --list in todomvc to show the tests split with none shared, runs each shard with --reporter=blob --project=chromium against the .env base URL, merges the three blob reports with merge-reports --reporter html and attaches the resulting index.html to the run, and checks the two workflow files with actionlint. The pull request links the first workflow run on the branch so a reviewer can see three todomvc shards and one merged report.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Shard the todomvc Playwright group across parallel GitHub Actions jobs. Read .github/workflows/test.yml, .github/workflows/template-test.yml, package.json, .gitignore, todomvc/playwright.config.ts, todomvc/tests/example.spec.ts, todomvc/.env and svgomg/playwright.config.ts first. In template-test.yml add a shards input defaulting to '[1]', a matrix from fromJSON(inputs.shards) with fail-fast off, pass --shard=<matrix.shard>/<strategy.job-total> to playwright test, upload each shard's blob-report under a name with the group and shard, and add a merge job that runs even when a shard failed, downloads the shard artifacts with merge-multiple, runs npx playwright merge-reports --reporter html, and uploads playwright-report under the group's existing artifact name. Switch the todomvc and svgomg configs to the blob reporter on CI only. In test.yml pass shards '[1, 2, 3]' for todomvc and change no other job. Add blob-report to .gitignore. Install Chromium in your sandbox, show --shard=1/3 --list and --shard=2/3 --list for todomvc, run the three shards with the blob reporter on Chromium, merge them, attach the merged index.html, and run actionlint on both workflow files. Change no test.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request changes .github/workflows/template-test.yml, .github/workflows/test.yml, todomvc/playwright.config.ts, svgomg/playwright.config.ts and .gitignore, and no test file. Its description shows the two --list outputs side by side so a reviewer can see each test lands in exactly one shard, the three local shard runs, the merge-reports command, the actionlint result, and a link to the branch's first workflow run with its three todomvc jobs and the merge job.

Every group that does not pass shards runs as before: one job, one --shard=1/1, and its report merged from a single blob, so the artifact a reviewer downloads has the same name and layout it always had. The chrome-extension, github-api, android and oauth jobs, which do not use the template, are untouched. Traces are still recorded per test and travel inside each blob into the merged report.

Before merging, a reviewer picks the shard count for todomvc against the runner minutes they want to spend, since three shards is a starting point rather than a measurement, decides whether svgomg should also shard or only switch reporters, and checks the environment URL still resolves to the merged artifact.

Run it on every Linear issue labelled ci

Add an automation to the Project with this prompt and a linear trigger on linear.issue.labeled with the label your CI work uses in its labels. Each labelled issue starts one run with the issue as its context, on its own branch; the trigger adds the issue after the prompt and fills in no placeholder, and a change made by an app integration never fires it, so the agent's own comment on the issue starts nothing.

The automation's max_runs_per_hour holds a burst of labels to that many runs an hour: a fire past it is created queued and starts when the hour has room, and one past a second hour's worth is skipped as rate_limited until the issue is labelled again. From your own server, the same run starts with this request and the issue URL in resources, with a request_id that is a UUID v5 of the issue id so a retried start returns the run already begun.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

Sharding is a workflow change that is only right when the matrix, the artifact names, the merge step and the reporter switch all agree, and a mistake shows up as a missing report rather than a failing build. The agent runs the shards and the merge in its own sandbox before touching CI, and checks the YAML with actionlint, so the first workflow run on the branch is a confirmation rather than the first test.

The transcript keeps the --list outputs, the shard runs and the merge with their output, so a reviewer reads the split and the merged report from one page. Several repositories that share this template pattern can be sharded at once, each run in its own sandbox with its own browsers, and nothing merges until a person reads the diff.

Frequently asked questions

Why not just raise workers instead of sharding?

workers parallelises on one machine, and the config sets it to one on CI on purpose: the demo runs three browsers with traces on, and a GitHub runner has limited cores. --shard splits the same list across machines, so each job keeps one worker and the wall clock drops with the number of jobs. The two can be combined later if a runner has room.

What does merge-reports need?

Every shard must write the blob reporter, and the merge must run on the same Playwright version, which the template guarantees through the pinned container image and npm ci. The merge job downloads all blob-report-<group>-* artifacts into one directory and produces an HTML report identical in shape to a single run's, traces included.

Will a failing shard still produce a report?

Yes. The test step keeps continue-on-error: true, the matrix has fail-fast: false so one red shard does not cancel the others, and the merge job runs with if: always(). A reviewer gets one report with the failures marked rather than two partial ones.

Does this change how tests run locally?

No. The reporter switch keys on process.env.CI, so npx playwright test in todomvc still prints the list and opens the HTML report. --shard is only passed in the workflow; locally the whole suite runs as before.

More Playwright jobs

The same job on other stacks

More jobs like this: Turn a Linear label into a PR

Keep reading