All jobs

Add Merged Test Reports to a Playwright Workflow

Yes. An agent switches your parallel Playwright CI jobs to the blob reporter, adds a job that merges them into one HTML report, proves the merge, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent starts with how the reports are produced now. In a repository laid out like MarcusFelling/demo.playwright, each folder, basic, todomvc, svgomg, visual-comparison, accessibility and the rest, has its own playwright.config.ts with reporter: [['list'], ['html']], trace: 'on' and workers: process.env.CI ? 1 : undefined. .github/workflows/test.yml runs one job per folder, most of them through the reusable template-test.yml, which runs npx playwright test --retries 0 in the folder with continue-on-error: true and uploads playwright-report as an artifact named after the group. So a pull request ends with eight separate playwright-report artifacts, each a zip to download and open, and a failure in todomvc on webkit is found by guessing which zip to open. The container image is mcr.microsoft.com/playwright:v1.46.0-jammy and package.json pins @playwright/test ^1.46.0, which has the blob reporter and merge-reports.

It changes what each job writes, not how it runs. Every folder's config keeps list and gains ['blob'] on CI in place of html, with outputDir left at the default blob-report; the HTML reporter stays for local runs through a process.env.CI check so a developer's npx playwright test still opens a report. In template-test.yml and the hand-written jobs the upload step changes to blob-report with a name of blob-${{ inputs.TESTGROUP }}, and because the basic folder runs with testIgnore and a different baseURL, the agent checks that its blob still merges with the others, which merge-reports allows as long as the Playwright version matches.

A new merge-reports job in test.yml runs after every test job with if: always(), since the test steps are continue-on-error, downloads every blob-* artifact into one directory with actions/download-artifact and pattern, runs npx playwright merge-reports --reporter html ./all-blob-reports, and uploads the one playwright-report that comes out, named html-report--attempt-${{ github.run_attempt }} so a re-run does not overwrite the first. The job runs in the same container image so the merge uses the same Playwright. The README's line about extracting the zip is updated to name the one artifact.

In its sandbox the agent installs with npm ci, runs npx playwright test in todomvc and svgomg with CI=1 so each writes a blob-report, copies both into one directory, runs the merge, and opens the resulting index.html to confirm both groups appear with their traces attached. It quotes the merge command's output and the list of files in playwright-report in the pull request, and runs npx playwright test --list in each changed folder to show no config error.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add merged test reports to this Playwright repository's CI. Read .github/workflows/test.yml, .github/workflows/template-test.yml, package.json, the README and every playwright.config.ts first. In each config use the blob reporter on CI and keep html for local runs. In template-test.yml and the inline jobs upload blob-report as blob-<group>. Add a merge-reports job that runs after all test jobs with if: always(), downloads every blob-* artifact, runs npx playwright merge-reports --reporter html, and uploads one playwright-report named with the run attempt; run it in the same Playwright container image. Update the README's artifact instructions. In the sandbox run todomvc and svgomg with CI set, merge their blobs, and quote the merge output and the report's file list in the pull request. Run npx playwright test --list in every changed folder. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request touching every playwright.config.ts, .github/workflows/test.yml, .github/workflows/template-test.yml and README.md. The description quotes the merge-reports output from the sandbox, the file listing of the merged playwright-report with both groups' trace zips in data/, and the --list output per folder showing the same test counts as before.

The run's record keeps the two sandbox test runs in full, so a reviewer can see each group writing its blob and the merge reading both. The agent ran against the public demo sites the configs name, which is what the workflow does too.

Before merging, a reviewer decides whether the per-group artifacts should also stay. The pull request drops them in favour of the one merged report; keeping both is one more upload step per job and doubles the artifact storage.

Run it on every pull request

Once the merge job is in, every pull request's checks end with one report, and the review of that report is a job an agent can do. In the Project for the repository, an automation with a github trigger on pull_request.opened and pull_request.synchronize and a prompt that reads the merged report starts a run per pull request; the run downloads the html-report artifact, reads the failures and their traces, and posts a review comment naming the test, the step and the screenshot.

A re-delivered event is answered deduped by the trigger, so it does not open a second review. The same automation on several repositories runs in parallel, one sandbox each.

The use case this job follows, with its automation and trigger: Review every pull request.

Why run it on Rush?

Eight artifacts become one through two reporter lines and a workflow job, and the thing that goes wrong is the plumbing: a blob written with one Playwright version and merged with another refuses to load, a download-artifact pattern that misses one group silently drops its tests from the report, a merge job without if: always() never runs on the pull request that needed it. In a Rush sandbox the agent writes two real blobs with the pinned version, merges them, and opens the result before touching CI.

The merged report and the commands stay with the run, so the next person adding a test group knows the artifact name pattern and why the merge job runs in the container. Across many repositories with per-folder Playwright projects the job runs in parallel, one sandbox each, with each pull request carrying its own merge output.

Frequently asked questions

Does this change how tests run or how many run?

No. Each job still runs its folder with the same workers, retries and projects; only the reporter changes from html to blob on CI. The --list output the agent quotes shows the same test counts per folder as before.

Will I still get an HTML report locally?

Yes. The configs choose blob only when CI is set, so npx playwright test on a laptop still writes playwright-report and npx playwright show-report still opens it.

What if one test job fails to start?

The merge job has if: always() and downloads whatever blob-* artifacts exist, so the report covers the groups that ran. The job's log lists the artifacts it found, and a missing group is visible there rather than silently absent.

Why run the merge in the Playwright container?

Because merge-reports reads blobs with the Playwright that wrote them, and a version mismatch is refused. The container image and the package.json pin are the same release, and the merge job uses the same image as the test jobs.

Can the merged report be published to GitHub Pages instead of an artifact?

Yes. The merge job's output directory is a static site; say so in the prompt and the agent adds a deploy step with actions/deploy-pages on the main branch and keeps the artifact for pull requests.

More Playwright jobs

More jobs like this: Review every pull request

Keep reading