All jobs

Upgrade Playwright and Its Browsers in a Test Suite

Yes. The agent moves @playwright/test, the lockfile and every CI image tag to one version, reruns the suite in each browser and opens a pull request.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

A Playwright version is written in more places than package.json, and the upgrade fails quietly when one of them is missed. In a suite like MarcusFelling/demo.playwright, devDependencies has @playwright/test at ^1.46.0 and package-lock.json resolves both @playwright/test and playwright to 1.46.0. The container image mcr.microsoft.com/playwright:v1.46.0-jammy is written once in .github/workflows/template-test.yml and three times in .github/workflows/test.yml. Every job then runs npx playwright install inside that container. That last step is why a mismatch would go unnoticed: if the package moved and the image tag did not, the job would download a second set of browsers on every run and still pass. The agent searches the repository for the version string and for mcr.microsoft.com/playwright and lists every hit before it changes one.

It reads the release notes for each minor between the pinned version and the target, which it takes from npm view @playwright/test version, and turns them into a checklist against this suite: the Node versions the target supports against the node-version: 18 the workflows set, which Ubuntu tags the target's image is published under, and any change to browsers, headless mode or APIs the specs use. It reads the specs for the parts of Playwright that move most between releases. Here that is _android in android/tests/example.spec.ts, which Playwright exports as experimental, launchPersistentContext with --load-extension and backgroundPages() in chrome-extension/tests/example.spec.ts, where the extension's manifest.json declares manifest_version 2, the CDP session in performance/tests/example.spec.ts, and toHaveScreenshot in visual-comparison.

Then it upgrades and runs. npm install --save-dev @playwright/test@<target> updates the manifest and the lock together, and npx playwright install --with-deps brings the matching browsers into the sandbox. The agent runs each folder's suite with its own config, in every project that config defines, with --retries=0. It runs the same commands on the base commit first, so a test that already failed before the upgrade is not blamed on it. Packages that ride on Playwright are checked against their peer ranges: @axe-core/playwright and playwright-lighthouse, which here sits beside lighthouse 9.

What breaks gets the documented replacement, not a workaround. A renamed option is renamed. A screenshot that differs after a browser bump is examined: the agent attaches the expected, actual and diff images, and regenerates the Linux baseline landing-chromium-linux.png only when the difference is rendering and not content. It cannot produce landing-chromium-win32.png and says so. If the target's Chromium no longer loads a Manifest V2 extension, that is reported as a finding with the error, since porting the extension is a separate job. Finally it moves all four image tags to the target version, on an Ubuntu tag the target publishes, and leaves unrelated workflow problems on a list for the team.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Upgrade Playwright in this repository to the latest stable release. Read package.json, package-lock.json, .gitignore, every playwright.config.ts, every workflow under .github/workflows, the devcontainer config and every spec first. List every place a Playwright version or a mcr.microsoft.com/playwright image tag is written, and every package that depends on Playwright with its peer range. Get the target from npm view @playwright/test version. Read the release notes for each minor between the pinned version and the target and write a checklist for this suite: supported Node versions against the workflows' node-version, published image tags, changes to bundled browsers and headless mode, and changed or removed APIs that the specs use. Before upgrading, run npm ci, npx playwright install --with-deps and each folder's suite with --retries=0 in every project its config defines, and record what already fails; do not run suites that need credentials or an Android device, and list them as not run. Then run npm install --save-dev @playwright/test@<target>, install the matching browsers and rerun the same suites. Fix each new failure with the replacement the release notes document. For a screenshot difference, attach expected, actual and diff, and regenerate only the Linux baseline and only when the content is unchanged. Update every image tag in the workflows to the target version on a tag that exists, and raise node-version only if the target requires it. Do not raise retries or timeouts, do not skip tests, and do not fix unrelated workflow problems; list them. Open a pull request with the checklist, each item marked applied or not applicable with the file it touched, and a before and after table of results per folder and browser.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that changes package.json and package-lock.json, the image tag in each workflow that names one, and the specs or configs the new release required. Its description opens with the list of every place the old version was written and what each became, so a reviewer can confirm none is left with one search.

Below that is the checklist built from the release notes, each item marked applied with the file it touched or not applicable with the search that found no use, and a table with one row per folder and browser: the result on the base commit and the result on the upgraded branch. A test that fails in both columns is marked as failing before the upgrade. Any regenerated screenshot is shown with its diff image. Suites the agent did not run are named with the reason: secrets it was not given, an emulator, a runner operating system the sandbox does not have.

A reviewer checks three things. The image tags and the lockfile name the same version. No spec lost an assertion while being adapted. And every new baseline image shows a rendering change only. The repository's own CI then runs on the pull request inside the new container image, which is the first time that image is exercised; the sandbox runs used browsers installed by npx playwright install, not the image.

Run it on every Playwright release

Playwright ships a minor about once a month, so a schedule fits better than a release hook. An automation in the Project that holds the repository, with this prompt and a schedule trigger whose cron picks one morning a month in your timezone, starts one run. When npm view reports the version already pinned, the agent ends the run with nothing to do and opens no pull request.

For many repositories with their own suites, a script on your server lists them with GET /o/{org}/github and sends this body to POST /o/{org}/sessions once per repository, each with a request_id that is a UUID v5 of the repository's full name and the target version, so running the script twice starts nothing new. A key runs a limited number at once and queues the rest.

The use case this job follows, with its automation and trigger: Upgrade dependencies weekly.

Why run it on Rush?

The upgrade itself is one command. The work is the two full passes of every suite in every browser, before and after, and the reading in between. Downloading three browsers and running a dozen configs twice ties up a laptop and gives a result that depends on its screen scaling and fonts. A sandbox starts clean each time, and the commands and their output, the failed attempts included, stay in the run's transcript at GET /o/{org}/sessions/{id}/messages.

The before and after table is what makes the pull request mergeable without rerunning it locally. A reviewer sees which failures the upgrade caused, which it inherited and which tests were never run, and the session read at GET /o/{org}/sessions/{id} names the pull request and what the run cost.

Frequently asked questions

Dependabot already bumps @playwright/test. What does this add?

An npm update from Dependabot changes package.json and the lockfile. The image tag in a workflow's container.image is a separate update, when one is configured at all, so the two can merge days apart, and neither adapts a spec to a changed API or looks at a screenshot diff. This run moves every place together and shows the suite's results on both sides.

Will it update our screenshot baselines?

Only the ones for the platform the sandbox runs, Linux, and only when the diff shows a rendering change with the same content. Baselines for other platforms, such as a -win32.png file, are left as they are and listed, because they can only be produced on that platform.

Can it run suites that need secrets, like an API token?

Not unless you provide them. A secret saved to the Project reaches every session started in that Project as an environment variable, so those suites can run there. Without that, the agent lists them as not run and leaves them to your CI on the pull request.

Does it also upgrade Node in the workflows?

Only when the target Playwright release does not support the Node version the workflows set. In that case it raises node-version to the lowest version the release notes name and says so at the top of the description. Otherwise it leaves Node alone.

More Playwright jobs

More jobs like this: Upgrade dependencies weekly

Keep reading