# Review Pull Requests in a Playwright Test Suite

Yes. For each pull request the agent runs the changed specs repeatedly in every browser, checks that each assertion can fail and posts one review.

[Run this on Rush](https://cloud.getrush.ai/run/review-pull-requests/playwright/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

A pull request to a test suite is hard to review by eye, because a test that cannot fail looks the same in a diff as one that can. So the agent begins with what the repository's own checks cover. In a suite like MarcusFelling/demo.playwright there is an `.eslintrc.js` with the `@typescript-eslint` recommended rules, but no lint script and no `tsconfig.json`, so no rule that needs type information, such as one that catches a promise nobody awaited, can run. Both workflows check out `ref: main`, so on a pull request CI tests the main branch and not the commits under review. The reusable workflow runs tests under `continue-on-error: true`. In a repository like that, the agent's run on the pull request's branch is the only execution the new tests get before merge, and the review says so at the top.

It then runs what the diff changed. After `npm ci` and `npx playwright install --with-deps`, it runs `npx playwright test --list` in each folder the pull request touches and confirms every new test is collected. That step matters here: `basic/playwright.config.ts` sets `testIgnore: ['4-*', '5-*']`, and the two specs it excludes open `/file-uploads.html` and `/network.html` from a local server in `basic/server/index.js` on port 4345 that no config starts, while `baseURL` points at todomvc.com. A new spec whose name matches the pattern would be merged without ever running. The changed specs then run with `--repeat-each=10 --retries=0` in every project their config defines, and a test that passes only some of the time is reported with the error from a failing repeat.

Next it reads each added assertion and asks whether it can fail. `expect(page.locator('#div2.#drag1')).toBeTruthy()`, as written in `drag-and-drop/tests/example.spec.ts`, cannot: a locator is an object and is truthy whether or not anything matches. The agent proves a suspicion by breaking the thing under test, for example pointing the locator at an element that does not exist, and rerunning; an assertion that still passes is a finding. It flags a test with no `expect` at all, a call to an async helper without `await` (the todomvc spec has one such call beside twenty-two awaited ones), `page.waitForTimeout`, and a `test.only`, which `forbidOnly: !!process.env.CI` refuses in CI but not on a laptop.

The rest of the review is about how the test will age. A selector on a generated class such as `.heroTitle_ohkl`, or an absolute path like `xpath=//html/body/section/main/div/label`, breaks when markup moves, and the agent suggests the role or text locator that finds the same element. A `goto` with a full URL where the config already has a `baseURL` is noted. A changed file under a `-snapshots` folder needs a sentence in the pull request on why the picture changed. A value added to a tracked `.env` file is flagged before anything else: `oauth/.env` here carries a comment telling contributors to stop tracking the file if they add credentials.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Review this pull request to our Playwright test suite. Read package.json, every playwright.config.ts, the workflows under .github/workflows and the ESLint config first, and state at the top of the review whether CI runs the pull request's own commits, whether a failing test fails the job, and whether lint runs. On the pull request's branch run npm ci and npx playwright install --with-deps. In each folder the diff touches, run npx playwright test --list and confirm every added or renamed test is collected by its config; report any that testIgnore, testMatch or testDir leaves out. Run the changed spec files with --repeat-each=10 --retries=0 in every project their config defines and report each test as stable, flaky or failing, with the error for the last two. Do not run tests that need credentials or devices we have not given you; say which. Then read the diff. For every added assertion decide whether it can fail: flag expect on a locator or promise with a generic matcher such as toBeTruthy, a matcher on a locator that is not awaited, an async helper called without await, and tests with no assertion. Where you suspect an assertion cannot fail, break the condition it checks, rerun and report the result. Flag waitForTimeout, test.only, raised timeouts or retries, selectors tied to generated class names or absolute XPath (suggest the role or text locator), full URLs where baseURL exists, changed snapshot images with no explanation, and any credential added to a tracked file. Post one review on the pull request: the CI facts, the run results, then findings by file and line with the suggested change. Do not push commits.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app/pull/42"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

One review on the pull request. Its first lines state what CI does and does not prove for this change: whether the workflow runs the pull request's commits, whether a failure fails the job, and whether lint runs at all. A results block follows with each changed test marked `stable`, `flaky` or `failing` per browser, the repeat counts, and any test the config never collects.

Findings come next, ordered by file and line, each with the reason and the replacement: the web-first assertion to use in place of a generic matcher, the missing `await`, the locator that survives a markup change. For an assertion the agent suspected could not fail, the finding includes what it broke and the output showing the test still passed. Tests it did not run, and why, are listed last.

The review comments and leaves the merge to a maintainer. It does not judge whether the scenario is worth testing or whether a third-party page is a sensible target; those are the team's calls. The full output of every command is in the session's transcript at `GET /o/{org}/sessions/{id}/messages` for anyone who wants to check a result.

## Run it on every pull request

An automation in the Project that holds the repository, with this prompt, read-only switched on and a `github` trigger on `pull_request.opened` and `pull_request.synchronize`, reviews a pull request when it opens and again on each push. The trigger puts the pull request after the prompt and fills in nothing. With read-only on, the run can read the repository and comment and cannot push. `branch_patterns` limits it to pull requests against the branches you name.

The trigger fires for work by an owner, member or collaborator and never for a fork or a bot. For a contributor's fork, a maintainer pushes the commits to a branch in the repository itself and opens a pull request from it, and that one starts the review. Teams that want a review only when asked use `pull_request.labeled` with a label in `labels`.

## Why run it on Rush?

Whether a new test is stable and whether its assertions can fail are questions answered by running it, many times and in each browser, and by breaking it on purpose. A reviewer with the diff open in a browser tab does neither. The agent does both in its own sandbox with the browsers installed, for every push, and several pull requests are reviewed at the same time in separate sandboxes.

The evidence stays attached. Each repeat count and each deliberate break is a command with output kept in the run, so an author who disagrees with a finding can read exactly what was run and reply with a fix or a reason, and the next push is reviewed the same way.

## Frequently asked questions

### How does it know an assertion can never fail?

It reads the matcher and the value: a generic matcher applied to a locator object or an unawaited promise is always satisfied. When reading is not conclusive it changes the condition the test checks, such as the selector or the expected text, reruns the test and reports whether it went red. The changed copy is discarded and never pushed.

### Will it fix the tests it flags?

Not in this setup. The automation is read-only, so the run comments with the suggested change for each finding and the author applies it. To have fixes written, start a separate run without read-only and ask for a pull request against the author's branch.

### Our tests need a login. Can it still review them?

It reviews the code in every case and runs what it can reach. Tests that need an account or a token are listed as not run unless the Project has those values saved as secrets, which reach each session in the Project as environment variables. The review names which tests were run and which were only read.

### Does it review changes to the application under test too?

This prompt is written for the tests: specs, fixtures, page objects, configs and snapshots. If a pull request also changes application code, the agent says which files it did not review. A broader prompt on the same trigger can cover both.

## More Playwright jobs

- [Upgrade Playwright and Its Browsers in a Test Suite](https://getrush.ai/run/upgrade-playwright/playwright/)
- [Add Accessibility Tests to a Playwright Suite](https://getrush.ai/run/add-accessibility-tests/playwright/)
- [Add Merged Test Reports to a Playwright Workflow](https://getrush.ai/run/add-report-merging/playwright/)

## The same job on other stacks

- [Review Pull Requests on a Python Library](https://getrush.ai/run/review-pull-requests/python/)
- [Review Pull Requests on a Quarkus Service](https://getrush.ai/run/review-pull-requests/quarkus/)
- [Review Pull Requests Automatically in a Rails App](https://getrush.ai/run/review-pull-requests/rails/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
