What the agent does
The agent starts with the arithmetic the config implies, because a timeout that can never fire is not a timeout. In an app laid out like vercel/chatbot, playwright.config.ts for @playwright/test 1.61.1 sets timeout: 240 * 1000 next to a comment that says 120 seconds, expect: { timeout: 240 * 1000 }, retries: 0, workers: 2 in and out of CI, fullyParallel: true, trace: "retain-on-failure" and a webServer that runs pnpm dev with its own timeout: 120 * 1000 and waits on /ping, which proxy.ts answers. The four files under tests/e2e/ hold 23 tests. With a 4-minute ceiling per test and per assertion, a hung expect sits for four minutes before it fails, and the suite's worst case on two workers is 46 minutes against the timeout-minutes: 30 the job in .github/workflows/playwright.yml gives it. The job dies first, and the HTML report it uploads from playwright-report/ is never written.
It then reads what the tests themselves wait for, since the right limits come from there. tests/e2e/api.test.ts waits up to 30 seconds for an assistant message to appear and 10 seconds for the URL to change to a chat, and tests/e2e/chat.test.ts clicks the stop button with a 5-second timeout wrapped in a .catch that swallows the miss. tests/pages/chat.ts waits on [data-testid='multimodal-input'] with no limit of its own. The agent keeps every explicit per-call timeout that is longer than the new default, because those are the places the test author already thought about, and removes none of them.
The new config sets limits by kind. timeout drops to a one-minute default per test, expect.timeout to ten seconds so a missing element fails in the time a person would notice, use.actionTimeout and use.navigationTimeout get their own values so a click that never resolves and a page that never loads fail with those words in the error, and globalTimeout caps the whole run a few minutes under the job's 30 so Playwright writes its report before GitHub kills the process. The tests that wait on a model answer are marked with test.slow() in api.test.ts, which triples their budget, rather than raising the default for all 23. The webServer.timeout stays at 120 seconds, because next dev --turbo compiling on a cold runner is the one place a long wait is honest. retries stays at 0: a retry hides a timeout, and this change is about surfacing them.
To prove the limits fire, the agent runs the suite in its sandbox with pnpm exec playwright test --list to show every test still collects, then adds a throwaway test that awaits an element that never appears and runs it alone, reading in the output that it fails at the expect limit and not at four minutes, and deletes it. The whole suite needs the app's POSTGRES_URL and AUTH_SECRET; when the Project holds them as secrets the agent runs it end to end with the new config and quotes the per-test durations from the report, and when it does not, it says so and quotes the targeted run instead. It runs pnpm check, the repository's lint, before opening the pull request.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Set sensible timeouts in the Playwright suite. Read playwright.config.ts, every file under tests/, proxy.ts, package.json and .github/workflows/playwright.yml first, and work out the suite's worst-case duration against the CI job's timeout-minutes. Replace the single long timeout with a per-test default, an expect timeout, actionTimeout and navigationTimeout, and a globalTimeout that leaves room under the job limit for the HTML report. Keep every explicit per-call timeout that is longer than the new default, mark the tests that wait on a model answer with test.slow(), and do not add retries. Prove the limits with a throwaway test that waits on an element that never appears, run alone, then delete it. Run the full suite if the Project's secrets allow it, run pnpm check, and open a pull request that quotes the old and new values and the proof run.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that changes playwright.config.ts and adds test.slow() to the tests in tests/e2e/api.test.ts that wait on a model, with no change to the workflow. Its description is a table of each timeout before and after, the worst-case arithmetic for the suite against the job's 30 minutes, and the output of the proof run showing the hung assertion failing at the new expect limit.
The run's record keeps the --list output, the proof run and the lint, so a reviewer can see that every test still collects and that the one meant to fail did so in seconds, not minutes.
Before merging, a reviewer checks the values against what the app actually needs on their runners, since a slower CI machine than the agent's sandbox may want a longer navigationTimeout, and decides whether the stop-button test in chat.test.ts, whose .catch swallows a miss today, should become a real assertion now that a failure would be fast. The pull request names both.
Run it on every pull request that changes a test or the config
An automation in the Project with a github trigger on pull_request.opened and pull_request.synchronize runs a prompt that reads the pull request's changes to playwright.config.ts and tests/, recomputes the worst-case duration against timeout-minutes in the workflow, and comments when a change pushes the suite past the job again or adds a per-call timeout longer than the test default. The trigger adds the event after the prompt and fills in nothing else.
When a nightly run of the suite times out, a handler on your server that receives the workflow's failure can start one run with this body and a request_id that is a UUID v5 of the workflow run id, so a retried delivery returns the same run. The key's webhook_url reports the result when the run finishes.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
The numbers in a Playwright config are easy to type and hard to justify, and a wrong one shows up only when CI is red for half an hour and the report is missing. The agent reads the tests, the config and the workflow together, works out what the current values imply, and proves the new ones by running a test that is meant to fail, in a sandbox it owns, with the output kept.
A team with several apps on Playwright runs this once per repository, in parallel, each in its own sandbox. Each pull request carries that suite's own arithmetic and proof, so a suite that already has sane limits gets a pull request saying so rather than a copy of someone else's config.
Frequently asked questions
Why not just raise the job's timeout-minutes?
Because the job limit is the backstop, not the diagnosis. Raising it to fit a four-minute-per-test ceiling makes a hung test cost more minutes, not fewer, and still leaves no report when the limit is hit. Per-kind timeouts fail the one test that hung and let the other 22 finish and report.
Will the tests that call the model now fail on a slow day?
Those tests keep their explicit 30-second waits, and test.slow() gives them three times the default test budget, so they have more room than the rest of the suite and the same room they had for the assertion itself. If the model is slower than that, the test should fail, and the error will say which wait expired.
Does globalTimeout stop the report from being written?
No, the opposite. When globalTimeout fires, Playwright stops scheduling tests, marks the rest as not run and writes the HTML reporter's output, so the upload step in the workflow has something to upload. A job killed by timeout-minutes writes nothing.
Why does the agent leave retries at 0?
A retry masks the timeout this change is trying to surface, and the config's trace: "retain-on-failure" already gives a reviewer the trace of the failing attempt. If the team wants retries in CI later, that is a one-line change the pull request mentions, made on purpose rather than bundled here.
More Playwright jobs
The same job on other stacks
- Add Test Timeouts to a pytest Suite (pytest)