What the agent does
The agent starts by listing the interactions, since in an htmx app they live in template attributes and not in a JavaScript bundle. In an app like bigskysoftware/contact-app, a Flask application with no tests, templates/index.html has a search box that sends hx-get="/contacts" on keyup delay:200ms changed and swaps the tbody, and a span that loads /contacts/count on revealed. archive_ui.html starts a job with hx-post and then polls with load delay:500ms until a download link appears. rows.html deletes a row behind hx-confirm with hx-swap="outerHTML swap:1s". edit.html validates the email field on change, keyup delay:200ms into next .error, and its delete-btn deletes and returns to the list. The body sets hx-boost="true", so ordinary links are requests too.
It then reads the server for what each test must assert beyond the visible page. app.py returns the rows.html partial only when the HX-Trigger request header is search, and redirects with a flash message only when it is delete-btn, so the tests check those headers on the captured request and that the search response is a fragment and not a whole document. Contact.count sleeps two seconds and the archive job takes several seconds with random steps. The tests therefore wait on the response or on the text they expect, with Playwright's expect_response, expect_download and auto-retrying assertions, and contain no fixed sleeps. The confirm dialogs are native, so each delete test registers a dialog handler, once accepting and once dismissing, and the dismissed case asserts the row is still there.
Isolation takes some care because the app keeps its data in a file. Contact.load_db opens contacts.json relative to the working directory at import and save_db rewrites it on every change, while Flask finds templates and static files relative to the module. The agent uses that split: a fixture copies contacts.json into a temporary directory and starts the server there with PYTHONPATH pointing at the repository, FLASK_APP=app and flask run on a free port, since the pinned Flask 2.1.2 is configured through that variable. Each test gets its own process, so the class-level contact store and archive status never leak between tests and the repository's data file is never modified.
It writes the suite in Python with pytest-playwright, so the project keeps one toolchain, and puts the two packages in a requirements-dev.txt beside the pinned requirements.txt. Tests cover search with a term that matches several seeded contacts and one that matches none, the lazy count, both validation messages (Email Required and Email Must Be Unique), the archive flow through to the downloaded file and the reset, single delete, delete from the edit page, bulk delete of two ticked rows, and a boosted link checked by a marker on window that survives the navigation. It runs the suite three times in the sandbox, then changes the search target once to show a red test and restores it.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Write Playwright end-to-end tests for this htmx app. Read every template and list each hx- attribute with its trigger, target and swap, and any direct htmx or Alpine calls. Read the server routes they call and note where the response depends on a request header such as HX-Trigger or HX-Request, where a handler is slow on purpose, and where state is kept. Use the Python Playwright plugin for pytest if the app is Python, added in a separate dev requirements file; do not change application code or pinned requirements. Start the app for each test as a subprocess on a free port, with any data file it rewrites copied to a temporary directory so the repository copy is never modified and tests share no state. Write one test per interaction: search as you type, including a term with no matches and the pushed URL; lazy-loaded content; inline field validation with each message; a polling flow through to its end state and any download; deletes behind a confirm dialog, accepted and dismissed; bulk actions on several selected rows; and a boosted link, proving no full page load happened. Assert request headers and that partial responses are fragments. Never use a fixed sleep: wait on the response, the download or the expected text. Fail a test on any unexpected 4xx or 5xx response, and report assets that are already missing instead of fixing them. Install the browser, run the suite three times, break one hx-target to show a failing test, restore it, and open a pull request with the output and a note on which tests need network access to a CDN.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request adds a tests/ directory with a conftest.py holding the server fixture and one test module per page, plus requirements-dev.txt and a short README section with the two commands that install the browser and run the suite. No template, route or pinned requirement changes. The description lists each test against the attribute it exercises, shows the three runs, and shows the red run from the deliberate break with Playwright's message naming the element that never updated.
It reports what the tests turned up without fixing it. Here that would include the spinner image requested from /static/img/, which the repository does not contain, so the unexpected-response check carries one documented exception; two elements on the index page sharing the id spinner, which a strict locator refuses; and the tests that rely on Alpine and a stylesheet loaded from a CDN, marked so a network failure is distinguishable from a regression. A reviewer checks that no test sleeps, that every delete test would fail if the server ignored the request, and that the working tree's contacts.json is unchanged after a run.
Run it on every pull request that changes a template
Once the suite exists, new interactions should arrive with a test. Create an automation in the Project that holds the repository with a github trigger on pull_request.labeled and needs-e2e in its labels. A maintainer adds the label to a pull request that introduces an hx- attribute without a test. The trigger puts the pull request after the saved prompt and fills in nothing, so the prompt says: read the pull request that follows, write Playwright tests in the style of the existing suite for the interactions it adds, and open a pull request against its branch. It fires only when the label is added by an owner, member or collaborator, and never for a fork.
To give several server-rendered apps their first suite in one go, a script on your own server sends this page's body once per repository, each with a request_id that is a UUID v5 of the repository's full name, so a retry returns the run already started. Each run reads its own templates, and an app written in Go or Ruby gets Playwright's test runner for Node in a tests/ package of its own.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
End-to-end tests for hypermedia apps go wrong in a predictable way: someone adds a sleep for the debounce, another for the polling, and the suite becomes slow and still fails now and then. Getting them right means reading each trigger and waiting for exactly the request it causes, then running the suite repeatedly to see whether it holds. The agent does both in a sandbox with its own browser, and the three runs and the deliberate failure are kept in the run's record, so the pull request arrives with evidence about flakiness and not only a green tick.
The reading matters as much as the running. The contract of an htmx page is split between an attribute in a template and a header check in a route, and a test that only looks at the screen misses half of it. The agent asserts on both sides, isolates each test from the file the app writes, and reports what it found on the way. A team with several such apps starts one run per repository and reviews the pull requests as they come in.
Frequently asked questions
How do the tests wait for htmx without sleeping?
They wait for the thing the trigger causes. A search test types and then waits for the response to /contacts?q=…; the lazy count waits for its text; the archive test waits for the download event. Playwright's assertions retry until their timeout, so a two second handler needs a longer timeout and no sleep.
Why Python Playwright and not the Node test runner?
Because the app is Python and has no package.json, so pytest-playwright adds a test dependency without adding a second toolchain. If your repository already has Node tooling or the backend is not Python, the agent uses @playwright/test instead.
Will the tests change my data file or database?
No. The fixture copies the data file to a temporary directory and starts the app there, and each test gets its own server process. For an app with a real database the agent uses a throwaway database per run and says how it is created.
Can it test the confirm dialogs?
Yes. hx-confirm uses the browser's native dialog, which Playwright exposes as an event. Each delete is tested twice, once accepting and once dismissing, and the dismissed case asserts that no request was sent and the row remains.
More htmx jobs
The same job on other stacks
- Write Playwright Tests for an Inertia.js App (Inertia.js)
- Write Playwright Tests for a Laravel App (Laravel)
- Write Playwright Tests for a Next.js App (Next.js)