What the agent does
The agent reads how the site runs and what it contains before writing a test. In a site shaped like wagtail/bakerydemo, requirements/base.txt pins Django>=6.0,<6.1 and wagtail>=8.0,<8.1, .python-version says 3.12 and .nvmrc says 24; package.json has only eslint, stylelint and prettier, no test runner, and the Python tests are two files under bakerydemo/base/tests. bakerydemo/settings/base.py uses DATABASE_URL when it is set and falls back to SQLite otherwise, settings/test.py sets ALLOWED_HOSTS to *, a local-memory cache and the immediate task backend, and .github/workflows/ci.yml runs manage.py check, makemigrations --check and manage.py test under that settings module, so a browser suite can run on SQLite with no service container. load_initial_data in bakerydemo/base/management/commands copies the fixture media and loads base/fixtures/bakerydemo.json, then runs update_index, which is what makes search return results.
The fixture is the test data, so the agent reads it rather than inventing pages. bakerydemo.json holds pages with slugs such as breads, blog, locations, recipes, gallery, about and contact-us, bread pages like baguette, bagel and anadama-bread, locations like reykjavik and hofn, and blog tags. The models say what each page shows: BreadPage in breads/models.py has an origin country and a bread type through snippets, LocationPage has opening hours and a map, BlogIndexPage is a RoutablePageMixin with tags/ and tags/<tag>/ routes in blog/models.py, the two page models in recipes/models.py carry the step blocks, and FormPage is the contact form. search/views.py reads q, searches the blog, bread, location and recipes/ page models with the database backend configured in WAGTAILSEARCH_BACKENDS, and paginates. templates/includes/header.html has a search input with id="search-input" and name="q", a data-theme-toggle button and a data-mobile-navigation-toggle, which gives the tests stable selectors without touching templates.
In the sandbox the agent creates a virtualenv with uv, installs requirements/development.txt, runs manage.py migrate and manage.py load_initial_data, and starts manage.py runserver on port 8000 with bakerydemo.settings.dev. It adds @playwright/test to package.json, a playwright.config.ts whose webServer runs that same runserver command with a readiness URL of / and baseURL set, and a test:e2e script. The tests cover one thing each: the home page renders the header navigation and the hero; /breads/ lists bread cards and /breads/baguette/ shows the origin and bread type from its snippets; /blog/ lists posts and a /blog/tags/<tag>/ route filters them; /locations/ lists locations and a location page shows its opening hours; /recipes/ and one page under it render the step blocks; /search/?q=bread returns results that link to bread pages and an empty query shows the empty state; the theme toggle changes the document's theme attribute; the mobile navigation toggle reveals the menu at a phone viewport; /sitemap.xml lists the live pages; and /admin/login/ renders the Wagtail login form.
It adds an e2e job to ci.yml beside lint-server, lint-client and test-server: set up Python with uv as the other jobs do, Node from .nvmrc, npm ci, npx playwright install --with-deps chromium, migrate and load_initial_data, then npm run test:e2e, with the Playwright HTML report uploaded as an artifact on failure. It changes no template and no model; a test that needs an element with no stable selector is written against visible text instead, and the pull request lists those so a developer can add a data-testid later if they prefer.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Write Playwright end-to-end tests for this Wagtail site and open a pull request. Read requirements/, .python-version, .nvmrc, package.json, bakerydemo/settings/base.py, dev.py and test.py, bakerydemo/urls.py, every models.py under bakerydemo/, bakerydemo/search/views.py, bakerydemo/templates/includes/header.html, the load_initial_data command and the slugs in base/fixtures/bakerydemo.json, and .github/workflows/ci.yml first. In the sandbox, create a virtualenv with uv, install requirements/development.txt, run manage.py migrate and manage.py load_initial_data, and start manage.py runserver with the dev settings. Add @playwright/test as a dev dependency, a playwright.config.ts whose webServer runs that server with baseURL set, and a test:e2e script. Write one test per behaviour, using the fixture's pages: the home page, the breads index and one bread page with its snippet fields, the blog index and one tag route, the locations index and one location's opening hours, the /recipes/ index and one page under it, search with a matching query and with an empty query, the theme toggle, the mobile navigation at a phone viewport, sitemap.xml and the admin login form. Use existing ids, data attributes and visible text; change no template or model. Add an e2e job to ci.yml that installs Python and Node as the other jobs do, loads the fixture, installs Chromium and runs the suite, uploading the HTML report on failure. Run the suite until it passes and open a pull request with the test list and the run output.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request adds playwright.config.ts, a tests/e2e/ directory (or e2e/, matching whatever the repository already uses for browser tests) with one spec per area, two lines in package.json for the dependency and the script, an updated package-lock.json, and an e2e job in ci.yml. The body lists each test with the page or route it covers and pastes the Playwright run output from the sandbox.
The new job runs on the pull request itself alongside test-server, lint-server and lint-client, so the first evidence is a green e2e check on the same workflow. Because the suite runs on SQLite with the fixture, it needs no Postgres service and no Elasticsearch, and a reviewer can run npm run test:e2e locally after load_initial_data with nothing else installed.
Before merging, a reviewer checks that the selectors are the header's existing ids and data attributes or visible text rather than new attributes in templates, that each test asserts something the fixture guarantees (a bread's origin, a location's hours) rather than a count that changes when editors add pages, and that the job's Chromium install is cached or at least bounded so ci.yml does not double in length.
Run it on every pull request that changes a template or page model
Add an automation to the Project with a github trigger on pull_request.opened and pull_request.synchronize, branch_patterns on main, and a prompt that says to run the Playwright suite on the pull request's branch and, when the diff touches bakerydemo/templates/ or a models.py, add a test for the changed page type before pushing. The trigger appends the pull request to the prompt, so the agent checks out that branch and pushes to it; max_runs_per_hour keeps a busy review day inside a budget.
From your own server, a handler for GitHub's pull_request webhook sends POST /o/{org}/sessions with the prompt and https://github.com/your-org/your-app/pull/<n> in resources, which clones the pull request's branch, and a request_id that is a UUID v5 of the pull request number and head commit so a redelivery returns the same run.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Browser tests for a CMS are mostly setup: the right Python and Node, the fixture loaded, media copied, the search index built, a browser installed, and a server that is up before the first test opens a page. On a laptop that is the part that goes wrong, and the suite gets written against a half-loaded site. In a Rush sandbox the agent does the whole sequence from the repository's own README and workflow, and every command with its output is in the transcript.
The fixture makes the tests honest: each one asserts something bakerydemo.json guarantees, so the suite proves the templates and page models, not a particular editor's content. Once the e2e job is in ci.yml, every pull request runs it, and the agent can be asked to add a test for each new page type.
Frequently asked questions
Does it need Postgres or Elasticsearch?
No. settings/base.py falls back to SQLite without DATABASE_URL, and WAGTAILSEARCH_BACKENDS uses the database backend, so load_initial_data plus update_index is enough for search to return bread and blog pages. The compose file's Postgres and Redis are for the dev container, not the tests.
Will it add test ids to my templates?
No. It uses what header.html already has, search-input, data-theme-toggle and data-mobile-navigation-toggle, and visible text elsewhere. Where a selector is fragile it says so in the pull request so you can add a data-testid on your own terms.
What about the admin?
The suite checks that /admin/login/ renders the login form, which proves the admin URL include and its static files. Logging in and editing pages is a different suite: say so in the prompt and the agent signs in with the fixture's admin account and tests the page editor.
Why run the server from playwright.config.ts instead of a separate step?
Playwright's webServer starts manage.py runserver, waits for the readiness URL and stops it after the run, so the same command works locally and in the e2e job without a background process and a sleep. The job still loads the fixture first, because that is data, not a server.