All jobs

Write Pytest Tests for a Reflex App

Yes. An agent writes browser tests for your Reflex app with AppHarness and Playwright, matching the layout your CI already discovers, runs them and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the app and its neighbours are tested before writing anything, because Reflex apps are tested by running them. In a repository like reflex-dev/reflex-examples, the todo app has rxconfig.py with app_name="todo" and the Radix themes plugin, requirements.txt pinning reflex>=0.9.2, and todo/todo.py with a State whose items list starts with three entries, an add_item handler that reads new_item from the submitted form, and a finish_item handler that pops an item; the page is a heading, a form with an input named new_item and an Add button with reset_on_submit=True, and an ordered list where each item has an icon button that finishes it. There is no tests directory and no requirements-dev.txt. The repository's .github/workflows/app_harness.yml discovers every example that has a tests directory, fails if it lacks requirements-dev.txt, installs reflex with both requirement files, runs playwright install chromium --with-deps --only-shell, reflex init and pytest tests -vv with APP_HARNESS_HEADLESS=1. Two neighbours show the house style: counter/tests/test_counter.py drives the app through AppHarness.create(root=...) and a Selenium driver from harness.frontend() with poll_for_content, while form-designer/tests/conftest.py keeps a session-scoped AppHarness fixture and the tests use Playwright's page fixture against harness.frontend_url.

The agent follows the newer pattern. todo/tests/conftest.py defines a session-scoped todo_app fixture that enters AppHarness.create(root=Path(__file__).parent.parent), which compiles the app, starts the backend under uvicorn in a thread and the frontend as a dev server subprocess, and exposes frontend_url once the frontend is listening. todo/requirements-dev.txt lists pytest, playwright, pytest-playwright, uvicorn and psutil, the same lines form-designer uses, so the existing workflow picks the app up with no change to CI.

todo/tests/test_todo.py then pins down the behaviour in todo.py. One test opens the page and asserts the three default items are listed in order. One fills the input with a new item, clicks Add, waits for a fourth list item and asserts the input is empty again because of reset_on_submit. One submits an empty form and asserts the list is unchanged, since add_item ignores a blank new_item. One clicks the check button inside the first list item and asserts that item is gone and the other two remain. Each test uses Playwright's expect with its own timeout rather than sleeps, and locates elements by role and text so a styling change does not break them.

Running them needs the frontend toolchain reflex init downloads and a Chromium that Playwright installs, both of which the agent sets up in its sandbox with the workflow's own commands. It runs pytest tests -vv under APP_HARNESS_HEADLESS=1, reads each failure itself, and also runs reflex export once, because the repository's check_export.yml builds every example that way and the test files must not break it. ruff format and ruff from .pre-commit-config.yaml run over the new files, and the app's own code is left unchanged.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Write pytest tests for the Reflex app in the todo directory and keep its code unchanged. Read todo/rxconfig.py, todo/requirements.txt, todo/todo/todo.py, .github/workflows/app_harness.yml, .github/workflows/check_export.yml, .pre-commit-config.yaml, counter/tests/test_counter.py, form-designer/tests/conftest.py and form-designer/requirements-dev.txt first. Add todo/requirements-dev.txt with the same lines as form-designer. Write todo/tests/conftest.py with a session-scoped AppHarness fixture rooted at the app, and todo/tests/test_todo.py using Playwright's page fixture against frontend_url: the three default items in order, adding an item and the input resetting, an empty submit changing nothing, and finishing the first item. Use expect with locators by role and text, no sleeps. In the sandbox run playwright install chromium --with-deps --only-shell, reflex init, pytest tests -vv with APP_HARNESS_HEADLESS=1, reflex export, and ruff over the new files, fix what fails, and open a pull request that lists each test and what it asserts.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds todo/requirements-dev.txt, todo/tests/conftest.py and todo/tests/test_todo.py and changes nothing else. The description lists the four tests and the line in todo.py each one pins down, notes that app_harness.yml discovers the new tests directory on its own, and reports what reflex export produced.

The run's record keeps the reflex init, pytest tests -vv, reflex export and ruff output, so a reviewer sees the tests pass in a headless browser rather than trusting a sentence.

Before merging, a reviewer decides whether the shared fixture should move up to a repository-level conftest.py so the next example gets it for free, and whether to pin pytest-playwright in requirements-dev.txt as the other apps leave it unpinned.

Run it on every night

In the console, add an automation to the Project with this prompt and a schedule trigger, a cron of 0 2 * * * and your timezone. Each night one run finds the apps that changed since the last run and still have no tests directory, writes tests for them the same way, and opens one pull request on its own branch. A run whose tests cannot be made to pass ends needs_review rather than merging anything.

The same body sent to POST /o/{org}/sessions with the repository in resources starts one run by hand, with the app directory written into the prompt. A request_id that is a UUID v5 of the repository and directory together makes a retried call return that run instead of a second pull request.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

A Reflex test is a running app: a backend, a compiled frontend, a browser and the toolchain reflex init fetches on first use. None of that belongs on a laptop that is also editing the code, and the agent's sandbox has room for all of it. It runs the suite after every file it writes and reads the failures itself, so a locator that matches two buttons or an expect that races the websocket is fixed before anyone reviews it.

Because every pytest run and the reflex export stay with the record, the pull request can say which handlers in todo.py are now covered and show the headless run that proved it, instead of asking a reviewer to install Chromium and find out.

Frequently asked questions

Can I test the State class without a browser?

Handlers like add_item are plain methods on a State subclass, but the list the page shows comes from the compiled frontend through a websocket, so this repository's own tests drive the app in a browser with AppHarness. The agent follows that pattern; pure handler tests are a separate, smaller layer it mentions in the pull request.

Why Playwright rather than the Selenium driver that AppHarness.frontend() returns?

Both work, and counter uses Selenium. The form-designer tests use Playwright's page fixture with frontend_url, its expect waits on its own, and app_harness.yml already installs Chromium for Playwright, so new tests follow the newer of the two patterns.

Does CI need any change?

No. The workflow lists every directory with a tests folder and requires requirements-dev.txt beside it; adding both to todo puts it in the matrix on the next push.

What does the agent need in its sandbox?

Python with the two requirement files, the frontend toolchain reflex init installs, and a Chromium from playwright install. It runs the same commands as the workflow under APP_HARNESS_HEADLESS=1, so what passes there is what CI runs.

More Reflex jobs

The same job on other stacks

Keep reading