All jobs

Write Integration Tests for a Reflex App

Yes. An agent writes browser tests for a Reflex app with reflex.testing.AppHarness and Playwright, runs them headless in its sandbox and opens a PR that CI picks up.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the app and the tests the repository already has before it writes new ones. In a repository like reflex-dev/reflex-examples, the todo app is one file, todo/todo/todo.py: a State with an items list that starts as Write Code, Sleep and Have Fun, an add_item handler that reads new_item from the submitted form data and appends it only when it is not empty, and a finish_item handler that removes an item by value. The page renders a Todos heading, an rx.form with an input named new_item, the placeholder Add a todo... and an Add button, with on_submit set to add_item and reset_on_submit true, then an ordered list built with rx.foreach where each item has an icon button with a check icon that calls finish_item. todo/requirements.txt pins reflex>=0.9.2, rxconfig.py names the app todo, and there is no tests directory and no requirements-dev.txt.

Two sibling apps show the house style. counter/tests/test_counter.py opens the app with AppHarness.create(root=Path(__file__).parent.parent) in a fixture, drives it with the Selenium driver from harness.frontend(), and waits with poll_for_content and AppHarness._poll_for. form-designer/tests/conftest.py makes the harness session-scoped and its tests use the page fixture from pytest-playwright, page.goto(harness.frontend_url), locators by role and placeholder, and Playwright's expect. AppHarness in reflex.testing compiles the app, starts the backend on a free port and the frontend dev server with PORT=0, and exposes frontend_url; its DEFAULT_TIMEOUT is 15 seconds and POLL_INTERVAL a quarter second. .github/workflows/app_harness.yml builds its matrix from every tests directory in the repository, requires a requirements-dev.txt beside it, installs reflex with both requirement files, runs playwright install chromium --with-deps --only-shell, then reflex init and pytest tests -vv with APP_HARNESS_HEADLESS set to 1 and REFLEX_TELEMETRY_ENABLED false.

The agent adds todo/tests/conftest.py with a session-scoped todo_app fixture that yields a started AppHarness for the app's root, and todo/tests/test_todo.py with four tests in the Playwright style: the page loads with the heading and the three starting items in order; filling the input and clicking Add appends a fourth item and clears the input, which checks reset_on_submit; clicking the check button inside the Sleep item removes that item and leaves the other two, which checks that finish_item removes by value and not by position; and submitting the form with an empty input adds nothing, which checks the guard in add_item. Each test waits with expect on the list rather than sleeping, and the assertions read the list items by role so a style change does not break them. todo/requirements-dev.txt lists pytest, playwright, pytest-playwright, uvicorn and psutil, the same five lines as the form designer's, which is what the workflow installs.

Then the agent runs it the way the workflow does. In its own sandbox it makes a virtual environment in todo/, installs reflex, the current 0.10.1, with both requirement files, installs the Chromium shell with Playwright, runs reflex init, which fetches the frontend toolchain the first time, and runs pytest tests -vv with APP_HARNESS_HEADLESS=1. It then breaks the app on purpose, dropping the if new_item: guard, reruns to see the empty-submit test fail, and restores it, so the pull request can show that the tests catch a regression. Because the workflow's matrix is built with find . -name tests -type d, the new directory joins it with no change to the workflow file.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Write integration tests for the todo app in this Reflex repository. Read todo/todo/todo.py, todo/rxconfig.py, todo/requirements.txt, counter/tests/test_counter.py, form-designer/tests/conftest.py, form-designer/tests/test_login.py, form-designer/requirements-dev.txt and .github/workflows/app_harness.yml first. Add todo/tests/conftest.py with a session-scoped AppHarness fixture for the app, and todo/tests/test_todo.py using the pytest-playwright page fixture: the starting items render in order, adding an item through the form appends it and clears the input, finishing an item removes exactly that item, and submitting an empty input adds nothing. Wait with Playwright's expect, not sleeps. Add todo/requirements-dev.txt matching the form designer's. Create a virtual environment, install reflex and both requirement files, install the Chromium shell with Playwright, run reflex init and pytest tests -vv headless, then remove the empty-input guard in the app to show a failing test and restore it. Open a pull request with the commands and their output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request adds todo/tests/conftest.py, todo/tests/test_todo.py and todo/requirements-dev.txt and changes nothing under todo/todo/. Its description quotes the pytest tests -vv output with the four tests, the failing run with the guard removed and the passing run after it is restored, and the reflex init and playwright install output, so a reviewer sees the environment the tests ran in. The workflow file has no diff, and the description shows the matrix line the next CI run will add for todo.

A reviewer checks that the locators are by role and placeholder rather than by generated class names, that the harness fixture is session-scoped so the app compiles once, and that the empty-submit test asserts on the list and not only on the input. After merging, the app-harness workflow runs the new directory on every push and pull request.

Run it on every nightly schedule

An automation in the Project that holds the repository takes a schedule trigger with cron set to a night-time hour. Its runs are read-only, so the saved prompt asks the run to find the example apps with no tests directory, write the tests in a scratch copy, run them and answer with the ones that pass and the diff that adds them, which the key's session read or the console shows in the morning. A run that should open the pull request is started from the team's own server with this body and a request_id that is a UUID v5 of the app's directory name, so a retried start returns the run already started.

A repository with several untested apps gets one such run per app, each with its own input naming the directory.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

A browser test is only evidence once it has run against the compiled app in a real browser, and that takes a toolchain: Python, the frontend bundler reflex init fetches, Chromium and the harness that starts both servers. The agent builds all of it in its own sandbox, runs the tests headless as CI will, makes them fail on purpose and keeps every command and its output in the run's record, so the pull request shows green and red rather than claiming them.

The tests follow the repository's own patterns, the session-scoped harness from the form designer, the requirement file the workflow insists on and the headless flag it sets, so they run in the existing matrix with no workflow change. Ten apps in the repository get ten pull requests, each with tests written against that app's state and handlers.

Frequently asked questions

Why Playwright rather than Selenium?

Both are in use in the repository: the counter test uses the Selenium driver from AppHarness.frontend(), and the form designer uses pytest-playwright. The workflow installs the Chromium shell through Playwright, and its expect waits for the list to change without hand-written polling, so the new tests follow the form designer.

How long do the tests take?

The first run pays for reflex init and the frontend build, which AppHarness does once per session because the fixture is session-scoped; the four tests then share one app and one browser. The pull request quotes the pytest timing from the sandbox run.

Will CI pick the new tests up?

Yes, with no workflow change. app_harness.yml lists every directory named tests in the repository to build its matrix and fails any app without a requirements-dev.txt, which is why the pull request adds that file alongside the tests.

What if the app's state changes later?

The tests read the page by role, text and placeholder, so a new colour or layout does not touch them, while a renamed button, a changed starting list or a dropped guard does, which is what they are for. A deliberate change updates one assertion in test_todo.py.

More Reflex jobs

The same job on other stacks

Keep reading