All jobs

Add Snapshot Tests to a Textual App

Yes. An agent adds pytest-textual-snapshot to a Textual app, writes snapshot tests for its screens and key bindings, and opens a pull request with the SVGs.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the app is built and started before it tries to drive it. In a log viewer like Textualize/toolong, pyproject.toml is a Poetry project, toolong 1.5.0, with python = "^3.8", click, textual = "^0.58.0" and typing-extensions as dependencies, textual-dev in the dev group, and one script, tl = "toolong.cli:run"; poetry.lock pins Textual 0.58.0 and textual-dev 1.5.1. There is no tests/ directory, no pytest in the lock, and no workflow under .github/. src/toolong/cli.py is a Click command that takes file paths and -m/--merge, builds UI(files, merge=merge, save_merge=output_merge) from ui.py and calls run(); when stdin is not a TTY it copies the pipe into a temporary file and relaunches itself, which a test will not go through.

Then it maps what a snapshot can show. UI.on_mount pushes LogScreen, which composes a TabbedContent with one TabPane per file, each holding a Lazy(LogView(...)), and hides the tab bar when there is one file. LogView in log_view.py composes LogLines, a LinePanel, a FindDialog, an InfoOverlay and a LogFooter, with bindings ctrl+t for tail, ctrl+l for line numbers, ctrl+f for find and ctrl+g for go to; LogLines in log_lines.py binds enter to select a line and m, o and d to jump by minute, hour and day; ui.py binds f1 to a HelpScreen. format_parser.py detects common, combined, JSON and default line formats, and highlighter.py colours them. Each of those is a state worth a picture: a plain file, a JSON Lines file, the find dialog open with a match, the line panel after enter, the help screen, and a merged view of two files.

The agent adds pytest and pytest-textual-snapshot, 1.1.0 at the time of writing, which needs Python 3.8.1 or newer, pytest 8 and Textual 0.28 or newer and pins syrupy 4.8.0, to the dev group, and checks that Poetry resolves them against the locked Textual 0.58.0. It writes small fixture logs under tests/fixtures/: a combined-format access log with timestamps spanning three days, a JSON Lines file, and a second access log for the merge case, each short enough that the whole file fits the terminal, so a snapshot shows every line. Each test uses the snap_compare fixture with a tiny tests/apps/ script that builds UI([...]) directly, bypassing Click and the TTY branch, plus press for the keys: ["ctrl+f", "G", "E", "T"] to open find and type a query, ["enter"] for the line panel, ["f1"] for help, ["ctrl+l"] for line numbers, and terminal_size=(100, 30) so the SVG is deterministic across machines. Timestamps in the footer come from the fixture files, not the clock, so the snapshots are stable.

It runs pytest --snapshot-update once to create the SVGs under tests/__snapshots__/, opens each in its browser to confirm the picture is the state it meant, then runs pytest again to prove the comparison passes clean. A test that depends on the file watcher or on a Lazy widget not yet mounted gets a run_before hook that waits for LogLines to appear, and the agent notes the one it had to add. Last it adds .github/workflows/ci.yml with poetry install and pytest on the Python versions the project claims, and uploads snapshot_report.html, the diff page the plugin writes on failure, as an artifact so a failed snapshot can be read without rerunning.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add snapshot tests to this Textual app. Read pyproject.toml, poetry.lock and every module under src/ first, and list each screen, dialog and key binding a snapshot can show. Add pytest and pytest-textual-snapshot to the dev dependencies and confirm they resolve against the locked Textual version. Write short fixture files under tests/fixtures that exercise each format the app parses, with timestamps that do not depend on the clock. For each state write a test with the snap_compare fixture, a small script under tests/apps that constructs the App class directly, press for the keys, and a fixed terminal_size. Run pytest --snapshot-update once, open every SVG in the browser to check it shows the intended state, then run pytest again to prove it passes. Add a CI workflow that runs the suite on the supported Python versions and uploads snapshot_report.html on failure. Open a pull request that lists each snapshot and the state it captures.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request with the dev dependency change in pyproject.toml and poetry.lock, tests/fixtures/ with the sample logs, tests/apps/ with the launcher scripts, tests/test_snapshots.py, the SVGs under tests/__snapshots__/, and the workflow. Its description lists every snapshot with the keys pressed and the state it shows, each SVG rendered inline so a reviewer sees the pictures on the pull request, and the pytest output for the clean second run; the browser checks of each SVG are in the run's record.

A reviewer looks at the pictures first, since a snapshot test asserts whatever the first run produced. The find dialog should show the match count, the line panel should show the selected line's fields, and the merge view should interleave two files by timestamp. Any picture that is not the intended state is a test to fix before merge, and the run_before hooks are where to look if a snapshot captured a widget still loading.

Run it on every nightly schedule

An automation in the Project with a schedule trigger on cron for the small hours in your timezone runs this prompt with a line added: add a snapshot only for a screen, dialog or binding that has none, and open no pull request when every state is covered. Each night's run reads the current modules, so a new dialog gets its picture the night it lands.

For several Textual apps, your own server sends this body once per repository, each POST /o/{org}/sessions with that repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started. Each run reads its own lock file, so an app on an older Textual gets the plugin version that fits it.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

A terminal app's correctness is what it draws, and the only honest check is to drive it with keys in a terminal of fixed size and compare the result. The agent does that in its own sandbox, where the terminal size, the fixture files and the Python version are all pinned, so the SVGs it commits are the ones CI will reproduce; the pull request shows each picture, and the browser pass over every SVG is in the run's record.

It also removes the part of snapshot testing people skip, which is looking at every snapshot before accepting it. The agent opens each one and names the state it shows, so a snapshot that captured a loading widget is caught on the branch and not six months later when it silently stops matching. Several apps get the same care at once.

Frequently asked questions

Will the snapshots be stable in CI?

Yes, as long as the inputs are fixed. The tests set terminal_size, use fixture files with their own timestamps rather than the clock, and construct the App directly instead of going through the CLI's TTY detection. The SVG is text, so a diff is readable and the plugin's HTML report shows both sides.

How do I update a snapshot after a deliberate change?

Run pytest --snapshot-update locally or in a follow-up to the same session, then review the SVG diff like any other file. The pull request's description names each snapshot and the state it captures, so a change to the find dialog tells you which pictures should move.

Does it test the file watcher or tailing?

No. Tailing depends on a file changing while the app runs, which a snapshot cannot capture deterministically. The tests cover the parsed formats, the dialogs, the key bindings and the merge view; tailing is listed in the pull request as out of scope for snapshots.

What if the app's pinned Textual is too old for the plugin?

The agent checks the resolution before writing a test. The plugin's floor is Textual 0.28, so an app on 0.58 resolves cleanly. For an app below that floor it reports the gap and stops rather than bumping Textual, since that upgrade is its own change.

More Textual jobs

The same job on other stacks

Keep reading