All jobs

Write Pytest Tests for a Textual Terminal App

Yes. An agent tests the parsers and file reader under your Textual app, drives the app itself with run_test, adds a workflow to run them and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the package before deciding what to test. In an app like Textualize/toolong, pyproject.toml is a Poetry project at version 1.5.0 that depends on click 8.1, textual 0.58 and typing-extensions, exposes the tl script as toolong.cli:run, and has textual-dev as its only development dependency. There is no tests/ directory, no pytest configuration and no workflow under .github/. The code under src/toolong/ splits into pure modules and Textual widgets: timestamps.py holds seventeen TimestampFormat regex and parser pairs and a TimestampScanner that moves the last matching format to the front of its list; format_parser.py holds JSONLogFormat, CommonLogFormat and CombinedLogFormat plus a FormatParser that rotates the format that matched to the front and falls back to DefaultLogFormat; log_file.py holds LogFile, which opens plain files with buffering=0, copies .gz and .bz2 files into a temporary file with can_tail set to False, and scans line breaks backwards with mmap.rfind in batches of a thousand; cli.py is a Click command with -m/--merge and -o/--output-merge that prints help when no files are given on a terminal.

The pure modules get most of the tests because they decide what the app shows. tests/test_timestamps.py feeds each of the seventeen regexes one matching line and checks the datetime that comes back, checks that TimestampScanner.scan reorders its formats after a match on an Apache-style 29/Jan/2024:13:45:19 +0000 line, and checks the ten-thousand-character truncation. One test records a behaviour the agent found while reading: the module-level parse function hands match.string, the whole input, to the parser instead of the matched group, so a JSON log line whose timestamp sits inside other text yields None. The agent writes that test as a strict xfail with the reason in its id and names it in the pull request rather than changing the production code. tests/test_format_parser.py checks that a JSON line parses through JSONLogFormat, that a Common Log Format line gets its status code highlighted and the request method in bold yellow, that a blank line falls through to DefaultLogFormat with an empty Text, and that the rotation puts the matching format first for the next line.

tests/test_log_file.py uses tmp_path to write small files and checks LogFile.open sets size and can_tail, get_line decodes with replacement and expands tabs to four spaces, scan_line_breaks yields the offsets of every newline for a file with and without a trailing newline, and that a .gz and a .bz2 copy of the same file report is_compressed, open through open_compressed into a temporary file and set can_tail to False, plus one case where the exit_event is already set and open returns False. tests/test_cli.py uses Click's CliRunner for --help and --version; the branch that copies piped stdin to a temporary file consults sys.__stdin__.isatty() and forks a subprocess, which the agent leaves to a manual check and says so in a comment.

The app itself gets one smoke test through Textual's test harness: UI([path]).run_test() with a two-line log file, await pilot.pause(), and assertions that the app mounted and shows the file's name, written as an async test so it needs pytest-asyncio with asyncio_mode = "auto" in [tool.pytest.ini_options]. The agent adds pytest and pytest-asyncio to the Poetry dev group, runs poetry install and poetry run pytest -q in its sandbox, and adds .github/workflows/test.yml that installs Poetry and runs the suite on the oldest and newest Python the project declares, because without a workflow a pull request with passing tests has no green check to show.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Write the missing tests for this Textual application. Read pyproject.toml and every module under src/toolong first, and keep the production code unchanged. Add pytest and pytest-asyncio to the Poetry dev dependency group and a [tool.pytest.ini_options] section with asyncio_mode set to auto. Write tests/test_timestamps.py covering every TimestampFormat entry, the TimestampScanner reordering and the long-line truncation; tests/test_format_parser.py covering JSON, Common Log Format, blank lines and format rotation; tests/test_log_file.py using tmp_path for open, get_line, scan_line_breaks with and without a trailing newline, gzip and bzip2 files and an already-set exit event; tests/test_cli.py using CliRunner for --help and --version; and one async smoke test that runs the UI app with run_test on a small file. Where a test exposes a bug, mark it as a strict xfail with the reason and list it in the pull request instead of fixing it. Add .github/workflows/test.yml that installs Poetry and runs pytest on the oldest and newest Python versions pyproject.toml allows. Run the suite until it passes and open a pull request that lists each test file, what it covers and any xfail.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds five files under tests/, two dev dependencies and a pytest section in pyproject.toml, an updated poetry.lock, and .github/workflows/test.yml. The description lists each test file with the functions and classes it covers, names the strict xfail on timestamps.parse with the line that causes it, and states which branch of cli.py is not covered and why.

The run's record keeps the poetry install output, every pytest run including the ones that failed while the tests were being written, and the final passing run, so a reviewer can read the assertions that were adjusted and why rather than only the end state.

Before merging, a reviewer reads the xfail and decides whether to fix parse in a separate change, confirms the workflow's Python matrix matches what pyproject.toml promises, and checks that no test writes outside tmp_path.

Run it on every night

In the console, add an automation to the Project with this prompt and a schedule trigger, a cron of 0 2 * * * and your timezone. Each night one run reads what changed since the last, writes tests for the untested paths it finds and opens one pull request on its own branch, so the morning review is a diff of tests rather than a request to write them. A run whose new tests cannot be made to pass ends needs_review instead of merging anything.

The same body sent to POST /o/{org}/sessions with the repository in resources starts one run by hand, for instance before a release, and a request_id that is a UUID v5 of the release tag makes a retried call return that run rather than a second pull request.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

Tests for a terminal app need a place where a headless Textual app, Poetry, two Python versions and gzip and bzip2 fixtures can all run without touching anyone's shell, and the agent's sandbox is that place. It installs the project, runs pytest after every file it writes, and reads the failures itself; a regex that matches only on one timezone or a scan_line_breaks offset that is off by one is fixed in the test before anyone sees it.

Because every pytest run stays with the record, the pull request can say exactly which behaviour it pinned down and which one it found broken, with the output to show for it, instead of asking a reviewer to trust that the suite was run.

Frequently asked questions

Does it change the application code?

No. The prompt keeps production code unchanged, so the parse behaviour the agent noticed becomes a strict xfail test with the reason in its id and a line in the pull request. Fixing it is a separate, small change once the team agrees it is a bug.

How does it test a Textual app without a terminal?

Textual's App.run_test() runs the app headless and returns a Pilot that can press keys, click and wait with pause(). The test must be async, which is why pytest-asyncio is added with asyncio_mode = "auto" so no per-test marker is needed.

Why add a workflow when the task is tests?

The repository has no .github/workflows/ directory, so a pull request with a passing suite would have no check to show it. The workflow installs Poetry and runs pytest on the oldest and newest Python the project declares, and later pull requests inherit it.

Will the compressed-file tests be slow or flaky?

No. They write a few hundred bytes with the standard library's gzip and bz2 modules into tmp_path, open them through LogFile, and check can_tail and the decoded lines. Nothing is downloaded and nothing depends on timing except the exit_event case, which sets the event before calling open.

More Textual jobs

The same job on other stacks

Keep reading