All jobs

Write CLI Tests for a Typer App

Yes. An agent writes pytest tests for a Typer CLI with CliRunner and tmp_path, pins exit codes, options and JSON output as they really behave, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the whole CLI before it writes a test, which in a repository like Conor-McLeod/filetype-stats is one file. main.py holds a FileTypeAnalyser class that walks a directory with os.walk under a Rich progress bar, groups files by extension, renders a Rich table or tree, and can export JSON; below it, app = typer.Typer() and a single @app.command() named analyze with a DIRECTORY argument defaulting to . and the options --max-ext-length, --include-hidden, --sort (an Enum of size, count, name), --top, --show-examples, --tree and --export-json. pyproject.toml requires Python 3.13 and pins nothing for tests: typer, rich and marimo are the only dependencies, there is no [build-system] table and no dev group, and uv.lock resolves typer 0.16.1 with click 8.2.1. There are no tests and no workflow.

Then it runs the CLI to learn what the tests must pin, because the README and the code disagree. With one command registered, Typer makes it the root, so --help prints Usage: analyze [OPTIONS] [DIRECTORY], and the README's filetype-stats analyze treats analyze as the directory and exits 1 with Error: Directory 'analyze' does not exist. The missing-directory and not-a-directory errors go to stdout through the Rich console, not stderr, with exit code 1; a bad --sort value is Click's own error, exit code 2. Because there is no build backend, uv run filetype-stats cannot find a console script, so the tests import main directly with pythonpath = ["."] under [tool.pytest.ini_options].

A fixture directory built with tmp_path shows the rest. Hidden directories are skipped unless --include-hidden is set, but a hidden file in a visited directory is always counted under a (hidden file) row; b.PY is folded into py; a file whose extension is longer than --max-ext-length lands in (no extension); --top 2 prints two rows while the TOTAL row still sums every file; and --export-json writes a document whose keys are the plain labels with the Rich markup stripped, with size_bytes, size_human, count and up to five example_files per type plus a timestamp. Each of these becomes one test, with a comment where the behaviour looks unintended, such as the hidden-file row.

The tests go in tests/test_cli.py using typer.testing.CliRunner, which runs app in-process and returns exit_code and output; since every message is printed to stdout, assertions read result.output rather than result.stderr. The agent adds pytest under a dev dependency group in pyproject.toml so uv sync installs it, runs uv run pytest on the Python 3.13 that .python-version names, and puts the output in the pull request along with the one-line README mismatch.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Write CLI tests for this Typer app. Read pyproject.toml, uv.lock, .python-version, README.md and main.py first, then run the CLI with uv run python main.py --help and against a scratch directory to see what it actually does: how a single @app.command is invoked, where error messages go and with which exit codes, how hidden files and directories are counted with and without --include-hidden, what --max-ext-length, --sort, --top, --tree and --show-examples change, and what --export-json writes. Add pytest to a dev dependency group and pythonpath = [\".\"] under tool.pytest.ini_options, then write tests/test_cli.py with typer.testing.CliRunner and tmp_path fixtures that pin each of those behaviours, including the exit code and message for a missing directory, a file path and an invalid --sort, and the JSON document's keys and counts. Pin current behaviour and mark anything that looks unintended with a comment; do not change main.py. Run uv run pytest on the pinned Python and open a pull request that pastes the output and notes any mismatch between the README and the CLI.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds tests/test_cli.py, adds a dev dependency group with pytest and a pytest section to pyproject.toml, updates uv.lock for the new group, and leaves main.py untouched. The description pastes the uv run pytest output from the sandbox on Python 3.13 and lists each behaviour the tests pin.

Two findings sit at the top of the description. The README's filetype-stats analyze does not work: with one command, Typer takes analyze as the directory argument and exits 1, so either the README drops the word or the command gains a callback so it becomes a subcommand; the tests pin the current form and say so. And a hidden file inside a visited directory is counted under (hidden file) even without --include-hidden, while hidden directories are skipped, which the test records as current behaviour with a comment rather than as the specification.

Before merging, a reviewer decides which of the pinned behaviours are bugs to fix, in which case the assertion changes in the same pull request as main.py, and whether to add a [build-system] table so the filetype-stats script works under uv run, which would let one more test invoke the installed entry point.

Run it on every pull request that adds an option

An automation in the Project with a github trigger on pull_request.opened and pull_request.synchronize runs this prompt against a pull request that touches main.py; the trigger adds the pull request to the prompt, and the run clones that branch, exercises the new option with CliRunner, adds its test and reports on the pull request. Pull requests from forks never start a run.

A team with several Python CLIs instead sends one POST /o/{org}/sessions per repository from a script, with this prompt and the repository in resources, and a request_id that is a UUID v5 of the repository's full name so a retried call returns the run already started. Each run opens at most one pull request.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

Tests for a CLI are tests of behaviour nobody wrote down: which stream an error goes to, which exit code a bad option gets, what a hidden file does, what the JSON looks like. Guessing any of these produces a test that fails on day one for the wrong reason. The agent runs the CLI in its own sandbox on the pinned Python, with scratch directories it builds and throws away, and pins what it observed, so the pull request describes the tool as it is and flags the two places where that differs from the README.

Every command and its output stays on the run, so a reviewer reads the pytest log and the CLI transcripts without installing Python 3.13 or the dependencies. Several CLIs can get first tests at once, each in its own sandbox, and a behaviour that looks like a bug is reported in the pull request, not silently turned into an assertion that blesses it.

Frequently asked questions

Why do the tests not pass the word analyze?

Because with a single @app.command() and no callback, Typer makes that command the root of the app: --help prints Usage: analyze [OPTIONS] [DIRECTORY], and the first positional argument is the directory. Running filetype-stats analyze as the README says exits 1 with Error: Directory 'analyze' does not exist. The tests pin the working form and the pull request quotes the README line that needs changing.

Why assert on result.output and not stderr?

Every message in main.py, including the errors for a missing or non-directory path, goes through a Rich Console() that writes to stdout, and CliRunner captures it there; result.stderr is empty for those cases. Only Click's own usage errors, such as an invalid --sort, are different, and the test for that checks the exit code of 2 and the message in result.output.

How are hidden files tested?

A tmp_path fixture gets a hidden directory with a file inside and a hidden file at the top level. Without --include-hidden, the directory's file is absent from the counts and the top-level hidden file appears as a (hidden file) row; with the flag, both are counted. The test pins both outcomes and carries a comment that the first looks unintended, so the reviewer can decide.

Why does the pull request change pyproject.toml?

The project has no test dependency and no build backend, so pytest is not installed and uv run filetype-stats cannot find a script to spawn. The agent adds pytest under a dev dependency group, which uv sync installs by default, and pythonpath = ["."] so the tests import main. It does not add a [build-system], since that changes how the tool installs; the pull request names it as an option.

Are Rich tables stable enough to assert on?

Under CliRunner there is no terminal, so Rich emits no colour and uses a fixed width, and the rows came out as plain text with box-drawing characters in the sandbox. The tests assert on the extension labels and counts within a row rather than on whole lines, and on the JSON export for exact numbers, so a change in column padding does not break them.

More Typer jobs

The same job on other stacks

More jobs like this: Write the missing tests

Keep reading