# Write CLI Tests for a Typer App

Yes. An agent writes pytest tests for a Typer CLI with CliRunner and tmp_path, pins exit codes, options and JSON output as they really behave, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/write-cli-tests/typer/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads the whole CLI before it writes a test, which in a repository like Conor-McLeod/filetype-stats is one file. `main.py` holds a `FileTypeAnalyser` class that walks a directory with `os.walk` under a Rich progress bar, groups files by extension, renders a Rich table or tree, and can export JSON; below it, `app = typer.Typer()` and a single `@app.command()` named `analyze` with a `DIRECTORY` argument defaulting to `.` and the options `--max-ext-length`, `--include-hidden`, `--sort` (an `Enum` of `size`, `count`, `name`), `--top`, `--show-examples`, `--tree` and `--export-json`. `pyproject.toml` requires Python 3.13 and pins nothing for tests: `typer`, `rich` and `marimo` are the only dependencies, there is no `[build-system]` table and no dev group, and `uv.lock` resolves `typer` 0.16.1 with `click` 8.2.1. There are no tests and no workflow.

Then it runs the CLI to learn what the tests must pin, because the README and the code disagree. With one command registered, Typer makes it the root, so `--help` prints `Usage: analyze [OPTIONS] [DIRECTORY]`, and the README's `filetype-stats analyze` treats `analyze` as the directory and exits 1 with `Error: Directory 'analyze' does not exist.` The missing-directory and not-a-directory errors go to stdout through the Rich console, not stderr, with exit code 1; a bad `--sort` value is Click's own error, exit code 2. Because there is no build backend, `uv run filetype-stats` cannot find a console script, so the tests import `main` directly with `pythonpath = ["."]` under `[tool.pytest.ini_options]`.

A fixture directory built with `tmp_path` shows the rest. Hidden directories are skipped unless `--include-hidden` is set, but a hidden file in a visited directory is always counted under a `(hidden file)` row; `b.PY` is folded into `py`; a file whose extension is longer than `--max-ext-length` lands in `(no extension)`; `--top 2` prints two rows while the `TOTAL` row still sums every file; and `--export-json` writes a document whose keys are the plain labels with the Rich markup stripped, with `size_bytes`, `size_human`, `count` and up to five `example_files` per type plus a `timestamp`. Each of these becomes one test, with a comment where the behaviour looks unintended, such as the hidden-file row.

The tests go in `tests/test_cli.py` using `typer.testing.CliRunner`, which runs `app` in-process and returns `exit_code` and `output`; since every message is printed to stdout, assertions read `result.output` rather than `result.stderr`. The agent adds `pytest` under a `dev` dependency group in `pyproject.toml` so `uv sync` installs it, runs `uv run pytest` on the Python 3.13 that `.python-version` names, and puts the output in the pull request along with the one-line README mismatch.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Write CLI tests for this Typer app. Read pyproject.toml, uv.lock, .python-version, README.md and main.py first, then run the CLI with uv run python main.py --help and against a scratch directory to see what it actually does: how a single @app.command is invoked, where error messages go and with which exit codes, how hidden files and directories are counted with and without --include-hidden, what --max-ext-length, --sort, --top, --tree and --show-examples change, and what --export-json writes. Add pytest to a dev dependency group and pythonpath = [\".\"] under tool.pytest.ini_options, then write tests/test_cli.py with typer.testing.CliRunner and tmp_path fixtures that pin each of those behaviours, including the exit code and message for a missing directory, a file path and an invalid --sort, and the JSON document's keys and counts. Pin current behaviour and mark anything that looks unintended with a comment; do not change main.py. Run uv run pytest on the pinned Python and open a pull request that pastes the output and notes any mismatch between the README and the CLI.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that adds `tests/test_cli.py`, adds a `dev` dependency group with `pytest` and a pytest section to `pyproject.toml`, updates `uv.lock` for the new group, and leaves `main.py` untouched. The description pastes the `uv run pytest` output from the sandbox on Python 3.13 and lists each behaviour the tests pin.

Two findings sit at the top of the description. The README's `filetype-stats analyze` does not work: with one command, Typer takes `analyze` as the directory argument and exits 1, so either the README drops the word or the command gains a callback so it becomes a subcommand; the tests pin the current form and say so. And a hidden file inside a visited directory is counted under `(hidden file)` even without `--include-hidden`, while hidden directories are skipped, which the test records as current behaviour with a comment rather than as the specification.

Before merging, a reviewer decides which of the pinned behaviours are bugs to fix, in which case the assertion changes in the same pull request as `main.py`, and whether to add a `[build-system]` table so the `filetype-stats` script works under `uv run`, which would let one more test invoke the installed entry point.

## Run it on every pull request that adds an option

An automation in the Project with a `github` trigger on `pull_request.opened` and `pull_request.synchronize` runs this prompt against a pull request that touches `main.py`; the trigger adds the pull request to the prompt, and the run clones that branch, exercises the new option with `CliRunner`, adds its test and reports on the pull request. Pull requests from forks never start a run.

A team with several Python CLIs instead sends one `POST /o/{org}/sessions` per repository from a script, with this prompt and the repository in `resources`, and a `request_id` that is a UUID v5 of the repository's full name so a retried call returns the run already started. Each run opens at most one pull request.

## Why run it on Rush?

Tests for a CLI are tests of behaviour nobody wrote down: which stream an error goes to, which exit code a bad option gets, what a hidden file does, what the JSON looks like. Guessing any of these produces a test that fails on day one for the wrong reason. The agent runs the CLI in its own sandbox on the pinned Python, with scratch directories it builds and throws away, and pins what it observed, so the pull request describes the tool as it is and flags the two places where that differs from the README.

Every command and its output stays on the run, so a reviewer reads the `pytest` log and the CLI transcripts without installing Python 3.13 or the dependencies. Several CLIs can get first tests at once, each in its own sandbox, and a behaviour that looks like a bug is reported in the pull request, not silently turned into an assertion that blesses it.

## Frequently asked questions

### Why do the tests not pass the word analyze?

Because with a single `@app.command()` and no callback, Typer makes that command the root of the app: `--help` prints `Usage: analyze [OPTIONS] [DIRECTORY]`, and the first positional argument is the directory. Running `filetype-stats analyze` as the README says exits 1 with `Error: Directory 'analyze' does not exist.` The tests pin the working form and the pull request quotes the README line that needs changing.

### Why assert on result.output and not stderr?

Every message in `main.py`, including the errors for a missing or non-directory path, goes through a Rich `Console()` that writes to stdout, and `CliRunner` captures it there; `result.stderr` is empty for those cases. Only Click's own usage errors, such as an invalid `--sort`, are different, and the test for that checks the exit code of 2 and the message in `result.output`.

### How are hidden files tested?

A `tmp_path` fixture gets a hidden directory with a file inside and a hidden file at the top level. Without `--include-hidden`, the directory's file is absent from the counts and the top-level hidden file appears as a `(hidden file)` row; with the flag, both are counted. The test pins both outcomes and carries a comment that the first looks unintended, so the reviewer can decide.

### Why does the pull request change pyproject.toml?

The project has no test dependency and no build backend, so `pytest` is not installed and `uv run filetype-stats` cannot find a script to spawn. The agent adds `pytest` under a `dev` dependency group, which `uv sync` installs by default, and `pythonpath = ["."]` so the tests import `main`. It does not add a `[build-system]`, since that changes how the tool installs; the pull request names it as an option.

### Are Rich tables stable enough to assert on?

Under `CliRunner` there is no terminal, so Rich emits no colour and uses a fixed width, and the rows came out as plain text with box-drawing characters in the sandbox. The tests assert on the extension labels and counts within a row rather than on whole lines, and on the JSON export for exact numbers, so a change in column padding does not break them.

## More Typer jobs

- [Generate CLI Reference Docs for a Typer App](https://getrush.ai/run/add-cli-reference-docs/typer/)
- [Read Typer CLI Options From Environment Variables](https://getrush.ai/run/add-env-var-options/typer/)
- [Fix a Failing CI Run for a Typer CLI](https://getrush.ai/run/fix-failing-ci/typer/)

## The same job on other stacks

- [Write Tests for a Go CLI](https://getrush.ai/run/write-cli-tests/go-cli/)
- [Write Integration Tests for a Rust CLI](https://getrush.ai/run/write-cli-tests/rust-cli/)

## More jobs like this: Write the missing tests

- [Write Codeception Tests for a Yii2 App](https://getrush.ai/run/write-codeception-tests/yii/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
