What the agent does
The agent starts by reading where the help text comes from and where documentation lives now. In a Typer project like fastapi/fastapi-cli, src/fastapi_cli/cli.py creates app = typer.Typer(rich_markup_mode="rich", ...) with -h added as a help flag, a callback that owns --version and --verbose, and two commands, dev and run, whose parameters are Annotated[..., typer.Option(help=...)] with help strings full of Rich markup such as [blue]127.0.0.1[/blue] and [bold]FastAPI[/bold]. The same file wraps app.add_typer(fastapi_cloud_cli) and a second add_typer for fastapi_new in try/except ImportError, so the set of commands depends on which optional packages are installed. README.md describes fastapi dev and fastapi run in prose and lists no options, and there is no docs/ directory. pyproject.toml requires typer >= 0.26.1, uv.lock resolves it, and scripts/ holds short shell wrappers such as lint.sh and test.sh.
Typer can write the reference itself. Its own command line has a utils docs subcommand that imports an app and emits Markdown, so the agent adds scripts/docs.sh in the style of the other scripts, running typer fastapi_cli.cli utils docs --name fastapi --output docs/cli-reference.md. The --name matters: without it the page is titled CLI and the usage lines start with no program name, where a user expects to see fastapi dev. The output has one section per command with its usage line, arguments and options, and each option carries what Typer knows about it, for example the port's [env var: PORT; default: 8000]. Nothing is hand-written, so a new option appears in the reference the next time the script runs.
Two properties of this codebase shape the result, and the agent handles both in the open. First, the Rich markup does not survive as Markdown: the generator renders [blue] and [bold] as inline HTML span elements with style attributes. That displays on GitHub but is noisy in a diff, so the pull request shows a sample and offers the alternative of post-processing the spans away in the script. Second, the optional sub-apps. The repository's pre-commit workflow installs with uv sync --locked --extra standard, which makes fastapi_cloud_cli importable, while a plain uv sync does not, and the two environments generate different files. The script therefore states which environment it documents and the check runs in that same one, and the reference says at the top which extras it covers.
The check is what keeps the file true. The agent adds a local hook to .pre-commit-config.yaml, next to the existing uv run hooks for ruff, mypy and ty, that regenerates the reference and fails if git diff --exit-code docs/cli-reference.md finds a difference, so a pull request that changes an option without regenerating is stopped by the workflow already in .github/workflows/pre-commit.yml. It links the reference from the README under the two command sections, runs the script twice in the sandbox to confirm the output is stable from one run to the next, runs bash scripts/lint.sh and bash scripts/test.sh, and then changes one help string on purpose to watch the hook fail before reverting it.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Generate a CLI reference for this Typer app and keep it from drifting. Read pyproject.toml, src/fastapi_cli/cli.py, README.md, the scripts directory, .pre-commit-config.yaml and .github/workflows/pre-commit.yml and test.yml first. Add scripts/docs.sh, in the style of the other scripts, that runs Typer's own generator: typer fastapi_cli.cli utils docs --name fastapi --output docs/cli-reference.md. Note that cli.py adds optional sub-apps only when their packages import, and that the pre-commit workflow installs with --extra standard: decide which environment the reference documents, generate it there, say so at the top of the file, and make the check run in the same environment. Look at how Rich markup in the help strings is rendered in the Markdown and show a sample in the pull request with the option of stripping it. Add a local pre-commit hook beside the existing uv run hooks that regenerates the file and fails on git diff --exit-code. Link the reference from the README. Run the script twice to confirm stable output, run scripts/lint.sh and scripts/test.sh, then change one help string to confirm the hook fails and revert it. Open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
A pull request with scripts/docs.sh, the generated docs/cli-reference.md, the new hook in .pre-commit-config.yaml and a link in README.md. The description names the environment the file was generated in and why, shows a few lines of the output including one option with its default and environment variable, shows how one marked-up help string came out as HTML, and includes the hook's failing output from the deliberate change. The repository's own pre-commit and test workflows run on the pull request, and the first of them now includes the drift check.
A reviewer reads the generated file as a user would, which is a quick way to spot help strings that made sense in a terminal and read badly on a page. They decide whether the inline HTML is acceptable or the script should strip it. They check that Typer's --install-completion and --show-completion lines belong in a public reference, since the generator includes them, and that the choice about optional commands matches what the project wants to promise. After that the file needs no reviewing again: it changes only when the options do, in the same pull request, and the hook enforces it.
Run it on every Linear issue labelled cli-docs
A team with several Typer tools can make this a standing request. In the console, add an automation on each tool's Project with a linear trigger on linear.issue.labeled, the label cli-docs, and this prompt with the module path and command name changed for that tool. When a person in your Rush org labels an issue like "our README is missing half the flags", the issue is added after the prompt and the run opens a pull request in that Project's repository. The automation's runner needs a Linear connection, which is made in the console under Settings → Connections.
From a script, send the body to POST /o/{org}/sessions once per repository with a request_id that is a UUID v5 of the repository's full name, so running the script again returns the runs already started. Once the hook is merged the reference maintains itself, so this is a job each repository needs only once.
The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.
Why run it on Rush?
The generator's output depends on what is installed when it runs, and here that changes which commands exist. The agent has a sandbox where it can build both environments from uv.lock, generate in each, and compare, and the run keeps both outputs and every command. The pull request can then say which environment the reference describes and show the difference, which a file produced on one developer's machine with unknown extras installed cannot.
The check is also proved, not assumed. The run changes a help string, records the hook failing with the diff, and reverts the change, all in the run's record. A reviewer does not have to trust that the drift check works or try it themselves; they can read the failure it produced.
Frequently asked questions
Does the reference include the default values and environment variables?
Yes. Typer writes each option with its type and, where they exist, its default and environment variable, so the port option reads [env var: PORT; default: 8000]. Boolean pairs are shown in both forms, such as --reload / --no-reload, with the default named.
What happens to Rich markup in help strings?
The generator converts it to inline HTML spans with colour and weight styles. GitHub renders that, but the source is harder to read. The pull request shows an example and can add a step to the script that removes the spans, leaving plain text.
Will optional subcommands appear in the docs?
Only if their packages are importable when the script runs. This app adds its cloud and project-creation commands inside try/except ImportError, so the reference lists them when generated with the standard extra and omits them otherwise. The file states which it is.
How does the check know the docs are stale?
A pre-commit hook reruns the generator and then runs git diff --exit-code on the reference. If the regenerated file differs from the committed one, the hook exits non-zero and the workflow fails, with the diff showing which option changed.