All jobs

Review Pull Requests on a Typer CLI

Yes. For each pull request the agent runs the lint and test scripts on two dependency resolutions, diffs every command's --help against main and posts one review.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

For a command-line tool the options are the public API, and a diff of Python does not show how they changed. So the agent reviews the interface as well as the code. It starts with how the project checks itself. In a Typer CLI like fastapi/fastapi-cli, scripts/lint.sh runs mypy src tests (strict, per pyproject.toml), ty check src tests, ruff check and ruff format --check; scripts/test.sh is coverage run -m pytest tests, with --strict-config, --strict-markers and xfail_strict set in the pytest options; and .github/workflows/test.yml runs that script across three operating systems, five Python versions and two uv resolution modes before a last job demands coverage report --fail-under=100 over both src and tests. A separate workflow builds the sdist and runs the tests from inside it.

On the pull request's branch the agent repeats what it can of that on Linux. It runs uv sync --no-dev --group tests --extra standard and both scripts twice: once on the locked versions with the newest Python in the matrix, and once with UV_PYTHON=3.10 and UV_RESOLUTION=lowest-direct, which installs the minimum Typer the project declares, 0.26.1. The second pass is the one that catches a pull request using a Typer feature newer than the floor. It combines the coverage of the two passes and lists every line the diff adds that no test executes, since a single uncovered line fails the project's CI. It also runs python -m build --sdist, unpacks the archive and runs the tests there, which catches a new test asset that the build does not ship.

Then the interface. The agent checks out the base branch beside the pull request, runs python -m fastapi_cli --help, dev --help and run --help on each, and puts the textual diff at the top of its review. A renamed option, a changed default, a new required argument or a different envvar (the port option reads PORT) is called out as a breaking change for anyone with the old flag in a script or a Dockerfile. In src/fastapi_cli/cli.py the dev and run commands declare nearly the same options and both hand them to _run, so an option added to one and not the other gets a question, unless it is like workers, which only run has, or reload_dir, which only dev has.

The reading of the Python follows the file's own patterns. User-facing failures print through the toolkit and end with raise typer.Exit(code=1); a new error path that lets an exception escape would show users a traceback, and the agent checks the exit code a test asserts for it. Optional integrations are imported inside try and except ImportError, as uvicorn, fastapi_cloud_cli and fastapi_new are, so a new hard import of an optional package is a finding. A new third-party import must be covered by dependencies in pyproject.toml; pydantic, which config.py imports, is not listed there and arrives with FastAPI, so the review asks about each new import of that kind instead of assuming. Tests never start a server: they patch uvicorn.run and assert on the keyword arguments it received, and a new option should appear in such an assertion. The agent posts one review and pushes nothing.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Review this pull request to our Typer command-line tool. Read pyproject.toml, uv.lock, scripts/lint.sh, scripts/test.sh, .github/workflows/test.yml and test-redistribute.yml, tests/conftest.py and src/ first. On the pull request's branch run uv sync --no-dev --group tests --extra standard, bash scripts/lint.sh and bash scripts/test.sh, first on the locked versions with the newest Python in the matrix and then with UV_PYTHON=3.10 and UV_RESOLUTION=lowest-direct. Combine coverage from both passes and list every line added by the diff that no test executes. Build the sdist, unpack it and run the tests inside it. Check out the base branch in a second directory, run --help for the app and for each command on both branches, and include the diff of the output. Flag any renamed or removed option, changed default, new required argument or changed environment variable as a breaking change. Then read the diff: errors shown to users must exit through typer.Exit with a non-zero code and have a test asserting it; optional packages must be imported inside try/except ImportError; a new third-party import must be declared in pyproject.toml; an option added to dev or run but not the other needs a reason; a new option needs a test asserting it reaches uvicorn.run. Post one review on the pull request with the command results and the help diff first, then findings by file and line. Do not push commits.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app/pull/42"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

One review on the pull request. It opens with the help diff: for each command, the lines of --help that the pull request adds, removes or changes, and a sentence on whether an existing invocation would still work. Next is a results block for the two passes, with lint, tests and coverage each marked pass or fail, the uncovered added lines by file and number, and the sdist result. Findings from reading the diff follow, each with its file and line.

A pull request that changes only internals gets a short review that says the help output is identical and both passes are green. One that fails only on lowest-direct gets the traceback and the two honest ways out: raise the minimum in pyproject.toml or avoid the newer call. The complete output of every command is in the session's transcript at GET /o/{org}/sessions/{id}/messages.

The review comments and leaves the decision to a maintainer. Whether a breaking flag change is acceptable, what the deprecation period should be and how it is worded in the release notes are choices for the people who answer the issues afterwards. The agent also cannot run the Windows and macOS cells, and says so, leaving them to the matrix on the pull request.

Run it on every pull request

An automation in the Project that holds the repository, with this prompt, read-only switched on and a github trigger on pull_request.opened and pull_request.synchronize, starts a review when a pull request opens and again on each push. The trigger places the pull request after the prompt and fills in nothing. With read-only on, the run can read the repository and comment and cannot push.

The trigger fires for owners, members and collaborators, not for forks or bots, and an open source CLI gets most of its pull requests from forks. A fork's branch is not in the repository itself, so for those a maintainer pushes the contributor's commits to a branch there and opens a pull request from it, and that pull request starts the review. A team that wants reviews only on request uses pull_request.labeled with a label in labels instead.

The use case this job follows, with its automation and trigger: Review every pull request.

Why run it on Rush?

The checks that matter for a CLI are tedious to run by hand: two environments with different dependency sets, a coverage merge, an sdist build, and two checkouts to compare help output. A maintainer with the branch open in an editor runs the tests once, on whatever is installed. The agent builds each environment fresh in its sandbox for every pull request, and several pull requests are reviewed at once, each in a separate sandbox.

Because the commands and their output are kept with the run, a finding can be argued with. A contributor who doubts the lowest-direct failure can read the resolved package list and the traceback; one who thinks a help change is harmless can see the exact before and after text and point at it.

Frequently asked questions

Why review the --help output at all?

Because it is what users and their scripts depend on, and Typer generates it from function signatures. A refactor that renames a parameter also renames its option, and nothing in the Python diff says so. The text diff of --help between the two branches shows it in one line.

What does the lowest-direct pass add?

It installs the oldest version of each direct dependency the project claims to support, here typer >= 0.26.1. Code that relies on something newer still passes on the locked versions and fails for a user whose environment has the old one. The repository's CI has cells for this; the agent runs the Linux one before the review, so the author hears about it in the review and not from a red matrix later.

Will it fix the problems it finds?

Not in this setup: the prompt forbids pushing and the automation is read-only. To have fixes made, start a separate run without read-only, with the pull request's URL in resources, and ask for commits on its branch.

Does it need our PyPI token or any secrets?

No. It installs from the public index, builds the sdist locally and publishes nothing. The release workflows in the repository are read for context and never run.

More Typer jobs

The same job on other stacks

Keep reading