# Review Every Pull Request to a Streamlit App

Yes. An agent runs ruff and AppTest on each pull request to your Streamlit app, runs every page headless with the pinned dependencies, and comments on changed lines.

[Run this on Rush](https://cloud.getrush.ai/run/review-pull-requests/streamlit/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads the checks the repository already runs so its review adds to them instead of repeating them. In an app laid out like streamlit/llm-examples, `.github/workflows/app-testing.yml` uses `streamlit/streamlit-app-action@v0.0.3` with `app-path: Chatbot.py`, `ruff: true` and `pytest-args`, which lints, runs `app_test.py` and smoke-runs every page; `.pre-commit-config.yaml` runs `ruff` and `ruff-format` at `v0.1.14` with `check-yaml`, `end-of-file-fixer` and `trailing-whitespace`; `.ruff.toml` sets `line-length = 105`. `app_test.py` holds two tests on `streamlit.testing.v1.AppTest`: one runs `Chatbot.py`, sets the chat input, expects the `Please add your OpenAI API key to continue.` info, then patches `openai.resources.chat.Completions.create` and checks the assistant message; the other runs `pages/3_Langchain_Quickstart.py` with `langchain.llms.OpenAI.__call__` patched. `requirements.txt` pins `langchain==0.0.350` and `langchain-community==0.0.3` while leaving `streamlit>=1.28`, `openai>=1.2` and `anthropic>=0.3.0` open, which is the first thing a review of any dependency change has to know.

On each pull request it installs `requirements.txt` and `requirements-dev.txt` fresh, runs `ruff check` and `ruff format --check` at the pre-commit's pinned version, runs `pytest app_test.py`, and then runs every file under `pages/` and the root script through `AppTest.from_file(...).run()` and reads `at.exception`, which is what the action's smoke test does but with the diff in front of it. A pull request that bumps `langchain` past 0.0.350 breaks the `from langchain.chat_models import ChatOpenAI` and `from langchain.llms import OpenAI` imports in `pages/2_Chat_with_search.py`, `3_Langchain_Quickstart.py` and `4_Langchain_PromptTemplate.py`, and the smoke run shows the `ImportError` on the exact page; the comment lands on the `requirements.txt` line and names the `langchain-openai` and `langchain-community` imports that replace them. A pull request that changes a widget key, such as `chatbot_api_key` in `Chatbot.py`, gets a comment if `app_test.py` still looks the key up by its old name.

The rest of the review is about what Streamlit apps get wrong. Each page keeps the API key in a `st.text_input` with `type="password"` and a `key`, so it lives in `st.session_state` for the session only; a change that writes that value to disk, puts it in a log line, in `st.cache_data` arguments or in a query parameter gets a comment on that line. A page that calls a provider before checking the key, where the existing pages `st.info` and `st.stop()`, gets a comment with the pattern the other pages use. `pages/5_Chat_with_user_feedback.py` reads `TRUBRICS_EMAIL` and `TRUBRICS_PASSWORD` from `st.secrets`, so a change that moves either into code is flagged. The agent also reads the provider calls the way the vendors' current SDKs expect them: `pages/1_File_Q&A.py` builds a prompt with `anthropic.HUMAN_PROMPT` and calls `client.completions.create` with `model="claude-v1"`, the legacy text completions API and a retired model, and `pages/4_Langchain_PromptTemplate.py` names `text-davinci-003`; a pull request that touches either file gets a comment that the Messages API and a current model are the replacement, and one that touches neither does not, because the review is of the change.

It also checks the things Streamlit decides by file name and order: a new page under `pages/` without the numeric prefix changes the sidebar order, a page whose `st.set_page_config` is not the first Streamlit call raises at run time, and a `st.session_state` key read before it is set fails on first load, which the smoke run catches. Each comment quotes the line, the test or ruff output behind it, and the fix; the review ends with a summary of which pages ran clean under `AppTest`, the ruff result, and the versions `pip` resolved for the open-ended pins. The run is read-only: it edits nothing and pushes nothing.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Review the open pull request to this Streamlit app. Read .github/workflows/app-testing.yml, .pre-commit-config.yaml, .ruff.toml, requirements.txt, requirements-dev.txt and app_test.py first. Install the requirements fresh, run ruff check and ruff format --check at the pre-commit's version, run pytest app_test.py, then run the root script and every file under pages/ with AppTest.from_file(...).run() and report any page whose at.exception is set, with the traceback. Comment only on changed lines: a dependency bump that breaks an import a page uses, a widget key that app_test.py still looks up by its old name, an API key from st.text_input that is written to disk, logged, cached or put in a query parameter, a provider call made before the key check and st.stop(), a secret moved out of st.secrets, a new page without the numeric prefix, st.set_page_config not first, a session_state key read before it is set, and a provider call on a retired API or model when the change touches that call. Quote the line and the ruff, pytest or AppTest output behind each comment and say the fix. End with a summary listing each page's AppTest result, the ruff result and the versions pip resolved. Do not edit any file.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app/pull/123"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

The result is a review on the pull request: one comment per changed line that matters, each quoting the ruff finding, the `pytest` failure or the `AppTest` traceback behind it and the fix, plus a summary comment listing the root script and each page with its `AppTest` result, the ruff and format results, and the versions `pip` resolved for `streamlit`, `openai`, `anthropic` and the pinned LangChain pair.

Nothing in the repository changes: the run is read-only, so it cannot push a fix or open a branch. The transcript holds the full install log, the ruff output and every `AppTest` run, so a comment the author questions can be traced to the exact output it came from.

Before merging, the reviewer reads the comments, decides which findings belong in this pull request and which, such as moving `pages/1_File_Q&A.py` off the legacy completions API, belong in a follow-up, and whether an open-ended pin that resolved to a new major version should be pinned.

## Run it on every pull request opened

In the console, add an automation to the Project with this prompt and a GitHub trigger on `pull_request.opened` and `pull_request.synchronize`. The trigger adds the event after the prompt, and only work by an owner, member or collaborator fires it; pull requests from forks never do. Each push starts a read-only run on the pull request's branch, and a pull request whose pages all run clean and whose diff raises nothing ends with the summary comment alone.

A team that wants the review in its own tooling sends this body from its server when its GitHub webhook handler sees a pull request, with the pull request URL in `resources` and a `request_id` that is a UUID v5 of the head SHA, so a redelivered webhook returns the run already started. Set a webhook URL on the API key and Rush POSTs `run.completed` when the review is posted.

## Why run it on Rush?

A Streamlit page fails at run time, not at import: a renamed widget key, a provider SDK that moved a class, or a `session_state` read on first load all look fine in a diff and break when the page runs. `AppTest` runs the page headless with the real dependencies resolved from the open-ended pins, and on Rush that happens in a sandbox per pull request, with no API key and no network call to a provider, so a review can say which page broke and why rather than guess.

Each comment is backed by the install log, the ruff output and the `AppTest` traceback in the transcript, so the review is something a small team can rely on for every change rather than a spot check. Many pull requests are reviewed at once, each in its own sandbox, and the judgement is the same whether the author is a person or a dependency bot.

## Frequently asked questions

### Does it need an OpenAI or Anthropic key to run the pages?

No. Every page stops with `Please add your ... API key to continue.` until a key is typed, and `app_test.py` patches the provider calls, so the smoke runs and the test suite need no key and make no provider call. The review reads the provider code; it does not execute it.

### What does it say about a dependency bump?

It installs the bumped set and runs every page. If `langchain` moves past 0.0.350, the three LangChain pages fail to import `langchain.chat_models` and `langchain.llms`, and the comment on the `requirements.txt` line quotes the traceback and names the `langchain-openai` and `langchain-community` modules the pages would need. A bump that runs clean gets a line in the summary and no comment.

### Does it comment on the retired Anthropic completions API every time?

Only when the pull request changes `pages/1_File_Q&A.py` or the call itself. The review is of the change, so a pull request about another page is not told about a problem it did not touch; the summary names it once as a pre-existing item a maintainer may want to fix separately.

### Can it push a fix?

No. The run is read-only: it installs, lints, runs the pages, posts comments and cannot commit. A fix the author wants is a separate run started with the repository in `resources` and the review's findings in its prompt, which opens its own pull request.

## More Streamlit jobs

- [Upgrade LangChain in a Streamlit App](https://getrush.ai/run/upgrade-langchain/streamlit/)
- [Upgrade Python Dependencies in a Streamlit App](https://getrush.ai/run/upgrade-python-dependencies/streamlit/)
- [Write Tests for a Streamlit App](https://getrush.ai/run/write-apptest-tests/streamlit/)

## The same job on other stacks

- [Review Pull Requests Automatically in a SvelteKit App](https://getrush.ai/run/review-pull-requests/sveltekit/)
- [Review Pull Requests Automatically in a Symfony App](https://getrush.ai/run/review-pull-requests/symfony/)
- [Review Pull Requests to Temporal Go Workflows for Determinism](https://getrush.ai/run/review-pull-requests/temporal/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
