All jobs

Review Every Pull Request to a Streamlit App

Yes. An agent runs ruff and AppTest on each pull request to your Streamlit app, runs every page headless with the pinned dependencies, and comments on changed lines.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the checks the repository already runs so its review adds to them instead of repeating them. In an app laid out like streamlit/llm-examples, .github/workflows/app-testing.yml uses streamlit/streamlit-app-action@v0.0.3 with app-path: Chatbot.py, ruff: true and pytest-args, which lints, runs app_test.py and smoke-runs every page; .pre-commit-config.yaml runs ruff and ruff-format at v0.1.14 with check-yaml, end-of-file-fixer and trailing-whitespace; .ruff.toml sets line-length = 105. app_test.py holds two tests on streamlit.testing.v1.AppTest: one runs Chatbot.py, sets the chat input, expects the Please add your OpenAI API key to continue. info, then patches openai.resources.chat.Completions.create and checks the assistant message; the other runs pages/3_Langchain_Quickstart.py with langchain.llms.OpenAI.__call__ patched. requirements.txt pins langchain==0.0.350 and langchain-community==0.0.3 while leaving streamlit>=1.28, openai>=1.2 and anthropic>=0.3.0 open, which is the first thing a review of any dependency change has to know.

On each pull request it installs requirements.txt and requirements-dev.txt fresh, runs ruff check and ruff format --check at the pre-commit's pinned version, runs pytest app_test.py, and then runs every file under pages/ and the root script through AppTest.from_file(...).run() and reads at.exception, which is what the action's smoke test does but with the diff in front of it. A pull request that bumps langchain past 0.0.350 breaks the from langchain.chat_models import ChatOpenAI and from langchain.llms import OpenAI imports in pages/2_Chat_with_search.py, 3_Langchain_Quickstart.py and 4_Langchain_PromptTemplate.py, and the smoke run shows the ImportError on the exact page; the comment lands on the requirements.txt line and names the langchain-openai and langchain-community imports that replace them. A pull request that changes a widget key, such as chatbot_api_key in Chatbot.py, gets a comment if app_test.py still looks the key up by its old name.

The rest of the review is about what Streamlit apps get wrong. Each page keeps the API key in a st.text_input with type="password" and a key, so it lives in st.session_state for the session only; a change that writes that value to disk, puts it in a log line, in st.cache_data arguments or in a query parameter gets a comment on that line. A page that calls a provider before checking the key, where the existing pages st.info and st.stop(), gets a comment with the pattern the other pages use. pages/5_Chat_with_user_feedback.py reads TRUBRICS_EMAIL and TRUBRICS_PASSWORD from st.secrets, so a change that moves either into code is flagged. The agent also reads the provider calls the way the vendors' current SDKs expect them: pages/1_File_Q&A.py builds a prompt with anthropic.HUMAN_PROMPT and calls client.completions.create with model="claude-v1", the legacy text completions API and a retired model, and pages/4_Langchain_PromptTemplate.py names text-davinci-003; a pull request that touches either file gets a comment that the Messages API and a current model are the replacement, and one that touches neither does not, because the review is of the change.

It also checks the things Streamlit decides by file name and order: a new page under pages/ without the numeric prefix changes the sidebar order, a page whose st.set_page_config is not the first Streamlit call raises at run time, and a st.session_state key read before it is set fails on first load, which the smoke run catches. Each comment quotes the line, the test or ruff output behind it, and the fix; the review ends with a summary of which pages ran clean under AppTest, the ruff result, and the versions pip resolved for the open-ended pins. The run is read-only: it edits nothing and pushes nothing.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Review the open pull request to this Streamlit app. Read .github/workflows/app-testing.yml, .pre-commit-config.yaml, .ruff.toml, requirements.txt, requirements-dev.txt and app_test.py first. Install the requirements fresh, run ruff check and ruff format --check at the pre-commit's version, run pytest app_test.py, then run the root script and every file under pages/ with AppTest.from_file(...).run() and report any page whose at.exception is set, with the traceback. Comment only on changed lines: a dependency bump that breaks an import a page uses, a widget key that app_test.py still looks up by its old name, an API key from st.text_input that is written to disk, logged, cached or put in a query parameter, a provider call made before the key check and st.stop(), a secret moved out of st.secrets, a new page without the numeric prefix, st.set_page_config not first, a session_state key read before it is set, and a provider call on a retired API or model when the change touches that call. Quote the line and the ruff, pytest or AppTest output behind each comment and say the fix. End with a summary listing each page's AppTest result, the ruff result and the versions pip resolved. Do not edit any file.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app/pull/123"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The result is a review on the pull request: one comment per changed line that matters, each quoting the ruff finding, the pytest failure or the AppTest traceback behind it and the fix, plus a summary comment listing the root script and each page with its AppTest result, the ruff and format results, and the versions pip resolved for streamlit, openai, anthropic and the pinned LangChain pair.

Nothing in the repository changes: the run is read-only, so it cannot push a fix or open a branch. The transcript holds the full install log, the ruff output and every AppTest run, so a comment the author questions can be traced to the exact output it came from.

Before merging, the reviewer reads the comments, decides which findings belong in this pull request and which, such as moving pages/1_File_Q&A.py off the legacy completions API, belong in a follow-up, and whether an open-ended pin that resolved to a new major version should be pinned.

Run it on every pull request opened

In the console, add an automation to the Project with this prompt and a GitHub trigger on pull_request.opened and pull_request.synchronize. The trigger adds the event after the prompt, and only work by an owner, member or collaborator fires it; pull requests from forks never do. Each push starts a read-only run on the pull request's branch, and a pull request whose pages all run clean and whose diff raises nothing ends with the summary comment alone.

A team that wants the review in its own tooling sends this body from its server when its GitHub webhook handler sees a pull request, with the pull request URL in resources and a request_id that is a UUID v5 of the head SHA, so a redelivered webhook returns the run already started. Set a webhook URL on the API key and Rush POSTs run.completed when the review is posted.

The use case this job follows, with its automation and trigger: Review every pull request.

Why run it on Rush?

A Streamlit page fails at run time, not at import: a renamed widget key, a provider SDK that moved a class, or a session_state read on first load all look fine in a diff and break when the page runs. AppTest runs the page headless with the real dependencies resolved from the open-ended pins, and on Rush that happens in a sandbox per pull request, with no API key and no network call to a provider, so a review can say which page broke and why rather than guess.

Each comment is backed by the install log, the ruff output and the AppTest traceback in the transcript, so the review is something a small team can rely on for every change rather than a spot check. Many pull requests are reviewed at once, each in its own sandbox, and the judgement is the same whether the author is a person or a dependency bot.

Frequently asked questions

Does it need an OpenAI or Anthropic key to run the pages?

No. Every page stops with Please add your ... API key to continue. until a key is typed, and app_test.py patches the provider calls, so the smoke runs and the test suite need no key and make no provider call. The review reads the provider code; it does not execute it.

What does it say about a dependency bump?

It installs the bumped set and runs every page. If langchain moves past 0.0.350, the three LangChain pages fail to import langchain.chat_models and langchain.llms, and the comment on the requirements.txt line quotes the traceback and names the langchain-openai and langchain-community modules the pages would need. A bump that runs clean gets a line in the summary and no comment.

Does it comment on the retired Anthropic completions API every time?

Only when the pull request changes pages/1_File_Q&A.py or the call itself. The review is of the change, so a pull request about another page is not told about a problem it did not touch; the summary names it once as a pre-existing item a maintainer may want to fix separately.

Can it push a fix?

No. The run is read-only: it installs, lints, runs the pages, posts comments and cannot commit. A fix the author wants is a separate run started with the repository in resources and the review's findings in its prompt, which opens its own pull request.

More Streamlit jobs

The same job on other stacks

Keep reading