What the agent does
Streamlit ships its own test harness, streamlit.testing.v1.AppTest, which runs a script without a browser and lets a test set widget values and read what the page rendered. The agent starts by listing what there is to test. In streamlit/llm-examples that is Chatbot.py plus five pages under pages/: 1_File_Q&A.py on the Anthropic client, 2_Chat_with_search.py with a LangChain agent and DuckDuckGoSearchRun, 3_Langchain_Quickstart.py, 4_Langchain_PromptTemplate.py, and 5_Chat_with_user_feedback.py with streamlit_feedback and trubrics. requirements.txt asks for streamlit>=1.28, openai>=1.2, langchain==0.0.350 and anthropic>=0.3.0.
It reads the one test file, app_test.py, and finds two tests: test_Chatbot and test_Langchain_Quickstart. Both replace the model call inside the process, with @patch("openai.resources.chat.Completions.create") and @patch("langchain.llms.OpenAI.__call__"). That skips the client code entirely, so a broken request, a wrong model name or a changed response shape would still pass. The other four pages have no test at all.
So the agent stubs at the HTTP boundary instead, as the prompt asks. Each page builds its client from the key a user types (OpenAI(api_key=openai_api_key), anthropic.Client(api_key=...)), so the tests set the client's base URL through the environment variables the SDKs read and run a small local HTTP server in the test that answers /v1/chat/completions and the Anthropic completions route with recorded JSON. The real openai and anthropic clients then parse a real response. For every page it writes two cases: no API key, where the page must show the st.info message such as "Please add your OpenAI API key to continue." and stop, and a round trip, where the test types a key, sets a chat_input or text_input, runs the script and checks the reply rendered in at.chat_message or at.info.
It runs pytest with the requirements-dev.txt tools installed, the same command the repository's app-testing.yml reaches through streamlit/streamlit-app-action, and keeps going until every page passes. Where AppTest cannot drive a widget, the agent says which one and tests what it can around it rather than skipping the page.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Write tests with streamlit.testing.v1.AppTest for each page of this Streamlit app: Chatbot.py and every file under pages/. Stub only the model APIs at the HTTP boundary, with a local server the OpenAI and Anthropic clients reach through their base URL environment variables, never by patching client methods. For each page cover a missing API key (the st.info message and stop) and a full round trip from input to rendered answer. Replace the existing @patch-based tests in app_test.py with the same approach. If AppTest cannot drive a widget, say which and test around it. Run pytest until everything passes and open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request adds a tests/ directory (or extends app_test.py, whichever the repository's CI action collects) with one test module per page, a conftest.py holding the fake HTTP server as a pytest fixture, and recorded JSON responses under tests/testdata/. The two existing @patch tests are rewritten to use the same fixture, so the whole file tests one way. No page script under pages/ changes.
The description lists every page with its tests and what each asserts, notes any widget AppTest could not drive and how that page is covered instead, and records the exact streamlit, openai and langchain versions the run installed. The session's record has each pytest run with its output, including the run where a page failed because its client called an endpoint the fixture had not recorded.
Before merging, a reviewer checks the recorded responses look like the provider's real ones, that no real API key appears anywhere in the fixtures, and that the repository's App testing workflow passes on the pull request.
Run it on every new page added
Multipage Streamlit apps grow a page at a time, and the new one is usually the untested one. A GitHub trigger on pull_request.opened can run this request for any pull request that adds a file under pages/, with a prompt that names the new page, so it arrives with its own AppTest tests and fixture responses before review.
To cover a whole set of Streamlit apps across repositories, send one POST /o/{org}/sessions per repository from a script, each with its repository in resources. A key runs four sessions at once by default and answers 429 past that, so queue the rest and start each when a webhook reports one finished.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Testing an LLM app on a laptop tends to end in one of two places: tests that hit the real API with a real key, or tests that patch the client away until they prove nothing. In a Rush sandbox the agent can run a fake provider on a local port, point the real SDKs at it and run every page headless, with no key of yours involved and nothing left running on your machine.
The pull request carries its own evidence. The repository's CI runs the new tests on GitHub, and the session's record shows every pytest run in the sandbox with its output, so a reviewer can see the stub is at the network edge and the pages really parse responses. The same request works on any number of Streamlit repositories at once, each in its own sandbox.
Frequently asked questions
Why not just patch the OpenAI client like the existing tests?
Patching Completions.create replaces the code most likely to break: the request your page builds and the response it parses. A local server at the HTTP boundary keeps the real client in the test, so a wrong model name or a changed response shape fails the test, and the test still runs offline.
Does AppTest need a browser?
No. AppTest runs the script in the test process and exposes the rendered elements as objects (at.chat_input, at.info, at.sidebar.text_input), so the suite runs anywhere pytest runs, including GitHub Actions, with no browser installed.
What about pages that call a search tool, like DuckDuckGo?
The same rule applies: the test answers the tool's HTTP requests from the local server with a recorded result. If a library gives no way to change its host, the agent says so in the pull request and patches as close to the network call as the library allows, naming exactly where.
Will it upgrade Streamlit or LangChain?
No. It installs what requirements.txt resolves to and tests that. If a pinned version cannot run under AppTest, it says which and why, and leaves the upgrade to a separate change.