# Write Tests for a Streamlit App

Yes. An agent writes pytest tests with Streamlit's AppTest for each page of your app, runs them headless in its sandbox and opens a pull request.

[Run this on Rush](https://cloud.getrush.ai/run/write-apptest-tests/streamlit/), in a Project that has your repository.

## What the agent does

Streamlit ships its own test harness, `streamlit.testing.v1.AppTest`, which runs a script without a browser and lets a test set widget values and read what the page rendered. The agent starts by listing what there is to test. In streamlit/llm-examples that is `Chatbot.py` plus five pages under `pages/`: `1_File_Q&A.py` on the Anthropic client, `2_Chat_with_search.py` with a LangChain agent and `DuckDuckGoSearchRun`, `3_Langchain_Quickstart.py`, `4_Langchain_PromptTemplate.py`, and `5_Chat_with_user_feedback.py` with `streamlit_feedback` and `trubrics`. `requirements.txt` asks for `streamlit>=1.28`, `openai>=1.2`, `langchain==0.0.350` and `anthropic>=0.3.0`.

It reads the one test file, `app_test.py`, and finds two tests: `test_Chatbot` and `test_Langchain_Quickstart`. Both replace the model call inside the process, with `@patch("openai.resources.chat.Completions.create")` and `@patch("langchain.llms.OpenAI.__call__")`. That skips the client code entirely, so a broken request, a wrong model name or a changed response shape would still pass. The other four pages have no test at all.

So the agent stubs at the HTTP boundary instead, as the prompt asks. Each page builds its client from the key a user types (`OpenAI(api_key=openai_api_key)`, `anthropic.Client(api_key=...)`), so the tests set the client's base URL through the environment variables the SDKs read and run a small local HTTP server in the test that answers `/v1/chat/completions` and the Anthropic completions route with recorded JSON. The real `openai` and `anthropic` clients then parse a real response. For every page it writes two cases: no API key, where the page must show the `st.info` message such as "Please add your OpenAI API key to continue." and stop, and a round trip, where the test types a key, sets a `chat_input` or `text_input`, runs the script and checks the reply rendered in `at.chat_message` or `at.info`.

It runs `pytest` with the `requirements-dev.txt` tools installed, the same command the repository's `app-testing.yml` reaches through `streamlit/streamlit-app-action`, and keeps going until every page passes. Where AppTest cannot drive a widget, the agent says which one and tests what it can around it rather than skipping the page.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Write tests with streamlit.testing.v1.AppTest for each page of this Streamlit app: Chatbot.py and every file under pages/. Stub only the model APIs at the HTTP boundary, with a local server the OpenAI and Anthropic clients reach through their base URL environment variables, never by patching client methods. For each page cover a missing API key (the st.info message and stop) and a full round trip from input to rendered answer. Replace the existing @patch-based tests in app_test.py with the same approach. If AppTest cannot drive a widget, say which and test around it. Run pytest until everything passes and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

The pull request adds a `tests/` directory (or extends `app_test.py`, whichever the repository's CI action collects) with one test module per page, a `conftest.py` holding the fake HTTP server as a pytest fixture, and recorded JSON responses under `tests/testdata/`. The two existing `@patch` tests are rewritten to use the same fixture, so the whole file tests one way. No page script under `pages/` changes.

The description lists every page with its tests and what each asserts, notes any widget AppTest could not drive and how that page is covered instead, and records the exact `streamlit`, `openai` and `langchain` versions the run installed. The session's record has each `pytest` run with its output, including the run where a page failed because its client called an endpoint the fixture had not recorded.

Before merging, a reviewer checks the recorded responses look like the provider's real ones, that no real API key appears anywhere in the fixtures, and that the repository's `App testing` workflow passes on the pull request.

## Run it on every new page added

Multipage Streamlit apps grow a page at a time, and the new one is usually the untested one. A GitHub trigger on `pull_request.opened` can run this request for any pull request that adds a file under `pages/`, with a prompt that names the new page, so it arrives with its own AppTest tests and fixture responses before review.

To cover a whole set of Streamlit apps across repositories, send one `POST /o/{org}/sessions` per repository from a script, each with its repository in `resources`. A key runs four sessions at once by default and answers `429` past that, so queue the rest and start each when a webhook reports one finished.

## Why run it on Rush?

Testing an LLM app on a laptop tends to end in one of two places: tests that hit the real API with a real key, or tests that patch the client away until they prove nothing. In a Rush sandbox the agent can run a fake provider on a local port, point the real SDKs at it and run every page headless, with no key of yours involved and nothing left running on your machine.

The pull request carries its own evidence. The repository's CI runs the new tests on GitHub, and the session's record shows every `pytest` run in the sandbox with its output, so a reviewer can see the stub is at the network edge and the pages really parse responses. The same request works on any number of Streamlit repositories at once, each in its own sandbox.

## Frequently asked questions

### Why not just patch the OpenAI client like the existing tests?

Patching `Completions.create` replaces the code most likely to break: the request your page builds and the response it parses. A local server at the HTTP boundary keeps the real client in the test, so a wrong model name or a changed response shape fails the test, and the test still runs offline.

### Does AppTest need a browser?

No. AppTest runs the script in the test process and exposes the rendered elements as objects (`at.chat_input`, `at.info`, `at.sidebar.text_input`), so the suite runs anywhere pytest runs, including GitHub Actions, with no browser installed.

### What about pages that call a search tool, like DuckDuckGo?

The same rule applies: the test answers the tool's HTTP requests from the local server with a recorded result. If a library gives no way to change its host, the agent says so in the pull request and patches as close to the network call as the library allows, naming exactly where.

### Will it upgrade Streamlit or LangChain?

No. It installs what `requirements.txt` resolves to and tests that. If a pinned version cannot run under AppTest, it says which and why, and leaves the upgrade to a separate change.