What the agent does
The agent reads each page as Streamlit runs it, top to bottom on every interaction, and lists what that repeats. In streamlit/llm-examples, Chatbot.py builds OpenAI(api_key=openai_api_key) inside the if prompt := st.chat_input() block, so a new client object is made for every message; pages/5_Chat_with_user_feedback.py does the same. pages/1_File_Q&A.py reads the uploaded file with uploaded_file.read().decode() and sends the whole article to Anthropic each time the question input changes, so editing one word of the question re-sends the document. pages/2_Chat_with_search.py constructs ChatOpenAI, DuckDuckGoSearchRun and initialize_agent on every message. requirements.txt pins nothing below streamlit>=1.28, openai>=1.2 and anthropic>=0.3.0, and app_test.py already tests two pages with streamlit.testing.v1.AppTest, patching openai.resources.chat.Completions.create.
It separates the two kinds of thing Streamlit caches. Objects that should exist once per process and are not data, the API clients and the search tool, go behind st.cache_resource: a get_openai_client(api_key) in a new utils.py returns one client per key, and get_search_tool() returns one DuckDuckGoSearchRun. Values that are the result of a computation go behind st.cache_data, which pickles and copies them: in File Q&A, an answer_question(article, question, _api_key) function whose cache key is the article text and the question, with the key argument named with a leading underscore so Streamlit leaves it out of the hash and a key rotation does not throw away answers, plus a ttl so a changed model does not serve old text for ever. The chat completions in Chatbot.py are left uncached on purpose: the request includes the whole conversation, so no two calls share a key, and caching would only hide the cost.
The pages change by a few lines each. The client construction moves out of the chat block into the cached helper; File Q&A calls the cached function and shows the answer, and gains a small Clear cached answers button that calls st.cache_data.clear() in the sidebar, since a cached wrong answer is the first thing a user will ask about. The agent keeps the API-key-missing checks exactly where they are, before any cached call, so a missing key still shows the st.info message the tests assert on. Line length stays within the 105 that .ruff.toml sets, and the pre-commit hooks in .pre-commit-config.yaml run ruff and ruff-format.
Then it proves both the speed-up and that nothing broke. app_test.py gains a case that drives File Q&A twice with the same upload and question through AppTest with anthropic.Client patched, and asserts the patched method was called once; and a case that two runs of Chatbot.py with the same key return the same client object from the cache. The existing test_Chatbot keeps passing because the patch is on the class method, which the cached client still calls. pytest runs them, ruff check . and ruff format --check . pass, and .github/workflows/app-testing.yml runs the same through streamlit-app-action with ruff: true.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add caching to this Streamlit app. Read requirements.txt, .ruff.toml, app_test.py and every page under the root and pages/ first, and list what each rerun rebuilds or re-sends. Put objects that should exist once per process (API clients, tools) behind st.cache_resource in a shared module, and put repeatable computations (the File Q&A answer, keyed on the article text and question) behind st.cache_data with a ttl and with the API key passed as an underscore-prefixed argument so it is not part of the cache key. Do not cache chat completions whose input is the whole conversation; explain why in the pull request. Keep every missing-API-key check in place before any cached call. Add a sidebar button that clears st.cache_data. Extend app_test.py with AppTest cases that show the patched provider call happens once for a repeated question and that the cached client is reused. Run pytest, ruff check and ruff format --check, paste the output in the pull request and open it.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
A pull request that adds utils.py, changes Chatbot.py, pages/1_File_Q&A.py, pages/2_Chat_with_search.py and pages/5_Chat_with_user_feedback.py by a handful of lines each, and extends app_test.py. The description lists each page with what it used to rebuild and what is now cached, quotes the pytest run with the new cases, and shows ruff passing.
A reviewer checks the cache keys more than the code: that the File Q&A key is the article text and the question and nothing else, that the ttl matches how often the model or prompt changes, and that no st.cache_data function returns an object that should not be copied. They also decide whether the uncached chat path should stay uncached, which the pull request argues for.
The new tests are the proof the behaviour holds: a second identical question reaches the provider zero times, and the existing assertions about the missing-key message still pass.
Run it on every Linear issue labelled performance
The next slow page gets the same treatment from an issue. In the console, add an automation on the Project that holds the repository with a linear trigger on linear.issue.labeled and the label performance, and a prompt that says: read the page the issue names, cache what each rerun rebuilds with the right decorator, keep the API-key checks, add an AppTest case, run pytest and ruff, and open a pull request. The issue is placed after the prompt as the event and fills in nothing, so the page comes from the issue text. The trigger needs a Linear connection for the automation's runner and fires only when the label is applied by someone in your Rush org.
The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.
Why run it on Rush?
Caching in Streamlit is easy to add and easy to get wrong: a resource cached as data gets copied on every read, a key cached with the secret in it empties when the secret rotates, and a cached chat answer looks fine until two users share one. The agent runs each page through AppTest in its own sandbox, with the provider calls patched, so the pull request can say how many times the provider was reached before and after, which a code read cannot.
Those runs, with their output, stay in the run's transcript at GET /o/{org}/sessions/{id}/messages, and every changed file is listed with its diff at GET /o/{org}/p/{project}/sessions/{id}/changes. The reviewer reads the test that proves the repeated question was served from cache, instead of trusting a sentence that says so.
Frequently asked questions
When is it st.cache_resource and when st.cache_data?
st.cache_resource is for things that should exist once and be shared, such as an API client or a tool object; it returns the same object every time. st.cache_data is for results, such as an answer string; it stores a pickled copy and hands each caller a fresh one. Putting a client behind cache_data copies it on every read, and putting a result behind cache_resource lets one caller mutate what another sees.
Why is the API key not part of the cache key?
Because the answer to a question about an article does not depend on which key paid for it, and keys rotate. The agent names the parameter with a leading underscore, which tells Streamlit to leave it out of the hash, so a new key reuses the cached answers. For the client itself the key is the argument, since a client is only valid for one key.
Why not cache the chatbot replies?
Each call sends the whole conversation, so no two calls have the same input and a cache would never hit. It would also store user conversations in process memory. The agent caches the client only and says so in the pull request.
Do the existing tests still pass?
Yes. app_test.py patches openai.resources.chat.Completions.create on the class, and the cached client calls that method, so the patch still applies. The missing-key checks stay before any cached call, so the st.info message the tests assert on is unchanged.
More Streamlit jobs
The same job on other stacks
- Add Caching to a Vapor App (Vapor)
- Add Caching to a Flask App (Flask)
- Add Response Caching to a Hono API (Hono)