What the agent does
The agent reads the app and its tests before changing either, because in this repository the test drives the page the way a user would. In an app laid out like streamlit/llm-examples, Chatbot.py builds a sidebar with one st.text_input for the OpenAI key under the key chatbot_api_key, keeps the conversation in st.session_state["messages"], and on each st.chat_input calls client.chat.completions.create with the model name gpt-3.5-turbo written into the call. app_test.py loads that file with AppTest.from_file("Chatbot.py"), patches openai.resources.chat.Completions.create, sets the key through at.text_input(key="chatbot_api_key"), types into at.chat_input[0] and asserts the markdown of the chat messages. .github/workflows/app-testing.yml runs that suite and ruff through streamlit/streamlit-app-action@v0.0.3 on Python 3.11.
The change is small and placed where the file already keeps its choices. A module-level list of model names goes at the top of Chatbot.py, seeded with the name the file uses today so the default behaviour does not move, and the sidebar gains st.selectbox with the key chatbot_model right under the key input. The create call takes model=st.session_state.chatbot_model instead of the literal. Switching models mid-conversation keeps the messages, since the history is a list of plain role and content dicts that any chat model accepts; the pull request says so rather than adding a reset button nobody asked for.
The test is extended in the same style. test_Chatbot already builds a fake ChatCompletion through create_chat_completion; the agent adds a step that picks a different entry with at.selectbox(key="chatbot_model"), sends a message, and asserts the patched create was called with that model keyword, which the mock records in its call arguments. The existing assertions on the info message and the chat bubbles stay as they are. ruff with the repository's line-length = 105 from .ruff.toml runs over the result, and the ruff-format hook in .pre-commit-config.yaml is what the formatting is checked against.
In its sandbox the agent installs requirements.txt and requirements-dev.txt, runs pytest app_test.py to see both tests pass with the new step, and runs ruff check and ruff format --check on the two files. The other pages under pages/, which call LangChain with their own model names, are left alone: the prompt scopes the change to Chatbot.py, and the pull request notes that pages/2_Chat_with_search.py names its model separately.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add a model selector to the Streamlit chatbot in Chatbot.py. Read Chatbot.py, app_test.py, requirements.txt, .ruff.toml and .github/workflows/app-testing.yml first. Put the model names in a module-level list with the model the file uses today first, add an st.selectbox in the sidebar with the key chatbot_model, and pass the chosen name to the chat completion call. Keep the conversation when the model changes. Extend test_Chatbot in app_test.py to pick another model and assert the patched create call received it as the model keyword. Do not touch the pages directory. Run pytest app_test.py, ruff check and ruff format --check, and quote the results. Open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request with two files changed, Chatbot.py and app_test.py: the list and the selectbox, the one-word change in the completion call, and the added assertions. The description quotes the pytest run with both tests passing and the two ruff commands reporting nothing, and names the model the selector defaults to, which is the one the app called before.
The App testing workflow runs the same suite on the pull request through streamlit-app-action, with ruff: true, so the checks tab shows the result a second time, and the junit summary from pmeier/pytest-results-action lists the test by name.
Before merging, a reviewer edits the list to the models the team actually wants to offer, since the agent added only what the prompt named, and decides whether the choice should persist across page loads.
Run it on every Linear issue labelled agent
Add an automation in the Project with this prompt and a linear trigger on linear.issue.labeled, filtered to the agent label. When a teammate files "let users pick the model in the demo" and labels it, the trigger puts the issue after the prompt and starts a run in the app's repository; the pull request links back to the issue. Only label changes made by a person in your Rush org fire it, never changes made by an app integration.
From a script, the same body goes to POST /o/{org}/sessions with the issue's URL in resources beside the repository, so the agent reads the issue through your Linear connection. Send a request_id, a UUID, so a retried call returns the run it already started rather than opening a second pull request.
The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.
Why run it on Rush?
A selectbox is ten minutes of work and an hour of making sure the test still drives the page, because AppTest addresses widgets by key and a new widget shifts nothing only if it is keyed. The agent reads how app_test.py finds each input and extends it the same way, then runs the suite and the linters in its own sandbox before anyone sees the diff.
The test output stays with the run and the pull request quotes it, so a reviewer reads a passing suite with the new assertion rather than a screenshot of a sidebar. A team keeping several Streamlit demos sends the same request to each, in parallel, one sandbox and one pull request per repository.
Frequently asked questions
Does switching models clear the conversation?
No. The history in st.session_state["messages"] is a list of role and content pairs that is sent as is, so the next reply simply comes from the newly chosen model. A reset button is a separate change if the team wants one.
How does the test know which model was used?
test_Chatbot already patches openai.resources.chat.Completions.create. A patched function records every call, so the added assertion reads the model keyword from the last call after the selectbox was set with at.selectbox(key="chatbot_model").
Why not change the LangChain pages too?
They name their models in their own files through ChatOpenAI and langchain.llms.OpenAI, and requirements.txt pins langchain==0.0.350, so a shared selector would mean touching an older API surface. The prompt scopes the change to Chatbot.py; the other pages are a separate request.
Where do the model names come from?
From the prompt and the file: the list starts with the name Chatbot.py already calls, and the team adds the others. The agent does not invent model names, and the pull request says the list is meant to be edited.