All jobs

Stream Chat Responses in a Streamlit App

Yes. An agent switches your Streamlit chatbot to streamed replies rendered with st.write_stream, keeps the AppTest suite green with a mocked stream, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the chat is wired before it changes a call. In an app shaped like streamlit/llm-examples, Chatbot.py keeps the history in st.session_state["messages"], replays it with st.chat_message(role).write(content), reads the key from a sidebar text_input with key chatbot_api_key, and on each st.chat_input calls client.chat.completions.create(model="gpt-3.5-turbo", messages=...), then writes response.choices[0].message.content in one go. pages/5_Chat_with_user_feedback.py makes the same non-streaming call and appends the whole reply before st.write. pages/2_Chat_with_search.py is the one place that already streams, by passing streaming=True to ChatOpenAI and a StreamlitCallbackHandler, so the app's users already see partial output on one page and not on the others.

requirements.txt pins streamlit>=1.28, openai>=1.2, langchain==0.0.350 and langchain-community==0.0.3, and .github/workflows/app-testing.yml runs streamlit/streamlit-app-action with ruff: true and pytest-args on Chatbot.py. app_test.py is the test the change must keep green: it loads the app with AppTest.from_file("Chatbot.py"), patches openai.resources.chat.Completions.create to return a ChatCompletion built by create_chat_completion, sets the key input and the chat input, and asserts that at.chat_message[2].markdown[0].value equals the joke. A streamed call returns an iterator of chunks, not a ChatCompletion, so the fixture has to change with the code.

The agent passes stream=True, wraps the result in a generator that yields chunk.choices[0].delta.content or "", and inside with st.chat_message("assistant"): calls st.write_stream(generator), which renders the tokens as they arrive and returns the full string; that string is what goes into st.session_state.messages, so the replay loop on the next rerun is unchanged. It applies the same shape to pages/5_Chat_with_user_feedback.py, keeping the feedback widget keyed on the message count, and raises the streamlit floor in requirements.txt to a release that has st.write_stream, which the pin at >=1.28 does not guarantee.

In app_test.py the agent adds a create_chat_completion_chunks helper that builds ChatCompletionChunk objects from openai.types.chat with a ChoiceDelta per word and a final chunk with finish_reason stop, makes the patched create return an iterator over them, and keeps the existing assertion on the assembled markdown. It runs ruff check . with the repository's .ruff.toml line length of 105 and pytest -v, the two things the workflow runs, and keeps both outputs.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Make this Streamlit chatbot stream its replies and open a pull request. Read Chatbot.py, pages/5_Chat_with_user_feedback.py, pages/2_Chat_with_search.py, app_test.py, requirements.txt, .ruff.toml and .github/workflows/app-testing.yml first. In Chatbot.py and the feedback page, call client.chat.completions.create with stream=True, wrap the chunks in a generator that yields each delta's content, render it inside st.chat_message with st.write_stream, and store the string it returns in st.session_state.messages so the history replay is unchanged. Raise the streamlit floor in requirements.txt to a release that has st.write_stream and say which one in the pull request. Update app_test.py so the patched Completions.create returns an iterator of ChatCompletionChunk objects ending with finish_reason stop, and keep the existing assertion on the assembled assistant message. Run ruff check . and pytest -v and paste the output in the pull request. Do not change the LangChain pages or the sidebar.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request with a few lines changed in Chatbot.py and pages/5_Chat_with_user_feedback.py, a new chunk helper and an updated patch in app_test.py, and one line in requirements.txt. The description shows the generator, names the Streamlit release the floor now points at, and pastes the ruff check and pytest -v output from the sandbox.

app-testing.yml runs the same ruff and pytest steps on the pull request through streamlit-app-action, with the results summarised on the check, so the mocked stream is exercised there as well.

A reviewer runs streamlit run Chatbot.py with a real key once and watches a reply arrive word by word, then sends a second message to confirm the first reply is still in the history. They check that the feedback page still records feedback for the streamed reply and that nothing on the LangChain pages changed. Then they merge.

Run it on every chat page that writes replies in one go

A team with several Streamlit apps sends one POST /o/{org}/sessions per repository with this prompt and the repository in resources. Each run reads its own pages and test file, so an app that uses a different client gets the generator written for that client's chunk shape, and each opens its own pull request.

Give the key a webhook_url and Rush POSTs run.completed when each turn ends, with session_url to read the pull request from. Send a request_id with each call so a retried request returns the run it already started.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

Streaming is a small code change and a larger testing change: the mocked response has to become a mocked stream, and the assertion on the assembled message has to keep passing. In a Rush sandbox the agent runs the app's own AppTest suite and ruff before pushing, and the output stays in the run's record beside the pull request, so the reviewer sees the suite passing on the new fixture rather than a note saying it should.

Doing it by hand means reading the chunk types, rewriting the fixture and then checking two pages by eye. The agent does that in one pass, and the same request works unchanged on the next Streamlit app whose replies still arrive all at once.

Frequently asked questions

Will the history replay change?

No. st.write_stream returns the full text once the generator is exhausted, and that string is appended to st.session_state.messages exactly as the non-streaming reply was, so the loop at the top of the page renders it the same way on the next rerun.

Why does requirements.txt change?

Because streamlit>=1.28 allows releases that do not have st.write_stream. The agent raises the floor to the first release that does and names it in the pull request, so pip install -r requirements.txt cannot produce an app that fails at the new call.

What does the test mock now?

An iterator of ChatCompletionChunk objects, one ChoiceDelta per word and a final chunk with finish_reason set to stop, returned by the same patched openai.resources.chat.Completions.create. The assertion still compares at.chat_message[2].markdown[0].value to the joke.

Does it touch the LangChain pages?

No. pages/2_Chat_with_search.py already streams through StreamlitCallbackHandler, and the quickstart and prompt-template pages are left alone; the prompt says so.

More Streamlit jobs

The same job on other stacks

More jobs like this: Hand off a long task

Keep reading