What the agent does
The agent starts with how the chat page makes its model call, because the token numbers come from that response and nowhere else. In an app like streamlit/llm-examples, Chatbot.py reads an API key from a sidebar st.text_input with type="password", keeps the conversation in st.session_state["messages"], replays it with st.chat_message, and on each st.chat_input calls client.chat.completions.create(model="gpt-3.5-turbo", messages=...) from the openai package pinned at >=1.2 in requirements.txt, then reads only response.choices[0].message.content. The same response carries a usage object with prompt_tokens, completion_tokens and total_tokens that the page discards. pages/5_Chat_with_user_feedback.py makes the identical call and discards the same object; pages/1_File_Q&A.py uses the anthropic client's legacy completions.create, whose response has no usage field, and the two LangChain pages route through langchain 0.0.350, so the agent scopes the change to the two pages that call OpenAI directly and says so.
It adds a small helper module, usage.py, with two functions: one that takes a completion response and returns the three counts as a dict, tolerating a missing usage so a mocked response without it does not crash the page, and one that renders a caption. In Chatbot.py each assistant message stored in st.session_state.messages gains a usage entry, the replay loop shows st.caption under an assistant message with its prompt and completion tokens when the entry is present, and a running total is kept in st.session_state["total_tokens"] and shown in the sidebar under the API key field with st.metric, so a long conversation's growing prompt cost is visible as it grows. The agent makes the same change in pages/5_Chat_with_user_feedback.py, which already stores the assistant reply before rendering it, and leaves the Anthropic and LangChain pages untouched.
Cost in currency is left out on purpose. The page does not know which price applies to the account's model, and requirements.txt pins no model list, so the agent shows tokens, which the API states, and puts a one-line comment where a per-model price table would plug in. It keeps the message dicts compatible with what the API accepts by stripping the usage key before sending st.session_state.messages back to client.chat.completions.create, since the API rejects unknown fields on a message, and that stripping is the one line most likely to be forgotten by hand.
The proof is the repository's own test. app_test.py already drives Chatbot.py with streamlit.testing.v1.AppTest and patches openai.resources.chat.Completions.create with a hand-built ChatCompletion; the agent extends the create_chat_completion helper to attach a CompletionUsage with known counts, asserts that the caption under the assistant reply shows them and that the sidebar metric shows the running total after a second turn, and adds a case with no usage to show the page still renders. It runs pytest app_test.py and ruff check . with the repository's .ruff.toml line length of 105, the same two checks .github/workflows/app-testing.yml runs through streamlit-app-action, and quotes both outputs in the pull request.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Show token usage in this Streamlit chat app. Read Chatbot.py, every file under pages/, app_test.py, requirements.txt and .github/workflows first, and change only the pages that call the OpenAI chat completions API directly. Store each assistant reply's prompt_tokens and completion_tokens from the response's usage field with the message in session state, show them as a caption under that reply, keep a running total in session state shown as a metric in the sidebar, and strip the usage data from messages before they are sent back to the API. Do not add currency costs. Extend the existing AppTest in app_test.py so the mocked completion carries usage and assert the caption and the total, including a case with no usage field. Run pytest and ruff as the CI workflow does and open a pull request that links the issue.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request on its own branch adding usage.py, editing Chatbot.py and pages/5_Chat_with_user_feedback.py, and extending app_test.py. requirements.txt is unchanged because the openai package already returns the usage object, and the Anthropic and LangChain pages are untouched, with the description saying why each was left alone.
The description shows the pytest output with the new assertions passing and the ruff check result, and a screenshot from the transcript at GET /o/{org}/sessions/{id}/messages of the chat page after two turns with captions under each reply and the sidebar total. It names the line that strips usage before the API call as the one a reviewer should read.
Before merging, a reviewer checks that the stored message shape still round-trips through client.chat.completions.create, which the test covers through the mock, and decides whether to add a price table later. A run whose tests fail ends needs_review with the output rather than a pull request claiming success.
Run it on every Linear issue labelled agent
In the console, an automation with a linear trigger on linear.issue.labeled and the label agent starts one run per labelled issue in the Project that holds the app, with the issue text as its context. An issue that says "show how many tokens each answer used" is enough: the trigger appends the issue after the prompt, and the agent finds the chat.completions.create calls on its own. Changes made by an app integration never fire the trigger, so the agent's own comments on the issue start nothing.
From your own server, the same run is one POST /o/{org}/sessions per issue with this body, the repository in resources and the issue's https://linear.app/<workspace>/issue/<ID> URL there too, read through your Linear connection into the prompt. Send a request_id, a UUID v5 of the issue id, so a redelivered event does not start a second run.
The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.
Why run it on Rush?
The feature is a dozen lines, and the mistakes are small and specific: sending a message dict with an extra usage key back to the API and getting a 400, crashing the page on a response that lacks the field, or counting the same tokens twice when st.session_state replays. The agent reads how the page stores and replays messages and how app_test.py mocks the API, then changes both together so the test proves the behaviour instead of the screenshot alone.
On Rush the tests and the app run in the agent's own sandbox on the Python version the CI workflow uses, and the pytest output and the screenshot stay in the transcript next to the diff. A Linear label becomes a pull request with its evidence attached, and several labelled issues on the same app run at once, each on its own branch.
Frequently asked questions
Why tokens and not dollars?
Because the API reports tokens and nothing in the repository states a price. A currency figure needs a per-model price table that goes out of date, so the agent shows the counts the response contains and leaves a marked place for a price lookup. Ask for it in the prompt with your prices and it adds the table.
Does it work with the Anthropic page too?
Not in this pull request. pages/1_File_Q&A.py uses the Anthropic client's legacy completions.create, whose response has no usage object. Moving that page to the Messages API, which does report input_tokens and output_tokens, is a separate change, and the description lists it as a follow-up rather than mixing two migrations.
Will the stored usage break the API call?
No. The one line most likely to be forgotten is stripping the usage key from each message before st.session_state.messages is sent back to client.chat.completions.create, and the agent adds it and tests it: the mock records what it was called with, and the test asserts no message carries usage.
How is the running total kept across reruns?
In st.session_state["total_tokens"], updated once when a response arrives rather than recomputed from the replay loop, so Streamlit's rerun on every interaction does not double count. The test sends two prompts and asserts the metric equals the sum of both mocked usages.