# Show Token Usage per Reply in a Streamlit Chat App

Yes. An agent adds a per-reply token count and a running total to a Streamlit chatbot from the usage the API already returns, with an AppTest.

[Run this on Rush](https://cloud.getrush.ai/run/add-token-usage-display/streamlit/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent starts with how the chat page makes its model call, because the token numbers come from that response and nowhere else. In an app like streamlit/llm-examples, `Chatbot.py` reads an API key from a sidebar `st.text_input` with `type="password"`, keeps the conversation in `st.session_state["messages"]`, replays it with `st.chat_message`, and on each `st.chat_input` calls `client.chat.completions.create(model="gpt-3.5-turbo", messages=...)` from the `openai` package pinned at `>=1.2` in `requirements.txt`, then reads only `response.choices[0].message.content`. The same response carries a `usage` object with `prompt_tokens`, `completion_tokens` and `total_tokens` that the page discards. `pages/5_Chat_with_user_feedback.py` makes the identical call and discards the same object; `pages/1_File_Q&A.py` uses the `anthropic` client's legacy `completions.create`, whose response has no usage field, and the two LangChain pages route through `langchain` 0.0.350, so the agent scopes the change to the two pages that call OpenAI directly and says so.

It adds a small helper module, `usage.py`, with two functions: one that takes a completion response and returns the three counts as a dict, tolerating a missing `usage` so a mocked response without it does not crash the page, and one that renders a caption. In `Chatbot.py` each assistant message stored in `st.session_state.messages` gains a `usage` entry, the replay loop shows `st.caption` under an assistant message with its prompt and completion tokens when the entry is present, and a running total is kept in `st.session_state["total_tokens"]` and shown in the sidebar under the API key field with `st.metric`, so a long conversation's growing prompt cost is visible as it grows. The agent makes the same change in `pages/5_Chat_with_user_feedback.py`, which already stores the assistant reply before rendering it, and leaves the Anthropic and LangChain pages untouched.

Cost in currency is left out on purpose. The page does not know which price applies to the account's model, and `requirements.txt` pins no model list, so the agent shows tokens, which the API states, and puts a one-line comment where a per-model price table would plug in. It keeps the message dicts compatible with what the API accepts by stripping the `usage` key before sending `st.session_state.messages` back to `client.chat.completions.create`, since the API rejects unknown fields on a message, and that stripping is the one line most likely to be forgotten by hand.

The proof is the repository's own test. `app_test.py` already drives `Chatbot.py` with `streamlit.testing.v1.AppTest` and patches `openai.resources.chat.Completions.create` with a hand-built `ChatCompletion`; the agent extends the `create_chat_completion` helper to attach a `CompletionUsage` with known counts, asserts that the caption under the assistant reply shows them and that the sidebar metric shows the running total after a second turn, and adds a case with no `usage` to show the page still renders. It runs `pytest app_test.py` and `ruff check .` with the repository's `.ruff.toml` line length of 105, the same two checks `.github/workflows/app-testing.yml` runs through `streamlit-app-action`, and quotes both outputs in the pull request.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Show token usage in this Streamlit chat app. Read Chatbot.py, every file under pages/, app_test.py, requirements.txt and .github/workflows first, and change only the pages that call the OpenAI chat completions API directly. Store each assistant reply's prompt_tokens and completion_tokens from the response's usage field with the message in session state, show them as a caption under that reply, keep a running total in session state shown as a metric in the sidebar, and strip the usage data from messages before they are sent back to the API. Do not add currency costs. Extend the existing AppTest in app_test.py so the mocked completion carries usage and assert the caption and the total, including a case with no usage field. Run pytest and ruff as the CI workflow does and open a pull request that links the issue.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request on its own branch adding `usage.py`, editing `Chatbot.py` and `pages/5_Chat_with_user_feedback.py`, and extending `app_test.py`. `requirements.txt` is unchanged because the `openai` package already returns the usage object, and the Anthropic and LangChain pages are untouched, with the description saying why each was left alone.

The description shows the `pytest` output with the new assertions passing and the `ruff check` result, and a screenshot from the transcript at `GET /o/{org}/sessions/{id}/messages` of the chat page after two turns with captions under each reply and the sidebar total. It names the line that strips `usage` before the API call as the one a reviewer should read.

Before merging, a reviewer checks that the stored message shape still round-trips through `client.chat.completions.create`, which the test covers through the mock, and decides whether to add a price table later. A run whose tests fail ends `needs_review` with the output rather than a pull request claiming success.

## Run it on every Linear issue labelled agent

In the console, an automation with a `linear` trigger on `linear.issue.labeled` and the label `agent` starts one run per labelled issue in the Project that holds the app, with the issue text as its context. An issue that says "show how many tokens each answer used" is enough: the trigger appends the issue after the prompt, and the agent finds the `chat.completions.create` calls on its own. Changes made by an app integration never fire the trigger, so the agent's own comments on the issue start nothing.

From your own server, the same run is one `POST /o/{org}/sessions` per issue with this body, the repository in `resources` and the issue's `https://linear.app/<workspace>/issue/<ID>` URL there too, read through your Linear connection into the prompt. Send a `request_id`, a UUID v5 of the issue id, so a redelivered event does not start a second run.

## Why run it on Rush?

The feature is a dozen lines, and the mistakes are small and specific: sending a message dict with an extra `usage` key back to the API and getting a 400, crashing the page on a response that lacks the field, or counting the same tokens twice when `st.session_state` replays. The agent reads how the page stores and replays messages and how `app_test.py` mocks the API, then changes both together so the test proves the behaviour instead of the screenshot alone.

On Rush the tests and the app run in the agent's own sandbox on the Python version the CI workflow uses, and the `pytest` output and the screenshot stay in the transcript next to the diff. A Linear label becomes a pull request with its evidence attached, and several labelled issues on the same app run at once, each on its own branch.

## Frequently asked questions

### Why tokens and not dollars?

Because the API reports tokens and nothing in the repository states a price. A currency figure needs a per-model price table that goes out of date, so the agent shows the counts the response contains and leaves a marked place for a price lookup. Ask for it in the prompt with your prices and it adds the table.

### Does it work with the Anthropic page too?

Not in this pull request. `pages/1_File_Q&A.py` uses the Anthropic client's legacy `completions.create`, whose response has no usage object. Moving that page to the Messages API, which does report `input_tokens` and `output_tokens`, is a separate change, and the description lists it as a follow-up rather than mixing two migrations.

### Will the stored usage break the API call?

No. The one line most likely to be forgotten is stripping the `usage` key from each message before `st.session_state.messages` is sent back to `client.chat.completions.create`, and the agent adds it and tests it: the mock records what it was called with, and the test asserts no message carries `usage`.

### How is the running total kept across reruns?

In `st.session_state["total_tokens"]`, updated once when a response arrives rather than recomputed from the replay loop, so Streamlit's rerun on every interaction does not double count. The test sends two prompts and asserts the metric equals the sum of both mocked usages.

## More Streamlit jobs

- [Add Type Checking to a Streamlit App](https://getrush.ai/run/add-type-checking/streamlit/)
- [Dockerize a Streamlit App](https://getrush.ai/run/dockerize-app/streamlit/)
- [Fix "AttributeError: module 'streamlit' has no attribute 'beta_columns'" and the beta_expander Rename](https://getrush.ai/run/fix-streamlit-beta-columns-attribute-error/streamlit/)

## More jobs like this: Turn a Linear label into a PR

- [Track Token Usage per User in an AI SDK Chatbot](https://getrush.ai/run/add-token-usage-tracking/ai-sdk/)
- [Wrap Multi-Step Writes in Transactions with Drizzle](https://getrush.ai/run/add-transactions/drizzle/)
- [Add Transactions to a Mongoose App](https://getrush.ai/run/add-transactions/mongoose/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
