All jobs

Track Token Usage per User in an AI SDK Chatbot

Yes. An agent adds a usage table, writes each turn's token counts from streamText's onEnd event, exposes a per-user total, tests it, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads what the chat route already knows and throws away. In an app laid out like vercel/chatbot, app/(chat)/api/chat/route.ts authenticates, checks checkIpRateLimit, counts the user's messages in the last hour with getMessageCountByUserId against entitlementsByUserType[userType].maxMessagesPerHour, which lib/ai/entitlements.ts sets to 10 for guest and regular alike, then calls streamText from ai 7.0.15 with stopWhen: isStepCount(5), five tools, and onEnd, onAbort, onChunk and onError callbacks that only manage the waiting-status indicator. lib/db/schema.ts has User, Chat, Message_v2, Vote_v2, Document, Suggestion and Stream tables and no column for tokens anywhere; lib/db/migrations/ holds one revision, 0000_initial, generated by drizzle-kit from drizzle.config.ts. Models come from the Vercel AI Gateway through gateway.languageModel in lib/ai/providers.ts, and in the Playwright environment from lib/ai/models.mock.ts, whose mockUsage returns fixed input and output counts.

The onEnd event of streamText carries usage for the whole response across its steps: inputTokens, outputTokens, totalTokens, inputTokenDetails.cacheReadTokens and outputTokenDetails.reasoningTokens. The agent adds a Usage table to schema.ts with id, userId, chatId, modelId, those five counts and createdAt, and generates 0001_usage.sql with pnpm db:generate, which pnpm build applies through lib/db/migrate.ts. lib/db/queries.ts gains saveUsage and getUsageByUserId({ id, differenceInHours }), written in the same shape as getMessageCountByUserId with its ChatbotError("bad_request:database") wrapping.

In the route, the existing onEnd() becomes onEnd({ usage }): it still stops the waiting status, then calls saveUsage with the session's user id, the chat id and chatModel, after the counts are read, so an aborted stream records what the provider billed up to the abort. A new GET /api/usage route in app/(chat)/api/usage/route.ts answers the signed-in user's totals for the last 24 hours and the last 30 days, and the ChatbotError surfaces gain nothing new because the route reuses unauthorized:chat and bad_request:database. The entitlements file is not changed: the counts exist now, and a token budget is a second decision the pull request leaves to the team.

The test follows tests/e2e/api.test.ts: a Playwright test posts a message to /api/chat, waits for the stream to end, reads /api/usage and asserts the totals equal the mock model's mockUsage counts, which is possible only because the test environment swaps the gateway for the mock provider. The agent runs pnpm check for the Biome rules in lint.yml and the Playwright suite against a PostgreSQL it brings up in its sandbox with POSTGRES_URL set, and quotes both.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Do the work the Linear issue describes: track token usage per user in this AI SDK chatbot. Read app/(chat)/api/chat/route.ts, lib/ai/entitlements.ts, lib/ai/providers.ts, lib/ai/models.mock.ts, lib/db/schema.ts, lib/db/queries.ts, lib/db/migrate.ts, drizzle.config.ts, lib/errors.ts, tests/e2e/api.test.ts, package.json and the workflows under .github first. Add a Usage table with userId, chatId, modelId, inputTokens, outputTokens, totalTokens, cacheReadTokens, reasoningTokens and createdAt, generate the migration with pnpm db:generate, and add saveUsage and getUsageByUserId queries in the style of getMessageCountByUserId. In the chat route read usage from streamText's onEnd event and save a row per response. Add GET /api/usage returning the signed-in user's totals for the last day and the last 30 days. Write a Playwright test that sends one message and asserts the usage endpoint matches the mock model's counts. Run pnpm check and the Playwright suite against a PostgreSQL you start, and open a pull request linking the issue and quoting the output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds the table and its migration, two queries, the usage route and the test, and changes one callback in the chat route. The description shows an example GET /api/usage response, quotes the Biome check and the Playwright run with the new test listed, and says how an aborted stream is recorded.

The run's record keeps the migration being generated and applied and the full test output, so a reviewer can see the mock counts come back through the endpoint rather than trust the join.

Before merging, a reviewer decides whether the per-hour message limit in entitlements.ts should become a token budget read from the new table, and whether guest users should get a lower one. Both are small changes on top of counts that now exist, and the pull request names the two lines involved.

Run it on every Linear issue labelled billing

This is ticket-shaped work. An automation in the Project with a linear trigger on linear.issue.labeled and billing in its config.labels starts a run with this prompt and the issue appended after it, so the run reads the ticket's own wording for which totals the product wants and opens the pull request linked to it. The trigger adds the issue; it fills in no placeholder in the prompt.

A team with several AI SDK apps starts one run per repository with one POST /o/{org}/sessions each, this prompt as input, the repository in resources and the issue's URL as a second resource, with a request_id that is a UUID v5 of the issue id so a retry returns the run it already started.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

The counts are already in the onEnd event; the work is a migration, a callback and a query that have to agree on names, plus proof that the row written is the row the provider reported. That proof needs a database and the mock provider in one place, which the agent has in its own sandbox and keeps the output of.

The pull request carries the migration, the example response and the test run, so review is a diff and one JSON body rather than a local PostgreSQL and a gateway key. Across several apps the job runs in parallel, one sandbox and one pull request each, with the callback fitted to how each route calls streamText.

Frequently asked questions

Why onEnd rather than reading the stream?

Because onEnd fires once with usage summed across every step, including tool calls, which is what the provider bills. Reading the finish part of the stream gives the same totals but would have to be merged into the UI stream the route already builds; the callback is one place.

Does an aborted response get a row?

Yes, with the tokens used up to the abort, since the event still fires with what was counted. The route's existing onAbort only stops the status indicator, so the agent reads usage in onEnd and leaves onAbort alone.

Why not add the budget in the same pull request?

Because the ticket asks for tracking, and the limit's numbers are a product decision. The message-per-hour check stays as it is; replacing it with a token budget is two lines once the team picks the figures, and the pull request names them.

Will the mock provider in tests report tokens?

Yes. lib/ai/models.mock.ts returns a fixed usage from doStream, which the SDK sums into the onEnd event, so the Playwright test can assert exact numbers without any gateway credit.

More AI SDK jobs

More jobs like this: Turn a Linear label into a PR

Keep reading