# Trim Long Chats to the Model's Context Window with the AI SDK

Yes. An agent adds a trimmer that drops the oldest turns and stale tool output before each AI SDK call, keyed by a per-model budget, tests it, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-context-window-trimming/ai-sdk/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads how a chat's history reaches the model today. In a Next.js chat app laid out like vercel/chatbot, `app/(chat)/api/chat/route.ts` loads every row of the chat with `getMessagesByChatId`, which selects from `Message_v2` ordered by `createdAt` with no limit, turns them into UI messages with `convertToUIMessages` in `lib/utils.ts`, appends the new user message, and hands the whole list to `convertToModelMessages` and then to `streamText`. The request schema in `app/(chat)/api/chat/schema.ts` caps one text part at 2000 characters but says nothing about how many messages a chat may hold, and the five tools in `lib/ai/tools` write document content into parts that are stored as JSON on the message and replayed on every later turn. So a long chat grows until the model refuses it, and the route's `onError` turns that refusal into "Oops, an error occurred!".

It adds the trimmer where the list is built, not inside the SDK call. A new `lib/ai/context.ts` exports a function that takes the UI messages and a budget and returns the messages to send: the newest user message is always kept, older turns are dropped whole from the oldest end so a tool call is never separated from its result, and in assistant messages that survive, the output of `createDocument`, `updateDocument` and `editDocument` parts older than the latest turn is replaced by a one-line note that the document exists, since the artifact itself is loaded from the `Document` table and not from the conversation. The budget is a `maxInputTokens` field the agent adds to each entry of `chatModels` in `lib/ai/models.ts`; it leaves the field unset where it cannot read a published context window for that model, and a model without one is not trimmed, so nothing changes until the team fills the numbers in.

Tokens are estimated before the call and corrected after it. The estimate counts characters of the text parts and the serialized tool parts, which is coarse but errs toward sending less. After each turn the route reads `usage.inputTokens.total` from the finished stream, the shape `lib/ai/models.mock.ts` already returns, and stores it on the assistant message's metadata next to `createdAt`, so the next request can compare its estimate with what the model actually counted and the pull request's description can show the two side by side from the agent's own runs.

Tests sit beside `lib/ai/models.test.ts`, which already builds models with `MockLanguageModelV3` from `ai/test`. The agent adds a model whose `doStream` records the prompt it receives, builds a chat of many turns with document tool parts, runs the route's message preparation with a small budget, and asserts that the oldest turns are gone, the latest user message is present, the surviving tool outputs are the short note, and a model with no budget gets the untrimmed list. It runs those tests under the `PLAYWRIGHT` flag the project uses for its mock environment, then `pnpm check` and `tsc --noEmit`.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Keep long chats inside the model's context window. Read app/(chat)/api/chat/route.ts, app/(chat)/api/chat/schema.ts, lib/ai/models.ts, lib/ai/models.mock.ts, lib/ai/models.test.ts, lib/utils.ts, lib/db/queries.ts, lib/db/schema.ts and lib/types.ts first. Add lib/ai/context.ts with a function that takes the UI messages and a token budget and returns what to send: always keep the newest user message, drop the oldest turns whole, and in older assistant messages replace the output of createDocument, updateDocument and editDocument parts with a one-line note that the document exists. Add an optional maxInputTokens to each chatModels entry, set it only where you can read the model's published context window, and skip trimming for a model without one. Estimate tokens from characters before the call, and after each turn store usage.inputTokens.total on the assistant message metadata so the estimate can be compared with the real count. Write tests with MockLanguageModelV3 that capture the prompt the model receives. Run the tests, pnpm check and tsc --noEmit, and open a pull request with the output and a table of estimated versus reported tokens from your runs.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that adds `lib/ai/context.ts` and its test, changes `lib/ai/models.ts`, the chat route and `lib/types.ts` for the new metadata field, and documents `maxInputTokens` beside `gatewayOrder`. The description explains the three rules the trimmer follows, lists which models got a budget and from which vendor page, and shows the estimated and reported token counts from the agent's test runs.

The run's record keeps the test output, the lint and the type check, so a reviewer reads what the model actually received in each case rather than a summary. A reviewer checks the drop order against a chat that ends in a tool approval, since the route's approval flow replays stored messages, and reads the one-line note the trimmer leaves to make sure it does not invite the model to call the tool again.

## Run it on every Linear issue labelled context-window

A team makes this a rule. In the Project for the repository, an automation with the prompt above and a `linear` trigger on `linear.issue.labeled` for the label `context-window` starts a run when someone files "long chats fail with a generic error" or "tool output fills the prompt". The trigger adds the issue after the prompt, so the run reads which model or which chat length the reporter saw and starts from that case in its tests.

Each run opens its own pull request on its own branch and links the issue. A repository that already trims gets a description saying so and a pull request only for the part the issue asks for, such as a budget for a newly added model.

## Why run it on Rush?

Every chat product hits this once a user keeps one conversation open for a week, and the usual fix, slicing the array to the last N messages, breaks tool calls apart and loses the document the user is still editing. The rules that work depend on the app's own message shape, which here is JSON parts with five tool types, and on numbers the vendors publish. The agent reads both in its sandbox, writes the trimmer against the project's own mock models, and keeps the estimated and reported counts so the budget is checked against the model's own accounting rather than guessed.

The pull request is one module, one field per model and a handful of tests, so review is reading three rules. A company with several AI SDK apps runs them all at once, each in its own sandbox with its own pull request.

## Frequently asked questions

### Why not summarise the old turns instead of dropping them?

A summary costs a model call on every long turn and changes what the model sees in ways that are hard to test. Dropping whole turns is deterministic and the test can assert exactly what was sent. The agent describes in the pull request where a summary step would slot in, since the trimmer already knows which turns it dropped.

### Does this change what the user sees in the chat?

No. The database and the UI keep every message; only the list sent to the model changes. The stored `inputTokens` metadata is the one visible addition, and the UI ignores metadata fields it does not read.

### Where do the per-model budgets come from?

From each vendor's published context window, which the agent reads during the run and cites in the pull request. Where it finds none it leaves the field unset and the model untrimmed, and lists that model in the description so the team can fill it in.

### What about the character estimate being wrong?

It will be, in both directions by model, which is why the route stores the real count after every turn. The description shows the two from the agent's runs; a team that sees the estimate run high for one model lowers that model's budget, and a later run can swap the estimate for a tokenizer if the gap matters.

## More AI SDK jobs

- [Add a Fallback Model to an AI SDK Chat Route](https://getrush.ai/run/add-model-fallbacks/ai-sdk/)
- [Track Token Usage per User in an AI SDK Chatbot](https://getrush.ai/run/add-token-usage-tracking/ai-sdk/)
- [Add Chat Export to an AI SDK Chatbot](https://getrush.ai/run/add-chat-export/ai-sdk/)

## More jobs like this: Turn a Linear label into a PR

- [Add CORS to a Flask API](https://getrush.ai/run/add-cors/flask/)
- [Add CORS and Rate Limiting to a Ktor Server from a Linear Issue](https://getrush.ai/run/add-cors/ktor/)
- [Add CORS to a Rocket JSON API](https://getrush.ai/run/add-cors/rocket/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
