All jobs

Trim Message History in a LangGraph Agent

Yes. An agent trims what call_model sends to a token budget, keeps tool calls with their results, caps one search result's size, tests it and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent measures the growth before it limits it. In a LangGraph project like langchain-ai/react-agent, State in src/react_agent/state.py keeps messages under the add_messages reducer, and call_model in graph.py sends the system prompt followed by *state.messages, all of them, on every step. One question with one search adds four messages: the question, the model's tool call, the tool's result and the answer. The result is the heavy one, since search in tools.py returns whatever Tavily sends for max_search_results, which context.py defaults to 10. On the LangGraph server threads are kept between requests, and the README says follow-ups are appended to the same thread, so nothing ever shrinks the list. With a stub in place of Tavily that returns ten results of about 4,800 characters each, count_tokens_approximately puts the first model request at 31 tokens and the eighth, four questions in, at about 48,000.

It trims what is sent, not what is stored. In call_model the agent passes state.messages through trim_messages from langchain_core.messages with strategy="last", token_counter=count_tokens_approximately, a max_tokens budget and start_on="human", and gives the model the result. The state is left whole, so the thread in Studio and any audit of the conversation still show every turn. start_on is the setting that keeps the request valid: without it the kept window can open on an assistant message or a tool result whose call was cut away, and providers refuse a tool result with no matching call. The approximate counter is deliberate. load_chat_model builds whatever provider/model string the Context names, Anthropic, OpenAI or Fireworks from the dependencies in pyproject.toml, so the agent uses a counter that needs no provider and leaves headroom in the budget.

Running it exposes two edges the agent handles in the same change. When the newest turn alone is over the budget, trim_messages returns an empty list, and the model would be asked to answer a system prompt with no question. Trimming cannot fix a single oversized message, so the agent bounds it at the source: search shortens each result's content to a fixed length before returning it, and a test proves one full turn fits the default budget. The second edge is in configuration. The budget becomes a field on Context, max_context_tokens, beside max_search_results, and Context.__post_init__ fills fields from the environment with os.environ.get, which returns text. MAX_SEARCH_RESULTS=5 already arrives as the string '5', and a string budget makes trim_messages fail with a TypeError comparing int and str, so the agent converts integer fields there and says so in the pull request.

The tests sit in tests/unit_tests/ and call no provider. A fake model records the messages it receives, and a stub search returns long results; both replace names fetched with importlib.import_module, since import react_agent.graph as g hands back the compiled graph that __init__.py re-exports. The cases: a long thread stays under the budget on every model call while state.messages keeps its full length; the first kept message is always a human one; a tool call is never sent without its result; one turn with ten maximum-length results still fits; and the budget read from the environment is an integer. Then uv sync --extra dev for ruff and mypy, which are in the dev extra, and the three commands from .github/workflows/unit-tests.yml: ruff check ., mypy --strict src/ and pytest tests/unit_tests.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Stop this LangGraph agent's prompt from growing without limit. Read pyproject.toml, uv.lock, the README, every file under src/react_agent, tests/conftest.py, both test directories, the Makefile and .github/workflows first. Measure the growth: with a fake model that records what it is sent and a stub in place of TavilySearch that returns long results, run several questions on one thread and report the approximate token count of each model request. Then, in call_model, pass state.messages through trim_messages (strategy last, token_counter count_tokens_approximately, start_on human) with a budget from a new Context field max_context_tokens, and send the trimmed list to the model. Do not remove anything from state. Check what trim_messages returns when the newest turn alone exceeds the budget, and prevent that case by shortening each search result's content to a fixed length in src/react_agent/tools.py. Context.__post_init__ reads fields from the environment as strings: make integer fields integers, and say in the pull request that this also changes max_search_results. Add unit tests under tests/unit_tests, with no API key: every model request on a long thread is under the budget while state keeps every message, the kept window starts on a human message, no tool call is sent without its result, one turn with maximum-length results fits, and MAX_CONTEXT_TOKENS from the environment is an int. Run uv sync --extra dev, ruff check ., mypy --strict src/ and pytest tests/unit_tests, do not run tests/integration_tests, paste the output and the before and after measurements in the pull request and open it.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request across three files under src/react_agent/ and one new test file. graph.py gains the trimming call inside call_model; context.py gains the max_context_tokens field and the integer conversion; tools.py shortens each result; state.py is untouched, because the stored history does not change. The description leads with two columns of numbers from the agent's measurement, the approximate size of each model request on a long thread before and after, so the effect is visible without running anything.

A reviewer checks the budget against the smallest context window among the models the team runs, remembering the count is an estimate. They ask what the agent loses when turn one falls out of the window: if users state a name or a constraint early and refer to it later, trimming alone will forget it, and the follow-up is a summary or a store. And they read the truncation in search for whether cutting a result mid-sentence is acceptable for their answers, or whether fewer results in full would serve better.

Run it on every context length error

The symptom arrives as a provider error on a long thread, usually through error tracking. Sentry signs its own webhook requests and cannot add the Authorization: Bearer <secret> header a Rush webhook trigger requires, so the event goes to a handler on your server first. The handler checks Sentry's signature, keeps issues whose message mentions context length, and sends POST /o/{org}/sessions with this page's body, the issue's title and stack trace appended to input, the agent's repository in resources, and a request_id that is a UUID v5 of the Sentry issue id. The same issue firing again then returns the run that is already working on it.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

The interesting cases here only appear when the graph is run for many turns with large tool results: the window that opens on an orphaned tool message, the empty list when one turn is too big, the string that should have been a number. A developer checking the change by hand asks two questions and sees it work. The agent drives the graph through a long scripted thread in its sandbox, records the size of every request, and keeps the commands and their output in the session's transcript, where the numbers in the pull request can be checked.

No model is called to do it. The fake records what it would have been sent, which is the thing being limited, so the measurement costs no provider tokens and needs no key in the sandbox. That also makes it safe to start from an alert at any hour: the run cannot spend against your model account or touch a live thread, and what it leaves is a branch, a diff and proof that the diff does what it says.

Frequently asked questions

Does trimming delete messages from the thread?

No. trim_messages returns a shorter list for one model call and leaves state.messages alone, so the checkpoint keeps the whole conversation. Deleting from state is a different operation, done by returning RemoveMessage entries from a node, and this change does not use it.

Why did trim_messages return an empty list?

With strategy="last" and start_on="human", it keeps the newest messages that fit and then drops anything before the first human message in that window. When the newest turn is larger than max_tokens, nothing qualifies. Raise the budget or make the largest message smaller, which is why this change also shortens search results.

How accurate is count_tokens_approximately?

It is an estimate from the text, with no tokenizer and no network call, so it is fast and works for any provider. It will differ from what a given model counts. Set the budget with room to spare, well under the smallest context window you run.

More LangGraph jobs

More jobs like this: Fix a bug from an alert

Keep reading