# Add Rate Limiting to a LangGraph Agent

Yes. An agent adds one shared InMemoryRateLimiter to your graph's model calls, makes the rate a Context field, proves it with a fake model, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-rate-limiting/langgraph/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent starts with where the model is built. In an agent laid out like langchain-ai/react-agent, `call_model` in `src/react_agent/graph.py` runs `load_chat_model(runtime.context.model).bind_tools(TOOLS)` on every step, and `load_chat_model` in `utils.py` is a thin `init_chat_model(model, model_provider=provider)`, so each turn of the `call_model` to `tools` cycle constructs a fresh client. That is the detail that defeats the obvious fix: a rate limiter attached inside `load_chat_model` would be a new limiter with a full bucket on every call and would never limit anything. `Context` in `context.py` is a `kw_only` dataclass with `system_prompt`, `model` and `max_search_results`, each overridable from an upper-cased environment variable in `__post_init__`, and the project pins `langgraph>=1.2.11` and `langchain>=1.3.17`.

It puts the limiter where it is shared. A module-level `InMemoryRateLimiter` from `langchain_core.rate_limiters`, built once in `utils.py` with `requests_per_second`, `check_every_n_seconds=0.1` and a `max_bucket_size`, is passed to `init_chat_model` as `rate_limiter=`, so every client the graph builds draws from the same bucket for the life of the process. The rate becomes a `model_requests_per_second: float` field on `Context` with a default of `2.0`, which also makes it `MODEL_REQUESTS_PER_SECOND` in the environment through the existing `__post_init__` loop; `load_chat_model` takes it as a second argument and keeps one limiter per distinct rate in a small dict, so two configurations in one process do not share a bucket by accident. The Tavily call in `tools.py` is left alone: it is a search API with its own quota, and limiting it is a separate change.

The limiter only delays; it never raises. So the agent also writes down what happens past the bucket: an `AIMessage` from the provider's `429` still surfaces as the exception it always was, and the pull request says so rather than pretending the limiter is a retry. The `README.md` gets a short section under the model setup naming the new field and the environment variable.

In its sandbox the agent installs the project with `uv pip install -r pyproject.toml`, then writes `tests/unit_tests/test_rate_limit.py` with `GenericFakeChatModel` from `langchain_core.language_models.fake_chat_models`, a limiter at 5 requests per second and a bucket of 1, and asserts that ten `ainvoke` calls take at least 1.8 s and that two `load_chat_model` calls with the same rate return clients holding the same limiter object. It runs `make test`, `make lint` so `ruff` and `mypy --strict` pass on the new field, and pastes the timings into the pull request.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add rate limiting to the model calls in this LangGraph agent. Read src/react_agent/graph.py, utils.py, context.py, tools.py, pyproject.toml and tests/unit_tests/test_configuration.py first. call_model rebuilds the chat model on every step, so the limiter must be shared: create one InMemoryRateLimiter per rate at module level in utils.py and pass it to init_chat_model as rate_limiter. Add model_requests_per_second to Context with a default of 2.0 so it is also settable as MODEL_REQUESTS_PER_SECOND, and have load_chat_model take it. Leave the Tavily tool alone. Write tests/unit_tests/test_rate_limit.py using GenericFakeChatModel: ten calls at 5 requests per second with a bucket of 1 must take at least 1.8 seconds, and two loads with the same rate must share one limiter. Run make test and make lint, quote the timings, and add a README section. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request touching `src/react_agent/utils.py`, `src/react_agent/context.py`, `README.md` and a new `tests/unit_tests/test_rate_limit.py`. The description quotes the `make test` run with the new test's duration, the `ruff` and `mypy --strict` lines from `make lint`, and a short table of how long ten calls took at 5 and at 50 requests per second so the reviewer can see the limiter is doing the waiting.

The run's record keeps the full install and test transcript, including the `uv` resolution that proves `langchain_core.rate_limiters` is present at the pinned versions. Nothing calls a paid model: the test uses the fake chat model, and the integration test under `tests/integration_tests` is not run.

Before merging, a reviewer sets the default rate. 2.0 per second suits a single agent against one provider key; a team sharing one key across several deployments sets a lower `MODEL_REQUESTS_PER_SECOND` per process and keeps the sum under the provider's limit.

## Run it on every customer request

A product that runs one LangGraph agent per customer starts this from its own code. When a customer turns on a feature that calls the model more often, the product sends this request with the customer's repository in `resources` and a `request_id` that is a UUID v5 of the customer id and the feature name, so a retried request returns the run it already started instead of a second one.

Each run opens its own pull request against that customer's fork or branch and reports back over the `run.completed` webhook, whose `session_url` reads the session and its pull request. The product can pass the customer's agreed rate in the prompt, and the agent sets it as the `Context` default for that deployment.

## Why run it on Rush?

The bug this change prevents is invisible until production: a limiter built inside the model loader looks right in a diff, passes a test that calls the loader once, and does nothing, because the graph rebuilds the model every step. Finding that takes reading `call_model`, not the limiter's docs. In a Rush sandbox the agent installs the real pinned `langchain_core`, times real calls through a fake model, and the pull request shows the wait.

The timings and the lint output stay with the run, so the next person who raises the rate can read what the test asserts and why the limiter lives in `utils.py` rather than in the node. Across a fleet of agents built from the same template, the job runs in parallel, one sandbox each, and every pull request carries its own numbers.

## Frequently asked questions

### Does the limiter retry on a 429 from the provider?

No. `InMemoryRateLimiter` only spaces requests out before they are sent; a `429` that still arrives raises as before. If you want retries, say so and the agent adds `max_retries` on the chat model, which is a separate knob with its own backoff.

### Why is the limiter not inside load_chat_model?

Because `call_model` calls `load_chat_model` on every step, so a limiter created there would be new each time with a full bucket. The limiter is created once per rate at module level and reused, which the test checks by comparing the two clients' `rate_limiter` attribute.

### Does this limit the Tavily search tool too?

Not by default. The search tool has its own API quota and its own `max_search_results` setting; wrapping it would mix two budgets. Ask for it in the prompt and the agent adds a second limiter around the `TavilySearch.ainvoke` call.

### What about LangGraph Server deployments?

The same code applies: `langgraph.json` points at `graph.py:graph`, and a module-level limiter is shared by every run the server handles in one process. Set `MODEL_REQUESTS_PER_SECOND` in the deployment's environment, as the server loads `.env` per `langgraph.json`.

### Will the test be flaky on a slow CI runner?

It asserts a lower bound only, that ten calls took at least 1.8 s at 5 per second, which a slow machine can only exceed. It does not assert an upper bound, so it cannot fail because the runner was busy.

## More LangGraph jobs

- [Add Structured Output to a LangGraph Agent](https://getrush.ai/run/add-structured-output/langgraph/)
- [Handle Tool Errors in a LangGraph Agent](https://getrush.ai/run/add-tool-error-handling/langgraph/)
- [Trim Message History in a LangGraph Agent](https://getrush.ai/run/trim-message-history/langgraph/)

## The same job on other stacks

- [Add Rate Limiting to a Leptos Server](https://getrush.ai/run/add-rate-limiting/leptos/)
- [Add Rate Limiting to a Litestar API](https://getrush.ai/run/add-rate-limiting/litestar/)
- [Add Rate Limiting to a Medusa Store API](https://getrush.ai/run/add-rate-limiting/medusa/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
