# Add a Fallback Model to a LangGraph Agent

Yes. An agent reads how your graph loads its chat model, adds a second provider it falls back to on an error, proves it with a failing fake model, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-model-fallbacks/langgraph/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent starts with where the model is chosen, because a fallback is a change to that one place and nowhere else. In an agent laid out like langchain-ai/react-agent, `src/react_agent/context.py` is a dataclass with `model` defaulting to `anthropic/claude-sonnet-4-5-20250929` and a `__post_init__` that reads each unset field from an environment variable of the same name in upper case; `utils.py` splits that `provider/model` string and calls `init_chat_model`; and `graph.py` calls `load_chat_model(runtime.context.model).bind_tools(TOOLS)` inside `call_model` on every step. `pyproject.toml` lists `langchain-openai`, `langchain-anthropic` and `langchain-fireworks`, so two providers are already installed; `tests/unit_tests/test_configuration.py` checks the context's env-var behaviour, and `.github/workflows/unit-tests.yml` runs `ruff`, `mypy --strict src/` and `pytest tests/unit_tests` on Python 3.11 and 3.12.

It then adds the fallback as one more field and one more line. `Context` gets `fallback_model`, an optional `provider/model` string read from `FALLBACK_MODEL` the same way as the others, with a description for the Studio sidebar like the existing fields. `load_chat_model` gains an optional second name and, when it is given, returns the primary with `with_fallbacks([secondary])`. The order of operations is the detail that matters: `with_fallbacks` returns a runnable with no `bind_tools`, so the agent binds `TOOLS` to each model first and wraps the two bound models, which means `call_model` in `graph.py` changes from `load_chat_model(...).bind_tools(TOOLS)` to one call that takes the tools.

Two things come from reading rather than from the API reference. The `is_last_step` branch in `call_model` inspects `response.tool_calls`, and a fallback model's response has the same `AIMessage` shape, so that branch needs no change. And the fallback should trigger on provider errors, not on every exception: the agent passes `exceptions_to_handle` with the rate-limit, timeout and server-error types rather than the default `Exception`, so a bug in the prompt format does not quietly switch providers.

In its sandbox the agent runs `uv sync`, `ruff check .`, `mypy --strict src/` and `pytest tests/unit_tests` as the workflow does. It adds unit tests that need no API key: one builds the chain from a `GenericFakeChatModel` that raises on invoke and a second fake that answers, and asserts the answer came from the second; one asserts that with `FALLBACK_MODEL` unset the loader returns the plain model so nothing changes for anyone who does not opt in; and one extends `test_configuration.py` for the new environment variable. The integration test, which calls Anthropic and runs on a daily schedule, is left as it is.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add a fallback model to this LangGraph agent. Read src/react_agent/context.py, utils.py, graph.py, pyproject.toml, tests/unit_tests/test_configuration.py and .github/workflows/unit-tests.yml first. Add an optional fallback_model field to Context in provider/model form, read from FALLBACK_MODEL like the other fields. Change load_chat_model so that, when a fallback is given, it binds the tools to both models and returns the primary with with_fallbacks over the secondary, handling provider errors only, not every exception; update call_model in graph.py accordingly. With no fallback set, behaviour must be unchanged. Run uv sync, ruff check, mypy --strict src/ and pytest tests/unit_tests. Add unit tests with GenericFakeChatModel that prove the fallback answers when the primary raises, that the plain model is returned when no fallback is set, and that the env var is read. Do not touch the integration tests. Open a pull request quoting the lint, type check and test output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that changes `context.py` (one field with its description), `utils.py` (the loader taking the tools and the optional second name), `graph.py` (one line in `call_model`), `.env.example` (a commented `FALLBACK_MODEL` line) and the unit tests. The description quotes the `ruff`, `mypy --strict` and `pytest` output, names the exception types the fallback handles, and says in one sentence that a deployment with no `FALLBACK_MODEL` runs exactly as before.

The run's record keeps every command and its output, so a reviewer can read the `mypy --strict` pass on the new signature and the test that shows the second fake model answering after the first raised.

Before merging, a reviewer picks the fallback itself, since a different provider's model may answer in a different style or with different tool-call limits, and decides whether the fallback should also cover `langchain-fireworks`, which is installed but unused, or be removed from the dependencies.

## Run it on every Linear issue labelled reliability

A team with several LangGraph services makes this a rule. In the Project for each repository, an automation with the prompt above and a `linear` trigger on `linear.issue.labeled` for the label `reliability` starts a run when someone files "the research agent returned 529s for twenty minutes last night". The trigger adds the issue after the prompt, so the run reads which error the issue saw and makes sure that type is in the handled set.

Each run opens its own pull request on its own branch and links the issue. A service whose loader already wraps a fallback gets a description saying so and a pull request that only widens the exception list if the issue names a new one.

## Why run it on Rush?

`with_fallbacks` is one line, and the three ways it goes wrong are all invisible until a provider actually fails: tools bound after the wrap and silently absent from the fallback, every exception caught so a prompt bug switches providers, and a context field that reads the wrong environment variable. In a Rush sandbox the agent makes the primary fail on purpose with a fake model and reads which model answered, under the same `mypy --strict` the repository's workflow runs.

The test output and the type check stay with the run and the pull request quotes them, so the reviewer reads the assertion that the fallback answered rather than a description of the change. Across several agents the job runs in parallel, one sandbox each, each with its own proof.

## Frequently asked questions

### Does the fallback need another API key?

Yes, for the second provider: `OPENAI_API_KEY` if the fallback is an OpenAI model while the primary is Anthropic. `.env.example` already lists both. With no `FALLBACK_MODEL` set, no second key is read and nothing changes.

### Will tool calling still work when the fallback answers?

Yes, because the tools are bound to each model before the two are wrapped. That order is the point of the change in `utils.py`; the pull request's unit test uses a fake model to show the fallback's answer flowing through the same `route_model_output` edge.

### Which errors trigger the fallback?

Provider-side ones: rate limits, timeouts and server errors, listed explicitly in `exceptions_to_handle`. A `ValueError` from a malformed message or a missing key is raised as before, so a bug cannot hide behind a silent provider switch.

### Does this change what LangGraph Studio shows?

Only by adding a `fallback_model` field to the configuration sidebar, with the description the dataclass carries, the same way `model` and `max_search_results` appear today. `langgraph.json` is untouched.

## More LangGraph jobs

- [Add Rate Limiting to a LangGraph Agent](https://getrush.ai/run/add-rate-limiting/langgraph/)
- [Add Structured Output to a LangGraph Agent](https://getrush.ai/run/add-structured-output/langgraph/)
- [Handle Tool Errors in a LangGraph Agent](https://getrush.ai/run/add-tool-error-handling/langgraph/)

## The same job on other stacks

- [Add a Fallback Model to an AI SDK Chat Route](https://getrush.ai/run/add-model-fallbacks/ai-sdk/)

## More jobs like this: Turn a Linear label into a PR

- [Add Model Grants to a dbt Project](https://getrush.ai/run/add-model-grants/dbt/)
- [Add a Model Selector to a Streamlit Chatbot](https://getrush.ai/run/add-model-selector/streamlit/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
