All jobs

Add a Fallback Model to a LangGraph Agent

Yes. An agent reads how your graph loads its chat model, adds a second provider it falls back to on an error, proves it with a failing fake model, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent starts with where the model is chosen, because a fallback is a change to that one place and nowhere else. In an agent laid out like langchain-ai/react-agent, src/react_agent/context.py is a dataclass with model defaulting to anthropic/claude-sonnet-4-5-20250929 and a __post_init__ that reads each unset field from an environment variable of the same name in upper case; utils.py splits that provider/model string and calls init_chat_model; and graph.py calls load_chat_model(runtime.context.model).bind_tools(TOOLS) inside call_model on every step. pyproject.toml lists langchain-openai, langchain-anthropic and langchain-fireworks, so two providers are already installed; tests/unit_tests/test_configuration.py checks the context's env-var behaviour, and .github/workflows/unit-tests.yml runs ruff, mypy --strict src/ and pytest tests/unit_tests on Python 3.11 and 3.12.

It then adds the fallback as one more field and one more line. Context gets fallback_model, an optional provider/model string read from FALLBACK_MODEL the same way as the others, with a description for the Studio sidebar like the existing fields. load_chat_model gains an optional second name and, when it is given, returns the primary with with_fallbacks([secondary]). The order of operations is the detail that matters: with_fallbacks returns a runnable with no bind_tools, so the agent binds TOOLS to each model first and wraps the two bound models, which means call_model in graph.py changes from load_chat_model(...).bind_tools(TOOLS) to one call that takes the tools.

Two things come from reading rather than from the API reference. The is_last_step branch in call_model inspects response.tool_calls, and a fallback model's response has the same AIMessage shape, so that branch needs no change. And the fallback should trigger on provider errors, not on every exception: the agent passes exceptions_to_handle with the rate-limit, timeout and server-error types rather than the default Exception, so a bug in the prompt format does not quietly switch providers.

In its sandbox the agent runs uv sync, ruff check ., mypy --strict src/ and pytest tests/unit_tests as the workflow does. It adds unit tests that need no API key: one builds the chain from a GenericFakeChatModel that raises on invoke and a second fake that answers, and asserts the answer came from the second; one asserts that with FALLBACK_MODEL unset the loader returns the plain model so nothing changes for anyone who does not opt in; and one extends test_configuration.py for the new environment variable. The integration test, which calls Anthropic and runs on a daily schedule, is left as it is.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add a fallback model to this LangGraph agent. Read src/react_agent/context.py, utils.py, graph.py, pyproject.toml, tests/unit_tests/test_configuration.py and .github/workflows/unit-tests.yml first. Add an optional fallback_model field to Context in provider/model form, read from FALLBACK_MODEL like the other fields. Change load_chat_model so that, when a fallback is given, it binds the tools to both models and returns the primary with with_fallbacks over the secondary, handling provider errors only, not every exception; update call_model in graph.py accordingly. With no fallback set, behaviour must be unchanged. Run uv sync, ruff check, mypy --strict src/ and pytest tests/unit_tests. Add unit tests with GenericFakeChatModel that prove the fallback answers when the primary raises, that the plain model is returned when no fallback is set, and that the env var is read. Do not touch the integration tests. Open a pull request quoting the lint, type check and test output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that changes context.py (one field with its description), utils.py (the loader taking the tools and the optional second name), graph.py (one line in call_model), .env.example (a commented FALLBACK_MODEL line) and the unit tests. The description quotes the ruff, mypy --strict and pytest output, names the exception types the fallback handles, and says in one sentence that a deployment with no FALLBACK_MODEL runs exactly as before.

The run's record keeps every command and its output, so a reviewer can read the mypy --strict pass on the new signature and the test that shows the second fake model answering after the first raised.

Before merging, a reviewer picks the fallback itself, since a different provider's model may answer in a different style or with different tool-call limits, and decides whether the fallback should also cover langchain-fireworks, which is installed but unused, or be removed from the dependencies.

Run it on every Linear issue labelled reliability

A team with several LangGraph services makes this a rule. In the Project for each repository, an automation with the prompt above and a linear trigger on linear.issue.labeled for the label reliability starts a run when someone files "the research agent returned 529s for twenty minutes last night". The trigger adds the issue after the prompt, so the run reads which error the issue saw and makes sure that type is in the handled set.

Each run opens its own pull request on its own branch and links the issue. A service whose loader already wraps a fallback gets a description saying so and a pull request that only widens the exception list if the issue names a new one.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

with_fallbacks is one line, and the three ways it goes wrong are all invisible until a provider actually fails: tools bound after the wrap and silently absent from the fallback, every exception caught so a prompt bug switches providers, and a context field that reads the wrong environment variable. In a Rush sandbox the agent makes the primary fail on purpose with a fake model and reads which model answered, under the same mypy --strict the repository's workflow runs.

The test output and the type check stay with the run and the pull request quotes them, so the reviewer reads the assertion that the fallback answered rather than a description of the change. Across several agents the job runs in parallel, one sandbox each, each with its own proof.

Frequently asked questions

Does the fallback need another API key?

Yes, for the second provider: OPENAI_API_KEY if the fallback is an OpenAI model while the primary is Anthropic. .env.example already lists both. With no FALLBACK_MODEL set, no second key is read and nothing changes.

Will tool calling still work when the fallback answers?

Yes, because the tools are bound to each model before the two are wrapped. That order is the point of the change in utils.py; the pull request's unit test uses a fake model to show the fallback's answer flowing through the same route_model_output edge.

Which errors trigger the fallback?

Provider-side ones: rate limits, timeouts and server errors, listed explicitly in exceptions_to_handle. A ValueError from a malformed message or a missing key is raised as before, so a bug cannot hide behind a silent provider switch.

Does this change what LangGraph Studio shows?

Only by adding a fallback_model field to the configuration sidebar, with the description the dataclass carries, the same way model and max_search_results appear today. langgraph.json is untouched.

More LangGraph jobs

The same job on other stacks

More jobs like this: Turn a Linear label into a PR

Keep reading