What the agent does
The agent reads how the model is chosen and how a failure reaches the user today. In a Next.js chat app laid out like vercel/chatbot, lib/ai/providers.ts returns gateway.languageModel(modelId) outside tests and a mock provider under Playwright; lib/ai/models.ts lists the chat models with a gatewayOrder per model, the providers the gateway should try for that model, and getModelAvailability, which reads the gateway's endpoint health. app/(chat)/api/chat/route.ts calls streamText with getLanguageModel(chatModel) and providerOptions.gateway.order, tracks hasModelActivity through onChunk so a waiting status can turn into "may be slow or unavailable" after a health check, and merges the result into a UI message stream whose onError maps every failure to Oops, an error occurred! except the gateway's missing-card message. package.json pins ai at 7.0.15 and @ai-sdk/provider at ^4.0.2. So when a model's providers all refuse a request, the user waits, then sees a generic error, and the only recourse is to pick another model by hand.
It adds the fallback at the model boundary rather than in the route. The AI SDK's wrapLanguageModel takes a LanguageModelMiddleware, and the agent writes one in lib/ai/fallback.ts whose wrapStream calls the primary doStream and, if that call throws an APICallError that is retryable or carries a 429 or 5xx status before any part of the stream has started, calls the fallback model's doStream with the same params. An error after the first chunk is not retried, because the user has already seen text and a second answer would duplicate it. Each model in chatModels gets an optional fallbackModelId naming another entry in the same list, and getLanguageModel returns the wrapped model when one is set, so the route's streamText call does not change.
The fallback is visible, not silent. The middleware emits a data-waiting-status through the existing status mechanism saying which model answered, so the waiting message that already says "may be slow or unavailable" can say "answering with" the fallback instead; the route also records the model that actually produced the answer on the assistant message's metadata so the saved chat says which model wrote it. The titleModel in models.ts gets the same wrapper, since a title that fails leaves the chat called "New chat".
Tests live beside the mock provider the project already has. lib/ai/models.mock.ts builds a LanguageModel with doStream; the agent adds a model whose doStream throws an APICallError with a retryable status and a second that streams, wraps the first with the middleware pointing at the second, and asserts the stream completes, the fallback status event is emitted, and a non-retryable 400 is rethrown untouched. It runs those under the PLAYWRIGHT environment flag the route already honours, then pnpm check and tsc --noEmit, and leaves the e2e suite as it is.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add a fallback model to the chat route. Read package.json, lib/ai/providers.ts, lib/ai/models.ts, lib/ai/models.mock.ts, lib/ai/generate-title.ts, app/(chat)/api/chat/route.ts and lib/errors.ts first. Write a LanguageModelMiddleware in lib/ai/fallback.ts whose wrapStream calls the primary doStream and, on an APICallError that is retryable or has a 429 or 5xx status before any stream part has been emitted, calls a fallback model's doStream with the same params; never retry after the first chunk. Add an optional fallbackModelId to each entry in chatModels and have getLanguageModel return the wrapped model with wrapLanguageModel when one is set, and wrap the title model the same way. Emit the existing data-waiting-status with the fallback model's name and record the answering model on the assistant message. Add tests using the mock provider pattern: a primary that throws a retryable error and a fallback that streams, and a non-retryable error that is rethrown. Run the tests, pnpm check and tsc --noEmit, and open a pull request with the output.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that adds lib/ai/fallback.ts and its test, changes models.ts, providers.ts and the chat route's status handling, and documents fallbackModelId beside gatewayOrder. The description shows the middleware, the condition that decides whether to fall back, and the test output for the three cases.
The run's record keeps the test runs and the lint and type checks, so a reviewer sees the retryable error swallowed and the 400 rethrown rather than reads that it would be.
Before merging, a reviewer picks the pairs: which model falls back to which, with the capability table in mind, since a fallback without tool support changes what the route can do mid-conversation. The pull request lists each pair with the capabilities the gateway reports for both.
Run it on every alert on model provider errors
This is the job to run when the monitor reports a rise in 429 or 5xx responses from the gateway, or a model's getModelAvailability reading impacted for a sustained period. In the Project for the repository, an automation with the prompt above and a webhook trigger takes the alert body from a Grafana contact point, which can send the trigger's bearer secret in an Authorization header and an Idempotency-Key made from the alert's fingerprint to the trigger's URL, so a re-fired alert does not start a second run. The trigger adds the payload after the prompt, so the run reads which model was failing.
The first run adds the middleware and the pairs. A later run on a repository that already has them reads the alert, checks that the failing model has a fallback with the capabilities the conversation needs, and opens a pull request that changes one pair with the reason in the description, each on its own branch and linked to the alert.
The use case this job follows, with its automation and trigger: Fix a bug from an alert.
Why run it on Rush?
A provider outage is the failure every chat product meets and few handle: the retry in the SDK covers a blip, the gateway's provider order covers one provider, and nothing covers a model that is down everywhere. The fix belongs at the model boundary, where the condition is a status code and a flag that says whether text has started, and the proof is a mock that throws. The agent writes both in its sandbox against the project's own mock provider and keeps the output.
The pull request is one middleware, one field per model and three tests, so review is reading the condition. A company with several AI SDK apps runs them all at once, each in its own sandbox with its own pull request.
Frequently asked questions
Why not retry after the first chunk too?
Because the user has already seen text from the first model. A second stream would either repeat it or continue from a different model's state, and neither reads well. Before the first chunk nothing has been shown, so switching is invisible except for the status line the middleware emits.
How is this different from gatewayOrder?
gatewayOrder tells the gateway which providers to try for the same model. The fallback tries a different model when every provider for the first one has failed. The two stack: the gateway walks its order, and only an error that comes back from that walk triggers the fallback.
What if the fallback lacks tools or vision?
The route already reads each model's capabilities from the gateway to decide activeTools and sendReasoning. The agent lists each pair with both models' capabilities in the pull request so the reviewer pairs like with like, and the middleware passes the same params, so a tool call the fallback cannot make fails the way it would for any model without tools.
Does the SDK's own retry still run?
Yes. streamText keeps its maxRetries on the primary model, so a transient error is retried on the same model first. The middleware sees only the error that survives those retries, which is the point at which switching models is the right call.
More AI SDK jobs
The same job on other stacks
- Add a Fallback Model to a LangGraph Agent (LangGraph)