What the agent does
The agent starts with how the graph ends, because structured output is a change to the last step and to what the caller reads. In an agent laid out like langchain-ai/react-agent, src/react_agent/graph.py builds a StateGraph over State with InputState as the input schema and Context as the context schema, cycles between call_model and a ToolNode over TOOLS, and route_model_output returns __end__ when the last AIMessage has no tool calls. state.py holds only messages and the managed is_last_step, with a comment listing other fields one could add. langgraph.json serves the compiled graph as agent, so Studio and the API read whatever the state exposes. The unit tests under tests/unit_tests/ cover the context only; the one integration test asserts a substring of the final message's content, which is exactly the fragility a schema removes.
It then adds a respond node after the loop. A pydantic model in a new schemas.py names the fields the caller wants, say an answer, a list of sources and a confidence; State gains an optional structured_response of that type; and respond calls the context's model with with_structured_output over the last few messages and writes the result into that field. route_model_output returns respond instead of __end__ when there are no tool calls, and respond has an edge to __end__. The is_last_step branch in call_model, which answers with an apology when the step budget runs out, now also flows into respond, so a caller always gets the schema, with the apology in answer.
Two things come from reading rather than from the API reference. The model is loaded through load_chat_model from a provider/model string in Context, so respond reuses that loader rather than naming a provider, and the structured call is made on the unbound model, since tools are not wanted at that step. And InputState is deliberately narrower than State; the new field goes on State only, so the input contract to the graph does not change and Studio's input form stays the same.
In its sandbox the agent runs uv sync, ruff check ., mypy --strict src/ and pytest tests/unit_tests as .github/workflows/unit-tests.yml does. It writes unit tests that need no API key: one invokes the compiled graph with the loader patched to a GenericFakeChatModel whose scripted answer has no tool calls, and a respond whose structured call is patched to return a known schema instance, asserting the field on the result; one asserts the is_last_step path still ends with a populated schema; one checks mypy accepts the new field's type by running on the test module too. The daily integration test is left as it is.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add structured output to this LangGraph agent. Read src/react_agent/graph.py, state.py, context.py, utils.py, langgraph.json, tests/unit_tests/ and .github/workflows/unit-tests.yml first. Add a pydantic schema in src/react_agent/schemas.py with answer, sources and confidence, an optional structured_response field of that type on State but not InputState, and a respond node that loads the context's model through load_chat_model, calls with_structured_output on it over the recent messages, and writes the result to that field. Route call_model to respond instead of __end__ when there are no tool calls, including the is_last_step path, and respond to __end__. Do not change the tools or the prompts. Run uv sync, ruff check, mypy --strict src/ and pytest tests/unit_tests, add unit tests that patch the loader with GenericFakeChatModel and the structured call with a fixed instance to assert the field on the graph's result on both paths, and open a pull request quoting the output.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that adds schemas.py and the unit tests, and changes state.py (one field), graph.py (the respond node, the routing change and one edge) and the README's section on the graph's output. The description quotes the ruff, mypy --strict and pytest output, shows the schema as the caller will read it from result["structured_response"], and says that messages is still there for anyone reading the free-text answer.
The run's record keeps every command and its output, so a reviewer can read the test that drove the graph end to end with fake models and the assertion on the populated schema, with no provider call made.
Before merging, a reviewer settles the schema's fields, since the agent chose three to prove the path, and decides whether confidence should be a bounded float with a validator or dropped, and whether the integration test should now assert on the schema instead of a substring of the text.
Run it on every customer request
This is the change that lets a product call the agent and read the answer as data. Once the schema is in, your server starts one run per customer request with POST /o/{org}/sessions, the request in input, and reads result from the session or the run.completed webhook; the structured answer is what your code parses, with messages kept for the audit trail.
Send a request_id that is a UUID v5 of your own request id so a retried call returns the run it started rather than a second one, and a key at its max_concurrent limit queues the rest in order, so a burst needs no retry loop.
The use case this job follows, with its automation and trigger: Run an agent per customer task.
Why run it on Rush?
Structured output in a hand-built graph is a routing change, a state change and a schema at once, and the wrong version runs: a with_structured_output call on the tool-bound model, a field put on InputState so every caller must send it, a step-budget path that still ends in free text. In a Rush sandbox the agent makes the change under the repository's mypy --strict and drives the compiled graph with fake models on both ending paths before anyone merges.
The test output and the type check stay with the run and the pull request quotes them, so the reviewer reads the assertion on structured_response rather than a description of the node. Across several agents the job runs in parallel, one sandbox each, each with its own proof.
Frequently asked questions
Does the free-text answer go away?
No. messages keeps the full conversation including the final AIMessage; structured_response is added beside it. A caller that reads text today keeps working, and one that wants data reads the new field.
Why a separate respond node rather than binding the schema to call_model?
Because call_model is bound to the tools, and a model asked for both tool calls and a schema on every step does neither well. The loop stays as it is and the schema is produced once, at the end, from the model with no tools attached.
What happens when the step budget runs out?
The existing is_last_step branch writes an apology message, and the pull request routes that path through respond too, so the caller still gets the schema with the apology in answer instead of a bare message.
Does this need a particular model?
with_structured_output needs a model that supports tool calling or JSON mode, which the Anthropic and OpenAI models the context already names both do. The node loads whatever Context.model says, so switching models is still a config change.