What the agent does
Most agent repositories have one honest test: it calls a real model and checks a word in the answer. The agent starts by finding what sits between that and nothing. In a LangGraph project like langchain-ai/react-agent, src/react_agent/graph.py builds a two-node loop: call_model binds TOOLS to the chat model named in the run's context and invokes it, route_model_output sends the result to tools when the last message has tool calls and to __end__ when it has none, and a branch in call_model swaps in an apology message when state.is_last_step is true and the model still wants a tool. The tests cover little of it. tests/unit_tests/test_configuration.py has three tests of the Context dataclass, and tests/integration_tests/test_graph.py has one test that asks who founded LangChain and looks for a name, which needs an Anthropic key and a Tavily key and so runs only in a daily workflow that has them.
The pure functions come first, since they need no model at all. route_model_output gets three cases: an AIMessage with tool calls, one without, and a last message that is not an AIMessage, where the code raises ValueError. get_message_text in utils.py handles string, dict and list content and gets a case for each. load_chat_model splits its argument on the first slash only, so a test patches init_chat_model and checks that a model name containing slashes survives intact. Context.__post_init__ replaces any field still at its default with the environment variable of the same name in upper case; the new tests set variables through monkeypatch, where the existing ones assign os.environ["MODEL"] and never clear it, and they pin two things the code does today: MAX_SEARCH_RESULTS arrives as a string, and an argument passed explicitly but equal to the default is still overridden.
Then the loop itself, without the network. call_model looks up load_chat_model in its own module at call time, so a test can replace it with a function that returns a scripted model. In the langchain-core version this project locks, the fake chat models do not implement bind_tools, which the node calls, so the test defines a small subclass of GenericFakeChatModel whose bind_tools returns itself. The script is two messages: an AIMessage asking for the search tool, then a plain answer. search in tools.py constructs TavilySearch when it is called, so patching that name with a stub that returns a fixed result keeps the tool node real and the HTTP call out. The test awaits graph.ainvoke with a Context and asserts the message sequence, human, model with tool call, tool result, model, and that the stub received max_search_results from the context. A second test repeats it with a low recursion_limit and a model that always asks for a tool, and expects the apology text.
The new tests follow the suite's conventions. Async tests carry pytest.mark.anyio, the marker the integration test uses with the anyio_backend fixture in tests/conftest.py, and no new plugin is added. Every test function is annotated so that mypy --strict passes on the new files as well as on src/, which is all CI type-checks. The agent installs the project the way .github/workflows/unit-tests.yml does, with uv pip install -r pyproject.toml, and runs ruff check ., mypy --strict src/ and pytest tests/unit_tests on Python 3.11 and 3.12, the two versions in the workflow's matrix. It does not run or edit the integration test.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Write unit tests for this LangGraph agent that pass with no API keys. Read pyproject.toml, the Makefile, .github/workflows, tests/conftest.py, the existing tests and every file under src/react_agent first. Add tests under tests/unit_tests for: route_model_output (tool calls, no tool calls, a last message that is not an AIMessage); get_message_text for string, dict and list content; load_chat_model, patching init_chat_model and checking the provider and model it receives, including a model name that contains slashes; and Context, setting environment variables with monkeypatch and pinning the current behaviour for MAX_SEARCH_RESULTS and for an explicit argument equal to the default. Then test the compiled graph end to end: patch load_chat_model in the graph module to return a scripted fake chat model (check the installed langchain-core for one that supports bind_tools and subclass GenericFakeChatModel if none does), patch TavilySearch in the tools module with a stub, await graph.ainvoke with a Context, and assert the message sequence and that the stub received max_search_results. Add a test for the is_last_step branch using a low recursion_limit. Use pytest.mark.anyio like the existing async test and add no plugins. Annotate every test. Do not change src/ and do not run or edit tests/integration_tests; if a test shows behaviour that looks wrong, keep the test, mark the behaviour in the pull request and leave the fix to us. Run ruff check ., mypy --strict src/, mypy --strict on the new test files and pytest tests/unit_tests on Python 3.11 and 3.12. Open a pull request listing each test and what it covers, with the output of each command.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
A pull request that adds test files under tests/unit_tests and changes nothing in src/. The description lists each test with the function or path it covers, and pastes the output of ruff check ., both mypy --strict passes and pytest tests/unit_tests for both Python versions. Because none of the new tests reads a key, the repository's existing unit-test workflow runs them on the pull request as it stands.
A second list in the description holds behaviour the tests pinned that a maintainer may not want: an integer setting that becomes a string when it comes from the environment, and an explicit default that the environment overrides. The agent writes the test for what the code does and flags it; it does not decide for the team that the code is wrong.
A reviewer checks that the fake model's script matches how the real graph is used, since a test that scripts two turns says nothing about a model that asks for three tools at once, and reads the stub for TavilySearch to confirm it returns the same shape the real tool does. The integration test still has its job: it is the only test in which a real model chooses the tool call.
Run it on every pull request labelled needs-tests
An automation in the Project that holds the repository, with a github trigger on pull_request.labeled and needs-tests in its labels, starts one run when a maintainer adds that label. Its prompt asks for unit tests of the nodes, tools and routing functions the pull request changes, written with the same fake model and stubs. The trigger puts the pull request after the prompt and fills in nothing; the run opens its own pull request with the tests, which the author can merge into their branch.
To cover a set of agent repositories in one pass, a script on your server sends this body to POST /o/{org}/sessions once per repository, each with a request_id that is a UUID v5 of the repository's full name so a retry starts nothing new. A webhook_url on the key receives run.completed with session_url for every run the key started.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Tests for an agent are easy to get subtly wrong: a fake that does not implement the method the node calls, a patch applied to the module where a name is defined when the code looks it up somewhere else, an environment variable leaking from one test into the next. Finding those takes running the suite, reading the failure and trying again, on each Python version the project supports. The agent does that loop in its own sandbox and the pull request arrives with the passing output of both versions.
No model key enters the picture. The tests it writes call no provider, the run does not execute the integration suite, and the commands with their output are kept in the session's transcript at GET /o/{org}/sessions/{id}/messages, so a reviewer can see that the suite passed with no key in the environment.
Frequently asked questions
Do these tests call a model?
No. The chat model is a scripted fake that returns the messages the test gives it, and the search tool's client is replaced with a stub. That makes the suite fast and repeatable, and it means the tests check the graph's wiring and your code, not the quality of a model's answers.
Can the run execute our integration test as well?
Not as this page sets it up. That test needs ANTHROPIC_API_KEY and TAVILY_API_KEY. A Project secret reaches a run as an environment variable, but Rush refuses secret names it sets itself, ANTHROPIC_* among them, with 422 SECRET_NAME_INVALID. So the run leaves the integration test to the repository's scheduled workflow, where the keys already are.
Why subclass the fake chat model?
call_model calls bind_tools on whatever load_chat_model returns, and the fake chat models in the langchain-core version this project locks raise NotImplementedError there. A small subclass that returns itself from bind_tools is enough. The agent checks the installed version first and uses a built-in fake if a newer release provides one.
What about the recorded cassette under tests/cassettes?
It is left alone. The file was recorded against claude-3-5-sonnet-20240620, while Context now defaults to anthropic/claude-sonnet-4-5-20250929, and the pull request mentions the mismatch so the team can decide whether to record it again.
More LangGraph jobs
The same job on other stacks
- Write Pytest Tests for a Litestar App (Litestar)
- Write Pytest Tests for a Reflex App (Reflex)
- Write Pytest Tests for a Textual Terminal App (Textual)