What the agent does
The agent counts the examples before it runs them, because a docstring example that no test executes drifts silently. In a library laid out like pallets/markupsafe (3.1.0.dev), src/markupsafe/__init__.py holds 15 >>> lines across the docstrings of escape_silent, soft_str, the Markup class, unescape and striptags, showing calls like Markup("Hello, <em>World</em>!") and their Markup('…') reprs. README.md has 10 more and docs/index.rst, docs/formatting.rst and docs/html.rst another 14. pyproject.toml sets testpaths = ["tests"] and filterwarnings = ["error"], so pytest collects only the five files under tests/ and none of those examples run anywhere, not locally, not in the tox envs, not in the nine-row matrix of .github/workflows/tests.yaml.
Where the doctests run from matters in this package. __init__.py imports _escape_inner from ._speedups, the C extension built by setup.py, and falls back to ._native. Tox is configured in [tool.tox] with package = "wheel", so the tests run against an installed wheel that has the extension, and tests/conftest.py swaps markupsafe._escape_inner between both implementations with a session-scoped parametrised fixture. Pointing --doctest-modules at src/markupsafe would import the source tree, where no extension exists, and shadow the installed package. The agent instead adds a second command to the tox env: pytest --doctest-modules --pyargs markupsafe, which collects the docstrings from the installed package, extension included, plus pytest --doctest-glob='*.rst' --doctest-glob='README.md' docs README.md for the prose examples. The fixture in tests/conftest.py does not apply to items collected outside tests/, so the doctests run once, on whichever implementation the wheel imported, and the pull request says so.
Then it runs them and reads what fails, because some examples were written to be read, not executed. An .rst example that omits its imports, shows elided output, or relies on a variable from an earlier block fails under doctest; the agent fixes the ones that are one line away from passing, adds # doctest: +SKIP with a reason to the ones that are illustrations, and leaves the docstrings in __init__.py as they are unless one has drifted from the code, in which case the drift is the finding and the pull request quotes it. filterwarnings = error applies to doctests too, so an example that triggers a DeprecationWarning fails and is reported rather than muted.
The CI change is nothing beyond the tox command, because tests.yaml already runs uv run --locked --no-default-groups --group dev tox run with TOX_ENV per matrix row, and the parallel env on the free-threaded build keeps its own command, since pytest-run-parallel and doctests are a separate question. The agent runs tox run -e py3.14 or the interpreter it has, reads the collected count, which should rise by the number of examples that are not skipped, and runs tox run -e style for ruff and the pre-commit hooks before opening the pull request.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Run the docstring and documentation examples as doctests in CI. Read pyproject.toml, tests/conftest.py, the files under tests/, src/markupsafe/__init__.py, setup.py, README.md, the .rst files under docs/ and .github/workflows/tests.yaml first, and count the >>> examples in each. Because tox installs a wheel with the C extension, collect the package doctests with pytest --doctest-modules --pyargs markupsafe rather than from src/, and collect README.md and docs/*.rst with --doctest-glob. Add the commands to the tox env that CI runs. Run them, fix examples that are one line from passing, mark illustrations with doctest: +SKIP and a reason, and report any docstring that has drifted from the code without changing the code. Run tox -e style and open a pull request quoting the collected counts before and after and the output of the doctest run.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that changes the tox command in pyproject.toml and whichever examples needed a line, with the count of examples per file in its description, the pytest output showing the doctest items collected and passing against the installed wheel, the list of examples marked as skipped with each reason, and the style env output.
The run's record keeps the tox run, so a reviewer can see the collected count rise and read any example that failed on the first attempt before the agent fixed or skipped it.
Before merging, a reviewer decides whether the docs examples belong in the docs tox env instead of the test env, since the docs build with sphinx-build -W and a failing example could sit next to a broken cross-reference, and whether the conftest fixture should be extended so doctests run on both _native and _speedups too, which the pull request names as a follow-up.
Run it on every pull request that changes a docstring
Once the tox command exists, CI runs every example on every push. An automation in the Project with a github trigger on pull_request.opened and pull_request.synchronize can run a prompt that reads the pull request's changes under src/ and docs/, finds a public function whose docstring has no >>> example, and posts a review comment proposing one from the function's tests. The trigger adds the event after the prompt and fills in nothing else.
A team that maintains several libraries starts one run per repository from a script, POST /o/{org}/sessions with this input and the repository in resources, and a request_id that is a UUID v5 of the repository's full name so a retried call returns the run already started. Each run reads that library's own test setup, so a pure-Python package without a wheel step gets --doctest-modules on its source directory instead.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
The hard part is not the flag but where it points: a package with a C extension and a wheel-based tox setup runs its doctests on the wrong implementation if the flag is pointed at src/, and the examples pass while the shipped code is untested. The agent reads the import fallback, the tox packaging and the conftest, picks --pyargs, and proves it by running the env in its own sandbox with the output kept.
Across many libraries the runs go in parallel, one sandbox each, and every pull request carries that library's own counts and doctest output rather than a config copied from another.
Frequently asked questions
Why --pyargs instead of pointing at src/?
Tox installs the package as a wheel with the compiled _speedups extension and the tests import that. Collecting doctests from src/markupsafe would import the source tree instead, which has only the pure-Python fallback, so the examples would run against code the wheel does not ship. --pyargs markupsafe collects from the installed package.
Will the docs examples that show partial output pass?
Not as written: doctest compares output exactly. Examples that need +ELLIPSIS or +NORMALIZE_WHITESPACE get the directive, ones that are illustrations get +SKIP with a reason, and the pull request lists each decision so a reviewer can disagree with any one of them.
Does filterwarnings = error affect doctests?
Yes. It is a pytest setting and doctest items are pytest items, so an example that triggers a warning fails. That is a feature: an example that calls a deprecated path is exactly the drift the change is meant to catch.
Does this slow the CI matrix down?
By the time it takes to run a few dozen one-line examples, which is well under a second per row. The free-threaded parallel env keeps its own command and is not changed.
More pytest jobs
More jobs like this: Write the missing tests
- Add Playwright End-to-End Tests to a Chrome Extension (Chrome extension)
- Add End-to-End Tests to a Tauri App (Tauri)
- Add libFuzzer Fuzz Tests to a CMake Project (CMake)