All jobs

Run Sphinx Doctests in CI for a Python Library

Yes. An agent turns the >>> examples in your Sphinx docs into doctests, runs them in its sandbox, fixes the ones that fail and adds the check to CI.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

In a library documented like pallets/markupsafe, the docs are Sphinx sources under docs/, built by the docs tox environment with sphinx-build -E -W -b dirhtml docs docs/_build/dirhtml and on Read the Docs by the command in .readthedocs.yaml. docs/conf.py loads sphinx.ext.autodoc, extlinks, intersphinx, sphinxcontrib.log_cabinet and the Pallets theme, and nothing that runs examples. The examples are of three kinds: docs/index.rst has a bare >>> block that escapes a string and adds to it; docs/formatting.rst and docs/html.rst put theirs in .. code-block:: pycon directives that follow a .. code-block:: python defining a User or Image class; and the docstrings of escape_silent, soft_str, Markup.unescape and Markup.striptags in src/markupsafe/__init__.py carry >>> examples that autodoc copies into the rendered pages. None of them has been executed by anything.

The agent adds sphinx.ext.doctest to extensions and a doctest_global_setup that imports escape, escape_silent, soft_str and Markup, so the bare block in index.rst and the docstring examples run with their names bound. The doctest builder ignores code-block directives, so in formatting.rst and html.rst it turns each pycon block into a .. doctest:: and each class definition before it into a .. testcode::, which keeps the class visible on the page while also running it. The two files define different User classes, one with __html_format__ and one with only __html__; doctest namespaces are per document, so they do not collide, and the agent says so in the pull request rather than renaming one.

Then it runs uv run --group docs sphinx-build -b doctest docs docs/_build/doctest in its sandbox and reads the report. Two things it read in the sources are bound to show up. In formatting.rst the expected output of Markup("<p>User: {user:link}").format(user=user) is written without its closing quote and parenthesis, so the comparison can never match as printed; and in html.rst the User.__html__ method builds a link under /user/ while the expected output below it reads /users/3. The agent corrects the expected text to what the code produces in both places, re-runs the builder, and keeps iterating until the summary reports no failures. The _speedups C extension is built by uv sync, so the examples run against the same package a user installs.

Last it adds the doctest build as a second command of the docs tox environment in pyproject.toml, after the dirhtml build, and a docs job to .github/workflows/tests.yaml that runs uv run --locked --no-default-groups --group dev tox run -e docs. That workflow currently lists docs/** under paths-ignore for both triggers, which made sense while a docs change could not break a test; the agent removes that entry so a pull request that edits an example runs the job, and leaves .readthedocs.yaml alone, since Read the Docs only needs the HTML build.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Make the >>> examples in this project's Sphinx documentation run as doctests and add them to CI. Read docs/conf.py, every .rst under docs/, the docstrings in the package under src/, pyproject.toml (the dependency groups and the tox docs environment), .readthedocs.yaml and .github/workflows/tests.yaml first. Enable sphinx.ext.doctest with a doctest_global_setup that imports the public names the examples use. Convert pycon code-blocks to doctest directives and the python code-blocks that define their helper classes to testcode directives, keeping the rendered pages the same. Run sphinx-build -b doctest in the sandbox, fix every example whose expected output does not match what the code produces, and re-run until the report is clean. Add the doctest build to the docs tox environment, add a docs job to the tests workflow, and remove docs/** from its paths-ignore. Do not change the Read the Docs configuration. Open a pull request whose description quotes the doctest summary before and after your fixes and lists each example you corrected with the old and new output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that changes docs/conf.py, docs/formatting.rst, docs/html.rst, pyproject.toml and .github/workflows/tests.yaml. The description quotes the doctest builder's summary from the first run and from the last, lists each example whose expected output was corrected with the old and new text side by side, and shows the dirhtml build still passing with -W, so the conversion from code-block to doctest and testcode produced no new warning.

The run's record keeps every builder invocation, including the first report with its failures, so a reviewer can see which examples were wrong in the documentation as published, not only the fixed versions.

Before merging, a reviewer opens the two rendered pages from the record to confirm the class definitions still appear where they did, checks that each corrected output is what the code does and not what the prose claims, and decides whether the job should run on every pull request or only on those that touch docs/ or src/.

Run it on every Python library with Sphinx docs

A team that maintains several libraries sends this body once per repository from a script, with the repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started. GET /o/{org}/github lists the repositories the Rush GitHub App reaches; the script keeps the ones with a docs/conf.py.

Each run reads its own project. One that already loads sphinx.ext.doctest gets only the missing conversions and the CI job; one whose docs build is a make html in docs/Makefile rather than a tox environment gets a make doctest target instead; one with no examples at all comes back with a short answer and no pull request. Give the key a webhook URL and Rush POSTs run.completed with each session_url as the runs finish.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

Documentation examples drift because nothing executes them: a method is renamed, an output format changes, a class in the prose stops matching the class in the example, and the page keeps saying what it said. Running the doctest builder once on a laptop finds the drift but leaves the fixing, the directive conversions and the CI wiring half done. In a Rush sandbox the agent installs the project with its lockfile, builds the real docs, runs the real doctest builder and reads its report, then iterates until it is clean.

The record keeps every builder run, so a reviewer sees the failures as they were, the fixes that answered them and the clean summary that followed. Across many libraries the same job runs in parallel, each with its own interpreter, its own build and its own pull request, and each pull request carries its own evidence.

Frequently asked questions

Why convert the code-blocks instead of leaving them?

The doctest builder only collects bare >>> blocks and doctest directives. A .. code-block:: pycon is rendered the same way but never run, which is how the examples drifted in the first place. The doctest and testcode directives render as before and are executed on every build.

Will the docstring examples run too?

Yes. Autodoc copies them into the pages, and the builder treats their >>> lines like any other doctest block. They need the names they use to exist, which is what doctest_global_setup provides; without it every docstring example fails with a NameError.

Does this change what Read the Docs builds?

No. .readthedocs.yaml runs its own sphinx-build -W -b dirhtml command and is left alone. The doctest build runs in the tox environment and in CI, where a failure blocks a pull request instead of a documentation deploy.

What if an example needs network or a file?

The agent leaves such an example out of the run with a :skipif: or a testsetup that provides what it needs, and says which in the pull request. A library that escapes strings has no such example; a client library might.

Will a failing example block pull requests that only touch code?

Yes, by design. The examples describe the code, so a code change that alters an output should update its example in the same pull request. That is why docs/** is removed from paths-ignore rather than the job being limited to documentation changes.

More Sphinx jobs

More jobs like this: Hand off a long task

Keep reading