# Run Sphinx Doctests in CI for a Python Library

Yes. An agent turns the >>> examples in your Sphinx docs into doctests, runs them in its sandbox, fixes the ones that fail and adds the check to CI.

[Run this on Rush](https://cloud.getrush.ai/run/add-doctest-to-ci/sphinx/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

In a library documented like pallets/markupsafe, the docs are Sphinx sources under `docs/`, built by the `docs` tox environment with `sphinx-build -E -W -b dirhtml docs docs/_build/dirhtml` and on Read the Docs by the command in `.readthedocs.yaml`. `docs/conf.py` loads `sphinx.ext.autodoc`, `extlinks`, `intersphinx`, `sphinxcontrib.log_cabinet` and the Pallets theme, and nothing that runs examples. The examples are of three kinds: `docs/index.rst` has a bare `>>>` block that escapes a string and adds to it; `docs/formatting.rst` and `docs/html.rst` put theirs in `.. code-block:: pycon` directives that follow a `.. code-block:: python` defining a `User` or `Image` class; and the docstrings of `escape_silent`, `soft_str`, `Markup.unescape` and `Markup.striptags` in `src/markupsafe/__init__.py` carry `>>>` examples that autodoc copies into the rendered pages. None of them has been executed by anything.

The agent adds `sphinx.ext.doctest` to `extensions` and a `doctest_global_setup` that imports `escape`, `escape_silent`, `soft_str` and `Markup`, so the bare block in `index.rst` and the docstring examples run with their names bound. The doctest builder ignores `code-block` directives, so in `formatting.rst` and `html.rst` it turns each `pycon` block into a `.. doctest::` and each class definition before it into a `.. testcode::`, which keeps the class visible on the page while also running it. The two files define different `User` classes, one with `__html_format__` and one with only `__html__`; doctest namespaces are per document, so they do not collide, and the agent says so in the pull request rather than renaming one.

Then it runs `uv run --group docs sphinx-build -b doctest docs docs/_build/doctest` in its sandbox and reads the report. Two things it read in the sources are bound to show up. In `formatting.rst` the expected output of `Markup("<p>User: {user:link}").format(user=user)` is written without its closing quote and parenthesis, so the comparison can never match as printed; and in `html.rst` the `User.__html__` method builds a link under `/user/` while the expected output below it reads `/users/3`. The agent corrects the expected text to what the code produces in both places, re-runs the builder, and keeps iterating until the summary reports no failures. The `_speedups` C extension is built by `uv sync`, so the examples run against the same package a user installs.

Last it adds the doctest build as a second command of the `docs` tox environment in `pyproject.toml`, after the `dirhtml` build, and a `docs` job to `.github/workflows/tests.yaml` that runs `uv run --locked --no-default-groups --group dev tox run -e docs`. That workflow currently lists `docs/**` under `paths-ignore` for both triggers, which made sense while a docs change could not break a test; the agent removes that entry so a pull request that edits an example runs the job, and leaves `.readthedocs.yaml` alone, since Read the Docs only needs the HTML build.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Make the >>> examples in this project's Sphinx documentation run as doctests and add them to CI. Read docs/conf.py, every .rst under docs/, the docstrings in the package under src/, pyproject.toml (the dependency groups and the tox docs environment), .readthedocs.yaml and .github/workflows/tests.yaml first. Enable sphinx.ext.doctest with a doctest_global_setup that imports the public names the examples use. Convert pycon code-blocks to doctest directives and the python code-blocks that define their helper classes to testcode directives, keeping the rendered pages the same. Run sphinx-build -b doctest in the sandbox, fix every example whose expected output does not match what the code produces, and re-run until the report is clean. Add the doctest build to the docs tox environment, add a docs job to the tests workflow, and remove docs/** from its paths-ignore. Do not change the Read the Docs configuration. Open a pull request whose description quotes the doctest summary before and after your fixes and lists each example you corrected with the old and new output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that changes `docs/conf.py`, `docs/formatting.rst`, `docs/html.rst`, `pyproject.toml` and `.github/workflows/tests.yaml`. The description quotes the doctest builder's summary from the first run and from the last, lists each example whose expected output was corrected with the old and new text side by side, and shows the `dirhtml` build still passing with `-W`, so the conversion from `code-block` to `doctest` and `testcode` produced no new warning.

The run's record keeps every builder invocation, including the first report with its failures, so a reviewer can see which examples were wrong in the documentation as published, not only the fixed versions.

Before merging, a reviewer opens the two rendered pages from the record to confirm the class definitions still appear where they did, checks that each corrected output is what the code does and not what the prose claims, and decides whether the job should run on every pull request or only on those that touch `docs/` or `src/`.

## Run it on every Python library with Sphinx docs

A team that maintains several libraries sends this body once per repository from a script, with the repository in `resources` and a `request_id` that is a UUID v5 of the repository's full name, so a retried call returns the run already started. `GET /o/{org}/github` lists the repositories the Rush GitHub App reaches; the script keeps the ones with a `docs/conf.py`.

Each run reads its own project. One that already loads `sphinx.ext.doctest` gets only the missing conversions and the CI job; one whose docs build is a `make html` in `docs/Makefile` rather than a tox environment gets a `make doctest` target instead; one with no examples at all comes back with a short answer and no pull request. Give the key a webhook URL and Rush POSTs `run.completed` with each `session_url` as the runs finish.

## Why run it on Rush?

Documentation examples drift because nothing executes them: a method is renamed, an output format changes, a class in the prose stops matching the class in the example, and the page keeps saying what it said. Running the doctest builder once on a laptop finds the drift but leaves the fixing, the directive conversions and the CI wiring half done. In a Rush sandbox the agent installs the project with its lockfile, builds the real docs, runs the real doctest builder and reads its report, then iterates until it is clean.

The record keeps every builder run, so a reviewer sees the failures as they were, the fixes that answered them and the clean summary that followed. Across many libraries the same job runs in parallel, each with its own interpreter, its own build and its own pull request, and each pull request carries its own evidence.

## Frequently asked questions

### Why convert the code-blocks instead of leaving them?

The doctest builder only collects bare `>>>` blocks and `doctest` directives. A `.. code-block:: pycon` is rendered the same way but never run, which is how the examples drifted in the first place. The `doctest` and `testcode` directives render as before and are executed on every build.

### Will the docstring examples run too?

Yes. Autodoc copies them into the pages, and the builder treats their `>>>` lines like any other doctest block. They need the names they use to exist, which is what `doctest_global_setup` provides; without it every docstring example fails with a `NameError`.

### Does this change what Read the Docs builds?

No. `.readthedocs.yaml` runs its own `sphinx-build -W -b dirhtml` command and is left alone. The doctest build runs in the tox environment and in CI, where a failure blocks a pull request instead of a documentation deploy.

### What if an example needs network or a file?

The agent leaves such an example out of the run with a `:skipif:` or a `testsetup` that provides what it needs, and says which in the pull request. A library that escapes strings has no such example; a client library might.

### Will a failing example block pull requests that only touch code?

Yes, by design. The examples describe the code, so a code change that alters an output should update its example in the same pull request. That is why `docs/**` is removed from `paths-ignore` rather than the job being limited to documentation changes.

## More Sphinx jobs

- [Add i18n to a Sphinx Docs Site](https://getrush.ai/run/add-i18n/sphinx/)
- [Add Last Updated Dates to Sphinx Docs](https://getrush.ai/run/add-last-updated-dates/sphinx/)
- [Check for Broken Links in Sphinx Docs with the linkcheck Builder](https://getrush.ai/run/add-link-checking/sphinx/)

## More jobs like this: Hand off a long task

- [Add Draft Posts to a Gatsby Blog](https://getrush.ai/run/add-draft-posts/gatsby/)
- [Add Draft Posts to a VitePress Blog](https://getrush.ai/run/add-draft-posts/vitepress/)
- [Add Drift Detection to Flux Helm Releases](https://getrush.ai/run/add-drift-detection/flux/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
