# Run Docstring Examples as Tests in CI With pytest

Yes. An agent runs the >>> examples in your package and docs as pytest doctests against the installed wheel in tox and CI, fixes what drifted, opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-doctests-to-ci/pytest/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent counts the examples before it runs them, because a docstring example that no test executes drifts silently. In a library laid out like pallets/markupsafe (3.1.0.dev), `src/markupsafe/__init__.py` holds 15 `>>>` lines across the docstrings of `escape_silent`, `soft_str`, the `Markup` class, `unescape` and `striptags`, showing calls like `Markup("Hello, <em>World</em>!")` and their `Markup('…')` reprs. `README.md` has 10 more and `docs/index.rst`, `docs/formatting.rst` and `docs/html.rst` another 14. `pyproject.toml` sets `testpaths = ["tests"]` and `filterwarnings = ["error"]`, so pytest collects only the five files under `tests/` and none of those examples run anywhere, not locally, not in the tox envs, not in the nine-row matrix of `.github/workflows/tests.yaml`.

Where the doctests run from matters in this package. `__init__.py` imports `_escape_inner` from `._speedups`, the C extension built by `setup.py`, and falls back to `._native`. Tox is configured in `[tool.tox]` with `package = "wheel"`, so the tests run against an installed wheel that has the extension, and `tests/conftest.py` swaps `markupsafe._escape_inner` between both implementations with a session-scoped parametrised fixture. Pointing `--doctest-modules` at `src/markupsafe` would import the source tree, where no extension exists, and shadow the installed package. The agent instead adds a second command to the tox env: `pytest --doctest-modules --pyargs markupsafe`, which collects the docstrings from the installed package, extension included, plus `pytest --doctest-glob='*.rst' --doctest-glob='README.md' docs README.md` for the prose examples. The fixture in `tests/conftest.py` does not apply to items collected outside `tests/`, so the doctests run once, on whichever implementation the wheel imported, and the pull request says so.

Then it runs them and reads what fails, because some examples were written to be read, not executed. An `.rst` example that omits its imports, shows elided output, or relies on a variable from an earlier block fails under doctest; the agent fixes the ones that are one line away from passing, adds `# doctest: +SKIP` with a reason to the ones that are illustrations, and leaves the docstrings in `__init__.py` as they are unless one has drifted from the code, in which case the drift is the finding and the pull request quotes it. `filterwarnings = error` applies to doctests too, so an example that triggers a `DeprecationWarning` fails and is reported rather than muted.

The CI change is nothing beyond the tox command, because `tests.yaml` already runs `uv run --locked --no-default-groups --group dev tox run` with `TOX_ENV` per matrix row, and the `parallel` env on the free-threaded build keeps its own command, since `pytest-run-parallel` and doctests are a separate question. The agent runs `tox run -e py3.14` or the interpreter it has, reads the collected count, which should rise by the number of examples that are not skipped, and runs `tox run -e style` for ruff and the pre-commit hooks before opening the pull request.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Run the docstring and documentation examples as doctests in CI. Read pyproject.toml, tests/conftest.py, the files under tests/, src/markupsafe/__init__.py, setup.py, README.md, the .rst files under docs/ and .github/workflows/tests.yaml first, and count the >>> examples in each. Because tox installs a wheel with the C extension, collect the package doctests with pytest --doctest-modules --pyargs markupsafe rather than from src/, and collect README.md and docs/*.rst with --doctest-glob. Add the commands to the tox env that CI runs. Run them, fix examples that are one line from passing, mark illustrations with doctest: +SKIP and a reason, and report any docstring that has drifted from the code without changing the code. Run tox -e style and open a pull request quoting the collected counts before and after and the output of the doctest run.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that changes the tox command in `pyproject.toml` and whichever examples needed a line, with the count of examples per file in its description, the pytest output showing the doctest items collected and passing against the installed wheel, the list of examples marked as skipped with each reason, and the `style` env output.

The run's record keeps the tox run, so a reviewer can see the collected count rise and read any example that failed on the first attempt before the agent fixed or skipped it.

Before merging, a reviewer decides whether the docs examples belong in the `docs` tox env instead of the test env, since the docs build with `sphinx-build -W` and a failing example could sit next to a broken cross-reference, and whether the conftest fixture should be extended so doctests run on both `_native` and `_speedups` too, which the pull request names as a follow-up.

## Run it on every pull request that changes a docstring

Once the tox command exists, CI runs every example on every push. An automation in the Project with a `github` trigger on `pull_request.opened` and `pull_request.synchronize` can run a prompt that reads the pull request's changes under `src/` and `docs/`, finds a public function whose docstring has no `>>>` example, and posts a review comment proposing one from the function's tests. The trigger adds the event after the prompt and fills in nothing else.

A team that maintains several libraries starts one run per repository from a script, `POST /o/{org}/sessions` with this `input` and the repository in `resources`, and a `request_id` that is a UUID v5 of the repository's full name so a retried call returns the run already started. Each run reads that library's own test setup, so a pure-Python package without a wheel step gets `--doctest-modules` on its source directory instead.

## Why run it on Rush?

The hard part is not the flag but where it points: a package with a C extension and a wheel-based tox setup runs its doctests on the wrong implementation if the flag is pointed at `src/`, and the examples pass while the shipped code is untested. The agent reads the import fallback, the tox packaging and the conftest, picks `--pyargs`, and proves it by running the env in its own sandbox with the output kept.

Across many libraries the runs go in parallel, one sandbox each, and every pull request carries that library's own counts and doctest output rather than a config copied from another.

## Frequently asked questions

### Why --pyargs instead of pointing at src/?

Tox installs the package as a wheel with the compiled `_speedups` extension and the tests import that. Collecting doctests from `src/markupsafe` would import the source tree instead, which has only the pure-Python fallback, so the examples would run against code the wheel does not ship. `--pyargs markupsafe` collects from the installed package.

### Will the docs examples that show partial output pass?

Not as written: doctest compares output exactly. Examples that need `+ELLIPSIS` or `+NORMALIZE_WHITESPACE` get the directive, ones that are illustrations get `+SKIP` with a reason, and the pull request lists each decision so a reviewer can disagree with any one of them.

### Does filterwarnings = error affect doctests?

Yes. It is a pytest setting and doctest items are pytest items, so an example that triggers a warning fails. That is a feature: an example that calls a deprecated path is exactly the drift the change is meant to catch.

### Does this slow the CI matrix down?

By the time it takes to run a few dozen one-line examples, which is well under a second per row. The free-threaded `parallel` env keeps its own command and is not changed.

## More pytest jobs

- [Add Mutation Testing to a pytest Suite](https://getrush.ai/run/add-mutation-testing/pytest/)
- [Run a pytest Suite in Parallel With pytest-xdist](https://getrush.ai/run/add-parallel-tests/pytest/)
- [Add Property-Based Tests with Hypothesis to a pytest Suite](https://getrush.ai/run/add-property-tests/pytest/)

## More jobs like this: Write the missing tests

- [Add Playwright End-to-End Tests to a Chrome Extension](https://getrush.ai/run/add-e2e-tests/chrome-extension/)
- [Add End-to-End Tests to a Tauri App](https://getrush.ai/run/add-e2e-tests/tauri/)
- [Add libFuzzer Fuzz Tests to a CMake Project](https://getrush.ai/run/add-fuzz-tests/cmake/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
