# Add a Docs Build Check for a Sphinx Site on Every Pull Request

Yes. An agent adds a GitHub Actions job that builds your Sphinx docs with warnings as errors on every pull request, matching your existing workflows, in a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-docs-build-check/sphinx/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent starts from how the docs are built today, because a check that uses a different command than the maintainers do will fail for reasons nobody cares about. In a project laid out like pallets/click, the documentation is 37 Markdown files under `docs/`, read by Sphinx through `myst_parser`; `docs/conf.py` also loads `sphinx.ext.autodoc`, `sphinx.ext.intersphinx` with a mapping to the Python docs, `sphinx_tabs.tabs`, `sphinxcontrib.log_cabinet` and `pallets_sphinx_themes`, with `html_theme = "click"`. `pyproject.toml` keeps the build's packages in a `docs` dependency group (`myst-parser`, `pallets-sphinx-themes`, `sphinx`, `sphinx-tabs`, `sphinxcontrib-log-cabinet`) and a tox environment, `[tool.tox.env.docs]`, whose command is `sphinx-build -E -W -b dirhtml docs docs/_build/dirhtml`. `.readthedocs.yaml` runs almost the same line, `uv run --group docs sphinx-build -W -b dirhtml docs $READTHEDOCS_OUTPUT/html`, on Ubuntu 24.04 with Python 3.13.

Then it reads `.github/workflows/` and finds that nothing there builds the docs. `tests.yaml` runs tox on a matrix of Python versions, with `paths-ignore: ['docs/**', 'README.md']`, so a pull request that touches only the docs starts no job at all; `pre-commit.yaml` runs the hooks; `zizmor.yaml` lints every workflow file; `nightly.yaml`, `lock.yaml` and `publish.yaml` do not read `docs/`. A broken cross-reference or a `-W` warning is first seen on Read the Docs, after merge. That gap is the job.

The agent adds `.github/workflows/docs.yaml`, written in the shape of the workflows beside it: `permissions: {}` at the top, a `concurrency` group keyed on the pull request number, `actions/checkout` with `persist-credentials: false`, `astral-sh/setup-uv` with its cache, and `actions/setup-python` with `python-version-file: pyproject.toml`, each action pinned to the same commit SHA and version comment the other files use. Its one step is `uv run --locked --no-default-groups --group dev tox run -e docs`, the same invocation `tests.yaml` uses for its matrix, so the check builds with the `-E -W -b dirhtml` flags the maintainers already chose and fails on any warning. It triggers on `pull_request` and on pushes to `main` and `stable`, with a `paths` filter for `docs/**`, `src/**`, `pyproject.toml`, `uv.lock` and the workflow itself, since autodoc reads docstrings from `src/` and a signature change there can break a reference in `docs/api.md`.

Before opening the pull request the agent runs the same command in its sandbox and reads the output, so the first run of the check is known to pass on the current tree rather than hoped to. It also runs `zizmor` on the new file, as `zizmor.yaml` will, and keeps the workflow free of anything that linter flags. It adds a line to `docs/contributing.md` only where that page already explains how to run tox, naming the new environment and nothing else.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add a GitHub Actions check that builds this project's Sphinx documentation on every pull request and fails on warnings. Read pyproject.toml (the docs dependency group and the tox docs environment), docs/conf.py, .readthedocs.yaml and every file in .github/workflows first, and note that tests.yaml ignores docs/** so docs-only pull requests currently run nothing. Add .github/workflows/docs.yaml in the same style as the existing workflows: permissions: {}, the same concurrency pattern, the same actions pinned to the same commit SHAs with their version comments, persist-credentials: false, uv with its cache, and setup-python reading python-version-file from pyproject.toml. Run the docs build through the existing tox environment with uv run --locked --no-default-groups --group dev tox run -e docs so the -E -W dirhtml flags stay in one place. Trigger it on pull requests and on pushes to main and stable, filtered to docs/**, src/**, pyproject.toml, uv.lock and the workflow file. Run that exact command in your sandbox first and paste its last lines into the pull request, run zizmor on the new workflow, and mention the new environment in docs/contributing.md only where tox is already described. Change no other workflow.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

The pull request adds one file, `.github/workflows/docs.yaml`, and touches `docs/contributing.md` if that page lists the tox environments. Its description shows the tail of the `sphinx-build` output from the sandbox, ending in the line that says the build succeeded, and the `zizmor` result for the new workflow.

On the pull request itself the new check appears beside the `pre-commit` and `zizmor` jobs, and because it changed only a workflow file and a docs page, `tests.yaml` stays skipped, which is the behaviour the check was added to complement. From then on a pull request that breaks a cross-reference, misses a `toctree` entry or leaves an autodoc target unresolved gets a red check with the Sphinx warning in the log, before Read the Docs ever builds it.

A reviewer checks three things: that the pinned action SHAs match the ones in `tests.yaml`, that the `paths` filter covers the directories whose content reaches the docs (`src/` for autodoc, not only `docs/`), and that the build runs through tox rather than a copied `sphinx-build` line, so a future change to the flags in `pyproject.toml` applies to the check without a second edit.

## Run it on every documentation site without a pull request build

An organisation with several Sphinx projects sends this body once per repository to `POST /o/{org}/sessions`, each with its own repository in `resources` and a `request_id` that is a UUID v5 of the repository's full name, so a retried call returns the run already started instead of a second pull request. Each run reads its own project: one builds through tox, another through a `make html` target in `docs/Makefile`, a third has a `requirements.txt` under `docs/` instead of a dependency group, and the workflow each run writes follows the convention it finds rather than a template.

A `webhook_url` on the key reports each `run.completed` with the pull request URL in `result`. A `run.failed` is worth reading too: it usually means the docs do not build warning-free today, which is the finding the check exists to surface.

## Why run it on Rush?

The hard part of this job is not the YAML. It is matching a project's conventions closely enough that the maintainers accept the file: the SHA pins, the `permissions: {}` line, the `uv` flags, the tox environment instead of a bare `sphinx-build`, the security linter that runs on every workflow change. The agent reads all of that from the repository and from the workflows beside the new one, and it proves the command passes by running it in its own sandbox before anyone sees the pull request.

The run keeps every command it ran and what came back, so the review is about the trigger filter and the flags, not about whether the build works. Many repositories can get the same check in one batch, each in its own sandbox, each with its own build output in its own pull request.

## Frequently asked questions

### Why run tox instead of calling sphinx-build directly in the workflow?

Because the project already defines the docs build once, in `[tool.tox.env.docs]`, with `-E -W -b dirhtml`. A workflow that repeats those flags drifts from the local command the next time someone changes them. Running `tox run -e docs` keeps the check and the maintainer's local build identical.

### Does the check need network access?

Yes, for two reasons: `uv` installs the `docs` dependency group from the lockfile, and `sphinx.ext.intersphinx` downloads the Python documentation inventory named in `docs/conf.py`. GitHub-hosted runners have that access. With `-W` a failed inventory fetch is an error, so a long outage of docs.python.org would show up as a red check rather than a silent gap.

### Will it slow down pull requests that do not touch the docs?

The workflow carries a `paths` filter, so a pull request that changes neither `docs/`, `src/`, `pyproject.toml`, `uv.lock` nor the workflow itself does not start it. `src/` is included on purpose: autodoc reads docstrings and signatures from the package, and a rename there can break a reference in `docs/api.md`.

### What about Read the Docs?

Nothing changes there. `.readthedocs.yaml` keeps building the published site with its own `sphinx-build -W` line. The new job is the earlier signal, on the pull request, with the same strictness; a project that also enables pull request previews on Read the Docs gets a rendered preview as well, but that is a setting on their side, not a file in the repository.

### Can it also check for broken external links?

Sphinx has a `linkcheck` builder for that, but it is slow and fails on sites that rate-limit or block bots, so the agent keeps it out of the pull request check. A separate scheduled workflow running `sphinx-build -b linkcheck` once a week is the usual place for it, and the agent can add one when asked.

## More Sphinx jobs

- [Run Sphinx Doctests in CI for a Python Library](https://getrush.ai/run/add-doctest-to-ci/sphinx/)
- [Add i18n to a Sphinx Docs Site](https://getrush.ai/run/add-i18n/sphinx/)
- [Add Last Updated Dates to Sphinx Docs](https://getrush.ai/run/add-last-updated-dates/sphinx/)

## The same job on other stacks

- [Add a Build Check for a VitePress Docs Site on Every Pull Request](https://getrush.ai/run/add-docs-build-check/vitepress/)
- [Add a Build Check for a Docusaurus Site on Every Pull Request](https://getrush.ai/run/add-docs-build-check/docusaurus/)
- [Build an mdBook on Every Pull Request Before It Deploys](https://getrush.ai/run/add-docs-build-check/mdbook/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
