What the agent does
The agent starts from how the docs are built today, because a check that uses a different command than the maintainers do will fail for reasons nobody cares about. In a project laid out like pallets/click, the documentation is 37 Markdown files under docs/, read by Sphinx through myst_parser; docs/conf.py also loads sphinx.ext.autodoc, sphinx.ext.intersphinx with a mapping to the Python docs, sphinx_tabs.tabs, sphinxcontrib.log_cabinet and pallets_sphinx_themes, with html_theme = "click". pyproject.toml keeps the build's packages in a docs dependency group (myst-parser, pallets-sphinx-themes, sphinx, sphinx-tabs, sphinxcontrib-log-cabinet) and a tox environment, [tool.tox.env.docs], whose command is sphinx-build -E -W -b dirhtml docs docs/_build/dirhtml. .readthedocs.yaml runs almost the same line, uv run --group docs sphinx-build -W -b dirhtml docs $READTHEDOCS_OUTPUT/html, on Ubuntu 24.04 with Python 3.13.
Then it reads .github/workflows/ and finds that nothing there builds the docs. tests.yaml runs tox on a matrix of Python versions, with paths-ignore: ['docs/**', 'README.md'], so a pull request that touches only the docs starts no job at all; pre-commit.yaml runs the hooks; zizmor.yaml lints every workflow file; nightly.yaml, lock.yaml and publish.yaml do not read docs/. A broken cross-reference or a -W warning is first seen on Read the Docs, after merge. That gap is the job.
The agent adds .github/workflows/docs.yaml, written in the shape of the workflows beside it: permissions: {} at the top, a concurrency group keyed on the pull request number, actions/checkout with persist-credentials: false, astral-sh/setup-uv with its cache, and actions/setup-python with python-version-file: pyproject.toml, each action pinned to the same commit SHA and version comment the other files use. Its one step is uv run --locked --no-default-groups --group dev tox run -e docs, the same invocation tests.yaml uses for its matrix, so the check builds with the -E -W -b dirhtml flags the maintainers already chose and fails on any warning. It triggers on pull_request and on pushes to main and stable, with a paths filter for docs/**, src/**, pyproject.toml, uv.lock and the workflow itself, since autodoc reads docstrings from src/ and a signature change there can break a reference in docs/api.md.
Before opening the pull request the agent runs the same command in its sandbox and reads the output, so the first run of the check is known to pass on the current tree rather than hoped to. It also runs zizmor on the new file, as zizmor.yaml will, and keeps the workflow free of anything that linter flags. It adds a line to docs/contributing.md only where that page already explains how to run tox, naming the new environment and nothing else.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add a GitHub Actions check that builds this project's Sphinx documentation on every pull request and fails on warnings. Read pyproject.toml (the docs dependency group and the tox docs environment), docs/conf.py, .readthedocs.yaml and every file in .github/workflows first, and note that tests.yaml ignores docs/** so docs-only pull requests currently run nothing. Add .github/workflows/docs.yaml in the same style as the existing workflows: permissions: {}, the same concurrency pattern, the same actions pinned to the same commit SHAs with their version comments, persist-credentials: false, uv with its cache, and setup-python reading python-version-file from pyproject.toml. Run the docs build through the existing tox environment with uv run --locked --no-default-groups --group dev tox run -e docs so the -E -W dirhtml flags stay in one place. Trigger it on pull requests and on pushes to main and stable, filtered to docs/**, src/**, pyproject.toml, uv.lock and the workflow file. Run that exact command in your sandbox first and paste its last lines into the pull request, run zizmor on the new workflow, and mention the new environment in docs/contributing.md only where tox is already described. Change no other workflow.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request adds one file, .github/workflows/docs.yaml, and touches docs/contributing.md if that page lists the tox environments. Its description shows the tail of the sphinx-build output from the sandbox, ending in the line that says the build succeeded, and the zizmor result for the new workflow.
On the pull request itself the new check appears beside the pre-commit and zizmor jobs, and because it changed only a workflow file and a docs page, tests.yaml stays skipped, which is the behaviour the check was added to complement. From then on a pull request that breaks a cross-reference, misses a toctree entry or leaves an autodoc target unresolved gets a red check with the Sphinx warning in the log, before Read the Docs ever builds it.
A reviewer checks three things: that the pinned action SHAs match the ones in tests.yaml, that the paths filter covers the directories whose content reaches the docs (src/ for autodoc, not only docs/), and that the build runs through tox rather than a copied sphinx-build line, so a future change to the flags in pyproject.toml applies to the check without a second edit.
Run it on every documentation site without a pull request build
An organisation with several Sphinx projects sends this body once per repository to POST /o/{org}/sessions, each with its own repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started instead of a second pull request. Each run reads its own project: one builds through tox, another through a make html target in docs/Makefile, a third has a requirements.txt under docs/ instead of a dependency group, and the workflow each run writes follows the convention it finds rather than a template.
A webhook_url on the key reports each run.completed with the pull request URL in result. A run.failed is worth reading too: it usually means the docs do not build warning-free today, which is the finding the check exists to surface.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
The hard part of this job is not the YAML. It is matching a project's conventions closely enough that the maintainers accept the file: the SHA pins, the permissions: {} line, the uv flags, the tox environment instead of a bare sphinx-build, the security linter that runs on every workflow change. The agent reads all of that from the repository and from the workflows beside the new one, and it proves the command passes by running it in its own sandbox before anyone sees the pull request.
The run keeps every command it ran and what came back, so the review is about the trigger filter and the flags, not about whether the build works. Many repositories can get the same check in one batch, each in its own sandbox, each with its own build output in its own pull request.
Frequently asked questions
Why run tox instead of calling sphinx-build directly in the workflow?
Because the project already defines the docs build once, in [tool.tox.env.docs], with -E -W -b dirhtml. A workflow that repeats those flags drifts from the local command the next time someone changes them. Running tox run -e docs keeps the check and the maintainer's local build identical.
Does the check need network access?
Yes, for two reasons: uv installs the docs dependency group from the lockfile, and sphinx.ext.intersphinx downloads the Python documentation inventory named in docs/conf.py. GitHub-hosted runners have that access. With -W a failed inventory fetch is an error, so a long outage of docs.python.org would show up as a red check rather than a silent gap.
Will it slow down pull requests that do not touch the docs?
The workflow carries a paths filter, so a pull request that changes neither docs/, src/, pyproject.toml, uv.lock nor the workflow itself does not start it. src/ is included on purpose: autodoc reads docstrings and signatures from the package, and a rename there can break a reference in docs/api.md.
What about Read the Docs?
Nothing changes there. .readthedocs.yaml keeps building the published site with its own sphinx-build -W line. The new job is the earlier signal, on the pull request, with the same strictness; a project that also enables pull request previews on Read the Docs gets a rendered preview as well, but that is a setting on their side, not a file in the repository.
Can it also check for broken external links?
Sphinx has a linkcheck builder for that, but it is slow and fails on sites that rate-limit or block bots, so the agent keeps it out of the pull request check. A separate scheduled workflow running sphinx-build -b linkcheck once a week is the usual place for it, and the agent can add one when asked.