All jobs

Fail CI on Sphinx Warnings and Broken Links for a Python Library

Yes. An agent adds a CI job that builds your Sphinx docs with warnings as errors and checks links, pinned and scoped like your other workflows.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent starts with what the repository already does for its documentation and its CI, because the new job must match both. In a library like psf/requests, docs/ holds a Sphinx project with conf.py enabling autodoc, intersphinx, todo and viewcode, fifteen .rst files under user/, dev/ and community/, a Makefile whose html target runs sphinx-build, and a requirements.txt that pins Sphinx==7.2.6 for Read the Docs, which .readthedocs.yaml builds with the dirhtml builder on Python 3.12 after installing the package itself. The top-level Makefile has a docs target that runs cd docs && make html. Under .github/workflows there are eight workflows: run-tests.yml with a matrix of eight Python versions on three operating systems, lint.yml running pre-commit, typecheck.yml running pyright, codeql-analysis.yml, publish.yml, zizmor.yml, and two issue-management workflows. None of them builds the docs, so a broken cross-reference or an autodoc import failure is found on Read the Docs after merge, if anyone looks.

It then reads the conventions in those workflows so the new one is indistinguishable from them: every uses: is pinned to a full commit SHA with the version in a trailing comment (actions/checkout@9c091bb… # v7.0.0, actions/setup-python@5fda3b9… # v7.0.0), each job sets permissions: contents: read at the top, uses persist-credentials: false on checkout, and has timeout-minutes: 10. The pre-commit configuration in .pre-commit-config.yaml excludes docs/ from Ruff, which tells the agent that the docs tree is deliberately outside the lint job and needs its own.

The agent writes .github/workflows/docs.yml, triggered on push and pull_request like lint.yml, with one job that checks out with the same pinned SHA and persist-credentials: false, sets up Python 3.12 to match .readthedocs.yaml, installs the package with python -m pip install . so autodoc can import requests, installs -r docs/requirements.txt so the Sphinx version is the one Read the Docs uses, and runs sphinx-build -W --keep-going -b dirhtml docs docs/_build/dirhtml, where -W turns warnings into a failing exit code and --keep-going reports all of them instead of the first. A second step runs sphinx-build -b linkcheck over the same tree, with linkcheck_ignore entries added to docs/conf.py for the hosts the agent finds rate-limiting or sign-in-only when it runs the checker itself. It also adds a docs entry to the top-level Makefile only if the existing docs target does not already match, which here it does, so the Makefile is left alone.

Before opening the pull request the agent runs exactly those commands in its sandbox on Python 3.12: install, sphinx-build -W --keep-going, linkcheck. If the strict build fails on existing warnings, as a docs tree that has never been built with -W often does, the agent fixes each one in the .rst or docstring it points at, in a separate commit from the workflow, and quotes the warning and the fix; it does not loosen -W to get green. It validates the workflow with actionlint and with zizmor, since the repository runs zizmor on its workflows and the new file must pass the same audit, then pushes, and the new job runs on the pull request alongside the existing eight.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add a GitHub Actions job that builds this Python library's Sphinx documentation with warnings as errors and checks its links. Read every workflow under .github/workflows, .readthedocs.yaml, docs/conf.py, docs/requirements.txt, docs/Makefile and the top-level Makefile first, and match the existing workflows' conventions: SHA-pinned actions with version comments, contents: read permissions, persist-credentials false, a job timeout. Install the package and the pinned Sphinx version, run sphinx-build with -W --keep-going using the same builder Read the Docs uses, then run the linkcheck builder. Run both in the sandbox first; fix any existing warning at its source in a separate commit rather than relaxing the flags, and add linkcheck_ignore only for hosts that block automated checks. Validate the workflow with actionlint and zizmor and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request on its own branch that adds .github/workflows/docs.yml and, when the strict build found warnings, a second commit touching only the .rst files or docstrings that caused them, plus any linkcheck_ignore lines in docs/conf.py. The eight existing workflows, .readthedocs.yaml and docs/requirements.txt are unchanged, so Read the Docs keeps building exactly as before.

The description quotes the sphinx-build -W --keep-going output and the linkcheck summary from the sandbox, lists each warning fixed with its file and line, explains each ignored host in a sentence, and shows the actionlint and zizmor results. The new job's status appears on the pull request itself, which is the first real run of the check.

Before merging, a reviewer scans the ignore list, since an over-broad pattern hides real dead links, and confirms the Python version and builder match .readthedocs.yaml so the job and the published docs cannot disagree. The transcript at GET /o/{org}/sessions/{id}/messages holds every build log.

Run it on every terminal hand-off

A maintainer who just found a broken cross-reference on the published docs starts this from the terminal with this body and the repository in resources, then goes back to the issue queue. The run reads the real workflows and the real Read the Docs configuration, so the job it adds is pinned, scoped and versioned the way the others are, and it has already run the strict build once before anyone reviews it.

Send a request_id, a UUID, in the body so a retried start returns the run it already began. When the turn ends, GET /o/{org}/sessions/{id} names the pull request. The same body sent for each library you maintain adds the same check to each, every run reading that repository's own docs configuration in its own sandbox.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

The workflow is twenty lines, but the first strict build of a mature docs tree is where the work is: a handful of warnings that have been scrolling past for years, an autodoc import that needs the package installed, a link checker that fails on hosts that block bots. Done by hand, that is an afternoon of re-running builds; done loosely, -W gets dropped to make the job green and the check means nothing. The agent runs the strict build first, fixes causes rather than flags, and matches the repository's pinning and permission conventions so the new file passes its own zizmor audit.

On Rush the installs and builds happen in the agent's own sandbox on the Python version the docs are published with, and every build log stays in the transcript beside the diff. A maintainer reads a pull request whose claims they can check line by line, and the same job lands on the next library with one more request.

Frequently asked questions

What if the strict build already fails?

The agent fixes the warnings at their source, in the .rst or the docstring Sphinx points at, in a commit separate from the workflow so the two are reviewable apart, and quotes each warning in the description. It does not remove -W or add suppress_warnings to get a green run; if a warning cannot be fixed without a larger change, the pull request says which and why.

Will linkcheck make CI flaky?

It can, which is why the agent runs it in the sandbox first and adds linkcheck_ignore patterns for hosts that return errors to automated requests, each with a comment. If you prefer, say so in the prompt and the agent puts linkcheck in a separate job with continue-on-error: true, so dead links are reported without blocking the merge.

Does it change how Read the Docs builds?

No. .readthedocs.yaml and docs/requirements.txt are left alone; the new job reads them to use the same Python version, Sphinx version and builder, so a passing job means the published build will pass too. The job is a gate on pull requests, not a replacement for the hosted build.

Why match the SHA pinning and permissions so carefully?

Because the repository runs zizmor over its workflows, and a new file with a floating tag or missing persist-credentials: false would fail that audit on the same pull request. The agent copies the pinned SHAs and version comments from lint.yml, sets permissions: contents: read and a timeout, and runs zizmor and actionlint locally before pushing.

More Python jobs

The same job on other stacks

Keep reading