All jobs

Add a Sitemap to a Sphinx Docs Site

Yes. An agent adds sphinx-sitemap to your Sphinx docs with a base URL that follows Read the Docs versions, a robots.txt that names it, a build check, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the docs are built in each place they are built. In a project laid out like pallets/click, docs/conf.py lists seven extensions (sphinx.ext.autodoc, sphinx.ext.extlinks, sphinx.ext.intersphinx, sphinx_tabs.tabs, sphinxcontrib.log_cabinet, pallets_sphinx_themes, myst_parser), uses the click theme, sets html_static_path, html_favicon and html_logo, and sets no html_baseurl. pyproject.toml keeps the docs dependencies in a [dependency-groups] docs list and uv.lock pins Sphinx 9.1.0 on Python 3.12 and later. Two commands build the site: [tool.tox.env.docs] runs sphinx-build -E -W -b dirhtml docs docs/_build/dirhtml, and .readthedocs.yaml overrides Read the Docs' own Sphinx step with uv run --group docs sphinx-build -W -b dirhtml docs $READTHEDOCS_OUTPUT/html on Python 3.13. Both pass -W, so a warning fails the build, and both use dirhtml, so every page is a directory with a trailing slash. .github/workflows/tests.yaml ignores docs/**, so nothing in CI builds the docs on a pull request.

The extension is sphinx-sitemap 2.9.0, added to the docs group and to extensions as sphinx_sitemap. It needs html_baseurl, and the right value differs by build: on Read the Docs the site is served under a language and version path, and the build exposes that as READTHEDOCS_CANONICAL_URL, the canonical base for the version being built. The agent sets html_baseurl = os.environ.get("READTHEDOCS_CANONICAL_URL", "https://docs.example.com/") so a stable build writes stable URLs and a latest build writes latest ones, with the fallback keeping the local tox build free of the missing-base-URL warning that -W would turn into a failure. Because the canonical URL already carries the language and version, sitemap_url_scheme is set to "{link}" instead of the default "{lang}{version}{link}", which would otherwise append them a second time.

With dirhtml the extension's links already end in /, so no suffix handling is needed, and sitemap_excludes drops search/ and genindex/, pages no search engine should index. sitemap_show_lastmod stays off: it reads dates from git history and Read the Docs clones shallowly, so turning it on there would produce wrong dates. A docs/_extra/robots.txt with a Sitemap: line pointing at the base URL's sitemap.xml is copied to the output root with html_extra_path, and the pull request notes that Read the Docs serves a robots.txt of its own unless the project provides one, which this does.

To keep the sitemap from rotting, the agent adds a docs job to tests.yaml that runs uv run --locked --no-default-groups --group dev tox run -e docs on pull requests that touch docs/**, the paths the workflow currently ignores, and asserts docs/_build/dirhtml/sitemap.xml exists and lists the index page. In its sandbox it runs that same tox environment with -W, reads the generated file, and runs it a second time with READTHEDOCS_CANONICAL_URL set to a versioned URL to see the links change with it.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add a sitemap to these Sphinx docs. Read docs/conf.py, pyproject.toml (the dependency groups and the tox docs environment), uv.lock for the Sphinx version, .readthedocs.yaml and .github/workflows/tests.yaml first. Add sphinx-sitemap to the docs dependency group and to extensions. Set html_baseurl from READTHEDOCS_CANONICAL_URL with a plain fallback for local builds, set sitemap_url_scheme to {link} because that URL already carries the language and version, exclude the search and genindex pages, and leave sitemap_show_lastmod off because Read the Docs clones shallowly. Add a robots.txt with a Sitemap line through html_extra_path. Add a docs job to the tests workflow that runs the tox docs environment on pull requests touching docs and checks that sitemap.xml exists and lists the index page. Run the tox docs environment with -W in the sandbox, once plain and once with READTHEDOCS_CANONICAL_URL set to a versioned URL, and open a pull request that quotes the first lines of each generated sitemap.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request changes docs/conf.py, pyproject.toml, uv.lock and .github/workflows/tests.yaml, and adds docs/_extra/robots.txt. Its description quotes the head of sitemap.xml from the local build and from the build with a versioned canonical URL, so the reviewer sees both forms of link, and the sphinx-build -W result for each.

A reviewer checks the fallback base URL is their real docs host, that the excluded pages are the ones they want hidden, and whether the new workflow job should run on every pull request rather than only on docs changes. On the next Read the Docs build they open /sitemap.xml under the version path and /robots.txt at the root and confirm one names the other.

The theme, the existing extensions and the page content are untouched; a build without the environment variable produces the same HTML as before plus one XML file.

Run it on every docs site without a sitemap

An automation on the Project with a schedule trigger set to cron: "0 7 * * 1" in your timezone runs read-only with a prompt that builds the docs in the sandbox and comments on the repository's tracking issue when sitemap.xml is missing from the output or does not list a page added that week. The schedule starts a fresh run each Monday; nothing from the previous week is carried over.

To fix a site the schedule flags, a handler on your own server sends this body to POST /o/{org}/sessions with the repository in resources and a request_id that is a UUID v5 of the repository's id and the week, so a retried call returns the run already started. The run pushes the change on its own branch and the key's webhook_url receives run.completed.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

The three lines of configuration are easy; the detail that decides whether the links are right is which base URL each build uses, and that is only seen by building twice, once as tox does and once as Read the Docs does with its environment variable. The agent runs both in its sandbox under the same -W the project uses and keeps the generated files in the run's record, so the pull request quotes real links rather than the ones the docs promise.

Every Sphinx project in an organisation hosts on the same service with the same version layout, so the same run, pointed at each repository, writes a sitemap that follows that layout in each, with the build output kept per run.

Frequently asked questions

Read the Docs already serves a sitemap. Why add one?

Read the Docs generates a sitemap at the project root that lists versions and languages, not pages. sphinx-sitemap writes one per version that lists every page in that version, which is what a search engine needs to index the content; the two complement each other.

Why read the base URL from an environment variable?

The docs are served under a language and a version, and each Read the Docs build serves a different one. READTHEDOCS_CANONICAL_URL is the canonical base for the version being built, so reading it makes the stable sitemap point at stable and the latest sitemap at latest without hard-coding either.

Does the -W flag break the build when the base URL is missing?

It would: sphinx-sitemap warns when html_baseurl is unset, and -W turns warnings into failures in both the tox environment and the Read the Docs command. The fallback value in conf.py means the warning never fires, locally or in CI.

Why not turn on last-modified dates?

sitemap_show_lastmod takes each page's date from git history, and its documentation says shallow clones are not supported. Read the Docs clones shallowly, so the dates would be wrong exactly where the sitemap is published. It stays off, and the pull request says how to enable it on a full clone.

What about the dirhtml builder?

Both builds use dirhtml, so each page is a directory and the extension's links end in a slash. The agent builds with that builder, reads the generated links, and the excludes are written in that form (search/, genindex/) so they match.

More Sphinx jobs

The same job on other stacks

Keep reading