# Add a Sitemap to a Sphinx Docs Site

Yes. An agent adds sphinx-sitemap to your Sphinx docs with a base URL that follows Read the Docs versions, a robots.txt that names it, a build check, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-sitemap/sphinx/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads how the docs are built in each place they are built. In a project laid out like pallets/click, `docs/conf.py` lists seven extensions (`sphinx.ext.autodoc`, `sphinx.ext.extlinks`, `sphinx.ext.intersphinx`, `sphinx_tabs.tabs`, `sphinxcontrib.log_cabinet`, `pallets_sphinx_themes`, `myst_parser`), uses the `click` theme, sets `html_static_path`, `html_favicon` and `html_logo`, and sets no `html_baseurl`. `pyproject.toml` keeps the docs dependencies in a `[dependency-groups] docs` list and `uv.lock` pins Sphinx 9.1.0 on Python 3.12 and later. Two commands build the site: `[tool.tox.env.docs]` runs `sphinx-build -E -W -b dirhtml docs docs/_build/dirhtml`, and `.readthedocs.yaml` overrides Read the Docs' own Sphinx step with `uv run --group docs sphinx-build -W -b dirhtml docs $READTHEDOCS_OUTPUT/html` on Python 3.13. Both pass `-W`, so a warning fails the build, and both use `dirhtml`, so every page is a directory with a trailing slash. `.github/workflows/tests.yaml` ignores `docs/**`, so nothing in CI builds the docs on a pull request.

The extension is `sphinx-sitemap` 2.9.0, added to the `docs` group and to `extensions` as `sphinx_sitemap`. It needs `html_baseurl`, and the right value differs by build: on Read the Docs the site is served under a language and version path, and the build exposes that as `READTHEDOCS_CANONICAL_URL`, the canonical base for the version being built. The agent sets `html_baseurl = os.environ.get("READTHEDOCS_CANONICAL_URL", "https://docs.example.com/")` so a `stable` build writes `stable` URLs and a `latest` build writes `latest` ones, with the fallback keeping the local tox build free of the missing-base-URL warning that `-W` would turn into a failure. Because the canonical URL already carries the language and version, `sitemap_url_scheme` is set to `"{link}"` instead of the default `"{lang}{version}{link}"`, which would otherwise append them a second time.

With `dirhtml` the extension's links already end in `/`, so no suffix handling is needed, and `sitemap_excludes` drops `search/` and `genindex/`, pages no search engine should index. `sitemap_show_lastmod` stays off: it reads dates from git history and Read the Docs clones shallowly, so turning it on there would produce wrong dates. A `docs/_extra/robots.txt` with a `Sitemap:` line pointing at the base URL's `sitemap.xml` is copied to the output root with `html_extra_path`, and the pull request notes that Read the Docs serves a `robots.txt` of its own unless the project provides one, which this does.

To keep the sitemap from rotting, the agent adds a `docs` job to `tests.yaml` that runs `uv run --locked --no-default-groups --group dev tox run -e docs` on pull requests that touch `docs/**`, the paths the workflow currently ignores, and asserts `docs/_build/dirhtml/sitemap.xml` exists and lists the `index` page. In its sandbox it runs that same tox environment with `-W`, reads the generated file, and runs it a second time with `READTHEDOCS_CANONICAL_URL` set to a versioned URL to see the links change with it.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add a sitemap to these Sphinx docs. Read docs/conf.py, pyproject.toml (the dependency groups and the tox docs environment), uv.lock for the Sphinx version, .readthedocs.yaml and .github/workflows/tests.yaml first. Add sphinx-sitemap to the docs dependency group and to extensions. Set html_baseurl from READTHEDOCS_CANONICAL_URL with a plain fallback for local builds, set sitemap_url_scheme to {link} because that URL already carries the language and version, exclude the search and genindex pages, and leave sitemap_show_lastmod off because Read the Docs clones shallowly. Add a robots.txt with a Sitemap line through html_extra_path. Add a docs job to the tests workflow that runs the tox docs environment on pull requests touching docs and checks that sitemap.xml exists and lists the index page. Run the tox docs environment with -W in the sandbox, once plain and once with READTHEDOCS_CANONICAL_URL set to a versioned URL, and open a pull request that quotes the first lines of each generated sitemap.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

The pull request changes `docs/conf.py`, `pyproject.toml`, `uv.lock` and `.github/workflows/tests.yaml`, and adds `docs/_extra/robots.txt`. Its description quotes the head of `sitemap.xml` from the local build and from the build with a versioned canonical URL, so the reviewer sees both forms of link, and the `sphinx-build -W` result for each.

A reviewer checks the fallback base URL is their real docs host, that the excluded pages are the ones they want hidden, and whether the new workflow job should run on every pull request rather than only on docs changes. On the next Read the Docs build they open `/sitemap.xml` under the version path and `/robots.txt` at the root and confirm one names the other.

The theme, the existing extensions and the page content are untouched; a build without the environment variable produces the same HTML as before plus one XML file.

## Run it on every docs site without a sitemap

An automation on the Project with a `schedule` trigger set to `cron: "0 7 * * 1"` in your timezone runs read-only with a prompt that builds the docs in the sandbox and comments on the repository's tracking issue when `sitemap.xml` is missing from the output or does not list a page added that week. The schedule starts a fresh run each Monday; nothing from the previous week is carried over.

To fix a site the schedule flags, a handler on your own server sends this body to `POST /o/{org}/sessions` with the repository in `resources` and a `request_id` that is a UUID v5 of the repository's id and the week, so a retried call returns the run already started. The run pushes the change on its own branch and the key's `webhook_url` receives `run.completed`.

## Why run it on Rush?

The three lines of configuration are easy; the detail that decides whether the links are right is which base URL each build uses, and that is only seen by building twice, once as tox does and once as Read the Docs does with its environment variable. The agent runs both in its sandbox under the same `-W` the project uses and keeps the generated files in the run's record, so the pull request quotes real links rather than the ones the docs promise.

Every Sphinx project in an organisation hosts on the same service with the same version layout, so the same run, pointed at each repository, writes a sitemap that follows that layout in each, with the build output kept per run.

## Frequently asked questions

### Read the Docs already serves a sitemap. Why add one?

Read the Docs generates a sitemap at the project root that lists versions and languages, not pages. `sphinx-sitemap` writes one per version that lists every page in that version, which is what a search engine needs to index the content; the two complement each other.

### Why read the base URL from an environment variable?

The docs are served under a language and a version, and each Read the Docs build serves a different one. `READTHEDOCS_CANONICAL_URL` is the canonical base for the version being built, so reading it makes the `stable` sitemap point at `stable` and the `latest` sitemap at `latest` without hard-coding either.

### Does the -W flag break the build when the base URL is missing?

It would: `sphinx-sitemap` warns when `html_baseurl` is unset, and `-W` turns warnings into failures in both the tox environment and the Read the Docs command. The fallback value in `conf.py` means the warning never fires, locally or in CI.

### Why not turn on last-modified dates?

`sitemap_show_lastmod` takes each page's date from git history, and its documentation says shallow clones are not supported. Read the Docs clones shallowly, so the dates would be wrong exactly where the sitemap is published. It stays off, and the pull request says how to enable it on a full clone.

### What about the dirhtml builder?

Both builds use `dirhtml`, so each page is a directory and the extension's links end in a slash. The agent builds with that builder, reads the generated links, and the excludes are written in that form (`search/`, `genindex/`) so they match.

## More Sphinx jobs

- [Add Social Cards to Sphinx Docs](https://getrush.ai/run/add-social-cards/sphinx/)
- [Add a Typo Check to a Sphinx Documentation Site](https://getrush.ai/run/add-typo-check/sphinx/)
- [Fix "The sphinxcontrib.applehelp extension used by this project needs at least Sphinx v5.0"](https://getrush.ai/run/fix-sphinxcontrib-applehelp-version-error/sphinx/)

## The same job on other stacks

- [Add a Sitemap to a SvelteKit App](https://getrush.ai/run/add-sitemap/sveltekit/)
- [Add an XML Sitemap to a Symfony App](https://getrush.ai/run/add-sitemap/symfony/)
- [Add a Sitemap to a VitePress Site](https://getrush.ai/run/add-sitemap/vitepress/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
