All jobs

Add a Sitemap to an mdBook Site

Yes. An agent generates a sitemap.xml from your book's rendered pages in the publish script, points robots.txt at it, checks the XML in its sandbox and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent starts with how the book is built and published, because mdBook has no sitemap output of its own and the right place to add one depends on that. In a repository laid out like rust-lang/mdBook, the user guide lives under guide/ with a book.toml that sets site-url = "/mdBook/", hash-files = true, a [output.html.redirect] entry sending /format/config.html to configuration/index.html, search options, and a guide-helper preprocessor run with cargo run --manifest-path guide-helper/Cargo.toml. guide/src/SUMMARY.md lists the chapters, from the introduction through the command-line, format, continuous-integration and developer sections to the contributors page, plus a 404.md. .gitignore ignores guide/book, so nothing rendered is committed.

Publishing is ci/publish-guide.sh, run by the pages job in .github/workflows/deploy.yml: it builds with cargo run --no-default-features -F search -- build guide, reads the crate version with cargo metadata, adds a gh-pages worktree, and either moves guide/book into a pre-release/ directory when the version has a pre-release suffix or clears the branch root and moves the book there, amending the commit and force-pushing. A sitemap therefore belongs at the branch root only for the release build, and the pre-release copy under pre-release/ must not get one, or search engines would index two copies of every chapter.

The agent weighs an [output.sitemap] renderer backend and sets it aside: adding a second backend makes mdBook write HTML to book/html/ instead of book/, which would change the paths publish-guide.sh moves. Instead it adds a small step to the script after the build, before the worktree is touched: it walks guide/book for every .html file except 404.html, print.html and the search and table-of-contents helper pages, turns each path into an absolute URL using the deploy host joined to site-url with index.html folded to its directory, and writes guide/book/sitemap.xml with one <url> per page and no lastmod, because the chapter file behind a page is not recoverable from its path and a wrong date is worse than none. The step runs only on the release branch of the if, so pre-release/ stays out. A guide/src/robots.txt with a Sitemap: line pointing at that URL rides into the output, since mdBook copies non-Markdown files from src as they are.

In its sandbox the agent installs the Rust toolchain with rustup, builds the guide with the exact cargo run line the script uses so the guide-helper preprocessor runs, runs the new step, checks the file with xmllint --noout against the sitemap schema's namespace, and compares the number of <loc> entries with the number of chapters SUMMARY.md lists so a page is neither missed nor doubled. The redirect page for /format/config.html is excluded on purpose: it exists only to forward.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add a sitemap to this mdBook site. Read guide/book.toml, guide/src/SUMMARY.md, ci/publish-guide.sh, .github/workflows/deploy.yml and .gitignore first. Do not add a second mdBook backend, because that moves the HTML output directory the publish script relies on. Instead, add a step to ci/publish-guide.sh after the build that writes guide/book/sitemap.xml from the rendered pages, with absolute URLs built from the deploy host and the site-url in book.toml, index.html folded to its directory, and the 404 page, the print page, the helper pages and the redirect-only pages left out; run it only for the release build so the pre-release copy gets no sitemap. Add guide/src/robots.txt with a Sitemap line. In the sandbox, install Rust, build the guide with the same cargo command the script uses, run the new step, validate the XML with xmllint, and compare the entry count with the chapters in SUMMARY.md. Open a pull request whose description quotes the validation and the count.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that changes ci/publish-guide.sh and adds guide/src/robots.txt; book.toml, SUMMARY.md, the chapters and deploy.yml are untouched. The description quotes the xmllint result, the number of <loc> entries the step wrote against the number of chapters in SUMMARY.md, and the first few entries so the reviewer can see the host, the /mdBook/ prefix and the folded index paths are right.

The run's record keeps the full guide build, including the guide-helper preprocessor's output and any warning mdBook printed, and the generated sitemap.xml as an artifact, so a reviewer reads the actual file rather than the script that makes it.

Before merging, a reviewer checks the deploy host the script joins to site-url, decides whether draft chapters, which mdBook renders as placeholder pages, should stay in or out, and confirms the step sits inside the release branch of the if and not before it.

Run it on every mdBook repository

A team with several books starts one run per repository from a script: GET /o/{org}/github lists every repository the installation reaches, the script keeps the ones with a book.toml, and sends this body with that repository in resources only where the output has no sitemap.xml today. Each call carries a request_id that is a UUID v5 of the repository's full name, so a second pass returns the runs already started instead of doubling them.

Each run reads its own book. One published with mdbook build straight from a workflow gets the step in that workflow instead of a publish script; one whose site-url is already absolute needs no host joined; one with no 404.md has nothing to exclude. The pull requests arrive one per book, each with its generated sitemap in the record.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

A sitemap step written without building the book gets the paths wrong in ways that only show in Search Console weeks later: the site-url prefix dropped, index.html left on every section URL so each page is listed twice, the print page included, or the pre-release copy indexed beside the release. In a Rush sandbox the agent builds the real guide with the real preprocessor, runs the real script and reads the file it produced before it opens anything.

The build log, the validator's output and the generated file stay with the run, so the reviewer reads the sitemap rather than imagining it. The same job across every book a team publishes runs in parallel, each with its own machine, its own toolchain install and its own pull request.

Frequently asked questions

Why not an mdBook renderer backend?

A backend is the documented way to add an output, but the moment a book has two backends mdBook writes the HTML to book/html/ instead of book/, and ci/publish-guide.sh moves guide/book by path. A post-build step in the script adds the file without moving anything. A book published some other way might take the backend instead, and the agent says which it chose and why.

Does the pre-release copy get a sitemap?

No. The script publishes a pre-release version under pre-release/ beside the released book, and listing both would tell search engines every chapter exists twice. The step runs only in the release branch of the script's if.

Why is there no lastmod?

The rendered page's path does not say which Markdown file produced it, and the build does not record it, so the step would have to guess from git history and could stamp a wrong date. Search engines ignore lastmod they find unreliable, so the sitemap leaves it out rather than mislead.

What about the search index, the redirect and hash-files?

None of them change. hash-files only renames the CSS and JavaScript assets, which a sitemap never lists; the search index is built as before; and the page the [output.html.redirect] table generates is excluded because it only forwards to another page that is listed.

Can I run the step locally?

Yes. It is a function in ci/publish-guide.sh that reads guide/book, so after mdbook build guide a maintainer can call it and open the file. The agent runs it that way in its sandbox and quotes the result.

More mdBook jobs

The same job on other stacks

Keep reading