All jobs

Add a Sitemap to a Jekyll Site

Yes. An agent adds jekyll-sitemap to your Gemfile and plugin list, points robots.txt at the generated sitemap.xml, builds the site to check it and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the site's configuration first, because a sitemap plugin only does the right thing when url and baseurl are right and the pages it should skip are marked. In a site built on Poole, as in poole/poole, _config.yml sets url: https://getpoole.com, baseurl: "", permalink: pretty and paginate: 1, lists jekyll-gist, jekyll-paginate and jekyll-seo-tag under plugins, and excludes the Gemfile, Gemfile.lock, README.md and LICENSE.md from the output. The Gemfile names the same three plugins and jekyll with no versions; Gemfile.lock resolves them to Jekyll 4.0.0 and jekyll-seo-tag 2.6.1. _includes/head.html already calls {% seo title=false %} and links atom.xml, which is a Liquid template with layout: null. The pages are index.html, about.md, archive.md and a 404.html whose front matter sets permalink: 404.html; the four posts under _posts/ carry dates in their filenames and no last_modified_at. There is no robots.txt and no sitemap anywhere.

The plugin is jekyll-sitemap, which the agent adds in three places: gem "jekyll-sitemap" in the Gemfile, jekyll-sitemap in the plugins list of _config.yml, and the lockfile, which it refreshes with bundle install so the pull request carries the resolved version, 1.4.0, which depends on jekyll >= 3.7, < 5.0 and so fits the Jekyll 4 this site pins. The plugin's own template, lib/sitemap.xml in jekyll/jekyll-sitemap, decides what goes in: every document of every collection with output, every HTML page, and static files, each as an absolute URL built from url and baseurl, skipping anything whose front matter says sitemap: false and the page whose URL is /404.html. That last rule means Poole's 404 page is left out without a change, and atom.xml and styles.scss are left out because neither renders to HTML. Posts get a lastmod from their date; pages get one only when they set last_modified_at, so the agent adds nothing invented there.

robots.txt is the second file. The agent adds one at the root with a layout: null front matter block, User-agent: *, Allow: / and a Sitemap: line written as {{ site.url }}{{ site.baseurl }}/sitemap.xml, so the address follows the configuration rather than being typed twice. Because jekyll-sitemap is one of the plugins the GitHub Pages build allows, with version 1.4.0 listed in pages.github.com/versions.json under the github-pages gem, the same two files work whether the site is built by Pages itself or by a workflow that runs the repository's own Jekyll 4.

Then the agent builds. It installs the gems with bundle install, runs JEKYLL_ENV=production bundle exec jekyll build and reads _site/sitemap.xml: it checks that the four posts, the home page, about/ and archive/ are present as https://getpoole.com/... URLs, that 404.html is not, and that each post's lastmod matches the date in its filename. It fetches _site/robots.txt and confirms the Sitemap: line resolves to the same host. Because Poole's lockfile dates from Jekyll 4.0.0 with an ffi and sassc of the same age, the agent notes in the pull request whether bundle install needed a bundle update to install on the Ruby in its sandbox, and includes that lockfile change only if it did.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add an XML sitemap to this Jekyll site with the jekyll-sitemap plugin. Read _config.yml, the Gemfile and Gemfile.lock, _includes/head.html, the pages at the root and the _posts directory first. Add gem \"jekyll-sitemap\" to the Gemfile and jekyll-sitemap to the plugins list in _config.yml, run bundle install so the lockfile records the version, and add a robots.txt at the root with layout: null, User-agent: *, Allow: / and a Sitemap line built from site.url and site.baseurl. Do not add sitemap: false to pages the plugin already skips, such as the 404 page, and do not invent last_modified_at dates. Build the site with JEKYLL_ENV=production bundle exec jekyll build, check that _site/sitemap.xml lists every post and page with absolute URLs and leaves out the 404 page, and that _site/robots.txt points at it. If bundle install needs a bundle update to work on the current Ruby, say so and include the lockfile change. Open a pull request with the commands and their output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request changes Gemfile, Gemfile.lock and _config.yml, adds robots.txt, and touches no layout, include or post. Its description quotes the jekyll build output and the generated sitemap.xml, with the URL list the reviewer can compare against the site, and names the plugin version the lockfile now carries. If the old lockfile would not install, the description says which gems moved and why, separately from the sitemap change, so the reviewer can judge the two on their own.

A reviewer checks that url in _config.yml is the address the site is really served from, because every loc in the sitemap is built from it, and that no page they consider private is listed; a page they want out gets sitemap: false in its front matter, which the pull request shows how to do. After merge, the next build publishes /sitemap.xml and /robots.txt, and the sitemap can be submitted in Google Search Console or Bing Webmaster Tools, which the pull request leaves to the reader because it needs their account.

Run it on every Jekyll site with no sitemap

A team that runs several Jekyll sites sends this body once per repository from its own server, each POST /o/{org}/sessions naming one repository in resources and carrying a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started. Each run reads its own configuration: a site with baseurl set gets a Sitemap: line that includes it, a site that already has a robots.txt gets one appended line, and a site built by the GitHub Pages pipeline rather than its own Jekyll gets a note that the plugin is already in the github-pages gem and needs only the plugins entry.

The key's webhook_url reports each run the key starts with the pull request URL in result, so a script that sent five requests collects five pull requests without polling.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

A sitemap is small, but whether it is right depends on the whole site: the host in url, the permalink style, which pages render to HTML and which do not, and whether the plugin's own exclusions already cover the 404 page. The agent reads those files, runs the real jekyll build in its own sandbox, reads the XML it produced and keeps the output in the run's record, so the pull request shows the list of URLs that will be submitted rather than a promise that a plugin was added.

It also meets the repository where it is. A lockfile from Jekyll 4.0.0 may not install on a current Ruby, and the agent finds that out by running bundle install rather than by assuming, then reports the lockfile change as its own decision for the reviewer. Several sites, each with its own baseurl and pages, get several pull requests, each checked against its own build.

Frequently asked questions

Does this work on GitHub Pages' own build?

Yes. jekyll-sitemap 1.4.0 is in the github-pages gem that the Pages pipeline uses, so a site built that way needs only the plugins entry in _config.yml. A site that pins Jekyll 4 in its Gemfile, as this one does, is built by its own workflow instead, and there the Gemfile line is what installs the plugin.

How do I keep a page out of the sitemap?

Add sitemap: false to its front matter. The plugin skips those pages, and it already skips the page at /404.html, which is why the pull request does not touch the 404 page.

Will the dates be right?

Posts get lastmod from their date, which Jekyll takes from the filename unless the front matter sets date. Pages have no date, so they get no lastmod unless they set last_modified_at; the agent adds none, because an invented date is worse than no date to a crawler.

Why not write sitemap.xml as a Liquid template like atom.xml?

The plugin's template is the same idea, maintained by the Jekyll team, and it handles collections, static files and the sitemap: false exclusion. A hand-written template is a fine alternative for a site that cannot add gems, and the pull request links the plugin's lib/sitemap.xml as the starting point for that.

Does the sitemap include the paginated index pages?

Pages that jekyll-paginate generates, such as /page2/, are HTML pages and so appear in the sitemap. For a blog that is harmless, since they are real pages a reader can open; a reader who wants them out can exclude them with sitemap: false once their site's pagination plugin lets generated pages carry front matter of their own, and the pull request notes this as a separate question.

More Jekyll jobs

The same job on other stacks

Keep reading