All jobs

Add llms.txt to a Docusaurus Docs Site

Yes. An agent adds docusaurus-plugin-llms to your Docusaurus site so every build writes llms.txt and llms-full.txt from the docs, checks the output, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the site is built before it adds a plugin, because a Docusaurus site can have several docs instances, versions and locales, and an llms.txt has to say which of them it covers. In a site like helm/helm-www, package.json pins packageManager to yarn@1.22.22, requires Node ^20.19.0 || >=22.12.0 with 22 in .nvmrc, depends on @docusaurus/core and @docusaurus/preset-classic resolved to 3.10.2 in yarn.lock with @docusaurus/faster, and builds with docusaurus build. docusaurus.config.js reads the site URL from process.env.SITE_URL, falling back to http://localhost:3000, and netlify.toml sets SITE_URL to https://helm.sh only in the production context, so any absolute URL written at build time depends on that variable. The config sets onBrokenLinks and onBrokenAnchors to throw, lists twelve locales under i18n, and turns on future.v4.

The classic preset's docs plugin uses ./sidebars.js, which autogenerates one sidebar from the docs/ tree, and keeps three versions: current labelled 4.3.1, 3 at path v3 from versioned_docs/version-3, and 2 at path v2. A second @docusaurus/plugin-content-docs instance with id community serves the community/ directory at /community; its files are pulled from the helm/community repository by docusaurus-plugin-remote-content with noRuntimeDownloads: true, so they are committed and present at build. docs/ holds 127 Markdown and MDX pages, versioned_docs/version-3 126 and version-2 94, and blog/ has 61 posts with RSS and Atom feeds. docs/index.mdx imports DocCardList from @theme/DocCardList, and docs/_v4-in-progress.mdx is an underscore-prefixed partial that is not a page. .gitignore excludes /build, .htmltest.yml checks links in build, and the only workflows are a Go build for sdkexamples/** and a typos check.

The agent installs docusaurus-plugin-llms with yarn add --dev docusaurus-plugin-llms, which updates package.json and yarn.lock, and registers it in the plugins array of docusaurus.config.js with options read from the config rather than defaults: docsDir as two sections, docs with routeBasePath docs and community with routeBasePath community and the label Community; ignoreFiles with **/_*.mdx so the partial stays out; excludeImports: true so the import DocCardList line in docs/index.mdx does not land in llms-full.txt; removeDuplicateHeadings: true; title and description from the site's own title and tagline; and includeBlog left off, since the posts are announcements rather than reference. The two versioned docs trees are left out by default and the pull request says that versions: "auto" writes a set per version under its own subdirectory when the maintainers want one. Nothing is added to static/: the files are generated into build/ on every docusaurus build.

Then the agent builds. It runs yarn install --frozen-lockfile, as the Makefile does, then SITE_URL=https://helm.sh yarn build --locale en, because building all twelve locales takes far longer and the plugin reads the English docs/ tree either way. It opens build/llms.txt, confirms the title and tagline at the top, counts the links against the pages under docs/ and community/, checks that every link starts with https://helm.sh/docs/ or https://helm.sh/community/ rather than http://localhost:3000, and searches build/llms-full.txt for import lines and <DocCardList tags to confirm the cleaning options took. It runs typos on the changed files, since the typos workflow runs on every pull request, and yarn build once more without --locale to confirm the localized builds still complete.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add llms.txt and llms-full.txt to this Docusaurus site with docusaurus-plugin-llms. Read package.json, .nvmrc, docusaurus.config.js, sidebars.js, versions.json, netlify.toml, the Makefile and .gitignore first, and list every docs plugin instance, version and locale the config defines. Install the plugin as a dev dependency with the package manager package.json names and register it in the plugins array, with docsDir covering every docs instance under its own routeBasePath, ignoreFiles for underscore-prefixed partials, excludeImports and removeDuplicateHeadings on, and the site's title and tagline. Leave versioned docs and the blog out and say in the pull request which options would include them. Build with the production SITE_URL and a single locale, then check build/llms.txt for the right absolute URLs and a link count that matches the docs, and check build/llms-full.txt for leftover import lines or component tags. Run the repository's typos check on your changes. Do not commit anything under build/. Open a pull request that shows the head of the generated llms.txt.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request changes package.json, yarn.lock and docusaurus.config.js and nothing else. Its description shows the first lines of the generated llms.txt, the link count per section, and the two options a maintainer would flip to add versioned docs or the blog.

The repository's typos workflow runs on the pull request. The site build is not a workflow here but a Netlify deploy, so the agent's own build output in the run's record is the proof that the plugin runs under Docusaurus 3.10.2 with @docusaurus/faster and that the throw settings for broken links and anchors are untouched by it.

Before merging, a reviewer confirms that Netlify's production context sets SITE_URL, since a deploy preview without it would write localhost URLs into the file, decides whether the community docs imported from another repository belong in a file that describes this site, and checks that the Netlify cache plugin, which keeps build between deploys, does not serve a stale copy, which it cannot, because the plugin rewrites both files on each build.

Run it on every Docusaurus site in your organization

A company with several Docusaurus sites starts one run per repository from its own server: a POST /o/{org}/sessions each, with this prompt as input and that repository in resources. GET /o/{org}/github lists every repository the Rush GitHub App installation reaches, so a script can pick the ones whose package.json depends on @docusaurus/core and has no docusaurus-plugin-llms. Each run reads its own config, so a site with one docs instance and no versions gets a one-line plugin entry, and a site with a custom routeBasePath gets that path in every generated link.

A team that tracks documentation work in Linear runs the same prompt from an automation in the Project with a linear trigger on issues labelled docs. The trigger adds the issue after the prompt, so an issue that asks for the blog to be included, or for one file per version, reaches the agent with that request, and the pull request links back to it.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

Whether an llms.txt is right is only visible after a full build: the URLs depend on an environment variable, the cleaning options depend on what the MDX pages import, and a count that is off by a few pages means a partial or a second docs instance was missed. The agent runs the build in its own sandbox, on the Node version the repository names, and keeps the generated files with the run so a reviewer reads the actual output rather than a description of it.

The same prompt runs across every docs site at once, each in its own sandbox with its own pull request, and each build takes the minutes it takes without occupying anyone's laptop. The run's record keeps every command, from yarn install to the grep over llms-full.txt, as proof of what was checked.

Frequently asked questions

Does it include the v3 and v2 docs?

Not by default. The plugin reads the docs/ tree, which is the current version, and the versioned trees under versioned_docs/ are left out so the file describes the release the site presents as latest. Setting versions to "auto" makes the plugin write a separate llms.txt and llms-full.txt per version under its own subdirectory, and the pull request names that option.

What about the twelve translations?

The plugin reads the English source directories, so the generated files are English only, and a locale build does not get its own copy. A site that wants a per-locale file points a second plugin instance at the i18n/<locale>/docusaurus-plugin-content-docs/current tree, which is a follow-up the agent notes rather than does.

Will the new files break the link check or the typos check?

.htmltest.yml checks the HTML under build, and a .txt file is not parsed by it. The typos workflow reads source files, and the only source change is in docusaurus.config.js, which the agent runs typos over before opening the pull request. The generated files live under build/, which .gitignore already excludes.

Why not commit a hand-written llms.txt under static/?

With 127 pages in the current docs alone, a static file is out of date on the next docs change and nobody notices. Generating it at build time means a renamed page or a new section shows up in the file on the next deploy, and llms-full.txt, which carries the page bodies, is not something anyone maintains by hand.

Does the blog go in?

includeBlog is off, so the 61 posts are not listed. Turning it on adds the posts under the blog route base path, which is useful for a site whose blog is documentation in disguise and noise for one whose posts are release announcements.

More Docusaurus jobs

The same job on other stacks

Keep reading