All jobs

Add llms.txt to a MkDocs Site

Yes. An agent adds the mkdocs-llmstxt plugin to your MkDocs site with sections from your nav, builds llms.txt and llms-full.txt, keeps your Python floor, opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the docs are built and published before it adds a plugin, because the plugin's own requirements have to fit the project's. In a library like encode/httpx, mkdocs.yml uses the material theme with a custom_dir of docs/overrides, sets site_url to https://www.python-httpx.org/ and a site_description, and lays out a five-section nav: Introduction with index.md and quickstart.md, Advanced with ten pages under advanced/, Guides with async.md, http2.md, logging.md, compatibility.md and troubleshooting.md, API Reference with api.md, exceptions.md and environment_variables.md, and Community with three more. api.md is rendered by the mkautodoc extension from docstrings, so its content exists only after a build. requirements.txt pins mkdocs==1.6.1, mkdocs-material==9.6.18 and mkautodoc==0.2.0 beside the test tooling, scripts/build runs python -m build, twine check dist/* and mkdocs build, and scripts/publish ends with mkdocs gh-deploy --force. .github/workflows/test-suite.yml runs scripts/build on Python 3.9 through 3.13, and publish.yml runs scripts/publish on 3.9.

The plugin is mkdocs-llmstxt. Its current release, 0.5.1, requires Python 3.11 and depends on beautifulsoup4, markdownify, mdformat and mdformat-tables, and its README says the project is in maintenance mode. That Python floor is the whole difficulty: requirements.txt is installed on every matrix entry, and pyproject.toml keeps requires-python >=3.9, which the library has no reason to raise for a docs plugin. The agent keeps the plugin off the 3.9 and 3.10 jobs with an environment marker, mkdocs-llmstxt==0.5.1; python_version >= "3.11", and keeps the default build untouched by putting the plugin in a second config, mkdocs.llms.yml, that starts with INHERIT: mkdocs.yml and adds a plugins list. Because declaring plugins at all drops MkDocs' implicit search plugin, that list names search first and llmstxt second.

The llmstxt block maps the nav onto sections: Introduction with index.md and quickstart.md, Advanced with advanced/*.md, Guides with the five guide files, API Reference with api.md, exceptions.md and environment_variables.md, and Community with the rest, each with a one-line description where the page's own first paragraph gives one. markdown_description carries a short paragraph about the library, and full_output: llms-full.txt turns on the single expanded file. The plugin converts each page's built HTML back to Markdown, which is why api.md comes through with the signatures and docstrings mkautodoc rendered, and it writes one .md beside each page in site/ so llms.txt can link to https://www.python-httpx.org/quickstart.md and the rest. scripts/publish changes one line, mkdocs gh-deploy --force -f mkdocs.llms.yml, and publish.yml moves from Python 3.9 to 3.11 so that deploy has the plugin; scripts/build and the five-version matrix run exactly as before.

To check the output the agent runs pip install -r requirements.txt on Python 3.11 and mkdocs build -f mkdocs.llms.yml --strict in its sandbox, then reads site/llms.txt: an # HTTPX heading, the site_description as a blockquote, the five sections with one link per page, and no page left out of the nav. It greps site/llms-full.txt for a signature from api.md to confirm the generated reference made it through, confirms site/index.html and the search index are unchanged against a plain mkdocs build, and adds a docs-llms job to test-suite.yml on Python 3.11 that runs the strict build with the second config and fails if site/llms.txt is missing. It runs scripts/check as well, since ruff and mypy cover the test tree where nothing changed.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add llms.txt to this MkDocs site with the mkdocs-llmstxt plugin. Read mkdocs.yml, requirements.txt, pyproject.toml, scripts/build, scripts/publish, scripts/install, .github/workflows/test-suite.yml and .github/workflows/publish.yml first. The plugin needs a newer Python than the project's floor, so pin it in requirements.txt behind a python_version marker and put it in a second config file that inherits mkdocs.yml, listing search explicitly so the default search plugin stays on. Map the nav sections onto the plugin's sections with short descriptions, set a markdown_description, and turn on llms-full.txt. Point the gh-deploy line in scripts/publish at the new config and raise the publish workflow's Python to match. Leave scripts/build and the test matrix alone. Build with the new config in strict mode, show llms.txt and check that the autodoc API page is present in llms-full.txt, add a CI job that builds with the new config on a supported Python, and open a pull request with the output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request adds mkdocs.llms.yml and changes requirements.txt, scripts/publish, .github/workflows/publish.yml and .github/workflows/test-suite.yml. Its description pastes the generated llms.txt in full, the first lines of llms-full.txt at the API reference section, and the strict build output, and lists the pages each section covers next to the nav so a reviewer can see nothing is missing.

The Test Suite workflow runs on the pull request with the five existing jobs unchanged and the new docs-llms job alongside, so a reviewer sees both the plain build and the plugin build pass. The run's record keeps the diff between site/ from the two configs, which touches only the new text files.

Before merging, a reviewer edits the section descriptions and the markdown_description, decides whether llms-full.txt is wanted given its size for a site with a generated API reference, and whether to raise the docs tooling to Python 3.11 across the board instead of the marker. The pages, the theme and the default build are unchanged.

Run it on every docs site without an llms.txt

A team with several MkDocs sites sends this body once per repository to POST /o/{org}/sessions, each with its own repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started rather than opening a second pull request. Each run reads its own mkdocs.yml, so a site that already lists plugins gets llmstxt appended to the existing list, one whose tooling is already on Python 3.11 gets the plugin in the main config with no second file, and one with mkdocstrings gets its API pages through the same HTML-to-Markdown path.

Once the plugin is in the deploy config, no further run is needed when pages change: the files are regenerated by every gh-deploy. A webhook_url on the key reports each run.completed with the pull request in result, and a run.failed names a site whose strict build broke on the plugin, for instance one with a nav entry that points at a file the plugin cannot find.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

The plugin is a five-line config change and a Python floor that collides with the project's own; getting the second part wrong breaks the install on half the CI matrix. The agent reads the matrix, the install script and the publish script before touching anything, isolates the plugin behind a marker and a second config, and runs both builds in its sandbox to show the default site is byte-for-byte what it was.

The strict build output, the generated files and the two-config diff stay with the run and are quoted in the pull request. Several documentation sites can get the same change at once, each in its own sandbox, each with its own evidence.

Frequently asked questions

Why a second config file?

Because mkdocs-llmstxt needs Python 3.11 and scripts/build runs mkdocs build on Python 3.9 and 3.10 too. A config that names a plugin that is not installed fails the build, so the plugin lives in mkdocs.llms.yml, which inherits everything from mkdocs.yml, and only the publish step and the new CI job use it. A project already on 3.11 everywhere can put the block straight into mkdocs.yml.

Does the generated API reference end up in llms-full.txt?

Yes. The plugin works from the built HTML and converts it back to Markdown, so pages rendered by mkautodoc or mkdocstrings are included with their signatures and docstrings. The agent greps the output for a known signature from api.md to confirm it.

Why does the config list search?

MkDocs enables its built-in search plugin only when no plugins key is set. As soon as you declare one, you must list search yourself or the search box disappears. The agent lists it first and checks the search index is still written.

The plugin's README says it is in maintenance mode. Is that a problem?

It is worth knowing, and the pull request says so. The plugin is pinned to an exact version in requirements.txt like the rest of the docs tooling, so nothing changes under you, and the output is plain text files that any replacement can regenerate. The agent names the pin as the line to change if you move to another tool.

More MkDocs jobs

The same job on other stacks

Keep reading