# Add llms.txt to a Pelican Site

Yes. An agent writes a small Pelican plugin that builds llms.txt and llms-full.txt from your articles and pages on every build, checks the output and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-llms-txt/pelican/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent starts with how the site is built. In a Pelican site like getpelican/pelican-blog, `pelicanconf.py` sets `PATH = "content"`, `THEME = "themes/notmyidea"`, `DEFAULT_PAGINATION = 5`, `STATIC_PATHS` of `images` and `extra`, and an `EXTRA_PATH_METADATA` table that moves `extra/robots.txt` and the touch icons to the site root; `publishconf.py` imports it, sets `SITEURL = "https://blog.getpelican.com"` and `DELETE_OUTPUT_DIRECTORY = True`. There is no `PLUGINS` or `PLUGIN_PATHS` setting, no `requirements.txt` and no workflow. The `Makefile` builds with `make html` (which runs `clean` first), `make publish` for the production settings, and `make upload`, which rsyncs `output/` to the server over SSH. Content is 18 reStructuredText release announcements under `content/news/`, from `pelican-3.0.rst` to `pelican-4.7.rst`, one Markdown post, `namespace-plugins.md`, with `Title`, `Date`, `Modified` and `Slug` in its header, and three pages under `content/pages/`: `index.rst` (with `:save_as: index.html` and an empty `:url:`, so the site root is this page rather than the article list), `gratitude.rst` and `not-found.rst`.

Then it decides where the file comes from. A hand-written `llms.txt` goes stale with the next post, so the agent writes a plugin, `plugins/llms_txt/__init__.py`, and registers it with two new lines in `pelicanconf.py`: `PLUGIN_PATHS = ["plugins"]` and `PLUGINS = ["llms_txt"]`. The plugin connects to Pelican's `finalized` signal, after every generator has run, and reads the articles and pages the generators collected, in the same order the theme's `index.html` lists them. It writes `output/llms.txt` in the llms.txt format: an H1 with `SITENAME`, a one-paragraph blockquote taken from the `index.rst` page's first paragraph, a `## Pages` list linking `gratitude` and the index page, and a `## News` list with one line per article: its title linked at `SITEURL` plus the article's `url`, its date, and its summary with the HTML stripped. `llms-full.txt` holds the same headings followed by each article's full text, converted from the rendered HTML with the standard library's `html.parser`, so no new dependency is added. Both files use `SITEURL` from whichever settings file ran, so `make html` writes relative links and `make publish` writes absolute ones.

The agent then builds the site both ways to see the result. The repository pins no Pelican version, so it installs the current Pelican release with the `markdown` extra, which the Markdown post and the `MARKDOWN` extension table in `pelicanconf.py` need, and runs `make html` and `make publish`. It checks that `output/llms.txt` lists 19 entries under `## News` and 2 under `## Pages`, that the `not-found.rst` page is left out, that no entry carries the `{filename}` or `{static}` link syntax unresolved, and that `content/extra/robots.txt`, which disallows `/drafts/` and `/feeds/`, does not block the two new files. Because `DELETE_OUTPUT_DIRECTORY` is set in the production settings, the plugin writes after the output directory is rebuilt rather than before.

Last it writes `plugins/llms_txt/README.md` with the two settings and the format, adds a `requirements.txt` with `pelican[markdown]` so the next person and the build install the same thing, and commits the plugin, the config lines and the two docs. The generated `output/` stays untracked, as the existing `.gitignore` already has it.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add llms.txt and llms-full.txt to this Pelican site. Read pelicanconf.py, publishconf.py, the Makefile, the theme's templates and the content tree first, and note which settings file sets SITEURL. Write a local plugin under plugins/ that runs on Pelican's finalized signal and writes output/llms.txt (an H1 with the site name, a short blockquote, a Pages section and a News section with one linked line per article, newest first, with its date and HTML-free summary) and output/llms-full.txt with each article's full text, using only the standard library. Register it in pelicanconf.py with PLUGIN_PATHS and PLUGINS. Install Pelican with the markdown extra, run make html and make publish, and check both files: every article and public page is listed, hidden or not-found pages are not, links resolve against SITEURL, and robots.txt does not block them. Add a requirements.txt and a short README for the plugin. Open a pull request that quotes the first lines of the generated llms.txt from the production build.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

A pull request with the plugin file, its README, the two config lines in `pelicanconf.py` and a `requirements.txt`. The description quotes the head of the production `llms.txt`, the counts under each section, and the two `make` commands it ran with their final lines, so a reviewer can see what the site will publish without building it.

The reviewer checks three things: that the article list matches the site's index page in order and count, which the quoted file shows; that the blockquote reads as a description of the site and not as the first sentence of an old release note; and that the plugin writes nothing when `SITEURL` is empty that would break once it is set, which the two builds in the run's record answer. Merging and running `make upload` publishes the files beside the rest of the site.

## Run it on every terminal hand-off

A maintainer who wants this done before the next release note starts it from the terminal with this body and the repository in `resources`, then goes back to writing. The run reads the real settings and content, so the file it writes matches the site, and the pull request waits for a review rather than a merge.

Send a `request_id`, a UUID, with the body so a retried start returns the run it already began. When the turn ends, `GET /o/{org}/sessions/{id}` names the pull request, and a follow-up to the same session can ask for a `## Documentation` section that links the docs site from `MENUITEMS`, or for the summary length to change.

## Why run it on Rush?

The useful part of this job is seeing the file a real build produces, with the site's own titles, dates and links, and that means installing Pelican, building twice and reading the output. The agent does that in its own sandbox and keeps every command and its output in the run's record, so the pull request shows the generated file rather than describing it.

The same run fits any Pelican site: a personal blog on a different theme, a project's news site, a documentation site with pages only. Each gets a plugin that reads its own generators and settings, and a pull request built from its own content.

## Frequently asked questions

### Why a plugin instead of a static llms.txt file in content/extra?

A file under `STATIC_PATHS` would be copied as is and go stale with the next post. The plugin reads the articles the build already collected, so every `make publish` rewrites `llms.txt` with the current list. The format is the same either way.

### Does it change my theme or templates?

No. The plugin writes two files into `output/` from Pelican's signal and touches nothing in `themes/notmyidea`. If you want a link to `llms.txt` in the footer, that is a one-line change to `base.html` the pull request leaves to you.

### What about drafts and hidden pages?

Articles with `Status: draft` and pages with `Status: hidden` are not in the lists the generators expose for publishing, so they are left out. The run checks the `not-found.rst` page in particular and says in the pull request whether it appears.

### Will the Markdown post and the reStructuredText posts render the same?

Both reach the plugin as rendered HTML, which it strips to text the same way. The run builds with the `markdown` extra installed so `namespace-plugins.md` is processed; without it Pelican skips the file with a warning, and the pull request says so.

## More Pelican jobs

- [Add Open Graph Tags to a Pelican Blog](https://getrush.ai/run/add-open-graph-tags/pelican/)
- [Add Reading Time to a Pelican Blog](https://getrush.ai/run/add-reading-time/pelican/)
- [Add Search to a Pelican Blog](https://getrush.ai/run/add-search/pelican/)

## The same job on other stacks

- [Add llms.txt to a Sphinx Docs Site](https://getrush.ai/run/add-llms-txt/sphinx/)
- [Add llms.txt to a Starlight Docs Site](https://getrush.ai/run/add-llms-txt/starlight/)
- [Add llms.txt to a VitePress Blog](https://getrush.ai/run/add-llms-txt/vitepress/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
