All jobs

Add llms.txt to a Pelican Site

Yes. An agent writes a small Pelican plugin that builds llms.txt and llms-full.txt from your articles and pages on every build, checks the output and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent starts with how the site is built. In a Pelican site like getpelican/pelican-blog, pelicanconf.py sets PATH = "content", THEME = "themes/notmyidea", DEFAULT_PAGINATION = 5, STATIC_PATHS of images and extra, and an EXTRA_PATH_METADATA table that moves extra/robots.txt and the touch icons to the site root; publishconf.py imports it, sets SITEURL = "https://blog.getpelican.com" and DELETE_OUTPUT_DIRECTORY = True. There is no PLUGINS or PLUGIN_PATHS setting, no requirements.txt and no workflow. The Makefile builds with make html (which runs clean first), make publish for the production settings, and make upload, which rsyncs output/ to the server over SSH. Content is 18 reStructuredText release announcements under content/news/, from pelican-3.0.rst to pelican-4.7.rst, one Markdown post, namespace-plugins.md, with Title, Date, Modified and Slug in its header, and three pages under content/pages/: index.rst (with :save_as: index.html and an empty :url:, so the site root is this page rather than the article list), gratitude.rst and not-found.rst.

Then it decides where the file comes from. A hand-written llms.txt goes stale with the next post, so the agent writes a plugin, plugins/llms_txt/__init__.py, and registers it with two new lines in pelicanconf.py: PLUGIN_PATHS = ["plugins"] and PLUGINS = ["llms_txt"]. The plugin connects to Pelican's finalized signal, after every generator has run, and reads the articles and pages the generators collected, in the same order the theme's index.html lists them. It writes output/llms.txt in the llms.txt format: an H1 with SITENAME, a one-paragraph blockquote taken from the index.rst page's first paragraph, a ## Pages list linking gratitude and the index page, and a ## News list with one line per article: its title linked at SITEURL plus the article's url, its date, and its summary with the HTML stripped. llms-full.txt holds the same headings followed by each article's full text, converted from the rendered HTML with the standard library's html.parser, so no new dependency is added. Both files use SITEURL from whichever settings file ran, so make html writes relative links and make publish writes absolute ones.

The agent then builds the site both ways to see the result. The repository pins no Pelican version, so it installs the current Pelican release with the markdown extra, which the Markdown post and the MARKDOWN extension table in pelicanconf.py need, and runs make html and make publish. It checks that output/llms.txt lists 19 entries under ## News and 2 under ## Pages, that the not-found.rst page is left out, that no entry carries the {filename} or {static} link syntax unresolved, and that content/extra/robots.txt, which disallows /drafts/ and /feeds/, does not block the two new files. Because DELETE_OUTPUT_DIRECTORY is set in the production settings, the plugin writes after the output directory is rebuilt rather than before.

Last it writes plugins/llms_txt/README.md with the two settings and the format, adds a requirements.txt with pelican[markdown] so the next person and the build install the same thing, and commits the plugin, the config lines and the two docs. The generated output/ stays untracked, as the existing .gitignore already has it.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add llms.txt and llms-full.txt to this Pelican site. Read pelicanconf.py, publishconf.py, the Makefile, the theme's templates and the content tree first, and note which settings file sets SITEURL. Write a local plugin under plugins/ that runs on Pelican's finalized signal and writes output/llms.txt (an H1 with the site name, a short blockquote, a Pages section and a News section with one linked line per article, newest first, with its date and HTML-free summary) and output/llms-full.txt with each article's full text, using only the standard library. Register it in pelicanconf.py with PLUGIN_PATHS and PLUGINS. Install Pelican with the markdown extra, run make html and make publish, and check both files: every article and public page is listed, hidden or not-found pages are not, links resolve against SITEURL, and robots.txt does not block them. Add a requirements.txt and a short README for the plugin. Open a pull request that quotes the first lines of the generated llms.txt from the production build.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request with the plugin file, its README, the two config lines in pelicanconf.py and a requirements.txt. The description quotes the head of the production llms.txt, the counts under each section, and the two make commands it ran with their final lines, so a reviewer can see what the site will publish without building it.

The reviewer checks three things: that the article list matches the site's index page in order and count, which the quoted file shows; that the blockquote reads as a description of the site and not as the first sentence of an old release note; and that the plugin writes nothing when SITEURL is empty that would break once it is set, which the two builds in the run's record answer. Merging and running make upload publishes the files beside the rest of the site.

Run it on every terminal hand-off

A maintainer who wants this done before the next release note starts it from the terminal with this body and the repository in resources, then goes back to writing. The run reads the real settings and content, so the file it writes matches the site, and the pull request waits for a review rather than a merge.

Send a request_id, a UUID, with the body so a retried start returns the run it already began. When the turn ends, GET /o/{org}/sessions/{id} names the pull request, and a follow-up to the same session can ask for a ## Documentation section that links the docs site from MENUITEMS, or for the summary length to change.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

The useful part of this job is seeing the file a real build produces, with the site's own titles, dates and links, and that means installing Pelican, building twice and reading the output. The agent does that in its own sandbox and keeps every command and its output in the run's record, so the pull request shows the generated file rather than describing it.

The same run fits any Pelican site: a personal blog on a different theme, a project's news site, a documentation site with pages only. Each gets a plugin that reads its own generators and settings, and a pull request built from its own content.

Frequently asked questions

Why a plugin instead of a static llms.txt file in content/extra?

A file under STATIC_PATHS would be copied as is and go stale with the next post. The plugin reads the articles the build already collected, so every make publish rewrites llms.txt with the current list. The format is the same either way.

Does it change my theme or templates?

No. The plugin writes two files into output/ from Pelican's signal and touches nothing in themes/notmyidea. If you want a link to llms.txt in the footer, that is a one-line change to base.html the pull request leaves to you.

What about drafts and hidden pages?

Articles with Status: draft and pages with Status: hidden are not in the lists the generators expose for publishing, so they are left out. The run checks the not-found.rst page in particular and says in the pull request whether it appears.

Will the Markdown post and the reStructuredText posts render the same?

Both reach the plugin as rendered HTML, which it strips to text the same way. The run builds with the markdown extra installed so namespace-plugins.md is processed; without it Pelican skips the file with a warning, and the pull request says so.

More Pelican jobs

The same job on other stacks

Keep reading