What the agent does
The agent reads how the site is built before it reads the diff, because a docs pull request breaks in the build long before anyone sees a typo. In a site laid out like testing-library/testing-library-docs, package.json depends on @docusaurus/core and @docusaurus/preset-classic 2.x with React 16, builds with docusaurus build, formats with prettier --write '**/*.{js,json,mdx}' through kcd-scripts' Prettier config, and runs lint-staged from a Husky pre-commit hook, so a contributor who committed with --no-verify can land unformatted MDX. .nvmrc pins Node 16.14, package-lock.json is checked in, and there is no GitHub workflow: netlify.toml builds with npm install; npm run format; npm run build and publishes build, so the only check before merge is a Netlify deploy preview. docusaurus.config.js sets onBrokenLinks: 'log', which means a broken internal link prints a line and the build still passes.
It then maps the content. docs/ holds 114 files: top-level pages such as intro.mdx, guiding-principles.mdx, the examples index and the example-* and ecosystem-* pages, plus directories per framework, dom-testing-library, react-testing-library, queries, angular-testing-library, cypress-testing-library and the others. sidebars.js lists pages by their front matter id, not their file name, so docs/example-formik.md carries id: example-react-formik and appears in the examples sidebar under that id, while example-findByText.md keeps an id equal to its name. The docs preset enables showLastUpdateAuthor and showLastUpdateTime, sets editUrl to the repository's edit/main/ path, and loads the remark-plugin-npm2yarn plugin with sync: true, so a npm2yarn code block must stay a plain npm install line for the tabs to render. Recent merged pull requests on the upstream site are typical of the work: a Vue Router FAQ rewrite, an Angular compatibility table row, a wrapper option for Svelte, grammar fixes across six files, and the removal of a deprecated ecosystem page.
The agent checks out the pull request branch and runs the build with the pinned Node, npm ci then npm run build, and reads the output for two things the config lets through: the onBrokenLinks lines, which it treats as findings, and MDX compile errors, which fail the build outright. It runs npx prettier --check on the changed files with the repository's config, since the Netlify build would reformat them silently. It then checks each changed or added page against the site's structure: a new page's id is in sidebars.js or the review says it is orphaned, a renamed id still matches every to: in the navbar and footer of docusaurus.config.js, a deleted page such as the removed jest-native entry has no remaining link from the examples index or a sibling page, a code block keeps its npm2yarn meta, and an editUrl still resolves.
Then it reads the prose the way a maintainer would. For an API page under docs/dom-testing-library or docs/queries it compares a described option or signature with the library's current README or type definitions, fetched read-only, and flags a claim that no longer matches; for an example page it runs the snippet's imports against the package versions the page names; for a table such as the Angular compatibility matrix it checks the new row against the package's release tags. It posts one review that cites file and line for each finding, quotes the build and Prettier output, and pushes nothing.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Review this pull request to our Docusaurus docs site. Read package.json, .nvmrc, netlify.toml, docusaurus.config.js and sidebars.js first. With the pinned Node, run npm ci and npm run build on the pull request branch and treat every broken-link line and MDX error as a finding, since the config only logs broken links. Run prettier --check on the changed files with the repository's config. For each changed page check that its front matter id is referenced in sidebars.js, that any id or path it renames is updated in the navbar and footer links of docusaurus.config.js and in other pages, that npm2yarn code blocks keep their meta, and that a deleted page leaves no link behind. Read the changed prose and compare any API option, signature or version it states with the library's current README or types, fetched read-only. Post one review on the pull request citing file and line for every finding, with the build and prettier output quoted. Do not push commits.",
"resources": [
{
"url": "https://github.com/your-org/your-app/pull/42"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is one review on the pull request. It opens with the build result on the pinned Node and the onBrokenLinks lines, if any, then the Prettier check, then findings ordered by file: a page whose id is in no sidebar, a footer link that still points at a renamed id, a code block that lost its npm2yarn tag, and any sentence whose option name or version does not match the library it documents, each with the file, the line and what the agent compared it against. A pull request that only fixes grammar across six files gets a short review that says the build is clean and the edits read correctly.
The session keeps the record: the npm ci and npm run build output in full, the Prettier diff, the pages it fetched to check a claim and what each said. GET /o/{org}/sessions/{id}/messages returns that transcript, so a maintainer who doubts a finding about an API option can see the README line it came from.
Before merging, a reviewer looks at the Netlify deploy preview for the pages the review names, since the agent's build proves the site compiles but not how a tab or admonition renders, and decides on any prose finding that is a judgement call. A follow-up message to the same session, such as asking it to check the same claim against an older release, continues from the same checkout.
Run it on every pull request opened
An automation in the Project that holds the docs site, with this prompt and a github trigger on pull_request.opened and pull_request.synchronize, reviews every pull request and every push to one. The trigger adds the pull request after the prompt and fills nothing else in; its runs are read-only, which is all a review needs. It fires only for pull requests from an owner, member or collaborator, and most docs contributions arrive from forks, so a maintainer sends this body from their own server with the fork's pull request URL in resources for those, and the agent checks out that branch through the GitHub App installation.
A maintainer who looks after several documentation sites adds the same automation to each Project, with max_runs_per_hour set for the pace of contributions, so a batch of all-contributors bot updates does not queue a dozen reviews.
The use case this job follows, with its automation and trigger: Review every pull request.
Why run it on Rush?
A docs site without CI gets reviewed by eye and a deploy preview, and the things that slip through are structural: an id nobody added to the sidebar, a footer link to a page that moved, an example that imports an option the library renamed two releases ago. In a Rush sandbox the agent builds with the pinned Node 16 the site expects, reads the broken-link lines the config only logs, and checks each claim against the library's own files, with nothing installed on a maintainer's laptop.
The review carries its evidence with it: the build output, the Prettier result and the transcript of what it compared each sentence against. Several pull requests get reviewed at the same time in separate sandboxes, so a weekend's worth of community edits is read by Monday instead of waiting in a queue.
Frequently asked questions
The site builds on Netlify. Why build it again?
Because the Netlify build runs npm run format first and sets onBrokenLinks to log, so it reformats unformatted files and passes with broken links. The agent builds the branch as committed, reports the broken-link lines as findings, and runs Prettier in check mode so the pull request shows what the deploy would have rewritten.
Can it tell whether the documentation is correct?
For the parts that have a source of truth. An option name, a function signature or a version number is checked against the library's README, type definitions or release tags, fetched read-only, and the review names what it compared. Whether an explanation is clear is left to the maintainer, and the review says so rather than guessing.
What about pull requests from forks?
A github trigger never fires for a fork. A maintainer starts the review by sending this body with the pull request URL in resources; the agent checks out the fork's branch through your GitHub App installation and posts the review the same way. For a community docs site that is most pull requests, so a small script on the maintainer's side does it on each new pull request.
Does it need the pinned Node version?
Yes. Docusaurus 2 with React 16 builds on the Node in .nvmrc, 16.14 in the example repository, and a newer Node can fail in ways that have nothing to do with the pull request. The agent installs that version in the sandbox and says which it used in the review.