What the agent does
The agent first finds out what already guards a pull request. In an app like payloadcms/website, Payload 3 and its Next.js front end share one folder, there is no test script, and the only workflow reacts to a Payload release by syncing documentation. Nothing runs on a pull request. What the repository does have is pnpm lint, ESLint with Payload's own config, and a tsconfig.json with noEmit set, so tsc is a pure type-check. The agent installs pnpm at the version in packageManager and the Node major in mise.toml, checks out the base branch in a second directory, and runs both commands on each side so the review reports a difference.
It then checks the generated files, which is the mistake specific to this stack. src/payload-types.ts is committed and is produced by pnpm generate:types from the collections, globals and blocks; the admin import map under src/app/(payload)/admin comes from pnpm generate:importmap. If the diff edits a field or adds a custom admin component and neither file moved, the agent runs both generators on the head and compares the result with what is committed. A stale types file compiles today and misleads every later change.
Access control gets the closest reading. Every collection in src/collections declares an access block, and the existing ones set the pattern: writes are isAdmin; collections with drafts read through publishedOnly, which returns a query limiting others to _status equal to published; readVersions is admin-only; and Users is publicly readable while its email and roles fields carry field-level rules. Against that pattern the agent flags a new collection with no access block, a drafts-enabled collection whose read returns true, a personal-data field added to a publicly readable collection without a field-level read, and an access function that returns true where its neighbours return a query.
It reads the code that goes around those rules too. Payload's Local API skips access control unless a call passes overrideAccess: false, so a new route handler that calls payload.find with input from the request is a finding. Custom endpoints in payload.config.ts authenticate themselves: createReleasePost compares an x-release-secret header with an environment variable and answers 401, and a new endpoint is held to that. A field rename with no file under src/migrations is noted, since the database is MongoDB and old documents keep the old key. The agent does not run pnpm build, which calls payload migrate and needs a database, and it says so in the review.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Review this pull request to our Payload CMS app. Read package.json, pnpm-workspace.yaml, mise.toml, tsconfig.json, eslint.config.js, .env.example, src/payload.config.ts, every file under src/access, src/access.ts, the access block of every collection under src/collections, src/migrations and the workflows under .github first. Note which commands exist and which need a database. Install pnpm at the version in packageManager and the Node major in mise.toml. Check out the base branch in a second directory and run pnpm install, pnpm lint and pnpm exec tsc on both base and head; report only what the pull request changes and list problems that already exist on the base separately. On the head run pnpm generate:types and pnpm generate:importmap and report any difference from the committed src/payload-types.ts and import map. Do not run pnpm build or start the app. Then read the diff and flag: a new collection or global without an access block; a collection with drafts whose read access is not publishedOnly or an equivalent query; a field holding an email, a token, an internal id or a role on a publicly readable collection without field-level read access; an access function that returns true for a signed-in user where the existing ones return a where query; a Local API call that takes request input without overrideAccess false; a new custom endpoint with no authentication check; a direct call to the database adapter's model that skips hooks and access, unless the diff says why; a renamed or removed field with no migration; a collection rendered on the site with no revalidation hook; and a new environment variable missing from .env.example. Post one review on the pull request with the command and generator differences first, then findings by file and line with one sentence of reason each. Do not push commits.",
"resources": [
{
"url": "https://github.com/your-org/your-app/pull/42"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
One review on the pull request. It opens with the commands: lint and type-check on base and on head, and whether the two generators reproduced the committed files. A content-model change that forgot pnpm generate:types gets that as its first line, with the names of the types that differ. A pull request that only touches a React component gets a short review saying the checks are unchanged and no collection, access rule or endpoint is in the diff.
Findings follow by file and line, each stated as what a caller could do. An anonymous request could read drafts of the new collection through the REST API. A signed-in user could update another user's document, because the function returns true where isAdminOrSelf returns a query on id. A route handler returns documents the caller's role should not see, because the Local API call runs with access control off. Where a finding leans on a convention, the review quotes the existing line that sets it. Every command's output is in the session's transcript at GET /o/{org}/sessions/{id}/messages.
The review does not decide policy. Whether a collection should be public, whether a counter may write to the database model directly as the docs-feedback vote endpoint does, and whether a migration is needed for a renamed field are raised as questions for the people who own the data. It states what it did not do: no build, no database, no admin panel opened.
Run it on every pull request
In the Project that holds the repository, add an automation with this page's prompt, read-only switched on, and a github trigger on pull_request.opened and pull_request.synchronize. Each pull request is reviewed when it opens and again after every push. The pull request is placed after the prompt as the event; nothing in the prompt is filled in. With read-only on, the run can read the repository and comment and cannot push, which is what a reviewer should be able to do.
Only pull requests by an owner, member or collaborator fire the trigger; forks and bot senders never do. For an open-source Payload project that takes outside contributions, a maintainer pushes the contributor's commits to a branch in the repository and opens a pull request from it. To limit reviews to content-model work, use pull_request.labeled with a label such as schema in the trigger's labels, and have a maintainer apply it.
The use case this job follows, with its automation and trigger: Review every pull request.
Why run it on Rush?
In a Payload app the security model is ordinary TypeScript: a function per operation that returns true, false or a query. That makes it easy to write and easy to get subtly wrong, and neither ESLint nor the compiler knows that read: () => true on a collection with drafts publishes the drafts. With no test suite and no check on pull requests, the only safeguard is a reviewer who remembers to open the collection file and compare it with its neighbours. An agent does that comparison on every pull request, with the whole repository checked out, and cites the neighbour it compared against.
The mechanical half benefits from a clean machine. Regenerating types and the import map on a laptop depends on which branch was last installed. In a Rush sandbox the base and head are installed fresh for each pull request with the pinned pnpm, the generators run against exactly the head commit, and several pull requests are reviewed at once in separate sandboxes. The transcript keeps each command and its output, so a claim that the types are stale comes with the diff that shows it.
Frequently asked questions
Does the review need MongoDB or my environment variables?
No. Lint, the type-check and the two generators read the config and the source, not the database. The request tells the agent not to build or start the app, so no DATABASE_URI, PAYLOAD_SECRET or third-party key has to be in the Project. If you want the review to boot the admin panel as well, that is a different prompt and the agent has to start a MongoDB in its sandbox first.
What does it check about access control that a linter cannot?
Meaning. A linter sees a function that returns a boolean. The agent reads which operation the function guards, whether the collection has drafts or versions, what the fields contain and what the sibling collections do, and says when a rule lets someone read or write more than the pattern in the repository allows.
Why flag Local API calls?
Because payload.find, payload.update and the rest run with access control off unless overrideAccess: false is passed, usually together with the requesting user. That default is right for scripts and seed code and wrong for a route handler that passes along a query from the browser. The review separates the two by reading where the input comes from.
Will it regenerate and commit the types for the author?
Not as a reviewer. With read-only on it cannot push, and the request says to push nothing. It reports that the committed file is stale and names the command. If you would rather have that fixed automatically, start a separate run on the pull request's branch that runs the generators and commits the result.
Does it work for Payload with Postgres?
Yes. The access, endpoint and generated-file checks are the same. The migration check gets stricter: with a SQL adapter a changed field needs a migration file for the schema itself, so the agent looks for one in the migrations folder whenever a collection's fields change.