What the agent does
The agent checks the pull request out in its own sandbox and reads the repository's shape before the diff. In a repository like microsoft/vscode-extension-samples that shape is unusual: 86 top-level directories, 81 of them extensions with their own package.json, and a root package.json whose scripts fan out over them with tsx .scripts/run-script.ts compile and lint. The root validate script, which is the only thing .github/workflows/ci.yml runs on a pull request after npm ci --workspaces=false, checks two things: that every directory is listed in .scripts/samples.ts, which has 82 path entries, and that the README's table between the SAMPLES_BEGIN and SAMPLES_END markers matches what .scripts/update-readme.ts would generate. CI does not compile a single extension, so the review has to.
From the diff the agent lists the extension directories touched and, in each, runs npm install, npm run compile and npm run lint with that extension's own package.json, the way a maintainer would. Each extension's eslint.config.mjs extends the typescript-eslint recommended and stylistic sets with curly on, semicolons required, imports named in camelCase or PascalCase and unused variables an error unless prefixed with an underscore; its tsconfig.json targets ES2024 as CommonJS with strict on and out as the output directory. Formatting follows .editorconfig and .prettierrc.json: tabs, width 4, single quotes, 92 columns. An extension with a test script, such as helloworld-test-sample, whose src/test/runTest.ts downloads VS Code through @vscode/test-electron and runs a Mocha suite, gets npm test under xvfb-run, because the sandbox has no display of its own.
Then it reads the diff with the extension's manifest open, looking for the mistakes this kind of repository collects. A command declared under contributes.commands with no matching registerCommand in extension.ts, or the reverse; a disposable created in activate and never pushed to context.subscriptions; a main entry that does not match the compiler's outDir; an engines.vscode range that disagrees with the @types/vscode version in devDependencies, as both are ^1.100.0 in the hello world sample; an activationEvents entry VS Code now infers from contributes; a sample README that no longer describes what the code does; and a new or renamed directory that samples.ts and the README table do not know about, which is the one failure validate would catch.
It posts one GitHub review. Each finding cites the file and line, says whether it blocks merge or is a suggestion, and quotes the compile or lint output it rests on. Where a finding is about runtime behaviour the agent writes a short Mocha case in the sandbox to confirm it before saying so, but does not add it to the branch, and it pushes no commits of its own.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Review this pull request on a VS Code extension samples repository. Check it out. Read the root package.json and .scripts/validate.ts, then list the extension directories the diff touches. In each, run npm install, npm run compile and npm run lint with that extension's own package.json, and npm test under xvfb-run where a test script exists. Run npm run validate at the root when a directory was added, renamed or removed. Read each changed extension.ts beside its package.json: check that every contributes.commands entry has a registerCommand and vice versa, that disposables are pushed to context.subscriptions, that main matches the tsconfig outDir, that engines.vscode agrees with the @types/vscode version, that activationEvents are not redundant, and that the sample's README still matches the code. Where a finding is about behaviour, confirm it with a short Mocha case in the sandbox without adding it to the branch. Post one review that quotes each compile, lint and test result, separates blocking issues from suggestions, and cites file and line for every finding. Do not push commits.",
"resources": [
{
"url": "https://github.com/your-org/your-app/pull/42"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a review on the pull request, not a commit. It opens with the extensions the diff touched and, for each, the result of npm run compile, npm run lint and npm test where one exists, quoted from the sandbox. Then the findings, each anchored to a file and line in the diff, grouped into what blocks merge and what is a suggestion, with the manifest or compiler line that supports it.
The run is listed in the Project with its full transcript, readable at GET /o/{org}/sessions/{id}/messages: every npm command, its output, and each file the agent opened. The repository's own CI workflow still runs validate on the pull request as before, so a maintainer sees both the registry check and the per-extension build that CI skips.
Before acting on the review, a maintainer checks that each blocking finding names a real break rather than a stylistic preference, and that a finding about a command or disposable points at the right extension.ts, since several samples share names. Anything the agent got wrong is answered on the review thread like any other reviewer's comment.
Run it on every pull request opened
Create an automation in the Project with a github trigger on pull_request.opened and pull_request.synchronize, branch_patterns set to main, and this prompt. Each fire adds the pull request after the prompt, so every pull request and every push to it gets its own sandbox, its own node_modules and its own review. Mark the automation read_only so its runs can comment and never push. Only work by an owner, member or collaborator fires it; pull requests from forks do not.
max_runs_per_hour, between 1 and 30, keeps a burst of dependency pull requests across many sample directories to a bounded number of installs; a fire past the cap is skipped as rate_limited and the pull request is reviewed on its next push. The review is the result on the pull request, and the Project lists the run beside it.
The use case this job follows, with its automation and trigger: Review every pull request.
Why run it on Rush?
A repository of 81 small extensions is expensive to review by hand because the cost is per directory: each has its own npm install, its own compile and its own lint, and the one workflow in .github/workflows/ci.yml runs none of them. A reviewer who does this properly spends the time on setup rather than on reading. In a Rush sandbox the agent pays that cost in its own clone for every pull request, in parallel across pull requests, and the maintainer reads a review that already has the compile and lint results in it.
The review is evidence rather than opinion. Every claim quotes the command that produced it, and the transcript keeps the rest, so a maintainer who doubts a finding can read the exact tsc or eslint output instead of re-running it. Several pull requests at once mean several sandboxes at once, each with its own VS Code download for the Mocha suites, and none of them on anyone's laptop.
Frequently asked questions
Does the agent compile every sample on each pull request?
No. It compiles and lints the directories the diff touches, which is what CI leaves out, and runs the root validate script when a directory was added, renamed or removed. Compiling all 81 extensions on every pull request would spend most of the run on code the pull request did not change; ask for it in the prompt if you want a periodic full sweep.
How does it run extension tests without a display?
Extensions with a test script, like helloworld-test-sample, use @vscode/test-electron, which downloads VS Code and launches it with the extension under test. The sandbox runs that under xvfb-run, which gives Electron a virtual display. The transcript keeps the download and the Mocha output.
Will it push fixes to the branch?
No. The prompt says not to push and the automation is marked read_only, which means its runs can read the repository and comment but cannot push. Suggested changes appear as review comments with the proposed code; a maintainer or the author applies them.
What does it do with a pull request that adds a new sample?
It runs the new directory's compile, lint and any tests, then npm run validate at the root, which fails unless the directory is listed in .scripts/samples.ts and the README table has been regenerated with update-readme. It also reads the new sample's README against its package.json contributions, since the samples table is built from the apis and contributions fields in samples.ts.
Can it review a pull request opened from a fork?
Not through the GitHub trigger; only work by an owner, member or collaborator fires an automation, and fork pull requests never do. A maintainer can still start one run by hand with this request body and the fork pull request's URL in resources, and the review posts the same way.
More VS Code extension jobs
The same job on other stacks
- Review Pull Requests Automatically in a Vue App (Vue)
- Review Pull Requests on a Wagtail Site (Wagtail)
- Review Pull Requests on a WordPress Plugin (WordPress plugin)