# Review Every Pull Request on a VS Code Extension Repository

Yes. Each pull request gets an agent that checks it out, compiles, lints and tests the extensions the diff touches, and posts one review citing file and line.

[Run this on Rush](https://cloud.getrush.ai/run/review-pull-requests/vscode-extension/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent checks the pull request out in its own sandbox and reads the repository's shape before the diff. In a repository like microsoft/vscode-extension-samples that shape is unusual: 86 top-level directories, 81 of them extensions with their own `package.json`, and a root `package.json` whose scripts fan out over them with `tsx .scripts/run-script.ts compile` and `lint`. The root `validate` script, which is the only thing `.github/workflows/ci.yml` runs on a pull request after `npm ci --workspaces=false`, checks two things: that every directory is listed in `.scripts/samples.ts`, which has 82 `path` entries, and that the README's table between the `SAMPLES_BEGIN` and `SAMPLES_END` markers matches what `.scripts/update-readme.ts` would generate. CI does not compile a single extension, so the review has to.

From the diff the agent lists the extension directories touched and, in each, runs `npm install`, `npm run compile` and `npm run lint` with that extension's own `package.json`, the way a maintainer would. Each extension's `eslint.config.mjs` extends the typescript-eslint recommended and stylistic sets with `curly` on, semicolons required, imports named in camelCase or PascalCase and unused variables an error unless prefixed with an underscore; its `tsconfig.json` targets ES2024 as CommonJS with `strict` on and `out` as the output directory. Formatting follows `.editorconfig` and `.prettierrc.json`: tabs, width 4, single quotes, 92 columns. An extension with a `test` script, such as `helloworld-test-sample`, whose `src/test/runTest.ts` downloads VS Code through `@vscode/test-electron` and runs a Mocha suite, gets `npm test` under `xvfb-run`, because the sandbox has no display of its own.

Then it reads the diff with the extension's manifest open, looking for the mistakes this kind of repository collects. A command declared under `contributes.commands` with no matching `registerCommand` in `extension.ts`, or the reverse; a disposable created in `activate` and never pushed to `context.subscriptions`; a `main` entry that does not match the compiler's `outDir`; an `engines.vscode` range that disagrees with the `@types/vscode` version in `devDependencies`, as both are `^1.100.0` in the hello world sample; an `activationEvents` entry VS Code now infers from `contributes`; a sample README that no longer describes what the code does; and a new or renamed directory that `samples.ts` and the README table do not know about, which is the one failure `validate` would catch.

It posts one GitHub review. Each finding cites the file and line, says whether it blocks merge or is a suggestion, and quotes the compile or lint output it rests on. Where a finding is about runtime behaviour the agent writes a short Mocha case in the sandbox to confirm it before saying so, but does not add it to the branch, and it pushes no commits of its own.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Review this pull request on a VS Code extension samples repository. Check it out. Read the root package.json and .scripts/validate.ts, then list the extension directories the diff touches. In each, run npm install, npm run compile and npm run lint with that extension's own package.json, and npm test under xvfb-run where a test script exists. Run npm run validate at the root when a directory was added, renamed or removed. Read each changed extension.ts beside its package.json: check that every contributes.commands entry has a registerCommand and vice versa, that disposables are pushed to context.subscriptions, that main matches the tsconfig outDir, that engines.vscode agrees with the @types/vscode version, that activationEvents are not redundant, and that the sample's README still matches the code. Where a finding is about behaviour, confirm it with a short Mocha case in the sandbox without adding it to the branch. Post one review that quotes each compile, lint and test result, separates blocking issues from suggestions, and cites file and line for every finding. Do not push commits.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app/pull/42"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a review on the pull request, not a commit. It opens with the extensions the diff touched and, for each, the result of `npm run compile`, `npm run lint` and `npm test` where one exists, quoted from the sandbox. Then the findings, each anchored to a file and line in the diff, grouped into what blocks merge and what is a suggestion, with the manifest or compiler line that supports it.

The run is listed in the Project with its full transcript, readable at `GET /o/{org}/sessions/{id}/messages`: every `npm` command, its output, and each file the agent opened. The repository's own `CI` workflow still runs `validate` on the pull request as before, so a maintainer sees both the registry check and the per-extension build that CI skips.

Before acting on the review, a maintainer checks that each blocking finding names a real break rather than a stylistic preference, and that a finding about a command or disposable points at the right `extension.ts`, since several samples share names. Anything the agent got wrong is answered on the review thread like any other reviewer's comment.

## Run it on every pull request opened

Create an automation in the Project with a `github` trigger on `pull_request.opened` and `pull_request.synchronize`, `branch_patterns` set to `main`, and this prompt. Each fire adds the pull request after the prompt, so every pull request and every push to it gets its own sandbox, its own `node_modules` and its own review. Mark the automation `read_only` so its runs can comment and never push. Only work by an owner, member or collaborator fires it; pull requests from forks do not.

`max_runs_per_hour`, between 1 and 30, keeps a burst of dependency pull requests across many sample directories to a bounded number of installs; a fire past the cap is skipped as `rate_limited` and the pull request is reviewed on its next push. The review is the result on the pull request, and the Project lists the run beside it.

## Why run it on Rush?

A repository of 81 small extensions is expensive to review by hand because the cost is per directory: each has its own `npm install`, its own compile and its own lint, and the one workflow in `.github/workflows/ci.yml` runs none of them. A reviewer who does this properly spends the time on setup rather than on reading. In a Rush sandbox the agent pays that cost in its own clone for every pull request, in parallel across pull requests, and the maintainer reads a review that already has the compile and lint results in it.

The review is evidence rather than opinion. Every claim quotes the command that produced it, and the transcript keeps the rest, so a maintainer who doubts a finding can read the exact `tsc` or `eslint` output instead of re-running it. Several pull requests at once mean several sandboxes at once, each with its own VS Code download for the Mocha suites, and none of them on anyone's laptop.

## Frequently asked questions

### Does the agent compile every sample on each pull request?

No. It compiles and lints the directories the diff touches, which is what CI leaves out, and runs the root `validate` script when a directory was added, renamed or removed. Compiling all 81 extensions on every pull request would spend most of the run on code the pull request did not change; ask for it in the prompt if you want a periodic full sweep.

### How does it run extension tests without a display?

Extensions with a `test` script, like `helloworld-test-sample`, use `@vscode/test-electron`, which downloads VS Code and launches it with the extension under test. The sandbox runs that under `xvfb-run`, which gives Electron a virtual display. The transcript keeps the download and the Mocha output.

### Will it push fixes to the branch?

No. The prompt says not to push and the automation is marked `read_only`, which means its runs can read the repository and comment but cannot push. Suggested changes appear as review comments with the proposed code; a maintainer or the author applies them.

### What does it do with a pull request that adds a new sample?

It runs the new directory's compile, lint and any tests, then `npm run validate` at the root, which fails unless the directory is listed in `.scripts/samples.ts` and the README table has been regenerated with `update-readme`. It also reads the new sample's README against its `package.json` contributions, since the samples table is built from the `apis` and `contributions` fields in `samples.ts`.

### Can it review a pull request opened from a fork?

Not through the GitHub trigger; only work by an owner, member or collaborator fires an automation, and fork pull requests never do. A maintainer can still start one run by hand with this request body and the fork pull request's URL in `resources`, and the review posts the same way.

## More VS Code extension jobs

- [Upgrade a VS Code Extension to the Latest API](https://getrush.ai/run/upgrade-vscode-engine/vscode-extension/)
- [Write Integration Tests for a VS Code Extension](https://getrush.ai/run/write-extension-tests/vscode-extension/)
- [Add Code Coverage to CI for a VS Code Extension](https://getrush.ai/run/add-coverage-to-ci/vscode-extension/)

## The same job on other stacks

- [Review Pull Requests Automatically in a Vue App](https://getrush.ai/run/review-pull-requests/vue/)
- [Review Pull Requests on a Wagtail Site](https://getrush.ai/run/review-pull-requests/wagtail/)
- [Review Pull Requests on a WordPress Plugin](https://getrush.ai/run/review-pull-requests/wordpress-plugin/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
