What the agent does
The agent reads how the extension is built and tested before it writes a workflow. In an extension shaped like helloworld-test-sample in microsoft/vscode-extension-samples, package.json sets engines.vscode to ^1.100.0, main to ./out/extension.js, and scripts compile for tsc -p ./, lint for eslint, pretest that runs compile, and test that runs node ./out/test/runTest.js. The dev dependencies are @vscode/test-electron, mocha, glob, eslint 9 with typescript-eslint, typescript ^5.9.2 and @types/node ^22, and package-lock.json makes npm the package manager. tsconfig.json compiles CommonJS to out/ with strict on and excludes .vscode-test, the directory where downloaded VS Code builds land.
src/test/runTest.ts calls runTests from @vscode/test-electron with the extension root as extensionDevelopmentPath and out/test/suite/index as extensionTestsPath, which downloads a VS Code build and launches it with the extension loaded; its catch prints Failed to run tests and exits 1 without the error. src/test/suite/index.ts builds a Mocha tdd runner over every **/**.test.js under out/test, and extension.test.ts holds one sample test. Nothing in the repository runs any of this on a pull request: the samples repository's own ci.yml runs only npm run validate, which checks the sample list. Because the test launches a real VS Code window, it needs a display, which on a Linux runner means xvfb.
The agent writes .github/workflows/ci.yml triggered on pull requests and pushes to the default branch. One job runs on a matrix of ubuntu-latest, windows-latest and macos-latest, checks out, sets up Node 22 to match @types/node with the npm cache keyed on package-lock.json, runs npm ci, npm run lint and npm run compile, then runs the tests with xvfb-run -a npm test on Linux and plain npm test on the other two, using an if on runner.os. It caches .vscode-test keyed on the engines.vscode range so the VS Code download is not repeated on every run, pins each action to a major version, and sets permissions: contents: read. It also makes runTest.ts print the caught error before exiting, so a failure in CI says what failed rather than only that it did.
Before pushing, the agent runs the same steps in its sandbox: npm ci, npm run lint, npm run compile, then installs xvfb and the shared libraries VS Code needs on a headless Linux box and runs xvfb-run -a npm test, reading the Mocha summary with its passing count. If lint reports a problem the sample never checked, it fixes it in a separate commit. Then it opens the pull request, which is the first run of the new workflow, and reads the three matrix legs before it calls the job done.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add GitHub Actions CI to this VS Code extension and open a pull request. Read package.json, package-lock.json, tsconfig.json, eslint.config.mjs, src/test/runTest.ts, src/test/suite/index.ts and src/test/suite/extension.test.ts first. Write .github/workflows/ci.yml that runs on pull requests and pushes to the default branch, with a matrix of ubuntu, windows and macos: setup-node on Node 22 with the npm cache keyed on the lockfile, npm ci, npm run lint, npm run compile, then npm test under xvfb-run on Linux and directly elsewhere. Cache the .vscode-test download, pin actions to a major version and set read-only permissions. Make runTest.ts log the error it catches. Run npm ci, lint, compile and the tests under xvfb in the sandbox before you push, and quote the Mocha summary in the pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request on its own branch that adds .github/workflows/ci.yml and changes src/test/runTest.ts to log the caught error. If npm run lint found something, a second commit fixes it. The description lists the matrix, the Node version and why, the cache keys, and the one command that differs by platform.
The proof is two-fold: the sandbox output quoted in the description and kept in the run's record, with npm run lint, npm run compile and the Mocha summary from xvfb-run -a npm test, and the workflow's own first run on the pull request, where the three legs report. The agent reads those legs and fixes a platform-specific failure, such as a path separator in a test, before it reports the job finished.
Before merging, a reviewer checks that the engines.vscode range in the cache key is the one they bump on upgrades, that running the matrix on three operating systems is worth the minutes for this extension, and that the action pins match the organisation's policy. Then they make the workflow a required check on the default branch.
Run it on every extension repository without CI
A team that maintains several extensions sends one POST /o/{org}/sessions per repository, with this prompt and the repository in resources. GET /o/{org}/github lists every repository the Rush GitHub App installation can reach, so a script can start a run for each one whose package.json has an engines.vscode field and whose .github/workflows/ directory is missing. Each run reads that extension's own scripts, so one on @vscode/test-cli gets a vscode-test step instead of node ./out/test/runTest.js, and each opens its own pull request.
Set a webhook URL on the API key and Rush POSTs run.completed or run.failed when a run's turn finishes, with session_url so your server can read the pull request link. Send a fresh UUID as request_id with each request so a retry returns the run already started.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
Extension tests are integration tests that download and launch VS Code, so the workflow that runs them fails for reasons that have nothing to do with the extension: no display on Linux, a download repeated on every run, a catch that hides the real error. On Rush the agent installs xvfb and the libraries in its own sandbox, runs the exact npm test the workflow will run, and reads the first real run of the workflow on the pull request, so the job is done when the checks are green rather than when the YAML is written.
Every command and its output stay in the run's record, the workflow is a single readable file, and nothing becomes a required check until a person merges. The same request can add CI to every extension the team maintains, each run reading its own test runner, one pull request each.
Frequently asked questions
Why does Linux need xvfb-run?
@vscode/test-electron launches a full VS Code window with the extension loaded, and on a Linux runner there is no display server. xvfb-run -a starts a virtual framebuffer for the duration of the command. Windows and macOS runners have a desktop session, so npm test runs there as it is.
Which Node version does the workflow use?
Node 22, because @types/node is pinned to ^22 and that is what tsc type-checks against. The agent says so in the pull request; if your extension bundles for a specific VS Code Electron version, name the Node version you want in the prompt.
Will the VS Code download happen on every run?
No. The workflow caches .vscode-test keyed on the engines.vscode range from package.json, so the build downloaded for ^1.100.0 is reused until that range changes. The first run on each operating system still downloads once.
Does the agent change my tests?
Only runTest.ts, and only to print the error it already catches before exiting, so a CI failure names the cause. The Mocha suite, the test files and the pretest compile step stay as they are.
More VS Code Extension jobs
The same job on other stacks
- Set Up GitHub Actions CI for an AdonisJS App (AdonisJS)
- Add GitHub Actions CI to an Airflow Project (Apache Airflow)
- Set Up GitHub Actions CI for an AWS CDK App (AWS CDK)