All jobs

Add Code Coverage to CI for a VS Code Extension

Yes. An agent moves your VS Code extension's tests to the VS Code test CLI, turns on its coverage, writes a real test, and publishes the report from CI in a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the extension is tested before it touches the config. In an extension laid out like helloworld-test-sample in microsoft/vscode-extension-samples, package.json declares engines.vscode ^1.100.0, one command extension.helloWorld, main at ./out/extension.js, and scripts where compile is tsc -p ./, pretest compiles and test runs node ./out/test/runTest.js. src/test/runTest.ts calls runTests from @vscode/test-electron ^2.3.9 with the extension root and ./suite/index, and src/test/suite/index.ts builds a Mocha tdd runner over **/**.test.js. The only test, src/test/suite/extension.test.ts, asserts that [1, 2, 3].indexOf(5) is -1, and its import of the extension is commented out. src/extension.ts registers the command and shows an information message. So the suite runs, passes, and executes none of the extension; a coverage number for it today would be zero, and nothing in the repository measures it. tsconfig.json emits CommonJS to out/ with source maps, which coverage needs.

The agent moves the runner to @vscode/test-cli, which is the VS Code team's current entry point and the one with built-in coverage. It adds .vscode-test.mjs with defineConfig({ files: 'out/test/**/*.test.js', mocha: { ui: 'tdd' } }), changes test to vscode-test, keeps pretest as the compile step, and deletes runTest.ts and suite/index.ts, whose jobs the CLI now does. @vscode/test-electron stays, since the CLI drives it. Coverage is then a flag: vscode-test --coverage --coverage-reporter text --coverage-reporter lcov --coverage-reporter json-summary --coverage-output coverage, and the coverage block in the config excludes out/test/**. coverage/ and .vscode-test/ go into .gitignore.

A number is only worth publishing when the suite reaches the code, so the agent replaces the placeholder test with one that does. extension.test.ts gets the extension by id from package.json's publisher and name, vscode-samples.helloworld-sample, awaits activate(), asserts extension.helloWorld is in vscode.commands.getCommands(true), and runs executeCommand('extension.helloWorld'), which resolves once the information message is shown. That exercises activate and the command callback, every line in extension.ts. The agent runs the suite under xvfb-run -a npm test in its sandbox, because the Electron host needs a display, reads the text report, and sets a floor from what it measured in a small script that reads coverage/coverage-summary.json and exits non-zero under the line percentage it saw, rounded down.

CI is the repository's own: the sample inherits nothing from the monorepo's .github/workflows/ci.yml, which only runs a validation script, so the agent adds .github/workflows/test.yml in the extension with actions/setup-node on Node 22, npm ci, xvfb-run -a npm test with the coverage flags, the floor script, coverage/lcov.info uploaded with actions/upload-artifact, and the text summary appended to $GITHUB_STEP_SUMMARY so the number shows on the job page without a paid service. permissions is contents: read. npm run lint with the repository's typescript-eslint and @stylistic config runs before the test step, and the agent runs actionlint on the workflow before opening the pull request.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add code coverage to CI for this VS Code extension. Read package.json, tsconfig.json, eslint.config.mjs, .vscode/launch.json, src/extension.ts and everything under src/test first. Move the test runner to @vscode/test-cli: add .vscode-test.mjs with the compiled test glob and the tdd ui, set the test script to vscode-test, delete runTest.ts and suite/index.ts, keep @vscode/test-electron. Replace the placeholder test with one that gets the extension by its publisher.name id, activates it, asserts the command is registered and executes it. Turn on coverage with the text, lcov and json-summary reporters into coverage/, exclude out/test from it, and ignore coverage/ and .vscode-test/ in git. Run the suite under xvfb-run, read the report, and add a script that fails when line coverage drops under the figure you measured, rounded down. Add .github/workflows/test.yml with Node 22, npm ci, npm run lint, the test under xvfb-run, the floor check, the lcov file as an artifact and the text summary in the step summary, with contents: read permissions. Run actionlint. Open a pull request with the coverage table and the floor you chose.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request adds .vscode-test.mjs, .github/workflows/test.yml and a coverage floor script, rewrites src/test/suite/extension.test.ts, deletes src/test/runTest.ts and src/test/suite/index.ts, and changes package.json's scripts and dev dependencies and .gitignore. Its description pastes the text coverage table from the sandbox run, names the floor the script enforces and why, and shows actionlint and npm run lint passing.

The test is the part worth reading: it activates the real extension inside the Electron host and executes its command, so the report covers extension.ts rather than a placeholder assertion, and a future command that is registered but never tested shows up as uncovered lines in the step summary.

Before merging, a reviewer runs the Run Extension Tests launch configuration once to confirm the updated .vscode/launch.json still opens the host with the new runner, decides whether the floor should rise as tests are added, and checks that the download of VS Code in CI is acceptable for the repository's runners.

Run it on every extension repository with no coverage report

A team that maintains several extensions sends this body once per repository to POST /o/{org}/sessions, each with its own repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started. Each run reads its own setup: an extension already on @vscode/test-cli gets only the coverage flags, the floor and the workflow, one bundled with esbuild or webpack gets source maps pointed at the bundle so the report names src/ files, and one with an existing workflow gets the steps added there.

An automation with a schedule trigger can run the same prompt nightly in the Project, so an extension whose tests fall behind its commands gets a pull request with the new gap listed, and a night with nothing uncovered ends with a note and no branch.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

Coverage for a VS Code extension is awkward to set up by hand: the tests run inside an Electron extension host that needs a display, the runner has changed twice in a few years, and a report over a placeholder test says nothing. The agent does the migration, runs the host under a virtual display in its own sandbox, and reads the real number before choosing a floor, with every command and its output kept on the run.

The result is a workflow that posts the table on the job page from the free tooling the VS Code team ships, and a test that actually activates the extension. Several extensions can get the same treatment at once, each in its own sandbox, each with its own measured floor.

Frequently asked questions

Why move to @vscode/test-cli instead of adding c8 around the old runner?

The tests execute inside the Electron extension host, a child process the old runTest.js starts, and the CLI is the tool the VS Code team built to collect coverage from that host with one flag. Wrapping the old runner would need the V8 coverage environment threaded through to Electron by hand. The CLI also keeps @vscode/test-electron underneath, so the download and launch behave as before.

Why does the test run under xvfb?

The extension host is a VS Code window, and on a Linux runner there is no display unless one is provided. xvfb-run -a starts a virtual one for the duration of the command. On macOS and Windows runners it is not needed, and the workflow runs Ubuntu only.

Where does the floor come from?

From the report the agent read in its sandbox after the new test ran, rounded down to the nearest whole percent. A floor set higher than the suite reaches fails every build from the first run; a floor set from the measured number fails only when coverage falls.

Does the launch configuration still work?

Run Extension Tests in .vscode/launch.json points --extensionTestsPath at out/test/suite/index, which the migration deletes. The pull request updates that path to the CLI's runner so the debug target keeps working, and says so in the description.

Will CI download VS Code on every run?

Yes, into .vscode-test/, which the agent adds to actions/cache keyed on the engines.vscode range so repeated runs reuse it. The first run downloads the matching build once.

More VS Code extension jobs

The same job on other stacks

Keep reading