# Add Code Coverage to CI for a VS Code Extension

Yes. An agent moves your VS Code extension's tests to the VS Code test CLI, turns on its coverage, writes a real test, and publishes the report from CI in a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-coverage-to-ci/vscode-extension/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads how the extension is tested before it touches the config. In an extension laid out like `helloworld-test-sample` in microsoft/vscode-extension-samples, `package.json` declares `engines.vscode` `^1.100.0`, one command `extension.helloWorld`, `main` at `./out/extension.js`, and scripts where `compile` is `tsc -p ./`, `pretest` compiles and `test` runs `node ./out/test/runTest.js`. `src/test/runTest.ts` calls `runTests` from `@vscode/test-electron` `^2.3.9` with the extension root and `./suite/index`, and `src/test/suite/index.ts` builds a Mocha `tdd` runner over `**/**.test.js`. The only test, `src/test/suite/extension.test.ts`, asserts that `[1, 2, 3].indexOf(5)` is -1, and its import of the extension is commented out. `src/extension.ts` registers the command and shows an information message. So the suite runs, passes, and executes none of the extension; a coverage number for it today would be zero, and nothing in the repository measures it. `tsconfig.json` emits CommonJS to `out/` with source maps, which coverage needs.

The agent moves the runner to `@vscode/test-cli`, which is the VS Code team's current entry point and the one with built-in coverage. It adds `.vscode-test.mjs` with `defineConfig({ files: 'out/test/**/*.test.js', mocha: { ui: 'tdd' } })`, changes `test` to `vscode-test`, keeps `pretest` as the compile step, and deletes `runTest.ts` and `suite/index.ts`, whose jobs the CLI now does. `@vscode/test-electron` stays, since the CLI drives it. Coverage is then a flag: `vscode-test --coverage --coverage-reporter text --coverage-reporter lcov --coverage-reporter json-summary --coverage-output coverage`, and the `coverage` block in the config excludes `out/test/**`. `coverage/` and `.vscode-test/` go into `.gitignore`.

A number is only worth publishing when the suite reaches the code, so the agent replaces the placeholder test with one that does. `extension.test.ts` gets the extension by id from `package.json`'s `publisher` and `name`, `vscode-samples.helloworld-sample`, awaits `activate()`, asserts `extension.helloWorld` is in `vscode.commands.getCommands(true)`, and runs `executeCommand('extension.helloWorld')`, which resolves once the information message is shown. That exercises `activate` and the command callback, every line in `extension.ts`. The agent runs the suite under `xvfb-run -a npm test` in its sandbox, because the Electron host needs a display, reads the text report, and sets a floor from what it measured in a small script that reads `coverage/coverage-summary.json` and exits non-zero under the line percentage it saw, rounded down.

CI is the repository's own: the sample inherits nothing from the monorepo's `.github/workflows/ci.yml`, which only runs a validation script, so the agent adds `.github/workflows/test.yml` in the extension with `actions/setup-node` on Node 22, `npm ci`, `xvfb-run -a npm test` with the coverage flags, the floor script, `coverage/lcov.info` uploaded with `actions/upload-artifact`, and the text summary appended to `$GITHUB_STEP_SUMMARY` so the number shows on the job page without a paid service. `permissions` is `contents: read`. `npm run lint` with the repository's `typescript-eslint` and `@stylistic` config runs before the test step, and the agent runs `actionlint` on the workflow before opening the pull request.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add code coverage to CI for this VS Code extension. Read package.json, tsconfig.json, eslint.config.mjs, .vscode/launch.json, src/extension.ts and everything under src/test first. Move the test runner to @vscode/test-cli: add .vscode-test.mjs with the compiled test glob and the tdd ui, set the test script to vscode-test, delete runTest.ts and suite/index.ts, keep @vscode/test-electron. Replace the placeholder test with one that gets the extension by its publisher.name id, activates it, asserts the command is registered and executes it. Turn on coverage with the text, lcov and json-summary reporters into coverage/, exclude out/test from it, and ignore coverage/ and .vscode-test/ in git. Run the suite under xvfb-run, read the report, and add a script that fails when line coverage drops under the figure you measured, rounded down. Add .github/workflows/test.yml with Node 22, npm ci, npm run lint, the test under xvfb-run, the floor check, the lcov file as an artifact and the text summary in the step summary, with contents: read permissions. Run actionlint. Open a pull request with the coverage table and the floor you chose.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

The pull request adds `.vscode-test.mjs`, `.github/workflows/test.yml` and a coverage floor script, rewrites `src/test/suite/extension.test.ts`, deletes `src/test/runTest.ts` and `src/test/suite/index.ts`, and changes `package.json`'s scripts and dev dependencies and `.gitignore`. Its description pastes the text coverage table from the sandbox run, names the floor the script enforces and why, and shows `actionlint` and `npm run lint` passing.

The test is the part worth reading: it activates the real extension inside the Electron host and executes its command, so the report covers `extension.ts` rather than a placeholder assertion, and a future command that is registered but never tested shows up as uncovered lines in the step summary.

Before merging, a reviewer runs the `Run Extension Tests` launch configuration once to confirm the updated `.vscode/launch.json` still opens the host with the new runner, decides whether the floor should rise as tests are added, and checks that the download of VS Code in CI is acceptable for the repository's runners.

## Run it on every extension repository with no coverage report

A team that maintains several extensions sends this body once per repository to `POST /o/{org}/sessions`, each with its own repository in `resources` and a `request_id` that is a UUID v5 of the repository's full name, so a retried call returns the run already started. Each run reads its own setup: an extension already on `@vscode/test-cli` gets only the coverage flags, the floor and the workflow, one bundled with esbuild or webpack gets source maps pointed at the bundle so the report names `src/` files, and one with an existing workflow gets the steps added there.

An automation with a `schedule` trigger can run the same prompt nightly in the Project, so an extension whose tests fall behind its commands gets a pull request with the new gap listed, and a night with nothing uncovered ends with a note and no branch.

## Why run it on Rush?

Coverage for a VS Code extension is awkward to set up by hand: the tests run inside an Electron extension host that needs a display, the runner has changed twice in a few years, and a report over a placeholder test says nothing. The agent does the migration, runs the host under a virtual display in its own sandbox, and reads the real number before choosing a floor, with every command and its output kept on the run.

The result is a workflow that posts the table on the job page from the free tooling the VS Code team ships, and a test that actually activates the extension. Several extensions can get the same treatment at once, each in its own sandbox, each with its own measured floor.

## Frequently asked questions

### Why move to @vscode/test-cli instead of adding c8 around the old runner?

The tests execute inside the Electron extension host, a child process the old `runTest.js` starts, and the CLI is the tool the VS Code team built to collect coverage from that host with one flag. Wrapping the old runner would need the V8 coverage environment threaded through to Electron by hand. The CLI also keeps `@vscode/test-electron` underneath, so the download and launch behave as before.

### Why does the test run under xvfb?

The extension host is a VS Code window, and on a Linux runner there is no display unless one is provided. `xvfb-run -a` starts a virtual one for the duration of the command. On macOS and Windows runners it is not needed, and the workflow runs Ubuntu only.

### Where does the floor come from?

From the report the agent read in its sandbox after the new test ran, rounded down to the nearest whole percent. A floor set higher than the suite reaches fails every build from the first run; a floor set from the measured number fails only when coverage falls.

### Does the launch configuration still work?

`Run Extension Tests` in `.vscode/launch.json` points `--extensionTestsPath` at `out/test/suite/index`, which the migration deletes. The pull request updates that path to the CLI's runner so the debug target keeps working, and says so in the description.

### Will CI download VS Code on every run?

Yes, into `.vscode-test/`, which the agent adds to `actions/cache` keyed on the `engines.vscode` range so repeated runs reuse it. The first run downloads the matching build once.

## More VS Code extension jobs

- [Add Dependabot to a VS Code Extension](https://getrush.ai/run/add-dependabot/vscode-extension/)
- [Bundle a VS Code Extension with esbuild](https://getrush.ai/run/add-esbuild-bundling/vscode-extension/)
- [Add User Settings to a VS Code Extension](https://getrush.ai/run/add-extension-settings/vscode-extension/)

## The same job on other stacks

- [Add Test Coverage Reporting to Wagtail CI](https://getrush.ai/run/add-coverage-to-ci/wagtail/)
- [Add Test Coverage to CI for an aiohttp Project](https://getrush.ai/run/add-coverage-to-ci/aiohttp/)
- [Add Test Coverage Reporting to Angular CI](https://getrush.ai/run/add-coverage-to-ci/angular/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
