What the agent does
Extension tests run inside a real VS Code: @vscode/test-electron downloads a VS Code build, @vscode/test-cli starts it with your extension loaded, and Mocha tests run in its extension host with the full vscode API. The agent starts by reading what the extension declares. In microsoft/vscode-extension-samples' tree-view-sample, package.json names the extension custom-view-samples from publisher vscode-samples, requires vscode ^1.100.0, compiles with tsc -p ./ into out/, and has no test script and no test dependencies. It contributes a package-explorer activity bar container holding the nodeDependencies view, five explorer views, and commands such as nodeDependencies.refreshEntry, addEntry, editEntry and deleteEntry.
Two details in that manifest decide how the tests are written. activationEvents lists only onLanguage:json and onLanguage:jsonc, so the extension does not activate when VS Code starts; a test must open a JSON document or call extension.activate() first, or every command lookup fails. And DepNodeProvider in src/nodeDependencies.ts builds its tree from the first workspace folder's package.json and that folder's node_modules, showing "No dependency in empty workspace" or "Workspace has no package.json" otherwise, so the tests need a fixture workspace with known contents.
The agent copies the setup from the repository's own helloworld-test-cli-sample, which uses @vscode/test-cli ^0.0.8, @vscode/test-electron ^2.3.9, Mocha and a .vscode-test.mjs with files: 'out/test/**/*.test.js'. It adds those to devDependencies, a test script that runs vscode-test, and a .vscode-test.mjs whose workspaceFolder points at src/test/fixtures/workspace: a package.json with two dependencies and matching stub folders under node_modules. Tests go in src/test/*.test.ts, which the existing tsconfig.json (rootDir: src, outDir: out) compiles to out/test.
The tests activate the extension through vscode.extensions.getExtension('vscode-samples.custom-view-samples'), check every contributed command is registered with vscode.commands.getCommands(true), open the view with the workbench.view.extension.package-explorer command, and call DepNodeProvider.getChildren() against the fixture to assert the tree lists the two dependencies with the versions in its package.json. They run nodeDependencies.refreshEntry and check the provider fires onDidChangeTreeData. The agent runs npm run compile and xvfb-run -a npm test in its sandbox until every test passes.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "codex",
"input": "Pick the tree-view-sample extension. Add integration tests with @vscode/test-cli and @vscode/test-electron, set up the way helloworld-test-cli-sample in this repository does (devDependencies, a test script, .vscode-test.mjs). Give the tests a fixture workspace with a package.json and node_modules entries, activate the extension explicitly (it only activates on JSON files), check every contributed command is registered, open the Package Explorer view, assert the Node Dependencies tree lists the fixture's dependencies, and run the refresh command. Compile, run them under xvfb-run until they pass, and open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request adds to tree-view-sample/: .vscode-test.mjs, src/test/extension.test.ts and src/test/nodeDependencies.test.ts, the fixture under src/test/fixtures/workspace/, the test dependencies and @types/mocha in package.json with the updated package-lock.json, a test script, and .vscode-test/ in .gitignore so the downloaded VS Code is not committed. extension.ts and the providers do not change.
The description lists each test and what it proves, quotes the Mocha summary from the xvfb-run -a npm test run in the sandbox, and names the VS Code version the tests ran against. It also notes what is not covered: the FTP explorer needs a live FTP server, and drag and drop in testViewDragAndDrop has no programmatic API to drive. The session's record keeps every compile and test run.
Before merging, a reviewer checks the fixture is small and committed, that the tests do not depend on the order Mocha runs files in, and whether to add a CI job, which the pull request includes only if the repository already has a workflow for this sample.
Run it on every command contributed
Each new command or view in package.json is a new thing that can silently fail to register. A GitHub trigger on pull_request.opened can start this run for pull requests that change an extension's package.json or src/extension.ts, asking for a test that the new contribution registers and works, so it is covered when it lands.
In a repository with dozens of extensions, send one request per extension directory that has no tests yet. Each run boots its own VS Code under its own virtual display in its own sandbox, so they proceed side by side and each opens a separate pull request.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
The integration test setup is the part extension authors give up on: a downloaded VS Code build, a headless display on Linux, a fixture workspace, and an activation event that quietly prevents the extension from loading in the test window. In a Rush sandbox the agent works through those failures on a disposable Linux machine with xvfb-run, and no extra VS Code window ever opens on your desktop.
What comes back is checkable: the pull request's tests, the Mocha output from the sandbox in the session's record, and the exact VS Code version they passed against. Because each extension gets its own sandbox, a samples repository can get tests across many extensions at once, each reviewed on its own.
Frequently asked questions
Why does the test have to activate the extension itself?
VS Code only loads an extension when one of its activationEvents fires. This sample activates on opening a JSON or JSONC file, so a fresh test window would never load it. The tests call extension.activate() first, which is what VS Code does when the event fires.
Can the tests read what the tree view shows?
The vscode API does not expose the rendered tree, so the tests call the TreeDataProvider the view uses, here DepNodeProvider.getChildren(), with the same fixture workspace. That checks the data the view displays, and a separate test opens the view to prove it is registered.
Which VS Code version do the tests run against?
By default @vscode/test-cli downloads the current stable build. Set version in .vscode-test.mjs to pin one, or add a second configuration with the oldest version your engines field allows, here 1.100.
Does it need a display?
VS Code is an Electron app, so on Linux it needs one, which xvfb-run provides. On macOS and Windows the same npm test runs without it.