What the agent does
The agent clones the repository into its own cloud sandbox and reads how the extension is built and activated before it adds a reporter. In an extension like aaron-bond/better-comments, package.json is version 3.0.2 with engines.vscode at ^1.65.0, activationEvents of onStartupFinished, a Node entry at out/extension from tsc and a web entry at out/web/extension.js from build/web-extension.webpack.config.js, which targets webworker with fallbacks for path, util and assert. The only contribution is configuration: better-comments.multilineComments, highlightPlainText and the tags array. src/extension.ts is 82 lines that wire onDidChangeActiveTextEditor, onDidChangeTextDocument and extensions.onDidChange to a 100 ms debounced updateDecorations, src/parser.ts finds single-line, block and JSDoc comments per language and sets supportedLanguage, and src/configuration.ts reads comment delimiters from every installed extension's contributes.languages. There is no telemetry, no .github/workflows/, and the test runner in src/test/index.ts is the old vscode/lib/testrunner.
It adds @vscode/extension-telemetry, whose current release asks for engines.vscode of ^1.75.0, so it raises that field and @types/vscode together and runs tsc to confirm the extension's own API use still compiles against the newer typings. A new src/telemetry.ts builds one TelemetryReporter from a connection string constant, which the package's README says is not a secret and can live in the source, and activate pushes the reporter onto context.subscriptions so disposal flushes the queue. The reporter itself checks the user's telemetry level before every send, so the extension does not read telemetry.telemetryLevel by hand.
The events are few and named for what the extension does: activated with a measurement of how many tags are configured and properties for the two boolean settings, language.unsupported with the languageId when parser.supportedLanguage is false after SetRegex, and sendTelemetryErrorEvent from a catch around the regex setup, which today has none. No document text, file name or tag content is ever a property; the pull request says so and lists each event with its fields in a new Telemetry section of README.md, next to the settings it already documents.
The package ships a browser entry, so webpack picks it for the web build without a new fallback; the agent runs npm run compile, npm run lint with the repository's tslint.json, and npm run package-web to show the webworker bundle still builds, then npx vsce package to produce a VSIX and confirm .vscodeignore still excludes src/ and the test files from it. It does not rewrite the Mocha test runner, and says in the pull request that extension host tests are a separate change.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add telemetry to this VS Code extension with @vscode/extension-telemetry. Read package.json, src/extension.ts, src/parser.ts, src/configuration.ts and the webpack config under build/ first. Raise the VS Code engine and typings only as far as the package requires, create one reporter in its own module and dispose it through context.subscriptions, and send a small set of named events: one on activation with the counts of configured settings, one when a language is unsupported, and an error event around the regex setup. Never include document text, paths or tag contents. Document every event and its fields in the README, run the compile, lint, web bundle and vsce package steps, and open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request on its own branch that changes package.json, package-lock.json, src/extension.ts, src/parser.ts and README.md, and adds src/telemetry.ts. The description lists the three events with their properties and measurements, explains the engine bump from ^1.65.0 to ^1.75.0 and what that means for users on older VS Code, and pastes the tsc, tslint, webpack and vsce package output from the sandbox.
The run's record keeps each command with its output, including the VSIX contents vsce prints, so a reviewer can check that no test or source file slipped into the package.
Before merging, a reviewer creates the Application Insights resource and pastes its connection string into src/telemetry.ts, decides whether the engine bump warrants a minor or major version in CHANGELOG.md, and installs the VSIX locally with telemetry set to all and then off to see events appear and stop.
Run it on every new extension repository
Start one run per extension from your own server: POST /o/{org}/sessions with this prompt and that repository in resources, one call each. GET /o/{org}/github lists every repository the Rush GitHub App installation reaches, so a script over an organisation's extensions can start a run for each that has no @vscode/extension-telemetry in its package.json, in its own sandbox and on its own branch.
Give the key a webhook_url and each run posts run.completed with its session_url when the turn finishes, so the pull request links collect on your side without polling. There is no need to wait for one extension to finish before starting the next: the key runs max_concurrent at once, 4 by default and up to 100, and any further request answers 202 with status: "queued" and starts, oldest first, when a slot frees.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
Telemetry in an extension is a small amount of code with two easy mistakes: an engine requirement raised further than needed, which drops users on older VS Code, and an event property that carries a file path or a line of someone's code. The agent reads what the package requires and what the extension handles, keeps the events to counts and language ids, and builds both the Node and web bundles in its own sandbox before the pull request exists.
Every command and its output stays in the run's record, so the reviewer sees the real tsc and webpack results rather than a sentence about them. Several extensions can be instrumented at once in separate sandboxes, each ending in its own pull request.
Frequently asked questions
Does this respect the user's telemetry setting?
Yes. @vscode/extension-telemetry checks VS Code's telemetry level before sending and drops events when it is off, and the reporter is disposed with the extension so nothing is sent after deactivation. The extension never calls the package's sendDangerous methods, which bypass that check.
Why does the VS Code engine version go up?
The telemetry package's current release declares engines.vscode of ^1.75.0, and vsce refuses to package an extension whose own engine is lower than a dependency requires. The pull request raises engines.vscode and @types/vscode to that version and no further, and notes it as a change for users on older builds.
Will the web extension build still work?
Yes. The package publishes a browser entry that webpack resolves in the existing webworker target, so build/web-extension.webpack.config.js needs no new fallback. The agent runs npm run package-web and includes its output.
What is never sent?
Document contents, file names, workspace paths and the user's configured tag strings. Properties are language ids and booleans, and measurements are counts. The Telemetry section the pull request adds to README.md lists every event, so a user can check the list against the code.