What the agent does
The agent reads how the app is tested today. In a Bolt app laid out like slack-samples/bolt-js-starter-template, package.json is an ES module with @slack/bolt 5 and dotenv as its only dependencies and Biome as its only dev dependency; npm test runs node --test tests/**/*.spec.js with no test framework beyond node:test, and npm run lint runs biome check. The code is six listener folders under listeners/, actions, commands, events, messages, shortcuts and views, each with a callback file and an index.js that registers it on the app, plus listeners/index.js that calls the six register functions, and two entry points, app.js for Socket Mode and app-oauth.js for an HTTP install flow. tests/ mirrors the callbacks one spec file each, built on mock.fn() fakes for ack, client, respond and logger, with a happy path and an error-logging case per callback.
.github/workflows/test.yml runs npm install, npm run lint and npm test on Node 20, 22 and 24 for every push to main and every pull request, with a five-minute timeout; dependencies.yml approves and auto-merges Dependabot's patch and minor updates, and dependabot.yml schedules npm and github-actions monthly. No step measures coverage, and the six index.js registration files and listeners/index.js have no tests at all, so a listener that is written and tested but never registered passes CI.
The agent adds c8 as a dev dependency and a test:coverage script, c8 npm test, which wraps the existing command through V8's built-in coverage rather than instrumenting files. A .c8rc.json at the root sets all: true so untested files count, include to listeners/**, exclude to the two entry points, since app.js and app-oauth.js start a real app and belong to a smoke test rather than a unit suite, reporters text for the job log and lcov for an artifact, and check-coverage with a lines floor chosen from the first report so the job passes on the current tree and fails on a drop. The coverage/ directory joins .gitignore beside node_modules.
To make the registration files count, the agent writes tests/listeners/register.spec.js in the same style as the existing specs: a fake app whose action, command, event, message, shortcut and view methods are mock.fn(), a call to registerListeners, and assertions that each method was called once with the identifier the manifest.json declares, /sample-command, sample_shortcut_id, sample_action_id, sample_view_id and app_home_opened. CI gets one more step in test.yml after npm test, npm run test:coverage, and an actions/upload-artifact step for coverage/lcov.info from the Node 24 job only. The agent runs npm run lint, because Biome checks the new spec and config too, and npm run test:coverage on the sandbox's Node before opening the pull request, as AGENTS.md asks.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add test coverage to CI for this Slack Bolt app. Read package.json, biome.json, manifest.json, AGENTS.md, app.js, every file under listeners/ and tests/, and .github/workflows/test.yml first. Add c8 as a dev dependency and a test:coverage script that runs c8 npm test. Write .c8rc.json with all: true, include listeners/**, exclude app.js and app-oauth.js, text and lcov reporters, and check-coverage with a lines floor set just under what the current suite reaches. Add coverage/ to .gitignore. Write tests/listeners/register.spec.js in the style of the existing specs: a fake app with mock.fn() methods, a call to registerListeners, and one assertion per listener that it was registered under the identifier manifest.json declares. Add an npm run test:coverage step to test.yml after npm test and upload coverage/lcov.info as an artifact from the newest Node in the matrix. Run npm run lint and npm run test:coverage until both pass, then open a pull request that includes the coverage table and explains the floor.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request adds c8 to devDependencies and the test:coverage script in package.json, updates package-lock.json, adds .c8rc.json, tests/listeners/register.spec.js and the coverage/ line in .gitignore, and extends .github/workflows/test.yml with the coverage and artifact steps. Its description carries the text table c8 prints, file by file, the floor it chose and why, and the Biome result.
The table is the first time the app has had a per-file number: the callbacks the existing specs cover, the registration files the new spec covers, and any branch in a callback that no test reaches, such as a catch block in a listener whose error path has no case yet. Those lines are listed in the description as the next tests worth writing rather than hidden behind a total.
Before merging, a reviewer checks that the floor is low enough to pass on the three Node versions in the matrix and high enough to catch an untested listener, decides whether the artifact should be uploaded from every matrix job or one, and whether to publish the lcov file to a coverage service later, which would be one more step and a token.
Run it on every pull request
A team with several Slack apps on the same template sends this body once per repository from its own server, each a POST /o/{org}/sessions with that repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started. Each run reads its own layout: an app that moved its listeners into TypeScript gets the same c8 wrapper around its compiled test command, and one that already uses Jest gets Jest's own collectCoverage instead.
Once the floor is in place, an automation on the Project with a github trigger on pull_request.opened and pull_request.synchronize and the prompt "if the coverage step failed, read the table, write the missing cases for the listener that dropped below the floor in the style of tests/, and push them to the pull request branch" starts one run per push. The trigger adds the pull request after the prompt, only work by an owner, member or collaborator fires it, and the run pushes to the contributor's branch only when the check failed.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Coverage on a small app is less about the number than about what it reveals: here, that every callback had a test and no registration did, which is the one bug a Bolt app can ship without a failing test. The agent reads the suite, finds that gap, writes the test in the project's own mock.fn() style, and sets the floor from a real run rather than a round number.
The sandbox runs the lint and the test commands for real on a Node version from the matrix, and their output stays with the run, so the pull request shows the table the reviewer will see in CI. A team with many Slack apps runs the same request across all of them at once, each in its own sandbox, and each gets a floor fitted to its own suite.
Frequently asked questions
Why c8 instead of Node's built-in --experimental-test-coverage?
The matrix runs Node 20, 22 and 24, and the built-in reporter and its threshold flags differ across those versions, while c8 reads the same V8 coverage data and behaves the same on all three. The description notes that a repository dropping Node 20 can switch to the built-in flags later with the same floor.
Why exclude app.js and app-oauth.js?
Both files construct an App and call app.start() at import, so loading them in a unit test opens a Socket Mode connection or an HTTP server. They are the place for a smoke test with a running Slack app, not for line coverage, and leaving them in would only lower the total without saying anything useful.
Does the new spec test Bolt itself?
No. It passes a fake app with mock.fn() methods to registerListeners and asserts which identifier each listener was registered under, so it catches a listener that is written but never wired, or wired under a name manifest.json does not declare. Bolt's own routing stays Bolt's responsibility.
Will the floor make every small pull request fail?
The floor is set just under the current total, so a change that adds a listener with a test keeps passing, and one that adds a listener without a test fails with the file named in the table. The text reporter prints that table in the job log, so the failing line is visible without downloading the artifact.