What the agent does
The agent reads the server before deciding what a test can reach. In a project like socketio/chat-example, package.json is an ES module with socket.io 4.7, express 4.18, sqlite and sqlite3, and @socket.io/cluster-adapter; its only script is start, there are no dev dependencies, no test runner and no CI. index.js does everything in one file: the cluster primary forks one worker per CPU with PORT set to 3000 plus the index and calls setupPrimary(), and each worker opens chat.db, creates a messages table whose client_offset column is UNIQUE, serves index.html, and builds a Server with connectionStateRecovery and the cluster adapter. The chat message handler inserts the message, treats a SQLite constraint error (errno 19) as a duplicate and acknowledges without re-emitting, otherwise broadcasts the message with result.lastID as the server offset; a socket that was not recovered gets every row with an id above socket.handshake.auth.serverOffset.
None of that is testable while it only exists inside cluster.isPrimary's else branch, so the first change is a small refactor. The agent moves the worker body into server.js as createServer({ dbFile, adapter }) that returns the HTTP server, the io instance and the database handle, and reduces index.js to the cluster bootstrap that calls it with chat.db and the cluster adapter. Behaviour is unchanged: the same table, the same handlers, the same PORT logic.
The tests use the runner already in Node, node:test with node:assert, which the Socket.IO testing guide shows alongside the other frameworks, so the only new dependency is socket.io-client as a dev dependency. test/chat.test.js starts the server on port 0 with dbFile set to :memory: and no adapter, reads the port from httpServer.address(), connects two clients, and checks four things: a message sent with an acknowledgement reaches the other client with a server offset; sending the same clientOffset twice stores one row, acknowledges both, and emits once; a client that connects with auth.serverOffset set receives only rows above it; and a client that connects with no offset receives the full history in id order. before and after hooks open and close the server and disconnect clients so the process exits.
The agent adds a test script, node --test, runs it in the sandbox, and runs npm start once to confirm the bootstrap still forks workers and serves the page after the refactor. chat.db stays in .gitignore. The pull request names each test and the behaviour in index.js it pins, and flags that the cluster adapter itself is not exercised in tests because it needs a primary process.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Write integration tests for this Socket.IO chat server. Read package.json, index.js, index.html and .gitignore first. Move the worker body of index.js into server.js as a createServer function that takes a database file path and an optional adapter and returns the HTTP server, the io instance and the database, and leave index.js as the cluster bootstrap that calls it with chat.db and the cluster adapter, with behaviour unchanged. Add socket.io-client as a dev dependency and a test script that runs node --test. In test/chat.test.js, start the server on port 0 with an in-memory SQLite database and no adapter, connect real clients, and test that a chat message with an acknowledgement is broadcast with a server offset, that a repeated clientOffset is stored once and acknowledged both times, that a client connecting with auth.serverOffset receives only later rows, and that a client with no offset receives the full history in order. Close everything in after hooks. Run the tests and npm start in the sandbox, and open a pull request that lists each test and the index.js behaviour it covers.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that adds server.js and test/chat.test.js, shrinks index.js to the cluster bootstrap, and adds socket.io-client and a test script to package.json with the matching package-lock.json. The description lists the four tests, states that each runs against a real socket over a real port with an in-memory database, and points out that the cluster adapter is wired in index.js only.
The run's record keeps the node --test output and the npm start log showing the workers coming up after the refactor, so a reviewer can see both the tests and the unchanged start path without checking the branch out.
Before merging, a reviewer reads server.js against the old index.js to confirm the handlers moved verbatim, decides whether test/ should also cover connectionStateRecovery with a disconnect and reconnect, and whether to add a workflow that runs npm test on pull requests, since the repository has none.
Run it on every pull request touching the socket handlers
Once the tests exist, an automation on the Project with a github trigger on pull_request.opened and pull_request.synchronize and the prompt "if this pull request changes server.js, add or update a test in test/chat.test.js for the changed event and run node --test" keeps them current. The trigger puts the pull request after the prompt; only pushes by an owner, member or collaborator fire it, never a fork's.
For a batch of socket services, send one POST /o/{org}/sessions per repository from your own server with that repository in resources and a request_id that is a UUID v5 of its full name, so a retried call returns the run already started rather than a second refactor.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Socket tests are flaky on a laptop for reasons that have nothing to do with the code: a port already bound, a chat.db left over from a manual run, a client that never disconnected and keeps the process alive. The agent runs on a fresh sandbox with port 0 and an in-memory database, and the tests have to pass there, cleanly, before the pull request opens.
The sandbox keeps the test output and the npm start log, so the pull request can show that both the tests and the cluster bootstrap work after the refactor rather than asking a reviewer to trust that moving the handlers changed nothing.
Frequently asked questions
Why change index.js at all?
Because the server is built inside the else branch of cluster.isPrimary, importing index.js from a test would fork workers instead of returning a server. Moving that body into an exported function is the smallest change that makes it reachable, and index.js keeps calling it exactly as before.
Why node:test instead of Jest or Mocha?
The repository has no test framework, and node:test ships with Node, so the only new dependency is socket.io-client. If you already use Jest or Vitest elsewhere, say so in the prompt and the agent writes the same tests for that runner; the Socket.IO testing guide covers all of them.
Do the tests cover the cluster adapter?
No. @socket.io/cluster-adapter needs a primary process to relay between workers, so the tests run a single server with no adapter and the pull request says so. The adapter stays wired in index.js, and npm start is run once to confirm the workers still come up.
What about connection state recovery?
The tests cover the offset replay that index.js performs for sockets that were not recovered, because that is application code. Exercising recovery itself means a disconnect and reconnect within the recovery window, which the pull request lists as a follow-up rather than a claim.