What the agent does
The agent clones the repository into its own sandbox and reads the service before writing anything. In the src/payment service of open-telemetry/opentelemetry-demo, package.json has one script, start, which runs node --require @opentelemetry/auto-instrumentations-node/register index.js, and no test script at all. index.js serves the gRPC PaymentService: its charge handler sets demo.payment.amount on the active span and calls charge() in charge.js. That function starts its own charge span from the payment tracer, sets demo.payment.card_type, demo.payment.card_valid, demo.user_context.loyalty_level and demo.payment.charged, and on any thrown error calls recordException, sets the status to ERROR and adds error.type. The same attribute names are declared in telemetry-schema/attributes/payment.yaml, which notes that the collector masks demo.payment.card_number and deletes demo.payment.card_cvv before export.
From that code it lists every case a test should pin: a valid Visa or Mastercard that is charged; a valid card of another type, which throws the "Sorry, we cannot process" error; a number simple-card-validator rejects, which throws "Credit card info is invalid."; an expiry month and year in the past; a request whose baggage carries synthetic_request=true, which must set user_agent.synthetic.type to test and demo.payment.charged to false; baggage with enduser.id, which is copied onto the span; the paymentFailure flag above zero, which fails with loyalty_level gold; and the emitRawPii flag, which adds the raw card number and CVV only when it is on. Each case names the span attributes and status it expects.
The tests use Node's own runner (node --test), so no test framework is added. The span side comes from the packages the lockfile already resolves through auto-instrumentations-node 0.82.0: a TracerProvider with a SimpleSpanProcessor and an InMemorySpanExporter from @opentelemetry/sdk-trace 3.0.0, and the AsyncLocalStorageContextManager from @opentelemetry/context-async-hooks 3.0.0 so that baggage set with context.with reaches charge(). Both become explicit dev dependencies at those versions. Requests follow ChargeRequest in pb/demo.proto: an amount with currencyCode, units and nanos, and a creditCard with number, CVV, expiration year and month.
Feature flags need care. charge.js calls OpenFeature.setProvider(new FlagdProvider()) when it loads, and the sandbox has no flagd, so every evaluation returns its default. For the two flag cases the test sets the InMemoryProvider that @openfeature/server-sdk 1.23.0 already ships, with paymentFailure and emitRawPii values per test, and closes OpenFeature afterwards so a provider retrying its connection does not keep the runner alive. The agent runs npm ci and node --test in src/payment until every test passes, adds a test script, and checks that the Dockerfile still installs with npm ci --omit=dev, so the new dev dependencies never reach the image.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Write unit tests for the spans the payment service emits, in src/payment. Read package.json, index.js, charge.js, logger.js, pb/demo.proto and telemetry-schema/attributes/payment.yaml first, and list every outcome charge() can have: a valid Visa or Mastercard, another card type, an invalid number, an expired card, synthetic_request baggage, enduser.id baggage, the paymentFailure flag above zero, and emitRawPii on and off. For each, assert the charge span's attributes, its status and any recorded exception. Use node --test and an InMemorySpanExporter on a TracerProvider registered before charge.js is required, with an AsyncLocalStorageContextManager so baggage reaches the code. Take the SDK packages at the versions package-lock.json already resolves and add them as dev dependencies only. Swap the flagd provider for OpenFeature's InMemoryProvider in the flag tests and close OpenFeature afterwards. Add a test script, run npm ci and npm test until they pass, and do not change charge.js or index.js. Report anything in the code that looks wrong instead of fixing it. Open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request on its own branch. For a service like src/payment, the diff adds a test file (for example charge.test.js) with one test per outcome above, a test script in package.json, and the two OpenTelemetry SDK packages under devDependencies with the matching package-lock.json change. charge.js and index.js are untouched. The description lists each test with the attributes and status it asserts, and says how the flag tests replace flagd.
The session transcript keeps every command with its output: npm ci, each node --test run including the ones that failed while the tests were being written, and the final passing run. Anything the agent noticed but did not change is written in the description as a finding, for example that demo.payment.amount is set by the gRPC handler in index.js rather than inside charge(), so a test of charge() alone cannot see it.
Before merging, a reviewer checks that the tests read spans from the exporter rather than from mocks of @opentelemetry/api, that the exporter is reset between tests so no test depends on order, and that the emitRawPii tests assert the card number is absent by default. If the maintainers want these tests in CI, the pull request names the workflow step that would run them; it adds no workflow unless input asks for one.
Run it on every night, on a schedule
A repository like this holds a dozen services in as many languages, and each one emits spans someone relies on in a dashboard. An automation in the Project that holds the repository, with a schedule trigger whose cron is 0 2 * * * in your timezone, can run a prompt that picks the next service under src/ with no span tests and does the same work there. Each night is one run on its own branch and one pull request, and a night with nothing left to cover ends with a short answer.
To cover several services at once instead, send one POST /o/{org}/sessions per service from a script, with the service path in input. A key runs at most its max_concurrent sessions at once (4 by default); a start past that answers 202 with status: "queued" and begins when a slot frees, oldest first.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Span tests are easy to skip because they need a tracer provider, a context manager and a feature-flag provider set up just so, and without a registered context manager the baggage cases quietly test nothing, since context.with does not carry baggage into the code. In a Rush sandbox the agent works through that setup with node --test until spans actually arrive in the exporter, on a machine that exists for this run, without installing anything in your checkout or needing the demo's Docker Compose stack.
The proof is the record. Each node --test run, failing and passing, is in the transcript with its output, and the pull request shows exactly which attributes a dashboard can rely on. When someone later renames demo.payment.charged or drops the ERROR status, the test fails in review instead of a panel going quiet in production.
Frequently asked questions
Does it need a collector or a running backend to test spans?
No. The tests export spans to an InMemorySpanExporter inside the test process and read them back with getFinishedSpans(). No OTLP endpoint, collector or Jaeger is started, so the tests run anywhere npm test runs.
Will it add Jest or Mocha to the service?
Not unless you ask. The payment service has no test framework, and Node's built-in node --test runner needs none, so the only new dev dependencies are the OpenTelemetry SDK packages the lockfile already resolves. If your service already uses a framework, name it in input and the agent writes the tests for it.
How does it test code behind feature flags?
It replaces the flagd provider with OpenFeature's InMemoryProvider inside the test and sets the flag values each case needs. The real FlagdProvider stays in charge.js; the tests only change which provider answers while they run.
Can it do the same for a Go or Python service?
Yes. Each OpenTelemetry SDK has an in-memory exporter, such as tracetest.NewInMemoryExporter in Go and InMemorySpanExporter in Python. Name the service directory in input and the agent uses that language's test runner and exporter.