All jobs

Is the New Store Ready to Launch? An Online Store's Daily Sort of Beta Testers' Jam Recordings

An online store relaunching its site shares a Jam Recording Link with beta testers, and its agent sorts each day's recordings into launch blockers and the rest.

Run this on RushOpens the console with this prompt ready to send. Connect Jam there first.

What the agent does

This is for an online store moving to a new storefront: a new theme, a new checkout, new product pages. Before switching the domain over, it invites its own staff and a group of regular customers to shop on the preview site for two weeks and report anything that goes wrong. Instead of a feedback form, it gives them a Jam Recording Link: a tester clicks it, records their screen while the problem happens, and Jam keeps the video with the browser's console, network requests and clicks. By the third day there are dozens of recordings and nobody can say whether the store is ready. The agent answers that every evening. The store connects Jam once with POST /me/connections {"type": "mcp", "name": "jam", "url": "https://mcp.jam.dev/mcp", "auth": {"mode": "oauth"}} and signs in on Jam's page; Jam's docs say the server's access mirrors that person's own Jam permissions.

The run calls getRecordingLink with the link's public id and listRecordingLinkJams to list the Jams recorded through it, following the cursor until it reaches recordings older than the last day. For each new Jam it calls getDetails for what happened and the timeline, getNetworkRequests filtered by statusCode for failed calls, getConsoleLogs with logLevel set to errors, and getUserEvents, which describes the tester's clicks, inputs and page changes in plain words. Where the tester talked while recording, getVideoTranscript gives what they said, which usually names what they expected to happen.

It sorts by what a launch cannot ship with. A blocker is anything that stops a purchase: an add-to-cart that fails, a payment step that errors or hangs, a discount code refused when it should apply, a shipping rate missing for a region the store ships to, an order confirmation that never appears. Next come problems that cost sales without stopping them (a broken product image, a size selector that resets) and then cosmetic ones. It groups recordings that show the same failure, matching on the endpoint and error in the network log and the page in the user events, and counts how many testers hit each, with their browser and device from the recording.

It writes launch-readiness.md: the open blockers first, each with the step to reproduce, the failing request and its status, and links to every Jam that shows it; then the other groups; then recordings it could not place, such as an Instant Replay Jam with no video. It also returns the count of blockers as JSON for the team's channel. It writes nothing back to Jam. Jam's server can comment, move and delete Jams and folders, and its docs warn that deleteJam and deleteFolder cannot be undone; the prompt forbids all of them.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Use the Jam connection. Read the Recording Link with public id rl_storebeta (getRecordingLink) and list its Jams with listRecordingLinkJams, following the cursor, keeping Jams created in the last 24 hours. For each: getDetails, getNetworkRequests with statusCode filters for 4xx and 5xx, getConsoleLogs at error level, getUserEvents, and getVideoTranscript when the Jam has video. Classify each as blocker (stops a purchase: add to cart, checkout, payment, discount code, shipping rate, order confirmation), costs_sales (product page, search, images, variant pickers) or cosmetic. Group Jams showing the same failing endpoint and error on the same page, and count testers, browsers and devices per group. Write launch-readiness.md: blockers first with steps to reproduce, the failing request and status, and links to each Jam; then the other groups; then Jams you could not classify and why. Return only JSON: {blockers: number, groups: [{title, class, jams, testers, failing_request}]}. Do not comment on, edit, move or delete any Jam or folder.",
  "connections": [
    "jam"
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The session returns launch-readiness.md as an artifact and the counts in result. The store's lead and its developers read it at the end of each beta day: the blockers are tomorrow's work, and the day the count reaches zero is the day they can plan the switch.

What the team checks: that a group really is one failure and not two that share an endpoint, by opening two of its Jams; recordings classed cosmetic where the tester sounded stuck in the transcript; and anything the agent could not classify. A failure that happens only in the preview, such as a payment provider in test mode refusing a real card, is the team's to recognise. The transcript lists every Jam tool the agent called, so a recording missing from the report can be traced.

Run it on every evening of each beta day

In the console, a Project automation with a schedule trigger, cron 0 19 * * * in the store's timezone, runs a saved agent whose connections name jam with this prompt every evening of the beta, with notify_each_run set to true so each day's result comes on its own. When the beta ends, the automation is turned off.

A store that wants the count on its own launch dashboard can start the same request from its backend instead and read the answer from the key's webhook, matching the execution_id it saved at start.

The use case this job follows, with its automation and trigger: Produce a report or file, no repo.

Why run it on Rush?

A form asks testers to describe what went wrong, and most describe it badly. A Jam recording already holds the failing request and the console error; what takes time is opening forty of them and seeing that eleven are the same broken discount call. The agent does that reading and grouping, and Jam's sign-in stays in Rush's connection rather than a personal token pasted into a script.

Each run has its own computer and a record of every tool call and its cost. When a developer asks why a blocker was counted twice, the evening's run shows which Jams it grouped and what in each network log it matched on.

Frequently asked questions

Do beta testers need a Jam account?

That is set in Jam, not in Rush: the store creates the Recording Link there and chooses its settings, such as a folder, a domain or an expiry, before sharing it. The agent only reads the Jams that arrive through the link, with the store's own sign-in.

Will the agent reply to testers or close their reports?

No. It writes a report for the store's team and changes nothing in Jam. Thanking a tester or telling them a fix is live is a person's job.

Can it tell us whether to launch?

It counts open blockers and shows the evidence for each. Whether to launch with a known issue, such as one browser that few customers use, is the store's decision.

Why do some recordings have no transcript or frames?

Jam's docs say the video tools return nothing for Instant Replay Jams, and frames need a video that has finished processing, which takes a few minutes after upload. The agent lists those recordings and works from their logs and clicks instead.

More Jam jobs

More jobs like this: Produce a report or file, no repo

Keep reading