All jobs

Add Replay Tests to Temporal Workflows

Yes. An agent records histories of your Temporal workflow's success and failure paths, replays them in a test with Worker.runReplayHistory, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the workflow is tested today. In a project laid out like temporalio/money-transfer-project-template-ts, src/workflows.ts exports moneyTransfer, which proxies withdraw, deposit and refund with a retry policy of up to 500 attempts, InvalidAccountError and InsufficientFundsError as non-retryable, and a one-minute startToCloseTimeout, and refunds the source account when the deposit fails. src/mocha/workflows.test.ts starts a TestWorkflowEnvironment.createLocal(), creates a Worker with the real activities or with withdraw and deposit stubbed, and asserts the result string or a WorkflowFailedError across four cases. package.json pins the SDK packages at ~1.14.1 with @temporalio/testing as a devDependency, test runs mocha --exit --require ts-node/register src/mocha/*.test.ts, .nvmrc says 18, and .eslintrc.js forbids Node built-ins other than assert in src/workflows.ts.

Those tests prove the workflow's logic. None of them proves that a change to workflows.ts can still be replayed against an execution that is already running, which is what a Temporal deploy breaks when someone reorders two activities or adds a timer. The agent adds that check in two parts. src/mocha/record-histories.ts runs the workflow in the test environment twice, once with stubbed activities for the success path and once with the real activities and an invalid target account for the refund path, fetches each history with handle.fetchHistory() and writes it as JSON under src/mocha/histories/. Those two files are committed as fixtures.

src/mocha/replay.test.ts then reads every file in that directory, turns it back into a history with the SDK's historyFromJSON, and calls Worker.runReplayHistory({ workflowsPath: require.resolve('../workflows') }, history) for each. A workflow whose code no longer matches the events throws DeterminismViolationError, and that is the assertion. The test needs no server and no activities, so it runs in a fraction of the time the environment tests take, and the existing src/mocha/*.test.ts glob picks it up under npm test as it is. A test:replay script runs it alone, and record-histories is a separate script because fixtures are regenerated on purpose, not on every run.

To show the test does its job, the agent makes one deliberate non-deterministic edit in its sandbox, moving the deposit call ahead of withdraw, runs the replay test, quotes the DeterminismViolationError it gets, and reverts the edit. It runs npm run lint and npm run build too, since tsconfig.json includes src/**/*.ts and the recorder is new TypeScript.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add replay tests to the Temporal workflows in this TypeScript project. Read package.json, .nvmrc, tsconfig.json, .eslintrc.js, src/workflows.ts, src/activities.ts, src/banking-client.ts, src/shared.ts and src/mocha/workflows.test.ts first. Write src/mocha/record-histories.ts that runs moneyTransfer in a TestWorkflowEnvironment for the success path with stubbed activities and for the deposit-failure refund path with the real activities, fetches each history with fetchHistory and writes it as JSON under src/mocha/histories, and commit those files. Write src/mocha/replay.test.ts that loads every history with historyFromJSON and replays it with Worker.runReplayHistory against src/workflows.ts, failing on DeterminismViolationError. Add test:replay and record-histories scripts. Prove the test catches a change by temporarily reordering the withdraw and deposit calls, quoting the failure, and reverting. Run npm test, npm run lint and npm run build, and open a pull request quoting the output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds the recorder, the replay test, two history fixtures and two scripts, and leaves workflows.ts untouched. The description quotes the passing npm test run with the replay cases listed beside the four existing ones, the DeterminismViolationError from the deliberate reorder, and the lint and build output.

The run's record keeps the recording run, the replay run and the reorder experiment, so a reviewer can read the error the test raises rather than trust that it would.

Before merging, a reviewer decides whether the fixtures should be refreshed from a real namespace instead of the test environment, which the recorder could do with temporal workflow show --output json, and whether the replay test should also run in CI on every pull request, which it can since it needs no server.

Run it on every pull request that touches a workflow file

Once the fixtures exist, the check belongs on every change to src/workflows.ts. An automation in the Project with a github trigger on pull_request.opened and pull_request.synchronize runs a prompt that checks out the pull request's branch, runs npm run test:replay, and posts the result as a review comment, naming the history that failed when one does. The trigger adds the event after the prompt, so the prompt names the script and the fixture directory itself.

A team with several Temporal services starts one run per repository with one POST /o/{org}/sessions each, this prompt as input and the repository in resources, with a request_id that is a UUID v5 of the repository's full name so a retry returns the run it already started. Each run reads its own workflow and its own tests, so a service with three workflows records histories for the paths each one has.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

A replay test is only as good as the histories it replays, and recording them means running the workflow through both of its paths with a real worker and a real test server. The agent does that in its own sandbox, commits the fixtures, and then breaks the workflow on purpose to show the test fails, which is the step most teams skip.

The pull request carries the fixtures, the passing run and the quoted failure, so review is reading three outputs rather than installing the Temporal CLI locally. Across several services the job runs in parallel, one sandbox and one pull request each, each with histories recorded from its own workflows.

Frequently asked questions

Why record from the test environment rather than production?

Because the test environment runs in the sandbox with no credentials and gives a history for each path on demand, including the refund path, which needs a failing deposit. A production history is a better fixture for an execution that is actually running; the recorder can be pointed at one with temporal workflow show, and the pull request says how.

Does the replay test need a Temporal server?

No. Worker.runReplayHistory feeds the recorded events to the workflow code in-process and checks that the commands it produces match. That is why it runs faster than the existing tests and can run on every pull request without a service container.

What changes will it catch?

Any change that alters the order or kind of commands the workflow issues for the recorded path: reordering activities, adding or removing a timer, changing an activity's name. It does not catch a change to the retry policy's numbers or to activity code, which the environment tests cover.

What happens when the fixtures go stale on purpose?

A change that is meant to alter the workflow fails the replay test, which is the signal to version it with patched or to re-record. Running npm run record-histories after the change rewrites the fixtures, and the diff of the JSON files shows the reviewer exactly which events moved.

More Temporal jobs

More jobs like this: Write the missing tests

Keep reading