What the agent does
The agent reads the program as Pulumi will run it. In pulumi/examples' aws-ts-s3-folder, index.ts creates an aws.s3.Bucket, a BucketWebsiteConfiguration with indexDocument.suffix set to index.html, a BucketPublicAccessBlock with blockPublicAcls: false, one BucketObject per file in www/ (index.html and favicon.png, typed with mime.getType), and a BucketPolicy that allows s3:GetObject to Principal: "*" on ${siteBucket.arn}/*, with dependsOn on the access block. It exports bucketName and websiteUrl. package.json pins @pulumi/pulumi 3.228.0 and @pulumi/aws 7.23.0, sets "type": "module", and has ts-node with ESM enabled in tsconfig.json, but no test script and no test runner.
Pulumi programs declare resources as a side effect of being imported, so the order matters. The agent writes a test file that calls pulumi.runtime.setMocks first, with a newResource that returns each resource's inputs as its state plus an id and an arn it builds from the name, and a call that answers any provider function. Only then does it await import the program. Because the package is ESM TypeScript, it runs Mocha through a TypeScript loader, adds that as the test script, and runs the tests from the project directory, since index.ts reads www as a relative path.
The assertions read resolved outputs through pulumi.Output apply and check what the program means rather than how it is spelled. The website test checks the index document suffix is index.html and the configuration targets the site bucket. The access test checks the policy grants only s3:GetObject, to every principal, on that bucket's ARN with /*, and that the access block leaves public ACLs unblocked. It also checks there is one BucketObject per file in www/, with text/html and image/png content types.
The request asks for tag checks too, and this program sets no tags on any resource. The agent does not invent a passing test for that: it writes the tag check as the team would want it (for example, every bucket carries the stack and project names), marks it as an expected failure, and says in the pull request that the program has no tags to check yet. A real gap found by a test is reported, not hidden.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "In aws-ts-s3-folder, write unit tests with pulumi.runtime.setMocks and Mocha that check the bucket's website settings, the public access policy and tags. Run them and open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-pulumi-programs"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request that adds a test file under aws-ts-s3-folder/tests/, a test script, and Mocha, its types and a TypeScript loader in package.json and the lockfile. The test file sets mocks before it imports index.ts, then checks the website configuration, the access block, the bucket policy document, and the objects created from www/. index.ts is unchanged.
The description lists each assertion with the resource it reads, explains why the tag test is marked as expected to fail, and notes that the tests run with no AWS credentials and no Pulumi backend login. The session's trajectory keeps each npm test run with its output.
Before merging, a reviewer checks the mocks return realistic state for any resource whose outputs the program reads, decides what tags the program should set, and checks the policy assertions match the intended exposure. With tags added in a follow-up, the expected failure becomes a passing test.
Run it on every pull request that changes a Pulumi program
A GitHub pull_request webhook can send a request when a pull request touches a Pulumi program's directory, with that pull request in resources, asking the agent to update or add unit tests for the resources it changed. Each pull request gets its own sandbox, so programs for different clouds are tested side by side.
Set a webhook URL on the API key and Rush POSTs run.completed or run.failed when the turn finishes, so the pull request gets the test result before pulumi preview ever runs against a real account.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Unit tests are the one kind of Pulumi test that needs no cloud account, which makes them the right place to start and the easiest place to get subtly wrong: mocks registered after the import, outputs never resolved, assertions that pass because nothing ran. In its own sandbox the agent runs the suite with no AWS credentials present at all, so a test that tried to reach a provider would fail there rather than quietly use your profile.
The run shows its work: every npm test and its output is in the trajectory, and the gap the tests found is named in the pull request. Because each run is independent, the same request can add unit tests to every program in a repository of infrastructure examples at once, each reviewed on its own.
Frequently asked questions
Do the tests need AWS credentials or a Pulumi Cloud login?
No. With pulumi.runtime.setMocks, every resource is answered by the mock in the test process, so there is no provider call and no state backend. The sandbox for this request has no cloud credentials at all.
Why mock here when real dependencies are better?
These are unit tests of what the program declares, and Pulumi's mocks are the framework's own way to run a program without a deployment. They check intent: the policy document, the website settings, the objects. Whether AWS accepts that policy is a job for an integration test with pulumi up against a test account.
Can it write tests for Python or Go Pulumi programs?
Yes. Pulumi has the same mocks in its Python and Go SDKs, with pytest or go test as the runner. Name the program and the runner in the prompt.
What if my program has no tags at all?
Then a passing tag test would be a lie, so the agent writes the check you asked for, marks it as expected to fail, and says why in the pull request. Ask it in a follow-up to add the tags and it changes the program and the test together.