What the agent does
The agent checks out the pull request's branch in its own cloud sandbox and first works out what the change touches, because a Pulumi repository is rarely one program. In a repository shaped like pulumi/examples, CONTRIBUTING.md names every example <cloud>-<language>-<name>, AGENTS.md says there are more than 300 of them, and .github/workflows/pull-request.yml starts with a changes job that runs scripts/changed-examples.sh against the pull request's base to decide which directories changed and which integration test sets (AwsTs, GcpPy, and so on) they map to. The agent runs the same script, after fetching the base branch so the three-dot diff has a merge base, and reads from its output that a change under misc/test/, .github/, scripts/, Makefile, package.json or eslint.config.mjs counts as shared and would send CI to the full sweep.
For each changed program it reads the files a reviewer would. In aws-ts-s3-folder, Pulumi.yaml declares runtime: nodejs and a template: block whose aws:region config carries a description and the default us-west-2, package.json pins @pulumi/aws 7.23.0 and @pulumi/pulumi 3.228.0 with "type": "module", tsconfig.json compiles only index.ts under strict, and index.ts creates an aws.s3.Bucket, a BucketWebsiteConfiguration, a BucketPublicAccessBlock, one BucketObject per file in www/ and a BucketPolicy that depends on the access block, then exports bucketName and websiteUrl. The agent compares that against the rules the repository writes down: the README type from CONTRIBUTING.md (a deployable example needs the deploy button, sentence-case headings and the Prerequisites, Deploying and Cleaning up sections of example-readme-template.md.txt), the .github/pull_request_template.md checklist (no hardcoded credentials, no Pulumi.*.yaml stack files, a runtime: field), and misc/test/AGENTS.md, which asks every new example for a TestAcc<Cloud><Lang><Name> function that calls helpers.ProgramTest rather than integration.ProgramTest, under a //go:build Aws || all tag, so the test is skipped when PULUMI_CHANGED_PATHS does not cover its directory.
Then it runs what it can run without cloud credentials, in the order the workflow does. npx eslint . applies eslint.config.mjs, whose header rule rejects any .ts file missing the Copyright 2016-YYYY, Pulumi Corporation. All rights reserved. line; make check_python_formatting runs Black with the 100-column black.toml; inside a changed TypeScript example it runs npm install and npx tsc --noEmit, inside a Go example go build ./... and go vet, and in misc/test it runs go build -tags all ./... and go test ./helpers/..., the two commands the unit-go job uses to make sure a bad test edit fails before a provider job does. scripts/pr-preview-changed.sh would go further and run pulumi preview on a temporary file backend after seeding missing config with placeholders, but that still needs the provider's credentials in the environment; with none in the Project's secrets the agent skips it and says so in the review rather than reporting a preview it did not run.
The review it posts on the pull request is organised by example directory. Each finding cites a file and line and quotes the command output behind it: a missing template: block, a README that opens with a heading in title case, a test that calls integration.ProgramTest and would therefore deploy on every pull request, a package.json that pins an older @pulumi/pulumi than the rest of the repository, or a changed shared path that will cost a full nightly-sized run. It never approves or merges, and it states plainly that the integration tests in misc/test deploy real resources and were not run.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Review this Pulumi pull request. Fetch the base branch, run scripts/changed-examples.sh to list the changed example directories and the test sets they map to, and say whether any shared path forces a full sweep. For every changed example read Pulumi.yaml, the README, the dependency file and the program entry point, and check them against CONTRIBUTING.md, example-readme-template.md.txt, .github/pull_request_template.md and misc/test/AGENTS.md: naming, runtime field, template block, README type and headings, no credentials, no Pulumi.*.yaml stack files, and a TestAcc function that calls helpers.ProgramTest. Run npx eslint . and make check_python_formatting at the root, npm install and npx tsc --noEmit in changed TypeScript examples, go build and go vet in changed Go examples, and go build -tags all ./... plus go test ./helpers/... in misc/test. Run pulumi preview only if provider credentials are present in the environment, and otherwise say it was skipped. Post one review on the pull request listing findings per example with file, line and the command output that supports each one. Do not approve, merge or push.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a review on the pull request, written per changed example. For each one it lists what passed, with the command and its last lines, and what did not, with the file, the line and the rule it breaks, quoting CONTRIBUTING.md or misc/test/AGENTS.md where the rule lives. A closing section says which test sets scripts/changed-examples.sh selected, whether a shared path widens the run, and that pulumi preview and the TestAcc deployment were not run in the sandbox.
The session read at GET /o/{org}/sessions/{id} names the repository and the pull request, and the transcript at GET /o/{org}/sessions/{id}/messages keeps every command: the diff against the base, each eslint, tsc, go build and Black run with its full output, and the files the agent opened. A reviewer who doubts a finding can read the exact output it rests on.
Before acting on the review, a maintainer checks that each cited line is on the pull request's branch and not on the base, runs make test_example.TestAcc<Name> with real credentials for the examples the agent flagged, and decides whether a shared-path change is worth the full sweep. The agent's comment is one input to that decision, not the decision.
Run it on every pull request
In the console, add an automation to the Project with this prompt and a github trigger on pull_request.opened and pull_request.synchronize, with branch_patterns set to the default branch if the repository also takes pull requests against release branches. Each push starts one run on that pull request's branch, with the event after the prompt; pull requests from forks and pushes by bots never fire it, and the automation starts at most max_runs_per_hour runs in an hour.
From your own server, the same run starts with POST /o/{org}/sessions, resources set to https://github.com/your-org/your-app/pull/<n> and a request_id made from the head commit SHA, so a retried call returns the run already started for that push. Give the key a webhook URL and Rush POSTs run.completed with data.session_url when the review is posted, which a handler can record beside the check run.
The use case this job follows, with its automation and trigger: Review every pull request.
Why run it on Rush?
A Pulumi review done by reading the diff misses what the repository's own tooling catches: the header rule in eslint.config.mjs, Black's column limit, a tsconfig.json that compiles only index.ts, a Go test file without its build tag. In its own sandbox the agent installs the same Node, Go and Python the setup action pins and runs those tools on the real branch, so every finding comes with output rather than an opinion.
Many pull requests can be reviewed at once, each in a sandbox of its own, with nothing shared between an AWS example and a Kubernetes one. The transcript keeps what was run and what was skipped, so the line between checks that happened in the sandbox and deployments that need credentials is visible to anyone reading the run later.
Frequently asked questions
Does it deploy anything?
No. The integration tests under misc/test call pulumi up against real accounts, and the review says they were not run. The agent runs lint, formatting, compilation and the harness's own unit tests, and runs pulumi preview only when provider credentials are already in the environment; otherwise it skips the preview and says so.
How does it know which examples a pull request changed?
It runs the repository's scripts/changed-examples.sh, the same script the changes job in pull-request.yml uses, after fetching the base branch so the three-dot diff has a merge base. The script also reports when a shared path such as misc/test/ or Makefile changed, which sends CI to the full set of test sets, and the review passes that on.
Will it comment on every README style point?
It checks the rules CONTRIBUTING.md and example-readme-template.md.txt write down: the example type, the deploy button for deployable examples, sentence-case headings and the required sections. It cites the rule with each finding so a maintainer can see which ones are conventions the repository asks for and which are judgement.
Can it approve or request changes?
No. It posts one review comment with findings and leaves the approval to a person. If you want a blocking signal, a handler on your server can read the run's result from data.session_url when the run.completed webhook arrives and set a status on the commit.