# Review Pulumi Pull Requests

Yes. An agent checks out each Pulumi pull request, works out which programs changed, runs the lint, build and unit checks CI runs, and posts a review with output.

[Run this on Rush](https://cloud.getrush.ai/run/review-pulumi-changes/pulumi/), in a Project that has your repository.

## What the agent does

The agent checks out the pull request's branch in its own cloud sandbox and first works out what the change touches, because a Pulumi repository is rarely one program. In a repository shaped like pulumi/examples, `CONTRIBUTING.md` names every example `<cloud>-<language>-<name>`, `AGENTS.md` says there are more than 300 of them, and `.github/workflows/pull-request.yml` starts with a `changes` job that runs `scripts/changed-examples.sh` against the pull request's base to decide which directories changed and which integration test sets (`AwsTs`, `GcpPy`, and so on) they map to. The agent runs the same script, after fetching the base branch so the three-dot diff has a merge base, and reads from its output that a change under `misc/test/`, `.github/`, `scripts/`, `Makefile`, `package.json` or `eslint.config.mjs` counts as shared and would send CI to the full sweep.

For each changed program it reads the files a reviewer would. In `aws-ts-s3-folder`, `Pulumi.yaml` declares `runtime: nodejs` and a `template:` block whose `aws:region` config carries a description and the default `us-west-2`, `package.json` pins `@pulumi/aws` 7.23.0 and `@pulumi/pulumi` 3.228.0 with `"type": "module"`, `tsconfig.json` compiles only `index.ts` under `strict`, and `index.ts` creates an `aws.s3.Bucket`, a `BucketWebsiteConfiguration`, a `BucketPublicAccessBlock`, one `BucketObject` per file in `www/` and a `BucketPolicy` that depends on the access block, then exports `bucketName` and `websiteUrl`. The agent compares that against the rules the repository writes down: the README type from `CONTRIBUTING.md` (a deployable example needs the deploy button, sentence-case headings and the Prerequisites, Deploying and Cleaning up sections of `example-readme-template.md.txt`), the `.github/pull_request_template.md` checklist (no hardcoded credentials, no `Pulumi.*.yaml` stack files, a `runtime:` field), and `misc/test/AGENTS.md`, which asks every new example for a `TestAcc<Cloud><Lang><Name>` function that calls `helpers.ProgramTest` rather than `integration.ProgramTest`, under a `//go:build Aws || all` tag, so the test is skipped when `PULUMI_CHANGED_PATHS` does not cover its directory.

Then it runs what it can run without cloud credentials, in the order the workflow does. `npx eslint .` applies `eslint.config.mjs`, whose header rule rejects any `.ts` file missing the `Copyright 2016-YYYY, Pulumi Corporation.  All rights reserved.` line; `make check_python_formatting` runs Black with the 100-column `black.toml`; inside a changed TypeScript example it runs `npm install` and `npx tsc --noEmit`, inside a Go example `go build ./...` and `go vet`, and in `misc/test` it runs `go build -tags all ./...` and `go test ./helpers/...`, the two commands the `unit-go` job uses to make sure a bad test edit fails before a provider job does. `scripts/pr-preview-changed.sh` would go further and run `pulumi preview` on a temporary file backend after seeding missing config with placeholders, but that still needs the provider's credentials in the environment; with none in the Project's secrets the agent skips it and says so in the review rather than reporting a preview it did not run.

The review it posts on the pull request is organised by example directory. Each finding cites a file and line and quotes the command output behind it: a missing `template:` block, a README that opens with a heading in title case, a test that calls `integration.ProgramTest` and would therefore deploy on every pull request, a `package.json` that pins an older `@pulumi/pulumi` than the rest of the repository, or a changed shared path that will cost a full nightly-sized run. It never approves or merges, and it states plainly that the integration tests in `misc/test` deploy real resources and were not run.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Review this Pulumi pull request. Fetch the base branch, run scripts/changed-examples.sh to list the changed example directories and the test sets they map to, and say whether any shared path forces a full sweep. For every changed example read Pulumi.yaml, the README, the dependency file and the program entry point, and check them against CONTRIBUTING.md, example-readme-template.md.txt, .github/pull_request_template.md and misc/test/AGENTS.md: naming, runtime field, template block, README type and headings, no credentials, no Pulumi.*.yaml stack files, and a TestAcc function that calls helpers.ProgramTest. Run npx eslint . and make check_python_formatting at the root, npm install and npx tsc --noEmit in changed TypeScript examples, go build and go vet in changed Go examples, and go build -tags all ./... plus go test ./helpers/... in misc/test. Run pulumi preview only if provider credentials are present in the environment, and otherwise say it was skipped. Post one review on the pull request listing findings per example with file, line and the command output that supports each one. Do not approve, merge or push.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a review on the pull request, written per changed example. For each one it lists what passed, with the command and its last lines, and what did not, with the file, the line and the rule it breaks, quoting `CONTRIBUTING.md` or `misc/test/AGENTS.md` where the rule lives. A closing section says which test sets `scripts/changed-examples.sh` selected, whether a shared path widens the run, and that `pulumi preview` and the `TestAcc` deployment were not run in the sandbox.

The session read at `GET /o/{org}/sessions/{id}` names the repository and the pull request, and the transcript at `GET /o/{org}/sessions/{id}/messages` keeps every command: the diff against the base, each `eslint`, `tsc`, `go build` and Black run with its full output, and the files the agent opened. A reviewer who doubts a finding can read the exact output it rests on.

Before acting on the review, a maintainer checks that each cited line is on the pull request's branch and not on the base, runs `make test_example.TestAcc<Name>` with real credentials for the examples the agent flagged, and decides whether a shared-path change is worth the full sweep. The agent's comment is one input to that decision, not the decision.

## Run it on every pull request

In the console, add an automation to the Project with this prompt and a `github` trigger on `pull_request.opened` and `pull_request.synchronize`, with `branch_patterns` set to the default branch if the repository also takes pull requests against release branches. Each push starts one run on that pull request's branch, with the event after the prompt; pull requests from forks and pushes by bots never fire it, and the automation starts at most `max_runs_per_hour` runs in an hour.

From your own server, the same run starts with `POST /o/{org}/sessions`, `resources` set to `https://github.com/your-org/your-app/pull/<n>` and a `request_id` made from the head commit SHA, so a retried call returns the run already started for that push. Give the key a webhook URL and Rush POSTs `run.completed` with `data.session_url` when the review is posted, which a handler can record beside the check run.

## Why run it on Rush?

A Pulumi review done by reading the diff misses what the repository's own tooling catches: the header rule in `eslint.config.mjs`, Black's column limit, a `tsconfig.json` that compiles only `index.ts`, a Go test file without its build tag. In its own sandbox the agent installs the same Node, Go and Python the setup action pins and runs those tools on the real branch, so every finding comes with output rather than an opinion.

Many pull requests can be reviewed at once, each in a sandbox of its own, with nothing shared between an AWS example and a Kubernetes one. The transcript keeps what was run and what was skipped, so the line between checks that happened in the sandbox and deployments that need credentials is visible to anyone reading the run later.

## Frequently asked questions

### Does it deploy anything?

No. The integration tests under `misc/test` call `pulumi up` against real accounts, and the review says they were not run. The agent runs lint, formatting, compilation and the harness's own unit tests, and runs `pulumi preview` only when provider credentials are already in the environment; otherwise it skips the preview and says so.

### How does it know which examples a pull request changed?

It runs the repository's `scripts/changed-examples.sh`, the same script the `changes` job in `pull-request.yml` uses, after fetching the base branch so the three-dot diff has a merge base. The script also reports when a shared path such as `misc/test/` or `Makefile` changed, which sends CI to the full set of test sets, and the review passes that on.

### Will it comment on every README style point?

It checks the rules `CONTRIBUTING.md` and `example-readme-template.md.txt` write down: the example type, the deploy button for deployable examples, sentence-case headings and the required sections. It cites the rule with each finding so a maintainer can see which ones are conventions the repository asks for and which are judgement.

### Can it approve or request changes?

No. It posts one review comment with findings and leaves the approval to a person. If you want a blocking signal, a handler on your server can read the run's result from `data.session_url` when the `run.completed` webhook arrives and set a status on the commit.