What the agent does
The agent checks out the pull request in its own sandbox and reads the files that define the app's contract with Shopify, since most review findings on an embedded app are a mismatch between two of them. In an app built from Shopify/shopify-app-template-react-router, app/shopify.server.ts configures shopifyApp with ApiVersion.October26, Prisma session storage and AppDistribution.AppStore; .graphqlrc.ts names the same API version for code generation; and shopify.app.toml declares scopes = "write_products", api_version = "2026-10" under both [webhooks] and [events], and one subscription per topic whose uri matches a flat route file, such as /webhooks/app/uninstalled and app/routes/webhooks.app.uninstalled.tsx. package.json requires Node 22.12 or newer and pins the five React Router packages at exactly 7.18.2. .github/workflows/ci.yml installs with npm, yarn and pnpm on Node 22 and 24 and runs npx tsc --noEmit, npm run lint, npx prisma generate && npx prisma validate and npm run build.
It runs those commands on the head, plus two the workflow does not. npm run typecheck runs react-router typegen before tsc, which catches a route whose generated types changed. For the database it applies prisma/migrations/ to a fresh SQLite file with prisma migrate deploy and then asks prisma migrate diff whether prisma/schema.prisma matches what the migrations built. prisma validate passes on a schema edit that has no migration; the diff does not, and npm run setup, which production runs as prisma generate && prisma migrate deploy, would deploy the app without the new column.
Then it reads the diff against the template's own rules. Every loader and action under app/routes/app.* must call authenticate.admin(request), as app.tsx, app._index.tsx and app.additional.tsx do, and every webhook route must call authenticate.webhook(request) and return a Response. A new webhook topic belongs in shopify.app.toml and not in an afterAuth hook, which the README explains runs only after install and when a token expires. A new Admin GraphQL operation is checked against the declared scopes, and a changed API version must move in the server config, the GraphQL config and both TOML sections together. The README's embedded-app rules are applied line by line: a plain <a> where Link should be used, redirect imported from react-router where the one returned by authenticate.admin is required. Webhook handlers are read for repeat delivery, following the uninstall handler's own comment that a webhook can fire more than once and after the session is gone.
Two repository-specific checks close the review. .github/dependabot.yml ignores React Router releases above 7.18.2 and explains at length that a later patch tightened action-origin validation and broke form posts under shopify app dev, so a pull request that bumps those packages gets that comment quoted back with a request for the fix it describes. And the pull request template's checklist asks for a CHANGELOG.md entry and README updates, which the agent verifies. It posts one review, and states what it could not do: shopify app dev needs a signed-in Shopify account and a development store, so the app was built and type-checked but not opened in an admin.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Review the pull request this run was started for, on a Shopify app built with React Router. Read the diff and every file it touches, plus package.json, shopify.app.toml, shopify.web.toml.liquid, app/shopify.server.ts, .graphqlrc.ts, app/routes.ts, every file under app/routes, prisma/schema.prisma and prisma/migrations, .eslintrc.cjs, .github/workflows/ci.yml, .github/dependabot.yml, .github/PULL_REQUEST_TEMPLATE.md and the Gotchas section of the README. In your sandbox install dependencies with npm and run npm run typecheck, npm run lint, npx prisma generate, npx prisma validate and npm run build on the head, then apply the migrations to a fresh SQLite file and use prisma migrate diff to report any difference from the schema. Check that every loader and action under the app routes calls authenticate.admin, every webhook route calls authenticate.webhook and tolerates repeat delivery, new topics are declared in shopify.app.toml with a matching route, new GraphQL operations fit the declared scopes, the API version agrees across the server config, the GraphQL config and the TOML, embedded navigation follows the README rules, and the React Router pin explained in dependabot.yml is respected. Check the template's checklist. Post one review with a verdict, findings citing file and line, and the command output. Push nothing, and say that the app was not run in a Shopify admin.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
One review on the pull request. It gives a verdict, then findings grouped by kind: authentication, webhooks and scopes, API version, database, navigation, dependencies. Each cites a file and line and says whether it blocks. Under the findings are the results of the type check, lint, Prisma commands and build, the migration diff when there is one, and a line for each checklist item in the pull request template.
Nothing is pushed, and the complete logs remain in the run's record. A maintainer acts on the blocking findings and then does the part a sandbox cannot: runs shopify app dev against a development store and clicks through the changed screen inside the admin, where session and iframe problems actually appear.
Run it on every pull request opened
Create an automation in the app's Project with this prompt and a github trigger on pull_request.opened and pull_request.synchronize, with read_only on so its runs can comment and cannot push. The pull request event is added after the prompt, and every push to the branch starts a review of the new head. The trigger's authors setting can limit reviews to named GitHub logins. Bot senders never fire it, so Dependabot's weekly pull requests are left for a person to look at.
From your own server, the same review starts with POST /o/{org}/sessions: put the pull request URL in resources so the clone is its branch, and send a request_id that is a UUID v5 of GitHub's delivery id so a redelivery starts nothing new. A webhook_url set on the key receives run.completed with the session_url when the review has been posted.
The use case this job follows, with its automation and trigger: Review every pull request.
Why run it on Rush?
The costly mistakes in a Shopify app are consistent ones: a route that skips authentication, a topic with no handler, a schema change with no migration. They pass a build. The agent runs the repository's commands and the migration replay in its own sandbox, reads the diff against the TOML and the server config, and keeps all of the output, so each finding points at a line and, where a command found it, at that command's message.
Because each review has its own sandbox with its own node_modules and SQLite file, reviews of several pull requests run at the same time without sharing state, and a contributor who pushes a fix gets the new head reviewed while the maintainer is elsewhere.
Frequently asked questions
Can the agent run the app inside the Shopify admin?
The review does not attempt it. shopify app dev signs in to a Shopify account and installs the app on a development store, which the sandbox has no credentials for. The review builds and type-checks the app and says plainly that the embedded behaviour was not exercised.
How does it know a GraphQL operation needs a new scope?
It reads the operation and the scopes line in shopify.app.toml and checks the objects and mutations against Shopify's documentation for the pinned API version. Where the documentation is clear it cites it; where it is not, the finding is marked as a question for the author.
Why replay migrations when CI already runs prisma validate?
prisma validate checks that the schema file is well formed. It does not compare the schema with the migration history. Applying the migrations to an empty database and diffing against the schema is what shows a model change that nobody generated a migration for.
Will it review Dependabot pull requests?
A github trigger does not fire for bot senders, so those are not reviewed automatically. A maintainer can start a run for one by hand. The React Router pin and its explanation in dependabot.yml are checked on every human pull request that touches package.json.