All jobs

Review Pull Requests for a GraphQL API

Yes. For each pull request the agent diffs the schema for breaking changes, checks the committed SDL is current, runs the tests and posts one review.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The question every GraphQL pull request has to answer is whether a query that works today still works after the merge, and a code diff does not show that. The agent builds the answer from the schema. In a server like graphql/swapi-graphql the schema is defined in code under src/schema, and a printed copy is committed as schema.graphql, written by npm run store-schema through printSchema. The agent checks out the base branch in a second directory, prints the schema on both sides with npm run print-schema, builds each with buildSchema, and runs findBreakingChanges and findDangerousChanges from the installed graphql-js over the pair. A removed field or type, a removed argument, a new required argument, or a field that changed type is breaking. An added enum value or a changed default is dangerous. Each is listed with the type and field it concerns.

It then checks that the committed SDL tells the truth. If the schema printed from the pull request's code differs from the schema.graphql in the same branch, the author changed one and not the other, and anything that reads the file, a client code generator or a reviewer, sees a schema that does not exist. The review says which command regenerates it. The agent also runs the eight documents under doc/example_queries against the new schema. They stand in for client queries, and one that no longer validates is a breaking change with a concrete victim.

The repository's own checks run next, on the pull request's branch. npm ci runs a postinstall that downloads data from swapi.tech into cache/data.json and builds lib, and npm test chains eslint, flow check and jest; jest.config.js sets collectCoverage: true, so the run prints a coverage table. Each type has a spec under src/schema/__tests__ with a fragment such as AllFilmProperties that selects every field of the type. A pull request that adds a field and does not add it to that fragment has shipped a field no test resolves, and the agent names the fragment and the missing line.

Last it reads the resolvers against the patterns the codebase already follows. Objects are fetched through getObjectFromUrl and getObjectsFromUrls in src/schema/apiHelper.js, which go through one DataLoader; a resolver that calls getFromLocalUrl directly bypasses the batching. List fields are Relay connections built by connectionFromUrls with a totalCount, and a bare list of objects on a new field is flagged for the missing pagination. A new object type needs interfaces: () => [nodeInterface], a globalIdField, and a case in the switch in swapiTypeToGraphQLType, or node(id:) throws for it. No field in src is wrapped in GraphQLNonNull, and helpers such as convertToNumber return null for upstream values like unknown, so a pull request that makes a field non-null is asked what happens to the parent object when the data is missing. Fields carry a description; a new one without it is noted.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Review this pull request to our GraphQL API. Read package.json, jest.config.js, the scripts directory, schema.graphql, doc/example_queries and everything under src first. Check out the base branch in a second directory. In both, run npm ci (note what postinstall does) and print the schema with the repository's print-schema script. Build both schemas and run graphql-js's findBreakingChanges and findDangerousChanges from base to head; list each result with its type, field and kind. Compare the schema printed from the pull request's code with the schema.graphql committed on the pull request's branch and report any difference, with the command that regenerates the file. Validate and execute every document under doc/example_queries against the pull request's schema and report any that fails. Run npm test on the pull request's branch and report lint, type check and test results and the coverage of changed files. Then read the diff. Flag: a new field missing from its type's All...Properties fragment in the specs; a resolver that fetches data without going through the DataLoader-backed helpers; a new list field that is not a connection with totalCount; a new object type that does not implement Node, has no global id or is missing from the type switch used by node(id:); a field made non-null whose data can be missing; a field or argument removed without first being deprecated with a deprecationReason; a field without a description; and resolver errors that would leak internal details to clients. Post one review on the pull request: the schema change summary first (breaking, dangerous, safe additions), then the SDL check, the example queries, the test results, and findings by file and line with the suggested change. Do not push commits.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app/pull/42"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

One review on the pull request. It opens with the schema change summary in three groups: breaking changes, dangerous changes and safe additions, each entry naming the type and field. A pull request that only adds fields shows an empty first group, and the review says in one line that existing queries are unaffected. When the first group is not empty, each entry comes with the way to make the same change without breaking clients: keep the old field with a deprecationReason and add the new one beside it.

Below the summary are the SDL check, with the diff between the printed schema and the committed file if they disagree, the result of each example query, and the output of npm test with coverage for the files the pull request changed. Findings from reading the resolvers follow by file and line, each tied to the existing code that shows the pattern.

The review does not decide whether a breaking change is acceptable. An API with one first-party client can coordinate a removal; a public one cannot, and the agent does not know which clients send which fields unless usage data is given to it. It also cannot see production traffic. The maintainer merges with the breaking-change list in front of them, and every command behind it is in the session's transcript at GET /o/{org}/sessions/{id}/messages.

Run it on every pull request

An automation in the Project that holds the repository, with this prompt, read-only switched on and a github trigger on pull_request.opened and pull_request.synchronize, reviews each pull request when it opens and after each push. The trigger appends the pull request to the prompt and fills in nothing else. With read-only on, the run reads the repository and comments and cannot push. Setting branch_patterns to the release branch keeps reviews to pull requests that target it.

Only an owner, member or collaborator fires the trigger; a pull request from a fork or a bot does not. For a fork, a maintainer pushes the contributor's commits to a branch in the repository itself and opens the pull request from there. A team that wants the schema check on every pull request and the full review only on request runs two automations: this one on pull_request.labeled with a label in labels, and a shorter prompt that does only the schema comparison on every push.

The use case this job follows, with its automation and trigger: Review every pull request.

Why run it on Rush?

The schema comparison needs two checkouts, two installs and two builds, and nobody does that by hand for a small pull request. A small pull request is exactly the kind that removes a field by accident. The agent does it for every push in its own sandbox, where a second checkout and a second node_modules cost nothing to keep apart, and reviews for several pull requests run at once without sharing a working directory.

The result is a list a maintainer can act on and verify. Each breaking change names a type and field, each stale-SDL finding comes with a diff, and each example query has a pass or fail. The commands that produced them are kept with the run, so an author who thinks a change is safe can read what was compared and answer with evidence.

Frequently asked questions

How does it detect a breaking change?

It prints the schema from the base branch and from the pull request, builds both, and runs graphql-js's findBreakingChanges over them. That function reports removed types and fields, changed field types, removed or newly required arguments, and removed enum values. The agent adds the example queries as a second check, since a document that stops validating shows the break from the client's side.

We use SDL files, not a schema built in code. Does that work?

Yes. With schema-first projects the agent builds each side from the SDL files on that branch and compares the same way. The stale-file check then runs in the other direction: it confirms the resolvers cover every field the SDL declares.

Can it tell whether a removed field is still used by clients?

Not from the repository alone. It can say the field is used by an example query or a test. If you start a run with your usage export attached as a file, or with a connection to the service that holds it, the prompt can ask the agent to check each removed field against it.

Does it need the API to be deployed somewhere?

No. The schema is built and queried in-process inside the sandbox, the same way the repository's tests do it with graphql(). Nothing is deployed and no request is sent to your running service.

More GraphQL jobs

The same job on other stacks

Keep reading