All jobs

Review Pull Requests on an MCP Server

Yes. An agent builds an MCP server pull request, compares the tool list before and after, checks schemas, hints and error results, and posts one review.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reviews the server as a client would meet it, not only as a diff. In a repository like modelcontextprotocol/quickstart-resources it first reads the checks the project already has: make check runs make lint and then make test, the Makefile has one lint target per language (ruff, tsc --noEmit, gofmt with go vet, cargo fmt with clippy -D warnings, ruby -wc), and tests/smoke-test.sh starts each server and drives it with tests/helpers/mcp-test-client.ts. It installs only the toolchains for the directories the pull request touches, then runs those targets on the base commit and on the head and keeps both outputs.

Then it compares the tool surface. It starts the built server from each commit, connects over stdio with the SDK client, and saves the answer to tools/list: every name, description, input schema, output schema and annotation. The difference between the two files is the part of the pull request a host application will feel. A renamed tool, a newly required argument, a narrowed type or a changed hint breaks callers even when every test is green, so each one is a finding with the before and after quoted. An added tool is checked for a description a model can choose from and for arguments with their own descriptions, as state and latitude have in weather-server-typescript/src/index.ts.

After that it reads the diff for mistakes this kind of server makes. Anything written to stdout in a stdio server corrupts the protocol, so a console.log or a bare print is a blocker; the existing code logs through console.error. A tool that declares an outputSchema must return structuredContent of that shape, and an array-rooted schema must return a top-level array: in Python that means a RootModel like Alerts, because a plain -> list[Alert] return is wrapped as {"result": [...]}. An upstream failure should be an error result, raised as ToolError or a thrown Error, while an empty list is a normal answer. Defaults for upstream fields must survive explicit nulls, which is why the code uses ?? and or and not a missing-key default. Hints must be true: both weather tools set readOnlyHint and idempotentHint, and a new tool that writes cannot copy them.

Last it checks what the tests will not notice. The helper picks call arguments by matching the tool name against /alert/i and /forecast/i and skips a tool it has no arguments for, so a new tool with an output schema is listed and never called unless the pull request adds an entry. It checks that a behaviour change lands in every language the repository keeps in step, since AGENTS.md asks for all five in one pull request, and that an SDK bump comes with its lockfile. It pushes nothing and posts one review.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Review the pull request this run was started for, on an MCP server. Read the diff and every file it touches, plus AGENTS.md, the README's contributing section, the Makefile, the CI workflow and the tests directory. Install only the toolchains the touched directories need. On the base commit and on the head, run the repository's lint and test targets for those directories and keep the output. Build the server at both commits, connect to each over stdio with the SDK client, save tools/list from each, and report every difference in tool names, descriptions, input schemas, output schemas and annotations, marking the ones that break existing callers. Read the diff for: anything written to stdout in a stdio server; a tool whose structuredContent does not match its declared outputSchema, including an array-rooted schema answered with a wrapped object; upstream failures returned as normal results, or empty results returned as errors; defaults that do not survive explicit nulls from the upstream API; annotations that are not true of the tool; arguments without descriptions or bounds; secrets or tokens in code, examples or logs; a new tool the smoke test lists but never calls; a behaviour change made in one language and not the others; and a dependency change without its lockfile. Post one review with a verdict, findings that cite file and line, the two tool lists' differences and the command output. Push nothing, and say which checks you could not run and why.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is one review on the pull request. It opens with a verdict and a short table of the tool surface: tools added, removed or renamed, and for each changed tool the schema or hint that moved, with the old and new value. Findings follow in order of severity, each with a file and line and one sentence on what a client or a model would see. Below them are the lint and test results for base and head, so a failure that already existed on the base is not blamed on the author.

The review separates what was run from what was read. A stdout write found by grep, a schema mismatch caught by calling the tool, and a judgment that a description is too vague for a model to choose the tool are labelled as such. It also says what was out of reach: a tool that needs a credential the sandbox does not have is listed and not called. A maintainer reads the surface table first, because that is the compatibility decision, and then decides whether the hints and error handling match what the tool really does.

Run it on every pull request opened

Create an automation in the Project that holds the server's repository, with this prompt and a github trigger on pull_request.opened and pull_request.synchronize. Make it read only, so its runs comment and never push. The trigger puts the pull request event after the saved prompt, which is how the run knows what to check out. To review only changes to the tools, set labels on a pull_request.labeled trigger and have a maintainer add the label.

The trigger fires only for work by an owner, member or collaborator, and never for a bot or a fork. For Dependabot's weekly SDK bumps, a handler on your own server receives GitHub's webhook and sends this body with the pull request URL in resources, using a request_id that is a UUID v5 of the repository, the pull request number and the head SHA. For a contributor's fork, a maintainer pushes the commits to a branch in the repository itself and opens the pull request from there.

The use case this job follows, with its automation and trigger: Review every pull request.

Why run it on Rush?

The risky part of an MCP server change is rarely in the lines that changed. It is in the tool list that host applications and the models behind them already depend on, and nobody diffs that by hand. The agent builds both commits in its own sandbox, asks each server what it offers, and puts the answer side by side. Because every command and its output is kept in the run's record, the review quotes the tool list and the linter and is not one reader's impression.

It also runs when the maintainers are not there. A server with five language versions needs five toolchains to check properly, which few laptops have ready, and a review that skips two of them is the usual result. Each run installs what the diff needs, and ten open pull requests are ten runs that do not wait on each other.

Frequently asked questions

CI already runs lint and smoke tests. What does the review add?

CI answers whether the head passes. The review runs the same targets on the base too, compares the tool list between the two commits, and reads for what no test asserts: stdout writes, hints that are not true, a new tool the test helper never calls. Those are the changes that pass CI and still break a client.

Does it call the tools, or only list them?

It calls the ones it can call safely. Read-only tools are called with arguments it takes from the tests or the schema, and the result is checked against the declared output schema. A tool that writes, sends or deletes is listed and read, and the review says it was not called.

Can it review a server that needs API keys?

It reviews the code and the tool list either way, since listing tools needs no upstream call. To have it call tools against a real account, save the key as a secret on the Project so the run reads it as an environment variable. Without one the review names each tool it could not exercise.

Will it push fixes to the pull request?

No. The automation is read only and the prompt says to push nothing, so the result is a review. If you want a fix, reply on the session with a follow-up or start a separate run without the read-only setting.

More MCP server jobs

The same job on other stacks

Keep reading