What the agent does
An Elixir review that only reads the diff misses what the compiler and the database would have said. So the agent first learns how this project is checked. In a Phoenix application like hexpm/hexpm, .github/workflows/main.yml pins OTP 29.1 and Elixir 1.20.4 and runs, in order, mix deps.get, mix compile --warnings-as-errors, mix format --check-formatted and mix ecto.setup, then mix test with MIX_ENV=test against a postgres:18 service. mix.exs shows what those names expand to: the test alias creates the database, loads priv/repo/structure.sql, runs pending migrations and only then runs ExUnit, and ecto.setup rebuilds the database and runs priv/repo/seeds.exs. .formatter.exs includes .heex templates through Phoenix.LiveView.HTMLFormatter, so templates are formatted code here too. The README adds that the schema needs the pg_trgm and pgcrypto extensions from postgresql-contrib.
It reproduces that in its sandbox on the pull request's branch. It installs the OTP and Elixir versions the workflow names, installs PostgreSQL with the contrib package through apt, and sets the postgres user's password to what the project's config expects. Then it runs the four checks. A new unused variable or a call to a function that does not exist fails --warnings-as-errors; a template indented by hand fails the format check; a migration that cannot run on top of structure.sql, or a seed script the change broke, fails mix ecto.setup. For the tests it runs the files the diff touches first and then the whole suite, and quotes failures with their file and line as ExUnit prints them.
Reading the diff comes after, with the project's own patterns as the yardstick. Calls that leave the system go through behaviours with Mox doubles declared in test/support/mocks.ex, for billing, breached-password lookup, geolocation, HTTP and docs search, so a new integration that calls an HTTP client directly from a context module is a finding even if its test passes today. A new route belongs in the right pipeline in lib/hexpm_web/router.ex: the :api pipeline authenticates, the :browser pipeline protects against forgery. A new migration under priv/repo/migrations is read for what it locks and whether it can be rolled back, and the agent runs mix ecto.rollback and mix ecto.migrate once to see. A new background job should sit beside the existing Oban workers with a test of its own.
One check is particular to an app with clients in the wild. This server is the integration target for the Hex client, and CI has a job that clones hexpm/hex at main and at two release branches and runs the client's integration tests against the server. When a pull request changes anything under lib/hexpm_web/controllers/api or the API views, the agent does the same for the client's main branch, read-only, and reports whether an older client would still get the responses it expects. It also notes what it did not find: a .credo.exs is in the repository but Credo is not among the dependencies in mix.exs, so the review does not invent a lint step CI lacks. All of it goes into one review, and the agent pushes nothing.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Review this pull request to our Elixir application. Read mix.exs with its aliases, .formatter.exs, .github/workflows/main.yml, the README's setup section, config/test.exs, lib/hexpm_web/router.ex and test/support first. Install the OTP and Elixir versions the workflow pins and PostgreSQL with its contrib extensions, then on the pull request branch run mix deps.get, mix compile --warnings-as-errors, mix format --check-formatted, mix ecto.setup and MIX_ENV=test mix test, the changed test files first. Quote every warning, formatting diff and failing test with file and line. Then read the diff: external calls must go through a behaviour with a Mox double in test/support/mocks.ex; new routes must be in the right router pipeline; a new migration is checked for locks and for rollback by running mix ecto.rollback and mix ecto.migrate; a new Oban worker needs a test. If the change touches the API controllers or views, clone the Hex client at main read-only and run its integration tests against this branch the way the test-hex job does, and report the result. Do not add tools the project does not use. Post one review on the pull request with file and line for every finding and the command output quoted. Do not push commits.",
"resources": [
{
"url": "https://github.com/your-org/your-app/pull/42"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
One review on the pull request. The top is a results block: compile, format, database setup and tests, each with pass or fail and the lines that failed. If the API was touched there is a fifth line for the client's integration suite. Findings follow by file, each naming the line, the problem and the pattern in this codebase it departs from, such as an existing behaviour module or the pipeline a sibling route uses. A pull request that changes one template and passes everything gets a short review saying so.
The session holds the evidence in full. GET /o/{org}/sessions/{id}/messages returns the transcript: the mix output, the Postgres setup, the rollback and re-migrate, the client test run. A contributor who disagrees with a finding about a migration can read exactly what the rollback printed. The prompt asks for a review that comments; it does not approve or block the pull request on the team's behalf.
A maintainer still owns the judgement calls a tool should not make: whether the feature belongs, whether a public API change is acceptable to older clients even if their tests pass, and whether a slow migration is tolerable for the size of the real table, which the sandbox database does not have.
Run it on every pull request labelled agent-review
Some teams want a review on request, not on every push. An automation in the Project with this prompt, read-only switched on, and a github trigger on pull_request.labeled with agent-review in its labels starts one run each time a maintainer adds that label. The trigger places the pull request after the prompt and fills nothing in. Its max runs per hour keeps a bulk relabel from starting dozens of reviews at once; events past the cap wait as queued runs.
The trigger fires only for work by an owner, member or collaborator, and never for pull requests from forks, which is where an open source project gets most of its contributions. For those a maintainer pushes the contributor's commits to a branch in the repository itself and opens a pull request from it; the label on that one starts the review like any other.
The use case this job follows, with its automation and trigger: Review every pull request.
Why run it on Rush?
Reviewing an Elixir change properly means having the right OTP and Elixir pair, a Postgres with the right extensions, a database built from the structure file plus migrations, and for an API change a second repository checked out beside the first. Maintainers have that on one machine, tied to whatever branch they are on. The agent builds it fresh in a sandbox for each pull request, several at a time when a queue has built up, and the commands with their output are kept in the run's record.
It also checks the things humans skip when busy because they are tedious to set up: whether the seeds still load, whether the migration rolls back, whether an older client still passes. Each result is in the review with the command that produced it, so the human reviewer starts from facts and spends their attention on design.
Frequently asked questions
Does it run Credo or Dialyzer?
Only if the project does. The agent reads mix.exs and the CI workflow and runs the checks defined there, so a repository with Credo or Dialyxir in its dependencies gets those too. It does not add a tool the team has not adopted, since findings from an unconfigured linter are mostly noise.
How does it get a database for the tests?
It installs PostgreSQL in its own sandbox, with the contrib package for extensions such as pg_trgm, and lets the project's mix test alias create, load and migrate the test database. Nothing connects to your staging or production data.
Can it review LiveView and HEEx changes?
Yes. Templates are covered by the format check when the formatter config includes the LiveView HTML formatter, compile warnings catch what the compiler can see in templates, and LiveView tests run with the rest of the suite. The agent reads template changes for the same things a person would: unescaped output, missing phx- bindings, forms without the fields their changeset requires.
Will it review pull requests from forks?
Not automatically, because the github trigger ignores them. A maintainer pushes the contributor's commits to a branch in the repository itself and opens a pull request from it, and adding the label there starts the same review.