All jobs

Review Pull Requests for a discord.py Bot

Yes. For each pull request the agent applies the migrations to a scratch PostgreSQL, runs pyright, reads the diff for bot-specific mistakes and posts one review.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

A bot's pull request can pass every check a generic reviewer runs and still let any member ban people, so the agent learns the bot's own conventions first. In a bot like Rapptz/RoboDanny those live in a few files. cogs/utils/checks.py defines is_mod(), is_admin() and is_manager() on top of hybrid_permissions_check, which applies a commands.check for prefix commands and app_commands.default_permissions for slash commands in one decorator. bot.py builds the client with an explicit discord.Intents(...) list, AllowedMentions(roles=False, everyone=False, users=True), and a tuple of 22 initial_extensions. launcher.py applies SQL files named V<n>__<description>.sql from migrations/ in version order, and its own comment says the numbers must rise with no gaps. There are no tests and no workflow, and pyproject.toml configures black, isort and pyright.

It runs what can run without a token. It installs requirements.txt in a virtual environment and runs pyright, black --check and isort --check on the files the pull request changes, on the base branch and on the pull request's branch, and reports only what the pull request added; a codebase with no CI rarely starts clean. It imports every changed module with a stand-in config module, because config.py is ignored by git and both bot.py and launcher.py import it, so a syntax error or a bad import in a cog is caught without starting the bot. For a pull request that touches migrations/, it installs PostgreSQL in the sandbox with sudo apt-get install -y postgresql, creates the role and database the README describes, with the pg_trgm extension the first revision's indexes need, and runs python launcher.py db init so every revision, the new one last, applies in order.

Then it reads the diff against the conventions. A new moderation or configuration command must carry one of the permission decorators, and a hybrid command must be guild-only when its body reads ctx.guild. A new cog needs an async def setup(bot) and an entry in initial_extensions; cogs/poll.py in this repository has the first and not the second, so it loads only when someone loads it by hand. A feature that listens for an event the current intents do not deliver is flagged with the intent it needs. A message built from user text that passes its own allowed_mentions is checked against the bot-wide default it overrides.

Two classes of finding get the closest reading. SQL: queries here pass values as $1 parameters to asyncpg, and the few f-string queries interpolate column names or clauses the code builds itself; in one place, cogs/stars.py, a clamped number and a unit are formatted into an interval with a comment explaining why. A pull request that formats a command argument into a query string is reported with the parameterised form. A write to a table that a @cache.cache() getter reads needs the matching invalidate call, as the existing writes have. Responsiveness: a slash command that queries the database or calls an HTTP API before it answers should defer first, as the existing cogs do, and blocking calls such as time.sleep or a synchronous HTTP client inside a coroutine are flagged with the aiohttp session on the bot as the replacement.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Review this pull request to our discord.py bot. Read README.md, requirements.txt, pyproject.toml, bot.py, launcher.py, cogs/utils/checks.py, cogs/utils/context.py and the migrations directory first, and write down the conventions: the permission decorators, the intents and allowed mentions the bot sets, how cogs are registered, how migrations are named and applied, and how queries pass values. Create a virtual environment and install requirements.txt plus pyright, black and isort. Run pyright, black --check and isort --check on the changed files on the base branch and on the pull request's branch and report only what the pull request introduced. Import each changed module with a stand-in config module registered in sys.modules; do not create config.py and do not start the bot. If the diff touches migrations/, install PostgreSQL with sudo apt-get install -y postgresql, create the role, database and extension the README describes, write a config module that points at it, and run python launcher.py db init; report whether every revision applied and whether the new file's number follows the last with no gap. Then read the diff. Flag: a moderation or configuration command with no permission decorator; a command that uses ctx.guild but can run in a direct message; a new cog missing setup() or missing from initial_extensions; an event listener whose intent is not enabled; SQL that formats a command argument into the query string, with the parameterised version; a write to a table read through a cached getter with no invalidate; a slash command that does slow work before responding without deferring; blocking calls in coroutines; a message built from user text that overrides allowed_mentions; and a new secret read from anywhere but the config module. Post one review on the pull request: tool results first, then findings by file and line with the suggested change. Do not push commits.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app/pull/42"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

One review on the pull request. It opens with the tool results: the pyright errors and formatting differences the pull request introduced, set apart from those already on the base branch, the result of importing each changed module, and, when migrations changed, the output of launcher.py db init against the scratch database with the revision count it applied.

Findings follow by file and line. Each names the convention it comes from and where that convention is defined, so the author can see that is_mod() in cogs/utils/checks.py is what the other moderation commands use, or that the other queries in the same cog pass their values as parameters. Each has the suggested change. A pull request that follows the conventions gets a short review saying which checks ran and that nothing was found.

The review does not prove the command works in a server. The agent has no bot token and connects to no gateway, so behaviour that depends on Discord, such as role hierarchy, rate limits or how a modal renders, is outside what it can check, and the review says so. A maintainer still runs the change in a test server before merging; the review is there so that session is spent on behaviour and not on a missing decorator.

Run it on every pull request

An automation in the Project that holds the repository, with this prompt, read-only switched on and a github trigger on pull_request.opened and pull_request.synchronize, posts a review when a pull request opens and again on every push to it. The trigger adds the pull request after the prompt and fills in no placeholder. Read-only lets the run read the repository and comment, and keeps it from pushing.

Community bots get most of their pull requests from forks, and the trigger never fires for a fork or a bot account, only for an owner, member or collaborator. A maintainer who wants a fork's change reviewed pushes its commits to a branch in the repository and opens a pull request from that branch. To review only when asked, use pull_request.labeled with the label in labels. The automation's max runs per hour limits how many reviews a burst of pushes can start.

The use case this job follows, with its automation and trigger: Review every pull request.

Why run it on Rush?

The checks that catch real bot bugs need an environment a volunteer maintainer does not want to rebuild for each pull request: a clean virtual environment, a throwaway PostgreSQL with the right extension, the migrations applied from zero. In a sandbox that environment is built fresh for every review and thrown away after, and pull requests from several contributors are reviewed at once without sharing a database.

A bot runs with powers in other people's servers, so the cost of a missed permission check is paid by its users. Having every pull request read against the same list of conventions, with the commands and their output kept in the session's transcript at GET /o/{org}/sessions/{id}/messages, means a finding can be traced to what was run and a clean review means the same thing each time.

Frequently asked questions

Does the review need my bot token or database credentials?

No. The agent never starts the bot. It imports changed modules with a stand-in config module and, when migrations change, applies them to a PostgreSQL it installs inside its own sandbox. Your production database and token are not part of the run.

Can it test a slash command end to end?

Not against Discord. Without a token there is no gateway connection and no interaction to answer. It checks what can be checked from code and tools: types, imports, permissions, SQL, migrations and response timing patterns. It says in the review that live behaviour was not exercised.

Our bot has no tests or CI. Is a review still useful?

That is the case this page is written for. The agent runs the tools the repository already configures in pyproject.toml and compares base against branch, so existing problems do not drown out new ones. It then reads the diff against conventions it found in the code, which need no test suite.

Will it push fixes to the pull request?

No. The automation is read-only, so the run comments and cannot push. Each finding carries the suggested change for the author to apply. If you want fixes written, start a separate run without read-only that opens a pull request against the author's branch.

More discord.py jobs

The same job on other stacks

Keep reading