What the agent does
The agent clones the application into its own cloud sandbox and reads how it is built and checked before it adds a tool. In a Phoenix app like hexpm/hexpm, mix.exs requires Elixir ~> 1.20, lists Phoenix, Plug, Ecto, Bandit, Oban, Sentry and a long tail of syntax-highlighting packages, and defines a test alias that creates, loads and migrates the database first. .github/workflows/main.yml runs mix compile --warnings-as-errors, mix format --check-formatted and mix ecto.setup in a format job and mix test in a test job, both on Elixir 1.20.4 and OTP 29.1 against a postgres:18 service, with every action pinned to a commit SHA. A separate codeql.yml already has security-events: write and uploads results through the CodeQL action, and a .credo.exs is checked in, but nothing scans for Phoenix-specific security problems.
The agent adds :sobelow to deps for :dev and :test with runtime: false, runs mix deps.get, then runs mix sobelow --verbose once and reads every finding against the code it points at. The checks it runs for configuration pass on this app for reasons the agent can name: lib/hexpm_web/router.ex puts :protect_from_forgery and :put_secure_browser_headers in both the :browser and :browser_api pipelines, config/prod.exs sets force_ssl with rewrite_on: [:x_forwarded_proto], and config/runtime.exs reads HEXPM_SECRET_KEY_BASE from the environment. Where the scanner points at the literal secret_key_base values in config/dev.exs and config/test.exs, the agent records them as development-only in the config rather than silencing the check.
Findings in code get the same treatment, one at a time. Where a template renders Markdown through raw and the content was produced by the app's own renderer, the agent adds a # sobelow_skip comment naming the check on that line with a short reason; where a finding is a real problem, it fixes it in its own commit; and anything it is unsure about is listed in the pull request as a question rather than skipped. The result is a .sobelow-conf with exit: "low" so any new finding fails the run, skip: true so the inline skips are honoured, and an ignore list that is as short as the code allows.
Then it wires the scan into CI the way the repository already does security work. A sobelow job in codeql.yml installs Elixir and OTP with erlef/setup-beam at the same pinned version the other jobs use, restores the same dependency cache key, runs mix sobelow --config --format sarif and uploads the file with the CodeQL upload-sarif action under the existing security-events: write permission, so findings appear on the Security tab and on pull requests as annotations. The agent runs mix sobelow --config until it exits clean, then mix format --check-formatted and mix compile --warnings-as-errors so the new dependency does not break the format job.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add Sobelow security scanning to this Phoenix application. Read mix.exs, .github/workflows/main.yml, .github/workflows/codeql.yml, lib/*_web/router.ex, lib/*_web/endpoint.ex and the config files first. Add :sobelow to deps for dev and test with runtime: false, run mix deps.get and mix sobelow --verbose, and triage every finding against the code: fix real problems in their own commits, add a sobelow_skip comment with a one-line reason where the code is safe, and list anything you are unsure about in the pull request as a question. Write a .sobelow-conf with exit on low, skip enabled and the shortest ignore list the code allows, recording development-only secrets as such. Add a job to codeql.yml that installs Elixir and OTP with the same pinned erlef/setup-beam the other workflows use, restores the dependency cache, runs mix sobelow --config --format sarif and uploads the file with the CodeQL upload-sarif action. Run mix sobelow --config until it exits clean, then mix format --check-formatted and mix compile --warnings-as-errors. Open a pull request listing each finding and what you did with it.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request on its own branch changing mix.exs and mix.lock for the dependency, adding .sobelow-conf, adding a job to .github/workflows/codeql.yml, and touching only the source files where a finding needed a fix or a # sobelow_skip comment, each in its own commit with the check name in the message. The description holds a table of findings with file, check, severity and outcome, and a short list of questions for the reviewer where the agent chose not to decide alone.
The transcript at GET /o/{org}/sessions/{id}/messages keeps the first mix sobelow --verbose run with every finding, the runs after each change, and the final mix sobelow --config, mix format --check-formatted and mix compile --warnings-as-errors output, so a reviewer can see the scanner go quiet rather than read that it did. The pull request's own checks then run the new SARIF job for the first time.
Before merging, a reviewer reads each skip comment and decides whether the reason holds, checks that exit: "low" is what the team wants on the first day, and confirms the SARIF upload appears on the Security tab. A follow-up on the same session can tighten the config, or add the scan to the pre-existing format job instead of a separate one.
Run it on every Linear issue labelled security
Add an automation to the Project with this prompt and a linear trigger on linear.issue.labeled, filtered to the security label. When someone labels an issue such as "add a Phoenix security scan before the audit", the trigger places the issue after the prompt and the pull request links back to it. Only label changes made by a person in your Rush org fire the trigger, and the automation starts at most max_runs_per_hour runs in an hour.
Once the scan is in, keep it honest with a second automation: a github trigger on pull_request.opened and pull_request.synchronize with a prompt that runs mix sobelow --config on the pull request's branch and explains each new finding in context; the trigger clones that branch and adds the event after the prompt. From a script, either body goes to POST /o/{org}/sessions with the repository in resources and a request_id for a safe retry.
The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.
Why run it on Rush?
The hard part of adding a security scanner is the first run, when it reports everything it has ever been taught to worry about and someone has to read each finding against the code. The agent does that reading in its own sandbox with the dependencies compiled and the scanner installed, and ships the decisions as a config with reasons and a table, so the team inherits a clean baseline instead of a wall of warnings to argue about.
Every scanner run and its output stay in the run's transcript, so a reviewer can check that a finding marked safe was read rather than suppressed. The compile and format checks run exactly as CI runs them, and the laptop that handed off the task is free while the dependency tree compiles.
Frequently asked questions
Will it suppress findings to get a clean run?
No. Each # sobelow_skip comment names the check and carries a one-line reason on the line it covers, fixes go in their own commits, and anything the agent cannot justify is listed in the pull request as a question. The ignore list in .sobelow-conf is kept as short as the code allows and each entry says why.
Why put the job in codeql.yml rather than the main CI workflow?
Because that workflow already has the security-events: write permission and the SARIF upload pattern, so findings land on the repository's Security tab and as pull request annotations beside CodeQL's. The agent reuses the pinned erlef/setup-beam version and the dependency cache key from main.yml so the job compiles nothing twice.
Does it flag the secret_key_base values in dev and test config?
Where the scanner does, the agent records them in .sobelow-conf as development-only values, with a note that config/runtime.exs reads the production secret from HEXPM_SECRET_KEY_BASE, rather than turning the secrets check off for the whole project.
Does the app already pass the configuration checks?
The agent reads the router and config before running the scanner, and in this app the browser pipelines include :protect_from_forgery and :put_secure_browser_headers and production sets force_ssl, so those checks should pass. Whether they do is shown by the scanner output in the pull request, not assumed.
More Elixir jobs
More jobs like this: Turn a Linear label into a PR
- Add Social Cards to an Eleventy Blog (Eleventy)
- Add Social Cards to Sphinx Docs (Sphinx)
- Add Social Cards to a VitePress Blog (VitePress)