What the agent does
The agent reads the workflow and the tests before it touches either. In a repository laid out like vapor/template, .github/workflows/test-template.yml runs on pull requests and pushes to main with permissions: contents: read, caches a build of the Vapor toolbox in a cache-toolbox job, then in test-new-and-build runs a matrix over the swift:6.3-noble image, four --fluent.db choices and --leaf or --no-leaf, generates a project with vapor new template-test and runs swift test --package-path template-test inside that container, with MySQL and Postgres services for the database cells and DATABASE_HOST pointing at them. Package.swift is swift-tools-version:6.3, depends on Vapor 4.121.4 and names a {{name}}Tests target on VaporTesting; Tests/AppTests/AppTests.swift uses Swift Testing, a @Suite with .serialized when Fluent is on, a withApp helper that runs configure, autoMigrate, the test and autoRevert, and @Test functions for GET /hello, listing, creating and deleting todos through app.testing(). .gitignore already lists default.profraw, so someone has run coverage locally before; nothing in CI reads it.
Coverage in SwiftPM is one flag and two files. The agent changes the test step to swift test --enable-code-coverage --package-path template-test, which makes the toolchain write .build/debug/codecov/default.profdata and a <name>PackageTests.xctest bundle beside it under .build/debug/. From those it exports lcov with the llvm-cov that ships in the same Swift toolchain: llvm-cov export -format=lcov -instr-profile .build/debug/codecov/default.profdata .build/debug/<name>PackageTests.xctest --ignore-filename-regex='\.build/|Tests/' > coverage.lcov, so dependency sources checked out under .build/checkouts and the test target itself do not count toward the app's number. The test bundle's name is the package name with PackageTests.xctest appended, so the step derives it from swift package describe rather than hard-coding a name a template fills in. swift test --show-codecov-path prints the JSON report path the same build produces, and the step keeps that file as an artifact for anyone who wants the raw numbers.
Then the upload. The agent adds codecov/codecov-action after the export with files: coverage.lcov, flags set from the matrix cell (sqlite-leaf, postgres-noleaf, and so on) so eight reports from one run merge into one number with per-cell breakdowns, fail_ci_if_error: true, and token: ${{ secrets.CODECOV_TOKEN }} because the action needs a token outside public forks. A codecov.yml at the root sets a project status with a threshold so a pull request whose change drops coverage by more than a point fails its check, and a patch status so new lines are expected to be tested. The job runs inside the Swift container, where the cache action already required installing zstd; curl and git are present in swift:6.3-noble, which is all the upload step needs.
It proves the pipeline in its own sandbox before opening anything: it generates a project with the toolbox the way the workflow does, runs the coverage build, exports the lcov, and reads it to check that Sources/App/Controllers/TodoController.swift and routes.swift carry line counts and that Tests/ and .build/ are absent. It then runs the Codecov uploader in dry-run mode against the file to confirm the format. The pull request shows the per-file summary from llvm-cov report on the same profile, explains why .serialized on the Fluent suite matters for a merged number, and lists what was not changed: the tests, Package.swift, and the test-new-and-container job, which builds the Docker image and runs no tests.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add test coverage reporting to this Vapor app's GitHub Actions workflow. Read Package.swift, .github/workflows/test-template.yml, Tests/AppTests/AppTests.swift, .gitignore and the Dockerfile first. In the job that runs swift test, add --enable-code-coverage, then export lcov with the toolchain's llvm-cov from .build/debug/codecov/default.profdata and the <package>PackageTests.xctest bundle, deriving the package name from swift package describe and ignoring .build/ and Tests/ paths, and keep the JSON report from swift test --show-codecov-path as an artifact. Upload the lcov with codecov/codecov-action, with flags from the matrix cell, fail_ci_if_error and the CODECOV_TOKEN secret, and add a codecov.yml with project and patch statuses and a one-point threshold. Run the coverage build in your sandbox the way the workflow does, read the lcov to confirm the app's sources are counted and dependencies are not, and dry-run the upload. Open a pull request with the llvm-cov report summary and change nothing in the tests or Package.swift.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request changes one step and adds two in .github/workflows/test-template.yml, adds codecov.yml, and nothing else. Its description shows the llvm-cov report table from the sandbox run, with a line for each file under Sources/App and the overall percentage, the first lines of coverage.lcov to show the paths are relative to the package, and the dry-run output of the uploader.
Nothing changes for a developer running swift test locally: the flag is only in the workflow, so the local build does not pay for instrumentation unless they add it. Once the CODECOV_TOKEN secret exists, each pull request gets a Codecov check with the project and patch numbers, and the eight matrix cells show as flags on the same report.
Before merging, a reviewer adds the Codecov token to the repository's secrets, decides whether the one-point threshold in codecov.yml is right for a template whose tests are deliberately small, and whether the coverage step should run in every matrix cell or only in one, which the pull request offers as a one-line if: on the step.
Run it on every repository with tests but no coverage number
A team with several Swift services sends one POST /o/{org}/sessions per repository from the terminal or a script, with this prompt and the repository in resources, and a request_id that is a UUID v5 of the repository's full name so a retry returns the run already started. GET /o/{org}/github lists every repository the Rush GitHub App installation reaches, so a script can start a run only where a workflow runs swift test and no step mentions coverage.
Each run reads its own workflow, so a service that runs tests on a macOS runner gets the xcrun llvm-cov form, one that already uploads to another service gets the lcov export added beside it, and one with several test targets gets each bundle exported and merged. Each opens its own pull request with its own report table.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
SwiftPM's coverage output is easy to turn on and easy to get wrong in CI: the profile lands under .build/debug/codecov, the bundle name depends on the package name, dependencies under .build/checkouts inflate or deflate the number, and inside a container the toolchain's llvm-cov is the one that matches the profile format. The agent runs the whole chain in its own sandbox with the same Swift image the workflow uses and reads the lcov before it writes the workflow, so the first pull request check is one it has already seen pass.
The transcript keeps the build output, the report table and the uploader's dry run, so a reviewer can see the number and how it was made without running Swift locally. Several services can get coverage at once, each in its own sandbox, and nothing merges until a person reads the diff.
Frequently asked questions
Why export lcov instead of uploading the JSON that swift test produces?
The JSON from --show-codecov-path is SwiftPM's own format, which not every service reads, and it includes dependency files. llvm-cov export -format=lcov with an ignore pattern for .build/ and Tests/ gives the standard format with only the app's sources, and the JSON is still kept as an artifact for anyone who wants the raw data.
Where does the test bundle name come from?
SwiftPM builds a single <PackageName>PackageTests.xctest for all test targets, and in this repository the package name is a template placeholder. The step reads the name from swift package describe at run time, so the same workflow works in the template and in a project generated from it.
Does every matrix cell need to upload?
Uploading from all eight cells gives Codecov one merged number plus a flag per cell, which is useful because the Fluent and no-Fluent builds compile different code. If that is more than the team wants, the pull request shows the if: that limits the coverage steps to the Postgres and Leaf cell.
Will the instrumented build slow the workflow?
Slightly: the test build is instrumented and the export runs once per cell, which the pull request measures from its sandbox run. The toolbox cache and the services are unchanged, and the Docker build job is untouched because it runs no tests.
What about the Swift Testing suite being serialized?
.serialized on the Fluent suite makes the database tests run one at a time, which is why the profile covers each path cleanly; the agent leaves the attribute alone and says so. Coverage is read from the merged profile after all tests finish, so parallel or serial makes no difference to the number.
More Vapor jobs
The same job on other stacks
- Add Vitest Coverage to CI for a Vite React App (Vite + React)
- Add Code Coverage to CI for a VS Code Extension (VS Code extension)
- Add Test Coverage Reporting to Wagtail CI (Wagtail)