All jobs

Add Google Benchmark to a CMake Project

Yes. An agent adds a benchmark target behind a CMake option, fetches Google Benchmark pinned to a tag, writes benchmarks for the parser, runs them and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent clones the project into its own cloud sandbox and reads the build before adding to it. In a library like jarro2783/cxxopts, the root CMakeLists.txt declares cmake_minimum_required(VERSION 3.5...3.19), reads the version out of include/cxxopts.hpp with a helper in cmake/cxxopts.cmake, defines an INTERFACE target cxxopts with the alias cxxopts::cxxopts, and gates examples, tests, install and warnings behind options that default to on only when the project is built standalone. The warnings helper appends -Wall -Werror -Wextra -Wshadow -Weffc++ -Wconversion and more to CMAKE_CXX_FLAGS for the whole build, and the standard helper sets CMAKE_CXX_STANDARD to 11 unless CXXOPTS_CXX_STANDARD says otherwise. src/CMakeLists.txt builds one example target with cxx_std_17.

test/CMakeLists.txt is where the project's testing habits show: options_test and a CXXOPTS_NO_REGEX variant built against the single-header Catch v2.13.11 in test/catch.hpp, no-exceptions smoke tests on GCC and Clang, UBSan compiled into the parser tests on non-Windows, build-and-test checks for find_package, add_subdirectory and pkg-config consumers, a link_test across two translation units, and a libFuzzer target on Clang and Linux. .github/workflows/cmake.yml builds and runs ctest on Ubuntu 22.04 and 24.04 with g++-9, g++-10 and clang++, on macOS, and on Windows with ClangCl; build.yml does a plain cmake .., make and ctest; meson.yml covers the Meson build. There is no benchmark anywhere, so a change to the parser in the 3192-line header has no number attached to it.

The agent adds an option CXXOPTS_BUILD_BENCHMARKS, default OFF, so no existing CI job changes, and a benchmark/ directory with its own CMakeLists.txt. That file requires a CMake new enough for FetchContent, which the root's 3.5 floor does not guarantee, and fails with a clear message when the option is on under an older CMake; it declares Google Benchmark with FetchContent_Declare pinned to a release tag, sets BENCHMARK_ENABLE_TESTING and BENCHMARK_ENABLE_GTEST_TESTS off, strips -Werror and -Weffc++ from CMAKE_CXX_FLAGS in that directory's scope so the fetched library builds under this project's warning set, and raises CMAKE_CXX_STANDARD in the same scope because the library's own build asks for a newer standard than the 11 default. Then cxxopts_benchmark links cxxopts::cxxopts and benchmark::benchmark.

benchmark/parse_benchmark.cpp measures what the library is for: building an Options object with a dozen typed options, parsing a short argv of two flags, parsing a long one with positional arguments and repeated values, and parsing with CXXOPTS_NO_REGEX defined in a second target so the two parsing paths the tests already distinguish get numbers too, each benchmark using benchmark::DoNotOptimize on the result. The agent configures with the option on, builds, and runs the binary with --benchmark_format=json and --benchmark_out, pastes the table into the pull request, and adds one step to build.yml that configures with -DCXXOPTS_BUILD_BENCHMARKS=ON and runs the binary with --benchmark_min_time low enough to be a smoke check, uploading the JSON as an artifact. The Meson build gets no benchmark; the pull request says so.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add Google Benchmark to this CMake project. Read the root CMakeLists.txt, cmake/, test/CMakeLists.txt, src/CMakeLists.txt and every workflow under .github/workflows first. Add an option, default off, that enables a benchmark/ directory. In that directory fetch Google Benchmark with FetchContent pinned to a release tag, turn off its tests, and make sure it builds under this project's warning flags and C++ standard without changing them for the rest of the build. Write benchmarks for the library's hot path: constructing an options object, parsing short and long argument lists, and the no-regex parsing path as a second target. Configure with the option on, build, run the benchmarks with JSON output and paste the table into the pull request. Add one CI step that builds and runs the benchmarks briefly as a smoke check and uploads the JSON. Do not change the existing test targets or the Meson build. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds one option to the root CMakeLists.txt, a benchmark/ directory with a CMakeLists.txt and parse_benchmark.cpp, and one step in build.yml, with test/, src/, meson.build and the header untouched. The description holds the benchmark table from the sandbox run, the tag Google Benchmark is pinned to, and the two flags stripped for the fetched library and why.

The run's record keeps the configure with the option on, the build log showing the fetched library compiling under the adjusted flags, the benchmark run and a second configure with the option off proving the default build is unchanged, so a reviewer can see the existing ctest targets build exactly as before. The numbers in the table are from the sandbox's hardware and are there to show the harness works, not to be a baseline.

Before merging, a reviewer picks the Google Benchmark tag they want to track, decides whether the CI smoke step belongs in build.yml or in a separate workflow, and chooses the first benchmark they want to add to the three the agent wrote.

Run it on every pull request to the parser

The benchmark earns its keep when a change to the header gets a number. An automation with a github trigger on pull_request.opened and pull_request.synchronize starts one run per pull request; the agent checks out the branch, and when the diff does not touch include/cxxopts.hpp or benchmark/, ends with a one-line note. Only work by an owner, member or collaborator fires the trigger; pull requests from forks never do.

When it does, the run builds the benchmark binary from the base branch and from the pull request branch in the same sandbox, runs both with JSON output, compares them with the compare.py tool that ships in the Google Benchmark checkout, and posts one review with the table and the benchmarks that moved beyond noise, citing the diff hunk nearest each. It pushes no commits.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

Adding a benchmark harness to a header-only library is mostly build engineering: a fetched dependency that must compile under someone else's -Werror -Weffc++, a C++ standard floor the dependency does not share, a CMake minimum that predates FetchContent. The agent works those out against the real CMakeLists.txt in its own sandbox, with both configurations built and the benchmark run kept in the record, so the pull request is a harness that already works rather than a sketch.

The same request across an organisation's C++ libraries returns one pull request each, every one built from that project's own options, warning flags and standard rather than a template, and each with its own first table. The projects that already set -Werror get the scoped flag fix; the ones that do not get a smaller diff.

Frequently asked questions

Will this slow down the existing CI jobs?

No. The option defaults to OFF, so the matrix in cmake.yml, the macOS and Windows jobs and the Meson workflow configure and build exactly as before. Only the one new step in build.yml turns it on, and it runs the benchmarks for a short minimum time as a smoke check.

Why strip -Werror for the fetched library?

The project appends -Werror and -Weffc++ to CMAKE_CXX_FLAGS globally, and a third-party library built under those flags fails on warnings its authors never targeted. The agent removes those two flags in the benchmark/ directory scope only, so the project's own sources, including the benchmark file, keep the full warning set.

Are the numbers in the pull request a baseline?

No. They come from the sandbox's hardware on one run and show the harness builds and executes. A baseline needs the same machine over time, which is what the pull-request comparison automation above provides by building both branches in one sandbox.

Does FetchContent work with the project's CMake minimum?

Not at the 3.5 floor the root file allows. The benchmark/CMakeLists.txt checks the running CMake version and stops with a message naming the minimum it needs when the option is on; with the option off, an old CMake never reaches that check.

More CMake jobs

More jobs like this: Hand off a long task

Keep reading