All jobs

Add libFuzzer Fuzz Tests to a CMake Project

Yes. An agent adds a libFuzzer harness as a CMake target behind an option, a seed corpus from the existing tests and a short fuzz job in CI, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent clones the repository into its own cloud sandbox and reads the build before it writes a harness. In a project like p-ranav/argparse, the library is one header, include/argparse/argparse.hpp, exposed by CMakeLists.txt as an INTERFACE target with cxx_std_17 under project(argparse VERSION 3.2.0), with ARGPARSE_BUILD_TESTS and ARGPARSE_INSTALL on when the project is top level and ARGPARSE_BUILD_SAMPLES off. test/CMakeLists.txt is its own project, argparse_tests, that compiles main.cpp and thirty-four test_*.cpp files into a doctest executable named tests, with -Wall -Wpedantic -Wconversion -Werror on GCC. .github/workflows/ci.yml runs on pull requests, configures from test/ on a seven-toolchain matrix and runs ./tests. A .clang-tidy and two tidy workflows exist; there is no fuzz target, no sanitizer build and no fuzz directory. The parser's entry points are parse_args, which takes a std::vector<std::string>, and parse_known_args, and both report bad input by throwing std::runtime_error or std::invalid_argument.

The agent adds a fuzz/ directory with fuzz_parse_args.cpp, which defines extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size), splits the bytes on NUL into an argument vector, builds an ArgumentParser with a fixed set of arguments that covers the features the doctest files exercise, a positional, a store_true flag, an option with scan<'i', int>, an nargs range, choices, append, a mutually exclusive group and a subparser, calls parse_args inside a try that catches std::exception, and lets anything else surface, since a crash, a sanitizer report or a hang is what the fuzzer is there to find. A second file, fuzz_help.cpp, streams help() and usage() for the parsed state so the formatting code is covered too. fuzz/CMakeLists.txt builds each file with -fsanitize=fuzzer,address,undefined behind a new option(ARGPARSE_BUILD_FUZZERS "Build libFuzzer targets" OFF) that the top-level CMakeLists.txt honours only when CMAKE_CXX_COMPILER_ID matches Clang, because libFuzzer ships with Clang's runtime.

A fuzzer finds more with a starting point, so the agent writes fuzz/corpus/ from argument vectors it reads out of the doctest files, such as the negative-number and compound-argument cases in test_negative_numbers.cpp and test_compound_arguments.cpp, one NUL-separated file each, and a fuzz/dictionary.txt of the tokens the parser treats specially: --, -, =, --help, --version. A fuzz/README.md explains how to build and run a target locally and how to turn a crash file into a doctest regression.

In ci.yml a new fuzz job on Ubuntu with Clang configures with -DARGPARSE_BUILD_FUZZERS=ON, builds, runs each target with the corpus and dictionary for a short -max_total_time and a -timeout, and uploads any crash-* or timeout-* file as an artifact when the job fails. In the sandbox the agent installs Clang, builds the targets and runs each fuzzer for the same short window, keeps the output, and if a run produces a crash file, minimizes it with -minimize_crash=1 and adds the input as a doctest case in test/ beside the fix or, when the fix is not obvious, as a documented reproducer in the pull request.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add libFuzzer fuzz tests to this CMake project. Read CMakeLists.txt, test/CMakeLists.txt, the doctest files under test/ and .github/workflows/ci.yml first. Create a fuzz/ directory with a harness that splits the input on NUL bytes into an argument vector and feeds it to parse_args on an ArgumentParser configured with a positional, a store_true flag, a scanned integer option, an nargs range, choices, append, a mutually exclusive group and a subparser, catching std::exception and nothing else, and a second harness that exercises help and usage output. Build them with libFuzzer, AddressSanitizer and UndefinedBehaviorSanitizer enabled, behind an ARGPARSE_BUILD_FUZZERS option that is only honoured with Clang. Seed a corpus from argument vectors in the existing tests and add a dictionary of parser tokens. Add a fuzz job to ci.yml that builds with Clang, runs each target for 60 seconds with a 10 second timeout and uploads crash files as artifacts. Install Clang in the sandbox, build and run each fuzzer for the same window, minimize and record any crash as a doctest regression, and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds fuzz/CMakeLists.txt, fuzz/fuzz_parse_args.cpp, fuzz/fuzz_help.cpp, fuzz/corpus/, fuzz/dictionary.txt and fuzz/README.md, adds the option to CMakeLists.txt, and adds a fuzz job to .github/workflows/ci.yml. The description shows the configure and build commands, pastes the tail of each fuzzer's output with its coverage counters, and either states that the window produced no crash or lists each minimized input with the doctest case that now pins it.

The run's record keeps the Clang installation, every build line and the full fuzzer output, so a reviewer can read the executions-per-second and coverage figures from the sandbox rather than trusting a summary.

Before merging, a reviewer decides whether the CI window is long enough to be useful without slowing every pull request, checks that the harness's argument set matches the features users rely on, and considers registering the project with OSS-Fuzz for continuous runs, which the pull request mentions but does not do.

Run it on every C++ repository with hand-written parsing

In the console, an automation in the Project that has the repository, with a github trigger on pull_request.opened and pull_request.synchronize and branch_patterns for your default branch, starts one run per pull request with the pull request as its context. The prompt tells the run to rebuild the fuzz targets, run them for the short window against the committed corpus plus any new seeds the pull request adds, and push a doctest regression to the branch when a crash appears.

From your own server, the same work is one POST /o/{org}/sessions per repository with this body and the repository in resources, with a request_id that is a UUID v5 of the repository name so a retried call returns the run already started. A webhook URL on the API key reports run.completed with the session_url to read the pull request link from.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

A fuzz harness is only useful when it reaches the code that matters and fails for the right reasons: one that catches every exception hides real bugs, one that catches none fails on every malformed flag, and one built without sanitizers misses the memory errors fuzzing exists to find. The agent reads the parser's entry points and the existing tests to choose what to catch and what to configure, builds with Clang and the sanitizers, and runs the fuzzer itself before the pull request exists.

The sandbox is its own computer where installing Clang and running a fuzzer for a minute leaves nothing behind, and every build and fuzzer line stays in the run's record. Several C++ libraries can get harnesses at once, each in its own sandbox with its own pull request.

Frequently asked questions

Why is the fuzz target off by default and Clang only?

Because -fsanitize=fuzzer is a Clang feature, and the project's CI builds on GCC and MSVC too. The ARGPARSE_BUILD_FUZZERS option keeps the default configure unchanged for every consumer who adds the library with add_subdirectory, and the Clang check turns the option into a clear message rather than a link error elsewhere.

What does the harness do with a thrown exception?

It catches std::exception, which covers the std::runtime_error and std::invalid_argument the parser throws on bad input, and returns normally, since those are the documented way the library reports a bad command line. Anything else, a crash, an AddressSanitizer report, an infinite loop past the timeout, fails the run and produces a reproducer file.

Does the run find bugs?

It might, and the pull request reports whatever the short window found, minimized, with a doctest case pinning each input. It also might find nothing in a minute, which is a normal result for a mature parser, and the value is then the harness, the corpus and the CI job that keep running on every pull request.

Does this register the project with OSS-Fuzz?

No. OSS-Fuzz integration lives in a separate repository and needs a project maintainer to apply. The harness the agent adds follows the libFuzzer entry-point convention OSS-Fuzz builds, so the pull request notes that the same files can be reused there.

More CMake jobs

The same job on other stacks

Keep reading