All jobs

Add Fuzz Tests to a Go CLI

Yes. An agent writes native Go fuzz targets for your CLI's URL, front matter and escape-sequence parsers, seeds them from existing tests, runs them, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads where the program parses text it did not write before it picks targets, because a fuzzer pointed at a function that only takes well-formed structs finds nothing. In a CLI shaped like charmbracelet/glow (module charm.land/glow/v3, go 1.26.7, Cobra 1.10.2, Bubble Tea v2, Glamour v2), url.go turns a github://owner/repo or gitlab://owner/repo shorthand into a URL: githubReadmeURL and gitlabReadmeURL strip the prefix, split on /, return nil unless there are exactly two parts, and call url.JoinPath on the host; isURL accepts anything url.ParseRequestURI takes that contains ://. utils/utils.go has RemoveFrontmatter, which runs the yamlPattern regexp twice over a byte slice and slices the content at the second match, and IsMarkdownFile, which treats a name with no extension as Markdown. utils/textsize.go is the largest parser: ApplyTextSizing scans rendered output for \x1b]6666; markers, scanEscape walks CSI and OSC sequences by hand looking for a final byte, a BEL or an ESC-backslash, and writeOSC66 splits text into chunks under osc66MaxPayload.

Then it reads what tests exist, so seeds come from them rather than from invention. url_test.go has eight shorthand and full URL cases, every one skipped with t.Skip("test uses network, sometimes fails for no reason") because readmeURL goes on to call the GitHub and GitLab APIs. utils/textsize_test.go has TestApplyTextSizingPassthrough, TestApplyTextSizingMalformedMarker, TestApplyTextSizingLevels, TestApplyTextSizingWrappedHeading, TestApplyTextSizingNestedStyles and TestScanEscape, each with literal marker strings. .golangci.yml sets run.tests: false, so test files are not linted, and Taskfile.yaml runs go test ./... {{.CLI_ARGS}}.

Go has fuzzing built in since 1.18, so no new dependency is needed. The agent writes url_fuzz_test.go with FuzzReadmeShorthand, which feeds the string to githubReadmeURL and gitlabReadmeURL and, when a URL comes back, asserts its host is github.com or gitlab.com and its path is / plus the two parts it was given, so an input such as github://a/ or a part of .. that JoinPath cleans into a different path fails the target rather than reaching the API; FuzzIsURL asserts the function never panics and agrees with url.ParseRequestURI. utils/utils_fuzz_test.go holds FuzzRemoveFrontmatter, asserting the result is always a suffix of the input and that a document with no leading --- comes back unchanged. utils/textsize_fuzz_test.go holds FuzzApplyTextSizing, asserting a string with no marker prefix is returned byte for byte, that the output contains no \x1b]6666; marker, and that the function never panics on a marker cut off mid-sequence, and FuzzScanEscape, asserting the returned prefix is never longer than the input. readmeURL and sourceFromArg are left out on purpose: both reach the network or the filesystem, and the pull request says so.

Each target seeds its corpus with f.Add from the inputs the existing tests use, the eight URL forms and the marker strings from textsize_test.go, so the fuzzer starts from the shapes the code expects. The agent runs each target with go test -run '^$' -fuzz=FuzzApplyTextSizing -fuzztime=60s ./utils and the same for the others, commits any input the fuzzer writes to testdata/fuzz/<Target>/ so it runs as a regular test from then on, and for a reproducible failure adds the input as a plain t.Run case with the behaviour it observed, leaving the fix out of this change unless it is one line. Without -fuzz, go test ./... runs every target over its seed corpus, so coverage.yml's go test -race -covermode atomic ./... and the shared build.yml cover them with no new job; the agent adds a fuzz task to Taskfile.yaml that runs every target for a bounded time, and runs golangci-lint run to confirm nothing changed for the linter.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add Go fuzz tests to this CLI and open a pull request. Read go.mod, main.go, url.go, url_test.go, github.go, gitlab.go, utils/utils.go, utils/textsize.go, utils/textsize_test.go, glow_test.go, Taskfile.yaml, .golangci.yml and the workflows under .github/workflows first. Use the native testing.F fuzzer, no extra dependency. Write targets for githubReadmeURL and gitlabReadmeURL (assert the returned host and that the path is exactly the two parts given), isURL (no panic, agrees with url.ParseRequestURI), RemoveFrontmatter (result is a suffix of the input, unchanged without a leading front matter), ApplyTextSizing (unchanged without the marker prefix, no marker left in the output, no panic on a truncated marker) and scanEscape (prefix never longer than the input). Do not fuzz readmeURL or sourceFromArg; say why. Seed every corpus with f.Add from the inputs the existing tests already use. Run each target with -fuzztime=60s, commit any inputs written under testdata/fuzz, and for a reproducible failure add the input as a regular test case and describe it in the pull request without changing behaviour beyond a one-line fix. Add a fuzz task to Taskfile.yaml. Run go test ./..., go test -race ./... and golangci-lint run, and paste the results and each fuzzer's summary line in the pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds url_fuzz_test.go, utils/utils_fuzz_test.go and utils/textsize_fuzz_test.go, a fuzz task in Taskfile.yaml, and whatever the bounded runs wrote under testdata/fuzz/. No file under ui/ and no non-test source changes unless a one-line fix was taken. The description lists each target, the property it asserts, where its seeds came from, the summary line the fuzzer printed for each run with its duration and execution count, and the two functions left unfuzzed with the reason.

On a repository with glow's workflows, the pull request runs the shared build job, the coverage job's go test -race -covermode atomic ./..., which executes every target over its seed corpus and any committed failing input, and the lint job. The run's record keeps each -fuzz invocation with its output, including a crash and the input file it produced.

A reviewer checks that each target asserts a property of the result rather than only surviving, that the assertions match what the code is meant to do (a JoinPath that cleans .. may be intended), and reads any input the fuzzer committed. Then they decide how long to fuzz on a schedule, since the pull request's runs are bounded.

Run it on every Go repository with hand-written parsers

A team with several Go tools sends one POST /o/{org}/sessions per repository with this prompt and that repository in resources. Each run finds its own parsers and writes targets for them, so a tool that parses a config format gets targets for that, and each run opens its own pull request on its own branch.

Give the API key a webhook_url and Rush POSTs run.completed when each turn ends, with the session_url to read the pull request and the fuzzer output from. Send a request_id per repository so a retried call returns the run it already started.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

A fuzz target is only worth adding if it has been run, because the first minute of fuzzing a parser that was never fuzzed is when the truncated escape sequence or the empty path segment turns up. That minute is CPU time on a machine that is not someone's laptop. In a Rush sandbox the agent runs every target for a bounded time, keeps the fuzzer's own summary and any crash input in the run's record, and the pull request arrives with the first findings instead of empty harnesses.

Native Go fuzzing means the targets cost nothing after merge: with no -fuzz flag they are ordinary tests over their seed corpus, so the existing coverage and build workflows run them on every pull request without a new job.

Frequently asked questions

Does this need cargo-fuzz style tooling or a nightly toolchain?

No. Go has fuzzing in the standard testing package since 1.18, and this module is on go 1.26.7. A target is a FuzzXxx(f *testing.F) function, go test -fuzz drives it, and without the flag it runs as a normal test over its seeds.

Why not fuzz readmeURL or sourceFromArg?

readmeURL calls the GitHub or GitLab API for any github.com or gitlab.com host, and sourceFromArg opens files, walks directories and fetches http URLs. A fuzzer would hit the network thousands of times a second. The targets cover the pure functions underneath them instead.

What goes into git?

The three test files, the Taskfile task, and any inputs the fuzzer writes under testdata/fuzz/<Target>/, which Go then treats as part of the seed corpus. The fuzzer's cache under $GOCACHE is not committed.

Will CI run the fuzzers?

The seed corpus runs on every go test ./..., which the coverage workflow already does with -race. Open-ended fuzzing is the fuzz task in Taskfile.yaml, run by hand or from a scheduled job you add later with the -fuzztime you want.

More Go CLI jobs

The same job on other stacks

Keep reading