All jobs

Add Property-Based Tests to an Elixir Library

Yes. An agent adds StreamData to an Elixir library, writes properties for its encode, decode and parse functions, and opens a pull request with mix test green.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent starts with the build and the tests the library already has. In a library like elixir-plug/plug, mix.exs declares app: :plug, elixir: "~> 1.15" and three runtime deps, mime, plug_crypto and telemetry, plus ex_doc for the :docs env only; there is no stream_data, and mix.lock lists nine Hex packages. .github/workflows/ci.yml runs two rows, Elixir 1.15 on OTP 24 and Elixir 1.19 on OTP 28, and on the newer row it also runs mix deps.get --check-locked, mix format --check-formatted and mix deps.unlock --check-unused, so a dependency added to mix.exs without a committed mix.lock update fails CI before a single test runs. test/test_helper.exs starts ExUnit with assert_receive_timeout: 200, and .formatter.exs covers test/**/*.exs, so new files are formatted like the rest.

Then it reads the existing suites to find functions with an inverse or an invariant worth stating. test/plug/conn/query_test.exs is 251 lines of examples for Plug.Conn.Query.decode/1 and encode/2, such as decode("foo[]=bar&foo[]=baz") giving ["bar", "baz"]; test/plug/conn/cookies_test.exs has 197 lines for Plug.Conn.Cookies.decode/1 and encode/2; test/plug/conn/utils_test.exs is six lines that only run the doctests of media_type/1, content_type/1, params/1, token/1 and list/1; and test/plug/html_test.exs covers Plug.HTML.html_escape/1 with a handful of strings. Each of these is a pure function of a binary, which is what property testing is for.

The agent adds {:stream_data, "~> 1.4", only: :test} to deps/0, runs mix deps.get so mix.lock gains the entry, and writes one property file beside each example file, using use ExUnitProperties and check all with the default 100 runs. For Plug.Conn.Query the property is a round trip: a map generated with map_of from string(:alphanumeric, min_length: 1) keys to printable strings, lists and nested maps encodes and decodes back to itself. The generators carry the format's own rules, written as comments rather than hidden: keys may not contain [ or ], because decode("foo[]") reads brackets as list syntax, and an empty list does not survive encode, so lists are generated non-empty. For Plug.Conn.Cookies the property is that encode(key, %{value: v}) followed by decode returns v for every token-safe key and value. For Plug.HTML, html_escape/1 of any printable string contains none of <, >, &, " or ' outside an entity, leaves a string without those characters unchanged, and equals IO.iodata_to_binary(html_escape_to_iodata(s)). For Plug.Conn.Utils the property is the absence of a crash: media_type/1, content_type/1, token/1 and list/1 return :error or a tagged tuple for any binary, never raise.

It runs mix format, mix test, mix deps.unlock --check-unused and mix deps.get --check-locked, then installs the older Elixir and OTP pair from the CI matrix and runs mix test there too, since StreamData's own floor is what decides whether the dependency fits the oldest row. When a property fails, StreamData shrinks the input to its smallest counterexample. The agent does not narrow the generator to make a failure disappear: a genuine counterexample goes into the pull request as a finding with the shrunk input and the assertion it broke, and the property stays marked @tag :skip with that reason until a maintainer decides.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add property-based tests with StreamData to this Elixir library. Read mix.exs, mix.lock, .formatter.exs, .github/workflows/ci.yml, test/test_helper.exs and every file under test/ first, and list the pure functions that have an inverse or an invariant: encode and decode pairs, parsers that must never raise, escaping functions. Add {:stream_data, \"~> 1.4\", only: :test} to the deps and commit the mix.lock change. Write one property file beside each existing example file, with use ExUnitProperties and check all. Put each format's rules into the generators and comment why. Run mix format, mix test, mix deps.unlock --check-unused and mix deps.get --check-locked, then install the oldest Elixir and OTP pair in the CI matrix and run mix test again. If a property finds a real counterexample, do not weaken the generator: report the shrunk input in the pull request and skip that property with the reason. Open a pull request that lists each property, what it states and how many runs it makes.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request on its own branch with the mix.exs and mix.lock change and the new property files, one per module: query_property_test.exs, cookies_property_test.exs, html_property_test.exs and utils_property_test.exs in the layout the repository already uses. Its description lists every property in one line each, the generator it draws from and the rule that generator encodes, and the output of mix test on both rows of the CI matrix is in the run's record for anyone who wants the raw lines rather than the summary.

A reviewer reads the generators before the assertions. A property is only as strong as the inputs it draws, so the question is whether string(:alphanumeric) keys and non-empty lists are the format's real rules or a shortcut around a bug, and the comments beside each constraint are where the agent has to make that case. Any shrunk counterexample in the description is the first thing to triage: it is a bug in the library or a rule the generator still misses, and either way it is worth more than the hundred passing runs around it.

Run it on every nightly schedule

An automation in the Project with a schedule trigger on cron for the small hours in your timezone runs this prompt with a line added: cover only modules that gained a public function since the last run, and open no pull request when nothing is uncovered. Each night's run reads the current mix.exs and test tree, so a library that adopted StreamData last week gets new properties beside the ones it has, not a second dependency line.

For several libraries at once, your own server sends this body once per repository, each POST /o/{org}/sessions with that repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started. Each run reads its own CI matrix and its own mix.lock, so a library that still supports Elixir 1.14 gets the StreamData version that fits it.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

Property tests are cheap to write and expensive to trust: the generator has to match the format, and the only way to know is to run the suite under the oldest and newest toolchain the project supports and read the shrunk failures. The agent does that in its own sandbox, installing the Elixir and OTP pairs from the CI matrix with no effect on anyone's machine, and the pull request carries the mix test output from each, so the reviewer judges the properties against evidence rather than against a claim that they pass.

It also scales the way a maintainer's attention does not. Twenty libraries with encode and decode pairs get the same treatment in the same hour, each with its own generators derived from its own test files, and every run keeps the full record of what it read and ran. A property that found a real counterexample in one library is a finding in that library's pull request, with the input that caused it, and nothing else.

Frequently asked questions

Does it change my existing ExUnit tests?

No. The example tests stay as they are; the properties go in new files beside them, following the same test/plug/conn/ layout and the same async: true setting. The only edits to existing files are mix.exs, for the dependency, and mix.lock.

What if a property fails because of a real bug?

The agent leaves the generator alone and reports the shrunk counterexample in the pull request, with the property skipped and the reason in its tag. Narrowing the inputs until the failure goes away would turn a finding into a hidden assumption, which is the opposite of what the suite is for.

Will the new dependency pass the CI checks for the lock file?

Yes. The agent commits the mix.lock change and runs mix deps.get --check-locked and mix deps.unlock --check-unused, the two checks the newer row of the workflow makes, so the pull request arrives with the lock in the state CI expects.

How many runs does each property make, and how long does the suite take?

StreamData's default is 100 runs per property, which the agent keeps unless a property needs max_runs to say otherwise. Each run is a few microseconds of encoding and decoding, so the suite stays within the time the existing mix test already takes; the exact time is in the run's record.

More Elixir jobs

The same job on other stacks

More jobs like this: Write the missing tests

Keep reading