# Add Property-Based Tests to an Elixir Library

Yes. An agent adds StreamData to an Elixir library, writes properties for its encode, decode and parse functions, and opens a pull request with mix test green.

[Run this on Rush](https://cloud.getrush.ai/run/add-property-tests/elixir/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent starts with the build and the tests the library already has. In a library like elixir-plug/plug, `mix.exs` declares `app: :plug`, `elixir: "~> 1.15"` and three runtime deps, `mime`, `plug_crypto` and `telemetry`, plus `ex_doc` for the `:docs` env only; there is no `stream_data`, and `mix.lock` lists nine Hex packages. `.github/workflows/ci.yml` runs two rows, Elixir 1.15 on OTP 24 and Elixir 1.19 on OTP 28, and on the newer row it also runs `mix deps.get --check-locked`, `mix format --check-formatted` and `mix deps.unlock --check-unused`, so a dependency added to `mix.exs` without a committed `mix.lock` update fails CI before a single test runs. `test/test_helper.exs` starts ExUnit with `assert_receive_timeout: 200`, and `.formatter.exs` covers `test/**/*.exs`, so new files are formatted like the rest.

Then it reads the existing suites to find functions with an inverse or an invariant worth stating. `test/plug/conn/query_test.exs` is 251 lines of examples for `Plug.Conn.Query.decode/1` and `encode/2`, such as `decode("foo[]=bar&foo[]=baz")` giving `["bar", "baz"]`; `test/plug/conn/cookies_test.exs` has 197 lines for `Plug.Conn.Cookies.decode/1` and `encode/2`; `test/plug/conn/utils_test.exs` is six lines that only run the doctests of `media_type/1`, `content_type/1`, `params/1`, `token/1` and `list/1`; and `test/plug/html_test.exs` covers `Plug.HTML.html_escape/1` with a handful of strings. Each of these is a pure function of a binary, which is what property testing is for.

The agent adds `{:stream_data, "~> 1.4", only: :test}` to `deps/0`, runs `mix deps.get` so `mix.lock` gains the entry, and writes one property file beside each example file, using `use ExUnitProperties` and `check all` with the default 100 runs. For `Plug.Conn.Query` the property is a round trip: a map generated with `map_of` from `string(:alphanumeric, min_length: 1)` keys to printable strings, lists and nested maps encodes and decodes back to itself. The generators carry the format's own rules, written as comments rather than hidden: keys may not contain `[` or `]`, because `decode("foo[]")` reads brackets as list syntax, and an empty list does not survive `encode`, so lists are generated non-empty. For `Plug.Conn.Cookies` the property is that `encode(key, %{value: v})` followed by `decode` returns `v` for every token-safe key and value. For `Plug.HTML`, `html_escape/1` of any printable string contains none of `<`, `>`, `&`, `"` or `'` outside an entity, leaves a string without those characters unchanged, and equals `IO.iodata_to_binary(html_escape_to_iodata(s))`. For `Plug.Conn.Utils` the property is the absence of a crash: `media_type/1`, `content_type/1`, `token/1` and `list/1` return `:error` or a tagged tuple for any binary, never raise.

It runs `mix format`, `mix test`, `mix deps.unlock --check-unused` and `mix deps.get --check-locked`, then installs the older Elixir and OTP pair from the CI matrix and runs `mix test` there too, since StreamData's own floor is what decides whether the dependency fits the oldest row. When a property fails, StreamData shrinks the input to its smallest counterexample. The agent does not narrow the generator to make a failure disappear: a genuine counterexample goes into the pull request as a finding with the shrunk input and the assertion it broke, and the property stays marked `@tag :skip` with that reason until a maintainer decides.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add property-based tests with StreamData to this Elixir library. Read mix.exs, mix.lock, .formatter.exs, .github/workflows/ci.yml, test/test_helper.exs and every file under test/ first, and list the pure functions that have an inverse or an invariant: encode and decode pairs, parsers that must never raise, escaping functions. Add {:stream_data, \"~> 1.4\", only: :test} to the deps and commit the mix.lock change. Write one property file beside each existing example file, with use ExUnitProperties and check all. Put each format's rules into the generators and comment why. Run mix format, mix test, mix deps.unlock --check-unused and mix deps.get --check-locked, then install the oldest Elixir and OTP pair in the CI matrix and run mix test again. If a property finds a real counterexample, do not weaken the generator: report the shrunk input in the pull request and skip that property with the reason. Open a pull request that lists each property, what it states and how many runs it makes.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

A pull request on its own branch with the `mix.exs` and `mix.lock` change and the new property files, one per module: `query_property_test.exs`, `cookies_property_test.exs`, `html_property_test.exs` and `utils_property_test.exs` in the layout the repository already uses. Its description lists every property in one line each, the generator it draws from and the rule that generator encodes, and the output of `mix test` on both rows of the CI matrix is in the run's record for anyone who wants the raw lines rather than the summary.

A reviewer reads the generators before the assertions. A property is only as strong as the inputs it draws, so the question is whether `string(:alphanumeric)` keys and non-empty lists are the format's real rules or a shortcut around a bug, and the comments beside each constraint are where the agent has to make that case. Any shrunk counterexample in the description is the first thing to triage: it is a bug in the library or a rule the generator still misses, and either way it is worth more than the hundred passing runs around it.

## Run it on every nightly schedule

An automation in the Project with a `schedule` trigger on `cron` for the small hours in your `timezone` runs this prompt with a line added: cover only modules that gained a public function since the last run, and open no pull request when nothing is uncovered. Each night's run reads the current `mix.exs` and test tree, so a library that adopted StreamData last week gets new properties beside the ones it has, not a second dependency line.

For several libraries at once, your own server sends this body once per repository, each `POST /o/{org}/sessions` with that repository in `resources` and a `request_id` that is a UUID v5 of the repository's full name, so a retried call returns the run already started. Each run reads its own CI matrix and its own `mix.lock`, so a library that still supports Elixir 1.14 gets the StreamData version that fits it.

## Why run it on Rush?

Property tests are cheap to write and expensive to trust: the generator has to match the format, and the only way to know is to run the suite under the oldest and newest toolchain the project supports and read the shrunk failures. The agent does that in its own sandbox, installing the Elixir and OTP pairs from the CI matrix with no effect on anyone's machine, and the pull request carries the `mix test` output from each, so the reviewer judges the properties against evidence rather than against a claim that they pass.

It also scales the way a maintainer's attention does not. Twenty libraries with encode and decode pairs get the same treatment in the same hour, each with its own generators derived from its own test files, and every run keeps the full record of what it read and ran. A property that found a real counterexample in one library is a finding in that library's pull request, with the input that caused it, and nothing else.

## Frequently asked questions

### Does it change my existing ExUnit tests?

No. The example tests stay as they are; the properties go in new files beside them, following the same `test/plug/conn/` layout and the same `async: true` setting. The only edits to existing files are `mix.exs`, for the dependency, and `mix.lock`.

### What if a property fails because of a real bug?

The agent leaves the generator alone and reports the shrunk counterexample in the pull request, with the property skipped and the reason in its tag. Narrowing the inputs until the failure goes away would turn a finding into a hidden assumption, which is the opposite of what the suite is for.

### Will the new dependency pass the CI checks for the lock file?

Yes. The agent commits the `mix.lock` change and runs `mix deps.get --check-locked` and `mix deps.unlock --check-unused`, the two checks the newer row of the workflow makes, so the pull request arrives with the lock in the state CI expects.

### How many runs does each property make, and how long does the suite take?

StreamData's default is 100 runs per property, which the agent keeps unless a property needs `max_runs` to say otherwise. Each run is a few microseconds of encoding and decoding, so the suite stays within the time the existing `mix test` already takes; the exact time is in the run's record.

## More Elixir jobs

- [Add a Release Workflow to an Elixir Library](https://getrush.ai/run/add-release-workflow/elixir/)
- [Add Sobelow Security Scanning to a Phoenix App](https://getrush.ai/run/add-sobelow/elixir/)
- [Explain the Architecture of an Elixir Codebase](https://getrush.ai/run/architecture-overview/elixir/)

## The same job on other stacks

- [Add Property-Based Tests with Hypothesis to a pytest Suite](https://getrush.ai/run/add-property-tests/pytest/)

## More jobs like this: Write the missing tests

- [Add Random Test Order to a pytest Suite](https://getrush.ai/run/add-random-test-order/pytest/)
- [Add Replay Tests to Temporal Workflows](https://getrush.ai/run/add-replay-tests/temporal/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
