# Add Property-Based Tests with Hypothesis to a pytest Suite

Yes. An agent adds Hypothesis to a pytest project, writes property tests for the invariants it finds in the code, runs them in its sandbox and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-property-tests/pytest/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads the code for invariants before it writes a test, because a property test is only as good as the property. In a library like pallets/markupsafe, `src/markupsafe/__init__.py` says that `escape` replaces `&`, `<`, `>`, `'` and `"`, returns a `Markup`, and calls `__html__` instead when the object has one; `Markup.unescape` reverses entities with `html.unescape`; `striptags` removes tags and comments and collapses whitespace; and the actual replacement lives in two implementations, `_native.py` and the C extension in `_speedups.c`. `tests/conftest.py` has a session-scoped autouse fixture parametrised over both modules that swaps `markupsafe._escape_inner`, so every existing test already runs twice. `tests/test_escape.py` is a `parametrize` table of hand-picked strings: empty, ASCII with the five characters at the start, middle and end, the same for two-byte and four-byte characters. That table is exactly the shape a property test generalises.

From that reading it writes `tests/test_properties.py` with four properties. For any text, `Markup(escape(s)).unescape()` gives back `s`, since every replacement is an entity that `html.unescape` knows. Escaping is idempotent: `escape(escape(s)) == escape(s)`, because the result carries `__html__` and is returned as is. The result is always a `Markup`, checked with `type(...) is Markup`, the way the existing `test_type_behavior` checks it. And the two implementations agree: `_native._escape_inner(s) == _speedups._escape_inner(s)` for any text, called directly rather than through the fixture, which is the property that matters most for a C extension. Strings come from `st.text()`, plus a second strategy drawn from a short alphabet of the five special characters and a few letters so that examples hit the replacements often instead of rarely.

The project's own settings shape the details. `hypothesis` goes into the `tests` dependency group in `pyproject.toml` and `uv.lock` is regenerated, so `uv run --locked tox run` finds it. mypy runs in strict mode over `tests` as well as `src`, so the new file is fully typed. `pytest-run-parallel` runs the suite with threads on free-threaded Python and `pyproject.toml` already declares a `thread_unsafe` marker for tests that must not; Hypothesis keeps per-test state, so the agent marks the new tests with it. Hypothesis writes its example database to `.hypothesis/`, which the agent adds to `.gitignore`.

Then it runs `uv sync` and `uv run pytest tests/test_properties.py -v` in its sandbox, where `uv sync` compiles `_speedups` so the differential test exercises the real extension. Each test runs twice because of the fixture. If Hypothesis finds a counterexample it shrinks it to a minimal string, and the agent quotes that string in the pull request and says what it reveals rather than loosening the property to make it pass; whether the two implementations disagree on some input is something the run shows, not something the page can promise.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add property-based tests with Hypothesis to this pytest project. Read the package under src/, tests/conftest.py, every existing test file, pyproject.toml (dependency groups, pytest markers, mypy and tox settings), uv.lock and .gitignore first. Add hypothesis to the tests dependency group and re-lock with uv. Write tests/test_properties.py with properties you derive from the code: round trips, idempotence, return types, and agreement between any alternative implementations of the same function. Use st.text() and a second strategy over the characters the code treats specially. Keep the file passing mypy in strict mode, mark the tests with the project's thread_unsafe marker, and ignore the Hypothesis example database in .gitignore. Run the new tests in the sandbox. If a property fails, keep the property, quote the shrunk counterexample in the pull request and explain what it means; do not weaken the assertion. Do not change existing tests. Open a pull request whose description lists each property in one sentence and quotes the pytest output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that adds `tests/test_properties.py` and changes `pyproject.toml`, `uv.lock` and `.gitignore`. The description states each property in one sentence with the strategy that feeds it, quotes the pytest output for the new file, which shows every test twice, once per implementation, and names the Hypothesis version the lock pinned.

The run's record keeps the full pytest session, including Hypothesis's own statistics for each test and any shrunk counterexample, so a reviewer who doubts a property can read the generated inputs rather than the agent's summary of them.

Before merging, a reviewer checks that each property is a statement about the function's contract and not a restatement of its code, that the alphabet strategy still includes ordinary characters so it does not test only the special cases, and that the `thread_unsafe` marker is on every new test if the parallel environment runs in CI.

## Run it on every repository with a pytest suite

A team with several Python packages sends this body once per repository from a script, with the repository in `resources` and a `request_id` that is a UUID v5 of the repository's full name, so a retried call returns the run already started. `GET /o/{org}/github` lists the repositories the Rush GitHub App reaches; the script keeps the ones with a `tests/` directory and a pytest configuration.

Each run derives its own properties: a parser gets a round trip through its serialiser, a string function gets idempotence and type checks, a module with a C fallback gets a differential test. Give the key a webhook URL and Rush POSTs `run.completed` with `data.session_url` as each run ends, so a handler can collect the pull request links in one place.

## Why run it on Rush?

Writing a property test means reading the implementation closely enough to say what must always hold, then running it enough times to trust it, and the second part is what a developer skips at the end of a day. In a Rush sandbox the agent builds the package, including its C extension, runs Hypothesis to completion and reads any counterexample before it opens anything, with nothing installed on the developer's machine.

The result carries its own proof. The record keeps the pytest session with Hypothesis's statistics and any shrunk failing input, so the reviewer sees what was generated, not a claim that something was. Several packages run in parallel, each in its own sandbox with its own interpreter and lockfile.

## Frequently asked questions

### Why both st.text() and a custom alphabet?

`st.text()` covers the whole of Unicode, which matters for a function whose fast path is written in C and handles multi-byte characters, but it rarely produces the five characters `escape` replaces. A second strategy drawn from a short alphabet of those characters plus a few letters hits the replacements in most examples. Together they cover breadth and depth.

### Will the differential test still run if the C extension is missing?

The existing fixture skips the speedups parameter when the import fails, and the new differential test is written to skip the same way. In the sandbox `uv sync` builds the extension, so the run in the pull request exercises both paths.

### What happens when a property fails?

Hypothesis shrinks the failing input to a minimal example and reports it. The agent keeps the property, quotes the example in the pull request and explains what it means, because a failing property is usually a bug in the code or a misunderstanding of the contract, and either one is worth a reviewer's attention more than a green run.

### Does this slow the suite down?

Each property runs Hypothesis's default number of examples per test, twice for the two implementations. The pull request quotes the run time of the new file from the pytest output so the reviewer can set a lower `max_examples` in a settings profile if it matters.

### Will it change my existing tests?

No. The parametrised tables in `tests/test_escape.py` stay as they are; they document specific cases a reviewer can read at a glance, which a property test does not replace.

## More pytest jobs

- [Add Random Test Order to a pytest Suite](https://getrush.ai/run/add-random-test-order/pytest/)
- [Add Test Timeouts to a pytest Suite](https://getrush.ai/run/add-test-timeouts/pytest/)
- [Roll Back the Database After Each pytest Test in a FastAPI App](https://getrush.ai/run/add-transactional-test-fixtures/pytest/)

## The same job on other stacks

- [Add Property-Based Tests to an Elixir Library](https://getrush.ai/run/add-property-tests/elixir/)

## More jobs like this: Write the missing tests

- [Add Replay Tests to Temporal Workflows](https://getrush.ai/run/add-replay-tests/temporal/)
- [Add a Playwright Smoke Suite to a Next.js App](https://getrush.ai/run/add-smoke-tests/playwright/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
