All jobs

Add Property-Based Tests with Hypothesis to a pytest Suite

Yes. An agent adds Hypothesis to a pytest project, writes property tests for the invariants it finds in the code, runs them in its sandbox and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the code for invariants before it writes a test, because a property test is only as good as the property. In a library like pallets/markupsafe, src/markupsafe/__init__.py says that escape replaces &, <, >, ' and ", returns a Markup, and calls __html__ instead when the object has one; Markup.unescape reverses entities with html.unescape; striptags removes tags and comments and collapses whitespace; and the actual replacement lives in two implementations, _native.py and the C extension in _speedups.c. tests/conftest.py has a session-scoped autouse fixture parametrised over both modules that swaps markupsafe._escape_inner, so every existing test already runs twice. tests/test_escape.py is a parametrize table of hand-picked strings: empty, ASCII with the five characters at the start, middle and end, the same for two-byte and four-byte characters. That table is exactly the shape a property test generalises.

From that reading it writes tests/test_properties.py with four properties. For any text, Markup(escape(s)).unescape() gives back s, since every replacement is an entity that html.unescape knows. Escaping is idempotent: escape(escape(s)) == escape(s), because the result carries __html__ and is returned as is. The result is always a Markup, checked with type(...) is Markup, the way the existing test_type_behavior checks it. And the two implementations agree: _native._escape_inner(s) == _speedups._escape_inner(s) for any text, called directly rather than through the fixture, which is the property that matters most for a C extension. Strings come from st.text(), plus a second strategy drawn from a short alphabet of the five special characters and a few letters so that examples hit the replacements often instead of rarely.

The project's own settings shape the details. hypothesis goes into the tests dependency group in pyproject.toml and uv.lock is regenerated, so uv run --locked tox run finds it. mypy runs in strict mode over tests as well as src, so the new file is fully typed. pytest-run-parallel runs the suite with threads on free-threaded Python and pyproject.toml already declares a thread_unsafe marker for tests that must not; Hypothesis keeps per-test state, so the agent marks the new tests with it. Hypothesis writes its example database to .hypothesis/, which the agent adds to .gitignore.

Then it runs uv sync and uv run pytest tests/test_properties.py -v in its sandbox, where uv sync compiles _speedups so the differential test exercises the real extension. Each test runs twice because of the fixture. If Hypothesis finds a counterexample it shrinks it to a minimal string, and the agent quotes that string in the pull request and says what it reveals rather than loosening the property to make it pass; whether the two implementations disagree on some input is something the run shows, not something the page can promise.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add property-based tests with Hypothesis to this pytest project. Read the package under src/, tests/conftest.py, every existing test file, pyproject.toml (dependency groups, pytest markers, mypy and tox settings), uv.lock and .gitignore first. Add hypothesis to the tests dependency group and re-lock with uv. Write tests/test_properties.py with properties you derive from the code: round trips, idempotence, return types, and agreement between any alternative implementations of the same function. Use st.text() and a second strategy over the characters the code treats specially. Keep the file passing mypy in strict mode, mark the tests with the project's thread_unsafe marker, and ignore the Hypothesis example database in .gitignore. Run the new tests in the sandbox. If a property fails, keep the property, quote the shrunk counterexample in the pull request and explain what it means; do not weaken the assertion. Do not change existing tests. Open a pull request whose description lists each property in one sentence and quotes the pytest output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds tests/test_properties.py and changes pyproject.toml, uv.lock and .gitignore. The description states each property in one sentence with the strategy that feeds it, quotes the pytest output for the new file, which shows every test twice, once per implementation, and names the Hypothesis version the lock pinned.

The run's record keeps the full pytest session, including Hypothesis's own statistics for each test and any shrunk counterexample, so a reviewer who doubts a property can read the generated inputs rather than the agent's summary of them.

Before merging, a reviewer checks that each property is a statement about the function's contract and not a restatement of its code, that the alphabet strategy still includes ordinary characters so it does not test only the special cases, and that the thread_unsafe marker is on every new test if the parallel environment runs in CI.

Run it on every repository with a pytest suite

A team with several Python packages sends this body once per repository from a script, with the repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started. GET /o/{org}/github lists the repositories the Rush GitHub App reaches; the script keeps the ones with a tests/ directory and a pytest configuration.

Each run derives its own properties: a parser gets a round trip through its serialiser, a string function gets idempotence and type checks, a module with a C fallback gets a differential test. Give the key a webhook URL and Rush POSTs run.completed with data.session_url as each run ends, so a handler can collect the pull request links in one place.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

Writing a property test means reading the implementation closely enough to say what must always hold, then running it enough times to trust it, and the second part is what a developer skips at the end of a day. In a Rush sandbox the agent builds the package, including its C extension, runs Hypothesis to completion and reads any counterexample before it opens anything, with nothing installed on the developer's machine.

The result carries its own proof. The record keeps the pytest session with Hypothesis's statistics and any shrunk failing input, so the reviewer sees what was generated, not a claim that something was. Several packages run in parallel, each in its own sandbox with its own interpreter and lockfile.

Frequently asked questions

Why both st.text() and a custom alphabet?

st.text() covers the whole of Unicode, which matters for a function whose fast path is written in C and handles multi-byte characters, but it rarely produces the five characters escape replaces. A second strategy drawn from a short alphabet of those characters plus a few letters hits the replacements in most examples. Together they cover breadth and depth.

Will the differential test still run if the C extension is missing?

The existing fixture skips the speedups parameter when the import fails, and the new differential test is written to skip the same way. In the sandbox uv sync builds the extension, so the run in the pull request exercises both paths.

What happens when a property fails?

Hypothesis shrinks the failing input to a minimal example and reports it. The agent keeps the property, quotes the example in the pull request and explains what it means, because a failing property is usually a bug in the code or a misunderstanding of the contract, and either one is worth a reviewer's attention more than a green run.

Does this slow the suite down?

Each property runs Hypothesis's default number of examples per test, twice for the two implementations. The pull request quotes the run time of the new file from the pytest output so the reviewer can set a lower max_examples in a settings profile if it matters.

Will it change my existing tests?

No. The parametrised tables in tests/test_escape.py stay as they are; they document specific cases a reviewer can read at a glance, which a property test does not replace.

More pytest jobs

The same job on other stacks

More jobs like this: Write the missing tests

Keep reading