All jobs

Write Property-Based Tests for a Python Library

Yes. An agent adds Hypothesis properties for core parsers, runs a high example count, and opens a PR without changing library code on failure.

Run this on RushIn a Project that has your repository.

What the agent does

The agent targets pure parsing and transformation helpers with clear invariants. In encode/httpx that means modules such as httpx/_urlparse.py (RFC 3986 parsing, percent-encoding sets, encode_host, MAX_URL_LENGTH = 65536) and httpx/_urls.py (URL and QueryParams, including QueryParams.merge and string round-trips). tests/models/test_url.py already holds example-based URL cases; there is no Hypothesis usage in the tree today (requirements.txt pins pytest and friends but not hypothesis).

It adds a dev dependency on Hypothesis (in the project's usual place: requirements.txt test extras or a tests optional extra if one exists) and new tests under tests/ that use @given strategies for URL strings, query key/value maps and path segments. Properties include: parsing then str(URL(...)) round-trips for valid inputs; QueryParams build/merge idempotence where the library guarantees it; invalid inputs raising httpx.InvalidURL rather than hanging; and encoding helpers being stable under repeated application when documented as such.

Runs use a high Hypothesis example count (pytest --hypothesis-seed=... and settings profiles with elevated max_examples) so edge cases surface in the sandbox. If a property fails, the agent keeps the shrinking counterexample in the test file or pull request body and does not patch httpx/ production modules to make the property pass.

It runs the focused pytest path for the new module after installing requirements, iterates only on flaky strategies or over-tight invariants, and opens a pull request that adds test code plus the Hypothesis dependency pin. For QueryParams, properties exercise construction from dicts, lists of pairs and raw query strings, then compare str(QueryParams(...)) with parsing that string back when the library treats them as equivalent. For hosts, strategies include IDNA labels and IPv6 literals in brackets, matching behaviours already illustrated in the URL docstring at the top of _urls.py, while staying inside MAX_URL_LENGTH.

Start it with one API call

POST https://api.prix.dev/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Write property-based tests with Hypothesis for the core parsing and transformation functions of this library: round trips, idempotence and invariants. Run them with a high example count. Open a pull request; if a property fails, include the shrunk counterexample and do not change the library.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-python-lib"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request adding Hypothesis tests (for httpx, typically beside tests/models/ or under tests/properties/) and a pinned hypothesis in the test requirements. Library sources under httpx/ are unchanged when a counterexample appears; the failing property and its shrunk input are documented for maintainers.

The pull request description states the pytest command, max_examples setting, and whether every property passed or which one found a counterexample. The trajectory includes Hypothesis reports (Falsifying example: ...) when a failure is left intentional.

Before merging, a reviewer confirms strategies only generate inputs in the documented domain, that failing properties are intentional bug reports rather than flaky tests marked xfail without discussion, and that CI installs the new Hypothesis pin.

Run it on every URL or query parsing code changed

On pulls that touch httpx/_urlparse.py or httpx/_urls.py (or your library's equivalent parsers), POST this session request so properties re-run at high max_examples in a sandbox before merge.

Pipe run.completed / run.failed webhooks to the library channel with the session URL. A found counterexample is a signal to open a tracking issue, not to silently weaken the property in an automated follow-up.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

Hypothesis at high example counts burns CPU exploring encodings and Unicode hosts. Rush runs that load in a sandbox so your laptop fans are not the test runner, and the agent can re-seed and re-run overnight-quality settings without blocking you. Existing pytest markers in pyproject.toml (including network) stay respected: property tests do not mark themselves as network tests and do not call the public internet.

Leaving production code untouched when a property fails is the point: you get a minimal counterexample in a pull request, with the full pytest log in the trajectory, ready for a human decision.

Frequently asked questions

Will it "fix" httpx if Hypothesis finds a bug?

No. The prompt forbids changing the library when a property fails. The shrunk counterexample stays in the pull request for maintainers to triage.

Does it replace the existing URL unit tests?

No. Example-based tests in tests/models/test_url.py remain. Properties add random exploration on top.

Which Python versions?

Those requires-python and classifiers name. httpx declares >=3.9 through 3.13; the agent uses a version matching CI or pyproject.toml.

How high is the example count?

Higher than Hypothesis defaults (hundreds to thousands of examples per property, set via @settings(max_examples=...) or a profile). The exact number is written in the pull request so CI can match it.