All jobs

Fix "Fixtures are not meant to be called directly" in a pytest Suite

Yes. The agent moves the setup a test module calls directly into a plain function, keeps the fixture for tests that inject it, and opens a pull request.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

Failed: Fixture "red" called directly. Fixtures are not meant to be called directly is pytest refusing a call that pytest 3 allowed with a warning: a function decorated with @pytest.fixture invoked like any other function. Since pytest 4 the call is an error, and when it happens at module import time it fails collection of the whole file. The run starts with the failing CI log, and the agent clones the repository into its own cloud sandbox.

In a library like PyCQA/redbaron, requirements.txt pins pytest<4 for exactly this reason, while the README says pip install pytest and py.test tests, and .travis.yml installs pytest unpinned, so a fresh environment gets a current release. tests/test_bounding_box.py defines the fixture red at line 11, returning a parsed RedBaron tree, then calls fst = red() at line 20 at module level to build bounding_boxes, a table of twenty-seven expected boxes over nodes of that tree. A second fixture, bounding_box_fixture, is parametrised over the table at line 51, and test_bounding_box at line 55 uses it. The direct call is the only one in the suite.

The agent reproduces it with pytest 9.2.0: collection of tests/test_bounding_box.py errors with the message and the link to pytest's deprecation page, and with that file ignored the rest of the suite runs: 1430 passed and one unrelated failure in tests/test_no_pygments.py, an assertion on terminal colour escapes. It then separates the two uses of red. The parsing moves into a plain function, make_red(), which the module-level table calls; the fixture red stays and returns make_red(), so any test that injects red is unchanged.

With that change the file collects and its parametrised test runs on pytest 9. The agent leaves the pytest<4 pin to the maintainer's choice but says in the pull request that nothing else in the suite needs it, and notes the test_no_pygments failure as separate so it is not mistaken for part of this change.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "CI fails with 'Fixture \"red\" called directly. Fixtures are not meant to be called directly' on a current pytest. Reproduce it, find every place a fixture function is called as a plain function, move the shared setup into an ordinary function that both the fixture and the direct caller use, run the whole suite on current pytest, report any failures that are unrelated, and open a pull request that quotes the error and explains the change.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request with a small change to tests/test_bounding_box.py: a plain make_red() function, the fixture returning it, and the module-level table built from the plain function. Its description quotes the collection error, shows the file collecting on pytest 9.2.0, and gives the suite's result with the unrelated failure named.

A reviewer checks that the table is built from the same source as before, that no test's behaviour changed (the fixture still yields a fresh tree per test), and that the pin in requirements.txt is either lifted in the same pull request or left with a sentence saying why. The one other failure is listed as outside this change.

The session keeps both pytest runs, before and after, so the reviewer can compare the collection error with the passing collection without installing anything.

Run it on every failed CI run

Create an automation in the Project with this prompt and a github trigger on workflow_run.failure. Each failed workflow run starts one run with the event after the prompt; the agent reads the job log, reproduces the failure on the pinned and the current pytest, and opens the pull request.

Repositories that still run on Travis or another CI can send the failure from a handler on your server to a webhook trigger's URL with Authorization: Bearer <secret>; the body becomes the event text the run reads.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

The error depends on which pytest a fresh install resolves, so it reproduces on a new laptop and not on an old one. A Rush run installs the pinned version and the current one in its own sandbox, keeps both outputs, and fixes the test file against the current behaviour, so the pull request shows the before and the after rather than a description of them.

A suite of more than a thousand tests takes seconds to run but minutes to read when it fails at collection. The run isolates the one direct call, confirms the rest of the suite passes, and reports the unrelated failure separately, so review is a diff of a dozen lines with its evidence attached.

Frequently asked questions

Why not drop the fixture and use the plain function everywhere?

Because tests may inject red by name and rely on a fresh tree each time. Keeping the fixture as a thin wrapper over make_red() changes no test. If a grep shows no test injects it, the agent removes the fixture and says so.

Will this work on the pinned pytest 3 as well?

Yes. A plain function called at module level is ordinary Python on every pytest release, so the file collects on pytest<4 and on 9.x alike. The pull request runs the suite on both and reports each.

My tests do not call fixtures. Where else does this error come from?

Module-level code is the usual place: building a table of cases, a cached object, or a conftest.py helper that another helper calls. The agent greps for @pytest.fixture names used as calls anywhere under tests/ and conftest.py, not only inside test functions.

Should the pytest<4 pin go?

After this change nothing in the suite needs it, and the pull request says so. Lifting it is a one-line edit the agent includes when the prompt asks for it; otherwise it stays, documented, for the maintainer to decide.

More pytest jobs

More jobs like this: Fix a bug from an alert

Keep reading