All jobs

Fix "ModuleNotFoundError: No module named 'pipes'" on Python 3.13

Yes. An agent swaps every pipes.quote call for shlex.quote, keeps older interpreters importing, and reports the next removed module waiting behind this one.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

Python 3.13 removed the pipes module, and almost every project that still imports it uses one function from it. The agent starts by proving that. In Yelp/mrjob there are five imports: line 23 of mrjob/util.py, line 23 of mrjob/bin.py, line 19 of mrjob/cloud.py, and the two test doubles tests/mockssh.py and tests/mockhadoop.py. Every use is pipes.quote(...). The one in mrjob/util.py sits inside cmd_line(), which mrjob/fs/ssh.py, mrjob/fs/hadoop.py, mrjob/bin.py and mrjob/local.py all import, so on 3.13 import mrjob.util stops at line 23 with the error in the title and nothing else in the package gets a chance to run.

The replacement is not a judgment call about behaviour. On Python 3.12, pipes.quote is shlex.quote evaluates to True: the removed name was an alias for the function that stays. The bootstrap script that mrjob/cloud.py assembles, where each uploaded URI and file name is quoted into a copy line and a chmod u+rx line, comes out character for character the same. What does need a decision is the range of interpreters. This project's setup.py still carries classifiers for Python 2.7 and 3.4 to 3.7, .travis.yml lists 2.7 and two PyPy builds, and there is a mrjob/py2.py module of version shims. shlex.quote arrived in Python 3.3, so a bare from shlex import quote ends 2.7 support in passing. The agent does not make that call silently. It puts one guarded import where the project keeps its other shims, points the five files at it, and says in the pull request that deleting the fallback is a one-line follow-up once 2.7 is formally dropped.

Then it looks for what the first error was hiding. Line 31 of the same file is from distutils.spawn import find_executable, and mrjob/compat.py imports LooseVersion from distutils.version on its line 20. distutils left the standard library in 3.12. On a 3.12 interpreter with no setuptools installed, import mrjob.util fails there with No module named 'distutils'; with setuptools installed the import succeeds and prints DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13. On 3.13 the pipes line fails first either way. So once pipes is fixed, a 3.13 environment without setuptools moves straight to the distutils error eight lines further down. The agent runs the import in four environments, 3.12 and 3.13 each with and without setuptools, and puts the table in the pull request so nobody reads a green result from one of them as the whole answer.

For proof it uses the command the project's CI file names, python -m unittest discover -v, starting with tests.test_util, and not the Makefile target, which calls python setup.py test, a command current setuptools refuses to run. No existing test calls cmd_line() directly, so the agent adds one that quotes an argument containing a space and one containing a single quote and compares the result with the string the old alias produced on 3.12.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "On Python 3.13 this project fails with \"ModuleNotFoundError: No module named 'pipes'\". List every import of the pipes module in the package and in the tests, and every attribute used from it. If the only use is pipes.quote, replace it with shlex.quote and confirm on Python 3.12 that the two names are the same function. Before choosing the import style, read setup.py, the CI file and any compatibility module to see which Python versions the project still claims; if Python 2.7 is among them, add one guarded import in the existing compatibility module and use it everywhere instead of dropping 2.7 silently, and say so. Then import the package's main modules on Python 3.12 and 3.13, each with and without setuptools installed, and report the result of all four as a table. If another removed standard-library module fails next, name the file and line and report it; fix it only if the change is as mechanical as this one, otherwise leave it for a separate pull request. Add a unit test for any shell-quoting helper that has none, covering an argument with a space and one with a single quote. Run the tests with the command the CI file uses, not with setup.py test. Open a pull request with the import table, the test results before and after, and what you left alone.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that changes the import lines in three library modules and two test doubles, adds the shim if older interpreters are still claimed, and adds a quoting test. The description opens with the four-row import table: which interpreter, whether setuptools was present, and the first error or none. On mrjob the tests.test_util module runs 30 tests on 3.13 once pipes is replaced and setuptools is installed, three of them skipped, and the pull request quotes that output beside the 3.12 run.

A reviewer checks three things. The quoting helper returns the same strings as before, which the new test pins. The choice about Python 2.7 is stated and matches what the maintainers want. And the distutils imports are either fixed in a clearly separate commit or listed as the next step, so merging this does not get reported as full 3.13 support when a clean 3.13 install still cannot import the package.

Run it on every failed CI run on a new Python version

The usual way this error arrives is a CI matrix that gains a 3.13 entry, or a base image that moves. If the project's checks run on GitHub Actions, add an automation in the Project with this prompt and a github trigger on workflow_run.failure. Rush places the failed run's event after the saved prompt as third-party data; it fills in nothing inside the prompt, so the prompt itself should say to read the failing job's log and act only when the first error is a missing standard-library module.

Give the automation a low max runs per hour. A matrix with several jobs can fail several times for the same import, and one pull request fixes all of them. Runs past the hourly number are created queued and wait until the hour has room. At most one more hour's worth waits and an event past that is refused, which costs nothing here, since each repeat reports the same import.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

The interesting result here is a table of four environments, and a laptop has one of them. Most development machines have setuptools somewhere on the path, which is exactly the condition that hides the second failure. In its own sandbox the agent builds each interpreter and each package set from nothing, runs the same import in all four, and the commands and their output are kept with the session, so the reviewer can see that the row marked clean really had no setuptools in it.

It is also a change that is easy to do halfway. A search and replace in the editor fixes the three library files and misses the two under tests, or fixes all five and quietly ends support for an interpreter the metadata still advertises. The run reads the packaging files before it edits, states the decision it took, and returns a pull request whose description can be checked line by line against the transcript.

Frequently asked questions

What replaces pipes.quote in Python 3.13?

shlex.quote. It is the same function: on Python 3.12, pipes.quote is shlex.quote is True. Code that used pipes.Template to build shell pipelines has no drop-in replacement and needs subprocess instead, which is a different and larger change.

Will the quoted command lines change?

No. Because the old name was an alias, the output is identical for every input. The agent still adds a test with a space and a single quote in the argument so the behaviour is pinned for the next refactor.

Why does it still fail on 3.13 after the fix?

Often because another removed module is imported a few lines later. In mrjob that is distutils, at line 31 of mrjob/util.py. It only shows on an interpreter without setuptools, so the agent tests with and without it and reports both.

Does this drop Python 2 support?

Only if you ask for that. shlex.quote does not exist on 2.7, so when the project's metadata still lists it the agent adds a guarded import that falls back to the old module. Removing the fallback later is one line.

More jobs like this: Fix a bug from an alert

Keep reading