All jobs

Fix "ValueError: invalid mode: 'rU'" on Python 3.11 and Later

An agent removes the U flag from every open() call, checks which of them were really codecs.open, and installs and tests the package on Python 3.11 and newer.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The U in open(path, 'rU') asked Python 2 for universal newlines: treat \r\n and \r as \n when reading. Python 3 does that for every text-mode file by default, so the flag became a no-op, was deprecated in 3.4 and was removed in 3.11. A call that still passes it raises ValueError: invalid mode: 'rU'. The fix is to delete one letter. Where the letter hides is the reason this error reaches users: it is most often in setup.py, in the few lines that read the README into long_description, so the failure appears during pip install and not in anything the package's tests exercise.

hosford42/xcs, a learning classifier system library, has it in exactly that place. setup.py reads README.rst with open(path.join(here, 'README.rst'), encoding='utf-8', mode='rU'). The file's imports matter here: it says from codecs import open, so this is codecs.open, which appends b to the mode when an encoding is given. On Python 3.11 the error a user sees is therefore ValueError: invalid mode: 'rUb', a spelling that appears nowhere in the source. Someone searching the repository for rUb finds nothing. The agent reads the imports before it reads the call, replaces the mode with 'r', and drops the codecs import in favour of the built-in open, which takes encoding directly in Python 3.

It then looks for the same flag elsewhere and finds a second use in package.py, the author's release script, which rewrites xcs.egg-info/PKG-INFO after building. That one uses the built-in open and would raise the plain 'rU' message. The agent fixes the mode there too and reports, without changing them, the other reasons that script cannot run on a clean machine: it calls os.chdir('.\\doc') with a Windows path separator and shells out to ipython nbconvert and python setup.py sdist bdist_wheel.

To prove the install works, the agent creates virtual environments on Python 3.11 and on the newest release, runs pip install . in each and then the unittest suite under tests/. It runs the same install on Python 3.10 first, to show the failure is new in 3.11 and that nothing else was already broken. setup.py also does from xcs import __version__, importing the package at build time, and its classifiers stop at Python 3.4. The agent leaves the import alone, proposes updated classifiers and a python_requires line, and lists them as suggestions.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "This project fails on Python 3.11 or later with \"ValueError: invalid mode: 'rU'\" (the message may show 'rUb' or 'Ur'). Find every call that opens a file with a U in its mode: open, io.open, codecs.open, fileinput.input, zipfile's ZipFile.open, and helper functions that pass a mode through. For each, read the file's imports first and say which open function it really is; codecs.open adds b to the mode when an encoding is given, so the error text can differ from the source text. Remove the U and keep the rest of the mode. Where the call is codecs.open with an encoding and nothing depends on codecs behaviour, switch to the built-in open with the same encoding. Do not add newline arguments unless the code compares raw line endings; if it does, explain. Check setup.py, setup helpers, release and packaging scripts, docs configuration and tests as well as the package. Then create environments on Python 3.10, Python 3.11 and the newest Python release. On each, run pip install . from a clean clone and run the test suite; show that 3.10 passes before and after and that 3.11 and newer fail before and pass after. If a file read this way contains Windows or old Mac line endings, show that the text read is the same before and after. Report anything else that fails on the newest Python with its traceback, without fixing it. If the packaging metadata lists only old Python versions, propose updated classifiers and a python_requires value. Open a pull request with the list of calls, the commands and their output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that lists each call with four facts: the file and line, which open it was after reading the imports, the mode before and the mode after. For xcs there are two rows. The description quotes the exact error text each produced, so the 'rUb' in a user's bug report can be matched to the setup.py line that caused it.

Below the list is a grid of results: three Python versions, before and after the change, for pip install . and for the tests. A reviewer checks that the only cells that changed are the ones expected, 3.11 and newer going from failing to passing at the install step. They then read the report section for things the newest interpreter rejects that this pull request did not touch, and the proposed metadata. Adding python_requires changes which versions pip will offer the package to, so the agent leaves that as a suggested diff in the description for the maintainer to apply.

Run it on every issue labelled python-3.11

Maintainers of several small packages tend to hear about this one through issues: a user pastes the pip install traceback and nothing else. Create an automation in the Project with the prompt above and a github trigger on issues.labeled, with python-3.11 in its labels. When a maintainer adds that label, Rush starts a run and puts the issue after the saved prompt, so the agent has the user's traceback and Python version next to its instructions. The saved prompt stays the only instruction; the issue text is treated as data.

The label step is deliberate. Triggers only fire for actions by an owner, member or collaborator, so it is the maintainer's labelling that starts the run, whoever wrote the issue. Set max runs per hour low enough that labelling a backlog of ten issues in one sitting queues the runs instead of starting them together.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

The evidence for this fix is a grid of interpreter versions, and each cell needs a clean environment, because a package that is already installed in development mode will import fine while pip install . fails. The agent builds a fresh virtual environment per version in its sandbox, installs from a clean clone into each and runs the tests. The transcript keeps every command with its output, including the failing ones from before the change, so the before column is real.

Install-time bugs are also the ones a maintainer's own machine hides best. A wheel cached by pip, a leftover egg-info directory or a README generated locally can each make the install succeed for the author and fail for everyone else. The sandbox has only what the repository contains, which is the same thing a new user has.

Frequently asked questions

What do I replace 'rU' with?

'r', or no mode at all, since 'r' is the default. Text mode in Python 3 already translates \r\n and \r to \n on reading, which is all the U flag ever asked for. The file's contents as read do not change.

My traceback says 'rUb' but I never wrote that. Where does it come from?

From codecs.open. When you pass it an encoding it opens the underlying file in binary and adds b to whatever mode you gave, so 'rU' becomes 'rUb'. Look for from codecs import open or codecs.open( in the file the traceback names.

Does removing U change how Windows line endings are handled?

No. Universal newline handling is the default behaviour of text mode, controlled by the newline argument, which defaults to translating all three endings to \n. The only code affected is code that passed newline='' on purpose to see raw endings, and such code was not using U in the first place.

The failing open() is inside a dependency's setup.py. What can the agent do?

It names the dependency and version from the build log and checks whether a newer release, or a prebuilt wheel, avoids running that setup.py at all. If so it raises your pin and reruns your tests. If not, it reports the upstream line that needs changing, since editing a dependency's source inside your repository is not a fix you can keep.

More Python jobs

More jobs like this: Fix a bug from an alert

Keep reading