All jobs

Fix "AttributeError: 'HTMLParser' object has no attribute 'unescape'" in Python 3.9+

Yes. An agent replaces HTMLParser().unescape with html.unescape, keeps a Python 2 fallback where one exists, and tests the path that reaches it.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

HTMLParser.unescape turned entities such as & and < back into characters. It was deprecated in favour of the html.unescape function and removed in Python 3.9. Unlike a removed module, a removed method does not fail on import. It fails only when the line runs, and that can be a line the tests never reach. The w3c_validate plugin in getpelican/pelican-plugins is an exact case. In w3c_validate/wc3_validate.py, validate() creates a parser on line 39 with the comment that it is for unescaping the validator's messages, and calls h.unescape(err['message']) on lines 52 and 58, once inside the loop over errors and once inside the loop over warnings.

The agent works out when those lines execute before running anything. Both sit inside loops over results from the W3C validator, so a page with no errors and no warnings never touches them. The plugin's single test, test_validate_ok in test_w3c_validate.py, validates test_content/getpelican.html and asserts one INFO line, which means it only ever exercises the clean path, and it needs the live validation service to do so. On a current Python the plugin therefore works on every valid page and raises the error in the title on the first page that has something to report. The failure is confined to the moment the plugin has a result to show.

To reproduce that without the network, the agent replaces the two outside pieces with stubs: pelican.signals, which the module imports at the top, and HTMLValidator from py_w3c, which validate() imports and calls. With a stub validator that returns one error whose message contains " and <, Python 3.8 logs the unescaped message and prints DeprecationWarning: The unescape method is deprecated and will be removed in 3.5, use html.unescape() instead. A current Python raises AttributeError. The same happens for a result with one warning, and a result with neither passes on both.

The file already carries a Python 2 fallback, importing HTMLParser from html.parser and from the old HTMLParser module on ImportError, so the agent keeps that shape. The import becomes from html import unescape, with the fallback branch binding unescape to the old parser method, and the two call sites call unescape(err['message']). The line that built the parser goes. It turns the stubbed reproduction into three tests, for an error, a warning and a clean page, runs them on Python 3.8 and on current versions, and compares the logged text on each.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "This project fails on Python 3.9 or later with \"AttributeError: 'HTMLParser' object has no attribute 'unescape'\". Find every call to the unescape method of an HTMLParser instance, however the class is imported (html.parser, the Python 2 HTMLParser module, six.moves.html_parser, or a compatibility module of our own). For each call, work out when it executes: which function, under which condition, and whether any existing test reaches it. Say so explicitly if the current tests only cover a path that skips the call, or need a network service to run. Reproduce the failure without network access by stubbing external services and returning data that contains HTML entities. Run the reproduction on Python 3.8, where the method still exists, and save what it outputs. Then replace the method with html.unescape. If the file already has a Python 2 import fallback, keep one: import unescape from html, and in the ImportError branch bind the same name to the old parser's method. If there is no such fallback, use html.unescape directly. Remove parser objects that were only created for unescaping. Do not change how many times a string is unescaped. Turn the reproduction into tests for each branch that reaches a call site and one that does not, and run them on Python 3.8 and a current Python, comparing the output with what you saved. Report other failures you see on the current Python as follow-ups. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that changes one function and adds tests. In the plugin, the import block inside validate() is rewritten, one line that constructed a parser is deleted, and two calls lose their h. prefix. The description shows the three cases on each interpreter before the change, clean page, one error and one warning, and the same three afterwards, with the logged message text identical to what Python 3.8 produced.

It says plainly what the existing test did and did not prove. A reviewer reading "tests pass on Python 3.12" could conclude the plugin was fine, so the description states that the only test covers a page with no findings, which never reaches the removed method, and that the new tests use a stub in place of the validation service. It does not claim a request to the real service was made.

The reviewer checks that the stub returns results in the same shape validate() reads, including the lastLine, firstColumn and lastColumn keys the function falls back to, and that reverting the import makes two of the three new tests fail. If the repository has dropped Python 2 everywhere else, they can ask for the fallback branch to be removed in a follow-up message to the same session.

Run it on every Linear issue moved to Todo

Compatibility fixes are small enough to plan and carry out straight from the tracker. Connect Linear in the console under Settings → Connections, then add an automation to the repository's Project with the prompt above and a linear trigger on linear.issue.state_changed. Its states setting maps a state name to plan or exec. Map your triage state to plan: moving an issue there starts a read-only run that posts its plan as a comment on the issue, which for this job is the list of call sites and when each one executes. Map your in-progress state to exec, and moving the issue again starts an ordinary run that opens the pull request.

The issue is added after the saved prompt as event text, and the prompt remains the only instruction. The trigger fires only when the person who moved the issue is in your Rush org, and changes made by an app integration never fire it, so a sync tool shuffling states does not start work. Use teams to restrict it to the team that owns the code.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

A bug that only appears when an external service returns bad news is awkward to reproduce by hand. You need the old interpreter for a baseline, the new one for the failure, and a stand-in for the service that returns the right shape of data. The agent builds all three in its sandbox, where installing a second Python and writing a throwaway stub costs nothing, and it keeps the commands and their output in the session. The pull request arrives with the failing case demonstrated, not described.

It is also the kind of fix where passing tests mislead. The plugin's own test exercises nothing that broke, so it cannot fail for this reason on a current Python. An agent told to find when each call executes reports that gap first, and the result includes tests for the paths that matter, so the next removed method in the same loop is caught by CI and not by a user whose site has a validation error.

Frequently asked questions

Is html.unescape identical to HTMLParser().unescape?

On Python 3 versions that had both, the results match for this plugin: the same message with ", <, > and & logs the same text through either on Python 3.8. Both handle named, decimal and hexadecimal character references. The function needs no parser object, which is why the line that built one can go.

The warning says the method will be removed in 3.5. Why did it work until 3.8?

The message names a release that came and went. The method kept working with that warning for several more versions and was removed in Python 3.9. A project whose CI stopped at 3.7, as this repository's .travis.yml does, never saw the failure.

Why did my tests pass on the new Python if the code is broken?

Because nothing called the method. Removing a method changes nothing at import time, and here the calls sit inside loops that run only when the validator returns an error or a warning. A test that feeds the function a clean page finishes without touching them.

Do I need the Python 2 fallback?

Only if the code must still run there, since html.unescape does not exist on Python 2. The agent keeps a fallback when the file already has one and leaves it out otherwise. Saying in the prompt that Python 2 support is over gets you the shorter version.

More jobs like this: Turn a Linear label into a PR

Keep reading