All jobs

Fix "ImportError: lxml.html.clean module is now a separate project lxml_html_clean"

Yes. An agent adds the lxml_html_clean package the import now needs, confirms the Cleaner imports on the lxml a fresh install resolves, and reports the next break.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

lxml 5.2.0, released on 31 March 2024, moved lxml.html.clean out of the main package into lxml_html_clean, a separate project first published a month earlier. On 5.1.1 the module is still a compiled file inside the wheel, lxml/html/clean.cpython-311-x86_64-linux-gnu.so. On 5.2.0 and every release since, including the current 6.1.3, from lxml.html.clean import Cleaner raises ImportError: lxml.html.clean module is now a separate project lxml_html_clean. Install lxml[html-clean] or lxml_html_clean directly. The message names the fix, but the error lands on code that has not changed in years, because lxml without a version in a requirements file now resolves to 6.x.

NikolaiT/GoogleScraper, a scraper for search engine result pages with nearly three thousand stars, shows the usual shape. Its requirements.txt is ten unpinned names, asyncio, lxml, selenium, cssselect, requests, PyMySql, sqlalchemy, aiohttp, fake_useragent and recommonmark, and setup.py reads that file into install_requires. GoogleScraper/parsing.py opens with import lxml.html and from lxml.html.clean import Cleaner on lines 6 and 7; _parse_lxml calls cleaner.clean_html(self.dom) before lxml.html.document_fromstring. The same import is at line 7 of parsing-new-version.py and, lazily, at line 464 of caching.py. A clean install on Python 3.11 resolves lxml 6.1.3, and import GoogleScraper.parsing fails on line 7 with the message above.

The agent reproduces that on a clean environment and reads which of the three forms of the fix fits the repository. lxml[html_clean] keeps one line in requirements.txt and lets pip pull the companion package; lxml_html_clean as its own line is explicit and shows up in a freeze; a pin lxml<5.2 keeps the old module and loses every lxml release since March 2024, which the request below rules out. It adds the package, reinstalls, and confirms that from lxml.html.clean import Cleaner now resolves to the shim lxml keeps for that import.

Then it imports the module again, and reports what happens next rather than stopping at a green import line. In GoogleScraper's case import GoogleScraper.parsing gets past line 7 and fails in output_converter.py line 19, where Link.__table__.columns._data.keys() reaches into a private SQLAlchemy attribute that the SQLAlchemy 2.1.8 a fresh install resolves no longer has (AttributeError: _data). That is a second unpinned dependency and a different change, so the agent names the file and line in the pull request as the next break and leaves it out of this one.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "This repository fails at import with \"ImportError: lxml.html.clean module is now a separate project lxml_html_clean\" after lxml was installed fresh. Read the dependency files and report how lxml is pinned, then create a clean virtual environment on a current Python, install the dependencies as a new user would, and report the lxml version that was resolved. Import the failing module and save the traceback. Search every .py file for lxml.html.clean and list each import with its file and line, including imports inside functions. Add the lxml_html_clean package to the dependency file, either as the lxml[html_clean] extra or as its own line, and say which you chose and why. Do not pin lxml below 5.2 to keep the old module. Reinstall, import the module again, and run the repository's tests or entry point if it has one. If the import now fails on a different library's removed API, report that traceback with its file and line as a separate finding and do not fix it in this change. Open a pull request with the resolved versions, the list of imports found, and the import result after the change.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that adds one dependency line and changes no Python. The description lists the lxml version a clean install resolved, the traceback before, every lxml.html.clean import in the repository with its file and line, and the form of the fix chosen with its reason: the lxml[html_clean] extra, or lxml_html_clean as its own requirement.

What a reviewer checks is that the fix is the companion package and not a pin. A lxml<5.2 line would also make the import work and would silently hold the parser on a release from March 2024; the description shows the pip freeze line for lxml-html-clean so the reviewer can see which path was taken.

The description also states how far the import got afterwards. For GoogleScraper that is output_converter.py line 19 and SQLAlchemy 2's removal of a private attribute, named as the next change with the traceback, so the reviewer merges the lxml fix knowing the package still does not import and exactly why.

Run it on every Monday morning

An unpinned lxml line breaks on an install day, not a commit day. Give the Project an automation with a schedule trigger, for example the cron expression 0 7 * * 1 in your timezone, and a prompt that tells the agent to install the dependencies fresh, import every module of the package, and open a pull request only when an import fails on a name a new release moved or removed. On a week when nothing broke, the run ends with a one-line answer and no pull request.

For a set of scrapers and parsers that share this lxml shape, start the request above once per repository from a script with an API key, and read each session's pr_url when its run.completed webhook arrives. Each run resolves its own lxml in its own sandbox, and runs past the key's max_concurrent limit are accepted and wait as queued.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

The dependency line takes a minute. What a fresh install also reveals is every other unpinned name that moved in the same years, and the only way to see the next one is to clear the first and import again. An agent in a sandbox does that in sequence and leaves each traceback in the session, so the pull request fixes one library and names the next instead of pretending the package starts.

It also starts from the state a new user is in. A machine that has had lxml 4.9 installed since the scraper was written never shows this error. A sandbox resolves what pip install lxml gives today, which is the environment the next clone will have.

Frequently asked questions

Which should I add, lxml[html_clean] or lxml_html_clean?

They install the same package. The extra keeps a single lxml line and ties the companion to it; the explicit line shows in pip freeze and in lockfiles as its own entry. The agent picks one, says why in the pull request, and either way the import resolves through the shim lxml keeps for lxml.html.clean.

Can I pin lxml below 5.2 instead?

5.1.1 still ships the module as a compiled file inside the wheel, so a pin works for now. It also holds the parser on a release from March 2024 for every future Python and security fix. The request above tells the agent not to pin; the fix is one line.

Why does the module still fail to import after the fix?

Because an unpinned requirements file moves every dependency at once. In the fixture the next failure is in output_converter.py, where SQLAlchemy 2 removed a private attribute the code reached into. The agent reports that with the file and line as a separate finding, since it is a different library and a code change rather than a dependency line.

Does the agent change the Cleaner calls?

No. Cleaner and clean_html keep their names and arguments in lxml_html_clean, and the old import path keeps working once the package is installed, so parsing.py is untouched. The pull request changes only the dependency file.

What about imports inside functions?

They fail only when the function runs, so a green import line does not prove them. The agent searches for lxml.html.clean across every file, including the lazy import at line 464 of caching.py in the fixture, and lists each one.

More jobs like this: Fix a bug from an alert

Keep reading