# Fix "ImportError: lxml.html.clean module is now a separate project lxml_html_clean"

Yes. An agent adds the lxml_html_clean package the import now needs, confirms the Cleaner imports on the lxml a fresh install resolves, and reports the next break.

[Run this on Rush](https://cloud.getrush.ai/run/fix-lxml-html-clean-separate-project-import-error/lxml/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

lxml 5.2.0, released on 31 March 2024, moved `lxml.html.clean` out of the main package into `lxml_html_clean`, a separate project first published a month earlier. On 5.1.1 the module is still a compiled file inside the wheel, `lxml/html/clean.cpython-311-x86_64-linux-gnu.so`. On 5.2.0 and every release since, including the current 6.1.3, `from lxml.html.clean import Cleaner` raises `ImportError: lxml.html.clean module is now a separate project lxml_html_clean. Install lxml[html-clean] or lxml_html_clean directly.` The message names the fix, but the error lands on code that has not changed in years, because `lxml` without a version in a requirements file now resolves to 6.x.

NikolaiT/GoogleScraper, a scraper for search engine result pages with nearly three thousand stars, shows the usual shape. Its `requirements.txt` is ten unpinned names, `asyncio`, `lxml`, `selenium`, `cssselect`, `requests`, `PyMySql`, `sqlalchemy`, `aiohttp`, `fake_useragent` and `recommonmark`, and `setup.py` reads that file into `install_requires`. `GoogleScraper/parsing.py` opens with `import lxml.html` and `from lxml.html.clean import Cleaner` on lines 6 and 7; `_parse_lxml` calls `cleaner.clean_html(self.dom)` before `lxml.html.document_fromstring`. The same import is at line 7 of `parsing-new-version.py` and, lazily, at line 464 of `caching.py`. A clean install on Python 3.11 resolves lxml 6.1.3, and `import GoogleScraper.parsing` fails on line 7 with the message above.

The agent reproduces that on a clean environment and reads which of the three forms of the fix fits the repository. `lxml[html_clean]` keeps one line in `requirements.txt` and lets pip pull the companion package; `lxml_html_clean` as its own line is explicit and shows up in a freeze; a pin `lxml<5.2` keeps the old module and loses every lxml release since March 2024, which the request below rules out. It adds the package, reinstalls, and confirms that `from lxml.html.clean import Cleaner` now resolves to the shim lxml keeps for that import.

Then it imports the module again, and reports what happens next rather than stopping at a green import line. In GoogleScraper's case `import GoogleScraper.parsing` gets past line 7 and fails in `output_converter.py` line 19, where `Link.__table__.columns._data.keys()` reaches into a private SQLAlchemy attribute that the SQLAlchemy 2.1.8 a fresh install resolves no longer has (`AttributeError: _data`). That is a second unpinned dependency and a different change, so the agent names the file and line in the pull request as the next break and leaves it out of this one.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "This repository fails at import with \"ImportError: lxml.html.clean module is now a separate project lxml_html_clean\" after lxml was installed fresh. Read the dependency files and report how lxml is pinned, then create a clean virtual environment on a current Python, install the dependencies as a new user would, and report the lxml version that was resolved. Import the failing module and save the traceback. Search every .py file for lxml.html.clean and list each import with its file and line, including imports inside functions. Add the lxml_html_clean package to the dependency file, either as the lxml[html_clean] extra or as its own line, and say which you chose and why. Do not pin lxml below 5.2 to keep the old module. Reinstall, import the module again, and run the repository's tests or entry point if it has one. If the import now fails on a different library's removed API, report that traceback with its file and line as a separate finding and do not fix it in this change. Open a pull request with the resolved versions, the list of imports found, and the import result after the change.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

A pull request that adds one dependency line and changes no Python. The description lists the lxml version a clean install resolved, the traceback before, every `lxml.html.clean` import in the repository with its file and line, and the form of the fix chosen with its reason: the `lxml[html_clean]` extra, or `lxml_html_clean` as its own requirement.

What a reviewer checks is that the fix is the companion package and not a pin. A `lxml<5.2` line would also make the import work and would silently hold the parser on a release from March 2024; the description shows the `pip freeze` line for `lxml-html-clean` so the reviewer can see which path was taken.

The description also states how far the import got afterwards. For GoogleScraper that is `output_converter.py` line 19 and SQLAlchemy 2's removal of a private attribute, named as the next change with the traceback, so the reviewer merges the lxml fix knowing the package still does not import and exactly why.

## Run it on every Monday morning

An unpinned `lxml` line breaks on an install day, not a commit day. Give the Project an automation with a `schedule` trigger, for example the cron expression `0 7 * * 1` in your timezone, and a prompt that tells the agent to install the dependencies fresh, import every module of the package, and open a pull request only when an import fails on a name a new release moved or removed. On a week when nothing broke, the run ends with a one-line answer and no pull request.

For a set of scrapers and parsers that share this lxml shape, start the request above once per repository from a script with an API key, and read each session's `pr_url` when its `run.completed` webhook arrives. Each run resolves its own lxml in its own sandbox, and runs past the key's `max_concurrent` limit are accepted and wait as `queued`.

## Why run it on Rush?

The dependency line takes a minute. What a fresh install also reveals is every other unpinned name that moved in the same years, and the only way to see the next one is to clear the first and import again. An agent in a sandbox does that in sequence and leaves each traceback in the session, so the pull request fixes one library and names the next instead of pretending the package starts.

It also starts from the state a new user is in. A machine that has had lxml 4.9 installed since the scraper was written never shows this error. A sandbox resolves what `pip install lxml` gives today, which is the environment the next clone will have.

## Frequently asked questions

### Which should I add, lxml[html_clean] or lxml_html_clean?

They install the same package. The extra keeps a single `lxml` line and ties the companion to it; the explicit line shows in `pip freeze` and in lockfiles as its own entry. The agent picks one, says why in the pull request, and either way the import resolves through the shim lxml keeps for `lxml.html.clean`.

### Can I pin lxml below 5.2 instead?

5.1.1 still ships the module as a compiled file inside the wheel, so a pin works for now. It also holds the parser on a release from March 2024 for every future Python and security fix. The request above tells the agent not to pin; the fix is one line.

### Why does the module still fail to import after the fix?

Because an unpinned requirements file moves every dependency at once. In the fixture the next failure is in `output_converter.py`, where SQLAlchemy 2 removed a private attribute the code reached into. The agent reports that with the file and line as a separate finding, since it is a different library and a code change rather than a dependency line.

### Does the agent change the Cleaner calls?

No. `Cleaner` and `clean_html` keep their names and arguments in `lxml_html_clean`, and the old import path keeps working once the package is installed, so `parsing.py` is untouched. The pull request changes only the dependency file.

### What about imports inside functions?

They fail only when the function runs, so a green `import` line does not prove them. The agent searches for `lxml.html.clean` across every file, including the lazy import at line 464 of `caching.py` in the fixture, and lists each one.

## More jobs like this: Fix a bug from an alert

- [Fix "TypeError: marked is not a function" After Upgrading marked](https://getrush.ai/run/fix-marked-is-not-a-function/marked/)
- [Fix "cannot import name 'soft_unicode' from 'markupsafe'"](https://getrush.ai/run/fix-markupsafe-soft-unicode-import-error/markupsafe/)
- [Fix "AttributeError: module 'matplotlib.cm' has no attribute 'get_cmap'" on Matplotlib 3.9](https://getrush.ai/run/fix-matplotlib-cm-get-cmap-attribute-error/matplotlib/)
- [Fix "TypeError: Missing parameter name at 1" After Upgrading to Express 5](https://getrush.ai/run/fix-missing-parameter-name-at-1/path-to-regexp/)
- [Fix "AttributeError: module 'bcrypt' has no attribute '__about__'" with passlib](https://getrush.ai/run/fix-module-bcrypt-has-no-attribute-about/passlib/)
- [Fix "AttributeError: module 'cv2' has no attribute 'xfeatures2d'" by Moving SIFT to cv2.SIFT_create](https://getrush.ai/run/fix-module-cv2-has-no-attribute-xfeatures2d/opencv-python/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
