All jobs

Fix "Cannot Import Name 'Imputer' from 'sklearn.preprocessing'" After a scikit-learn Upgrade

Yes. An agent upgrades scikit-learn, reproduces the ImportError, moves the code to SimpleImputer with its changed arguments, fixes the rest and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent clones the repository into its own cloud sandbox and reads the script and its pins first. In a research script like iamaziz/ar-embeddings, which classifies Arabic sentiment with word embeddings, requirements.txt pins scikit_learn==0.18.1, gensim==1.0.1, numpy==1.21.0, pandas==0.19.2, nltk==3.6.6 and matplotlib==2.0.0. asa.py imports Imputer at line 23 and uses it once, in remove_nan at line 241: Imputer(missing_values='NaN', strategy='mean', axis=0), then fit_transform(x) on the averaged word vectors. The same file loads the embeddings with KeyedVectors.load_word2vec_format and reads w2v_model.syn0.shape at line 102. The dataset files under datasets/ are in the repository; embeddings/arabic-news.tar.gz is a Git LFS pointer to a 179 MB model, so the agent can import and unit-test the code but not run the full training.

Then it reproduces the break on current releases. In a current Python environment with the current scikit-learn, numpy, pandas and gensim, python -c "import asa" fails at line 23: ImportError: cannot import name 'Imputer' from 'sklearn.preprocessing'. Imputer was replaced by sklearn.impute.SimpleImputer in 0.20 and removed in 0.22, so any upgrade past 0.21 hits this line before a single sample is read.

The fix is three lines, and two of its details are easy to get wrong. The import becomes from sklearn.impute import SimpleImputer, and the call becomes SimpleImputer(missing_values=np.nan, strategy="mean"): the string 'NaN' is no longer a valid missing_values, and axis=0 does not exist because SimpleImputer always works per column, which is what axis=0 meant. The agent checks remove_nan on a small array with a NaN and gets the column mean filled in, under -W error so no deprecation slips by. It also fixes what the same bump breaks on the next line it would reach: gensim 4 has no syn0, so w2v_model.syn0.shape at line 102 becomes w2v_model.vectors.shape, confirmed with a two-word KeyedVectors built in the sandbox.

It stops where proof stops. With the model only available through LFS and the README's HuggingFace link, the agent does not claim a training run; the pull request lists init_sims(replace=True) as a gensim call to check on the first real run, and leaves the classifier list, the ROC plot and the train_test_split call alone, since the current API accepts them as written. The six pinned versions in requirements.txt are moved to the releases the run resolves.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "This Python script fails with \"ImportError: cannot import name 'Imputer' from 'sklearn.preprocessing'\" after upgrading scikit-learn. Read requirements.txt and every file that imports sklearn first. Install the current releases of the pinned packages, reproduce the import error and quote it with its line. Replace Imputer with sklearn.impute.SimpleImputer, translating each argument and saying which ones changed meaning, and check the replacement on a small array with missing values. Then import the module again and fix any other break the same upgrade causes in code the import reaches, each with its line. Say clearly which parts you could not run and why. Open a pull request that quotes the failing and passing runs.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request touching asa.py and requirements.txt: the import moved to sklearn.impute, remove_nan rewritten around SimpleImputer(missing_values=np.nan, strategy="mean"), the gensim syn0 read changed to vectors, and the six pins moved to current releases. The description quotes the ImportError with its line from the upgraded environment, the clean import asa afterwards, and the small array before and after imputation.

A reviewer checks the argument translation, that 'NaN' became np.nan and that dropping axis=0 keeps per-column means, reads the list of what was not run and why, and decides whether to run the full script once with the real model before merging. The run's record holds each command with its output, so the import failure and the fixed import can be opened rather than taken on trust.

Run it on every Monday morning

Add an automation to the Project with this prompt and a schedule trigger, cron set to 0 9 * * 1 in your timezone. Each Monday one run moves the pins to current releases, imports the code and opens one pull request on its own branch; a week where an upgrade breaks something the agent cannot repair ends needs_review with the error attached, and a week with nothing to bump opens nothing.

The same body works from your own code with POST /o/{org}/sessions and an API key when a dependency bot's scikit-learn bump turns red; the key's webhook_url receives run.completed or run.failed with the session URL and the pull request when the run ends.

The use case this job follows, with its automation and trigger: Upgrade dependencies weekly.

Why run it on Rush?

A pin from 2016 holds a whole environment in place, and the first upgrade breaks several things at once. The agent installs the current releases in its own sandbox, hits the Imputer import first, fixes it, imports again and hits the gensim rename next, so the pull request covers what the upgrade actually breaks rather than the one error in the ticket. Every command and its output stays in the run's record.

The honest boundary matters as much as the fix. The model behind this script is a 179 MB file outside the repository, so the agent proves the import and the imputer on data it can make, names the call it could not exercise, and leaves the training run to the reviewer instead of reporting a success it did not see.

Frequently asked questions

What happened to Imputer?

scikit-learn 0.20 moved imputation into its own sklearn.impute module as SimpleImputer and removed sklearn.preprocessing.Imputer in 0.22. The new class takes missing_values=np.nan instead of the string 'NaN' and has no axis argument; it always fills by column.

Why did the agent change gensim code in a scikit-learn fix?

Because the prompt asks it to import the module again after the first fix, and gensim 4, which the same upgrade pulls in, has no syn0 attribute. Fixing only the first error would leave the script failing one line later on the next run.

Can it run the whole training?

Only if the model file is reachable. Here it is a Git LFS pointer to a 179 MB download, so the agent tested remove_nan and the embedding reads on data it built and said so. If your repository holds the model or the run can fetch it, the same prompt runs the script end to end.

Will it pin the new versions?

It replaces the six exact pins with the releases the run resolves, so the file stays reproducible. If you prefer ranges, say so in the prompt.

More scikit-learn jobs

More jobs like this: Upgrade dependencies weekly

Keep reading