What the agent does
scikit-learn 0.18 introduced sklearn.model_selection and deprecated three older modules: cross_validation, grid_search and learning_curve. Version 0.20 removed them. A script that still says from sklearn.cross_validation import train_test_split fails at import with ModuleNotFoundError: No module named 'sklearn.cross_validation'. For train_test_split, cross_val_score and GridSearchCV the fix is the import line alone. The splitter classes are different: KFold, StratifiedKFold and ShuffleSplit used to take the data size or labels in the constructor and are now built with n_splits and asked for indices through .split(X, y). The agent changes the import, rewrites any splitter that needs it, and then does the part people skip, which is running the code on a current scikit-learn and reading the numbers.
faizann24/Fwaf-Machine-Learning-driven-Web-Application-Firewall is one file, script.py, with two data files beside it. It imports train_test_split from the old module and calls it once, with test_size=0.2 and random_state=42. The agent changes that import to sklearn.model_selection and the script gets past its imports. Nothing else in it names a removed module, so the diff that fixes the reported error is one line.
Getting past the import puts years of other changes in play, and the agent runs the script to see them. The model is LogisticRegression with a class_weight and no solver named. The default solver changed from liblinear to lbfgs in 0.22, with a limit of 100 iterations, so the same line now fits a different optimiser and may print a ConvergenceWarning. The agent reports the warning if it appears and the five figures the script prints, accuracy, precision, recall, F1 and AUC, under the default solver and with solver='liblinear' set explicitly. It does not choose between them. That is the author's decision, and the pull request lays both results side by side.
It also finds a reason the figures move even with no library change. loadFile reads each file and passes the lines through list(set(data)), and the script does it again with list(set(badQueries)). Set order for strings varies from one Python process to the next, so random_state=42 splits a differently ordered list each run and the printed metrics differ run to run. The agent demonstrates this by running the script twice with different PYTHONHASHSEED values, and proposes sorting after deduplication in a separate commit. Along the way it notes that matplotlib.pyplot is imported and never used, and that the files are opened with no encoding, so reading them depends on the platform default.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "This project fails with \"ModuleNotFoundError: No module named 'sklearn.cross_validation'\" (or sklearn.grid_search, sklearn.learning_curve). Find every import from these removed modules in scripts, packages and notebooks and change it to sklearn.model_selection. For functions whose call is unchanged (train_test_split, cross_val_score, cross_val_predict, GridSearchCV, RandomizedSearchCV, learning_curve), change only the import. For splitter classes whose constructor changed (KFold, StratifiedKFold, LabelKFold now GroupKFold, ShuffleSplit, StratifiedShuffleSplit, LeaveOneOut), rewrite the construction to use n_splits and the iteration to use .split(X, y), keeping shuffle and random_state as they were, and show old and new code for each. Where code reads grid_scores_ from a search object, move it to cv_results_. Then install the newest scikit-learn and run every script you touched from start to finish. Capture all warnings. For each estimator created without an explicit solver or other parameter whose default has changed since scikit-learn 0.19, say what the default was and is, and run the script both ways, reporting the printed metrics for each; do not pick one. Check whether the results are reproducible: run twice with different PYTHONHASHSEED values and report whether the metrics match; if they do not, find the source of the ordering and propose a fix in a separate commit. If you can install scikit-learn 0.19 on an older Python, run the original script there as a baseline; if you cannot, say so. Add a requirements file with the versions you ran if none exists. Open a pull request with the import changes, the metrics table, the warnings and the commands.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
A pull request whose first commit is the import change, one line for the repository above. The description then gives a small table: the five metrics the script prints, in a column for the current default solver and a column for liblinear, with the scikit-learn version at the top and any ConvergenceWarning text beneath. If the agent managed to install 0.19 on an older interpreter, a third column shows the original behaviour; if not, the description says that the baseline could not be produced and why.
The second commit, if the reviewer wants it, sorts the deduplicated lines so a fixed random_state means a fixed split. The description shows the two runs with different hash seeds that motivated it. A reviewer's job is mostly reading: whether the metric differences between solvers matter for what the model is used for, and whether to pin the solver in code. They also check the new requirements file, which lists what the agent ran with and is the first statement of dependencies the repository has had.
Run it on every Monday morning
Teams with a folder of analysis scripts or course notebooks rarely know which ones still run. Create an automation in the Project with the prompt above and a schedule trigger with the cron 0 7 * * 1 and your timezone. Each week Rush starts one run that installs the newest scikit-learn, executes the scripts and opens a pull request when an import or a signature has gone stale. Say in the saved prompt that it should add to its previous pull request if that is still open, and finish with a one-line answer when everything runs.
A schedule fires at most hourly and this job has no reason to run that often; weekly matches how often scikit-learn's changes reach a fresh install. If the scripts live across many repositories, a small script of your own can send the request per repository instead, each with a request_id that is a UUID v5 of the repository name and the ISO week, so a rerun in the same week starts nothing new.
The use case this job follows, with its automation and trigger: Fix a bug from an alert.
Why run it on Rush?
The import fix is trivial and the consequences are not, and the only way to see them is to run the script on a clean install of the current library. This script reads more than a million lines of queries, builds character n-gram features and fits a model, several times over if both solvers and two hash seeds are compared. In its own sandbox the agent can do all of those runs, and the session transcript keeps the printed metrics and warnings from each, so the table in the pull request is copied from output and not from memory.
A baseline on scikit-learn 0.19 needs a Python old enough to install it, which few developers keep around. The agent can try to build that environment without touching anyone's working setup, and if the attempt fails, the failed commands are in the transcript as well. Either way the reviewer knows whether the before column is measured or missing.
Frequently asked questions
What replaces sklearn.cross_validation?
sklearn.model_selection. train_test_split, cross_val_score and cross_val_predict moved there with the same arguments. sklearn.grid_search and sklearn.learning_curve were folded into the same module, so GridSearchCV and learning_curve are imported from it as well.
Why does KFold raise a TypeError after I fix the import?
The class changed shape when it moved. The old form was KFold(n, n_folds=5) and you iterated over the object. The new form is KFold(n_splits=5) and you iterate over kf.split(X). StratifiedKFold likewise no longer takes y in its constructor; pass it to .split(X, y).
My accuracy changed after upgrading. Did the fix break something?
The import did not. Defaults did: LogisticRegression switched its default solver in 0.22, RandomForestClassifier its default number of trees, and several others moved too. The agent lists each estimator in your code that relies on a changed default and runs it both ways so you can see how much of the difference comes from that.
Can I just install an old scikit-learn?
Only on an old Python, since releases before 0.20 have no wheels for current interpreters, and then every other package in the environment has to be old too. That is workable for reproducing a published result once. For code you intend to keep running, changing the import is one line and removes the constraint.