What the agent does
The agent reads the requirements before the traceback, because the error only appears when nothing pins scikit-learn. In a training repository like iterative/cml_base_case, requirements.txt lists setuptools, scikit-learn and matplotlib with no versions, and the workflow in .github/workflows/cml.yaml runs pip3 install -r requirements.txt and python train.py on python-version: '3.x', then posts metrics.txt to the pull request with cml comment create --pr --publish. So every run of that workflow installs whatever scikit-learn is newest.
A clean install today resolves scikit-learn 1.9.1, NumPy 2.4.6 and matplotlib 3.11.2, and python train.py stops at line 5, from sklearn.metrics import plot_confusion_matrix, with ImportError: cannot import name 'plot_confusion_matrix' from 'sklearn.metrics'. To date the change, the agent installs two older releases in their own environments. On 1.1.3 the import works and the call on line 32 prints FutureWarning: Function plot_confusion_matrix is deprecated; Function plot_confusion_matrix is deprecated in 1.0 and will be removed in 1.2. Use one of the class methods: ConfusionMatrixDisplay.from_predictions or ConfusionMatrixDisplay.from_estimator. On 1.2.0 the import already raises the same ImportError as 1.9.1.
The replacement keeps the call's shape. Line 32 is disp = plot_confusion_matrix(clf, X_test, y_test, normalize="true", cmap=plt.cm.Blues), followed by plt.savefig("plot.png"). ConfusionMatrixDisplay.from_estimator takes estimator, X and y positionally and normalize, cmap, display_labels and the rest as keywords, so the agent changes the import to from sklearn.metrics import ConfusionMatrixDisplay and the call to ConfusionMatrixDisplay.from_estimator(clf, X_test, y_test, normalize="true", cmap=plt.cm.Blues), and nothing else in the file.
Proof is the script's own output. train.py reads the four CSV files under data/ with np.genfromtxt, fits a RandomForestClassifier with max_depth 5, prints the test accuracy, writes metrics.txt and saves plot.png. On the branch it prints 0.864, writes metrics.txt with Accuracy: 0.8640 and the image link, and produces plot.png, which matches the accuracy line already committed in the repository's metrics.txt. The agent does not commit the regenerated image or metrics file, since the workflow regenerates both on every push.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Our CML training workflow fails with \"ImportError: cannot import name 'plot_confusion_matrix' from 'sklearn.metrics'\". Read requirements.txt, the workflow and train.py, say which scikit-learn a clean install resolves, and find the release that deprecated the function and the one that removed it by installing each and running the script. Replace the call with ConfusionMatrixDisplay.from_estimator with the same arguments, run train.py on the current release, and open a pull request quoting the traceback before, the deprecation message from the older release, and the script's printed accuracy and written files after. Do not commit regenerated outputs.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
A pull request that changes two lines of train.py: the import and the plotting call. Its description quotes the ImportError on scikit-learn 1.9.1, the FutureWarning from 1.1.3 that names the replacement, the ImportError on 1.2.0, and the accuracy and file list from running the script on the branch.
A reviewer checks that normalize="true" and cmap=plt.cm.Blues survive unchanged, that plt.savefig("plot.png") still follows the call, and that requirements.txt is untouched. Because the workflow posts metrics.txt to pull requests, the pull request's own CML comment shows the plot rendered from the new call.
If the team wants the install reproducible, the description notes that the three unpinned lines in requirements.txt are why a release from years ago changed the outcome, and leaves pinning as its own decision.
Run it on every failed training workflow
The failure surfaces as a red workflow, so the trigger is that failure. A handler on your server receives GitHub's workflow-run webhook and, when the job log holds an ImportError from sklearn, starts this run with POST /o/{org}/sessions and a UUID v5 of the workflow run id as request_id, so a redelivered webhook returns the run already started.
For a sweep across training repositories, your server sends this request once per repository, each with its own UUID as request_id, and reads the run.completed webhooks; a key runs at most max_concurrent at once and queues the rest.
The use case this job follows, with its automation and trigger: Fix a bug from an alert.
Why run it on Rush?
Dating a removal takes several installed releases, and the deprecation message that names the replacement only prints when the old call actually runs. In a Rush sandbox the agent builds one environment per release, runs the script in each and keeps every line, so the pull request shows the warning text from 1.1.3 and the error from 1.2.0 rather than a changelog citation.
The script writes files and an accuracy, which is the natural check that the new call does what the old one did. The agent runs it on the branch, compares the printed accuracy with the one committed in metrics.txt, and reports the generated plot, so the reviewer reads a result rather than trusting that two keyword arguments were carried over.
Frequently asked questions
Which scikit-learn release removed plot_confusion_matrix?
The agent installs releases to find out rather than guessing. On 1.1.3 the function imports and prints a FutureWarning saying it is deprecated in 1.0 and will be removed in 1.2; on 1.2.0 the import fails with the same error you see on 1.9.1.
Is ConfusionMatrixDisplay.from_estimator a direct replacement?
For this call, yes. It takes the estimator, the features and the labels positionally and accepts normalize, cmap, display_labels, values_format, ax and colorbar as keywords, so the existing arguments move over unchanged. from_predictions is the alternative when you already have predicted labels.
Will the agent pin scikit-learn instead?
Not unless asked. Pinning would hide the removal rather than fix the code, and the repository's unpinned requirements.txt is a choice the team made. The pull request names the unpinned lines so pinning can be its own change.
Why not commit the regenerated plot and metrics?
The workflow regenerates metrics.txt and plot.png on every push and publishes them in a pull request comment. Committing them would add a binary diff that the next run overwrites, so the pull request quotes the accuracy and file names instead.