What the agent does
The import is one line and the agent changes it first: from sklearn.externals import joblib becomes import joblib, with joblib added to the requirements. Then it keeps going, because in a project that saves models the import is rarely the only thing that aged. amirziai/sklearnflask is a Flask API that loads model/model.pkl at start-up and serves /predict, with /train to rebuild the model from data/titanic.csv. Its requirements.txt pins scikit-learn==0.20.3, pandas==0.24.2, numpy==1.16.3 and Flask==1.0.2. None of those has a prebuilt package for Python 3.12, so anyone setting it up today takes current releases, and line 9 of main.py raises the error in the title.
With the import fixed, the committed model is the next failure. Before loading a pickle it did not write, the agent disassembles it with pickletools, which lists the classes the file refers to without executing anything. model.pkl names sklearn.ensemble.forest RandomForestClassifier, sklearn.tree.tree DecisionTreeClassifier and sklearn.externals.joblib.numpy_pickle NumpyArrayWrapper. Those module paths no longer exist, and joblib.load on a current scikit-learn fails with ModuleNotFoundError: No module named 'sklearn.ensemble.forest'. The small model_columns.pkl, a plain list, loads fine. main.py catches the exception at start-up and prints No model here, so the server would come up looking healthy with nothing to predict with.
A pickled estimator cannot be carried across that many releases, so the model has to be rebuilt, and the repository has what is needed: the data and the /train route. That route fails too. On pandas 3 it answers 500 from line 66, df_[col].fillna(0, inplace=True), with Invalid value '0' for dtype 'str'. The loop decides which columns are categorical with col_type == 'O', and the Sex and Embarked columns now read as dtype str, so they fall into the numeric branch. The same line is also a chained in-place fill on a slice, which pandas now warns never updates the original. The agent changes three lines: copy the slice, test the dtype with pd.api.types.is_numeric_dtype, and assign the filled column back.
To know whether the rebuilt model is a fair replacement it needs the old one's behaviour, so it builds a Python 3.7 environment with the pinned versions and loads the committed file there. That model has 10 trees, scores 0.8361 on the 891 training rows, and answers the README's four sample passengers with [1, 1, 1, 0], which is not the [0, 1, 1, 0] the README prints. The retrained model has 100 trees, because the library's default changed, scores 0.8418 on the same rows and gives the same four answers as the old file. The agent reports all of this and flags what it cannot fix by itself: training sets no random seed, and tests.py holds one placeholder test that asserts True equals False.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "This service fails at start-up with \"ImportError: cannot import name 'joblib' from 'sklearn.externals'\". Replace the import with the standalone joblib package and add it to the requirements. Then check everything that depended on the old versions. For every pickled model or artifact in the repository: do not load it first. Disassemble it with pickletools and list the classes and module paths it references, then try loading it on current scikit-learn and report the exact result. Build a second environment with the versions the requirements file pins, on a Python old enough to install them, load the committed model there, and record its type, its main hyperparameters, its score on the training data and its predictions for the sample request in the README. If the model cannot load on current releases, retrain it with the project's own training code and data. Fix whatever stops training on current pandas and scikit-learn with the smallest change, and explain each line. Compare old and new model: hyperparameters whose defaults changed, score on the same rows, predictions for the README sample, and the feature column list. Make the service fail loudly at start-up when a model file exists but cannot be loaded. Add tests that start the app with the test client, train, and predict the README sample; replace any placeholder test. Update the pinned requirements to versions you ran. Do not commit a retrained model unless the repository already commits one, and if you do, say which library versions wrote it. Open a pull request with the comparison table.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
A pull request with a short diff and a long description. The code changes are the import, three lines in the training loop, a start-up check that no longer swallows a model that fails to load, real tests in place of the placeholder, and requirements moved to versions the run installed. Because the repository commits its model, the retrained model/model.pkl and model_columns.pkl are included, with the scikit-learn, joblib and Python versions that wrote them stated next to the files.
The description gives the reviewer a table: the old model loaded on the pinned stack against the new one, with tree count, training score, the four sample predictions and the column list for each. It includes the pickletools listing and the two error messages that followed the import fix. A reviewer decides things the agent only flags, such as whether 100 trees is acceptable where 10 were deployed, whether to set a seed, and whether the README's sample output should be corrected. If training cannot be made to work, the run ends for review with the failing step and its traceback, and no model is committed.
Run it on every Linear issue labelled agent
An old model service usually comes back to life as a ticket: move it to the new base image, or get it running for a new teammate. Create an automation in the Project that has the repository, give it this prompt, and add a linear trigger on linear.issue.labeled for a label of your choice. When someone in your org applies the label, a run starts, and the issue's text is added after the saved prompt. The trigger needs a Linear connection for the member the automation runs as, made in the console under Settings → Connections.
The prompt stays the same for every such issue, so keep the specifics in the ticket: which service, which Python it must run on, whether a retrained model may be committed. A state mapped to plan mode makes the first pass read-only, with the plan posted as a comment on the issue before any branch exists.
The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.
Why run it on Rush?
The comparison depends on having both worlds at once: the pinned versions from years ago on an interpreter that can still install them, and today's releases. That is two interpreters and two sets of compiled packages, and the old set is the kind of install that fights back on a current laptop. A Rush run builds both in its own sandbox, loads the old model where it can be loaded, trains the new one where it will run, and keeps each command and its output in the session.
It also makes the stopping point visible. Someone fixing the import at a desk sees the server start, sees No model here, and may reasonably think the job is done or the model is merely missing. The run follows the failure through three layers, import, pickle and training code, and the pull request shows each one with its message, so the reviewer knows the service was exercised through a prediction and not only through start-up.
Frequently asked questions
What replaces sklearn.externals.joblib?
The standalone joblib package: import joblib, with the same dump and load calls. scikit-learn used to bundle a copy and stopped. Add joblib to your requirements explicitly, even though scikit-learn installs it as a dependency.
Why does my old model.pkl fail to load after I fixed the import?
A pickle stores the import paths of the classes inside it, and scikit-learn has since renamed its internal modules. In the fixture the file refers to sklearn.ensemble.forest, which no longer exists. The model has to be retrained on the new version, or loaded in an environment with the old one.
Is it safe for the agent to load a pickle from my repository?
Loading a pickle runs code, so the prompt has it inspect the file with pickletools first, which only reads the opcodes. The run also happens in a sandbox of its own and not on your machine. For files from outside your team, keep the inspection step.
Will the retrained model give the same predictions?
Not necessarily. Defaults change between releases, such as the number of trees in a random forest, and training without a fixed seed varies from run to run. The pull request compares scores and sample predictions between old and new so the difference is measured, not assumed.
Can I avoid retraining by pinning the old scikit-learn?
Only together with an old Python, since releases from that period have no builds for current interpreters. That keeps the service on unsupported versions. The agent uses such an environment once, to read the old model's behaviour, and does not leave the project there.
More jobs like this: Turn a Linear label into a PR
- Fix You Must Pass the Application as an Import String to Enable Reload in Uvicorn (Uvicorn)
- Turn Labeled GitHub Issues Into Pull Requests for an Ansible Role (Ansible)
- Turn Labeled GitHub Issues Into Pull Requests for a Django App (Django)
- Turn Labeled GitHub Issues Into Pull Requests for an Express API (Express)
- Turn Labeled GitHub Issues Into Pull Requests for a FastAPI App (FastAPI)
- Turn GitHub Issues into Pull Requests for a Fastify API (Fastify)