All jobs

Fix "_pickle.UnpicklingError: Weights only load failed" from torch.load in PyTorch 2.6 and Later

Yes. An agent classifies every torch.load call by what the file holds and who wrote it, then fixes each the right way and round-trips a checkpoint to prove it.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

PyTorch 2.6 changed one default. torch.load(path) used to unpickle anything in the file; it now behaves as torch.load(path, weights_only=True) and accepts only tensors, state dictionaries and a short list of plain types. A checkpoint that contains a whole model object, an optimizer, an argparse.Namespace or a NumPy scalar raises _pickle.UnpicklingError: Weights only load failed, followed by a line naming the first type it would not accept. The message itself offers weights_only=False as a way out, and that is where people stop. An agent treats each call as a decision about trust, because the old behaviour runs whatever code the file contains.

mila-iqia/babyai is the hard version of this problem, not the easy one. babyai/utils/model.py saves with torch.save(model, path), the entire ACModel object, and loads with a bare torch.load(path) when CUDA is present or torch.load(path, map_location=torch.device("cpu")) otherwise. Neither call passes weights_only. setup.py asks for torch>=0.4.1 with no upper bound, while environment.yaml names pytorch=1.4, so a plain pip install today gets a PyTorch on the new default and the first load_model fails. The error would name babyai.model.ACModel as an unsupported global.

The agent first maps the calls. In this repository load_model is the only wrapper around torch.load, and it is reached from scripts/train_rl.py, from babyai/imitation.py and from the agent class in babyai/utils/agent.py. The files it reads are ones the project's own training scripts wrote to the models folder under BABYAI_STORAGE. That provenance is what decides the fix. For a file your own code produced on your own disk, passing weights_only=False explicitly is a legitimate choice: it restores the behaviour the code was written for and records, in the source, that the file is trusted. The agent makes that change in the two calls, adds a comment stating the assumption, and does not apply it anywhere a checkpoint could come from a URL, an upload or another party.

It then lays out the better fix without forcing it. Saving model.state_dict() and rebuilding the model from its constructor arguments would let the default safe loader work. Here that is a real refactor: ACModel takes obs_space, action_space, three dimension sizes, arch and several flags, none of which are stored beside model.pt, and every existing checkpoint would need converting. Allow-listing with torch.serialization.add_safe_globals looks like a middle path and is not one for whole-model files, because after the model class the loader stops at the next type inside it, such as torch.nn.modules.linear.Linear, and then the next. The agent writes this up as a follow-up with a conversion script outline. To verify the change it made, it installs a current CPU build of PyTorch, constructs a small ACModel, saves it with the project's save_model, and loads it back with load_model, first on the unmodified code to capture the error and then on the fixed code.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Loading a checkpoint fails on PyTorch 2.6 or later with \"_pickle.UnpicklingError: Weights only load failed\". Find every call to torch.load in the repository, including through wrappers, and every torch.save that writes the files those calls read. For each load, build a row: file and line, what object is saved (a state_dict, a dict of tensors and plain values, a whole nn.Module, an optimizer, anything else), where the file comes from (written by this project's own code, shipped in the repository, downloaded from a URL, supplied by a user), and whether weights_only is already passed. Then fix each call according to its row. If the file holds only tensors and plain containers, leave the default and confirm it loads. If it holds a few extra known types such as argparse.Namespace or NumPy scalars, keep weights_only=True and allow-list exactly those types with torch.serialization.safe_globals around the call. If it holds a whole pickled model or other arbitrary objects and is written only by this project's own code to local storage, pass weights_only=False explicitly with a one-line comment stating that the file is trusted and why. If such a file can come from a download, an upload or any third party, do not pass weights_only=False: stop, report the call, and propose converting that checkpoint format to a state_dict. Do not pin PyTorch to an older version. Install a current CPU build of PyTorch and, for each load you changed, create a small object of the saved type with the project's own code, save it with the project's save function, and load it with the project's load function, once before your change to capture the error and once after. Include the commands and output. Do not download any checkpoint from the internet. In the pull request description include the table, and for every weights_only=False you added, describe what a state_dict-based format would require (constructor arguments to store, a conversion script for existing files) without implementing it.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request with a small diff and a table that explains it. Each row is one torch.load call: what the file contains, who writes it, and which of the three treatments it received. For a project that pickles whole models to its own storage, the diff is an explicit weights_only=False on those calls, each with a comment naming the assumption it rests on.

The description shows the round trip. First the project's own save and load functions on the unmodified code, ending in the UnpicklingError and the name of the unsupported global. Then the same commands after the change, ending with the loaded object's type. A reviewer can see that the error reproduced was the one reported, and that the fix was exercised through the project's code path and not a toy script.

The part to read slowly is the provenance column. Every weights_only=False is a statement that the file can only have been written by code the team controls. If any of those files can arrive another way, a shared bucket anyone can write to or a model zoo link in the README, the reviewer should reject that row. The agent lists the calls it refused to relax for that reason, and the state-dictionary migration it recommends for them.

Run it on every Grafana alert firing

Training and evaluation jobs usually fail inside a scheduler or a batch queue, and an alert rule on their logs is the first thing to see it. A Grafana webhook contact point can send a custom Authorization header, so it can call a Rush webhook trigger without a relay. Create the trigger on an automation that holds the prompt above, copy its secret when the console shows it, and configure the contact point with the trigger's URL and Authorization: Bearer followed by that secret. The alert's JSON payload is added after the prompt as event text.

Route only the rule that matches Weights only load failed to this contact point, and set the automation's max runs per hour to 1. Grafana re-sends while an alert stays firing and sends again when it resolves. With the cap at 1, no more than one run starts in an hour: one further notification waits for the next hour and any beyond it are refused, so a long outage cannot open a pull request per repeat.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

The fix has to be shown on the PyTorch version that fails, and research code is often developed in an environment years older. In this example the environment file names PyTorch 1.4 on Python 3.6. Nobody wants to rebuild their working setup to reproduce an error that only appears on a fresh install. The agent's sandbox is that fresh install: it puts a current PyTorch beside the project, runs the save and load functions as they are, and keeps the failing and passing output in the session record.

The judgement about trust also benefits from a full read of the code, which is cheap for an agent and easy to skip under deadline. Tracing every torch.load back to the torch.save that feeds it, and to the README sections that tell users where files come from, is what separates a safe weights_only=False from a dangerous one. The table in the pull request is that trace, written down.

Frequently asked questions

Is weights_only=False safe?

Only for files you trust completely. With it, loading a checkpoint can execute arbitrary code embedded in the file, which is why the default changed. It is acceptable for checkpoints your own training code wrote to storage you control, and it is the wrong fix for anything downloaded or user-supplied.

The error names numpy.core.multiarray.scalar or argparse.Namespace. What then?

Those are ordinary values saved next to the weights. Keep the safe loader and allow them: wrap the call in torch.serialization.safe_globals([...]) with exactly the types the message names. The agent adds only the types your checkpoints actually contain and lists them in the pull request.

Can I allow-list my model class and keep weights_only=True?

Not usefully when the file holds a whole pickled module. After your class, the loader rejects the next type inside it, such as torch.nn.modules.linear.Linear, and you end up allow-listing much of the library. The durable fix for that case is to save a state dictionary and rebuild the model in code.

Why not pin torch below 2.6?

A pin keeps the old default and postpones the decision. It also holds back every other fix in newer releases and eventually conflicts with new Python versions. The code change is one argument per call, and it makes the trust decision visible where a pin hides it.

Does map_location affect this?

No. map_location only decides which device the tensors land on. A call such as torch.load(path, map_location='cpu') still uses the new default and raises the same error for a file that holds more than weights.

More jobs like this: Fix a bug from an alert

Keep reading