All jobs

Fix "AttributeError: The vocab attribute was removed from KeyedVector in Gensim 4.0.0"

Yes. An agent finds every model.wv.vocab use, replaces it with the key_to_index mapping, and trains a small model to show the script running end to end.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the script that fails and every other file that touches the word vectors, because a half-migrated repository is the usual cause. In a research codebase like alexeyev/abae-pytorch, requirements.txt already asks for gensim>=4.1.0 and the README says so since December 2021, and most of the code was moved: word2vec.py trains with vector_size=200, the Gensim 4 argument name, reader.py line 97 iterates w2v_model.wv.key_to_index, and model.py line 145 reads wv.index_to_key. One line was missed: word2vec.py line 50, in the __main__ block that prints neighbours after training, tests if word in model.wv.vocab:.

Then it reproduces without the 100 MB Amazon reviews file example_run.sh downloads. The agent writes a 400-line corpus of a dozen words, runs python word2vec.py tiny.txt with a clean install, which resolves Gensim 4.4.0, and gets the full message after training completes: AttributeError: The vocab attribute was removed from KeyedVector in Gensim 4.0.0. Use KeyedVector's .key_to_index dict, .index_to_key list, and methods .get_vecattr(key, attr) and .set_vecattr(key, attr, new_val) instead. See https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4. The trained model is saved before the failure, so the training itself is fine and only the check breaks.

The fix is if word in model.wv.key_to_index:, matching what reader.py already does; word in model.wv also works, since KeyedVectors supports membership directly, and the agent notes both. With the change the script prints Training done. and then ten nearest neighbours for each of the five probe words from similar_by_word, which still exists in Gensim 4. The model loads back with Word2Vec.load, has 15 keys for the tiny corpus, and type(model.wv) is KeyedVectors.

Proof is the before-and-after run on the same corpus plus a grep showing no other .vocab, index2word or syn0 reference remains outside comments; model.py keeps a commented-out wv.syn0 line the agent leaves as is. Nothing in main.py or the Hydra configs changes, and torch is not needed to exercise word2vec.py, so the sandbox run stays small.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Our training script fails with \"AttributeError: The vocab attribute was removed from KeyedVector in Gensim 4.0.0\". Find every use of .vocab, index2word, syn0 and other Gensim 3 attributes across the repository and say which were already migrated, then reproduce the error on a small corpus you write yourself rather than the full dataset, fix the remaining uses with key_to_index or index_to_key, and run the script again to show it completing and printing its neighbours. Open a pull request with the grep results, the gensim version installed and both runs.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that changes one line in word2vec.py from model.wv.vocab to model.wv.key_to_index. The description lists the grep over the repository, naming reader.py and model.py as already on the Gensim 4 API, quotes the full AttributeError with its migration link, names Gensim 4.4.0 as the installed release, and shows the neighbour output after the fix.

A reviewer checks that the replacement keeps the same meaning, a membership test on the trained vocabulary, and that similar_by_word and Word2Vec.load are untouched. They see the tiny corpus the agent used and can rerun the two commands in a minute.

The description records what did not change: requirements.txt already requires gensim>=4.1.0, the vector_size and max_vocab_size training arguments are current, and main.py was not run because it needs torch and the full dataset.

Run it on every failed training job

A training job that fails after the model is saved wastes the run, so the job runner is the trigger. A handler on your server receives the job's failure notice and forwards it to a Rush webhook trigger with Authorization: Bearer and the trigger's secret, and the traceback follows the prompt as data; the run opens the pull request with the fix.

For a group of research repositories that share the same word2vec helpers, your server sends this request once per repository, each with its own UUID as request_id so a retry returns the run already started, and reads each run.completed webhook; a key starts at most max_concurrent runs at once and queues the rest.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

The error says what to use instead, but it says nothing about where else the old attribute hides, and the full pipeline needs a dataset download and a GPU to reach the failing line. In a Rush sandbox the agent writes a small corpus, trains in seconds, greps the whole repository, and keeps both runs, so the pull request shows the one missed line and that the rest of the code was already migrated.

A one-line fix is only convincing with the script seen running past the point that failed. The agent shows Training done. followed by the neighbour lists on the same corpus, so the reviewer reads a diff with the output that proves it.

Frequently asked questions

Why did only one line fail if the project already requires Gensim 4?

Because the rest was migrated: reader.py uses key_to_index and model.py uses index_to_key. The __main__ block of word2vec.py runs only when the script is invoked directly, so its model.wv.vocab check survived until someone trained vectors from the command line.

key_to_index or in model.wv?

Both are membership tests on the same vocabulary; word in model.wv is the shortest, and model.wv.key_to_index matches the style already used in reader.py. The pull request uses key_to_index for consistency and mentions the alternative.

Do I need the full dataset to check the fix?

No. The agent writes a few hundred lines of repeated words, trains with the script's own main, and reaches the failing line in seconds. The real dataset only changes the neighbours printed, not the code path.

Does similar_by_word still exist in Gensim 4?

Yes. The after-fix run prints ten neighbours per probe word from model.wv.similar_by_word, and Word2Vec.load reads the saved model back with 15 keys for the tiny corpus.

More Gensim jobs

More jobs like this: Fix a bug from an alert

Keep reading