# Fix "LookupError: Resource punkt_tab not found" in word_tokenize

Yes. An agent reproduces the NLTK 3.9 punkt_tab lookup in an empty data directory, fixes the download your code makes, and proves the tokenizer on 3.9 and current.

[Run this on Rush](https://cloud.getrush.ai/run/fix-resource-punkt-tab-not-found/nltk/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

NLTK 3.9 moved the Punkt sentence model that `word_tokenize` loads to a new resource, `punkt_tab`. In a clean data directory on 3.9.1, `nltk.download("punkt")` downloads and unzips `tokenizers/punkt`, returns `True`, and the very next `word_tokenize("Hello world.")` raises `LookupError: Resource punkt_tab not found`, with the downloader's own hint to run `nltk.download('punkt_tab')`. The traceback runs from `word_tokenize` into `sent_tokenize` and `PunktTokenizer.load_lang`, which looks for `tokenizers/punkt_tab/english/`. On 3.8.1 the same two lines tokenize. On 3.10.4 `download("punkt")` fetches `punkt_tab` beside `punkt`, so the error belongs to the 3.9 releases, and a project lands on one through a loose pin or a rebuilt image.

cmooredev/RepoReader is a command-line tool that clones a GitHub repository, indexes it and answers questions about it with OpenAI. `utils.py` line 6 runs `nltk.download("punkt")` at import time, and `clean_and_tokenize` at line 8 strips markup and URLs from a document and returns `nltk.word_tokenize(text)` on line 17. `file_processing.py` imports that function on line 10 and uses it while building a BM25 index over the cloned files, so the error does not show at start-up: it shows after `main.py` prints "Repository cloned. Indexing files..." and the first document is tokenized. `requirements.txt` pins `nltk==3.8.1` among about a hundred other pins, `langchain==0.0.136`, `openai==0.27.4` and `torch==2.0.0` among them, which is what keeps the tool on the release where `punkt` was enough.

The agent reproduces the failure without the rest of the stack. It installs `nltk==3.9.1` in a virtual environment, points `NLTK_DATA` at an empty temporary directory, imports `utils` and calls `clean_and_tokenize` on a sentence, and keeps the `LookupError`. Then it repeats the same steps on the current release and on the pinned 3.8.1, so the pull request can say on which releases the project's own download call is sufficient. The fix in `utils.py` adds `nltk.download("punkt_tab")` beside the existing `punkt` download, which works on 3.8.1 as well as 3.9.1 since the downloader index is the same, and moves both calls out of module import into a function that checks `nltk.data.find` first and downloads quietly only when a resource is missing, so importing `utils` no longer prints downloader output or touches the network on every process start.

The test the agent writes runs `clean_and_tokenize` in a subprocess with `NLTK_DATA` set to a fresh temporary directory, which proves the downloads the module makes are the ones `word_tokenize` needs rather than relying on whatever is cached in the sandbox's home directory. It runs that test under the pinned release and under 3.9.1, and the pin in `requirements.txt` moves to a release where both the download and the tokenizer were checked. `main.py`, which needs an `OPENAI_API_KEY`, is not run; nothing in the fix touches it.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "This project fails with LookupError: Resource punkt_tab not found when word_tokenize runs. Find every nltk.download call and every tokenizer call (word_tokenize, sent_tokenize) with file and line, and note which run at import time. Reproduce the error in a virtual environment with nltk 3.9.1 and NLTK_DATA set to an empty temporary directory, calling the project's own tokenizing function; then repeat on the release pinned in requirements.txt and on the current nltk release, and record for each whether the project's download call is enough. Fix the download so the resource word_tokenize needs is fetched, and move downloads out of import time into a function that checks nltk.data.find first and downloads quietly only when missing. Write a test that runs the tokenizing function in a subprocess with NLTK_DATA pointed at a fresh empty directory. Update the nltk pin in requirements.txt to a release you verified. Do not run the main script and do not use an OpenAI key. Open a pull request with the version table, the traceback, and the test output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

A pull request with a small diff in `utils.py`, a new test, and a changed pin in `requirements.txt`. Its description is led by a table of NLTK releases the agent tried in the sandbox and, for each, whether `download("punkt")` alone let `word_tokenize` run, followed by the traceback it captured, so a reader who only skims learns that the failure is a 3.9 behaviour and not a broken machine.

The session's record shows each virtual environment being created, the empty data directory, the failing call, and the test passing on the releases it was run against. Because the test runs in a subprocess with its own `NLTK_DATA`, the transcript shows the downloads happening inside the test, which is the evidence that the fix is the download and not a leftover file.

Before merging, a reviewer checks that the download now runs where the team wants it: at first use, as the pull request does, or in an image build step with `NLTK_DATA` baked in, which the description offers as the alternative for deployments with no outbound network. The reviewer also confirms the new pin is one the rest of the pinned stack installs beside.

## Run it on every exception captured in production

A tokenizer that fails on first use is the kind of error an error tracker catches once per deployment. Sentry signs its own requests and cannot add the bearer header a Rush `webhook` trigger needs, so a small handler on your server receives Sentry's issue webhook and forwards the payload to the trigger's URL with `Authorization: Bearer <its secret>`, sending the Sentry issue id as the `Idempotency-Key` of that trigger call so a redelivered alert starts no second run. The payload reaches the agent after the prompt, as data.

From a script instead, one `POST /o/{org}/sessions` per affected repository carries this request, the repository in `resources`, and a `request_id` that is a UUID v5 of the issue id. A webhook set on the API key reports `run.completed` or `run.failed` for each run that key started, with the `session_url` to read the pull request from.

## Why run it on Rush?

The error depends on what is already in a data directory, which is why it is hard to reproduce on a developer's machine: a laptop that has had NLTK for years has `punkt_tab` from some earlier project, and the bug hides. A sandbox starts empty. The agent controls `NLTK_DATA` for every run it makes, so the version table in the pull request comes from clean environments, not from one machine's cache.

It also keeps the fix proportionate. A project like this one has a hundred pins and a model dependency that takes a while to install; the agent installs only NLTK for the reproduction, runs the test matrix in minutes, and leaves the record of every command with the pull request. Several tools forked from the same tutorial get the same request, one run each, in parallel up to the key's `max_concurrent` limit.

## Frequently asked questions

### Why did nltk.download("punkt") stop being enough?

Because NLTK 3.9 loads the Punkt model from `tokenizers/punkt_tab/<language>/` and `punkt` holds the older files. In a clean directory on 3.9.1, downloading `punkt` succeeds and `word_tokenize` still raises; downloading `punkt_tab` makes it pass. On 3.10.4 the `punkt` download brings `punkt_tab` with it.

### Will downloading punkt_tab break NLTK 3.8?

No. On 3.8.1 in a clean directory `nltk.download("punkt_tab")` returns `True` and `word_tokenize` runs, since the downloader index is shared across releases. The agent keeps the `punkt` download for the older code path and adds `punkt_tab`, and runs the test on the pinned release and on 3.9.1.

### Which tokenizers need the resource?

`word_tokenize` calls `sent_tokenize` first, and that is where Punkt loads, so both need `punkt_tab`. A regular-expression tokenizer such as `wordpunct_tokenize` loads no resource and runs in an empty data directory. The agent lists which the project uses before changing anything.

### Does the fix need network access at runtime?

At first use, yes: the download runs when a resource is missing. For a deployment without outbound network, the pull request describes the alternative of downloading during the image build and pointing `NLTK_DATA` at that directory, which the test's subprocess already exercises.

## More jobs like this: Fix a bug from an alert

- [Fix "ImportError: cannot import name 'SafeConfigParser' from 'configparser'" on Python 3.12](https://getrush.ai/run/fix-safeconfigparser-import-error/configparser/)
- [Fix "DEPRECATION WARNING [legacy-js-api]: The legacy JS API is deprecated and will be removed in Dart Sass 2.0.0"](https://getrush.ai/run/fix-sass-legacy-js-api-deprecation-warning/sass/)
- [Fix a Scrapy spider whose start_requests() is never called and scrapes 0 items](https://getrush.ai/run/fix-scrapy-start-requests-not-called/scrapy/)
- [Fix "distplot is a deprecated function and will be removed in seaborn v0.14.0"](https://getrush.ai/run/fix-seaborn-distplot-deprecated-warning/seaborn/)
- [Fix Actix Web Errors from Sentry Alerts](https://getrush.ai/run/fix-sentry-errors/actix/)
- [Turn Sentry Errors Into Fix Pull Requests for an aiohttp App](https://getrush.ai/run/fix-sentry-errors/aiohttp/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
