What the agent does
NLTK 3.9 moved the Punkt sentence model that word_tokenize loads to a new resource, punkt_tab. In a clean data directory on 3.9.1, nltk.download("punkt") downloads and unzips tokenizers/punkt, returns True, and the very next word_tokenize("Hello world.") raises LookupError: Resource punkt_tab not found, with the downloader's own hint to run nltk.download('punkt_tab'). The traceback runs from word_tokenize into sent_tokenize and PunktTokenizer.load_lang, which looks for tokenizers/punkt_tab/english/. On 3.8.1 the same two lines tokenize. On 3.10.4 download("punkt") fetches punkt_tab beside punkt, so the error belongs to the 3.9 releases, and a project lands on one through a loose pin or a rebuilt image.
cmooredev/RepoReader is a command-line tool that clones a GitHub repository, indexes it and answers questions about it with OpenAI. utils.py line 6 runs nltk.download("punkt") at import time, and clean_and_tokenize at line 8 strips markup and URLs from a document and returns nltk.word_tokenize(text) on line 17. file_processing.py imports that function on line 10 and uses it while building a BM25 index over the cloned files, so the error does not show at start-up: it shows after main.py prints "Repository cloned. Indexing files..." and the first document is tokenized. requirements.txt pins nltk==3.8.1 among about a hundred other pins, langchain==0.0.136, openai==0.27.4 and torch==2.0.0 among them, which is what keeps the tool on the release where punkt was enough.
The agent reproduces the failure without the rest of the stack. It installs nltk==3.9.1 in a virtual environment, points NLTK_DATA at an empty temporary directory, imports utils and calls clean_and_tokenize on a sentence, and keeps the LookupError. Then it repeats the same steps on the current release and on the pinned 3.8.1, so the pull request can say on which releases the project's own download call is sufficient. The fix in utils.py adds nltk.download("punkt_tab") beside the existing punkt download, which works on 3.8.1 as well as 3.9.1 since the downloader index is the same, and moves both calls out of module import into a function that checks nltk.data.find first and downloads quietly only when a resource is missing, so importing utils no longer prints downloader output or touches the network on every process start.
The test the agent writes runs clean_and_tokenize in a subprocess with NLTK_DATA set to a fresh temporary directory, which proves the downloads the module makes are the ones word_tokenize needs rather than relying on whatever is cached in the sandbox's home directory. It runs that test under the pinned release and under 3.9.1, and the pin in requirements.txt moves to a release where both the download and the tokenizer were checked. main.py, which needs an OPENAI_API_KEY, is not run; nothing in the fix touches it.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "This project fails with LookupError: Resource punkt_tab not found when word_tokenize runs. Find every nltk.download call and every tokenizer call (word_tokenize, sent_tokenize) with file and line, and note which run at import time. Reproduce the error in a virtual environment with nltk 3.9.1 and NLTK_DATA set to an empty temporary directory, calling the project's own tokenizing function; then repeat on the release pinned in requirements.txt and on the current nltk release, and record for each whether the project's download call is enough. Fix the download so the resource word_tokenize needs is fetched, and move downloads out of import time into a function that checks nltk.data.find first and downloads quietly only when missing. Write a test that runs the tokenizing function in a subprocess with NLTK_DATA pointed at a fresh empty directory. Update the nltk pin in requirements.txt to a release you verified. Do not run the main script and do not use an OpenAI key. Open a pull request with the version table, the traceback, and the test output.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
A pull request with a small diff in utils.py, a new test, and a changed pin in requirements.txt. Its description is led by a table of NLTK releases the agent tried in the sandbox and, for each, whether download("punkt") alone let word_tokenize run, followed by the traceback it captured, so a reader who only skims learns that the failure is a 3.9 behaviour and not a broken machine.
The session's record shows each virtual environment being created, the empty data directory, the failing call, and the test passing on the releases it was run against. Because the test runs in a subprocess with its own NLTK_DATA, the transcript shows the downloads happening inside the test, which is the evidence that the fix is the download and not a leftover file.
Before merging, a reviewer checks that the download now runs where the team wants it: at first use, as the pull request does, or in an image build step with NLTK_DATA baked in, which the description offers as the alternative for deployments with no outbound network. The reviewer also confirms the new pin is one the rest of the pinned stack installs beside.
Run it on every exception captured in production
A tokenizer that fails on first use is the kind of error an error tracker catches once per deployment. Sentry signs its own requests and cannot add the bearer header a Rush webhook trigger needs, so a small handler on your server receives Sentry's issue webhook and forwards the payload to the trigger's URL with Authorization: Bearer <its secret>, sending the Sentry issue id as the Idempotency-Key of that trigger call so a redelivered alert starts no second run. The payload reaches the agent after the prompt, as data.
From a script instead, one POST /o/{org}/sessions per affected repository carries this request, the repository in resources, and a request_id that is a UUID v5 of the issue id. A webhook set on the API key reports run.completed or run.failed for each run that key started, with the session_url to read the pull request from.
The use case this job follows, with its automation and trigger: Fix a bug from an alert.
Why run it on Rush?
The error depends on what is already in a data directory, which is why it is hard to reproduce on a developer's machine: a laptop that has had NLTK for years has punkt_tab from some earlier project, and the bug hides. A sandbox starts empty. The agent controls NLTK_DATA for every run it makes, so the version table in the pull request comes from clean environments, not from one machine's cache.
It also keeps the fix proportionate. A project like this one has a hundred pins and a model dependency that takes a while to install; the agent installs only NLTK for the reproduction, runs the test matrix in minutes, and leaves the record of every command with the pull request. Several tools forked from the same tutorial get the same request, one run each, in parallel up to the key's max_concurrent limit.
Frequently asked questions
Why did nltk.download("punkt") stop being enough?
Because NLTK 3.9 loads the Punkt model from tokenizers/punkt_tab/<language>/ and punkt holds the older files. In a clean directory on 3.9.1, downloading punkt succeeds and word_tokenize still raises; downloading punkt_tab makes it pass. On 3.10.4 the punkt download brings punkt_tab with it.
Will downloading punkt_tab break NLTK 3.8?
No. On 3.8.1 in a clean directory nltk.download("punkt_tab") returns True and word_tokenize runs, since the downloader index is shared across releases. The agent keeps the punkt download for the older code path and adds punkt_tab, and runs the test on the pinned release and on 3.9.1.
Which tokenizers need the resource?
word_tokenize calls sent_tokenize first, and that is where Punkt loads, so both need punkt_tab. A regular-expression tokenizer such as wordpunct_tokenize loads no resource and runs in an empty data directory. The agent lists which the project uses before changing anything.
Does the fix need network access at runtime?
At first use, yes: the download runs when a resource is missing. For a deployment without outbound network, the pull request describes the alternative of downloading during the image build and pointing NLTK_DATA at that directory, which the test's subprocess already exercises.
More jobs like this: Fix a bug from an alert
- Fix "ImportError: cannot import name 'SafeConfigParser' from 'configparser'" on Python 3.12 (configparser)
- Fix "DEPRECATION WARNING [legacy-js-api]: The legacy JS API is deprecated and will be removed in Dart Sass 2.0.0" (Sass)
- Fix a Scrapy spider whose start_requests() is never called and scrapes 0 items (Scrapy)
- Fix "distplot is a deprecated function and will be removed in seaborn v0.14.0" (seaborn)
- Fix Actix Web Errors from Sentry Alerts (Actix Web)
- Turn Sentry Errors Into Fix Pull Requests for an aiohttp App (aiohttp)