What the agent does
The agent starts with what the crawler sends today. In a project laid out like scrapy/quotesbot, quotesbot/settings.py has four live lines, BOT_NAME, SPIDER_MODULES, NEWSPIDER_MODULE and ROBOTSTXT_OBEY = True, and everything else, including DOWNLOAD_DELAY, CONCURRENT_REQUESTS_PER_DOMAIN and the whole AUTOTHROTTLE_* block, is a commented default from the template. So both spiders under quotesbot/spiders/, toscrape-css and toscrape-xpath, run at Scrapy's defaults: sixteen concurrent requests, eight per domain, no delay, following li.next through every page of quotes.toscrape.com as fast as the site answers. There is no requirements.txt, no custom_settings on either spider, and the README's only run instruction is scrapy crawl toscrape-css.
It enables the extension and sets the three numbers that matter. AUTOTHROTTLE_ENABLED = True, AUTOTHROTTLE_START_DELAY = 1.0 so the first requests are polite before any latency has been measured, AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0 so the throttle aims for two requests in flight per server, and AUTOTHROTTLE_MAX_DELAY = 30.0 so a slow or failing site cannot push the delay past half a minute. DOWNLOAD_DELAY = 0.5 is set too, because AutoThrottle treats it as the floor the delay never goes under, and CONCURRENT_REQUESTS_PER_DOMAIN = 4 stays as the ceiling the throttle cannot exceed. The agent writes a comment above each line saying which of the two it is, floor or cap, since the interaction is the part people get wrong. AUTOTHROTTLE_DEBUG is left off in settings and turned on from the command line for the proof run.
The spiders are untouched. AutoThrottle is a downloader extension keyed on the download slot, so both spiders pick it up from settings, and a per-spider override, if one is ever needed, is a custom_settings dict the agent mentions in the README rather than adding. A requirements.txt pins the Scrapy release the proof ran on, because the extension's TARGET_CONCURRENCY setting replaced the older AUTOTHROTTLE_START_DELAY-only behaviour in Scrapy 1.1 and the project never said which version it expected.
In its sandbox the agent installs the pinned Scrapy, runs scrapy list, then scrapy crawl toscrape-css -s AUTOTHROTTLE_DEBUG=True -s CLOSESPIDER_PAGECOUNT=10, and reads the [scrapy.extensions.throttle] lines that print the slot, the measured latency and the new delay after each response. It pastes a few of those lines and the closing stats, downloader/request_count, elapsed_time_seconds and finish_reason, into the pull request, next to the same crawl's stats without the extension for comparison.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add AutoThrottle to this Scrapy project. Read scrapy.cfg, quotesbot/settings.py, both spiders under quotesbot/spiders/ and the README first. In settings.py set AUTOTHROTTLE_ENABLED True, AUTOTHROTTLE_START_DELAY 1.0, AUTOTHROTTLE_TARGET_CONCURRENCY 2.0 and AUTOTHROTTLE_MAX_DELAY 30.0, set DOWNLOAD_DELAY 0.5 as the floor and CONCURRENT_REQUESTS_PER_DOMAIN 4 as the cap, with a comment on each saying which it is. Leave AUTOTHROTTLE_DEBUG off in settings and do not change the spiders. Add a requirements.txt pinning the Scrapy you install. Run scrapy list, then one crawl of toscrape-css with AUTOTHROTTLE_DEBUG=True and CLOSESPIDER_PAGECOUNT=10, and the same crawl with AUTOTHROTTLE_ENABLED=False, and quote the throttle debug lines and both crawls' request count, elapsed time and finish_reason in the pull request. Add a short README section on the settings. Open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request touching quotesbot/settings.py, README.md and a new requirements.txt. The description quotes the scrapy list output, five or six [scrapy.extensions.throttle] debug lines showing the delay moving with the measured latency, and a two-row table of the throttled and unthrottled crawls with downloader/request_count, elapsed_time_seconds and finish_reason from each closing stats block.
The run's record keeps both crawl logs in full, so a reviewer can see every page request, the robots.txt fetch and the moment closespider stopped each run at ten pages. Nothing hit a production target: quotes.toscrape.com is built for scraping practice.
Before merging, a reviewer picks the numbers for the real sites. A target concurrency of 2.0 and a 30 s cap suit a small site; a large one with an explicit rate limit in its terms gets a lower target, and a site behind a CDN that answers fast gets a higher floor so the throttle does not speed up to the cap.
Run it on every Linear issue labelled throttle
A team that keeps several Scrapy projects turns this into a rule. In the Project for each repository, an automation with the prompt above and a linear trigger on linear.issue.labeled for the label throttle starts a run whenever someone files an issue like "the catalogue spider is getting 429s from the supplier site". The trigger adds the issue after the prompt, so the run reads which spider and which site the issue names and sets the numbers for that domain.
Each run opens its own pull request on its own branch and links the issue. On a repository that already has AutoThrottle enabled, the run adjusts the target concurrency or the cap for the named site and quotes the debug lines that justify the new value.
The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.
Why run it on Rush?
AutoThrottle is five settings, and the mistake is in the two it interacts with: a DOWNLOAD_DELAY left at zero gives the throttle no floor, a concurrency cap left at eight lets it climb to a rate the site blocks, and the only way to see the delay actually adapting is to run a crawl with debug on and read the throttle lines. On a laptop that is a virtualenv and a few minutes against a live site. In a Rush sandbox the agent does exactly that and the pull request quotes the lines.
The two crawl logs stay with the run, so the next person who tunes the numbers can see what the delay did under the previous settings. Across a dozen spider repositories the job runs in parallel, one sandbox each, with each pull request carrying its own before-and-after.
Frequently asked questions
Will the crawl take much longer?
Somewhat, and that is the point: the throttle starts at one second between requests and settles on whatever keeps two requests in flight per server. The pull request's table shows the elapsed time for ten pages with and without it, so the cost is a number rather than a guess.
Why set DOWNLOAD_DELAY if AutoThrottle adapts the delay?
Because AutoThrottle never goes below DOWNLOAD_DELAY; with it at zero the throttle can speed up to no delay at all on a fast site. Half a second is the floor the pull request sets, and the comment above it says so.
Does the debug output end up in production logs?
No. AUTOTHROTTLE_DEBUG stays off in settings.py and is passed with -s only for the proof crawl. Turn it on the same way when tuning a new site.
Can one spider have different throttle settings?
Yes. A custom_settings dict on the spider class overrides any of these for that spider alone, and the README section the agent writes shows the shape. Both spiders here crawl the same site, so the project-wide settings are enough.
What about sites that return 429 or 503?
AutoThrottle only measures latency; it does not slow down for a status code. Scrapy's retry middleware backs off on those, and if the prompt asks for it the agent also sets RETRY_HTTP_CODES and a higher AUTOTHROTTLE_MAX_DELAY for the site that sends them.
More Scrapy jobs
More jobs like this: Turn a Linear label into a PR
- Add Background Jobs to a Litestar App (Litestar)
- Add Oban Background Jobs to a Phoenix LiveView App (Phoenix LiveView)
- Add Background Jobs to a Vapor App from a Linear Issue (Vapor)