What the agent does
The agent reads the project before it enables anything, because the cache middleware has settings that depend on how the spiders are run. In a project laid out like scrapy/quotesbot, scrapy.cfg points at quotesbot.settings, which sets BOT_NAME, the spider modules and ROBOTSTXT_OBEY = True, and leaves every other key commented out, including the five HTTPCACHE_* lines the startproject template ships. Two spiders in quotesbot/spiders/, toscrape-css and toscrape-xpath, each start at http://quotes.toscrape.com/, yield a dict per div.quote and follow li.next > a until the last page. pipelines.py is the empty template class, items.py defines no fields, and there is no test directory and no requirements file. .gitignore already lists .scrapy, which is where the project-level cache directory lands.
The change is in settings.py: HTTPCACHE_ENABLED = True, HTTPCACHE_DIR = 'httpcache', which Scrapy resolves under the .scrapy project directory so the ignore rule already covers it, HTTPCACHE_EXPIRATION_SECS set to a day rather than the template's 0, which means never expire, and HTTPCACHE_IGNORE_HTTP_CODES = [500, 502, 503, 504] so an outage page is not served for a day. The storage stays FilesystemCacheStorage, with HTTPCACHE_GZIP = True added since the pages are text. The policy stays the default DummyPolicy, which serves everything it has: the site sets no cache headers, so RFC2616Policy would cache nothing, and the pull request says that in one sentence.
Two things come from reading the spiders rather than the settings reference. Both spiders fetch the same ten pages, so one cache serves both; the agent notes that HTTPCACHE_DIR is shared per project, not per spider, which is what you want here. And ROBOTSTXT_OBEY makes a first request for robots.txt on every run; that request goes through the same middleware and is cached too, which the second run's stats confirm.
In its sandbox the agent installs Scrapy, runs scrapy crawl toscrape-css -o quotes.json once and reads the closing stats: downloader/request_count of 11 and httpcache/miss of 11. It runs the same crawl again and reads httpcache/hit of 11 and no downloader/request_count at all, with the two output files identical. It then runs toscrape-xpath and shows it hits the cache the first spider filled. Finally it sets HTTPCACHE_EXPIRATION_SECS to 1 for one run to show a miss after expiry, restores the value, and quotes all four stats blocks. A tests/ directory is not added, since the project has none, but the four runs are the evidence.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add HTTP caching to this Scrapy project. Read scrapy.cfg, quotesbot/settings.py, both spiders in quotesbot/spiders, quotesbot/pipelines.py and .gitignore first. In settings.py enable HttpCacheMiddleware with the filesystem storage, a one-day expiry, gzip, and ignore 5xx responses; keep the default policy and say why RFC2616Policy would not cache this site. Confirm the cache directory is already ignored by git. Install Scrapy, run scrapy crawl toscrape-css -o quotes.json twice and quote the httpcache and downloader stats from both runs, run toscrape-xpath once to show it hits the same cache, then run once with a 1 second expiry to show a miss and restore the setting. Open a pull request quoting the stats.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request with one file changed, quotesbot/settings.py: the five cache keys uncommented and set, and two added, all in the block the template already reserved for them. No spider changes, no new dependency, no .gitignore change because .scrapy was already there. The description quotes the stats of four crawls: eleven misses, then eleven hits and no downloads, the second spider hitting the first's cache, and one miss after a one-second expiry.
The description also says what the cache does not do: it does not respect the site's headers, because the default policy ignores them and the site sends none, and it does not cache a 5xx page, because of the ignore list.
Before merging, a reviewer sets the expiry to the real freshness the data needs, a day being the agent's default rather than a measurement, and decides whether a production deploy should keep the cache on or leave it for development, which is one HTTPCACHE_ENABLED override in the deploy's settings.
Run it on every nightly
The cache makes repeated runs cheap, which is what a scheduled crawl wants. A schedule trigger with cron set to a quiet hour in the trigger's timezone starts a run that executes scrapy crawl toscrape-css -o quotes.json and compares the output with the previous night's, publishing a diff of added and removed quotes as the result and nothing when nothing changed.
Each run starts in a fresh sandbox, so the cache begins empty each night and the expiry setting never masks a change on the site; the run's stats block, kept in its record, shows the request count a reviewer expects to see.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
Turning on the cache is uncommenting five lines, and getting it wrong is as easy: an expiry of zero caches an outage page forever, and a policy chosen from the docs caches nothing on a site without headers. In a Rush sandbox the agent reads the spiders, sets the values for this site, and proves the behaviour with the middleware's own stats rather than describing it.
The record of the run keeps every crawl's stats, and the pull request quotes them, so the reviewer sees eleven hits and no downloads rather than a settings diff. Across every Scrapy project a team keeps, the same request runs in parallel, one sandbox and one pull request each.
Frequently asked questions
Why not RFC2616Policy?
That policy honours Cache-Control and Expires headers, and quotes.toscrape.com sends none, so it would store nothing. The default DummyPolicy serves every stored response until HTTPCACHE_EXPIRATION_SECS passes, which is what a development crawl wants. The pull request says so, and switching is one setting if the target site sends headers.
Where does the cache live, and is it committed?
Under the project's .scrapy directory, as .scrapy/httpcache, because HTTPCACHE_DIR is relative to the project data directory. The repository's .gitignore already lists .scrapy, which the agent checks and quotes, so nothing in the pull request changes what git tracks.
Will a server error be cached for a day?
No. HTTPCACHE_IGNORE_HTTP_CODES lists 500, 502, 503 and 504, so those responses are passed through and never stored. A 404 is stored, since a missing page is a result the next run can reuse.
Do the two spiders share the cache?
Yes. The cache is keyed by request fingerprint within the project directory, not per spider, and both spiders request the same ten pages plus robots.txt. The pull request quotes toscrape-xpath hitting on its first run after toscrape-css filled the cache.
More Scrapy jobs
More jobs like this: Write the missing tests
- Add Integration Tests to a Flutter App (Flutter)
- Add a Locust Load Test to CI (Locust)
- Add a Migration Check to CI for a TypeORM Project (TypeORM)