# Add HTTP Caching to a Scrapy Project

Yes. An agent enables Scrapy's HttpCacheMiddleware with a filesystem store and an expiry, proves a second crawl makes no requests, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-http-caching/scrapy/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads the project before it enables anything, because the cache middleware has settings that depend on how the spiders are run. In a project laid out like scrapy/quotesbot, `scrapy.cfg` points at `quotesbot.settings`, which sets `BOT_NAME`, the spider modules and `ROBOTSTXT_OBEY = True`, and leaves every other key commented out, including the five `HTTPCACHE_*` lines the `startproject` template ships. Two spiders in `quotesbot/spiders/`, `toscrape-css` and `toscrape-xpath`, each start at `http://quotes.toscrape.com/`, yield a dict per `div.quote` and follow `li.next > a` until the last page. `pipelines.py` is the empty template class, `items.py` defines no fields, and there is no test directory and no requirements file. `.gitignore` already lists `.scrapy`, which is where the project-level cache directory lands.

The change is in `settings.py`: `HTTPCACHE_ENABLED = True`, `HTTPCACHE_DIR = 'httpcache'`, which Scrapy resolves under the `.scrapy` project directory so the ignore rule already covers it, `HTTPCACHE_EXPIRATION_SECS` set to a day rather than the template's `0`, which means never expire, and `HTTPCACHE_IGNORE_HTTP_CODES = [500, 502, 503, 504]` so an outage page is not served for a day. The storage stays `FilesystemCacheStorage`, with `HTTPCACHE_GZIP = True` added since the pages are text. The policy stays the default `DummyPolicy`, which serves everything it has: the site sets no cache headers, so `RFC2616Policy` would cache nothing, and the pull request says that in one sentence.

Two things come from reading the spiders rather than the settings reference. Both spiders fetch the same ten pages, so one cache serves both; the agent notes that `HTTPCACHE_DIR` is shared per project, not per spider, which is what you want here. And `ROBOTSTXT_OBEY` makes a first request for `robots.txt` on every run; that request goes through the same middleware and is cached too, which the second run's stats confirm.

In its sandbox the agent installs Scrapy, runs `scrapy crawl toscrape-css -o quotes.json` once and reads the closing stats: `downloader/request_count` of 11 and `httpcache/miss` of 11. It runs the same crawl again and reads `httpcache/hit` of 11 and no `downloader/request_count` at all, with the two output files identical. It then runs `toscrape-xpath` and shows it hits the cache the first spider filled. Finally it sets `HTTPCACHE_EXPIRATION_SECS` to 1 for one run to show a miss after expiry, restores the value, and quotes all four stats blocks. A `tests/` directory is not added, since the project has none, but the four runs are the evidence.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add HTTP caching to this Scrapy project. Read scrapy.cfg, quotesbot/settings.py, both spiders in quotesbot/spiders, quotesbot/pipelines.py and .gitignore first. In settings.py enable HttpCacheMiddleware with the filesystem storage, a one-day expiry, gzip, and ignore 5xx responses; keep the default policy and say why RFC2616Policy would not cache this site. Confirm the cache directory is already ignored by git. Install Scrapy, run scrapy crawl toscrape-css -o quotes.json twice and quote the httpcache and downloader stats from both runs, run toscrape-xpath once to show it hits the same cache, then run once with a 1 second expiry to show a miss and restore the setting. Open a pull request quoting the stats.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request with one file changed, `quotesbot/settings.py`: the five cache keys uncommented and set, and two added, all in the block the template already reserved for them. No spider changes, no new dependency, no `.gitignore` change because `.scrapy` was already there. The description quotes the stats of four crawls: eleven misses, then eleven hits and no downloads, the second spider hitting the first's cache, and one miss after a one-second expiry.

The description also says what the cache does not do: it does not respect the site's headers, because the default policy ignores them and the site sends none, and it does not cache a 5xx page, because of the ignore list.

Before merging, a reviewer sets the expiry to the real freshness the data needs, a day being the agent's default rather than a measurement, and decides whether a production deploy should keep the cache on or leave it for development, which is one `HTTPCACHE_ENABLED` override in the deploy's settings.

## Run it on every nightly

The cache makes repeated runs cheap, which is what a scheduled crawl wants. A `schedule` trigger with `cron` set to a quiet hour in the trigger's `timezone` starts a run that executes `scrapy crawl toscrape-css -o quotes.json` and compares the output with the previous night's, publishing a diff of added and removed quotes as the result and nothing when nothing changed.

Each run starts in a fresh sandbox, so the cache begins empty each night and the expiry setting never masks a change on the site; the run's stats block, kept in its record, shows the request count a reviewer expects to see.

## Why run it on Rush?

Turning on the cache is uncommenting five lines, and getting it wrong is as easy: an expiry of zero caches an outage page forever, and a policy chosen from the docs caches nothing on a site without headers. In a Rush sandbox the agent reads the spiders, sets the values for this site, and proves the behaviour with the middleware's own stats rather than describing it.

The record of the run keeps every crawl's stats, and the pull request quotes them, so the reviewer sees eleven hits and no downloads rather than a settings diff. Across every Scrapy project a team keeps, the same request runs in parallel, one sandbox and one pull request each.

## Frequently asked questions

### Why not RFC2616Policy?

That policy honours Cache-Control and Expires headers, and quotes.toscrape.com sends none, so it would store nothing. The default DummyPolicy serves every stored response until HTTPCACHE_EXPIRATION_SECS passes, which is what a development crawl wants. The pull request says so, and switching is one setting if the target site sends headers.

### Where does the cache live, and is it committed?

Under the project's .scrapy directory, as .scrapy/httpcache, because HTTPCACHE_DIR is relative to the project data directory. The repository's .gitignore already lists .scrapy, which the agent checks and quotes, so nothing in the pull request changes what git tracks.

### Will a server error be cached for a day?

No. HTTPCACHE_IGNORE_HTTP_CODES lists 500, 502, 503 and 504, so those responses are passed through and never stored. A 404 is stored, since a missing page is a result the next run can reuse.

### Do the two spiders share the cache?

Yes. The cache is keyed by request fingerprint within the project directory, not per spider, and both spiders request the same ten pages plus robots.txt. The pull request quotes toscrape-xpath hitting on its first run after toscrape-css filled the cache.

## More Scrapy jobs

- [Scrape JavaScript-Rendered Pages with Scrapy and Playwright](https://getrush.ai/run/add-javascript-rendering/scrapy/)
- [Fix a Scrapy Spider That Stopped Returning Items](https://getrush.ai/run/fix-broken-spiders/scrapy/)
- [Fix a Scrapy spider whose start_requests() is never called and scrapes 0 items](https://getrush.ai/run/fix-scrapy-start-requests-not-called/scrapy/)

## More jobs like this: Write the missing tests

- [Add Integration Tests to a Flutter App](https://getrush.ai/run/add-integration-tests/flutter/)
- [Add a Locust Load Test to CI](https://getrush.ai/run/add-load-test-to-ci/locust/)
- [Add a Migration Check to CI for a TypeORM Project](https://getrush.ai/run/add-migration-check-to-ci/typeorm/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
