# Fix a Scrapy spider whose start_requests() is never called and scrapes 0 items

Yes. An agent finds the Scrapy release that stopped calling start_requests(), adds the async start() method, and shows the spider making its first request again.

[Run this on Rush](https://cloud.getrush.ai/run/fix-scrapy-start-requests-not-called/scrapy/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads the spider and how it is launched before installing anything, because a crawl that ends `finished` with zero items looks like a site problem until the versions are checked. In a data project like turicas/socios-brasil, `requirements.txt` lists `scrapy` with no version, and `run.sh` has an `extract_cnae` function that runs `scrapy runspider` six times, once per CNAE table version from 1.0 to 2.3, with `-s HTTPCACHE_ENABLED=true`, a `RETRY_HTTP_CODES` list, `-a versao=...` and `-o` to a CSV. `cnae.py` defines `CNAESpider`, whose `__init__` raises `ValueError` without `versao`, and whose `start_requests` returns a one-element list: `scrapy.Request(url=self.versoes[self.versao], callback=self.parse)`. Everything else is `parse`, `parse_items` and `parse_subclasse` recursing through the IBGE hierarchy.

Then it reproduces with a clean install, which today resolves Scrapy 2.20.0 on Python 3.11 with Twisted 26.10.0 and lxml 6.1.3. Running the spider with `-a versao=2.3 -s CLOSESPIDER_PAGECOUNT=1` ends with `finish_reason: finished`, no `downloader/` statistics at all, zero items and no warning about the spider. Grepping the installed package for `start_requests` finds nothing: the method is simply never looked up. The only deprecation Scrapy 2.20 prints is about the `FEED_MODE` setting's default changing in a future version.

To date the change the agent installs an earlier release. Scrapy 2.15.2 runs the same command and prints a `ScrapyDeprecationWarning` naming `cnae.CNAESpider`: `start_requests()` has been deprecated in favour of a new method, `start()`, to support asynchronous code execution, will stop being called in a future version, and spiders that need older Scrapy should override both. That run makes one request, which the IBGE site answers with 403 for Scrapy's default user agent, so the stats show `downloader/request_count: 1`. Scrapy's release notes for 2.16.0 (2026-05-19) list the removal under deprecation removals, referencing pull request 7490: `start_requests()` is removed and no longer called.

The fix keeps the spider working on both sides of 2.16: add `async def start(self)` that yields the same `scrapy.Request`, and keep `start_requests` for Scrapy below 2.13. With that, 2.20.0 and 2.15.2 both show `downloader/request_count: 1` and `closespider_pagecount` as the finish reason, and 2.15.2 no longer prints the deprecation. The 403 itself is a separate finding the agent reports with the response status rather than hiding it.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Our Scrapy spider finishes immediately with zero items and no requests since dependencies were reinstalled. Read the spider and the script that launches it, say which Scrapy version a clean install resolves today, and check whether that release still calls start_requests(). If it does not, find the release notes that removed it, add an async start() method that yields the same initial requests while keeping start_requests() for older Scrapy, and run the spider with a one-page limit before and after, quoting the downloader statistics. Open a pull request with the change, the versions, and any other deprecation the run prints.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

A pull request that adds three lines to the spider: an `async def start(self)` yielding the version-specific request, above the existing `start_requests`. The description quotes the 2.20.0 run with no downloader statistics, the 2.15.2 deprecation warning that names the class, the 2.16.0 release note that removed the method, and the after-fix statistics showing one request on each version.

A reviewer checks that `start()` yields exactly what `start_requests` returned, that the `versao` argument is still required, and that nothing in `parse_items` or `parse_subclasse` changed. They also see the 403 from the target site reported as a separate item, with the note that a `USER_AGENT` setting is a decision for the maintainers, not part of this fix.

The description lists the `FEED_MODE` deprecation 2.20 prints as a second, unrelated finding and leaves `requirements.txt` unpinned unless the team asks for a floor such as `scrapy>=2.13`.

## Run it on every scheduled crawl that reports zero items

A crawl that runs on a schedule is the natural place to catch this. An automation with a `schedule` trigger runs the spider with a page limit in the Project's repository and checks the `item_scraped_count` and `downloader/request_count` statistics; when both are zero after a dependency change, it opens the pull request with the versions and the fix.

A pipeline that already runs the crawl can post to a Rush `webhook` trigger when the item count drops to zero, with `Authorization: Bearer` and the trigger's secret, and the payload with the Scrapy version and statistics follows the prompt as data; a run starts per alert.

## Why run it on Rush?

A silently removed method leaves no error to search for: the crawl ends `finished`, the log is clean, and the output file is empty. In a Rush sandbox the agent installs the current release and an older one side by side, runs the same command under both, greps the installed package, and reads the project's release notes, so the pull request shows the exact release where the behaviour changed instead of guessing at the site.

The fix must keep the spider running under whatever Scrapy the next machine installs. The agent proves the request fires on 2.20.0 and 2.15.2 and keeps the old method for releases before 2.13, so the reviewer reads a three-line change with the statistics from both runs.

## Frequently asked questions

### Why is there no error or warning in Scrapy 2.20?

Because the method was removed, not deprecated, in 2.16.0: the engine calls `start()` and never looks for `start_requests()`. The deprecation warning existed from 2.13 to 2.15, so a project that jumped straight to a current release never saw it.

### Do I need to keep start_requests() at all?

Only if the spider must run on Scrapy older than 2.13, where `start()` does not exist. The pull request keeps both, as the 2.13 warning text recommends; a team that pins `scrapy>=2.13` can delete the old method.

### Can start() yield exactly what start_requests returned?

Yes. `start()` is an async generator and yields `scrapy.Request` objects, or items, one at a time. This spider's one request with its `callback=self.parse` moves across unchanged.

### Why does the test run show a 403?

The IBGE site refuses Scrapy's default user agent. That status proves the request was made, which is the point of the check, and the agent reports it rather than changing the `USER_AGENT` setting in a pull request about the start method.

## More Scrapy jobs

- [Save Scrapy Items to PostgreSQL](https://getrush.ai/run/save-items-to-postgres/scrapy/)
- [Upgrade a Scrapy Project to the Latest Scrapy](https://getrush.ai/run/upgrade-scrapy/scrapy/)
- [Write Tests for Scrapy Spiders](https://getrush.ai/run/write-spider-tests/scrapy/)

## More jobs like this: Fix a bug from an alert

- [Fix "distplot is a deprecated function and will be removed in seaborn v0.14.0"](https://getrush.ai/run/fix-seaborn-distplot-deprecated-warning/seaborn/)
- [Fix Actix Web Errors from Sentry Alerts](https://getrush.ai/run/fix-sentry-errors/actix/)
- [Turn Sentry Errors Into Fix Pull Requests for an aiohttp App](https://getrush.ai/run/fix-sentry-errors/aiohttp/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
