All jobs

Fix a Scrapy spider whose start_requests() is never called and scrapes 0 items

Yes. An agent finds the Scrapy release that stopped calling start_requests(), adds the async start() method, and shows the spider making its first request again.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the spider and how it is launched before installing anything, because a crawl that ends finished with zero items looks like a site problem until the versions are checked. In a data project like turicas/socios-brasil, requirements.txt lists scrapy with no version, and run.sh has an extract_cnae function that runs scrapy runspider six times, once per CNAE table version from 1.0 to 2.3, with -s HTTPCACHE_ENABLED=true, a RETRY_HTTP_CODES list, -a versao=... and -o to a CSV. cnae.py defines CNAESpider, whose __init__ raises ValueError without versao, and whose start_requests returns a one-element list: scrapy.Request(url=self.versoes[self.versao], callback=self.parse). Everything else is parse, parse_items and parse_subclasse recursing through the IBGE hierarchy.

Then it reproduces with a clean install, which today resolves Scrapy 2.20.0 on Python 3.11 with Twisted 26.10.0 and lxml 6.1.3. Running the spider with -a versao=2.3 -s CLOSESPIDER_PAGECOUNT=1 ends with finish_reason: finished, no downloader/ statistics at all, zero items and no warning about the spider. Grepping the installed package for start_requests finds nothing: the method is simply never looked up. The only deprecation Scrapy 2.20 prints is about the FEED_MODE setting's default changing in a future version.

To date the change the agent installs an earlier release. Scrapy 2.15.2 runs the same command and prints a ScrapyDeprecationWarning naming cnae.CNAESpider: start_requests() has been deprecated in favour of a new method, start(), to support asynchronous code execution, will stop being called in a future version, and spiders that need older Scrapy should override both. That run makes one request, which the IBGE site answers with 403 for Scrapy's default user agent, so the stats show downloader/request_count: 1. Scrapy's release notes for 2.16.0 (2026-05-19) list the removal under deprecation removals, referencing pull request 7490: start_requests() is removed and no longer called.

The fix keeps the spider working on both sides of 2.16: add async def start(self) that yields the same scrapy.Request, and keep start_requests for Scrapy below 2.13. With that, 2.20.0 and 2.15.2 both show downloader/request_count: 1 and closespider_pagecount as the finish reason, and 2.15.2 no longer prints the deprecation. The 403 itself is a separate finding the agent reports with the response status rather than hiding it.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Our Scrapy spider finishes immediately with zero items and no requests since dependencies were reinstalled. Read the spider and the script that launches it, say which Scrapy version a clean install resolves today, and check whether that release still calls start_requests(). If it does not, find the release notes that removed it, add an async start() method that yields the same initial requests while keeping start_requests() for older Scrapy, and run the spider with a one-page limit before and after, quoting the downloader statistics. Open a pull request with the change, the versions, and any other deprecation the run prints.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that adds three lines to the spider: an async def start(self) yielding the version-specific request, above the existing start_requests. The description quotes the 2.20.0 run with no downloader statistics, the 2.15.2 deprecation warning that names the class, the 2.16.0 release note that removed the method, and the after-fix statistics showing one request on each version.

A reviewer checks that start() yields exactly what start_requests returned, that the versao argument is still required, and that nothing in parse_items or parse_subclasse changed. They also see the 403 from the target site reported as a separate item, with the note that a USER_AGENT setting is a decision for the maintainers, not part of this fix.

The description lists the FEED_MODE deprecation 2.20 prints as a second, unrelated finding and leaves requirements.txt unpinned unless the team asks for a floor such as scrapy>=2.13.

Run it on every scheduled crawl that reports zero items

A crawl that runs on a schedule is the natural place to catch this. An automation with a schedule trigger runs the spider with a page limit in the Project's repository and checks the item_scraped_count and downloader/request_count statistics; when both are zero after a dependency change, it opens the pull request with the versions and the fix.

A pipeline that already runs the crawl can post to a Rush webhook trigger when the item count drops to zero, with Authorization: Bearer and the trigger's secret, and the payload with the Scrapy version and statistics follows the prompt as data; a run starts per alert.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

A silently removed method leaves no error to search for: the crawl ends finished, the log is clean, and the output file is empty. In a Rush sandbox the agent installs the current release and an older one side by side, runs the same command under both, greps the installed package, and reads the project's release notes, so the pull request shows the exact release where the behaviour changed instead of guessing at the site.

The fix must keep the spider running under whatever Scrapy the next machine installs. The agent proves the request fires on 2.20.0 and 2.15.2 and keeps the old method for releases before 2.13, so the reviewer reads a three-line change with the statistics from both runs.

Frequently asked questions

Why is there no error or warning in Scrapy 2.20?

Because the method was removed, not deprecated, in 2.16.0: the engine calls start() and never looks for start_requests(). The deprecation warning existed from 2.13 to 2.15, so a project that jumped straight to a current release never saw it.

Do I need to keep start_requests() at all?

Only if the spider must run on Scrapy older than 2.13, where start() does not exist. The pull request keeps both, as the 2.13 warning text recommends; a team that pins scrapy>=2.13 can delete the old method.

Can start() yield exactly what start_requests returned?

Yes. start() is an async generator and yields scrapy.Request objects, or items, one at a time. This spider's one request with its callback=self.parse moves across unchanged.

Why does the test run show a 403?

The IBGE site refuses Scrapy's default user agent. That status proves the request was made, which is the point of the check, and the agent reports it rather than changing the USER_AGENT setting in a pull request about the start method.

More Scrapy jobs

More jobs like this: Fix a bug from an alert

Keep reading