All jobs

Write Tests for Scrapy Spiders

Yes. The agent saves real pages as fixtures, tests each spider's parse callback offline with pytest, and adds contracts that scrapy check runs on the live site.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

A spider is a function from a page to items and more requests, which makes it easier to test than most scraping code admits. The agent starts by reading what there is. In a Scrapy project like scrapy/quotesbot, scrapy.cfg points at quotesbot.settings, and quotesbot/spiders/ holds two spiders, toscrape-css and toscrape-xpath. Each has one parse callback that yields a dict with text, author and tags for every div.quote on the page and then follows the link inside li.next. There is no test directory, no dependency file that names a Scrapy version, and scrapy check reports that it ran 0 contracts. The agent installs Scrapy and pytest, writes down the versions it got, and runs scrapy list to confirm both spiders load.

The offline tests come first. The agent fetches the first and the last listing page once and saves them under tests/fixtures/. A test reads a saved file, wraps the bytes in an HtmlResponse with the page's URL and a Request, calls parse on a spider instance and sorts what comes out into items and requests. For the first page it asserts the number of items, the fields of the first one including its list of tags, and that the only request goes to /page/2/ on the same host. For the last page it asserts that no request is yielded, which is the case that decides whether a crawl ever stops. The README says both spiders extract the same data, so one more test feeds the same fixture to both and requires equal items.

Two details of this repository shape the code. The spider files are named toscrape-css.py and toscrape-xpath.py, and a hyphen is not legal in an import statement, so the tests load each class through Scrapy's SpiderLoader with the project settings, the same way scrapy crawl finds them. And the callbacks use extract_first(), which returns None when nothing matches. The agent writes small hand-made fixtures for a quote with no tags and a quote with no author, and pins what the code does today: an empty list in the first case and an item whose author is None in the second. It flags the second in the pull request and leaves the decision to the team.

Last come contracts, Scrapy's own check against the live site. The agent adds a docstring to each parse with @url for the start page, @returns items and @returns requests with bounds, and @scrapes text author tags. scrapy check then fetches the page and exits non-zero when a contract fails. These need the network and the offline tests do not, so they stay separate: pytest for every pull request, contracts for a schedule.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Write tests for the spiders in this Scrapy project. Read scrapy.cfg, the settings module, items.py, pipelines.py and every file under the spiders package first, then run scrapy list. For each spider, fetch its start page and one page from the end of its pagination once, save them under tests/fixtures, and write pytest tests that build an HtmlResponse from the saved bytes and call the spider's callback directly: assert the number of items, the fields of the first item, the URL of each follow-up request, and that the last page yields no request. Load spider classes through scrapy.spiderloader.SpiderLoader with the project settings, since a spider module's file name may not be importable. Where two spiders scrape the same site, add a test that they return equal items from the same fixture. Add hand-made fixtures for a record with a missing field and one with an empty list field, and pin what the code does today; if that behaviour looks wrong, keep the test and list it in the pull request without changing the spider. Then add contracts to each callback's docstring (@url, @returns items, @returns requests, @scrapes) and run scrapy check. The pytest suite must make no network request. If the repository pins no Scrapy version, add a requirements file with the versions you ran. Do not change selectors or settings. Open a pull request with the output of pytest and scrapy check.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that adds a tests/ directory with the saved pages and the test module, a docstring with contracts on each callback, and a requirements file when the project had none. No selector and no setting changes. The description lists each test with the callback and the case it covers, and pastes the output of pytest and of scrapy check.

A short second list holds behaviour the tests pinned that someone may want changed, such as an item that is still yielded when its author is missing. The agent tests what the code does and marks it; whether a half-empty item should be dropped in a pipeline is the project's call.

A reviewer opens the saved pages and checks that they are the listing pages the spider is meant for and not an error page or a consent wall, since a fixture of the wrong page makes every assertion meaningless. They also check the bounds in @returns: a contract that allows zero items passes on a broken selector.

Run it on every pull request labelled spider

An automation in the Project that holds the repository, with a github trigger on pull_request.labeled and spider in its labels, starts one run when a maintainer adds that label to a teammate's pull request that brings a new spider. Its prompt asks for fixtures, offline tests and contracts for the spiders the pull request adds. The trigger puts the event after the prompt and fills in nothing, and the run opens its own pull request with the tests.

To cover a folder of scraping projects in one pass, a script on your server sends this body to POST /o/{org}/sessions once per repository, with a request_id that is a UUID v5 of the repository's full name so a retry starts nothing new. A webhook_url on the key receives run.completed with session_url for each run the key started.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

Writing these tests means fetching pages from a site you do not own, and doing it from a laptop mixes that traffic with everything else on your connection. The run has its own sandbox: it fetches each page once, saves it, and from then on the suite runs against files. Every command and its output is kept in the transcript at GET /o/{org}/sessions/{id}/messages, so a reviewer can see each page fetched once and the suite then pass from the saved files.

It also scales the dull way. A team with forty spiders in ten repositories can start ten runs at once, each in its own sandbox with its own checkout, and get ten pull requests whose tests were executed before anyone read them.

Frequently asked questions

Do the tests hit the website every time they run?

No. The pytest suite reads saved HTML and builds responses in memory, so it runs with no network. Only the contracts fetch the live page, and those run through scrapy check, which you can put on a schedule instead of on every pull request.

What is the difference between a contract and a pytest test here?

A contract lives in the callback's docstring and checks the live site: this URL returns between this many items with these fields. A pytest test checks your code against a page that never changes. The first tells you the site moved; the second tells you a refactor broke the spider.

My spiders yield Item classes, not dicts. Does that change anything?

The approach is the same. The tests compare fields through itemadapter so dicts, Item subclasses and dataclasses read alike. In scrapy/quotesbot the spiders yield plain dicts and the QuotesbotItem class in items.py is empty, so the tests there compare dicts.

Will it test pipelines and middlewares too?

Only if you ask, and only those that are enabled. In scrapy/quotesbot ITEM_PIPELINES is commented out in settings.py, so QuotesbotPipeline never runs and the agent leaves it untested and says so. A pipeline that is switched on gets a unit test that calls process_item with an item and checks what comes back.

More Scrapy jobs

More jobs like this: Write the missing tests

Keep reading