# Scrape JavaScript-Rendered Pages with Scrapy and Playwright

Yes. An agent confirms the data is rendered by JavaScript, adds scrapy-playwright for only the requests that need a browser, and proves it with item counts.

[Run this on Rush](https://cloud.getrush.ai/run/add-javascript-rendering/scrapy/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent starts by proving that JavaScript is the problem, because a browser is the most expensive fix for an empty crawl. In a project like scrapy/quotesbot, the spiders read `div.quote` elements from `http://quotes.toscrape.com/`. The same site has a variant under `/js/` where that selector matches nothing in the downloaded HTML: the quotes sit in a `var data = [...]` array inside an inline script that writes them into the page, while the `li.next` pagination link is in the static markup. A plain Scrapy spider pointed there follows every page, yields no items and logs no error. The agent shows this with `scrapy fetch` and a count of `div.quote` in the saved response, then opens the page in the sandbox's browser and counts the same selector in the rendered document.

With the cause confirmed, it adds scrapy-playwright, whose documentation asks for Python 3.10 or later, Scrapy 2.7 or later and Playwright 1.40 or later. The project pins nothing, so the agent writes a requirements file with the versions it installed and downloads one browser with `playwright install --with-deps chromium`. Two settings go into `quotesbot/settings.py`. `DOWNLOAD_HANDLERS` points at `scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler`, and for both schemes: the documentation says `https` is usually enough, and this project's start URLs are plain `http`, so a handler registered for `https` alone would never see them. `TWISTED_REACTOR` is set to the asyncio reactor, which projects created with Scrapy 2.7 or later already have and this older settings file does not.

The handler only renders requests that ask for it with `meta={"playwright": True}`, and the agent keeps it that way. It adds a spider for the rendered pages and leaves `toscrape-css` and `toscrape-xpath` on the fast path. The new spider cannot use `start_urls` alone, since those requests carry no meta, so it builds its first request itself, with a `PageMethod("wait_for_selector", "div.quote")` so the response is returned after the quotes exist. The follow-up request is where this goes wrong most often: a spider that yields `scrapy.Request(response.urljoin(next_page_url))` for the next page without the same meta gets page 1 rendered and every later page empty. The agent carries the meta forward and adds a `PLAYWRIGHT_ABORT_REQUEST` predicate that skips images, which the items do not need.

The proof is a pair of counts. The agent crawls the static site with an existing spider and the `/js/` variant with the new one, each with `-o` to a file, and compares `item_scraped_count`, the number of pages requested and the first item of each. It also runs the new spider once with the meta removed from the follow-up request to show the failure the fix prevents, and puts that number in the pull request. Offline tests save one rendered page and one raw page as fixtures and assert that the callback yields items from the first and none from the second. Because the data on this site is embedded as JSON in the page, the description also says how many lines a parser for that array would take and when it would be the better choice.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "codex",
  "input": "A crawl in this Scrapy project finished with no items and no errors on pages that show data in a browser. If an alert naming the spider and URL follows this prompt, start from it; if none does, check the pages under /js/ of the site the spiders crawl. Read scrapy.cfg, quotesbot/settings.py and every spider first. Prove the cause before changing anything: fetch the page with scrapy fetch, count the spider's selector in the raw response, then open the page in the browser and count it in the rendered document. Say in the pull request whether the data is rendered by script, and if it is embedded in the HTML as JSON, say so and how large a parser for it would be. Add scrapy-playwright with a requirements file pinning the versions you installed, and install Chromium with playwright install --with-deps chromium. In settings.py register ScrapyPlaywrightDownloadHandler for http and https and set TWISTED_REACTOR to the asyncio reactor; say why each is needed in this project. Add a new spider for the rendered pages and leave the existing spiders unrendered. Send playwright meta on the first request and on every follow-up request, wait for the item selector with a PageMethod, and abort image requests. Crawl the static pages with an existing spider and the rendered pages with the new one, each with -o, and report item_scraped_count, request counts and the first item of each. Run the new spider once without meta on the follow-up request and report that count too. Add pytest tests that build responses from one saved rendered page and one saved raw page. Open a pull request with the numbers.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

The pull request adds a spider, a requirements file and tests with their two saved pages, and changes `quotesbot/settings.py`. The description opens with the diagnosis as two numbers, the selector count in the raw response and in the rendered page, and a screenshot of the page as the browser drew it. Then comes a small table: for the static crawl and the rendered crawl, the items scraped, the requests made and the first item, with the deliberately broken run on a third line.

It is plain about cost. The description gives the elapsed time Scrapy logged for each crawl, so the price of rendering is a measured figure from this run and not an estimate, and it lists what a deployment now needs: a Chromium download and its system libraries on every machine that runs the new spider. It restates that only requests carrying the meta key use the browser, and names the cheaper alternative it found for this site, with the reason it would or would not hold up if the page changed.

A reviewer compares the two item counts and reads the wait condition, since a wait on the wrong selector either times out or returns early. They also decide whether the new spider belongs in the scheduled crawls yet, given what it needs installed.

## Run it on every crawl that returns no items

The signal for this job is a number your scheduler already has: a crawl whose stats end with no `item_scraped_count`. In the console, add an automation to the Project with a `webhook` trigger and a saved prompt like this one. The console gives the trigger's URL and shows its secret once, so store it then. From the script that runs your crawls, when a spider finishes empty, `POST` its name, start URL and stats to that URL with `Authorization: Bearer <secret>`. The body, up to 64 KB, reaches the agent after the prompt as data.

Send an `Idempotency-Key` header on the call to the trigger's URL, built from the spider name and the crawl's date, so a retried delivery starts nothing new. The call answers 202 with a `status`: `queued` with an `execution_id` when a run was created, `deduped` for a repeat, or `skipped` with a reason.

## Why run it on Rush?

This job needs a browser twice: once to establish that the page really shows data a plain download lacks, and once inside the crawl itself. Every Rush sandbox has one, along with the network access to fetch the site and room to install Chromium for Playwright. The agent's counts from the raw response, the rendered page and both crawls are kept with the run, so the diagnosis in the pull request is backed by output a reviewer can read line by line.

It also runs the failing variant on purpose. Showing that the spider drops to one page of items when the follow-up request loses its meta is the difference between a fix that works today and one whose weak point is documented. That extra crawl costs nothing in attention when it happens in a sandbox, and several empty-crawl alerts on one morning can each get their own run.

## Frequently asked questions

### Will every request go through the browser now?

No. The download handler renders only requests whose meta sets `playwright` to true and passes the rest to Scrapy's normal downloader. The existing spiders are untouched and run at the speed they did before.

### Why not Splash or Selenium?

scrapy-playwright plugs into Scrapy as a download handler, so scheduling, middlewares and pipelines work as usual, and it needs no separate rendering service. If your project already runs Splash, say so and the agent extends that setup instead of adding a second renderer.

### What if the data is in a JSON endpoint or embedded in the page?

Then a browser is often unnecessary, and the agent says so. It checks the raw response and the page's network requests first. For data embedded in a script tag or served by an API the page calls, a few lines that parse it are faster and less fragile than rendering.

### Does it work on Windows or with an older Scrapy?

The agent checks your versions against scrapy-playwright's stated requirements and reports any gap before it writes code. Its sandbox is Linux, so the crawl output in the pull request is from Linux; the plugin's documentation has its own section on Windows support.

## More Scrapy jobs

- [Fix a Scrapy Spider That Stopped Returning Items](https://getrush.ai/run/fix-broken-spiders/scrapy/)
- [Fix a Scrapy spider whose start_requests() is never called and scrapes 0 items](https://getrush.ai/run/fix-scrapy-start-requests-not-called/scrapy/)
- [Save Scrapy Items to PostgreSQL](https://getrush.ai/run/save-items-to-postgres/scrapy/)

## More jobs like this: Fix a bug from an alert

- [Redact Tokens and Passwords from Winston Logs in an Express API](https://getrush.ai/run/add-log-redaction/winston/)
- [Add a Fallback Model to an AI SDK Chat Route](https://getrush.ai/run/add-model-fallbacks/ai-sdk/)
- [Add Query Timeouts to a Drizzle and postgres.js Setup in Next.js](https://getrush.ai/run/add-query-timeouts/drizzle/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
