# Upgrade a Scrapy Project to the Latest Scrapy

Yes. The agent pins the release it tested, crawls every spider on it, clears each deprecation warning in the log and opens a pull request with the item counts.

[Run this on Rush](https://cloud.getrush.ai/run/upgrade-scrapy/scrapy/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The first question in a Scrapy upgrade is what the project runs on now, and old projects often cannot answer it. In a project like scrapy/quotesbot there is no `requirements.txt`, no `pyproject.toml` and no `setup.py`: `scrapy.cfg` and the `import scrapy` lines are the only places the framework is named, so whichever release `pip install scrapy` resolves on a given day is the version in use. The files carry their age in other ways. Every module opens with a `# -*- coding: utf-8 -*-` line, `pipelines.py` declares `class QuotesbotPipeline(object)`, and `settings.py` is the template an early `scrapy startproject` wrote, with links to documentation hosts in its comments and only four settings uncommented. The agent records all of that, installs the current release into a fresh virtual environment and notes the exact version of Scrapy and Twisted from the first lines of a crawl log.

Then it runs every spider `scrapy list` prints, each with `CLOSESPIDER_PAGECOUNT` set to a few pages and `-o` to a file, and reads three things in each log: the item count, the response statuses, and any line containing `ScrapyDeprecationWarning`. Scrapy reports deprecations through the log at warning level, so a search of the saved log is the check. In this repository the two spiders rely on `start_urls`, which the base class still turns into the first requests, and on `extract_first()` and `extract()`, which still work; a crawl of either prints no deprecation. The agent says that plainly instead of rewriting code that has no reason to change.

What is not exercised gets a second look, because a clean crawl only covers the code that ran. `ITEM_PIPELINES` is commented out here, so `QuotesbotPipeline` never executes. Enabled on the command line with `-s`, its `process_item(self, item, spider)` signature draws a `ScrapyDeprecationWarning` saying the spider argument will stop being passed. The agent applies the same test to every pipeline, middleware and extension the settings mention, enabled or not, and to any spider that overrides `start_requests`, since current releases document an asynchronous `start()` method as the entry point and describe `start_requests()` as what older versions call.

The settings get a comparison, not a rewrite. The agent generates a throwaway project with the installed release's `scrapy startproject` and diffs its `settings.py` against yours. The current template sets `FEED_EXPORT_ENCODING = "utf-8"`, a download delay and a per-domain concurrency of one, none of which this project has. Without the first, a JSON feed writes each curly quote in the scraped text as a `\u201c` escape. The agent proposes the encoding setting with a before and after sample of the feed and lists the other differences for a person to decide, because a new delay changes how long every crawl takes.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Upgrade this Scrapy project to the latest Scrapy release. Read scrapy.cfg, the settings module, items.py, pipelines.py, any middlewares and every spider first, and find every file that pins or names a Scrapy version; if none does, say so. If a version is pinned, create one virtual environment with it and crawl every spider from scrapy list with -s CLOSESPIDER_PAGECOUNT=3 and -o to a file, as the baseline. Create a second environment with the latest release and run the same crawls. For each spider report the Scrapy and Twisted versions from the log, the item count, the response status counts and every log line containing ScrapyDeprecationWarning, and compare the items with the baseline when there is one. Enable each pipeline, middleware and extension that the settings mention but leave commented out, one at a time with -s, and record the warnings that appear. Fix every deprecation in code the project owns, with the smallest change the warning asks for. Generate a fresh project with scrapy startproject in a temporary directory and diff its settings.py against ours: apply FEED_EXPORT_ENCODING if it is missing and show the feed before and after, and list every other difference without applying it. Pin the Scrapy version you tested in a requirements file. Do not change selectors, delays, concurrency or the robots.txt setting. Open a pull request with the table of results per spider and the log excerpts.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

A pull request that pins the tested release in a requirements file, changes the signatures or imports that drew a deprecation warning, and adds the feed encoding setting when it was missing. The description has a table with one row per spider: versions, item count, response statuses and the number of deprecation lines before and after the change. Where a pinned older version existed, the table also says whether the items from both versions were equal.

Below the table is the list the agent did not act on: each setting where your file differs from the current template, with what the template has and what it would change in a crawl. Code that still works without a warning, such as `extract_first()`, is listed as left alone.

A reviewer reads the log excerpts for the status counts first, since a crawl that was refused by the site proves nothing about the upgrade. Then they check that every component the settings mention appears in the results, including the ones that are switched off, because that is where a signature change hides until someone enables it in production.

## Run it on every Monday morning

An automation in the Project that holds the repository, with this prompt and a `schedule` trigger whose `cron` is `0 7 * * 1` in your `timezone`, starts one run each week. Most weeks there is no new release; the run ends with a short answer saying the pin is current and opens nothing. The week a release lands, the pull request is waiting with the crawl results on it.

Teams that keep one repository per client send this body from a script to `POST /o/{org}/sessions` for each repository, with a `request_id` that is a UUID v5 of the repository name and the ISO week, so running the script twice in a week starts each upgrade once.

## Why run it on Rush?

An upgrade check for a crawler has to crawl, and it has to do so twice under different versions without one environment leaking into the other. The run builds both virtual environments in its own sandbox, runs the bounded crawls there, and keeps each log in the transcript at `GET /o/{org}/sessions/{id}/messages`. The table in the pull request can be checked against those logs line for line.

It also covers the part people skip. Reading the release notes tells you what changed in Scrapy; only running each disabled pipeline and middleware tells you which of those changes land in your code. That is slow, mechanical work with a clear finish, which is what a run is for.

## Frequently asked questions

### My project has no pinned Scrapy version. What is the baseline?

There is none to compare against, and the pull request says so. The agent treats the current release as the first known-good version: it crawls every spider on it, records the counts and pins it. From then on each upgrade has a real before and after.

### Will it replace extract_first() with get()?

Not unless you ask. Both names work and the old ones print no warning, so changing them is a style edit across every spider with no effect on the crawl. The agent keeps the upgrade diff to what the new release requires and lists the rest.

### Does the crawl during the upgrade hammer the target site?

No. Each spider stops after the few pages `CLOSESPIDER_PAGECOUNT` allows, and the project's delay, concurrency and `ROBOTSTXT_OBEY` settings are left as they are. With a pinned older version that is two short crawls per spider.

### What about the Python version?

The agent reports the Python it ran on and the minimum the new Scrapy release declares, and stops there. Python 2 leftovers such as the coding line and `class Name(object)` are harmless on Python 3; it removes them only if the prompt asks, in a separate commit.

## More Scrapy jobs

- [Write Tests for Scrapy Spiders](https://getrush.ai/run/write-spider-tests/scrapy/)
- [Add AutoThrottle to a Scrapy Project](https://getrush.ai/run/add-autothrottle/scrapy/)
- [Add Feed Exports to a Scrapy Project](https://getrush.ai/run/add-feed-exports/scrapy/)

## More jobs like this: Upgrade dependencies weekly

- [Upgrade Python Selenium Tests to Selenium 4 (find_element_by_* and PhantomJS Removed)](https://getrush.ai/run/upgrade-selenium/selenium/)
- [Upgrade SQLx From 0.5 to the Current Release in a Rust API](https://getrush.ai/run/upgrade-sqlx/sqlx/)
- [Upgrade Storybook to the Latest Version](https://getrush.ai/run/upgrade-storybook/storybook/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
