# Add Feed Exports to a Scrapy Project

Yes. An agent reads your spiders and settings, declares the output in a FEEDS setting with an item class and field order, runs a crawl to prove it, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-feed-exports/scrapy/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent starts with what the project yields and how it is written today. In a project laid out like scrapy/quotesbot, two spiders in `quotesbot/spiders/`, `toscrape-css` and `toscrape-xpath`, each yield a plain dict with `text`, `author` and `tags`, and the only way to get a file out is the `-o quotes.json` flag the README shows under `scrapy crawl`. `quotesbot/items.py` still holds the generated `QuotesbotItem` with a `pass` body, `quotesbot/pipelines.py` returns every item unchanged, and `quotesbot/settings.py` has `ROBOTSTXT_OBEY = True` with every feed, throttle and cache setting left commented out. There is no requirements file, so the Scrapy version is whatever is installed; the spiders call `extract_first()` and `extract()`, the older spellings of `get()` and `getall()`.

It then writes the export where it belongs. `FEEDS` in `settings.py` maps an output URI to its options: a `jsonlines` feed at `output/%(name)s-%(time)s.jl` with `encoding` set to `utf8`, so each quote is one line and a crawl that stops halfway still leaves a readable file, plus a `csv` feed of the same items for whoever opens them in a spreadsheet. `FEED_EXPORT_FIELDS` fixes the column order to `text`, `author`, `tags`, and `FEED_EXPORT_ENCODING` is set once so the curly quotes in `text` are not escaped as `\u201c`. Because `%(name)s` is in the URI, both spiders write to their own files without touching either spider.

The dicts become an item. `QuotesbotItem` gets the three `scrapy.Field()` declarations, both spiders yield it instead of a dict, and the exporter now refuses a misspelled key at the point it is set rather than writing a column nobody asked for. A `requirements.txt` pins the Scrapy release the agent ran against, since `FEEDS` replaced `FEED_URI` and `FEED_FORMAT` in Scrapy 2.1 and the project had nothing saying which side of that line it was on. The `extract_first()` calls are left alone: they still work, and changing them is a separate pull request.

In its sandbox the agent installs the pinned Scrapy, runs `scrapy list` to confirm both spiders still load, then `scrapy crawl toscrape-css` with `CLOSESPIDER_ITEMCOUNT` set low enough to finish in a minute against quotes.toscrape.com, and checks the `.jl` and `.csv` files it produced: line count, the field order, and that `tags` is a list in JSON lines and a joined string in CSV. The README's `-o` example is updated to say the feeds now run on every crawl and `-O` overwrites them.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add feed exports to this Scrapy project. Read scrapy.cfg, quotesbot/settings.py, quotesbot/items.py, quotesbot/pipelines.py and both spiders under quotesbot/spiders/ first. In settings.py add a FEEDS setting with a jsonlines feed at output/%(name)s-%(time)s.jl and a csv feed of the same items, FEED_EXPORT_FIELDS in the order text, author, tags, and FEED_EXPORT_ENCODING utf-8. Give QuotesbotItem the three fields and make both spiders yield it instead of a dict. Add a requirements.txt that pins the Scrapy version you install. Do not change the selectors or the extract_first calls. Run scrapy list, then one short crawl of toscrape-css with CLOSESPIDER_ITEMCOUNT set so it finishes quickly, and quote the feed files' line counts and first lines in the pull request. Update the README's crawl section to describe the feeds. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request touching `quotesbot/settings.py`, `quotesbot/items.py`, both spider files, a new `requirements.txt`, the README and `.gitignore` with `output/` added. The description quotes the `scrapy list` output, the crawl's closing stats block (items scraped, requests made, the `finish_reason`), and the first two lines of each feed file so a reviewer sees the exact shape without running anything.

The run's record keeps the full crawl log, so the `robots.txt` fetch, every page request and the exporter's open and close messages are there to read. Nothing is written to the repository's tracked files by the crawl itself; the feed files land under `output/`, which the pull request ignores.

Before merging, a reviewer checks two things: that the `%(time)s` in the URI is what they want (one file per crawl rather than one file overwritten), and that the CSV join of `tags` with a comma suits whoever reads the spreadsheet, since a quote's tags can themselves contain commas.

## Run it on every Linear issue labelled export

A team that keeps several Scrapy projects turns this into a rule rather than a one-off. In the Project for each repository, an automation with the prompt above and a `linear` trigger on `linear.issue.labeled` for the label `export` starts a run whenever someone files an issue such as "write the product spider's output to S3 as JSON lines". The trigger adds the issue after the prompt, so the run reads the bucket name and format from the issue text and writes that feed instead of the local one.

Each run opens its own pull request on its own branch and links the issue. A feed URI on `s3://` needs `boto3` in the sandbox and credentials the Project holds as secrets, which the agent reads as environment variables; the crawl in the sandbox then proves the upload rather than the local write.

## Why run it on Rush?

The feed settings are easy to write and easy to get subtly wrong: a `FEED_URI` from an old tutorial that Scrapy 2.1 stopped reading, an encoding left at the default so every curly quote comes out escaped, a CSV with the columns in dict order. Running the crawl is the only way to see the file, and on a laptop that means a virtualenv, a Scrapy install and a few minutes against a live site. In a Rush sandbox the agent does exactly that, against the real target, and the file it reads is the one the pull request describes.

The crawl log and the feed samples stay with the run, so the reviewer sees the exporter open, the item count and the first lines rather than a promise. Across a dozen spider repositories the job runs in parallel, one sandbox each, and each pull request carries its own proof.

## Frequently asked questions

### Will this change what the spiders scrape?

No. The selectors, the `start_urls` and the pagination in both spiders stay as they are. The only spider change is yielding `QuotesbotItem(...)` with the same three keys instead of a dict, which makes a misspelled field an error at the spider rather than a silent extra column.

### Can the feed go to S3 or GCS instead of a local file?

Yes. `FEEDS` takes an `s3://bucket/path.jl` or `gs://` URI with the same options; Scrapy needs `boto3` or `google-cloud-storage` installed and credentials in the environment. Say so in the prompt and the agent pins the extra dependency in `requirements.txt` and proves the upload from the sandbox.

### Why pin Scrapy when the project never did?

Because `FEEDS` exists only from Scrapy 2.1, and a project with no requirements file runs on whatever version the machine has. The pin records the version the crawl in the pull request ran on, so the next person installs the same one.

### Does the crawl hit the real site on every run?

It fetches quotes.toscrape.com, a site built for scraping practice, with `ROBOTSTXT_OBEY` still on and the item count capped so it stops after a page or two. For a production target, add `HTTPCACHE_ENABLED` to the prompt so a second run replays the cached responses.

### What happens to the -o flag in the README?

It still works and overrides nothing: `-o` appends to the feeds the settings declare, and `-O` overwrites a file. The README's crawl section is rewritten to say the feeds run on every crawl and where the files land.

## More Scrapy jobs

- [Add HTTP Caching to a Scrapy Project](https://getrush.ai/run/add-http-caching/scrapy/)
- [Scrape JavaScript-Rendered Pages with Scrapy and Playwright](https://getrush.ai/run/add-javascript-rendering/scrapy/)
- [Fix a Scrapy Spider That Stopped Returning Items](https://getrush.ai/run/fix-broken-spiders/scrapy/)

## More jobs like this: Turn a Linear label into a PR

- [Add Open and Save File Dialogs to an Electron App from a Linear Issue](https://getrush.ai/run/add-file-dialogs/electron/)
- [Add Native File Dialogs to a Tauri App](https://getrush.ai/run/add-file-dialogs/tauri/)
- [Add File Uploads to a React Router App](https://getrush.ai/run/add-file-uploads/react-router/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
