What the agent does
The agent starts with what the project yields and how it is written today. In a project laid out like scrapy/quotesbot, two spiders in quotesbot/spiders/, toscrape-css and toscrape-xpath, each yield a plain dict with text, author and tags, and the only way to get a file out is the -o quotes.json flag the README shows under scrapy crawl. quotesbot/items.py still holds the generated QuotesbotItem with a pass body, quotesbot/pipelines.py returns every item unchanged, and quotesbot/settings.py has ROBOTSTXT_OBEY = True with every feed, throttle and cache setting left commented out. There is no requirements file, so the Scrapy version is whatever is installed; the spiders call extract_first() and extract(), the older spellings of get() and getall().
It then writes the export where it belongs. FEEDS in settings.py maps an output URI to its options: a jsonlines feed at output/%(name)s-%(time)s.jl with encoding set to utf8, so each quote is one line and a crawl that stops halfway still leaves a readable file, plus a csv feed of the same items for whoever opens them in a spreadsheet. FEED_EXPORT_FIELDS fixes the column order to text, author, tags, and FEED_EXPORT_ENCODING is set once so the curly quotes in text are not escaped as \u201c. Because %(name)s is in the URI, both spiders write to their own files without touching either spider.
The dicts become an item. QuotesbotItem gets the three scrapy.Field() declarations, both spiders yield it instead of a dict, and the exporter now refuses a misspelled key at the point it is set rather than writing a column nobody asked for. A requirements.txt pins the Scrapy release the agent ran against, since FEEDS replaced FEED_URI and FEED_FORMAT in Scrapy 2.1 and the project had nothing saying which side of that line it was on. The extract_first() calls are left alone: they still work, and changing them is a separate pull request.
In its sandbox the agent installs the pinned Scrapy, runs scrapy list to confirm both spiders still load, then scrapy crawl toscrape-css with CLOSESPIDER_ITEMCOUNT set low enough to finish in a minute against quotes.toscrape.com, and checks the .jl and .csv files it produced: line count, the field order, and that tags is a list in JSON lines and a joined string in CSV. The README's -o example is updated to say the feeds now run on every crawl and -O overwrites them.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add feed exports to this Scrapy project. Read scrapy.cfg, quotesbot/settings.py, quotesbot/items.py, quotesbot/pipelines.py and both spiders under quotesbot/spiders/ first. In settings.py add a FEEDS setting with a jsonlines feed at output/%(name)s-%(time)s.jl and a csv feed of the same items, FEED_EXPORT_FIELDS in the order text, author, tags, and FEED_EXPORT_ENCODING utf-8. Give QuotesbotItem the three fields and make both spiders yield it instead of a dict. Add a requirements.txt that pins the Scrapy version you install. Do not change the selectors or the extract_first calls. Run scrapy list, then one short crawl of toscrape-css with CLOSESPIDER_ITEMCOUNT set so it finishes quickly, and quote the feed files' line counts and first lines in the pull request. Update the README's crawl section to describe the feeds. Open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request touching quotesbot/settings.py, quotesbot/items.py, both spider files, a new requirements.txt, the README and .gitignore with output/ added. The description quotes the scrapy list output, the crawl's closing stats block (items scraped, requests made, the finish_reason), and the first two lines of each feed file so a reviewer sees the exact shape without running anything.
The run's record keeps the full crawl log, so the robots.txt fetch, every page request and the exporter's open and close messages are there to read. Nothing is written to the repository's tracked files by the crawl itself; the feed files land under output/, which the pull request ignores.
Before merging, a reviewer checks two things: that the %(time)s in the URI is what they want (one file per crawl rather than one file overwritten), and that the CSV join of tags with a comma suits whoever reads the spreadsheet, since a quote's tags can themselves contain commas.
Run it on every Linear issue labelled export
A team that keeps several Scrapy projects turns this into a rule rather than a one-off. In the Project for each repository, an automation with the prompt above and a linear trigger on linear.issue.labeled for the label export starts a run whenever someone files an issue such as "write the product spider's output to S3 as JSON lines". The trigger adds the issue after the prompt, so the run reads the bucket name and format from the issue text and writes that feed instead of the local one.
Each run opens its own pull request on its own branch and links the issue. A feed URI on s3:// needs boto3 in the sandbox and credentials the Project holds as secrets, which the agent reads as environment variables; the crawl in the sandbox then proves the upload rather than the local write.
The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.
Why run it on Rush?
The feed settings are easy to write and easy to get subtly wrong: a FEED_URI from an old tutorial that Scrapy 2.1 stopped reading, an encoding left at the default so every curly quote comes out escaped, a CSV with the columns in dict order. Running the crawl is the only way to see the file, and on a laptop that means a virtualenv, a Scrapy install and a few minutes against a live site. In a Rush sandbox the agent does exactly that, against the real target, and the file it reads is the one the pull request describes.
The crawl log and the feed samples stay with the run, so the reviewer sees the exporter open, the item count and the first lines rather than a promise. Across a dozen spider repositories the job runs in parallel, one sandbox each, and each pull request carries its own proof.
Frequently asked questions
Will this change what the spiders scrape?
No. The selectors, the start_urls and the pagination in both spiders stay as they are. The only spider change is yielding QuotesbotItem(...) with the same three keys instead of a dict, which makes a misspelled field an error at the spider rather than a silent extra column.
Can the feed go to S3 or GCS instead of a local file?
Yes. FEEDS takes an s3://bucket/path.jl or gs:// URI with the same options; Scrapy needs boto3 or google-cloud-storage installed and credentials in the environment. Say so in the prompt and the agent pins the extra dependency in requirements.txt and proves the upload from the sandbox.
Why pin Scrapy when the project never did?
Because FEEDS exists only from Scrapy 2.1, and a project with no requirements file runs on whatever version the machine has. The pin records the version the crawl in the pull request ran on, so the next person installs the same one.
Does the crawl hit the real site on every run?
It fetches quotes.toscrape.com, a site built for scraping practice, with ROBOTSTXT_OBEY still on and the item count capped so it stops after a page or two. For a production target, add HTTPCACHE_ENABLED to the prompt so a second run replays the cached responses.
What happens to the -o flag in the README?
It still works and overrides nothing: -o appends to the feeds the settings declare, and -O overwrites a file. The README's crawl section is rewritten to say the feeds run on every crawl and where the files land.