What the agent does
A spider whose selectors stopped matching does not crash. In a Scrapy project like scrapy/quotesbot, the toscrape-css spider loops over response.css("div.quote"); if the site renames that class, the loop body never runs, the crawl closes with finish_reason set to finished, the stats contain no item_scraped_count at all, and the log has no line at error level. The only evidence is an empty feed. So the agent does not begin by reading the traceback, because there is none. It begins by reproducing: scrapy crawl for the named spider with CLOSESPIDER_PAGECOUNT set to a handful of pages and -o to a scratch file, then the stats dump at the end of the log.
The stats say which kind of failure this is before any selector is touched. Responses with status 200 and no items point at markup. A 403 or 429, or a redirect to a login or consent page, means the site is refusing the crawler, and changing selectors would be the wrong fix. A count of requests far below normal with items still arriving means pagination broke, not extraction. The agent also reads settings.py: this project sets ROBOTSTXT_OBEY = True, so a new robots.txt rule that disallows the listing would drop every request before it is sent, and the log's robots lines would show it.
For a markup change the agent saves the page the spider received and works through the selectors from the outside in: the container (div.quote), then each field inside it (span.text::text, small.author::text, div.tags > a.tag::text), then the next-page link (li.next > a::attr(href)). It runs each against the saved page and finds the first that returns nothing, reads the surrounding HTML for where that data moved, and writes the smallest selector that matches the new structure. This repository has a second spider, toscrape-xpath, that scrapes the same pages with XPath, so the agent checks it against the same saved page and repairs it in the same pull request if it broke too.
Then it proves the repair. The saved page goes into tests/fixtures/ with a pytest test that builds an HtmlResponse from it and asserts the items and the follow-up request, so the fix has a test that fails on the old selector. The bounded crawl runs again and the item count goes in the pull request next to the count from the failing run. If the data is no longer in the HTML at all, because the page now renders it from JavaScript, the agent stops and reports what it found in the response instead of guessing at a rewrite.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "The spider toscrape-css in this Scrapy project finished its last crawl with no items and no errors. Find out why and fix it. Read scrapy.cfg, the settings module and the spider first. Reproduce with scrapy crawl toscrape-css -s CLOSESPIDER_PAGECOUNT=5 -o /tmp/before.jsonl and read the stats at the end of the log: response status counts, robots.txt lines, request count and item_scraped_count. Decide which case this is and say so: markup changed, the site is blocking or redirecting the crawler, robots.txt now disallows the pages, pagination broke, or the data moved into JavaScript. Only for a markup or pagination change, save the page the spider received, test each selector against it from the container inward, and change the first one that no longer matches to the smallest selector that fits the new markup. Check every other spider that scrapes the same site against the same page and fix it in the same change if it is broken. Add the saved page under tests/fixtures with a pytest test that builds an HtmlResponse from it and asserts the items and the next request; the test must fail on the old selector. Run the bounded crawl again and report the item count before and after. Do not raise concurrency, remove the robots.txt setting, add proxies or change the user agent. If the site is blocking the crawler or the data is no longer in the HTML, change nothing and report what the responses contain. Open a pull request with the diagnosis, the selector diff and the command output.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
For a markup change, a pull request with a small diff in the spider, the saved page as a fixture and one test. The description opens with the diagnosis in a sentence, names the selector that stopped matching and the element the data now sits in, and gives the item count of the bounded crawl before and after with the stats lines it read them from.
For the other cases there is no pull request, and that is the correct result. The run's answer says which case it found, with the status codes or the robots.txt rule or the script tag that now holds the data, and what the options are. A site that started returning 403 to the crawler is a decision about access for a person to make, and the prompt forbids the agent from working around it.
A reviewer compares the new selector with the saved page to see that it is tied to something stable, a class or attribute that names the content, and not to position in the document. They also read the items in the after file for fields that are present but empty, which a count alone does not show.
Run it on every crawl that scraped nothing
Scrapy does not call anyone when a crawl comes back empty, so the event has to come from whatever runs your crawls. Put an automation in the Project that holds the repository, with this prompt minus the spider's name and a webhook trigger. The script that runs the nightly crawl counts the lines in the feed afterwards; when the count is zero it sends a POST to the trigger's URL with Authorization: Bearer and the trigger's secret, and a JSON document with the spider's name and the stats dump. That document, up to 64 KB, becomes the event text after the prompt, which is how the agent learns which spider to look at.
Send an Idempotency-Key header to the trigger's URL made of the spider's name and the crawl date, so a rerun of the same failed job starts nothing new. The automation's max runs per hour keeps a site-wide redesign that empties every spider at once from starting more investigations than you want to read.
The use case this job follows, with its automation and trigger: Fix a bug from an alert.
Why run it on Rush?
The diagnosis depends on what the site sends to a crawler today, and that has to be fetched from somewhere. A run fetches it from its own sandbox, saves the page, and keeps every scrapy crawl command with its log in the transcript at GET /o/{org}/sessions/{id}/messages. The person who reviews the fix can read the same stats the agent read.
Empty crawls also arrive in groups, because one redesign breaks every spider for that site. Each spider gets its own run and its own pull request, and they proceed at the same time without sharing a checkout, so the morning after a redesign starts with fixes to review.
Frequently asked questions
Will it try to get around a block or a CAPTCHA?
No. The prompt tells it to change nothing when the site is refusing the crawler and to report the responses instead. It does not rotate user agents, add proxies or switch off ROBOTSTXT_OBEY. Whether and how to keep crawling a site that has started refusing you is your decision.
How does it know the selector is wrong and not the site temporarily down?
From the stats of its own crawl. A page that returns status 200 with the usual size and no items is a selector problem; a 5xx or a timeout is not, and the agent says so and leaves the code alone. It saves the response it judged, so you can check the call.
Does it crawl the whole site to test the fix?
No. It sets CLOSESPIDER_PAGECOUNT so the crawl closes after a few pages, which is enough to show items and pagination working. The project's own delay and concurrency settings stay as they are.
What if only some fields are empty and items still arrive?
Name the field in the prompt. The same method applies one level down: the container still matches, so the agent tests the field selectors against the saved page and repairs the one that returns nothing. A check on field fill rate in your crawl job catches this kind of failure, which an item count misses.