All jobs

Fix "DeprecationWarning: Call to deprecated method findAll" in Beautiful Soup

Yes. An agent renames every camel-case Beautiful Soup call to find_all, pins the release that warns, and makes your suite fail on the next deprecation.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

Beautiful Soup 4.13.0 started warning on the old camel-case names. With warnings shown, soup.findAll('a') prints DeprecationWarning: Call to deprecated method findAll. (Replaced by find_all) -- Deprecated since version 4.0.0. from bs4/_deprecation.py; the same call on 4.12.3 is silent. A second one is older: since 4.11.0, find('a', text='t') prints The 'text' argument to find()-type methods is deprecated. Use 'string' instead. Both calls still return the right tags, so the first time most people see the message is when a CI job runs with -W error or when a test runner turns deprecations into failures.

In a scraper like MayankPandey01/BrokenLinkHijacker, requirements.txt lists requests, urllib3, bs4, beautifulsoup4, colorama and argparse, all unpinned, so a fresh install resolves beautifulsoup4 4.15.0 with soupsieve 2.10, requests 2.34.2 and urllib3 2.8.0. BLH.py is one 344-line file: main_webpage_links at line 85 parses requests.get(url, headers=headers, verify=...).content with html.parser on line 90, then loops soup.findAll("a") on line 92 for links and soup.findAll('img') on line 117 for images; status_check on line 163 requests each link in a thread and main_proc on line 179 runs one to three levels of crawl. The README says to run it as BLH.py https://example.com and recommends Python 3.8.

The agent reproduces it against a page it serves itself. It writes a small HTML file with two links and an image, serves it with python -m http.server, and runs python -W default BLH.py http://localhost:8765/: the output shows the two inbound links and the image link, with the deprecation line printed once for line 92 and once for line 117. Then it runs with -W error::DeprecationWarning to see what a strict CI would do, and finds that the file does not even compile under that flag: the ASCII banner at line 43 contains \|, an invalid escape sequence that Python 3.11 reports as a DeprecationWarning, so the run dies with a SyntaxError before any HTML is fetched. On 3.12 the same escape is a SyntaxWarning.

The port is small and the agent keeps it small: findAll becomes find_all on both lines, the banner becomes a raw string, and requirements.txt loses the duplicate bs4 entry and the stdlib argparse and pins beautifulsoup4>=4.13. It greps for the other camel-case names Beautiful Soup aliases (findAllNext, findParent, renderContents) and for text= passed to find or find_all, so the pull request lists every hit or says there were none. A test serves the fixture page, runs the script as a subprocess with -W error::DeprecationWarning, and asserts it exits 0 and prints the three links, so the next deprecation fails the suite instead of scrolling past.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "This scraper prints DeprecationWarning: Call to deprecated method findAll when run with warnings enabled. In a clean virtual environment, install requirements.txt and report the resolved beautifulsoup4 version; then serve a small HTML page with two links and an image using python -m http.server and run the script against it with -W default, capturing every warning with its file and line. Run it again with -W error::DeprecationWarning and report what fails first. Rename every findAll and other camel-case Beautiful Soup alias to its snake_case name, replace any text= argument to find-type methods with string=, and fix any invalid escape sequence the strict run reports, changing nothing else on those lines. Clean requirements.txt of duplicate and standard-library entries and pin beautifulsoup4 to a release that emits the findAll warning or newer. Add a test that serves the fixture page, runs the script as a subprocess with -W error::DeprecationWarning and asserts it exits 0 and lists the links. Run the test and open a pull request with both outputs and the list of renamed calls.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that changes the scraper file, requirements.txt and a new test file, with the test's fixture page beside it. The description opens with the resolved package versions, the -W default run showing each deprecation with its line, and the -W error run showing what stopped it first, which for this example is the banner's escape sequence rather than a Beautiful Soup call.

Below that is the list of renamed calls, file and line, and the grep results for the other aliases and for text=, so a reviewer sees that the two findAll calls were the whole exposure and nothing was renamed by guesswork. The session's transcript keeps the server command, both script runs and the test output.

Before merging, a reviewer checks that the pinned lower bound matches what the rest of the team installs, that the raw-string banner still prints the same art, and that the subprocess test does not depend on a port that CI already uses.

Run it on every failed CI run with -W error

The warning turns into a failure the day someone adds -W error to the test command or a test runner starts treating deprecations as errors. When that job fails, a handler on your server reads the failure, and sends this request with POST /o/{org}/sessions with the repository in resources and the failing command's output appended to input. Give it a request_id that is a UUID v5 of the CI run id, so a retried delivery returns the run that already started instead of opening a second pull request.

For a fleet of scrapers that share this pattern, a bot sender never fires an automation's github trigger, so give the automation a webhook trigger and add a workflow step on the bump that runs when its title names beautifulsoup4 and posts the pull request number to the trigger's URL with Authorization: Bearer and the trigger's secret; the event lands after the prompt, and the run renames what the new release warns about before the bump merges.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

The rename takes a minute; knowing it is complete does not. Showing that nothing else warns means a clean install, a page to crawl, two runs under different warning filters and a grep for every alias Beautiful Soup still carries, and the strict run here surfaces an unrelated escape-sequence problem that a quick edit would miss. In a Rush sandbox the agent serves its own page, runs both filters and keeps the output, and the machine is gone when the run ends.

What comes back is evidence rather than a description: the transcript holds the versions, the warning lines with their numbers and the passing strict run. A team with several scrapers starts one run per repository with the same request, up to the key's max_concurrent limit at once, and the rest queue.

Frequently asked questions

Does findAll still work, or is my scraper broken?

It still works. On beautifulsoup4 4.15.0 the call returns the same tags and only prints a DeprecationWarning, which Python hides by default. It breaks only under -W error or a test runner that fails on deprecations, which is why the agent reproduces it with the warning filter set explicitly.

Which release started warning?

4.13.0 for findAll; 4.12.3 is silent on the same call. The text= argument has warned since 4.11.0. The agent pins the lower bound to 4.13 so the suite runs against a release that warns, which is what makes the strict test meaningful.

Why does the strict run fail on a line that has nothing to do with Beautiful Soup?

Because -W error::DeprecationWarning turns every deprecation into an error, and an invalid escape sequence in a normal string is one on Python 3.11. The example's banner at line 43 has one, so the file fails to compile before any page is fetched. Making the banner a raw string clears it; the pull request shows that run too.

Will the agent crawl a real site to test this?

No. It serves a fixture page from the sandbox with python -m http.server and points the script at it, so the test is deterministic and makes no outbound requests. The same fixture ships with the test in the pull request.

More jobs like this: Fix a bug from an alert

Keep reading