All jobs

Replace the Deprecated request Package With Native fetch

Yes. An agent replaces the deprecated request package with Node's built-in fetch, keeps cookies, redirects and error codes working, and opens a pull request.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

request has been deprecated on npm for years and still sits in the dependencies of many small packages. The agent reads the one call that matters and every option on it, because each option is a behaviour to keep. In a library like samholmes/node-open-graph, index.js exports a callback function, og(url, cb, options), whose getHTML step calls request with encoding: 'utf8', gzip: true, jar: true and a User-Agent header, treats status 200 as the only success and turns anything else into Error("Request failed with HTTP status code: N"). package.json asks for request ^2.73.0 and cheerio ^0.22.0. The lockfile, at lockfileVersion 1, resolves request to 2.88.0. The test script is the npm placeholder, and test.js prints the result of fetching a page from the network without asserting anything.

So the agent writes the test bed first: a local HTTP server with a page that has Open Graph tags, the same page gzipped, one in Latin-1, a 404, a plain redirect, and a page that answers with Set-Cookie and a redirect to itself until the cookie comes back. It runs the unchanged getHTML against each, then a direct fetch version, and compares. Most rows match. Both send accept-encoding: gzip, deflate and decompress the reply, both decode the body as UTF-8 whatever the page declares, both follow the redirect, and the 404 message is the same string.

Two rows differ. jar: true gives request a cookie jar, so the cookie-gated page succeeds on the second request. fetch has no jar. It asks 21 times and gives up with fetch failed. A scraper meets consent and session redirects of exactly this kind, so the agent keeps the behaviour with a manual redirect loop that carries Set-Cookie values forward and stops after ten hops, the limit request used. The second difference is the error object. For a host that does not resolve, request passes an error whose code is ENOTFOUND and whose message is getaddrinfo ENOTFOUND plus the host. fetch throws a TypeError that says fetch failed, with the code one level down on cause. The agent rethrows the cause, so a caller that checks err.code still works.

It keeps the public interface as it is. The package is published as open-graph and its users pass a callback, so the callback signature, the userAgent option and the parse export do not move. It adds engines with node >=18, the first release line where fetch is a global, and replaces the placeholder test script with node --test over the local server. While testing the error path it finds a defect that predates the change: when parse throws, the callback is called with the error and then called again with null, because the catch block does not return. That fix goes in its own commit.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Remove the deprecated request package from this Node.js project and use the built-in fetch instead. Read package.json, the lockfile and every file that requires request, and list each call with every option it passes (encoding, gzip, jar, headers, followRedirect, timeout, json, form, auth, proxy) and how the callback uses err, res.statusCode and body. Before editing, write a local HTTP server with cases for: a normal page, a gzipped page, a non-UTF-8 page, a 404, a redirect, and a page that sets a cookie and redirects to itself until the cookie is sent back. Run the unchanged code against each case and against a host name that does not resolve, and record the result or the error with its message and code. Then replace request with fetch. Keep the exported functions, their callback signatures and their error messages. If the old call used a cookie jar, keep cookies across redirects with a manual redirect loop limited to the number of hops request allowed. Make network errors expose the same code property as before. Run the same cases again and show both columns. Do not add a timeout or change which status codes count as success unless I ask; report those as suggestions. Add an engines field for the lowest Node version with global fetch, turn the cases into a node --test suite wired to npm test, and report npm audit and the installed package count before and after. Put any bug that existed before in a separate commit. Open a pull request with every command and its output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that changes index.js, package.json and package-lock.json, replaces test.js with a suite that needs no network, and carries the double-callback fix as a second commit. The description lists the request options found and how each is preserved, then the comparison table with one row per server case and columns for the old and new result. It reports the install: from the lockfile, npm ci adds 75 packages and npm audit lists 12 findings, four of them critical. Without request, 28 packages remain and npm audit lists four, all in the tree under cheerio 0.22.

A reviewer reads the two rows that needed code, cookies and error codes, and decides whether the redirect loop is worth keeping; dropping it is a one-line choice if no caller depends on cookie-gated pages. They check the engines field against the Node versions their users run, since that is the one visible change for consumers. The description also lists what was left alone on purpose: there is still no timeout, and a page that declares another charset is still decoded as UTF-8, as before.

Run it on every GitHub issue labelled dependencies

Deprecated packages usually surface as an issue someone files after an npm install warning. Add an automation to the Project with a github trigger on issues.labeled, set its labels to the one you use for dependency work, and save a prompt like the one above. When a repository owner, member or collaborator adds the label, a run starts with the issue appended after the prompt as event text. Bot senders never fire the trigger, so a label applied by an app does nothing.

The saved prompt is the only instruction the run follows, so keep it general: name the package to remove in the prompt, or tell the agent to remove whichever deprecated HTTP client package.json lists. Each labelled issue produces one pull request.

The use case this job follows, with its automation and trigger: Upgrade dependencies weekly.

Why run it on Rush?

Replacing an HTTP client is a claim that two implementations behave alike, and the cases where they do not are the ones nobody has a test for. The run builds those cases as a local server in its own sandbox, drives both clients through them, and keeps the output. The cookie-gated redirect that takes two requests with one client and twenty-one with the other is a measured row in a table, not a guess.

Nothing in the run depends on the public internet behaving. The old test.js fetched a page from a remote host; the new suite serves its own pages. That makes the proof repeatable in CI, and it is the reason the pull request can state its results as facts a reviewer can rerun.

Frequently asked questions

Is native fetch a drop-in replacement for request?

For a plain GET that follows redirects and reads text, nearly. It differs where request had options: there is no cookie jar, no json: true, no proxy option, and errors are a TypeError with the system code on cause. The agent lists the options your code uses and tests each one.

Will this be a breaking change for people who use my package?

Only through the Node version. The exported functions, callback arguments and error messages stay the same, and network errors keep their code. Consumers on a Node release older than 18 have no global fetch, which the new engines field states, so publish it as a major version if you still support them.

How are cookies kept without a cookie jar?

By following redirects by hand. The agent calls fetch with redirect: 'manual', reads the Set-Cookie headers of each redirect response, and sends them as a Cookie header on the next hop, up to ten hops. It handles the single-host case this library needs and is not a general cookie store.

Does removing request clear npm audit?

It clears what request brought in, which here is eight of twelve findings and 47 packages. The remaining four come from the old cheerio release this package pins. Upgrading that changes parsing behaviour and is better done as its own pull request with its own tests.

More jobs like this: Upgrade dependencies weekly

Keep reading