All jobs

Migrate From boto to boto3 in a Python Service

Yes. An agent ports S3 code from boto 2 to boto3, compares the requests both libraries send to a local stub, and opens a pull request.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

boto 2 has a hard stop date built into Python itself. Its last release, 2.49.0, is from July 2018, its repository is archived, and on Python 3.12 import boto fails with ModuleNotFoundError: No module named 'boto.vendored.six.moves'. On 3.11 the same import works, so the break arrives with the interpreter upgrade and not before. The agent first finds every boto call. In a service like propublica/politwoops-tweet-collector, requirements.txt pins boto==2.45.0 and one file uses it: bin/screenshot-worker.py imports S3Connection and Key, and a single method, upload_image, does the work for both screenshots and tweet images.

That method is short and nearly every line of it carries a decision. It reads access_key, secret_access_key, bucket_name and url_prefix from the [aws] section of the worker's ini file and passes the keys to S3Connection. It calls conn.create_bucket(bucket_name) on every upload to get a bucket object. It uploads with key.set_contents_from_filename(tmp_path, policy='public-read', headers={'Content-Type': content_type, 'Max-Age': 604800}), returns a URL it builds by hand as http://s3.amazonaws.com/<bucket>/<path>, and catches only IOError. The caller stores that URL in MySQL.

The worker cannot run whole in a sandbox: it needs beanstalkd, MySQL and a phantomjs binary, and there are no tests. So the agent isolates the upload. It starts a small HTTP server on localhost that accepts S3-style PUT requests and logs the path and headers, points boto at it, and runs the method's exact calls on Python 3.11. The log shows two requests for one image. The first is PUT /<bucket>/, the bucket creation, sent every time. The second is the object, with x-amz-acl: public-read, the content type, and a header literally named Max-Age, which is not a caching header. S3 takes caching instructions from Cache-Control, so by Amazon's documentation the week of caching this code asks for is not being applied. Both requests are signed with the older AWS key:signature scheme.

The boto3 version is one call, upload_file(tmp_path, bucket_name, dest_path, ExtraArgs={...}), and the same server shows one PUT, signed with AWS4-HMAC-SHA256. Two details need a human decision and the agent raises both. boto3 refuses unknown upload arguments, so Max-Age cannot be carried over as it is: ExtraArgs={'Max-Age': ...} raises ValueError: Invalid extra_args key 'Max-Age'. The agent sends CacheControl='max-age=604800', which is what the original meant, and flags it as a behaviour change, since objects will now be cached. And the error handling changes shape. A missing local file still raises an IOError subclass, but a failed upload raises a boto3 or botocore exception that the existing except IOError does not catch, so the agent widens that clause to keep the method returning None on failure as its caller expects.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Port this Python project from boto (version 2) to boto3. Read the requirements files and every module that imports boto. For each use, write down the connection arguments and where credentials come from, every bucket and key call with its arguments, headers and ACL, what the function returns, and which exceptions it catches. Confirm on the Python versions you can install whether `import boto` works. Do not use real AWS credentials. Start a local HTTP server that accepts S3-style requests and logs method, path and headers; run the existing boto calls against it and save the log. Port to boto3 with the same credential source passed explicitly, the same bucket, key, ACL and content type. Do not create the bucket on each upload. Where the old code sent a header boto3 will not accept, say what the header did, choose the closest supported argument, and flag it as a behaviour change. Keep return values, including any URL the code builds and stores, exactly as they are. Make sure every failure the old except clause handled is still handled, and list the boto3 exception types involved. Run the boto3 version against the same server and show both request logs in the pull request. State which Python versions the new dependency supports and whether the rest of the project runs on them.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that changes one method in bin/screenshot-worker.py, its two imports, and one line of requirements.txt. The description shows the two request logs next to each other: two requests against one, the old signature scheme against the new, Max-Age against Cache-Control. It states that the stored URL format is untouched, so rows already in the database and rows written after the change look the same.

Three notes are for the person who owns the AWS account, because a local stub cannot answer them. The first is permissions: the worker's credentials no longer need to create buckets, and the bucket must already exist. The second is the ACL. The upload still asks for public-read, and Amazon's documentation says buckets created with the current defaults have ACLs disabled and reject such a request, so a move to a new bucket would need a bucket policy instead. The third is the region. boto3 signs for a region, and the ini file has no region setting, so the agent adds one with a default and says which.

The description also reports what the port does not solve. The current boto3 requires Python 3.10 or newer, which suits a 3.12 upgrade, but three other scripts under bin/ are still Python 2 syntax and fail to compile on any Python 3. They are outside this change and are listed so nobody assumes the repository as a whole is ready.

Run it on every repository on your Python 3.12 upgrade list

boto tends to survive in the oldest corners of a codebase: cron jobs, backup scripts, workers nobody has deployed in a year. When a Python upgrade is planned, search for boto== and import boto across your repositories and send the body above to POST /o/{org}/sessions once for each hit. Use a request_id that is a UUID v5 of the repository name, so a second pass of the script returns the earlier runs instead of starting new ones.

With a webhook_url set on the key, each run posts run.completed or run.failed when it ends. Runs past the key's max_concurrent wait as queued and start in order.

The use case this job follows, with its automation and trigger: Upgrade dependencies weekly.

Why run it on Rush?

Code that talks to S3 is risky to test where real credentials are within reach. A laptop often has an AWS profile configured, and the quickest manual test of an upload function is to upload something. The agent's sandbox has no AWS access unless you give it one, and the prompt tells it to prove the port against a server it starts itself. What that server logs is better evidence than a successful real upload would be, because it shows the exact headers each library sent.

It is also the only way most teams will ever see what their boto code does on the wire. Few people reading headers={'Max-Age': 604800} would stop to ask whether S3 knows that header. A request log makes it obvious, and puts the choice between fixing it and preserving it in front of a reviewer.

Frequently asked questions

Why does import boto fail on Python 3.12?

boto bundles an old copy of the six compatibility library, and the way that copy registers its moves module stopped working with Python 3.12's import system. The error is No module named 'boto.vendored.six.moves'. No boto release fixes it; the last one came out in 2018.

Can boto and boto3 be installed side by side during the migration?

Yes. They are separate packages with separate import names, and a large project can port one module per pull request. They only have to share an interpreter that boto still imports on, which means Python 3.11 or older until the last module is done.

What replaces Key.set_contents_from_filename?

upload_file on the S3 client or on a Bucket resource. The ACL moves from policy='public-read' to ExtraArgs={'ACL': 'public-read'}, and content type and caching become ContentType and CacheControl in the same dictionary. Arbitrary HTTP headers are not accepted there.

Do I still need to call create_bucket or get_bucket first?

No. boto3's client calls take the bucket name as an argument and make no request until the operation itself. Code that called create_bucket only to obtain a bucket object, as this worker does, loses a request per upload and no longer needs permission to create buckets.

Will my S3 URLs change?

Only if your code asks the library for them. This worker formats its own URL string and stores it, so the agent leaves it alone. Where code used generate_url, the agent ports it to generate_presigned_url and shows an old and a new URL, because the query parameters differ between the two signature versions.

More jobs like this: Upgrade dependencies weekly

Keep reading