All jobs

Upgrade a Python Library from pandas 1.x to pandas 3

Yes. The agent moves the pin, the Python version and the tests that used removed arguments, then proves the numbers the library computes did not change.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

A pandas pin that is four years old says little about how much work the upgrade is, so the agent measures before it edits. In a library like bukosabino/ta the pin lives in requirements-core.txt as pandas==1.3.5 and numpy==1.21.5, and the test, coverage and docs requirement files all include it with -r. setup.py is different: its install_requires names numpy and pandas with no version, so anyone who installs the published package already gets the newest pandas. Only the project's own tests still run on the old one. The agent writes that down first, because it changes what the pull request is for: catching the tests up with what users already run.

Next comes the interpreter. pandas 1.3.5 ships wheels for CPython 3.7 to 3.10 and numpy 1.21.5 declares Requires-Python >=3.7,<3.11, so the baseline needs Python 3.10 or older. When this page was written the newest pandas was 3.1.1, which requires Python 3.11 and numpy 2.0.2 or later. No single interpreter runs both sides. The agent builds two environments, one per side, and runs the suite the way the Makefile does, python -m unittest discover, in each. On the old pins all 134 tests pass. Run again with -W error, two fail on a FutureWarning: test_tsi and test_tsi2 in test/unit/momentum.py pass check_less_precise=True to pd.testing.assert_series_equal.

On pandas 3.1.1 with numpy 2.4.6 those same two tests are the only ones that break, now with TypeError: assert_series_equal() got an unexpected keyword argument 'check_less_precise'. The other 132 pass, and with the two fixed the whole suite passes under -W error, so nothing else in the library is on a deprecation path. A search of ta/ for removed spellings (fillna(method=...), append, np.NaN, iteritems) finds none; ta/utils.py already calls ffill() and bfill(), and the release table in RELEASE.md records two earlier rounds of warning fixes. The fix to the tests is an explicit rtol=0.5e-3 in place of the removed argument, which passes on both sides. Deleting the argument instead fails both tests, because the reference values in test/data carry fewer digits than the default tolerance compares.

Passing tests only cover the rows the reference files hold, so the agent also compares full outputs. It runs ta.add_all_ta_features on test/data/datas.csv in both environments, with and without fillna, writes every column at 17 significant digits and diffs the two. On this repository that is 46,306 rows and 94 columns; 86 columns come out bit for bit identical, and the other 8 differ by less than 1e-11, the largest in volume_sma_em. Then it follows the pin outward: the test, coverage and docs requirement files still install on Python 3.11, while requirements-play.txt does not, because matplotlib==3.1.1 has wheels for CPython 3.6 and 3.7 only.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Upgrade pandas in this repository to the newest release, and numpy to the newest release that pandas and the chosen Python support. Before editing, find every place a version is declared: requirement files and the files they include with -r, setup.py or pyproject.toml, classifiers, CI config and docs. Say which of them governs what users install and which governs only this project's tests. Read the Requires-Python and wheel tags of the pinned and the target versions on PyPI and pick an interpreter for each side; build one environment per side. Run the test suite exactly as the Makefile or CI runs it in the old environment, then again with -W error, and record totals and every warning. Run it in the new environment and list each failure with its cause. Fix only what the upgrade breaks: replace removed arguments and methods with their documented replacements, keep every assertion as strict as it was, and do not delete a tolerance or loosen one. After the tests pass under -W error on the new versions, compare outputs: run the library's widest public entry point on the sample data in both environments, write every column with 17 significant digits, and report how many columns are identical and the largest absolute difference in the rest. Then try to install every other requirements file against the new pin and report which ones no longer install and why; do not rewrite CI or docs tooling in this pull request. Update the pins and any Python version classifiers the change makes false. Open a pull request whose description has both test totals, the list of changed lines by cause, the output comparison, and the follow-ups you did not do.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that is smaller than the version jump suggests. In a repository like the one above it changes two lines of requirements-core.txt, two arguments in test/unit/momentum.py, and the Python classifiers in setup.py, which still list 3.6 and 3.7. The description opens with the two test runs side by side: the count on the old pins, the count on the new ones, and the names of the tests that failed in between with the error each raised.

Below that sits the output comparison, as a table: rows and columns computed, columns that are identical, and for each column that differs its largest absolute difference. A reviewer who maintains numeric code reads this section first. A difference of 1e-12 in a rolling indicator is floating point order of operations; a difference in the third decimal would be a behaviour change, and the agent is told to stop and report one, not to adjust a tolerance until the suite is green.

The last section lists what the pin exposed and the agent left alone: a requirements file that no longer installs, a CI image on a Python the new pandas does not support, a docs build pinned to an old Sphinx. Each has the command that failed and its first error line. Before merging, the reviewer checks that no assertion lost precision, and decides whether the oldest supported Python in the classifiers matches what the project wants to promise.

Run it on every Monday morning

A pin goes stale because nobody is asked to move it. An automation in the Project that holds the repository, with a schedule trigger set to a weekly cron, starts this job without anyone asking. Its saved prompt is the instruction above with one addition: if the pinned versions already equal the newest ones the interpreter supports, answer in one line and open nothing. Most weeks that is the whole run. In a week with a pandas release, the pull request is there before anyone has read the release notes.

For several repositories, a script sends this body once per repository to POST /o/{org}/sessions, each with a request_id that is a UUID v5 of the repository name and the ISO week, so a retried script starts nothing twice. A key runs a set number of sessions at once and queues the rest in order, and a webhook_url on the key receives run.completed for each run the key started.

The use case this job follows, with its automation and trigger: Upgrade dependencies weekly.

Why run it on Rush?

This upgrade needs two interpreters and two sets of binary wheels that cannot share an environment, and it needs the old side to keep working until the comparison is done. A laptop has one Python on its path and a virtual environment that somebody will overwrite. The run's sandbox is a clean machine for this one job: the agent installs both sides, runs the suite four times and the comparison once, and every command with its output stays in the transcript at GET /o/{org}/sessions/{id}/messages.

The proof is also the part people skip by hand. Diffing 94 columns over 46,306 rows at full precision is ten lines of script and a few minutes of compute, and it is the difference between a pull request that says the tests pass and one that says the numbers are the same. If the reviewer wants one more check, a follow-up to POST /o/{org}/sessions/{id}/messages wakes the same sandbox with both environments still installed.

Frequently asked questions

Should the upgrade go through pandas 2 first?

As a diagnostic step, yes. The last 2.x release still raises warnings for most things pandas 3 removes, so a run under -W error there names each problem with a message instead of a traceback. In bukosabino/ta the result at pandas 2.3.3 is the same two test errors as at 3.1.1, so the pull request goes straight to 3.

Why not just delete the removed test argument?

Because the test would then compare at the default tolerance, and the reference data was written for a looser one. In this repository deleting check_less_precise makes both tests fail on values that are correct. An explicit rtol keeps the comparison the test always made.

Will users of my package be affected?

That depends on which file the pin is in. A pin in a requirements file used for development changes nothing for users; a bound in install_requires or pyproject.toml changes what pip resolves for them. The agent reports which case applies before it edits anything.

More pandas jobs

More jobs like this: Upgrade dependencies weekly

Keep reading