All jobs

Migrate a Python Test Suite From unittest to pytest

Yes. An agent moves unittest classes to pytest functions and fixtures in steps, keeping every test running at each step and explaining any change in the test count.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

pytest runs unittest.TestCase classes as they are, so the agent's first step is to run the existing suite under pytest unchanged. That shows what the migration is really made of. In a project like pglass/cube, a Rubik's cube solver, all tests sit in one file, tests/test.py: 84 test methods in five classes (TestPoint, TestMatrix, TestCube, TestSolver, TestOptimize), run by python tests/test.py through the unittest.main() call on the last line. Typing pytest in that repository collects nothing, because pytest only looks for files named test_*.py or *_test.py and this one is test.py. pytest tests/test.py fails differently, with No module named 'rubik', unless the package is installed, while python -m pytest tests/test.py passes because that form puts the current directory on the import path. None of this is about assertions, and all of it has to be settled before a single test is rewritten.

So the first commit changes no test code. The agent renames the file to tests/test_cube.py, adds a pytest section to the project's configuration with testpaths and pythonpath, and replaces the command in the three places that hard-code it: the Unit test step of .github/workflows/test.yaml, the test target of the Makefile, and commands in tox.ini. It adds pytest where each of those installs its tools, since setup.py has an empty install_requires and no test extra. After this commit the suite reports 84 passed under pytest, and that number is the baseline for everything after.

The second commit is the mechanical rewrite, and the file's own counts say how mechanical: 117 self.assertEqual calls, a handful of assertTrue, assertFalse, assertNotEqual and assertIsNone, one assertRaises and one assertRaisesRegex. Each becomes a bare assert or a pytest.raises block, with match= carrying the message "Stuck in loop - unsolvable cube" for the regex case. The three setUp methods become fixtures. TestPoint.setUp builds three points and TestCube.setUp builds a solved cube and a debug cube; the agent turns each attribute into a named fixture, so a test asks for solved_cube instead of reading self.solved_cube, and drops the classes where nothing else holds them together.

The third commit is the part worth doing by judgement, and the agent keeps it separate so you can decline it. TestSolver.test_cube_solver loops over five scrambled cubes inside one test and wraps each in a try that calls self.fail(traceback.format_exc() + "original cube: " + orig), hand-built reporting that exists only because a loop hides which input failed. With pytest.mark.parametrize each cube is its own test with its own id, and the wrapper goes. The same applies to the five unsolvable cubes and to the twelve move pairs TestOptimize iterates over. This is the one place the test count changes, and the agent states the arithmetic in the pull request so that a larger number reads as the same checks reported separately, not as new coverage.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Migrate this project's tests from unittest to pytest, in separate commits. First read how tests run today: the test files and their names, the command in the CI workflow, the Makefile, tox.ini or noxfile, and how the package gets onto the import path. Run the existing suite the documented way and record the number of tests. Commit 1, no test code changes: make plain `pytest` from the repository root collect and pass the same number of tests. Rename files that pytest's default patterns would not collect, add pytest configuration (testpaths, and pythonpath if the package is not installed), add pytest to the development dependencies, and replace the test command everywhere it is written. Commit 2, mechanical: turn self.assert* calls into plain asserts, assertRaises and assertRaisesRegex into pytest.raises with match, setUp and tearDown into fixtures, and remove TestCase classes that no longer share anything; remove unittest.main(). The count must equal the baseline. Commit 3, optional and separate: where one test loops over a list of inputs, use pytest.mark.parametrize, and remove try/except blocks that only existed to say which input failed. State the new count and the arithmetic behind it. Do not change the code under test. Do not change what any test checks; if an assertion looks wrong, report it instead of fixing it. Run the linter the project uses on the result. Open a pull request with the pytest summary after each commit.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request with three commits that can be reviewed, and reverted, one at a time. The first touches the workflow, the Makefile, tox.ini, the configuration file and one rename, and its message carries the output 84 passed. The second is a long diff of one file in which every hunk is the same few substitutions. The third is short and holds the parametrized tests. The description gives the test count after each commit and, for the last, a line per parametrized test saying how many cases it now reports.

A reviewer reads the second commit for the two things a mechanical rewrite can get wrong. One is argument order: assertEqual(expected, actual) becomes assert expected == actual, and the agent keeps the order the file used so failure messages still read the same way. The other is an assertion that carried a msg= argument, which must survive as the message after the comma. The pull request also says what it did not do: the python-version matrix in the workflow and the ruff configuration in .ruff.toml are unchanged.

Run it on every repository still on unittest

This is a one-time job per repository, so the usual way to repeat it is a loop over repositories on your side. Send the request once per repository, with that repository as the resources URL and a fresh request_id, a UUID you generate per repository, so that a retried call returns the run it already started. Runs past the key's max_concurrent limit are created queued and start as slots free, so the loop needs no pacing of its own.

To let people ask for it, add an automation on each Project with a github trigger on issues.labeled and a label such as pytest-migration. The labelled issue is appended after the prompt as event text; the prompt above stays the only instruction.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

A test migration is only trustworthy if the suite was run before, between and after, and the counts were compared. That is tedious on a laptop and nobody attaches the output. The agent runs the suite at every commit in a sandbox with nothing preinstalled, which is also how it finds problems like a package that imports only because of how the command was typed. Every command and its output stays with the run, so the 84 passed in the commit message can be traced to the invocation that printed it.

The work is also long and uniform, the kind a person starts carefully and finishes carelessly. Handing it off from a terminal means the hundred-odd substitutions happen while you do something else, and what comes back is three commits whose claims you can check by running one command each.

Frequently asked questions

Do I have to rewrite my tests to use pytest at all?

No. pytest collects and runs unittest.TestCase subclasses, including setUp and tearDown, so switching the runner is the first commit and can be the only one. What you gain from rewriting is fixtures you can share across files, parametrize, and failure output that shows both sides of a plain assert. Fixtures and parametrize do not work on TestCase methods, which is the reason to convert the classes.

Why does pytest say no tests ran?

Usually the file name. By default pytest collects test_*.py and *_test.py, so a file called test.py or tests.py is skipped unless you pass its path. Rename the file or set python_files in the pytest configuration. The agent renames, because a default that works for every contributor is better than a setting they must know about.

Will the number of tests change?

Not in the first two commits; the agent treats a different number there as a bug in its own work. It changes only where a loop inside one test becomes a parametrized test, because each input is then counted separately. The pull request lists those tests and their case counts so the total can be checked by hand.

Can it keep some tests on unittest?

Yes. Both styles run in one pytest session, so a class that depends on TestCase features your team wants to keep, such as subTest, can stay as it is. Say which files or classes to leave alone in the prompt and the agent converts the rest.

More unittest jobs

More jobs like this: Hand off a long task

Keep reading