What the agent does
The agent reads the build before the tests. In a library like kevinsawicki/http-request the root pom.xml is a parent with one module, and lib/pom.xml declares junit:junit 4.10 in test scope, Jetty 8.1.9 for an embedded test server, maven-compiler-plugin 2.3.2 with source and target set to 1.5, and no maven-surefire-plugin entry at all. The tests are three files under lib/src/test/java: EncodeTest with 2 tests, HttpRequestTest with 161, and their base class ServerTestCase, which starts the server. They use @BeforeClass, @AfterClass, @Before, @After and four @Test(expected = ...) methods. There are no rules and no runners, and no assertion passes a message, so the changed position of the message argument in Jupiter's assertions, a classic source of silent mistakes, cannot bite here. The agent checks that by counting arguments, not by assuming.
It runs mvn test before changing anything, and on a current JDK the baseline fails before a single test: javac in JDK 17 answers Source option 5 is no longer supported. Use 7 or later. The agent raises the compiler level in a scratch commit only as far as needed to get a baseline, which then shows all 163 tests passing on JUnit 4. It keeps that as its reference and flags the level as a decision for the maintainers, because the same POM stamps the jar with Bundle-RequiredExecutionEnvironment J2SE-1.5 and JUnit Jupiter itself needs Java 8, as do the lambdas that assertThrows wants.
The mechanical conversion is the easy part. OpenRewrite's JUnit4to5Migration rewrites the four files: the dependency becomes org.junit.jupiter:junit-jupiter, lifecycle annotations and imports change, and each expected-exception test becomes an assertThrows call. Then comes the first trap. mvn test reports BUILD SUCCESS, and the summary reads Tests run: 0. The POM pins no Surefire, so the version comes from Maven's defaults, and Maven 3.8 binds Surefire 2.12.4, which predates JUnit Platform support (added in 2.22.0) and simply finds nothing to run. A green build with zero tests is exactly what a hurried migration merges. The agent treats a test count lower than the baseline as a failure and pins a current maven-surefire-plugin in the POM.
With tests running again, the second trap appears: nine of the 163 fail on code that passed minutes earlier. Jupiter runs methods in a deterministic order that is deliberately not the declaration order, and this suite has one test that leaks. customConnectionFactory installs a static factory through HttpRequest.setConnectionFactory that sends every request to the test server's base URL, and never puts the default back; only a later test, nullConnectionFactory, did that by accident of ordering. Under the new order other tests run in between, lose their query strings and paths, and one fails a cast to HttpsURLConnection. The agent traces the failures to that shared state and resets the factory in the existing @AfterEach method. It does not pin a method order to hide the dependency. All 163 pass.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "codex",
"input": "Migrate this Maven project's tests from JUnit 4 to JUnit Jupiter. Read every pom.xml, the CI config and all test sources first. Report the JUnit version, whether maven-surefire-plugin is pinned and to what, the compiler source and target, and an inventory of JUnit 4 features in use: lifecycle annotations, expected exceptions, timeouts, rules, runners, categories, assumptions, and assertions that pass a message argument (count arguments, do not guess). Install a JDK and Maven from system packages if they are missing. Run mvn test on the untouched code and record the number of tests run; if the build fails for a reason unrelated to JUnit, such as a source level the JDK no longer supports, make the smallest change that yields a baseline, keep it in its own commit and flag it. Convert the tests, using OpenRewrite's JUnit 4 to 5 migration or by hand, and stay on the latest 5.x release unless the project already requires Java 17. After converting, run mvn test and compare the number of tests run with the baseline: a lower number, including zero with a successful build, is a failure you must fix by pinning a Surefire version that supports the JUnit Platform. If tests that passed before now fail, find the cause; if it is shared static state exposed by the new method order, fix the cleanup rather than forcing an order. Do not change production code or weaken assertions. Remove the junit:junit dependency and confirm no org.junit import outside org.junit.jupiter remains. Open a pull request with the inventory, the baseline and final test counts, each trap you hit with the output that showed it, and the open question about the minimum Java version.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request changes lib/pom.xml and the three test files. In the POM, junit:junit is replaced by junit-jupiter, a maven-surefire-plugin version is pinned for the first time, and the compiler level change sits in its own commit so it can be discussed or replaced on its own. In the tests, the diff is imports, annotations, four assertThrows conversions and one added line in the cleanup method. No assertion is removed or loosened, and nothing under src/main is touched.
The description is built around two numbers a reviewer can verify: tests run before and tests run after, with the Surefire summary lines for each. Between them it shows the intermediate states, the run that reported zero tests with a successful build and the run with nine failures, each with the output and the explanation. That record is the reason to trust the final count; a migration that only shows a green build at the end has not shown that the tests executed.
One question is left for a person and stated as such: what Java version the library should target now. The baseline needed a level above 1.5 just to compile on a current JDK, and Jupiter needs Java 8 for the tests. Whether the published artifact's own minimum moves with it, and whether the bundle metadata should change, is a compatibility decision the agent does not make on its own. The stale .travis.yml, which names no JDK, is mentioned and left alone.
Run it on every repository that still depends on junit:junit
Most organisations have this dependency in many repositories. List them from your dependency graph or a search for <artifactId>junit</artifactId>, and for each one send POST /o/{org}/sessions with that repository in resources and the prompt above as input. Each run works in its own sandbox with its own Maven cache, so one repository's odd plugin setup cannot affect another's result. Runs past the key's max_concurrent are accepted as queued and start in order as earlier ones finish.
Make the batch safe to re-run by sending a request_id with each call: a UUID v5 of the repository's full name works, since the same name always yields the same UUID and a repeated call returns the run it already started. Sort the results by the difference between the two test counts. Equal counts and a short trap list can be reviewed quickly; a run that reports shared state it had to untangle deserves a careful read.
The use case this job follows, with its automation and trigger: Upgrade dependencies weekly.
Why run it on Rush?
Every hard part of this migration only shows when the build runs: the JDK refusing an old source level, the runner that silently finds no tests, the failures that depend on execution order. None of them is visible in the diff, and two of them end in a green build. The agent runs Maven at each step in its sandbox and keeps the output, so the pull request can quote the line that said zero tests ran and the line that said 163 did. The full transcript of commands is at GET /o/{org}/sessions/{id}/messages.
A clean machine also helps the diagnosis. Maven's default plugin versions depend on the Maven that runs the build, and a developer's ~/.m2 and IDE can mask what CI will do. The sandbox starts without that history, which is how a build that works on a laptop and reports no tests elsewhere is caught before merge instead of after.
Frequently asked questions
Can JUnit 4 and JUnit 5 tests run side by side while I migrate?
Yes. Adding the junit-vintage-engine lets the JUnit Platform run existing JUnit 4 tests next to Jupiter ones, so a large suite can move a package at a time. For a suite of three files the agent converts everything at once and removes junit:junit, which leaves nothing on the old API to maintain.
Why not go straight to JUnit 6?
JUnit 6 requires Java 17. A project whose build still declares an older Java level would have to raise its minimum first, which is a larger decision than a test migration. The agent moves to the latest 5.x release, which needs Java 8, and notes that the next step is small once the Java baseline allows it.
Does it handle rules and custom runners?
It inventories them first, because they do not convert mechanically. Common rules such as temporary folders and expected exceptions have direct Jupiter replacements. A custom runner needs an extension written for it, and the agent lists each one with the approach it took or the reason it left that class on the Vintage engine.
How do I know the tests still run in CI after merging?
Check the test count in the CI log against the count in the pull request. The agent pins the Surefire version in the POM so the count does not depend on which Maven the CI image ships, and that pin is the line to keep if anyone later tidies the build.
More jobs like this: Upgrade dependencies weekly
- Migrate dgrijalva/jwt-go to golang-jwt in a Go Auth Service (jwt-go)
- Replace lazy_static With std::sync::LazyLock in a Rust Project (lazy_static)
- Migrate a chi API from Logrus to log/slog (chi)
- Migrate a Sphinx Project From m2r to m2r2 (m2r)
- Migrate a React App From Material-UI v4 to MUI v5 (Material-UI)
- Migrate From mgo to the Official MongoDB Go Driver (mgo)