# Fix Flaky Tests in a Laravel App An agent reruns your flaky Laravel test until it fails, finds the cause in ordering, time or shared data, fixes it, and proves it with 50 clean runs. [Run this on Rush](https://cloud.getrush.ai/run/fix-flaky-tests/laravel/), in a Project that has your repository. ## What the agent does In a Laravel app like BookStackApp/BookStack, the agent first builds the environment the suite expects. phpunit.xml sets DB_CONNECTION to mysql_testing, CACHE_DRIVER and SESSION_DRIVER to array and QUEUE_CONNECTION to sync, and dev/docs/php-testing.md says the testing database must exist, then be migrated and seeded with composer refresh-test-database, which runs migrate:refresh and the DummyContentSeeder on mysql_testing. The agent starts MySQL in the sandbox, creates the bookstack-test database and user the docs name, runs composer install and that script, and only then runs the test named in the alert. Then it makes the failure happen on demand. It runs php vendor/bin/phpunit --filter with the test's name in a loop and counts failures, and if the test passes alone it runs the whole file and then the whole suite with --order-by=random, recording each --random-order-seed that fails. Those counts are the before numbers the pull request reports. A test that only fails after a particular neighbour is an order dependency; one that fails alone at random is time, randomness or a race. BookStack's tests have a few patterns the agent checks first. tests/TestCase.php wraps each test in DatabaseTransactions over the seeded data, and Tests\Helpers\EntityProvider returns the first page, chapter or book not yet fetched, ordered however the database returns it, so a test that edits a seeded record can depend on what ran before it. Time comes in through Carbon::setTestNow in tests/Settings/RecycleBinTest.php and $this->travel(2)->days() in tests/Auth/RegistrationTest.php, randomness through Str::random in tests such as tests/Uploads/ImageTest.php, and wall-clock timing through a sleep(4) command in tests/Exports/PdfExportTest.php. Once it has a cause, the agent fixes that cause: freezing time in the test instead of reading the clock, giving the test its own record instead of a shared seeded one, adding an explicit order to a query, or fixing the application code when the test exposed a real bug. It never adds a retry, a longer sleep or a skip. Then it runs the test 50 times alone and the file under several random seeds, and records the after counts. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "claude", "input": "The test named in the alert below fails intermittently. Set up the mysql_testing database the way dev/docs/php-testing.md describes and run composer refresh-test-database. Run the test in a loop with php vendor/bin/phpunit --filter until it fails, then the file and the suite with --order-by=random, recording failing seeds. Find why it fails (ordering, time, randomness, shared seeded data or a race), fix the test or the code, run it 50 times to show it now passes every time, and open a pull request with the before and after counts and the seeds you tried. Never add a retry, a sleep or a skip.\n\nAlert: ", "resources": [ { "url": "https://github.com/your-org/your-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back The pull request is usually small: the test method, sometimes a helper in tests/Helpers, and the application file only when the test found a real bug. The description names the cause in one sentence (for example, the test relied on the first seeded page not having been edited by an earlier test), shows the loop that reproduced it with the failure count out of the runs, the random seeds that failed, and the 50 passing runs after the fix. A reviewer checks that the fix removes the cause rather than hiding it: no retry, no longer sleep, no assertion relaxed to accept both outcomes. If the agent could not reproduce the failure in the sandbox at all, it says so with the number of runs it tried and the seeds it used, and does not open a pull request that only guesses. The full output of every loop is in the session transcript. ## Run it on every flaky test alert Point your CI's failure webhook, or whatever notices a test that failed and then passed on rerun, at a small handler that sends POST /o/{org}/sessions with the repository in resources and the test name and log pasted into the prompt. Use the test name as the Idempotency-Key for a day or so, so the same flake reported by ten builds starts one run, not ten. When the run finishes, the webhook on your key posts run.completed with the session_url, and the session carries the pull request. A run that could not reproduce the flake reports that too, which is still useful when deciding whether to quarantine the test. ## Why run it on Rush? Chasing a flaky test means running it hundreds of times against a real MySQL database, often for an hour, while the machine is useless for anything else. A Rush sandbox gives the agent its own MySQL server and PHP install, and every phpunit loop is kept with its output, so the before and after counts in the pull request are the actual runs, not a claim. Flakes rarely come one at a time. Each alert can start its own run in its own sandbox, so two flaky tests in different parts of the suite are investigated at once without sharing a database, and each comes back as a separate, reviewable change. ## Frequently asked questions ### Why not just retry the test in CI? A retry hides the failure without removing its cause, and the cause is sometimes a real bug in the app. The prompt forbids retries, sleeps and skips, so the only way the agent can make the test pass reliably is to fix what makes it fail. ### What if the test only fails after other tests run? The agent runs the file and then the suite with PHPUnit's --order-by=random and records which seeds fail. A seed that fails every time reproduces the order dependency, and the agent narrows it to the test that leaves state behind. ### Does it need my production database? No. The suite runs against the mysql_testing connection seeded from the DummyContentSeeder, which the agent builds inside the sandbox. Nothing outside the sandbox is touched. ### Why 50 runs? A test that failed one time in ten is very unlikely to pass 50 times in a row by luck. You can raise the number in the prompt for rarer flakes, and the pull request reports whatever count was used.