# Fix Flaky Tests in a Rails App An agent loops your flaky RSpec example until it fails, finds the order, clock or shared-record cause, fixes it, and shows 50 passing runs in a PR. [Run this on Rush](https://cloud.getrush.ai/run/fix-flaky-tests/rails/), in a Project that has your repository. ## What the agent does In a Rails app like lobsters/lobsters, the agent sets up the suite the way CI does. .github/workflows/check.yml copies config/database.yml.sample to config/database.yml, runs ./bin/rails db:schema:load db:migrate and then bundle exec rspec. The app runs Ruby 4.0.0 from .ruby-version, Rails 8.1.3.1 and rspec-rails 8.0.2 from Gemfile.lock, and SQLite through the sqlite3 gem, so the sandbox needs no database server: the agent installs that Ruby, runs bundle install and the same two commands, then runs the spec named in the alert. Next it reproduces the failure. It runs bundle exec rspec on the example's file and line in a loop and counts failures. Lobsters runs specs in defined order, since config.order = :random is commented out in spec/spec_helper.rb, so a flake that never fails alone is a strong hint of order dependency; the agent reruns with --order random and records each --seed that fails, then uses rspec --bisect with a failing seed to find the smallest set of examples that reproduces it. The suite has specific places where state crosses examples, and the agent reads them before guessing. spec/rails_helper.rb creates a System user and category1 with three tags once in before(:suite), uses transactional fixtures and DatabaseCleaner with :transaction, but switches the strategy to :truncation for :js and :truncate examples and never sets it back, so the agent checks whether examples that run after one of those still find the suite-level records. It resets InactiveUser.instance_variable_set(:@inactive_user, nil) after each example because app/models/inactive_user.rb memoizes a User in a module instance variable. Time is controlled through ActiveSupport::Testing::TimeHelpers, and recorded HTTP goes through spec/support/vcr.rb and webmock. With the cause found, the agent fixes it at the source: freeze_time or travel_to around an assertion that reads the clock, a factory record instead of one created in before(:suite), a reset for a memoized value it found, or an order clause on a query whose result order was never defined. Retries, longer waits and pending marks are off the table. It then runs the example 50 times alone and the file under the failing seeds, and records the after counts. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "codex", "input": "The spec named in the alert below fails intermittently. Set up the app as .github/workflows/check.yml does (copy config/database.yml.sample, run bin/rails db:schema:load db:migrate), then run the example in a loop with bundle exec rspec until it fails. If it never fails alone, run with --order random, record the failing seeds and use --bisect to find the examples that cause it. Find why it fails (ordering, time, randomness, shared records, memoized state or a race), fix the spec or the code, run it 50 times to show it now passes every time, and open a pull request with the before and after counts and the seeds. Never add a retry, a sleep or a pending mark.\n\nAlert: ", "resources": [ { "url": "https://github.com/your-org/your-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back The pull request changes the spec and, when the spec exposed a real bug, the model or controller it tests. Its description states the cause in a sentence, the rspec loop with how many of its runs failed before the fix, any failing seed and the --bisect output that narrowed it down, and the 50 passing runs after. If the cause was global state, such as a memoized value in a module, the description says where it lives and how the reset was added. A reviewer checks that the change addresses the cause, not the symptom. A fix that stubs Time.now for one example is fine; a fix that wraps the assertion in a retry or loosens it to accept two values is not, and the prompt forbids it. When the agent cannot reproduce the failure, the result says so with the number of runs and seeds it tried, which is the evidence you need before quarantining the spec instead. ## Run it on every spec that fails then passes Most CI systems can tell you when an example failed and then passed on a rerun. Send those events to a handler that calls POST /o/{org}/sessions with the repository in resources and the spec path and log in the prompt, and use the spec path as the Idempotency-Key so repeated reports of the same flake start a single run. Your key's webhook posts when each run finishes. Store the pull request link against the spec, so the next report of the same flake points at the open fix instead of starting another investigation. ## Why run it on Rush? Bisecting an order-dependent spec means running large parts of the suite many times, and on a laptop that blocks everything else for as long as it takes. In a Rush sandbox the agent has its own Ruby, gems and SQLite files, can run rspec --bisect for as long as it needs, and keeps each command with its output, so the failure counts and seeds in the pull request are the ones it actually saw. Each flaky spec gets its own sandbox, so two investigations never share a database file or a seed. You can start one run per flaky example the day they are reported and review the fixes as separate pull requests. ## Frequently asked questions ### What if the spec passes every time alone? That usually means another example leaves state behind. The agent runs the file and suite with --order random, keeps the seeds that fail, and runs rspec --bisect with one of them to find the example that causes it. ### Will it turn on random ordering for the whole suite? Not by default, since that can expose other order dependencies in the same pull request. It uses random order to investigate and suggests enabling it permanently as a separate change. ### Does it work with system specs that use a browser? Yes. The sandbox has a browser, so Capybara specs tagged :js can run there. Those specs switch DatabaseCleaner to truncation in this app, which is one of the first things the agent checks when a nearby spec flakes. ### What does the agent need from the alert? The spec file and line or example name, and the CI log if you have it. The log helps it see the failure message and seed CI used, but the agent reproduces the failure itself before changing anything.