All jobs

Write Rack::Test Tests for a Sinatra Service That Has None

Yes. An agent adds Minitest and Rack::Test, posts real payloads to your routes, checks the response, the email and the span, and opens a pull request.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent clones the repository into its own sandbox and reads the service as a whole, since a Sinatra app with no tests usually does real work when it loads. In open-telemetry/opentelemetry-demo, src/email is one classic-style file, email_server.rb, with a Gemfile for Sinatra 4.2, Pony, Puma, the OpenTelemetry SDK and exporters, and the OpenFeature SDK with the flagd provider, and no test gem at all. The Dockerfile builds on Ruby 4.0.5 and Gemfile.lock says BUNDLED WITH 4.0.10, so the agent installs that Ruby and that Bundler in the sandbox before bundle install. Loading the file configures a flagd client for FLAGD_HOST, calls OpenTelemetry::SDK.configure with the Sinatra instrumentation, and attaches an OTLP metrics exporter.

It reads the one route end to end. POST /send_order_confirmation parses the body with JSON.parse(..., object_class: OpenStruct), sets demo.order.id on the request span, adds one to the demo.notification.confirmations counter and calls send_email. That opens a send_email span, reads the emailMemoryLeak flag (default 0), renders views/confirmation.erb, pads the body with spaces when the flag is above 1, and sends it with Pony via: :test to data.email from noreply@example.com. When the flag is below 1 it clears Mail::TestMailer.deliveries straight away, so with the default flag a test could never see the email it sent. The payload comes from the checkout service in Go, which marshals the OrderResult protobuf with encoding/json, so keys are snake case and zero values are left out; the test fixtures follow that.

The tests use Minitest and Rack::Test, added to the Gemfile in a test group. Before requiring the app they set APP_ENV=test and set OTEL_TRACES_EXPORTER and OTEL_LOGS_EXPORTER to none, which the locked SDKs read, so nothing tries to reach a collector, then add a SimpleSpanProcessor with the SDK's InMemorySpanExporter to OpenTelemetry.tracer_provider. Each test sets OpenFeature's InMemoryProvider from openfeature-sdk 0.8.1 with the emailMemoryLeak value it needs: 1 to keep the delivery so the test can read its recipient, sender, subject and rendered order, and 2 to check the padding doubles the body. Sinatra sets raise_errors in the test environment, so the malformed-JSON and missing-order tests turn it off to check the 500 and that the error block recorded the exception on the span.

Rendering the template with real values exposes how it prints money: <%= order.shipping_cost.units %>.<%= order.shipping_cost.nanos %> prints nanos as a raw integer, so a cost of 5 units and 50,000,000 nanos, five cents, renders as 5.50000000, and a whole-unit cost, whose zero nanos the Go side omits, renders as 5.. The agent writes tests that pin today's output and lists both as findings rather than changing the view. It runs bundle exec ruby -Itest over the test files, or a Rakefile task if it adds one, until everything passes, and adds bundle config set --local without test to the Dockerfile's builder stage so the test gems stay out of the image.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Write tests for the Sinatra email service in src/email, which has none. Read email_server.rb, views/confirmation.erb, the Gemfile, Gemfile.lock, the Dockerfile and how src/checkout builds the request it posts first. Install the Ruby version the Dockerfile uses and the Bundler version in Gemfile.lock. Add minitest and rack-test in a test group. Before loading the app, set APP_ENV=test and set OTEL_TRACES_EXPORTER and OTEL_LOGS_EXPORTER to none; collect spans with an InMemorySpanExporter on a SimpleSpanProcessor. Replace the flagd provider with OpenFeature's InMemoryProvider in each test and set emailMemoryLeak as the case needs. Cover a valid order with the email's recipient, sender, subject and rendered order, the demo.order.id attribute and the send_email span, the padding when the flag is 2, malformed JSON and a missing order as 500s with the exception recorded. Build fixtures the way checkout serializes them. Do not change email_server.rb or the view; list anything that looks wrong as a finding. Keep the test gems out of the Docker image. Run the tests until they pass. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request scoped to src/email: a test group in the Gemfile with the matching Gemfile.lock change, a test/ directory with a helper that sets the environment and the span exporter and a test file for the route, a fixture order shaped like the checkout service's JSON, and one line in the Dockerfile that keeps the test group out of the image. email_server.rb and views/confirmation.erb are unchanged.

The description lists each test with what it asserts, and the findings: the money formatting in the template, and anything else the agent noticed, such as the route returning an empty 200 body because send_email ends with puts. The session transcript keeps the Ruby and Bundler installs, bundle install, every test run including the failures, and the final passing run.

Before merging, a reviewer checks that the tests clear Mail::TestMailer.deliveries themselves between cases, that no test depends on a flagd server, and whether the money formatting should be fixed in a follow-up now that a test pins it.

Run it on every service with no tests

This repository has services in many languages, and a small one like src/email is easy to leave without a unit test. A script on your own server can send one POST /o/{org}/sessions per service directory with the directory and its language in input, each in its own sandbox with its own toolchain: Ruby for src/email, PHP for src/quote, Python for src/recommendation. A key runs at most its max_concurrent sessions at once (4 by default), and later starts answer 202 with status: "queued" until a slot frees.

Set a webhook_url on the key and Rush posts run.completed or run.failed as each run's turn ends, with a session_url whose read returns the pull request link and the agent's summary of the findings.

The use case this job follows, with its automation and trigger: Write the missing tests.

Why run it on Rush?

Testing this service on a laptop means installing Ruby 4.0.5 and Bundler 4.0.10 beside whatever Ruby the laptop already has, compiling native gems, and making sure nothing in the test process tries to reach flagd or an OpenTelemetry collector. In a Rush sandbox the agent installs exactly the versions the Dockerfile and lockfile name, on a machine that exists for this run, and the test run proves the service loads without the rest of the demo.

The record is the evidence. Every install and test run is in the transcript with its output, and the findings in the pull request come from rendered output a test pins, not from reading the template and guessing.

Frequently asked questions

Will it convert the app to the modular Sinatra::Base style?

No. Rack::Test works with a classic app through Sinatra::Application, so the tests load email_server.rb as it is. Rewriting the app's style is a separate change you can ask for.

Does it need flagd or a collector running?

No. The tests replace the flagd provider with OpenFeature's InMemoryProvider and set the trace and log exporters to none, collecting spans in memory instead. Nothing outside the test process is called.

Why not RSpec?

Minitest ships with Ruby and needs no configuration, which suits a one-file service with no test setup at all. If your other Ruby services use RSpec, say so in input and the agent writes the same tests as specs.

Will the test gems end up in the production image?

No. The agent puts them in a test group and adds bundle config set --local without test before bundle install in the Dockerfile's builder stage, and says so in the pull request.

The same job on other stacks

Keep reading