What the agent does
The agent reads the API and the one test file that exists. In a Flask app like miguelgrinberg/microblog, app/api/__init__.py registers a blueprint mounted at /api with users, errors and tokens. app/api/auth.py builds HTTPBasicAuth and HTTPTokenAuth from Flask-HTTPAuth 4.8.0: verify_password looks the user up and calls check_password, verify_token calls User.check_token, and both error handlers answer error_response(status), a JSON body with the status text. app/api/tokens.py has POST /api/tokens behind basic auth, answering {'token'}, and DELETE /api/tokens behind token auth, answering 204. In app/models.py, get_token reuses a token with more than a minute left and otherwise issues secrets.token_hex(16) for an hour, revoke_token moves the expiry one second into the past, and check_token answers no user for an expired one. app/api/users.py has user, users, followers and following reads, each paginated through to_collection_dict into items, _meta and _links, a POST /users that answers 400 with must include username, email and password fields or please use a different username, and a PUT /users/<id> that aborts 403 for anyone but the user. tests.py has a TestConfig with TESTING, an in-memory SQLite sqlite:// and ELASTICSEARCH_URL = None, and four tests, all on models; nothing requests a route. create_app builds the Redis client lazily, so no Redis is needed to call these routes.
The new tests go in tests.py beside UserModelCase, as an ApiCase with the same setUp and a self.client = self.app.test_client(). Auth first: POST /api/tokens with a wrong password answers 401 and the {'error': 'Unauthorized'} body from error_response; with the right one it answers a 32-character token, and a second call within the hour returns the same token, which pins the reuse rule in get_token. A request to /api/users with no header answers 401, with a made-up token 401, and with the issued token 200. After DELETE /api/tokens the same token answers 401, and a token whose token_expiration the test moves into the past is refused by check_token.
Then the routes. GET /api/users/<id> returns the to_dict shape with _links for self, followers, following and avatar and no email for another user; GET /api/users?per_page=2 returns two items, _meta.total_items, and a _links.next that leads to the rest. Followers and following reflect follow() calls made in the test. POST /api/users with a missing field answers the exact 400 message, with a taken username the other one, and with a good body answers 201 with a Location header pointing at the new user. PUT /api/users/<id> by another user answers 403, by the user updates about_me, and with a username already taken answers 400. Each assertion names the message the route produces, so a wording change is a test change on purpose.
Running them needs nothing beyond requirements.txt: the agent runs pip install -r requirements.txt in the sandbox and python tests.py, and the in-memory database starts empty for every test through setUp and tearDown. It also runs the four original tests, so the pull request shows the whole file passing and the count going from four to the new total. Where a test exposes a behavior worth a second look, such as DELETE /api/tokens answering 204 for an already-revoked token because expiry only moves backwards, the pull request says so rather than changing the route.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "tests.py only tests the models; the REST API under app/api has no tests. Add an ApiCase to tests.py using the same TestConfig and a test client. Cover: POST /api/tokens with a wrong password (401 with the error_response body) and a right one (a token, and the same token on a second call within the hour); /api/users with no header, a bad token and the real token; DELETE /api/tokens making the token invalid; an expired token refused. Cover the user routes: get user with its _links and no email for another user; the paginated list with per_page and _links.next; followers and following after follow(); POST /api/users with the exact 400 messages for a missing field and a taken username, and 201 with a Location header; PUT by another user 403, by the user updating about_me, and a taken username 400. Run python tests.py and open a pull request quoting the output and noting any route behavior worth a look, without changing the routes.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request changes tests.py only, adding ApiCase under the existing model tests. Its description quotes the unittest output with the number of tests run and the result, and a short list of what the tests pinned about the routes, such as the token reuse window and the 204 on a second revoke.
A reviewer reads the assertions for the exact 400 messages and the _links shape, checks that each test builds its own users rather than depending on order, and confirms no route changed. The transcript at GET /o/{org}/sessions/{id}/messages has the dependency install and the test run in full.
No Redis, Elasticsearch or mail server is involved: TestConfig turns search off, the Redis client is built lazily and never used by these routes, and TESTING keeps Flask-Mail from sending.
Run it on every nightly
Once the API has tests, the gaps that remain are the ones new routes open. Add an automation on the Project with a schedule trigger (cron and timezone) and a prompt that lists routes under app/api with no test in tests.py, writes tests for them in the same ApiCase style and opens a pull request, or reports that every route is covered and opens nothing.
A webhook URL on the API key reports runs your server starts with this body; a run an automation starts is not tied to a key, and its pull request shows up on the repository like any other. From your own server the same run starts with this body and a request_id that is a UUID v5 of the night's date, so a retried send returns the run already started.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
API tests are cheap to write and expensive to get right by hand: the status text in error_response, the reuse window in get_token, the Location header on a 201 are all details a test must match exactly, and the only way to know is to run them. The sandbox installs the pinned requirements, runs the whole file, and keeps the output in the transcript; a laptop's virtualenv and a half-run suite leave no record.
Several services with the same blueprint pattern are several runs, each on its own branch, so a reviewer reads one test file per service and the same unittest summary in each description.
Frequently asked questions
unittest or pytest?
unittest, because tests.py already uses it and requirements.txt does not list pytest. The new case follows the file's setUp and tearDown exactly. If you prefer pytest, say so and the agent adds it with a conftest.py and converts the file.
Why run against SQLite instead of the production database?
TestConfig already points at an in-memory SQLite, and these routes use nothing database-specific, so every test starts from an empty schema in milliseconds. A Postgres-backed run would prove the same assertions more slowly and needs a service the test file does not expect.
Will the tests change if I rename a route?
Yes, on purpose: each test requests the real path and asserts the real message, so a rename or a reworded error fails the suite until the test is updated with it. That is the behavior a test file for an API should have.
Can it fix what the tests expose?
In a follow-up message to the same session, POST /o/{org}/sessions/{id}/messages. This run keeps the routes as they are so the pull request shows today's behavior pinned first; the follow-up changes a route with its test beside it.
The same job on other stacks
- Write API Integration Tests for a Hono App (Hono)
- Write API Integration Tests for a NestJS App (NestJS)
- Write API Tests for a Strapi App (Strapi)
- Write API Tests for an Axum App (Axum)
- Write API Tests for a Django REST Framework API (Django REST Framework)
More jobs like this: Write the missing tests
- Write Tests for a Streamlit App (Streamlit)