What the agent does
The agent reads how the app starts and stops before it changes anything. In an app laid out like litestar-org/litestar-fullstack, src/py/app/server/asgi.py builds Litestar(plugins=[ApplicationCore()]), and ApplicationCore.on_app_init in src/py/app/server/core.py extends the plugin list with structlog, granian, alchemy, vite, the SAQ plugin, problem_details, oauth2_provider, email and domain. uv.lock pins granian 2.6.1 and litestar 2.19.0, and src/py/app/__main__.py sets LITESTAR_GRANIAN_IN_SUBPROCESS=false so the server runs in the CLI process. SaqSettings in src/py/app/lib/settings.py sets use_server_lifespan from SAQ_USE_SERVER_LIFESPAN, true by default, so the background-tasks queue with its two cron jobs, cleanup_auth_tokens hourly and refresh_oauth_tokens every 15 minutes, each with timeout=600, starts and stops with the web process. src/py/app/lib/worker.py already logs on_shutdown and logs a job that ends ABORTED in after_process, so the worker side has hooks; what is missing is an order, a bound and proof.
The agent then finds what each layer does today. tools/deploy/docker/Dockerfile ends with STOPSIGNAL SIGTERM, ENTRYPOINT ["tini", "--"] and CMD ["litestar", "run", "--host", "0.0.0.0", "--port", "8000"]; Dockerfile.worker runs app workers run the same way; docker-compose.yml starts the worker with litestar workers run and has health checks but no stop_grace_period, so Compose gives every container its default ten seconds before SIGKILL. Granian's --workers-kill-timeout, read from GRANIAN_WORKERS_KILL_TIMEOUT, is disabled by default, so a worker that refuses to stop is never killed by Granian itself. Advanced Alchemy's Litestar config disposes the engine and the session maker on shutdown already, so the agent does not add a second dispose.
The change is small and ordered. In on_app_init the agent appends an on_shutdown handler that runs first: it flips a draining flag on app.state, which SystemController.check_system_health in src/py/app/domain/system/controllers.py reads so /health answers 503 while the process drains, and it logs shutdown_started through the existing structlog logger. Then it sets GRANIAN_WORKERS_KILL_TIMEOUT in both Dockerfiles, so a request stuck past that bound ends the worker instead of blocking the deploy, and adds stop_grace_period to the app and worker services in docker-compose.yml, longer for the worker because a refresh_oauth_tokens job may be mid-flight. The new values are documented in the settings docstrings beside LITESTAR_KEEPALIVE, which the app already keeps at 65 seconds to outlast a load balancer's idle timeout.
The proof is two tests under src/py/tests. A unit test uses litestar.testing.TestClient as a context manager, which runs the lifespan, and asserts the draining flag is false inside the block and true after it, with shutdown_started in the captured log. An integration test starts litestar run as a subprocess on a free port, opens a request to a slow route, sends SIGTERM to the process, and asserts the response still arrives with status 200 and the process exits 0 within the grace period. Both run with make test, which is uv run pytest src/py/tests --quiet, the same command .github/workflows/ci.yaml runs on Python 3.12 and 3.13.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add graceful shutdown to this Litestar application. Read src/py/app/server/asgi.py, src/py/app/server/core.py, src/py/app/server/plugins.py, src/py/app/lib/settings.py, src/py/app/lib/worker.py, src/py/app/domain/system/controllers.py, tools/deploy/docker/Dockerfile, Dockerfile.worker and docker-compose.yml, and uv.lock for the granian and litestar versions. Add an on_shutdown handler in ApplicationCore.on_app_init that sets a draining flag on app.state first and logs shutdown_started; make /health answer 503 while draining. Do not add a second engine dispose: advanced-alchemy already disposes the engine. Set GRANIAN_WORKERS_KILL_TIMEOUT in both Dockerfiles and add stop_grace_period to the app and worker services in docker-compose.yml, longer for the worker because SAQ cron jobs run with timeout=600. Add a TestClient unit test for the flag and the log line, and an integration test that starts litestar run as a subprocess, sends SIGTERM during a slow request, and asserts the response completes and the process exits 0. Run make test and open a pull request with the output and the new environment variables listed.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The pull request changes src/py/app/server/core.py and src/py/app/domain/system/controllers.py, adds the two tests under src/py/tests/unit/server/ and src/py/tests/integration/system/, and edits tools/deploy/docker/Dockerfile, Dockerfile.worker and docker-compose.yml. Its description pastes the make test output and lists GRANIAN_WORKERS_KILL_TIMEOUT and the two stop_grace_period values with the reason for each number.
On a deploy, the sequence is now visible in the logs: shutdown_started, the SAQ worker's Stopping background workers for queue, and the engine dispose, in that order, and /health answers 503 from the first line on so a load balancer stops sending new requests before the socket closes. A job that was running when the signal arrived either completes inside the worker's grace period or is logged as aborted by after_process, never silently lost.
Before merging, a reviewer checks the grace periods against the platform's own termination timeout, since a Kubernetes terminationGracePeriodSeconds or an ECS stop timeout shorter than Compose's value wins, and decides whether the slow route the integration test uses should stay as a test-only route or be replaced by a sleep inside an existing one.
Run it on every service that restarts under load
A platform team with several Litestar services sends this body once per repository to POST /o/{org}/sessions from a script, each with its own repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retry returns the run already started. Each run reads its own deploy files: a service on uvicorn gets --timeout-graceful-shutdown instead of the Granian variable, one without SAQ gets no worker grace period, and one whose health check already reads a flag gets only the tests.
Each run opens its own pull request on its own branch, and GET /o/{org}/sessions lists them newest first, so the person who started the batch reads every test result from one place and merges the services in the order they deploy.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
Shutdown bugs only show up on a deploy, where nobody is watching the one request that was cut off. The agent reproduces the moment on purpose: it starts the real server in its sandbox, sends the real signal during a real request, and keeps the exit code and the log lines as the test's assertions, so the behaviour is checked on every pull request instead of discovered from a 502 in production.
The run keeps every command and its output, so the reviewer sees the subprocess test pass with the signal timing without checking out the branch or installing uv, Postgres and Redis locally. Several services can be fixed at once, each in its own sandbox, and a service whose shutdown is already ordered is left with a note rather than a redundant hook.
Frequently asked questions
Why does /health go to 503 before the server stops listening?
A load balancer keeps routing to an instance until a health check fails, so a server that closes its socket while still marked healthy drops the requests in flight between the two. Flipping the flag in the first shutdown hook gives the balancer a failed check to act on while the in-flight requests finish. The flag lives on app.state, which the existing health controller can read without a new dependency.
Does the SAQ worker finish its job or abandon it?
With use_server_lifespan true the worker stops with the app, and a running job gets the grace period to complete. The cron jobs carry timeout=600, so the worker container's stop_grace_period is set longer than the web container's. A job that still does not finish is logged as aborted by after_process, which already handles the ABORTED status.
What does the Granian kill timeout change?
Nothing on a clean stop. It is the upper bound for a worker that refuses to stop, for example one blocked in a synchronous call. Without it Granian waits, and the container runtime's own SIGKILL arrives with no log line explaining why. With it, the worker is killed at a known point and the event appears in Granian's own output.
Will the integration test be flaky in CI?
It binds a free port, waits for the health route to answer before sending the signal, and asserts with a timeout rather than a fixed sleep. It needs Postgres, which the existing test setup already provides through pytest-databases, so it runs where the rest of the integration suite runs. If the sandbox cannot start the server, the run says so rather than skipping the test.
Does this change the distroless image?
Dockerfile.distroless keeps its tini entrypoint and app run command; the agent adds the same GRANIAN_WORKERS_KILL_TIMEOUT there so the three images behave alike. No new package is installed in any image.