# Add Graceful Shutdown to a Litestar App

Yes. An agent adds ordered shutdown hooks, a Granian kill timeout, a Docker stop grace period and a test that SIGTERM finishes in-flight work, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-graceful-shutdown/litestar/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads how the app starts and stops before it changes anything. In an app laid out like litestar-org/litestar-fullstack, `src/py/app/server/asgi.py` builds `Litestar(plugins=[ApplicationCore()])`, and `ApplicationCore.on_app_init` in `src/py/app/server/core.py` extends the plugin list with `structlog`, `granian`, `alchemy`, `vite`, the SAQ plugin, `problem_details`, `oauth2_provider`, `email` and `domain`. `uv.lock` pins `granian` 2.6.1 and `litestar` 2.19.0, and `src/py/app/__main__.py` sets `LITESTAR_GRANIAN_IN_SUBPROCESS=false` so the server runs in the CLI process. `SaqSettings` in `src/py/app/lib/settings.py` sets `use_server_lifespan` from `SAQ_USE_SERVER_LIFESPAN`, true by default, so the `background-tasks` queue with its two cron jobs, `cleanup_auth_tokens` hourly and `refresh_oauth_tokens` every 15 minutes, each with `timeout=600`, starts and stops with the web process. `src/py/app/lib/worker.py` already logs `on_shutdown` and logs a job that ends `ABORTED` in `after_process`, so the worker side has hooks; what is missing is an order, a bound and proof.

The agent then finds what each layer does today. `tools/deploy/docker/Dockerfile` ends with `STOPSIGNAL SIGTERM`, `ENTRYPOINT ["tini", "--"]` and `CMD ["litestar", "run", "--host", "0.0.0.0", "--port", "8000"]`; `Dockerfile.worker` runs `app workers run` the same way; `docker-compose.yml` starts the worker with `litestar workers run` and has health checks but no `stop_grace_period`, so Compose gives every container its default ten seconds before `SIGKILL`. Granian's `--workers-kill-timeout`, read from `GRANIAN_WORKERS_KILL_TIMEOUT`, is disabled by default, so a worker that refuses to stop is never killed by Granian itself. Advanced Alchemy's Litestar config disposes the engine and the session maker on shutdown already, so the agent does not add a second dispose.

The change is small and ordered. In `on_app_init` the agent appends an `on_shutdown` handler that runs first: it flips a `draining` flag on `app.state`, which `SystemController.check_system_health` in `src/py/app/domain/system/controllers.py` reads so `/health` answers 503 while the process drains, and it logs `shutdown_started` through the existing structlog logger. Then it sets `GRANIAN_WORKERS_KILL_TIMEOUT` in both Dockerfiles, so a request stuck past that bound ends the worker instead of blocking the deploy, and adds `stop_grace_period` to the `app` and `worker` services in `docker-compose.yml`, longer for the worker because a `refresh_oauth_tokens` job may be mid-flight. The new values are documented in the settings docstrings beside `LITESTAR_KEEPALIVE`, which the app already keeps at 65 seconds to outlast a load balancer's idle timeout.

The proof is two tests under `src/py/tests`. A unit test uses `litestar.testing.TestClient` as a context manager, which runs the lifespan, and asserts the draining flag is false inside the block and true after it, with `shutdown_started` in the captured log. An integration test starts `litestar run` as a subprocess on a free port, opens a request to a slow route, sends `SIGTERM` to the process, and asserts the response still arrives with status 200 and the process exits 0 within the grace period. Both run with `make test`, which is `uv run pytest src/py/tests --quiet`, the same command `.github/workflows/ci.yaml` runs on Python 3.12 and 3.13.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add graceful shutdown to this Litestar application. Read src/py/app/server/asgi.py, src/py/app/server/core.py, src/py/app/server/plugins.py, src/py/app/lib/settings.py, src/py/app/lib/worker.py, src/py/app/domain/system/controllers.py, tools/deploy/docker/Dockerfile, Dockerfile.worker and docker-compose.yml, and uv.lock for the granian and litestar versions. Add an on_shutdown handler in ApplicationCore.on_app_init that sets a draining flag on app.state first and logs shutdown_started; make /health answer 503 while draining. Do not add a second engine dispose: advanced-alchemy already disposes the engine. Set GRANIAN_WORKERS_KILL_TIMEOUT in both Dockerfiles and add stop_grace_period to the app and worker services in docker-compose.yml, longer for the worker because SAQ cron jobs run with timeout=600. Add a TestClient unit test for the flag and the log line, and an integration test that starts litestar run as a subprocess, sends SIGTERM during a slow request, and asserts the response completes and the process exits 0. Run make test and open a pull request with the output and the new environment variables listed.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

The pull request changes `src/py/app/server/core.py` and `src/py/app/domain/system/controllers.py`, adds the two tests under `src/py/tests/unit/server/` and `src/py/tests/integration/system/`, and edits `tools/deploy/docker/Dockerfile`, `Dockerfile.worker` and `docker-compose.yml`. Its description pastes the `make test` output and lists `GRANIAN_WORKERS_KILL_TIMEOUT` and the two `stop_grace_period` values with the reason for each number.

On a deploy, the sequence is now visible in the logs: `shutdown_started`, the SAQ worker's `Stopping background workers for queue`, and the engine dispose, in that order, and `/health` answers 503 from the first line on so a load balancer stops sending new requests before the socket closes. A job that was running when the signal arrived either completes inside the worker's grace period or is logged as aborted by `after_process`, never silently lost.

Before merging, a reviewer checks the grace periods against the platform's own termination timeout, since a Kubernetes `terminationGracePeriodSeconds` or an ECS stop timeout shorter than Compose's value wins, and decides whether the slow route the integration test uses should stay as a test-only route or be replaced by a sleep inside an existing one.

## Run it on every service that restarts under load

A platform team with several Litestar services sends this body once per repository to `POST /o/{org}/sessions` from a script, each with its own repository in `resources` and a `request_id` that is a UUID v5 of the repository's full name, so a retry returns the run already started. Each run reads its own deploy files: a service on uvicorn gets `--timeout-graceful-shutdown` instead of the Granian variable, one without SAQ gets no worker grace period, and one whose health check already reads a flag gets only the tests.

Each run opens its own pull request on its own branch, and `GET /o/{org}/sessions` lists them newest first, so the person who started the batch reads every test result from one place and merges the services in the order they deploy.

## Why run it on Rush?

Shutdown bugs only show up on a deploy, where nobody is watching the one request that was cut off. The agent reproduces the moment on purpose: it starts the real server in its sandbox, sends the real signal during a real request, and keeps the exit code and the log lines as the test's assertions, so the behaviour is checked on every pull request instead of discovered from a 502 in production.

The run keeps every command and its output, so the reviewer sees the subprocess test pass with the signal timing without checking out the branch or installing uv, Postgres and Redis locally. Several services can be fixed at once, each in its own sandbox, and a service whose shutdown is already ordered is left with a note rather than a redundant hook.

## Frequently asked questions

### Why does /health go to 503 before the server stops listening?

A load balancer keeps routing to an instance until a health check fails, so a server that closes its socket while still marked healthy drops the requests in flight between the two. Flipping the flag in the first shutdown hook gives the balancer a failed check to act on while the in-flight requests finish. The flag lives on `app.state`, which the existing health controller can read without a new dependency.

### Does the SAQ worker finish its job or abandon it?

With `use_server_lifespan` true the worker stops with the app, and a running job gets the grace period to complete. The cron jobs carry `timeout=600`, so the worker container's `stop_grace_period` is set longer than the web container's. A job that still does not finish is logged as aborted by `after_process`, which already handles the `ABORTED` status.

### What does the Granian kill timeout change?

Nothing on a clean stop. It is the upper bound for a worker that refuses to stop, for example one blocked in a synchronous call. Without it Granian waits, and the container runtime's own `SIGKILL` arrives with no log line explaining why. With it, the worker is killed at a known point and the event appears in Granian's own output.

### Will the integration test be flaky in CI?

It binds a free port, waits for the health route to answer before sending the signal, and asserts with a timeout rather than a fixed sleep. It needs Postgres, which the existing test setup already provides through `pytest-databases`, so it runs where the rest of the integration suite runs. If the sandbox cannot start the server, the run says so rather than skipping the test.

### Does this change the distroless image?

`Dockerfile.distroless` keeps its `tini` entrypoint and `app run` command; the agent adds the same `GRANIAN_WORKERS_KILL_TIMEOUT` there so the three images behave alike. No new package is installed in any image.

## More Litestar jobs

- [Add a Health Check Endpoint to a Litestar App](https://getrush.ai/run/add-health-checks/litestar/)
- [Add JWT Authentication to a Litestar App](https://getrush.ai/run/add-jwt-authentication/litestar/)
- [Add OpenTelemetry Tracing to a Litestar App](https://getrush.ai/run/add-opentelemetry/litestar/)

## The same job on other stacks

- [Add Graceful Shutdown to an Express and Mongoose API](https://getrush.ai/run/add-graceful-shutdown/mongoose/)
- [Add Graceful Shutdown to a Node.js Server](https://getrush.ai/run/add-graceful-shutdown/nodejs/)
- [Add Graceful Shutdown and Zero-Downtime Reloads with PM2](https://getrush.ai/run/add-graceful-shutdown/pm2/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
