# Add Graceful Shutdown to Services Running on Kubernetes

Yes. An agent finds which of your services drop requests on SIGTERM, adds signal handling, preStop hooks and grace periods that match, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-graceful-shutdown/kubernetes/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent clones the repository into its own cloud sandbox and reads both the manifests and the code behind them, because a graceful shutdown lives in both places. In a repository like GoogleCloudPlatform/microservices-demo, `kubernetes-manifests/` holds one file per service and eight of them set `terminationGracePeriodSeconds: 5`, while `frontend.yaml`, `checkoutservice.yaml` and `shippingservice.yaml` set none and so get the 30-second default. No manifest has a `lifecycle.preStop` hook. On the code side the picture is uneven: `src/frontend/main.go` ends in `log.Fatal(http.ListenAndServe(addr+":"+srvPort, handler))` with no signal handling, `src/checkoutservice/main.go` and `src/shippingservice/main.go` call `srv.Serve(lis)` on a `grpc.NewServer()` and never `GracefulStop`, `src/productcatalogservice/server.go` catches only `SIGUSR1` and `SIGUSR2` to toggle catalog reloading, the two Node services (`currencyservice/server.js`, `paymentservice/server.js`) call `server.start()` after `bindAsync` with no `process.on('SIGTERM')`, and the two Python services sleep in a `while True` loop that only `KeyboardInterrupt` breaks. Only `adservice` registers a JVM shutdown hook that clears its health status and calls `server.shutdown()`, and `cartservice` relies on the .NET generic host, whose default shutdown timeout is longer than the 5 seconds its manifest allows.

For each service the agent makes the same three changes in that service's own language. In the Go services it replaces the fatal listen with an `http.Server` or the existing `grpc.Server`, listens for `SIGTERM` and `SIGINT` with `signal.NotifyContext`, and on a signal first flips the gRPC health server to `NOT_SERVING` (the health servers are already registered in checkout and shipping) or the frontend's `/_healthz` to a failing answer, then calls `Shutdown(ctx)` or `GracefulStop()` with a deadline. In the Node services it adds a `SIGTERM` handler that calls `server.tryShutdown` and exits when the callback fires, with a fallback `forceShutdown` after a timeout. In the Python services it replaces the sleep loop with `server.wait_for_termination()` and a `signal.signal(SIGTERM, ...)` handler that calls `server.stop(grace)`. The Java service keeps its hook. Every `ENTRYPOINT` in the Dockerfiles is already in exec form, so the process is PID 1 and receives the signal; the agent checks each one and says so.

In the manifests it adds a `lifecycle.preStop` with a short `sleep` to every Deployment that serves traffic, so the endpoint has time to leave the Service before the container gets `SIGTERM`, and sets `terminationGracePeriodSeconds` to the sleep plus the in-process drain plus a margin, raising the eight services that sit at 5 seconds and setting an explicit value on the three that have none. The same edits go into `kustomize/base/` and `helm-chart/templates/`, which mirror the manifests file for file, and the agent runs `kustomize build kustomize/` and `helm template helm-chart/` in the sandbox to show the three sources agree. `loadgenerator` is a client, not a server, and keeps its manifest.

To check the code it builds each service it changed (`go build ./...` in the Go directories, `npm ci` in the Node ones, `python -m py_compile` for the Python servers) and runs the shutdown path where it can without a cluster: start the binary in the sandbox, open a request against it, send `SIGTERM`, and record whether the in-flight request completes and the process exits within the grace period. For the gRPC services it uses `grpcurl` against the health service to show the `NOT_SERVING` flip before the listener closes. The output of those runs goes in the pull request, service by service.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add graceful shutdown to every service in this repository that serves traffic on Kubernetes. For each service, read its Deployment manifest and its entry point, then: handle SIGTERM in the process so it stops accepting new work, marks itself unhealthy, drains in-flight requests and exits; add a lifecycle.preStop sleep to the Deployment; and set terminationGracePeriodSeconds to cover the sleep and the drain. Apply the manifest changes to every copy of the manifests (kubernetes-manifests, kustomize, helm-chart) and prove they render identically. Build each service you changed, exercise the shutdown path in the sandbox where the service can run standalone, and open a pull request that reports, per service, what it did on SIGTERM before and after.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request on its own branch that changes the entry point of each service that lacked a handler (`src/frontend/main.go`, `src/checkoutservice/main.go`, `src/shippingservice/main.go`, `src/productcatalogservice/server.go`, `src/currencyservice/server.js`, `src/paymentservice/server.js`, `src/emailservice/email_server.py`, `src/recommendationservice/recommendation_server.py`) and the matching Deployment in `kubernetes-manifests/`, `kustomize/base/` and `helm-chart/templates/`. The description has a table with one row per service: the language, what happened on `SIGTERM` before, what happens now, the `preStop` sleep and the grace period chosen, and the build and local shutdown output from the sandbox.

The run's record keeps every command and its output, including the `kustomize build` and `helm template` diffs that show the three manifest sets still agree and the standalone shutdown runs, so a reviewer can check a claim about any one service without re-running it.

Before merging, a reviewer sets the grace periods to match the slowest real request in each service, decides whether the `cartservice` host options should shorten the .NET shutdown timeout to fit its manifest, and plans a rollout where the deploy is watched for `5xx` during pod replacement, which is the signal that started this work.

## Run it on every 5xx spike during a rollout

When an alert on request errors fires in the minutes around a deploy, a handler on your server receives it from your alerting system and sends one `POST /o/{org}/sessions` with this body and the repository in `resources`, with a `request_id` that is a UUID v5 of the alert id so a repeated notification returns the run already started. The run reads the service the alert names first and reports whether its shutdown path explains the errors before it changes anything else. Grafana contact points can call a `webhook` trigger directly with its `Authorization: Bearer` header; most other alerting tools cannot, so the handler is the usual path.

Set a webhook URL on the API key and Rush posts `run.completed` or `run.failed` when a run's turn finishes, with `session_url` so the on-call channel gets the pull request link and the per-service table.

## Why run it on Rush?

Shutdown bugs are cross-cutting: eight services in four languages, three copies of every manifest, and a failure that only shows as a short burst of errors during a rollout. Doing it by hand means eight small changes that each need a different runtime to test. The agent has every runtime in its sandbox, makes the change in each service's own idiom, and runs the signal test for each one so the pull request says what each process does on `SIGTERM` rather than what it should do.

The run keeps every build and every shutdown trace with its output, so the review is about the grace period numbers and the rollout plan, not about whether the handler was wired. Several repositories or clusters can be done at once in separate sandboxes, each with its own pull request.

## Frequently asked questions

### Why add a preStop sleep if the process already handles SIGTERM?

Because endpoint removal and container termination run in parallel on Kubernetes. Without the sleep the process can receive `SIGTERM` while kube-proxy and the Service still route new connections to it, and those are the requests that fail. The sleep gives the endpoint change time to propagate; the in-process handler then drains what is already open.

### How does the agent pick the grace period?

It adds the `preStop` sleep, the drain deadline it put in the handler and a margin, and writes that sum into `terminationGracePeriodSeconds` for each Deployment. The pull request shows the arithmetic per service so a reviewer can replace the drain deadline with the real slowest request and get a new total.

### Will the health checks still work during shutdown?

Yes, and they are part of the change. On the signal, the gRPC services set their health service to `NOT_SERVING` and the frontend's `/_healthz` starts failing, so the readiness probe already in each manifest takes the pod out of rotation while the drain completes.

### What about the service written in C#?

The .NET generic host handles `SIGTERM` itself and waits for hosted services to stop, with a default timeout longer than the 5 seconds the `cartservice` manifest allows. The agent raises the manifest's grace period to cover the host's timeout rather than editing `Program.cs`, and the pull request notes `HostOptions.ShutdownTimeout` as the place to shorten it instead.

## More Kubernetes jobs

- [Add Horizontal Pod Autoscalers to Kubernetes Manifests from a Linear Issue](https://getrush.ai/run/add-horizontal-pod-autoscaler/kubernetes/)
- [Add an Ingress to a Kubernetes App](https://getrush.ai/run/add-ingress/kubernetes/)
- [Split Kubernetes Manifests into Kustomize Overlays for Dev and Prod](https://getrush.ai/run/add-kustomize-overlays/kubernetes/)

## The same job on other stacks

- [Add Graceful Shutdown to a Leptos Axum Server](https://getrush.ai/run/add-graceful-shutdown/leptos/)
- [Add Graceful Shutdown to a Litestar App](https://getrush.ai/run/add-graceful-shutdown/litestar/)
- [Add Graceful Shutdown to an Express and Mongoose API](https://getrush.ai/run/add-graceful-shutdown/mongoose/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
