All jobs

Add Graceful Shutdown to Services Running on Kubernetes

Yes. An agent finds which of your services drop requests on SIGTERM, adds signal handling, preStop hooks and grace periods that match, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent clones the repository into its own cloud sandbox and reads both the manifests and the code behind them, because a graceful shutdown lives in both places. In a repository like GoogleCloudPlatform/microservices-demo, kubernetes-manifests/ holds one file per service and eight of them set terminationGracePeriodSeconds: 5, while frontend.yaml, checkoutservice.yaml and shippingservice.yaml set none and so get the 30-second default. No manifest has a lifecycle.preStop hook. On the code side the picture is uneven: src/frontend/main.go ends in log.Fatal(http.ListenAndServe(addr+":"+srvPort, handler)) with no signal handling, src/checkoutservice/main.go and src/shippingservice/main.go call srv.Serve(lis) on a grpc.NewServer() and never GracefulStop, src/productcatalogservice/server.go catches only SIGUSR1 and SIGUSR2 to toggle catalog reloading, the two Node services (currencyservice/server.js, paymentservice/server.js) call server.start() after bindAsync with no process.on('SIGTERM'), and the two Python services sleep in a while True loop that only KeyboardInterrupt breaks. Only adservice registers a JVM shutdown hook that clears its health status and calls server.shutdown(), and cartservice relies on the .NET generic host, whose default shutdown timeout is longer than the 5 seconds its manifest allows.

For each service the agent makes the same three changes in that service's own language. In the Go services it replaces the fatal listen with an http.Server or the existing grpc.Server, listens for SIGTERM and SIGINT with signal.NotifyContext, and on a signal first flips the gRPC health server to NOT_SERVING (the health servers are already registered in checkout and shipping) or the frontend's /_healthz to a failing answer, then calls Shutdown(ctx) or GracefulStop() with a deadline. In the Node services it adds a SIGTERM handler that calls server.tryShutdown and exits when the callback fires, with a fallback forceShutdown after a timeout. In the Python services it replaces the sleep loop with server.wait_for_termination() and a signal.signal(SIGTERM, ...) handler that calls server.stop(grace). The Java service keeps its hook. Every ENTRYPOINT in the Dockerfiles is already in exec form, so the process is PID 1 and receives the signal; the agent checks each one and says so.

In the manifests it adds a lifecycle.preStop with a short sleep to every Deployment that serves traffic, so the endpoint has time to leave the Service before the container gets SIGTERM, and sets terminationGracePeriodSeconds to the sleep plus the in-process drain plus a margin, raising the eight services that sit at 5 seconds and setting an explicit value on the three that have none. The same edits go into kustomize/base/ and helm-chart/templates/, which mirror the manifests file for file, and the agent runs kustomize build kustomize/ and helm template helm-chart/ in the sandbox to show the three sources agree. loadgenerator is a client, not a server, and keeps its manifest.

To check the code it builds each service it changed (go build ./... in the Go directories, npm ci in the Node ones, python -m py_compile for the Python servers) and runs the shutdown path where it can without a cluster: start the binary in the sandbox, open a request against it, send SIGTERM, and record whether the in-flight request completes and the process exits within the grace period. For the gRPC services it uses grpcurl against the health service to show the NOT_SERVING flip before the listener closes. The output of those runs goes in the pull request, service by service.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add graceful shutdown to every service in this repository that serves traffic on Kubernetes. For each service, read its Deployment manifest and its entry point, then: handle SIGTERM in the process so it stops accepting new work, marks itself unhealthy, drains in-flight requests and exits; add a lifecycle.preStop sleep to the Deployment; and set terminationGracePeriodSeconds to cover the sleep and the drain. Apply the manifest changes to every copy of the manifests (kubernetes-manifests, kustomize, helm-chart) and prove they render identically. Build each service you changed, exercise the shutdown path in the sandbox where the service can run standalone, and open a pull request that reports, per service, what it did on SIGTERM before and after.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request on its own branch that changes the entry point of each service that lacked a handler (src/frontend/main.go, src/checkoutservice/main.go, src/shippingservice/main.go, src/productcatalogservice/server.go, src/currencyservice/server.js, src/paymentservice/server.js, src/emailservice/email_server.py, src/recommendationservice/recommendation_server.py) and the matching Deployment in kubernetes-manifests/, kustomize/base/ and helm-chart/templates/. The description has a table with one row per service: the language, what happened on SIGTERM before, what happens now, the preStop sleep and the grace period chosen, and the build and local shutdown output from the sandbox.

The run's record keeps every command and its output, including the kustomize build and helm template diffs that show the three manifest sets still agree and the standalone shutdown runs, so a reviewer can check a claim about any one service without re-running it.

Before merging, a reviewer sets the grace periods to match the slowest real request in each service, decides whether the cartservice host options should shorten the .NET shutdown timeout to fit its manifest, and plans a rollout where the deploy is watched for 5xx during pod replacement, which is the signal that started this work.

Run it on every 5xx spike during a rollout

When an alert on request errors fires in the minutes around a deploy, a handler on your server receives it from your alerting system and sends one POST /o/{org}/sessions with this body and the repository in resources, with a request_id that is a UUID v5 of the alert id so a repeated notification returns the run already started. The run reads the service the alert names first and reports whether its shutdown path explains the errors before it changes anything else. Grafana contact points can call a webhook trigger directly with its Authorization: Bearer header; most other alerting tools cannot, so the handler is the usual path.

Set a webhook URL on the API key and Rush posts run.completed or run.failed when a run's turn finishes, with session_url so the on-call channel gets the pull request link and the per-service table.

The use case this job follows, with its automation and trigger: Fix a bug from an alert.

Why run it on Rush?

Shutdown bugs are cross-cutting: eight services in four languages, three copies of every manifest, and a failure that only shows as a short burst of errors during a rollout. Doing it by hand means eight small changes that each need a different runtime to test. The agent has every runtime in its sandbox, makes the change in each service's own idiom, and runs the signal test for each one so the pull request says what each process does on SIGTERM rather than what it should do.

The run keeps every build and every shutdown trace with its output, so the review is about the grace period numbers and the rollout plan, not about whether the handler was wired. Several repositories or clusters can be done at once in separate sandboxes, each with its own pull request.

Frequently asked questions

Why add a preStop sleep if the process already handles SIGTERM?

Because endpoint removal and container termination run in parallel on Kubernetes. Without the sleep the process can receive SIGTERM while kube-proxy and the Service still route new connections to it, and those are the requests that fail. The sleep gives the endpoint change time to propagate; the in-process handler then drains what is already open.

How does the agent pick the grace period?

It adds the preStop sleep, the drain deadline it put in the handler and a margin, and writes that sum into terminationGracePeriodSeconds for each Deployment. The pull request shows the arithmetic per service so a reviewer can replace the drain deadline with the real slowest request and get a new total.

Will the health checks still work during shutdown?

Yes, and they are part of the change. On the signal, the gRPC services set their health service to NOT_SERVING and the frontend's /_healthz starts failing, so the readiness probe already in each manifest takes the pod out of rotation while the drain completes.

What about the service written in C#?

The .NET generic host handles SIGTERM itself and waits for hosted services to stop, with a default timeout longer than the 5 seconds the cartservice manifest allows. The agent raises the manifest's grace period to cover the host's timeout rather than editing Program.cs, and the pull request notes HostOptions.ShutdownTimeout as the place to shorten it instead.

More Kubernetes jobs

The same job on other stacks

Keep reading