All jobs

Add Graceful Shutdown and Zero-Downtime Reloads with PM2

Yes. An agent makes your Express app finish in-flight requests and close MongoDB on SIGTERM, signal readiness to PM2, set reload timeouts, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads what happens when the process is told to stop. In a Node service laid out like hagopj13/node-express-boilerplate, src/index.js connects Mongoose, then listens, and on SIGTERM logs a line and calls server.close() with no callback: the process stops accepting connections but never disconnects Mongoose and never exits, so PM2 waits out its kill timeout and sends SIGKILL. exitHandler, used for uncaughtException and unhandledRejection, closes the server and then exits with code 1 even after a clean close. ecosystem.config.json runs one instance of src/index.js with autorestart and time set and nothing about readiness or timeouts, and package.json starts it with pm2 start ecosystem.config.json --no-daemon, which docker-compose.prod.yml runs as yarn start on a node:alpine image. Mongoose is 5.7 and Express 4.17.

It rewrites the shutdown as one function with a deadline. On SIGTERM or SIGINT the process logs the signal, calls server.close so new connections are refused while open ones finish, waits for the close callback, then closes the Mongoose connection with mongoose.connection.close(false) so queued operations drain first, and exits 0. A timer created with unref fires if that takes longer than the app's own deadline, logs which step was still running, and exits 1, so a stuck connection cannot hold the old process open for ever. The existing exitHandler is folded in: an uncaught exception runs the same sequence and keeps exit code 1, and a clean signal no longer reports failure. A second signal during shutdown is ignored rather than starting the sequence twice.

Then the PM2 side. The agent adds process.send('ready') after listen succeeds, guarded by whether process.send exists so yarn dev under nodemon still works, and sets wait_ready: true with a listen_timeout in the ecosystem file so PM2 counts the app as up only once MongoDB is connected and the port is open. It sets kill_timeout a little above the app's own deadline, since PM2's documented default is 1600 milliseconds and the Mongoose close alone can take longer, and switches exec_mode to cluster with instances still at 1, because pm2 reload starts the new process and waits for its ready message before stopping the old one only in cluster mode; in fork mode a reload is a restart with a gap.

The proof is a test and a reload. A new tests/integration/shutdown.test.js spawns src/index.js with child_process, waits for the listening line, opens a request to a route the agent adds for the test that answers after a delay, sends SIGTERM, and asserts the response arrives and the exit code is 0; the suite needs MongoDB, so the agent installs a server in its sandbox and sets MONGODB_URL. Then it runs pm2 start ecosystem.config.json, loops curl against /v1/docs while running pm2 reload app, and counts refused connections, which should be none. It runs yarn test, yarn lint and yarn prettier and keeps all of it.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add graceful shutdown and zero-downtime reloads. Read package.json, ecosystem.config.json, src/index.js, src/app.js, src/config/config.js, docker-compose.prod.yml, Dockerfile and tests/utils/setupTestDB.js first. On SIGTERM and SIGINT stop accepting connections with server.close, wait for in-flight requests, close the Mongoose connection without forcing, and exit 0; add an unref'd deadline timer that logs the step still running and exits 1; keep exit code 1 for uncaughtException and unhandledRejection; ignore a second signal during shutdown. Send process.send('ready') after listen when process.send exists, and in ecosystem.config.json set wait_ready, listen_timeout, a kill_timeout above the app's deadline, and exec_mode cluster with instances 1. Write an integration test that spawns the process, holds a slow request open, sends SIGTERM and asserts the response completes and the exit code is 0. Run yarn test against a local MongoDB, then start under pm2 and run pm2 reload while a curl loop hits /v1/docs and count refused connections. Run yarn lint and yarn prettier, and open a pull request with the output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that changes src/index.js and ecosystem.config.json and adds one integration test. The description shows the shutdown sequence with its deadline, the three ecosystem keys with the values chosen and why they are ordered the way they are, the integration test output, and the curl loop's count of refused connections across a pm2 reload.

The run's record keeps the pm2 logs from the reload, so a reviewer can see the ready message, the new process listening and the old one exiting 0 in order. A reviewer checks that the deadline is longer than the slowest request the app serves and shorter than the orchestrator's own grace period, and that the Docker image's yarn start still runs with --no-daemon, which a cluster-mode app honours.

Run it on every service that restarts with dropped requests

A team with several Node services on this boilerplate starts one run per repository with one POST /o/{org}/sessions each, this prompt as input and the repository in resources. Each run reads its own index.js and ecosystem file, so a service that already drains connections gets only the readiness and timeout settings, and one that runs without PM2 gets the shutdown sequence and a description of what its process manager needs instead.

Send a request_id that is a UUID v5 of the repository's full name so a retried call returns the run it already started. The key's webhook_url reports when each run finishes, with session_url to read the pull request from.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

Graceful shutdown is a sequence with an order and a deadline, and it is almost never tested, because testing it means a real process, a real database and a signal sent at the right moment. Most services find out at the first deploy under load that server.close without an exit leaves PM2 to kill them. The agent writes the sequence, runs it as a spawned process with a request held open, and reloads under PM2 while counting refused connections, so the pull request carries the two results that matter.

The pull request is one entry file and one config file, so review is reading the order of the steps and three numbers. A company with many services on this boilerplate runs them all at once, each in its own sandbox with its own pull request.

Frequently asked questions

Why cluster mode with one instance?

Because pm2 reload overlaps the old and new process only in cluster mode; in fork mode it is a restart with a gap. One instance keeps memory and behaviour as they are. Raising instances later is a one-line change, and the description says what to check first, such as anything the app keeps in process memory.

What does wait_ready change?

Without it PM2 treats the app as up the moment the process starts, so a reload can stop the old process before the new one has connected to MongoDB. With wait_ready and the ready message sent after listen, the old process is kept until the new one can serve, and listen_timeout bounds how long PM2 waits for that message.

Does the test need PM2?

No. The integration test spawns src/index.js directly and sends the signal itself, so it runs under yarn test like the rest of the suite. The PM2 reload check is a separate step the agent runs in its sandbox and reports, since it depends on the ecosystem file rather than the code.

What about Kubernetes or Docker stop?

Both send SIGTERM and wait a grace period before SIGKILL, so the same sequence applies; the deadline in the app and kill_timeout in PM2 both need to sit under that period. The agent names the three values in the description so an operator can line them up with the platform's.

The same job on other stacks

Keep reading