All jobs

Add Request ID Logging to an Express and Mongoose API

Yes. An agent gives every request to your Express and Mongoose API an id, puts it in each log line, the response header and error bodies, tests it, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the app logs today before adding an id to it, because the id only helps if every line that belongs to a request carries it. In an API built from hagopj13/node-express-boilerplate, src/app.js registers two morgan handlers from src/config/morgan.js first of all, outside the test environment: a success handler for responses under 400 and an error handler for the rest, both writing through the winston logger in src/config/logger.js, whose printf format prints only level: message. Morgan's formats print the method, URL, status and response time, with the client address in production and res.locals.errorMessage on errors, which src/middlewares/error.js sets before sending {code, message} and a stack in development. Mongoose connects in src/index.js from config.mongoose.url, which src/config/config.js validates with Joi and suffixes with -test under NODE_ENV=test. package.json pins engines.node at >=12.0.0, installs with yarn.lock, runs jest serially with jest -i, and .travis.yml runs yarn lint and yarn test on Node 12 with a MongoDB service.

The id has to exist before the first log line and be readable from code that never sees req, such as a service that logs inside a Mongoose query, so the agent adds src/middlewares/requestId.js built on AsyncLocalStorage from async_hooks, which the declared Node floor supports. The middleware takes an incoming X-Request-Id when the header is present and matches a strict pattern of letters, digits, dots, underscores and hyphens up to a fixed length, so a proxy's id is kept but a hostile value is never written into logs, and otherwise makes a v4 id with the uuid package, added to dependencies rather than relying on crypto.randomUUID, which the engines floor does not promise. It stores the id in req.id, sets the X-Request-Id response header at once so even a failed response carries it, and runs the rest of the chain inside als.run. In src/app.js it is registered before the morgan handlers, which otherwise would log the first line without an id.

Three places then read the id. src/config/logger.js gains a winston format that pulls the store from AsyncLocalStorage and prefixes the message with the id in brackets when one exists, so logger.info and logger.error calls anywhere in src/services or src/models inherit it with no signature change. src/config/morgan.js registers a morgan.token('id', (req) => req.id) and puts :id at the front of both format strings. src/middlewares/error.js adds requestId to the JSON body it sends, so a client can quote the id from a 500 and support can find the lines; the unit tests in tests/unit/middlewares/error.test.js match with objectContaining, so they keep passing. src/docs/components.yml gets the new header and field described for Swagger.

Tests prove the three behaviors. tests/unit/middlewares/requestId.test.js uses the repository's node-mocks-http to check a generated id is a UUID, a valid incoming header is kept, an invalid one is replaced, and the response header matches req.id. An integration test using supertest against src/app.js requests /v1/users without a token and asserts the 401 body carries the same requestId as the X-Request-Id header. The integration suite connects to MongoDB through tests/utils/setupTestDB.js, and the sandbox has no MongoDB and no Docker, so the agent installs docker.io, starts dockerd, brings up the mongodb service from docker-compose.yml with docker-compose, points MONGODB_URL at it with the other variables from .env.example, and runs yarn test, yarn lint and yarn prettier, then starts the app once and shows two requests with different ids in the log and a third with a supplied header echoed back, before opening the pull request.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add request id logging to this Express and Mongoose API. Read package.json, src/app.js, src/index.js, src/config/logger.js, src/config/morgan.js, src/config/config.js, src/middlewares/error.js, tests/unit/middlewares/error.test.js, tests/utils/setupTestDB.js, .env.example, docker-compose.yml and .travis.yml first. Add src/middlewares/requestId.js that keeps an incoming X-Request-Id when it matches a strict pattern of safe characters and a maximum length, otherwise makes a v4 id with the uuid package (add it to dependencies; do not depend on crypto.randomUUID because engines.node is >=12), stores it on req.id, sets the X-Request-Id response header immediately, and runs the rest of the chain inside an AsyncLocalStorage store. Register it in src/app.js before the morgan handlers. Make the winston format in src/config/logger.js prefix every message with the id from the store when there is one, add a morgan token id used at the front of both formats in src/config/morgan.js, and add requestId to the error body in src/middlewares/error.js. Document the header and field in src/docs/components.yml. Add tests/unit/middlewares/requestId.test.js with node-mocks-http and an integration test with supertest that checks a 401 from /v1/users carries the same id in the header and body. MongoDB is not running here and Docker is not installed: install docker.io, start dockerd, bring up the mongodb service from docker-compose.yml with docker-compose, set MONGODB_URL and the other variables from .env.example, and run yarn test, yarn lint and yarn prettier. Start the app once and show log lines for two requests with different ids and one with a supplied header. Open a pull request that pastes those lines.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that adds src/middlewares/requestId.js and tests/unit/middlewares/requestId.test.js, adds a test to the integration suite, and changes src/app.js, src/config/logger.js, src/config/morgan.js, src/middlewares/error.js, src/docs/components.yml, package.json and yarn.lock for the uuid dependency. The description pastes a few lines of the running app's log, each starting with its id in brackets, one of them a value supplied in the request header, and the matching X-Request-Id response headers.

The run's record holds the Docker install, the MongoDB container start, the full yarn test output with the existing integration suites and the new tests, and the lint and prettier runs. The Mongoose models, services and routes are unchanged; the id reaches their log calls through the store, not through new parameters.

Before merging, a reviewer decides whether the proxy or load balancer in front of the API already sets X-Request-Id, in which case the keep-if-valid branch is the one that runs in production, whether the error body should carry the id in every environment or only outside production, and whether the pm2 setup in ecosystem.config.json should print the id in its own timestamped lines, which it does because the format is applied inside winston.

Run it on every new Express service

A team with several services built from the same boilerplate starts one run per repository from its own server: one POST /o/{org}/sessions each, with that repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started instead of a second pull request. Each run reads that service's own logger and morgan setup, so a service that already prints JSON logs gets the id as a field rather than a prefix.

Afterwards, an automation on the Project with a github trigger on pull_request.opened and pull_request.synchronize and the prompt "if this pull request adds a middleware before the request id middleware in src/app.js, or logs with console instead of the winston logger, comment with the line and why the id would be missing" keeps later changes from logging lines nobody can trace. The trigger adds the pull request after the prompt; only work by an owner, member or collaborator fires it, never a fork's.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

Request ids are the kind of change that is easy to write and hard to prove: the middleware looks fine, and the first production incident shows that a service's log calls had no id because they ran in a Mongoose callback outside the request. The agent runs the app in its own sandbox with a real MongoDB behind it, sends requests, and pastes the lines, so the reviewer sees the id on the morgan line, on a service log line and on the error body before merging.

The sandbox keeps the container start, the test output and the log excerpt with the run, and the pull request cites them. For a team with many services on this boilerplate, each gets its own run and its own pull request, with the same id format across all of them so a trace can be followed from one service into the next.

Frequently asked questions

Why AsyncLocalStorage instead of passing the id around?

Because the services in src/services and the Mongoose plugins log without access to req, and threading an id through every signature would touch most of the codebase. AsyncLocalStorage follows the request through awaited Mongoose calls and catchAsync wrappers, and a winston format reads the store at write time. The Node floor in package.json supports it.

What happens when a client sends its own X-Request-Id?

It is kept when it matches a strict pattern and length, so an id from a gateway or a retrying client is traceable end to end. Anything else is replaced with a fresh UUID, so a log line can never contain a value chosen to break log parsing. The unit test covers both branches.

Does the morgan line change for existing dashboards?

Yes, by one field at the front: :id precedes :method :url :status. The rest of both formats, including the client address in production and the error message on failures, is as it was. The pull request shows a before and after line so a parsing rule can be adjusted.

Why add uuid rather than use crypto.randomUUID?

engines.node is >=12.0.0 and .travis.yml tests on Node 12, where crypto.randomUUID does not exist. The uuid package works on every version the repository claims to support; the pull request notes that a reviewer who raises the floor can drop the dependency.

More Mongoose jobs

The same job on other stacks

Keep reading