# Investigate a NestJS API Incident With Datadog Yes. An agent reads the error spike in Datadog, traces it to recent commits in your NestJS auth and users modules, and writes a sourced incident.md. [Run this on Rush](https://cloud.getrush.ai/run/investigate-incidents-with-datadog/nestjs/), in a Project that has your repository. ## What the agent does The agent opens Datadog first. Through the Datadog connection it reads the window you name: error logs for the API service grouped by route and status, request counts and latency for the same minutes, and the monitors that fired. NestJS routes in a project like brocoders/nestjs-boilerplate are prefixed by `API_PREFIX=api` and URI versioning (`app.enableVersioning({ type: VersioningType.URI })` in `src/main.ts`), so Datadog shows paths like `/api/v1/auth/email/login` and `/api/v1/auth/refresh`, and the agent maps each one back to a decorator in `src/auth/auth.controller.ts`. Then it narrows the code. `AuthController` has `@Post('email/login')`, `email/register`, `email/confirm`, `forgot/password`, `reset/password`, `refresh` and `logout`, plus `@Get('me')`, `@Patch('me')` and `@Delete('me')`. Logins go through `AuthService.validateLogin`, which hashes a random value and stores it through `SessionService.create`; the users module keeps its data behind `src/users/infrastructure/persistence/` with `relational` (TypeORM 0.3.28, Postgres per `DATABASE_TYPE=postgres`) and `document` variants. A burst of 422s with `emailNotVerified` points at a different branch of `auth.service.ts` than a burst of 500s from the session repository, and the agent follows what the logs show. It lines the window up with what changed. It runs `git log --since='7 days ago' -- src/auth src/users src/session` and reads each diff that touches the failing endpoint, its DTO or validation pipe, or the token settings in `src/auth/config/auth.config.ts`. A change to `AUTH_JWT_TOKEN_EXPIRES_IN` (15m in `env-example-relational`) or to how `refresh` reads the session hash can explain a wave of 401s exactly 15 minutes after a deploy. The agent names a commit only when its timing and the symptom agree, and says when the evidence is thin. It leaves the code as it is. To test a hypothesis it can install from `package-lock.json` on the Node version in `.nvmrc` (24.14.1), run the unit tests with `npm test`, or bring up Postgres in its sandbox and run the e2e spec `test/user/auth.e2e-spec.ts` with `npm run test:e2e`. Whatever it runs, it puts the command and output in the report. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "claude", "input": "Error rates for the NestJS API service rose between 09:12 and 09:50 UTC today. Use the Datadog connection to read that window: error logs grouped by route and status code, request rate and latency for the /api/v1/auth and /api/v1/users routes, and any monitors that triggered. Compare it with the commits to src/auth, src/users and src/session in the last 7 days and read the diff of each one that touches a failing route, its DTO, the session repository or src/auth/config. You may run `npm test` or the auth e2e spec to check a hypothesis. Write incident.md with a summary, a UTC timeline, the most likely cause with commit and file, the evidence for each claim (query or command and its result), the causes you ruled out, and every query you ran. Change no code and change nothing in Datadog.", "connections": [ "datadog" ], "resources": [ { "url": "https://github.com/your-org/your-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back What comes back is `incident.md`, listed on the session's artifacts with a download link that works for an hour and refreshes each time you read the session. Its timeline puts the deploy, the first failing request, each monitor trigger and the recovery in UTC order, and its cause section names a commit and a file under `src/auth/` or `src/users/`, with the route and status code that commit explains. Each statement in the report cites its evidence: the Datadog log or metric query with its time range and the count it returned, or the `git show`, `npm test` or e2e command and its output. The ruled-out section is just as concrete, for instance a Postgres connection error that also appears in the hour before the incident and so cannot be its cause. The session's changed-files list is empty for the repository, which confirms the run changed no code. The on-call engineer reads the cause, reruns the query that supports it in Datadog, and decides whether to roll back the deploy or ship a fix, perhaps as a second run that takes `incident.md` as its input. ## Run it on every error-rate monitor firing Add an automation to the Project with this prompt and a `webhook` trigger. Point a Datadog webhook at the trigger's URL with its secret as a bearer token, and reference that webhook from the error-rate monitor on the API service. Each firing starts one run with the monitor's payload after the prompt, so the window and service come from the alert instead of being typed by hand. Because the same alert can fire, recover and fire again, cap the automation's max runs per hour and send an `Idempotency-Key` with each delivery so a retried webhook is deduplicated. When the report is ready, the key's webhook gets `run.completed` with the `session_url`, and your incident bot can attach the report to the incident. ## Why run it on Rush? A NestJS incident usually crosses the controller, the service, a repository and a config file, and the first half hour goes to finding which. An agent in its own Rush sandbox does that reading against Datadog and the repository while people work on mitigation. It brings up Postgres and runs the e2e spec in the sandbox if it needs to, so nobody's laptop or staging database is touched. The report holds up because it cites everything: the Datadog query behind each number and the command behind each code claim, all kept on the session. It is read-only by instruction and by record, and parallel incidents each get their own run and their own sandbox. ## Frequently asked questions ### Does the agent need our production secrets to investigate? No. It reads production behavior through Datadog, and for local checks it uses the placeholders in `env-example-relational`, which this repository's own config warns are not for deployment. It never needs `AUTH_JWT_SECRET` or a production database URL. ### We use the document (MongoDB) persistence, not relational. Does that change anything? Only which files it reads. The users and session modules keep both variants under `infrastructure/persistence/`, and the agent follows the one your errors name. For a local check it can bring up MongoDB with the `docker-compose.document.test.yaml` setup instead of Postgres. ### Can I get the report somewhere other than a file? The file is the artifact. The agent's final answer, which the session returns as `result` and the `run.completed` webhook carries when it fits in 16 KiB, holds the summary and the likely cause. Name your Slack sign-in in `connections` and ask in the prompt, and the agent can post that summary to the incident channel as well. ### What if the cause is not in our code at all? Then the report says so. If the errors line up with a Datadog-visible dependency, such as database latency rising before any request failed, it names that and lists the commits it checked and ruled out, rather than blaming the most recent one.