# Investigate an Express API Incident With Grafana Yes. An agent queries Loki and Prometheus in Grafana for the incident window, ties it to recent Express route commits, and writes incident.md. [Run this on Rush](https://cloud.getrush.ai/run/investigate-incidents-with-grafana/express/), in a Project that has your repository. ## What the agent does The agent reads Grafana before it reads code. Through the Grafana connection it queries Loki for the API's log lines in the window you give and Prometheus for request rate, error ratio and latency, and lists alerts that fired. Runs that reach Grafana's MCP server also load Grafana's own `grafana-cloud-mcp-tools` skill, so the agent knows how to phrase LogQL and PromQL against your data sources instead of trying queries at random. It knows what the logs contain because it reads the logging code. In an Express API like hagopj13/node-express-boilerplate, `src/config/morgan.js` writes one line per request through winston: `:method :url :status - :response-time ms`, and for status 400 and above it adds `- message: :message` from `res.locals.errorMessage`. `src/config/logger.js` prints `level: message` to the console with errors on stderr. In production, `errorHandler` in `src/middlewares/error.js` does not call `logger.error(err)`, so there are no stack traces in Loki: the morgan line with its message is the evidence, and the agent searches for that. Then it maps status codes to code paths. Routes mount under `/v1`: `/v1/auth` for `register`, `login`, `logout`, `refresh-tokens`, `forgot-password`, `reset-password`, `send-verification-email` and `verify-email`, and `/v1/users` for `/` and `/:userId`. In production `app.js` also puts `authLimiter` on `/v1/auth`, which allows 20 failed requests per 15 minutes (`skipSuccessfulRequests: true`), so a wave of 429s on `/v1/auth/login` is the limiter, not a crash. `errorConverter` wraps any error that is not already an `ApiError`, making it a 400 when it has a `statusCode` or is a Mongoose error and a 500 otherwise, and `errorHandler` masks non-operational errors as 500 in production. The agent uses these rules to read what each status class means. Last, it lines up the window with the commits to `src/routes/v1/`, `src/controllers/`, `src/services/` and `src/middlewares/` from the last 7 days and reads each relevant diff. To test a theory it can install with `yarn` from `yarn.lock`, start MongoDB in its sandbox, and run `yarn test` (Jest with `-i`, using `tests/integration/auth.test.js` and `user.test.js`), which connects to `MONGODB_URL` with a `-test` suffix. It writes everything down in `incident.md` and changes no code. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "claude", "input": "Error rates for the Express API rose between 11:30 and 12:10 UTC today. Use the Grafana connection to read that window: Loki log lines for the API grouped by route and status, including the morgan 'message:' text on 4xx and 5xx lines; Prometheus request rate, error ratio and latency; and any alerts that fired. Remember that in production the error handler logs no stack traces and authLimiter returns 429 after 20 failed /v1/auth requests in 15 minutes. Compare the window with the commits to src/routes/v1, src/controllers, src/services and src/middlewares in the last 7 days. You may run `yarn test` against a local MongoDB. Write incident.md with a summary, a UTC timeline, the most likely cause with commit and file, evidence for each claim (query or command and result), causes ruled out, and every query you ran. Change no code and nothing in Grafana.", "connections": [ "grafana" ], "resources": [ { "url": "https://github.com/your-org/your-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back What comes back is `incident.md` on the session's artifacts. The summary says which route failed and how, for example 500s on `PATCH /v1/users/:userId` beginning two minutes after a deploy. The timeline lists the deploy, the first failing Loki line, each alert and the recovery, all in UTC. Each claim carries the LogQL or PromQL that backs it, its time range and what it returned, or the `git show` or `yarn test` command and its output. The ruled-out list matters in this codebase: it states, with the query, whether 429s came from `authLimiter` and whether 400s came from validation in `src/validations/` rather than from the change under suspicion. The session shows no changed files in the repository and only read calls to Grafana. The on-call engineer checks the query behind the cause in Grafana Explore, confirms the commit was in the deployed build, and decides whether to roll back. ## Run it on every Grafana alert firing Add an automation to the Project with this prompt and a `webhook` trigger, then create a Grafana contact point of type webhook that posts to the trigger's URL with the trigger's secret as its bearer token, and route the API's error-rate alert to it. Each firing starts one run, with the alert's labels and values after the prompt as data, so the window and service come from the alert. Grafana resends an alert while it stays firing, so set the automation's max runs per hour low; extra fires are reported as `skipped` with `rate_limited` rather than starting duplicate investigations. Give the API key a webhook URL to get `run.completed` with the `session_url` when the report is written. ## Why run it on Rush? The first part of an Express incident is translation: which morgan line in Loki belongs to which route, whether a 429 is the limiter, and why production shows a bare 500 with no stack. An agent that has read `morgan.js`, `logger.js` and `error.js` does that translation in its own Rush sandbox and checks each idea against Grafana, while the engineers on call work on mitigation. The report cites every query and command, kept with its output on the session, and the run changes nothing, which the session's changed files and tool calls show. Running `yarn test` against MongoDB happens inside the sandbox, so no shared test database is touched, and separate incidents get separate runs at the same time. ## Frequently asked questions ### Our logs have no stack traces. Can the agent still find the cause? Often, yes. In this setup each failed request still logs its method, URL, status, response time and error message through morgan. The agent groups those lines by route and message, finds when they started, and reads the commits to that route's controller and service. ### How do we connect Grafana? Connect Grafana Cloud's MCP server once with `POST /me/connections {"type": "mcp", "name": "grafana", "url": "https://mcp.grafana.com/mcp", "auth": {"mode": "oauth"}}`, open the returned `authorize_url` while signed in to the console, and approve. Then name `grafana` in `connections` on each run. ### Can it tell rate limiting from a real failure? Yes, because it reads `src/middlewares/rateLimiter.js` and `app.js`. 429s on `/v1/auth` match `authLimiter`'s 20 failed requests per 15 minutes, which only runs in production. The report says whether the 429s were expected behavior or a symptom. ### Will it touch our production MongoDB? No. To test a theory it starts MongoDB inside its sandbox and runs the Jest integration tests there. The test config appends `-test` to the database name, and the prompt gives the agent no production connection string, so it has nothing to connect to.