What the agent does
The agent reads Grafana before it reads code. Through the Grafana connection it queries Loki for the API's log lines in the window you give and Prometheus for request rate, error ratio and latency, and lists alerts that fired. Runs that reach Grafana's MCP server also load Grafana's own grafana-cloud-mcp-tools skill, so the agent knows how to phrase LogQL and PromQL against your data sources instead of trying queries at random.
It knows what the logs contain because it reads the logging code. In an Express API like hagopj13/node-express-boilerplate, src/config/morgan.js writes one line per request through winston: :method :url :status - :response-time ms, and for status 400 and above it adds - message: :message from res.locals.errorMessage. src/config/logger.js prints level: message to the console with errors on stderr. In production, errorHandler in src/middlewares/error.js does not call logger.error(err), so there are no stack traces in Loki: the morgan line with its message is the evidence, and the agent searches for that.
Then it maps status codes to code paths. Routes mount under /v1: /v1/auth for register, login, logout, refresh-tokens, forgot-password, reset-password, send-verification-email and verify-email, and /v1/users for / and /:userId. In production app.js also puts authLimiter on /v1/auth, which allows 20 failed requests per 15 minutes (skipSuccessfulRequests: true), so a wave of 429s on /v1/auth/login is the limiter, not a crash. errorConverter wraps any error that is not already an ApiError, making it a 400 when it has a statusCode or is a Mongoose error and a 500 otherwise, and errorHandler masks non-operational errors as 500 in production. The agent uses these rules to read what each status class means.
Last, it lines up the window with the commits to src/routes/v1/, src/controllers/, src/services/ and src/middlewares/ from the last 7 days and reads each relevant diff. To test a theory it can install with yarn from yarn.lock, start MongoDB in its sandbox, and run yarn test (Jest with -i, using tests/integration/auth.test.js and user.test.js), which connects to MONGODB_URL with a -test suffix. It writes everything down in incident.md and changes no code.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Error rates for the Express API rose between 11:30 and 12:10 UTC today. Use the Grafana connection to read that window: Loki log lines for the API grouped by route and status, including the morgan 'message:' text on 4xx and 5xx lines; Prometheus request rate, error ratio and latency; and any alerts that fired. Remember that in production the error handler logs no stack traces and authLimiter returns 429 after 20 failed /v1/auth requests in 15 minutes. Compare the window with the commits to src/routes/v1, src/controllers, src/services and src/middlewares in the last 7 days. You may run `yarn test` against a local MongoDB. Write incident.md with a summary, a UTC timeline, the most likely cause with commit and file, evidence for each claim (query or command and result), causes ruled out, and every query you ran. Change no code and nothing in Grafana.",
"connections": [
"grafana"
],
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is incident.md on the session's artifacts. The summary says which route failed and how, for example 500s on PATCH /v1/users/:userId beginning two minutes after a deploy. The timeline lists the deploy, the first failing Loki line, each alert and the recovery, all in UTC.
Each claim carries the LogQL or PromQL that backs it, its time range and what it returned, or the git show or yarn test command and its output. The ruled-out list matters in this codebase: it states, with the query, whether 429s came from authLimiter and whether 400s came from validation in src/validations/ rather than from the change under suspicion.
The session shows no changed files in the repository and only read calls to Grafana. The on-call engineer checks the query behind the cause in Grafana Explore, confirms the commit was in the deployed build, and decides whether to roll back.
Run it on every Grafana alert firing
Add an automation to the Project with this prompt and a webhook trigger, then create a Grafana contact point of type webhook that posts to the trigger's URL with the trigger's secret as its bearer token, and route the API's error-rate alert to it. Each firing starts one run, with the alert's labels and values after the prompt as data, so the window and service come from the alert.
Grafana resends an alert while it stays firing, so set the automation's max runs per hour low; extra fires are reported as skipped with rate_limited rather than starting duplicate investigations. Give the API key a webhook URL to get run.completed with the session_url when the report is written.
The use case this job follows, with its automation and trigger: Investigate an incident.
Why run it on Rush?
The first part of an Express incident is translation: which morgan line in Loki belongs to which route, whether a 429 is the limiter, and why production shows a bare 500 with no stack. An agent that has read morgan.js, logger.js and error.js does that translation in its own Rush sandbox and checks each idea against Grafana, while the engineers on call work on mitigation.
The report cites every query and command, kept with its output on the session, and the run changes nothing, which the session's changed files and tool calls show. Running yarn test against MongoDB happens inside the sandbox, so no shared test database is touched, and separate incidents get separate runs at the same time.
Frequently asked questions
Our logs have no stack traces. Can the agent still find the cause?
Often, yes. In this setup each failed request still logs its method, URL, status, response time and error message through morgan. The agent groups those lines by route and message, finds when they started, and reads the commits to that route's controller and service.
How do we connect Grafana?
Connect Grafana Cloud's MCP server once with POST /me/connections {"type": "mcp", "name": "grafana", "url": "https://mcp.grafana.com/mcp", "auth": {"mode": "oauth"}}, open the returned authorize_url while signed in to the console, and approve. Then name grafana in connections on each run.
Can it tell rate limiting from a real failure?
Yes, because it reads src/middlewares/rateLimiter.js and app.js. 429s on /v1/auth match authLimiter's 20 failed requests per 15 minutes, which only runs in production. The report says whether the 429s were expected behavior or a symptom.
Will it touch our production MongoDB?
No. To test a theory it starts MongoDB inside its sandbox and runs the Jest integration tests there. The test config appends -test to the database name, and the prompt gives the agent no production connection string, so it has nothing to connect to.