What the agent does
In a set of Spring Boot services, the service that shows errors is often not the one that broke, so the agent reads Datadog across all of them before it picks one. Through the Datadog connection it pulls error logs, request rates and latency per service for the window you give, plus triggered monitors. In spring-petclinic/spring-petclinic-microservices that means the api-gateway, customers-service, visits-service and vets-service, with config-server, discovery-server and admin-server behind them, all on Spring Boot 4.0.1, Java 17 and Spring Cloud 2025.1.0.
It reads how the services call each other, because that decides where errors surface. ApiGatewayController.getOwnerDetails fetches the owner from customers-service, then calls VisitsServiceClient.getVisitsForPets, which requests http://visits-service/pets/visits?petId=..., wrapped in a ReactiveCircuitBreaker named getOwnerDetails whose fallback returns empty visits. So when visits-service fails, the gateway may log no error at all and owners simply show no visits. The agent checks the visits-service error rate and the circuit breaker's state, not only gateway 5xx counts.
It uses the metrics the code already names. OwnerResource is annotated @Timed("petclinic.owner"), PetResource @Timed("petclinic.pet") and VisitResource @Timed("petclinic.visit"), and the services pull in micrometer-registry-prometheus, spring-boot-starter-zipkin and datasource-micrometer-spring-boot. Whether those reach Datadog through the Datadog Agent scraping Prometheus or through a Micrometer Datadog registry, the agent queries the timers by name and splits them by endpoint, such as POST owners/*/pets/{petId}/visits against GET pets/visits.
Then it lines that up with the code. It runs git log --since='7 days ago' -- spring-petclinic-customers-service spring-petclinic-visits-service and reads each diff that touches a resource class, a repository or application.yml. It may run a module's tests to check a theory, for example ./mvnw -pl spring-petclinic-visits-service test for VisitResourceTest, and records the output. It changes no code.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Owners stopped showing pet visits and error rates rose between 16:20 and 17:05 UTC today. Use the Datadog connection to read that window for the api-gateway, customers-service and visits-service: error logs by service and endpoint, the petclinic.owner, petclinic.pet and petclinic.visit timers, request rates, and any monitors that triggered. Remember the gateway's getOwnerDetails circuit breaker returns empty visits on failure, so check visits-service directly. Compare the window with the commits to spring-petclinic-customers-service and spring-petclinic-visits-service in the last 7 days. You may run `./mvnw -pl <module> test`. Write incident.md with a summary, a UTC timeline, the most likely cause with commit, service and file, the evidence for each claim (query or command and result), causes ruled out, and every query you ran. Change no code and nothing in Datadog.",
"connections": [
"datadog"
],
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The run returns incident.md as a session artifact. Its summary names the service where the fault started, which may differ from the one users noticed, and its timeline in UTC covers deploys per service, the first failing request, circuit breaker activity at the gateway, monitor triggers and recovery.
The cause section points at one commit in one module, such as a change in VisitResource that broke the pets/visits query the gateway relies on, and shows the evidence: the petclinic.visit timer split by endpoint, the visits-service error logs with their query and time range, and the diff. Every claim has its Datadog query or command next to it, and ruled-out causes, such as a config-server outage or a discovery problem, list the query that excluded them.
The repository is untouched, which the session's changed-files list shows. The engineer who reads the report reruns the key query, checks the deploy time of the named commit, and chooses between rolling back that one service and shipping a fix.
Run it on every monitor alert on any service
Make an automation in the Project with this prompt and a webhook trigger, and add its URL to Datadog as a webhook that sends the trigger's secret as a bearer token. Mention the webhook in each service's error-rate monitor. The monitor's payload follows the prompt as data, so the run knows which service alerted and when, and reads the others itself.
When one fault trips monitors on several services at once, several alerts arrive within a minute. Cap the automation's max runs per hour at a low number so the first alert starts the investigation and the rest are skipped with rate_limited, which the fire outcome reports.
The use case this job follows, with its automation and trigger: Investigate an incident.
Why run it on Rush?
Tracing a fault across Spring Cloud services by hand means opening a Datadog view per service, remembering which client calls which, and reading the circuit breaker logic to explain why errors vanished at the gateway. The agent does that reading in a sandbox of its own, with its own clone and JDK, and can run ./mvnw there without filling your local Maven repository or competing with your IDE.
Everything the report says is backed by something you can rerun: a Datadog query with its range or a command with its output, all kept in the session. It is read-only by instruction, with a changed-files list to prove it, and a second incident at the same time gets a run of its own.
Frequently asked questions
Our services export metrics to Prometheus, not Datadog. Will the agent find them?
Only if Datadog receives them. The Datadog Agent can scrape the Prometheus endpoints these services expose, and then the agent queries the timers by name in Datadog. If your metrics live only in Prometheus and Grafana, connect Grafana instead and point the prompt at it.
Does it start all the services in its sandbox?
Not by default. Running a module's tests with ./mvnw -pl <module> test is usually enough to check a theory. Starting the full set with docker-compose.yml is possible in the sandbox, and the agent does it only if the prompt asks.
How does it tell a gateway problem from a downstream one?
It reads the code path first. Here the gateway's getOwnerDetails circuit breaker hides visits-service failures behind empty visits, so the agent compares each service's own error rate and latency rather than trusting the gateway's status codes.
Can it read Zipkin traces?
It reads traces that reach Datadog. These services send spans through spring-boot-starter-zipkin; if Datadog ingests them, the agent uses the spans to follow one request from the gateway to the failing service and cites the trace in the report.