# Investigate Thread Pool Starvation in an ASP.NET Core App Yes. An agent reads your ASP.NET Core code, configuration and recent commits and returns an incident report ranking each suspect with the check that confirms it. [Run this on Rush](https://cloud.getrush.ai/run/investigate-thread-pool-starvation/aspnet-core/), in a Project that has your repository. ## What the agent does The agent starts from the symptom and reads what the app actually declares. In an ASP.NET Core app like dotnet/eShop it opens global.json, which pins SDK 10.0.302 with rollForward latestFeature, then the solution file eShop.slnx and Directory.Packages.props, where Npgsql.EntityFrameworkCore.PostgreSQL 10.0.3, MediatR 13.1.0 and Asp.Versioning.Http 10.2.3 are pinned. That version context matters: the default command timeout, the Kestrel defaults and the thread-pool tuning knobs differ across .NET releases, so the report states which runtime the app builds for before it names a cause. Next it hunts for blocking work with the searches an on-call engineer would run, keeping every hit in context. It greps the repository for sync-over-async patterns such as .Result, .Wait() and GetAwaiter().GetResult(); in eShop the only .Result reads are in src/Ordering.API/Extensions/LinqSelectExtensions.cs on a custom CaughtException wrapper, so the report can clear them instead of listing a false suspect. It then reads the request path shown in the alert: src/Ordering.API/Apis/OrdersApi.cs maps the order endpoints, and src/Ordering.API/Application/Queries/OrderQueries.cs shows GetOrderAsync and GetOrdersFromUserAsync running without a CancellationToken, so an abandoned HTTP request still holds its async flow and database call to completion, which is the classic amplifier when latency climbs while CPU stays below 30%. Configuration and recent changes come third. appsettings.json and appsettings.Development.json carry the connection strings but no Kestrel limits, no thread-pool settings and no command-timeout overrides; src/eShop.ServiceDefaults/Extensions.cs adds AddStandardResilienceHandler() to every HttpClient, and src/EventBusRabbitMQ/RabbitMQEventBus.cs retries publishes through a ResiliencePipeline, both of which can multiply queued work once latency rises. The agent runs git log --oneline -30 and git show on the suspects to correlate the climb with a recent change, such as a commit that dropped a cancellation token from a hot path. If the run names a datadog connection (an admin signs in once with POST /me/connections and the run lists it in connections), the agent also reads the .NET runtime thread-pool counters and slow traces so the report can say whether queued work sits in HTTP waits, database calls or event-bus publishes. One place it checks is src/Ordering.API/Application/Behaviors/TransactionBehavior.cs, which wraps every MediatR command in an execution-strategy transaction before publishing integration events, so a slow database holds a thread across the whole command. Each suspect it writes up carries the check that would confirm or clear it, such as dotnet-counters monitor against the process with the System.Runtime counters for ThreadPool Queue Length and Thread Count, or the Datadog query for the matching spans. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "claude", "input": "Incident: Datadog shows .NET thread pool queue length climbing and request latency rising to 10 s on the ordering API under load, while CPU stays below 30%. Read the code, the configuration and the last 30 commits, find the most likely causes, and write incident.md with a timeline, each suspect with file and line evidence, and the check that would confirm or clear it. Change no code.", "resources": [ { "url": "https://github.com/your-org/your-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back What comes back is incident.md as a session artifact: the finished session lists it with its name, mime_type and a url on api.prix.dev that downloads with no key for an hour. The report opens with a timeline built from the last 30 commits and any Datadog data the run read, then ranks the suspects with file and line evidence; on an eShop-shaped codebase that is the tokenless queries in src/Ordering.API/Application/Queries/OrderQueries.cs, the per-command transaction in src/Ordering.API/Application/Behaviors/TransactionBehavior.cs, and the retry pipelines in src/eShop.ServiceDefaults/Extensions.cs and src/EventBusRabbitMQ/RabbitMQEventBus.cs. Each suspect ends with a check the on-call engineer can run before changing anything: the dotnet-counters command and counters to watch, the Datadog query that would show the queue length and latency together, or the git show that confirms a suspect commit. Because the prompt says change no code, the deliverable is read-only by construction. If the first pass points at one suspect, a follow-up POST to /o/{org}/sessions/{id}/messages asks the agent to dig into just that one, and the sandbox wakes where it left off. ## Run it on every thread-pool alert Point your monitoring at a run per alert: when the Datadog monitor on .NET thread-pool queue length fires, your server POSTs this same request with the alert text folded into input, one run per alert. Each run works in its own sandbox on its own clone and writes its own incident.md, so a latency spike gets a cited first pass before the on-call engineer opens Datadog. Set a webhook URL on the API key and Rush POSTs run.completed or run.failed when each turn finishes, with the session URL your server can follow to download the report. A key runs at most max_concurrent sessions at once (4 by default) and answers 429 past that, so a burst of alerts should be queued on your side and retried with the same Idempotency-Key. Because resources names the repository, the Rush GitHub App has to be installed on it for the run to clone it; a run that only reads metrics needs no repository at all. ## Why run it on Rush? On a laptop this first pass competes with everything else you are doing during the incident, and its notes die in a scratch buffer. On Rush the run works in its own cloud sandbox with its own clone: it runs the greps, the git log and the file reads against the real repository and keeps every command's output in the trajectory, so a second engineer can verify each citation. Nothing it reads is production, and the prompt forbids code changes, so the investigation is safe to start while the incident is still open. The proof is built in. During a multi-service incident you can start one run per service, the ordering API and the services it calls over HTTP and RabbitMQ, and each comes back with its own ranked report and its own checks, run in parallel instead of serialized on one laptop. When the follow-up work is a fix, the same session can take it on and the result arrives as a pull request a reviewer checks before merging. ## Frequently asked questions ### Can the agent see my Datadog metrics? Yes, when the run names your Datadog connection. An admin signs in once with POST /me/connections against Datadog's official OAuth MCP server, and the run lists it in connections so the agent can read thread-pool counters and slow traces. Without it, the report works from the numbers you paste into input, which is enough for the code-level suspects. ### Will it change any code? No. The prompt says change no code, and the deliverable is a report. Each suspect carries a check that confirms or clears it, so nothing in the report asks you to trust an unverified guess. If you later want the fix, send a follow-up to the same session and the change comes back as a pull request you review before merging. ### What if the report blames the wrong thing? Every suspect is written with the evidence and the check that would clear it, ranked rather than asserted, and the transcript keeps every command and file read behind each citation. An engineer can clear a suspect with one of the listed checks in minutes, and a follow-up asks the agent to re-rank with the new information. ### Does my app need a particular .NET version for this to work? No. The investigation is read-only: it greps source, reads configuration and walks recent commits. The agent notes the pinned versions it finds, such as the SDK in global.json and the packages in Directory.Packages.props, because the right suspect list depends on them, but it does not need to build or run your app.