All jobs

Investigate Connection Pool Exhaustion in a Spring Boot App

Yes. An agent reads your Spring Boot code, datasource settings and recent commits, and reports what holds HikariCP connections, with file, line and a check.

Run this on RushIn a Project that has your repository.

What the agent does

The agent clones the repository into its own cloud sandbox and starts from the error you paste: Connection is not available, request timed out after 30000ms, with active connections pinned at the maximum during a spike. That message is HikariCP giving up after its default connectionTimeout of 30 seconds, so the agent asks two questions of the code: how many connections the pool has, and what holds one for longer than it should.

In an app like spring-projects/spring-petclinic it reads pom.xml (the spring-boot-starter-parent 4.1.0, Java 17) and every properties file. application.properties sets spring.jpa.open-in-view=false, which means a connection is released when the repository call returns instead of being held until the view finishes rendering, and hibernate.default_batch_fetch_size=16. application-mysql.properties and application-postgres.properties set only the URL, user and password, with no spring.datasource.hikari.* keys, so the pool runs at HikariCP's default of 10 connections. The agent notes that one pod at 10 connections handles only as many concurrent database calls as that, however many request threads Tomcat has.

Then it reads what each request does with a connection. owner/Owner.java maps pets with @OneToMany(fetch = FetchType.EAGER) at line 67 and owner/Pet.java maps visits the same way at line 56, so loading owners also loads their pets and every pet's visits. vet/Vet.java loads specialties eagerly at line 47. vet/VetRepository.java caches both findAll() and findAll(Pageable) under the one vets cache, configured in system/CacheConfiguration.java; if a recent commit removed @Cacheable or the cache bean, every vet list would go back to the database. The agent also looks for anything that holds a connection across slow work: a @Transactional method that calls an HTTP API or sends mail, a stream left open, or a JdbcTemplate call inside a loop.

Last, it reads the 30 most recent commits with git log -p, looking for a smaller pool, a new eager association, a removed cache, open-in-view turned back on, or a new query in a hot controller such as OwnerController.findPaginatedForOwnersLastName. Where it helps, it starts the app in the sandbox against the H2 database the default profile uses, sends concurrent requests and reads /actuator/metrics/hikaricp.connections.active (the app exposes every actuator endpoint), to show which page holds connections longest. It writes incident.md and changes no code.

Start it with one API call

POST https://api.prix.dev/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Incident: Grafana shows HikariCP active connections pinned at the maximum and requests failing with Connection is not available, request timed out after 30000ms, during a traffic spike. Read the code, the configuration and the last 30 commits, find the most likely causes, and write incident.md with a timeline, each suspect with file and line evidence, and the check that would confirm or clear it. Change no code.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is incident.md, attached to the session as an artifact. It starts with the pool as configured (size, timeouts, where each value comes from, and which are HikariCP defaults because nothing sets them), then a timeline of the commits in the window, then the suspects in order. Each suspect names the file and line, explains how it keeps a connection checked out, and gives the check that settles it: hikaricp.connections.pending and hikaricp.connections.usage from Micrometer during the spike, HikariCP's leakDetectionThreshold turned on for one pod, a thread dump showing request threads waiting in HikariPool.getConnection, or the database's own view of sessions (pg_stat_activity or SHOW PROCESSLIST).

When the agent measured something in the sandbox, the report includes the command and output and marks it as a sandbox number on H2, not a production one. The session's transcript keeps every file the agent read and every command it ran.

A reviewer checks the top suspect against production metrics before changing anything, and is wary of the easy fix: raising maximum-pool-size hides a connection held during slow work and moves the pressure to the database. If the evidence points at a code path, a follow-up to the same session can ask the agent to fix it in a pull request.

Run it on every pool saturation alert

Alert on hikaricp_connections_pending above zero for a few minutes, or on hikaricp_connections_active at the maximum, and point the alert's webhook at a handler on your server that sends this request with the service, the time and the numbers in the prompt. The run starts from the repository as it is, so the report reflects the code that is deployed.

With a webhook URL on the API key, Rush POSTs run.completed with the session_url when the turn finishes; your handler fetches the report from the session's artifacts and posts it to the incident.

The use case this job follows, with its automation and trigger: Investigate an incident.

Why run it on Rush?

Pool exhaustion is rarely the pool's fault. Finding the cause means reading entity mappings, transaction boundaries, cache configuration and properties across the codebase, then checking which recent change moved one of them. In a Rush sandbox the agent does that reading with a JDK and Maven at hand, so it can also start the app and watch the pool under load without touching a shared environment or your laptop.

The report is checkable: each suspect carries its file and line, and each measurement its command and output in the session's transcript. When several services hit the same database at once, each can get its own run, up to the key's concurrent run limit, and the reports can be compared side by side.

Frequently asked questions

Will it just tell me to increase maximum-pool-size?

Only if the evidence supports it. The report first looks for what holds connections too long, such as eager associations, slow work inside a transaction or a missing cache, because a bigger pool hides those and shifts load to the database. If the pool is simply too small for the traffic, it says so and shows the arithmetic.

Does it need access to my production database?

No. It reads the code and configuration, and can run the app in its sandbox against an in-memory or local database. The report names the database and Micrometer checks your team runs in production.

Does it work with Gradle builds and Kotlin?

Yes. It reads build.gradle or build.gradle.kts the same way it reads pom.xml, and Kotlin JPA entities and repositories the same way as Java ones.

Can it read our HikariCP leak detection logs?

Yes, if you attach them to the request as a file. A leakDetectionThreshold warning includes the stack trace where the connection was taken, and the agent matches it to the code.