# Investigate a Queue Backlog in a Laravel App Yes. An agent reads your Laravel jobs, schedule, queue config and recent commits, and reports why the queue backs up, with file and line for each cause. [Run this on Rush](https://cloud.getrush.ai/run/investigate-queue-backlog/laravel/), in a Project that has your repository. ## What the agent does The agent clones the repository into its own cloud sandbox and starts from the pattern you describe: queue depth past 20,000 every night at 02:00, draining only by 09:00, with worker memory climbing. A backlog that starts at the same time every night is almost always the scheduler, so the agent reads what is scheduled first, then what each scheduled task dispatches, then how the workers that drain it are configured. In an app like koel/koel it reads `routes/console.php`, where six `Schedule::job(new RunCommandJob(...))->daily()` entries run `koel:scan`, `koel:prune`, `koel:podcasts:sync`, the two clean-up commands and `model:prune`. `config/app.php` sets the timezone to `UTC`, so `daily()` fires at midnight UTC, and the agent works out whether that matches a 02:00 alert in your Grafana timezone. It reads `app/Jobs/RunCommandJob.php`: the job sets `$timeout = 0`, wraps itself in `WithoutOverlapping` with `dontRelease()` and a 24-hour lock, and runs the command with `Artisan::call` inside the queue worker. A full library scan therefore runs inside one long-lived worker process, which explains memory rising in that worker. Then it follows what the scan dispatches per song. `app/Services/Transcoding/TranscodeOnScan.php` dispatches a `TranscodeSongJob` at line 45 for each song that needs transcoding when `TRANSCODE_ON_SCAN` is on, and `app/Services/AudioAnalysis/AnalyzeAudioOnScan.php` dispatches an `AnalyzeSongAudioJob` at line 36 when `ANALYZE_AUDIO_ON_SCAN` is on. `app/Observers/AlbumObserver.php` dispatches thumbnail and small-cover jobs at lines 27 and 31 whenever an album with a cover is saved. Each transcode job may run up to `TRANSCODE_TIMEOUT` (300 seconds by default) plus a 30-second margin, and each analysis job up to 330 seconds. The agent counts how many jobs one scan can enqueue and how long they take with the worker count you run. It reads `config/queue.php`, where the `database` connection is the default and `retry_after` is 360 seconds, and checks it against each job's `$timeout`, because a job that runs past `retry_after` is handed to a second worker and done twice. Last, it reads the 30 most recent commits with `git log -p` for a new scheduled task, a new per-song job, a changed default or a removed `WithoutOverlapping`, and writes `incident.md`. It changes no code. ## Start it with one API call `POST https://api.prix.dev/o/{org}/sessions` ```json { "agent": "claude", "input": "Incident: Grafana shows the Laravel queue depth climbing past 20,000 jobs every night at 02:00 and not draining until 09:00, with worker memory rising steadily. Read the code, the configuration and the last 30 commits, find the most likely causes, and write incident.md with a timeline, each suspect with file and line evidence, and the check that would confirm or clear it. Change no code.", "resources": [ { "url": "https://github.com/your-org/your-app" } ] } ``` 1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token. 2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox. 3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory. ## What comes back What comes back is `incident.md`, attached to the session as an artifact. It opens with the nightly timeline as the code defines it: what the scheduler starts and when, in your timezone, and what each of those dispatches. The suspects follow in order, each with the file and line, the estimated number of jobs or seconds it contributes, and the check that confirms it: `SELECT queue, count(*) FROM jobs GROUP BY queue` or a count by `payload->displayName` at 03:00, `php artisan queue:monitor`, the worker's memory per job from your process metrics, or the `failed_jobs` table for jobs that timed out and were retried. The report separates what the code proves from what depends on your settings, since a per-song job only exists when its environment flag is on. Every file the agent read and every command it ran are in the session's transcript. A reviewer checks the counts against the real `jobs` table before changing anything. Typical fixes are moving the scan's follow-up jobs to their own queue with their own workers, running workers with `--max-jobs` or `--memory` so they restart cleanly, or aligning `retry_after` with the longest job timeout. A follow-up message to the same session can ask the agent to make one of them as a pull request. ## Run it on every morning queue depth check Run it on a schedule: a cron job on your server at 03:00 that reads the queue depth and, when it is above your threshold, sends this request with the number in the prompt. Or point a Grafana alert on queue depth at a handler that does the same. Each run reads the repository as it is that night. With a webhook URL on the API key, Rush POSTs `run.completed` when the turn finishes, so the report is waiting for whoever starts work in the morning. ## Why run it on Rush? A nightly backlog spans the scheduler, the jobs, observers, queue config and environment flags, and the person who wrote each piece may not be around at 02:00. In a Rush sandbox the agent reads all of it at once, with PHP and Composer available to run `php artisan schedule:list` or the test suite, without touching your workers or your database. The result is a report a team can act on: each suspect with its file and line, each count with how it was derived, and the full record of the work in the session's transcript. If several apps share the problem, each gets its own run, up to the key's concurrent run limit. ## Frequently asked questions ### Does it work with Horizon and Redis queues? Yes. With Horizon it also reads `config/horizon.php` for supervisors, queues, `maxProcesses` and balancing, which often explain why one queue drains slowly while others sit idle. The checks it suggests use Horizon's dashboard and metrics instead of the `jobs` table. ### Why would worker memory keep rising? A queue worker is one long-lived PHP process, so anything a job leaves in memory stays until the worker restarts. A job that runs a whole Artisan command, as `RunCommandJob` does, is the usual culprit. The report points at the job and suggests `--max-jobs` or `--memory` on `queue:work` as the check and the mitigation. ### Can it look at our failed_jobs table or logs? Not on its own, since it works from the repository. Attach an export of `failed_jobs` or a log file to the request's input and the agent ties each failure to the job class and line. ### Will it change the schedule or the jobs? No. The prompt says to change no code, and the result is a Markdown report. Ask for a fix in a follow-up once you have confirmed the cause.