What the agent does
The run uses the GitLab connection: GitLab's MCP server at https://gitlab.com/api/v4/mcp, from the Rush connector catalog, connected once with an OAuth sign-in and named gitlab on the request. It reads pipelines and job logs only. Nothing is cloned, because flakiness shows up in pipeline history, not in the code.
The agent lists the group's projects with list_projects, then each project's pipelines from the last 14 days with list_pipelines and created_after. For each pipeline it calls get_pipeline with the jobs facet, which returns every job with its name, stage, status, duration, commit and whether it was retried. That gives a table in the sandbox of every job run in the window.
Two numbers come out of that table. The failure rate per job name per project: failed runs over all runs. And the flaky count: the number of times a job failed and then passed on a retry at the same commit SHA, with no new push in between. The second is the real signal, because a job that fails on a broken commit and passes on the fix is doing its job. The agent also adds up the minutes those failed attempts took, so the list can be ranked by time wasted.
For the top five flaky jobs it reads one failed attempt's log with get_job, asking for the log and limiting the bytes to the end of it, where the error usually is. It quotes the failing line: a timeout, a connection refused from a service container, a test name. It does not call save_pipeline to retry or cancel anything.
Start it with one API call
POST https://api.prix.dev/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Use the GitLab connection to read pipelines and their jobs from the last 14 days across every project in the group my-group. Write pipelines.md with: failure rate per job per project; jobs that failed and then passed on retry at the same commit, with how many times and the minutes those failed attempts took, ranked by minutes; and for the top five, one failing log line from a failed attempt with a link to the job. Change nothing: no retries, cancellations or comments.",
"connections": [
"gitlab"
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
The session returns pipelines.md. The first table ranks flaky jobs by minutes lost, with project, job name, stage, flaky count, total runs and failure rate. Under it, the top five each get a short section with a link to one failed job and the log line that failed, quoted. A last table lists jobs with a high failure rate that never passed on retry, which are real breakages rather than flakes.
A reviewer checks the log lines before filing tickets: the same job can fail for two reasons, and the quoted line is one example, not every cause. The transcript holds every pipeline and job the agent read, so a count that looks wrong can be traced to the pipelines behind it.
Run it on every two-week sprint
Run it at the end of each sprint, with the same 14-day window, so each report covers a sprint's pipelines. A Project automation on a schedule can send it, or a scheduled GitLab pipeline can call your endpoint, which posts the request.
Keep each report and the trend shows whether the fixes worked: attach last sprint's pipelines.md as a document block and ask which flakes went away and which are new. The run.completed webhook tells your server when to fetch it.
The use case this job follows, with its automation and trigger: Produce a report or file, no repo.
Why run it on Rush?
GitLab shows each pipeline's jobs, but finding the jobs that fail and pass at the same commit across a group means opening hundreds of pipelines. The agent reads them all through GitLab's own server, does the counting in its own sandbox and keeps the data it counted from, so the ranking is something you can check.
It runs without a CI runner or anyone's laptop, it only reads, and every call is recorded. A platform team can run it for several groups at once, one session each, and hand each team its own list.
Frequently asked questions
How does it tell a flaky job from a broken one?
A flaky job failed and then passed at the same commit with no new push. A job that fails until a new commit lands is counted in the failure rate but not as flaky.
Will it retry pipelines or open issues?
No. The prompt rules out changes and the agent uses only reading tools. Ask in a follow-up for issues to be drafted for the top flakes once you have reviewed them.
Can it read long job logs?
It reads a slice of each log, near the end where the failure usually is, and only for the top five jobs. Ask for more lines or more jobs in the prompt if you need them.
Does it work with GitLab self-managed?
Connect your instance's /api/v4/mcp address under the name gitlab instead of the gitlab.com one, if your version has the MCP server enabled.