All jobs

Add Priority Classes to Kubernetes Manifests

Yes. An agent adds PriorityClass objects, sets priorityClassName on the Deployments that must survive node pressure, and proves kustomize and helm still build.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent maps where the workloads are declared before it decides which of them are important. In GoogleCloudPlatform/microservices-demo the same application is written three times: kubernetes-manifests/ holds one file per service with a Deployment, a Service and a ServiceAccount, and cartservice.yaml also carries the redis-cart Deployment; kustomize/base/ is a copy of those files with kustomize/components/ layered on top, and kustomize/kustomization.yaml lists the components a reader can uncomment; helm-chart/ renders the same services from values.yaml with a template per service and the shared common.yaml. release/kubernetes-manifests.yaml is not a fourth copy: docs/releasing/make-release-artifacts.sh concatenates every file in kubernetes-manifests/ except the kustomization and rewrites the image tags, so a change there reaches the release file at the next release. A search for priorityClassName across all of them finds nothing, so every pod runs at the default priority and under node pressure the scheduler has no reason to keep checkout alive before the load generator.

It then chooses the classes and which pods get them, and writes the reasoning down because this is a policy, not a bug fix. Two PriorityClass objects, both with globalDefault: false so nothing changes for pods that do not ask: one for the request path a customer hits during checkout, frontend, checkoutservice, paymentservice, cartservice and its redis-cart, and a lower one for loadgenerator, whose only job is to make traffic and which loadgenerator.yaml already runs with one replica and larger requests than most services. The remaining services keep the default. The values and the gap between them are the agent's proposal, left for review; the file comments say that both must sit below the built-in system-cluster-critical and system-node-critical classes so the kube-system pods always win.

Three edits follow the three layouts. In kubernetes-manifests/, a new priority-classes.yaml and a priorityClassName line in each chosen Deployment's pod spec, plus the new file in that directory's kustomization.yaml, which the release script does not read but kubectl apply -k does. For Kustomize, the agent follows the repository's own pattern instead of editing base/: a components/priority-classes/ with the two objects and one patch per Deployment, listed as a commented line in kustomize/kustomization.yaml like network-policies and without-loadgenerator, and a new scenario under kustomize/tests/ so the component is built in CI. For Helm, a priorityClasses.create switch in values.yaml with a priorityClassName field under each service, the objects added to common.yaml behind the switch, and a {{- with }} block in each service template, so a chart user who already has classes can name theirs.

The checks are the ones the repository runs. .github/workflows/kustomize-build-ci.yaml runs kubectl kustomize . in kustomize/ and kustomize build for every directory under kustomize/tests/, and helm-chart-ci.yaml runs helm lint --strict and several helm template variants piped through kustomize build; the agent runs each of those locally and adds helm template . --set priorityClasses.create=true to show the rendered objects. PriorityClass is cluster-scoped, so the agent also notes that a cluster where the names already exist with other values will refuse the apply, and that an existing pod does not change priority until it is recreated. With no cluster in its sandbox it does not claim an apply; it says what it rendered.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add priority classes to this Kubernetes application. First find every place the workloads are declared: plain manifests, kustomize base and components, a helm chart, and any generated release file, and read how the release file is produced. Confirm no priorityClassName exists today. Propose two PriorityClass objects with globalDefault false, one for the customer request path and one lower class for the load generator, with comments explaining that both stay below the system-cluster-critical and system-node-critical classes, and list in the pull request which Deployments get which class and why. Apply the change in each layout the way that layout already extends itself: a new manifest file plus priorityClassName lines in the plain manifests, a kustomize component with patches rather than edits to base, and a values switch with per-service priorityClassName fields in the helm chart. Run the same build checks the CI workflows run (kubectl kustomize, kustomize build for every test scenario, helm lint --strict, helm template through kustomize build) and a helm template with the switch on, paste the output in the pull request and open it. Do not claim to have applied anything to a cluster.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that adds kubernetes-manifests/priority-classes.yaml, a kustomize/components/priority-classes/ directory with its patches and a kustomize/tests/ scenario, and priorityClasses values with template changes under helm-chart/; it edits the pod spec of the chosen Deployments in kubernetes-manifests/ and touches no other service. The description holds a table of Deployment to class, the rendered PriorityClass objects from helm template, and the output of every kustomize and helm check.

A reviewer decides the policy the agent proposed: whether paymentservice belongs with checkout, whether recommendationservice and adservice should be left at the default, and what the two numeric values should be for the cluster this runs on, since a class value is compared with every other class on that cluster and not only with the ones in this repository. They also confirm the names do not collide with classes the platform team already created.

They then check the thing the diff cannot show: that the release file will carry the new objects, which it does because the release script reads every YAML file in kubernetes-manifests/, and that the kustomize component is listed in the commented block of kustomize/kustomization.yaml so readers find it.

Run it on every Linear issue labelled scheduling

Once the classes exist, later requests are a service to move between them or a class to add. In the console, add an automation on the Project that holds the repository with a linear trigger on linear.issue.labeled and the label scheduling, and a prompt that says: change the priority class of the service the issue names in all three layouts, run the kustomize and helm checks, and open a pull request. The issue is placed after the prompt as the event and fills in nothing, so the service and the class come from the issue text. The trigger needs a Linear connection for the automation's runner and fires only when the label is applied by someone in your Rush org.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

The work is mostly reading: three copies of the same application, a release script that decides which copy ships, and a CI that builds each kustomize scenario. Missing one copy produces a cluster where helm users and manifest users get different behaviour under pressure. The agent reads all of it in its own sandbox with kubectl, kustomize and helm installed, and runs the repository's own build checks rather than guessing that the templates still render.

Every render and check stays in the run's transcript at GET /o/{org}/sessions/{id}/messages, so the PriorityClass objects the pull request quotes are the ones helm template printed, and the changed files are listed with their diff at GET /o/{org}/p/{project}/sessions/{id}/changes. A platform reviewer who wants to argue the values down argues with rendered YAML, not with a description of it.

Frequently asked questions

Which services get the higher class?

The agent proposes the checkout path: frontend, checkoutservice, paymentservice, cartservice and redis-cart, because losing any of them fails a purchase. loadgenerator gets the lower class so it is the first pod evicted when a node runs short. Everything else stays at the default, and the pull request lists the table so a reviewer can move a service.

Will this change pods that are already running?

No. A pod's priority is set when it is admitted, so existing pods keep the default until their Deployment rolls them. Applying the change rolls the Deployments whose pod spec changed, which is the chosen services only.

Why a kustomize component instead of editing base?

Because that is how this repository adds every optional behaviour: network-policies, without-loadgenerator and the others are components a reader enables in kustomize/kustomization.yaml, and kustomize/tests/ builds combinations of them in CI. A component keeps the base identical to the plain manifests and gives the choice to whoever deploys.

What happens if a class with the same name already exists on the cluster?

PriorityClass is cluster-scoped, so an apply with a different value for an existing name is refused. The pull request names the two classes so a platform team can rename them or point priorityClassName at classes they already manage, which the helm values make a one-line change.

More Kubernetes jobs

More jobs like this: Turn a Linear label into a PR

Keep reading