All jobs

Set Up Workload Identity on GKE with Terraform

Yes. An agent moves a GKE app's Google service accounts, IAM roles and Workload Identity bindings into Terraform and Kustomize, validates both and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads what the cluster already has before it writes any IAM, because on GKE the answer depends on the cluster mode. In a deployment like GoogleCloudPlatform/microservices-demo, terraform/main.tf creates google_container_cluster.my_cluster with enable_autopilot = true, an empty ip_allocation_policy and deletion_protection left at its default, after a project_services module enables container, monitoring, cloudtrace and cloudprofiler and, with var.memorystore, redis; a gcloud module fetches credentials and a null_resource runs kubectl apply -k ${var.filepath_manifest} followed by a kubectl wait. providers.tf pins the google provider at 7.46.1, variables.tf defines gcp_project_id, name, region, namespace, filepath_manifest and memorystore, and terraform.tfvars holds a project id placeholder. There is no google_service_account, no google_project_iam_member and no workload_identity_config block, which is correct for Autopilot: Workload Identity Federation is on by default there, so the missing half is the Google side. Every service in kustomize/base/ runs as its own Kubernetes ServiceAccount (adservice, cartservice, checkoutservice, currencyservice, emailservice, frontend, loadgenerator, paymentservice, productcatalogservice, recommendationservice, shippingservice), and three components need Google APIs: google-cloud-operations deploys an opentelemetrycollector Deployment with no serviceAccountName, so it runs as default, and its README tells the reader to bind default/default to a Google service account with gcloud by hand; spanner and alloydb patch the cartservice ServiceAccount with an iam.gke.io/gcp-service-account annotation whose value is a placeholder their READMEs replace with sed after gcloud iam service-accounts create and two add-iam-policy-binding commands. .github/workflows/terraform-validate-ci.yaml runs terraform init -backend=false and terraform validate; kustomize-build-ci.yaml runs kubectl kustomize . and builds every directory under kustomize/tests/.

The Terraform change replaces the by-hand commands. The agent adds terraform/workload-identity.tf with a google_service_account for the telemetry collector and one for the boutique services, google_project_iam_member resources granting roles/cloudtrace.agent, roles/monitoring.metricWriter and roles/cloudprofiler.agent to the collector account and the profiler role to the services account, since the operations component sets ENABLE_PROFILER on checkoutservice and frontend and clears DISABLE_PROFILER on the others, and a google_service_account_iam_member per Kubernetes service account with roles/iam.workloadIdentityUser and a member of the form serviceAccount:${var.gcp_project_id}.svc.id.goog[${var.namespace}/<ksa>], built with for_each over a local list of the eleven base accounts plus the collector's. The Spanner and AlloyDB accounts stay where their READMEs put them, behind a cartservice_database variable that adds roles/spanner.databaseUser or roles/alloydb.client and roles/secretmanager.secretAccessor to a third account, so a team that uses neither creates nothing. output.tf gains the three emails.

The Kubernetes side becomes a component like the others. kustomize/components/workload-identity/ holds a Component that gives the collector its own ServiceAccount and sets serviceAccountName on its Deployment, and patches every base ServiceAccount with the iam.gke.io/gcp-service-account annotation, using a placeholder value in the same style as SPANNER_DB_USER_GSA_ID, with a README that shows the sed line fed from terraform output. The null_resource that applies the manifests gets that substitution before kubectl apply, so terraform apply leaves nothing to do by hand. The google-cloud-operations README's Workload Identity section is rewritten to point at the component, and kustomize/tests/ gains a directory that combines workload-identity with google-cloud-operations, so the existing build job covers it.

The agent has no Google project, so it validates what can be validated without one and says so. It runs terraform fmt -check, terraform init -backend=false and terraform validate as CI does, and kubectl kustomize on the root, on the new test directory and on each of the spanner and alloydb combinations, diffing the rendered ServiceAccount objects before and after to show that only the annotation and the collector's account changed. The pull request states that terraform plan and the bindings themselves were not applied, and lists the gcloud commands from the three READMEs that the Terraform now replaces, one by one.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Set up Workload Identity for this GKE deployment in Terraform and Kustomize. Read terraform/main.tf, providers.tf, variables.tf, output.tf, terraform.tfvars, kustomize/kustomization.yaml, every ServiceAccount in kustomize/base, the google-cloud-operations, spanner and alloydb components with their READMEs, and the two CI workflows under .github/workflows first. Note the cluster mode and what it already enables. Add a Terraform file with Google service accounts for the telemetry collector and the services, the trace, metric and profiler roles they need, and a workloadIdentityUser binding per Kubernetes service account built from the project id and namespace variables; keep the database roles behind a variable. Add a Kustomize component that gives the collector its own service account and annotates every base service account with a placeholder the apply step substitutes from terraform output, add a kustomize/tests directory that combines it with google-cloud-operations, and rewrite the README sections that currently do this with gcloud. Run terraform fmt -check, init with no backend and validate, render the root, the test directories and the database combinations with kubectl kustomize, diff the ServiceAccount objects, and open a pull request that states what could not be applied without a project.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

A pull request that adds terraform/workload-identity.tf, a variable and three outputs, the kustomize/components/workload-identity/ component with its README, a kustomize/tests/ combination, a substitution step in the apply resource, and rewritten Workload Identity sections in three READMEs. The description shows the terraform validate result, the rendered ServiceAccount diff per combination, the mapping from each old gcloud command to the resource that replaces it, and a clear line that no plan or apply ran.

A reviewer checks the role list against what each service calls, decides whether the services should share one Google account or get one each, and merges. The next terraform apply creates the accounts and bindings, substitutes the emails and applies the manifests, and the collector exports traces without anyone running kubectl annotate by hand.

Run it on every terminal hand-off

A platform engineer who has just followed a README's gcloud steps for the second time starts this from the terminal with this body and the repository in resources, and goes back to the cluster. The run reads the real cluster resource and the real components, so it adds IAM rather than a redundant cluster flag, and the bindings name the service accounts the manifests actually use.

Send a request_id, a UUID, with the body so a retried start returns the run it already began. When the turn ends, GET /o/{org}/sessions/{id} names the pull request, and a follow-up to the same session can ask for one Google account per service, or for the same bindings expressed with principal:// IAM members instead of annotations.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

Workload Identity is a mapping between two sets of names, and a mistake in either set fails silently until a pod gets a permission error. The agent reads both sets from the repository, generates the bindings from the same variables the cluster uses, renders every manifest combination in its own sandbox, and keeps each command and diff in the run's record, so the pull request shows which accounts gained which annotation rather than a description of IAM.

The same run fits any GKE deployment: a Standard cluster where the workload_identity_config block and node pool metadata setting have to be added first, a Helm-based app where the annotation goes into values.yaml, or a multi-namespace setup where the member string carries each namespace. Each gets Terraform derived from its own cluster resource and manifests checked against its own build job.

Frequently asked questions

Does the cluster need a workload_identity_config block?

Not on Autopilot, where Workload Identity Federation is enabled by default, which is why the run adds no cluster flag. On a Standard cluster the block is required, along with the metadata setting on each node pool, and the run adds both when it reads enable_autopilot is absent.

Why does the collector get its own service account?

The component's Deployment has no serviceAccountName, so it runs as the namespace's default account, and binding that account to a Google account would grant trace and metric write access to every pod with no account of its own. A dedicated account keeps the grant to the one workload that needs it.

Can this be applied without the placeholder substitution?

The annotation needs the Google account's email, which exists only after Terraform creates it, so either the apply step substitutes it or a person runs the sed line from the component's README once. The pull request wires the substitution into the existing apply resource so terraform apply does both.

What was not verified?

Anything that needs a Google project: terraform plan, the IAM bindings themselves and a pod exporting a trace. The run validates the Terraform, renders every Kustomize combination and diffs the service accounts, and the pull request says that the first real apply is where the bindings are proven.

More GKE jobs

More jobs like this: Hand off a long task

Keep reading