Cloud Run with GKE Autopilot: "Required parameter namespace was null or undefined" error despite nam

[chuhongyang99]

I'm encountering a persistent error when trying to create pods in my GKE Autopilot cluster from my Cloud Run service. The error message is:

RequiredError: Required parameter namespace was null or undefined when calling CoreV1Api.createNamespacedPod.

However, I've verified that the namespace variable is correctly set in my Cloud Run code immediately before the createNamespacedPod call. Here's the relevant code snippet:

JavaScript

// ... other code ...

const namespace = process.env.NAMESPACE || 'default'; // Or hardcoded 'default' for testing

console.log("Namespace just before createNamespacedPod:", namespace); // The CRITICAL check

const createdPod = await k8sApi.createNamespacedPod(namespace, pod);

// ... rest of the code ...

The console.log statement shows that the namespace variable is indeed set to default.

I'm using Workload Identity Federation with GKE Autopilot. My setup is as follows:

1. Kubernetes Service Account: I created a service account in my GKE cluster:

   Bash

   kubectl create serviceaccount cloudrun-sa2 -n default

2. Annotation: I annotated the service account with my GCP service account email:

   Bash

   kubectl annotate serviceaccount cloudrun-sa2 -n default iam.gke.io/gcp-service-account=gke-cloud-runner@gke-pod-test.iam.gserviceaccount.com

3. RBAC: I've configured RBAC to grant the cloudrun-sa2 service account the necessary permissions to create pods:

   YAML

   # Role
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: pod-creator2
     namespace: default
   rules:
   - apiGroups: [""]
     resources: ["pods"]
     verbs: ["create"]
   
   # RoleBinding
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: cloudrun-sa-binding2
     namespace: default
   subjects:
   - kind: ServiceAccount
     name: cloudrun-sa2
     namespace: default
   roleRef:
     kind: Role
     name: pod-creator2
     apiGroup: rbac.authorization.k8s.io
   

   I've confirmed that the RBAC is working correctly using kubectl auth can-i:

   Bash

   kubectl auth can-i create pods -n default --as=system:serviceaccount:default:cloudrun-sa2
   yes

4. Cloud Run Service: My Cloud Run service is associated with the same GCP service account that I used in the annotation. I've redeployed the service after every change.

5. Kubernetes Client Library: I'm using the latest version of @kubernetes/client-node (version: @kubernetes/client-node@1.0.0). I've also tried other versions but the issue persists.

6. Node.js Version: My Cloud Run service is using Node.js version 18.

7. Simplified Test: I've created a minimal Cloud Run service that only tries to create a pod with a hardcoded default namespace. Even this simplified service fails with the same error.

8. Environment Variables: I've checked and there are no conflicting environment variables set in my Cloud Run service.

9. Cloud Run Logs: Here's the full error message and stack trace from my Cloud Run logs:

2025-02-17 21:18:14.945 HKT
Cloud RunReplaceServicecloudrun-gke-pod-00014-z5x {@type: type.googleapis.com/google.cloud.audit.AuditLog, methodName: /Services.ReplaceService, resourceName: namespaces/gke-pod-test/revisions/cloudrun-gke-pod-00014-z5x, response: {…}, serviceName: run.googleapis.com, status: {…}}
2025-02-17 21:18:16.242 HKT
Cloud RunReplaceServicecloudrun-gke-pod {@type: type.googleapis.com/google.cloud.audit.AuditLog, methodName: /Services.ReplaceService, resourceName: namespaces/gke-pod-test/services/cloudrun-gke-pod, response: {…}, serviceName: run.googleapis.com, status: {…}}
2025-02-17 21:18:20.967 HKT
GET500311 B52 msPostmanRuntime/7.43.0 https://cloudrun-gke-pod-923754347161.asia-southeast1.run.app/create-pod?Authorization=Bearer%20AIzaSyCwf6VdGLRnlvIpTYTUH8QtV0wIBh_bCN0
2025-02-17 21:18:21.002 HKT
Permission check failed: RequiredError: Required parameter namespace was null or undefined when calling CoreV1Api.createNamespacedPod.
2025-02-17 21:18:21.002 HKT
at CoreV1ApiRequestFactory.createNamespacedPod (file:///usr/src/app/node_modules/@kubernetes/client-node/dist/gen/apis/CoreV1Api.js:2484:19)
2025-02-17 21:18:21.002 HKT
at ObservableCoreV1Api.createNamespacedPodWithHttpInfo (file:///usr/src/app/node_modules/@kubernetes/client-node/dist/gen/types/ObservableAPI.js:12017:59)
2025-02-17 21:18:21.002 HKT
at ObservableCoreV1Api.createNamespacedPod (file:///usr/src/app/node_modules/@kubernetes/client-node/dist/gen/types/ObservableAPI.js:12042:21)
2025-02-17 21:18:21.002 HKT
at ObjectCoreV1Api.createNamespacedPod (file:///usr/src/app/node_modules/@kubernetes/client-node/dist/gen/types/ObjectParamAPI.js:4568:25)
2025-02-17 21:18:21.002 HKT
at file:///usr/src/app/index.js:31:41
2025-02-17 21:18:21.002 HKT
at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
2025-02-17 21:18:21.002 HKT
at next (/usr/src/app/node_modules/express/lib/router/route.js:149:13)
2025-02-17 21:18:21.002 HKT
at Route.dispatch (/usr/src/app/node_modules/express/lib/router/route.js:119:3)
2025-02-17 21:18:21.002 HKT
at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
2025-02-17 21:18:21.002 HKT
at /usr/src/app/node_modules/express/lib/router/index.js:284:15 {
2025-02-17 21:18:21.002 HKT
api: 'CoreV1Api',
2025-02-17 21:18:21.002 HKT
method: 'createNamespacedPod',
2025-02-17 21:18:21.002 HKT
field: 'namespace'
2025-02-17 21:18:21.002 HKT
}

I've tried numerous troubleshooting steps, including:

• Double-checking the annotation and RBAC.
• Simplifying the Cloud Run service code.
• Updating the @ kubernetes/client-node library.
• Checking for conflicting environment variables.
• Verifying the Cloud Run service account.

I'm at a loss as to why I'm still getting this error, especially since the namespace variable is correctly set and kubectl auth can-i returns yes. Any help would be greatly appreciated!