Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

Cloud Function connect to GKE over VPC Connector across Projects

Need some help regarding VPC Serverless Connectors. So we are using one Cloud function to connect to Kubernetes API which is across multiple projects, recently got a notification from GCP that any public endpoint connectivity to Kubernetes API server will be blocked and we need to set serverless connector to connect over private endpoint. Now we tried to configure and seems like the cloud function is not able to connect to network across projects with the connector. We are also not using shared VPC in the organization. Our team desperately need a solution on this. We tried using peering option , since GKE is actually peered to the project VPC, so because of transitivity we are suspecting cloud function could not able to connect to GKE
Below are the Logs when we ran Cloud function after peering and VPC connector setup

 

[
  {
    "textPayload": "\tdetails = \"Required \"container.clusters.get\" permission(s) for \"projects/omn-closely-novel-monster/locations/australia-southeast1/clusters/omneo-sandbox\".\"",
    "insertId": "635b740c000c0bc6390a1e58",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "configuration_name": "kube-workload-downscaling-5",
        "service_name": "kube-workload-downscaling-5",
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "project_id": "omn-basically-discrete-monarch",
        "location": "australia-southeast1"
      }
    },
    "timestamp": "2022-10-28T06:17:48.789446Z",
    "labels": {
      "goog-managed-by": "cloudfunctions",
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-10-28T06:17:48.792185980Z"
  },
  {
    "textPayload": "\tdebug_error_string = \"UNKNOWN:Error received from peer ipv4:142.250.204.10:443 {created_time:\"2022-10-28T06:17:48.788321459+00:00\", grpc_status:7, grpc_message:\"Required \\\"container.clusters.get\\\" permission(s) for \\\"projects/omn-closely-novel-monster/locations/australia-southeast1/clusters/omneo-sandbox\\\".\"}\"",
    "insertId": "635b740c000c0bce9114072d",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "location": "australia-southeast1",
        "configuration_name": "kube-workload-downscaling-5",
        "service_name": "kube-workload-downscaling-5",
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "project_id": "omn-basically-discrete-monarch"
      }
    },
    "timestamp": "2022-10-28T06:17:48.789454Z",
    "labels": {
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588",
      "goog-managed-by": "cloudfunctions"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-10-28T06:17:49.125916167Z"
  },
  {
    "textPayload": ">",
    "insertId": "635b740c000c0bd3317a12a9",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "location": "australia-southeast1",
        "project_id": "omn-basically-discrete-monarch",
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "service_name": "kube-workload-downscaling-5",
        "configuration_name": "kube-workload-downscaling-5"
      }
    },
    "timestamp": "2022-10-28T06:17:48.789459Z",
    "labels": {
      "goog-managed-by": "cloudfunctions",
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-10-28T06:17:49.125916167Z"
  },
  {
    "textPayload": "The above exception was the direct cause of the following exception:",
    "insertId": "635b740c000c0bde293e6d75",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "location": "australia-southeast1",
        "project_id": "omn-basically-discrete-monarch",
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "service_name": "kube-workload-downscaling-5",
        "configuration_name": "kube-workload-downscaling-5"
      }
    },
    "timestamp": "2022-10-28T06:17:48.789470Z",
    "labels": {
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588",
      "goog-managed-by": "cloudfunctions"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-10-28T06:17:49.125916167Z"
  },
  {
    "textPayload": "Traceback (most recent call last):\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py\", line 2525, in wsgi_app\n    response = self.full_dispatch_request()\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py\", line 1822, in full_dispatch_request\n    rv = self.handle_user_exception(e)\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py\", line 1820, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py\", line 1796, in dispatch_request\n    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/__init__.py\", line 100, in view_func\n    return function(request._get_current_object())\n  File \"/workspace/main.py\", line 16, in scaledown_kubernetes_workload\n    list_2 = suspend_cron_job(row[0],row[1],row[2],row[3])\n  File \"/workspace/main.py\", line 37, in suspend_cron_job\n    client = configure_authentication(projectid,zonename,clusterid,nodepool)\n  File \"/workspace/main.py\", line 100, in configure_authentication\n    cluster = cluster_manager.get_cluster(name=name)\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/container_v1/services/cluster_manager/client.py\", line 671, in get_cluster\n    response = rpc(\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py\", line 154, in __call__\n    return wrapped_func(*args, **kwargs)\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py\", line 283, in retry_wrapped_func\n    return retry_target(\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py\", line 190, in retry_target\n    return target()\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/grpc_helpers.py\", line 74, in error_remapped_callable\n    raise exceptions.from_grpc_error(exc) from exc\ngoogle.api_core.exceptions.PermissionDenied: 403 Required \"container.clusters.get\" permission(s) for \"projects/omn-closely-novel-monster/locations/australia-southeast1/clusters/omneo-sandbox\".",
    "insertId": "635b740c000c0bf1f85e79c3",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "location": "australia-southeast1",
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "service_name": "kube-workload-downscaling-5",
        "project_id": "omn-basically-discrete-monarch",
        "configuration_name": "kube-workload-downscaling-5"
      }
    },
    "timestamp": "2022-10-28T06:17:48.789489Z",
    "severity": "ERROR",
    "labels": {
      "goog-managed-by": "cloudfunctions",
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-10-28T06:17:49.125916167Z"
  },
  {
    "insertId": "635b740c000c1097715b5b78",
    "httpRequest": {
      "requestMethod": "POST",
      "requestUrl": "https://kube-workload-downscaling-5-yntiha3jka-ts.a.run.app/",
      "requestSize": "1145",
      "status": 500,
      "responseSize": "859",
      "userAgent": "curl/7.74.0",
      "remoteIp": "34.87.167.23",
      "serverIp": "216.239.34.53",
      "latency": "0.391999942s",
      "protocol": "HTTP/1.1"
    },
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "configuration_name": "kube-workload-downscaling-5",
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "service_name": "kube-workload-downscaling-5",
        "location": "australia-southeast1",
        "project_id": "omn-basically-discrete-monarch"
      }
    },
    "timestamp": "2022-10-28T06:17:48.790679Z",
    "severity": "ERROR",
    "labels": {
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588",
      "goog-managed-by": "cloudfunctions"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Frequests",
    "trace": "projects/omn-basically-discrete-monarch/traces/d5ae92077a46d5add0eedaa97fbe3d56",
    "receiveTimestamp": "2022-10-28T06:17:48.794318918Z",
    "spanId": "6804815165236499073",
    "traceSampled": true
  },
  {
    "textPayload": "[2022-10-28 06:24:36,192] ERROR in app: Exception on / [POST]",
    "insertId": "635b75a40002f0c5401ba031",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "location": "australia-southeast1",
        "project_id": "omn-basically-discrete-monarch",
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "service_name": "kube-workload-downscaling-5",
        "configuration_name": "kube-workload-downscaling-5"
      }
    },
    "timestamp": "2022-10-28T06:24:36.192709Z",
    "labels": {
      "goog-managed-by": "cloudfunctions",
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-10-28T06:24:36.196307657Z"
  },
  {
    "textPayload": "Traceback (most recent call last):\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/grpc_helpers.py\", line 72, in error_remapped_callable\n    return callable_(*args, **kwargs)\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/grpc/_channel.py\", line 946, in __call__\n    return _end_unary_response_blocking(state, call, False, None)\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/grpc/_channel.py\", line 849, in _end_unary_response_blocking\n    raise _InactiveRpcError(state)\ngrpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:",
    "insertId": "635b75a40002f0e9743d5cb1",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "service_name": "kube-workload-downscaling-5",
        "location": "australia-southeast1",
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "configuration_name": "kube-workload-downscaling-5",
        "project_id": "omn-basically-discrete-monarch"
      }
    },
    "timestamp": "2022-10-28T06:24:36.192745Z",
    "severity": "ERROR",
    "labels": {
      "goog-managed-by": "cloudfunctions",
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-10-28T06:24:36.196307657Z"
  },
  {
    "textPayload": "\tstatus = StatusCode.PERMISSION_DENIED",
    "insertId": "635b75a40002f0f1ac33a841",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "service_name": "kube-workload-downscaling-5",
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "configuration_name": "kube-workload-downscaling-5",
        "location": "australia-southeast1",
        "project_id": "omn-basically-discrete-monarch"
      }
    },
    "timestamp": "2022-10-28T06:24:36.192753Z",
    "labels": {
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588",
      "goog-managed-by": "cloudfunctions"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-10-28T06:24:36.196307657Z"
  },
  {
    "textPayload": "\tdetails = \"Required \"container.clusters.get\" permission(s) for \"projects/omn-closely-novel-monster/locations/australia-southeast1/clusters/omneo-sandbox\".\"",
    "insertId": "635b75a40002f0faf27bc317",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "configuration_name": "kube-workload-downscaling-5",
        "service_name": "kube-workload-downscaling-5",
        "location": "australia-southeast1",
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "project_id": "omn-basically-discrete-monarch"
      }
    },
    "timestamp": "2022-10-28T06:24:36.192762Z",
    "labels": {
      "goog-managed-by": "cloudfunctions",
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-10-28T06:24:36.196307657Z"
  },
  {
    "textPayload": "\tdebug_error_string = \"UNKNOWN:Error received from peer ipv4:172.217.167.106:443 {created_time:\"2022-10-28T06:24:36.191598764+00:00\", grpc_status:7, grpc_message:\"Required \\\"container.clusters.get\\\" permission(s) for \\\"projects/omn-closely-novel-monster/locations/australia-southeast1/clusters/omneo-sandbox\\\".\"}\"",
    "insertId": "635b75a40002f103c87599b0",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "project_id": "omn-basically-discrete-monarch",
        "service_name": "kube-workload-downscaling-5",
        "configuration_name": "kube-workload-downscaling-5",
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "location": "australia-southeast1"
      }
    },
    "timestamp": "2022-10-28T06:24:36.192771Z",
    "labels": {
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588",
      "goog-managed-by": "cloudfunctions"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-10-28T06:24:36.196307657Z"
  },
  {
    "textPayload": ">",
    "insertId": "635b75a40002f109e94bdfba",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "project_id": "omn-basically-discrete-monarch",
        "service_name": "kube-workload-downscaling-5",
        "location": "australia-southeast1",
        "configuration_name": "kube-workload-downscaling-5"
      }
    },
    "timestamp": "2022-10-28T06:24:36.192777Z",
    "labels": {
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588",
      "goog-managed-by": "cloudfunctions"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-10-28T06:24:36.196307657Z"
  },
  {
    "textPayload": "The above exception was the direct cause of the following exception:",
    "insertId": "635b75a40002f1138e2cfbbb",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "service_name": "kube-workload-downscaling-5",
        "configuration_name": "kube-workload-downscaling-5",
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "project_id": "omn-basically-discrete-monarch",
        "location": "australia-southeast1"
      }
    },
    "timestamp": "2022-10-28T06:24:36.192787Z",
    "labels": {
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588",
      "goog-managed-by": "cloudfunctions"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-10-28T06:24:36.196307657Z"
  },
  {
    "textPayload": "Traceback (most recent call last):\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py\", line 2525, in wsgi_app\n    response = self.full_dispatch_request()\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py\", line 1822, in full_dispatch_request\n    rv = self.handle_user_exception(e)\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py\", line 1820, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py\", line 1796, in dispatch_request\n    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/__init__.py\", line 100, in view_func\n    return function(request._get_current_object())\n  File \"/workspace/main.py\", line 16, in scaledown_kubernetes_workload\n    list_2 = suspend_cron_job(row[0],row[1],row[2],row[3])\n  File \"/workspace/main.py\", line 37, in suspend_cron_job\n    client = configure_authentication(projectid,zonename,clusterid,nodepool)\n  File \"/workspace/main.py\", line 100, in configure_authentication\n    cluster = cluster_manager.get_cluster(name=name)\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/container_v1/services/cluster_manager/client.py\", line 671, in get_cluster\n    response = rpc(\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py\", line 154, in __call__\n    return wrapped_func(*args, **kwargs)\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py\", line 283, in retry_wrapped_func\n    return retry_target(\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py\", line 190, in retry_target\n    return target()\n  File \"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/grpc_helpers.py\", line 74, in error_remapped_callable\n    raise exceptions.from_grpc_error(exc) from exc\ngoogle.api_core.exceptions.PermissionDenied: 403 Required \"container.clusters.get\" permission(s) for \"projects/omn-closely-novel-monster/locations/australia-southeast1/clusters/omneo-sandbox\".",
    "insertId": "635b75a40002f126593502de",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "configuration_name": "kube-workload-downscaling-5",
        "service_name": "kube-workload-downscaling-5",
        "revision_name": "kube-workload-downscaling-5-00005-deq",
        "location": "australia-southeast1",
        "project_id": "omn-basically-discrete-monarch"
      }
    },
    "timestamp": "2022-10-28T06:24:36.192806Z",
    "severity": "ERROR",
    "labels": {
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588",
      "goog-managed-by": "cloudfunctions"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2022-10-28T06:24:36.196307657Z"
  },
  {
    "insertId": "635b75a40002f60b3c9a679d",
    "httpRequest": {
      "requestMethod": "POST",
      "requestUrl": "https://kube-workload-downscaling-5-yntiha3jka-ts.a.run.app/",
      "requestSize": "1144",
      "status": 500,
      "responseSize": "859",
      "userAgent": "curl/7.74.0",
      "remoteIp": "34.87.167.23",
      "serverIp": "216.239.34.53",
      "latency": "0.291148723s",
      "protocol": "HTTP/1.1"
    },
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "location": "australia-southeast1",
        "configuration_name": "kube-workload-downscaling-5",
        "project_id": "omn-basically-discrete-monarch",
        "service_name": "kube-workload-downscaling-5",
        "revision_name": "kube-workload-downscaling-5-00005-deq"
      }
    },
    "timestamp": "2022-10-28T06:24:36.194059Z",
    "severity": "ERROR",
    "labels": {
      "instanceId": "001c9ea28b8bc2ab08bf670c73ac58eac53540910fb67926396ba4443e351fc16d99953e6a2feb610bf6d9a8aa0076d39f872091d47b51c65f98e1260f0d309588",
      "goog-managed-by": "cloudfunctions"
    },
    "logName": "projects/omn-basically-discrete-monarch/logs/run.googleapis.com%2Frequests",
    "trace": "projects/omn-basically-discrete-monarch/traces/dc2c59fc21cad6748144351f8ec19be2",
    "receiveTimestamp": "2022-10-28T06:24:36.196576815Z",
    "spanId": "10949821666663960749",
    "traceSampled": true
  }
]

 



 

0 1 1,446
1 REPLY 1

Since your cluster’s control plane does no longer have a public endpoint, only the VPC where your nodes are located, and any authorized networks in the VPC will be able to access the control plane. This is due to the control plane for your cluster being located in a Google-owned project, and reachable through VPC peering. 

Peering the VPC network on your second project to your node VPC will result in transitive peering, which is not supported. It’s also not possible to manually peer your second project’s VPC to the Google-owned project hosting the control plane.

A workaround is detailed in this guide and it involves using a proxy service to route access to the control plane, so you can access the API server from an on-premises location or a different VPC.