Unable to apply cloudrun to a private Autopilot cluster in a fleet

[zaphod72]

I have a private cluster with a bastion host. Started the host with:

gcloud compute ssh demo-cluster-bastion --tunnel-through-iap --zone us-central1-a -- -fqN -L8888:localhost:8888

I set the proxy in kubectl config:

kubectl config set clusters.gke_darren-dev-17e3_us-central1_demo-cluster.proxy-url http://localhost:8888

And `kubectl get namespaces` test works (without setting a HTTPS_PROXY env var).

However I can not run `gcloud container fleet cloudrun apply --gke-cluster=us-central1/demo-cluster`

> gcloud container fleet cloudrun apply --gke-cluster=us-central1/demo-cluster --verbosity=debug
DEBUG: Running [gcloud.container.fleet.cloudrun.apply] with arguments: [--gke-cluster: "us-central1/demo-cluster", --verbosity: "debug"]
DEBUG: Starting new HTTPS connection (1): container.googleapis.com:443
DEBUG: https://container.googleapis.com:443 "GET /v1/projects/darren-dev-17e3/locations/us-central1/clusters/demo-cluster?alt=json HTTP/1.1" 200 None
DEBUG: Saved kubeconfig to /var/folders/y7/j26tsq3j26n35h88f7syf9pr0000gn/T/tmpho6gu1nz/kubeconfig
kubeconfig entry generated for demo-cluster.
DEBUG: Executing command: ['/Users/darrenk/google-cloud-sdk/bin/kubectl', '--kubeconfig', '/var/folders/y7/j26tsq3j26n35h88f7syf9pr0000gn/T/tmpho6gu1nz/kubeconfig', '--request-timeout', '20s', 'apply', '-f', '-']
DEBUG: (gcloud.container.fleet.cloudrun.apply) Failed to apply manifest to cluster: error: error validating "STDIN": error validating data: failed to download openapi: Get "https://10.10.10.2/openapi/v2?timeout=20s": net/http: TLS handshake timeout; if you choose to ignore these errors, turn validation off with --validate=false
Traceback (most recent call last):
  File "/Users/darrenk/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 998, in Execute
    resources = calliope_command.Run(cli=self, args=args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/darrenk/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 838, in Run
    resources = command_instance.Run(args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/darrenk/google-cloud-sdk/lib/surface/container/fleet/cloudrun/apply.py", line 76, in Run
    _apply_cr_to_membership_cluster(kube_client, yaml_string)
  File "/Users/darrenk/google-cloud-sdk/lib/surface/container/fleet/cloudrun/apply.py", line 90, in _apply_cr_to_membership_cluster
    raise exceptions.Error(
googlecloudsdk.core.exceptions.Error: Failed to apply manifest to cluster: error: error validating "STDIN": error validating data: failed to download openapi: Get "https://10.10.10.2/openapi/v2?timeout=20s": net/http: TLS handshake timeout; if you choose to ignore these errors, turn validation off with --validate=false

ERROR: (gcloud.container.fleet.cloudrun.apply) Failed to apply manifest to cluster: error: error validating "STDIN": error validating data: failed to download openapi: Get "https://10.10.10.2/openapi/v2?timeout=20s": net/http: TLS handshake timeout; if you choose to ignore these errors, turn validation off with --validate=false

also tried with `HTTPS_PROXY`:

> HTTPS_PROXY=http://localhost:8888 gcloud container fleet cloudrun apply --gke-cluster=us-central1/demo-cluster --verbosity=debug
DEBUG: Running [gcloud.container.fleet.cloudrun.apply] with arguments: [--gke-cluster: "us-central1/demo-cluster", --verbosity: "debug"]
DEBUG: Starting new HTTPS connection (1): container.googleapis.com:443
DEBUG: Caught socket error, retrying: HTTPSConnectionPool(host='container.googleapis.com', port=443): Max retries exceeded with url: /v1/projects/darren-dev-17e3/locations/us-central1/clusters/demo-cluster?alt=json (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1086fec90>: Failed to establish a new connection: [Errno 61] Connection refused')))
DEBUG: Retrying request to url https://container.googleapis.com/v1/projects/darren-dev-17e3/locations/us-central1/clusters/demo-cluster?alt=json after exception HTTPSConnectionPool(host='container.googleapis.com', port=443): Max retries exceeded with url: /v1/projects/darren-dev-17e3/locations/us-central1/clusters/demo-cluster?alt=json (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1086fec90>: Failed to establish a new connection: [Errno 61] Connection refused')))

[KikoZam]

Hi @zaphod72,

The error message indicates that the gcloud is trying to establish a secure connection (TLS handshake) with the Google Kubernetes Engine (GKE) API server, possibly due to connectivity issues between your bastion host, the private cluster, and Cloud Run's managed APIs. Here are some possible troubleshooting that may help:

• Ensure that you have enabled and installed Knative serving in your fleet.  For other details and additional options, you can check the gcloud container fleet cloudrun enable for your reference.
• I found some information that can be helpful, as it addresses a similar issue. Although the post is a year older, it could still provide a solution for your case: Cloud Run for Anthos: You don't have any clusters with Cloud Run for Anthos enabled.

If you've tried the above steps and the issue persists, it might be an edge case or a bug. In this case, contacting Google Cloud support with detailed logs and your configuration might be necessary.

I hope the above information is helpful.

[zaphod72 · Accepted solution]

The `kubectl` command run by gcloud does not get the proxy-url from my kube config.

- cluster:
    certificate-authority-data: ...
    proxy-url: http://localhost:8888
    server: https://10.10.10.3
  name: gke...

But specifying it as an env var works.

HTTPS_PROXY=http://localhost:8888 gcloud container fleet cloudrun apply --gke-cluster=us-central1/asm-cluster