Problem - gcloud compute ssh, then "Client closed connection from [stdin]" in logon

[jimo]

Hello community.

I am new here, looking for help. I've been working with GCP SDK for ~year or so, and now all of sudden it stopped working for me in the sense that when I want to connect to target VM with gcloud compute ssh --tunnel-through-iap (...), it is executing the packaged putty as it should, and then it's just waiting few seconds with blank session and then it ends with error message

"Remote side unexpectedly closed network connection."

I have tried the connection with  --verbosity=debug, then I get more logs both in the console and in the putty session, but nothing really apparent to me.

I've tried even to uninstall the GCP SDK and install it from scratch, it didn't help.

When I try to authenticate and connect from a colleague's PC, it works fine.

I will post logs below this message, as they are longer.

Any help is highly appreciated, thank you in advance.

- jimo

gcloud command:

DEBUG: Running [gcloud.compute.ssh] with arguments: [--project: "(...)", --tunnel-through-iap: "True", --verbosity: "debug", --zone: "(...)", [USER@]INSTANCE: "(...)"]
DEBUG: Making request: POST https://oauth2.googleapis.com/token
DEBUG: Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG: https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
DEBUG: Starting new HTTPS connection (1): compute.googleapis.com:443
DEBUG: https://compute.googleapis.com:443 "POST /batch/compute/v1 HTTP/1.1" 200 None
DEBUG: Starting new HTTPS connection (1): compute.googleapis.com:443
DEBUG: https://compute.googleapis.com:443 "POST /batch/compute/v1 HTTP/1.1" 200 None
DEBUG: Starting new HTTPS connection (1): compute.googleapis.com:443
DEBUG: https://compute.googleapis.com:443 "POST /batch/compute/v1 HTTP/1.1" 200 None
DEBUG: Starting new HTTPS connection (1): oslogin.googleapis.com:443
DEBUG: https://oslogin.googleapis.com:443 "GET /v1/users/(...)/loginProfile?alt=json&projectId=(...) HTTP/1.1" 200 None
INFO: Using OS Login user [(...)] instead of default user [(...)]
DEBUG: OS Login Enabled: True
Security Keys Enabled: None
Username: (...)
SSH Security Key Support: None
Environment: putty
Security Keys:

Existing host keys found in C:\Users\(...)\.ssh\google_compute_known_hosts

putty:
2023-02-28 19:10:07 Leaving host lookup to proxy of "(...)" (for SSH connection)
2023-02-28 19:10:07 Starting local proxy command: "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin\..\platform\bundledpython\python.exe" "-S" "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\gcloud.py" compute start-iap-tunnel (...) 22 --listen-on-stdin --project=(...) --zone=(...) --verbosity=debug
2023-02-28 19:10:07 We claim version: SSH-2.0-PuTTY_Release_0.76
2023-02-28 19:10:16 proxy: DEBUG: Metrics collector initialized...
2023-02-28 19:10:16 proxy: DEBUG: Loaded Command Group: ['gcloud']
2023-02-28 19:10:16 proxy: DEBUG: Loaded Command Group: ['gcloud', 'container']
2023-02-28 19:10:16 proxy: DEBUG: Loaded Command Group: ['gcloud', 'compute']
2023-02-28 19:10:16 proxy: DEBUG: Loaded Command Group: ['gcloud', 'compute', 'start_iap_tunnel']
2023-02-28 19:10:16 proxy: DEBUG: Running [gcloud.compute.start-iap-tunnel] with arguments: [(...)]
2023-02-28 19:10:16 proxy: DEBUG: Starting new HTTPS connection (1): compute.googleapis.com:443
2023-02-28 19:10:17 proxy: DEBUG: https://compute.googleapis.com:443 "POST /batch/compute/v1 HTTP/1.1" 200 None
2023-02-28 19:10:17 proxy: WARNING:
2023-02-28 19:10:17 proxy:
2023-02-28 19:10:17 proxy: To increase the performance of the tunnel, consider installing NumPy. For instructions,
2023-02-28 19:10:17 proxy: please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidth
2023-02-28 19:10:17 proxy:
2023-02-28 19:10:17 proxy: DEBUG: credentials type for _GetAccessTokenCallback is [<googlecloudsdk.core.credentials.google_auth_credentials.Credentials object at 0x000001DC18DDC430>].
2023-02-28 19:10:17 proxy: DEBUG: Using new websocket library
2023-02-28 19:10:17 proxy: INFO: Connecting with URL ['wss://tunnel.cloudproxy.app/v4/connect?project=(...)&port=22&newWebsocket=True&zone=(...)&instance=(...)&interface=nic0']
2023-02-28 19:10:17 proxy: DEBUG: Connecting proxy...
2023-02-28 19:10:17 proxy: DEBUG: RECV opcode [2] data_len [337] binary_data[:20] [b'\x00\x01\x00\x00\x01KABFEQCzpZOFzGe']
2023-02-28 19:10:17 proxy: DEBUG: RECV opcode [2] data_len [27] binary_data[:20] [b'\x00\x04\x00\x00\x00\x15SSH-2.0-OpenSS']
2023-02-28 19:10:17 Remote version: SSH-2.0-OpenSSH_8.0
2023-02-28 19:10:17 Using SSH protocol version 2
2023-02-28 19:10:17 No GSSAPI security context available
2023-02-28 19:10:17 proxy: DEBUG: ENQUEUED data_len [28] bytes_to_send[:20] [b'SSH-2.0-PuTTY_Releas']
2023-02-28 19:10:17 proxy: INFO: Client closed connection from [stdin].
2023-02-28 19:10:17 proxy: DEBUG: CLOSE
2023-02-28 19:10:17 proxy: DEBUG: SEND data_len [34] send_data[:20] b'\x00\x04\x00\x00\x00\x1cSSH-2.0-PuTTY_'
2023-02-28 19:10:17 proxy: DEBUG: SEND data_len [10] send_data[:20] b'\x00\x07\x00\x00\x00\x00\x00\x00\x00\x15'
2023-02-28 19:10:17 proxy: INFO: Unable to send WebSocket ack [EOF occurred in violation of protocol (_ssl.c:2483)]
2023-02-28 19:10:17 proxy: DEBUG: [Errno 5] stdin ReadFile failed
2023-02-28 19:10:17 proxy: Traceback (most recent call last):
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\calliope\cli.py", line 987, in Execute
2023-02-28 19:10:17 proxy: resources = calliope_command.Run(cli=self, args=args)
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\calliope\backend.py", line 807, in Run
2023-02-28 19:10:17 proxy: resources = command_instance.Run(args)
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\surface\compute\start_iap_tunnel.py", line 144, in Run
2023-02-28 19:10:17 proxy: iap_tunnel_helper.Run()
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\compute\iap_tunnel.py", line 825, in Run
2023-02-28 19:10:17 proxy: self._RunReceiveLocalData(_StdinSocket(), 'stdin', user_agent)
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\compute\iap_tunnel.py", line 657, in _RunReceiveLocalData
2023-02-28 19:10:17 proxy: data = conn.recv(utils.SUBPROTOCOL_MAX_DATA_FRAME_SIZE)
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\compute\iap_tunnel.py", line 422, in recv
2023-02-28 19:10:17 proxy: return self._RecvWindows(bufsize)
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\compute\iap_tunnel.py", line 496, in _RecvWindows
2023-02-28 19:10:17 proxy: six.reraise(msg_data[0], msg_data[1], msg_data[2])
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\third_party\six\__init__.py", line 719, in reraise
2023-02-28 19:10:17 proxy: raise value
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\compute\iap_tunnel.py", line 459, in _ReadFromStdinAndEnqueueMessageWindows
2023-02-28 19:10:17 proxy: raise socket.error(errno.EIO, 'stdin ReadFile failed')
2023-02-28 19:10:17 proxy: OSError: [Errno 5] stdin ReadFile failed
2023-02-28 19:10:17 proxy: INFO: Error while receiving from WebSocket.
2023-02-28 19:10:17 proxy: Traceback (most recent call last):
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 237, in run_forever
2023-02-28 19:10:17 proxy: frame = self.recv()
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 149, in recv
2023-02-28 19:10:17 proxy: return self.frame_buffer.recv_frame()
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 102, in recv_frame
2023-02-28 19:10:17 proxy: (fin, rsv1, rsv2, rsv3, opcode, has_mask, length_bits) = self._recv_header()
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 66, in _recv_header
2023-02-28 19:10:17 proxy: header = self.recv(2)
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 281, in _recv_bytes
2023-02-28 19:10:17 proxy: self._throw_or_wait_for_retry(attempt=attempt, exception=e)
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 331, in _throw_or_wait_for_retry
2023-02-28 19:10:17 proxy: raise exception
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 275, in _recv_bytes
2023-02-28 19:10:17 proxy: self.close()
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 189, in close
2023-02-28 19:10:17 proxy: self.send_close(close_code, close_message)
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 210, in send_close
2023-02-28 19:10:17 proxy: self.send(
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 183, in send
2023-02-28 19:10:17 proxy: self._throw_or_wait_for_retry(attempt=attempt, exception=e)
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 331, in _throw_or_wait_for_retry
2023-02-28 19:10:17 proxy: raise exception
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 172, in send
2023-02-28 19:10:17 proxy: bytes_sent = self.sock.send(frame_data)
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\bundledpython\lib\ssl.py", line 1173, in send
2023-02-28 19:10:17 proxy: return self._sslobj.write(data)
2023-02-28 19:10:17 proxy: ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2483)
2023-02-28 19:10:17 proxy:
2023-02-28 19:10:17 proxy: During handling of the above exception, another exception occurred:
2023-02-28 19:10:17 proxy:
2023-02-28 19:10:17 proxy: Traceback (most recent call last):
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_websocket_helper.py", line 246, in _ReceiveFromWebSocket
2023-02-28 19:10:17 proxy: self._websocket.run_forever(
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 249, in run_forever
2023-02-28 19:10:17 proxy: self.close()
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 189, in close
2023-02-28 19:10:17 proxy: self.send_close(close_code, close_message)
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 210, in send_close
2023-02-28 19:10:17 proxy: self.send(
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 183, in send
2023-02-28 19:10:17 proxy: self._throw_or_wait_for_retry(attempt=attempt, exception=e)
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 331, in _throw_or_wait_for_retry
2023-02-28 19:10:17 proxy: raise exception
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\compute\iap_tunnel_lightweight_websocket.py", line 172, in send
2023-02-28 19:10:17 proxy: bytes_sent = self.sock.send(frame_data)
2023-02-28 19:10:17 proxy: File "C:\Users\(...)\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\bundledpython\lib\ssl.py", line 1173, in send
2023-02-28 19:10:17 proxy: return self._sslobj.write(data)
2023-02-28 19:10:17 proxy: ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2483)
2023-02-28 19:10:17 proxy: ERROR: gcloud crashed (OSError): [Errno 5] stdin ReadFile failed
2023-02-28 19:10:17 proxy:
2023-02-28 19:10:17 proxy: If you would like to report this issue, please run the following command:
2023-02-28 19:10:17 proxy: gcloud feedback
2023-02-28 19:10:17 proxy:
2023-02-28 19:10:17 proxy: To check gcloud for common problems, please run the following command:
2023-02-28 19:10:17 proxy: gcloud info --run-diagnostics
2023-02-28 19:10:17 proxy: DEBUG: Metrics reporting process started...
2023-02-28 19:10:17 Remote side unexpectedly closed network connection

[dionv]

Hello jimo,

Checking on similar concerns, I saw a post in StackOverflow by Sinto providing these steps "

To create and save a new keep-alive connection, follow these steps:

• Open the PuTTY application, and go to the Options panel (labeled "Category") on the left of the window.
• Select (click) the "Connection" item.
• In the ​​"Sending of null packets to keep the session active" area on the right, change the default value of "Seconds between keepalives" from 0 (turn off) to 1800 (30 minutes).
• Select the "Enable TCP keepalives (SO_KEEPALIVE option)" check box. Note: This option may not be available in older versions of the PuTTY client.
• On the topmost left side of the Options panel, select (click) "Session".
• In the "Host Name (or IP Address)" field, enter the destination host name or IP address (e.g., "destination.ipaddress.here.com" or "192.168.1.1").
• In the "Saved Sessions" text-entry box, provide a name for the session (e.g., "savedsession").
• Select "Save".

To use the modified session settings, select it from the "Saved sessions" list, then click the buttons marked "Load" and "Open".

If your connected sessions still time out, enter a lower number of seconds into the "Seconds between keepalives" value.

On searching more I also got a point that:

"Uncheck 'Attempt GSSAPI authentication (SSH-2 only)' in Putty: Category - Connection - SSH - Auth - GSSAPI

In my case it seems to be GSSAPI is incompatible to a Ubuntu host that uses Beyond trust (formerly: likewise open)."

"

[jimo]

Hello dionv, 

thank you for trying. The problem here looks more like something with proxy, which is not fully accepted by gcloud, even though when it's set with 

gcloud config set proxy/type http

gcloud config set proxy/address XXXXXXXXX

gcloud config set proxy/port XXXXXX

I am still not able to use the GCP from the standard GCP SDK. I got a workaround for now to execute gcloud commands from "cmder" tool. All my saved putty sessions are not working, had to re-create shell aliases for all machines in cmder, but at least I can access.

The case is under investigation here - https://issuetracker.google.com/issues/270909803 , they've accepted that as bug yesterday, so let's see if there will be some fix and when.