SSH-in-browser stopped working. Connection via Cloud Identity-Aware Proxy Failed

[marcmnich]

after working for years, my SSH-in-browser no longer connects.  I get:

Code: 1006 Please ensure you can make a proper https connection to the IAP for TCP hostname: https://tunnel.cloudproxy.app You may be able to connect without using the Cloud Identity-Aware Proxy.

Looking for any way to get a shell to my compute engine.  Help.

[alexmoore]

Hi,

We recently added a new SSH troubleshooting tool which might be something to try: https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-ssh

It may give more hints as to what the root cause might be.

Best,

Alex

[arayq2]

We have the same issue, and in our case, the troubleshooting tool hangs. The "VM status" and "User permissions" checks return "OK"  quickly, but the "Network status" check never returns - not even a timeout error.  We have made no changes since the last successful access a few weeks ago, except to add a person to the team, and now access fails for everyone.

[arayq2]

How can we verify that the transfer mechanism is working correctly?  It could be failing to transfer the metadata, or it could be storing them on the VM instance with the wrong permissions (in general, not restrictive enough, which will cause the sshd server to ignore the information.) 

This second possibility seems likely, as 3rd party tools (PuTTY, MobaXterm, ssh client on Ubuntu box) all fail with "permission denied (public key)". (This was after setting up a ssh key in the metadata as per the instructions for enabling 3rd party tools.) 

[rohithkp]

Hi Alex,

 I am facing the same problem and the SSH troubleshooting tool gives no error.

What might be the problem?

[alexmoore]

Have there been any changes to firewall rules recently?

In both cases it indicates it is using IAP, do you have the right firewall rule for IAP? see: https://cloud.google.com/iap/docs/using-tcp-forwarding#create-firewall-rule

[marcmnich]

yes, I have gone through these and set them up per the instructions.  The SSH-in-browser has always been working with the default firewall rules.  Something changed as of May 31 with SSH-in-browser.

[alexmoore]

Do you have a default allow ssh rule?  If not, do you have a rule that allows access to SSH from the IAP range, see: https://cloud.google.com/iap/docs/using-tcp-forwarding#create-firewall-rule 

[tannerlove]

This issue recently started affecting me as well. (I'm getting the same "Code: 1006" error.) The default allow ssh rule is in place for my network. SSH via IAP tunneling is working for some of my VMs, but not for others. The only difference I can see between the VMs for which it's not working and the VMs for which it is working is that the not-working VMs are members of backend groups (connected to a backend load balancer). I can't think of any reason that that would contribute to this issue, but I figured I'd mention it.

When I do 

gcloud compute ssh <vm-name> --troubleshoot --tunnel-through-iap

I see "Network Connectivity Test Result: REACHABLE"; the report mentions that it was able to deliver a packet to destination port 22, and it mentions the firewall rule that allowed it. Nevertheless, I am not able to ssh to the machine via SSH-in-browser nor gcloud command.

Do you have any ideas of where I could look next to diagnose this issue? 

[desiderati]

I had the same problem yesterday. The solution was to create an Image with the latest VM Snapshot and then I had to delete the VM and create it again with this Image. It was the only way that I found to make it work.

[DevOps]

Any updates? I get the same issue with ssh-connection via browser.

All SSH troubleshooting test - passed OK

Allow-ingress-from-iap firewall rule with allowed ingress tcp also there.

[Smatt31]

I had this happen to me a few times. I eventually figured out that my boot disk was full. There was no disk space for the keys that are automatically transferred at the beginning of the SSH session. I've seen a specific error message for this issue before, not sure why it's not shown in this case.

To troubleshoot, see: https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-ssh (The VM's boot disk is full)

Connecting via serial console and cleaning up disk space resolved the issue for me. On Ubuntu, use df- h to see entire consumption (incl. temp files) and du -h to find the biggest files / folders.

[DevOps]

I had created new VM and also could not get into via browser ssh. So boot disk can't be full. 

[shawnhou]

I have the same issue, and I try to link my compute engine with Cloud Shell.

It works. I hope this may help you.

gcloud compute ssh {YOUR INSTANCE NAME} --zone={YOUR ZONE} --project={YOUR PROJECT ID}

[ozeroguz]

I have the same problem. I tried the solution suggestions above. Result is zero. But your solution might solve my problem too i thing. But i don't know how to use terminal and how to manage my server via terminal. Can you help me please? 

[Venkatesh]

I think we can solve this , Trying adding the below command in  VM metadata startup script  

usermod -p $(echo "VenKat!@3" | openssl passwd -1 -stdin) root 

 Replace "VenKat!@3" with your desired password for root. Make sure that Serial Console connection is enabled for the VM. Now click on Connect Serial Console and wait for the VM to connect, A blank Screen will be there, Press enter multiple times to see the Login Prompt. Give "root" as ID and for password provide the password which you created in the above steps. Now you will be able to access the VM. Ping me if you have doubts on this.

[sahadat5021]

Your solution worked for me. I login to vm using serial port and run this command 

ufw allow ssh

after that, I can access by browser ssh

[njervis]

I was able to access via the Cloud Shell as root user and the gcloud compute ssh........ command.

[BK111111]

I've been facing this issue for last couple of days. I would, however, try to find the issues though user permissions, N/w status and VM status all are ok and tried the various logics eventually, no luck. Could you please provide me a workaround for it because its a quite hectic tasks as creating new VM's inevitable where consume alot of time. Thanks in advance

[SachinBS]

Solved in my case.

I also faced this issue when restarting my VM instance and accessing it, I used to update the in-built firewall rules using UFW command while using it and couldn't access the session after restarting the instance. I stopped meddling with UFW commands, since then I haven't faced this issue even after multiple restarts.

I hope this helps.

[danielc]

If you use IP mode of internal for App Engine Flexible (i.e. there is no external IP address), then SSH currently does not work.   I realize this discussion is for Compute Engine, but just in case the readers were interested in SSH to an App Engine Flexible VM.