ssh to Google Compute Engine instance no longer works

[Tord]

We have a Compute Engine VM to which we have always connected using the web SSH window accessible from the Cloud Console. This has always worked perfectly, and we successfully used it a few days ago. As of today, it suddenly no longer works, even though nothing has changed on our side. Pushing the SSH button in the Cloud Console gives a window with this message:

"Connection via Cloud Identity-Aware Proxy Failed. Code: 4010 Reason: destination read failed".

The VM itself appears to be alive and healthy; the web servers running on it respond as usual.

I also tried installing gcloud locally on my Mac (I never needed it before, as the web SSH console satisfied all our needs) and tried to set up ssh access to our VM there. I tried both the approaches described in the "OS login" and the "metadata-based SSH keys" sections of the documentation, with no success. I get a "Permission denied (publickey)" error, and generating a new key and adding it to the metadata section of the cloud console does not help.

I've spent almost the whole day browsing the documentation, reading Stack Overflow posts, and trying everything, and nothing seems to work. 

Any ideas for how to solve this issue?

[DanGCP]

Hello Tord,

It seems to me you describe an issue with an instance or possibly the network, as well as an ssh issue.

The 4010 error seems uncommon.  Perhaps restarting the instance would help.

Also, checking if there have been any recent changes to the Identity-Aware-Proxy configuration, the firewall or the routing.

Have you tried connecting using the Cloud Shell?  This would be the terminal that is on the webpage (top-right) of the GCP Console, not the ssh button. This terminal is directly connected to your project, making troubleshooting simpler in this case. You could try from the Cloud Shell, logged in with a user who has the compute instance admin role.  From that point, you could try revoking and re-creating the ssh keys, if you are using metadata keys to access the instance:

1) gcloud auth revoke --all

2) gcloud auth login [IAM-USER] 

3)  gcloud compute ssh ...

Dan

[Pixelizedch]

Hello, I've the same problem, it seems that the SSH interface has change, because there is a different graphic and since then it give me the same error. But I did nothing since last week when I connected without any problem.

[vs1]

We're also seeing this issue but the one account we have seems to be able to SSH without any issues. That account is also where the compute engine was created so that could be why. The account that has issues we see a Permissions denied (publickey) error when trying to SSH via cloud console. 

Both accounts have owner level access and the compute admin role (which gives full control of compute engine resources).

Is there anything permission wise we should be looking into?  

[Pixelizedch]

After losing days trying to find a solution (without any result) since yesterday I can access again, I just get some errors and the login is slower than before, but at least it works.

[DanGCP]

Some ideas for troubleshooting ssh on GCP:

1) If you can connect to your network via ssh, use "gcloud compute ssh" function
2) Verify ssh access is open via firewall
3) Verify the keys in "Compute Engine Metadata" using the Console, or else that they are in the project with the following command:

gcloud compute project-info describe --format flattened \
| grep commonInstanceMetadata.items | grep ssh | grep expireOn

4) The procedure I wrote initially can help when you have ssh keys that have expired.

5) Contact support and mention from where you are trying to contact your instance, if you are using "gcloud compute ssh" or just "ssh", and what error message you have. If you are using ssh, add the output from your command with the "-vvv" option for more verbosity.

" Permissions denied (publickey)" is an error from the ssh service itself.  This would be a good indication there is a key mismatch or the key has expired. In either case, try to create a new key.

Regards,
Dan

[kumards]

You could also use the SSH troubleshooting tool, which was released recently: https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-ssh#ssh_troubleshooting_tool

[saumya003]

https://www.linkedin.com/pulse/access-gcp-vms-via-serial-port-saumya-saumya/ 

please check this. This method always works for Ubuntu/linux VMs.

[jofre]

Related: https://issuetracker.google.com/228091306

[Pixelizedch]

This problem happened again... why Google just randomly disable SSH access to users? Even in that case I didn't edit. I used SSH 10 days ago without problem, I had to enter yesterday and it gave the error again because it's using the different interface... any ideas on how to avoid that? Or I just have to move everything away of this cloud?

Error: sometime "1006", sometime just "Connection failed" without number.

If I do "troubleshoot" it check VM, network and user. All of three are ok and "I have no problem", but still cannot access.

[DanGCP]

Hello Pixelizedch,
If you are having issues with error 1006, that is likely related to IAP and not to the original post, which is about error 4010.  Error 1006 is a more common error and can be debugged with additional context (such as from where you are connect to which resource), and error messages form the logs if possible.  I propose to focus this thread on error 4010.

[Fred-Niyi]

I am having similar issue and tried to connect via console port but getting the following error. 

It seems to be an issue with the network or firewall, I don't really understand. 

Strange thing though, I have another vm in the same us-east1-b zone, same network tag, which i can access via ssh no problem. 

I enabled OsLogin project wide and tunnel through iap and have a firewall rule ingress allow port 22 from 35.235.240.0/20.

The vm i cannot access is running a lamp stack, the other vm nothing.