SSH to private instance asks for (user@compute.instanceID) password

[cherriford]

I'm experiencing a strange issue where SSH'ing to a private GCE instance over Identity Aware Proxy responds with: (user@compute.12345678912345) Password: 

There is no known password for the instance, and nothing I input in this field works. What's especially odd about this issue is that, after multiple retries, it will randomly establish connection.

My SSH command is: gcloud compute ssh --tunnel-through-iap --zone "us-west1-a" --ssh-key-file=key user@instanceName 

Other notes:
- This is a Palo Alto Panorama Management Server that was created with the 

panorama-byol-1000 image.
- I am attaching the public SSH key to the instance through Terraform metadata
- The key in --ssh-key-file=key is the openssh private key for the instance

Could this be related to disk space? Or with SSH key propagation to the instance's ~/.ssh/authorized_key files? Any other thoughts?

I've read through every forum I could find and none of the suggestions netted any results.

[Holla]

Hi @cherriford,

It sounds like you're facing a frustrating SSH connection issue with your GCE instance over Identity Aware Proxy (IAP). Here are several steps and considerations to help troubleshoot and potentially resolve the problem:

Check SSH Key Propagation:

Make sure the public SSH key is being correctly added to the metadata of the instance. You can verify this by checking the ~/.ssh/authorized_keys file on the instance (if you can access it through another method).
Use the gcloud compute project-info describe command to confirm that the keys are registered correctly in the project's metadata as well.
Instance Disk Space:

Disk space issues can potentially cause problems, including issues with user profiles and SSH keys. Use the serial console or any other method to check available disk space on the instance. If it's low, try clearing up some space.
IAP Configuration:

Ensure that IAP is set up correctly for your project and that the user account you are using has the necessary IAP and IAM permissions. The required permissions are typically iap.tunnelInstances.accessViaIAP.
Firewall Rules:

Check that your firewall rules allow SSH traffic. Ensure that the IAP allows outbound access to your instance.
SSH Key Format:

Ensure that the SSH keys you’re generating and using are in the correct format. They should be OpenSSH format if you're using OpenSSH tools.
Use Verbose Mode:

Run your SSH command with the -v option for verbose output to get more insight into where the connection is failing. This can sometimes provide useful debug information.
Instance State:

Verify that your GCE instance is in a healthy state. Check the Google Cloud Console to see if there are any issues reported with the instance.
Check for Multiple SSH Keys:

If there are multiple public SSH keys associated with the instance, this can sometimes lead to confusion. Ensure that only the intended keys are present in authorized_keys.
Firewall or Security Policies:

If you're using a firewall or security policies, review them to ensure no rules are blocking SSH connections or IAP traffic.
Test with Different User:

If possible, try connecting with a different user account that has access configured to see if the problem is user-specific.

Best, 

[dimbuido]

Hi @cherriford,

The whole idea of Identity Aware Proxy (IAP) is to eliminate the dependency of having to use ssh keys that will require you to push your public keys in the target VM instance. It also eliminates the need for allowing 0.0.0.0/0 ssh firewall rules as you only need to allow 35.235.240.0/20 Ingress on port 22.

What I normally do is to use gcloud cli to authenticate. Once I am authenticated then I can ssh to the target VMs I have access to using the command: gcloud compute ssh <name of vm> --tunnel-through-iap --project <project_id> --zone=<availability zone where the vm is deployed>.

There are also IAM permission requirements needed for it to work which you can find here

Here's the link for more information. This means you don't need to use ssh keys anymore.

Hope this helps.