Persistent SSH Public Key Authentication Failure from Cloud Build to Windows VM - GCE Agent Scope C

[cbcbcb]

Hello Cloud Community,

I'm hitting a wall trying to get SSH public key authentication working consistently from Google Cloud Build to a Windows Server VM. The connection always fails with "Permission denied" (reported as "Failed password" in the VM's OpenSSH logs), and I've exhausted standard troubleshooting steps. I'm hoping someone here might have encountered a similar very specific issue or have insights.

My Setup:

• Cloud Build: Using gcloud compute ssh [username]@[vm-name] in cloudbuild.yaml with --log-http. Cloud Build generates a fresh SSH key for each run.

The Core Problem: The SSH connection fails because the OpenSSH server (sshd) on the Windows VM consistently falls back to password authentication, indicating it's not correctly using the provided public key.

Troubleshooting Steps & Findings (Detailed):

1. SSH Key Propagation (Cloud Build & GCE Agent):

   • gcloud compute ssh --log-http confirms that Cloud Build successfully pushes the generated SSH public key to the project's metadata for the user. The key is unique per build.

2. Windows VM OpenSSH Server (sshd) Configuration:

   • Confirmed C:\ProgramData\ssh\sshd_config has:
     • PubkeyAuthentication yes (uncommented)
     • PasswordAuthentication no (uncommented, explicitly to force key-only auth)
     • AuthorizedKeysFile .ssh\authorized_keys
   • The sshd service is verified as "Running".

3. authorized_keys File & Directory Permissions (icacls):

   • Executed multiple icacls commands to enforce strict permissions, including resetting them.
   • Current icacls output for C:\Users\[username]\.ssh\authorized_keys: C:\Users\[username]\.ssh\authorized_keys [machine name]\[username]:(F)

4. Hostname Mismatch (Addressed in cloudbuild.yaml):

   • Discovered the GCP VM resource name  differs from the Windows OS hostname

5. Google Compute Engine Agent (GCE Agent) Issues - The Biggest Mystery:

   • The authorized_keys file's timestamp on the VM consistently does not update after Cloud Build runs, confirming the GCE Agent isn't writing the new keys.
   • Initially suspected file locking (e.g., by Notepad), but confirmed it's not the case.
   • CRITICAL CONTRADICTION: Event Viewer logs (Windows Logs > Application, filtered by "Google Compute Engine Agent" source) show:

     GCEGuestAgent: Error watching metadata: context canceled
     GCEGuestAgent: Cannot connect to cloud logging, skipping flush: rpc error: code = PermissionDenied desc = Request had insufficient authentication scopes.
     error details: name = ErrorInfo reason = ACCESS_TOKEN_SCOPE_INSUFFICIENT domain = googleapis.com metadata = map[method:google.logging.v2.LoggingServiceV2.WriteLogEntries service:logging.googleapis.com]

   • However, my VM's "Service account" ([project number]-compute@developer.gserviceaccount.com) is explicitly set to "Allow full access to all Cloud APIs" in the VM instance details. This seems to contradict the agent's error message.
   • The "Google Compute Engine Agent" service is confirmed to be "Running" in Windows Services.

6. Manual SSH Test (Confirms Basic Connectivity, Not Key Auth):

   • I can successfully SSH into the VM using the [username] password I set up. This confirms basic SSH connectivity, firewall rules, and the user account are fine.
   • However, when I manually generated an SSH key pair on my local machine, added its public key to C:\Users\[username]\.ssh\authorized_keys on the VM, restarted sshd, and then tried to SSH from my local machine using that specific private key (ssh -i ...), it still prompted me for a password. This indicates public key authentication is failing even with a manually placed key that should be readable.

My Questions to the Community:

1. Has anyone encountered the "GCEGuestAgent: Request had insufficient authentication scopes" error when their VM's service account is already set to "Allow full access to all Cloud APIs"? What was the resolution?
2. Given that authorized_keys permissions are seemingly correct and a manually placed key still results in a password prompt, are there any less common sshd_config settings on Windows or user profile issues that could be causing sshd to ignore authorized_keys for public key authentication?
3. Are there any known issues with the GCE Agent or OpenSSH that might explain this behavior?

Any insights or suggestions would be greatly appreciated! Thank you in advance for any help.

[reinc]

Hi @cbcbcb, 

Welcome to Google Cloud Community!

The VM's access scopes can only be changed while the VM is STOPPED.

This cli command tells the VM instance to allow the attached service account to use all of its granted permissions.

gcloud compute instances set-service-account YOUR_VM_NAME \
    --zone=YOUR_ZONE \
    --scopes=cloud-platform

Via the Cloud Console:

1. Stop the VM.

2. Click "Edit".

3. Scroll to "Access scopes" and select "Allow full access to all Cloud APIs".

4. Save and Start the VM.

To answers your questions:

1. Has anyone encountered the "GCEGuestAgent: Request had insufficient authentication scopes" error when their VM's service account is already set to "Allow full access to all Cloud APIs"? What was the resolution?

Yes, this is common. The cause is almost always that the scope change was made after the VM was created, and the VM was never stopped and started from the GCP console. A simple OS-level reboot does not apply the new scopes. The resolution is the STOP/START cycle described above.

After fixing the VM's access scopes, you need to trigger the GCE Agent to re-sync the keys from GCP and then verify that it has successfully updated the local authorized_keys file.

2. Given that authorized_keys permissions are seemingly correct and a manually placed key still results in a password prompt, are there any less common sshd_config settings on Windows or user profile issues that could be causing sshd to ignore authorized_keys for public key authentication?

Here are the less common settings and profile issues that could be causing this behavior:

• sshd_config are more specific than the basics and can override them.

• If authorized_keys file must be UTF-8 (without BOM). Windows Notepad can save it as UTF-16 or add a hidden "Byte Order Mark" (BOM) at the beginning of the file, which makes it unreadable to sshd.

• If the user account is locked out due to previous failed password attempts.

3. Are there any known issues with the GCE Agent or OpenSSH that might explain this behavior?

Known issue to GCE Agent scope:
On the error filter logs you’ve shared the “ACCESS_TOKEN_SCOPE_INSUFFICIENT” means the GCE Agent running on the VM will continue to operate with the old, more restrictive scopes until the instance is fully stopped and restarted.

Know issue to OpenSSH:
Your manual ssh key test with locally placed keys also will fail and can't continue to access your VMs. 

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.