Issue with user managed GCP public key for Service Account - treaded as Google managed instead

[leoezhil]

We are encountering an urgent issue with uploading public keys to service accounts in Google Cloud Platform. When we upload a public key to a newly created service account, it is incorrectly labeled as "Google managed" (with the Google icon on the left) instead of "Uploaded" (arrow-up icon). This issue does not occur with older service accounts, where the certificates are uploaded and labeled correctly.

Consequences:

This mislabeling has significant consequences for our product, causing breaking changes. Specifically:

1. The certificate map exposed at: https://www.googleapis.com/robot/v1/metadata/x509/<sa_account@project>.iam.gserviceaccount.com has a different key/id for the uploaded certificate.
2. The expected behavior for uploaded certificates is for the ID to be the SHA-1 checksum of the certificate file (shasum /path/to/file), whereas the expected ID for a Google managed certificate is the SHA-1 checksum of the certificate itself (openssl x509 -in ./public_key.pem -noout -fingerprint).

By changing how the ID gets generated, our product breaks as it cannot find the certificate in the endpoint above.

We are curious to know why this issue is occurring with newly created service accounts and looking for guidance on how to ensure uploaded public keys are correctly labeled as "Uploaded"? Is there a GCP setting or policy that we need to change to resolve this?

[Hilda_Arteaga]

Hello @leoezhil 

Thanks for sharing your question in the Apigee community.

First, let's mention that

GOOGLE_MANAGED keys cannot be downloaded, and are automatically rotated. They’re not visible in the Cloud console, but will show up in the output of “gcloud iam service-accounts keys list” command. By default the gcloud output does not distinguish between USER-MANAGER keys and GOOGLE-MANAGER keys which can cause confusion to users. 

Now, as you mention there are some policies that can help to limit the use of these keys.

In this case, you could start having a look to “constraints/iam.disableServiceAccountKeyUpload”

This prevents principles from uploading a public key for a service account. Which disables the feature that allows uploading public keys to service accounts,  so this will happen if the constraint is set up as “True”(you could change it)

Usually users can upload public keys to service accounts based on their Could IAM roles and permission. 

Hopefully this information is helpful

[leoezhil]

Thank you, Hilda, for your response. Let me take a stab at explaining the issue again. We can upload the user-generated public keys to our service accounts, so the constraints and permissions are good.

In our older service accounts, the “keyOrigin” would appear as "USER_PROVIDED" when we upload the public key. Please see the screenshot below, as we have 4 keys, and 3 of them with the upload icon next to it show up as "USER_PROVIDED".

Now the issue is with Service accounts which are newly created, we can still upload the keys, please see below example of upload and the response.

The key is uploaded successfully, and response says "USER_PROVIDED".

BUT, when we do list keys on GCP or through APIs, those keys show up as “GOOGLE_PROVIDED”.

The way google generates IDs of keys for “USER_PROVIDED” is different from how it generates for “GOOGLE_PROVIDED”. This leads to our integration issue, where we look for user generated key mechanisms.

Consequences:

This mislabeling has significant consequences for our product, causing breaking changes. Specifically:

1. The certificate map exposed at: https://www.googleapis.com/robot/v1/metadata/x509/<sa_account@project>.iam.gserviceaccount.com has a different key/id for the uploaded certificate.
2. The expected behavior for uploaded certificates is for the ID to be the SHA-1 checksum of the certificate file (shasum /path/to/file), whereas the expected ID for a Google managed certificate is the SHA-1 checksum of the certificate itself (openssl x509 -in ./public_key.pem -noout -fingerprint).

By changing how the ID gets generated, our product breaks as it cannot find the certificate in the endpoint above. I hope this clarifies the post and gives significant evidence on the underlying cause of it. I am looking forward to your response. 

Thank you again for your help.

[Hilda_Arteaga]

Hi @leoezhil  thanks for providing more details and screens.

I’m double checking this, once I have further information myself or someone from Google Apigee’s team will jump in with more details. Apologies for the delay or the issues this may cause 

Thanks @Cdillard  noted and checking

[Cdillard]

We can confirm this bug/behavior as well and it is impacting operations. 

[acheng_jh]

This is impacting workload migrations into GCP.  Prompt remediation would be greatly appreciated.

[leoezhil]

@Hilda_Arteaga. Ty.  We have also opened a high impact support case for the same scenario. Case # 52686529.  You can check with Google support team as well. 

 

[acheng_jh]

We also have case Google support case 52685694 opened as well.

[leoezhil · Accepted solution]

Google cloud product and engineering team has acknowledged this issue and released a fix. We have verified the fix so closing this issue as solved. 

[acheng_jh]

Thank you for validating!!