Undocumented Breaking Cloud Run Changes in v461.0....

VlGr · 01-27-2024 08:50 PM

We have had the same CI/CD process for over a year and a half now, with the same service accounts and IAM policies, and it recently started failing. I pinned this down in what appears to be an undocumented change in v461.0.0 . It all started with the following error:

Permission 'run.operations.get' denied on resource 'projects/{projectName}/locations/us-central1/operations/{operationId}' (or resource may not exist).

After figuring out that there was no issues with v460 I saw the following change in the log.

Whereas in v460 and before there was a call like this:

024-01-28 03:17:27,724 INFO     ___FILE_ONLY___ ==== request start ====

2024-01-28 03:17:27,724 INFO     ___FILE_ONLY___ uri: https://run.googleapis.com/v1/projects/{projectId}/locations/us-central1/services/{serviceId}:setIamPolicy?alt=json

2024-01-28 03:17:27,724 INFO     ___FILE_ONLY___ method: POST

In v461 there is now this:

2024-01-28 03:22:00,294 INFO     ___FILE_ONLY___ ==== request start ====

2024-01-28 03:22:00,294 INFO     ___FILE_ONLY___ uri: https://us-central1-run.googleapis.com/v1/projects/{projectId}/locations/us-central1/ope
rations/{operationId}:wait?alt=json

2024-01-28 03:22:00,294 INFO     ___FILE_ONLY___ method: POST

Was this documented anywhere? I can't even figure out which call this is supposed to be.

gdash

Started getting this error in the last couple of days as well, similar situation.

robertcarlos

Hi @VlGr,

Welcome to Google Cloud Community!

You are getting this error as run.operations.get is not properly declared, as this permission is recommended in order to read the status of the service. Role should be at least set to roles/run.viewer as this contains the run.operations.get permission. Please check this documentation on Cloud Run IAM roles for more information on predefined roles and deployment permissions for Cloud Run.

You may also check these similar Stack Overflow questions that could be related to your concern:

Hope this helps.

DChiesa2023

hi robertcarlos,

The 2nd link you offered refers to permission run.services.get not run.operations.get. So I think that one is a different issue. The first link did not provide clear guidance on run.operations.get either.

Also Any insight as to why this behavior would change from gcloud CLI v460 to v461 ?

iskandertajines

Hello @robertcarlos

I have also started encountered this issue, even though there have been no changes to IAM permissions for my CICD service account. It already has the roles/run.developer role.

Do you have any solution in mind? thank you

DChiesa2023

It seems that

https://us-central1-run.googleapis.com/v1/projects/{projectId}/locations/us-central1
/operations/{operationId}:wait?alt=json

...is waiting for the resolution for a long-running-operation (so called LRO). (reference)

But I don't have insight into

why it changed from v460 to v461
why it is using a regional cloud-run endpoint (us-central1-run.googleapis.com) versus the global one (run.googleapis.com)

Based on the name, it seems likely that the "run.operations.get" permission is required to "wait" on an LRO. And maybe the principal you are using doesn't have that permission. See the answer from robertcarlos. Why v461 introduced the change is not clear. and I agree with you, there's nothing documented for Cloud Run in the release notes for v461.

VlGr

This is an official issue now with discussion here:

https://issuetracker.google.com/issues/322167526

Undocumented Breaking Cloud Run Changes in v461.0.0