Service account stopped working

[arik]

I've been using a service account to load data into my gcp bucket.
It was working fine for a year, and suddenly I started getting a permission saying:

AccessDeniedException: 403 ***@***.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist).

I checked the service account roles, and it has "Storage Object Admin"  
In addition it wasn't changed recently.

Any idea what's the issue here?

[VishalBulbule]

@arik 

The error you mentioned is at bucket level , while accessing buckets not during object.

any operation which directly listing buckets or similar rather than object level operation?

[arik]

Thanks for your reply @VishalBulbule 

I'm running:
gsutil rsync -R  ./LOCAL_FOLDER "gs://MY_BUCKET"

It was working or more than a year, than suddenly stopped

[VishalBulbule]

Can you try adding / at the end?

gsutil rsync -R  ./LOCAL_FOLDER "gs://MY_BUCKET/"

[kolban]

If we look here ... we can look at the permissions contained within the Storage Object Admin role.  It contains no permissions for working with buckets.  To work with buckets, your service account would appear to need Storage Admin (note Storage Admin <> Storage Object Admin).   I hear you say that "it" wasn't changed recently ... can we clarify what you mean by that?  From my understanding, you have two options:

1. Add the Storage Admin role to the service account at the Project IAM level.  This would mean that the service account has Storage Admin for ALL buckets in your project.

2. Add the Storage Admin role to the service account for the SPECIFIC bucket in question.

[arik]

Thanks for your replies 
You're right @kolban , buckets aren't listed for Storage Object Admin.

By it I mean there was no change to the service account.
I authenticated using the same service account on my local machine, and the same command worked. (was able to push files to the same bucket)
From github actions, it still fails.
I'll just point again, that this service account is used to push files from github action to the same bucket for about 1.5 years.
There's no change that I'm aware of, to the gcp bucket, service account, or github secrets

[kolban]

Aha my friend ... a new clue.  I don't think I picked up on the notion that the request was coming from github.  What I think we should do now is look at the path where the request originates from.  When I saw "gsutil" I immediately thought that you were running from the local workstation.   Now we need to drill into the environment where gsutil is being invoked from.  Sadly I'm no github expert so if you can describe the source environment slowly and with links, that would help me (at least).

What I am sensing is:

1. You are able to execute a gsutil command on your local workstation.  What is the EXACT command you are running?  Who EXACTLY are you running as when you run gsutil?

2. What is the environment where the request is failing?  What are the exact error messages?

3. What evidence do we have that the request that is failing is being executed as the same identity that works?

4. What evidence do we have that you are targeting the same target bucket?

[arik · Accepted solution]

Thanks again @kolban , I found the root cause.
On my local machine I was running gsutil version 4.59. 
I assumed the cause might be that github actions use the lates gsutil version each time it runs.
So I updated my local gsutil version to latest, which is 5.18.
And voila... now it failed locally as well.
On the old version, gsutil wasn't asking for the storage.buckets.get permission. (verified that on the gcp bucket access logs)

Adding this permission to my service account solved the issue.

Thanks for the help!

[Tshepo]

Hi @arik 

Can you advise what one can do, if the role cannot be edited to add the permission, since "Role cannot be edited as it is inherited from another resource"?

I ran into above issue while "Deploying a website with Cloud Run" https://codelabs.developers.google.com/codelabs/cloud-run-deploy#3

[thiloh]

We noticed the same issue after upgrading the cloud sdk.

Why is the additional `storage.buckets.get` permission necessary for a `gsutil rm gs://${BUCKET_NAME}/myfile` command? according to https://cloud.google.com/storage/docs/access-control/iam-gsutil?hl=de object admin permissions should be sufficient.

thanks a lot & kind regards