Vertex Ai service account unable to access GCS buckets though it has StorageObjectViewer roles

[Kiki26]

We have a Kube Flow pipeline job submitted via google-ai-platform. The job  that uses the vertex-ai service account reads from a GCS bucket that it has ObjectViewer and LegacyObjectReader permissions. We still get a 403. I would really appreciate leads to debug this futher, >Thanks!

[KirTitievsky]

Can you check Logging to see which specific method call fails?   We can then make sure that that method is covered by the roles (see this reference).   My guess is that Vertex is trying to do something with the buckets (e.g. list buckets). But that's a guess. 

[Kiki26]

Thank you for your response.
In the logs I see:
bucket = client.get_bucket(bucket_name)
calls:

File "/opt/app/.venv/lib/python3.8/site-packages/google/cloud/storage/client.py", line 773, in get_bucket

2023-11-01 21:37:11.277

workerpool0-0

bucket.reload(

2023-11-01 21:37:11.277

workerpool0-0

File "/opt/app/.venv/lib/python3.8/site-packages/google/cloud/storage/bucket.py", line 1077, in reload

2023-11-01 21:37:11.277

workerpool0-0

super(Bucket, self).reload(

2023-11-01 21:37:11.277

workerpool0-0

File "/opt/app/.venv/lib/python3.8/site-packages/google/cloud/storage/_helpers.py", line 246, in reload

2023-11-01 21:37:11.278

workerpool0-0

api_response = client._get_resource(

2023-11-01 21:37:11.278

workerpool0-0

File "/opt/app/.venv/lib/python3.8/site-packages/google/cloud/storage/client.py", line 378, in _get_resource

return self._connection.api_request(

Here is a snippet of my code:

service_account_info = json.loads(service_account_key_content) # service account's key JSON string
client = storage.Client.from_service_account_info(service_account_info)

bucket = client.get_bucket(bucket_name)
blobs = bucket.list_blobs(prefix=prefix)
for blob in blobs:
    blob_content = blob.download_as_text()

[KirTitievsky · Accepted solution]

I see. It's trying to get the bucket metadata. But you only gave it permissions to do things with objects. You need to grant it a role that has bucket.* permissions.  It lokos like roles/storage.legacyBucketReader grants that permission and not much more.

[Kiki26]

Worked like a charm!  How can I know, next time, which method requires what role ? Is there some guide, please ?

[KirTitievsky]

https://cloud.google.com/storage/docs/access-control/iam-roles -- there is no method -> role map that I know. But you can guess from this roles -> permissions map pretty well, since the permissions are pretty well aligned with methods.