gcsfuse permission issue from GKE pod to delete and put files from GCS bucket

[nishakushwaha07]

Hi, I have done setup of SFTP application on GKE cluster to mount the GCS bucket using gcsfuse.

I am able to get or read the files/folders from SFTP application or at POD level. But, having below issue to delete or put files/folders to GCS bucket.

time="16/06/2024 10:46:20.098012" severity=TRACE message="gcs: Req 0x3: <- DeleteObject(\"Hadoopshare/ExtUpload/FB1/override-me.txt\")"
time="16/06/2024 10:46:20.106916" severity=TRACE message="gcs: Req 0x3: -> DeleteObject(\"Hadoopshare/ExtUpload/FB1/override-me.txt\") (8.891388ms): googleapi: Error 403: Access denied., forbidden"
time="16/06/2024 10:46:20.107022" severity=ERROR message="Unlink: permission denied, DeleteChildFile: DeleteObject: googleapi: Error 403: Access denied., forbidden"
time="16/06/2024 10:46:20.107102" severity=TRACE message="fuse_debug: Op 0x00000034 connection.go:515] -> Error: \"permission denied\""
time="16/06/2024 10:46:20.107122" severity=ERROR message="fuse: *fuseops.UnlinkOp error: permission denied

1.
gcloud container clusters update cloud-migration-factory \
--workload-pool=cloud-migration-factory-422911.svc.id.goog \
--project=cloud-migration-factory-422911 \
--location=northamerica-northeast1-a

2.
kubectl create serviceaccount first-bucket-sa -n sftp-demo

3.
gcloud iam service-accounts create first-bucket-sa \
--display-name="GCS Bucket Access Service Account" \
--project=cloud-migration-factory-422911

4.
gcloud projects add-iam-policy-binding cloud-migration-factory-422911 \
--member="serviceAccount: first-bucket-sa@cloud-migration-factory-422911.iam.gserviceaccount.com" \
--role="roles/storage.objectAdmin"

5.
gcloud aim service-accounts add-aim-policy-binding first-bucket-sa@cloud-migration-factory-422911.iam.gserviceaccount.com \
--member="serviceAccount:cloud-migration-factory-422911.svc.id.goog[sftp-demo/first-bucket-sa]" \
--role="roles/iam.workloadIdentityUser" \
--project="cloud-migration-factory-422911"

6.
kubectl annotate serviceaccount first-bucket-sa iam.gke.io/gcp-service-account=first-bucket-sa@cloud-migration-factory-422911.iam.gserviceaccount.com -n sftp-demo

[Manish_B · Accepted solution]

Hello @nishakushwaha07,

Please note that if all the required permissions are given correctly to the service account, one of the reasons that could cause the permission error here in this case could be that the GKE metadata server is disabled in the node pool.

Please follow this to know how to properly enable the Workload Identity and Metadata Server on GKE using Terraform: https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/blob/main/docs/terraform.md

I hope that the above provided information is helpful. If not, to ensure a faster resolution and dedicated support for your issue, I kindly request you to file a support ticket by clicking here as this would require more specific debugging and analysis. Our support team will prioritize your request and provide you with the assistance you need.

Thanks & Regards,  
Manish Baviredddy