How can I give my VMs write access to a cloud storage bucket (when using gcsfuse)?

[vedantroy-genmo]

When using Google Cloud batch, I can mount my Google cloud storage bucket to a VM without issue. The programming running in the container successfully reads and writes from the bucket.

I launched a VM with the goal of manually using gcsfuse to mount a bucket to the VM. To do this, I used the following flags:

sudo gcsfuse --debug_fuse --debug_fs --debug_gcs --debug_http --foreground --log-file gcs_fuse.log --implicit-dirs --file-mode=777 --dir-mode=777 --implicit-dirs my_bucket_name /mnt/my_mount_point

But, while listing objects works without issue, when I try to create a file using `sudo touch /mnt/my_mount_point/test.txt`, I get the following logs:

{"timestamp":{"seconds":1713035751,"nanos":623371316},"severity":"INFO","message":"Start gcsfuse/2.0.0 (Go version go1.22.1) for app \"\" using mount point: /mnt/training_data\n"}
{"timestamp":{"seconds":1713035751,"nanos":623587162},"severity":"INFO","message":"GCSFuse mount command flags: {\"AppName\":\"\",\"Foreground\":true,\"ConfigFile\":\"\",\"MountOptions\":{},\"DirMode\":511,\"FileMode\":511,\"Uid\":-1,\"Gid\":-1,\"ImplicitDirs\":true,\"OnlyDir\":\"\",\"RenameDirLimit\":0,\"CustomEndpoint\":null,\"BillingProject\":\"\",\"KeyFile\":\"\",\"TokenUrl\":\"\",\"ReuseTokenFromUrl\":true,\"EgressBandwidthLimitBytesPerSecond\":-1,\"OpRateLimitHz\":-1,\"SequentialReadSizeMb\":200,\"MaxRetrySleep\":30000000000,\"StatCacheCapacity\":20460,\"StatCacheTTL\":60000000000,\"TypeCacheTTL\":60000000000,\"HttpClientTimeout\":0,\"MaxRetryDuration\":-1000000000,\"RetryMultiplier\":2,\"LocalFileCache\":false,\"TempDir\":\"\",\"ClientProtocol\":\"http1\",\"MaxConnsPerHost\":100,\"MaxIdleConnsPerHost\":100,\"EnableNonexistentTypeCache\":false,\"StackdriverExportInterval\":0,\"OtelCollectorAddress\":\"\",\"LogFile\":\"/home/ved/gcs_fuse.log\",\"LogFormat\":\"json\",\"ExperimentalEnableJsonRead\":false,\"DebugFuseErrors\":true,\"DebugFuse\":true,\"DebugFS\":true,\"DebugGCS\":true,\"DebugHTTP\":true,\"DebugInvariants\":false,\"DebugMutex\":false}"}
{"timestamp":{"seconds":1713035751,"nanos":623676033},"severity":"INFO","message":"GCSFuse mount config flags: {\"CreateEmptyFile\":false,\"Severity\":\"TRACE\",\"Format\":\"json\",\"FilePath\":\"/home/ved/gcs_fuse.log\",\"LogRotateConfig\":{\"MaxFileSizeMB\":512,\"BackupFileCount\":10,\"Compress\":true},\"MaxSizeMB\":-1,\"CacheFileForRangeRead\":false,\"CacheDir\":\"\",\"TtlInSeconds\":-9223372036854775808,\"TypeCacheMaxSizeMB\":4,\"StatCacheMaxSizeMB\":-9223372036854775808,\"EnableEmptyManagedFolders\":false}"}
{"timestamp":{"seconds":1713035751,"nanos":623701996},"severity":"INFO","message":"Creating Storage handle..."}
{"timestamp":{"seconds":1713035751,"nanos":623710575},"severity":"INFO","message":"UserAgent = gcsfuse/2.0.0 (Go version go1.22.1) (GPN:gcsfuse) (Cfg:0:0)\n"}
{"timestamp":{"seconds":1713035751,"nanos":625876567},"severity":"INFO","message":"Creating a mount at \"/mnt/training_data\"\n"}
{"timestamp":{"seconds":1713035751,"nanos":625978861},"severity":"INFO","message":"Creating a new server...\n"}
{"timestamp":{"seconds":1713035751,"nanos":626041939},"severity":"INFO","message":"Set up root directory for bucket training_data_ved"}
{"timestamp":{"seconds":1713035751,"nanos":626102748},"severity":"TRACE","message":"gcs: Req              0x0: <- ListObjects(\"\")"}
{"timestamp":{"seconds":1713035751,"nanos":670801757},"severity":"TRACE","message":"gcs: Req              0x0: -> ListObjects(\"\") (44.696127ms): OK"}
{"timestamp":{"seconds":1713035751,"nanos":670892276},"severity":"INFO","message":"Mounting file system \"training_data_ved\"..."}
{"timestamp":{"seconds":1713035751,"nanos":670936018},"severity":"TRACE","message":"fuse_debug: Beginning the mounting kickoff process"}
{"timestamp":{"seconds":1713035751,"nanos":670946995},"severity":"TRACE","message":"fuse_debug: Parsing fuse file descriptor"}
{"timestamp":{"seconds":1713035751,"nanos":670957420},"severity":"TRACE","message":"fuse_debug: Preparing for direct mounting"}
{"timestamp":{"seconds":1713035751,"nanos":670981840},"severity":"TRACE","message":"fuse_debug: Successfully opened the /dev/fuse in blocking mode"}
{"timestamp":{"seconds":1713035751,"nanos":670998986},"severity":"TRACE","message":"fuse_debug: Starting the unix mounting"}
{"timestamp":{"seconds":1713035751,"nanos":672289937},"severity":"TRACE","message":"fuse_debug: Unix mounting completed successfully"}
{"timestamp":{"seconds":1713035751,"nanos":672330128},"severity":"TRACE","message":"fuse_debug: Completed the mounting kickoff process"}
{"timestamp":{"seconds":1713035751,"nanos":672341828},"severity":"TRACE","message":"fuse_debug: Creating a connection object"}
{"timestamp":{"seconds":1713035751,"nanos":672732543},"severity":"TRACE","message":"fuse_debug: Op 0x00000002        connection.go:415] <- init"}
{"timestamp":{"seconds":1713035751,"nanos":672781660},"severity":"TRACE","message":"fuse_debug: Op 0x00000002        connection.go:497] -> OK ()"}
{"timestamp":{"seconds":1713035751,"nanos":672802875},"severity":"TRACE","message":"fuse_debug: Successfully created the connection"}
{"timestamp":{"seconds":1713035751,"nanos":672822244},"severity":"TRACE","message":"fuse_debug: Waiting for mounting process to complete"}
{"timestamp":{"seconds":1713035751,"nanos":672833028},"severity":"INFO","message":"File system has been successfully mounted."}
{"timestamp":{"seconds":1713035759,"nanos":83738127},"severity":"TRACE","message":"fuse_debug: Op 0x00000004        connection.go:415] <- GetInodeAttributes (inode 1, PID 14288)"}
{"timestamp":{"seconds":1713035759,"nanos":83889790},"severity":"TRACE","message":"fuse_debug: Op 0x00000004        connection.go:497] -> OK ()"}
{"timestamp":{"seconds":1713035759,"nanos":84591448},"severity":"TRACE","message":"fuse_debug: Op 0x00000006        connection.go:415] <- LookUpInode (parent 1, name \"test.txt\", PID 14288)"}
{"timestamp":{"seconds":1713035759,"nanos":84702891},"severity":"TRACE","message":"gcs: Req              0x1: <- ListObjects(\"test.txt/\")"}
{"timestamp":{"seconds":1713035759,"nanos":84833690},"severity":"TRACE","message":"gcs: Req              0x2: <- StatObject(\"test.txt\")"}
{"timestamp":{"seconds":1713035759,"nanos":99425085},"severity":"TRACE","message":"gcs: Req              0x1: -> ListObjects(\"test.txt/\") (14.729932ms): OK"}
{"timestamp":{"seconds":1713035759,"nanos":101432648},"severity":"TRACE","message":"gcs: Req              0x2: -> StatObject(\"test.txt\") (16.582257ms): gcs.NotFoundError: storage: object doesn't exist"}
{"timestamp":{"seconds":1713035759,"nanos":101523751},"severity":"TRACE","message":"fuse_debug: Op 0x00000006        connection.go:499] -> Error: \"no such file or directory\""}
{"timestamp":{"seconds":1713035759,"nanos":101624140},"severity":"TRACE","message":"fuse_debug: Op 0x00000008        connection.go:415] <- CreateFile (parent 1, name \"test.txt\", PID 14288)"}
{"timestamp":{"seconds":1713035759,"nanos":101819131},"severity":"TRACE","message":"fuse_debug: Op 0x00000008        connection.go:497] -> OK (inode 2)"}
{"timestamp":{"seconds":1713035759,"nanos":101931379},"severity":"TRACE","message":"fuse_debug: Op 0x0000000a        connection.go:415] <- FlushFile (inode 2, PID 14288)"}
{"timestamp":{"seconds":1713035759,"nanos":101992770},"severity":"TRACE","message":"gcs: Req              0x3: <- StatObject(\"test.txt\")"}
{"timestamp":{"seconds":1713035759,"nanos":114849417},"severity":"TRACE","message":"gcs: Req              0x3: -> StatObject(\"test.txt\") (12.861969ms): gcs.NotFoundError: storage: object doesn't exist"}
{"timestamp":{"seconds":1713035759,"nanos":120625342},"severity":"TRACE","message":"gcs: Req              0x4: <- CreateObject(\"test.txt\")"}
{"timestamp":{"seconds":1713035759,"nanos":133314288},"severity":"TRACE","message":"gcs: Req              0x4: -> CreateObject(\"test.txt\") (12.686702ms): error in closing writer : googleapi: Error 403: Access denied., forbidden"}
{"timestamp":{"seconds":1713035759,"nanos":133429043},"severity":"ERROR","message":"FlushFile: permission denied, FileInode.Sync: SyncObject: CreateObject: error in closing writer : googleapi: Error 403: Access denied., forbidden"}
{"timestamp":{"seconds":1713035759,"nanos":133492320},"severity":"TRACE","message":"fuse_debug: Op 0x0000000a        connection.go:499] -> Error: \"permission denied\""}
{"timestamp":{"seconds":1713035759,"nanos":133507663},"severity":"ERROR","message":"fuse: *fuseops.FlushFileOp error: permission denied"}
{"timestamp":{"seconds":1713035759,"nanos":133644551},"severity":"TRACE","message":"fuse_debug: Op 0x0000000c        connection.go:415] <- SetInodeAttributes (inode 2, PID 14288, atime 2024-04-13 19:15:59.131644036 +0000 UTC, mtime 2024-04-13 19:15:59.131644036 +0000 UTC)"}
{"timestamp":{"seconds":1713035759,"nanos":133710819},"severity":"TRACE","message":"gcs: Req              0x5: <- StatObject(\"test.txt\")"}
{"timestamp":{"seconds":1713035759,"nanos":146293744},"severity":"TRACE","message":"gcs: Req              0x5: -> StatObject(\"test.txt\") (12.595671ms): gcs.NotFoundError: storage: object doesn't exist"}
{"timestamp":{"seconds":1713035759,"nanos":146355139},"severity":"TRACE","message":"fuse_debug: Op 0x0000000c        connection.go:497] -> OK ()"}
{"timestamp":{"seconds":1713035759,"nanos":146466238},"severity":"TRACE","message":"fuse_debug: Op 0x0000000e        connection.go:415] <- FlushFile (inode 2, PID 14288)"}
{"timestamp":{"seconds":1713035759,"nanos":146570969},"severity":"TRACE","message":"gcs: Req              0x6: <- StatObject(\"test.txt\")"}
{"timestamp":{"seconds":1713035759,"nanos":160472228},"severity":"TRACE","message":"gcs: Req              0x6: -> StatObject(\"test.txt\") (13.900051ms): gcs.NotFoundError: storage: object doesn't exist"}
{"timestamp":{"seconds":1713035759,"nanos":160517073},"severity":"TRACE","message":"gcs: Req              0x7: <- CreateObject(\"test.txt\")"}
{"timestamp":{"seconds":1713035759,"nanos":166213653},"severity":"TRACE","message":"gcs: Req              0x7: -> CreateObject(\"test.txt\") (5.693125ms): error in closing writer : googleapi: Error 403: Access denied., forbidden"}
{"timestamp":{"seconds":1713035759,"nanos":166253720},"severity":"ERROR","message":"FlushFile: permission denied, FileInode.Sync: SyncObject: CreateObject: error in closing writer : googleapi: Error 403: Access denied., forbidden"}
{"timestamp":{"seconds":1713035759,"nanos":166293271},"severity":"TRACE","message":"fuse_debug: Op 0x0000000e        connection.go:499] -> Error: \"permission denied\""}
{"timestamp":{"seconds":1713035759,"nanos":166305328},"severity":"ERROR","message":"fuse: *fuseops.FlushFileOp error: permission denied"}
{"timestamp":{"seconds":1713035759,"nanos":166389137},"severity":"TRACE","message":"fuse_debug: Op 0x00000010        connection.go:415] <- ReleaseFileHandle (PID 0)"}
{"timestamp":{"seconds":1713035759,"nanos":166696391},"severity":"TRACE","message":"fuse_debug: Op 0x00000010        connection.go:497] -> OK ()"}

What do I need to do to give my VM write permissions for my bucket? 

[LeAnt]

Hi @vedantroy-genmo 

Welcome to Google Cloud Community!

You can find the required roles and permissions, along with the instructions on how to mount Cloud Storage buckets using Cloud Storage FUSE, in this Google Cloud document.

I hope this information is helpful.

If you need further assistance, you can always file a case with our support team.

[sj0]

Hi

I am also a newbie, and face a similar issue.  The problem is that I am learning on my own, and not as a part of my compnay - using a personal gmail account. 

While I could setup a gcs account, create VMs, buckets, etc. - I have not been able to create an "Organization", "Users" or "Groups", other than the defaults created by Google.  So, I assumed that I am an admin, and dont need special access rights, etc.  I am able to mount the bucket to a directory, using "rw" permission in the mount command - but like the previous poster, I am unable to write to this directory.   

The help document wants me to talk to my admin to give me permission - which does not apply to my case.

Appreciate your help on this "corner case".