How can I give my VMs write access to a cloud storage bucket (when using gcsfuse)?

[vedantroy-genmo]

When using Google Cloud batch, I can mount my Google cloud storage bucket to a VM without issue. The programming running in the container successfully reads and writes from the bucket.

I launched a VM with the goal of manually using gcsfuse to mount a bucket to the VM. To do this, I used the following flags:

sudo gcsfuse --debug_fuse --debug_fs --debug_gcs --debug_http --foreground --log-file gcs_fuse.log --implicit-dirs --file-mode=777 --dir-mode=777 --implicit-dirs my_bucket_name /mnt/my_mount_point

But, while listing objects works without issue, when I try to create a file using `sudo touch /mnt/my_mount_point/test.txt`, I get the following logs:

{"timestamp":{"seconds":1713035751,"nanos":623371316},"severity":"INFO","message":"Start gcsfuse/2.0.0 (Go version go1.22.1) for app \"\" using mount point: /mnt/training_data\n"}
{"timestamp":{"seconds":1713035751,"nanos":623587162},"severity":"INFO","message":"GCSFuse mount command flags: {\"AppName\":\"\",\"Foreground\":true,\"ConfigFile\":\"\",\"MountOptions\":{},\"DirMode\":511,\"FileMode\":511,\"Uid\":-1,\"Gid\":-1,\"ImplicitDirs\":true,\"OnlyDir\":\"\",\"RenameDirLimit\":0,\"CustomEndpoint\":null,\"BillingProject\":\"\",\"KeyFile\":\"\",\"TokenUrl\":\"\",\"ReuseTokenFromUrl\":true,\"EgressBandwidthLimitBytesPerSecond\":-1,\"OpRateLimitHz\":-1,\"SequentialReadSizeMb\":200,\"MaxRetrySleep\":30000000000,\"StatCacheCapacity\":20460,\"StatCacheTTL\":60000000000,\"TypeCacheTTL\":60000000000,\"HttpClientTimeout\":0,\"MaxRetryDuration\":-1000000000,\"RetryMultiplier\":2,\"LocalFileCache\":false,\"TempDir\":\"\",\"ClientProtocol\":\"http1\",\"MaxConnsPerHost\":100,\"MaxIdleConnsPerHost\":100,\"EnableNonexistentTypeCache\":false,\"StackdriverExportInterval\":0,\"OtelCollectorAddress\":\"\",\"LogFile\":\"/home/ved/gcs_fuse.log\",\"LogFormat\":\"json\",\"ExperimentalEnableJsonRead\":false,\"DebugFuseErrors\":true,\"DebugFuse\":true,\"DebugFS\":true,\"DebugGCS\":true,\"DebugHTTP\":true,\"DebugInvariants\":false,\"DebugMutex\":false}"}
{"timestamp":{"seconds":1713035751,"nanos":623676033},"severity":"INFO","message":"GCSFuse mount config flags: {\"CreateEmptyFile\":false,\"Severity\":\"TRACE\",\"Format\":\"json\",\"FilePath\":\"/home/ved/gcs_fuse.log\",\"LogRotateConfig\":{\"MaxFileSizeMB\":512,\"BackupFileCount\":10,\"Compress\":true},\"MaxSizeMB\":-1,\"CacheFileForRangeRead\":false,\"CacheDir\":\"\",\"TtlInSeconds\":-9223372036854775808,\"TypeCacheMaxSizeMB\":4,\"StatCacheMaxSizeMB\":-9223372036854775808,\"EnableEmptyManagedFolders\":false}"}
{"timestamp":{"seconds":1713035751,"nanos":623701996},"severity":"INFO","message":"Creating Storage handle..."}
{"timestamp":{"seconds":1713035751,"nanos":623710575},"severity":"INFO","message":"UserAgent = gcsfuse/2.0.0 (Go version go1.22.1) (GPN:gcsfuse) (Cfg:0:0)\n"}
{"timestamp":{"seconds":1713035751,"nanos":625876567},"severity":"INFO","message":"Creating a mount at \"/mnt/training_data\"\n"}
{"timestamp":{"seconds":1713035751,"nanos":625978861},"severity":"INFO","message":"Creating a new server...\n"}
{"timestamp":{"seconds":1713035751,"nanos":626041939},"severity":"INFO","message":"Set up root directory for bucket training_data_ved"}
{"timestamp":{"seconds":1713035751,"nanos":626102748},"severity":"TRACE","message":"gcs: Req              0x0: <- ListObjects(\"\")"}
{"timestamp":{"seconds":1713035751,"nanos":670801757},"severity":"TRACE","message":"gcs: Req              0x0: -> ListObjects(\"\") (44.696127ms): OK"}
{"timestamp":{"seconds":1713035751,"nanos":670892276},"severity":"INFO","message":"Mounting file system \"training_data_ved\"..."}
{"timestamp":{"seconds":1713035751,"nanos":670936018},"severity":"TRACE","message":"fuse_debug: Beginning the mounting kickoff process"}
{"timestamp":{"seconds":1713035751,"nanos":670946995},"severity":"TRACE","message":"fuse_debug: Parsing fuse file descriptor"}
{"timestamp":{"seconds":1713035751,"nanos":670957420},"severity":"TRACE","message":"fuse_debug: Preparing for direct mounting"}
{"timestamp":{"seconds":1713035751,"nanos":670981840},"severity":"TRACE","message":"fuse_debug: Successfully opened the /dev/fuse in blocking mode"}
{"timestamp":{"seconds":1713035751,"nanos":670998986},"severity":"TRACE","message":"fuse_debug: Starting the unix mounting"}
{"timestamp":{"seconds":1713035751,"nanos":672289937},"severity":"TRACE","message":"fuse_debug: Unix mounting completed successfully"}
{"timestamp":{"seconds":1713035751,"nanos":672330128},"severity":"TRACE","message":"fuse_debug: Completed the mounting kickoff process"}
{"timestamp":{"seconds":1713035751,"nanos":672341828},"severity":"TRACE","message":"fuse_debug: Creating a connection object"}
{"timestamp":{"seconds":1713035751,"nanos":672732543},"severity":"TRACE","message":"fuse_debug: Op 0x00000002        connection.go:415] <- init"}
{"timestamp":{"seconds":1713035751,"nanos":672781660},"severity":"TRACE","message":"fuse_debug: Op 0x00000002        connection.go:497] -> OK ()"}
{"timestamp":{"seconds":1713035751,"nanos":672802875},"severity":"TRACE","message":"fuse_debug: Successfully created the connection"}
{"timestamp":{"seconds":1713035751,"nanos":672822244},"severity":"TRACE","message":"fuse_debug: Waiting for mounting process to complete"}
{"timestamp":{"seconds":1713035751,"nanos":672833028},"severity":"INFO","message":"File system has been successfully mounted."}
{"timestamp":{"seconds":1713035759,"nanos":83738127},"severity":"TRACE","message":"fuse_debug: Op 0x00000004        connection.go:415] <- GetInodeAttributes (inode 1, PID 14288)"}
{"timestamp":{"seconds":1713035759,"nanos":83889790},"severity":"TRACE","message":"fuse_debug: Op 0x00000004        connection.go:497] -> OK ()"}
{"timestamp":{"seconds":1713035759,"nanos":84591448},"severity":"TRACE","message":"fuse_debug: Op 0x00000006        connection.go:415] <- LookUpInode (parent 1, name \"test.txt\", PID 14288)"}
{"timestamp":{"seconds":1713035759,"nanos":84702891},"severity":"TRACE","message":"gcs: Req              0x1: <- ListObjects(\"test.txt/\")"}
{"timestamp":{"seconds":1713035759,"nanos":84833690},"severity":"TRACE","message":"gcs: Req              0x2: <- StatObject(\"test.txt\")"}
{"timestamp":{"seconds":1713035759,"nanos":99425085},"severity":"TRACE","message":"gcs: Req              0x1: -> ListObjects(\"test.txt/\") (14.729932ms): OK"}
{"timestamp":{"seconds":1713035759,"nanos":101432648},"severity":"TRACE","message":"gcs: Req              0x2: -> StatObject(\"test.txt\") (16.582257ms): gcs.NotFoundError: storage: object doesn't exist"}
{"timestamp":{"seconds":1713035759,"nanos":101523751},"severity":"TRACE","message":"fuse_debug: Op 0x00000006        connection.go:499] -> Error: \"no such file or directory\""}
{"timestamp":{"seconds":1713035759,"nanos":101624140},"severity":"TRACE","message":"fuse_debug: Op 0x00000008        connection.go:415] <- CreateFile (parent 1, name \"test.txt\", PID 14288)"}
{"timestamp":{"seconds":1713035759,"nanos":101819131},"severity":"TRACE","message":"fuse_debug: Op 0x00000008        connection.go:497] -> OK (inode 2)"}
{"timestamp":{"seconds":1713035759,"nanos":101931379},"severity":"TRACE","message":"fuse_debug: Op 0x0000000a        connection.go:415] <- FlushFile (inode 2, PID 14288)"}
{"timestamp":{"seconds":1713035759,"nanos":101992770},"severity":"TRACE","message":"gcs: Req              0x3: <- StatObject(\"test.txt\")"}
{"timestamp":{"seconds":1713035759,"nanos":114849417},"severity":"TRACE","message":"gcs: Req              0x3: -> StatObject(\"test.txt\") (12.861969ms): gcs.NotFoundError: storage: object doesn't exist"}
{"timestamp":{"seconds":1713035759,"nanos":120625342},"severity":"TRACE","message":"gcs: Req              0x4: <- CreateObject(\"test.txt\")"}
{"timestamp":{"seconds":1713035759,"nanos":133314288},"severity":"TRACE","message":"gcs: Req              0x4: -> CreateObject(\"test.txt\") (12.686702ms): error in closing writer : googleapi: Error 403: Access denied., forbidden"}
{"timestamp":{"seconds":1713035759,"nanos":133429043},"severity":"ERROR","message":"FlushFile: permission denied, FileInode.Sync: SyncObject: CreateObject: error in closing writer : googleapi: Error 403: Access denied., forbidden"}
{"timestamp":{"seconds":1713035759,"nanos":133492320},"severity":"TRACE","message":"fuse_debug: Op 0x0000000a        connection.go:499] -> Error: \"permission denied\""}
{"timestamp":{"seconds":1713035759,"nanos":133507663},"severity":"ERROR","message":"fuse: *fuseops.FlushFileOp error: permission denied"}
{"timestamp":{"seconds":1713035759,"nanos":133644551},"severity":"TRACE","message":"fuse_debug: Op 0x0000000c        connection.go:415] <- SetInodeAttributes (inode 2, PID 14288, atime 2024-04-13 19:15:59.131644036 +0000 UTC, mtime 2024-04-13 19:15:59.131644036 +0000 UTC)"}
{"timestamp":{"seconds":1713035759,"nanos":133710819},"severity":"TRACE","message":"gcs: Req              0x5: <- StatObject(\"test.txt\")"}
{"timestamp":{"seconds":1713035759,"nanos":146293744},"severity":"TRACE","message":"gcs: Req              0x5: -> StatObject(\"test.txt\") (12.595671ms): gcs.NotFoundError: storage: object doesn't exist"}
{"timestamp":{"seconds":1713035759,"nanos":146355139},"severity":"TRACE","message":"fuse_debug: Op 0x0000000c        connection.go:497] -> OK ()"}
{"timestamp":{"seconds":1713035759,"nanos":146466238},"severity":"TRACE","message":"fuse_debug: Op 0x0000000e        connection.go:415] <- FlushFile (inode 2, PID 14288)"}
{"timestamp":{"seconds":1713035759,"nanos":146570969},"severity":"TRACE","message":"gcs: Req              0x6: <- StatObject(\"test.txt\")"}
{"timestamp":{"seconds":1713035759,"nanos":160472228},"severity":"TRACE","message":"gcs: Req              0x6: -> StatObject(\"test.txt\") (13.900051ms): gcs.NotFoundError: storage: object doesn't exist"}
{"timestamp":{"seconds":1713035759,"nanos":160517073},"severity":"TRACE","message":"gcs: Req              0x7: <- CreateObject(\"test.txt\")"}
{"timestamp":{"seconds":1713035759,"nanos":166213653},"severity":"TRACE","message":"gcs: Req              0x7: -> CreateObject(\"test.txt\") (5.693125ms): error in closing writer : googleapi: Error 403: Access denied., forbidden"}
{"timestamp":{"seconds":1713035759,"nanos":166253720},"severity":"ERROR","message":"FlushFile: permission denied, FileInode.Sync: SyncObject: CreateObject: error in closing writer : googleapi: Error 403: Access denied., forbidden"}
{"timestamp":{"seconds":1713035759,"nanos":166293271},"severity":"TRACE","message":"fuse_debug: Op 0x0000000e        connection.go:499] -> Error: \"permission denied\""}
{"timestamp":{"seconds":1713035759,"nanos":166305328},"severity":"ERROR","message":"fuse: *fuseops.FlushFileOp error: permission denied"}
{"timestamp":{"seconds":1713035759,"nanos":166389137},"severity":"TRACE","message":"fuse_debug: Op 0x00000010        connection.go:415] <- ReleaseFileHandle (PID 0)"}
{"timestamp":{"seconds":1713035759,"nanos":166696391},"severity":"TRACE","message":"fuse_debug: Op 0x00000010        connection.go:497] -> OK ()"}

What do I need to do to give my VM write permissions for my bucket?