Cannot delete from source during transfer job using Google Storage Transfer Service

[ceelor]

Hello,

Here is the context of my issue:

I have a VM with a mounted POSIX file system. WIthin the file system, I've created a directory called "test" (along with some files within) under the user "main" (does not have root access) to be used as the source directory for Google Storage Transfer Service. What I want to do is start a transfer job that transfers from the source to a bucket and then delete the files from the source directory.

After configuring and installing the transfer agent (rootless; using terraform script) under the same account, "main", I proceeded to start the transfer job process and fortunately it is able to transfer successfully and delete files from the source with no issues.

Another user called "alternate" from the same group called "together" that "main" is also apart of has created another directory called "attempt" to be transferred to the bucket as well.

So...
drwxrwxr--.  1    main        together    test
drwxrwxr--.  1    alternate together    attempt

When I start another transfer job to have the "attempt" directory transferred over, it fails. It actually appears that the transfer of the source files to the bucket occurs BUT the agent cannot delete any of the source files 

After making sure they are in the same group and the directory and files of "attempt" have the right permissions (RW) for "main" to delete, it still looks like there's an issue.

Like I mentioned previously, when everything is done against the "main" account (creating directories, files, transfers jobs), everything works smoothly. But when another user tries to move their directories or files into the source directory for the transfer, they cannot be deleted.

Edit: I would also like to note that I configured and am using an IAM service account in GCP with the correct roles because it works in one of the end-to-end test scenarios successfully, noted above.

Can someone please help me shed some light on the issue?

[KyleMari]

Hi @ceelor,

Welcome to Google Cloud Community!

Though both users main and alternate belong to the same group, the issue encountered suggests that there could be something with the way POSIX file system permission works rather than the behavior of the Storage Job Transfer execution.

Based on my understanding of your observations, the alternate account is the one that’s having an issue deleting a file inside attempt folder even if it already has a directory-level permission of drwxrwxr--. If so, this seems odd.

Here are the initial troubleshooting tips to identify the root cause:

• Manually test the account’s permission on a file or other folders: Try running ls -l <path_of_file> command inside the attempt directory to verify what alternate account is permitted to do,  then execute rm command manually inside attempt directory to ensure that alternate can delete any file in there. You could also test things further by creating or deleting files from other directories owned by the same group.
• Verify your configurations: Given that the transfer job involves Terraform, check if the Terraform job can somehow affect permissions or ownership of files in the affected directories that the alternate account is dealing with. Though this is unlikely, see to it that any possible file ownership/permission changes post-transfer isn’t happening, and Terraform’s configuration around POSIX file systems isn’t ending up restricting the alternate account.
• Manually execute the transfer job: For better testing, you may try manually executing the transfer job via gcloud CLI to see if there’s any difference.

If you observe that there’s a potential bug during the test, I’d also suggest creating a new Storage Transfer Service issue. This will help our Google Cloud team to dig deeper into the issue you encountered when performing the transfer job. Note though that a specific timeline for when the issue will be resolved is something I can’t provide.

I hope the above information is helpful.