Hello, I need to recursively copy many videos to our gcloud storage bucket from a local Ubuntu system while maintaining metadata, providing for continuation and validation. I am setting up some equipment to validate the following two options, but I was hoping someone could indicate which one aligns well with our goals.
Thank you for your interest and help!
cross-post: StackOverflow
OPTIONS:
1) After mounting the gcloud storage bucket to a local mount-point and issuing the rsync command (see following) to transfer the file through the mount-point:
a) Should the local mount-point have the sync attribute? I believe that it should not have this attribute since rsync will be performing the synchronization and validation.
b) Will this configuration be fast or require several exchanges between rsync and the cloud?
c) Will metadata be preserved after the rsync has been completed?
d) Will an interrupted upload continue from the partially uploaded data?
sudo rsync --archive --human-readable --verbose --partial --progress --human-readable --itemize-changes --stats src dst
and the following command to validate the transferred file(s).
sudo rsync --recursive --checksum --verbose --human-readable --itemize-changes --stats --dry-run src dst
2) Alternatively, if I used gcloud's cli to perform the transfer directly from the local Ubuntu system to the bucket:
a) Will this configuration be faster than question #1, or will it require several exchanges between rsync and the cloud?
b) Will metadata be preserved after the transfer has been completed?
c) Will an interrupted upload continue from the partially uploaded data?
sudo gcloud storage rsync --recursive --delete-unmatched-destination-objects src dst
and the following command to validate the transferred file(s).
sudo gcloud storage rsync --recursive --dry-run src dst
LinkedIn
Twitter
