How do I transfer multiple S3 buckets to one GCS bucket?

[rudra-deb]

I want to transfer the multiple S3 buckets (100+) in my AWS account to one single bucket under GCS. I was thinking of using the GCP Transfer service but it looks like it needs a single bucket name as Source.

Any suggestions how I can transfer all my S3 buckets in one-go/using one job?

[alexmoore · Accepted solution]

Hi,

You are looking in the right place, I would suggest Storage Transfer Service, each transfer job requires a specific source and destination, but you can create multiple transfer jobs, one for each source bucket for example.

If you have a list of source buckets in a text file you could automate the job creation using the gcloud CLI and a simple for loop in a bash script. See: https://cloud.google.com/storage-transfer/docs/create-transfers#gcloud-cli

Keep in mind there is a limit to the number of simultaneous transfer jobs on a project level, but when this is exceeded the jobs are simply queued, see:  https://cloud.google.com/storage-transfer/quotas

Also take a look here for some guidance on performance at scale: https://cloud.google.com/storage-transfer/docs/performance 

Hope that helps,

Alex

[faso]

Give ricedrive a try and see if it can meet your needs.