gsutil cp from AWS to GS - Traffic network

[juancarlos_la]

Hi there,

I need to move files from S3 to GCP, and the destination does not maintain the directory structure (index), just the file copied. I trried the process using Transfer Service but it always replicates the source directory structure in the destination. Therefore, I am trying to use a process in AWS that executes the `gsutil cp` command.

I understand that when a Transfer Service job is created, the transfer is made directly from S3 to GCP. If the `gsutil cp` command is used, there is a data transfer from the origin to the process where the command is executed, and then from there towards the destination?

S3 -> EC2 (gsutil cp) -> GCP

I did the test by transferring a 1GB file, executing the command on my laptop (simulating the process that would run in AWS) using a slow internet connection and the transfer rate was approximately 2MB/s, which makes me think that if there is a data transfer to the process, and then the destination.

I appreciate your comments on this.

[christianpaula]

Hi @juancarlos_la,

Welcome to Google Cloud Community! 

Using gsutil cp within an AWS instance adds an extra data transfer step, slowing down transfers and increasing costs.

 Consider these alternatives for transferring data from S3 to GCS without replicating the directory structure:

• AWS Transfer Service with filtering rules
• Cloud Storage Transfer Service
• AWS CLI with aws s3 sync command Choose the option that best suits your needs based on cost, performance, and security considerations.